From 71cf084e8bec79cb44da7cfb0c0f6e2f45fd8e30 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 4 Oct 2026 11:11:40 +0000 Subject: [PATCH] Adopt native PromptKit configuration capabilities in the roadmap --- docs/roadmap/promptkit.md | 134 +++++++++++++++++++++----------------- 1 file changed, 74 insertions(+), 60 deletions(-) diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 27a68e3a..07f1293e 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -8,8 +8,9 @@ configuration boundary. Make the new composition, inheritance, presence, and captured-configuration capabilities useful without introducing a second loader or changing Notarius's domain extraction and pipeline acceptance policies. -This is a feature roadmap, not an execution sequence. A separate implementation -plan should be prepared after the open decisions below are settled. Release +This is a feature roadmap, not an execution sequence. The operator-source and +native-overlay decisions were accepted on 2026-10-04 and are incorporated below. +A separate implementation plan will define the execution sequence. Release preparation and publication are separate work. ## Reviewed Baseline @@ -92,7 +93,7 @@ Do not add replacement Notarius configuration fields merely because the upstream constructor has changed. Document the changed external file format. Use one shared source-construction path for execution and profile preflight. -The proposed precedence, subject to the decisions below, is: +The source precedence is: 1. PromptKit's maintained catalog layer, supplied by PromptKit itself. 2. Registered application prompt and fallback-profile assets. @@ -101,6 +102,14 @@ The proposed precedence, subject to the decisions below, is: `promptkit.local_backend` is configured, so those explicit Notarius values win over conflicting file declarations of those fields. +The existing selected file/directory accepts native combined `profiles`, +`backends`, and `prompts` documents. Enable explicit ConfigKit imports on these +operator inputs. Describe the familiar `profile_file` and `profile_dir` fields +as PromptKit configuration sources whose principal workflow is profile +selection; do not add a section whitelist or a custom merge language. +Domain modules remain responsible for interpreting and validating model output, +even when an operator supplies a prompt overlay. + Inputs within each group use native strict composition. Directory discovery is recursive over YAML files and has no hidden per-file priority. No unselected directory is scanned. Construct filesystem-backed inputs with usable resource @@ -108,6 +117,11 @@ contexts so referenced message files retain their declaring-source base. Do not combine host profile files with embedded content under a fabricated common filesystem root. +Document import layouts that do not accidentally select the same definitions +through both directory discovery and imports: a file-root import layout is +often the simplest choice. Use native import/resource rules rather than +inventing deduplication, filename override order, or broader host access. + Use `FilesystemInputs` for the existing flattened embedded views, `DirectoryInputs` or an individual ConfigKit file input for the operator source, and `BackendSource` with the presence-aware local-backend helper for @@ -115,12 +129,14 @@ programmatic backend configuration. Keep schemas on the existing schema filesystem boundary. There is no need to deserialize catalog assets, register catalog backends individually, or recreate PromptKit inheritance. -The profile-only preflight engine does not need dummy prompt directories or -synthetic prompts. It can construct an engine from catalog, fallback, operator, -and local-backend sources, then call `InspectProfile`. If an operator source -also supplies prompts under the selected source policy, native construction -validation applies to them as well. Preflight does not resolve credential -values, prepare request artifacts, or contact providers. +Profile preflight does not need dummy prompt directories or synthetic prompts. +Both construction paths include the same catalog, application prompt/fallback, +operator, and local-backend layers. This matters for partial operator prompt +overlays: inspecting a layer without its application base would validate a +different effective definition. Call `InspectProfile` on the constructed engine +without preparing a synthetic request. Construction still captures and +validates the effective prompt definitions; preflight does not resolve +credential values, prepare request artifacts, or contact providers. ### Keyed Assets Without Prompt Behavior Changes @@ -152,7 +168,7 @@ logic into the root `assets` package. ### Presence, Inheritance, And Operator Migration -The recommended policy is native recursive overlay, not a compatibility shim +The adopted policy is native recursive overlay, not a compatibility shim for complete-definition replacement. Operators can override only the fields they want. Omitted fields retain lower-layer values, and `extends` uses native literal sibling-key lookup. Changing a backend or model does not erase @@ -177,6 +193,42 @@ Native eager validation applies to final effective values, not independently to each superseded contribution. Do not add a source-history audit or raw-layer validation that prevents legitimate overrides from repairing lower layers. +### Native Operator Capabilities + +Support the capabilities of the unified format directly, without new Notarius +configuration knobs or model-specific adaptation: + +- Custom or multiple backend definitions, including endpoint, credential + environment name, provider parameters, concurrency, and queue capacity. + Operators may also overlay maintained backend settings, for example changing + OpenRouter's admission limit. Backend IDs are configuration keys, not a + closed registration list. Notarius's global scheduler still applies. +- Profile inheritance, partial overrides, and recursive provider-parameter + composition. Explicit null suppression and zero/false values follow native + field rules. Native sibling inheritance is not limited by a Notarius + profiles-only whitelist; prompt/backend mappings follow upstream rules too. +- File-configured `api_key_required`, with PromptKit's exact credential + precedence and runtime checks. In particular, true suppresses a backend's + implicit environment source; an explicitly configured profile environment + source remains usable. This is an operator choice, not a new mandatory + credential policy or a reason to require credentials during preflight. +- Deployment prompt overlays and declaring-source-relative message files, + including imported content. Prompt version selection remains caller-owned; + this upgrade does not infer a new module version from file order or add + automatic latest-version selection. + +The existing `local_backend` object remains a convenient highest-priority +contribution for that conventional backend. Other backend settings remain +available through the unified operator source; do not copy every upstream field +into a parallel Notarius backend schema. + +Keep the two maintained top-level configuration examples. Use the complete +example's existing PromptKit deployment file and small canonical documentation +snippets to demonstrate native inheritance/partial overrides, custom backend +configuration, imports, null clearing, and prompt editing without adding a +large unrelated example suite. Not every capability needs its own runnable +pipeline. + ### Captured Execution And Checkpoint Identity One engine represents one captured configuration for its lifetime. Editing @@ -188,8 +240,8 @@ execution engines are one transactional filesystem snapshot. Compute and retain checkpoint identity from the configuration actually captured by the execution engine. Do not identify an old captured engine by files reread after it was constructed. Identity must account for effective -profile/backend semantics and the pinned PromptKit/catalog revisions; if -operator prompt definitions are admitted, it must also account for their +profile/backend semantics and the pinned PromptKit/catalog revisions. It must +also account for operator prompt definitions and their captured prompt content, not just `content_file` path strings. Use public captured inspection and prompt-definition equality values where @@ -234,7 +286,8 @@ not by exposing arbitrary upstream configuration content. configuration system beside them. - Migrate every registered prompt and fallback profile, the maintained deployment profile, and affected test fixtures to keyed definitions. -- Adopt the agreed source policy, presence/null behavior, and captured +- Adopt the unified source policy, explicit operator imports, presence/null + behavior, native operator capabilities, and captured lifetime; replace live-file checkpoint fingerprinting with captured identity. - Update behavioral tests at the adapter, CLI preflight, asset registry, and representative assembled-module boundaries. Reuse real ConfigKit/PromptKit @@ -251,8 +304,8 @@ not by exposing arbitrary upstream configuration content. unchanged standard texts and preserve the project's own license grant. Native layered overrides change an existing documented operator contract. -A concise Notarius ADR is appropriate if that recommendation is accepted, -recording the ownership/compatibility decision rather than duplicating the +Add a concise Notarius ADR recording the accepted native composition and +unified-source ownership/compatibility decision rather than duplicating the upstream manual. Existing historical ADRs need not be rewritten. ## Verification And Acceptance @@ -265,11 +318,14 @@ Acceptance is behavioral, not an inventory of private helper names: their intended behavior. Do not add exact shared-prefix-length detector tests. - Operator overrides, inheritance, explicit zero/null, duplicate definitions within a group, local backend precedence, endpoint-only profiles, and invalid - unused final definitions follow the agreed native configuration policy. - Credential-free preflight and execution resolve the same source ordering. + unused final definitions follow the native configuration policy. Cover + representative imported combined definitions, custom/backend catalog + overlays, partial prompt overlays with their embedded base, and optional + versus explicitly required credentials. Credential-free preflight and + execution resolve the same source ordering and application definitions. - Engine/file mutation tests demonstrate captured settings and content; fingerprints stay tied to that engine and change when a new engine captures - different execution-relevant settings or admitted prompt content. + different execution-relevant settings or operator prompt content. - Structural repair, semantic feedback, sessions/reasoning, capacity failure, transient errors, safe diagnostics, and observations retain existing provider-neutral behavior using deterministic boundary tests. @@ -287,45 +343,3 @@ Live provider testing is optional follow-up evidence, not required acceptance. This work warrants a minor Notarius release because the accepted external profile format and override behavior change. Keep Notarius configuration at version 5; select the actual release number during separate release preparation. - -## Open Decisions - -### 1. Native Overlay Or Preserve Complete Profile Replacement? - -**Recommendation:** adopt PromptKit's recursive field overlay and native -`extends`/null semantics. It removes bespoke configuration machinery, makes -small deployment overrides useful, and keeps Notarius aligned with upstream. -The tradeoff is explicit operator migration: omitted fields no longer erase -fallback settings, so local/model switches may require null clears. - -**Alternative:** preserve whole-profile replacement by explicitly clearing -lower definitions before applying operator definitions. This protects the old -replacement expectation, but requires Notarius to inspect/transform selected -sources and account for inheritance and origins. It undermines the simple -native handoff and is harder to maintain. Merely placing files in a later -underlay does not implement whole replacement. - -### 2. Scope Of Operator PromptKit Documents - -**Recommendation:** allow native combined `profiles`, `backends`, and `prompts` -documents through the existing selected file/directory settings, and enable -explicit ConfigKit imports on those operator inputs. Keep the familiar fields -and describe them as PromptKit configuration sources whose principal workflow -is profile selection. Do not invent a Notarius section whitelist or custom -merge language. Custom backends and deployment prompt edits become useful -native capabilities; source-relative content capture and checkpoint identity -must cover them. Module response decoding and artifact validation still define -what a supplied prompt can successfully produce. - -Directory selection discovers all YAML beneath its root. Imported fragments -that would also be discovered must not accidentally become duplicate inputs; -document a file-root import layout or keep fragments outside the discovered -directory, using upstream resource/import rules. Do not promise filename -override order or unrestricted host resource access beyond upstream behavior. - -**Alternative:** retain profile-only operator inputs and leave imports disabled -for this upgrade, introducing broader source selection later. This narrows -the initial migration and leaves module prompts exclusively embedded, but -requires an explicit locally chosen restriction on the unified upstream format -and postpones custom backend/configuration composition. It should be a -deliberate product decision, not an accidental consequence of old helper names.