From e327a5eef75072363be7ea7c84cb47cf9dbe33a7 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 4 Oct 2026 00:34:43 +0000 Subject: [PATCH] Plan the PromptKit configuration upgrade --- docs/roadmap/promptkit.md | 331 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 331 insertions(+) create mode 100644 docs/roadmap/promptkit.md diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md new file mode 100644 index 00000000..27a68e3a --- /dev/null +++ b/docs/roadmap/promptkit.md @@ -0,0 +1,331 @@ +# PromptKit 0.12 Configuration Migration + +## Purpose + +Adopt PromptKit v0.12.0 after the completed ConfigKit upgrade, replacing the +removed configuration APIs and asset formats with PromptKit's ConfigKit-backed +configuration boundary. Make the new composition, inheritance, presence, and +captured-configuration capabilities useful without introducing a second loader +or changing Notarius's domain extraction and pipeline acceptance policies. + +This is a feature roadmap, not an execution sequence. A separate implementation +plan should be prepared after the open decisions below are settled. Release +preparation and publication are separate work. + +## Reviewed Baseline + +Reviewed on 2026-10-04 against Notarius commit +`af62b3b`, with ConfigKit v0.5.0 and PromptKit v0.11.2 selected in `go.mod`. +The remote PromptKit v0.12.0 tag resolves to full source commit +`4b8169da286917fde7875425bb32b799b12c6af3`, matching the sibling checkout. +Primary upstream references are the +[release guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/releases/v0.12.0.md), +[consumer migration guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/consumers/pkg-promptkit.md#migrate-from-the-previous-configuration-api), +and [format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/formats.md). +These were reviewed in the local checkout; the remote tag was verified directly. + +The current integration has these migration-sensitive boundaries: + +- `internal/framework/llm/asset_registry.go` flattens module-owned prompt, + schema, and fallback-profile filesystems and constructs `WithPromptFS`, + `WithSchemaFS`, and `WithFallbackProfileFS` options. +- `promptkit_profiles.go` shares profile-source construction between the + execution client and credential-free explicit-profile inspection. It uses + `Config.ProfileDir`, `WithProfileFile`, and the backend registration option. + The inspection engine supplies `PromptDir: "."` without preparing a prompt. +- `promptkit_client.go` constructs the engine and uses prepared execution, + semantic correction messages, structural repair overrides, observations, + direct sessions, reasoning overrides, and provider-neutral error adaptation. +- `promptkit_profile_fingerprint.go` combines a pinned catalog marker with + reread operator profile files and fallback asset bytes. The optional local + endpoint contributes a separate identity; local concurrency is excluded. +- Embedded prompt definitions throughout `assets/dnd/` and + `assets/generic/normalize/deduplication/` use top-level `id` and `version`. + `assets/dnd/profiles/dnd-extraction.yaml` and the maintained deployment file + `examples/llm-profiles/dnd-extraction.yml` use top-level `id`. +- The public profile contract says a matching operator profile completely + replaces a fallback or catalog profile. It documents `base_profile` + inheritance. These claims do not match v0.12.0's native semantics. + +The current focused LLM, CLI, and module-integration tests pass with +`GOWORK=off`. This establishes the baseline only: no trial dependency upgrade, +live inference, or production model-quality evaluation has been performed. + +## Upstream Changes Relevant To Notarius + +PromptKit v0.12.0 is an incompatible configuration/API release: + +- `Config.Sources` accepts ordered ConfigKit underlays and an optional main + input. Legacy prompt/profile directory fields and configuration-source + options are removed. Schema and injected-runtime options remain available. +- Documents use keyed `backends`, `profiles`, and `prompts` mappings, with a + second version mapping beneath each prompt ID. Embedded `id`, `version`, and + `base_profile` fields are invalid; profile inheritance uses `extends`. +- Files within one source group compose strictly, not by filename precedence. + Later groups overlay earlier groups recursively. Explicit null, zero, + false, and empty values can differ from omission; `{}` does not clear a map. +- Engine construction validates every final effective definition, including + unused definitions, and captures message content and static templates. + Ordinary execution and inspection do not reload them. Schemas, request + artifacts, credentials, and provider operations retain their runtime timing. +- `InspectConfiguration` exposes captured presence, sources, diagnostics, + explanations, and redacted effective output. These reports are not + automatically safe to publish: ordinary prompt/provider content remains data. +- Maintained catalogs move to `/v3`, both at `v3.20260930.0`. + PromptKit and Notarius already share the required ConfigKit v0.5.0 and Go + 1.25.5 baseline; no additional Go-version upgrade is required by this release. + +## Target End State + +### One Native Configuration Boundary + +PromptKit owns the effective backend/profile/prompt configuration and all +upstream composition and inheritance. Notarius selects its sources and adapts +the resulting engine to the existing provider-neutral completion contract. +ConfigKit and PromptKit types remain inside the LLM adapter's explicit +integration boundary, not in domain modules or pipeline contracts. + +Retain Notarius configuration `version: 5`, `promptkit.profile_file`, +`promptkit.profile_dir`, their mutual exclusion and declaring-file-relative +path handling, `transient_retries`, and the optional conventional local backend. +Do not add replacement Notarius configuration fields merely because the +upstream constructor has changed. Document the changed external file format. + +Use one shared source-construction path for execution and profile preflight. +The proposed precedence, subject to the decisions below, is: + +1. PromptKit's maintained catalog layer, supplied by PromptKit itself. +2. Registered application prompt and fallback-profile assets. +3. The explicitly selected operator file or directory. +4. A programmatic `local` backend contribution when Notarius's + `promptkit.local_backend` is configured, so those explicit Notarius values + win over conflicting file declarations of those fields. + +Inputs within each group use native strict composition. Directory discovery is +recursive over YAML files and has no hidden per-file priority. No unselected +directory is scanned. Construct filesystem-backed inputs with usable resource +contexts so referenced message files retain their declaring-source base. +Do not combine host profile files with embedded content under a fabricated +common filesystem root. + +Use `FilesystemInputs` for the existing flattened embedded views, +`DirectoryInputs` or an individual ConfigKit file input for the operator +source, and `BackendSource` with the presence-aware local-backend helper for +programmatic backend configuration. Keep schemas on the existing schema +filesystem boundary. There is no need to deserialize catalog assets, register +catalog backends individually, or recreate PromptKit inheritance. + +The profile-only preflight engine does not need dummy prompt directories or +synthetic prompts. It can construct an engine from catalog, fallback, operator, +and local-backend sources, then call `InspectProfile`. If an operator source +also supplies prompts under the selected source policy, native construction +validation applies to them as well. Preflight does not resolve credential +values, prepare request artifacts, or contact providers. + +### Keyed Assets Without Prompt Behavior Changes + +Every maintained prompt declaration becomes a keyed document of this shape: + +```yaml +prompts: + dnd.spells: + v1: + # Existing definition body, without embedded id/version fields. +``` + +Profiles become keyed entries beneath `profiles`. Preserve literal prompt IDs, +the `v1` version, logical profile IDs, selected response schemas, content paths, +message order, roles, shared fragment bytes, cache-control metadata, and +configured structural repair budgets. Do not conflate this migration with +prompt rewriting, schema changes, or durable artifact version changes. + +The application-owned `dnd-extraction` fallback continues to explicitly select +`openrouter`, `openai/gpt-5.6-luna`, timeout 240, and service tier `flex`, with +reasoning effort unset. Do not replace that choice with an upstream catalog +alias or adopt unrelated model defaults. Migrate the maintained operator +example and relevant inline Go/YAML test fixtures as well as production assets. + +Module ownership, ordered asset manifests, scoped mounting, and selected-asset +hashing remain intact. Check virtual filenames and source-relative +`content_file` resolution after wrapping definitions; do not move business +logic into the root `assets` package. + +### Presence, Inheritance, And Operator Migration + +The recommended policy is native recursive overlay, not a compatibility shim +for complete-definition replacement. Operators can override only the fields +they want. Omitted fields retain lower-layer values, and `extends` uses native +literal sibling-key lookup. Changing a backend or model does not erase +explicit inherited settings; backend-derived defaults are applied to the +final backend after inheritance. + +Document intentional clearing with `null`, including reasoning/service tier, +sampling controls, credential lookup, and provider maps where supported. +Distinguish null from zero/false/blank and from an empty mapping. For example, +switching an operator's `dnd-extraction` overlay to a local model may also need +`service_tier: null` to remove the fallback's explicit `flex` setting. Do not +automatically scrub settings based on model-name heuristics. + +Translate `base_profile` to `extends` in maintained examples and explain that +operators must migrate their external files. Do not retain legacy aliases or +an alternate parser. Preserve exact PromptKit keys; do not introduce a new +identifier normalization pass. Update tests that assumed whole replacement +or selection-time validation deliberately, rather than treating those +intentional upstream changes as regressions. + +Native eager validation applies to final effective values, not independently +to each superseded contribution. Do not add a source-history audit or raw-layer +validation that prevents legitimate overrides from repairing lower layers. + +### Captured Execution And Checkpoint Identity + +One engine represents one captured configuration for its lifetime. Editing +operator definitions or message files requires constructing a new engine; +there is no watcher or automatic hot reload in this scope. Preserve the normal +CLI lifecycle without promising that independently constructed preflight and +execution engines are one transactional filesystem snapshot. + +Compute and retain checkpoint identity from the configuration actually +captured by the execution engine. Do not identify an old captured engine by +files reread after it was constructed. Identity must account for effective +profile/backend semantics and the pinned PromptKit/catalog revisions; if +operator prompt definitions are admitted, it must also account for their +captured prompt content, not just `content_file` path strings. + +Use public captured inspection and prompt-definition equality values where +appropriate; do not reach into PromptKit internals. `InspectPrompt.PromptHash` +includes captured content and is an opaque equality value. Inspection reports +alone do not embed referenced message-file bodies. Preserve application +module/schema fingerprints, actual-used profile provenance, and the existing +exclusion of concurrency-only local settings from semantic checkpoint identity. +A conservative identity across effective profiles is acceptable; per-selected- +profile optimization is not a prerequisite for this upgrade. + +Raw sources, configuration history, prompt bodies, endpoints, and credential +values must not be added to ordinary manifests or checkpoint metadata. The +new inspection API is useful privately for the migration and captured identity; +it does not require a new public CLI report or automatic dump into output or +debug bundles. Hash-only identity is distinct from publishing an inspection +report, and redaction is not a substitute for covering execution semantics. + +### Existing Execution Policies Remain Unchanged + +Preserve prepared execution, the exact latest-response semantic correction +pair, prompt-prefix caching, direct sessions, reasoning overrides, cancellation, +the shared scheduler, local/backend admission, observer accounting, error +classification, and credential redaction. Structural repairs, transient HTTP +retries, outer producer attempts, and validator execution budgets remain +separate. The existing one-repair maintained prompt default remains unchanged. + +The migration does not change extraction/normalization semantics, warning +classification, artifact contracts, subprocess receipts, or reusable-state +eligibility. Construction errors may occur earlier under native eager +validation; adapt them through the existing safe adapter/CLI error boundary, +not by exposing arbitrary upstream configuration content. + +## Required Work + +- Update `go.mod` and `go.sum` to PromptKit v0.12.0 and its selected `/v3` + catalogs. Keep ConfigKit v0.5.0 and unrelated dependencies unless module + resolution requires a change. Remove obsolete `/v2` requirements after tidy. +- Replace removed constructor/source/backend APIs in the LLM adapter and + shared profile-preflight construction; retain schema and injected-client + options. Remove obsolete adapter helpers rather than layering a parallel + configuration system beside them. +- Migrate every registered prompt and fallback profile, the maintained + deployment profile, and affected test fixtures to keyed definitions. +- Adopt the agreed source policy, presence/null behavior, and captured + lifetime; replace live-file checkpoint fingerprinting with captured identity. +- Update behavioral tests at the adapter, CLI preflight, asset registry, and + representative assembled-module boundaries. Reuse real ConfigKit/PromptKit + loading and offline injected clients or local HTTP fixtures. +- Update canonical documentation when implementation lands: + `docs/config.md` owns selected sources, precedence, profile syntax and + migration; `docs/operations.md` owns deployment and reload guidance; + `docs/internal/llm.md` owns construction, capture, and checkpoint mechanics; + `docs/integrations/pkg-promptkit.md` owns the supported upstream boundary and + pinned links. Revise consumer/CLI examples only where actually affected. +- Update `docs/oss/attribution.md` for PromptKit and the two `/v3` catalogs, + checking their actual notices and exact source commits. PromptKit remains + LGPL-3.0-or-later; the catalogs remain MIT at the reviewed revisions. Reuse + unchanged standard texts and preserve the project's own license grant. + +Native layered overrides change an existing documented operator contract. +A concise Notarius ADR is appropriate if that recommendation is accepted, +recording the ownership/compatibility decision rather than duplicating the +upstream manual. Existing historical ADRs need not be rewritten. + +## Verification And Acceptance + +Acceptance is behavioral, not an inventory of private helper names: + +- All registered D&D and generic prompts construct and prepare with migrated + assets, resolving shared/module content in the correct source context. + Request identities, message content/order, schemas, and cache controls retain + their intended behavior. Do not add exact shared-prefix-length detector tests. +- Operator overrides, inheritance, explicit zero/null, duplicate definitions + within a group, local backend precedence, endpoint-only profiles, and invalid + unused final definitions follow the agreed native configuration policy. + Credential-free preflight and execution resolve the same source ordering. +- Engine/file mutation tests demonstrate captured settings and content; + fingerprints stay tied to that engine and change when a new engine captures + different execution-relevant settings or admitted prompt content. +- Structural repair, semantic feedback, sessions/reasoning, capacity failure, + transient errors, safe diagnostics, and observations retain existing + provider-neutral behavior using deterministic boundary tests. +- Maintained minimal, complete, and inherited development pipelines validate + without credentials or provider calls. External legacy profile files receive + clear migration guidance, not silent reinterpretation. + +Use focused LLM, CLI, and module-integration tests during implementation, then +`GOWORK=off go test ./...`, `GOWORK=off go vet ./...`, a temporary native build, +and `GOWORK=off go mod tidy -diff` for this shared-boundary migration. Verify +selected module versions, attribution links, maintained examples, and stale +API/format claims. These are migration checks, not new permanent release gates. +Live provider testing is optional follow-up evidence, not required acceptance. + +This work warrants a minor Notarius release because the accepted external +profile format and override behavior change. Keep Notarius configuration at +version 5; select the actual release number during separate release preparation. + +## Open Decisions + +### 1. Native Overlay Or Preserve Complete Profile Replacement? + +**Recommendation:** adopt PromptKit's recursive field overlay and native +`extends`/null semantics. It removes bespoke configuration machinery, makes +small deployment overrides useful, and keeps Notarius aligned with upstream. +The tradeoff is explicit operator migration: omitted fields no longer erase +fallback settings, so local/model switches may require null clears. + +**Alternative:** preserve whole-profile replacement by explicitly clearing +lower definitions before applying operator definitions. This protects the old +replacement expectation, but requires Notarius to inspect/transform selected +sources and account for inheritance and origins. It undermines the simple +native handoff and is harder to maintain. Merely placing files in a later +underlay does not implement whole replacement. + +### 2. Scope Of Operator PromptKit Documents + +**Recommendation:** allow native combined `profiles`, `backends`, and `prompts` +documents through the existing selected file/directory settings, and enable +explicit ConfigKit imports on those operator inputs. Keep the familiar fields +and describe them as PromptKit configuration sources whose principal workflow +is profile selection. Do not invent a Notarius section whitelist or custom +merge language. Custom backends and deployment prompt edits become useful +native capabilities; source-relative content capture and checkpoint identity +must cover them. Module response decoding and artifact validation still define +what a supplied prompt can successfully produce. + +Directory selection discovers all YAML beneath its root. Imported fragments +that would also be discovered must not accidentally become duplicate inputs; +document a file-root import layout or keep fragments outside the discovered +directory, using upstream resource/import rules. Do not promise filename +override order or unrestricted host resource access beyond upstream behavior. + +**Alternative:** retain profile-only operator inputs and leave imports disabled +for this upgrade, introducing broader source selection later. This narrows +the initial migration and leaves module prompts exclusively embedded, but +requires an explicit locally chosen restriction on the unified upstream format +and postpones custom backend/configuration composition. It should be a +deliberate product decision, not an accidental consequence of old helper names.