Plan the PromptKit configuration upgrade

2026-10-04 00:34:43 +00:00
parent af62b3bacc
commit e327a5eef7
1 changed files with 331 additions and 0 deletions
--- a/docs/roadmap/promptkit.md
+++ b/docs/roadmap/promptkit.md
@@ -0,0 +1,331 @@
+# PromptKit 0.12 Configuration Migration
+
+## Purpose
+
+Adopt PromptKit v0.12.0 after the completed ConfigKit upgrade, replacing the
+removed configuration APIs and asset formats with PromptKit's ConfigKit-backed
+configuration boundary. Make the new composition, inheritance, presence, and
+captured-configuration capabilities useful without introducing a second loader
+or changing Notarius's domain extraction and pipeline acceptance policies.
+
+This is a feature roadmap, not an execution sequence. A separate implementation
+plan should be prepared after the open decisions below are settled. Release
+preparation and publication are separate work.
+
+## Reviewed Baseline
+
+Reviewed on 2026-10-04 against Notarius commit
+`af62b3b`, with ConfigKit v0.5.0 and PromptKit v0.11.2 selected in `go.mod`.
+The remote PromptKit v0.12.0 tag resolves to full source commit
+`4b8169da286917fde7875425bb32b799b12c6af3`, matching the sibling checkout.
+Primary upstream references are the
+[release guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/releases/v0.12.0.md),
+[consumer migration guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/consumers/pkg-promptkit.md#migrate-from-the-previous-configuration-api),
+and [format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/formats.md).
+These were reviewed in the local checkout; the remote tag was verified directly.
+
+The current integration has these migration-sensitive boundaries:
+
+- `internal/framework/llm/asset_registry.go` flattens module-owned prompt,
+  schema, and fallback-profile filesystems and constructs `WithPromptFS`,
+  `WithSchemaFS`, and `WithFallbackProfileFS` options.
+- `promptkit_profiles.go` shares profile-source construction between the
+  execution client and credential-free explicit-profile inspection. It uses
+  `Config.ProfileDir`, `WithProfileFile`, and the backend registration option.
+  The inspection engine supplies `PromptDir: "."` without preparing a prompt.
+- `promptkit_client.go` constructs the engine and uses prepared execution,
+  semantic correction messages, structural repair overrides, observations,
+  direct sessions, reasoning overrides, and provider-neutral error adaptation.
+- `promptkit_profile_fingerprint.go` combines a pinned catalog marker with
+  reread operator profile files and fallback asset bytes. The optional local
+  endpoint contributes a separate identity; local concurrency is excluded.
+- Embedded prompt definitions throughout `assets/dnd/` and
+  `assets/generic/normalize/deduplication/` use top-level `id` and `version`.
+  `assets/dnd/profiles/dnd-extraction.yaml` and the maintained deployment file
+  `examples/llm-profiles/dnd-extraction.yml` use top-level `id`.
+- The public profile contract says a matching operator profile completely
+  replaces a fallback or catalog profile. It documents `base_profile`
+  inheritance. These claims do not match v0.12.0's native semantics.
+
+The current focused LLM, CLI, and module-integration tests pass with
+`GOWORK=off`. This establishes the baseline only: no trial dependency upgrade,
+live inference, or production model-quality evaluation has been performed.
+
+## Upstream Changes Relevant To Notarius
+
+PromptKit v0.12.0 is an incompatible configuration/API release:
+
+- `Config.Sources` accepts ordered ConfigKit underlays and an optional main
+  input. Legacy prompt/profile directory fields and configuration-source
+  options are removed. Schema and injected-runtime options remain available.
+- Documents use keyed `backends`, `profiles`, and `prompts` mappings, with a
+  second version mapping beneath each prompt ID. Embedded `id`, `version`, and
+  `base_profile` fields are invalid; profile inheritance uses `extends`.
+- Files within one source group compose strictly, not by filename precedence.
+  Later groups overlay earlier groups recursively. Explicit null, zero,
+  false, and empty values can differ from omission; `{}` does not clear a map.
+- Engine construction validates every final effective definition, including
+  unused definitions, and captures message content and static templates.
+  Ordinary execution and inspection do not reload them. Schemas, request
+  artifacts, credentials, and provider operations retain their runtime timing.
+- `InspectConfiguration` exposes captured presence, sources, diagnostics,
+  explanations, and redacted effective output. These reports are not
+  automatically safe to publish: ordinary prompt/provider content remains data.
+- Maintained catalogs move to `/v3`, both at `v3.20260930.0`.
+  PromptKit and Notarius already share the required ConfigKit v0.5.0 and Go
+  1.25.5 baseline; no additional Go-version upgrade is required by this release.
+
+## Target End State
+
+### One Native Configuration Boundary
+
+PromptKit owns the effective backend/profile/prompt configuration and all
+upstream composition and inheritance. Notarius selects its sources and adapts
+the resulting engine to the existing provider-neutral completion contract.
+ConfigKit and PromptKit types remain inside the LLM adapter's explicit
+integration boundary, not in domain modules or pipeline contracts.
+
+Retain Notarius configuration `version: 5`, `promptkit.profile_file`,
+`promptkit.profile_dir`, their mutual exclusion and declaring-file-relative
+path handling, `transient_retries`, and the optional conventional local backend.
+Do not add replacement Notarius configuration fields merely because the
+upstream constructor has changed. Document the changed external file format.
+
+Use one shared source-construction path for execution and profile preflight.
+The proposed precedence, subject to the decisions below, is:
+
+1. PromptKit's maintained catalog layer, supplied by PromptKit itself.
+2. Registered application prompt and fallback-profile assets.
+3. The explicitly selected operator file or directory.
+4. A programmatic `local` backend contribution when Notarius's
+   `promptkit.local_backend` is configured, so those explicit Notarius values
+   win over conflicting file declarations of those fields.
+
+Inputs within each group use native strict composition. Directory discovery is
+recursive over YAML files and has no hidden per-file priority. No unselected
+directory is scanned. Construct filesystem-backed inputs with usable resource
+contexts so referenced message files retain their declaring-source base.
+Do not combine host profile files with embedded content under a fabricated
+common filesystem root.
+
+Use `FilesystemInputs` for the existing flattened embedded views,
+`DirectoryInputs` or an individual ConfigKit file input for the operator
+source, and `BackendSource` with the presence-aware local-backend helper for
+programmatic backend configuration. Keep schemas on the existing schema
+filesystem boundary. There is no need to deserialize catalog assets, register
+catalog backends individually, or recreate PromptKit inheritance.
+
+The profile-only preflight engine does not need dummy prompt directories or
+synthetic prompts. It can construct an engine from catalog, fallback, operator,
+and local-backend sources, then call `InspectProfile`. If an operator source
+also supplies prompts under the selected source policy, native construction
+validation applies to them as well. Preflight does not resolve credential
+values, prepare request artifacts, or contact providers.
+
+### Keyed Assets Without Prompt Behavior Changes
+
+Every maintained prompt declaration becomes a keyed document of this shape:
+
+```yaml
+prompts:
+  dnd.spells:
+    v1:
+      # Existing definition body, without embedded id/version fields.
+```
+
+Profiles become keyed entries beneath `profiles`. Preserve literal prompt IDs,
+the `v1` version, logical profile IDs, selected response schemas, content paths,
+message order, roles, shared fragment bytes, cache-control metadata, and
+configured structural repair budgets. Do not conflate this migration with
+prompt rewriting, schema changes, or durable artifact version changes.
+
+The application-owned `dnd-extraction` fallback continues to explicitly select
+`openrouter`, `openai/gpt-5.6-luna`, timeout 240, and service tier `flex`, with
+reasoning effort unset. Do not replace that choice with an upstream catalog
+alias or adopt unrelated model defaults. Migrate the maintained operator
+example and relevant inline Go/YAML test fixtures as well as production assets.
+
+Module ownership, ordered asset manifests, scoped mounting, and selected-asset
+hashing remain intact. Check virtual filenames and source-relative
+`content_file` resolution after wrapping definitions; do not move business
+logic into the root `assets` package.
+
+### Presence, Inheritance, And Operator Migration
+
+The recommended policy is native recursive overlay, not a compatibility shim
+for complete-definition replacement. Operators can override only the fields
+they want. Omitted fields retain lower-layer values, and `extends` uses native
+literal sibling-key lookup. Changing a backend or model does not erase
+explicit inherited settings; backend-derived defaults are applied to the
+final backend after inheritance.
+
+Document intentional clearing with `null`, including reasoning/service tier,
+sampling controls, credential lookup, and provider maps where supported.
+Distinguish null from zero/false/blank and from an empty mapping. For example,
+switching an operator's `dnd-extraction` overlay to a local model may also need
+`service_tier: null` to remove the fallback's explicit `flex` setting. Do not
+automatically scrub settings based on model-name heuristics.
+
+Translate `base_profile` to `extends` in maintained examples and explain that
+operators must migrate their external files. Do not retain legacy aliases or
+an alternate parser. Preserve exact PromptKit keys; do not introduce a new
+identifier normalization pass. Update tests that assumed whole replacement
+or selection-time validation deliberately, rather than treating those
+intentional upstream changes as regressions.
+
+Native eager validation applies to final effective values, not independently
+to each superseded contribution. Do not add a source-history audit or raw-layer
+validation that prevents legitimate overrides from repairing lower layers.
+
+### Captured Execution And Checkpoint Identity
+
+One engine represents one captured configuration for its lifetime. Editing
+operator definitions or message files requires constructing a new engine;
+there is no watcher or automatic hot reload in this scope. Preserve the normal
+CLI lifecycle without promising that independently constructed preflight and
+execution engines are one transactional filesystem snapshot.
+
+Compute and retain checkpoint identity from the configuration actually
+captured by the execution engine. Do not identify an old captured engine by
+files reread after it was constructed. Identity must account for effective
+profile/backend semantics and the pinned PromptKit/catalog revisions; if
+operator prompt definitions are admitted, it must also account for their
+captured prompt content, not just `content_file` path strings.
+
+Use public captured inspection and prompt-definition equality values where
+appropriate; do not reach into PromptKit internals. `InspectPrompt.PromptHash`
+includes captured content and is an opaque equality value. Inspection reports
+alone do not embed referenced message-file bodies. Preserve application
+module/schema fingerprints, actual-used profile provenance, and the existing
+exclusion of concurrency-only local settings from semantic checkpoint identity.
+A conservative identity across effective profiles is acceptable; per-selected-
+profile optimization is not a prerequisite for this upgrade.
+
+Raw sources, configuration history, prompt bodies, endpoints, and credential
+values must not be added to ordinary manifests or checkpoint metadata. The
+new inspection API is useful privately for the migration and captured identity;
+it does not require a new public CLI report or automatic dump into output or
+debug bundles. Hash-only identity is distinct from publishing an inspection
+report, and redaction is not a substitute for covering execution semantics.
+
+### Existing Execution Policies Remain Unchanged
+
+Preserve prepared execution, the exact latest-response semantic correction
+pair, prompt-prefix caching, direct sessions, reasoning overrides, cancellation,
+the shared scheduler, local/backend admission, observer accounting, error
+classification, and credential redaction. Structural repairs, transient HTTP
+retries, outer producer attempts, and validator execution budgets remain
+separate. The existing one-repair maintained prompt default remains unchanged.
+
+The migration does not change extraction/normalization semantics, warning
+classification, artifact contracts, subprocess receipts, or reusable-state
+eligibility. Construction errors may occur earlier under native eager
+validation; adapt them through the existing safe adapter/CLI error boundary,
+not by exposing arbitrary upstream configuration content.
+
+## Required Work
+
+- Update `go.mod` and `go.sum` to PromptKit v0.12.0 and its selected `/v3`
+  catalogs. Keep ConfigKit v0.5.0 and unrelated dependencies unless module
+  resolution requires a change. Remove obsolete `/v2` requirements after tidy.
+- Replace removed constructor/source/backend APIs in the LLM adapter and
+  shared profile-preflight construction; retain schema and injected-client
+  options. Remove obsolete adapter helpers rather than layering a parallel
+  configuration system beside them.
+- Migrate every registered prompt and fallback profile, the maintained
+  deployment profile, and affected test fixtures to keyed definitions.
+- Adopt the agreed source policy, presence/null behavior, and captured
+  lifetime; replace live-file checkpoint fingerprinting with captured identity.
+- Update behavioral tests at the adapter, CLI preflight, asset registry, and
+  representative assembled-module boundaries. Reuse real ConfigKit/PromptKit
+  loading and offline injected clients or local HTTP fixtures.
+- Update canonical documentation when implementation lands:
+  `docs/config.md` owns selected sources, precedence, profile syntax and
+  migration; `docs/operations.md` owns deployment and reload guidance;
+  `docs/internal/llm.md` owns construction, capture, and checkpoint mechanics;
+  `docs/integrations/pkg-promptkit.md` owns the supported upstream boundary and
+  pinned links. Revise consumer/CLI examples only where actually affected.
+- Update `docs/oss/attribution.md` for PromptKit and the two `/v3` catalogs,
+  checking their actual notices and exact source commits. PromptKit remains
+  LGPL-3.0-or-later; the catalogs remain MIT at the reviewed revisions. Reuse
+  unchanged standard texts and preserve the project's own license grant.
+
+Native layered overrides change an existing documented operator contract.
+A concise Notarius ADR is appropriate if that recommendation is accepted,
+recording the ownership/compatibility decision rather than duplicating the
+upstream manual. Existing historical ADRs need not be rewritten.
+
+## Verification And Acceptance
+
+Acceptance is behavioral, not an inventory of private helper names:
+
+- All registered D&D and generic prompts construct and prepare with migrated
+  assets, resolving shared/module content in the correct source context.
+  Request identities, message content/order, schemas, and cache controls retain
+  their intended behavior. Do not add exact shared-prefix-length detector tests.
+- Operator overrides, inheritance, explicit zero/null, duplicate definitions
+  within a group, local backend precedence, endpoint-only profiles, and invalid
+  unused final definitions follow the agreed native configuration policy.
+  Credential-free preflight and execution resolve the same source ordering.
+- Engine/file mutation tests demonstrate captured settings and content;
+  fingerprints stay tied to that engine and change when a new engine captures
+  different execution-relevant settings or admitted prompt content.
+- Structural repair, semantic feedback, sessions/reasoning, capacity failure,
+  transient errors, safe diagnostics, and observations retain existing
+  provider-neutral behavior using deterministic boundary tests.
+- Maintained minimal, complete, and inherited development pipelines validate
+  without credentials or provider calls. External legacy profile files receive
+  clear migration guidance, not silent reinterpretation.
+
+Use focused LLM, CLI, and module-integration tests during implementation, then
+`GOWORK=off go test ./...`, `GOWORK=off go vet ./...`, a temporary native build,
+and `GOWORK=off go mod tidy -diff` for this shared-boundary migration. Verify
+selected module versions, attribution links, maintained examples, and stale
+API/format claims. These are migration checks, not new permanent release gates.
+Live provider testing is optional follow-up evidence, not required acceptance.
+
+This work warrants a minor Notarius release because the accepted external
+profile format and override behavior change. Keep Notarius configuration at
+version 5; select the actual release number during separate release preparation.
+
+## Open Decisions
+
+### 1. Native Overlay Or Preserve Complete Profile Replacement?
+
+**Recommendation:** adopt PromptKit's recursive field overlay and native
+`extends`/null semantics. It removes bespoke configuration machinery, makes
+small deployment overrides useful, and keeps Notarius aligned with upstream.
+The tradeoff is explicit operator migration: omitted fields no longer erase
+fallback settings, so local/model switches may require null clears.
+
+**Alternative:** preserve whole-profile replacement by explicitly clearing
+lower definitions before applying operator definitions. This protects the old
+replacement expectation, but requires Notarius to inspect/transform selected
+sources and account for inheritance and origins. It undermines the simple
+native handoff and is harder to maintain. Merely placing files in a later
+underlay does not implement whole replacement.
+
+### 2. Scope Of Operator PromptKit Documents
+
+**Recommendation:** allow native combined `profiles`, `backends`, and `prompts`
+documents through the existing selected file/directory settings, and enable
+explicit ConfigKit imports on those operator inputs. Keep the familiar fields
+and describe them as PromptKit configuration sources whose principal workflow
+is profile selection. Do not invent a Notarius section whitelist or custom
+merge language. Custom backends and deployment prompt edits become useful
+native capabilities; source-relative content capture and checkpoint identity
+must cover them. Module response decoding and artifact validation still define
+what a supplied prompt can successfully produce.
+
+Directory selection discovers all YAML beneath its root. Imported fragments
+that would also be discovered must not accidentally become duplicate inputs;
+document a file-root import layout or keep fragments outside the discovered
+directory, using upstream resource/import rules. Do not promise filename
+override order or unrestricted host resource access beyond upstream behavior.
+
+**Alternative:** retain profile-only operator inputs and leave imports disabled
+for this upgrade, introducing broader source selection later. This narrows
+the initial migration and leaves module prompts exclusively embedded, but
+requires an explicit locally chosen restriction on the unified upstream format
+and postpones custom backend/configuration composition. It should be a
+deliberate product decision, not an accidental consequence of old helper names.