Plan ConfigKit and PromptKit upgrades

docs/roadmap/upgrade-kit.md

@@ -0,0 +1,336 @@
+# ConfigKit And PromptKit Upgrades
+
+## Purpose
+
+Adopt ConfigKit v0.4.0 and PromptKit v0.11.0 in one coordinated work set:
+
+- Consistently call ConfigKit's selected configuration layer an **overlay**,
+  distinct from PromptKit LLM profiles.
+- Adopt configurable transient HTTP retries, provider-neutral failure
+  classifications, and execution observations from PromptKit.
+
+Preserve application/framework boundaries and keep the two upgrades as distinct
+implementation workstreams. Neither requires a redesign of pipeline execution.
+
+This is a feature roadmap describing the target state and required changes,
+not an implementation sequence. It does not authorize release publication.
+
+## Current Baseline
+
+The reviewed baseline is Notarius v0.7.1 (`53d9284`):
+
+- `go.mod` pins ConfigKit v0.3.0 and PromptKit v0.10.0.
+- `internal/core/config/load.go` passes `LoadOptions.Profile` into
+  `configkit.Options[Document].Profile`.
+- `--config-profile` selects `profiles/<name>.yml` beneath the configuration
+  root for `run`, `config validate`, `config effective`, `config explain`, and
+  `pipelines list`.
+- The maintained selected-layer example is
+  `examples/profiles/development.yml`; configuration and CLI tests create
+  equivalent directories and exercise selection and inspection.
+- ConfigKit owns composition and inspection metadata. Notarius owns its
+  version 5 document, defaults, validation, source-relative path resolution,
+  runtime conversion, and CLI discovery.
+
+A dependency-only trial against v0.4.0 fails to compile at `load.go:33` because
+`configkit.Options.Profile` no longer exists. The review found no direct
+Notarius references to `LayerProfile`, `CodeProfile*`, or `LoadSecretsDir`.
+
+An isolated dependency-only trial against PromptKit v0.11.0 passes the complete
+Notarius test suite and `go vet ./...`. That proves baseline compatibility, not
+adoption of the new controls or observations. The current adapter does not
+configure transient retries or attach an execution observer. It classifies
+generation errors broadly and records successful-result usage and structural
+repair counts. The outer scheduler holds a permit for the whole completion.
+
+## Upstream Changes
+
+### ConfigKit v0.4.0
+
+The [ConfigKit v0.4.0 release guide](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.4.0/docs/releases/v0.4.0.md)
+defines four breaking naming changes:
+
+| Surface | Previous | Target |
+| --- | --- | --- |
+| Selection API | `Options.Profile` | `Options.Overlay` |
+| Selected file | `profiles/<name>.yml` | `overlays/<name>.yml` |
+| Provenance layer | `LayerProfile` / `profile` | `LayerOverlay` / `overlay` |
+| Diagnostic identifiers | `CodeProfile*` / `profile.*` | `CodeOverlay*` / `overlay.*` |
+
+There are no legacy aliases or directory fallback. Explicit imports from a
+directory named `profiles` remain valid ordinary imports. Merge rules,
+precedence, lifecycle, source-origin APIs, minimum Go version, and YAML parser
+dependency are unchanged.
+
+The intervening v0.3.1 release makes the opt-in `LoadSecretsDir` API ignore
+dot-prefixed entries. Notarius does not use this API; adding secrets-directory
+bootstrap is outside this work.
+
+### PromptKit v0.11.0
+
+The [PromptKit v0.11.0 release guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/releases/v0.11.0.md)
+introduces additive runtime APIs and one changed default:
+
+- Built-in generation defaults to two additional transient HTTP attempts for
+  statuses 429, 500, 502, 503, and 504. Engine and request controls can override
+  the count, including zero. Injected `LLMClient` implementations are not
+  automatically replayed; a custom HTTP client on the built-in path still is.
+- Backoff is cancellable and honors valid provider `Retry-After` timing.
+  Generation and HTTP timeouts apply per attempt; the caller context bounds
+  the whole operation. Transport failures, malformed responses, other HTTP
+  statuses, and invalid generated content are not transient-retry candidates.
+- `RunRequest.Observer` reports lifecycle metadata, attempts, waits, outcomes,
+  and presence-aware usage, including work before a failed execution. It works
+  with prepared execution; preparation itself emits no execution events.
+- `ErrTransport`, `ErrMalformedResponse`, `ErrRetriesExhausted`, and
+  `RetryExhaustedError` support typed classification without parsing error text.
+- The license changes to LGPL-3.0-or-later. Catalog selections and the minimum
+  Go version are unchanged from v0.10.0.
+
+## Target End State
+
+### Dependency And Ownership
+
+Pin ConfigKit v0.4.0 and PromptKit v0.11.0 and tidy the dependency graph.
+Continue using only ConfigKit's
+public root API behind `internal/core/config`. No compatibility parser,
+application-owned composition engine, or new configuration abstraction is
+needed. PromptKit remains behind `internal/framework/llm`; its types must not
+leak into domain modules or pipeline contracts. Retain the existing prepared
+execution path, profile choices, and D&D model settings.
+
+### Consistent Overlay Selection
+
+- Replace the configuration-selection CLI flag with `--config-overlay` on all
+  commands that currently accept `--config-profile`.
+- Rename application-owned fields, variables, helpers, help text, and test
+  descriptions that specifically represent this selection to overlay terms.
+- Pass the selection through `configkit.Options.Overlay`.
+- Select files only from `overlays/<name>.yml`, following ConfigKit's contract.
+- Preserve empty and `default` selection as the composed base configuration
+  without a selected overlay.
+- Use a clean cutover: no old CLI alias, directory fallback, automatic file
+  moves, or custom migration machinery. Normal unknown-flag handling is
+  sufficient for the retired flag; explain migration in documentation.
+
+This rename is deliberately narrow. Preserve `llm_profile`, `--llm-profile`,
+`promptkit.profile_dir`, `promptkit.profile_file`, `base_profile`, embedded
+`dnd-extraction` profiles, and `examples/llm-profiles/`. Other application
+concepts named `Profile`, including pipeline/runtime types, must not be renamed
+merely because they share the word.
+
+### Preserved Behavior
+
+Retain configuration version 5, with the additive retry setting described below.
+Preserve existing defaults other than the explicitly adopted transient retry policy,
+imports, overlay merge behavior (including sequence replacement), explicit CLI
+override precedence, strict decoding, domain validation, and secret redaction.
+Do not add new restrictions or relax existing explicit contracts as part of the
+rename.
+
+Relative paths continue to resolve from the file supplying the surviving
+declaration, including overlay files. Invocation-relative CLI paths and
+main-file-relative built-in defaults retain their current behavior.
+
+Effective configuration, explanations, and configuration-load diagnostics use
+the new upstream overlay provenance and diagnostic vocabulary. Preserve source
+file labels and actionable error context; do not translate the new metadata
+back into legacy profile terminology.
+
+Pipeline stage order, semantic acceptance, LLM selection, domain artifact
+schemas, and subprocess result receipts remain unchanged. No blanket checkpoint
+invalidation is added merely for the ConfigKit rename. Update the existing
+PromptKit-version identity used by checkpoint fingerprinting with its dependency
+pin, following the current adapter convention; no cache redesign is needed.
+
+### Transient Retry Policy
+
+Expose `promptkit.transient_retries` as a nonnegative integer setting with a
+Notarius default of **2 additional attempts**. Explicit zero disables transient
+retries. Apply the resolved setting to the shared PromptKit engine so modules,
+normalizers, validators, and semantic correction calls receive the same policy.
+Include the effective value in configuration inspection. Do not impose an
+arbitrary application maximum or add per-binding controls in this work set.
+
+Maintain three separate budgets and owners:
+
+| Budget | Owner | Repeated work |
+| --- | --- | --- |
+| Transient retries | PromptKit | The same generation request after eligible HTTP failures |
+| Structural repairs | PromptKit | A corrective generation for invalid structured output |
+| Producer/validator retries | Notarius | The configured stage or validator operation, with semantic feedback where applicable |
+
+Do not silently reduce, repurpose, or remove existing stage/validator budgets.
+Each structural repair generation receives its own transient budget. For a
+single-completion producer, two additional stage attempts, one structural
+repair, and two transient retries permit up to 18 HTTP attempts, excluding any
+separate validator calls. Document this multiplication and the possibility of
+additional cost; retries do not imply exactly-once provider execution.
+
+Preserve caller cancellation throughout generation and waiting. Do not describe
+a profile's generation timeout as a total completion deadline: provider
+`Retry-After` can extend waits beyond local backoff, and each HTTP attempt has
+its own timeout. A new total-deadline configuration surface is outside this
+upgrade; accurately document the current controls and limitations.
+
+Retain Notarius's completion-scoped scheduler permit during upstream backoff.
+PromptKit releases its active backend permit during that wait, but this does
+not release the outer Notarius permit. This remains bounded and correct; do
+not add an attempt-level scheduler redesign.
+
+### Provider-Neutral Failure Diagnostics
+
+Translate the new typed errors into application-owned classifications for
+transport failures and malformed provider responses. Represent transient retry
+exhaustion separately from its underlying failure/status, retaining the actual
+attempt count where available. Preserve existing HTTP-status reporting,
+capacity classification, cancellation, deadline handling, and redaction.
+
+Do not turn a malformed provider envelope into a semantic rejection or confuse
+it with a structurally invalid model response. Do not expose PromptKit error
+types across the adapter boundary or parse prose to recover classifications.
+Existing stage disposition policies continue to decide what happens after a
+completion fails; this work does not redesign those policies.
+
+### Execution Observations
+
+Attach a request-local observer through the existing prepared-execution path.
+Translate selected metadata into Notarius-owned completion/debug summaries:
+
+- dispatched attempt, transient retry, and structural repair-call counts;
+- capacity-wait, generation, retry-delay, and total execution durations;
+- terminal outcome, failure classification, HTTP status, and exhaustion state;
+- observed token counts with explicit known/complete indicators.
+
+Retain available summaries on failed completions, including cancellation, rather
+than discarding them with the absent `RunResult`. Preparation failures have no
+execution observation; do not fabricate one. Keep existing successful-result
+usage fields and their meanings unchanged. Observation totals are separate
+presence-aware metadata, not a second set of tokens to add to existing usage.
+Unknown counts are not known zero; partial totals are not a billing ledger.
+
+Use lightweight per-request accumulation with no unbounded event history or
+asynchronous queue. Callbacks are synchronous: avoid blocking I/O or acquisition
+of scheduler resources inside them, and isolate concurrent requests. Persist
+the projection through the existing opt-in debug path, including safe terminal
+summaries where applicable. Do not serialize upstream event structs directly;
+their JSON layout is not a public contract. Allowlist metadata and preserve
+existing identifier/content handling and redaction boundaries.
+
+Successful transient recovery is ordinary success, not a warning. This work
+does not add routine console messages, change the warning policy, introduce a
+new durable telemetry artifact, or add complete cost accounting.
+
+## Required Changes
+
+### Code And Examples
+
+- Update `go.mod` and `go.sum` and the configuration loader/options boundary.
+- Update CLI selection plumbing in `internal/cli/run.go` and
+  `configkit_commands.go`, including usage, flag parsing, argument classification,
+  explicit empty-value handling, and error paths.
+- Move the maintained example to `examples/overlays/development.yml` and update
+  example discovery and related fixtures. Inspect relative paths when moving
+  files; do not assume all occurrences of `profiles` are configuration overlays.
+- Update affected configuration and CLI tests, particularly document paths,
+  operational defaults, safe inspection, command selection, diagnostic source
+  labels, run contracts, and maintained-example coverage.
+- Add the retry setting through the typed configuration document, defaults,
+  validation, runtime conversion, and CLI client-construction boundary. Update
+  the complete example to demonstrate it without adding unnecessary settings
+  to the minimal example.
+- Update the PromptKit adapter's engine construction, request observation,
+  error conversion, and checkpoint version identity. Extend transport-neutral
+  response/debug contracts only with the application-owned metadata needed.
+- Carry observations through the pipeline debug wrapper and safe terminal
+  reporting without double-counting usage or losing failed-call summaries.
+
+### Documentation
+
+Update current documentation with the implementation, keeping canonical
+ownership under [Documentation Policy](../policy/documentation.md):
+
+- `docs/cli.md`: new flag on every affected command and matching help behavior.
+- `docs/config.md`: overlay terminology, directory layout, precedence, and the
+  explicit distinction from LLM profiles; transient retry setting and default.
+- `docs/integrations/pkg-configkit.md`: pinned release and supported upstream
+  naming/inspection contract.
+- `docs/internal/configuration.md` and relevant CLI internals: new selection
+  terminology and unchanged boundaries.
+- `docs/consumers/subprocess.md` and any other current callers/examples: updated
+  invocation syntax. Review operations and README references where applicable.
+- `docs/integrations/pkg-promptkit.md`: updated pin, typed errors, observer
+  support, and retry contract. Refresh pinned upstream links, including its
+  relocated release-policy document.
+- `docs/internal/llm.md` and relevant pipeline/debug internals: observation
+  projection, failure handling, scheduler interaction, and separate retry
+  budgets. Remove the obsolete claim that PromptKit has no provider retry loop.
+- `docs/operations.md`: latency/cost implications, timeout limits, opt-in debug
+  evidence, and how to disable transient retries.
+
+Preserve historical release notes and the accepted text of ADR-0018. Its
+ConfigKit ownership decision remains valid; a terminology migration alone does
+not require a new ADR or supersession. Current documentation must not portray
+the old dependency pin as the current supported version.
+
+Record PromptKit's LGPL-3.0-or-later change in the integration documentation and
+eventual release note. Review its notices and distribution implications within
+Notarius's source-only release model. Do not silently relicense Notarius, copy
+upstream release policy into this repository, or introduce binary packaging.
+Any future binary distribution needs its own license-compliance review.
+
+## Operator Migration And Release Impact
+
+This work belongs in the next **minor** Notarius release, not a patch, because
+both the documented selected-file layout and CLI syntax change.
+
+Operators move selected configuration files from `profiles/` to `overlays/`
+and replace `--config-profile` with `--config-overlay` in scripts, service
+definitions, and subprocess callers. Preserve unrelated PromptKit profile
+directories and explicit imports. Review relative paths and explicit imports
+that refer to moved files. No YAML body version migration is required.
+
+The eventual release note must explain this cutover, the newly enabled transient
+retries and zero-value opt-out, and the dependency license change. PromptKit's
+catalogs are unchanged from v0.10.0; this upgrade adds no further profile-name
+migration or change to the D&D model defaults. Release preparation,
+tagging, and publication remain a separately requested task under
+[Source Releases](../release.md).
+
+## Acceptance And Verification
+
+- Notarius builds against ConfigKit v0.4.0 and PromptKit v0.11.0 without
+  compatibility shims.
+- All five affected commands use the new flag; non-default overlay selection,
+  omitted selection, empty selection, and `default` retain their intended
+  behavior. The old flag is not accepted.
+- Selected overlays load from the new directory. A legacy-only selected file
+  does not silently load; ordinary explicit imports remain supported.
+- Representative tests demonstrate overlay precedence, CLI overrides,
+  declaration-relative paths, redacted inspection, and source-aware failures.
+- Maintained examples validate, including the development overlay. LLM profile
+  loading and defaults remain unaffected.
+- Tests prove the default transient retry policy, explicit zero, configured
+  counts, and negative-value rejection through the relevant configuration and
+  adapter boundaries. Use local HTTP test servers for eligible/noneligible
+  statuses, exhaustion, cancellation, and interaction with structural repair;
+  an injected LLM fake alone does not exercise built-in HTTP retries.
+- Provider-neutral errors preserve underlying status and cancellation identity
+  without leaking credentials or upstream types.
+- Observations are available for success and failure, preserve unknown versus
+  known-zero usage, avoid aggregate double-counting, and stay isolated under
+  concurrent completions. Successful retry recovery emits no warning.
+- Representative scheduler/repair/retry checks preserve bounded execution and
+  cancellation; use existing coverage where it already proves the contract.
+- Current documentation, help text, examples, and tests agree. Searches for old
+  vocabulary are reviewed semantically, excluding intentional historical and
+  LLM-profile references rather than enforcing a blanket text ban.
+- Run focused `go test ./internal/core/config ./internal/cli ./internal/framework/llm ./internal/framework/pipeline`, followed by
+  `go test ./...`, `go vet ./...`, `go build ./...`, `go mod tidy -diff`, and
+  `git diff --check`, with `GOWORK=off`. Run race-enabled tests for the modified
+  adapter, scheduler, and observation integration. Validate both maintained configurations
+  and the complete configuration with `--config-overlay development`.
+
+Use existing behavioral tests where possible, following
+[Testing Policy](../policy/testing.md). No live model calls or production API
+credentials are needed to validate this migration.