diff --git a/docs/roadmap/upgrade-kit.md b/docs/roadmap/upgrade-kit.md new file mode 100644 index 00000000..6549b526 --- /dev/null +++ b/docs/roadmap/upgrade-kit.md @@ -0,0 +1,336 @@ +# ConfigKit And PromptKit Upgrades + +## Purpose + +Adopt ConfigKit v0.4.0 and PromptKit v0.11.0 in one coordinated work set: + +- Consistently call ConfigKit's selected configuration layer an **overlay**, + distinct from PromptKit LLM profiles. +- Adopt configurable transient HTTP retries, provider-neutral failure + classifications, and execution observations from PromptKit. + +Preserve application/framework boundaries and keep the two upgrades as distinct +implementation workstreams. Neither requires a redesign of pipeline execution. + +This is a feature roadmap describing the target state and required changes, +not an implementation sequence. It does not authorize release publication. + +## Current Baseline + +The reviewed baseline is Notarius v0.7.1 (`53d9284`): + +- `go.mod` pins ConfigKit v0.3.0 and PromptKit v0.10.0. +- `internal/core/config/load.go` passes `LoadOptions.Profile` into + `configkit.Options[Document].Profile`. +- `--config-profile` selects `profiles/.yml` beneath the configuration + root for `run`, `config validate`, `config effective`, `config explain`, and + `pipelines list`. +- The maintained selected-layer example is + `examples/profiles/development.yml`; configuration and CLI tests create + equivalent directories and exercise selection and inspection. +- ConfigKit owns composition and inspection metadata. Notarius owns its + version 5 document, defaults, validation, source-relative path resolution, + runtime conversion, and CLI discovery. + +A dependency-only trial against v0.4.0 fails to compile at `load.go:33` because +`configkit.Options.Profile` no longer exists. The review found no direct +Notarius references to `LayerProfile`, `CodeProfile*`, or `LoadSecretsDir`. + +An isolated dependency-only trial against PromptKit v0.11.0 passes the complete +Notarius test suite and `go vet ./...`. That proves baseline compatibility, not +adoption of the new controls or observations. The current adapter does not +configure transient retries or attach an execution observer. It classifies +generation errors broadly and records successful-result usage and structural +repair counts. The outer scheduler holds a permit for the whole completion. + +## Upstream Changes + +### ConfigKit v0.4.0 + +The [ConfigKit v0.4.0 release guide](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.4.0/docs/releases/v0.4.0.md) +defines four breaking naming changes: + +| Surface | Previous | Target | +| --- | --- | --- | +| Selection API | `Options.Profile` | `Options.Overlay` | +| Selected file | `profiles/.yml` | `overlays/.yml` | +| Provenance layer | `LayerProfile` / `profile` | `LayerOverlay` / `overlay` | +| Diagnostic identifiers | `CodeProfile*` / `profile.*` | `CodeOverlay*` / `overlay.*` | + +There are no legacy aliases or directory fallback. Explicit imports from a +directory named `profiles` remain valid ordinary imports. Merge rules, +precedence, lifecycle, source-origin APIs, minimum Go version, and YAML parser +dependency are unchanged. + +The intervening v0.3.1 release makes the opt-in `LoadSecretsDir` API ignore +dot-prefixed entries. Notarius does not use this API; adding secrets-directory +bootstrap is outside this work. + +### PromptKit v0.11.0 + +The [PromptKit v0.11.0 release guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/releases/v0.11.0.md) +introduces additive runtime APIs and one changed default: + +- Built-in generation defaults to two additional transient HTTP attempts for + statuses 429, 500, 502, 503, and 504. Engine and request controls can override + the count, including zero. Injected `LLMClient` implementations are not + automatically replayed; a custom HTTP client on the built-in path still is. +- Backoff is cancellable and honors valid provider `Retry-After` timing. + Generation and HTTP timeouts apply per attempt; the caller context bounds + the whole operation. Transport failures, malformed responses, other HTTP + statuses, and invalid generated content are not transient-retry candidates. +- `RunRequest.Observer` reports lifecycle metadata, attempts, waits, outcomes, + and presence-aware usage, including work before a failed execution. It works + with prepared execution; preparation itself emits no execution events. +- `ErrTransport`, `ErrMalformedResponse`, `ErrRetriesExhausted`, and + `RetryExhaustedError` support typed classification without parsing error text. +- The license changes to LGPL-3.0-or-later. Catalog selections and the minimum + Go version are unchanged from v0.10.0. + +## Target End State + +### Dependency And Ownership + +Pin ConfigKit v0.4.0 and PromptKit v0.11.0 and tidy the dependency graph. +Continue using only ConfigKit's +public root API behind `internal/core/config`. No compatibility parser, +application-owned composition engine, or new configuration abstraction is +needed. PromptKit remains behind `internal/framework/llm`; its types must not +leak into domain modules or pipeline contracts. Retain the existing prepared +execution path, profile choices, and D&D model settings. + +### Consistent Overlay Selection + +- Replace the configuration-selection CLI flag with `--config-overlay` on all + commands that currently accept `--config-profile`. +- Rename application-owned fields, variables, helpers, help text, and test + descriptions that specifically represent this selection to overlay terms. +- Pass the selection through `configkit.Options.Overlay`. +- Select files only from `overlays/.yml`, following ConfigKit's contract. +- Preserve empty and `default` selection as the composed base configuration + without a selected overlay. +- Use a clean cutover: no old CLI alias, directory fallback, automatic file + moves, or custom migration machinery. Normal unknown-flag handling is + sufficient for the retired flag; explain migration in documentation. + +This rename is deliberately narrow. Preserve `llm_profile`, `--llm-profile`, +`promptkit.profile_dir`, `promptkit.profile_file`, `base_profile`, embedded +`dnd-extraction` profiles, and `examples/llm-profiles/`. Other application +concepts named `Profile`, including pipeline/runtime types, must not be renamed +merely because they share the word. + +### Preserved Behavior + +Retain configuration version 5, with the additive retry setting described below. +Preserve existing defaults other than the explicitly adopted transient retry policy, +imports, overlay merge behavior (including sequence replacement), explicit CLI +override precedence, strict decoding, domain validation, and secret redaction. +Do not add new restrictions or relax existing explicit contracts as part of the +rename. + +Relative paths continue to resolve from the file supplying the surviving +declaration, including overlay files. Invocation-relative CLI paths and +main-file-relative built-in defaults retain their current behavior. + +Effective configuration, explanations, and configuration-load diagnostics use +the new upstream overlay provenance and diagnostic vocabulary. Preserve source +file labels and actionable error context; do not translate the new metadata +back into legacy profile terminology. + +Pipeline stage order, semantic acceptance, LLM selection, domain artifact +schemas, and subprocess result receipts remain unchanged. No blanket checkpoint +invalidation is added merely for the ConfigKit rename. Update the existing +PromptKit-version identity used by checkpoint fingerprinting with its dependency +pin, following the current adapter convention; no cache redesign is needed. + +### Transient Retry Policy + +Expose `promptkit.transient_retries` as a nonnegative integer setting with a +Notarius default of **2 additional attempts**. Explicit zero disables transient +retries. Apply the resolved setting to the shared PromptKit engine so modules, +normalizers, validators, and semantic correction calls receive the same policy. +Include the effective value in configuration inspection. Do not impose an +arbitrary application maximum or add per-binding controls in this work set. + +Maintain three separate budgets and owners: + +| Budget | Owner | Repeated work | +| --- | --- | --- | +| Transient retries | PromptKit | The same generation request after eligible HTTP failures | +| Structural repairs | PromptKit | A corrective generation for invalid structured output | +| Producer/validator retries | Notarius | The configured stage or validator operation, with semantic feedback where applicable | + +Do not silently reduce, repurpose, or remove existing stage/validator budgets. +Each structural repair generation receives its own transient budget. For a +single-completion producer, two additional stage attempts, one structural +repair, and two transient retries permit up to 18 HTTP attempts, excluding any +separate validator calls. Document this multiplication and the possibility of +additional cost; retries do not imply exactly-once provider execution. + +Preserve caller cancellation throughout generation and waiting. Do not describe +a profile's generation timeout as a total completion deadline: provider +`Retry-After` can extend waits beyond local backoff, and each HTTP attempt has +its own timeout. A new total-deadline configuration surface is outside this +upgrade; accurately document the current controls and limitations. + +Retain Notarius's completion-scoped scheduler permit during upstream backoff. +PromptKit releases its active backend permit during that wait, but this does +not release the outer Notarius permit. This remains bounded and correct; do +not add an attempt-level scheduler redesign. + +### Provider-Neutral Failure Diagnostics + +Translate the new typed errors into application-owned classifications for +transport failures and malformed provider responses. Represent transient retry +exhaustion separately from its underlying failure/status, retaining the actual +attempt count where available. Preserve existing HTTP-status reporting, +capacity classification, cancellation, deadline handling, and redaction. + +Do not turn a malformed provider envelope into a semantic rejection or confuse +it with a structurally invalid model response. Do not expose PromptKit error +types across the adapter boundary or parse prose to recover classifications. +Existing stage disposition policies continue to decide what happens after a +completion fails; this work does not redesign those policies. + +### Execution Observations + +Attach a request-local observer through the existing prepared-execution path. +Translate selected metadata into Notarius-owned completion/debug summaries: + +- dispatched attempt, transient retry, and structural repair-call counts; +- capacity-wait, generation, retry-delay, and total execution durations; +- terminal outcome, failure classification, HTTP status, and exhaustion state; +- observed token counts with explicit known/complete indicators. + +Retain available summaries on failed completions, including cancellation, rather +than discarding them with the absent `RunResult`. Preparation failures have no +execution observation; do not fabricate one. Keep existing successful-result +usage fields and their meanings unchanged. Observation totals are separate +presence-aware metadata, not a second set of tokens to add to existing usage. +Unknown counts are not known zero; partial totals are not a billing ledger. + +Use lightweight per-request accumulation with no unbounded event history or +asynchronous queue. Callbacks are synchronous: avoid blocking I/O or acquisition +of scheduler resources inside them, and isolate concurrent requests. Persist +the projection through the existing opt-in debug path, including safe terminal +summaries where applicable. Do not serialize upstream event structs directly; +their JSON layout is not a public contract. Allowlist metadata and preserve +existing identifier/content handling and redaction boundaries. + +Successful transient recovery is ordinary success, not a warning. This work +does not add routine console messages, change the warning policy, introduce a +new durable telemetry artifact, or add complete cost accounting. + +## Required Changes + +### Code And Examples + +- Update `go.mod` and `go.sum` and the configuration loader/options boundary. +- Update CLI selection plumbing in `internal/cli/run.go` and + `configkit_commands.go`, including usage, flag parsing, argument classification, + explicit empty-value handling, and error paths. +- Move the maintained example to `examples/overlays/development.yml` and update + example discovery and related fixtures. Inspect relative paths when moving + files; do not assume all occurrences of `profiles` are configuration overlays. +- Update affected configuration and CLI tests, particularly document paths, + operational defaults, safe inspection, command selection, diagnostic source + labels, run contracts, and maintained-example coverage. +- Add the retry setting through the typed configuration document, defaults, + validation, runtime conversion, and CLI client-construction boundary. Update + the complete example to demonstrate it without adding unnecessary settings + to the minimal example. +- Update the PromptKit adapter's engine construction, request observation, + error conversion, and checkpoint version identity. Extend transport-neutral + response/debug contracts only with the application-owned metadata needed. +- Carry observations through the pipeline debug wrapper and safe terminal + reporting without double-counting usage or losing failed-call summaries. + +### Documentation + +Update current documentation with the implementation, keeping canonical +ownership under [Documentation Policy](../policy/documentation.md): + +- `docs/cli.md`: new flag on every affected command and matching help behavior. +- `docs/config.md`: overlay terminology, directory layout, precedence, and the + explicit distinction from LLM profiles; transient retry setting and default. +- `docs/integrations/pkg-configkit.md`: pinned release and supported upstream + naming/inspection contract. +- `docs/internal/configuration.md` and relevant CLI internals: new selection + terminology and unchanged boundaries. +- `docs/consumers/subprocess.md` and any other current callers/examples: updated + invocation syntax. Review operations and README references where applicable. +- `docs/integrations/pkg-promptkit.md`: updated pin, typed errors, observer + support, and retry contract. Refresh pinned upstream links, including its + relocated release-policy document. +- `docs/internal/llm.md` and relevant pipeline/debug internals: observation + projection, failure handling, scheduler interaction, and separate retry + budgets. Remove the obsolete claim that PromptKit has no provider retry loop. +- `docs/operations.md`: latency/cost implications, timeout limits, opt-in debug + evidence, and how to disable transient retries. + +Preserve historical release notes and the accepted text of ADR-0018. Its +ConfigKit ownership decision remains valid; a terminology migration alone does +not require a new ADR or supersession. Current documentation must not portray +the old dependency pin as the current supported version. + +Record PromptKit's LGPL-3.0-or-later change in the integration documentation and +eventual release note. Review its notices and distribution implications within +Notarius's source-only release model. Do not silently relicense Notarius, copy +upstream release policy into this repository, or introduce binary packaging. +Any future binary distribution needs its own license-compliance review. + +## Operator Migration And Release Impact + +This work belongs in the next **minor** Notarius release, not a patch, because +both the documented selected-file layout and CLI syntax change. + +Operators move selected configuration files from `profiles/` to `overlays/` +and replace `--config-profile` with `--config-overlay` in scripts, service +definitions, and subprocess callers. Preserve unrelated PromptKit profile +directories and explicit imports. Review relative paths and explicit imports +that refer to moved files. No YAML body version migration is required. + +The eventual release note must explain this cutover, the newly enabled transient +retries and zero-value opt-out, and the dependency license change. PromptKit's +catalogs are unchanged from v0.10.0; this upgrade adds no further profile-name +migration or change to the D&D model defaults. Release preparation, +tagging, and publication remain a separately requested task under +[Source Releases](../release.md). + +## Acceptance And Verification + +- Notarius builds against ConfigKit v0.4.0 and PromptKit v0.11.0 without + compatibility shims. +- All five affected commands use the new flag; non-default overlay selection, + omitted selection, empty selection, and `default` retain their intended + behavior. The old flag is not accepted. +- Selected overlays load from the new directory. A legacy-only selected file + does not silently load; ordinary explicit imports remain supported. +- Representative tests demonstrate overlay precedence, CLI overrides, + declaration-relative paths, redacted inspection, and source-aware failures. +- Maintained examples validate, including the development overlay. LLM profile + loading and defaults remain unaffected. +- Tests prove the default transient retry policy, explicit zero, configured + counts, and negative-value rejection through the relevant configuration and + adapter boundaries. Use local HTTP test servers for eligible/noneligible + statuses, exhaustion, cancellation, and interaction with structural repair; + an injected LLM fake alone does not exercise built-in HTTP retries. +- Provider-neutral errors preserve underlying status and cancellation identity + without leaking credentials or upstream types. +- Observations are available for success and failure, preserve unknown versus + known-zero usage, avoid aggregate double-counting, and stay isolated under + concurrent completions. Successful retry recovery emits no warning. +- Representative scheduler/repair/retry checks preserve bounded execution and + cancellation; use existing coverage where it already proves the contract. +- Current documentation, help text, examples, and tests agree. Searches for old + vocabulary are reviewed semantically, excluding intentional historical and + LLM-profile references rather than enforcing a blanket text ban. +- Run focused `go test ./internal/core/config ./internal/cli ./internal/framework/llm ./internal/framework/pipeline`, followed by + `go test ./...`, `go vet ./...`, `go build ./...`, `go mod tidy -diff`, and + `git diff --check`, with `GOWORK=off`. Run race-enabled tests for the modified + adapter, scheduler, and observation integration. Validate both maintained configurations + and the complete configuration with `--config-overlay development`. + +Use existing behavioral tests where possible, following +[Testing Policy](../policy/testing.md). No live model calls or production API +credentials are needed to validate this migration.