notarius/docs/roadmap/upgrade-kit.md

ConfigKit And PromptKit Upgrades

Purpose

Adopt ConfigKit v0.4.0 and PromptKit v0.11.0 in one coordinated work set:

• Consistently call ConfigKit's selected configuration layer an overlay, distinct from PromptKit LLM profiles.
• Adopt configurable transient HTTP retries, provider-neutral failure classifications, and execution observations from PromptKit.

Preserve application/framework boundaries and keep the two upgrades as distinct implementation workstreams. Neither requires a redesign of pipeline execution.

This is a feature roadmap describing the target state and required changes, not an implementation sequence. It does not authorize release publication.

Current Baseline

The reviewed baseline is Notarius v0.7.1 (53d9284):

• go.mod pins ConfigKit v0.3.0 and PromptKit v0.10.0.
• internal/core/config/load.go passes LoadOptions.Profile into configkit.Options[Document].Profile.
• --config-profile selects profiles/<name>.yml beneath the configuration root for run, config validate, config effective, config explain, and pipelines list.
• The maintained selected-layer example is examples/profiles/development.yml; configuration and CLI tests create equivalent directories and exercise selection and inspection.
• ConfigKit owns composition and inspection metadata. Notarius owns its version 5 document, defaults, validation, source-relative path resolution, runtime conversion, and CLI discovery.

A dependency-only trial against v0.4.0 fails to compile at load.go:33 because configkit.Options.Profile no longer exists. The review found no direct Notarius references to LayerProfile, CodeProfile*, or LoadSecretsDir.

An isolated dependency-only trial against PromptKit v0.11.0 passes the complete Notarius test suite and go vet ./.... That proves baseline compatibility, not adoption of the new controls or observations. The current adapter does not configure transient retries or attach an execution observer. It classifies generation errors broadly and records successful-result usage and structural repair counts. The outer scheduler holds a permit for the whole completion.

Upstream Changes

ConfigKit v0.4.0

The ConfigKit v0.4.0 release guide defines four breaking naming changes:

Surface	Previous	Target
Selection API	Options.Profile	Options.Overlay
Selected file	profiles/<name>.yml	overlays/<name>.yml
Provenance layer	LayerProfile / profile	LayerOverlay / overlay
Diagnostic identifiers	CodeProfile* / profile.*	CodeOverlay* / overlay.*

There are no legacy aliases or directory fallback. Explicit imports from a directory named profiles remain valid ordinary imports. Merge rules, precedence, lifecycle, source-origin APIs, minimum Go version, and YAML parser dependency are unchanged.

The intervening v0.3.1 release makes the opt-in LoadSecretsDir API ignore dot-prefixed entries. Notarius does not use this API; adding secrets-directory bootstrap is outside this work.

PromptKit v0.11.0

The PromptKit v0.11.0 release guide introduces additive runtime APIs and one changed default:

• Built-in generation defaults to two additional transient HTTP attempts for statuses 429, 500, 502, 503, and 504. Engine and request controls can override the count, including zero. Injected LLMClient implementations are not automatically replayed; a custom HTTP client on the built-in path still is.
• Backoff is cancellable and honors valid provider Retry-After timing. Generation and HTTP timeouts apply per attempt; the caller context bounds the whole operation. Transport failures, malformed responses, other HTTP statuses, and invalid generated content are not transient-retry candidates.
• RunRequest.Observer reports lifecycle metadata, attempts, waits, outcomes, and presence-aware usage, including work before a failed execution. It works with prepared execution; preparation itself emits no execution events.
• ErrTransport, ErrMalformedResponse, ErrRetriesExhausted, and RetryExhaustedError support typed classification without parsing error text.
• The license changes to LGPL-3.0-or-later. Catalog selections and the minimum Go version are unchanged from v0.10.0.

Target End State

Dependency And Ownership

Pin ConfigKit v0.4.0 and PromptKit v0.11.0 and tidy the dependency graph. Continue using only ConfigKit's public root API behind internal/core/config. No compatibility parser, application-owned composition engine, or new configuration abstraction is needed. PromptKit remains behind internal/framework/llm; its types must not leak into domain modules or pipeline contracts. Retain the existing prepared execution path, profile choices, and D&D model settings.

Consistent Overlay Selection

• Replace the configuration-selection CLI flag with --config-overlay on all commands that currently accept --config-profile.
• Rename application-owned fields, variables, helpers, help text, and test descriptions that specifically represent this selection to overlay terms.
• Pass the selection through configkit.Options.Overlay.
• Select files only from overlays/<name>.yml, following ConfigKit's contract.
• Preserve empty and default selection as the composed base configuration without a selected overlay.
• Use a clean cutover: no old CLI alias, directory fallback, automatic file moves, or custom migration machinery. Normal unknown-flag handling is sufficient for the retired flag; explain migration in documentation.

This rename is deliberately narrow. Preserve llm_profile, --llm-profile, promptkit.profile_dir, promptkit.profile_file, base_profile, embedded dnd-extraction profiles, and examples/llm-profiles/. Other application concepts named Profile, including pipeline/runtime types, must not be renamed merely because they share the word.

Preserved Behavior

Retain configuration version 5, with the additive retry setting described below. Preserve existing defaults other than the explicitly adopted transient retry policy, imports, overlay merge behavior (including sequence replacement), explicit CLI override precedence, strict decoding, domain validation, and secret redaction. Do not add new restrictions or relax existing explicit contracts as part of the rename.

Relative paths continue to resolve from the file supplying the surviving declaration, including overlay files. Invocation-relative CLI paths and main-file-relative built-in defaults retain their current behavior.

Effective configuration, explanations, and configuration-load diagnostics use the new upstream overlay provenance and diagnostic vocabulary. Preserve source file labels and actionable error context; do not translate the new metadata back into legacy profile terminology.

Pipeline stage order, semantic acceptance, LLM selection, domain artifact schemas, and subprocess result receipts remain unchanged. No blanket checkpoint invalidation is added merely for the ConfigKit rename. Update the existing PromptKit-version identity used by checkpoint fingerprinting with its dependency pin, following the current adapter convention; no cache redesign is needed.

Transient Retry Policy

Expose promptkit.transient_retries as a nonnegative integer setting with a Notarius default of 2 additional attempts. Explicit zero disables transient retries. Apply the resolved setting to the shared PromptKit engine so modules, normalizers, validators, and semantic correction calls receive the same policy. Include the effective value in configuration inspection. Do not impose an arbitrary application maximum or add per-binding controls in this work set.

Maintain three separate budgets and owners:

Budget	Owner	Repeated work
Transient retries	PromptKit	The same generation request after eligible HTTP failures
Structural repairs	PromptKit	A corrective generation for invalid structured output
Producer/validator retries	Notarius	The configured stage or validator operation, with semantic feedback where applicable

Do not silently reduce, repurpose, or remove existing stage/validator budgets. Each structural repair generation receives its own transient budget. For a single-completion producer, two additional stage attempts, one structural repair, and two transient retries permit up to 18 HTTP attempts, excluding any separate validator calls. Document this multiplication and the possibility of additional cost; retries do not imply exactly-once provider execution.

Preserve caller cancellation throughout generation and waiting. Do not describe a profile's generation timeout as a total completion deadline: provider Retry-After can extend waits beyond local backoff, and each HTTP attempt has its own timeout. A new total-deadline configuration surface is outside this upgrade; accurately document the current controls and limitations.

Retain Notarius's completion-scoped scheduler permit during upstream backoff. PromptKit releases its active backend permit during that wait, but this does not release the outer Notarius permit. This remains bounded and correct; do not add an attempt-level scheduler redesign.

Provider-Neutral Failure Diagnostics

Translate the new typed errors into application-owned classifications for transport failures and malformed provider responses. Represent transient retry exhaustion separately from its underlying failure/status, retaining the actual attempt count where available. Preserve existing HTTP-status reporting, capacity classification, cancellation, deadline handling, and redaction.

Do not turn a malformed provider envelope into a semantic rejection or confuse it with a structurally invalid model response. Do not expose PromptKit error types across the adapter boundary or parse prose to recover classifications. Existing stage disposition policies continue to decide what happens after a completion fails; this work does not redesign those policies.

Execution Observations

Attach a request-local observer through the existing prepared-execution path. Translate selected metadata into Notarius-owned completion/debug summaries:

• dispatched attempt, transient retry, and structural repair-call counts;
• capacity-wait, generation, retry-delay, and total execution durations;
• terminal outcome, failure classification, HTTP status, and exhaustion state;
• observed token counts with explicit known/complete indicators.

Retain available summaries on failed completions, including cancellation, rather than discarding them with the absent RunResult. Preparation failures have no execution observation; do not fabricate one. Keep existing successful-result usage fields and their meanings unchanged. Observation totals are separate presence-aware metadata, not a second set of tokens to add to existing usage. Unknown counts are not known zero; partial totals are not a billing ledger.

Use lightweight per-request accumulation with no unbounded event history or asynchronous queue. Callbacks are synchronous: avoid blocking I/O or acquisition of scheduler resources inside them, and isolate concurrent requests. Persist the projection through the existing opt-in debug path, including safe terminal summaries where applicable. Do not serialize upstream event structs directly; their JSON layout is not a public contract. Allowlist metadata and preserve existing identifier/content handling and redaction boundaries.

Successful transient recovery is ordinary success, not a warning. This work does not add routine console messages, change the warning policy, introduce a new durable telemetry artifact, or add complete cost accounting.

Required Changes

Code And Examples

• Update go.mod and go.sum and the configuration loader/options boundary.
• Update CLI selection plumbing in internal/cli/run.go and configkit_commands.go, including usage, flag parsing, argument classification, explicit empty-value handling, and error paths.
• Move the maintained example to examples/overlays/development.yml and update example discovery and related fixtures. Inspect relative paths when moving files; do not assume all occurrences of profiles are configuration overlays.
• Update affected configuration and CLI tests, particularly document paths, operational defaults, safe inspection, command selection, diagnostic source labels, run contracts, and maintained-example coverage.
• Add the retry setting through the typed configuration document, defaults, validation, runtime conversion, and CLI client-construction boundary. Update the complete example to demonstrate it without adding unnecessary settings to the minimal example.
• Update the PromptKit adapter's engine construction, request observation, error conversion, and checkpoint version identity. Extend transport-neutral response/debug contracts only with the application-owned metadata needed.
• Carry observations through the pipeline debug wrapper and safe terminal reporting without double-counting usage or losing failed-call summaries.

Documentation

Update current documentation with the implementation, keeping canonical ownership under Documentation Policy:

• docs/cli.md: new flag on every affected command and matching help behavior.
• docs/config.md: overlay terminology, directory layout, precedence, and the explicit distinction from LLM profiles; transient retry setting and default.
• docs/integrations/pkg-configkit.md: pinned release and supported upstream naming/inspection contract.
• docs/internal/configuration.md and relevant CLI internals: new selection terminology and unchanged boundaries.
• docs/consumers/subprocess.md and any other current callers/examples: updated invocation syntax. Review operations and README references where applicable.
• docs/integrations/pkg-promptkit.md: updated pin, typed errors, observer support, and retry contract. Refresh pinned upstream links, including its relocated release-policy document.
• docs/internal/llm.md and relevant pipeline/debug internals: observation projection, failure handling, scheduler interaction, and separate retry budgets. Remove the obsolete claim that PromptKit has no provider retry loop.
• docs/operations.md: latency/cost implications, timeout limits, opt-in debug evidence, and how to disable transient retries.

Preserve historical release notes and the accepted text of ADR-0018. Its ConfigKit ownership decision remains valid; a terminology migration alone does not require a new ADR or supersession. Current documentation must not portray the old dependency pin as the current supported version.

Record PromptKit's LGPL-3.0-or-later change in the integration documentation and eventual release note. Review its notices and distribution implications within Notarius's source-only release model. Do not silently relicense Notarius, copy upstream release policy into this repository, or introduce binary packaging. Any future binary distribution needs its own license-compliance review.

Operator Migration And Release Impact

This work belongs in the next minor Notarius release, not a patch, because both the documented selected-file layout and CLI syntax change.

Operators move selected configuration files from profiles/ to overlays/ and replace --config-profile with --config-overlay in scripts, service definitions, and subprocess callers. Preserve unrelated PromptKit profile directories and explicit imports. Review relative paths and explicit imports that refer to moved files. No YAML body version migration is required.

The eventual release note must explain this cutover, the newly enabled transient retries and zero-value opt-out, and the dependency license change. PromptKit's catalogs are unchanged from v0.10.0; this upgrade adds no further profile-name migration or change to the D&D model defaults. Release preparation, tagging, and publication remain a separately requested task under Source Releases.

Acceptance And Verification

• Notarius builds against ConfigKit v0.4.0 and PromptKit v0.11.0 without compatibility shims.
• All five affected commands use the new flag; non-default overlay selection, omitted selection, empty selection, and default retain their intended behavior. The old flag is not accepted.
• Selected overlays load from the new directory. A legacy-only selected file does not silently load; ordinary explicit imports remain supported.
• Representative tests demonstrate overlay precedence, CLI overrides, declaration-relative paths, redacted inspection, and source-aware failures.
• Maintained examples validate, including the development overlay. LLM profile loading and defaults remain unaffected.
• Tests prove the default transient retry policy, explicit zero, configured counts, and negative-value rejection through the relevant configuration and adapter boundaries. Use local HTTP test servers for eligible/noneligible statuses, exhaustion, cancellation, and interaction with structural repair; an injected LLM fake alone does not exercise built-in HTTP retries.
• Provider-neutral errors preserve underlying status and cancellation identity without leaking credentials or upstream types.
• Observations are available for success and failure, preserve unknown versus known-zero usage, avoid aggregate double-counting, and stay isolated under concurrent completions. Successful retry recovery emits no warning.
• Representative scheduler/repair/retry checks preserve bounded execution and cancellation; use existing coverage where it already proves the contract.
• Current documentation, help text, examples, and tests agree. Searches for old vocabulary are reviewed semantically, excluding intentional historical and LLM-profile references rather than enforcing a blanket text ban.
• Run focused go test ./internal/core/config ./internal/cli ./internal/framework/llm ./internal/framework/pipeline, followed by go test ./..., go vet ./..., go build ./..., go mod tidy -diff, and git diff --check, with GOWORK=off. Run race-enabled tests for the modified adapter, scheduler, and observation integration. Validate both maintained configurations and the complete configuration with --config-overlay development.

Use existing behavioral tests where possible, following Testing Policy. No live model calls or production API credentials are needed to validate this migration.