notarius/docs/roadmap/implementation.md

# ConfigKit And PromptKit Upgrade Implementation Plan

## Purpose And Scope

Implement [ConfigKit And PromptKit Upgrades](upgrade-kit.md). That document
owns the desired behavior and operator migration; this document owns execution
order. Each numbered stage is one bounded implementation prompt for a
gpt-5.6-terra coding agent. Complete and validate stages in order.

The inspected baseline pins ConfigKit v0.3.0 and PromptKit v0.10.0. Adopt
exactly ConfigKit v0.4.0 and PromptKit v0.11.0, not whatever release happens to
be newest during implementation. The release guides and public APIs are
available in the sibling checkouts `../ConfigKit` and `../promptkit`; consult
the target tags or downloaded modules if those checkouts have advanced.
Build and test Notarius with `GOWORK=off`, without local dependency replacements.

Read `docs/development.md` and all five policies in `docs/policy/` before
implementation. In particular, simplify machinery without narrowing operator
choices or elevating incidental details into requirements. Preserve existing
contracts unless this roadmap expressly changes them. Use graph discovery for
code, then inspect the relevant source and tests; locations below identify the
reviewed baseline, not a requirement to preserve private file structure.

## Decisions And Boundaries

- Configuration selection becomes `--config-overlay` and `overlays/<name>.yml`,
  with no legacy alias or fallback. Keep empty, omitted, and `default` selection
  behavior. Do not rename PromptKit profiles or pipeline/runtime profile types.
- Keep configuration version 5, source-file-relative declarations, CLI override
  precedence, strict loading, inspection provenance, and redaction.
- Add only the global `promptkit.transient_retries` setting: nonnegative integer,
  default **2 additional attempts**, explicit zero disables. No arbitrary cap,
  per-binding setting, new CLI retry flag, or new total-deadline setting.
- PromptKit owns transient HTTP retries and structural repairs; Notarius keeps
  stage/validator retry budgets and disposition unchanged. The existing shared
  scheduler retains its completion-scoped permit through upstream waiting.
- Add provider-neutral failure metadata and request-local execution summaries.
  Do not change domain artifacts, result receipts, prompt content, validators,
  acceptance policy, or introduce a new telemetry service/artifact.
- Successful retry recovery is ordinary success, not a warning. Debug remains
  opt-in. Existing token fields keep their meaning; observed usage is separate.
- No release, tag, push, source-license change, or binary packaging is part of
  these stages. Record upgrade implications in current integration/operator
  documentation; the eventual release note is a separate release task.

For each stage, inspect the starting worktree, implement code with its focused
tests and current-behavior documentation, run the listed relevant checks, and
make one coherent local commit under agent policy. Report checks and any gaps.
Do not repeat passed checks for unchanged content merely to make a commit.

## Stage 1 — Cut Over Configuration Selection To Overlays

**Status:** Completed 2026-09-25.

**Goal:** Adopt ConfigKit v0.4.0 and finish its narrow operator-facing rename in
one coherent change. This is one prompt: it is a naming migration, not a new
configuration engine.

**Inspect:** `internal/core/config/load.go`, `loaded_config.go`, configuration
document/path/operational tests; `internal/cli/run.go`,
`configkit_commands.go`, CLI configuration and example contract tests;
`examples/profiles/development.yml`; upstream v0.4.0 migration guide.

**Work:**

1. Upgrade ConfigKit to v0.4.0 in `go.mod`/`go.sum`. Change `LoadOptions.Profile`
   to `Overlay` and pass it into `configkit.Options[Document].Overlay`. Rename
   only helpers, fields, locals, fixtures, and descriptions representing the
   selected configuration layer.
2. Replace `--config-profile` with `--config-overlay` for `run`, `config validate`,
   `config effective`, `config explain`, and `pipelines list`. Update usage,
   flag parsing, argument classification, explicit-empty handling, and error
   paths together. Retired flags use normal unknown-flag behavior.
3. Move `examples/profiles/development.yml` to
   `examples/overlays/development.yml`; update maintained-example discovery and
   selected-layer fixtures. Check file-relative paths after moving. Leave
   `examples/llm-profiles/`, PromptKit profile sources, and ordinary explicit
   imports alone unless an import actually points at the moved example.
4. Consume upstream overlay provenance and diagnostic names directly. Do not
   rebuild composition, implement directory fallback, or translate metadata
   back to profile vocabulary.
5. Update `docs/config.md`, `docs/cli.md`, `docs/integrations/pkg-configkit.md`,
   `docs/internal/configuration.md`, relevant portions of `docs/internal/cli.md`,
   and `docs/consumers/subprocess.md`. Review README, operations, and other
   consumer examples for affected invocations. Explain the clean migration and
   distinction from LLM profiles. Preserve accepted ADR-0018 and release history.

**Acceptance:** All five commands select overlays consistently. Representative
existing tests cover selected/base/empty/default behavior, overlay and explicit
override precedence, declaration-relative paths, redacted effective/explain
output, and source-aware failure labels. Include a selected legacy-only file
case proving there is no silent fallback. Retain ordinary-import behavior and
LLM profile coverage; do not duplicate every upstream parser case.

**Validate:** `GOWORK=off go test ./internal/core/config ./internal/cli`,
`GOWORK=off go mod tidy -diff`, `git diff --check`. Review occurrences of
`config-profile` and `examples/profiles` in current docs/code semantically;
historical docs and explicit legacy-rejection tests are not defects.

## Stage 2 — Upgrade The PromptKit Runtime Baseline

**Status:** Completed 2026-09-25.

**Goal:** Adopt the compatible upstream dependency and document its changed
default before adding Notarius controls. One small prompt.

**Inspect:** `internal/framework/llm/promptkit_client.go`, asset/profile identity
helpers and tests, `go.mod`, and upstream v0.11.0 release/consumer documentation.

**Work:**

1. Pin PromptKit v0.11.0 and tidy. Keep the Go version and provider catalog
   selections unless the actual dependency graph requires a change; the target
   release does not require either. Do not alter embedded D&D model settings.
2. Update `promptKitBuiltinProfileCatalogID` to the v0.11.0 identity in accordance
   with the existing checkpoint fingerprint convention. No other cache policy
   change or blanket invalidation is part of this work.
3. Update the pinned integration links in `docs/integrations/pkg-promptkit.md`,
   including the relocated upstream `docs/policy/release.md`. Record the
   LGPL-3.0-or-later dependency license and reference its license/notices; do not
   relicense Notarius or invent a new distribution workflow.
4. Remove the obsolete no-provider-retry-loop statement in `docs/internal/llm.md`.
   Describe the now-active upstream default and per-attempt timeouts in the
   appropriate integration/operations sections. Do not yet describe the setting
   or observations from later stages as implemented.

**Acceptance:** The prepared-execution adapter builds and its existing tests
pass without prompt/catalog changes. Current documentation acknowledges the
dependency's default retry behavior and license accurately.

**Validate:** `GOWORK=off go test ./internal/framework/llm ./internal/cli`,
`GOWORK=off go mod tidy -diff`, `git diff --check`.

## Stage 3 — Wire The Explicit Transient Retry Setting

**Status:** Completed 2026-09-25.

**Goal:** Make retry policy inspectable and configurable through the one shared
engine. One prompt covering a single vertical configuration slice.

**Inspect:** `internal/core/config/{document,config,load,operational_document,
loaded_config,validation}.go`, effective-config tests;
`internal/cli/catalog.go:buildProductionLLMClient`;
`internal/framework/llm/promptkit_client.go:NewPromptKitClient` and HTTP tests.

**Work:**

1. Add `TransientRetries *int` to `PromptKitDocument` and a resolved integer to
   `PromptKitConfig`. Supply 2 through both the document default path and the
   runtime `Default()` path. Preserve explicit zero; never use zero as omission.
   Include it in document-to-runtime conversion and effective configuration.
2. Reject negative values through both existing document and direct runtime
   validation entry points. Follow current source-aware diagnostics; do not
   impose a new maximum. Keep ConfigKit's existing decoding behavior.
3. Add `TransientRetries *int` to `PromptKitClientConfig`: nil follows upstream
   default, non-nil includes explicit zero. Production construction always
   supplies the resolved configuration value. Pass it into `promptkit.Config`,
   not per-request options. All calls through the injected shared client thus
   share the setting without module changes.
4. Add the setting to `examples/dnd-complete.config.yml` and `docs/config.md`;
   keep the minimal example minimal. Update configuration/LLM internal guides
   for the new plumbing and operations for the zero-value opt-out.

**Acceptance/tests:** Demonstrate omitted default, explicit zero, a positive
override, negative rejection, and effective/explain inspection. Use an existing
CLI factory test to prove the resolved value reaches construction. Exercise
built-in generation with a local HTTP server: eligible failure then success,
disabled retries, exhausted configured count, and a noneligible 400 response.
An injected `LLMClient` stub does not test built-in HTTP retry wiring. Avoid
testing the upstream jitter formula or every equivalent HTTP status.

**Validate:** `GOWORK=off go test ./internal/core/config ./internal/cli ./internal/framework/llm`
and `git diff --check`.

## Stage 4 — Preserve Typed Failure And Exhaustion Metadata

**Status:** Completed 2026-09-25.

**Goal:** Distinguish provider failures without coupling callers to PromptKit.
One prompt; defer execution observations to the next stage.

**Inspect:** `internal/framework/contracts/errors.go`, adapter error conversion
in `promptkit_client.go`, redaction helpers, and existing pipeline handling of
`ErrLLMGeneration`, `ErrLLMCapacityExceeded`, and invalid structured output.

**Work:**

1. Extend the existing application-owned `LLMGenerationError` rather than
   creating a parallel provider-error hierarchy. Retain `StatusCode()` and
   `errors.Is(err, ErrLLMGeneration)`. Add application-owned failure-kind,
   exhaustion, and exhausted-generation attempt metadata/accessors; use uint64
   for upstream attempt counts. Preserve the existing constructor's callers
   with sensible unspecified/generation metadata, adding a metadata constructor
   or private adapter helper as needed.
2. Define kinds for `transport`, `provider_status`, `malformed_response`, and
   generic `generation`. Exhaustion is an independent boolean and attempt count,
   not a replacement kind. Zero attempts means unavailable in this error
   projection, not proof no request happened. The count comes from
   `RetryExhaustedError.Attempts()` and covers the exhausted generation, not all
   repair generations combined.
3. Use `errors.Is`/`errors.As` on upstream sentinels/types. Keep cancellation and
   deadline identity first, capacity handling intact, and HTTP status available
   through an exhaustion wrapper. Do not unwrap application errors to upstream
   provider objects or classify by prose. Preserve sanitized diagnostic text.
4. A malformed provider envelope is a generation failure; schema-invalid model
   content stays `ErrInvalidStructuredOutput`. Do not change outer retry or
   validator disposition logic. Document classification at the integration and
   internal LLM boundaries.

**Acceptance/tests:** Adapter tests cover transport, malformed response,
noneligible status, exhausted eligible status with count/status preserved, and
cancellation/deadline/capacity regression behavior. Check typed fields and
identities, not full messages. Existing credential-redaction cases remain valid
for nested exhaustion diagnostics; use fake secrets only.

**Validate:** `GOWORK=off go test ./internal/framework/contracts ./internal/framework/llm ./internal/framework/pipeline`
and `git diff --check`.

## Stage 5 — Project Request-Local Execution Observations

**Goal:** Return a lightweight terminal summary even when generation fails.
One prompt focused on contracts and adapter mapping, without persistence changes.

**Inspect:** `internal/framework/contracts/contracts.go`, `CompleteStructured`,
`responseFromResult`, prepared execution tests, upstream `observations.go` and
prepared-execution observer documentation.

**Work:**

1. Add an optional `Execution *LLMExecutionSummary` field to
   `StructuredCompletionResponse` (`execution,omitempty`). Own these DTOs in
   Notarius contracts; do not alias or serialize upstream event types.
2. The summary contains uint64 `attempts`, `transient_retries`, `repair_calls`;
   integer millisecond `capacity_wait_ms`, `generation_ms`, `retry_delay_ms`,
   `duration_ms`; `outcome`, `failure_kind`, `status_code`, `retries_exhausted`;
   and `usage`. Usage contains prompt/completion/total/cached/cache-write counts,
   each with `value`, `known`, `complete` fields. Do not omit these booleans or
   known-zero values. The summary needs no prompts, response text, endpoints,
   credentials, raw errors, or new copies of run/profile identifiers.
3. Map terminal outcome/failure values explicitly to application-owned strings.
   Preserve the distinctions exposed by v0.11.0 (including canceled, deadline,
   capacity, invalid output, configuration/load/render failures); unknown future
   failures map to `other`. Reuse relevant application failure vocabulary from
   Stage 4, without forcing every execution error into a generation error.
4. Attach a local observer to the `RunRequest` before `PrepareExecution` freezes
   it. Retain the `ExecutionFinished` projection only: upstream supplies aggregate
   counts, durations, and usage. Do not sum attempt events and terminal totals,
   retain an event log, or reconstruct a second accounting system. Synchronous
   request-local collection needs no global state or asynchronous queue.
5. Attach that summary to every subsequent return from `RunPrepared`, including
   context/capacity/provider failure and post-result decode/validation failure.
   Preparation failure has no execution event and returns no summary. Preserve
   upstream execution outcome as such: a later Notarius decode error remains a
   separate returned error, not a fabricated upstream terminal event.
6. Keep existing successful-result token and repair fields unchanged. Observed
   `repair_calls` represents execution activity and is not a renamed legacy
   `RepairAttempts`. No observation field is a billable-cost guarantee.

**Acceptance/tests:** Success, failed execution, cancellation, and no-observation
preparation failure retain the right presence. Test known zero versus unknown
and partial aggregate usage with controlled provider responses. One concurrent
completion case proves no summary cross-contamination. Check relationships and
nonnegative durations, not exact timing. Document the DTO and mapping in
`docs/internal/llm.md`; public dependency semantics stay in pkg-promptkit.

**Validate:** `GOWORK=off go test ./internal/framework/contracts ./internal/framework/llm`
and `GOWORK=off go test -race ./internal/framework/llm`.

## Stage 6 — Carry Summaries Through Existing Debug Paths

**Goal:** Make observations useful to operators without a new output channel.
One prompt, limited to debug propagation and terminal allowlists.

**Inspect:** `internal/framework/pipeline/debug.go`, its response-content/fallback
helpers, terminal recording tests, and `debugLLMScope.references`. Trace
`debugLLMCallReference` through the existing timed stage/terminal envelopes;
updating only the detailed response file is insufficient. The CLI's separate
run-summary files do not need a new run-wide accounting mechanism.

**Work:**

1. Project the optional execution summary into existing debug response artifacts
   and per-call references used by stage/terminal records. Explicitly preserve it
   when `Debug.Response` or `RunResult` is absent and through content splitting
   and fallback construction. Keep copies detached if existing clone paths
   could otherwise share mutable pointers.
2. Use explicit typed fields as the allowlist for this metadata-only terminal
   projection. Do not carry raw upstream events, newly copied provider diagnostics, or model
   content into safe terminal summaries. Preserve established redaction and
   detailed-trace content boundaries.
3. Leave warnings, normal console output, receipts, domain artifacts, checkpoint
   payloads, and no-debug behavior unchanged. No new telemetry file or aggregation
   of observed token totals into legacy totals.
4. Update `docs/internal/state.md`/relevant debug internals and `docs/operations.md`
   to identify where this diagnostic information is available. Link to the
   canonical explanation rather than repeating every field in consumer docs.

**Acceptance/tests:** Through the real debug wrapper and existing recorder seam,
verify success and failed/canceled completions retain summaries in response and
safe terminal records. Verify content splitting/fallback does not discard them,
and debug disabled writes nothing. Reuse existing redaction tests, adding only
the new projection assertions. No new warnings result from recovered retries.

**Validate:** `GOWORK=off go test ./internal/framework/pipeline ./internal/cli`,
`GOWORK=off go test -race ./internal/framework/pipeline`, `git diff --check`.

## Stage 7 — Verify Retry Composition And Finish Operator Guidance

**Goal:** Validate the cross-boundary behavior that individual stages cannot
prove, then reconcile current documentation. One prompt; no generalized retry
test framework or runtime redesign.

**Inspect:** Existing adapter HTTP fixtures, scheduler cancellation tests,
pipeline stage retry/debug tests, CLI production tests; current config, LLM,
operations, and consumer documentation.

**Work and acceptance:**

- Add one representative local-HTTP scenario combining a recoverable transient
  status with structurally invalid generated content followed by a successful
  repair. Prove transient budgets apply independently per generation and
  terminal attempt/repair/usage accounting is not doubled. Use small explicit
  budgets rather than executing the illustrative 18-request worst case.
- Exercise cancellation during provider-directed retry waiting without a long
  real sleep: signal that the request was received, cancel a context while a
  long Retry-After is pending, and wait with a bounded test timeout. Preserve
  cancellation identity and the terminal observation. Do not build a second
  backoff implementation or test exact random delays.
- Review shared scheduler tests for completion-scoped permit ownership and
  cancellation/release. Add a focused integration case only where existing
  coverage does not protect that boundary across the new retry path. Keep
  upstream backend capacity algorithms upstream.
- Confirm a recovered generation passes through ordinary success without
  producing process warnings. Preserve existing outer stage retries and semantic
  feedback. Do not alter limits to hide budget multiplication.
- Finish operations guidance: distinguish transient, structural, and stage/
  validator budgets; give the 18-attempt single-producer illustration with its
  assumptions; explain per-attempt timeout versus overall caller cancellation,
  Retry-After waits, cost/latency implications, incomplete usage accounting, and
  the outer scheduler holding its permit during waits. No new total deadline or
  exactly-once guarantee is implied.
- Review all canonical docs changed earlier for consistency. Keep exact fields
  and defaults in config, flags in CLI, dependency semantics in integrations,
  mechanism in internals, and operating implications in operations. Keep the
  migration note about overlays distinct from unchanged LLM profiles. Retain
  historical ADR/release content; no new ADR is needed for these upgrades.

**Validate:** Run the focused adapter/pipeline/CLI tests and race tests covering
modified concurrency paths. Review tests for unnecessary upstream duplication
or exact-prose/change-detector assertions; consolidate rather than multiply them.

## Stage 8 — Final Integration Verification And Handoff

**Goal:** Verify the complete upgraded application and documentation against the
feature roadmap. One final verification prompt; fix only defects within scope.

**Work:** Review all feature acceptance items and staged diffs. Confirm no
upstream types escaped the LLM boundary, no references to the retired config
flag remain in current examples/help, no unintended LLM profile rename occurred,
and no source-only/license/release scope expanded. Check edited relative links.
Leave the roadmap available for the user's completion review; do not publish a
release or delete planning documents automatically.

**Validation commands:**

```sh
GOWORK=off go test ./...
GOWORK=off go vet ./...
GOWORK=off go build ./...
GOWORK=off go mod tidy -diff
GOWORK=off go test -race ./internal/framework/llm ./internal/framework/pipeline
GOWORK=off go run ./cmd/notarius config validate --config examples/dnd-minimal.config.yml
GOWORK=off go run ./cmd/notarius config validate --config examples/dnd-complete.config.yml
GOWORK=off go run ./cmd/notarius config validate --config examples/dnd-complete.config.yml --config-overlay development
git diff --check
```

Reuse already-passed checks for the same unchanged candidate where appropriate.
The example/CLI tests own maintained examples; the commands above are a one-time
migration smoke check, not new permanent release gates. No live model calls or
real API keys are required. Report environment/dependency-access failures
separately from implementation failures; do not claim blocked checks passed.

**Completion criteria:** Both exact dependency pins are active; overlay migration,
retry setting, typed errors, observations, debug propagation, tests, examples,
and current documentation satisfy the feature roadmap. Report commits and
checks, plus any residual limitation. Future release preparation should explain
the breaking overlay cutover, retry default/opt-out, and upstream license change
in a minor-release note, but is not part of this implementation.

## Open Questions

None. The feature roadmap supplies the policy choices needed for implementation.
The DTO naming and stage boundaries above settle remaining implementation choices
without adding new product restrictions or compatibility promises.