audita/docs/roadmap/implementation.md

# Implementation Plan: Testing Overlays and Migration Handoff

**Last updated:** 2026-09-24

**Status:** Complete (2026-09-24). All six stages below are finished. This plan implements the next
unit in the [prototype roadmap](library-boundary.md#next-unit-testing-overlays-and-production-migration-plan),
not the production ConfigKit or PromptKit migrations.

**Production handoff revision:** The completed prototype stages remain recorded
below. The current [production plan](configkit.md#ordered-production-migration-handoff)
targets both libraries in one refactor; intermediate commits need not build or run.
It supersedes the earlier interim-runtime design. No temporary adapter,
compatibility shim, or processing gate is required.

Execute Stage 1 through Stage 6 in order. Each stage is sized as a separate coding
prompt for a gpt-5.6-terra agent and ends with a reviewable result. When assigned
one stage, finish that stage and report the next stage; when assigned the entire
plan, continue through all stages. Do not restart completed work: inspect current
files, recorded results, and Git state before deciding what remains.

Before each stage, follow `AGENTS.md`, [development guidance](../development.md),
and the [agent policy](../policy/agents.md). Preserve unrelated changes. Use the
knowledge graph for code discovery; examples may be excluded from its index, in
which case inspect the known prototype files directly. Follow the repository's
testing and documentation policies. Complete relevant checks and make a plain
English local commit for each coherent stage under the standing commit policy.
No push, tag, release, or upstream-library edit is included.

Update each stage's status line with its result and verification evidence when
finished; keep detailed design/results in their owning roadmap documents. Do not
mark a stage complete while required checks are failing. Fix in-scope failures;
record any genuine upstream gap and its effect on later stages before continuing
work that depends on it. Routine implementation choices do not need fresh approval.

The [ConfigKit roadmap](configkit.md#selected-overlays-for-testing) owns the overlay
contract. The [boundary findings](library-boundary-findings.md) own existing choices
and evidence. Preserve the original v0.3.1 results as historical. ConfigKit v0.4.0
renames selected configuration profiles to overlays; do not rename PromptKit's
`profiles/`, `profiles_dir`, or execution-profile fields.

All implementation in this plan stays under `examples/library-adoption/`, plus
relevant roadmap/README updates. Keep the root Go module, production CLI/runtime,
correction algorithms, and upstream libraries unchanged. Use the existing built-in
PromptKit HTTP path with a controlled transport, synthetic credentials, and no live
provider calls. Keep imports and secret-directory bootstrap disabled. This work
does not introduce general environment overlays or backwards compatibility.

Unless explicitly stated otherwise, shell commands below run from
`examples/library-adoption/`. Use an installed Go 1.25.5 or newer. Acquire dependencies
with `GOWORK=off go mod download all` when needed; then use the offline command
prefix below. Root `go test ./...` does not cover this nested module.

## Stage 1: Upgrade the Prototype Dependency

**Status:** Complete (2026-09-24). **Depends on:** nothing. ConfigKit v0.4.0 pinned;
the nested offline tests and demonstration passed with the expected 1/4/0/2 result.

Read the prototype README, `go.mod`, `go.sum`, and the current ConfigKit roadmap.
Confirm the working tree and run the existing nested-module tests once as a
baseline. Upgrade ConfigKit to the published v0.4.0 release using module tooling;
leave PromptKit and Audita dependency versions unchanged. Review any indirect
module changes and retain only changes needed by this upgrade. Do not add local
replacements or an active Go workspace.

No source changes were needed in the earlier isolated review. If a change is now
necessary, investigate and record the concrete reason instead of assuming that
ConfigKit execution overlays and PromptKit profiles share an API.

Verification:

```sh
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test ./...
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run .
```

Acceptance: the manifest and sums pin v0.4.0, tests pass, and the demonstration
still yields one accepted correction, four attempts, zero transient retries, and
two structural repairs. Update the README/findings to distinguish the new pin from
historical v0.3.1 evidence. Do not claim overlay integration yet.

## Stage 2: Add Overlay Loading and Configuration Fixtures

**Status:** Complete (2026-09-24). **Depends on:** Stage 1. On-disk overlay and
profile fixtures, precedence, inheritance, rejection, and runtime assembly tests
pass in the nested module (`go test ./...`).

Start with `config.go`, `config_test.go`, `config.yaml`, and `profiles/`. Extend
`loadConfig` with an explicit overlay-selection argument or a small invocation
options value. Pass it to ConfigKit `Options.Overlay`; selection does not belong
in the persistent `Config` struct. Update existing callers to request no overlay
without changing their behavior. Avoid a parallel merge implementation.

Add these prototype fixtures:

- `overlays/testing.yml`, selecting `proposal-testing` and `assessment-testing`
  through the existing two `llm` role fields. Do not put `version`, model settings,
  or ConfigKit directives in this overlay.
- Complete PromptKit profiles under `profiles/` with those IDs and distinct
  synthetic model IDs, such as `synthetic-grammar-testing` and
  `synthetic-review-testing`. Both use the existing `offline` backend and keep the
  current positive repair policy and finite shared capacity.

Preserve base `config.yaml` as the no-overlay fixture, including its explicitly
selected assessment profile. This makes the risk of accidentally leaving the
validation role on its base model observable. Profile paths remain relative to
main configuration according to the selected application contract, not relative
to the overlay file.

Add focused configuration tests proving:

- Omitted selection, empty selection, and `default` select the base configuration;
  `testing` selects both new profiles.
- Explicit ConfigKit overrides win over overlay values. Only supplied flags will
  become overrides in Stage 3; parser defaults must not enter this layer.
- With validation selection absent from both base and overlay, validation inherits
  the final proposal selection, including an explicit override. An existing
  explicit base validation selection remains explicit unless replaced by overlay
  or CLI. Empty/null validation values retain the established rejection behavior.
- Runtime profile assembly resolves both selected profiles to the shared backend.

Use temporary on-disk layouts for overlay tests so ConfigKit owns file discovery.
Existing byte-source tests can remain for cases that do not select an overlay;
if a byte source needs an overlay, supply ConfigKit's supported filesystem context
rather than relying on the process working directory.

Acceptance: the new fixtures load and assemble correctly, the focused precedence/
inheritance tests pass, and `go test ./...` in the nested module still passes.
Do not yet claim that the testing models reached the HTTP transport.

## Stage 3: Demonstrate and Verify Model Switching

**Status:** Complete (2026-09-24). **Depends on:** Stage 2. Nested tests inspect
outbound base/testing/override model IDs, including repairs; all three documented
offline demonstrations passed with one accepted correction and 4/0/2 attempts,
retries, and repairs.

Start with `main.go`, `offline.go`, `runtime.go`, and `workflow_test.go`. Add minimal
standard-library argument handling to the prototype executable:

- `--overlay <name>` selects the ConfigKit overlay.
- `--proposal-profile <id>` and `--validation-profile <id>` produce ConfigKit
  overrides only when explicitly supplied.
- Keep the existing fixture-based invocation from the prototype directory. There
  is no need to reproduce production file discovery or its subcommand tree here.

Use a small testable entry function if needed to check flag handling without
repeated subprocess builds. Preserve ordinary flag error handling. Keep execution
on the synthetic transport for every supported demonstration invocation; do not
add a live-network mode. Extend that transport to recognize the testing models and
preserve the initial-invalid-output/repair behavior for each role.

Add a compact safe selection summary to demonstration output: requested overlay
and effective profile, backend, and model for each role, alongside the existing
domain outcome and attempt/retry/repair totals. It must reflect the assembled
configuration rather than echoing flag values. Inspection APIs and tests in the
next stage suffice; do not create production config commands in this prototype.

Test through the assembled path and inspect actual outbound HTTP request bodies:

- No overlay sends the base proposal and assessment models.
- `testing` sends the testing proposal and assessment models, including repair
  requests, and still produces one accepted correction with two repairs.
- An explicitly supplied role-profile flag overrides that role's testing selection;
  the other role retains its overlay selection. An omitted flag preserves overlay
  selection. Match reported selections to observed outbound model IDs.

Use distinct models per role for these demonstration cases. Do not expand the fake
provider into a general simulation framework. Reuse existing prompt/schema/domain
helpers and existing capacity/observation coverage.

Acceptance: nested-module tests pass and these invocations visibly show the expected
selections and successful domain result:

```sh
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run .
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing --proposal-profile proposal
```

Update the prototype README with these implemented commands and fixture layout.
State that synthetic testing IDs demonstrate operator-selectable routing, not
measured price, speed, or model quality.

## Stage 4: Verify Failure, Credential, and Inspection Boundaries

**Status:** Complete (2026-09-24). **Depends on:** Stage 3. Focused offline tests
cover ConfigKit overlay errors, backend validation, credential replacement and
null handling, provenance, inspection, and redaction. Nested `go test -race ./...`
and `go vet ./...` passed on linux/amd64 with Go 1.26.8.

Extend focused tests in the prototype; keep them offline and independent of host
configuration and credentials. Exercise the consumer's assembled loader/runtime
rather than duplicating ConfigKit's exhaustive parser or merge tests.

Required evidence:

- Missing and invalid overlay selections fail with ConfigKit classifications.
  Representative attempts to replace the schema version or introduce ConfigKit
  directives fail. Do not create application fallbacks for those failures.
- An overlay can change an ordinary non-profile setting, such as a backend limit,
  and the effective setting still undergoes application validation. This proves
  overlays have not been accidentally restricted to profile selection.
- A replaced credential reference resolves only its surviving replacement; an
  unresolved displaced reference does not make the load fail. A surviving missing
  or empty secret reference still fails. Explicit null retains unauthenticated
  backend semantics. Use synthetic environment names and values under test control.
- Switching role profiles alone does not remove other credential references still
  present in configuration. Document this distinction and test the relevant case;
  do not bypass ConfigKit validation for unselected backends as an incidental fix.
- `WriteEffective` and relevant explanation/metadata APIs identify overlay origin;
  a CLI replacement identifies CLI origin. Inspect selected PromptKit profiles from
  that same loaded result and verify model/backend agreement with runtime selection.
- Synthetic credential values do not appear in effective configuration, explanations,
  profile inspection, observations, or surfaced errors exercised by these scenarios.
  Confirm the controlled transport receives the effective synthetic credential.

Maintain the existing retry/repair/capacity tests. Add only the overlay-specific
integration coverage needed to protect this boundary; do not repeat the complete
scheduler matrix for each fixture.

Verification:

```sh
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test -race ./...
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go vet ./...
```

Acceptance: all cases above have passing behavioral evidence, the existing suite
remains green, and concurrent observation collection remains race-free. Record
actual gaps instead of silently relaxing tests or claiming complete integration.

## Stage 5: Record Overlay Findings and Executable Instructions

**Status:** Complete (2026-09-24). **Depends on:** Stage 4. Dated v0.4.0 evidence,
test coverage, commands, limitations, README guidance, and roadmap links updated;
affected links and `git diff --check` verified.

This stage changes documentation, not production code. Read the new tests/results,
then update the existing owners rather than creating another parallel design:

- Append a dated v0.4.0/overlay result section to `library-boundary-findings.md`.
  Record commands, toolchain/platform, outcomes, test-to-requirement coverage,
  limitations, and any unresolved gaps. Keep original v0.3.1 evidence identifiable.
- Update the prototype README to match the actual executable and fixtures; check
  every documented command. Explain the separate ConfigKit overlay and PromptKit
  profile directories and the credential behavior proved in Stage 4.
- Record the overlay implementation as verified in `library-boundary.md` and link
  the new evidence from `library-adoption.md`. The next unit still has its Stage 6
  planning deliverable outstanding; keep production migration pending.

Acceptance: the README commands are reproducible, the new overlay evidence is
clearly separated from historical results, and each Stage 2–4 requirement maps
to a passing test or observed demonstration result. Verify affected links and
`git diff --check`. Do not repeat unchanged code checks solely for this commit.

## Stage 6: Prepare the Production Migration Plan and Close the Unit

**Status:** Complete (2026-09-24). **Depends on:** Stage 5. The ordered ConfigKit
production handoff records bounded packages, runtime integration, tests,
dependencies, and acceptance criteria. Prototype evidence and scope audit passed;
links and `git diff --check` verified.

Expand `configkit.md` into a concrete, ordered production migration plan using
the existing inventory and verified overlay evidence. Follow the development
reading guide for configuration/CLI and LLM runtime work; use graph-first discovery
to verify affected production components and their tests. This is a planning stage:
do not implement the production migration or overwrite this execution plan with
new production stages. Keep future work packages explicitly pending.

The production handoff must specify bounded work packages, dependencies, affected
components, tests, and acceptance criteria for:

- The full version-2 configuration definition: existing domain fields, defaults,
  presence/null/zero behavior, role inheritance, overlays, secrets, and validation.
- File discovery and invocation parsing; `--overlay` behavior shared by process,
  config validation, and effective inspection; explicit-setting override mapping;
  removal of legacy environment overlays, aliases, and direct-key flags.
- Profile loading/inspection and configuration snapshots, including relative-path
  bases, secret redaction, provenance, and clear separation of ConfigKit settings
  from PromptKit execution controls.
- Direct integration of the final configuration with PromptKit execution. Specify
  the final behavior and tests for each setting; do not preserve the old runtime
  through temporary adapters or add interim processing gates.
- Removal of superseded configuration machinery, retained domain checks, maintained
  examples, canonical documentation updates, and the combined migration guide.
- Removal of superseded LLM machinery and release follow-up. Preserve
  the agreed ordering and defer release compliance/publication to the final unit.

Resolve final integration decisions against the evidence. Record genuine open
questions without creating scaffolding to support intermediate commits.

Final audit: map every acceptance criterion in the prototype extension to actual
files and command results. Verify new relative links/anchors, run `git diff --check`,
and review that no root runtime/dependency or upstream files changed. Reuse passing
checks for unchanged code; rerun the affected tests if this stage changes code or
fixtures to correct a finding. A root build/test run is not a substitute for the
nested-module checks, and this documentation stage adds no release gate.

Mark this prototype-extension/planning unit complete in `library-boundary.md` only
after the audit passes, and link the production handoff from `library-adoption.md`.
Production implementation remains pending.

Acceptance: a reviewer can reproduce base/testing/override selection offline,
trace each new claim to evidence, and hand the production ConfigKit work to a coding
agent in bounded units. Commit the completed documentation, report remaining
limitations and working-tree state, and stop at the end of this plan.