audita/docs/roadmap/implementation.md

Implementation Plan: Testing Overlays and Migration Handoff

Last updated: 2026-09-24

Status: Planned. All stages below are pending. This plan implements the next unit in the prototype roadmap, not the production ConfigKit or PromptKit migrations.

Execute Stage 1 through Stage 6 in order. Each stage is sized as a separate coding prompt for a gpt-5.6-terra agent and ends with a reviewable result. When assigned one stage, finish that stage and report the next stage; when assigned the entire plan, continue through all stages. Do not restart completed work: inspect current files, recorded results, and Git state before deciding what remains.

Before each stage, follow AGENTS.md, development guidance, and the agent policy. Preserve unrelated changes. Use the knowledge graph for code discovery; examples may be excluded from its index, in which case inspect the known prototype files directly. Follow the repository's testing and documentation policies. Complete relevant checks and make a plain English local commit for each coherent stage under the standing commit policy. No push, tag, release, or upstream-library edit is included.

Update each stage's status line with its result and verification evidence when finished; keep detailed design/results in their owning roadmap documents. Do not mark a stage complete while required checks are failing. Fix in-scope failures; record any genuine upstream gap and its effect on later stages before continuing work that depends on it. Routine implementation choices do not need fresh approval.

The ConfigKit roadmap owns the overlay contract. The boundary findings own existing choices and evidence. Preserve the original v0.3.1 results as historical. ConfigKit v0.4.0 renames selected configuration profiles to overlays; do not rename PromptKit's profiles/, profiles_dir, or execution-profile fields.

All implementation in this plan stays under examples/library-adoption/, plus relevant roadmap/README updates. Keep the root Go module, production CLI/runtime, correction algorithms, and upstream libraries unchanged. Use the existing built-in PromptKit HTTP path with a controlled transport, synthetic credentials, and no live provider calls. Keep imports and secret-directory bootstrap disabled. This work does not introduce general environment overlays or backwards compatibility.

Unless explicitly stated otherwise, shell commands below run from examples/library-adoption/. Use an installed Go 1.25.5 or newer. Acquire dependencies with GOWORK=off go mod download all when needed; then use the offline command prefix below. Root go test ./... does not cover this nested module.

Stage 1: Upgrade the Prototype Dependency

Status: Complete (2026-09-24). Depends on: nothing. ConfigKit v0.4.0 pinned; the nested offline tests and demonstration passed with the expected 1/4/0/2 result.

Read the prototype README, go.mod, go.sum, and the current ConfigKit roadmap. Confirm the working tree and run the existing nested-module tests once as a baseline. Upgrade ConfigKit to the published v0.4.0 release using module tooling; leave PromptKit and Audita dependency versions unchanged. Review any indirect module changes and retain only changes needed by this upgrade. Do not add local replacements or an active Go workspace.

No source changes were needed in the earlier isolated review. If a change is now necessary, investigate and record the concrete reason instead of assuming that ConfigKit execution overlays and PromptKit profiles share an API.

Verification:

GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test ./...
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run .

Acceptance: the manifest and sums pin v0.4.0, tests pass, and the demonstration still yields one accepted correction, four attempts, zero transient retries, and two structural repairs. Update the README/findings to distinguish the new pin from historical v0.3.1 evidence. Do not claim overlay integration yet.

Stage 2: Add Overlay Loading and Configuration Fixtures

Status: Complete (2026-09-24). Depends on: Stage 1. On-disk overlay and profile fixtures, precedence, inheritance, rejection, and runtime assembly tests pass in the nested module (go test ./...).

Start with config.go, config_test.go, config.yaml, and profiles/. Extend loadConfig with an explicit overlay-selection argument or a small invocation options value. Pass it to ConfigKit Options.Overlay; selection does not belong in the persistent Config struct. Update existing callers to request no overlay without changing their behavior. Avoid a parallel merge implementation.

Add these prototype fixtures:

• overlays/testing.yml, selecting proposal-testing and assessment-testing through the existing two llm role fields. Do not put version, model settings, or ConfigKit directives in this overlay.
• Complete PromptKit profiles under profiles/ with those IDs and distinct synthetic model IDs, such as synthetic-grammar-testing and synthetic-review-testing. Both use the existing offline backend and keep the current positive repair policy and finite shared capacity.

Preserve base config.yaml as the no-overlay fixture, including its explicitly selected assessment profile. This makes the risk of accidentally leaving the validation role on its base model observable. Profile paths remain relative to main configuration according to the selected application contract, not relative to the overlay file.

Add focused configuration tests proving:

• Omitted selection, empty selection, and default select the base configuration; testing selects both new profiles.
• Explicit ConfigKit overrides win over overlay values. Only supplied flags will become overrides in Stage 3; parser defaults must not enter this layer.
• With validation selection absent from both base and overlay, validation inherits the final proposal selection, including an explicit override. An existing explicit base validation selection remains explicit unless replaced by overlay or CLI. Empty/null validation values retain the established rejection behavior.
• Runtime profile assembly resolves both selected profiles to the shared backend.

Use temporary on-disk layouts for overlay tests so ConfigKit owns file discovery. Existing byte-source tests can remain for cases that do not select an overlay; if a byte source needs an overlay, supply ConfigKit's supported filesystem context rather than relying on the process working directory.

Acceptance: the new fixtures load and assemble correctly, the focused precedence/ inheritance tests pass, and go test ./... in the nested module still passes. Do not yet claim that the testing models reached the HTTP transport.

Stage 3: Demonstrate and Verify Model Switching

Status: Complete (2026-09-24). Depends on: Stage 2. Nested tests inspect outbound base/testing/override model IDs, including repairs; all three documented offline demonstrations passed with one accepted correction and 4/0/2 attempts, retries, and repairs.

Start with main.go, offline.go, runtime.go, and workflow_test.go. Add minimal standard-library argument handling to the prototype executable:

• --overlay <name> selects the ConfigKit overlay.
• --proposal-profile <id> and --validation-profile <id> produce ConfigKit overrides only when explicitly supplied.
• Keep the existing fixture-based invocation from the prototype directory. There is no need to reproduce production file discovery or its subcommand tree here.

Use a small testable entry function if needed to check flag handling without repeated subprocess builds. Preserve ordinary flag error handling. Keep execution on the synthetic transport for every supported demonstration invocation; do not add a live-network mode. Extend that transport to recognize the testing models and preserve the initial-invalid-output/repair behavior for each role.

Add a compact safe selection summary to demonstration output: requested overlay and effective profile, backend, and model for each role, alongside the existing domain outcome and attempt/retry/repair totals. It must reflect the assembled configuration rather than echoing flag values. Inspection APIs and tests in the next stage suffice; do not create production config commands in this prototype.

Test through the assembled path and inspect actual outbound HTTP request bodies:

• No overlay sends the base proposal and assessment models.
• testing sends the testing proposal and assessment models, including repair requests, and still produces one accepted correction with two repairs.
• An explicitly supplied role-profile flag overrides that role's testing selection; the other role retains its overlay selection. An omitted flag preserves overlay selection. Match reported selections to observed outbound model IDs.

Use distinct models per role for these demonstration cases. Do not expand the fake provider into a general simulation framework. Reuse existing prompt/schema/domain helpers and existing capacity/observation coverage.

Acceptance: nested-module tests pass and these invocations visibly show the expected selections and successful domain result:

GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run .
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing --proposal-profile proposal

Update the prototype README with these implemented commands and fixture layout. State that synthetic testing IDs demonstrate operator-selectable routing, not measured price, speed, or model quality.

Stage 4: Verify Failure, Credential, and Inspection Boundaries

Status: Pending. Depends on: Stage 3.

Extend focused tests in the prototype; keep them offline and independent of host configuration and credentials. Exercise the consumer's assembled loader/runtime rather than duplicating ConfigKit's exhaustive parser or merge tests.

Required evidence:

• Missing and invalid overlay selections fail with ConfigKit classifications. Representative attempts to replace the schema version or introduce ConfigKit directives fail. Do not create application fallbacks for those failures.
• An overlay can change an ordinary non-profile setting, such as a backend limit, and the effective setting still undergoes application validation. This proves overlays have not been accidentally restricted to profile selection.
• A replaced credential reference resolves only its surviving replacement; an unresolved displaced reference does not make the load fail. A surviving missing or empty secret reference still fails. Explicit null retains unauthenticated backend semantics. Use synthetic environment names and values under test control.
• Switching role profiles alone does not remove other credential references still present in configuration. Document this distinction and test the relevant case; do not bypass ConfigKit validation for unselected backends as an incidental fix.
• WriteEffective and relevant explanation/metadata APIs identify overlay origin; a CLI replacement identifies CLI origin. Inspect selected PromptKit profiles from that same loaded result and verify model/backend agreement with runtime selection.
• Synthetic credential values do not appear in effective configuration, explanations, profile inspection, observations, or surfaced errors exercised by these scenarios. Confirm the controlled transport receives the effective synthetic credential.

Maintain the existing retry/repair/capacity tests. Add only the overlay-specific integration coverage needed to protect this boundary; do not repeat the complete scheduler matrix for each fixture.

Verification:

GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test -race ./...
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go vet ./...

Acceptance: all cases above have passing behavioral evidence, the existing suite remains green, and concurrent observation collection remains race-free. Record actual gaps instead of silently relaxing tests or claiming complete integration.

Stage 5: Record Overlay Findings and Executable Instructions

Status: Pending. Depends on: Stage 4.

This stage changes documentation, not production code. Read the new tests/results, then update the existing owners rather than creating another parallel design:

• Append a dated v0.4.0/overlay result section to library-boundary-findings.md. Record commands, toolchain/platform, outcomes, test-to-requirement coverage, limitations, and any unresolved gaps. Keep original v0.3.1 evidence identifiable.
• Update the prototype README to match the actual executable and fixtures; check every documented command. Explain the separate ConfigKit overlay and PromptKit profile directories and the credential behavior proved in Stage 4.
• Record the overlay implementation as verified in library-boundary.md and link the new evidence from library-adoption.md. The next unit still has its Stage 6 planning deliverable outstanding; keep production migration pending.

Acceptance: the README commands are reproducible, the new overlay evidence is clearly separated from historical results, and each Stage 2–4 requirement maps to a passing test or observed demonstration result. Verify affected links and git diff --check. Do not repeat unchanged code checks solely for this commit.

Stage 6: Prepare the Production Migration Plan and Close the Unit

Status: Pending. Depends on: Stage 5.

Expand configkit.md into a concrete, ordered production migration plan using the existing inventory and verified overlay evidence. Follow the development reading guide for configuration/CLI and LLM runtime work; use graph-first discovery to verify affected production components and their tests. This is a planning stage: do not implement the production migration or overwrite this execution plan with new production stages. Keep future work packages explicitly pending.

The production handoff must specify bounded work packages, dependencies, affected components, tests, and acceptance criteria for:

• The full version-2 configuration definition: existing domain fields, defaults, presence/null/zero behavior, role inheritance, overlays, secrets, and validation.
• File discovery and invocation parsing; --overlay behavior shared by process, config validation, and effective inspection; explicit-setting override mapping; removal of legacy environment overlays, aliases, and direct-key flags.
• Profile loading/inspection and configuration snapshots, including relative-path bases, secret redaction, provenance, and clear separation of ConfigKit settings from PromptKit execution controls.
• Runtime conversion and the temporary connection to the existing LLM runtime. For each new setting, state whether the interim runtime honors it or cannot yet do so. Decide the explicit interim behavior and its tests; do not silently map per-backend limits to the old global/role pools or pretend old retries implement PromptKit repair/zero semantics. Avoid recreating PromptKit inside the bridge.
• Removal of superseded configuration machinery, retained domain checks, maintained examples, canonical documentation updates, and the combined migration guide.
• The later PromptKit replacement/removal point and release follow-up. Preserve the agreed ordering and defer release compliance/publication to the final unit.

If an interim behavior cannot be settled from the evidence, identify that concrete open decision as a prerequisite to its production work package. Do not hide it in an adapter or describe an unresolved production step as implementation-ready.

Final audit: map every acceptance criterion in the prototype extension to actual files and command results. Verify new relative links/anchors, run git diff --check, and review that no root runtime/dependency or upstream files changed. Reuse passing checks for unchanged code; rerun the affected tests if this stage changes code or fixtures to correct a finding. A root build/test run is not a substitute for the nested-module checks, and this documentation stage adds no release gate.

Mark this prototype-extension/planning unit complete in library-boundary.md only after the audit passes, and link the production handoff from library-adoption.md. Production implementation remains pending.

Acceptance: a reviewer can reproduce base/testing/override selection offline, trace each new claim to evidence, and hand the production ConfigKit work to a coding agent in bounded units. Commit the completed documentation, report remaining limitations and working-tree state, and stop at the end of this plan.