diff --git a/docs/roadmap/configkit.md b/docs/roadmap/configkit.md index f7f93f4..6404485 100644 --- a/docs/roadmap/configkit.md +++ b/docs/roadmap/configkit.md @@ -1,6 +1,6 @@ # ConfigKit Adoption Roadmap -**Last updated:** 2026-09-23 +**Last updated:** 2026-09-24 **Status:** High-level, provisional planning only. Adoption direction is agreed; this document does not authorize migration implementation. Revisit the scope, @@ -103,6 +103,11 @@ effective-config inspection, selected-profile inspection, and processing must al use the same selected overlay. Preserve overlay provenance in configuration explanations and diagnostics while retaining secret redaction. +The immediate prototype extension and production-planning handoff are specified +in the [prototype roadmap](library-boundary.md#next-unit-testing-overlays-and-production-migration-plan) +and its ordered [implementation plan](implementation.md). Production CLI changes +below remain subsequent migration work. + Implementation and acceptance work: - Upgrade the isolated prototype to ConfigKit v0.4.0 and add a testing-overlay diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..f4b833d --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,289 @@ +# Implementation Plan: Testing Overlays and Migration Handoff + +**Last updated:** 2026-09-24 + +**Status:** Planned. All stages below are pending. This plan implements the next +unit in the [prototype roadmap](library-boundary.md#next-unit-testing-overlays-and-production-migration-plan), +not the production ConfigKit or PromptKit migrations. + +Execute Stage 1 through Stage 6 in order. Each stage is sized as a separate coding +prompt for a gpt-5.6-terra agent and ends with a reviewable result. When assigned +one stage, finish that stage and report the next stage; when assigned the entire +plan, continue through all stages. Do not restart completed work: inspect current +files, recorded results, and Git state before deciding what remains. + +Before each stage, follow `AGENTS.md`, [development guidance](../development.md), +and the [agent policy](../policy/agents.md). Preserve unrelated changes. Use the +knowledge graph for code discovery; examples may be excluded from its index, in +which case inspect the known prototype files directly. Follow the repository's +testing and documentation policies. Complete relevant checks and make a plain +English local commit for each coherent stage under the standing commit policy. +No push, tag, release, or upstream-library edit is included. + +Update each stage's status line with its result and verification evidence when +finished; keep detailed design/results in their owning roadmap documents. Do not +mark a stage complete while required checks are failing. Fix in-scope failures; +record any genuine upstream gap and its effect on later stages before continuing +work that depends on it. Routine implementation choices do not need fresh approval. + +The [ConfigKit roadmap](configkit.md#selected-overlays-for-testing) owns the overlay +contract. The [boundary findings](library-boundary-findings.md) own existing choices +and evidence. Preserve the original v0.3.1 results as historical. ConfigKit v0.4.0 +renames selected configuration profiles to overlays; do not rename PromptKit's +`profiles/`, `profiles_dir`, or execution-profile fields. + +All implementation in this plan stays under `examples/library-adoption/`, plus +relevant roadmap/README updates. Keep the root Go module, production CLI/runtime, +correction algorithms, and upstream libraries unchanged. Use the existing built-in +PromptKit HTTP path with a controlled transport, synthetic credentials, and no live +provider calls. Keep imports and secret-directory bootstrap disabled. This work +does not introduce general environment overlays or backwards compatibility. + +Unless explicitly stated otherwise, shell commands below run from +`examples/library-adoption/`. Use an installed Go 1.25.5 or newer. Acquire dependencies +with `GOWORK=off go mod download all` when needed; then use the offline command +prefix below. Root `go test ./...` does not cover this nested module. + +## Stage 1: Upgrade the Prototype Dependency + +**Status:** Pending. **Depends on:** nothing. + +Read the prototype README, `go.mod`, `go.sum`, and the current ConfigKit roadmap. +Confirm the working tree and run the existing nested-module tests once as a +baseline. Upgrade ConfigKit to the published v0.4.0 release using module tooling; +leave PromptKit and Audita dependency versions unchanged. Review any indirect +module changes and retain only changes needed by this upgrade. Do not add local +replacements or an active Go workspace. + +No source changes were needed in the earlier isolated review. If a change is now +necessary, investigate and record the concrete reason instead of assuming that +ConfigKit execution overlays and PromptKit profiles share an API. + +Verification: + +```sh +GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test ./... +GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . +``` + +Acceptance: the manifest and sums pin v0.4.0, tests pass, and the demonstration +still yields one accepted correction, four attempts, zero transient retries, and +two structural repairs. Update the README/findings to distinguish the new pin from +historical v0.3.1 evidence. Do not claim overlay integration yet. + +## Stage 2: Add Overlay Loading and Configuration Fixtures + +**Status:** Pending. **Depends on:** Stage 1. + +Start with `config.go`, `config_test.go`, `config.yaml`, and `profiles/`. Extend +`loadConfig` with an explicit overlay-selection argument or a small invocation +options value. Pass it to ConfigKit `Options.Overlay`; selection does not belong +in the persistent `Config` struct. Update existing callers to request no overlay +without changing their behavior. Avoid a parallel merge implementation. + +Add these prototype fixtures: + +- `overlays/testing.yml`, selecting `proposal-testing` and `assessment-testing` + through the existing two `llm` role fields. Do not put `version`, model settings, + or ConfigKit directives in this overlay. +- Complete PromptKit profiles under `profiles/` with those IDs and distinct + synthetic model IDs, such as `synthetic-grammar-testing` and + `synthetic-review-testing`. Both use the existing `offline` backend and keep the + current positive repair policy and finite shared capacity. + +Preserve base `config.yaml` as the no-overlay fixture, including its explicitly +selected assessment profile. This makes the risk of accidentally leaving the +validation role on its base model observable. Profile paths remain relative to +main configuration according to the selected application contract, not relative +to the overlay file. + +Add focused configuration tests proving: + +- Omitted selection, empty selection, and `default` select the base configuration; + `testing` selects both new profiles. +- Explicit ConfigKit overrides win over overlay values. Only supplied flags will + become overrides in Stage 3; parser defaults must not enter this layer. +- With validation selection absent from both base and overlay, validation inherits + the final proposal selection, including an explicit override. An existing + explicit base validation selection remains explicit unless replaced by overlay + or CLI. Empty/null validation values retain the established rejection behavior. +- Runtime profile assembly resolves both selected profiles to the shared backend. + +Use temporary on-disk layouts for overlay tests so ConfigKit owns file discovery. +Existing byte-source tests can remain for cases that do not select an overlay; +if a byte source needs an overlay, supply ConfigKit's supported filesystem context +rather than relying on the process working directory. + +Acceptance: the new fixtures load and assemble correctly, the focused precedence/ +inheritance tests pass, and `go test ./...` in the nested module still passes. +Do not yet claim that the testing models reached the HTTP transport. + +## Stage 3: Demonstrate and Verify Model Switching + +**Status:** Pending. **Depends on:** Stage 2. + +Start with `main.go`, `offline.go`, `runtime.go`, and `workflow_test.go`. Add minimal +standard-library argument handling to the prototype executable: + +- `--overlay ` selects the ConfigKit overlay. +- `--proposal-profile ` and `--validation-profile ` produce ConfigKit + overrides only when explicitly supplied. +- Keep the existing fixture-based invocation from the prototype directory. There + is no need to reproduce production file discovery or its subcommand tree here. + +Use a small testable entry function if needed to check flag handling without +repeated subprocess builds. Preserve ordinary flag error handling. Keep execution +on the synthetic transport for every supported demonstration invocation; do not +add a live-network mode. Extend that transport to recognize the testing models and +preserve the initial-invalid-output/repair behavior for each role. + +Add a compact safe selection summary to demonstration output: requested overlay +and effective profile, backend, and model for each role, alongside the existing +domain outcome and attempt/retry/repair totals. It must reflect the assembled +configuration rather than echoing flag values. Inspection APIs and tests in the +next stage suffice; do not create production config commands in this prototype. + +Test through the assembled path and inspect actual outbound HTTP request bodies: + +- No overlay sends the base proposal and assessment models. +- `testing` sends the testing proposal and assessment models, including repair + requests, and still produces one accepted correction with two repairs. +- An explicitly supplied role-profile flag overrides that role's testing selection; + the other role retains its overlay selection. An omitted flag preserves overlay + selection. Match reported selections to observed outbound model IDs. + +Use distinct models per role for these demonstration cases. Do not expand the fake +provider into a general simulation framework. Reuse existing prompt/schema/domain +helpers and existing capacity/observation coverage. + +Acceptance: nested-module tests pass and these invocations visibly show the expected +selections and successful domain result: + +```sh +GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . +GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing +GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing --proposal-profile proposal +``` + +Update the prototype README with these implemented commands and fixture layout. +State that synthetic testing IDs demonstrate operator-selectable routing, not +measured price, speed, or model quality. + +## Stage 4: Verify Failure, Credential, and Inspection Boundaries + +**Status:** Pending. **Depends on:** Stage 3. + +Extend focused tests in the prototype; keep them offline and independent of host +configuration and credentials. Exercise the consumer's assembled loader/runtime +rather than duplicating ConfigKit's exhaustive parser or merge tests. + +Required evidence: + +- Missing and invalid overlay selections fail with ConfigKit classifications. + Representative attempts to replace the schema version or introduce ConfigKit + directives fail. Do not create application fallbacks for those failures. +- An overlay can change an ordinary non-profile setting, such as a backend limit, + and the effective setting still undergoes application validation. This proves + overlays have not been accidentally restricted to profile selection. +- A replaced credential reference resolves only its surviving replacement; an + unresolved displaced reference does not make the load fail. A surviving missing + or empty secret reference still fails. Explicit null retains unauthenticated + backend semantics. Use synthetic environment names and values under test control. +- Switching role profiles alone does not remove other credential references still + present in configuration. Document this distinction and test the relevant case; + do not bypass ConfigKit validation for unselected backends as an incidental fix. +- `WriteEffective` and relevant explanation/metadata APIs identify overlay origin; + a CLI replacement identifies CLI origin. Inspect selected PromptKit profiles from + that same loaded result and verify model/backend agreement with runtime selection. +- Synthetic credential values do not appear in effective configuration, explanations, + profile inspection, observations, or surfaced errors exercised by these scenarios. + Confirm the controlled transport receives the effective synthetic credential. + +Maintain the existing retry/repair/capacity tests. Add only the overlay-specific +integration coverage needed to protect this boundary; do not repeat the complete +scheduler matrix for each fixture. + +Verification: + +```sh +GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test -race ./... +GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go vet ./... +``` + +Acceptance: all cases above have passing behavioral evidence, the existing suite +remains green, and concurrent observation collection remains race-free. Record +actual gaps instead of silently relaxing tests or claiming complete integration. + +## Stage 5: Record Overlay Findings and Executable Instructions + +**Status:** Pending. **Depends on:** Stage 4. + +This stage changes documentation, not production code. Read the new tests/results, +then update the existing owners rather than creating another parallel design: + +- Append a dated v0.4.0/overlay result section to `library-boundary-findings.md`. + Record commands, toolchain/platform, outcomes, test-to-requirement coverage, + limitations, and any unresolved gaps. Keep original v0.3.1 evidence identifiable. +- Update the prototype README to match the actual executable and fixtures; check + every documented command. Explain the separate ConfigKit overlay and PromptKit + profile directories and the credential behavior proved in Stage 4. +- Record the overlay implementation as verified in `library-boundary.md` and link + the new evidence from `library-adoption.md`. The next unit still has its Stage 6 + planning deliverable outstanding; keep production migration pending. + +Acceptance: the README commands are reproducible, the new overlay evidence is +clearly separated from historical results, and each Stage 2–4 requirement maps +to a passing test or observed demonstration result. Verify affected links and +`git diff --check`. Do not repeat unchanged code checks solely for this commit. + +## Stage 6: Prepare the Production Migration Plan and Close the Unit + +**Status:** Pending. **Depends on:** Stage 5. + +Expand `configkit.md` into a concrete, ordered production migration plan using +the existing inventory and verified overlay evidence. Follow the development +reading guide for configuration/CLI and LLM runtime work; use graph-first discovery +to verify affected production components and their tests. This is a planning stage: +do not implement the production migration or overwrite this execution plan with +new production stages. Keep future work packages explicitly pending. + +The production handoff must specify bounded work packages, dependencies, affected +components, tests, and acceptance criteria for: + +- The full version-2 configuration definition: existing domain fields, defaults, + presence/null/zero behavior, role inheritance, overlays, secrets, and validation. +- File discovery and invocation parsing; `--overlay` behavior shared by process, + config validation, and effective inspection; explicit-setting override mapping; + removal of legacy environment overlays, aliases, and direct-key flags. +- Profile loading/inspection and configuration snapshots, including relative-path + bases, secret redaction, provenance, and clear separation of ConfigKit settings + from PromptKit execution controls. +- Runtime conversion and the temporary connection to the existing LLM runtime. + For each new setting, state whether the interim runtime honors it or cannot yet + do so. Decide the explicit interim behavior and its tests; do not silently map + per-backend limits to the old global/role pools or pretend old retries implement + PromptKit repair/zero semantics. Avoid recreating PromptKit inside the bridge. +- Removal of superseded configuration machinery, retained domain checks, maintained + examples, canonical documentation updates, and the combined migration guide. +- The later PromptKit replacement/removal point and release follow-up. Preserve + the agreed ordering and defer release compliance/publication to the final unit. + +If an interim behavior cannot be settled from the evidence, identify that concrete +open decision as a prerequisite to its production work package. Do not hide it in +an adapter or describe an unresolved production step as implementation-ready. + +Final audit: map every acceptance criterion in the prototype extension to actual +files and command results. Verify new relative links/anchors, run `git diff --check`, +and review that no root runtime/dependency or upstream files changed. Reuse passing +checks for unchanged code; rerun the affected tests if this stage changes code or +fixtures to correct a finding. A root build/test run is not a substitute for the +nested-module checks, and this documentation stage adds no release gate. + +Mark this prototype-extension/planning unit complete in `library-boundary.md` only +after the audit passes, and link the production handoff from `library-adoption.md`. +Production implementation remains pending. + +Acceptance: a reviewer can reproduce base/testing/override selection offline, +trace each new claim to evidence, and hand the production ConfigKit work to a coding +agent in bounded units. Commit the completed documentation, report remaining +limitations and working-tree state, and stop at the end of this plan. diff --git a/docs/roadmap/library-boundary.md b/docs/roadmap/library-boundary.md index 8a580ee..8e5c918 100644 --- a/docs/roadmap/library-boundary.md +++ b/docs/roadmap/library-boundary.md @@ -1,17 +1,67 @@ # Library Adoption: Shared Boundary and Prototype -**Last updated:** 2026-09-23 +**Last updated:** 2026-09-24 -**Status:** Completed bounded boundary-design and offline-prototype unit. +**Status:** Original boundary-design and offline-prototype unit completed; +overlay extension and production-planning handoff planned. Production migration has not started. The work packages below retain the original scope; [decisions, evidence, limitations, and handoff](library-boundary-findings.md) record the outcome. The [executable prototype](../../examples/library-adoption/README.md) is isolated from production. No publication or upstream changes were made. -**Subsequent scope decision:** Enable ConfigKit v0.4.0 selected overlays for testing -in the next configuration unit. The [overlay plan](configkit.md#selected-overlays-for-testing) -supersedes the original optional-feature exclusion below; the completed prototype -has not yet exercised overlays. +## Next Unit: Testing Overlays and Production Migration Plan + +**Status:** Planned; not implemented. Execute the ordered stages in +[implementation.md](implementation.md). The original work packages below remain +historical scope for the completed first unit. + +Extend the existing prototype to prove the [selected-overlay contract](configkit.md#selected-overlays-for-testing) +with ConfigKit v0.4.0, then turn the verified findings into a detailed production +ConfigKit migration plan. Do not repeat the original inventory or redesign the +established library boundary without new evidence. + +Required work: + +- Upgrade only the nested prototype module from ConfigKit v0.3.1 to v0.4.0. +- Pass an explicit overlay selection through ConfigKit and add `overlays/testing.yml` + plus two PromptKit testing profiles. Select both roles and retain one shared + finite backend. Synthetic model names demonstrate routing; they do not benchmark + real model cost, speed, or quality. +- Make the offline executable accept overlay and explicit role-profile selections, + and show the selected models alongside domain outcomes and retry/repair counts. +- Verify base/overlay/CLI precedence, role inheritance, invalid selections, + surviving credential references, safe inspection, and overlay provenance. + Prove the actual HTTP request models for both roles; loading changed strings + alone is insufficient evidence. +- Record new results separately from the historical v0.3.1 findings. Expand the + existing ConfigKit roadmap into a concrete production implementation handoff, + including the interim runtime boundary and removal of legacy configuration. + +Target end state: reviewers can run the prototype with and without the testing +selection and see both LLM roles switch models. The offline test suite establishes +that selection, inspection, and execution agree, and the production ConfigKit work +has bounded tasks with explicit acceptance criteria. + +Acceptance criteria: + +- The published v0.4.0 dependency is pinned without local replacements; the original + no-overlay path and existing integration tests continue to pass. +- `testing` switches both roles, explicit profile flags override its selections, + and omission/empty/`default` retains base selection according to ConfigKit. +- Invalid overlays fail clearly; inheritance and secret resolution use surviving + effective settings. Inspection records overlay origin without disclosing keys. +- Both selected testing models reach the controlled HTTP transport; structural + repair, shared capacity, observations, and Audita domain checks remain active. +- Updated examples, executable instructions, findings, and a detailed production + migration plan are reviewable. Exact verification commands and limitations are + recorded, including the fact that root tests do not cover the nested module. +- Production code, root dependencies, and upstream libraries remain unchanged. + No release or live provider call is part of this unit. + +This unit ends with a production migration **plan**, not production implementation. +The ConfigKit → PromptKit → combined release sequence remains unchanged. Imports +and secret-directory bootstrap remain disabled; PromptKit execution profiles keep +their existing names and meaning. ## Objective