Plan staged implementation of the testing overlay prototype
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# ConfigKit Adoption Roadmap
|
||||
|
||||
**Last updated:** 2026-09-23
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** High-level, provisional planning only. Adoption direction is agreed;
|
||||
this document does not authorize migration implementation. Revisit the scope,
|
||||
@@ -103,6 +103,11 @@ effective-config inspection, selected-profile inspection, and processing must al
|
||||
use the same selected overlay. Preserve overlay provenance in configuration
|
||||
explanations and diagnostics while retaining secret redaction.
|
||||
|
||||
The immediate prototype extension and production-planning handoff are specified
|
||||
in the [prototype roadmap](library-boundary.md#next-unit-testing-overlays-and-production-migration-plan)
|
||||
and its ordered [implementation plan](implementation.md). Production CLI changes
|
||||
below remain subsequent migration work.
|
||||
|
||||
Implementation and acceptance work:
|
||||
|
||||
- Upgrade the isolated prototype to ConfigKit v0.4.0 and add a testing-overlay
|
||||
|
||||
289
docs/roadmap/implementation.md
Normal file
289
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,289 @@
|
||||
# Implementation Plan: Testing Overlays and Migration Handoff
|
||||
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Planned. All stages below are pending. This plan implements the next
|
||||
unit in the [prototype roadmap](library-boundary.md#next-unit-testing-overlays-and-production-migration-plan),
|
||||
not the production ConfigKit or PromptKit migrations.
|
||||
|
||||
Execute Stage 1 through Stage 6 in order. Each stage is sized as a separate coding
|
||||
prompt for a gpt-5.6-terra agent and ends with a reviewable result. When assigned
|
||||
one stage, finish that stage and report the next stage; when assigned the entire
|
||||
plan, continue through all stages. Do not restart completed work: inspect current
|
||||
files, recorded results, and Git state before deciding what remains.
|
||||
|
||||
Before each stage, follow `AGENTS.md`, [development guidance](../development.md),
|
||||
and the [agent policy](../policy/agents.md). Preserve unrelated changes. Use the
|
||||
knowledge graph for code discovery; examples may be excluded from its index, in
|
||||
which case inspect the known prototype files directly. Follow the repository's
|
||||
testing and documentation policies. Complete relevant checks and make a plain
|
||||
English local commit for each coherent stage under the standing commit policy.
|
||||
No push, tag, release, or upstream-library edit is included.
|
||||
|
||||
Update each stage's status line with its result and verification evidence when
|
||||
finished; keep detailed design/results in their owning roadmap documents. Do not
|
||||
mark a stage complete while required checks are failing. Fix in-scope failures;
|
||||
record any genuine upstream gap and its effect on later stages before continuing
|
||||
work that depends on it. Routine implementation choices do not need fresh approval.
|
||||
|
||||
The [ConfigKit roadmap](configkit.md#selected-overlays-for-testing) owns the overlay
|
||||
contract. The [boundary findings](library-boundary-findings.md) own existing choices
|
||||
and evidence. Preserve the original v0.3.1 results as historical. ConfigKit v0.4.0
|
||||
renames selected configuration profiles to overlays; do not rename PromptKit's
|
||||
`profiles/`, `profiles_dir`, or execution-profile fields.
|
||||
|
||||
All implementation in this plan stays under `examples/library-adoption/`, plus
|
||||
relevant roadmap/README updates. Keep the root Go module, production CLI/runtime,
|
||||
correction algorithms, and upstream libraries unchanged. Use the existing built-in
|
||||
PromptKit HTTP path with a controlled transport, synthetic credentials, and no live
|
||||
provider calls. Keep imports and secret-directory bootstrap disabled. This work
|
||||
does not introduce general environment overlays or backwards compatibility.
|
||||
|
||||
Unless explicitly stated otherwise, shell commands below run from
|
||||
`examples/library-adoption/`. Use an installed Go 1.25.5 or newer. Acquire dependencies
|
||||
with `GOWORK=off go mod download all` when needed; then use the offline command
|
||||
prefix below. Root `go test ./...` does not cover this nested module.
|
||||
|
||||
## Stage 1: Upgrade the Prototype Dependency
|
||||
|
||||
**Status:** Pending. **Depends on:** nothing.
|
||||
|
||||
Read the prototype README, `go.mod`, `go.sum`, and the current ConfigKit roadmap.
|
||||
Confirm the working tree and run the existing nested-module tests once as a
|
||||
baseline. Upgrade ConfigKit to the published v0.4.0 release using module tooling;
|
||||
leave PromptKit and Audita dependency versions unchanged. Review any indirect
|
||||
module changes and retain only changes needed by this upgrade. Do not add local
|
||||
replacements or an active Go workspace.
|
||||
|
||||
No source changes were needed in the earlier isolated review. If a change is now
|
||||
necessary, investigate and record the concrete reason instead of assuming that
|
||||
ConfigKit execution overlays and PromptKit profiles share an API.
|
||||
|
||||
Verification:
|
||||
|
||||
```sh
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test ./...
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run .
|
||||
```
|
||||
|
||||
Acceptance: the manifest and sums pin v0.4.0, tests pass, and the demonstration
|
||||
still yields one accepted correction, four attempts, zero transient retries, and
|
||||
two structural repairs. Update the README/findings to distinguish the new pin from
|
||||
historical v0.3.1 evidence. Do not claim overlay integration yet.
|
||||
|
||||
## Stage 2: Add Overlay Loading and Configuration Fixtures
|
||||
|
||||
**Status:** Pending. **Depends on:** Stage 1.
|
||||
|
||||
Start with `config.go`, `config_test.go`, `config.yaml`, and `profiles/`. Extend
|
||||
`loadConfig` with an explicit overlay-selection argument or a small invocation
|
||||
options value. Pass it to ConfigKit `Options.Overlay`; selection does not belong
|
||||
in the persistent `Config` struct. Update existing callers to request no overlay
|
||||
without changing their behavior. Avoid a parallel merge implementation.
|
||||
|
||||
Add these prototype fixtures:
|
||||
|
||||
- `overlays/testing.yml`, selecting `proposal-testing` and `assessment-testing`
|
||||
through the existing two `llm` role fields. Do not put `version`, model settings,
|
||||
or ConfigKit directives in this overlay.
|
||||
- Complete PromptKit profiles under `profiles/` with those IDs and distinct
|
||||
synthetic model IDs, such as `synthetic-grammar-testing` and
|
||||
`synthetic-review-testing`. Both use the existing `offline` backend and keep the
|
||||
current positive repair policy and finite shared capacity.
|
||||
|
||||
Preserve base `config.yaml` as the no-overlay fixture, including its explicitly
|
||||
selected assessment profile. This makes the risk of accidentally leaving the
|
||||
validation role on its base model observable. Profile paths remain relative to
|
||||
main configuration according to the selected application contract, not relative
|
||||
to the overlay file.
|
||||
|
||||
Add focused configuration tests proving:
|
||||
|
||||
- Omitted selection, empty selection, and `default` select the base configuration;
|
||||
`testing` selects both new profiles.
|
||||
- Explicit ConfigKit overrides win over overlay values. Only supplied flags will
|
||||
become overrides in Stage 3; parser defaults must not enter this layer.
|
||||
- With validation selection absent from both base and overlay, validation inherits
|
||||
the final proposal selection, including an explicit override. An existing
|
||||
explicit base validation selection remains explicit unless replaced by overlay
|
||||
or CLI. Empty/null validation values retain the established rejection behavior.
|
||||
- Runtime profile assembly resolves both selected profiles to the shared backend.
|
||||
|
||||
Use temporary on-disk layouts for overlay tests so ConfigKit owns file discovery.
|
||||
Existing byte-source tests can remain for cases that do not select an overlay;
|
||||
if a byte source needs an overlay, supply ConfigKit's supported filesystem context
|
||||
rather than relying on the process working directory.
|
||||
|
||||
Acceptance: the new fixtures load and assemble correctly, the focused precedence/
|
||||
inheritance tests pass, and `go test ./...` in the nested module still passes.
|
||||
Do not yet claim that the testing models reached the HTTP transport.
|
||||
|
||||
## Stage 3: Demonstrate and Verify Model Switching
|
||||
|
||||
**Status:** Pending. **Depends on:** Stage 2.
|
||||
|
||||
Start with `main.go`, `offline.go`, `runtime.go`, and `workflow_test.go`. Add minimal
|
||||
standard-library argument handling to the prototype executable:
|
||||
|
||||
- `--overlay <name>` selects the ConfigKit overlay.
|
||||
- `--proposal-profile <id>` and `--validation-profile <id>` produce ConfigKit
|
||||
overrides only when explicitly supplied.
|
||||
- Keep the existing fixture-based invocation from the prototype directory. There
|
||||
is no need to reproduce production file discovery or its subcommand tree here.
|
||||
|
||||
Use a small testable entry function if needed to check flag handling without
|
||||
repeated subprocess builds. Preserve ordinary flag error handling. Keep execution
|
||||
on the synthetic transport for every supported demonstration invocation; do not
|
||||
add a live-network mode. Extend that transport to recognize the testing models and
|
||||
preserve the initial-invalid-output/repair behavior for each role.
|
||||
|
||||
Add a compact safe selection summary to demonstration output: requested overlay
|
||||
and effective profile, backend, and model for each role, alongside the existing
|
||||
domain outcome and attempt/retry/repair totals. It must reflect the assembled
|
||||
configuration rather than echoing flag values. Inspection APIs and tests in the
|
||||
next stage suffice; do not create production config commands in this prototype.
|
||||
|
||||
Test through the assembled path and inspect actual outbound HTTP request bodies:
|
||||
|
||||
- No overlay sends the base proposal and assessment models.
|
||||
- `testing` sends the testing proposal and assessment models, including repair
|
||||
requests, and still produces one accepted correction with two repairs.
|
||||
- An explicitly supplied role-profile flag overrides that role's testing selection;
|
||||
the other role retains its overlay selection. An omitted flag preserves overlay
|
||||
selection. Match reported selections to observed outbound model IDs.
|
||||
|
||||
Use distinct models per role for these demonstration cases. Do not expand the fake
|
||||
provider into a general simulation framework. Reuse existing prompt/schema/domain
|
||||
helpers and existing capacity/observation coverage.
|
||||
|
||||
Acceptance: nested-module tests pass and these invocations visibly show the expected
|
||||
selections and successful domain result:
|
||||
|
||||
```sh
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run .
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing --proposal-profile proposal
|
||||
```
|
||||
|
||||
Update the prototype README with these implemented commands and fixture layout.
|
||||
State that synthetic testing IDs demonstrate operator-selectable routing, not
|
||||
measured price, speed, or model quality.
|
||||
|
||||
## Stage 4: Verify Failure, Credential, and Inspection Boundaries
|
||||
|
||||
**Status:** Pending. **Depends on:** Stage 3.
|
||||
|
||||
Extend focused tests in the prototype; keep them offline and independent of host
|
||||
configuration and credentials. Exercise the consumer's assembled loader/runtime
|
||||
rather than duplicating ConfigKit's exhaustive parser or merge tests.
|
||||
|
||||
Required evidence:
|
||||
|
||||
- Missing and invalid overlay selections fail with ConfigKit classifications.
|
||||
Representative attempts to replace the schema version or introduce ConfigKit
|
||||
directives fail. Do not create application fallbacks for those failures.
|
||||
- An overlay can change an ordinary non-profile setting, such as a backend limit,
|
||||
and the effective setting still undergoes application validation. This proves
|
||||
overlays have not been accidentally restricted to profile selection.
|
||||
- A replaced credential reference resolves only its surviving replacement; an
|
||||
unresolved displaced reference does not make the load fail. A surviving missing
|
||||
or empty secret reference still fails. Explicit null retains unauthenticated
|
||||
backend semantics. Use synthetic environment names and values under test control.
|
||||
- Switching role profiles alone does not remove other credential references still
|
||||
present in configuration. Document this distinction and test the relevant case;
|
||||
do not bypass ConfigKit validation for unselected backends as an incidental fix.
|
||||
- `WriteEffective` and relevant explanation/metadata APIs identify overlay origin;
|
||||
a CLI replacement identifies CLI origin. Inspect selected PromptKit profiles from
|
||||
that same loaded result and verify model/backend agreement with runtime selection.
|
||||
- Synthetic credential values do not appear in effective configuration, explanations,
|
||||
profile inspection, observations, or surfaced errors exercised by these scenarios.
|
||||
Confirm the controlled transport receives the effective synthetic credential.
|
||||
|
||||
Maintain the existing retry/repair/capacity tests. Add only the overlay-specific
|
||||
integration coverage needed to protect this boundary; do not repeat the complete
|
||||
scheduler matrix for each fixture.
|
||||
|
||||
Verification:
|
||||
|
||||
```sh
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test -race ./...
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go vet ./...
|
||||
```
|
||||
|
||||
Acceptance: all cases above have passing behavioral evidence, the existing suite
|
||||
remains green, and concurrent observation collection remains race-free. Record
|
||||
actual gaps instead of silently relaxing tests or claiming complete integration.
|
||||
|
||||
## Stage 5: Record Overlay Findings and Executable Instructions
|
||||
|
||||
**Status:** Pending. **Depends on:** Stage 4.
|
||||
|
||||
This stage changes documentation, not production code. Read the new tests/results,
|
||||
then update the existing owners rather than creating another parallel design:
|
||||
|
||||
- Append a dated v0.4.0/overlay result section to `library-boundary-findings.md`.
|
||||
Record commands, toolchain/platform, outcomes, test-to-requirement coverage,
|
||||
limitations, and any unresolved gaps. Keep original v0.3.1 evidence identifiable.
|
||||
- Update the prototype README to match the actual executable and fixtures; check
|
||||
every documented command. Explain the separate ConfigKit overlay and PromptKit
|
||||
profile directories and the credential behavior proved in Stage 4.
|
||||
- Record the overlay implementation as verified in `library-boundary.md` and link
|
||||
the new evidence from `library-adoption.md`. The next unit still has its Stage 6
|
||||
planning deliverable outstanding; keep production migration pending.
|
||||
|
||||
Acceptance: the README commands are reproducible, the new overlay evidence is
|
||||
clearly separated from historical results, and each Stage 2–4 requirement maps
|
||||
to a passing test or observed demonstration result. Verify affected links and
|
||||
`git diff --check`. Do not repeat unchanged code checks solely for this commit.
|
||||
|
||||
## Stage 6: Prepare the Production Migration Plan and Close the Unit
|
||||
|
||||
**Status:** Pending. **Depends on:** Stage 5.
|
||||
|
||||
Expand `configkit.md` into a concrete, ordered production migration plan using
|
||||
the existing inventory and verified overlay evidence. Follow the development
|
||||
reading guide for configuration/CLI and LLM runtime work; use graph-first discovery
|
||||
to verify affected production components and their tests. This is a planning stage:
|
||||
do not implement the production migration or overwrite this execution plan with
|
||||
new production stages. Keep future work packages explicitly pending.
|
||||
|
||||
The production handoff must specify bounded work packages, dependencies, affected
|
||||
components, tests, and acceptance criteria for:
|
||||
|
||||
- The full version-2 configuration definition: existing domain fields, defaults,
|
||||
presence/null/zero behavior, role inheritance, overlays, secrets, and validation.
|
||||
- File discovery and invocation parsing; `--overlay` behavior shared by process,
|
||||
config validation, and effective inspection; explicit-setting override mapping;
|
||||
removal of legacy environment overlays, aliases, and direct-key flags.
|
||||
- Profile loading/inspection and configuration snapshots, including relative-path
|
||||
bases, secret redaction, provenance, and clear separation of ConfigKit settings
|
||||
from PromptKit execution controls.
|
||||
- Runtime conversion and the temporary connection to the existing LLM runtime.
|
||||
For each new setting, state whether the interim runtime honors it or cannot yet
|
||||
do so. Decide the explicit interim behavior and its tests; do not silently map
|
||||
per-backend limits to the old global/role pools or pretend old retries implement
|
||||
PromptKit repair/zero semantics. Avoid recreating PromptKit inside the bridge.
|
||||
- Removal of superseded configuration machinery, retained domain checks, maintained
|
||||
examples, canonical documentation updates, and the combined migration guide.
|
||||
- The later PromptKit replacement/removal point and release follow-up. Preserve
|
||||
the agreed ordering and defer release compliance/publication to the final unit.
|
||||
|
||||
If an interim behavior cannot be settled from the evidence, identify that concrete
|
||||
open decision as a prerequisite to its production work package. Do not hide it in
|
||||
an adapter or describe an unresolved production step as implementation-ready.
|
||||
|
||||
Final audit: map every acceptance criterion in the prototype extension to actual
|
||||
files and command results. Verify new relative links/anchors, run `git diff --check`,
|
||||
and review that no root runtime/dependency or upstream files changed. Reuse passing
|
||||
checks for unchanged code; rerun the affected tests if this stage changes code or
|
||||
fixtures to correct a finding. A root build/test run is not a substitute for the
|
||||
nested-module checks, and this documentation stage adds no release gate.
|
||||
|
||||
Mark this prototype-extension/planning unit complete in `library-boundary.md` only
|
||||
after the audit passes, and link the production handoff from `library-adoption.md`.
|
||||
Production implementation remains pending.
|
||||
|
||||
Acceptance: a reviewer can reproduce base/testing/override selection offline,
|
||||
trace each new claim to evidence, and hand the production ConfigKit work to a coding
|
||||
agent in bounded units. Commit the completed documentation, report remaining
|
||||
limitations and working-tree state, and stop at the end of this plan.
|
||||
@@ -1,17 +1,67 @@
|
||||
# Library Adoption: Shared Boundary and Prototype
|
||||
|
||||
**Last updated:** 2026-09-23
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Completed bounded boundary-design and offline-prototype unit.
|
||||
**Status:** Original boundary-design and offline-prototype unit completed;
|
||||
overlay extension and production-planning handoff planned.
|
||||
Production migration has not started. The work packages below retain the original
|
||||
scope; [decisions, evidence, limitations, and handoff](library-boundary-findings.md)
|
||||
record the outcome. The [executable prototype](../../examples/library-adoption/README.md)
|
||||
is isolated from production. No publication or upstream changes were made.
|
||||
|
||||
**Subsequent scope decision:** Enable ConfigKit v0.4.0 selected overlays for testing
|
||||
in the next configuration unit. The [overlay plan](configkit.md#selected-overlays-for-testing)
|
||||
supersedes the original optional-feature exclusion below; the completed prototype
|
||||
has not yet exercised overlays.
|
||||
## Next Unit: Testing Overlays and Production Migration Plan
|
||||
|
||||
**Status:** Planned; not implemented. Execute the ordered stages in
|
||||
[implementation.md](implementation.md). The original work packages below remain
|
||||
historical scope for the completed first unit.
|
||||
|
||||
Extend the existing prototype to prove the [selected-overlay contract](configkit.md#selected-overlays-for-testing)
|
||||
with ConfigKit v0.4.0, then turn the verified findings into a detailed production
|
||||
ConfigKit migration plan. Do not repeat the original inventory or redesign the
|
||||
established library boundary without new evidence.
|
||||
|
||||
Required work:
|
||||
|
||||
- Upgrade only the nested prototype module from ConfigKit v0.3.1 to v0.4.0.
|
||||
- Pass an explicit overlay selection through ConfigKit and add `overlays/testing.yml`
|
||||
plus two PromptKit testing profiles. Select both roles and retain one shared
|
||||
finite backend. Synthetic model names demonstrate routing; they do not benchmark
|
||||
real model cost, speed, or quality.
|
||||
- Make the offline executable accept overlay and explicit role-profile selections,
|
||||
and show the selected models alongside domain outcomes and retry/repair counts.
|
||||
- Verify base/overlay/CLI precedence, role inheritance, invalid selections,
|
||||
surviving credential references, safe inspection, and overlay provenance.
|
||||
Prove the actual HTTP request models for both roles; loading changed strings
|
||||
alone is insufficient evidence.
|
||||
- Record new results separately from the historical v0.3.1 findings. Expand the
|
||||
existing ConfigKit roadmap into a concrete production implementation handoff,
|
||||
including the interim runtime boundary and removal of legacy configuration.
|
||||
|
||||
Target end state: reviewers can run the prototype with and without the testing
|
||||
selection and see both LLM roles switch models. The offline test suite establishes
|
||||
that selection, inspection, and execution agree, and the production ConfigKit work
|
||||
has bounded tasks with explicit acceptance criteria.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- The published v0.4.0 dependency is pinned without local replacements; the original
|
||||
no-overlay path and existing integration tests continue to pass.
|
||||
- `testing` switches both roles, explicit profile flags override its selections,
|
||||
and omission/empty/`default` retains base selection according to ConfigKit.
|
||||
- Invalid overlays fail clearly; inheritance and secret resolution use surviving
|
||||
effective settings. Inspection records overlay origin without disclosing keys.
|
||||
- Both selected testing models reach the controlled HTTP transport; structural
|
||||
repair, shared capacity, observations, and Audita domain checks remain active.
|
||||
- Updated examples, executable instructions, findings, and a detailed production
|
||||
migration plan are reviewable. Exact verification commands and limitations are
|
||||
recorded, including the fact that root tests do not cover the nested module.
|
||||
- Production code, root dependencies, and upstream libraries remain unchanged.
|
||||
No release or live provider call is part of this unit.
|
||||
|
||||
This unit ends with a production migration **plan**, not production implementation.
|
||||
The ConfigKit → PromptKit → combined release sequence remains unchanged. Imports
|
||||
and secret-directory bootstrap remain disabled; PromptKit execution profiles keep
|
||||
their existing names and meaning.
|
||||
|
||||
## Objective
|
||||
|
||||
|
||||
Reference in New Issue
Block a user