From 590604bf3874a216d9c00b959e4424e9f9cd7045 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 24 Sep 2026 00:24:19 +0000 Subject: [PATCH] Document verified prototype overlay behavior --- docs/roadmap/implementation.md | 4 +- docs/roadmap/library-adoption.md | 6 ++- docs/roadmap/library-boundary-findings.md | 52 ++++++++++++++++++++++- docs/roadmap/library-boundary.md | 5 ++- examples/library-adoption/README.md | 3 ++ 5 files changed, 64 insertions(+), 6 deletions(-) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index f6ea073..b3d1a6b 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -225,7 +225,9 @@ actual gaps instead of silently relaxing tests or claiming complete integration. ## Stage 5: Record Overlay Findings and Executable Instructions -**Status:** Pending. **Depends on:** Stage 4. +**Status:** Complete (2026-09-24). **Depends on:** Stage 4. Dated v0.4.0 evidence, +test coverage, commands, limitations, README guidance, and roadmap links updated; +affected links and `git diff --check` verified. This stage changes documentation, not production code. Read the new tests/results, then update the existing owners rather than creating another parallel design: diff --git a/docs/roadmap/library-adoption.md b/docs/roadmap/library-adoption.md index e85ffa4..a65f772 100644 --- a/docs/roadmap/library-adoption.md +++ b/docs/roadmap/library-adoption.md @@ -1,9 +1,11 @@ # Coordinated ConfigKit and PromptKit Adoption -**Last updated:** 2026-09-23 +**Last updated:** 2026-09-24 **Status:** Shared boundary and offline prototype completed; production migrations -remain planned. See the [findings](library-boundary-findings.md). Detailed design, +remain planned. The [v0.4.0 overlay evidence](library-boundary-findings.md#selected-overlay-extension-2026-09-24) +extends the completed prototype; its production-planning handoff remains pending. +Detailed design, implementation tasks, dependency versions, and acceptance criteria must be revisited before migration begins. This document authorizes neither implementation nor release publication and sets no schedule. diff --git a/docs/roadmap/library-boundary-findings.md b/docs/roadmap/library-boundary-findings.md index 225dde5..554103d 100644 --- a/docs/roadmap/library-boundary-findings.md +++ b/docs/roadmap/library-boundary-findings.md @@ -102,7 +102,7 @@ Selected for the prototype and recommended for production: Repeated modules retain their existing ordered meaning. - The original prototype disabled ConfigKit imports, selected configuration layers (then called configuration profiles), and secret-directory bootstrap. Selected - overlays are now planned under the linked overlay decision; the other two + overlays are now verified under the linked overlay decision; the other two features remain disabled. PromptKit execution profiles retain their meaning. Configuration validation/inspection must assemble and inspect both selected profiles @@ -369,3 +369,53 @@ and rebuild/relink materials needed for Audita's Go binary/container distributio This unit does not publish binaries or claim that release compliance is complete. No upstream relicensing request or library implementation change is required for this prototype. + +## Selected-Overlay Extension: 2026-09-24 + +The isolated prototype now pins published ConfigKit v0.4.0. ConfigKit loads +`overlays/testing.yml` relative to the selected main file; the overlay selects +complete PromptKit execution profiles from the separate `profiles/` directory. +The original v0.3.1 observations above are historical and do not prove this +extension. No production Audita behavior changed. + +With Go 1.26.8 on linux/amd64, from `examples/library-adoption/`, these offline +checks passed with `GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off`: + +```sh +go test ./... +go test -race ./... +go vet ./... +go run . +go run . --overlay testing +go run . --overlay testing --proposal-profile proposal +``` + +Each demonstration reported one accepted correction, four attempts, zero +transient retries, and two structural repairs. The base run selected +`synthetic-grammar`/`synthetic-review`; testing selected +`synthetic-grammar-testing`/`synthetic-review-testing`; the explicit proposal +override restored only `synthetic-grammar`. These are synthetic routing IDs, +not price, latency, quality, or provider-compatibility measurements. + +| Requirement | Consumer evidence | +| --- | --- | +| Base, empty/default, testing selection, role overrides, final inheritance, and relative profile resolution | `TestSelectedOverlayConfiguration` loads temporary on-disk layouts and assembles both roles. | +| Actual base/testing/override HTTP models and repairs | `TestDemonstrationSelectsOutboundModels` captures every request body and matches model IDs to the reported selections and domain outcome. | +| Missing/invalid selection, version/directive rejection, and ordinary backend-setting validation | `TestOverlayFailuresAndApplicationValidation` checks ConfigKit diagnostic codes and effective backend limits. | +| Surviving credential resolution, absent/empty failures, null unauthenticated use, and profile-only switching | `TestOverlayCredentialResolutionAndInspection` controls synthetic environment values and observes HTTP Authorization. | +| Redacted effective output, explanation and metadata origin, profile inspection, and observations | `TestOverlayCredentialResolutionAndInspection` and `TestOverlaySelectionProvenance` inspect safe output and overlay/CLI origins. | +| Existing retry, repair, domain, capacity, and concurrent-observation behavior | The unchanged nested suite passes, including under `-race`. | + +Changing a profile does not remove a backend credential reference from the +effective configuration. ConfigKit validates a surviving reference when only +profile selection changes. Replacing the reference in an +overlay resolves only the replacement; explicit null allows unauthenticated use. +This distinction is part of the selected configuration boundary, not an +application fallback. Imports and secret-directory bootstrap remain disabled. + +The tests use a controlled built-in HTTP transport and synthetic credentials; +there were no live provider calls. The prototype still covers one representative +proposal/assessment pair, not all production modules, prompts, CLI paths, or +diagnostic artifacts. Root `go test ./...` does not run this nested module. No +upstream API gap appeared in this extension. The production migration handoff is +the remaining work for this unit. diff --git a/docs/roadmap/library-boundary.md b/docs/roadmap/library-boundary.md index 8e5c918..d28e4b8 100644 --- a/docs/roadmap/library-boundary.md +++ b/docs/roadmap/library-boundary.md @@ -3,7 +3,7 @@ **Last updated:** 2026-09-24 **Status:** Original boundary-design and offline-prototype unit completed; -overlay extension and production-planning handoff planned. +overlay extension verified; production-planning handoff pending. Production migration has not started. The work packages below retain the original scope; [decisions, evidence, limitations, and handoff](library-boundary-findings.md) record the outcome. The [executable prototype](../../examples/library-adoption/README.md) @@ -11,7 +11,8 @@ is isolated from production. No publication or upstream changes were made. ## Next Unit: Testing Overlays and Production Migration Plan -**Status:** Planned; not implemented. Execute the ordered stages in +**Status:** Overlay implementation verified; production-planning handoff pending. +Execute the remaining ordered work in [implementation.md](implementation.md). The original work packages below remain historical scope for the completed first unit. diff --git a/examples/library-adoption/README.md b/examples/library-adoption/README.md index 48dda41..c846f13 100644 --- a/examples/library-adoption/README.md +++ b/examples/library-adoption/README.md @@ -36,6 +36,9 @@ from `profiles/`. An explicitly supplied `--proposal-profile` or requested overlay and each effective profile, backend, and model. Synthetic model IDs demonstrate operator-selected routing; they are not measurements of price, speed, or model quality. +`overlays/` contains ConfigKit application-setting layers; `profiles/` contains +PromptKit execution profiles. Profile paths resolve from the main `config.yaml` +directory, including when an overlay is selected. An overlay can also change ordinary backend settings and credential references. ConfigKit resolves only the surviving reference at a replaced credential path; changing role profiles alone leaves other configured backend credential references