From 5b7d2e6e81f597bb022eb3d44de1b336f3baa218 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 24 Sep 2026 00:27:47 +0000 Subject: [PATCH] Plan production ConfigKit migration from verified prototype --- docs/roadmap/configkit.md | 214 +++++++++++++++++++--- docs/roadmap/implementation.md | 7 +- docs/roadmap/library-adoption.md | 6 +- docs/roadmap/library-boundary-findings.md | 6 +- docs/roadmap/library-boundary.md | 8 +- 5 files changed, 204 insertions(+), 37 deletions(-) diff --git a/docs/roadmap/configkit.md b/docs/roadmap/configkit.md index 6404485..25a3f2f 100644 --- a/docs/roadmap/configkit.md +++ b/docs/roadmap/configkit.md @@ -2,9 +2,9 @@ **Last updated:** 2026-09-24 -**Status:** High-level, provisional planning only. Adoption direction is agreed; -this document does not authorize migration implementation. Revisit the scope, -sequence, design, and acceptance criteria in detail before starting work. +**Status:** Prototype boundary and selected overlays verified. The ordered +production handoff below is planned, not implemented; no production migration or +release has begun. ## Purpose and Direction @@ -20,10 +20,9 @@ guide instead of legacy loaders, aliases, dual operating modes, or an ongoing compatibility layer. Retain existing syntax only when it independently suits the new model, not because compatibility requires it. -The initial review and committed prototype used ConfigKit v0.3.1. The migration -baseline is now v0.4.0, whose selected configuration feature is named overlays. -An isolated upgrade passed the prototype tests and demonstration without source -changes; the committed prototype dependency has not yet been updated. The installed +The initial review and prototype used ConfigKit v0.3.1. The prototype now pins +v0.4.0 and verifies its selected-overlay path; original v0.3.1 findings remain +historical. The installed application must depend on a versioned module, not a sibling checkout or local module replacement. @@ -103,16 +102,17 @@ effective-config inspection, selected-profile inspection, and processing must al use the same selected overlay. Preserve overlay provenance in configuration explanations and diagnostics while retaining secret redaction. -The immediate prototype extension and production-planning handoff are specified +The completed prototype extension and production-planning handoff are recorded in the [prototype roadmap](library-boundary.md#next-unit-testing-overlays-and-production-migration-plan) and its ordered [implementation plan](implementation.md). Production CLI changes below remain subsequent migration work. Implementation and acceptance work: -- Upgrade the isolated prototype to ConfigKit v0.4.0 and add a testing-overlay - fixture before completing production configuration migration. Preserve the - original v0.3.1 evidence as historical; overlay behavior is not yet proven by it. +- The isolated prototype pins ConfigKit v0.4.0 and includes the testing-overlay + fixture. Preserve the original v0.3.1 evidence as historical; the + [extension results](library-boundary-findings.md#selected-overlay-extension-2026-09-24) + establish the selected-overlay behavior. - Verify unchanged base selection when no overlay is requested, both testing role selections, and explicit setting flags taking precedence over the overlay. - Verify role inheritance after overlay application, effective profile/backend @@ -130,7 +130,173 @@ prototype's original scope or change the ConfigKit → PromptKit → combined re sequence. The production LLM migration remains necessary before promising the full new runtime behavior. -## Provisional Stages +## Ordered Production Migration Handoff + +The [migration inventory](library-boundary-findings.md#migration-inventory) is the +field-by-field old-to-new map. The [selected boundary](library-boundary-findings.md#configuration-and-profile-contract) +owns defaults and semantic choices; the [v0.4.0 overlay results](library-boundary-findings.md#selected-overlay-extension-2026-09-24) +prove the representative load, inspection, and HTTP path. These packages are +future work. Complete each dependency and its tests before removing the code it +replaces. Do not treat the isolated prototype as a second production runtime. + +### 1. Define and load the complete version-2 configuration + +**Depends on:** the verified prototype. **Owns:** `internal/core/config` and its +tests. Replace the version-1 YAML/env application path with one ConfigKit v0.4.0 +definition and a version-2 rule. Pin the published module in the root Go module +and raise the production Go minimum as needed. Keep the existing YAML dependency +used for glossary parsing. ConfigKit imports and secret-directory bootstrap stay +disabled. + +The definition must cover every retained domain setting in the inventory: +ordered/repeated `pipeline.modules`, `output.schema`, validation prompt budget, +chunking, normalization, thresholds, context, and diagnostics, plus `profiles_dir`, +backend endpoint/concurrency/queue capacity/optional credential, and LLM proposal +and validation profile selection, transient retries, repair attempts, and total +operation timeout. Apply the inventory's defaults once. Preserve presence: +omitted validation profile inherits the **final** proposal profile; explicit +empty/null validation fails. Omitted or null section target means automatic, zero +fails. Explicit empty module list means no modules. Optional credential null means +unauthenticated; supplied references must resolve to nonempty values. Keep the +selected numeric-second normalization fields numeric, and use a duration string +for `llm.operation_timeout`; do not revive version-1 duration coercion. Validate +backend limits, references, profile selections, thresholds, and cross-field domain +rules on the resolved result. Preserve ConfigKit provenance for derived values. + +**Acceptance/tests:** `internal/core/config` tests use temporary files to cover +version rejection, strict fields, defaults, omission/null/zero, module order, +final role inheritance, overlays and explicit override precedence, secret +replacement/missing/null, and retained domain validation. Compare behavior to the +prototype extension rather than duplicating ConfigKit's parser matrix. No legacy +environment overlay or old-schema acceptance remains. + +### 2. Select files and map invocation input + +**Depends on:** package 1. **Owns:** `internal/cli/run.go`, +`internal/cli/process_flags.go`, `internal/cli` tests, and `cmd/audita` subprocess +tests. Select exactly one main file: explicit `--config`, then the two system +locations in the selected boundary. Remove `AUDITA_CONFIG`; a missing explicit +path fails, while absent system files are skipped. Require an explicit file for +`config validate`. Parse help before loading configuration. + +Add invocation-only `--overlay ` to `process`, `config validate`, and +`config print-effective`, passing it to `Options.Overlay` for the same selected +main file. Omitted, empty, or `default` selects the base. Keep transcript, +glossary, output, and report paths outside persistent configuration. Map only +flags actually supplied by the user into ConfigKit overrides; parser defaults +must never defeat file or overlay values. Retain useful domain-setting flags from +the inventory. Add profile-selection flags. Remove direct API-key/model/endpoint, +old retry/role/global concurrency flags and aliases rather than translating them +silently. Retire all general `AUDITA_*` configuration overlays listed in the +inventory; environment names retain meaning only when explicitly referenced as +secrets. + +**Acceptance/tests:** CLI and subprocess tests cover discovery order, explicit +missing files, all three commands with base/testing/default/missing selections, +overlay then explicit-setting precedence, omitted flags, usage/error streams and +exit classifications. Reject old flags and implicit environment configuration. +The same loaded result must feed validation, effective inspection, and processing. + +### 3. Assemble profiles and safe inspection + +**Depends on:** packages 1–2. **Owns:** a small CLI/runtime assembly boundary, +config inspection, and diagnostic snapshot wiring in `internal/cli` and +`internal/core/diagnostics`; verify against `internal/framework/processreport` +where snapshots are referenced. Resolve `profiles_dir` relative to the selected +**main** file, including when an overlay is selected; absolute paths remain valid. +Inspect both selected PromptKit profiles and verify their named registered backend, +endpoint equality, and absence of a competing profile credential lookup. Treat +ConfigKit settings and PromptKit generation controls as separate views. + +Use `WriteEffective` for redacted effective YAML and expose relevant +explanation/provenance and selected-profile model/backend controls through an +Audita-owned safe view. A ConfigKit snapshot alone cannot claim to describe the +full execution target. Validation and inspection resolve surviving secret +references without provider calls; a missing or empty surviving reference fails, +while a replaced reference is not resolved. Snapshot and surfaced error paths +must redact secret **values** and preserve overlay/CLI origin. Avoid serializing +raw library objects into stable report formats. + +**Acceptance/tests:** config-command and diagnostics tests verify relative paths, +same selection across views, overlay/CLI origin, surviving credentials, +unauthenticated null, and no secret values in YAML, explanations, selected-profile +inspection, diagnostics, or errors. Update `internal/cli/run_test.go`, +`internal/core/config/*_test.go`, and report/diagnostics tests that own these +outputs. + +### 4. Convert runtime inputs without claiming unavailable execution + +**Depends on:** packages 1–3. **Owns:** a narrow conversion at the CLI to +`internal/framework/llm`/runner boundary, with focused `internal/cli` and +`internal/framework/llm` tests. Convert retained domain settings to their current +domain structs. Convert selected profile/backend endpoint, model, resolved secret, +and per-attempt timeout to existing client setup values for assembly checks; keep +library types out of modules, validators, transcript schemas, and ledgers. + +The interim conversion has this explicit scope; "assemble" means values can be +validated and converted, not that processing is enabled: + +| New setting or control | Interim disposition | +| --- | --- | +| `profiles_dir`, proposal/validation profile IDs | Honor for relative-path profile loading, final inheritance, inspection, and target assembly. | +| `backends..endpoint` and `credential` | Honor for selected target validation and client setup; keep ConfigKit secret resolution and redaction. | +| Profile model and `timeout_seconds` | Convert for old client setup checks; old transport can consume these per-attempt values. | +| `validation.max_prompt_tokens`; retained domain, output, and diagnostics fields | Honor through existing domain/batching/report inputs after validation. | +| `backends..concurrency` and `queue_capacity` | Validate and inspect; old global/role pools cannot enforce independent per-backend admission or queue semantics. | +| `llm.transient_retries` | Validate and inspect; old retry eligibility and explicit-zero behavior differ. | +| `llm.repair_attempts` | Validate and inspect; old runtime has no PromptKit structural-repair contract. | +| `llm.operation_timeout` | Validate and inspect; old per-call timeout is not the full logical-operation deadline. | +| Other PromptKit profile generation controls and execution observations | Inspect only; old transport/reporting cannot claim equivalent behavior. | + +The old global/role pools and retry loop must not be presented as equivalents. The +selected interim behavior is to reject `process` before any LLM request with a +clear unsupported-runtime classification while ConfigKit has landed but PromptKit +has not. `config validate` and effective/profile inspection remain usable. This +is an unreleased development interval, not a user-visible mode or compatibility +switch. The process gate is removed only when the PromptKit package below proves +the complete settings through controlled calls. + +**Acceptance/tests:** conversion tests verify the representable values and +redaction; a process subprocess test proves the interim rejection occurs before +provider transport or output mutation. No test should assert that old retries or +schedulers implement new settings. The combined migration cannot release with +this gate in place. + +### 5. Remove superseded configuration paths and update maintained contracts + +**Depends on:** packages 1–4. **Owns:** removal of old file/env merge, aliases, +direct-key flags, legacy redaction/inspection paths, and obsolete tests in +`internal/core/config` and `internal/cli`. Retain Audita's domain checks, +normalization/chunking, output encoding, module and validator behavior. Update +`examples/` with version-2 base configuration, `overlays/testing.yml`, complete +PromptKit profiles, and secret-free credentials by reference. Update the canonical +`docs/config.md`, `docs/cli.md`, operations/integration/internal references as +their contracts change, and provide one migration guide with the complete +inventory's file/flag/environment mapping, duration and empty-list changes, +credentials, profiles/overlays, inspection, and changed runtime/diagnostic +semantics. Do not leave dual loaders or a permanent translation mode. + +**Acceptance/tests:** maintained examples validate offline; removed flags and old +schema are rejected; relevant config/CLI/subprocess/report tests pass; documentation +links and `git diff --check` pass. The interim process restriction remains explicit +until package 6 and is not documented as a released product feature. + +### 6. Replace execution with PromptKit, then prepare the combined release + +**Depends on:** packages 1–5. This is the later PromptKit work unit, not part of +this ConfigKit implementation. Replace the old provider transport, prompt loader, +structural-output handling, and scheduler composition as described by the +[coordinated adoption roadmap](library-adoption.md#4-complete-promptkit-adoption) +and [execution/observation handoff](library-boundary-findings.md#execution-and-failure-mapping). +Prove both roles' profile/backend settings, per-backend capacity, explicit-zero +retry, positive repair, total-operation deadline, domain decisions, and safe +observations with controlled transport. Remove the interim converter/process gate +and superseded LLM machinery only after those tests pass. Update report and +diagnostic contracts and the migration guide. The combined release and license/ +distribution review follow last under the release policy; no publication is +authorized by this plan. + +## Historical Provisional Outline The completed [shared-boundary and prototype unit](library-boundary.md) supplies the initial target settings model and integration evidence. Extend that evidence @@ -205,26 +371,26 @@ questions and the selected retry direction. The coordinated plan adopts PromptKit's explicit retry semantics and enables structural repair from the outset. Include separate transient-retry and positive repair budgets, plus the total-operation deadline policy, in the shared settings -design. The reviewed PromptKit baseline is v0.11.0; exact defaults and settings -ownership remain detailed-planning decisions. Do not implement the runtime change +design. The selected defaults and ownership are recorded in the +[boundary findings](library-boundary-findings.md#configuration-and-profile-contract). +Do not implement the runtime change merely as a side effect of replacing the configuration loader. Retention and transcript timestamp decisions are not automatically included. -## Open Design Decisions and Boundaries +## Implementation Boundaries -- Pin ConfigKit v0.4.0 and align Go/toolchain requirements. ConfigKit requires +- Pin ConfigKit v0.4.0 in production and align Go/toolchain requirements. ConfigKit requires Go 1.25; the combined PromptKit prototype requires Go 1.25.5. Config-only adoption does not automatically remove Audita's existing YAML dependency used for glossary parsing. -- Decide the configuration type shape, domain inheritance rules, and runtime - conversion boundary without leaking library concerns throughout the pipeline. +- Implement the selected type shape, final role inheritance, and interim runtime + gate above without leaking library concerns throughout the pipeline. - Enable selected overlays as specified above. Keep imports and secret-directory bootstrap disabled unless a separate concrete requirement justifies them. -- Define the desired behavior of validation and inspection without available - credentials under ConfigKit's secret-reference requirements. -- Decide which inspection and diagnostic interfaces benefit from provenance and - explanation, and account for their changed formats in the migration guide. -- Finalize task breakdown, test scope, and acceptance criteria after the other - planned large changes are understood. No schedule or effort estimate is committed. +- Validation and inspection resolve surviving credential references and fail on + missing or empty values; no credential-free bypass is planned. +- Add the safe selected-profile view and ConfigKit provenance to inspection and + diagnostics as specified in package 3, then account for changed formats in the + migration guide. No schedule or effort estimate is committed. The eventual completion test is that Audita follows the suite's ConfigKit conventions while retaining its intended domain capabilities, with a clear diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index b3d1a6b..0c19301 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -2,7 +2,7 @@ **Last updated:** 2026-09-24 -**Status:** Planned. All stages below are pending. This plan implements the next +**Status:** Complete (2026-09-24). All six stages below are finished. This plan implements the next unit in the [prototype roadmap](library-boundary.md#next-unit-testing-overlays-and-production-migration-plan), not the production ConfigKit or PromptKit migrations. @@ -249,7 +249,10 @@ to a passing test or observed demonstration result. Verify affected links and ## Stage 6: Prepare the Production Migration Plan and Close the Unit -**Status:** Pending. **Depends on:** Stage 5. +**Status:** Complete (2026-09-24). **Depends on:** Stage 5. The ordered ConfigKit +production handoff records bounded packages, interim runtime behavior, tests, +dependencies, and acceptance criteria. Prototype evidence and scope audit passed; +links and `git diff --check` verified. Expand `configkit.md` into a concrete, ordered production migration plan using the existing inventory and verified overlay evidence. Follow the development diff --git a/docs/roadmap/library-adoption.md b/docs/roadmap/library-adoption.md index a65f772..b68fa37 100644 --- a/docs/roadmap/library-adoption.md +++ b/docs/roadmap/library-adoption.md @@ -4,10 +4,8 @@ **Status:** Shared boundary and offline prototype completed; production migrations remain planned. The [v0.4.0 overlay evidence](library-boundary-findings.md#selected-overlay-extension-2026-09-24) -extends the completed prototype; its production-planning handoff remains pending. -Detailed design, -implementation tasks, dependency versions, and acceptance criteria must be revisited -before migration begins. This document authorizes neither implementation nor release +extends the completed prototype, and the [ordered ConfigKit handoff](configkit.md#ordered-production-migration-handoff) +specifies the next implementation work. This document authorizes no release publication and sets no schedule. ## Goal and Ownership diff --git a/docs/roadmap/library-boundary-findings.md b/docs/roadmap/library-boundary-findings.md index 554103d..4e3b918 100644 --- a/docs/roadmap/library-boundary-findings.md +++ b/docs/roadmap/library-boundary-findings.md @@ -404,7 +404,7 @@ not price, latency, quality, or provider-compatibility measurements. | Missing/invalid selection, version/directive rejection, and ordinary backend-setting validation | `TestOverlayFailuresAndApplicationValidation` checks ConfigKit diagnostic codes and effective backend limits. | | Surviving credential resolution, absent/empty failures, null unauthenticated use, and profile-only switching | `TestOverlayCredentialResolutionAndInspection` controls synthetic environment values and observes HTTP Authorization. | | Redacted effective output, explanation and metadata origin, profile inspection, and observations | `TestOverlayCredentialResolutionAndInspection` and `TestOverlaySelectionProvenance` inspect safe output and overlay/CLI origins. | -| Existing retry, repair, domain, capacity, and concurrent-observation behavior | The unchanged nested suite passes, including under `-race`. | +| Existing retry, repair, domain, capacity, and concurrent-observation behavior | `TestWorkflowBoundaries`, `TestSharedCapacityAndRecovery`, `TestQueuedCancellationAndDeadline`, `TestDeadlineStopsRetryAfter`, and `TestPartialUsageOnFailedRun` pass, including under `-race`. | Changing a profile does not remove a backend credential reference from the effective configuration. ConfigKit validates a surviving reference when only @@ -417,5 +417,5 @@ The tests use a controlled built-in HTTP transport and synthetic credentials; there were no live provider calls. The prototype still covers one representative proposal/assessment pair, not all production modules, prompts, CLI paths, or diagnostic artifacts. Root `go test ./...` does not run this nested module. No -upstream API gap appeared in this extension. The production migration handoff is -the remaining work for this unit. +upstream API gap appeared in this extension. The [ordered production handoff](configkit.md#ordered-production-migration-handoff) +keeps implementation and release pending. diff --git a/docs/roadmap/library-boundary.md b/docs/roadmap/library-boundary.md index d28e4b8..c2eef63 100644 --- a/docs/roadmap/library-boundary.md +++ b/docs/roadmap/library-boundary.md @@ -3,7 +3,7 @@ **Last updated:** 2026-09-24 **Status:** Original boundary-design and offline-prototype unit completed; -overlay extension verified; production-planning handoff pending. +overlay extension and production-planning handoff completed, 2026-09-24. Production migration has not started. The work packages below retain the original scope; [decisions, evidence, limitations, and handoff](library-boundary-findings.md) record the outcome. The [executable prototype](../../examples/library-adoption/README.md) @@ -11,9 +11,9 @@ is isolated from production. No publication or upstream changes were made. ## Next Unit: Testing Overlays and Production Migration Plan -**Status:** Overlay implementation verified; production-planning handoff pending. -Execute the remaining ordered work in -[implementation.md](implementation.md). The original work packages below remain +**Status:** Complete. The [ordered production handoff](configkit.md#ordered-production-migration-handoff) +is ready for subsequent work; [implementation.md](implementation.md) records this +unit's checks. The original work packages below remain historical scope for the completed first unit. Extend the existing prototype to prove the [selected-overlay contract](configkit.md#selected-overlays-for-testing)