From 187dc4cb1cf7ee8aec2eca406d9945b8001af9d2 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 23 Sep 2026 23:04:46 +0000 Subject: [PATCH] Plan ConfigKit overlays for testing with cheaper models --- docs/roadmap/configkit.md | 91 +++++++++++++++++++---- docs/roadmap/library-adoption.md | 10 ++- docs/roadmap/library-boundary-findings.md | 12 ++- docs/roadmap/library-boundary.md | 7 +- 4 files changed, 100 insertions(+), 20 deletions(-) diff --git a/docs/roadmap/configkit.md b/docs/roadmap/configkit.md index 32e9c87..f7f93f4 100644 --- a/docs/roadmap/configkit.md +++ b/docs/roadmap/configkit.md @@ -20,8 +20,10 @@ guide instead of legacy loaders, aliases, dual operating modes, or an ongoing compatibility layer. Retain existing syntax only when it independently suits the new model, not because compatibility requires it. -The initial review used the sibling `../ConfigKit` checkout at v0.3.1. Reassess the -supported release and current conventions during detailed planning. The installed +The initial review and committed prototype used ConfigKit v0.3.1. The migration +baseline is now v0.4.0, whose selected configuration feature is named overlays. +An isolated upgrade passed the prototype tests and demonstration without source +changes; the committed prototype dependency has not yet been updated. The installed application must depend on a versioned module, not a sibling checkout or local module replacement. @@ -59,11 +61,75 @@ Transcript/glossary formats, processing algorithms, and output schemas are outsi this migration unless later planning identifies a concrete dependency or an explicitly coordinated change. +## Selected Overlays for Testing + +Enable ConfigKit's optional selected-overlay feature in the migration. A testing +configuration that selects cheaper, faster LLMs is the initial use case. ConfigKit +**overlays** modify application settings; PromptKit **execution profiles** still +own model and generation settings. Keep `profiles_dir`, the PromptKit `profiles/` +directory, and proposal/validation profile names unchanged by this terminology +change. + +The planned interface is `--overlay ` on `process`, `config validate`, and +`config print-effective`. Treat selection as invocation state passed to +`Options.Overlay`, not as another persistent setting or environment overlay. +Follow ConfigKit v0.4.0's selection contract: omission, an empty name, or `default` +selects no overlay; a named selection loads `overlays/.yml` beneath the selected +configuration root. A missing selected overlay is an error. Use the library's +single-overlay semantics without adding stacking, inheritance, or legacy-directory +fallback. + +Precedence is application defaults, main configuration, selected overlay, then +explicit setting flags. Perform Audita's role inheritance and semantic validation +on the resolved configuration. ConfigKit's base parsing and schema-version checks +still precede overlay application; overlays cannot replace the schema version or +contain ConfigKit directives. General environment-variable overrides remain retired. +Imports and secret-directory bootstrap remain disabled. + +Provide a maintained `overlays/testing.yml` example that selects testing execution +profiles through `llm.proposal_profile` and `llm.validation_profile`. Model names and +generation controls belong in the corresponding PromptKit profile files. Set both +role selections in this example so a base configuration's explicit validation +profile cannot accidentally retain the production model. Normal omission-based +validation-profile inheritance continues to apply to the final effective settings. +The example must make its backend/credential requirements clear and use no real +secrets. Model choice is operator-controlled; “testing” does not promise a price, +performance level, or equivalent correction quality. + +Overlays may modify other ordinary application settings supported by the schema; +they are not restricted to profile selection. The resulting configuration must +still satisfy shared backend capacity, credential, and domain rules. Validation, +effective-config inspection, selected-profile inspection, and processing must all +use the same selected overlay. Preserve overlay provenance in configuration +explanations and diagnostics while retaining secret redaction. + +Implementation and acceptance work: + +- Upgrade the isolated prototype to ConfigKit v0.4.0 and add a testing-overlay + fixture before completing production configuration migration. Preserve the + original v0.3.1 evidence as historical; overlay behavior is not yet proven by it. +- Verify unchanged base selection when no overlay is requested, both testing role + selections, and explicit setting flags taking precedence over the overlay. +- Verify role inheritance after overlay application, effective profile/backend + assembly, and secret resolution from the surviving configuration. +- Reject missing/invalid overlay selections and prohibited version/directive changes + through ConfigKit; rely on its tests for exhaustive merge/parser cases. +- Verify redacted inspection and overlay provenance, then run both LLM roles through + the offline provider using the selected testing profiles. Provider requests must + use their model selections; no paid calls are needed. +- Add production CLI coverage for the three commands and document selection, + precedence, example layout, and migration guidance when implemented. + +This expands the next configuration unit; it does not reopen the completed +prototype's original scope or change the ConfigKit → PromptKit → combined release +sequence. The production LLM migration remains necessary before promising the full +new runtime behavior. + ## Provisional Stages -First complete the [shared-boundary and prototype unit](library-boundary.md) from -the coordinated adoption plan. It supplies the target settings model and integration -evidence for the production configuration migration described here. +The completed [shared-boundary and prototype unit](library-boundary.md) supplies +the initial target settings model and integration evidence. Extend that evidence +with the selected-overlay work above during the next configuration unit. ### 1. Define the Contract and Coordinate Sequencing @@ -85,8 +151,8 @@ an implementation plan and concrete acceptance criteria before beginning migrati After detailed planning, implement a bounded path from file selection and CLI inputs through ConfigKit to Audita's runtime configuration. Exercise defaults, -proposal/validation inheritance, concurrency, duration values, secrets, and -redacted inspection using representative valid and invalid inputs. +selected overlays, proposal/validation inheritance, concurrency, duration values, +secrets, and redacted inspection using representative valid and invalid inputs. Use the result to assess whether domain rules remain clear and substantial old infrastructure can be removed. Resolve genuine library gaps deliberately, including @@ -141,14 +207,13 @@ Retention and transcript timestamp decisions are not automatically included. ## Open Design Decisions and Boundaries -- Select and pin the ConfigKit release and align Go/toolchain requirements. The - reviewed version requires Go 1.25. Config-only adoption does not automatically - remove Audita's existing YAML dependency used for glossary parsing. +- Pin ConfigKit v0.4.0 and align Go/toolchain requirements. ConfigKit requires + Go 1.25; the combined PromptKit prototype requires Go 1.25.5. Config-only + adoption does not automatically remove Audita's existing YAML dependency used for glossary parsing. - Decide the configuration type shape, domain inheritance rules, and runtime conversion boundary without leaking library concerns throughout the pipeline. -- Start with imports, profiles, and secrets-directory bootstrap disabled unless a - concrete suite or Audita requirement justifies enabling them. Adoption does not - require activating every library capability. +- Enable selected overlays as specified above. Keep imports and secret-directory + bootstrap disabled unless a separate concrete requirement justifies them. - Define the desired behavior of validation and inspection without available credentials under ConfigKit's secret-reference requirements. - Decide which inspection and diagnostic interfaces benefit from provenance and diff --git a/docs/roadmap/library-adoption.md b/docs/roadmap/library-adoption.md index 5cd3f31..e85ffa4 100644 --- a/docs/roadmap/library-adoption.md +++ b/docs/roadmap/library-adoption.md @@ -57,9 +57,10 @@ avoid duplicate model, endpoint, credential, or timeout definitions with competi precedence. Keep ConfigKit application configuration distinct from PromptKit execution profiles. -In particular, optional ConfigKit configuration profiles and PromptKit execution -profiles are different concepts. Select an explicit profile-ownership model; -complete Audita-owned execution profiles are the initial recommendation, with +ConfigKit v0.4.0 calls its optional configuration layers overlays. Enable selected +overlays for testing as specified in the [ConfigKit roadmap](configkit.md#selected-overlays-for-testing); +these select application settings, including PromptKit execution profiles. +Select an explicit profile-ownership model; complete Audita-owned execution profiles are the initial recommendation, with catalog-driven selection a deliberate alternative. Capacity needs are simple: protect local backends from exceeding their available @@ -103,7 +104,8 @@ ConfigKit-facing settings around the old LLM runtime only to redesign them later Implement the agreed application configuration contract following the [ConfigKit roadmap](configkit.md). Retire legacy environment overlays and obsolete CLI/configuration conventions, update maintained examples and canonical references, -and begin the combined migration guide. +and begin the combined migration guide. Include the selected testing-overlay path +and its offline integration evidence from the ConfigKit roadmap. If the existing LLM runtime remains temporarily during this stage, keep translation into it small and explicitly transitional. Do not preserve its old settings merely diff --git a/docs/roadmap/library-boundary-findings.md b/docs/roadmap/library-boundary-findings.md index ff681fe..9066f14 100644 --- a/docs/roadmap/library-boundary-findings.md +++ b/docs/roadmap/library-boundary-findings.md @@ -4,6 +4,12 @@ production migration target; not the current Audita configuration contract. The [bounded roadmap](library-boundary.md) owns scope and completion criteria. +**Subsequent scope decision:** The [ConfigKit overlay plan](configkit.md#selected-overlays-for-testing) +adds optional selected overlays, with v0.4.0 as the migration baseline. It supersedes +the original no-overlay decision and adds an input layer between the main file and +explicit setting overrides. The v0.3.1 prototype results below remain historical +evidence; they do not establish overlay integration. + ## Selected Boundary The prototype lives in [examples/library-adoption](../../examples/library-adoption/README.md), @@ -87,8 +93,10 @@ Selected for the prototype and recommended for production: credentials and the future optional section target accept null. Explicit empty module lists will mean an empty correction pipeline, not “restore defaults.” Repeated modules retain their existing ordered meaning. -- Do not enable ConfigKit imports, configuration profiles, or secret-directory - bootstrap. These are distinct from PromptKit execution profiles. +- The original prototype disabled ConfigKit imports, selected configuration layers + (then called configuration profiles), and secret-directory bootstrap. Selected + overlays are now planned under the linked overlay decision; the other two + features remain disabled. PromptKit execution profiles retain their meaning. Configuration validation/inspection must assemble and inspect both selected profiles without provider calls. ConfigKit's effective YAML describes only its loaded state; diff --git a/docs/roadmap/library-boundary.md b/docs/roadmap/library-boundary.md index 863b483..8a580ee 100644 --- a/docs/roadmap/library-boundary.md +++ b/docs/roadmap/library-boundary.md @@ -8,6 +8,11 @@ scope; [decisions, evidence, limitations, and handoff](library-boundary-findings record the outcome. The [executable prototype](../../examples/library-adoption/README.md) is isolated from production. No publication or upstream changes were made. +**Subsequent scope decision:** Enable ConfigKit v0.4.0 selected overlays for testing +in the next configuration unit. The [overlay plan](configkit.md#selected-overlays-for-testing) +supersedes the original optional-feature exclusion below; the completed prototype +has not yet exercised overlays. + ## Objective Define and prove the boundary between Audita, ConfigKit, and PromptKit before @@ -80,7 +85,7 @@ Resolve these design decisions explicitly: | Role selection and inheritance | Specify how proposal and validation choose profiles, including omission and any inheritance. Do not inherit old behavior solely for compatibility. | | Credentials | Prefer one ConfigKit-owned secret-resolution path passed through PromptKit's supported request credential field. Define backend/profile credential association and avoid competing environment lookups. Validate this against profile requirements. | | Retry, repair, and deadlines | Assign one owner and explicit defaults for each control. Use positive structural-repair defaults and document combined limits and total-operation scope. | -| Optional library features | Leave ConfigKit imports, configuration profiles, and secret-directory bootstrap out unless a concrete requirement is identified. These are separate from PromptKit execution profiles. | +| Optional library features | The original prototype omitted imports, selected configuration layers, and secret-directory bootstrap. Selected overlays are now planned for testing; see the [overlay plan](configkit.md#selected-overlays-for-testing). Imports and secret-directory bootstrap remain out. | | Inspection | Define effective configuration and selected-profile inspection without implying that one library's snapshot contains the other's complete state. | Choose concrete field names, example values, defaults, units, and omission/null/zero