Plan ConfigKit overlays for testing with cheaper models

This commit is contained in:
2026-09-23 23:04:46 +00:00
parent 228087dec8
commit 187dc4cb1c
4 changed files with 100 additions and 20 deletions

View File

@@ -20,8 +20,10 @@ guide instead of legacy loaders, aliases, dual operating modes, or an ongoing
compatibility layer. Retain existing syntax only when it independently suits the
new model, not because compatibility requires it.
The initial review used the sibling `../ConfigKit` checkout at v0.3.1. Reassess the
supported release and current conventions during detailed planning. The installed
The initial review and committed prototype used ConfigKit v0.3.1. The migration
baseline is now v0.4.0, whose selected configuration feature is named overlays.
An isolated upgrade passed the prototype tests and demonstration without source
changes; the committed prototype dependency has not yet been updated. The installed
application must depend on a versioned module, not a sibling checkout or local
module replacement.
@@ -59,11 +61,75 @@ Transcript/glossary formats, processing algorithms, and output schemas are outsi
this migration unless later planning identifies a concrete dependency or an
explicitly coordinated change.
## Selected Overlays for Testing
Enable ConfigKit's optional selected-overlay feature in the migration. A testing
configuration that selects cheaper, faster LLMs is the initial use case. ConfigKit
**overlays** modify application settings; PromptKit **execution profiles** still
own model and generation settings. Keep `profiles_dir`, the PromptKit `profiles/`
directory, and proposal/validation profile names unchanged by this terminology
change.
The planned interface is `--overlay <name>` on `process`, `config validate`, and
`config print-effective`. Treat selection as invocation state passed to
`Options.Overlay`, not as another persistent setting or environment overlay.
Follow ConfigKit v0.4.0's selection contract: omission, an empty name, or `default`
selects no overlay; a named selection loads `overlays/<name>.yml` beneath the selected
configuration root. A missing selected overlay is an error. Use the library's
single-overlay semantics without adding stacking, inheritance, or legacy-directory
fallback.
Precedence is application defaults, main configuration, selected overlay, then
explicit setting flags. Perform Audita's role inheritance and semantic validation
on the resolved configuration. ConfigKit's base parsing and schema-version checks
still precede overlay application; overlays cannot replace the schema version or
contain ConfigKit directives. General environment-variable overrides remain retired.
Imports and secret-directory bootstrap remain disabled.
Provide a maintained `overlays/testing.yml` example that selects testing execution
profiles through `llm.proposal_profile` and `llm.validation_profile`. Model names and
generation controls belong in the corresponding PromptKit profile files. Set both
role selections in this example so a base configuration's explicit validation
profile cannot accidentally retain the production model. Normal omission-based
validation-profile inheritance continues to apply to the final effective settings.
The example must make its backend/credential requirements clear and use no real
secrets. Model choice is operator-controlled; “testing” does not promise a price,
performance level, or equivalent correction quality.
Overlays may modify other ordinary application settings supported by the schema;
they are not restricted to profile selection. The resulting configuration must
still satisfy shared backend capacity, credential, and domain rules. Validation,
effective-config inspection, selected-profile inspection, and processing must all
use the same selected overlay. Preserve overlay provenance in configuration
explanations and diagnostics while retaining secret redaction.
Implementation and acceptance work:
- Upgrade the isolated prototype to ConfigKit v0.4.0 and add a testing-overlay
fixture before completing production configuration migration. Preserve the
original v0.3.1 evidence as historical; overlay behavior is not yet proven by it.
- Verify unchanged base selection when no overlay is requested, both testing role
selections, and explicit setting flags taking precedence over the overlay.
- Verify role inheritance after overlay application, effective profile/backend
assembly, and secret resolution from the surviving configuration.
- Reject missing/invalid overlay selections and prohibited version/directive changes
through ConfigKit; rely on its tests for exhaustive merge/parser cases.
- Verify redacted inspection and overlay provenance, then run both LLM roles through
the offline provider using the selected testing profiles. Provider requests must
use their model selections; no paid calls are needed.
- Add production CLI coverage for the three commands and document selection,
precedence, example layout, and migration guidance when implemented.
This expands the next configuration unit; it does not reopen the completed
prototype's original scope or change the ConfigKit → PromptKit → combined release
sequence. The production LLM migration remains necessary before promising the full
new runtime behavior.
## Provisional Stages
First complete the [shared-boundary and prototype unit](library-boundary.md) from
the coordinated adoption plan. It supplies the target settings model and integration
evidence for the production configuration migration described here.
The completed [shared-boundary and prototype unit](library-boundary.md) supplies
the initial target settings model and integration evidence. Extend that evidence
with the selected-overlay work above during the next configuration unit.
### 1. Define the Contract and Coordinate Sequencing
@@ -85,8 +151,8 @@ an implementation plan and concrete acceptance criteria before beginning migrati
After detailed planning, implement a bounded path from file selection and CLI
inputs through ConfigKit to Audita's runtime configuration. Exercise defaults,
proposal/validation inheritance, concurrency, duration values, secrets, and
redacted inspection using representative valid and invalid inputs.
selected overlays, proposal/validation inheritance, concurrency, duration values,
secrets, and redacted inspection using representative valid and invalid inputs.
Use the result to assess whether domain rules remain clear and substantial old
infrastructure can be removed. Resolve genuine library gaps deliberately, including
@@ -141,14 +207,13 @@ Retention and transcript timestamp decisions are not automatically included.
## Open Design Decisions and Boundaries
- Select and pin the ConfigKit release and align Go/toolchain requirements. The
reviewed version requires Go 1.25. Config-only adoption does not automatically
remove Audita's existing YAML dependency used for glossary parsing.
- Pin ConfigKit v0.4.0 and align Go/toolchain requirements. ConfigKit requires
Go 1.25; the combined PromptKit prototype requires Go 1.25.5. Config-only
adoption does not automatically remove Audita's existing YAML dependency used for glossary parsing.
- Decide the configuration type shape, domain inheritance rules, and runtime
conversion boundary without leaking library concerns throughout the pipeline.
- Start with imports, profiles, and secrets-directory bootstrap disabled unless a
concrete suite or Audita requirement justifies enabling them. Adoption does not
require activating every library capability.
- Enable selected overlays as specified above. Keep imports and secret-directory
bootstrap disabled unless a separate concrete requirement justifies them.
- Define the desired behavior of validation and inspection without available
credentials under ConfigKit's secret-reference requirements.
- Decide which inspection and diagnostic interfaces benefit from provenance and