Plan ConfigKit overlays for testing with cheaper models
This commit is contained in:
@@ -20,8 +20,10 @@ guide instead of legacy loaders, aliases, dual operating modes, or an ongoing
|
||||
compatibility layer. Retain existing syntax only when it independently suits the
|
||||
new model, not because compatibility requires it.
|
||||
|
||||
The initial review used the sibling `../ConfigKit` checkout at v0.3.1. Reassess the
|
||||
supported release and current conventions during detailed planning. The installed
|
||||
The initial review and committed prototype used ConfigKit v0.3.1. The migration
|
||||
baseline is now v0.4.0, whose selected configuration feature is named overlays.
|
||||
An isolated upgrade passed the prototype tests and demonstration without source
|
||||
changes; the committed prototype dependency has not yet been updated. The installed
|
||||
application must depend on a versioned module, not a sibling checkout or local
|
||||
module replacement.
|
||||
|
||||
@@ -59,11 +61,75 @@ Transcript/glossary formats, processing algorithms, and output schemas are outsi
|
||||
this migration unless later planning identifies a concrete dependency or an
|
||||
explicitly coordinated change.
|
||||
|
||||
## Selected Overlays for Testing
|
||||
|
||||
Enable ConfigKit's optional selected-overlay feature in the migration. A testing
|
||||
configuration that selects cheaper, faster LLMs is the initial use case. ConfigKit
|
||||
**overlays** modify application settings; PromptKit **execution profiles** still
|
||||
own model and generation settings. Keep `profiles_dir`, the PromptKit `profiles/`
|
||||
directory, and proposal/validation profile names unchanged by this terminology
|
||||
change.
|
||||
|
||||
The planned interface is `--overlay <name>` on `process`, `config validate`, and
|
||||
`config print-effective`. Treat selection as invocation state passed to
|
||||
`Options.Overlay`, not as another persistent setting or environment overlay.
|
||||
Follow ConfigKit v0.4.0's selection contract: omission, an empty name, or `default`
|
||||
selects no overlay; a named selection loads `overlays/<name>.yml` beneath the selected
|
||||
configuration root. A missing selected overlay is an error. Use the library's
|
||||
single-overlay semantics without adding stacking, inheritance, or legacy-directory
|
||||
fallback.
|
||||
|
||||
Precedence is application defaults, main configuration, selected overlay, then
|
||||
explicit setting flags. Perform Audita's role inheritance and semantic validation
|
||||
on the resolved configuration. ConfigKit's base parsing and schema-version checks
|
||||
still precede overlay application; overlays cannot replace the schema version or
|
||||
contain ConfigKit directives. General environment-variable overrides remain retired.
|
||||
Imports and secret-directory bootstrap remain disabled.
|
||||
|
||||
Provide a maintained `overlays/testing.yml` example that selects testing execution
|
||||
profiles through `llm.proposal_profile` and `llm.validation_profile`. Model names and
|
||||
generation controls belong in the corresponding PromptKit profile files. Set both
|
||||
role selections in this example so a base configuration's explicit validation
|
||||
profile cannot accidentally retain the production model. Normal omission-based
|
||||
validation-profile inheritance continues to apply to the final effective settings.
|
||||
The example must make its backend/credential requirements clear and use no real
|
||||
secrets. Model choice is operator-controlled; “testing” does not promise a price,
|
||||
performance level, or equivalent correction quality.
|
||||
|
||||
Overlays may modify other ordinary application settings supported by the schema;
|
||||
they are not restricted to profile selection. The resulting configuration must
|
||||
still satisfy shared backend capacity, credential, and domain rules. Validation,
|
||||
effective-config inspection, selected-profile inspection, and processing must all
|
||||
use the same selected overlay. Preserve overlay provenance in configuration
|
||||
explanations and diagnostics while retaining secret redaction.
|
||||
|
||||
Implementation and acceptance work:
|
||||
|
||||
- Upgrade the isolated prototype to ConfigKit v0.4.0 and add a testing-overlay
|
||||
fixture before completing production configuration migration. Preserve the
|
||||
original v0.3.1 evidence as historical; overlay behavior is not yet proven by it.
|
||||
- Verify unchanged base selection when no overlay is requested, both testing role
|
||||
selections, and explicit setting flags taking precedence over the overlay.
|
||||
- Verify role inheritance after overlay application, effective profile/backend
|
||||
assembly, and secret resolution from the surviving configuration.
|
||||
- Reject missing/invalid overlay selections and prohibited version/directive changes
|
||||
through ConfigKit; rely on its tests for exhaustive merge/parser cases.
|
||||
- Verify redacted inspection and overlay provenance, then run both LLM roles through
|
||||
the offline provider using the selected testing profiles. Provider requests must
|
||||
use their model selections; no paid calls are needed.
|
||||
- Add production CLI coverage for the three commands and document selection,
|
||||
precedence, example layout, and migration guidance when implemented.
|
||||
|
||||
This expands the next configuration unit; it does not reopen the completed
|
||||
prototype's original scope or change the ConfigKit → PromptKit → combined release
|
||||
sequence. The production LLM migration remains necessary before promising the full
|
||||
new runtime behavior.
|
||||
|
||||
## Provisional Stages
|
||||
|
||||
First complete the [shared-boundary and prototype unit](library-boundary.md) from
|
||||
the coordinated adoption plan. It supplies the target settings model and integration
|
||||
evidence for the production configuration migration described here.
|
||||
The completed [shared-boundary and prototype unit](library-boundary.md) supplies
|
||||
the initial target settings model and integration evidence. Extend that evidence
|
||||
with the selected-overlay work above during the next configuration unit.
|
||||
|
||||
### 1. Define the Contract and Coordinate Sequencing
|
||||
|
||||
@@ -85,8 +151,8 @@ an implementation plan and concrete acceptance criteria before beginning migrati
|
||||
|
||||
After detailed planning, implement a bounded path from file selection and CLI
|
||||
inputs through ConfigKit to Audita's runtime configuration. Exercise defaults,
|
||||
proposal/validation inheritance, concurrency, duration values, secrets, and
|
||||
redacted inspection using representative valid and invalid inputs.
|
||||
selected overlays, proposal/validation inheritance, concurrency, duration values,
|
||||
secrets, and redacted inspection using representative valid and invalid inputs.
|
||||
|
||||
Use the result to assess whether domain rules remain clear and substantial old
|
||||
infrastructure can be removed. Resolve genuine library gaps deliberately, including
|
||||
@@ -141,14 +207,13 @@ Retention and transcript timestamp decisions are not automatically included.
|
||||
|
||||
## Open Design Decisions and Boundaries
|
||||
|
||||
- Select and pin the ConfigKit release and align Go/toolchain requirements. The
|
||||
reviewed version requires Go 1.25. Config-only adoption does not automatically
|
||||
remove Audita's existing YAML dependency used for glossary parsing.
|
||||
- Pin ConfigKit v0.4.0 and align Go/toolchain requirements. ConfigKit requires
|
||||
Go 1.25; the combined PromptKit prototype requires Go 1.25.5. Config-only
|
||||
adoption does not automatically remove Audita's existing YAML dependency used for glossary parsing.
|
||||
- Decide the configuration type shape, domain inheritance rules, and runtime
|
||||
conversion boundary without leaking library concerns throughout the pipeline.
|
||||
- Start with imports, profiles, and secrets-directory bootstrap disabled unless a
|
||||
concrete suite or Audita requirement justifies enabling them. Adoption does not
|
||||
require activating every library capability.
|
||||
- Enable selected overlays as specified above. Keep imports and secret-directory
|
||||
bootstrap disabled unless a separate concrete requirement justifies them.
|
||||
- Define the desired behavior of validation and inspection without available
|
||||
credentials under ConfigKit's secret-reference requirements.
|
||||
- Decide which inspection and diagnostic interfaces benefit from provenance and
|
||||
|
||||
Reference in New Issue
Block a user