Finalize the ConfigKit and PromptKit implementation plan
This commit is contained in:
@@ -2,9 +2,9 @@
|
||||
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Prototype boundary and selected overlays verified. The ordered
|
||||
production handoff below is planned, not implemented; no production migration or
|
||||
release has begun.
|
||||
**Status:** Final target selected; production implementation pending. The boundary
|
||||
and selected overlays are verified. The [implementation plan](implementation.md)
|
||||
owns execution stages; no production migration or release has begun.
|
||||
|
||||
## Purpose and Direction
|
||||
|
||||
@@ -27,10 +27,10 @@ application must depend on a versioned module, not a sibling checkout or local
|
||||
module replacement.
|
||||
|
||||
The shared boundary/prototype unit is complete. Its [selected contract, inventory,
|
||||
and implementation handoff](library-boundary-findings.md) refine the provisional
|
||||
directions here; production configuration migration has not begun.
|
||||
and evidence](library-boundary-findings.md) define the detailed target alongside
|
||||
this roadmap. Production configuration migration has not begun.
|
||||
|
||||
## Proposed End State
|
||||
## Target End State
|
||||
|
||||
- ConfigKit owns strict YAML loading, input presence, default application, explicit
|
||||
override mechanics, recorded resolution, secret references, and safe inspection.
|
||||
@@ -40,17 +40,17 @@ directions here; production configuration migration has not begun.
|
||||
Audita's current per-layer validation and environment-overlay behavior.
|
||||
- General configuration environment overrides are retired. Credentials use typed
|
||||
environment-name secret references; supplied references must resolve according
|
||||
to ConfigKit's contract. Review config-file selection separately as an
|
||||
application-owned startup decision.
|
||||
to ConfigKit's contract. File selection follows the application-owned
|
||||
discovery rules in the selected boundary.
|
||||
- Standardize duration representations, explicit empty/null/zero handling, and
|
||||
inheritance on the shared conventions. Express genuine Audita domain rules
|
||||
through the supported resolution and validation APIs.
|
||||
- Review the configuration-related CLI afresh. Useful explicit flags can become
|
||||
ConfigKit override records; old names and aliases are not requirements. Direct
|
||||
credential-value flags should give way to the secret-reference convention.
|
||||
- Retained domain-setting flags and proposal/validation profile flags become
|
||||
explicit ConfigKit overrides according to the migration inventory. Remove old
|
||||
provider/key/retry/concurrency flags and aliases. Credentials use secret references.
|
||||
- Use ConfigKit's redacted effective output and provide source/explanation access
|
||||
where useful. Existing JSON inspection output need not be preserved. Coordinate
|
||||
diagnostic configuration snapshots with the new model.
|
||||
through the selected views below. Legacy JSON inspection is replaced.
|
||||
Diagnostic configuration snapshots use the same new model.
|
||||
- Delete superseded loading and override machinery. Do not hide the old model
|
||||
behind adapters that defeat the adoption's consistency benefit.
|
||||
|
||||
@@ -102,228 +102,69 @@ effective-config inspection, selected-profile inspection, and processing must al
|
||||
use the same selected overlay. Preserve overlay provenance in configuration
|
||||
explanations and diagnostics while retaining secret redaction.
|
||||
|
||||
The completed prototype extension and production-planning handoff are recorded
|
||||
in the [prototype roadmap](library-boundary.md#next-unit-testing-overlays-and-production-migration-plan)
|
||||
and its ordered [implementation plan](implementation.md). Production CLI changes
|
||||
below remain subsequent migration work.
|
||||
The [recorded overlay results](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
|
||||
prove the representative load, inspection, and HTTP path. Production acceptance
|
||||
requires the same selection to reach both actual LLM roles, with explicit flags
|
||||
winning over overlays and surviving secret references resolved/redacted correctly.
|
||||
The isolated prototype is evidence, not a second supported runtime.
|
||||
|
||||
Implementation and acceptance work:
|
||||
## Inspection and Invocation Contract
|
||||
|
||||
- The isolated prototype pins ConfigKit v0.4.0 and includes the testing-overlay
|
||||
fixture. Preserve the original v0.3.1 evidence as historical; the
|
||||
[extension results](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
|
||||
establish the selected-overlay behavior.
|
||||
- Verify unchanged base selection when no overlay is requested, both testing role
|
||||
selections, and explicit setting flags taking precedence over the overlay.
|
||||
- Verify role inheritance after overlay application, effective profile/backend
|
||||
assembly, and secret resolution from the surviving configuration.
|
||||
- Reject missing/invalid overlay selections and prohibited version/directive changes
|
||||
through ConfigKit; rely on its tests for exhaustive merge/parser cases.
|
||||
- Verify redacted inspection and overlay provenance, then run both LLM roles through
|
||||
the offline provider using the selected testing profiles. Provider requests must
|
||||
use their model selections; no paid calls are needed.
|
||||
- Add production CLI coverage for the three commands and document selection,
|
||||
precedence, example layout, and migration guidance when implemented.
|
||||
`process`, `config validate`, and `config print-effective` share file/overlay
|
||||
selection and retained setting overrides. Invocation I/O paths remain process-only.
|
||||
Only explicitly supplied flags override settings. Validation requires an explicit
|
||||
config file; processing and inspection use the selected discovery order. Help is
|
||||
available before configuration loading. All three commands validate selected
|
||||
profiles without requiring a provider call during setup.
|
||||
|
||||
This expands the next configuration unit; it does not reopen the completed
|
||||
prototype's original scope or change the ConfigKit → PromptKit → combined release
|
||||
sequence. The production LLM migration remains necessary before promising the full
|
||||
new runtime behavior.
|
||||
`config print-effective --view settings|profiles|explain` exposes three YAML views:
|
||||
|
||||
## Ordered Production Migration Handoff
|
||||
- `settings` (default): ConfigKit's redacted effective configuration.
|
||||
- `profiles`: an Audita-owned summary of each selected role's profile, backend,
|
||||
model, temperature, token limit, top-p, timeout, service tier, and reasoning
|
||||
effort where present. Arbitrary extra provider parameters and credentials are
|
||||
omitted; this is a selected-controls view, not every execution input.
|
||||
- `explain`: field paths, sources, and derivation records from ConfigKit, without
|
||||
resolved secret values.
|
||||
|
||||
The [migration inventory](library-boundary-findings.md#migration-inventory) is the
|
||||
field-by-field old-to-new map. The [selected boundary](library-boundary-findings.md#configuration-and-profile-contract)
|
||||
owns defaults and semantic choices; the [v0.4.0 overlay results](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
|
||||
prove the representative load, inspection, and HTTP path. These packages are
|
||||
future work within one coordinated refactor to adopt both libraries. The order
|
||||
organizes implementation dependencies, not independently runnable product states.
|
||||
Intermediate commits need not build or run; do not add temporary scaffolding,
|
||||
compatibility shims, or a processing gate to support them. Remove superseded code
|
||||
as its replacement lands. Run focused checks as components become testable, and
|
||||
validate the complete application before declaring the refactor finished. The
|
||||
isolated prototype is evidence, not a second production runtime.
|
||||
A ConfigKit snapshot alone does not describe profile-owned controls. Profile
|
||||
inspection must enforce the registered-backend and credential boundary. Diagnostic
|
||||
snapshots use these safe views and preserve overlay/explicit-override provenance.
|
||||
The migration guide explains the intentional replacement of legacy inspection JSON.
|
||||
|
||||
### 1. Define and load the complete version-2 configuration
|
||||
## Integration and Completion Criteria
|
||||
|
||||
**Depends on:** the verified prototype. **Owns:** `internal/core/config` and its
|
||||
tests. Replace the version-1 YAML/env application path with one ConfigKit v0.4.0
|
||||
definition and a version-2 rule. Pin the published module in the root Go module
|
||||
and raise the production Go minimum as needed. Keep the existing YAML dependency
|
||||
used for glossary parsing. ConfigKit imports and secret-directory bootstrap stay
|
||||
disabled.
|
||||
ConfigKit and PromptKit form one intentional breaking refactor. Intermediate
|
||||
commits need not build or run. The target contains no temporary old-runtime adapter,
|
||||
compatibility shim, processing gate, dual loader, or automatic legacy conversion.
|
||||
The [combined feature roadmap](library-adoption.md) owns execution/reporting intent;
|
||||
[implementation.md](implementation.md) alone owns stage order and implementation tasks.
|
||||
|
||||
The definition must cover every retained domain setting in the inventory:
|
||||
ordered/repeated `pipeline.modules`, `output.schema`, validation prompt budget,
|
||||
chunking, normalization, thresholds, context, and diagnostics, plus `profiles_dir`,
|
||||
backend endpoint/concurrency/queue capacity/optional credential, and LLM proposal
|
||||
and validation profile selection, transient retries, repair attempts, and total
|
||||
operation timeout. Apply the inventory's defaults once. Preserve presence:
|
||||
omitted validation profile inherits the **final** proposal profile; explicit
|
||||
empty/null validation fails. Omitted or null section target means automatic, zero
|
||||
fails. Explicit empty module list means no modules. Optional credential null means
|
||||
unauthenticated; supplied references must resolve to nonempty values. Keep the
|
||||
selected numeric-second normalization fields numeric, and use a duration string
|
||||
for `llm.operation_timeout`; do not revive version-1 duration coercion. Validate
|
||||
backend limits, references, profile selections, thresholds, and cross-field domain
|
||||
rules on the resolved result. Preserve ConfigKit provenance for derived values.
|
||||
The final application loads all retained domain settings, assembles both selected
|
||||
profiles, and processes through PromptKit. Controlled HTTP evidence must connect
|
||||
configuration selection to actual requests, domain outcomes, safe diagnostics,
|
||||
and shared finite backend capacity. Maintained examples and one migration guide
|
||||
cover the complete file/flag/environment transition. Obsolete application-owned
|
||||
loading, transport, rendering, and scheduler machinery is removed.
|
||||
|
||||
**Acceptance/tests:** `internal/core/config` tests use temporary files to cover
|
||||
version rejection, strict fields, defaults, omission/null/zero, module order,
|
||||
final role inheritance, overlays and explicit override precedence, secret
|
||||
replacement/missing/null, and retained domain validation. Compare behavior to the
|
||||
prototype extension rather than duplicating ConfigKit's parser matrix. No legacy
|
||||
environment overlay or old-schema acceptance remains.
|
||||
The combined target is Audita v2.0.0. Release compliance and publication remain
|
||||
last and separate from implementing the refactor. Retention and transcript timestamp
|
||||
changes are not included; the selected retry/repair contract resolves the retry
|
||||
follow-up in [runtime behavior](runtime-behavior.md).
|
||||
|
||||
### 2. Select files and map invocation input
|
||||
|
||||
**Depends on:** package 1. **Owns:** `internal/cli/run.go`,
|
||||
`internal/cli/process_flags.go`, `internal/cli` tests, and `cmd/audita` subprocess
|
||||
tests. Select exactly one main file: explicit `--config`, then the two system
|
||||
locations in the selected boundary. Remove `AUDITA_CONFIG`; a missing explicit
|
||||
path fails, while absent system files are skipped. Require an explicit file for
|
||||
`config validate`. Parse help before loading configuration.
|
||||
|
||||
Add invocation-only `--overlay <name>` to `process`, `config validate`, and
|
||||
`config print-effective`, passing it to `Options.Overlay` for the same selected
|
||||
main file. Omitted, empty, or `default` selects the base. Keep transcript,
|
||||
glossary, output, and report paths outside persistent configuration. Map only
|
||||
flags actually supplied by the user into ConfigKit overrides; parser defaults
|
||||
must never defeat file or overlay values. Retain useful domain-setting flags from
|
||||
the inventory. Add profile-selection flags. Remove direct API-key/model/endpoint,
|
||||
old retry/role/global concurrency flags and aliases rather than translating them
|
||||
silently. Retire all general `AUDITA_*` configuration overlays listed in the
|
||||
inventory; environment names retain meaning only when explicitly referenced as
|
||||
secrets.
|
||||
|
||||
**Acceptance/tests:** CLI and subprocess tests cover discovery order, explicit
|
||||
missing files, all three commands with base/testing/default/missing selections,
|
||||
overlay then explicit-setting precedence, omitted flags, usage/error streams and
|
||||
exit classifications. Reject old flags and implicit environment configuration.
|
||||
The same loaded result must feed validation, effective inspection, and processing.
|
||||
|
||||
### 3. Assemble profiles and safe inspection
|
||||
|
||||
**Depends on:** packages 1–2. **Owns:** a small CLI/runtime assembly boundary,
|
||||
config inspection, and diagnostic snapshot wiring in `internal/cli` and
|
||||
`internal/core/diagnostics`; verify against `internal/framework/processreport`
|
||||
where snapshots are referenced. Resolve `profiles_dir` relative to the selected
|
||||
**main** file, including when an overlay is selected; absolute paths remain valid.
|
||||
Inspect both selected PromptKit profiles and verify their named registered backend,
|
||||
endpoint equality, and absence of a competing profile credential lookup. Treat
|
||||
ConfigKit settings and PromptKit generation controls as separate views.
|
||||
|
||||
Use `WriteEffective` for redacted effective YAML and expose relevant
|
||||
explanation/provenance and selected-profile model/backend controls through an
|
||||
Audita-owned safe view. A ConfigKit snapshot alone cannot claim to describe the
|
||||
full execution target. Validation and inspection resolve surviving secret
|
||||
references without provider calls; a missing or empty surviving reference fails,
|
||||
while a replaced reference is not resolved. Snapshot and surfaced error paths
|
||||
must redact secret **values** and preserve overlay/CLI origin. Avoid serializing
|
||||
raw library objects into stable report formats.
|
||||
|
||||
**Acceptance/tests:** config-command and diagnostics tests verify relative paths,
|
||||
same selection across views, overlay/CLI origin, surviving credentials,
|
||||
unauthenticated null, and no secret values in YAML, explanations, selected-profile
|
||||
inspection, diagnostics, or errors. Update `internal/cli/run_test.go`,
|
||||
`internal/core/config/*_test.go`, and report/diagnostics tests that own these
|
||||
outputs.
|
||||
|
||||
### 4. Connect the final configuration directly to PromptKit execution
|
||||
|
||||
**Depends on:** packages 1–3. **Owns:** the final CLI/runtime assembly boundary,
|
||||
`internal/framework/llm`, prompt/schema loading, runner wiring, and relevant tests.
|
||||
Implement PromptKit adoption as part of the same refactor. Convert domain settings
|
||||
into Audita's existing domain structs and supply the selected profiles, registered
|
||||
backends, resolved secrets, retry/repair budgets, and logical-operation deadlines
|
||||
directly to PromptKit. Keep library types out of correction algorithms and ledgers.
|
||||
|
||||
Migrate all module and validator prompts, preserve their domain checks, and map
|
||||
PromptKit results, classified failures, and observations into Audita reporting and
|
||||
redacted diagnostics. Replace the old provider transport, prompt loader,
|
||||
structural-output handling, and global/role scheduler composition. Follow the
|
||||
[coordinated execution scope](library-adoption.md#4-complete-promptkit-adoption)
|
||||
and [execution/observation contract](library-boundary-findings.md#execution-and-failure-mapping).
|
||||
Do not build an adapter to feed the old LLM runtime or introduce an interim
|
||||
`process` rejection mode.
|
||||
|
||||
**Acceptance/tests:** controlled HTTP tests prove both roles' final model/backend
|
||||
selection, overlays and explicit overrides, finite per-backend capacity, explicit
|
||||
zero retries, positive structural repair, total-operation deadlines, cancellation,
|
||||
domain acceptance/rejection, and safe observations including partial failures.
|
||||
Processing uses the new configuration and execution path together. Reuse existing
|
||||
domain coverage and the prototype's integration evidence; update tests tied to the
|
||||
retired transport and scheduler contracts.
|
||||
|
||||
### 5. Remove superseded configuration paths and update maintained contracts
|
||||
|
||||
**Depends on:** packages 1–4. **Owns:** removal of old file/env merge, aliases,
|
||||
direct-key flags, legacy redaction/inspection paths, and obsolete tests in
|
||||
`internal/core/config` and `internal/cli`. Retain Audita's domain checks,
|
||||
normalization/chunking, output encoding, module and validator behavior. Update
|
||||
`examples/` with version-2 base configuration, `overlays/testing.yml`, complete
|
||||
PromptKit profiles, and secret-free credentials by reference. Update the canonical
|
||||
`docs/config.md`, `docs/cli.md`, operations/integration/internal references as
|
||||
their contracts change, and provide one migration guide with the complete
|
||||
inventory's file/flag/environment mapping, duration and empty-list changes,
|
||||
credentials, profiles/overlays, inspection, and changed runtime/diagnostic
|
||||
semantics. Do not leave dual loaders or a permanent translation mode.
|
||||
|
||||
**Acceptance/tests:** maintained examples validate offline; removed flags and old
|
||||
schema are rejected; relevant config/CLI/subprocess/report tests pass; documentation
|
||||
links and `git diff --check` pass. Remove the isolated prototype when its useful
|
||||
coverage and examples have moved into the production suite.
|
||||
|
||||
### 6. Validate the combined end state and prepare release
|
||||
|
||||
**Depends on:** packages 1–5. Verify the assembled version-2 configuration,
|
||||
overlay, profile, execution, reporting, and artifact workflows. Run the relevant
|
||||
full build/test checks, confirm obsolete configuration and LLM machinery is gone,
|
||||
and verify maintained examples and the migration guide against the final behavior.
|
||||
There is no requirement to validate or preserve intermediate product states.
|
||||
|
||||
The refactor is complete when both ConfigKit and PromptKit are adopted and the
|
||||
complete application passes its required checks. Release compliance, license/
|
||||
distribution review, and publication follow last under the established release
|
||||
procedure. No publication is authorized by this plan.
|
||||
|
||||
## Sequencing Considerations
|
||||
|
||||
Changes to module configuration, runtime settings, CLI design, or diagnostic
|
||||
reporting are likely to overlap this migration. Design their shared target model
|
||||
first. If another change will substantially reshape those areas, avoid migrating
|
||||
the same concepts twice. Work independent of configuration can proceed separately.
|
||||
|
||||
The [coordinated library-adoption roadmap](library-adoption.md) owns the proposed
|
||||
sequence with PromptKit: shared boundary design and a PromptKit prototype, then
|
||||
ConfigKit and PromptKit implementation in dependency order within one refactor,
|
||||
followed by a combined v2.0.0 release. Intermediate commits are work in progress,
|
||||
not supported configurations or runnable milestones.
|
||||
|
||||
The separate [runtime behavior decisions](runtime-behavior.md) track remaining
|
||||
questions and the selected retry direction.
|
||||
The coordinated plan adopts PromptKit's explicit retry semantics and enables
|
||||
structural repair from the outset. Include separate transient-retry and positive
|
||||
repair budgets, plus the total-operation deadline policy, in the shared settings
|
||||
design. The selected defaults and ownership are recorded in the
|
||||
[boundary findings](library-boundary-findings.md#configuration-and-profile-contract).
|
||||
Implement the runtime change explicitly as part of the combined refactor.
|
||||
Retention and transcript timestamp decisions are not automatically included.
|
||||
|
||||
## Implementation Boundaries
|
||||
## Scope Boundaries
|
||||
|
||||
- Pin ConfigKit v0.4.0 in production and align Go/toolchain requirements. ConfigKit requires
|
||||
Go 1.25; the combined PromptKit prototype requires Go 1.25.5. Config-only
|
||||
adoption does not automatically remove Audita's existing YAML dependency used for glossary parsing.
|
||||
Go 1.25; the combined PromptKit prototype requires Go 1.25.5. Adopting these libraries
|
||||
does not automatically remove Audita's existing YAML dependency used for glossary parsing.
|
||||
- Implement the selected configuration and final PromptKit runtime directly,
|
||||
without temporary runtime adapters, compatibility shims, or process gates.
|
||||
- Enable selected overlays as specified above. Keep imports and secret-directory
|
||||
bootstrap disabled unless a separate concrete requirement justifies them.
|
||||
- Validation and inspection resolve surviving credential references and fail on
|
||||
missing or empty values; no credential-free bypass is planned.
|
||||
- Add the safe selected-profile view and ConfigKit provenance to inspection and
|
||||
diagnostics as specified in package 3, then account for changed formats in the
|
||||
migration guide. No schedule or effort estimate is committed.
|
||||
- Inspection and diagnostics use the safe selected-profile view and ConfigKit
|
||||
provenance described above. Changed formats are covered by the migration guide.
|
||||
|
||||
The eventual completion test is that Audita follows the suite's ConfigKit
|
||||
conventions while retaining its intended domain capabilities, with a clear
|
||||
|
||||
@@ -1,306 +1,527 @@
|
||||
# Implementation Plan: Testing Overlays and Migration Handoff
|
||||
# Implementation Plan: Adopt ConfigKit and PromptKit
|
||||
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Complete (2026-09-24). All six stages below are finished. This plan implements the next
|
||||
unit in the [prototype roadmap](library-boundary.md#next-unit-testing-overlays-and-production-migration-plan),
|
||||
not the production ConfigKit or PromptKit migrations.
|
||||
**Status:** Ready for implementation. All stages below are pending. This replaces
|
||||
the completed testing-overlay prototype plan; its results remain in the
|
||||
[boundary findings](library-boundary-findings.md#selected-overlay-extension-2026-09-24).
|
||||
|
||||
**Production handoff revision:** The completed prototype stages remain recorded
|
||||
below. The current [production plan](configkit.md#ordered-production-migration-handoff)
|
||||
targets both libraries in one refactor; intermediate commits need not build or run.
|
||||
It supersedes the earlier interim-runtime design. No temporary adapter,
|
||||
compatibility shim, or processing gate is required.
|
||||
## Scope and Working Instructions
|
||||
|
||||
Execute Stage 1 through Stage 6 in order. Each stage is sized as a separate coding
|
||||
prompt for a gpt-5.6-terra agent and ends with a reviewable result. When assigned
|
||||
one stage, finish that stage and report the next stage; when assigned the entire
|
||||
plan, continue through all stages. Do not restart completed work: inspect current
|
||||
files, recorded results, and Git state before deciding what remains.
|
||||
Implement the final configuration and execution model in the
|
||||
[combined feature roadmap](library-adoption.md) and [configuration roadmap](configkit.md).
|
||||
The [selected boundary](library-boundary-findings.md#configuration-and-profile-contract),
|
||||
[migration inventory](library-boundary-findings.md#migration-inventory), and
|
||||
[failure mapping](library-boundary-findings.md#execution-and-failure-mapping)
|
||||
are the detailed behavioral inputs. Do not repeat the prototype or reopen settled
|
||||
choices without a concrete contradiction in the pinned libraries.
|
||||
|
||||
Before each stage, follow `AGENTS.md`, [development guidance](../development.md),
|
||||
and the [agent policy](../policy/agents.md). Preserve unrelated changes. Use the
|
||||
knowledge graph for code discovery; examples may be excluded from its index, in
|
||||
which case inspect the known prototype files directly. Follow the repository's
|
||||
testing and documentation policies. Complete relevant checks and make a plain
|
||||
English local commit for each coherent stage under the standing commit policy.
|
||||
No push, tag, release, or upstream-library edit is included.
|
||||
This plan is for a gpt-5.6-terra coding agent. Execute Stage 1 through Stage 12 in
|
||||
order. Each stage is one bounded implementation prompt. When assigned the whole
|
||||
plan, continue between stages without a permission round. When assigned one stage,
|
||||
finish that stage's scope and report the next stage. Mark completed stages with a
|
||||
short result and actual validation evidence; do not call a pending integration
|
||||
check passed.
|
||||
|
||||
Update each stage's status line with its result and verification evidence when
|
||||
finished; keep detailed design/results in their owning roadmap documents. Do not
|
||||
mark a stage complete while required checks are failing. Fix in-scope failures;
|
||||
record any genuine upstream gap and its effect on later stages before continuing
|
||||
work that depends on it. Routine implementation choices do not need fresh approval.
|
||||
Read `AGENTS.md`, [development guidance](../development.md), and all documents under
|
||||
`docs/policy/`. Use graph-first code discovery. Follow the
|
||||
[agent](../policy/agents.md), [architecture](../policy/architecture.md),
|
||||
[testing](../policy/testing.md), [documentation](../policy/documentation.md), and
|
||||
[release](../policy/release.md) policies. The intentional dependency, configuration,
|
||||
and scheduler changes here supersede the corresponding old implementation choices;
|
||||
update canonical descriptions as the new behavior lands.
|
||||
|
||||
The [ConfigKit roadmap](configkit.md#selected-overlays-for-testing) owns the overlay
|
||||
contract. The [boundary findings](library-boundary-findings.md) own existing choices
|
||||
and evidence. Preserve the original v0.3.1 results as historical. ConfigKit v0.4.0
|
||||
renames selected configuration profiles to overlays; do not rename PromptKit's
|
||||
`profiles/`, `profiles_dir`, or execution-profile fields.
|
||||
The stages divide one refactor, not supported intermediate releases. Intermediate
|
||||
commits may fail to build or run. Do not write temporary adapters, compatibility
|
||||
shims, duplicate execution paths, stubs returning invented successes, or an interim
|
||||
processing gate. Remove superseded code when replacing it. Run focused checks
|
||||
where the affected packages are testable; record checks deferred because later
|
||||
stages must reconnect callers. Such a completed stage is still work in progress
|
||||
on the overall refactor. Commit coherent stages with plain-English messages under
|
||||
the standing policy. Do not spend a stage making an intermediate application work.
|
||||
|
||||
All implementation in this plan stays under `examples/library-adoption/`, plus
|
||||
relevant roadmap/README updates. Keep the root Go module, production CLI/runtime,
|
||||
correction algorithms, and upstream libraries unchanged. Use the existing built-in
|
||||
PromptKit HTTP path with a controlled transport, synthetic credentials, and no live
|
||||
provider calls. Keep imports and secret-directory bootstrap disabled. This work
|
||||
does not introduce general environment overlays or backwards compatibility.
|
||||
Keep tests deterministic, offline, and credential-independent. Reuse prototype
|
||||
scenarios at their production boundary rather than copy the nested application or
|
||||
retest all library internals. Preserve domain tests unless the selected contract
|
||||
actually changes. Update each affected canonical reference alongside its owning
|
||||
stage; Stage 11 finishes the cross-document migration guide and examples.
|
||||
|
||||
Unless explicitly stated otherwise, shell commands below run from
|
||||
`examples/library-adoption/`. Use an installed Go 1.25.5 or newer. Acquire dependencies
|
||||
with `GOWORK=off go mod download all` when needed; then use the offline command
|
||||
prefix below. Root `go test ./...` does not cover this nested module.
|
||||
No upstream edits, paid calls, push, tag, publication, retention-policy change,
|
||||
timestamp fix, or unrelated pipeline redesign is included. Release compliance
|
||||
and publication remain a later release unit. Completion here means both libraries
|
||||
are adopted, obsolete machinery is removed, and the final application is verified.
|
||||
|
||||
## Stage 1: Upgrade the Prototype Dependency
|
||||
## Implementation Decisions
|
||||
|
||||
**Status:** Complete (2026-09-24). **Depends on:** nothing. ConfigKit v0.4.0 pinned;
|
||||
the nested offline tests and demonstration passed with the expected 1/4/0/2 result.
|
||||
These decisions close the remaining application-design details; background and
|
||||
user-facing policy belong to the feature roadmaps.
|
||||
|
||||
Read the prototype README, `go.mod`, `go.sum`, and the current ConfigKit roadmap.
|
||||
Confirm the working tree and run the existing nested-module tests once as a
|
||||
baseline. Upgrade ConfigKit to the published v0.4.0 release using module tooling;
|
||||
leave PromptKit and Audita dependency versions unchanged. Review any indirect
|
||||
module changes and retain only changes needed by this upgrade. Do not add local
|
||||
replacements or an active Go workspace.
|
||||
- Pin the published modules used by the verified prototype: ConfigKit v0.4.0 and
|
||||
PromptKit v0.11.0. Set the root minimum to Go 1.25.5. Keep yaml.v3 while glossary
|
||||
parsing uses it. No sibling replacements or dependency on Audita's own old release.
|
||||
- Keep application loading in `internal/core/config`, embedded prompt definitions
|
||||
in `internal/prompts`, schemas in `internal/framework/responseschema`, and the
|
||||
PromptKit integration in `internal/framework/llm`. Use one engine per process
|
||||
invocation, shared by both role-bound clients; neither modules nor validators
|
||||
construct engines. The CLI composes these pieces. Avoid a generic service layer.
|
||||
- Retain the `CompleteStructured(ctx, request, out)` boundary, using Audita-owned
|
||||
prompt ID/version, string variables, schema identity, and diagnostic context.
|
||||
Replace the old rendered-message/model request fields. Model/profile/credential
|
||||
selection belongs to the role-bound adapter, not individual callers. Consolidate
|
||||
the equivalent validator types onto the framework contract where needed, without
|
||||
an import cycle. No PromptKit types in correction algorithms or ledger records.
|
||||
- Store PromptKit definitions for all nine existing prompt IDs with their existing
|
||||
`v1` selection. Replace the custom `hardening` template function with an
|
||||
application-supplied `HardeningPolicy` string variable sourced from the one
|
||||
embedded hardening fragment. The adapter supplies that reserved variable after
|
||||
caller variables. PromptKit renders all messages; do not retain an Audita template
|
||||
engine or introduce generated prompt files requiring a second build step.
|
||||
- Use one shared configuration-input parser for `process`, `config validate`, and
|
||||
`config print-effective`. All three accept `--config`, `--overlay`, and the
|
||||
retained setting overrides in the inventory, including the two profile flags.
|
||||
Only `process` accepts transcript/glossary/output/report paths. Only explicitly
|
||||
visited setting flags become ConfigKit overrides. Do not add retry/capacity/key
|
||||
flags beyond the agreed inventory.
|
||||
- `config validate` requires an explicit nonempty config path. The other two
|
||||
commands use the selected discovery order. All validate the resolved settings
|
||||
and both selected profiles without provider calls, including an empty module
|
||||
pipeline. Help succeeds before file discovery or secret/profile loading.
|
||||
- `config print-effective` adds `--view settings|profiles|explain`, defaulting to
|
||||
`settings`. Settings uses ConfigKit `WriteEffective` YAML. Profiles emits a safe
|
||||
Audita-owned YAML summary for the proposal and validation roles. Explain emits
|
||||
a YAML sequence of field path, source, and derivation records from ConfigKit's
|
||||
recorded resolution, with no resolved secret values. Unknown views are usage
|
||||
errors. This avoids mixing multiple formats into the default output or adding
|
||||
another command tree. Profiles includes selected ID, backend, model, temperature,
|
||||
max tokens, top-p, per-attempt timeout, service tier, and reasoning effort when
|
||||
present; omit arbitrary extra provider parameters and credentials. It is a
|
||||
selected-controls summary, not a dump of every profile input.
|
||||
- Keep CLI exit classes 0/1/2 and transcript/report stream behavior. Invalid user
|
||||
settings/profiles are configuration errors (2); broken embedded assets and
|
||||
runtime/provider failures are runtime errors (1). Serialization/write failures
|
||||
remain runtime errors. Do not classify malformed HTTP envelopes as repairable
|
||||
model output.
|
||||
- Publish process reports as `audita-process-report` schema `v2`. Preserve existing
|
||||
domain fields and ledger ordering, add a top-level `llm_summary`, and replace
|
||||
obsolete scheduler utilization fields as specified in Stage 10. Transcript
|
||||
encoder versions stay unchanged. No legacy report mode is needed.
|
||||
|
||||
No source changes were needed in the earlier isolated review. If a change is now
|
||||
necessary, investigate and record the concrete reason instead of assuming that
|
||||
ConfigKit execution overlays and PromptKit profiles share an API.
|
||||
## Stage 1: Replace the Configuration Model and Loader
|
||||
|
||||
Verification:
|
||||
**Status:** Pending. **Depends on:** the completed prototype only.
|
||||
**Owns:** root `go.mod`/`go.sum`, `internal/core/config`, and config tests.
|
||||
|
||||
```sh
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test ./...
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run .
|
||||
```
|
||||
1. Pin both modules and the Go minimum above. Align build/CI toolchain declarations
|
||||
with that minimum; preserve the existing platform set and release gates.
|
||||
2. Replace version-1 YAML/env merge and duration coercion with one ConfigKit
|
||||
definition covering every retained field in the migration inventory. Use the
|
||||
prototype's types/lifecycle as evidence, then add the full domain configuration.
|
||||
Remove legacy environment loaders and alias resolution; do not keep two loaders.
|
||||
3. Implement defaults, presence-aware backend defaults, final validation-profile
|
||||
inheritance, and recorded derivations. Keep `chunking.target_sections` optional,
|
||||
explicit empty module lists empty, and module order/repeats intact. Validate
|
||||
domain constraints after overlay and explicit override resolution. Preserve
|
||||
numeric normalization seconds; operation timeout is a duration string.
|
||||
4. Use typed optional secret references. Resolve every surviving reference, including
|
||||
unselected backends; missing/empty references fail, replacement resolves only the
|
||||
replacement, and null permits unauthenticated use. Enable selected overlays;
|
||||
leave imports and secret-directory bootstrap disabled.
|
||||
5. Expose the ConfigKit result to CLI composition for effective output and provenance;
|
||||
convert retained settings into existing domain inputs without transporting the
|
||||
library result through correction code. Update `docs/config.md`'s implemented
|
||||
loader/schema description, clearly marking unfinished CLI integration if needed.
|
||||
|
||||
Acceptance: the manifest and sums pin v0.4.0, tests pass, and the demonstration
|
||||
still yields one accepted correction, four attempts, zero transient retries, and
|
||||
two structural repairs. Update the README/findings to distinguish the new pin from
|
||||
historical v0.3.1 evidence. Do not claim overlay integration yet.
|
||||
**Acceptance and validation:** focused config tests cover strict version/fields,
|
||||
selected defaults, omission/null/zero, final role inheritance, empty/repeated
|
||||
modules, overlay/override precedence, secret replacement/null/missing, and retained
|
||||
cross-field rules. Old schema fails with migration guidance. General environment
|
||||
variables have no implicit effect. Run `go test ./internal/core/config` when this
|
||||
package is independently buildable; record caller compile failures for later stages.
|
||||
Do not duplicate ConfigKit's exhaustive YAML/merge tests.
|
||||
|
||||
## Stage 2: Add Overlay Loading and Configuration Fixtures
|
||||
## Stage 2: Replace CLI Configuration Selection and Overrides
|
||||
|
||||
**Status:** Complete (2026-09-24). **Depends on:** Stage 1. On-disk overlay and
|
||||
profile fixtures, precedence, inheritance, rejection, and runtime assembly tests
|
||||
pass in the nested module (`go test ./...`).
|
||||
**Status:** Pending. **Depends on:** Stage 1.
|
||||
**Owns:** `internal/cli/run.go`, `process_flags.go`, configuration command parsing,
|
||||
and corresponding CLI/subprocess tests.
|
||||
|
||||
Start with `config.go`, `config_test.go`, `config.yaml`, and `profiles/`. Extend
|
||||
`loadConfig` with an explicit overlay-selection argument or a small invocation
|
||||
options value. Pass it to ConfigKit `Options.Overlay`; selection does not belong
|
||||
in the persistent `Config` struct. Update existing callers to request no overlay
|
||||
without changing their behavior. Avoid a parallel merge implementation.
|
||||
1. Implement the shared configuration-input parser described above. Preserve the
|
||||
existing supported positional-argument arrangement for `process`, required
|
||||
glossary, output/report behavior, and root/subcommand help. Remove obsolete
|
||||
LLM/model/key/timeout/retry/concurrency flags and aliases outright.
|
||||
2. Select exactly one file: explicit `--config`, then
|
||||
`/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`. Missing system
|
||||
candidates are skipped; selected inaccessible files and explicit empty/missing
|
||||
paths fail. With no file, report a configuration error; do not select a provider.
|
||||
Remove `AUDITA_CONFIG`. Test discovery using an injected path list/filesystem
|
||||
seam rather than modifying the host's system configuration.
|
||||
3. Pass invocation overlay selection to ConfigKit and only supplied setting flags
|
||||
to its overrides. `--modules ''` must clear the pipeline; explicit numeric zero
|
||||
reaches validation rather than being treated as absent. Relative profile bases
|
||||
derive from the selected main file, never the overlay directory.
|
||||
4. Wire the parsed result into the final process/config command entry points.
|
||||
Profile-dependent command completion belongs to Stage 4; do not emulate the old
|
||||
runtime to make the process command work now. Update `docs/cli.md` for the final
|
||||
input contract as parsing lands.
|
||||
|
||||
Add these prototype fixtures:
|
||||
**Acceptance and validation:** parser/loading tests prove help without usable
|
||||
configuration, discovery/error cases, explicit-only overrides, retained domain
|
||||
flags, removed flag rejection, and ignored legacy environment values. Table-drive
|
||||
shared cases; retain representative subprocess checks rather than triplicate every
|
||||
parser test across commands. Config commands reject positional arguments and
|
||||
process-only flags. Defer assembled command checks until Stage 8 if imports still
|
||||
prevent compilation.
|
||||
|
||||
- `overlays/testing.yml`, selecting `proposal-testing` and `assessment-testing`
|
||||
through the existing two `llm` role fields. Do not put `version`, model settings,
|
||||
or ConfigKit directives in this overlay.
|
||||
- Complete PromptKit profiles under `profiles/` with those IDs and distinct
|
||||
synthetic model IDs, such as `synthetic-grammar-testing` and
|
||||
`synthetic-review-testing`. Both use the existing `offline` backend and keep the
|
||||
current positive repair policy and finite shared capacity.
|
||||
## Stage 3: Migrate Embedded Prompt Assets and Call Contracts
|
||||
|
||||
Preserve base `config.yaml` as the no-overlay fixture, including its explicitly
|
||||
selected assessment profile. This makes the risk of accidentally leaving the
|
||||
validation role on its base model observable. Profile paths remain relative to
|
||||
main configuration according to the selected application contract, not relative
|
||||
to the overlay file.
|
||||
**Status:** Pending. **Depends on:** Stage 1; execute after Stage 2.
|
||||
**Owns:** `internal/prompts`, `internal/framework/responseschema`,
|
||||
`internal/framework/contracts`, validator contract aliases, and prompt tests.
|
||||
|
||||
Add focused configuration tests proving:
|
||||
1. Convert all four module and five validator prompts listed in
|
||||
[the prompt registry](../internal/prompts.md) to PromptKit definitions. Preserve
|
||||
instructions, variables, glossary/description context, prompt IDs, and the two
|
||||
existing JSON response schemas. Embed assets once and expose their filesystem
|
||||
to the runtime. Do not copy the prototype's duplicate schema directory.
|
||||
2. Keep output contracts JSON Schema based, but supply the full effective output
|
||||
contract and repair count on each runtime request. Do not create separate
|
||||
operator-facing repair defaults per prompt. Use the hardening variable decision
|
||||
above to retain a single hardening source.
|
||||
3. Change the structured-call request to prompt identity, variables, response schema
|
||||
identity, and diagnostic context. Keep decoded output and typed Audita metadata
|
||||
as the return boundary. Include role/module instance/section/batch/validator
|
||||
context where available for attribution; do not derive identities from messages.
|
||||
4. Replace the old parser/renderer and hash assumptions. Keep small metadata and
|
||||
schema lookup helpers where Audita needs them. Update the prompt reference.
|
||||
|
||||
- Omitted selection, empty selection, and `default` select the base configuration;
|
||||
`testing` selects both new profiles.
|
||||
- Explicit ConfigKit overrides win over overlay values. Only supplied flags will
|
||||
become overrides in Stage 3; parser defaults must not enter this layer.
|
||||
- With validation selection absent from both base and overlay, validation inherits
|
||||
the final proposal selection, including an explicit override. An existing
|
||||
explicit base validation selection remains explicit unless replaced by overlay
|
||||
or CLI. Empty/null validation values retain the established rejection behavior.
|
||||
- Runtime profile assembly resolves both selected profiles to the shared backend.
|
||||
**Acceptance and validation:** a table covering all nine definitions prepares
|
||||
representative variable sets through PromptKit with controlled profile fixtures;
|
||||
checks include hardening, section/glossary/description data, missing variables,
|
||||
correct schema association, and rendered content preserving the previous task
|
||||
instructions. Use reviewed expectations, not a retained legacy renderer or old
|
||||
Audita module dependency. Existing schema/domain tests remain meaningful. Callers
|
||||
are deliberately migrated in Stages 6–7.
|
||||
|
||||
Use temporary on-disk layouts for overlay tests so ConfigKit owns file discovery.
|
||||
Existing byte-source tests can remain for cases that do not select an overlay;
|
||||
if a byte source needs an overlay, supply ConfigKit's supported filesystem context
|
||||
rather than relying on the process working directory.
|
||||
## Stage 4: Assemble Profiles and Finish Offline Inspection
|
||||
|
||||
Acceptance: the new fixtures load and assemble correctly, the focused precedence/
|
||||
inheritance tests pass, and `go test ./...` in the nested module still passes.
|
||||
Do not yet claim that the testing models reached the HTTP transport.
|
||||
**Status:** Pending. **Depends on:** Stages 1–3.
|
||||
**Owns:** engine/profile assembly in `internal/framework/llm`, CLI config commands,
|
||||
and inspection tests.
|
||||
|
||||
## Stage 3: Demonstrate and Verify Model Switching
|
||||
1. Construct one PromptKit engine with the embedded assets, complete profile files,
|
||||
and registered application backends. Reuse the prototype's supported filesystem,
|
||||
backend, retry-config, and inspection APIs. Accept an HTTP client at the external
|
||||
boundary for tests; production uses the library transport.
|
||||
2. Resolve `profiles_dir` against the main config file. Inspect both selected roles,
|
||||
require a registered backend, reject endpoint-only bypass and profile credential
|
||||
lookup, and compare endpoints after trimming surrounding whitespace and trailing
|
||||
slashes, as in the prototype. Do not invent profile inheritance/catalog wiring
|
||||
or reject unrelated unselected profile content beyond library behavior.
|
||||
3. Complete `config validate` and the three effective-output views. Settings,
|
||||
profiles, and explanations share the same load/assembly path used by process.
|
||||
Keep the `config is valid` success message. Redact configured secret values in
|
||||
any surfaced library error; do not stringify raw configuration/profile objects.
|
||||
4. Add the internal safe-view helpers later used for snapshots. Document the view
|
||||
interface and its limits in `docs/cli.md` and `docs/config.md`.
|
||||
|
||||
**Status:** Complete (2026-09-24). **Depends on:** Stage 2. Nested tests inspect
|
||||
outbound base/testing/override model IDs, including repairs; all three documented
|
||||
offline demonstrations passed with one accepted correction and 4/0/2 attempts,
|
||||
retries, and repairs.
|
||||
**Acceptance and validation:** offline command tests prove all views and validation
|
||||
agree for base/testing/explicit-profile selections; main-file-relative and absolute
|
||||
profile paths work; missing profiles, backend mismatch, competing credentials,
|
||||
missing surviving secrets, and invalid views fail with the selected exit class.
|
||||
Use a transport that fails the test if called. Inspect outputs/errors for synthetic
|
||||
secret absence and overlay/CLI provenance, including inherited validation selection.
|
||||
|
||||
Start with `main.go`, `offline.go`, `runtime.go`, and `workflow_test.go`. Add minimal
|
||||
standard-library argument handling to the prototype executable:
|
||||
## Stage 5: Implement the Final PromptKit Structured-Call Adapter
|
||||
|
||||
- `--overlay <name>` selects the ConfigKit overlay.
|
||||
- `--proposal-profile <id>` and `--validation-profile <id>` produce ConfigKit
|
||||
overrides only when explicitly supplied.
|
||||
- Keep the existing fixture-based invocation from the prototype directory. There
|
||||
is no need to reproduce production file discovery or its subcommand tree here.
|
||||
**Status:** Pending. **Depends on:** Stages 3–4.
|
||||
**Owns:** `internal/framework/llm`, the typed failure boundary in
|
||||
`internal/framework/structuredoutput`, and adapter tests.
|
||||
|
||||
Use a small testable entry function if needed to check flag handling without
|
||||
repeated subprocess builds. Preserve ordinary flag error handling. Keep execution
|
||||
on the synthetic transport for every supported demonstration invocation; do not
|
||||
add a live-network mode. Extend that transport to recognize the testing models and
|
||||
preserve the initial-invalid-output/repair behavior for each role.
|
||||
1. Implement role-bound clients over the shared engine. Start the operation context
|
||||
and Audita logical-call record before preparation; inherit earlier caller deadlines.
|
||||
Select the role profile and backend secret, supply reserved hardening variables,
|
||||
and pass an explicit full schema/repair contract. Use `PrepareExecution`, its
|
||||
`Details`, then `RunPrepared` so diagnostic messages/hashes describe the actual
|
||||
execution snapshot. Defer handle discard; do not call `Prepare` and then `Run`
|
||||
with an independently reloaded profile. Before dispatch, verify the prepared
|
||||
target still matches the role's registered backend/endpoint and has no competing
|
||||
credential lookup. A changed target fails safely rather than sending a backend
|
||||
key to a different target. This is the final diagnostic/security boundary, not
|
||||
an intermediate adapter.
|
||||
2. Check both the returned error and `result.Validation.IsValid`: exhausted repair
|
||||
can return a result with failed validation and no error. Decode only validated
|
||||
JSON into the requested Audita output. Use the selected failure table:
|
||||
exhausted output repair becomes the Audita malformed-output classification; provider/transport/protocol/retry/admission/deadline failures
|
||||
remain runtime failures. Preserve typed causes for classification and safe
|
||||
messages for users. Explicit zero retries must not become retries-exhausted.
|
||||
3. Preserve prepared messages/hashes and available final content and profile/model
|
||||
metadata through Audita-owned types. Provide a per-call observer hook and logical
|
||||
start/finish record bound to Audita diagnostic context for Stage 9; do not
|
||||
create another retry loop, scheduler, provider envelope parser, or schema engine.
|
||||
4. Remove the old OpenAI-compatible client and its role config resolution. Retain
|
||||
only applicable redaction and diagnostics helpers. Update the LLM integration
|
||||
reference to delegate wire/retry mechanics to the pinned library.
|
||||
|
||||
Add a compact safe selection summary to demonstration output: requested overlay
|
||||
and effective profile, backend, and model for each role, alongside the existing
|
||||
domain outcome and attempt/retry/repair totals. It must reflect the assembled
|
||||
configuration rather than echoing flag values. Inspection APIs and tests in the
|
||||
next stage suffice; do not create production config commands in this prototype.
|
||||
**Acceptance and validation:** controlled HTTP tests cover successful decode,
|
||||
positive repair and exhaustion for both schema types, 429/5xx retry recovery,
|
||||
explicit zero, exhausted positive budget, transport/malformed-envelope failure,
|
||||
combined retry-plus-repair bounds, preparation failure without dispatch, exact
|
||||
prepared request metadata, target-change rejection, and operation cancellation
|
||||
during a long retry wait. Assert classifications and meaningful counts, not timing-sensitive sleeps
|
||||
or complete error strings. Reuse the prototype sequences; no need to reproduce
|
||||
PromptKit's entire status-code matrix.
|
||||
|
||||
Test through the assembled path and inspect actual outbound HTTP request bodies:
|
||||
## Stage 6: Migrate All Proposal Callers
|
||||
|
||||
- No overlay sends the base proposal and assessment models.
|
||||
- `testing` sends the testing proposal and assessment models, including repair
|
||||
requests, and still produces one accepted correction with two repairs.
|
||||
- An explicitly supplied role-profile flag overrides that role's testing selection;
|
||||
the other role retains its overlay selection. An omitted flag preserves overlay
|
||||
selection. Match reported selections to observed outbound model IDs.
|
||||
**Status:** Pending. **Depends on:** Stage 5.
|
||||
**Owns:** `internal/modules`, `internal/framework/proposal_generation`, and their tests.
|
||||
|
||||
Use distinct models per role for these demonstration cases. Do not expand the fake
|
||||
provider into a general simulation framework. Reuse existing prompt/schema/domain
|
||||
helpers and existing capacity/observation coverage.
|
||||
1. Switch glossary, homophones, spoken_word, and grammar proposal construction to
|
||||
prompt IDs and variables. Preserve section/glossary encoding, context trimming,
|
||||
replacement policy, confidence filtering, and applicability checks.
|
||||
2. Feed repaired structured output into the existing domain path. Exhausted
|
||||
structural repair produces the existing section warning and no proposals;
|
||||
runtime failures propagate. Schema-valid but inapplicable proposals do not
|
||||
trigger semantic retries or additional repairs.
|
||||
3. Update fake clients and module prompt tests to the final request contract.
|
||||
Delete duplicate message rendering and obsolete transport/schema checks; retain
|
||||
domain protection and prompt variable coverage. Update module internals docs.
|
||||
|
||||
Acceptance: nested-module tests pass and these invocations visibly show the expected
|
||||
selections and successful domain result:
|
||||
**Acceptance and validation:** all four modules have an exercised variable/schema
|
||||
mapping; existing domain suites retain their assertions. Representative controlled
|
||||
adapter coverage proves repaired proposals reach domain checks and that a low-
|
||||
confidence/inapplicable proposal remains rejected. Preserve section/proposal
|
||||
ordering. Run the module and proposal-generation suites as imports become buildable.
|
||||
|
||||
```sh
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run .
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing --proposal-profile proposal
|
||||
```
|
||||
## Stage 7: Migrate LLM Validators Without Changing Domain Decisions
|
||||
|
||||
Update the prototype README with these implemented commands and fixture layout.
|
||||
State that synthetic testing IDs demonstrate operator-selectable routing, not
|
||||
measured price, speed, or model quality.
|
||||
**Status:** Pending. **Depends on:** Stage 6.
|
||||
**Owns:** `internal/framework/validators`, validation prompt builders/batching,
|
||||
and validator tests.
|
||||
|
||||
## Stage 4: Verify Failure, Credential, and Inspection Boundaries
|
||||
1. Migrate spoken_form_plausibility, meaning_reversal_review, editorial_review,
|
||||
grammar_review, and spoken_word_review to prompt IDs/variables and the common
|
||||
structured boundary. Keep deterministic validators independent of PromptKit.
|
||||
2. Preserve batching by `validation.max_prompt_tokens`, proposal preview, stable
|
||||
proposal indexes, exact decision cardinality, and unknown/duplicate index checks.
|
||||
Keep the current domain token-estimation method; do not add a second template
|
||||
renderer or treat this budget as a provider output-token limit.
|
||||
3. Apply the failure mapping: exhausted repairs reject/warn the batch; schema-valid
|
||||
bad cardinality or negative assessments reject without semantic retry. Provider,
|
||||
malformed HTTP, deadline, and admission failures propagate as runtime errors.
|
||||
Update any old tests treating malformed provider envelopes as warning-only.
|
||||
4. Remove redundant validator client/model configuration types and keep execution
|
||||
class distinctions. Update validator reference/internal documentation.
|
||||
|
||||
**Status:** Complete (2026-09-24). **Depends on:** Stage 3. Focused offline tests
|
||||
cover ConfigKit overlay errors, backend validation, credential replacement and
|
||||
null handling, provenance, inspection, and redaction. Nested `go test -race ./...`
|
||||
and `go vet ./...` passed on linux/amd64 with Go 1.26.8.
|
||||
**Acceptance and validation:** preserve existing batching no-drop/no-duplicate,
|
||||
domain reason, glossary protection, and cardinality tests. Cover all five prompt
|
||||
mappings, positive repair before domain validation, exhausted repair, and runtime
|
||||
failure propagation through a representative validator. Run validator tests;
|
||||
full runner behavior is Stage 8.
|
||||
|
||||
Extend focused tests in the prototype; keep them offline and independent of host
|
||||
configuration and credentials. Exercise the consumer's assembled loader/runtime
|
||||
rather than duplicating ConfigKit's exhaustive parser or merge tests.
|
||||
## Stage 8: Connect the Runner and Remove the Old Scheduling Hierarchy
|
||||
|
||||
Required evidence:
|
||||
**Status:** Pending. **Depends on:** Stages 2 and 4–7.
|
||||
**Owns:** CLI process assembly, `internal/framework/runner`, old scheduler contracts,
|
||||
and runner/CLI integration tests.
|
||||
|
||||
- Missing and invalid overlay selections fail with ConfigKit classifications.
|
||||
Representative attempts to replace the schema version or introduce ConfigKit
|
||||
directives fail. Do not create application fallbacks for those failures.
|
||||
- An overlay can change an ordinary non-profile setting, such as a backend limit,
|
||||
and the effective setting still undergoes application validation. This proves
|
||||
overlays have not been accidentally restricted to profile selection.
|
||||
- A replaced credential reference resolves only its surviving replacement; an
|
||||
unresolved displaced reference does not make the load fail. A surviving missing
|
||||
or empty secret reference still fails. Explicit null retains unauthenticated
|
||||
backend semantics. Use synthetic environment names and values under test control.
|
||||
- Switching role profiles alone does not remove other credential references still
|
||||
present in configuration. Document this distinction and test the relevant case;
|
||||
do not bypass ConfigKit validation for unselected backends as an incidental fix.
|
||||
- `WriteEffective` and relevant explanation/metadata APIs identify overlay origin;
|
||||
a CLI replacement identifies CLI origin. Inspect selected PromptKit profiles from
|
||||
that same loaded result and verify model/backend agreement with runtime selection.
|
||||
- Synthetic credential values do not appear in effective configuration, explanations,
|
||||
profile inspection, observations, or surfaced errors exercised by these scenarios.
|
||||
Confirm the controlled transport receives the effective synthetic credential.
|
||||
1. Pass the shared engine's two role clients into the runner. Remove global and
|
||||
role schedulers, scheduler interfaces/fields, legacy concurrency configuration,
|
||||
and measured scheduler wrappers. PromptKit backend admission is the only LLM
|
||||
service-capacity mechanism. Do not remove unrelated deterministic work or alter
|
||||
module sequencing merely because the scheduler disappears.
|
||||
2. Preserve concurrent section work, deterministic collection/indexing, ordered
|
||||
module application, validator execution classes, and cancellation of remaining
|
||||
work on runtime failure. Capacity rejection fails the run as selected; do not
|
||||
silently retry or add another queue to conceal it.
|
||||
3. Wire final loaded domain settings, role clients, and diagnostics context into
|
||||
normal processing. No old-runtime branch or interim gate remains. Keep existing
|
||||
transcript encoding, output channels, partial domain results, and artifact
|
||||
retention behavior. Ensure terminal failures cannot emit a purportedly complete
|
||||
corrected transcript.
|
||||
4. Update architecture policy for shared library dependencies and engine-local
|
||||
backend admission; preserve the existing ownership, determinism, and redaction
|
||||
invariants. Update pipeline/runtime internal guidance.
|
||||
|
||||
Maintain the existing retry/repair/capacity tests. Add only the overlay-specific
|
||||
integration coverage needed to protect this boundary; do not repeat the complete
|
||||
scheduler matrix for each fixture.
|
||||
**Acceptance and validation:** controlled concurrent requests from both roles to
|
||||
one backend never exceed its configured cap, including retries and repairs;
|
||||
separate backend identities have independent capacity. Queued cancellation,
|
||||
deadline and zero-wait-capacity rejection do not leak capacity; later work can
|
||||
succeed. Existing deterministic runner/domain tests pass after deleting scheduler-
|
||||
implementation assertions where the affected packages compile. Exercise a real
|
||||
CLI-to-local-HTTP workflow with both roles when reporting dependencies permit;
|
||||
otherwise record that check for Stage 12. Do not restore old utilization fields
|
||||
or add placeholders merely to make this intermediate commit build.
|
||||
|
||||
Verification:
|
||||
## Stage 9: Collect Execution Observations Safely
|
||||
|
||||
```sh
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test -race ./...
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go vet ./...
|
||||
```
|
||||
**Status:** Pending. **Depends on:** Stage 8.
|
||||
**Owns:** Audita-owned execution record types and collector in the LLM boundary,
|
||||
runner attribution, and concurrency/observation tests.
|
||||
|
||||
Acceptance: all cases above have passing behavioral evidence, the existing suite
|
||||
remains green, and concurrent observation collection remains race-free. Record
|
||||
actual gaps instead of silently relaxing tests or claiming complete integration.
|
||||
1. Translate PromptKit events into small Audita-owned records at the observer
|
||||
boundary. Use the prototype's short mutex-protected collection pattern; callbacks
|
||||
do no filesystem I/O, provider calls, or waiting on their own run. Freeze/copy
|
||||
snapshots after active work has joined.
|
||||
2. Bind observations to logical call, role (`proposal` or `validation`), module
|
||||
instance, section, validator and batch where applicable, then retain library
|
||||
run/backend/profile and attempt/generation identity. Keep provider attempts distinct from logical calls.
|
||||
3. Record dispatch, queue/generation/retry/wall durations, outcome/failure category,
|
||||
HTTP status, retry exhaustion, attempts/retries/repairs, and usage value/known/
|
||||
complete per field. Use terminal events once for provider totals; never add
|
||||
attempt usage on top. `PrepareExecution` emits no events and `RunPrepared` timing
|
||||
excludes preparation. Audita's surrounding call record therefore owns the logical
|
||||
count and wall time, including preparation failures; do not invent library events.
|
||||
A failure before dispatch has zero attempts and known-zero complete usage.
|
||||
Preserve completed observations on nil-result failures and mark usage incomplete
|
||||
when dispatched attempts lack measurements. Across calls, sum known values,
|
||||
set known if any contributing dispatched usage is known, and set complete only
|
||||
when all contributing dispatched usage is complete, independently per field.
|
||||
An aggregate with no dispatched attempts is known-zero and complete.
|
||||
4. Keep non-LLM runner wall/validator timing. Remove instrumentation that counts
|
||||
scheduler permits as calls. Do not infer queue peaks, exact active concurrency,
|
||||
billing totals, or timing fields the released event API does not provide.
|
||||
|
||||
## Stage 5: Record Overlay Findings and Executable Instructions
|
||||
**Acceptance and validation:** adapter/collector tests prove known zero differs
|
||||
from unknown, partial usage survives failed repairs, undispatched admissions are
|
||||
not HTTP attempts, terminal totals are not doubled, and parallel callbacks retain
|
||||
attribution safely. Run targeted race tests for the affected LLM/runner packages.
|
||||
No raw provider messages, URLs, keys, prompts, or payloads enter observation records.
|
||||
|
||||
**Status:** Complete (2026-09-24). **Depends on:** Stage 4. Dated v0.4.0 evidence,
|
||||
test coverage, commands, limitations, README guidance, and roadmap links updated;
|
||||
affected links and `git diff --check` verified.
|
||||
## Stage 10: Update Reports and Diagnostic Artifacts
|
||||
|
||||
This stage changes documentation, not production code. Read the new tests/results,
|
||||
then update the existing owners rather than creating another parallel design:
|
||||
**Status:** Pending. **Depends on:** Stage 9.
|
||||
**Owns:** `internal/core/reporting`, `internal/core/diagnostics`,
|
||||
`internal/framework/processreport`, runtime diagnostic writers, and CLI artifact wiring.
|
||||
|
||||
- Append a dated v0.4.0/overlay result section to `library-boundary-findings.md`.
|
||||
Record commands, toolchain/platform, outcomes, test-to-requirement coverage,
|
||||
limitations, and any unresolved gaps. Keep original v0.3.1 evidence identifiable.
|
||||
- Update the prototype README to match the actual executable and fixtures; check
|
||||
every documented command. Explain the separate ConfigKit overlay and PromptKit
|
||||
profile directories and the credential behavior proved in Stage 4.
|
||||
- Record the overlay implementation as verified in `library-boundary.md` and link
|
||||
the new evidence from `library-adoption.md`. The next unit still has its Stage 6
|
||||
planning deliverable outstanding; keep production migration pending.
|
||||
1. Bump the process report version to `v2`; keep the existing transcript/ledger
|
||||
structures. Add `llm_summary` with `logical_calls`, `provider_attempts`,
|
||||
`transient_retries`, `structural_repairs`, and `usage`. A logical call counts
|
||||
once even when it fails before dispatch. Usage has `prompt`, `completion`,
|
||||
`total`, `cached`, and `cache_write`, each `{value, known, complete}`.
|
||||
Omit the whole summary only when execution observations are unavailable; a
|
||||
successfully initialized run with no LLM calls has known-zero complete totals.
|
||||
2. Replace obsolete utilization concurrency/pool fields with `backends`, keyed by
|
||||
backend ID, containing configured `concurrency`/`queue_capacity` and the same
|
||||
execution summary plus `capacity_wait_ms`, `generation_ms`, `retry_delay_ms`,
|
||||
and `logical_wall_ms`. Sum library terminal timings once per call for capacity,
|
||||
generation, and retry time; use Audita's whole-call elapsed time for logical wall
|
||||
time. Attribute pre-dispatch failures to the already selected role backend.
|
||||
Durations can overlap and must not be presented as a decomposition of elapsed
|
||||
process time. Preserve domain module/validator timing and run wall time.
|
||||
Existing role/module call counts now count logical calls; remove their obsolete
|
||||
scheduler-wait and transport-execution fields rather than assign misleading new
|
||||
meanings. Remove estimated active-concurrency fields throughout utilization.
|
||||
3. Write `llm-execution.json` as `{calls: [...], events: [...]}` in the run directory.
|
||||
Calls contain Audita logical identity, attribution, elapsed milliseconds, final
|
||||
classification, summary, and available prompt/profile/model metadata. Events
|
||||
contain the safe Stage 9 records, including attempt and terminal records, joined
|
||||
by call ID. A preparation failure has a call record and no fabricated events.
|
||||
Use millisecond integer durations in serialized records; absent attempt/section/
|
||||
batch identity is omitted, not encoded as a misleading zero. Add optional
|
||||
`llm_execution_path` and `selected_profiles_path` to diagnostic path metadata.
|
||||
Write `selected-profiles.yaml` from the safe view and use ConfigKit YAML for
|
||||
the effective snapshot (`effective-config.redacted.yaml`). Keep invocation
|
||||
overlay selection and recorded origins in safe invocation/config diagnostics.
|
||||
Adjust centralized filename constants and readers together; do not emit old and
|
||||
new snapshots in parallel.
|
||||
4. Preserve redacted request/response/error diagnostic content from prepared
|
||||
details and available final results, including prompt definition/rendered hashes
|
||||
and selected profile/model. Do not promise legacy hash equality or complete
|
||||
provider payloads for every retry. Use the separate observation artifact for retry/repair
|
||||
attempts. Redact all resolved backend secrets, including unselected ones, at
|
||||
every serialization/error boundary; observation records remain payload-free.
|
||||
5. Update output-file, operations, and diagnostics/reporting contracts. Preserve
|
||||
failure report/ledger behavior and the distinction between primary execution
|
||||
failures and secondary diagnostic write errors. Do not change retention policy.
|
||||
|
||||
Acceptance: the README commands are reproducible, the new overlay evidence is
|
||||
clearly separated from historical results, and each Stage 2–4 requirement maps
|
||||
to a passing test or observed demonstration result. Verify affected links and
|
||||
`git diff --check`. Do not repeat unchanged code checks solely for this commit.
|
||||
**Acceptance and validation:** report/diagnostic tests check v2 mapping, summary
|
||||
semantics, per-backend grouping, safe serialization, nil-result partial failures,
|
||||
and existing deterministic ledger order. A CLI failure fixture echoes a synthetic
|
||||
secret in a provider error; stdout/stderr/report and retained artifacts must not
|
||||
leak it. Inspect known-zero and unavailable summaries structurally, not by complete
|
||||
JSON golden dumps. Reuse Stage 9 evidence rather than retest every event sequence.
|
||||
|
||||
## Stage 6: Prepare the Production Migration Plan and Close the Unit
|
||||
## Stage 11: Finish Examples, Migration Guidance, and Removal
|
||||
|
||||
**Status:** Complete (2026-09-24). **Depends on:** Stage 5. The ordered ConfigKit
|
||||
production handoff records bounded packages, runtime integration, tests,
|
||||
dependencies, and acceptance criteria. Prototype evidence and scope audit passed;
|
||||
links and `git diff --check` verified.
|
||||
**Status:** Pending. **Depends on:** Stages 1–10.
|
||||
**Owns:** maintained examples, affected canonical documentation, obsolete code/tests,
|
||||
prototype removal, and final dependency cleanup.
|
||||
|
||||
Expand `configkit.md` into a concrete, ordered production migration plan using
|
||||
the existing inventory and verified overlay evidence. Follow the development
|
||||
reading guide for configuration/CLI and LLM runtime work; use graph-first discovery
|
||||
to verify affected production components and their tests. This is a planning stage:
|
||||
do not implement the production migration or overwrite this execution plan with
|
||||
new production stages. Keep future work packages explicitly pending.
|
||||
1. Replace maintained config examples with schema 2, complete PromptKit profile
|
||||
files, and `overlays/testing.yml` selecting both roles. Use one shared backend
|
||||
where roles share a service; provide secret-reference and unauthenticated local
|
||||
examples without real secrets. Explain that operators choose real models and
|
||||
testing cost/quality is not guaranteed. Validate examples offline with controlled
|
||||
synthetic environment values; no service is required for profile inspection.
|
||||
2. Write `docs/migrations/v2.md` as the one migration guide. Cover every old file/
|
||||
flag/environment item in the inventory, durations, explicit empties/null,
|
||||
credentials, profiles/overlays, inspection views, finite capacity, retry/repair/
|
||||
deadline semantics, and changed report/diagnostic shapes. Do not add conversion
|
||||
tooling or compatibility aliases. Link current contracts for complete references.
|
||||
3. Finish README prerequisites/examples and canonical CLI/config, integration,
|
||||
internal component, and architecture references. Audit build/CI toolchain images
|
||||
for Go 1.25.5 or newer; keep only required Linux amd64/arm64 targets and best-effort
|
||||
macOS arm64. Do not rewrite immutable historical release notes.
|
||||
4. Remove `examples/library-adoption` once its unique useful scenarios live in
|
||||
production tests. Remove the nested dependency on old Audita and duplicate
|
||||
assets/fixtures. Update historical roadmap links to recorded evidence or Git
|
||||
history rather than leaving broken links to deleted files.
|
||||
5. Remove superseded loaders, client, renderers, schedulers, aliases, and tests
|
||||
asserting retired behavior. Tidy/verify the root dependency graph and confirm
|
||||
no local replacements. Keep yaml.v3 if glossary parsing still uses it. Mark the
|
||||
retry-count follow-up in `runtime-behavior.md` resolved without including its
|
||||
unrelated retention/timestamp proposals.
|
||||
|
||||
The production handoff must specify bounded work packages, dependencies, affected
|
||||
components, tests, and acceptance criteria for:
|
||||
**Acceptance and validation:** maintained examples validate and all guide links
|
||||
resolve. Search for old CLI/env inputs, loader/client/scheduler names and prototype
|
||||
references; remaining mentions must be intentional migration/history material.
|
||||
Review deleted tests for lost domain/concurrency/redaction protection. Run affected
|
||||
example/config/subprocess checks and `git diff --check`; final full-suite execution
|
||||
belongs to Stage 12.
|
||||
|
||||
- The full version-2 configuration definition: existing domain fields, defaults,
|
||||
presence/null/zero behavior, role inheritance, overlays, secrets, and validation.
|
||||
- File discovery and invocation parsing; `--overlay` behavior shared by process,
|
||||
config validation, and effective inspection; explicit-setting override mapping;
|
||||
removal of legacy environment overlays, aliases, and direct-key flags.
|
||||
- Profile loading/inspection and configuration snapshots, including relative-path
|
||||
bases, secret redaction, provenance, and clear separation of ConfigKit settings
|
||||
from PromptKit execution controls.
|
||||
- Direct integration of the final configuration with PromptKit execution. Specify
|
||||
the final behavior and tests for each setting; do not preserve the old runtime
|
||||
through temporary adapters or add interim processing gates.
|
||||
- Removal of superseded configuration machinery, retained domain checks, maintained
|
||||
examples, canonical documentation updates, and the combined migration guide.
|
||||
- Removal of superseded LLM machinery and release follow-up. Preserve
|
||||
the agreed ordering and defer release compliance/publication to the final unit.
|
||||
## Stage 12: Verify the Combined Application and Close the Refactor
|
||||
|
||||
Resolve final integration decisions against the evidence. Record genuine open
|
||||
questions without creating scaffolding to support intermediate commits.
|
||||
**Status:** Pending. **Depends on:** Stages 1–11.
|
||||
**Owns:** final integration tests, final defect fixes within this scope, and status/evidence.
|
||||
|
||||
Final audit: map every acceptance criterion in the prototype extension to actual
|
||||
files and command results. Verify new relative links/anchors, run `git diff --check`,
|
||||
and review that no root runtime/dependency or upstream files changed. Reuse passing
|
||||
checks for unchanged code; rerun the affected tests if this stage changes code or
|
||||
fixtures to correct a finding. A root build/test run is not a substitute for the
|
||||
nested-module checks, and this documentation stage adds no release gate.
|
||||
1. Complete representative production subprocess workflows against controlled HTTP:
|
||||
base selection, testing overlay, explicit profile override, and a retained
|
||||
domain override. Observe actual outbound models for both roles, an accepted
|
||||
correction after repair, a warning-only domain/repair rejection, and a runtime
|
||||
failure with safe partial diagnostics. Cover an empty module pipeline and a
|
||||
repeated-module sequence without duplicating all adapter scenarios.
|
||||
2. Run root `go test ./...`, `go build -o ./bin/audita ./cmd/audita`, and focused race
|
||||
checks for the final concurrent adapter/runner/collector changes. Verify both
|
||||
Linux targets compile using `bash scripts/build-release.sh v2.0.0 <temp-dir>`;
|
||||
this names local build outputs and creates no tag or publication. No native Mac, paid-provider, or CI-completion gate.
|
||||
If the supported minimum toolchain was not exercised earlier, verify the root
|
||||
suite with installed Go 1.25.5; do not silently claim this from a newer compiler.
|
||||
3. Fix actual failures and rerun affected checks; repeat the full suite when those
|
||||
fixes materially change assembled behavior. Do not rerun unchanged tests solely
|
||||
to mark multiple planning steps complete. Verify maintained documentation links,
|
||||
examples, and whitespace once for the final content.
|
||||
4. Record concise final evidence and known limits here, mark the production feature
|
||||
roadmaps implemented, and commit the completed refactor. Both libraries must
|
||||
own their selected responsibilities, processing must work, and no temporary
|
||||
or legacy compatibility machinery may remain.
|
||||
5. Hand off release preparation separately: proposed Audita v2.0.0, one migration
|
||||
guide, actual pinned production dependency/license inventory, and the remaining
|
||||
LGPL notice/source/rebuild-distribution review. Do not claim license compliance
|
||||
from prototype tests or publish a release. The later release unit uses the
|
||||
existing minimal automated procedure and ends on a successful source/tag push,
|
||||
without waiting for CI.
|
||||
|
||||
Mark this prototype-extension/planning unit complete in `library-boundary.md` only
|
||||
after the audit passes, and link the production handoff from `library-adoption.md`.
|
||||
Production implementation remains pending.
|
||||
**Acceptance:** the final production application and required Linux builds pass;
|
||||
configuration selection, both LLM roles, domain outcomes, cancellation/capacity,
|
||||
secret handling, reports and retained artifacts match the selected contract.
|
||||
Recorded checks describe what actually ran. The workspace is committed and clean.
|
||||
Release-specific compliance and publication are explicitly still pending.
|
||||
|
||||
Acceptance: a reviewer can reproduce base/testing/override selection offline,
|
||||
trace each new claim to evidence, and hand the production ConfigKit work to a coding
|
||||
agent in bounded units. Commit the completed documentation, report remaining
|
||||
limitations and working-tree state, and stop at the end of this plan.
|
||||
## Open Questions
|
||||
|
||||
None block this implementation plan. Configuration, inspection, runtime boundaries,
|
||||
report representation, and stage ownership are selected above. Release distribution
|
||||
compliance remains outside this implementation sprint and must be resolved against
|
||||
the final production dependency graph before publishing binaries; it is not a
|
||||
reason to build compatibility scaffolding or delay the refactor.
|
||||
|
||||
@@ -2,179 +2,128 @@
|
||||
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Shared boundary and offline prototype completed; production migrations
|
||||
remain planned. The [v0.4.0 overlay evidence](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
|
||||
extends the completed prototype, and the [ordered ConfigKit handoff](configkit.md#ordered-production-migration-handoff)
|
||||
specifies the next implementation work. This document authorizes no release
|
||||
publication and sets no schedule.
|
||||
**Status:** Final target selected; production implementation pending. The shared
|
||||
boundary and offline prototype, including ConfigKit v0.4.0 overlays, are verified.
|
||||
[Implementation stages](implementation.md) are owned by the implementation plan.
|
||||
This roadmap defines the end state and user intent; it authorizes no publication.
|
||||
|
||||
## Goal and Ownership
|
||||
|
||||
Adopt ConfigKit and PromptKit to align Audita with the application suite's shared
|
||||
configuration and LLM execution conventions. Remove overlapping application
|
||||
infrastructure while keeping transcript-correction responsibilities in Audita.
|
||||
configuration and LLM execution conventions. Consistency is the primary benefit;
|
||||
removing overlapping application infrastructure is a secondary benefit. Preserve
|
||||
Audita's transcript-correction capabilities and domain protections.
|
||||
|
||||
| Owner | Intended responsibilities |
|
||||
| Owner | Responsibilities |
|
||||
| --- | --- |
|
||||
| ConfigKit | Application configuration loading, presence/default semantics, explicit overrides, recorded resolution, secret references, and safe configuration inspection |
|
||||
| PromptKit | Prompt/profile formats and rendering, provider calls, structural output validation, and shared execution capabilities supplied by the selected release |
|
||||
| Audita | Configuration field meanings, CLI parsing and file selection, transcript normalization/chunking, module ordering, domain validators, correction application, reports, and diagnostic artifact lifecycle |
|
||||
| ConfigKit | Strict application configuration loading, presence/default semantics, explicit overrides, recorded resolution, secret references, and safe inspection |
|
||||
| PromptKit | Prompt/profile formats and rendering, provider calls, structural output validation and repair, transient retries, engine-local backend admission, and execution observations |
|
||||
| Audita | Setting meanings, file/CLI selection, role/profile association, normalization/chunking, ordered modules, domain validation, correction application, reports, diagnostic redaction, and artifact lifecycle |
|
||||
|
||||
This roadmap owns sequencing across the two adoptions. The
|
||||
[ConfigKit roadmap](configkit.md) owns the configuration migration's proposed scope.
|
||||
PromptKit v0.11.0 is the reviewed baseline for the adoption prototype. It supplies
|
||||
transient HTTP retries, execution observations, typed failure categories, an offline
|
||||
proposal/assessment example, and LGPL-3.0-or-later licensing. The earlier upstream
|
||||
consumer requests are no longer outstanding prerequisites on those points.
|
||||
Audita has not yet adopted either library; recheck the selected releases during
|
||||
detailed planning.
|
||||
The [configuration roadmap](configkit.md) owns configuration and inspection policy.
|
||||
The [boundary findings](library-boundary-findings.md) own the selected defaults,
|
||||
full migration inventory, failure mapping, and prototype evidence. The selected
|
||||
production versions are ConfigKit v0.4.0 and PromptKit v0.11.0, with Go 1.25.5 as
|
||||
the minimum. Use published modules, never sibling checkouts or local replacements.
|
||||
The existing YAML dependency remains while glossary parsing needs it.
|
||||
|
||||
## Migration Posture
|
||||
|
||||
Treat adoption as an intentional breaking transition. Preserve intended domain
|
||||
capabilities, not legacy configuration files, flags, environment variables, or
|
||||
configuration-inspection formats. Supply one migration guide rather than parallel
|
||||
legacy loaders, compatibility aliases, or permanent translation layers.
|
||||
This is an intentional breaking transition targeting Audita v2.0.0. No backward
|
||||
compatibility is required for existing installations, config files, CLI flags,
|
||||
environment overlays, or inspection output. Provide one migration guide, without
|
||||
legacy loaders, aliases, dual modes, or automatic conversion tooling.
|
||||
|
||||
Plan to ship both adoptions together as Audita v2.0.0, without an intermediate
|
||||
public release. Implementation should still consist of coherent, reviewable units.
|
||||
Revisit the final release scope and sequencing alongside other large planned
|
||||
changes, especially changes to module or pipeline configuration.
|
||||
Both libraries belong to one refactor. Intermediate commits need not build or run;
|
||||
there is no supported intermediate product state. No temporary adapters, compatibility
|
||||
shims, or processing gates are needed. The complete application must work and pass
|
||||
its checks before the refactor is finished. Release compliance and publication
|
||||
come last, with no intermediate public release.
|
||||
|
||||
## Proposed Sequence
|
||||
## Configuration and Execution Target
|
||||
|
||||
The [shared-boundary and prototype roadmap](library-boundary.md) records the completed
|
||||
unit covering stages 1 and 2 below. The [findings](library-boundary-findings.md)
|
||||
select its target contract and identify the next ConfigKit implementation work.
|
||||
ConfigKit selected overlays enable a testing configuration that chooses cheaper or
|
||||
faster models. The example selects both proposal and validation profiles. ConfigKit
|
||||
application overlays and PromptKit execution profiles remain separate concepts.
|
||||
Complete operator-maintained profiles own model/generation controls; embedded
|
||||
application prompts and schemas are not operator-selected files. No catalog wiring,
|
||||
profile-inheritance feature, ConfigKit imports, or secret-directory bootstrap is
|
||||
part of this migration.
|
||||
|
||||
### 1. Design the Shared Configuration Boundary
|
||||
Capacity controls serve two needs: respecting a local service's hard simultaneous-
|
||||
query limit and bounding concurrent paid API activity. One finite limit per backend
|
||||
is shared across both roles and every initial, retry, and repair attempt. Limits
|
||||
are engine-local, not coordinated across processes. No additional global or
|
||||
per-role limiter or fine-grained scheduling policy is required. Profiles must use
|
||||
registered backends and ConfigKit-resolved credentials; endpoint-only bypass and
|
||||
competing profile credential lookup are excluded.
|
||||
|
||||
Define the target settings model for both libraries before implementing either
|
||||
migration. Decide how Audita selects proposal and validation execution profiles,
|
||||
supplies credentials, selects transient-retry and structural-repair budgets, and
|
||||
assigns per-backend concurrency limits. Give each setting one authoritative home;
|
||||
avoid duplicate model, endpoint, credential, or timeout definitions with competing
|
||||
precedence.
|
||||
Structural repair is enabled by default for both roles because additional attempts
|
||||
to obtain valid output are desired. Transient retries and structural repairs have
|
||||
separate budgets; explicit zero disables the respective behavior. Each generation,
|
||||
including a repair, receives its own transient-retry budget. The Audita logical-call
|
||||
deadline covers preparation, waiting, backoff, all generations, and repairs; profile
|
||||
HTTP timeouts apply per attempt. Exact defaults and bounds are in the
|
||||
[selected contract](library-boundary-findings.md#configuration-and-profile-contract).
|
||||
No Audita outer retry loop restores the old broader retry policy.
|
||||
|
||||
Keep ConfigKit application configuration distinct from PromptKit execution profiles.
|
||||
ConfigKit v0.4.0 calls its optional configuration layers overlays. Enable selected
|
||||
overlays for testing as specified in the [ConfigKit roadmap](configkit.md#selected-overlays-for-testing);
|
||||
these select application settings, including PromptKit execution profiles.
|
||||
Select an explicit profile-ownership model; complete Audita-owned execution profiles are the initial recommendation, with
|
||||
catalog-driven selection a deliberate alternative.
|
||||
Audita retains all four correction modules, five LLM-validator prompts,
|
||||
deterministic validators, confidence/applicability checks, glossary protection,
|
||||
validation batching/cardinality, deterministic proposal and ledger ordering,
|
||||
transcript encoders, and artifact retention. PromptKit-valid JSON still passes
|
||||
Audita's domain checks. Repair exhaustion, provider failure, and domain rejection
|
||||
follow the [selected failure table](library-boundary-findings.md#execution-and-failure-mapping).
|
||||
Retention and transcript timestamp changes remain separate work; this migration
|
||||
resolves the [retry-count follow-up](runtime-behavior.md).
|
||||
|
||||
Capacity needs are simple: protect local backends from exceeding their available
|
||||
simultaneous-query capacity and give paid API backends a reasonable finite cap.
|
||||
Proposal and validation calls using the same backend share its limit. Separate
|
||||
proposal/validation budgets and additional fine-grained Audita scheduling controls
|
||||
are not required. Account for the engine-local scope of PromptKit's existing limits.
|
||||
## Reporting and Diagnostic Target
|
||||
|
||||
### 2. Prove the PromptKit Boundary
|
||||
Keep stable Audita-owned serialized records; do not expose raw library structs.
|
||||
Use process report schema `v2` with the existing domain outcomes and a new
|
||||
`llm_summary` separating logical calls, dispatched provider attempts, transient
|
||||
retries, structural repairs, and usage. Transcript schema versions remain unchanged.
|
||||
The implementation plan specifies the concrete new fields and artifact wiring.
|
||||
|
||||
Before finalizing the new application configuration schema, exercise one proposal
|
||||
call and one LLM-validator call through PromptKit using controlled offline responses.
|
||||
Verify profile selection, credentials, embedded prompts and schemas, shared backend
|
||||
capacity, failed output validation, cancellation, and diagnostic mapping.
|
||||
Usage distinguishes known zero, unknown, and incomplete totals. Preserve observations
|
||||
from failed calls even when no final result exists. Count terminal totals once;
|
||||
attempt observations provide diagnosis without being added to those totals again.
|
||||
This is operational evidence, not a complete billing ledger.
|
||||
|
||||
Enable structural repair from the outset for proposal and LLM-validator outputs.
|
||||
Additional attempts to obtain valid output are desired behavior, not a deferred
|
||||
optimization. Select explicit positive repair budgets during detailed planning and
|
||||
exercise invalid output that becomes valid after repair, as well as exhausted
|
||||
repair budgets.
|
||||
Replace global/role scheduler utilization with per-backend configured capacity and
|
||||
observed queue/generation/retry/logical-call timing. Do not fabricate measurements
|
||||
that the event API does not provide. Keep domain timing and deterministic correction
|
||||
ledger ordering. Record prompt definition/rendered hashes and selected profile/model
|
||||
without promising the old hash values. Callbacks are short, concurrency-safe, and
|
||||
free of blocking I/O or waits on their own execution.
|
||||
|
||||
Keep transient HTTP retries and structural repair distinct. With the reviewed
|
||||
library, each generation, including a repair, has its own transient-retry budget;
|
||||
plan and document the combined request allowance and overall deadline. Adopt
|
||||
PromptKit's explicit zero-disables-retries semantics instead of Audita's legacy
|
||||
zero-as-default behavior. The library's narrower transient retry policy does not
|
||||
retry transport failures or malformed provider responses; do not restore the old
|
||||
policy through an overlapping application retry loop.
|
||||
Effective configuration and selected-profile snapshots remain distinct safe views.
|
||||
Secret values must not escape through inspection, logs, reports, runtime errors,
|
||||
or retained diagnostic files. Observation records exclude payloads, URLs, raw
|
||||
provider messages, prompts, and credentials. Existing artifact lifecycle and
|
||||
partial-failure handling remain Audita responsibilities.
|
||||
|
||||
Verify that initial, retry, and repair calls share backend limits and that their
|
||||
observations reach Audita without double-counting usage. Audita's domain acceptance
|
||||
checks remain necessary after structural repair succeeds; valid JSON does not imply
|
||||
an acceptable correction or validator decision.
|
||||
## Completion Criteria and Release Boundary
|
||||
|
||||
Use the prototype to resolve genuine boundary gaps and inform configuration design.
|
||||
It is evidence for the migration, not a permanent alternate runtime. Avoid finalizing
|
||||
ConfigKit-facing settings around the old LLM runtime only to redesign them later.
|
||||
- Both libraries own their selected responsibilities in the production application.
|
||||
Superseded loaders, transport, renderers, schedulers, aliases, and the isolated
|
||||
prototype are gone once its useful evidence is represented in production tests.
|
||||
- Offline tests prove configuration-to-execution routing for both roles, selected
|
||||
overlays and explicit overrides, shared backend capacity, retry/repair outcomes,
|
||||
cancellation/deadlines, domain decisions, redaction, and truthful partial reports.
|
||||
Tests exercise real internal collaborators and controlled HTTP boundaries.
|
||||
- Maintained examples and canonical docs describe the implemented final model.
|
||||
One migration guide covers files, flags, environment inputs, profiles, credentials,
|
||||
overlays, inspection, and changed runtime/report/diagnostic semantics.
|
||||
- The assembled application passes its required checks; Linux amd64 and arm64 build.
|
||||
macOS arm64 remains best effort. No live-provider or CI-success gate is added.
|
||||
|
||||
### 3. Complete ConfigKit Adoption
|
||||
PromptKit v0.11.0's reviewed retry, observation, and capacity APIs remove the earlier
|
||||
upstream feature prerequisites; no upstream edit is required for adoption. The
|
||||
prototype proved representative behavior, not release compliance or model quality.
|
||||
|
||||
Implement the agreed application configuration contract following the
|
||||
[ConfigKit roadmap](configkit.md). Retire legacy environment overlays and obsolete
|
||||
CLI/configuration conventions, update maintained examples and canonical references,
|
||||
and begin the combined migration guide. Include the selected testing-overlay path
|
||||
and its offline integration evidence from the ConfigKit roadmap.
|
||||
|
||||
ConfigKit and PromptKit adoption form one refactor. These sections describe
|
||||
implementation order, not separately runnable milestones. Intermediate commits
|
||||
need not build or run. Build directly toward the final integration without
|
||||
temporary adapters, compatibility shims, or an interim processing gate; validate
|
||||
the complete application before declaring the refactor finished.
|
||||
|
||||
### 4. Complete PromptKit Adoption
|
||||
|
||||
Move prompt rendering, execution profiles, provider calls, and structural output
|
||||
validation onto the selected PromptKit release. Remove overlapping transport,
|
||||
prompt-loading, and scheduling machinery once the replacement is verified.
|
||||
|
||||
Retain Audita's transcript pipeline, semantic and deterministic validators,
|
||||
correction application, reports, ledger, and artifact retention. Map PromptKit
|
||||
results, errors, metadata, and execution observations into those responsibilities.
|
||||
Connect the new configuration directly to this execution path and remove the
|
||||
superseded configuration and runtime machinery.
|
||||
|
||||
### 5. Validate and Release the Combined Migration
|
||||
|
||||
Validate the assembled configuration-to-execution workflows, including local and
|
||||
API backend limits, successful and exhausted repairs, transient retries,
|
||||
failure/cancellation behavior, credential handling, diagnostics, and domain outcomes.
|
||||
Use deterministic tests and controlled endpoints rather than paid calls as acceptance
|
||||
gates. Update documentation and examples together with
|
||||
implemented behavior and remove obsolete contracts and code.
|
||||
|
||||
Complete one migration guide covering files, CLI, environment settings, credentials,
|
||||
execution profiles, inspection, and changed diagnostics. Release the combined
|
||||
breaking transition under the established [release procedure](../release.md), with
|
||||
no CI waiting added to release completion.
|
||||
|
||||
## Reviewed Upstream Readiness
|
||||
|
||||
The v0.11.0 review covered the relevant implementation, the full test suite,
|
||||
targeted race tests, and the offline proposal/assessment example; those checks
|
||||
passed. Existing per-backend controls cover the requested capacity model, including
|
||||
retry and repair interactions. No further upstream feature request is currently a
|
||||
prerequisite to the prototype. Keep the planned adoption sequence unchanged.
|
||||
|
||||
Remaining integration work includes selecting a total-operation deadline alongside
|
||||
per-attempt timeouts, using shared backend identities with finite limits, and mapping
|
||||
PromptKit observations to Audita-owned diagnostic records. Observation callbacks are
|
||||
synchronous and may overlap across runs; collection must be short and safe for
|
||||
concurrent use. Distinguish unknown or incomplete usage from known zero and do not treat
|
||||
observations as a complete billing ledger or a stable serialized report format.
|
||||
|
||||
The licensing change has shipped as LGPL-3.0-or-later. Account for applicable
|
||||
notices and source/rebuild distribution requirements in Audita's release design;
|
||||
relicensing no longer needs to be requested upstream. Recheck released APIs,
|
||||
toolchain requirements, and dependency versions at implementation time. Pin
|
||||
published library releases; production builds must not depend on sibling checkouts
|
||||
or local module replacements.
|
||||
|
||||
If the prototype reveals a genuine remaining gap, evaluate it explicitly rather
|
||||
than quietly recreate shared library functionality as permanent Audita infrastructure.
|
||||
|
||||
## Coordination and Open Decisions
|
||||
|
||||
- Use the selected configuration/profile boundary, defaults, credential flow, and
|
||||
retry/repair/deadline contract in the [findings](library-boundary-findings.md)
|
||||
when detailing the ConfigKit production migration.
|
||||
- Implement the outcome and observation mapping recorded in the findings, retaining
|
||||
Audita domain validation and diagnostic redaction.
|
||||
- Resolve the [runtime retry-count follow-up](runtime-behavior.md) through this migration.
|
||||
Retention and transcript timestamp questions are not automatically included.
|
||||
- Revisit ordering when other major changes are defined. Prefer designing shared
|
||||
module/pipeline settings once, while keeping unrelated work independent.
|
||||
- Flesh out task-level scope, tests, and acceptance criteria before implementation.
|
||||
Completion requires the intended ownership split, removed transitional machinery,
|
||||
verified assembled behavior, and one coherent migration path for users.
|
||||
The selected PromptKit release uses LGPL-3.0-or-later. A later release unit must
|
||||
review the actual production dependency graph and distribution requirements,
|
||||
including notices and source/rebuild materials, before binary publication. This
|
||||
implementation plan does not claim that review is complete. Release publication
|
||||
uses the existing [procedure](../release.md) and completes on successful source/tag
|
||||
push, without waiting for CI or artifact delivery.
|
||||
|
||||
@@ -311,51 +311,23 @@ claims of completion for the prototype.
|
||||
|
||||
The new profile/credential/capacity and deadline choices are explicit above; no
|
||||
unresolved upstream requirement prevents starting the ConfigKit implementation
|
||||
plan. Production report field names and final runtime wiring follow the outcome
|
||||
and observation contract here. The coordinated refactor targets both libraries
|
||||
directly; intermediate commits do not require a working legacy runtime.
|
||||
plan. Production report field names and final runtime wiring are selected in the
|
||||
[implementation plan](implementation.md), following the outcome and observation
|
||||
contract here. The coordinated refactor targets both libraries directly; intermediate commits do not require a working legacy runtime.
|
||||
|
||||
### Component-Level Next Unit
|
||||
### Production Implementation Reference
|
||||
|
||||
Implement ConfigKit adoption first, against this target model:
|
||||
The [implementation plan](implementation.md) owns ordered production work,
|
||||
component changes, tests, and acceptance criteria. The [combined roadmap](library-adoption.md)
|
||||
and [configuration roadmap](configkit.md) own the final target. This evidence does
|
||||
not require an interim runtime bridge or separately runnable configuration migration.
|
||||
|
||||
| Component | Migration action |
|
||||
| --- | --- |
|
||||
| `internal/core/config` | Replace file/env merge and custom duration parsing with ConfigKit definitions, presence-aware resolution, domain validation, and explicit override paths. Retain domain defaults and validation rules identified in the inventory. |
|
||||
| `internal/cli` | Separate invocation paths from persistent settings; select one file; parse only supplied setting flags into overrides; remove legacy LLM/role-budget/direct-key flags and all environment overlays. Assemble/inspect selected profiles for validation. |
|
||||
| CLI config inspection and diagnostics snapshot | Use ConfigKit redacted output plus Audita-owned selected-profile summary; update command tests and diagnostic snapshot consumers. |
|
||||
| Configuration tests/examples | Replace v1 discovery/overlay/alias expectations with v2 presence, inheritance, secret, profile, and override contracts; preserve domain validation tests. Add actual production discovery and CLI tests not exercised by this prototype. |
|
||||
|
||||
The PromptKit portion of the refactor replaces `internal/framework/llm` provider execution
|
||||
and scheduler composition, `internal/prompts` parsing/rendering, and duplicated
|
||||
structural-output validation. Keep a small Audita adapter behind the existing
|
||||
structured-call boundary where that contract remains sufficient; revise its return
|
||||
metadata only as needed for repair and observation outcomes. Avoid library types
|
||||
in module proposals, domain validators, transcript schemas, and correction ledgers.
|
||||
|
||||
Retain `internal/core/schema`, normalization/chunking/output encoding,
|
||||
`internal/framework/proposals`, deterministic validators, module identity/order,
|
||||
LLM-validator domain batching and cardinality, runner ordering/apply logic, report
|
||||
assembly, and artifact lifecycle. Update their integration wiring rather than
|
||||
rewriting those algorithms. Move all nine prompt definitions and schema references,
|
||||
checking each module/validator's variables and helpers; this prototype proves only
|
||||
one pair. Migrate actual diagnostic redaction and error classification, then remove
|
||||
the old transport/schedulers/prompt loader. No temporary runtime bridge is needed.
|
||||
|
||||
Documentation work during the respective implementation stages: update
|
||||
`docs/config.md`, `docs/cli.md`, examples, README prerequisites, internal config/prompt/
|
||||
LLM/runner guidance, and output/LLM/subprocess integration contracts. Create one v2
|
||||
migration guide covering file shape, duration/empty-list changes, removed environment
|
||||
and CLI inputs, backend/profile/secret setup, effective inspection, retry/repair
|
||||
behavior, and changed usage/concurrency diagnostics. Revise architecture policy's
|
||||
existing scheduler language when production uses PromptKit; the current policy
|
||||
continues to describe the current runtime. Retention and timestamp fixes remain
|
||||
separate roadmap work.
|
||||
|
||||
The combined release remains last, with no intermediate publication. Raise the
|
||||
production Go minimum when dependencies land; review CI/build images and supported
|
||||
Linux amd64/arm64 builds then. Prototype execution on Linux amd64 is not proof of
|
||||
arm64 release readiness or live model output quality.
|
||||
Retained domain scope includes transcript/glossary schemas, normalization/chunking,
|
||||
output encoding, proposals and apply-time checks, deterministic validators, ordered
|
||||
modules, LLM-validator batching/cardinality, ledger ordering, and artifact lifecycle.
|
||||
The production integration replaces loading, provider execution, prompt rendering,
|
||||
and scheduler composition while preserving those responsibilities. Retention and
|
||||
timestamp changes remain separate work.
|
||||
|
||||
### Distribution Follow-Up
|
||||
|
||||
@@ -416,5 +388,5 @@ The tests use a controlled built-in HTTP transport and synthetic credentials;
|
||||
there were no live provider calls. The prototype still covers one representative
|
||||
proposal/assessment pair, not all production modules, prompts, CLI paths, or
|
||||
diagnostic artifacts. Root `go test ./...` does not run this nested module. No
|
||||
upstream API gap appeared in this extension. The [ordered production handoff](configkit.md#ordered-production-migration-handoff)
|
||||
upstream API gap appeared in this extension. The [production implementation plan](implementation.md)
|
||||
keeps implementation and release pending.
|
||||
|
||||
@@ -11,9 +11,9 @@ is isolated from production. No publication or upstream changes were made.
|
||||
|
||||
## Next Unit: Testing Overlays and Production Migration Plan
|
||||
|
||||
**Status:** Complete. The [ordered production handoff](configkit.md#ordered-production-migration-handoff)
|
||||
is ready for subsequent work; [implementation.md](implementation.md) records this
|
||||
unit's checks. The original work packages below remain
|
||||
**Status:** Complete. The [production implementation plan](implementation.md)
|
||||
owns subsequent work; the [overlay findings](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
|
||||
record this unit's checks. The original work packages below remain
|
||||
historical scope for the completed first unit.
|
||||
|
||||
Extend the existing prototype to prove the [selected-overlay contract](configkit.md#selected-overlays-for-testing)
|
||||
@@ -68,7 +68,7 @@ their existing names and meaning.
|
||||
|
||||
Define and prove the boundary between Audita, ConfigKit, and PromptKit before
|
||||
replacing Audita's production configuration or LLM runtime. This unit covers the
|
||||
shared design and representative prototype in stages 1 and 2 of the
|
||||
shared design and representative prototype supporting the
|
||||
[coordinated adoption roadmap](library-adoption.md). The
|
||||
[ConfigKit roadmap](configkit.md) continues to own the broader configuration scope.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user