From 1ad211928357485eafe11c858eb19481f150069a Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 24 Sep 2026 01:23:32 +0000 Subject: [PATCH] Finalize the ConfigKit and PromptKit implementation plan --- docs/roadmap/configkit.md | 275 ++------- docs/roadmap/implementation.md | 709 ++++++++++++++-------- docs/roadmap/library-adoption.md | 251 +++----- docs/roadmap/library-boundary-findings.md | 58 +- docs/roadmap/library-boundary.md | 8 +- 5 files changed, 642 insertions(+), 659 deletions(-) diff --git a/docs/roadmap/configkit.md b/docs/roadmap/configkit.md index 3fb2456..49b19d4 100644 --- a/docs/roadmap/configkit.md +++ b/docs/roadmap/configkit.md @@ -2,9 +2,9 @@ **Last updated:** 2026-09-24 -**Status:** Prototype boundary and selected overlays verified. The ordered -production handoff below is planned, not implemented; no production migration or -release has begun. +**Status:** Final target selected; production implementation pending. The boundary +and selected overlays are verified. The [implementation plan](implementation.md) +owns execution stages; no production migration or release has begun. ## Purpose and Direction @@ -27,10 +27,10 @@ application must depend on a versioned module, not a sibling checkout or local module replacement. The shared boundary/prototype unit is complete. Its [selected contract, inventory, -and implementation handoff](library-boundary-findings.md) refine the provisional -directions here; production configuration migration has not begun. +and evidence](library-boundary-findings.md) define the detailed target alongside +this roadmap. Production configuration migration has not begun. -## Proposed End State +## Target End State - ConfigKit owns strict YAML loading, input presence, default application, explicit override mechanics, recorded resolution, secret references, and safe inspection. @@ -40,17 +40,17 @@ directions here; production configuration migration has not begun. Audita's current per-layer validation and environment-overlay behavior. - General configuration environment overrides are retired. Credentials use typed environment-name secret references; supplied references must resolve according - to ConfigKit's contract. Review config-file selection separately as an - application-owned startup decision. + to ConfigKit's contract. File selection follows the application-owned + discovery rules in the selected boundary. - Standardize duration representations, explicit empty/null/zero handling, and inheritance on the shared conventions. Express genuine Audita domain rules through the supported resolution and validation APIs. -- Review the configuration-related CLI afresh. Useful explicit flags can become - ConfigKit override records; old names and aliases are not requirements. Direct - credential-value flags should give way to the secret-reference convention. +- Retained domain-setting flags and proposal/validation profile flags become + explicit ConfigKit overrides according to the migration inventory. Remove old + provider/key/retry/concurrency flags and aliases. Credentials use secret references. - Use ConfigKit's redacted effective output and provide source/explanation access - where useful. Existing JSON inspection output need not be preserved. Coordinate - diagnostic configuration snapshots with the new model. + through the selected views below. Legacy JSON inspection is replaced. + Diagnostic configuration snapshots use the same new model. - Delete superseded loading and override machinery. Do not hide the old model behind adapters that defeat the adoption's consistency benefit. @@ -102,228 +102,69 @@ effective-config inspection, selected-profile inspection, and processing must al use the same selected overlay. Preserve overlay provenance in configuration explanations and diagnostics while retaining secret redaction. -The completed prototype extension and production-planning handoff are recorded -in the [prototype roadmap](library-boundary.md#next-unit-testing-overlays-and-production-migration-plan) -and its ordered [implementation plan](implementation.md). Production CLI changes -below remain subsequent migration work. +The [recorded overlay results](library-boundary-findings.md#selected-overlay-extension-2026-09-24) +prove the representative load, inspection, and HTTP path. Production acceptance +requires the same selection to reach both actual LLM roles, with explicit flags +winning over overlays and surviving secret references resolved/redacted correctly. +The isolated prototype is evidence, not a second supported runtime. -Implementation and acceptance work: +## Inspection and Invocation Contract -- The isolated prototype pins ConfigKit v0.4.0 and includes the testing-overlay - fixture. Preserve the original v0.3.1 evidence as historical; the - [extension results](library-boundary-findings.md#selected-overlay-extension-2026-09-24) - establish the selected-overlay behavior. -- Verify unchanged base selection when no overlay is requested, both testing role - selections, and explicit setting flags taking precedence over the overlay. -- Verify role inheritance after overlay application, effective profile/backend - assembly, and secret resolution from the surviving configuration. -- Reject missing/invalid overlay selections and prohibited version/directive changes - through ConfigKit; rely on its tests for exhaustive merge/parser cases. -- Verify redacted inspection and overlay provenance, then run both LLM roles through - the offline provider using the selected testing profiles. Provider requests must - use their model selections; no paid calls are needed. -- Add production CLI coverage for the three commands and document selection, - precedence, example layout, and migration guidance when implemented. +`process`, `config validate`, and `config print-effective` share file/overlay +selection and retained setting overrides. Invocation I/O paths remain process-only. +Only explicitly supplied flags override settings. Validation requires an explicit +config file; processing and inspection use the selected discovery order. Help is +available before configuration loading. All three commands validate selected +profiles without requiring a provider call during setup. -This expands the next configuration unit; it does not reopen the completed -prototype's original scope or change the ConfigKit → PromptKit → combined release -sequence. The production LLM migration remains necessary before promising the full -new runtime behavior. +`config print-effective --view settings|profiles|explain` exposes three YAML views: -## Ordered Production Migration Handoff +- `settings` (default): ConfigKit's redacted effective configuration. +- `profiles`: an Audita-owned summary of each selected role's profile, backend, + model, temperature, token limit, top-p, timeout, service tier, and reasoning + effort where present. Arbitrary extra provider parameters and credentials are + omitted; this is a selected-controls view, not every execution input. +- `explain`: field paths, sources, and derivation records from ConfigKit, without + resolved secret values. -The [migration inventory](library-boundary-findings.md#migration-inventory) is the -field-by-field old-to-new map. The [selected boundary](library-boundary-findings.md#configuration-and-profile-contract) -owns defaults and semantic choices; the [v0.4.0 overlay results](library-boundary-findings.md#selected-overlay-extension-2026-09-24) -prove the representative load, inspection, and HTTP path. These packages are -future work within one coordinated refactor to adopt both libraries. The order -organizes implementation dependencies, not independently runnable product states. -Intermediate commits need not build or run; do not add temporary scaffolding, -compatibility shims, or a processing gate to support them. Remove superseded code -as its replacement lands. Run focused checks as components become testable, and -validate the complete application before declaring the refactor finished. The -isolated prototype is evidence, not a second production runtime. +A ConfigKit snapshot alone does not describe profile-owned controls. Profile +inspection must enforce the registered-backend and credential boundary. Diagnostic +snapshots use these safe views and preserve overlay/explicit-override provenance. +The migration guide explains the intentional replacement of legacy inspection JSON. -### 1. Define and load the complete version-2 configuration +## Integration and Completion Criteria -**Depends on:** the verified prototype. **Owns:** `internal/core/config` and its -tests. Replace the version-1 YAML/env application path with one ConfigKit v0.4.0 -definition and a version-2 rule. Pin the published module in the root Go module -and raise the production Go minimum as needed. Keep the existing YAML dependency -used for glossary parsing. ConfigKit imports and secret-directory bootstrap stay -disabled. +ConfigKit and PromptKit form one intentional breaking refactor. Intermediate +commits need not build or run. The target contains no temporary old-runtime adapter, +compatibility shim, processing gate, dual loader, or automatic legacy conversion. +The [combined feature roadmap](library-adoption.md) owns execution/reporting intent; +[implementation.md](implementation.md) alone owns stage order and implementation tasks. -The definition must cover every retained domain setting in the inventory: -ordered/repeated `pipeline.modules`, `output.schema`, validation prompt budget, -chunking, normalization, thresholds, context, and diagnostics, plus `profiles_dir`, -backend endpoint/concurrency/queue capacity/optional credential, and LLM proposal -and validation profile selection, transient retries, repair attempts, and total -operation timeout. Apply the inventory's defaults once. Preserve presence: -omitted validation profile inherits the **final** proposal profile; explicit -empty/null validation fails. Omitted or null section target means automatic, zero -fails. Explicit empty module list means no modules. Optional credential null means -unauthenticated; supplied references must resolve to nonempty values. Keep the -selected numeric-second normalization fields numeric, and use a duration string -for `llm.operation_timeout`; do not revive version-1 duration coercion. Validate -backend limits, references, profile selections, thresholds, and cross-field domain -rules on the resolved result. Preserve ConfigKit provenance for derived values. +The final application loads all retained domain settings, assembles both selected +profiles, and processes through PromptKit. Controlled HTTP evidence must connect +configuration selection to actual requests, domain outcomes, safe diagnostics, +and shared finite backend capacity. Maintained examples and one migration guide +cover the complete file/flag/environment transition. Obsolete application-owned +loading, transport, rendering, and scheduler machinery is removed. -**Acceptance/tests:** `internal/core/config` tests use temporary files to cover -version rejection, strict fields, defaults, omission/null/zero, module order, -final role inheritance, overlays and explicit override precedence, secret -replacement/missing/null, and retained domain validation. Compare behavior to the -prototype extension rather than duplicating ConfigKit's parser matrix. No legacy -environment overlay or old-schema acceptance remains. +The combined target is Audita v2.0.0. Release compliance and publication remain +last and separate from implementing the refactor. Retention and transcript timestamp +changes are not included; the selected retry/repair contract resolves the retry +follow-up in [runtime behavior](runtime-behavior.md). -### 2. Select files and map invocation input - -**Depends on:** package 1. **Owns:** `internal/cli/run.go`, -`internal/cli/process_flags.go`, `internal/cli` tests, and `cmd/audita` subprocess -tests. Select exactly one main file: explicit `--config`, then the two system -locations in the selected boundary. Remove `AUDITA_CONFIG`; a missing explicit -path fails, while absent system files are skipped. Require an explicit file for -`config validate`. Parse help before loading configuration. - -Add invocation-only `--overlay ` to `process`, `config validate`, and -`config print-effective`, passing it to `Options.Overlay` for the same selected -main file. Omitted, empty, or `default` selects the base. Keep transcript, -glossary, output, and report paths outside persistent configuration. Map only -flags actually supplied by the user into ConfigKit overrides; parser defaults -must never defeat file or overlay values. Retain useful domain-setting flags from -the inventory. Add profile-selection flags. Remove direct API-key/model/endpoint, -old retry/role/global concurrency flags and aliases rather than translating them -silently. Retire all general `AUDITA_*` configuration overlays listed in the -inventory; environment names retain meaning only when explicitly referenced as -secrets. - -**Acceptance/tests:** CLI and subprocess tests cover discovery order, explicit -missing files, all three commands with base/testing/default/missing selections, -overlay then explicit-setting precedence, omitted flags, usage/error streams and -exit classifications. Reject old flags and implicit environment configuration. -The same loaded result must feed validation, effective inspection, and processing. - -### 3. Assemble profiles and safe inspection - -**Depends on:** packages 1–2. **Owns:** a small CLI/runtime assembly boundary, -config inspection, and diagnostic snapshot wiring in `internal/cli` and -`internal/core/diagnostics`; verify against `internal/framework/processreport` -where snapshots are referenced. Resolve `profiles_dir` relative to the selected -**main** file, including when an overlay is selected; absolute paths remain valid. -Inspect both selected PromptKit profiles and verify their named registered backend, -endpoint equality, and absence of a competing profile credential lookup. Treat -ConfigKit settings and PromptKit generation controls as separate views. - -Use `WriteEffective` for redacted effective YAML and expose relevant -explanation/provenance and selected-profile model/backend controls through an -Audita-owned safe view. A ConfigKit snapshot alone cannot claim to describe the -full execution target. Validation and inspection resolve surviving secret -references without provider calls; a missing or empty surviving reference fails, -while a replaced reference is not resolved. Snapshot and surfaced error paths -must redact secret **values** and preserve overlay/CLI origin. Avoid serializing -raw library objects into stable report formats. - -**Acceptance/tests:** config-command and diagnostics tests verify relative paths, -same selection across views, overlay/CLI origin, surviving credentials, -unauthenticated null, and no secret values in YAML, explanations, selected-profile -inspection, diagnostics, or errors. Update `internal/cli/run_test.go`, -`internal/core/config/*_test.go`, and report/diagnostics tests that own these -outputs. - -### 4. Connect the final configuration directly to PromptKit execution - -**Depends on:** packages 1–3. **Owns:** the final CLI/runtime assembly boundary, -`internal/framework/llm`, prompt/schema loading, runner wiring, and relevant tests. -Implement PromptKit adoption as part of the same refactor. Convert domain settings -into Audita's existing domain structs and supply the selected profiles, registered -backends, resolved secrets, retry/repair budgets, and logical-operation deadlines -directly to PromptKit. Keep library types out of correction algorithms and ledgers. - -Migrate all module and validator prompts, preserve their domain checks, and map -PromptKit results, classified failures, and observations into Audita reporting and -redacted diagnostics. Replace the old provider transport, prompt loader, -structural-output handling, and global/role scheduler composition. Follow the -[coordinated execution scope](library-adoption.md#4-complete-promptkit-adoption) -and [execution/observation contract](library-boundary-findings.md#execution-and-failure-mapping). -Do not build an adapter to feed the old LLM runtime or introduce an interim -`process` rejection mode. - -**Acceptance/tests:** controlled HTTP tests prove both roles' final model/backend -selection, overlays and explicit overrides, finite per-backend capacity, explicit -zero retries, positive structural repair, total-operation deadlines, cancellation, -domain acceptance/rejection, and safe observations including partial failures. -Processing uses the new configuration and execution path together. Reuse existing -domain coverage and the prototype's integration evidence; update tests tied to the -retired transport and scheduler contracts. - -### 5. Remove superseded configuration paths and update maintained contracts - -**Depends on:** packages 1–4. **Owns:** removal of old file/env merge, aliases, -direct-key flags, legacy redaction/inspection paths, and obsolete tests in -`internal/core/config` and `internal/cli`. Retain Audita's domain checks, -normalization/chunking, output encoding, module and validator behavior. Update -`examples/` with version-2 base configuration, `overlays/testing.yml`, complete -PromptKit profiles, and secret-free credentials by reference. Update the canonical -`docs/config.md`, `docs/cli.md`, operations/integration/internal references as -their contracts change, and provide one migration guide with the complete -inventory's file/flag/environment mapping, duration and empty-list changes, -credentials, profiles/overlays, inspection, and changed runtime/diagnostic -semantics. Do not leave dual loaders or a permanent translation mode. - -**Acceptance/tests:** maintained examples validate offline; removed flags and old -schema are rejected; relevant config/CLI/subprocess/report tests pass; documentation -links and `git diff --check` pass. Remove the isolated prototype when its useful -coverage and examples have moved into the production suite. - -### 6. Validate the combined end state and prepare release - -**Depends on:** packages 1–5. Verify the assembled version-2 configuration, -overlay, profile, execution, reporting, and artifact workflows. Run the relevant -full build/test checks, confirm obsolete configuration and LLM machinery is gone, -and verify maintained examples and the migration guide against the final behavior. -There is no requirement to validate or preserve intermediate product states. - -The refactor is complete when both ConfigKit and PromptKit are adopted and the -complete application passes its required checks. Release compliance, license/ -distribution review, and publication follow last under the established release -procedure. No publication is authorized by this plan. - -## Sequencing Considerations - -Changes to module configuration, runtime settings, CLI design, or diagnostic -reporting are likely to overlap this migration. Design their shared target model -first. If another change will substantially reshape those areas, avoid migrating -the same concepts twice. Work independent of configuration can proceed separately. - -The [coordinated library-adoption roadmap](library-adoption.md) owns the proposed -sequence with PromptKit: shared boundary design and a PromptKit prototype, then -ConfigKit and PromptKit implementation in dependency order within one refactor, -followed by a combined v2.0.0 release. Intermediate commits are work in progress, -not supported configurations or runnable milestones. - -The separate [runtime behavior decisions](runtime-behavior.md) track remaining -questions and the selected retry direction. -The coordinated plan adopts PromptKit's explicit retry semantics and enables -structural repair from the outset. Include separate transient-retry and positive -repair budgets, plus the total-operation deadline policy, in the shared settings -design. The selected defaults and ownership are recorded in the -[boundary findings](library-boundary-findings.md#configuration-and-profile-contract). -Implement the runtime change explicitly as part of the combined refactor. -Retention and transcript timestamp decisions are not automatically included. - -## Implementation Boundaries +## Scope Boundaries - Pin ConfigKit v0.4.0 in production and align Go/toolchain requirements. ConfigKit requires - Go 1.25; the combined PromptKit prototype requires Go 1.25.5. Config-only - adoption does not automatically remove Audita's existing YAML dependency used for glossary parsing. + Go 1.25; the combined PromptKit prototype requires Go 1.25.5. Adopting these libraries + does not automatically remove Audita's existing YAML dependency used for glossary parsing. - Implement the selected configuration and final PromptKit runtime directly, without temporary runtime adapters, compatibility shims, or process gates. - Enable selected overlays as specified above. Keep imports and secret-directory bootstrap disabled unless a separate concrete requirement justifies them. - Validation and inspection resolve surviving credential references and fail on missing or empty values; no credential-free bypass is planned. -- Add the safe selected-profile view and ConfigKit provenance to inspection and - diagnostics as specified in package 3, then account for changed formats in the - migration guide. No schedule or effort estimate is committed. +- Inspection and diagnostics use the safe selected-profile view and ConfigKit + provenance described above. Changed formats are covered by the migration guide. The eventual completion test is that Audita follows the suite's ConfigKit conventions while retaining its intended domain capabilities, with a clear diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 7409ff0..d2d7b9a 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,306 +1,527 @@ -# Implementation Plan: Testing Overlays and Migration Handoff +# Implementation Plan: Adopt ConfigKit and PromptKit **Last updated:** 2026-09-24 -**Status:** Complete (2026-09-24). All six stages below are finished. This plan implements the next -unit in the [prototype roadmap](library-boundary.md#next-unit-testing-overlays-and-production-migration-plan), -not the production ConfigKit or PromptKit migrations. +**Status:** Ready for implementation. All stages below are pending. This replaces +the completed testing-overlay prototype plan; its results remain in the +[boundary findings](library-boundary-findings.md#selected-overlay-extension-2026-09-24). -**Production handoff revision:** The completed prototype stages remain recorded -below. The current [production plan](configkit.md#ordered-production-migration-handoff) -targets both libraries in one refactor; intermediate commits need not build or run. -It supersedes the earlier interim-runtime design. No temporary adapter, -compatibility shim, or processing gate is required. +## Scope and Working Instructions -Execute Stage 1 through Stage 6 in order. Each stage is sized as a separate coding -prompt for a gpt-5.6-terra agent and ends with a reviewable result. When assigned -one stage, finish that stage and report the next stage; when assigned the entire -plan, continue through all stages. Do not restart completed work: inspect current -files, recorded results, and Git state before deciding what remains. +Implement the final configuration and execution model in the +[combined feature roadmap](library-adoption.md) and [configuration roadmap](configkit.md). +The [selected boundary](library-boundary-findings.md#configuration-and-profile-contract), +[migration inventory](library-boundary-findings.md#migration-inventory), and +[failure mapping](library-boundary-findings.md#execution-and-failure-mapping) +are the detailed behavioral inputs. Do not repeat the prototype or reopen settled +choices without a concrete contradiction in the pinned libraries. -Before each stage, follow `AGENTS.md`, [development guidance](../development.md), -and the [agent policy](../policy/agents.md). Preserve unrelated changes. Use the -knowledge graph for code discovery; examples may be excluded from its index, in -which case inspect the known prototype files directly. Follow the repository's -testing and documentation policies. Complete relevant checks and make a plain -English local commit for each coherent stage under the standing commit policy. -No push, tag, release, or upstream-library edit is included. +This plan is for a gpt-5.6-terra coding agent. Execute Stage 1 through Stage 12 in +order. Each stage is one bounded implementation prompt. When assigned the whole +plan, continue between stages without a permission round. When assigned one stage, +finish that stage's scope and report the next stage. Mark completed stages with a +short result and actual validation evidence; do not call a pending integration +check passed. -Update each stage's status line with its result and verification evidence when -finished; keep detailed design/results in their owning roadmap documents. Do not -mark a stage complete while required checks are failing. Fix in-scope failures; -record any genuine upstream gap and its effect on later stages before continuing -work that depends on it. Routine implementation choices do not need fresh approval. +Read `AGENTS.md`, [development guidance](../development.md), and all documents under +`docs/policy/`. Use graph-first code discovery. Follow the +[agent](../policy/agents.md), [architecture](../policy/architecture.md), +[testing](../policy/testing.md), [documentation](../policy/documentation.md), and +[release](../policy/release.md) policies. The intentional dependency, configuration, +and scheduler changes here supersede the corresponding old implementation choices; +update canonical descriptions as the new behavior lands. -The [ConfigKit roadmap](configkit.md#selected-overlays-for-testing) owns the overlay -contract. The [boundary findings](library-boundary-findings.md) own existing choices -and evidence. Preserve the original v0.3.1 results as historical. ConfigKit v0.4.0 -renames selected configuration profiles to overlays; do not rename PromptKit's -`profiles/`, `profiles_dir`, or execution-profile fields. +The stages divide one refactor, not supported intermediate releases. Intermediate +commits may fail to build or run. Do not write temporary adapters, compatibility +shims, duplicate execution paths, stubs returning invented successes, or an interim +processing gate. Remove superseded code when replacing it. Run focused checks +where the affected packages are testable; record checks deferred because later +stages must reconnect callers. Such a completed stage is still work in progress +on the overall refactor. Commit coherent stages with plain-English messages under +the standing policy. Do not spend a stage making an intermediate application work. -All implementation in this plan stays under `examples/library-adoption/`, plus -relevant roadmap/README updates. Keep the root Go module, production CLI/runtime, -correction algorithms, and upstream libraries unchanged. Use the existing built-in -PromptKit HTTP path with a controlled transport, synthetic credentials, and no live -provider calls. Keep imports and secret-directory bootstrap disabled. This work -does not introduce general environment overlays or backwards compatibility. +Keep tests deterministic, offline, and credential-independent. Reuse prototype +scenarios at their production boundary rather than copy the nested application or +retest all library internals. Preserve domain tests unless the selected contract +actually changes. Update each affected canonical reference alongside its owning +stage; Stage 11 finishes the cross-document migration guide and examples. -Unless explicitly stated otherwise, shell commands below run from -`examples/library-adoption/`. Use an installed Go 1.25.5 or newer. Acquire dependencies -with `GOWORK=off go mod download all` when needed; then use the offline command -prefix below. Root `go test ./...` does not cover this nested module. +No upstream edits, paid calls, push, tag, publication, retention-policy change, +timestamp fix, or unrelated pipeline redesign is included. Release compliance +and publication remain a later release unit. Completion here means both libraries +are adopted, obsolete machinery is removed, and the final application is verified. -## Stage 1: Upgrade the Prototype Dependency +## Implementation Decisions -**Status:** Complete (2026-09-24). **Depends on:** nothing. ConfigKit v0.4.0 pinned; -the nested offline tests and demonstration passed with the expected 1/4/0/2 result. +These decisions close the remaining application-design details; background and +user-facing policy belong to the feature roadmaps. -Read the prototype README, `go.mod`, `go.sum`, and the current ConfigKit roadmap. -Confirm the working tree and run the existing nested-module tests once as a -baseline. Upgrade ConfigKit to the published v0.4.0 release using module tooling; -leave PromptKit and Audita dependency versions unchanged. Review any indirect -module changes and retain only changes needed by this upgrade. Do not add local -replacements or an active Go workspace. +- Pin the published modules used by the verified prototype: ConfigKit v0.4.0 and + PromptKit v0.11.0. Set the root minimum to Go 1.25.5. Keep yaml.v3 while glossary + parsing uses it. No sibling replacements or dependency on Audita's own old release. +- Keep application loading in `internal/core/config`, embedded prompt definitions + in `internal/prompts`, schemas in `internal/framework/responseschema`, and the + PromptKit integration in `internal/framework/llm`. Use one engine per process + invocation, shared by both role-bound clients; neither modules nor validators + construct engines. The CLI composes these pieces. Avoid a generic service layer. +- Retain the `CompleteStructured(ctx, request, out)` boundary, using Audita-owned + prompt ID/version, string variables, schema identity, and diagnostic context. + Replace the old rendered-message/model request fields. Model/profile/credential + selection belongs to the role-bound adapter, not individual callers. Consolidate + the equivalent validator types onto the framework contract where needed, without + an import cycle. No PromptKit types in correction algorithms or ledger records. +- Store PromptKit definitions for all nine existing prompt IDs with their existing + `v1` selection. Replace the custom `hardening` template function with an + application-supplied `HardeningPolicy` string variable sourced from the one + embedded hardening fragment. The adapter supplies that reserved variable after + caller variables. PromptKit renders all messages; do not retain an Audita template + engine or introduce generated prompt files requiring a second build step. +- Use one shared configuration-input parser for `process`, `config validate`, and + `config print-effective`. All three accept `--config`, `--overlay`, and the + retained setting overrides in the inventory, including the two profile flags. + Only `process` accepts transcript/glossary/output/report paths. Only explicitly + visited setting flags become ConfigKit overrides. Do not add retry/capacity/key + flags beyond the agreed inventory. +- `config validate` requires an explicit nonempty config path. The other two + commands use the selected discovery order. All validate the resolved settings + and both selected profiles without provider calls, including an empty module + pipeline. Help succeeds before file discovery or secret/profile loading. +- `config print-effective` adds `--view settings|profiles|explain`, defaulting to + `settings`. Settings uses ConfigKit `WriteEffective` YAML. Profiles emits a safe + Audita-owned YAML summary for the proposal and validation roles. Explain emits + a YAML sequence of field path, source, and derivation records from ConfigKit's + recorded resolution, with no resolved secret values. Unknown views are usage + errors. This avoids mixing multiple formats into the default output or adding + another command tree. Profiles includes selected ID, backend, model, temperature, + max tokens, top-p, per-attempt timeout, service tier, and reasoning effort when + present; omit arbitrary extra provider parameters and credentials. It is a + selected-controls summary, not a dump of every profile input. +- Keep CLI exit classes 0/1/2 and transcript/report stream behavior. Invalid user + settings/profiles are configuration errors (2); broken embedded assets and + runtime/provider failures are runtime errors (1). Serialization/write failures + remain runtime errors. Do not classify malformed HTTP envelopes as repairable + model output. +- Publish process reports as `audita-process-report` schema `v2`. Preserve existing + domain fields and ledger ordering, add a top-level `llm_summary`, and replace + obsolete scheduler utilization fields as specified in Stage 10. Transcript + encoder versions stay unchanged. No legacy report mode is needed. -No source changes were needed in the earlier isolated review. If a change is now -necessary, investigate and record the concrete reason instead of assuming that -ConfigKit execution overlays and PromptKit profiles share an API. +## Stage 1: Replace the Configuration Model and Loader -Verification: +**Status:** Pending. **Depends on:** the completed prototype only. +**Owns:** root `go.mod`/`go.sum`, `internal/core/config`, and config tests. -```sh -GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test ./... -GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . -``` +1. Pin both modules and the Go minimum above. Align build/CI toolchain declarations + with that minimum; preserve the existing platform set and release gates. +2. Replace version-1 YAML/env merge and duration coercion with one ConfigKit + definition covering every retained field in the migration inventory. Use the + prototype's types/lifecycle as evidence, then add the full domain configuration. + Remove legacy environment loaders and alias resolution; do not keep two loaders. +3. Implement defaults, presence-aware backend defaults, final validation-profile + inheritance, and recorded derivations. Keep `chunking.target_sections` optional, + explicit empty module lists empty, and module order/repeats intact. Validate + domain constraints after overlay and explicit override resolution. Preserve + numeric normalization seconds; operation timeout is a duration string. +4. Use typed optional secret references. Resolve every surviving reference, including + unselected backends; missing/empty references fail, replacement resolves only the + replacement, and null permits unauthenticated use. Enable selected overlays; + leave imports and secret-directory bootstrap disabled. +5. Expose the ConfigKit result to CLI composition for effective output and provenance; + convert retained settings into existing domain inputs without transporting the + library result through correction code. Update `docs/config.md`'s implemented + loader/schema description, clearly marking unfinished CLI integration if needed. -Acceptance: the manifest and sums pin v0.4.0, tests pass, and the demonstration -still yields one accepted correction, four attempts, zero transient retries, and -two structural repairs. Update the README/findings to distinguish the new pin from -historical v0.3.1 evidence. Do not claim overlay integration yet. +**Acceptance and validation:** focused config tests cover strict version/fields, +selected defaults, omission/null/zero, final role inheritance, empty/repeated +modules, overlay/override precedence, secret replacement/null/missing, and retained +cross-field rules. Old schema fails with migration guidance. General environment +variables have no implicit effect. Run `go test ./internal/core/config` when this +package is independently buildable; record caller compile failures for later stages. +Do not duplicate ConfigKit's exhaustive YAML/merge tests. -## Stage 2: Add Overlay Loading and Configuration Fixtures +## Stage 2: Replace CLI Configuration Selection and Overrides -**Status:** Complete (2026-09-24). **Depends on:** Stage 1. On-disk overlay and -profile fixtures, precedence, inheritance, rejection, and runtime assembly tests -pass in the nested module (`go test ./...`). +**Status:** Pending. **Depends on:** Stage 1. +**Owns:** `internal/cli/run.go`, `process_flags.go`, configuration command parsing, +and corresponding CLI/subprocess tests. -Start with `config.go`, `config_test.go`, `config.yaml`, and `profiles/`. Extend -`loadConfig` with an explicit overlay-selection argument or a small invocation -options value. Pass it to ConfigKit `Options.Overlay`; selection does not belong -in the persistent `Config` struct. Update existing callers to request no overlay -without changing their behavior. Avoid a parallel merge implementation. +1. Implement the shared configuration-input parser described above. Preserve the + existing supported positional-argument arrangement for `process`, required + glossary, output/report behavior, and root/subcommand help. Remove obsolete + LLM/model/key/timeout/retry/concurrency flags and aliases outright. +2. Select exactly one file: explicit `--config`, then + `/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`. Missing system + candidates are skipped; selected inaccessible files and explicit empty/missing + paths fail. With no file, report a configuration error; do not select a provider. + Remove `AUDITA_CONFIG`. Test discovery using an injected path list/filesystem + seam rather than modifying the host's system configuration. +3. Pass invocation overlay selection to ConfigKit and only supplied setting flags + to its overrides. `--modules ''` must clear the pipeline; explicit numeric zero + reaches validation rather than being treated as absent. Relative profile bases + derive from the selected main file, never the overlay directory. +4. Wire the parsed result into the final process/config command entry points. + Profile-dependent command completion belongs to Stage 4; do not emulate the old + runtime to make the process command work now. Update `docs/cli.md` for the final + input contract as parsing lands. -Add these prototype fixtures: +**Acceptance and validation:** parser/loading tests prove help without usable +configuration, discovery/error cases, explicit-only overrides, retained domain +flags, removed flag rejection, and ignored legacy environment values. Table-drive +shared cases; retain representative subprocess checks rather than triplicate every +parser test across commands. Config commands reject positional arguments and +process-only flags. Defer assembled command checks until Stage 8 if imports still +prevent compilation. -- `overlays/testing.yml`, selecting `proposal-testing` and `assessment-testing` - through the existing two `llm` role fields. Do not put `version`, model settings, - or ConfigKit directives in this overlay. -- Complete PromptKit profiles under `profiles/` with those IDs and distinct - synthetic model IDs, such as `synthetic-grammar-testing` and - `synthetic-review-testing`. Both use the existing `offline` backend and keep the - current positive repair policy and finite shared capacity. +## Stage 3: Migrate Embedded Prompt Assets and Call Contracts -Preserve base `config.yaml` as the no-overlay fixture, including its explicitly -selected assessment profile. This makes the risk of accidentally leaving the -validation role on its base model observable. Profile paths remain relative to -main configuration according to the selected application contract, not relative -to the overlay file. +**Status:** Pending. **Depends on:** Stage 1; execute after Stage 2. +**Owns:** `internal/prompts`, `internal/framework/responseschema`, +`internal/framework/contracts`, validator contract aliases, and prompt tests. -Add focused configuration tests proving: +1. Convert all four module and five validator prompts listed in + [the prompt registry](../internal/prompts.md) to PromptKit definitions. Preserve + instructions, variables, glossary/description context, prompt IDs, and the two + existing JSON response schemas. Embed assets once and expose their filesystem + to the runtime. Do not copy the prototype's duplicate schema directory. +2. Keep output contracts JSON Schema based, but supply the full effective output + contract and repair count on each runtime request. Do not create separate + operator-facing repair defaults per prompt. Use the hardening variable decision + above to retain a single hardening source. +3. Change the structured-call request to prompt identity, variables, response schema + identity, and diagnostic context. Keep decoded output and typed Audita metadata + as the return boundary. Include role/module instance/section/batch/validator + context where available for attribution; do not derive identities from messages. +4. Replace the old parser/renderer and hash assumptions. Keep small metadata and + schema lookup helpers where Audita needs them. Update the prompt reference. -- Omitted selection, empty selection, and `default` select the base configuration; - `testing` selects both new profiles. -- Explicit ConfigKit overrides win over overlay values. Only supplied flags will - become overrides in Stage 3; parser defaults must not enter this layer. -- With validation selection absent from both base and overlay, validation inherits - the final proposal selection, including an explicit override. An existing - explicit base validation selection remains explicit unless replaced by overlay - or CLI. Empty/null validation values retain the established rejection behavior. -- Runtime profile assembly resolves both selected profiles to the shared backend. +**Acceptance and validation:** a table covering all nine definitions prepares +representative variable sets through PromptKit with controlled profile fixtures; +checks include hardening, section/glossary/description data, missing variables, +correct schema association, and rendered content preserving the previous task +instructions. Use reviewed expectations, not a retained legacy renderer or old +Audita module dependency. Existing schema/domain tests remain meaningful. Callers +are deliberately migrated in Stages 6–7. -Use temporary on-disk layouts for overlay tests so ConfigKit owns file discovery. -Existing byte-source tests can remain for cases that do not select an overlay; -if a byte source needs an overlay, supply ConfigKit's supported filesystem context -rather than relying on the process working directory. +## Stage 4: Assemble Profiles and Finish Offline Inspection -Acceptance: the new fixtures load and assemble correctly, the focused precedence/ -inheritance tests pass, and `go test ./...` in the nested module still passes. -Do not yet claim that the testing models reached the HTTP transport. +**Status:** Pending. **Depends on:** Stages 1–3. +**Owns:** engine/profile assembly in `internal/framework/llm`, CLI config commands, +and inspection tests. -## Stage 3: Demonstrate and Verify Model Switching +1. Construct one PromptKit engine with the embedded assets, complete profile files, + and registered application backends. Reuse the prototype's supported filesystem, + backend, retry-config, and inspection APIs. Accept an HTTP client at the external + boundary for tests; production uses the library transport. +2. Resolve `profiles_dir` against the main config file. Inspect both selected roles, + require a registered backend, reject endpoint-only bypass and profile credential + lookup, and compare endpoints after trimming surrounding whitespace and trailing + slashes, as in the prototype. Do not invent profile inheritance/catalog wiring + or reject unrelated unselected profile content beyond library behavior. +3. Complete `config validate` and the three effective-output views. Settings, + profiles, and explanations share the same load/assembly path used by process. + Keep the `config is valid` success message. Redact configured secret values in + any surfaced library error; do not stringify raw configuration/profile objects. +4. Add the internal safe-view helpers later used for snapshots. Document the view + interface and its limits in `docs/cli.md` and `docs/config.md`. -**Status:** Complete (2026-09-24). **Depends on:** Stage 2. Nested tests inspect -outbound base/testing/override model IDs, including repairs; all three documented -offline demonstrations passed with one accepted correction and 4/0/2 attempts, -retries, and repairs. +**Acceptance and validation:** offline command tests prove all views and validation +agree for base/testing/explicit-profile selections; main-file-relative and absolute +profile paths work; missing profiles, backend mismatch, competing credentials, +missing surviving secrets, and invalid views fail with the selected exit class. +Use a transport that fails the test if called. Inspect outputs/errors for synthetic +secret absence and overlay/CLI provenance, including inherited validation selection. -Start with `main.go`, `offline.go`, `runtime.go`, and `workflow_test.go`. Add minimal -standard-library argument handling to the prototype executable: +## Stage 5: Implement the Final PromptKit Structured-Call Adapter -- `--overlay ` selects the ConfigKit overlay. -- `--proposal-profile ` and `--validation-profile ` produce ConfigKit - overrides only when explicitly supplied. -- Keep the existing fixture-based invocation from the prototype directory. There - is no need to reproduce production file discovery or its subcommand tree here. +**Status:** Pending. **Depends on:** Stages 3–4. +**Owns:** `internal/framework/llm`, the typed failure boundary in +`internal/framework/structuredoutput`, and adapter tests. -Use a small testable entry function if needed to check flag handling without -repeated subprocess builds. Preserve ordinary flag error handling. Keep execution -on the synthetic transport for every supported demonstration invocation; do not -add a live-network mode. Extend that transport to recognize the testing models and -preserve the initial-invalid-output/repair behavior for each role. +1. Implement role-bound clients over the shared engine. Start the operation context + and Audita logical-call record before preparation; inherit earlier caller deadlines. + Select the role profile and backend secret, supply reserved hardening variables, + and pass an explicit full schema/repair contract. Use `PrepareExecution`, its + `Details`, then `RunPrepared` so diagnostic messages/hashes describe the actual + execution snapshot. Defer handle discard; do not call `Prepare` and then `Run` + with an independently reloaded profile. Before dispatch, verify the prepared + target still matches the role's registered backend/endpoint and has no competing + credential lookup. A changed target fails safely rather than sending a backend + key to a different target. This is the final diagnostic/security boundary, not + an intermediate adapter. +2. Check both the returned error and `result.Validation.IsValid`: exhausted repair + can return a result with failed validation and no error. Decode only validated + JSON into the requested Audita output. Use the selected failure table: + exhausted output repair becomes the Audita malformed-output classification; provider/transport/protocol/retry/admission/deadline failures + remain runtime failures. Preserve typed causes for classification and safe + messages for users. Explicit zero retries must not become retries-exhausted. +3. Preserve prepared messages/hashes and available final content and profile/model + metadata through Audita-owned types. Provide a per-call observer hook and logical + start/finish record bound to Audita diagnostic context for Stage 9; do not + create another retry loop, scheduler, provider envelope parser, or schema engine. +4. Remove the old OpenAI-compatible client and its role config resolution. Retain + only applicable redaction and diagnostics helpers. Update the LLM integration + reference to delegate wire/retry mechanics to the pinned library. -Add a compact safe selection summary to demonstration output: requested overlay -and effective profile, backend, and model for each role, alongside the existing -domain outcome and attempt/retry/repair totals. It must reflect the assembled -configuration rather than echoing flag values. Inspection APIs and tests in the -next stage suffice; do not create production config commands in this prototype. +**Acceptance and validation:** controlled HTTP tests cover successful decode, +positive repair and exhaustion for both schema types, 429/5xx retry recovery, +explicit zero, exhausted positive budget, transport/malformed-envelope failure, +combined retry-plus-repair bounds, preparation failure without dispatch, exact +prepared request metadata, target-change rejection, and operation cancellation +during a long retry wait. Assert classifications and meaningful counts, not timing-sensitive sleeps +or complete error strings. Reuse the prototype sequences; no need to reproduce +PromptKit's entire status-code matrix. -Test through the assembled path and inspect actual outbound HTTP request bodies: +## Stage 6: Migrate All Proposal Callers -- No overlay sends the base proposal and assessment models. -- `testing` sends the testing proposal and assessment models, including repair - requests, and still produces one accepted correction with two repairs. -- An explicitly supplied role-profile flag overrides that role's testing selection; - the other role retains its overlay selection. An omitted flag preserves overlay - selection. Match reported selections to observed outbound model IDs. +**Status:** Pending. **Depends on:** Stage 5. +**Owns:** `internal/modules`, `internal/framework/proposal_generation`, and their tests. -Use distinct models per role for these demonstration cases. Do not expand the fake -provider into a general simulation framework. Reuse existing prompt/schema/domain -helpers and existing capacity/observation coverage. +1. Switch glossary, homophones, spoken_word, and grammar proposal construction to + prompt IDs and variables. Preserve section/glossary encoding, context trimming, + replacement policy, confidence filtering, and applicability checks. +2. Feed repaired structured output into the existing domain path. Exhausted + structural repair produces the existing section warning and no proposals; + runtime failures propagate. Schema-valid but inapplicable proposals do not + trigger semantic retries or additional repairs. +3. Update fake clients and module prompt tests to the final request contract. + Delete duplicate message rendering and obsolete transport/schema checks; retain + domain protection and prompt variable coverage. Update module internals docs. -Acceptance: nested-module tests pass and these invocations visibly show the expected -selections and successful domain result: +**Acceptance and validation:** all four modules have an exercised variable/schema +mapping; existing domain suites retain their assertions. Representative controlled +adapter coverage proves repaired proposals reach domain checks and that a low- +confidence/inapplicable proposal remains rejected. Preserve section/proposal +ordering. Run the module and proposal-generation suites as imports become buildable. -```sh -GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . -GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing -GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go run . --overlay testing --proposal-profile proposal -``` +## Stage 7: Migrate LLM Validators Without Changing Domain Decisions -Update the prototype README with these implemented commands and fixture layout. -State that synthetic testing IDs demonstrate operator-selectable routing, not -measured price, speed, or model quality. +**Status:** Pending. **Depends on:** Stage 6. +**Owns:** `internal/framework/validators`, validation prompt builders/batching, +and validator tests. -## Stage 4: Verify Failure, Credential, and Inspection Boundaries +1. Migrate spoken_form_plausibility, meaning_reversal_review, editorial_review, + grammar_review, and spoken_word_review to prompt IDs/variables and the common + structured boundary. Keep deterministic validators independent of PromptKit. +2. Preserve batching by `validation.max_prompt_tokens`, proposal preview, stable + proposal indexes, exact decision cardinality, and unknown/duplicate index checks. + Keep the current domain token-estimation method; do not add a second template + renderer or treat this budget as a provider output-token limit. +3. Apply the failure mapping: exhausted repairs reject/warn the batch; schema-valid + bad cardinality or negative assessments reject without semantic retry. Provider, + malformed HTTP, deadline, and admission failures propagate as runtime errors. + Update any old tests treating malformed provider envelopes as warning-only. +4. Remove redundant validator client/model configuration types and keep execution + class distinctions. Update validator reference/internal documentation. -**Status:** Complete (2026-09-24). **Depends on:** Stage 3. Focused offline tests -cover ConfigKit overlay errors, backend validation, credential replacement and -null handling, provenance, inspection, and redaction. Nested `go test -race ./...` -and `go vet ./...` passed on linux/amd64 with Go 1.26.8. +**Acceptance and validation:** preserve existing batching no-drop/no-duplicate, +domain reason, glossary protection, and cardinality tests. Cover all five prompt +mappings, positive repair before domain validation, exhausted repair, and runtime +failure propagation through a representative validator. Run validator tests; +full runner behavior is Stage 8. -Extend focused tests in the prototype; keep them offline and independent of host -configuration and credentials. Exercise the consumer's assembled loader/runtime -rather than duplicating ConfigKit's exhaustive parser or merge tests. +## Stage 8: Connect the Runner and Remove the Old Scheduling Hierarchy -Required evidence: +**Status:** Pending. **Depends on:** Stages 2 and 4–7. +**Owns:** CLI process assembly, `internal/framework/runner`, old scheduler contracts, +and runner/CLI integration tests. -- Missing and invalid overlay selections fail with ConfigKit classifications. - Representative attempts to replace the schema version or introduce ConfigKit - directives fail. Do not create application fallbacks for those failures. -- An overlay can change an ordinary non-profile setting, such as a backend limit, - and the effective setting still undergoes application validation. This proves - overlays have not been accidentally restricted to profile selection. -- A replaced credential reference resolves only its surviving replacement; an - unresolved displaced reference does not make the load fail. A surviving missing - or empty secret reference still fails. Explicit null retains unauthenticated - backend semantics. Use synthetic environment names and values under test control. -- Switching role profiles alone does not remove other credential references still - present in configuration. Document this distinction and test the relevant case; - do not bypass ConfigKit validation for unselected backends as an incidental fix. -- `WriteEffective` and relevant explanation/metadata APIs identify overlay origin; - a CLI replacement identifies CLI origin. Inspect selected PromptKit profiles from - that same loaded result and verify model/backend agreement with runtime selection. -- Synthetic credential values do not appear in effective configuration, explanations, - profile inspection, observations, or surfaced errors exercised by these scenarios. - Confirm the controlled transport receives the effective synthetic credential. +1. Pass the shared engine's two role clients into the runner. Remove global and + role schedulers, scheduler interfaces/fields, legacy concurrency configuration, + and measured scheduler wrappers. PromptKit backend admission is the only LLM + service-capacity mechanism. Do not remove unrelated deterministic work or alter + module sequencing merely because the scheduler disappears. +2. Preserve concurrent section work, deterministic collection/indexing, ordered + module application, validator execution classes, and cancellation of remaining + work on runtime failure. Capacity rejection fails the run as selected; do not + silently retry or add another queue to conceal it. +3. Wire final loaded domain settings, role clients, and diagnostics context into + normal processing. No old-runtime branch or interim gate remains. Keep existing + transcript encoding, output channels, partial domain results, and artifact + retention behavior. Ensure terminal failures cannot emit a purportedly complete + corrected transcript. +4. Update architecture policy for shared library dependencies and engine-local + backend admission; preserve the existing ownership, determinism, and redaction + invariants. Update pipeline/runtime internal guidance. -Maintain the existing retry/repair/capacity tests. Add only the overlay-specific -integration coverage needed to protect this boundary; do not repeat the complete -scheduler matrix for each fixture. +**Acceptance and validation:** controlled concurrent requests from both roles to +one backend never exceed its configured cap, including retries and repairs; +separate backend identities have independent capacity. Queued cancellation, +deadline and zero-wait-capacity rejection do not leak capacity; later work can +succeed. Existing deterministic runner/domain tests pass after deleting scheduler- +implementation assertions where the affected packages compile. Exercise a real +CLI-to-local-HTTP workflow with both roles when reporting dependencies permit; +otherwise record that check for Stage 12. Do not restore old utilization fields +or add placeholders merely to make this intermediate commit build. -Verification: +## Stage 9: Collect Execution Observations Safely -```sh -GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go test -race ./... -GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off go vet ./... -``` +**Status:** Pending. **Depends on:** Stage 8. +**Owns:** Audita-owned execution record types and collector in the LLM boundary, +runner attribution, and concurrency/observation tests. -Acceptance: all cases above have passing behavioral evidence, the existing suite -remains green, and concurrent observation collection remains race-free. Record -actual gaps instead of silently relaxing tests or claiming complete integration. +1. Translate PromptKit events into small Audita-owned records at the observer + boundary. Use the prototype's short mutex-protected collection pattern; callbacks + do no filesystem I/O, provider calls, or waiting on their own run. Freeze/copy + snapshots after active work has joined. +2. Bind observations to logical call, role (`proposal` or `validation`), module + instance, section, validator and batch where applicable, then retain library + run/backend/profile and attempt/generation identity. Keep provider attempts distinct from logical calls. +3. Record dispatch, queue/generation/retry/wall durations, outcome/failure category, + HTTP status, retry exhaustion, attempts/retries/repairs, and usage value/known/ + complete per field. Use terminal events once for provider totals; never add + attempt usage on top. `PrepareExecution` emits no events and `RunPrepared` timing + excludes preparation. Audita's surrounding call record therefore owns the logical + count and wall time, including preparation failures; do not invent library events. + A failure before dispatch has zero attempts and known-zero complete usage. + Preserve completed observations on nil-result failures and mark usage incomplete + when dispatched attempts lack measurements. Across calls, sum known values, + set known if any contributing dispatched usage is known, and set complete only + when all contributing dispatched usage is complete, independently per field. + An aggregate with no dispatched attempts is known-zero and complete. +4. Keep non-LLM runner wall/validator timing. Remove instrumentation that counts + scheduler permits as calls. Do not infer queue peaks, exact active concurrency, + billing totals, or timing fields the released event API does not provide. -## Stage 5: Record Overlay Findings and Executable Instructions +**Acceptance and validation:** adapter/collector tests prove known zero differs +from unknown, partial usage survives failed repairs, undispatched admissions are +not HTTP attempts, terminal totals are not doubled, and parallel callbacks retain +attribution safely. Run targeted race tests for the affected LLM/runner packages. +No raw provider messages, URLs, keys, prompts, or payloads enter observation records. -**Status:** Complete (2026-09-24). **Depends on:** Stage 4. Dated v0.4.0 evidence, -test coverage, commands, limitations, README guidance, and roadmap links updated; -affected links and `git diff --check` verified. +## Stage 10: Update Reports and Diagnostic Artifacts -This stage changes documentation, not production code. Read the new tests/results, -then update the existing owners rather than creating another parallel design: +**Status:** Pending. **Depends on:** Stage 9. +**Owns:** `internal/core/reporting`, `internal/core/diagnostics`, +`internal/framework/processreport`, runtime diagnostic writers, and CLI artifact wiring. -- Append a dated v0.4.0/overlay result section to `library-boundary-findings.md`. - Record commands, toolchain/platform, outcomes, test-to-requirement coverage, - limitations, and any unresolved gaps. Keep original v0.3.1 evidence identifiable. -- Update the prototype README to match the actual executable and fixtures; check - every documented command. Explain the separate ConfigKit overlay and PromptKit - profile directories and the credential behavior proved in Stage 4. -- Record the overlay implementation as verified in `library-boundary.md` and link - the new evidence from `library-adoption.md`. The next unit still has its Stage 6 - planning deliverable outstanding; keep production migration pending. +1. Bump the process report version to `v2`; keep the existing transcript/ledger + structures. Add `llm_summary` with `logical_calls`, `provider_attempts`, + `transient_retries`, `structural_repairs`, and `usage`. A logical call counts + once even when it fails before dispatch. Usage has `prompt`, `completion`, + `total`, `cached`, and `cache_write`, each `{value, known, complete}`. + Omit the whole summary only when execution observations are unavailable; a + successfully initialized run with no LLM calls has known-zero complete totals. +2. Replace obsolete utilization concurrency/pool fields with `backends`, keyed by + backend ID, containing configured `concurrency`/`queue_capacity` and the same + execution summary plus `capacity_wait_ms`, `generation_ms`, `retry_delay_ms`, + and `logical_wall_ms`. Sum library terminal timings once per call for capacity, + generation, and retry time; use Audita's whole-call elapsed time for logical wall + time. Attribute pre-dispatch failures to the already selected role backend. + Durations can overlap and must not be presented as a decomposition of elapsed + process time. Preserve domain module/validator timing and run wall time. + Existing role/module call counts now count logical calls; remove their obsolete + scheduler-wait and transport-execution fields rather than assign misleading new + meanings. Remove estimated active-concurrency fields throughout utilization. +3. Write `llm-execution.json` as `{calls: [...], events: [...]}` in the run directory. + Calls contain Audita logical identity, attribution, elapsed milliseconds, final + classification, summary, and available prompt/profile/model metadata. Events + contain the safe Stage 9 records, including attempt and terminal records, joined + by call ID. A preparation failure has a call record and no fabricated events. + Use millisecond integer durations in serialized records; absent attempt/section/ + batch identity is omitted, not encoded as a misleading zero. Add optional + `llm_execution_path` and `selected_profiles_path` to diagnostic path metadata. + Write `selected-profiles.yaml` from the safe view and use ConfigKit YAML for + the effective snapshot (`effective-config.redacted.yaml`). Keep invocation + overlay selection and recorded origins in safe invocation/config diagnostics. + Adjust centralized filename constants and readers together; do not emit old and + new snapshots in parallel. +4. Preserve redacted request/response/error diagnostic content from prepared + details and available final results, including prompt definition/rendered hashes + and selected profile/model. Do not promise legacy hash equality or complete + provider payloads for every retry. Use the separate observation artifact for retry/repair + attempts. Redact all resolved backend secrets, including unselected ones, at + every serialization/error boundary; observation records remain payload-free. +5. Update output-file, operations, and diagnostics/reporting contracts. Preserve + failure report/ledger behavior and the distinction between primary execution + failures and secondary diagnostic write errors. Do not change retention policy. -Acceptance: the README commands are reproducible, the new overlay evidence is -clearly separated from historical results, and each Stage 2–4 requirement maps -to a passing test or observed demonstration result. Verify affected links and -`git diff --check`. Do not repeat unchanged code checks solely for this commit. +**Acceptance and validation:** report/diagnostic tests check v2 mapping, summary +semantics, per-backend grouping, safe serialization, nil-result partial failures, +and existing deterministic ledger order. A CLI failure fixture echoes a synthetic +secret in a provider error; stdout/stderr/report and retained artifacts must not +leak it. Inspect known-zero and unavailable summaries structurally, not by complete +JSON golden dumps. Reuse Stage 9 evidence rather than retest every event sequence. -## Stage 6: Prepare the Production Migration Plan and Close the Unit +## Stage 11: Finish Examples, Migration Guidance, and Removal -**Status:** Complete (2026-09-24). **Depends on:** Stage 5. The ordered ConfigKit -production handoff records bounded packages, runtime integration, tests, -dependencies, and acceptance criteria. Prototype evidence and scope audit passed; -links and `git diff --check` verified. +**Status:** Pending. **Depends on:** Stages 1–10. +**Owns:** maintained examples, affected canonical documentation, obsolete code/tests, +prototype removal, and final dependency cleanup. -Expand `configkit.md` into a concrete, ordered production migration plan using -the existing inventory and verified overlay evidence. Follow the development -reading guide for configuration/CLI and LLM runtime work; use graph-first discovery -to verify affected production components and their tests. This is a planning stage: -do not implement the production migration or overwrite this execution plan with -new production stages. Keep future work packages explicitly pending. +1. Replace maintained config examples with schema 2, complete PromptKit profile + files, and `overlays/testing.yml` selecting both roles. Use one shared backend + where roles share a service; provide secret-reference and unauthenticated local + examples without real secrets. Explain that operators choose real models and + testing cost/quality is not guaranteed. Validate examples offline with controlled + synthetic environment values; no service is required for profile inspection. +2. Write `docs/migrations/v2.md` as the one migration guide. Cover every old file/ + flag/environment item in the inventory, durations, explicit empties/null, + credentials, profiles/overlays, inspection views, finite capacity, retry/repair/ + deadline semantics, and changed report/diagnostic shapes. Do not add conversion + tooling or compatibility aliases. Link current contracts for complete references. +3. Finish README prerequisites/examples and canonical CLI/config, integration, + internal component, and architecture references. Audit build/CI toolchain images + for Go 1.25.5 or newer; keep only required Linux amd64/arm64 targets and best-effort + macOS arm64. Do not rewrite immutable historical release notes. +4. Remove `examples/library-adoption` once its unique useful scenarios live in + production tests. Remove the nested dependency on old Audita and duplicate + assets/fixtures. Update historical roadmap links to recorded evidence or Git + history rather than leaving broken links to deleted files. +5. Remove superseded loaders, client, renderers, schedulers, aliases, and tests + asserting retired behavior. Tidy/verify the root dependency graph and confirm + no local replacements. Keep yaml.v3 if glossary parsing still uses it. Mark the + retry-count follow-up in `runtime-behavior.md` resolved without including its + unrelated retention/timestamp proposals. -The production handoff must specify bounded work packages, dependencies, affected -components, tests, and acceptance criteria for: +**Acceptance and validation:** maintained examples validate and all guide links +resolve. Search for old CLI/env inputs, loader/client/scheduler names and prototype +references; remaining mentions must be intentional migration/history material. +Review deleted tests for lost domain/concurrency/redaction protection. Run affected +example/config/subprocess checks and `git diff --check`; final full-suite execution +belongs to Stage 12. -- The full version-2 configuration definition: existing domain fields, defaults, - presence/null/zero behavior, role inheritance, overlays, secrets, and validation. -- File discovery and invocation parsing; `--overlay` behavior shared by process, - config validation, and effective inspection; explicit-setting override mapping; - removal of legacy environment overlays, aliases, and direct-key flags. -- Profile loading/inspection and configuration snapshots, including relative-path - bases, secret redaction, provenance, and clear separation of ConfigKit settings - from PromptKit execution controls. -- Direct integration of the final configuration with PromptKit execution. Specify - the final behavior and tests for each setting; do not preserve the old runtime - through temporary adapters or add interim processing gates. -- Removal of superseded configuration machinery, retained domain checks, maintained - examples, canonical documentation updates, and the combined migration guide. -- Removal of superseded LLM machinery and release follow-up. Preserve - the agreed ordering and defer release compliance/publication to the final unit. +## Stage 12: Verify the Combined Application and Close the Refactor -Resolve final integration decisions against the evidence. Record genuine open -questions without creating scaffolding to support intermediate commits. +**Status:** Pending. **Depends on:** Stages 1–11. +**Owns:** final integration tests, final defect fixes within this scope, and status/evidence. -Final audit: map every acceptance criterion in the prototype extension to actual -files and command results. Verify new relative links/anchors, run `git diff --check`, -and review that no root runtime/dependency or upstream files changed. Reuse passing -checks for unchanged code; rerun the affected tests if this stage changes code or -fixtures to correct a finding. A root build/test run is not a substitute for the -nested-module checks, and this documentation stage adds no release gate. +1. Complete representative production subprocess workflows against controlled HTTP: + base selection, testing overlay, explicit profile override, and a retained + domain override. Observe actual outbound models for both roles, an accepted + correction after repair, a warning-only domain/repair rejection, and a runtime + failure with safe partial diagnostics. Cover an empty module pipeline and a + repeated-module sequence without duplicating all adapter scenarios. +2. Run root `go test ./...`, `go build -o ./bin/audita ./cmd/audita`, and focused race + checks for the final concurrent adapter/runner/collector changes. Verify both + Linux targets compile using `bash scripts/build-release.sh v2.0.0 `; + this names local build outputs and creates no tag or publication. No native Mac, paid-provider, or CI-completion gate. + If the supported minimum toolchain was not exercised earlier, verify the root + suite with installed Go 1.25.5; do not silently claim this from a newer compiler. +3. Fix actual failures and rerun affected checks; repeat the full suite when those + fixes materially change assembled behavior. Do not rerun unchanged tests solely + to mark multiple planning steps complete. Verify maintained documentation links, + examples, and whitespace once for the final content. +4. Record concise final evidence and known limits here, mark the production feature + roadmaps implemented, and commit the completed refactor. Both libraries must + own their selected responsibilities, processing must work, and no temporary + or legacy compatibility machinery may remain. +5. Hand off release preparation separately: proposed Audita v2.0.0, one migration + guide, actual pinned production dependency/license inventory, and the remaining + LGPL notice/source/rebuild-distribution review. Do not claim license compliance + from prototype tests or publish a release. The later release unit uses the + existing minimal automated procedure and ends on a successful source/tag push, + without waiting for CI. -Mark this prototype-extension/planning unit complete in `library-boundary.md` only -after the audit passes, and link the production handoff from `library-adoption.md`. -Production implementation remains pending. +**Acceptance:** the final production application and required Linux builds pass; +configuration selection, both LLM roles, domain outcomes, cancellation/capacity, +secret handling, reports and retained artifacts match the selected contract. +Recorded checks describe what actually ran. The workspace is committed and clean. +Release-specific compliance and publication are explicitly still pending. -Acceptance: a reviewer can reproduce base/testing/override selection offline, -trace each new claim to evidence, and hand the production ConfigKit work to a coding -agent in bounded units. Commit the completed documentation, report remaining -limitations and working-tree state, and stop at the end of this plan. +## Open Questions + +None block this implementation plan. Configuration, inspection, runtime boundaries, +report representation, and stage ownership are selected above. Release distribution +compliance remains outside this implementation sprint and must be resolved against +the final production dependency graph before publishing binaries; it is not a +reason to build compatibility scaffolding or delay the refactor. diff --git a/docs/roadmap/library-adoption.md b/docs/roadmap/library-adoption.md index 5c4eeb9..7136c13 100644 --- a/docs/roadmap/library-adoption.md +++ b/docs/roadmap/library-adoption.md @@ -2,179 +2,128 @@ **Last updated:** 2026-09-24 -**Status:** Shared boundary and offline prototype completed; production migrations -remain planned. The [v0.4.0 overlay evidence](library-boundary-findings.md#selected-overlay-extension-2026-09-24) -extends the completed prototype, and the [ordered ConfigKit handoff](configkit.md#ordered-production-migration-handoff) -specifies the next implementation work. This document authorizes no release -publication and sets no schedule. +**Status:** Final target selected; production implementation pending. The shared +boundary and offline prototype, including ConfigKit v0.4.0 overlays, are verified. +[Implementation stages](implementation.md) are owned by the implementation plan. +This roadmap defines the end state and user intent; it authorizes no publication. ## Goal and Ownership Adopt ConfigKit and PromptKit to align Audita with the application suite's shared -configuration and LLM execution conventions. Remove overlapping application -infrastructure while keeping transcript-correction responsibilities in Audita. +configuration and LLM execution conventions. Consistency is the primary benefit; +removing overlapping application infrastructure is a secondary benefit. Preserve +Audita's transcript-correction capabilities and domain protections. -| Owner | Intended responsibilities | +| Owner | Responsibilities | | --- | --- | -| ConfigKit | Application configuration loading, presence/default semantics, explicit overrides, recorded resolution, secret references, and safe configuration inspection | -| PromptKit | Prompt/profile formats and rendering, provider calls, structural output validation, and shared execution capabilities supplied by the selected release | -| Audita | Configuration field meanings, CLI parsing and file selection, transcript normalization/chunking, module ordering, domain validators, correction application, reports, and diagnostic artifact lifecycle | +| ConfigKit | Strict application configuration loading, presence/default semantics, explicit overrides, recorded resolution, secret references, and safe inspection | +| PromptKit | Prompt/profile formats and rendering, provider calls, structural output validation and repair, transient retries, engine-local backend admission, and execution observations | +| Audita | Setting meanings, file/CLI selection, role/profile association, normalization/chunking, ordered modules, domain validation, correction application, reports, diagnostic redaction, and artifact lifecycle | -This roadmap owns sequencing across the two adoptions. The -[ConfigKit roadmap](configkit.md) owns the configuration migration's proposed scope. -PromptKit v0.11.0 is the reviewed baseline for the adoption prototype. It supplies -transient HTTP retries, execution observations, typed failure categories, an offline -proposal/assessment example, and LGPL-3.0-or-later licensing. The earlier upstream -consumer requests are no longer outstanding prerequisites on those points. -Audita has not yet adopted either library; recheck the selected releases during -detailed planning. +The [configuration roadmap](configkit.md) owns configuration and inspection policy. +The [boundary findings](library-boundary-findings.md) own the selected defaults, +full migration inventory, failure mapping, and prototype evidence. The selected +production versions are ConfigKit v0.4.0 and PromptKit v0.11.0, with Go 1.25.5 as +the minimum. Use published modules, never sibling checkouts or local replacements. +The existing YAML dependency remains while glossary parsing needs it. ## Migration Posture -Treat adoption as an intentional breaking transition. Preserve intended domain -capabilities, not legacy configuration files, flags, environment variables, or -configuration-inspection formats. Supply one migration guide rather than parallel -legacy loaders, compatibility aliases, or permanent translation layers. +This is an intentional breaking transition targeting Audita v2.0.0. No backward +compatibility is required for existing installations, config files, CLI flags, +environment overlays, or inspection output. Provide one migration guide, without +legacy loaders, aliases, dual modes, or automatic conversion tooling. -Plan to ship both adoptions together as Audita v2.0.0, without an intermediate -public release. Implementation should still consist of coherent, reviewable units. -Revisit the final release scope and sequencing alongside other large planned -changes, especially changes to module or pipeline configuration. +Both libraries belong to one refactor. Intermediate commits need not build or run; +there is no supported intermediate product state. No temporary adapters, compatibility +shims, or processing gates are needed. The complete application must work and pass +its checks before the refactor is finished. Release compliance and publication +come last, with no intermediate public release. -## Proposed Sequence +## Configuration and Execution Target -The [shared-boundary and prototype roadmap](library-boundary.md) records the completed -unit covering stages 1 and 2 below. The [findings](library-boundary-findings.md) -select its target contract and identify the next ConfigKit implementation work. +ConfigKit selected overlays enable a testing configuration that chooses cheaper or +faster models. The example selects both proposal and validation profiles. ConfigKit +application overlays and PromptKit execution profiles remain separate concepts. +Complete operator-maintained profiles own model/generation controls; embedded +application prompts and schemas are not operator-selected files. No catalog wiring, +profile-inheritance feature, ConfigKit imports, or secret-directory bootstrap is +part of this migration. -### 1. Design the Shared Configuration Boundary +Capacity controls serve two needs: respecting a local service's hard simultaneous- +query limit and bounding concurrent paid API activity. One finite limit per backend +is shared across both roles and every initial, retry, and repair attempt. Limits +are engine-local, not coordinated across processes. No additional global or +per-role limiter or fine-grained scheduling policy is required. Profiles must use +registered backends and ConfigKit-resolved credentials; endpoint-only bypass and +competing profile credential lookup are excluded. -Define the target settings model for both libraries before implementing either -migration. Decide how Audita selects proposal and validation execution profiles, -supplies credentials, selects transient-retry and structural-repair budgets, and -assigns per-backend concurrency limits. Give each setting one authoritative home; -avoid duplicate model, endpoint, credential, or timeout definitions with competing -precedence. +Structural repair is enabled by default for both roles because additional attempts +to obtain valid output are desired. Transient retries and structural repairs have +separate budgets; explicit zero disables the respective behavior. Each generation, +including a repair, receives its own transient-retry budget. The Audita logical-call +deadline covers preparation, waiting, backoff, all generations, and repairs; profile +HTTP timeouts apply per attempt. Exact defaults and bounds are in the +[selected contract](library-boundary-findings.md#configuration-and-profile-contract). +No Audita outer retry loop restores the old broader retry policy. -Keep ConfigKit application configuration distinct from PromptKit execution profiles. -ConfigKit v0.4.0 calls its optional configuration layers overlays. Enable selected -overlays for testing as specified in the [ConfigKit roadmap](configkit.md#selected-overlays-for-testing); -these select application settings, including PromptKit execution profiles. -Select an explicit profile-ownership model; complete Audita-owned execution profiles are the initial recommendation, with -catalog-driven selection a deliberate alternative. +Audita retains all four correction modules, five LLM-validator prompts, +deterministic validators, confidence/applicability checks, glossary protection, +validation batching/cardinality, deterministic proposal and ledger ordering, +transcript encoders, and artifact retention. PromptKit-valid JSON still passes +Audita's domain checks. Repair exhaustion, provider failure, and domain rejection +follow the [selected failure table](library-boundary-findings.md#execution-and-failure-mapping). +Retention and transcript timestamp changes remain separate work; this migration +resolves the [retry-count follow-up](runtime-behavior.md). -Capacity needs are simple: protect local backends from exceeding their available -simultaneous-query capacity and give paid API backends a reasonable finite cap. -Proposal and validation calls using the same backend share its limit. Separate -proposal/validation budgets and additional fine-grained Audita scheduling controls -are not required. Account for the engine-local scope of PromptKit's existing limits. +## Reporting and Diagnostic Target -### 2. Prove the PromptKit Boundary +Keep stable Audita-owned serialized records; do not expose raw library structs. +Use process report schema `v2` with the existing domain outcomes and a new +`llm_summary` separating logical calls, dispatched provider attempts, transient +retries, structural repairs, and usage. Transcript schema versions remain unchanged. +The implementation plan specifies the concrete new fields and artifact wiring. -Before finalizing the new application configuration schema, exercise one proposal -call and one LLM-validator call through PromptKit using controlled offline responses. -Verify profile selection, credentials, embedded prompts and schemas, shared backend -capacity, failed output validation, cancellation, and diagnostic mapping. +Usage distinguishes known zero, unknown, and incomplete totals. Preserve observations +from failed calls even when no final result exists. Count terminal totals once; +attempt observations provide diagnosis without being added to those totals again. +This is operational evidence, not a complete billing ledger. -Enable structural repair from the outset for proposal and LLM-validator outputs. -Additional attempts to obtain valid output are desired behavior, not a deferred -optimization. Select explicit positive repair budgets during detailed planning and -exercise invalid output that becomes valid after repair, as well as exhausted -repair budgets. +Replace global/role scheduler utilization with per-backend configured capacity and +observed queue/generation/retry/logical-call timing. Do not fabricate measurements +that the event API does not provide. Keep domain timing and deterministic correction +ledger ordering. Record prompt definition/rendered hashes and selected profile/model +without promising the old hash values. Callbacks are short, concurrency-safe, and +free of blocking I/O or waits on their own execution. -Keep transient HTTP retries and structural repair distinct. With the reviewed -library, each generation, including a repair, has its own transient-retry budget; -plan and document the combined request allowance and overall deadline. Adopt -PromptKit's explicit zero-disables-retries semantics instead of Audita's legacy -zero-as-default behavior. The library's narrower transient retry policy does not -retry transport failures or malformed provider responses; do not restore the old -policy through an overlapping application retry loop. +Effective configuration and selected-profile snapshots remain distinct safe views. +Secret values must not escape through inspection, logs, reports, runtime errors, +or retained diagnostic files. Observation records exclude payloads, URLs, raw +provider messages, prompts, and credentials. Existing artifact lifecycle and +partial-failure handling remain Audita responsibilities. -Verify that initial, retry, and repair calls share backend limits and that their -observations reach Audita without double-counting usage. Audita's domain acceptance -checks remain necessary after structural repair succeeds; valid JSON does not imply -an acceptable correction or validator decision. +## Completion Criteria and Release Boundary -Use the prototype to resolve genuine boundary gaps and inform configuration design. -It is evidence for the migration, not a permanent alternate runtime. Avoid finalizing -ConfigKit-facing settings around the old LLM runtime only to redesign them later. +- Both libraries own their selected responsibilities in the production application. + Superseded loaders, transport, renderers, schedulers, aliases, and the isolated + prototype are gone once its useful evidence is represented in production tests. +- Offline tests prove configuration-to-execution routing for both roles, selected + overlays and explicit overrides, shared backend capacity, retry/repair outcomes, + cancellation/deadlines, domain decisions, redaction, and truthful partial reports. + Tests exercise real internal collaborators and controlled HTTP boundaries. +- Maintained examples and canonical docs describe the implemented final model. + One migration guide covers files, flags, environment inputs, profiles, credentials, + overlays, inspection, and changed runtime/report/diagnostic semantics. +- The assembled application passes its required checks; Linux amd64 and arm64 build. + macOS arm64 remains best effort. No live-provider or CI-success gate is added. -### 3. Complete ConfigKit Adoption +PromptKit v0.11.0's reviewed retry, observation, and capacity APIs remove the earlier +upstream feature prerequisites; no upstream edit is required for adoption. The +prototype proved representative behavior, not release compliance or model quality. -Implement the agreed application configuration contract following the -[ConfigKit roadmap](configkit.md). Retire legacy environment overlays and obsolete -CLI/configuration conventions, update maintained examples and canonical references, -and begin the combined migration guide. Include the selected testing-overlay path -and its offline integration evidence from the ConfigKit roadmap. - -ConfigKit and PromptKit adoption form one refactor. These sections describe -implementation order, not separately runnable milestones. Intermediate commits -need not build or run. Build directly toward the final integration without -temporary adapters, compatibility shims, or an interim processing gate; validate -the complete application before declaring the refactor finished. - -### 4. Complete PromptKit Adoption - -Move prompt rendering, execution profiles, provider calls, and structural output -validation onto the selected PromptKit release. Remove overlapping transport, -prompt-loading, and scheduling machinery once the replacement is verified. - -Retain Audita's transcript pipeline, semantic and deterministic validators, -correction application, reports, ledger, and artifact retention. Map PromptKit -results, errors, metadata, and execution observations into those responsibilities. -Connect the new configuration directly to this execution path and remove the -superseded configuration and runtime machinery. - -### 5. Validate and Release the Combined Migration - -Validate the assembled configuration-to-execution workflows, including local and -API backend limits, successful and exhausted repairs, transient retries, -failure/cancellation behavior, credential handling, diagnostics, and domain outcomes. -Use deterministic tests and controlled endpoints rather than paid calls as acceptance -gates. Update documentation and examples together with -implemented behavior and remove obsolete contracts and code. - -Complete one migration guide covering files, CLI, environment settings, credentials, -execution profiles, inspection, and changed diagnostics. Release the combined -breaking transition under the established [release procedure](../release.md), with -no CI waiting added to release completion. - -## Reviewed Upstream Readiness - -The v0.11.0 review covered the relevant implementation, the full test suite, -targeted race tests, and the offline proposal/assessment example; those checks -passed. Existing per-backend controls cover the requested capacity model, including -retry and repair interactions. No further upstream feature request is currently a -prerequisite to the prototype. Keep the planned adoption sequence unchanged. - -Remaining integration work includes selecting a total-operation deadline alongside -per-attempt timeouts, using shared backend identities with finite limits, and mapping -PromptKit observations to Audita-owned diagnostic records. Observation callbacks are -synchronous and may overlap across runs; collection must be short and safe for -concurrent use. Distinguish unknown or incomplete usage from known zero and do not treat -observations as a complete billing ledger or a stable serialized report format. - -The licensing change has shipped as LGPL-3.0-or-later. Account for applicable -notices and source/rebuild distribution requirements in Audita's release design; -relicensing no longer needs to be requested upstream. Recheck released APIs, -toolchain requirements, and dependency versions at implementation time. Pin -published library releases; production builds must not depend on sibling checkouts -or local module replacements. - -If the prototype reveals a genuine remaining gap, evaluate it explicitly rather -than quietly recreate shared library functionality as permanent Audita infrastructure. - -## Coordination and Open Decisions - -- Use the selected configuration/profile boundary, defaults, credential flow, and - retry/repair/deadline contract in the [findings](library-boundary-findings.md) - when detailing the ConfigKit production migration. -- Implement the outcome and observation mapping recorded in the findings, retaining - Audita domain validation and diagnostic redaction. -- Resolve the [runtime retry-count follow-up](runtime-behavior.md) through this migration. - Retention and transcript timestamp questions are not automatically included. -- Revisit ordering when other major changes are defined. Prefer designing shared - module/pipeline settings once, while keeping unrelated work independent. -- Flesh out task-level scope, tests, and acceptance criteria before implementation. - Completion requires the intended ownership split, removed transitional machinery, - verified assembled behavior, and one coherent migration path for users. +The selected PromptKit release uses LGPL-3.0-or-later. A later release unit must +review the actual production dependency graph and distribution requirements, +including notices and source/rebuild materials, before binary publication. This +implementation plan does not claim that review is complete. Release publication +uses the existing [procedure](../release.md) and completes on successful source/tag +push, without waiting for CI or artifact delivery. diff --git a/docs/roadmap/library-boundary-findings.md b/docs/roadmap/library-boundary-findings.md index fae2ea1..7d5de5f 100644 --- a/docs/roadmap/library-boundary-findings.md +++ b/docs/roadmap/library-boundary-findings.md @@ -311,51 +311,23 @@ claims of completion for the prototype. The new profile/credential/capacity and deadline choices are explicit above; no unresolved upstream requirement prevents starting the ConfigKit implementation -plan. Production report field names and final runtime wiring follow the outcome -and observation contract here. The coordinated refactor targets both libraries -directly; intermediate commits do not require a working legacy runtime. +plan. Production report field names and final runtime wiring are selected in the +[implementation plan](implementation.md), following the outcome and observation +contract here. The coordinated refactor targets both libraries directly; intermediate commits do not require a working legacy runtime. -### Component-Level Next Unit +### Production Implementation Reference -Implement ConfigKit adoption first, against this target model: +The [implementation plan](implementation.md) owns ordered production work, +component changes, tests, and acceptance criteria. The [combined roadmap](library-adoption.md) +and [configuration roadmap](configkit.md) own the final target. This evidence does +not require an interim runtime bridge or separately runnable configuration migration. -| Component | Migration action | -| --- | --- | -| `internal/core/config` | Replace file/env merge and custom duration parsing with ConfigKit definitions, presence-aware resolution, domain validation, and explicit override paths. Retain domain defaults and validation rules identified in the inventory. | -| `internal/cli` | Separate invocation paths from persistent settings; select one file; parse only supplied setting flags into overrides; remove legacy LLM/role-budget/direct-key flags and all environment overlays. Assemble/inspect selected profiles for validation. | -| CLI config inspection and diagnostics snapshot | Use ConfigKit redacted output plus Audita-owned selected-profile summary; update command tests and diagnostic snapshot consumers. | -| Configuration tests/examples | Replace v1 discovery/overlay/alias expectations with v2 presence, inheritance, secret, profile, and override contracts; preserve domain validation tests. Add actual production discovery and CLI tests not exercised by this prototype. | - -The PromptKit portion of the refactor replaces `internal/framework/llm` provider execution -and scheduler composition, `internal/prompts` parsing/rendering, and duplicated -structural-output validation. Keep a small Audita adapter behind the existing -structured-call boundary where that contract remains sufficient; revise its return -metadata only as needed for repair and observation outcomes. Avoid library types -in module proposals, domain validators, transcript schemas, and correction ledgers. - -Retain `internal/core/schema`, normalization/chunking/output encoding, -`internal/framework/proposals`, deterministic validators, module identity/order, -LLM-validator domain batching and cardinality, runner ordering/apply logic, report -assembly, and artifact lifecycle. Update their integration wiring rather than -rewriting those algorithms. Move all nine prompt definitions and schema references, -checking each module/validator's variables and helpers; this prototype proves only -one pair. Migrate actual diagnostic redaction and error classification, then remove -the old transport/schedulers/prompt loader. No temporary runtime bridge is needed. - -Documentation work during the respective implementation stages: update -`docs/config.md`, `docs/cli.md`, examples, README prerequisites, internal config/prompt/ -LLM/runner guidance, and output/LLM/subprocess integration contracts. Create one v2 -migration guide covering file shape, duration/empty-list changes, removed environment -and CLI inputs, backend/profile/secret setup, effective inspection, retry/repair -behavior, and changed usage/concurrency diagnostics. Revise architecture policy's -existing scheduler language when production uses PromptKit; the current policy -continues to describe the current runtime. Retention and timestamp fixes remain -separate roadmap work. - -The combined release remains last, with no intermediate publication. Raise the -production Go minimum when dependencies land; review CI/build images and supported -Linux amd64/arm64 builds then. Prototype execution on Linux amd64 is not proof of -arm64 release readiness or live model output quality. +Retained domain scope includes transcript/glossary schemas, normalization/chunking, +output encoding, proposals and apply-time checks, deterministic validators, ordered +modules, LLM-validator batching/cardinality, ledger ordering, and artifact lifecycle. +The production integration replaces loading, provider execution, prompt rendering, +and scheduler composition while preserving those responsibilities. Retention and +timestamp changes remain separate work. ### Distribution Follow-Up @@ -416,5 +388,5 @@ The tests use a controlled built-in HTTP transport and synthetic credentials; there were no live provider calls. The prototype still covers one representative proposal/assessment pair, not all production modules, prompts, CLI paths, or diagnostic artifacts. Root `go test ./...` does not run this nested module. No -upstream API gap appeared in this extension. The [ordered production handoff](configkit.md#ordered-production-migration-handoff) +upstream API gap appeared in this extension. The [production implementation plan](implementation.md) keeps implementation and release pending. diff --git a/docs/roadmap/library-boundary.md b/docs/roadmap/library-boundary.md index 67d611d..cfaaf95 100644 --- a/docs/roadmap/library-boundary.md +++ b/docs/roadmap/library-boundary.md @@ -11,9 +11,9 @@ is isolated from production. No publication or upstream changes were made. ## Next Unit: Testing Overlays and Production Migration Plan -**Status:** Complete. The [ordered production handoff](configkit.md#ordered-production-migration-handoff) -is ready for subsequent work; [implementation.md](implementation.md) records this -unit's checks. The original work packages below remain +**Status:** Complete. The [production implementation plan](implementation.md) +owns subsequent work; the [overlay findings](library-boundary-findings.md#selected-overlay-extension-2026-09-24) +record this unit's checks. The original work packages below remain historical scope for the completed first unit. Extend the existing prototype to prove the [selected-overlay contract](configkit.md#selected-overlays-for-testing) @@ -68,7 +68,7 @@ their existing names and meaning. Define and prove the boundary between Audita, ConfigKit, and PromptKit before replacing Audita's production configuration or LLM runtime. This unit covers the -shared design and representative prototype in stages 1 and 2 of the +shared design and representative prototype supporting the [coordinated adoption roadmap](library-adoption.md). The [ConfigKit roadmap](configkit.md) continues to own the broader configuration scope.