567 lines
37 KiB
Markdown
567 lines
37 KiB
Markdown
# Implementation Plan: Adopt ConfigKit and PromptKit
|
||
|
||
**Last updated:** 2026-09-24
|
||
|
||
**Status:** Ready for implementation. All stages below are pending. This replaces
|
||
the completed testing-overlay prototype plan; its results remain in the
|
||
[boundary findings](library-boundary-findings.md#selected-overlay-extension-2026-09-24).
|
||
|
||
## Scope and Working Instructions
|
||
|
||
Implement the final configuration and execution model in the
|
||
[combined feature roadmap](library-adoption.md) and [configuration roadmap](configkit.md).
|
||
The [selected boundary](library-boundary-findings.md#configuration-and-profile-contract),
|
||
[migration inventory](library-boundary-findings.md#migration-inventory), and
|
||
[failure mapping](library-boundary-findings.md#execution-and-failure-mapping)
|
||
are the detailed behavioral inputs. Do not repeat the prototype or reopen settled
|
||
choices without a concrete contradiction in the pinned libraries.
|
||
|
||
This plan is for a gpt-5.6-terra coding agent. Execute Stage 1 through Stage 12 in
|
||
order. Each stage is one bounded implementation prompt. When assigned the whole
|
||
plan, continue between stages without a permission round. When assigned one stage,
|
||
finish that stage's scope and report the next stage. Mark completed stages with a
|
||
short result and actual validation evidence; do not call a pending integration
|
||
check passed.
|
||
|
||
Read `AGENTS.md`, [development guidance](../development.md), and all documents under
|
||
`docs/policy/`. Use graph-first code discovery. Follow the
|
||
[agent](../policy/agents.md), [architecture](../policy/architecture.md),
|
||
[testing](../policy/testing.md), [documentation](../policy/documentation.md), and
|
||
[release](../policy/release.md) policies. The intentional dependency, configuration,
|
||
and scheduler changes here supersede the corresponding old implementation choices;
|
||
update canonical descriptions as the new behavior lands.
|
||
|
||
The stages divide one refactor, not supported intermediate releases. Intermediate
|
||
commits may fail to build or run. Do not write temporary adapters, compatibility
|
||
shims, duplicate execution paths, stubs returning invented successes, or an interim
|
||
processing gate. Remove superseded code when replacing it. Run focused checks
|
||
where the affected packages are testable; record checks deferred because later
|
||
stages must reconnect callers. Such a completed stage is still work in progress
|
||
on the overall refactor. Commit coherent stages with plain-English messages under
|
||
the standing policy. Do not spend a stage making an intermediate application work.
|
||
|
||
Keep tests deterministic, offline, and credential-independent. Reuse prototype
|
||
scenarios at their production boundary rather than copy the nested application or
|
||
retest all library internals. Preserve domain tests unless the selected contract
|
||
actually changes. Update each affected canonical reference alongside its owning
|
||
stage; Stage 11 finishes the cross-document migration guide and examples.
|
||
|
||
No upstream edits, paid calls, push, tag, publication, retention-policy change,
|
||
timestamp fix, or unrelated pipeline redesign is included. Release compliance
|
||
and publication remain a later release unit. Completion here means both libraries
|
||
are adopted, obsolete machinery is removed, and the final application is verified.
|
||
|
||
## Implementation Decisions
|
||
|
||
These decisions close the remaining application-design details; background and
|
||
user-facing policy belong to the feature roadmaps.
|
||
|
||
- Pin the published modules used by the verified prototype: ConfigKit v0.4.0 and
|
||
PromptKit v0.11.0. Set the root minimum to Go 1.25.5. Keep yaml.v3 while glossary
|
||
parsing uses it. No sibling replacements or dependency on Audita's own old release.
|
||
- Keep application loading in `internal/core/config`, embedded prompt definitions
|
||
in `internal/prompts`, schemas in `internal/framework/responseschema`, and the
|
||
PromptKit integration in `internal/framework/llm`. Use one engine per process
|
||
invocation, shared by both role-bound clients; neither modules nor validators
|
||
construct engines. The CLI composes these pieces. Avoid a generic service layer.
|
||
- Retain the `CompleteStructured(ctx, request, out)` boundary, using Audita-owned
|
||
prompt ID/version, string variables, schema identity, and diagnostic context.
|
||
Replace the old rendered-message/model request fields. Model/profile/credential
|
||
selection belongs to the role-bound adapter, not individual callers. Consolidate
|
||
the equivalent validator types onto the framework contract where needed, without
|
||
an import cycle. No PromptKit types in correction algorithms or ledger records.
|
||
- Store PromptKit definitions for all nine existing prompt IDs with their existing
|
||
`v1` selection. Replace the custom `hardening` template function with an
|
||
application-supplied `HardeningPolicy` string variable sourced from the one
|
||
embedded hardening fragment. The adapter supplies that reserved variable after
|
||
caller variables. PromptKit renders all messages; do not retain an Audita template
|
||
engine or introduce generated prompt files requiring a second build step.
|
||
- Use one shared configuration-input parser for `process`, `config validate`, and
|
||
`config print-effective`. All three accept `--config`, `--overlay`, and the
|
||
retained setting overrides in the inventory, including the two profile flags.
|
||
Only `process` accepts transcript/glossary/output/report paths. Only explicitly
|
||
visited setting flags become ConfigKit overrides. Do not add retry/capacity/key
|
||
flags beyond the agreed inventory.
|
||
- `config validate` requires an explicit nonempty config path. The other two
|
||
commands use the selected discovery order. All validate the resolved settings
|
||
and both selected profiles without provider calls, including an empty module
|
||
pipeline. Help succeeds before file discovery or secret/profile loading.
|
||
- `config print-effective` adds `--view settings|profiles|explain`, defaulting to
|
||
`settings`. Settings uses ConfigKit `WriteEffective` YAML. Profiles emits a safe
|
||
Audita-owned YAML summary for the proposal and validation roles. Explain emits
|
||
a YAML sequence of field path, source, and derivation records from ConfigKit's
|
||
recorded resolution, with no resolved secret values. Unknown views are usage
|
||
errors. This avoids mixing multiple formats into the default output or adding
|
||
another command tree. Profiles includes selected ID, backend, model, temperature,
|
||
max tokens, top-p, per-attempt timeout, service tier, and reasoning effort when
|
||
present; omit arbitrary extra provider parameters and credentials. It is a
|
||
selected-controls summary, not a dump of every profile input.
|
||
- Keep CLI exit classes 0/1/2 and transcript/report stream behavior. Invalid user
|
||
settings/profiles are configuration errors (2); broken embedded assets and
|
||
runtime/provider failures are runtime errors (1). Serialization/write failures
|
||
remain runtime errors. Do not classify malformed HTTP envelopes as repairable
|
||
model output.
|
||
- Publish process reports as `audita-process-report` schema `v2`. Preserve existing
|
||
domain fields and ledger ordering, add a top-level `llm_summary`, and replace
|
||
obsolete scheduler utilization fields as specified in Stage 10. Transcript
|
||
encoder versions stay unchanged. No legacy report mode is needed.
|
||
|
||
## Stage 1: Replace the Configuration Model and Loader
|
||
|
||
**Status:** Complete (2026-09-24). Replaced the version-1 loader with the ConfigKit
|
||
schema-2 model, selected overlays, typed secret references, final role inheritance,
|
||
and retained domain validation. Pinned both libraries and Go 1.25.5, and aligned
|
||
the release build image. `go test ./internal/core/config` and `git diff --check`
|
||
passed. `go test ./...` currently fails in old CLI, validator, and diagnostic
|
||
callers of removed fields; those connections are assigned to later stages.
|
||
**Depends on:** the completed prototype only.
|
||
**Owns:** root `go.mod`/`go.sum`, `internal/core/config`, and config tests.
|
||
|
||
1. Pin both modules and the Go minimum above. Align build/CI toolchain declarations
|
||
with that minimum; preserve the existing platform set and release gates.
|
||
2. Replace version-1 YAML/env merge and duration coercion with one ConfigKit
|
||
definition covering every retained field in the migration inventory. Use the
|
||
prototype's types/lifecycle as evidence, then add the full domain configuration.
|
||
Remove legacy environment loaders and alias resolution; do not keep two loaders.
|
||
3. Implement defaults, presence-aware backend defaults, final validation-profile
|
||
inheritance, and recorded derivations. Keep `chunking.target_sections` optional,
|
||
explicit empty module lists empty, and module order/repeats intact. Validate
|
||
domain constraints after overlay and explicit override resolution. Preserve
|
||
numeric normalization seconds; operation timeout is a duration string.
|
||
4. Use typed optional secret references. Resolve every surviving reference, including
|
||
unselected backends; missing/empty references fail, replacement resolves only the
|
||
replacement, and null permits unauthenticated use. Enable selected overlays;
|
||
leave imports and secret-directory bootstrap disabled.
|
||
5. Expose the ConfigKit result to CLI composition for effective output and provenance;
|
||
convert retained settings into existing domain inputs without transporting the
|
||
library result through correction code. Update `docs/config.md`'s implemented
|
||
loader/schema description, clearly marking unfinished CLI integration if needed.
|
||
|
||
**Acceptance and validation:** focused config tests cover strict version/fields,
|
||
selected defaults, omission/null/zero, final role inheritance, empty/repeated
|
||
modules, overlay/override precedence, secret replacement/null/missing, and retained
|
||
cross-field rules. Old schema fails with migration guidance. General environment
|
||
variables have no implicit effect. Run `go test ./internal/core/config` when this
|
||
package is independently buildable; record caller compile failures for later stages.
|
||
Do not duplicate ConfigKit's exhaustive YAML/merge tests.
|
||
|
||
## Stage 2: Replace CLI Configuration Selection and Overrides
|
||
|
||
**Status:** Complete (2026-09-24). Added one parser for all three commands,
|
||
strict file discovery, optional overlay selection, and visited-only setting
|
||
overrides; removed retired flags and legacy process flag extraction. Updated CLI
|
||
and configuration references. Isolated parser/loading tests passed with
|
||
`go test internal/cli/config_input.go internal/cli/config_input_test.go`, and
|
||
`git diff --check` passed. `go test ./internal/cli` remains unbuildable because
|
||
the old runtime, validators, and diagnostics still use removed configuration
|
||
fields; assembled command checks remain deferred to later stages.
|
||
**Depends on:** Stage 1.
|
||
**Owns:** `internal/cli/run.go`, `process_flags.go`, configuration command parsing,
|
||
and corresponding CLI/subprocess tests.
|
||
|
||
1. Implement the shared configuration-input parser described above. Preserve the
|
||
existing supported positional-argument arrangement for `process`, required
|
||
glossary, output/report behavior, and root/subcommand help. Remove obsolete
|
||
LLM/model/key/timeout/retry/concurrency flags and aliases outright.
|
||
2. Select exactly one file: explicit `--config`, then
|
||
`/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`. Missing system
|
||
candidates are skipped; selected inaccessible files and explicit empty/missing
|
||
paths fail. With no file, report a configuration error; do not select a provider.
|
||
Remove `AUDITA_CONFIG`. Test discovery using an injected path list/filesystem
|
||
seam rather than modifying the host's system configuration.
|
||
3. Pass invocation overlay selection to ConfigKit and only supplied setting flags
|
||
to its overrides. `--modules ''` must clear the pipeline; explicit numeric zero
|
||
reaches validation rather than being treated as absent. Relative profile bases
|
||
derive from the selected main file, never the overlay directory.
|
||
4. Wire the parsed result into the final process/config command entry points.
|
||
Profile-dependent command completion belongs to Stage 4; do not emulate the old
|
||
runtime to make the process command work now. Update `docs/cli.md` for the final
|
||
input contract as parsing lands.
|
||
|
||
**Acceptance and validation:** parser/loading tests prove help without usable
|
||
configuration, discovery/error cases, explicit-only overrides, retained domain
|
||
flags, removed flag rejection, and ignored legacy environment values. Table-drive
|
||
shared cases; retain representative subprocess checks rather than triplicate every
|
||
parser test across commands. Config commands reject positional arguments and
|
||
process-only flags. Defer assembled command checks until Stage 8 if imports still
|
||
prevent compilation.
|
||
|
||
## Stage 3: Migrate Embedded Prompt Assets and Call Contracts
|
||
|
||
**Status:** Complete (2026-09-24). All nine embedded prompts are PromptKit
|
||
definitions with the shared hardening variable, existing `v1` IDs, and canonical
|
||
JSON Schema associations. Replaced the template renderer and its hash metadata;
|
||
the structured-call request now carries prompt, schema, variables, and attribution
|
||
through one framework contract. `go test ./internal/prompts
|
||
./internal/framework/responseschema ./internal/framework/structuredoutput` and
|
||
`git diff --check` passed. `go test ./...` still fails at old module, validator,
|
||
runtime, and diagnostic callers awaiting their assigned migrations.
|
||
**Depends on:** Stage 1; execute after Stage 2.
|
||
**Owns:** `internal/prompts`, `internal/framework/responseschema`,
|
||
`internal/framework/contracts`, validator contract aliases, and prompt tests.
|
||
|
||
1. Convert all four module and five validator prompts listed in
|
||
[the prompt registry](../internal/prompts.md) to PromptKit definitions. Preserve
|
||
instructions, variables, glossary/description context, prompt IDs, and the two
|
||
existing JSON response schemas. Embed assets once and expose their filesystem
|
||
to the runtime. Do not copy the prototype's duplicate schema directory.
|
||
2. Keep output contracts JSON Schema based, but supply the full effective output
|
||
contract and repair count on each runtime request. Do not create separate
|
||
operator-facing repair defaults per prompt. Use the hardening variable decision
|
||
above to retain a single hardening source.
|
||
3. Change the structured-call request to prompt identity, variables, response schema
|
||
identity, and diagnostic context. Keep decoded output and typed Audita metadata
|
||
as the return boundary. Include role/module instance/section/batch/validator
|
||
context where available for attribution; do not derive identities from messages.
|
||
4. Replace the old parser/renderer and hash assumptions. Keep small metadata and
|
||
schema lookup helpers where Audita needs them. Update the prompt reference.
|
||
|
||
**Acceptance and validation:** a table covering all nine definitions prepares
|
||
representative variable sets through PromptKit with controlled profile fixtures;
|
||
checks include hardening, section/glossary/description data, missing variables,
|
||
correct schema association, and rendered content preserving the previous task
|
||
instructions. Use reviewed expectations, not a retained legacy renderer or old
|
||
Audita module dependency. Existing schema/domain tests remain meaningful. Callers
|
||
are deliberately migrated in Stages 6–7.
|
||
|
||
## Stage 4: Assemble Profiles and Finish Offline Inspection
|
||
|
||
**Status:** Complete (2026-09-24). Added one engine and profile inspection path
|
||
for both roles, requiring registered matching backends and backend-owned
|
||
credentials. Config commands and process select that path; settings, safe selected
|
||
profiles, and provenance are exposed as separate YAML views with secret redaction.
|
||
Focused offline checks passed with `go test internal/framework/llm/runtime.go
|
||
internal/framework/llm/safe_view.go internal/framework/llm/runtime_inspection_test.go`
|
||
and `go test internal/cli/config_input.go internal/cli/config_input_test.go`;
|
||
`git diff --check` passed. Assembled command tests remain deferred until the
|
||
old caller packages are reconnected in later stages.
|
||
**Depends on:** Stages 1–3.
|
||
**Owns:** engine/profile assembly in `internal/framework/llm`, CLI config commands,
|
||
and inspection tests.
|
||
|
||
1. Construct one PromptKit engine with the embedded assets, complete profile files,
|
||
and registered application backends. Reuse the prototype's supported filesystem,
|
||
backend, retry-config, and inspection APIs. Accept an HTTP client at the external
|
||
boundary for tests; production uses the library transport.
|
||
2. Resolve `profiles_dir` against the main config file. Inspect both selected roles,
|
||
require a registered backend, reject endpoint-only bypass and profile credential
|
||
lookup, and compare endpoints after trimming surrounding whitespace and trailing
|
||
slashes, as in the prototype. Do not invent profile inheritance/catalog wiring
|
||
or reject unrelated unselected profile content beyond library behavior.
|
||
3. Complete `config validate` and the three effective-output views. Settings,
|
||
profiles, and explanations share the same load/assembly path used by process.
|
||
Keep the `config is valid` success message. Redact configured secret values in
|
||
any surfaced library error; do not stringify raw configuration/profile objects.
|
||
4. Add the internal safe-view helpers later used for snapshots. Document the view
|
||
interface and its limits in `docs/cli.md` and `docs/config.md`.
|
||
|
||
**Acceptance and validation:** offline command tests prove all views and validation
|
||
agree for base/testing/explicit-profile selections; main-file-relative and absolute
|
||
profile paths work; missing profiles, backend mismatch, competing credentials,
|
||
missing surviving secrets, and invalid views fail with the selected exit class.
|
||
Use a transport that fails the test if called. Inspect outputs/errors for synthetic
|
||
secret absence and overlay/CLI provenance, including inherited validation selection.
|
||
|
||
## Stage 5: Implement the Final PromptKit Structured-Call Adapter
|
||
|
||
**Status:** Complete (2026-09-24). Added role-bound prepared execution with
|
||
target verification, validated-only decoding, typed safe failures, protected
|
||
prepared/final-content snapshots, and logical-call and execution-event hooks.
|
||
Removed the old direct HTTP client and role resolver; updated LLM integration
|
||
documentation. `go test ./internal/framework/llm` and
|
||
`go test ./internal/framework/structuredoutput` passed, including controlled
|
||
HTTP repair/retry, target-drift, transport/protocol, and cancellation cases;
|
||
`git diff --check` passed. Caller integration remains for Stages 6–8.
|
||
**Depends on:** Stages 3–4.
|
||
**Owns:** `internal/framework/llm`, the typed failure boundary in
|
||
`internal/framework/structuredoutput`, and adapter tests.
|
||
|
||
1. Implement role-bound clients over the shared engine. Start the operation context
|
||
and Audita logical-call record before preparation; inherit earlier caller deadlines.
|
||
Select the role profile and backend secret, supply reserved hardening variables,
|
||
and pass an explicit full schema/repair contract. Use `PrepareExecution`, its
|
||
`Details`, then `RunPrepared` so diagnostic messages/hashes describe the actual
|
||
execution snapshot. Defer handle discard; do not call `Prepare` and then `Run`
|
||
with an independently reloaded profile. Before dispatch, verify the prepared
|
||
target still matches the role's registered backend/endpoint and has no competing
|
||
credential lookup. A changed target fails safely rather than sending a backend
|
||
key to a different target. This is the final diagnostic/security boundary, not
|
||
an intermediate adapter.
|
||
2. Check both the returned error and `result.Validation.IsValid`: exhausted repair
|
||
can return a result with failed validation and no error. Decode only validated
|
||
JSON into the requested Audita output. Use the selected failure table:
|
||
exhausted output repair becomes the Audita malformed-output classification; provider/transport/protocol/retry/admission/deadline failures
|
||
remain runtime failures. Preserve typed causes for classification and safe
|
||
messages for users. Explicit zero retries must not become retries-exhausted.
|
||
3. Preserve prepared messages/hashes and available final content and profile/model
|
||
metadata through Audita-owned types. Provide a per-call observer hook and logical
|
||
start/finish record bound to Audita diagnostic context for Stage 9; do not
|
||
create another retry loop, scheduler, provider envelope parser, or schema engine.
|
||
4. Remove the old OpenAI-compatible client and its role config resolution. Retain
|
||
only applicable redaction and diagnostics helpers. Update the LLM integration
|
||
reference to delegate wire/retry mechanics to the pinned library.
|
||
|
||
**Acceptance and validation:** controlled HTTP tests cover successful decode,
|
||
positive repair and exhaustion for both schema types, 429/5xx retry recovery,
|
||
explicit zero, exhausted positive budget, transport/malformed-envelope failure,
|
||
combined retry-plus-repair bounds, preparation failure without dispatch, exact
|
||
prepared request metadata, target-change rejection, and operation cancellation
|
||
during a long retry wait. Assert classifications and meaningful counts, not timing-sensitive sleeps
|
||
or complete error strings. Reuse the prototype sequences; no need to reproduce
|
||
PromptKit's entire status-code matrix.
|
||
|
||
## Stage 6: Migrate All Proposal Callers
|
||
|
||
**Status:** Pending. **Depends on:** Stage 5.
|
||
**Owns:** `internal/modules`, `internal/framework/proposal_generation`, and their tests.
|
||
|
||
1. Switch glossary, homophones, spoken_word, and grammar proposal construction to
|
||
prompt IDs and variables. Preserve section/glossary encoding, context trimming,
|
||
replacement policy, confidence filtering, and applicability checks.
|
||
2. Feed repaired structured output into the existing domain path. Exhausted
|
||
structural repair produces the existing section warning and no proposals;
|
||
runtime failures propagate. Schema-valid but inapplicable proposals do not
|
||
trigger semantic retries or additional repairs.
|
||
3. Update fake clients and module prompt tests to the final request contract.
|
||
Delete duplicate message rendering and obsolete transport/schema checks; retain
|
||
domain protection and prompt variable coverage. Update module internals docs.
|
||
|
||
**Acceptance and validation:** all four modules have an exercised variable/schema
|
||
mapping; existing domain suites retain their assertions. Representative controlled
|
||
adapter coverage proves repaired proposals reach domain checks and that a low-
|
||
confidence/inapplicable proposal remains rejected. Preserve section/proposal
|
||
ordering. Run the module and proposal-generation suites as imports become buildable.
|
||
|
||
## Stage 7: Migrate LLM Validators Without Changing Domain Decisions
|
||
|
||
**Status:** Pending. **Depends on:** Stage 6.
|
||
**Owns:** `internal/framework/validators`, validation prompt builders/batching,
|
||
and validator tests.
|
||
|
||
1. Migrate spoken_form_plausibility, meaning_reversal_review, editorial_review,
|
||
grammar_review, and spoken_word_review to prompt IDs/variables and the common
|
||
structured boundary. Keep deterministic validators independent of PromptKit.
|
||
2. Preserve batching by `validation.max_prompt_tokens`, proposal preview, stable
|
||
proposal indexes, exact decision cardinality, and unknown/duplicate index checks.
|
||
Keep the current domain token-estimation method; do not add a second template
|
||
renderer or treat this budget as a provider output-token limit.
|
||
3. Apply the failure mapping: exhausted repairs reject/warn the batch; schema-valid
|
||
bad cardinality or negative assessments reject without semantic retry. Provider,
|
||
malformed HTTP, deadline, and admission failures propagate as runtime errors.
|
||
Update any old tests treating malformed provider envelopes as warning-only.
|
||
4. Remove redundant validator client/model configuration types and keep execution
|
||
class distinctions. Update validator reference/internal documentation.
|
||
|
||
**Acceptance and validation:** preserve existing batching no-drop/no-duplicate,
|
||
domain reason, glossary protection, and cardinality tests. Cover all five prompt
|
||
mappings, positive repair before domain validation, exhausted repair, and runtime
|
||
failure propagation through a representative validator. Run validator tests;
|
||
full runner behavior is Stage 8.
|
||
|
||
## Stage 8: Connect the Runner and Remove the Old Scheduling Hierarchy
|
||
|
||
**Status:** Pending. **Depends on:** Stages 2 and 4–7.
|
||
**Owns:** CLI process assembly, `internal/framework/runner`, old scheduler contracts,
|
||
and runner/CLI integration tests.
|
||
|
||
1. Pass the shared engine's two role clients into the runner. Remove global and
|
||
role schedulers, scheduler interfaces/fields, legacy concurrency configuration,
|
||
and measured scheduler wrappers. PromptKit backend admission is the only LLM
|
||
service-capacity mechanism. Do not remove unrelated deterministic work or alter
|
||
module sequencing merely because the scheduler disappears.
|
||
2. Preserve concurrent section work, deterministic collection/indexing, ordered
|
||
module application, validator execution classes, and cancellation of remaining
|
||
work on runtime failure. Capacity rejection fails the run as selected; do not
|
||
silently retry or add another queue to conceal it.
|
||
3. Wire final loaded domain settings, role clients, and diagnostics context into
|
||
normal processing. No old-runtime branch or interim gate remains. Keep existing
|
||
transcript encoding, output channels, partial domain results, and artifact
|
||
retention behavior. Ensure terminal failures cannot emit a purportedly complete
|
||
corrected transcript.
|
||
4. Update architecture policy for shared library dependencies and engine-local
|
||
backend admission; preserve the existing ownership, determinism, and redaction
|
||
invariants. Update pipeline/runtime internal guidance.
|
||
|
||
**Acceptance and validation:** controlled concurrent requests from both roles to
|
||
one backend never exceed its configured cap, including retries and repairs;
|
||
separate backend identities have independent capacity. Queued cancellation,
|
||
deadline and zero-wait-capacity rejection do not leak capacity; later work can
|
||
succeed. Existing deterministic runner/domain tests pass after deleting scheduler-
|
||
implementation assertions where the affected packages compile. Exercise a real
|
||
CLI-to-local-HTTP workflow with both roles when reporting dependencies permit;
|
||
otherwise record that check for Stage 12. Do not restore old utilization fields
|
||
or add placeholders merely to make this intermediate commit build.
|
||
|
||
## Stage 9: Collect Execution Observations Safely
|
||
|
||
**Status:** Pending. **Depends on:** Stage 8.
|
||
**Owns:** Audita-owned execution record types and collector in the LLM boundary,
|
||
runner attribution, and concurrency/observation tests.
|
||
|
||
1. Translate PromptKit events into small Audita-owned records at the observer
|
||
boundary. Use the prototype's short mutex-protected collection pattern; callbacks
|
||
do no filesystem I/O, provider calls, or waiting on their own run. Freeze/copy
|
||
snapshots after active work has joined.
|
||
2. Bind observations to logical call, role (`proposal` or `validation`), module
|
||
instance, section, validator and batch where applicable, then retain library
|
||
run/backend/profile and attempt/generation identity. Keep provider attempts distinct from logical calls.
|
||
3. Record dispatch, queue/generation/retry/wall durations, outcome/failure category,
|
||
HTTP status, retry exhaustion, attempts/retries/repairs, and usage value/known/
|
||
complete per field. Use terminal events once for provider totals; never add
|
||
attempt usage on top. `PrepareExecution` emits no events and `RunPrepared` timing
|
||
excludes preparation. Audita's surrounding call record therefore owns the logical
|
||
count and wall time, including preparation failures; do not invent library events.
|
||
A failure before dispatch has zero attempts and known-zero complete usage.
|
||
Preserve completed observations on nil-result failures and mark usage incomplete
|
||
when dispatched attempts lack measurements. Across calls, sum known values,
|
||
set known if any contributing dispatched usage is known, and set complete only
|
||
when all contributing dispatched usage is complete, independently per field.
|
||
An aggregate with no dispatched attempts is known-zero and complete.
|
||
4. Keep non-LLM runner wall/validator timing. Remove instrumentation that counts
|
||
scheduler permits as calls. Do not infer queue peaks, exact active concurrency,
|
||
billing totals, or timing fields the released event API does not provide.
|
||
|
||
**Acceptance and validation:** adapter/collector tests prove known zero differs
|
||
from unknown, partial usage survives failed repairs, undispatched admissions are
|
||
not HTTP attempts, terminal totals are not doubled, and parallel callbacks retain
|
||
attribution safely. Run targeted race tests for the affected LLM/runner packages.
|
||
No raw provider messages, URLs, keys, prompts, or payloads enter observation records.
|
||
|
||
## Stage 10: Update Reports and Diagnostic Artifacts
|
||
|
||
**Status:** Pending. **Depends on:** Stage 9.
|
||
**Owns:** `internal/core/reporting`, `internal/core/diagnostics`,
|
||
`internal/framework/processreport`, runtime diagnostic writers, and CLI artifact wiring.
|
||
|
||
1. Bump the process report version to `v2`; keep the existing transcript/ledger
|
||
structures. Add `llm_summary` with `logical_calls`, `provider_attempts`,
|
||
`transient_retries`, `structural_repairs`, and `usage`. A logical call counts
|
||
once even when it fails before dispatch. Usage has `prompt`, `completion`,
|
||
`total`, `cached`, and `cache_write`, each `{value, known, complete}`.
|
||
Omit the whole summary only when execution observations are unavailable; a
|
||
successfully initialized run with no LLM calls has known-zero complete totals.
|
||
2. Replace obsolete utilization concurrency/pool fields with `backends`, keyed by
|
||
backend ID, containing configured `concurrency`/`queue_capacity` and the same
|
||
execution summary plus `capacity_wait_ms`, `generation_ms`, `retry_delay_ms`,
|
||
and `logical_wall_ms`. Sum library terminal timings once per call for capacity,
|
||
generation, and retry time; use Audita's whole-call elapsed time for logical wall
|
||
time. Attribute pre-dispatch failures to the already selected role backend.
|
||
Durations can overlap and must not be presented as a decomposition of elapsed
|
||
process time. Preserve domain module/validator timing and run wall time.
|
||
Existing role/module call counts now count logical calls; remove their obsolete
|
||
scheduler-wait and transport-execution fields rather than assign misleading new
|
||
meanings. Remove estimated active-concurrency fields throughout utilization.
|
||
3. Write `llm-execution.json` as `{calls: [...], events: [...]}` in the run directory.
|
||
Calls contain Audita logical identity, attribution, elapsed milliseconds, final
|
||
classification, summary, and available prompt/profile/model metadata. Events
|
||
contain the safe Stage 9 records, including attempt and terminal records, joined
|
||
by call ID. A preparation failure has a call record and no fabricated events.
|
||
Use millisecond integer durations in serialized records; absent attempt/section/
|
||
batch identity is omitted, not encoded as a misleading zero. Add optional
|
||
`llm_execution_path` and `selected_profiles_path` to diagnostic path metadata.
|
||
Write `selected-profiles.yaml` from the safe view and use ConfigKit YAML for
|
||
the effective snapshot (`effective-config.redacted.yaml`). Keep invocation
|
||
overlay selection and recorded origins in safe invocation/config diagnostics.
|
||
Adjust centralized filename constants and readers together; do not emit old and
|
||
new snapshots in parallel.
|
||
4. Preserve redacted request/response/error diagnostic content from prepared
|
||
details and available final results, including prompt definition/rendered hashes
|
||
and selected profile/model. Do not promise legacy hash equality or complete
|
||
provider payloads for every retry. Use the separate observation artifact for retry/repair
|
||
attempts. Redact all resolved backend secrets, including unselected ones, at
|
||
every serialization/error boundary; observation records remain payload-free.
|
||
5. Update output-file, operations, and diagnostics/reporting contracts. Preserve
|
||
failure report/ledger behavior and the distinction between primary execution
|
||
failures and secondary diagnostic write errors. Do not change retention policy.
|
||
|
||
**Acceptance and validation:** report/diagnostic tests check v2 mapping, summary
|
||
semantics, per-backend grouping, safe serialization, nil-result partial failures,
|
||
and existing deterministic ledger order. A CLI failure fixture echoes a synthetic
|
||
secret in a provider error; stdout/stderr/report and retained artifacts must not
|
||
leak it. Inspect known-zero and unavailable summaries structurally, not by complete
|
||
JSON golden dumps. Reuse Stage 9 evidence rather than retest every event sequence.
|
||
|
||
## Stage 11: Finish Examples, Migration Guidance, and Removal
|
||
|
||
**Status:** Pending. **Depends on:** Stages 1–10.
|
||
**Owns:** maintained examples, affected canonical documentation, obsolete code/tests,
|
||
prototype removal, and final dependency cleanup.
|
||
|
||
1. Replace maintained config examples with schema 2, complete PromptKit profile
|
||
files, and `overlays/testing.yml` selecting both roles. Use one shared backend
|
||
where roles share a service; provide secret-reference and unauthenticated local
|
||
examples without real secrets. Explain that operators choose real models and
|
||
testing cost/quality is not guaranteed. Validate examples offline with controlled
|
||
synthetic environment values; no service is required for profile inspection.
|
||
2. Write `docs/migrations/v2.md` as the one migration guide. Cover every old file/
|
||
flag/environment item in the inventory, durations, explicit empties/null,
|
||
credentials, profiles/overlays, inspection views, finite capacity, retry/repair/
|
||
deadline semantics, and changed report/diagnostic shapes. Do not add conversion
|
||
tooling or compatibility aliases. Link current contracts for complete references.
|
||
3. Finish README prerequisites/examples and canonical CLI/config, integration,
|
||
internal component, and architecture references. Audit build/CI toolchain images
|
||
for Go 1.25.5 or newer; keep only required Linux amd64/arm64 targets and best-effort
|
||
macOS arm64. Do not rewrite immutable historical release notes.
|
||
4. Remove `examples/library-adoption` once its unique useful scenarios live in
|
||
production tests. Remove the nested dependency on old Audita and duplicate
|
||
assets/fixtures. Update historical roadmap links to recorded evidence or Git
|
||
history rather than leaving broken links to deleted files.
|
||
5. Remove superseded loaders, client, renderers, schedulers, aliases, and tests
|
||
asserting retired behavior. Tidy/verify the root dependency graph and confirm
|
||
no local replacements. Keep yaml.v3 if glossary parsing still uses it. Mark the
|
||
retry-count follow-up in `runtime-behavior.md` resolved without including its
|
||
unrelated retention/timestamp proposals.
|
||
|
||
**Acceptance and validation:** maintained examples validate and all guide links
|
||
resolve. Search for old CLI/env inputs, loader/client/scheduler names and prototype
|
||
references; remaining mentions must be intentional migration/history material.
|
||
Review deleted tests for lost domain/concurrency/redaction protection. Run affected
|
||
example/config/subprocess checks and `git diff --check`; final full-suite execution
|
||
belongs to Stage 12.
|
||
|
||
## Stage 12: Verify the Combined Application and Close the Refactor
|
||
|
||
**Status:** Pending. **Depends on:** Stages 1–11.
|
||
**Owns:** final integration tests, final defect fixes within this scope, and status/evidence.
|
||
|
||
1. Complete representative production subprocess workflows against controlled HTTP:
|
||
base selection, testing overlay, explicit profile override, and a retained
|
||
domain override. Observe actual outbound models for both roles, an accepted
|
||
correction after repair, a warning-only domain/repair rejection, and a runtime
|
||
failure with safe partial diagnostics. Cover an empty module pipeline and a
|
||
repeated-module sequence without duplicating all adapter scenarios.
|
||
2. Run root `go test ./...`, `go build -o ./bin/audita ./cmd/audita`, and focused race
|
||
checks for the final concurrent adapter/runner/collector changes. Verify both
|
||
Linux targets compile using `bash scripts/build-release.sh v2.0.0 <temp-dir>`;
|
||
this names local build outputs and creates no tag or publication. No native Mac, paid-provider, or CI-completion gate.
|
||
If the supported minimum toolchain was not exercised earlier, verify the root
|
||
suite with installed Go 1.25.5; do not silently claim this from a newer compiler.
|
||
3. Fix actual failures and rerun affected checks; repeat the full suite when those
|
||
fixes materially change assembled behavior. Do not rerun unchanged tests solely
|
||
to mark multiple planning steps complete. Verify maintained documentation links,
|
||
examples, and whitespace once for the final content.
|
||
4. Record concise final evidence and known limits here, mark the production feature
|
||
roadmaps implemented, and commit the completed refactor. Both libraries must
|
||
own their selected responsibilities, processing must work, and no temporary
|
||
or legacy compatibility machinery may remain.
|
||
5. Hand off release preparation separately: proposed Audita v2.0.0, one migration
|
||
guide, actual pinned production dependency/license inventory, and the remaining
|
||
LGPL notice/source/rebuild-distribution review. Do not claim license compliance
|
||
from prototype tests or publish a release. The later release unit uses the
|
||
existing minimal automated procedure and ends on a successful source/tag push,
|
||
without waiting for CI.
|
||
|
||
**Acceptance:** the final production application and required Linux builds pass;
|
||
configuration selection, both LLM roles, domain outcomes, cancellation/capacity,
|
||
secret handling, reports and retained artifacts match the selected contract.
|
||
Recorded checks describe what actually ran. The workspace is committed and clean.
|
||
Release-specific compliance and publication are explicitly still pending.
|
||
|
||
## Open Questions
|
||
|
||
None block this implementation plan. Configuration, inspection, runtime boundaries,
|
||
report representation, and stage ownership are selected above. Release distribution
|
||
compliance remains outside this implementation sprint and must be resolved against
|
||
the final production dependency graph before publishing binaries; it is not a
|
||
reason to build compatibility scaffolding or delay the refactor.
|