eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Remove completed roadmaps

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-03 16:58:43 +00:00
parent 2023c91ec3
commit 23e1df4cfb
13 changed files with 6 additions and 2203 deletions
Download Patch File Download Diff File Expand all files Collapse all files

162
docs/roadmap/decision-set-reconciliation.md
View File

@@ -1,162 +0,0 @@
# Partial Success for Validator Decision Sets
**Last updated:** 2026-09-25
**Status:** Implemented. This document owns the target behavior and user intent.
[historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/91d9be68a7bb3a102900253fdfd8438f0a43f666/docs/roadmap/implementation.md) owns concrete decisions, ordered work,
and verification.
## Purpose and User Intent
Preserve usable validator decisions when other entries in the same model response
have missing, duplicate, or unexpected correction indices. One defective entry
should not prevent unrelated candidates from completing their required validation.
This extends Audita's existing partial-success approach without another model call.
Implement this simpler improvement before reconsidering guided domain recovery.
Missing validation must never authorize applying a correction, but neither should
it invalidate a different candidate's unambiguous approval or editorial rejection.
The existing [resilience design](resilience.md) remains the foundation.
## Historical Baseline
The reviewed application baseline is `d2a6fcf`:
- PromptKit validates the complete model response against Audita's embedded JSON
Schema and performs its configured structural repairs. Only structurally valid,
decoded responses reach domain decision mapping.
- `mapLLMResponseToDecisions` in
[`llm_validators.go`](../../internal/framework/validators/llm_validators.go)
requires exactly one decision for every requested correction index. An empty
decision list, missing index, duplicate index, or unexpected index returns an
error instead of retaining usable decisions.
- `LLMBackedValidator.Validate` consequently marks the entire batch unavailable
with `validation_decisions_invalid`. Other batches continue, but usable decisions
within that batch are lost.
- The runner's cardinality checks enforce an internal contract: each candidate
receives one outcome from each validator it reaches. These checks apply to
Audita-owned results, not directly to raw model entries.
- Deterministic validators already filter individual proposals. Unpreviewable
candidates and oversized validation inputs are isolated; recoverable provider
failures remain batch-local; application skips individual conflicts.
- Ordinary reports distinguish failed operations from successful operations with
recoveries. Several failed operations in one stage/chunk share one work-unit
error record. The [output contract](../integrations/output-files.md) owns the
current counts and serialized representation.
The defect addressed here is the all-or-nothing domain mapping boundary, not
validation itself or the runner's internal consistency checks.
## Target Decision Semantics
Reconcile each structurally valid response against the indices actually requested
in that particular validator batch. Correction indices, rather than segment IDs
or positions in the response array, identify candidates. Several corrections may
legitimately target the same segment.
| Response condition | Target behavior |
| --- | --- |
| Exactly one valid decision for a requested index | Preserve its approval or rejection and associated decision evidence. |
| Requested index omitted | Produce an explicit validation-unavailable outcome for that candidate. |
| Requested index appears more than once | Treat that candidate's validation as unavailable; preserve unrelated decisions. |
| Index was not requested in this batch | Ignore the extraneous entry and record the handled defect. |
| Empty decision list for a nonempty request | Mark all requested candidates unavailable; do not fail the run. |
| Decisions appear in a different order | Match by identity and retain deterministic internal results. |
Duplicate handling does not choose the first or last answer, nor infer a verdict
by voting. Even identical duplicates follow the agreed duplicate-index handling;
accepting them specially is not needed for this feature. Duplicate extraneous
indices remain extraneous entries, not reasons to invalidate requested candidates.
An unavailable candidate remains unapplied and does not proceed as approved to
later validators. It is distinct from a valid `approved: false` decision, which
is ordinary editorial processing. Valid approvals continue through the remaining
required validators and apply-time checks; this feature grants no bypass.
The mapper returns one internal outcome for every requested candidate and no
outcome for an extraneous index. Preserve strict runner cardinality checks so
application bugs are still detected. Do not convert genuine internal invariant
failures, cancellation, or fatal setup failures into tolerated model defects.
## Outcomes and Reporting
Keep existing work-unit aggregation and operation accounting:
- If any requested candidate lacks an unambiguous usable decision, record a
validation-operation error identifying the affected requested indices. Other
candidates' usable decisions from that operation remain effective. An operation
error does not mean every candidate in its batch failed.
- If all requested candidates have usable decisions and extra entries were
ignored, record a successful operation with a recovery for that handled defect.
This remains a recovery even if all legitimate decisions are editorial rejections.
- If missing/duplicate requested decisions and ignored extras coexist, retain the
operation error. Follow the existing rule that failed operations do not also
produce recovery records; retain extra-entry details in debug evidence.
- A valid complete response with no handled defect adds neither error nor recovery.
Valid editorial rejection alone never adds either category.
- Preserve actual structural repair and service-retry counts. On successful
operations, consolidate applicable recovery reasons in one recovery record.
Do not count reconciliation as another LLM operation or structural repair.
Error attribution must distinguish affected candidates from the rest of the batch.
Debug evidence should preserve supplied decisions, ignored entries, and the reason
an expected candidate became unavailable. Do not fabricate candidate ledger entries
for unknown indices or mislabel missing decisions as deliberate editorial rejection.
Ordinary reports must retain safe metadata without raw transcript or response text.
Use a decision-specific recovery reason rather than treating ignored validator
entries as defective correction proposals. Concrete reason names and minimal data
model changes belong in the implementation plan. Keep YAML, Markdown, and CLI
counts consistent, with the existing reporting-disable and independent debug options.
## Ownership and Target Architecture
Audita's validator layer reconciles structurally valid model responses into
complete internal candidate outcomes. The runner preserves strict internal
cardinality and advances only approved candidates. Reporting consumes operation
facts without treating partial validation failure as loss of every decision.
Debug evidence distinguishes unavailable validation from editorial rejection.
The schema/transport boundary remains with PromptKit; reconciliation needs no
additional model call. Concrete helper structures, reason names, mapping changes,
and verification steps belong in the [historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/91d9be68a7bb3a102900253fdfd8438f0a43f666/docs/roadmap/implementation.md).
## Scope Boundaries
PromptKit's whole-response structural validation and repair remain unchanged.
A malformed entry, including an extraneous entry that violates the schema, may
still invalidate the response and exhaust structural repairs before reconciliation
can run. Ignoring schema-valid extras does not promise salvage from malformed JSON,
missing required fields, wrong types, or out-of-range values.
General per-entry structural salvage is a separate design question, not part of
this fix. Likewise, guided regeneration remains deferred for reconsideration after
this change; no new retries, feedback prompts, budgets, upstream dependency changes,
or operator controls are required here.
Preserve module scope, literal replacement behavior, fixed chunks, stage barriers,
span-deletion safeguards, backend capacity limits, and fatal cancellation behavior.
No additional restrictions on valid corrections or guarantees about model judgment
are introduced. This work does not itself authorize tagging or publishing v2.0.0.
## Acceptance Criteria
- A schema-valid response can retain usable decisions despite missing, duplicate,
or unexpected indices elsewhere in the same batch.
- Only the affected requested candidates become unavailable; extras never create
candidates or attach decisions to a different batch's candidates.
- Valid negative decisions remain editorial outcomes, and every applied correction
completes all required validation.
- Internal validator results satisfy existing strict cardinality checks.
- Missing/duplicate requested decisions yield accurately attributed errors without
losing valid work. Extra-only defects yield a consolidated recovery; mixed
defects follow existing failed-operation reporting semantics.
- Final transcript, debug evidence, YAML reports, Markdown counts, and CLI summary
agree about applied work and failures/recoveries.
- Reconciliation makes no additional provider calls. Existing structural failures,
provider failures, and cancellation retain their documented handling.
## Open Questions
No unresolved product-policy questions block implementation. Guided recovery and
structural salvage remain separate future decisions.

74
docs/roadmap/feature-request-narratio.md
View File

@@ -1,74 +0,0 @@
# Independent output destinations for Narratio integration
## Request
Allow CLI callers to independently select destinations for:
- the completed transcript;
- ordinary reports; and
- debug captures, including retained working material.
Narratio would like to use the existing `--output`, `--report-dir`, and
`--work-dir` controls together, with `--debug` supplied when it explicitly
requests debug capture. This request concerns external CLI behavior only.
## Current CLI limitation
The [CLI contract](../cli.md) documents that `--report-dir` selects the report
directory for stdout transcript output, but has no effect with file transcript
output. Reports then accompany the transcript file. This prevents Narratio from
directing the transcript and ordinary reports to independent locations in the
same invocation.
The existing transcript and debug destination controls already provide much of
the requested separation.
## Desired behavior
- With `--output FILE --report-dir DIR`, write the transcript to `FILE` and any
ordinary reports produced to `DIR`.
- Continue supporting `--report-dir DIR` when the transcript goes to stdout.
- Apply the selected report destination to reports produced on unsuccessful
runs as well as successful runs. This does not require reports for failure
conditions that would not otherwise produce them.
- Allow the debug destination to be selected independently of both the
transcript and ordinary report destinations. Audita may retain its own
filenames and nested directory structure beneath that destination.
- Keep debug enablement separate from destination selection. Narratio needs
to select a debug destination while leaving capture enablement to Audita's
configuration or defaults, and to pass `--debug` when explicitly requesting
capture. Narratio does not need an additional debug-disable mechanism.
- Preserve existing defaults when destination overrides are omitted. Selecting
a report or debug destination does not itself enable that category of output.
Production Narratio runs use Audita's default ordinary reporting behavior. They
do not pass `--reporting=false` or enable debug unless requested. Narratio
captures stdout and stderr itself; this request does not require a separate
Audita flag for redirecting those streams.
## Supporting rationale
Narratio's planned workspace separates products and diagnostics. Operators inspect
completed transcripts under `transcripts/` and other completed products under
`artifacts/`. Logs and Debug have run-specific storage and convenient
latest-attempt views organized by stage.
Independent destinations let Narratio supply a transcript staging path, a
run-specific polish log directory for ordinary reports, and a separate
run-specific polish debug directory. Reports can be inspected at their selected
location as they are produced, including after failures, without Narratio relocating them
from the transcript directory. Normal reporting can remain enabled without
mixing diagnostic files with completed products.
Narratio owns these destinations and their operator-facing views. Audita owns
the report and debug formats and its internal working-state lifecycle. This
request does not prescribe Audita's internal design, cache locations, retention
policies, or how it organizes material within a debug capture.
## Observable outcome
A caller can select three distinct destinations in one invocation. The
transcript appears at the selected output file, ordinary reports appear at the
selected report directory, and enabled debug captures appear beneath the
selected debug root. The same report placement applies to whatever reports
Audita produces before an unsuccessful exit.

12
docs/roadmap/future.md
View File

@@ -1,15 +1,15 @@
# Deferred Feature Work
**Last updated:** 2026-09-25
**Last updated:** 2026-10-03
**Status:** Deferred or exploratory work; not implemented or scheduled. These
entries preserve scope outside the [resilience roadmap](resilience.md) and
[span-deletion consistency roadmap](span-deletion.md). Each entry states its
intent; inclusion does not commit us to implementing every capability.
entries preserve work beyond the completed resilience and span-deletion
features. Each entry states its intent; inclusion does not commit us to
implementing every capability.
## Domain-Aware Response Recovery
The [partial-success decision reconciliation](decision-set-reconciliation.md)
The implemented [partial-success decision reconciliation](../internal/validators.md#llm-validator-batching)
preserves usable decisions without another model call. Guided recovery remains
deferred pending evidence from failures that remain. General per-entry
salvage from structurally invalid responses is also a separate design question;
@@ -59,7 +59,7 @@ No new scheduler, global limiter, or execution mode is selected by this entry.
Potential future work, subject first to evaluating whether it is worth implementing
at all. Empty replacement text and deletion of a span within a surviving segment
are covered by the accepted [span-deletion consistency roadmap](span-deletion.md).
are covered by the implemented [span-deletion behavior](../internal/modules.md#proposal-generation-ownership).
That feature does not remove segment records or permit blank resulting segments.
Before designing segment removal, review representative production examples to

137
docs/roadmap/implementation.md
View File

@@ -1,137 +0,0 @@
# Implementation Plan: Independent Output Destinations
**Last updated:** 2026-10-03
**Status:** Complete; Stages 1–2 implemented and verified.
**Audience:** gpt-5.6-terra coding agent.
## Scope and Execution
Implement the target behavior in the [Narratio feature roadmap](narratio-feature.md),
in Stage 1 then Stage 2. That roadmap owns the user intent and external contract;
this plan owns the implementation sequence and verification. The original
[Narratio request](feature-request-narratio.md) supplies consumer context. The
completed policy-adoption plan previously at this path remains available at
[its final committed revision](https://gitea.maximumdirect.net/eric/audita/src/commit/61bc467a03da17d52481faa5db96d9e048815be7/docs/roadmap/implementation.md).
Begin with [development guidance](../development.md) and the applicable
[agent](../policy/agents.md), [architecture](../policy/architecture.md),
[documentation](../policy/documentation.md), and
[testing](../policy/testing.md) policies. Use graph-first code discovery as the
agent policy directs. Commit each completed unit with a short, plain English
message; do not release, tag, or push as part of this feature. No approval gate
is needed between stages.
Keep the change at the CLI destination-selection boundary. `--report-dir` is
already parsed as a ConfigKit override of `reporting.stdout_dir`; retain that
mapping so effective-configuration views and flag precedence remain intact.
For `process` only, also retain whether the caller explicitly supplied the flag.
The configured `reporting.stdout_dir` value alone must not redirect file-output
reports. Do not add a configuration key, alter report schemas, add output-root
separation rules, or change the output session's existing lifecycle merely to
support this selection.
Use this selection rule after configuration is loaded:
| Reporting state and transcript destination | Selected ordinary report directory |
| --- | --- |
| Reporting disabled | None; do not preflight or create the chosen report directory |
| Reporting enabled, stdout transcript | Effective `reporting.stdout_dir`, including any `--report-dir` override |
| Reporting enabled, file transcript, explicit `--report-dir` | Effective `reporting.stdout_dir` from that flag |
| Reporting enabled, file transcript, no explicit `--report-dir` | Parent of `--output FILE` |
An active selected report directory must be nonblank. A blank
`reporting.stdout_dir` must not newly invalidate file output when no explicit
`--report-dir` was supplied. Continue to let existing configuration validation
handle invalid setting values; this plan does not add special handling for an
invalid but inactive flag. Existing `--work-dir` and effective
`diagnostics.debug` controls already select and enable debug capture
independently. Selecting `--report-dir` or `--work-dir` alone never turns on
reporting or debug capture. A failed report still appears in the selected
directory only when the existing output lifecycle would produce it; failures
before report initialization remain outside that guarantee.
## Stage 1 — Implement Explicit Report-Destination Selection
**Completion:** Complete — targeted CLI and subprocess tests pass; reviewed diff and whitespace checks.
**Files:** `internal/cli/config_input.go`, `internal/cli/process.go`,
`internal/cli/run.go`, `internal/cli/config_input_test.go`, and the existing
affected subprocess test in `cmd/audita/main_integration_test.go`.
1. In `configInput`, record explicit `--report-dir` presence from the parsed
flag set's `Visit` callback. Preserve the existing ConfigKit override and
its `--report-dir` provenance. Do not infer explicitness from the effective
config value, because a default, file, or overlay can yield the same value.
Other commands may continue accepting the override for inspection without
changing their behavior.
2. Extend `selectedReportDir` to accept that explicitness and implement the
table above. Make `runProcess` pass the new input to it, then continue
passing the selected path to the existing `diagnostics.PrepareOutput` call.
Leave `PrepareOutput` collision, preflight, initialization, failure, and
finalization behavior in place. Leave the existing `cfg.Diagnostics.Debug`
and `cfg.Diagnostics.WorkDir` path into debug startup intact.
3. Update focused CLI tests for explicit presence and selection: file output
with explicit flag uses the selected directory; file output with only a
configured or overlay `reporting.stdout_dir` still uses the output parent;
stdout uses the effective report directory; disabled reporting selects no
active report directory. Test active blank values only where they affect
selection. Retain coverage that the flag remains a ConfigKit override.
4. Replace the existing subprocess assertion in
`TestSubprocessReportingAndDebugSelections` that relies on an ignored
`--report-dir` with the new behavior. Keep its separate check that an
inactive destination is not probed. Avoid duplicating every unit-selection
case at the subprocess layer.
5. Update `audita process` help so it no longer says `--report-dir` is ignored
with file output. Keep help aligned with the selection rule above.
**Stage completion:** Run targeted tests for the changed CLI and subprocess
packages (`go test ./internal/cli ./cmd/audita`) and `git diff --check`. Review
the diff for accidental configuration-schema changes or newly strengthened
destination constraints, then commit the coherent code/test unit.
## Stage 2 — Verify End-to-End Behavior and Update the Public Contract
**Files:** `cmd/audita/main_integration_test.go`, `docs/cli.md`,
`docs/config.md`, `docs/operations.md`,
`docs/integrations/subprocess.md`, and `docs/roadmap/narratio-feature.md`.
**Completion:** Complete — the full Go test suite, maintained configuration
examples, affected Markdown links, and whitespace checks pass.
Touch another document only if its current statement actually conflicts with
the new behavior.
1. Add one representative subprocess success case with distinct transcript
file, report directory, and debug root, enabling debug explicitly. Verify
the transcript, ordinary reports, and `run-<id>` debug capture appear at
their selected destinations and that the report files are not placed beside
the transcript. Reuse the existing test fixture; do not prescribe a new
report format or a Narratio-specific directory layout.
2. Add a subprocess case based on the invalid-input fixture used by
`TestSubprocessInvalidInputLeavesFailedReports`. It fails **after** report
initialization; check that its available failed-status reports use an
explicit report directory. Preserve the existing test of default placement
and the existing early-failure semantics. Rely on current output-session
collision and inactive-debug tests where they already prove the behavior;
add a narrow test only if this new CLI route exposes an uncovered interaction.
3. Change the CLI, configuration, operations, and subprocess documents to state
the four selection cases above. Explain that a config-file or overlay value
for `reporting.stdout_dir` remains stdout-only, while an explicit
`--report-dir` flag selects reports for either transcript destination.
Clarify that `--work-dir` chooses a
debug root without enabling capture and that configuration or `--debug` can
enable it. Describe reports after failures only to the extent the current
output lifecycle supplies them. Leave the canonical output-file schema
document unchanged unless this work exposes a factual error in it.
4. Review the completed behavior against the feature roadmap's acceptance
criteria. Run `go test ./...`, check documentation links and examples, and
run `git diff --check`. If a test fails, resolve that failure before marking
the feature complete; do not add unrelated gates. Mark the feature roadmap
implemented once the final behavior and documentation are verified, then
review the staged diff and commit the completed unit with a short, plain
English message. Leave a clean workspace.
No open policy or design decision is required to execute these stages. The
explicit-flag precedence, preserved defaults, inactive-destination behavior,
failure-report limit, and existing debug controls are settled in the feature
roadmap. If implementation uncovers a conflict with an established contract,
report the concrete conflict instead of silently broadening this feature.

128
docs/roadmap/library-adoption.md
View File

@@ -1,128 +0,0 @@
# Coordinated ConfigKit and PromptKit Adoption
**Last updated:** 2026-09-28
**Status:** Production implementation complete. ConfigKit v0.4.2 and PromptKit
v0.11.2 are adopted; [historical implementation evidence](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md) records the
completed checks. Release compliance and publication remain separate work.
## Goal and Ownership
Adopt ConfigKit and PromptKit to align Audita with the application suite's shared
configuration and LLM execution conventions. Consistency is the primary benefit;
removing overlapping application infrastructure is a secondary benefit. Preserve
Audita's transcript-correction capabilities and domain protections.
| Owner | Responsibilities |
| --- | --- |
| ConfigKit | Strict application configuration loading, presence/default semantics, explicit overrides, recorded resolution, secret references, and safe inspection |
| PromptKit | Prompt/profile formats and rendering, provider calls, structural output validation and repair, transient retries, engine-local backend admission, and execution observations |
| Audita | Setting meanings, file/CLI selection, role/profile association, normalization/chunking, ordered modules, domain validation, correction application, reports, diagnostic redaction, and artifact lifecycle |
The [configuration roadmap](configkit.md) owns configuration and inspection policy.
The [boundary findings](library-boundary-findings.md) own the selected defaults,
full migration inventory, failure mapping, and prototype evidence. The selected
production versions are ConfigKit v0.4.2 and PromptKit v0.11.2, with Go 1.25.5 as
the minimum. Use published modules, never sibling checkouts or local replacements.
The existing YAML dependency remains while glossary parsing needs it.
## Migration Posture
This is an intentional breaking transition targeting Audita v2.0.0. No backward
compatibility is required for existing installations, config files, CLI flags,
environment overlays, or inspection output. Provide one migration guide, without
legacy loaders, aliases, dual modes, or automatic conversion tooling.
Both libraries belong to one refactor. Intermediate commits need not build or run;
there is no supported intermediate product state. No temporary adapters, compatibility
shims, or processing gates are needed. The complete application must work and pass
its checks before the refactor is finished. Release compliance and publication
come last, with no intermediate public release.
## Configuration and Execution Target
ConfigKit selected overlays enable a testing configuration that chooses cheaper or
faster models. The example selects both proposal and validation profiles. ConfigKit
application overlays and PromptKit execution profiles remain separate concepts.
Complete operator-maintained profiles own model/generation controls; embedded
application prompts and schemas are not operator-selected files. No catalog wiring,
profile-inheritance feature, ConfigKit imports, or secret-directory bootstrap is
part of this migration.
Capacity controls serve two needs: respecting a local service's hard simultaneous-
query limit and bounding concurrent paid API activity. One finite limit per backend
is shared across both roles and every initial, retry, and repair attempt. Limits
are engine-local, not coordinated across processes. No additional global or
per-role limiter or fine-grained scheduling policy is required. Profiles must use
registered backends and ConfigKit-resolved credentials; endpoint-only bypass and
competing profile credential lookup are excluded.
Structural repair is enabled by default for both roles because additional attempts
to obtain valid output are desired. Transient retries and structural repairs have
separate budgets; explicit zero disables the respective behavior. Each generation,
including a repair, receives its own transient-retry budget. The Audita logical-call
deadline covers preparation, waiting, backoff, all generations, and repairs; profile
HTTP timeouts apply per attempt. Exact defaults and bounds are in the
[selected contract](library-boundary-findings.md#configuration-and-profile-contract).
No Audita outer retry loop restores the old broader retry policy.
Audita retains all four correction modules, five LLM-validator prompts,
deterministic validators, confidence/applicability checks, glossary protection,
validation batching/cardinality, deterministic proposal and ledger ordering,
transcript encoders, and artifact retention. PromptKit-valid JSON still passes
Audita's domain checks. Repair exhaustion, provider failure, and domain rejection
follow the [selected failure table](library-boundary-findings.md#execution-and-failure-mapping).
Retention and transcript timestamp changes remain separate work; this migration
resolves the [retry-count follow-up](runtime-behavior.md).
## Reporting and Diagnostic Target
Keep stable Audita-owned serialized records; do not expose raw library structs.
Use process report schema `v2` with the existing domain outcomes and a new
`llm_summary` separating logical calls, dispatched provider attempts, transient
retries, structural repairs, and usage. Transcript schema versions remain unchanged.
The implementation plan specifies the concrete new fields and artifact wiring.
Usage distinguishes known zero, unknown, and incomplete totals. Preserve observations
from failed calls even when no final result exists. Count terminal totals once;
attempt observations provide diagnosis without being added to those totals again.
This is operational evidence, not a complete billing ledger.
Replace global/role scheduler utilization with per-backend configured capacity and
observed queue/generation/retry/logical-call timing. Do not fabricate measurements
that the event API does not provide. Keep domain timing and deterministic correction
ledger ordering. Record prompt definition/rendered hashes and selected profile/model
without promising the old hash values. Callbacks are short, concurrency-safe, and
free of blocking I/O or waits on their own execution.
Effective configuration and selected-profile snapshots remain distinct safe views.
Secret values must not escape through inspection, logs, reports, runtime errors,
or retained diagnostic files. Observation records exclude payloads, URLs, raw
provider messages, prompts, and credentials. Existing artifact lifecycle and
partial-failure handling remain Audita responsibilities.
## Completion Criteria and Release Boundary
- Both libraries own their selected responsibilities in the production application.
Superseded loaders, transport, renderers, schedulers, aliases, and the isolated
prototype are gone once its useful evidence is represented in production tests.
- Offline tests prove configuration-to-execution routing for both roles, selected
overlays and explicit overrides, shared backend capacity, retry/repair outcomes,
cancellation/deadlines, domain decisions, redaction, and truthful partial reports.
Tests exercise real internal collaborators and controlled HTTP boundaries.
- Maintained examples and canonical docs describe the implemented final model.
One migration guide covers files, flags, environment inputs, profiles, credentials,
overlays, inspection, and changed runtime/report/diagnostic semantics.
- The assembled application passes its required checks; Linux amd64 and arm64 build.
macOS arm64 remains best effort. No live-provider or CI-success gate is added.
PromptKit v0.11.0's reviewed retry, observation, and capacity APIs remove the earlier
upstream feature prerequisites; no upstream edit is required for adoption. The
prototype proved representative behavior, not release compliance or model quality.
The selected PromptKit release uses LGPL-3.0-or-later. A later release unit must
review the actual production dependency graph and distribution requirements,
including notices and source/rebuild materials, before binary publication. This
implementation plan does not claim that review is complete. Release publication
uses the existing [procedure](../release.md) and completes on successful source/tag
push, without waiting for CI or artifact delivery.

392
docs/roadmap/library-boundary-findings.md
View File

@@ -1,392 +0,0 @@
# Shared Library Boundary: Decisions and Evidence
**Status:** Boundary and offline prototype completed, 2026-09-23. Selected prototype contract and proposed
production migration target; not the current Audita configuration contract.
The [bounded roadmap](library-boundary.md) owns scope and completion criteria.
**Subsequent scope decision:** The [ConfigKit overlay plan](configkit.md#selected-overlays-for-testing)
adds optional selected overlays, with v0.4.0 as the migration baseline. It supersedes
the original no-overlay decision and adds an input layer between the main file and
explicit setting overrides. The v0.3.1 prototype results below remain historical
evidence; they do not establish overlay integration.
## Selected Boundary
The former prototype is retained in Git history at `590604b`; it was a separate
Go module using published ConfigKit v0.4.0, PromptKit v0.11.0, and
Audita v1.1.0. The Audita dependency lets this consumer reuse actual internal domain
helpers through its Audita module namespace without a local replacement. It does
not make those internal packages a new public API. Remove this dependency along
with the prototype after equivalent production integration tests landed.
The initial prototype results below were recorded with ConfigKit v0.3.1. On
2026-09-24, upgrading only its ConfigKit pin and sums to v0.4.0 required no source
changes. With Go 1.26.8 on linux/amd64, the nested offline `go test ./...` passed
and `go run .` still reported one accepted correction, four attempts, zero
transient retries, and two structural repairs. This verifies the dependency
upgrade, not selected-overlay behavior.
ConfigKit owns loading, presence, explicit overrides, secret resolution, and
redacted effective settings. PromptKit owns profile parsing, rendering, provider
execution, transient retries, structural repair, and backend admission. Audita
owns file selection, setting meanings, role selection, domain acceptance, and
mapping execution facts into reports. Production CLI and runtime remain unchanged.
## Configuration and Profile Contract
Selected for the prototype and recommended for production:
- Main YAML requires integer `version: 2`; absent, null, unknown fields, and older
versions fail. No legacy parser, aliases, environment overlays, or automatic
conversion. Defaults precede the selected file; explicitly supplied CLI settings
become ConfigKit overrides. Validation runs on the resolved result, so an override
may repair an invalid lower-precedence setting when decoding succeeds.
- Production file selection: explicit `--config`, then
`/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`. Select one file;
do not merge them. Missing system files are skipped; explicit missing or empty
paths and inaccessible selected files fail. No selected file fails with a request
to configure backend/profile selection: no hidden provider or model default.
Remove `AUDITA_CONFIG`. The prototype loads its explicit fixture only; discovery
and command parsing belong to the ConfigKit implementation stage.
- `profiles_dir` defaults to `profiles`, resolved relative to the selected main
configuration file, with absolute paths accepted. PromptKit reads that directory.
Prompts and schemas are application-embedded, not operator-selected prompt files.
The prototype uses complete application-owned profile files. Catalog profiles and
profile inheritance are not needed for this migration; do not add catalog wiring.
- `backends.<id>.endpoint` is required. Use one application backend identity per
service capacity pool. Do not split proposal and validation into different backend
IDs when they use the same service. Limits are engine-local, not shared across
multiple Audita processes or other applications.
- `backends.<id>.concurrency` defaults to **1**, must be positive, and is shared by
both roles and all initial/retry/repair attempts. Operators can select **4** for
the described local backend or a suitable finite paid API limit. There is no
additional global or per-role limiter. `queue_capacity` defaults to **1024**,
following PromptKit; non-negative, with zero meaning no waiting admission.
This is an operational escape hatch rather than a second scheduling hierarchy.
- `backends.<id>.credential` is an optional ConfigKit secret reference: a scalar
environment variable name. Omission or null means no credential. A supplied
reference must resolve to a nonempty value, including during validation and
inspection. There is no direct key field or CLI key flag. ConfigKit resolves it
once and Audita supplies `RunRequest.APIKey`. Profiles must not introduce another
`api_key_env` lookup. Backend credentials also support unauthenticated local use.
- Profiles own model, temperature, output token limit, top-p, per-generation timeout,
service tier, reasoning effort, and extra provider parameters using PromptKit's
format. They select the application `backend`; their effective endpoint must
equal that backend's endpoint. Endpoint-only execution is rejected because it
bypasses the requested finite capacity boundary. A named registered backend is
mandatory even if both roles use the same profile.
- `llm.proposal_profile` is required. Omitted `llm.validation_profile` inherits the
final proposal profile as a whole, with ConfigKit recording the derivation.
Explicit empty or null values fail; no individual model/endpoint/key inheritance.
- `llm.transient_retries` defaults to **2** additional attempts per generation;
explicit zero disables retries; negative is invalid. PromptKit owns which failures
are eligible. Do not add an Audita retry loop.
- `llm.repair_attempts` defaults to **2** additional generations for both roles;
explicit zero disables repair when deliberately requested. PromptKit supports
0–3. Positive defaults are exercised in the prototype. Audita owns this common
setting and supplies a complete request output contract with the role's embedded
schema, avoiding separate user-facing repair defaults in each prompt.
- `llm.operation_timeout` defaults to **15m**, must be positive, and uses a Go
duration string. It starts before each logical proposal/assessment call and covers
preparation, capacity waiting, transient backoff, generation, and repairs. It is
separate from whole-process cancellation. Profile `timeout_seconds` defaults to
PromptKit's **600** seconds and is applied afresh to each built-in HTTP attempt.
It does not bound the combined retry/repair sequence. A stricter caller/transport timeout can
also end execution. The two defaults are selected starting points, not throughput
measurements. With the default T=2 and R=2, each logical call dispatches at most
**9** attempts before deadline/cancellation; proposal plus assessment may use 18.
- Omission retains static defaults or invokes the recorded role/backend derivations.
Null is not a general synonym for omission: nonoptional scalars reject it. Optional
credentials and the future optional section target accept null. Explicit empty
module lists will mean an empty correction pipeline, not “restore defaults.”
Repeated modules retain their existing ordered meaning.
- The original prototype disabled ConfigKit imports, selected configuration layers
(then called configuration profiles), and secret-directory bootstrap. Selected
overlays are now verified under the linked overlay decision; the other two
features remain disabled. PromptKit execution profiles retain their meaning.
Configuration validation/inspection must assemble and inspect both selected profiles
without provider calls. ConfigKit's effective YAML describes only its loaded state;
a separate selected-profile view shows effective model controls/backend selection.
Do not imply either snapshot alone represents the full execution configuration.
The prototype exercises the two APIs but does not implement production subcommands.
Do not serialize raw profile/library objects as a permanent Audita report schema.
## Migration Inventory
This inventory covers the current configuration and process flags documented in
[configuration](../config.md) and [CLI](../cli.md), including environment-only fields.
“Retain” below means retain domain meaning in the new schema, not accept old files.
All legacy environment overlays listed afterward are removed.
| Current file field / setting | Current CLI flag(s) | Target and rationale |
| --- | --- | --- |
| File selection | `--config` | Retain explicit invocation flag; discovery described above. |
| Transcript input | positional path | Invocation only; retain one transcript path. |
| Glossary input | `--glossary` | Invocation only; retain required glossary path. |
| Corrected output | `--output` | Invocation only; retain stdout fallback. |
| Report output | `--report-json` | Invocation only; retain separate report file. |
| `pipeline.modules` | `--modules` | Retain sequence/default `[glossary, homophones, glossary, spoken_word, grammar]`; explicit empty clears rather than inherits. |
| `output.schema` | `--output-schema` | Retain encoder selection/default `bare-segments`. |
| `llm.proposal.base_url`, `llm.validation.base_url` | `--base-url`, `--validation-base-url` | Replace with backend endpoint; remove per-role endpoint flags. |
| Both `llm.*.model` | `--model`, `--validation-model` | Replace with profile model; new `--proposal-profile` / `--validation-profile` select whole profiles. |
| Both `llm.*.api_key_env`, direct runtime keys | `--llm-api-key`, `--validation-llm-api-key` | Replace with backend ConfigKit secret reference; remove direct key flags. |
| Both `llm.*.timeout` | `--llm-timeout-seconds`, `--validation-llm-timeout-seconds` | Replace with profile `timeout_seconds`, plus `llm.operation_timeout`; no duplicate timeout flags. |
| Both `llm.*.max_retries` | `--max-retries`, `--validation-max-retries` | Replace with common `llm.transient_retries` and distinct `llm.repair_attempts`; remove misleading old flags. |
| `concurrency.total_llm` | `--total-llm-concurrency`, `--llm-concurrency` | Replace with per-backend concurrency; remove global/alias flags. |
| `concurrency.proposal_llm`, `concurrency.validation_llm` | `--proposal-llm-concurrency`, `--validation-llm-concurrency` | Remove role budgets; shared backend handles service capacity. |
| Validation prompt budget (no old YAML field) | `--validation-max-prompt-tokens` | Add `validation.max_prompt_tokens`, default 2048, positive; retain explicit flag override. Audita owns batching. |
| `chunking.target_sections` | `--target-sections` | Retain optional positive target; omission/null means automatic, zero invalid. |
| `chunking.max_section_tokens` | `--max-section-tokens` | Retain 8192, positive. |
| `chunking.min_section_tokens` | `--min-section-tokens` | Retain 2048, positive, no greater than maximum. |
| `normalization.max_segment_gap` | `--normalize-max-segment-gap` | Retain 4.0 numeric seconds, non-negative; drop alternate YAML duration syntax for these domain numeric fields. |
| `normalization.ellipsis_gap` | `--normalize-ellipsis-gap` | Retain 3.5 numeric seconds, non-negative. |
| `normalization.max_segment_duration` | `--normalize-max-segment-duration` | Retain 60.0 numeric seconds, positive. |
| `normalization.max_segment_tokens` | `--normalize-max-segment-tokens` | Retain 2048, positive. |
| `thresholds.glossary` | `--glossary-confidence-threshold` | Retain 0.8 in [0,1]. |
| `thresholds.homophones` | `--homophones-confidence-threshold` | Retain 0.8 in [0,1]. |
| `thresholds.spoken_word` | `--spoken-word-confidence-threshold` | Retain 0.8 in [0,1]. |
| `thresholds.grammar` | `--grammar-confidence-threshold` | Retain 0.8 in [0,1]. |
| `context.description` | `--transcript-description` | Retain trimming and 500-byte limit; empty clears context. |
| `diagnostics.work_dir` | `--work-dir` | Retain `/tmp/audita`, nonempty; artifact lifecycle remains Audita-owned. |
| `diagnostics.retention` | `--work-dir-retention` | Retain `auto` default and selection; existing `never` follow-up is separate. |
| Help/version/config commands | `--help`, `-h` | Retain command purpose; help should parse before loading configuration. Effective-config output intentionally becomes ConfigKit YAML, not legacy JSON. |
Retained setting flags become explicit ConfigKit overrides only when supplied;
CLI parser defaults must not overwrite file values. Normalization flag values stay
numeric seconds. Domain values convert into existing domain structs after validation;
do not spread ConfigKit or PromptKit types into correction algorithms.
Complete environment removal map (names remain permissible if explicitly named as
secret references; none have implicit configuration meaning):
| Old environment names | Replacement |
| --- | --- |
| `AUDITA_CONFIG` | File selection above. |
| `AUDITA_MODULES` | `pipeline.modules` / explicit flag. |
| `AUDITA_LLM_API_KEY`, `OPENROUTER_API_KEY`, `AUDITA_VALIDATION_LLM_API_KEY` | Explicit backend credential references; no implicit fallback. |
| `AUDITA_MODEL`, `AUDITA_VALIDATION_MODEL` | Profile files and role selection. |
| `AUDITA_BASE_URL`, `AUDITA_VALIDATION_BASE_URL` | Backend endpoints. |
| `AUDITA_LLM_TIMEOUT_SECONDS`, `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS` | Profile timeout and operation deadline. |
| `AUDITA_MAX_RETRIES`, `AUDITA_VALIDATION_MAX_RETRIES` | Common transient/repair settings. |
| `AUDITA_TOTAL_LLM_CONCURRENCY`, `AUDITA_LLM_CONCURRENCY`, `AUDITA_PROPOSAL_LLM_CONCURRENCY`, `AUDITA_VALIDATION_LLM_CONCURRENCY` | Backend concurrency; remove alias/role budgets. |
| `AUDITA_VALIDATION_MAX_PROMPT_TOKENS` | `validation.max_prompt_tokens` / explicit flag. |
| `AUDITA_TARGET_SECTIONS`, `AUDITA_MAX_SECTION_TOKENS`, `AUDITA_MIN_SECTION_TOKENS` | Corresponding chunking fields / flags. |
| `AUDITA_NORMALIZE_MAX_SEGMENT_GAP`, `AUDITA_NORMALIZE_ELLIPSIS_GAP`, `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION`, `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS` | Corresponding normalization fields / flags. |
| `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD`, `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD`, `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD`, `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD` | Corresponding threshold fields / flags. |
| `AUDITA_WORK_DIR`, `AUDITA_WORK_DIR_RETENTION` | Diagnostics fields / flags. |
## Execution and Failure Mapping
The executable uses grammar proposal plus grammar review, current response schemas,
current hardening text, transcript-section/description helpers, proposal preview, and
actual decision-cardinality enforcement from Audita v1.1.0. PromptKit renders the
adapted templates; the custom HTTP transport supplies OpenAI-compatible responses.
The prototype pre-expands Audita's `hardening` template function into the embedded
PromptKit definitions; it passes existing named string variables unchanged. No
provider, paid call, or injected high-level LLM client participates.
Generated JSON is structurally validated/repaired by PromptKit before Audita decodes
it. Audita then checks proposal applicability, confidence, and decision identity and
cardinality. The demonstration previews corrections without mutation. Production
retains every module's full validator chain, glossary protection, batching, and
apply-time checks; the representative prototype does not claim to replace them.
| Outcome | Section/batch effect | Overall run |
| --- | --- | --- |
| Structural repair succeeds | Continue domain validation. | Normal execution. |
| Proposal repair exhausted | No proposals from this section; malformed-output warning. | Continue other work; successful completion may include warnings. |
| Assessment repair exhausted | Reject this batch; malformed-output warning. | Continue other work. |
| Schema-valid invalid/missing/duplicate decision indices | Reject batch and warn; no semantic retry/repair. | Continue other work. |
| Inapplicable/low-confidence proposal or negative assessment | Reject affected proposal; preserve domain reasons. | Continue other work. |
| Provider failure, retry exhaustion, transport or malformed HTTP response | Stop affected operation; retain available observations. | Runtime failure; cancel remaining work, retain existing Audita diagnostic handling. |
| Deadline, caller cancellation, capacity admission failure | Stop affected operation; no outer retry or fabricated output. | Runtime failure with distinct classified cause. |
| Invalid settings/profile/schema setup | Fail before processing where detectable. | Configuration error for user input, runtime error for broken embedded assets. |
Zero transient retries produces one ordinary provider failure, not
`ErrRetriesExhausted`; the latter signals an exhausted **positive** budget.
Provider failures retain a typed cause. Do not expose raw provider messages through
user diagnostics: continue Audita's separate secret-redaction boundary. A failed
logical call may return no result while its observations still record completed
attempts or an earlier successful generation. Already completed domain results can
remain in diagnostics; failure must not be presented as a fully corrected output.
## Observation and Report Handoff
The prototype maps library events into Audita-owned records under a short mutex.
Records contain logical run/role/backend/profile identity, generation and attempt
indices, dispatch state, queue/generation/retry/wall timing, classified outcomes,
HTTP status, exhaustion, repair/retry counts, and each usage field's value,
knownness, and completeness. It omits URLs, credentials, payloads, raw provider
errors, and prompt text. Identifiers themselves are metadata, not a secret channel.
Use terminal records once for logical totals; retain attempt records for diagnosis
without adding them to terminal totals. A known zero is distinct from unknown;
partial totals remain incomplete after unmetered failed attempts. These observations
are not a billing ledger. Callback code must not wait on its own run's progress.
Production report changes must distinguish logical calls from dispatched provider
attempts. Replace scheduler utilization fields based on global/proposal/validation
pools with per-backend admission/active timing. Preserve deterministic ledger and
validator ordering. Map prompt definition/rendered hashes and selected profile/model
into request metadata, but do not promise equality with legacy prompt hashes or
serialize PromptKit's raw event struct as the stable report contract. Update the
output integration reference, report mapper tests, and downstream migration guide
when the production representation is implemented.
## Verification and Remaining Handoff
The focused matrix is implemented in the nested module:
| Roadmap scenario | Evidence |
| --- | --- |
| Successful assembled path | `go run .`; `TestWorkflowBoundaries/repair_both_roles`; actual Audita domain helpers and both selected profiles. |
| Presence, overrides, inheritance | `TestConfigurationContract`: omitted role/backend values, explicit zero, null/empty, version, unknown keys, and explicit overrides. |
| Credentials | Missing/unset and empty references fail; optional null accepted; effective/profile inspection redacted; HTTP authorization verified; no secret/prompt text in observations. `TestProfilesRespectSharedBackendBoundary` rejects competing lookup and capacity bypass. |
| Both roles repaired | `repair_both_roles`: four attempts, two repairs, one accepted correction. |
| Repair exhausted | Proposal and assessment exhaustion subtests: bounded counts and selected warning/rejection outcomes. |
| Semantic boundary | Missing span, missing/duplicate decision indices, negative assessment; no semantic repair loop. |
| Retry/zero/exhaustion | 429 recovery, zero-budget 503, positive-budget exhaustion after completed proposal; typed error assertions. |
| Combined budgets | 503 → invalid JSON → repair 503 → valid proposal → assessment: five attempts, two transient retries, one repair. |
| Shared capacity | `TestSharedCapacityAndRecovery`: caps four and two, both roles, a retry and repair, queue rejection, measured peak equal to cap, later successful call. |
| Waiting/cancellation/deadline | `TestQueuedCancellationAndDeadline`; `TestDeadlineStopsRetryAfter`: no queued dispatch after cancellation/deadline and a 60-second provider retry delay interrupted by the operation deadline. |
| Failed-run observations | `TestPartialUsageOnFailedRun`: valid HTTP output with known-zero usage followed by repair failure; nil result, retained events, incomplete known aggregate, correct counts. |
| Representative assets | `TestRealPromptRenderingParity` compares both rendered message pairs to Audita v1.1.0; `TestSchemaFixturesMatchAudita` compares schema bytes. |
Commands executed from `examples/library-adoption` on Linux amd64 with Go 1.26.8:
```sh
GOWORK=off go mod download all
GOWORK=off GOPROXY=off GOSUMDB=off go mod verify
GOWORK=off GOPROXY=off GOSUMDB=off go run .
GOWORK=off GOPROXY=off GOSUMDB=off go test -race ./...
GOWORK=off GOPROXY=off GOSUMDB=off go vet ./...
```
All passed. The demonstration reports accepted=1, rejected=0, attempts=4,
transient_retries=0, structural_repairs=2. Module verification reports all modules
verified. Published tag availability was checked with `git ls-remote --tags` for
ConfigKit v0.3.1 and PromptKit v0.11.0; successful module acquisition verifies this
is not merely a local-checkout experiment. The module has no replacements.
The minimum Go 1.25.5 toolchain was also exercised with offline tests and the same
successful demonstration. The exact installed-toolchain invocation was:
```sh
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off /home/eric/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.5.linux-amd64/bin/go test ./...
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off /home/eric/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.5.linux-amd64/bin/go run .
```
An initial attempt to select that cached toolchain through `GOTOOLCHAIN=go1.25.5`
with checksum verification disabled was rejected by Go's toolchain verifier. Using
the installed binary with `GOTOOLCHAIN=local` proved offline execution without
weakening dependency checks. The portable README commands assume an installed
sufficiently new Go executable on PATH and disable toolchain switching.
From the Audita root, `go test ./...` and
`go build -o ./bin/audita ./cmd/audita` passed. These check existing production code;
they do not cover the nested module. Affected relative documentation links,
inventory coverage, and `git diff --check` were checked separately.
### Findings and Limits
No adoption-blocking upstream API gap was found. Two integration details deserve
explicit attention: PromptKit's profile GoDoc calls `timeout_seconds` a generation
deadline, but its built-in HTTP path applies it separately to each attempt; the
consumer therefore needs the selected total-operation deadline. Also,
`APIKeyRequired` is an in-memory profile facility, not a profile-YAML field. The
file-profile design uses the presence of ConfigKit's optional credential reference
to require local secret resolution; it does not invent a new PromptKit YAML key.
The chosen defaults are proposed operational policy, not measured model-quality or
cost recommendations. Tests use controlled transport responses and synthetic data;
they do not establish real provider compatibility, model quality, network behavior,
throughput, cross-process coordination, complete billing, or release compliance.
Only the representative LLM-facing subset is implemented in the prototype; the
inventory specifies the remaining domain fields for the ConfigKit migration.
Production file discovery, CLI parsing, full module chains, all nine prompts,
filesystem diagnostics/redaction, and serialized report migration remain their
assigned production tasks. Those are deliberate scope boundaries, not missing
claims of completion for the prototype.
The new profile/credential/capacity and deadline choices are explicit above; no
unresolved upstream requirement prevents starting the ConfigKit implementation
plan. Production report field names and final runtime wiring are selected in the
[implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md), following the outcome and observation
contract here. The coordinated refactor targets both libraries directly; intermediate commits do not require a working legacy runtime.
### Production Implementation Reference
The [implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md) owns ordered production work,
component changes, tests, and acceptance criteria. The [combined roadmap](library-adoption.md)
and [configuration roadmap](configkit.md) own the final target. This evidence does
not require an interim runtime bridge or separately runnable configuration migration.
Retained domain scope includes transcript/glossary schemas, normalization/chunking,
output encoding, proposals and apply-time checks, deterministic validators, ordered
modules, LLM-validator batching/cardinality, ledger ordering, and artifact lifecycle.
The production integration replaces loading, provider execution, prompt rendering,
and scheduler composition while preserving those responsibilities. Retention and
timestamp changes remain separate work.
### Distribution Follow-Up
The pinned graph includes ConfigKit, PromptKit, both PromptKit backend catalogs,
jsonschema/v6, yaml.v3, yaml.v4, x/text, and kr/text (including module/test dependency
edges). Before distribution, review the actual production dependency graph and
its license files, including PromptKit/catalog LGPL-3.0-or-later terms. Decide and
verify the notices, license copies, corresponding library source/modifications,
and rebuild/relink materials needed for Audita's Go binary/container distribution.
This unit does not publish binaries or claim that release compliance is complete.
No upstream relicensing request or library implementation change is required for
this prototype.
## Selected-Overlay Extension: 2026-09-24
The isolated prototype now pins published ConfigKit v0.4.0. ConfigKit loads
`overlays/testing.yml` relative to the selected main file; the overlay selects
complete PromptKit execution profiles from the separate `profiles/` directory.
The original v0.3.1 observations above are historical and do not prove this
extension. No production Audita behavior changed.
With Go 1.26.8 on linux/amd64, from `examples/library-adoption/`, these offline
checks passed with `GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off`:
```sh
go test ./...
go test -race ./...
go vet ./...
go run .
go run . --overlay testing
go run . --overlay testing --proposal-profile proposal
```
Each demonstration reported one accepted correction, four attempts, zero
transient retries, and two structural repairs. The base run selected
`synthetic-grammar`/`synthetic-review`; testing selected
`synthetic-grammar-testing`/`synthetic-review-testing`; the explicit proposal
override restored only `synthetic-grammar`. These are synthetic routing IDs,
not price, latency, quality, or provider-compatibility measurements.
| Requirement | Consumer evidence |
| --- | --- |
| Base, empty/default, testing selection, role overrides, final inheritance, and relative profile resolution | `TestSelectedOverlayConfiguration` loads temporary on-disk layouts and assembles both roles. |
| Actual base/testing/override HTTP models and repairs | `TestDemonstrationSelectsOutboundModels` captures every request body and matches model IDs to the reported selections and domain outcome. |
| Missing/invalid selection, version/directive rejection, and ordinary backend-setting validation | `TestOverlayFailuresAndApplicationValidation` checks ConfigKit diagnostic codes and effective backend limits. |
| Surviving credential resolution, absent/empty failures, null unauthenticated use, and profile-only switching | `TestOverlayCredentialResolutionAndInspection` controls synthetic environment values and observes HTTP Authorization. |
| Redacted effective output, explanation and metadata origin, profile inspection, and observations | `TestOverlayCredentialResolutionAndInspection` and `TestOverlaySelectionProvenance` inspect safe output and overlay/CLI origins. |
| Existing retry, repair, domain, capacity, and concurrent-observation behavior | `TestWorkflowBoundaries`, `TestSharedCapacityAndRecovery`, `TestQueuedCancellationAndDeadline`, `TestDeadlineStopsRetryAfter`, and `TestPartialUsageOnFailedRun` pass, including under `-race`. |
Changing a profile does not remove a backend credential reference from the
effective configuration. ConfigKit validates a surviving reference when only
profile selection changes. Replacing the reference in an
overlay resolves only the replacement; explicit null allows unauthenticated use.
This distinction is part of the selected configuration boundary, not an
application fallback. Imports and secret-directory bootstrap remain disabled.
The tests use a controlled built-in HTTP transport and synthetic credentials;
there were no live provider calls. The prototype still covers one representative
proposal/assessment pair, not all production modules, prompts, CLI paths, or
diagnostic artifacts. Root `go test ./...` does not run this nested module. No
upstream API gap appeared in this extension. The [production implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md)
keeps implementation and release pending.

269
docs/roadmap/library-boundary.md
View File

@@ -1,269 +0,0 @@
# Library Adoption: Shared Boundary and Prototype
**Last updated:** 2026-09-24
**Status:** Original boundary-design and offline-prototype unit completed;
overlay extension and production-planning handoff completed, 2026-09-24.
The work packages below retain the original scope;
[decisions, evidence, limitations, and handoff](library-boundary-findings.md)
record the outcome. The executable prototype was removed after production
integration; Git history at `590604b` retains it. No publication or upstream
changes were made in that historical unit.
## Next Unit: Testing Overlays and Production Migration Plan
**Status:** Complete. The [production implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md)
owns subsequent work; the [overlay findings](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
record this unit's checks. The original work packages below remain
historical scope for the completed first unit.
Extend the existing prototype to prove the [selected-overlay contract](configkit.md#selected-overlays-for-testing)
with ConfigKit v0.4.0, then turn the verified findings into a detailed production
ConfigKit migration plan. Do not repeat the original inventory or redesign the
established library boundary without new evidence.
Required work:
- Upgrade only the nested prototype module from ConfigKit v0.3.1 to v0.4.0.
- Pass an explicit overlay selection through ConfigKit and add `overlays/testing.yml`
plus two PromptKit testing profiles. Select both roles and retain one shared
finite backend. Synthetic model names demonstrate routing; they do not benchmark
real model cost, speed, or quality.
- Make the offline executable accept overlay and explicit role-profile selections,
and show the selected models alongside domain outcomes and retry/repair counts.
- Verify base/overlay/CLI precedence, role inheritance, invalid selections,
surviving credential references, safe inspection, and overlay provenance.
Prove the actual HTTP request models for both roles; loading changed strings
alone is insufficient evidence.
- Record new results separately from the historical v0.3.1 findings. Expand the
existing ConfigKit roadmap into a concrete production implementation handoff,
including final runtime integration and removal of legacy configuration.
Target end state: reviewers can run the prototype with and without the testing
selection and see both LLM roles switch models. The offline test suite establishes
that selection, inspection, and execution agree, and the production ConfigKit work
has bounded tasks with explicit acceptance criteria.
Acceptance criteria:
- The published v0.4.0 dependency is pinned without local replacements; the original
no-overlay path and existing integration tests continue to pass.
- `testing` switches both roles, explicit profile flags override its selections,
and omission/empty/`default` retains base selection according to ConfigKit.
- Invalid overlays fail clearly; inheritance and secret resolution use surviving
effective settings. Inspection records overlay origin without disclosing keys.
- Both selected testing models reach the controlled HTTP transport; structural
repair, shared capacity, observations, and Audita domain checks remain active.
- Updated examples, executable instructions, findings, and a detailed production
migration plan are reviewable. Exact verification commands and limitations are
recorded, including the fact that root tests do not cover the nested module.
- Production code, root dependencies, and upstream libraries remain unchanged.
No release or live provider call is part of this unit.
This unit ends with a production migration **plan**, not production implementation.
The ConfigKit → PromptKit → combined release sequence remains unchanged. Imports
and secret-directory bootstrap remain disabled; PromptKit execution profiles keep
their existing names and meaning.
## Objective
Define and prove the boundary between Audita, ConfigKit, and PromptKit before
replacing Audita's production configuration or LLM runtime. This unit covers the
shared design and representative prototype supporting the
[coordinated adoption roadmap](library-adoption.md). The
[ConfigKit roadmap](configkit.md) continues to own the broader configuration scope.
The deliverable is a concrete target contract supported by an offline executable
prototype, sufficient to implement the ConfigKit migration without subsequently
redesigning its LLM-facing settings. A successful prototype is not completion of
either production migration.
## Established Direction
- This is an intentional breaking migration with a migration guide, not legacy
configuration support, aliases, or a permanent compatibility mode.
- ConfigKit owns shared configuration mechanics; PromptKit owns shared prompt and
generation mechanics; Audita retains domain meaning and transcript correction.
- Structural repair is enabled from the outset for proposal and LLM-validator calls.
Positive default repair budgets must be selected, not silently left at zero.
- Transient retries and structural repair are separate. Explicit zero disables
transient retries; Audita must not add a second automatic retry loop.
- One finite concurrency limit per backend protects local capacity and bounds paid
API activity. Calls from both roles share that limit. Fine-grained role budgets
and an additional global concurrency control are unnecessary.
- Keep the sequence: boundary/prototype, ConfigKit migration, PromptKit migration,
then combined major release. No intermediate public release is planned.
## Scope and Deliverables
| Deliverable | Required result |
| --- | --- |
| Configuration inventory and migration map | Every existing setting and invocation option has a proposed destination, replacement, or removal, with rationale. |
| Boundary specification | One authoritative source for each setting, explicit precedence/defaults, credential flow, runtime conversion, and failure handling. |
| Representative executable prototype | New-style configuration drives one proposal and one assessment through released libraries and a controlled provider. |
| Focused behavioral evidence | Retry, repair, capacity, credential, observation, and domain-boundary scenarios below are demonstrated. |
| Decision and integration handoff | Concrete choices, measured limitations, affected components, removal plan, and next implementation tasks are recorded. |
Do not switch the production CLI to the prototype, migrate every module/prompt,
redesign transcript/glossary formats, change correction algorithms, modify retention
or timestamp behavior, or publish a release in this unit. Do not reproduce the
libraries' full test suites. Upstream changes require a separately scoped task if
an actual gap is found.
## Work Package 1: Inventory and Decide Ownership
Read the current [configuration](../config.md), [CLI](../cli.md),
[LLM runtime](../internal/llm-runtime.md), [prompts](../internal/prompts.md),
[validators](../internal/validators.md), and [output contracts](../integrations/output-files.md).
Use graph-first discovery to verify the relevant code and tests, especially
configuration application, prompt construction, structured responses, scheduling,
and report assembly.
Build a migration table grouped by application/domain settings, invocation state,
backend settings, execution profiles, output contracts, and inspection. Include
current environment settings and aliases without treating their existence as a
requirement to preserve them. Identify report fields whose meanings change when
one logical generation contains multiple provider attempts.
Resolve these design decisions explicitly:
| Decision | Proposed direction to prove |
| --- | --- |
| File selection and version | Audita selects the main configuration and validates a new schema version; specify discovery and the treatment of old versions. |
| Application versus invocation state | ConfigKit loads persistent settings; Audita parses invocation-only input/output paths and translates explicitly supplied setting flags into supported overrides. |
| Backend identity | One registration represents the service capacity shared by proposal and validation calls. Avoid separate identities for the same service merely because roles differ. |
| Endpoint and capacity ownership | Define them once per backend, with finite limits and an explicit policy for waiting capacity/exhaustion. Do not expose endpoint-only unlimited execution accidentally. |
| Execution profiles | Prefer complete application-owned profiles using PromptKit conventions. Decide their source and selection without duplicating model/settings fields in application configuration. Catalog selection remains a deliberate option. |
| Role selection and inheritance | Specify how proposal and validation choose profiles, including omission and any inheritance. Do not inherit old behavior solely for compatibility. |
| Credentials | Prefer one ConfigKit-owned secret-resolution path passed through PromptKit's supported request credential field. Define backend/profile credential association and avoid competing environment lookups. Validate this against profile requirements. |
| Retry, repair, and deadlines | Assign one owner and explicit defaults for each control. Use positive structural-repair defaults and document combined limits and total-operation scope. |
| Optional library features | The original prototype omitted imports, selected configuration layers, and secret-directory bootstrap. Selected overlays are now planned for testing; see the [overlay plan](configkit.md#selected-overlays-for-testing). Imports and secret-directory bootstrap remain out. |
| Inspection | Define effective configuration and selected-profile inspection without implying that one library's snapshot contains the other's complete state. |
Choose concrete field names, example values, defaults, units, and omission/null/zero
semantics for the prototype. In particular, decide what validation and inspection
do when configured credentials are unavailable. Classify choices as proposed,
selected, or unresolved; do not present candidates as implemented contracts.
## Work Package 2: Specify Execution and Reporting Semantics
Document the flow from selected configuration through backend/profile assembly,
prompt inputs, generation, structural validation/repair, typed decoding, and Audita
domain validation. Identify which values cross the boundary and which component
owns each conversion. Keep library types out of unrelated domain code where a
small existing domain value is sufficient.
Use the current correction-set and validator-decision schemas as representative
inputs. Reuse domain checks where practical. Select one real proposal/validator
pair; account for prompt helpers, template variables, shared fragments, and schema
references rather than testing only an unrelated hello-world prompt.
Record the following distinctions:
- Repair seeks output conforming to its structural contract. Success does not
establish semantic correctness or authorize applying a correction.
- A schema-valid decision set still requires Audita's cardinality and proposal
identity checks. Semantic rejection is not automatically another repair request.
- Exhausted structural repair yields failed content validation; provider failures,
retry exhaustion, cancellation, and capacity rejection have separate outcomes.
- Specify each outcome's intended effect on an Audita section/batch and overall run,
including warnings, rejected work, partial domain results, and terminal failures.
Preserve domain intent without preserving obsolete transport behavior by accident.
- For transient retry budget T and structural repair budget R, one logical call can
dispatch up to `(T + 1) * (R + 1)` provider attempts. For example, T=2 and R=2
permits nine attempts; these illustrative values do not select production defaults.
- Choose the total-operation deadline scope for a logical proposal or assessment;
distinguish it from per-attempt timeouts and whole-process cancellation. Provider
retry timing may exceed local backoff, so deadline behavior must be explicit.
Design the Audita observation record and aggregation rules. Include logical-call
and attempt identity, role/backend correlation, queue/generation/retry timing,
repair and retry counts, outcomes, and available usage. Preserve unknown/incomplete
usage rather than inventing zeros or complete costs. Do not add terminal totals to
attempt totals. Collect events safely across concurrent calls, without blocking
callbacks on progress by the same run or persisting raw library events as a stable
Audita schema. Transcript-bearing artifacts remain separately governed by Audita.
## Work Package 3: Build the Bounded Offline Prototype
Use ConfigKit v0.3.1 and PromptKit v0.11.0 as reviewed starting points; verify their
published availability and APIs before pinning the prototype. Recheck the combined
Go minimum (the reviewed PromptKit release requires Go 1.25.5). Do not depend on
sibling checkouts, an active Go workspace, or committed local module replacements.
A proposed home is `examples/library-adoption/` as an explicitly experimental,
standalone Go module, isolating prototype dependencies from the production binary.
Confirm this placement during execution. Its README must give exact offline run
and test commands after dependency acquisition; root `go test ./...` must not be
claimed to cover a nested module. Keep complete proposed configuration and prompt
fixtures with it and clearly label them as prototype-only.
The prototype should:
1. Load representative application settings with ConfigKit, using synthetic secret
values supplied by the test environment and an application-owned version rule.
2. Assemble shared backend registrations and proposal/validation profile selections.
3. Supply representative transcript/proposal inputs to PromptKit definitions and
actual response schemas using supported sources.
4. Use the built-in HTTP generation path with a controlled local endpoint or
transport. An injected LLM client alone cannot prove built-in HTTP retry behavior.
5. Run proposal generation, decode and check candidates, run assessment, and perform
the selected Audita decision checks without modifying real transcript files.
6. Collect selected safe observations and show a compact result understandable to a
reviewer, including repaired output and final domain acceptance/rejection.
No paid calls, real credentials, host environment assumptions, persistent caches,
or general-purpose test framework are needed. Make concurrency and cancellation
checks deterministic through controlled synchronization. Keep delays bounded and
use library-supported controls; do not patch library internals to make tests pass.
## Focused Verification Matrix
| Scenario | Evidence required |
| --- | --- |
| Successful assembled path | ConfigKit-loaded selections drive both real prompt roles through PromptKit and Audita checks. |
| Presence, overrides, and inheritance | Representative omission, explicit values, and role selection follow the proposed contract; unknown fields and unsupported versions fail clearly. |
| Credentials | Required missing/empty references fail as intended; controlled calls receive the chosen synthetic credential; inspection and observations do not expose it. |
| Structural repair succeeds | Invalid proposal output and invalid assessment output are repaired in their respective workflows; positive budgets are actually exercised. |
| Repair exhausted | Repeated invalid output produces the selected Audita outcome and accurate accounting, without an unbounded outer retry. |
| Domain rejection | Structurally valid but unacceptable correction or decision is rejected by Audita, demonstrating the semantic boundary. |
| Transient retry and explicit zero | An eligible temporary status recovers; zero makes one attempt. Exhaustion retains a useful classified cause. |
| Combined repair and retry | A repaired generation encountering a transient failure respects both budgets and reports distinct counts. |
| Shared capacity | Concurrent calls from both roles, including retry/repair activity, stay within one selected backend cap; test a representative local cap of four and a finite API cap. |
| Waiting, cancellation, and deadline | Cancellation/deadline stops pending work and retry waits; capacity exhaustion has the intended application outcome and later calls can still proceed. |
| Failed-run observations | Completed work remains observable after later failure; unknown usage, known zero, and aggregate counts are mapped correctly. |
Tests should protect Audita's integration decisions and important assembled behavior.
Rely on PromptKit's tests for exhaustive HTTP-status, backoff, and scheduler details;
rely on ConfigKit's tests for its parser and presence machinery. Use race testing for
new concurrent observation collection. Keep consumer checks offline and credential-free.
## Completion Criteria and Handoff
This unit is complete only when:
- The settings/ownership table and migration inventory are concrete enough to start
the ConfigKit migration, including file/profile sources, CLI scope, credentials,
defaults, finite backend limits, positive repair budgets, and deadline policy.
- The offline prototype and focused matrix demonstrate the selected boundary with
both libraries together, including both LLM roles and Audita domain checks.
- Exact verification commands, results, limitations, and unresolved upstream gaps
are recorded. Missing evidence is not reported as successful integration.
- No adoption-critical decision remains hidden in prototype code. If a gap prevents
settling the boundary, record it and revise the plan before declaring readiness.
- The existing Audita production behavior remains intact in this unit. Temporary
code is clearly isolated and has a removal/integration destination during the
production migrations; no second supported runtime is introduced.
- A component-level follow-up identifies code to replace, domain code to retain,
tests to update, documentation/migration-guide work, and direct integration of
the final configuration and PromptKit runtime.
- Licensing/distribution follow-up for the selected dependency graph is identified
for the later release stage; this prototype does not claim release compliance.
- Roadmap links and examples resolve, appropriate checks pass, and commits contain
only this unit's work. Significant settled architectural choices may be recorded
in an ADR under the [documentation policy](../policy/documentation.md), while
implementation status stays in the roadmaps.
Retain the overall adoption order unless evidence or the other planned large changes
justifies revising it. Completing this unit enables detailed production migration;
it does not authorize an intermediate release or require unrelated runtime fixes.

109
docs/roadmap/narratio-feature.md
View File

@@ -1,109 +0,0 @@
# Independent Output Destinations for Narratio
**Last updated:** 2026-10-03
**Status:** Implemented and verified. This roadmap records the external behavior
and the reasons for it. The [implementation plan](implementation.md) records the
completed work; the [Narratio request](feature-request-narratio.md) provides the
consumer context.
## Purpose
Allow one `audita process` invocation to place the completed transcript,
ordinary reports, and enabled debug captures at independently selected
locations. Narratio can then keep completed products, ordinary run diagnostics,
and detailed debug evidence in its own workspace without relocating Audita's
files after the run. Audita continues to own report formats and its debug
working-state lifecycle; Narratio owns its destination paths and operator views.
## Current Baseline
- `--output FILE` writes the transcript to `FILE`, or stdout when omitted.
- Ordinary reporting is enabled by default. With file output, `errors.yml`,
`recoveries.yml`, and `report.md` go beside the transcript. With stdout output,
`--report-dir DIR` or `reporting.stdout_dir` selects their directory. An
explicit `--report-dir` also selects reports for file output; without it, the
configured `reporting.stdout_dir` value remains stdout-only.
- Reports are initialized before input processing, then updated on completion or
failure where possible. Early configuration or destination failures can occur
before report initialization and therefore leave no report files.
- `--work-dir DIR` already overrides the debug root independently. It does not
enable debug capture. Capture is enabled by `diagnostics.debug` or an explicit
`--debug`; when enabled, Audita retains a `run-<id>` directory beneath the
selected root. Debug remains optional, including after unsuccessful runs.
- The output session receives distinct transcript and report paths and checks
active destinations for conflicts with each other and protected inputs. The
CLI selects the report destination while the output session retains its
existing collision checks, preflight, initialization, and finalization.
The current contracts are in [CLI](../cli.md), [configuration](../config.md),
[operations](../operations.md), [subprocess guidance](../integrations/subprocess.md),
and [output files](../integrations/output-files.md).
## Targeted Behavior
An explicit `--report-dir DIR` selects the destination for ordinary reports
whether the transcript goes to `--output FILE` or stdout. With all three paths
selected, Audita writes the transcript to `FILE`, any enabled ordinary reports
to `DIR`, and any enabled debug captures beneath the `--work-dir` root. The
report and debug destinations may be different from the transcript's parent
and from each other.
| Selection | Transcript | Ordinary reports when enabled | Debug when enabled |
| --- | --- | --- | --- |
| No destination override | stdout | `reporting.stdout_dir` (default `.`) | `diagnostics.work_dir` (default `/tmp/audita`) |
| `--output FILE` only | `FILE` | Parent of `FILE` | Configured/default debug root |
| `--report-dir DIR` only | stdout | `DIR` | Configured/default debug root |
| `--output FILE --report-dir DIR --work-dir ROOT` | `FILE` | `DIR` | Beneath `ROOT` |
An explicit CLI report directory takes precedence over file-output placement.
Without that explicit flag, `reporting.stdout_dir` remains a setting for stdout
output only. This preserves existing file-output defaults and avoids changing
the meaning of a configuration field outside Narratio's CLI request. A config
file or overlay can continue to enable or disable ordinary reporting and debug
capture under the existing precedence rules. Selecting `--report-dir` or
`--work-dir` alone does not enable either category. An explicit `--debug` can
enable capture; no new debug-disable control is needed.
The selected report directory applies to whatever reports Audita currently
produces on both successful and unsuccessful runs. It does not make new failure
conditions produce reports, change early-failure behavior, or guarantee that an
unwritable destination yields a complete bundle. Existing per-file replacement,
failed-status finalization, output ordering, exit codes, and redacted path
announcements remain the governing contracts.
Active destinations retain existing safety rules: report files must not collide
with the transcript or protected inputs, and their directory must satisfy the
current preflight requirements. With reporting disabled, the report directory
is inactive and should not be probed merely because it was selected. With debug
disabled, the debug root is inactive and should not be created merely because it
was selected. No new rule requires the three roots to be distinct if their
actual output files do not conflict.
Audita continues to write stdout and stderr under its existing stream contract.
Narratio captures those streams itself. Audita retains its report filenames and
its nested debug layout.
## Acceptance Criteria
- A CLI caller can provide `--output FILE --report-dir DIR --work-dir ROOT` in
one invocation. The transcript appears at `FILE`, enabled ordinary reports at
`DIR`, and enabled debug captures beneath `ROOT`.
- The same explicit report directory works with stdout transcript output and
applies to reports produced after an unsuccessful run.
- Omitted overrides preserve the current stdout and file-output placement;
disabled reporting and debug do not create their respective artifacts solely
because destination flags were supplied.
- Existing protection against conflicting destinations and source overwrites,
report/debug formats, output lifecycle, streams, and exit behavior still hold.
- Current documentation and focused tests describe and verify the new CLI
contract. No Narratio-specific path convention or Audita-managed latest-run
view is introduced.
## Scope Boundary
This request changes external CLI destination selection. It does not call for
new output formats, config-file fields, a relocation or retention service,
changes to cache ownership, additional stream-redirection flags, or new debug
capture controls. Future release versioning follows Audita's release policy;
this roadmap does not itself schedule or publish a release.

254
docs/roadmap/policy.md
View File

@@ -1,254 +0,0 @@
# Standard Policy Alignment
**Last updated:** 2026-09-28
**Status:** Implemented. This roadmap replaces its earlier licensing-only scope
and records the accepted end state. The
[historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/61bc467a03da17d52481faa5db96d9e048815be7/docs/roadmap/implementation.md)
records the ordered work and verification; the adopted policies are in
`docs/policy/`.
## Purpose and Review Revision
Align Audita with [Standard Repository Policies](https://gitea.maximumdirect.net/eric/policy),
using identical policy language except where a substantive, locally significant
requirement or decision justifies a variation. Adopt upstream wording verbatim
for non-substantive differences. Perform the required template customizations
without inventing application contracts, restrictions, or guarantees.
The accepted review compared Audita's current documents directly against upstream
commit **`d84a38ebab26f048cafbe59a2246700c35d29843`**, beginning with the
[adoption and update guide at that revision](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/adoption.md).
Use that revision for this adoption. Historical adoption baselines need not be
reconstructed or assessed. A later upstream revision requires deliberate review,
not an automatic substitution during implementation.
The public upstream repository, discovery links, and copied-policy attribution
are part of the accepted direction. This supersedes the earlier instruction to
omit references to the policy repository.
## Current Baseline
- Audita has five adopted policies: agents 1.0, architecture 1.0, documentation
1.1, testing 1.0, and release 1.0. None has per-document `Upstream Source`
metadata. `meta.md` and `oss.md` are absent.
- [AGENTS.md](../../AGENTS.md) routes readers to the contributor guide and agent
policy. [docs/development.md](../development.md) has component-specific reading
guidance, but lacks the upstream selective Policy Reading Guide and currently
directs all tasks to the architecture policy.
- [COPYRIGHT](../../COPYRIGHT) and [LICENSE](../../LICENSE) already establish
Audita's GPL-3.0-or-later application grant and full GPLv3 text. The grant does
not yet identify the scope of separately licensed copied policy documents.
- There is no canonical `docs/oss/attribution.md` or local dependency license-text
collection. Relevant component notices still require an evidence-based inventory.
- The [README](../../README.md) lacks the prescribed final licensing statements.
The [release procedure](../release.md) lacks licensing-footer instructions, and
the [release workflow](../../.woodpecker/release.yml) does not explicitly reuse
checked-in release notes as hosted release bodies.
- Audita already releases through one publication command with an atomic source
and tag push. Linux amd64 and arm64 are required; macOS arm64 is best-effort
development support. Successful publication completes the release without
waiting for CI or artifact delivery.
## Targeted Policy Set and Justified Customizations
All seven policy files belong under `docs/policy/`. The versions below are those
at the selected upstream revision. No existing policy is wholly unchanged;
existing architectural decisions and platform commitments need no substantive
revision merely to align the text.
| Policy | Target version | Required alignment and justified local treatment |
| --- | --- | --- |
| `agents.md` | 1.3 | Adopt upstream substantive language verbatim. Retain Audita's graph-first Codebase Discovery section because it is an explicitly selected repository workflow. |
| `architecture.md` | 1.3 | Adopt general ownership, boundary, and constraint guidance, then perform the template's required application-specific customization. Preserve Audita's ordered pipeline, ConfigKit/PromptKit responsibilities, deterministic processing, recovery, concurrency, and secret-handling decisions. These describe real project requirements rather than stylistic preferences. |
| `documentation.md` | 1.11 | Adopt upstream prose and applicable ownership rows. Retain concrete output-contract ownership, contributor change recipes, and the decision to place operating/recovery guidance in operations instead of restoring troubleshooting documentation. Omit HTTP API and public Go API ownership rows because Audita has neither responsibility. |
| `testing.md` | 1.3 | Adopt substantive text verbatim. Its distinction between established contracts and invented test requirements needs no local exception. |
| `release.md` | 1.4 | Adopt upstream substantive language. Preserve Linux amd64/arm64 requirements, best-effort macOS arm64, unsupported Windows/macOS amd64, and Audita's explicit decision against human release gates. These are selected support and workflow commitments. |
| `meta.md` | 1.0 | Adopt provenance, local authority, and public discovery guidance with accurate metadata and local attribution navigation. No additional substantive exception is needed. |
| `oss.md` | 1.8 | Adopt the House Format and maintenance rules, with required project-specific facts and reference links. Preserve Audita's GPL-3.0-or-later application grant, Gitea host, and existing distribution channels. |
Remove each `Base Policy Template` section after carrying out its instructions.
Keep local customizations identifiable and explain their project-specific reason
in the appropriate policy; do not maintain alternate paraphrases of shared rules.
Architecture customization does not authorize a package redesign or additional
threat-model guarantees. Supporting documents are warranted by existing
responsibilities and substantive content, not by unused template examples.
Successful-tag-push completion, asynchronous artifacts, minimal automated gates,
and proportionate validation are now shared upstream rules. Replace Audita's
parallel explanations with the upstream language rather than describing these
as continuing exceptions. Keep concrete commands, remotes, build targets, and
publication mechanics in the release procedure with appropriate policy links.
## Provenance and Selective Reading
For every adopted policy, retain the selected upstream `Base Policy Version`,
set `Last Updated` to its actual local adoption date, and introduce `Upstream
Source` linking to that exact file at the selected full commit. The URL pattern is:
```text
https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/<filename>.md
```
Substitute each of `agents`, `architecture`, `documentation`, `testing`, `release`,
`meta`, and `oss`; a repository-root or shared generic source link is insufficient.
These links describe the source actually incorporated, not a reconstructed claim
about older local documents. Git and per-document metadata are sufficient; no
separate synchronization registry or recurring check is required.
In `meta.md`, retain these public discovery destinations:
- [Standard Repository Policies](https://gitea.maximumdirect.net/eric/policy).
- [Current adoption and update guide](https://gitea.maximumdirect.net/eric/policy/src/branch/main/docs/adoption.md).
Discovery links follow the current upstream project and guide. Adopted-source,
notice, and example links remain commit-pinned. Preserve valid pinned example
references supplied by the selected policy, or pin them to this review revision
after checking the referenced assets; do not turn them into moving-branch links.
Adapt upstream `agents/AGENTS.md` into Audita's root orientation, retaining its
existing applicable instructions. Add the adoption guide's Policy Reading Guide
to `docs/development.md` and retain the component-specific task guide. Route
policy adoption and updates to `meta.md`, repository recommendations and changes
to agents, application-code or structure work to architecture, validation choices
to testing, documentation work to documentation, dependency/licensing work to
OSS, and release/versioning work to release policy. Read applicable policies
before their decisions, reuse unchanged context, and expand reading with scope.
Do not require reading the entire suite for every task.
## Copied Policy Licensing and Local Material
Follow the selected revision's
[canonical policy attribution entry](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/examples/licensing/policy-attribution.md)
and [project grant](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/COPYRIGHT).
Insert the entry beneath the title in `docs/oss/attribution.md`, substitute the
full review commit in all three URLs, and accurately identify copied/adapted
material, including orientation or reading-guide text where copied. Preserve
supplied creator, copyright, license, disclaimer notices, and modification
indications. Keep the material statement concise.
Copy `CC-BY-SA-4.0.txt` into `docs/oss/licenses/`. Add the adoption guide's scoped
notice to `COPYRIGHT`, identifying the adapted policies as CC-BY-SA-4.0 and
routing readers to attribution for the affected material. Account accurately for
other copied guidance in that scope. Audita's application code remains
GPL-3.0-or-later; unrelated documentation is not automatically relicensed.
The root GPLv3 `LICENSE` needs no change.
| Material | Destination or treatment |
| --- | --- |
| Applicable policies, including `meta.md` and `oss.md` | Copy and customize under `docs/policy/`, retaining provenance. |
| Upstream `agents/AGENTS.md` | Adapt into root `AGENTS.md`. |
| Selective reading guidance | Integrate into Audita's own `docs/development.md`. |
| Upstream README, root AGENTS.md, developer guide, and adoption guide | Keep upstream and consult through links; do not copy as local guides. |
| Licensing example collection | Keep upstream with commit-pinned references; use it to create actual notices. Original example formatting is CC0 and alone needs no policy attribution entry. |
| Applicable standard and companion license texts | Copy complete, unmodified texts locally; no blanket collection. |
## Component Attribution
Complete the previously agreed inventory of relevant incorporated or distributed
components: direct and transitive dependencies, copied code/documentation/assets,
and applicable Go runtime notices. Determine inclusion from actual material and
use, not simply presence in `go.mod`; tooling or consultation alone does not
require an entry. Inspect selected sources directly rather than copying a
dependency's attribution document wholesale.
Use the exact House Format in OSS policy 1.8: the prescribed title followed by
component entries, factual identification, applicable supplied copyrights and
SPDX expressions, standard license/notice labels, and `[Source](COMMIT_URL).`
links. Omit unavailable optional fields instead of adding explanatory filler.
Preserve required upstream material, license alternatives, and exceptions.
Keep research narratives, compliance conclusions, and exception reasoning out
of attribution; place reasoning in the OSS policy or an appropriate decision
record. This replaces the older roadmap's permission to put exception reasoning
in a component entry.
Upstream component and notice links identify exact imported commits. Local
standard texts include directly reachable companions, such as GPLv3 for LGPLv3.
Apply the adopted house interpretation of notice links without blanket extra
local-copy requirements or repeated approval. Concrete obligations beyond notice
reproduction receive their own applicable treatment; the format alone does not
certify compatibility or resolve every distribution obligation.
The reviewed dependency revisions remain useful starting evidence, subject to
checking the dependencies selected when implementation occurs:
| Component | Reviewed version | Source commit |
| --- | --- | --- |
| ConfigKit | v0.4.1 | `0adb94f59e44827908eff401d4381a398fe17107` |
| PromptKit | v0.11.1 | `15e40679546f2e21b6f51cc84b6f0a087a8486db` |
| OpenRouter catalog | v2.20260926.0 | `4f731d217e0bc39151c010f50fb5912706e7b6b3` |
| Rakestrawhome catalog | v2.20260926.0 | `48918a3ad39fd99843cda94a86133b1b3881d596` |
These library patches changed licensing/documentation rather than library code
or catalog profiles. ConfigKit and PromptKit identify LGPL-3.0-or-later; the
selected catalogs identify MIT. Other relevant dependencies still need inventory.
## Discoverability and Release Integration
End README with the project-license and attribution statements prescribed by the
OSS policy, using Audita's actual grant. `COPYRIGHT` owns the separately licensed
policy scope; README links to that grant and the attribution document.
Update `docs/release.md` to specify the final `COPYRIGHT | LICENSE | ATTRIBUTION`
links in every new release note and hosted download page. Use absolute Gitea URLs
at the release's immutable tag or source commit. Reuse the checked-in note as the
hosted body where supported; verify the publication plugin's supported mechanism
and make the necessary focused `.woodpecker/release.yml` change. Avoid separately
maintained release prose or notice lists.
Preserve the current atomic source/tag publication approach, supported targets,
and completion point. Align procedure wording with upstream's preparation,
ambiguous-push recovery, and asynchronous retry rules. Handle completed-roadmap
retirement during preparation, preserving planned/deferred work and repairing
links; this roadmap update does not authorize deleting other roadmaps now.
Check intended future-tag paths against the candidate source without requiring
the tag to resolve before publication. Apply the formatting convention
prospectively; leave published tags, notes, artifacts, and checksums intact.
Concrete historical omissions, if discovered, receive deliberate treatment under
their applicable terms and release correction rules.
## Work Required and Acceptance Criteria
Implementation covers the seven policy documents, `AGENTS.md`,
`docs/development.md`, `COPYRIGHT`, `docs/oss/attribution.md`, applicable files in
`docs/oss/licenses/`, README, the release procedure, and focused hosted-body
publication wiring. Update this roadmap's implementation status when appropriate.
The
[historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/61bc467a03da17d52481faa5db96d9e048815be7/docs/roadmap/implementation.md)
owns the editing sequence and verification steps.
The work is complete when:
- Shared policy language matches the selected upstream revision after required
template removal and customization; retained variations have substantive
Audita-specific reasons as identified above.
- All seven policies have accurate versions, adoption dates, and file-specific
commit-pinned source links. Public discovery links and selective orientation
are in place, with no leftover template-only instructions or broken links.
- Copied policy/guidance scope, modification status, credit, CC-BY-SA text, and
root grant notice follow the documented upstream arrangement while preserving
Audita's application license.
- Relevant components have verified House Format entries, exact source/notice
commits, and complete applicable license texts, without editorial commentary.
- README and future release documentation provide the required navigation, and
hosted release content comes from the checked-in note through a verified
publication mechanism.
- Audita's application contracts, architectural decisions, platform commitments,
release completion semantics, and published history are preserved.
- Affected text, links, license copies, metadata, and publication configuration
pass proportionate review and validation. No release is needed solely to test
adoption; behavioral workflow changes receive focused validation if required.
## Scope Boundaries and Open Questions
No policy decisions remain open for the agreed scope. Component-specific notices
and the publication plugin's supported release-body mechanism require factual
verification during implementation. Raise a decision only for a concrete
unresolved obligation or material tradeoff uncovered by that work.
This feature does not require application refactoring, dependency upgrades,
mandatory scanners, a full SBOM, recurring legal approvals, synchronization
checks, additional human release gates, publication-time network checks, or CI
waiting. The documentation convention alone does not require downloadable
license assets, source bundles, or a different binary package format.

161
docs/roadmap/remediation.md
View File

@@ -1,161 +0,0 @@
# Maintainability Audit: Findings and Recommended Changes
**Reviewed:** 2026-09-25, source commit `011b021`.
**Scope:** Recent partial-decision reconciliation, validation context, prompt
alignment, and request-local reference changes, including related existing code.
**Status:** R27–R30 addressed; see the resolution record below. Previous findings and resolution records remain in Git
history through `011b021`.
## Architecture Assessment
The recent changes fit Audita's existing boundaries. Proposal generation,
validation, prompt assets, and outcome reporting already have suitable owners;
this audit does not recommend a new common package or a broad refactor.
The strongest opportunities are shared prompt instructions and request-scoped
validation context preparation. Two smaller improvements concern recovery-reason
classification and partial-result assembly; both predate the latest commits.
These are bounded maintenance recommendations, not new release gates or changes
to correction policy, response acceptance, or reporting semantics.
## R27 — P2: Consolidate Shared Prompt Instructions
**Category:** Repeated behavioral-contract text; risk of future drift.
**Evidence:** The four [proposal prompts](../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml)
repeat literal source-span matching, request-local identity, replacement-text,
source-based proposal, and confidence instructions. The three active
[validator prompts](../../internal/prompts/assets/prompts/validators.editorial_review.yaml)
repeat candidate-reference, supplemental-context, and verdict-confidence guidance.
Recent contract changes required coordinated edits across these assets. The
[prompt registry](../../internal/prompts/registry.go) already provides a shared
hardening fragment supplied through the role-bound adapter.
**Recommended change:** Extract genuinely common passages into embedded fragments
under `internal/prompts/assets/shared`, following the existing fragment-supply
approach. Keep the fragments owned by `internal/prompts` and use the existing
PromptKit rendering path. Keep module-specific examples, replacement-policy
instructions, protected-name exceptions, and semantic criteria in their individual
prompt definitions. Do not merge similar-looking rules with different meanings.
**Acceptance:** Common instructions have one maintained source and appear in all
intended rendered prompts. The existing module and validator distinctions remain
intact, including glossary replace-all behavior and the spoken-word protected-name
exception. Focused rendering checks cover fragment inclusion and representative
module-specific guidance without requiring exact-prose snapshots or introducing
a separate templating framework.
## R28 — P2: Prepare Validation Context Once Per Request
**Category:** Repeated computation and overlapping segment lookup.
**Evidence:** [BuildLLMValidationRequest](../../internal/framework/validators/llm_validators.go)
already indexes transcript segments, but
[validationContext](../../internal/framework/validators/llm_context.go) scans the
transcript again for each candidate. It also calls
[NewProtectedVocabulary](../../internal/framework/validators/protected_terms.go)
for each candidate, rebuilding the same vocabulary and compiling its matching
expression repeatedly. This is duplicated preparation rather than copied code.
**Recommended change:** Prepare a segment-position lookup and protected vocabulary
once per validation request, then pass that prepared context into the existing
context-selection helper. Keep the implementation within `framework/validators`.
Continue selecting glossary evidence for each candidate individually because its
proposed replacement can mention different terms. A process-wide cache or new
context-management package is unnecessary for this cleanup.
**Acceptance:** Candidates in one validation request reuse the prepared lookup and
vocabulary. Context still consists of the available immediate neighbors within the
current chunk and relevant glossary entries in glossary order. Candidate-specific
replacement text still participates in glossary selection. Existing optional-context
trimming, token estimation, and oversized-input handling retain their behavior.
Verify these outcomes with focused behavioral tests rather than assertions about
private helper call counts or unmeasured performance guarantees.
## R29 — P3: Share Execution-Recovery Classification
**Category:** Duplicate reporting rule; predates the latest commits.
**Evidence:** [Proposal generation](../../internal/framework/proposal_generation/generate.go)
and [LLM validation](../../internal/framework/validators/llm_validators.go)
independently translate structural-repair and transient-retry counts into
`output_repaired` and `service_recovered` reasons. The conversion is the same in
both locations; the surrounding success and domain-outcome decisions differ.
**Recommended change:** Place a small conversion helper in the existing
`framework/outcomes` package and use it from both callers. Keep the decision to
record recoveries at the call sites, along with domain-specific reasons such as
filtered validator decisions. Do not combine proposal and validation orchestration
or move domain-success classification into this helper.
**Acceptance:** Both callers use the same execution-count conversion. Existing
reason ordering, successful-operation reporting, and domain-specific additions
remain intact. A partially unavailable validation batch continues to record its
failure and execution counts without acquiring recovery records merely because a
repair or retry occurred. Existing outcome tests should continue to protect these
distinctions.
## R30 — P3: Centralize Partial Validator Result Assembly
**Category:** Repeated early-return construction; predates the latest commits.
**Evidence:** Several early returns in
[LLMBackedValidator.Validate](../../internal/framework/validators/llm_validators.go)
repeat nested slice appends to combine immediate decisions, oversized-input
decisions, and decisions from completed model batches, together with accumulated
operation outcomes. This makes it easier for a future failure path to omit facts
already collected by the validator.
**Recommended change:** Use a small local result-building helper to assemble the
partial result consistently. Keep error handling, batching, success-path ordering,
and cardinality enforcement explicit. A general-purpose result-aggregation package
is not needed.
**Acceptance:** Cancellation and failure paths retain the same previously collected
decisions and operations. Result ordering and cardinality behavior remain unchanged;
the helper does not enforce complete cardinality on interrupted work. Retain the
existing partial-progress and fatal-failure coverage, adding tests only for a
material gap exposed by the refactor.
## Boundaries to Preserve
- Keep proposal and validator reference mapping separate. Proposals can target the
same segment repeatedly; duplicate validator decisions make that candidate
unavailable. Similar indexing arithmetic alone does not justify a generic
reference-management abstraction.
- Keep distinct protected-term policy checks separate. They already share the
vocabulary implementation and intentionally apply different policies.
- Preserve exact source-span matching, request-local references, batching, and
partial-success behavior. Consolidation is not a reason to strengthen contracts
or broaden the accepted correction scope.
## Review Verification and Limits
- This is a source-level maintainability audit, not a finding of demonstrated
production failures or a measured performance assessment.
- No application tests were rerun for the original read-only audit; its roadmap
update used documentation link and whitespace checks. Implementation validation
is recorded below.
- Future remediation should run focused behavioral tests and the broader suite
appropriate to the actual change, following the testing policy. No live-provider
evaluation or new evaluation framework is required by these recommendations.
- The findings above describe the audited baseline. Implementation and validation
are recorded below; no release publication is included.
## Resolution Record
Implemented as focused changes within the existing packages. Proposal and validator
reference mapping and distinct protected-term policies remain separate.
| Finding | Resolution and evidence |
| --- | --- |
| R27 | Common proposal and validation instructions live in two embedded shared fragments, supplied through the existing role-bound PromptKit adapter. Individual YAML definitions retain policy differences and examples. Rendering tests check fragment inclusion once per applicable prompt and representative module-specific guidance. |
| R28 | Validation request construction builds one segment-position lookup and protected vocabulary, reused by context selection for every candidate. A new test verifies that two replacements of the same source span receive their own relevant glossary entries. Existing neighbor, glossary-order, chunk-boundary, and budget tests pass. |
| R29 | Both proposal generation and validation use `outcomes.ExecutionRecoveries`. Call sites still control success and domain-specific recovery reasons. Existing tests cover combined recoveries and failed validation with repair counts but no recovery record. |
| R30 | A local result builder combines collected decisions and operation outcomes for partial returns and completed validation. Sorting and full cardinality checks remain outside it. Existing fatal-batch, partial-progress, runner, and reporting tests pass. |
Validation: `go test ./...` passed, including PromptKit preparation, partial-decision
reconciliation, subprocess reports, and fatal-progress coverage. Documentation
links and whitespace checks passed. No live model evaluation or performance
measurement was performed; the changes make no new model-quality or latency
promise.

290
docs/roadmap/resilience.md
View File

@@ -1,290 +0,0 @@
# Resilient Processing and Focused Run Reports
**Last updated:** 2026-09-24
**Status:** Implemented. This document records the selected behavior and user
intent. Concrete design decisions and completion evidence are in
[historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/a83a518/docs/roadmap/implementation.md).
## Purpose and User Intent
Audita should reliably produce a useful transcript even when its LLM services
cannot improve it. Exhausted retries or unusable model responses must not turn
an otherwise processable transcript into a failed production run.
> Failure of every LLM operation across every stage and chunk still produces a
> successful output containing the deterministically normalized transcript, with
> no LLM corrections applied. Partial success preserves as many fully validated,
> applicable corrections as possible.
Deterministic normalization remains intentional and may change segment layout,
identifiers, and text before LLM work begins. Output uses the selected transcript
encoder; byte-for-byte preservation of the original input and mapping corrections
back to pre-normalization segmentation are not requirements.
The overriding goal is to minimize how much useful processing is lost because of
one problematic transcript segment. A stage/chunk work unit is an attribution
boundary, not an all-or-nothing transaction.
Existing reports contain useful information, but there are no compatibility
obligations for reporting or debug output. Reuse existing machinery only when it
makes the implementation simpler and clearer; preservation is not itself a goal.
Internal structures and measurements must serve Audita's processing workflow.
The reporting package adapts those outcomes to structured and human-facing
presentations, rather than reshaping processing around a reporting convenience.
## Historical Baseline and Implemented Change
The [pipeline](../internal/pipeline.md), [LLM integration](../integrations/openai-compatible-llm.md),
[report assembly](../internal/diagnostics-reporting.md), and
[artifact lifecycle](../operations.md) describe current behavior.
Before this work, exhausted structural repairs could become warnings and skipped
proposals or rejected validation batches. Other model runtime failures canceled
peer work, stopped subsequent modules, and prevented final transcript output.
Successful repairs appeared in execution observations rather than a dedicated
recovery classification. Detailed artifacts were written during ordinary runs
and could later be removed by retention. The working transcript was re-chunked
between modules.
This feature changed those behaviors. It builds on the completed
[library adoption](library-adoption.md) and [review remediation](remediation.md).
Fatal cancellation remains necessary; unsuccessful model work must be separated
from failures that actually require cancellation of the run.
## Fixed Chunks and Ordered Stages
Normalize first, then partition the transcript once. Preserve chunk boundaries,
segment membership, segment identity, and ordering throughout LLM processing.
Text evolves as accepted corrections are applied. Recombine chunks in original
order after the LLM stages finish.
A stage is a configured module instance. Repeated instances of the same module
have distinct identities. Proposal generation and each validator are operations
within a stage; validation may use multiple batches within a chunk. A work unit
is one stage instance applied to one chunk.
Preserve the overall stage sequence: finish a stage across its chunks before
advancing to the next stage. Concurrent chunk work shares PromptKit's backend
capacity. Failure in one unit does not prevent other chunks or later stages
from executing.
Retain operator control over chunk size and the current defaults. Production
experience indicates that chunk size affects model latency nonlinearly and that
useful settings are well below backend context limits. Tuning is an empirical
operator decision. Do not add automatic headroom calculations, cross-profile
sizing, adaptive repartitioning, or automatic tuning in this feature. A later
operation that cannot process its input follows the same safe error handling;
it does not silently alter chunk boundaries. Validation batching within a chunk
remains supported.
Independent whole-pipeline traversal by chunks and domain-aware response repair
are explicitly deferred in [future work](future.md). Read-only neighboring
context, cross-chunk edits, and transcript-wide restructuring are outside this
unit, without becoming permanent prohibitions on future capabilities.
## Partial Success and Failure Isolation
- Keep bounded transport retries, structural repairs, operation deadlines, and
shared backend capacity. Completion comes from safe fallback, not unlimited
attempts or an outer retry loop around PromptKit.
- Apply a correction only after it passes every required validator and the
application checks. Failed validation must never be treated as approval.
- If a validation batch fails, exclude its affected candidates while allowing
other candidates to continue through the remaining validators. Preserve fully
validated, applicable corrections from successful batches in the same unit.
- Track candidates by stable proposal identity. Batch membership can change
between validators; equal batch numbers do not establish candidate identity.
- A failed proposal-generation operation leaves its chunk unchanged for that
stage because there are no reliable proposals to salvage. Failures later in
validation need not discard unrelated successful candidates.
- Preserve earlier stages' corrections and other chunks' successful work.
Record partial success explicitly; a unit can apply corrections and have errors.
- Continue despite model-operation failures, including exhausted repairs/retries,
unusable model decisions, provider/transport failures, local operation deadlines,
and admission failures. An operation need not exhaust its attempt budget to fail.
- A valid empty proposal list or valid assessments rejecting all corrections is
successful processing. Correction count alone does not determine failure.
- Keep safe filtering of defective proposals. Do not retry ordinary editorial
rejections. For now, an unusable validator decision set excludes the affected
candidates and records an error after existing permitted recovery. New bounded
feedback/regeneration for domain-invalid responses belongs to future work.
- Retain fatal treatment for invalid input/configuration, internal defects,
explicit run cancellation, and inability to produce/write the transcript.
Caller cancellation is distinct from a local model-operation timeout.
- Completed runs with model errors return success. Required report-write failures
are an additional fatal output condition when reporting is enabled, as below.
PromptKit owns transport, transient retries, structural repair, and admission.
Audita owns domain acceptance, partial outcomes, fallback, and classification.
ConfigKit owns configuration resolution. Debug and reporting choices must not
change correction decisions, retry budgets, or continuation.
## Errors, Recoveries, and Debug
Classify by eventual outcome and impact. These are three reporting categories;
debug is evidence, not a failure severity. Whether execution continues is a
separate property.
| Category | Meaning | Representative cases |
| --- | --- | --- |
| Errors | Intended work could not complete reliably, or a fatal run-level operation failed. Affected candidates are excluded; other corrections can succeed. | Exhausted repair; unavailable provider; unusable validation decisions; operation deadline. |
| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient service failure recovered; defective proposals safely discarded. |
| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompts/responses, individual attempts, execution traces. |
Normal editorial disagreement belongs in debug. A validator correctly rejecting
a meaning-changing edit is expected behavior. A proposal referring to nonexistent
original text is defective; safely filtering it is a recovery. An unresolved
missing/duplicate decision-index response is an error, not a recovery.
Record an operation's eventual outcome once. Several failed attempts followed by
success produce a consolidated recovery. Attempts ending in failure contribute
to error details, not additional recoveries. Earlier operations that recovered
remain recorded if a later operation introduces an error in the same unit.
Separate output-quality recovery, defective-proposal filtering, and service
recovery with structured reasons. Network instability must not look like a prompt
defect. Do not introduce an arbitrary warning threshold or a fourth category.
### Counting and Processing Volume
- Count error-bearing work units once per stage/chunk, consolidating underlying
operation failures as details. A partially successful unit still counts once.
- Count recoveries once per recovered operation. Aggregate defective-proposal
filtering per proposal response, retaining affected-proposal counts as detail.
Attempts and individual discarded proposals are not independent incidents.
- Keep work-unit coverage separate from operation/recovery counts. Show processing
volume with explicitly labeled denominators; do not divide recovered-operation
counts by work-unit counts and call that a recovery rate.
- Report fatal run-level errors separately from the stage/chunk grid when they
cannot be attributed to a work unit. Do not invent a chunk to accommodate them.
- Derive YAML records, Markdown summaries, and CLI counts from actual processing
outcomes with deterministic ordering, rather than independently calculating them.
## Production Outputs, Placement, and Lifecycle
Reporting is enabled by default. The ordinary bundle is:
- `errors.yml`: structured error-bearing units, their underlying failures, and
applicable run-level failures.
- `recoveries.yml`: structured recovered operations and pertinent counts.
- `report.md`: a concise human-facing summary of outcome, corrections, processing
volume, errors, recoveries, and their locations in the pipeline.
Write all three documents, including empty event lists for uneventful completed
runs. Use Audita-owned schemas with safe explanations and stable reason codes;
retain run, stage-instance, chunk, operation, and validator/batch identities where
applicable. The implementation plan specifies the versioned schemas; neither old
report formats nor raw library structures define the new output contract.
Place reports beside the output transcript. Retain transcript-to-stdout support
with an explicit report-directory selection, defaulting to the invocation working
directory. Preserve the existing overwrite convention: a new run can replace files
at its selected destinations. Do not add a mandatory fresh-directory restriction
or compatibility exports.
Before model work, create initial versions of all three reports carrying a common
run identifier and explicit `in_progress` status (rendered as prose in Markdown).
Fail immediately if initialization fails. Also check the transcript destination;
creating neighboring reports does not prove that an existing transcript file is
writable. Preflight must not prematurely destroy an existing transcript.
Finalize all required outputs and check every final write. Initial write success
cannot guarantee later success if space or permissions change. Failure to write
the transcript or finalize a required report returns nonzero and must not erase
an already produced transcript. Where possible, finalize available reports with
an explicit failed status on fatal termination. Abrupt termination may leave
`in_progress`; elaborate crash-recovery machinery is not required.
Provide an option to disable reporting entirely. It suppresses all three files
(`errors.yml`, `recoveries.yml`, and `report.md`), including initialization and
final writes, while retaining the brief CLI summary and independently selectable
debug output. Disabled reporting must not create or require a writable report
destination, and must not remove reports from earlier runs.
Transcript output failures remain fatal regardless of reporting selection.
Ordinary reports survive independently of debug cleanup. Reporting configuration
and finalization must not depend on retaining a debug run directory.
## Debug and Internal Boundaries
Production issue records are compact. Full responses, transcript excerpts,
correction lists, and complete validation decisions are opt-in debug evidence.
Use a single ConfigKit-backed debug boolean with a `--debug` CLI override. Debug
artifacts may use JSON where useful. Preserve configured-secret redaction in every
presentation.
Debug-off prevents detailed artifact persistence; writing and later deleting
those files is insufficient. In-memory domain state needed to process candidates
correctly is not debug output. Debug and ordinary reporting are independent
selections and do not alter processing behavior.
There is no obligation to retain `--report-json`, old artifact layouts, parallel
exports, or old retention settings. Reuse code only where useful to the new
workflow. Enabled debug output is retained on success or failure; disabled debug
creates no debug directory. This supersedes the retention question in the
[runtime follow-up](runtime-behavior.md). The legacy `never` behavior is not the
meaning of the new reporting-disable option. Optional debug-write failures produce
a safe stderr diagnostic without changing correction decisions or the run exit
status; ordinary report and transcript-write failures retain their fatal semantics.
## Deterministic Human-Facing Presentation
Use the CLI style:
> Completed: 18 corrections applied; 3 work units with errors; 7 recoveries.
Do not say that all error-bearing units were skipped: some apply valid corrections.
Include report locations when reporting is enabled and the files exist. Send the
summary to stderr so stdout remains exclusively available for transcript output.
Disabled reporting does not suppress the summary or fatal-error messages.
Keep `report.md` concise and high-level. Use separate error and recovery matrices
with stages as columns and chunks as rows. Include every chunk in fixed order;
retain configured stage order and distinct labels for repeated module instances.
No automatic width detection, thresholds, adaptive omission, or table splitting.
A wide or long table is acceptable.
The error matrix counts error-bearing work units: each attempted cell is 0 or 1.
Recovery cells count recovered operations and can exceed 1. An error cell may
coexist with recoveries and successfully applied corrections in the same unit.
Distinguish completed work with zero events from unattempted work; do not portray
canceled/unavailable processing as clean completion. Include totals, processing
volume, and a short reason breakdown. Detailed inspection belongs in YAML or debug.
## Target Acceptance Criteria
- Total LLM failure still returns success and the encoded normalized transcript,
provided required output writes succeed.
- Mixed-success runs preserve independently validated, applicable corrections
within the same stage/chunk as a failed batch. No candidate bypasses a required
validator; later validators handle survivors by stable proposal identity.
- Subsequent stages and unaffected chunks execute after recoverable failures.
Fatal errors and explicit cancellation stop and join active work appropriately.
- Chunk boundaries and identity remain fixed, with operator-selected sizing and
unchanged defaults; final recombination preserves ordering.
- Valid no-change/editorial-rejection outcomes are successful. Repaired/filtered
output and unresolved failures receive distinct classifications and counts.
- YAML, Markdown, and CLI agree on partial outcomes and totals; error matrices
count affected work units, recovery matrices count recovered operations.
- With reporting enabled, all three reports are initialized before model work,
finalized and retained, including empty lists. Initial or final required-output
write failures produce nonzero exits without destroying completed transcript output.
- Reporting can be disabled without requiring its destination. Debug can be enabled
independently; neither selection changes processing outcomes. Debug-off writes
no detailed debug artifacts, and stdout transcript output remains supported.
- Reports use the fixed deterministic layout without adaptive presentation logic.
- Meaningful controlled-provider tests cover exhausted recovery, failed validation
batches with successful survivors, total failure, fatal cancellation, output
failures, and reporting/debug selections. Preserve domain safeguards, capacity
limits, and redaction.
- Canonical CLI, configuration, operations, integration, internal, and architecture
documentation describes the implemented final behavior.
## Implementation Reference
The product decisions are settled. The [historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/a83a518/docs/roadmap/implementation.md) defines
the selected settings and precedence, YAML schemas, counts and ordering, interrupted
work presentation, destination checks, and output finalization. It also owns the
ordered implementation stages and validation work. This document remains the
reference for the desired behavior and its acceptance criteria.

18
docs/roadmap/runtime-behavior.md
View File

@@ -1,18 +0,0 @@
# Runtime Behavior Decisions
**Last updated:** 2026-09-24
The documentation migration identified existing behaviors for separate product
consideration. The resilience implementation resolved the diagnostics-retention
question; the input-timestamp question remains open.
- Resolved: legacy `auto`/`always`/`never` retention settings were removed.
Debug is explicitly enabled with `diagnostics.debug` or `--debug`, creates no
directory when disabled, and retains its directory on every outcome when enabled.
- Resolved: [configuration](../config.md) now treats explicit zero as disabling
transient retries or structural repair independently; PromptKit performs both.
- Decide whether missing or null input timestamps should be rejected; current
decoding is documented in the [input contract](../integrations/transcript-glossary-files.md).
These decisions are independent of the completed documentation migration and
release workflow.

203
docs/roadmap/span-deletion.md
View File

@@ -1,203 +0,0 @@
# Consistent Span Deletion
**Last updated:** 2026-09-25
**Status:** Implemented. This document owns the desired behavior and user intent.
The [historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md)
records the concrete steps, selected implementation details, and completion
evidence.
## Purpose and User Intent
Align proposal prompts, the LLM response schema, internal validation, and
application behavior so legitimate text-span deletions can be considered and
applied through the ordinary correction workflow. An empty replacement should
not trigger structural repair merely because it is empty, nor invalidate other
usable corrections returned alongside it.
This is a consistency fix within Audita's current editing scope. It does not
authorize whole-segment removal, broaden module responsibilities, or introduce
new recovery machinery. The accepted [resilience behavior](resilience.md) remains
in effect, including partial success, fixed chunks, and validation before mutation.
## Historical Baseline (`0b6cd98`)
The source reviewed at `0b6cd98` contains a mismatch:
- The [spoken-word prompt](../../internal/prompts/assets/prompts/modules.spoken_word.proposal.yaml)
permits removing fillers and asks for the exact affected span and replacement,
without explaining how to represent deletion. Grammar permits punctuation and
spacing changes, which can also naturally involve deletion.
- The shared [correction response schema](../../internal/framework/responseschema/assets/correction_set.v1.json)
requires `corrected_text` to have at least one character. Every built-in proposal
module uses that schema. An empty replacement invalidates the response before
Audita evaluates individual proposals.
- PromptKit performs bounded structural repair. If repair does not produce a
valid response, the proposal operation fails and yields no candidates, including
otherwise valid proposals in the same response. The current resilience path
records the error and continues other work.
- [CorrectionProposal](../../internal/framework/proposals/proposal.go) already
allows empty replacement text. The
[preview/application logic](../../internal/framework/proposals/preview.go)
supports literal span deletion when the resulting segment remains nonblank.
- Domain shape checks currently reject whitespace-only original spans, although
the response schema permits them and grammar prompts permit spacing cleanup.
This is another restriction to align with the target contract below.
- The validator named `non_empty_corrected_text` actually checks the resulting
segment, not the replacement string. Both this check and application prevent
a segment from becoming empty or whitespace-only. Application changes segment
text; it does not remove segment records.
A temporary controlled-provider review reproduced whole-response rejection for
an empty replacement alongside a valid correction, and immediate schema acceptance
for a space-only replacement. Existing proposal tests confirm internal support for
partial deletion and rejection of blank resulting segments. These observations
explain a plausible production failure mechanism; they do not establish the cause
of any particular historical incident without its payload and running version.
## Target Correction Contract
- `original_text` identifies an existing, nonempty literal span in the current
target segment. Whitespace-only spans are eligible; an empty string identifies
no source span and remains invalid. Matching and ambiguity checks still apply
under the module's replacement policy: `require_unique` for grammar, spoken-word,
and homophones; `replace_all` for glossary.
- `corrected_text` remains a required string. An empty string means delete the
identified span; it does not mean remove the segment record or skip validation.
Missing fields and non-string values remain structurally invalid.
- After applying a correction, the segment must contain non-whitespace text.
Enforce this using the source segment and replacement result in Audita, not a
minimum replacement length in the structural schema.
- Every accepted deletion passes the same required validators as other edits in
its module, including confidence, protected terms, editorial review, and meaning
preservation where applicable. Empty replacement is neither automatic approval
nor an automatic rejection.
- Application preserves segment identity, speaker, timing, membership, ordering,
and fixed chunk boundaries. Recheck applicability against text produced by
earlier accepted corrections, retaining existing conflict handling.
- Preserve literal replacement behavior. Prompts should select spans and
replacements that handle adjacent whitespace and punctuation appropriately;
this feature does not add automatic trimming or a second normalization pass.
Neither side of a correction is rejected solely for containing only whitespace.
For example, reducing a uniquely identified run of two spaces to one space may
be appropriate; validity still depends on the resulting text and domain checks.
Preserve the intended safeguards and outcomes, not every detail of the current
implementation. Validator ordering, internal structure, and payload construction
may change when needed to achieve this consistent contract. Existing behavior or
a passing test does not justify retaining a conflicting restriction.
Illustrative cases:
| Segment | Original span | Replacement | Intended handling |
| --- | --- | --- | --- |
| `uh hello` | `uh ` | empty string | Structurally valid; may produce `hello` after required validation. |
| `uh hello` | `uh hello` | `hello` | Remains a valid representation of the same cleanup, subject to validation. |
| `hello world` | two spaces | one space | Eligible spacing correction, subject to unique matching and required validation. |
| `uh` | `uh` | empty string | Structurally valid proposal, then filtered because the resulting segment would be blank. |
| `hello world` | `hello world` | spaces only | Filtered because the resulting segment would be blank. |
These examples describe eligibility and mechanical results, not unconditional
editorial approval. An unchanged segment or valid empty correction list remains
a successful no-change outcome.
## Prompt and Module Alignment
Review relevant prompt definitions for consistency and edit only those needing
clarification. Where needed, explain empty-string deletion, whitespace-only spans,
and the resulting-segment limit with a concise, module-appropriate example. The
model should omit a correction that would leave a blank segment rather than invent
padding or replacement content. Consistent guidance is the goal; adding deletion
language to every prompt is not a requirement.
Retain each module's existing scope:
- Spoken-word cleanup can propose appropriate filler or dysfluency deletion while
preserving substantive content and intentional emphasis.
- Grammar can propose permitted punctuation or spacing deletion; allowing an
empty replacement does not authorize word removal or spoken-word cleanup.
- Glossary and homophone modules retain their existing correction purposes.
Shared deletion syntax does not grant them new editorial responsibilities.
Review the active LLM-validator prompts and their input construction as well as
proposal prompts. Validators must understand the before/after effect of deletion
and assess it under their existing criteria. A correct negative editorial decision
remains ordinary processing, not a defect that should be regenerated until approved.
The resulting-segment validator's name and documented identity should describe
that responsibility accurately, rather than imply that the replacement string
itself must be nonempty. Preserve the existing blank-result failure reason and
safeguard while making that deliberate identity change; current chain position
and preview implementation are not independent product requirements.
## Outcomes and Reporting
A legitimate deletion is an ordinary correction. Empty replacement alone must
produce neither structural repair nor a recovery classification. If another
problem required successful repair or filtering, retain that actual recovery.
A schema-valid proposal that would blank its stage-input segment is a defective
proposal handled through existing safe filtering. Preserve the
`empty_resulting_segment` reason
and consolidated `proposals_filtered` recovery behavior; other usable proposals
in the response continue through required validation. Ordinary editorial rejection
remains debug evidence rather than a model error or filtering recovery.
A stage can propose several edits to one segment in the same response. Each edit
is evaluated against the stage input, then approved edits are applied sequentially.
If a proposal was applicable to that input but an earlier accepted edit makes it
unsafe, skip it as an apply-time conflict. Do not count that conflict as a defective
proposal, model error, or filtering recovery. For example, in `"uh um"`, deleting
`"uh "` and deleting `"um"` each leaves text individually; after the first deletion,
the second must be skipped to prevent a blank result. Retain the skipped decision
in debug evidence. The existing runner already distinguishes this situation.
Debug evidence must accurately retain the empty replacement and the corresponding
validation/application outcome. Ordinary reports retain their existing counts and
purpose; no new report schema, incident category, or deletion-specific counter is
needed. Unrelated structural defects may still exhaust PromptKit repair and produce
an error for the operation, as today.
## Acceptance Criteria
- A controlled valid response containing an empty replacement reaches Audita's
domain validation without structural repair solely due to that empty string.
- A permitted span deletion can pass the required validators and reach the final
transcript while preserving segment metadata and fixed chunk membership.
- A mixed response can apply an approved deletion and another approved correction;
an empty replacement does not invalidate the whole response.
- A proposal that would blank its stage-input segment is filtered independently,
retains the existing reason, and does not prevent unrelated valid candidates
from succeeding. A proposal made unsafe only by an earlier accepted edit is
skipped without adding an error or filtering recovery; a blank result is still
prevented and the conflict remains visible in debug evidence.
- A uniquely matching whitespace-only original span can reach validation and
application. Empty originals remain invalid, ambiguous matches remain subject
to module policy, and whitespace-only resulting segments remain disallowed.
- Protected-term and meaning-changing deletions remain subject to their normal
rejection rules. No candidate bypasses required validation.
- A deletion needing no recovery counts as an ordinary applied correction; actual
repair/filtering and editorial rejection retain their existing classifications.
Debug evidence represents the empty replacement faithfully.
- Schema and domain tests distinguish an empty string from missing/non-string
replacement values and distinguish empty replacement from blank resulting text.
- Prompt guidance and validator naming express the same contract as the code.
Tests protect outcomes rather than exact prompt prose or incidental formatting.
Use deterministic, offline controlled responses for behavioral integration tests;
no live-provider or paid evaluation is a prerequisite. Representative production
examples can inform prompt quality without expanding this into an evaluation
framework or guaranteeing that models always follow the instructions.
## Deferred Work
[Whole-segment removal](future.md#whole-segment-removal) is only potential future
work and first requires deciding whether its benefit warrants implementation.
Domain-aware regeneration and independent whole-pipeline traversal also remain
in [future.md](future.md). This feature changes neither retry budgets nor the
Audita/PromptKit ownership boundary and requires no upstream library feature.
Implementation sequencing and concrete names belong to the
[historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md).
The target remains consistent deletion of spans within surviving segments, with
no new operator controls or report types.