72 Commits

Author SHA1 Message Date
4cf1b6fbca Document the Audita v2 release
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
2026-09-25 13:30:02 +00:00
07a4b65570 Share prompt instructions and simplify validation helpers 2026-09-25 13:26:51 +00:00
5feec1bf14 Record maintainability audit recommendations 2026-09-25 13:23:12 +00:00
011b021d47 Simplify references and output instructions in LLM prompts 2026-09-25 12:38:23 +00:00
325c4ea7d3 Align prompts and strengthen validation context 2026-09-25 12:23:10 +00:00
3eb73e70fc Record prompt audit findings and recommendations 2026-09-25 12:15:44 +00:00
b8089f4a50 Verify partial validator decisions through CLI workflows 2026-09-25 11:34:26 +00:00
22f0932389 Track unavailable validator decisions in reports and debug records 2026-09-25 11:28:44 +00:00
d726ea8f6d Reconcile partial validator decision responses 2026-09-25 11:25:22 +00:00
7199cb565f Plan partial validator decision handling 2026-09-25 03:27:41 +00:00
60e2fe74b5 Define partial success for validator decisions 2026-09-25 03:23:03 +00:00
d2a6fcfd41 Fix deletion conflict accounting coverage 2026-09-25 03:06:57 +00:00
5e2e863fbd Record the deletion conflict test gap 2026-09-25 03:03:14 +00:00
24b899281c Verify span deletion through complete CLI workflows 2026-09-25 02:56:44 +00:00
2a58db04c2 Preserve empty deletion spans in debug ledger 2026-09-25 02:52:09 +00:00
93f62c9095 Accept span deletions in structured responses 2026-09-25 02:50:46 +00:00
640f663ace Align validator identity and span domain checks 2026-09-25 02:48:57 +00:00
31979d4081 Refine the span deletion contract and implementation plan 2026-09-25 02:44:38 +00:00
f219309617 Plan the span deletion implementation 2026-09-25 01:21:25 +00:00
a83a51848b Define consistent span deletion and defer whole-segment removal 2026-09-25 01:16:17 +00:00
0b6cd98262 Remove unnecessary output restrictions and simplify report tests 2026-09-25 00:36:44 +00:00
efaefdbf3d Record unnecessary restrictions found in the resilience review 2026-09-25 00:30:23 +00:00
ec78b5a5db Fix the five resilience review findings 2026-09-24 21:48:18 +00:00
9fc4de22b7 Record resilience implementation review findings 2026-09-24 19:30:22 +00:00
6523d7ac74 Verify the integrated resilient processing contract 2026-09-24 18:03:25 +00:00
a770c77a23 Document resilient reports and remove obsolete warning material 2026-09-24 17:55:37 +00:00
2f036b0356 Compose resilient CLI processing and report lifecycle 2026-09-24 17:48:52 +00:00
87b8d9ad5c Make debug evidence opt-in and retain all candidates 2026-09-24 17:41:10 +00:00
a80eed4745 Preflight and publish required outputs through an output session 2026-09-24 17:37:04 +00:00
989b28e8c5 Render fixed Markdown reports and shared completion counts 2026-09-24 17:34:03 +00:00
61af29ecb8 Map processing outcomes to versioned YAML reports 2026-09-24 17:31:45 +00:00
63fcfc487d Add reporting and debug settings with context-aware CLI selection 2026-09-24 17:27:11 +00:00
b98604e5ca Integrate partial processing outcomes and fatal cancellation 2026-09-24 17:23:59 +00:00
bb21a81805 Keep transcript sections fixed throughout processing 2026-09-24 17:17:14 +00:00
0a19f163c6 Preserve validation decisions across failed batches 2026-09-24 17:14:33 +00:00
42bde59827 Preserve proposal outcomes and classify filtered proposals 2026-09-24 17:09:35 +00:00
51169ff9e9 Classify model failures and capture execution counts 2026-09-24 17:07:01 +00:00
7dc1be5a08 Define processing outcomes and failure contracts 2026-09-24 17:01:19 +00:00
8ffbf41b89 Plan resilient processing and focused run reports 2026-09-24 16:50:21 +00:00
cb405f6fc7 Record empty replacement handling as future work 2026-09-24 16:36:11 +00:00
053b714ce6 Finalize resilience decisions and record deferred work 2026-09-24 16:34:35 +00:00
aa2f486845 Record the resilience and reporting roadmap 2026-09-24 16:01:15 +00:00
665ee85215 Fix the five library adoption review findings 2026-09-24 15:32:13 +00:00
c42bc01052 Record library adoption review findings 2026-09-24 03:07:36 +00:00
7cefb3fb04 Verify the integrated application and complete the migration 2026-09-24 02:40:48 +00:00
80804a038f Migrate examples and guidance and remove legacy test harness 2026-09-24 02:37:20 +00:00
9561e80efb Add version two LLM reports and safe execution diagnostics 2026-09-24 02:31:37 +00:00
4bfc5f3456 Collect PromptKit execution observations with logical attribution 2026-09-24 02:23:57 +00:00
4812032c9e Connect processing to shared PromptKit backend capacity 2026-09-24 02:18:31 +00:00
a4a72168d5 Move LLM validators to PromptKit prompt variables 2026-09-24 02:07:17 +00:00
f577847ca2 Route proposal modules through embedded PromptKit definitions 2026-09-24 02:03:08 +00:00
c8a78bf024 Use PromptKit prepared execution for structured model calls 2026-09-24 01:57:25 +00:00
96d3d03bee Assemble PromptKit profiles for offline configuration inspection 2026-09-24 01:50:10 +00:00
f8c18247a0 Move embedded prompts to PromptKit definitions 2026-09-24 01:44:23 +00:00
51e370b4c8 Unify CLI configuration selection and overrides 2026-09-24 01:38:49 +00:00
9fb89057ce Replace legacy configuration loading with ConfigKit schema two 2026-09-24 01:33:03 +00:00
1ad2119283 Finalize the ConfigKit and PromptKit implementation plan 2026-09-24 01:23:32 +00:00
116648fa78 Simplify the library migration to target the final integration 2026-09-24 01:09:43 +00:00
5b7d2e6e81 Plan production ConfigKit migration from verified prototype 2026-09-24 00:27:47 +00:00
590604bf38 Document verified prototype overlay behavior 2026-09-24 00:24:19 +00:00
7b809878d6 Verify prototype overlay boundaries 2026-09-24 00:22:56 +00:00
228ac4e26d Demonstrate overlay model routing in prototype 2026-09-24 00:19:51 +00:00
2d8b441802 Add prototype testing overlay configuration 2026-09-24 00:17:38 +00:00
2c55bf5f1e Upgrade prototype to ConfigKit 0.4.0 2026-09-24 00:15:27 +00:00
c97a6fa70a Plan staged implementation of the testing overlay prototype 2026-09-24 00:11:18 +00:00
187dc4cb1c Plan ConfigKit overlays for testing with cheaper models 2026-09-23 23:04:46 +00:00
228087dec8 Prototype ConfigKit and PromptKit adoption and record the findings 2026-09-23 18:48:18 +00:00
f70bb27636 Simplify the agent commit policy wording 2026-09-23 18:22:48 +00:00
a379cc915e Detail the shared library boundary and prototype plan 2026-09-23 18:21:05 +00:00
837efc78a7 Plan PromptKit adoption with structural repair enabled 2026-09-23 18:18:45 +00:00
6b33254a82 Plan coordinated ConfigKit and PromptKit adoption 2026-09-23 15:38:41 +00:00
e8bc005040 Outline the breaking ConfigKit migration 2026-09-23 15:17:02 +00:00
251 changed files with 12866 additions and 16576 deletions

View File

@@ -3,7 +3,7 @@ when:
steps:
- name: build-release-assets
image: golang:1.25
image: golang:1.25.5
commands:
- bash scripts/build-release.sh "$CI_COMMIT_TAG" dist

View File

@@ -6,19 +6,27 @@ reports for review and subprocess integration.
## Quickstart
From a checkout, build with Go 1.24 or newer:
From a checkout, build with Go 1.25.5 or newer:
```sh
go build -o ./bin/audita ./cmd/audita
```
Processing requires a reachable [compatible LLM endpoint](docs/integrations/openai-compatible-llm.md)
and access to its configured model. Supply its credentials through the
[configuration mechanisms](docs/config.md#secrets); the
[minimal config](examples/minimal-config.yml) reads `AUDITA_LLM_API_KEY` and uses
the [default model and endpoint](docs/config.md#llm-settings). Set that variable
in your environment before using an endpoint that requires authentication.
For another endpoint or model, edit a copy of the config.
and access to a model you select. The [minimal config](examples/minimal-config.yml)
targets an unauthenticated local endpoint; edit its endpoint and
[local profiles](examples/profiles/) to match your service and models. The
[service example](examples/production-config.yml) demonstrates a named credential
reference and a shared backend. Set its referenced environment variable before
validating it. The [testing overlay](examples/overlays/testing.yml) selects both
role profiles, but model cost, speed, and quality depend on your provider choices.
Check configuration and profile selection offline:
```sh
./bin/audita config validate --config examples/minimal-config.yml
./bin/audita config print-effective --config examples/minimal-config.yml --view profiles
```
Run the included small input with the minimal config:
@@ -29,8 +37,14 @@ Run the included small input with the minimal config:
--output corrected.json
```
This makes LLM requests. Inherited configuration environment overrides still
apply; see [precedence](docs/config.md#effective-precedence).
This makes LLM requests. Explicit setting flags override the selected file and
optional overlay; general environment setting overrides are no longer read.
By default the command writes `errors.yml`, `recoveries.yml`, and `report.md`
beside `corrected.json`. Use `--reporting=false` to suppress those files, or
`--debug` to retain detailed run artifacts under the configured debug work
directory. Model failures affecting some operations can still yield a completed
transcript with applied corrections; the summary and reports identify them.
See the [v2 migration guide](docs/migrations/v2.md) for old configurations.
## Documentation

View File

@@ -1,11 +1,20 @@
package main
import (
"context"
"os"
"os/signal"
"syscall"
"gitea.maximumdirect.net/eric/audita/internal/cli"
)
func main() {
os.Exit(cli.Run(os.Args[1:], os.Stdout, os.Stderr))
os.Exit(run())
}
func run() int {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
return cli.Run(ctx, os.Args[1:], os.Stdout, os.Stderr)
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,198 +1,95 @@
# Audita CLI Reference
## Shortest Useful Command
The shared version-2 configuration parser and offline profile inspection serve
`process`, `config validate`, and `config print-effective`.
The [configuration reference](config.md) owns fields, defaults, and validation;
the [migration guide](migrations/v2.md) maps retired inputs.
## Commands and help
```sh
audita process transcript.json --glossary glossary.yaml --output corrected.json
audita process <transcript.json> --glossary <glossary.yml> [flags]
audita config validate --config <config.yml> [flags]
audita config print-effective [flags]
```
This command runs the core workflow with your input files. For a copyable example
using maintained inputs and explicit prerequisites, see the [README](../README.md).
Commands here assume `audita` is on `PATH`; use `./bin/audita` after a local build.
Root invocation, `--help`, `-h`, and command help succeed without loading a config
file or resolving secrets. Unknown commands and flags return exit code `2`.
`process` accepts exactly one transcript path, either before the flags or after
them. With flags first, put the transcript last. Both config commands reject
positional arguments and process-only flags.
## Command Overview
All three commands accept `--config <path>` and `--overlay <name>`. `config validate`
requires an explicit nonempty `--config`; the other commands select the explicit
file, then `/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`.
Missing system candidates are skipped; inaccessible selected files are errors.
No file means a configuration error. `AUDITA_CONFIG` is ignored. The selected
overlay is loaded beside the main file, not used to locate profile files.
- `audita process`: process one transcript JSON file.
- `audita config validate`: validate a versioned YAML config file.
- `audita config print-effective`: print redacted effective config JSON.
## Setting overrides
General help:
Only flags actually supplied override file or overlay settings. `--modules ''`
clears the module pipeline, and an explicit numeric zero reaches configuration
validation. The shared setting flags are:
```sh
audita --help
audita process --help
audita config --help
```
| Setting | Flags |
| --- | --- |
| Profile selection | `--proposal-profile`, `--validation-profile` |
| Pipeline and output | `--modules`, `--output-schema` |
| Validation budget | `--validation-max-prompt-tokens` |
| Chunking | `--target-sections`, `--max-section-tokens`, `--min-section-tokens` |
| Normalization (numeric seconds) | `--normalize-max-segment-gap`, `--normalize-ellipsis-gap`, `--normalize-max-segment-duration`, `--normalize-max-segment-tokens` |
| Confidence thresholds | `--glossary-confidence-threshold`, `--homophones-confidence-threshold`, `--spoken-word-confidence-threshold`, `--grammar-confidence-threshold` |
| Context | `--transcript-description` |
| Reporting | `--reporting[=true\|false]`, `--report-dir DIR` |
| Debug | `--debug[=true\|false]`, `--work-dir DIR` |
## `process`
`process` alone accepts `--glossary` (required) and `--output`.
An omitted or empty output value selects stdout. Nonempty output filenames are
used literally, including whitespace.
Reporting defaults to enabled. `--reporting=false` disables all three ordinary
reports; an explicit false overrides a true file or overlay setting. With file
output, reports go beside the transcript and `--report-dir` has no effect. With
stdout output, `--report-dir` selects the report directory, defaulting to the
invocation working directory. `--debug` separately enables retained detailed
artifacts under `--work-dir`. Configuration inspection does not create or probe
these output directories.
Per-role model, base URL, key, timeout, retry, and concurrency flags have been
removed. Models and generation controls belong to PromptKit profiles, backend
credentials and capacity to configuration, and retry/deadline settings to the
shared LLM section. Legacy configuration environment overrides are ignored.
Usage:
## Streams and exit status
```sh
audita process <transcript.json> [flags]
```
`config validate` loads settings and both selected PromptKit profiles offline,
then prints `config is valid` on success. `config print-effective` accepts
`--view settings|profiles|explain` and defaults to `settings`. Settings writes
ConfigKit's redacted effective-settings YAML. Profiles writes an Audita-owned
YAML summary of selected ID, backend, model, sampling controls, per-attempt
timeout, service tier, and reasoning effort for each role. It omits credentials,
endpoints, and arbitrary extra provider parameters. Explain writes a YAML
sequence of field paths, source origins, and recorded derivations without
resolved setting values. These views share the same settings and profile
validation path as `process`. Unknown views are usage errors. The effective
settings YAML is for inspection and is not guaranteed reloadable.
Input requirements:
- exactly one transcript JSON positional argument is required;
- `--glossary <path>` is required.
For `process`, `--output` writes the corrected transcript to a file and leaves
stdout empty on success. Without it, stdout carries transcript JSON only on
success. Enabled reporting creates `errors.yml`, `recoveries.yml`, and
`report.md` in the selected directory. Successful stderr begins with
`Completed: N corrections applied; E work units with errors; R recoveries.`
Report and debug locations follow only when created. Model-operation failures
can coexist with a completed run and applied corrections; read the YAML reports
for details. Debug artifacts are created only with `--debug`. See the
[report contract](integrations/output-files.md).
Place the transcript path first, followed by flags, as shown above. Flag values
may use `--name value` or `--name=value`. Unknown flags or extra positional
arguments are errors. Flags before a final transcript path are also accepted;
do not interleave further flags after a positional argument consumed by the flag
parser.
Exit codes are `0` for success, `1` for runtime or output failure, and `2` for
usage or configuration input errors. Invalid settings and profiles return `2`;
broken embedded assets, caller cancellation, fatal processing, and required
output failures return `1`. A late report-write failure can follow a published
transcript; exit status and report status remain authoritative.
Configuration commands inspect profiles without contacting a provider.
[Configuration discovery and precedence](config.md#loading-model) determine the
initial settings; explicit process flags supply overrides. Configuration is loaded
before process help is printed, so invalid discovered configuration can also make
process help fail. Input formats belong to the
[input-file contract](integrations/transcript-glossary-files.md).
### `process` Flag Reference
Core I/O flags:
- `--config <path>`: path to versioned YAML config file.
- `--glossary <path>`: glossary YAML input path (required).
- `--output <path>`: corrected transcript JSON output file path.
- `--report-json <path>`: machine-readable report JSON output path.
- `--output-schema <key>`: output encoder selection; see [configuration](config.md#pipeline-and-output).
- `--modules <csv>`: comma-separated module sequence override.
Primary LLM flags:
- `--llm-api-key <value>`: primary LLM API key.
- `--model <name>`: primary LLM model name.
- `--base-url <url>`: primary OpenAI-compatible base URL.
- `--llm-timeout-seconds <int>`: primary timeout in seconds.
- `--max-retries <int>`: primary structured-output retries.
Validation LLM flags:
- `--validation-llm-api-key <value>`: validation LLM API key.
- `--validation-model <name>`: validation LLM model name.
- `--validation-base-url <url>`: validation OpenAI-compatible base URL.
- `--validation-llm-timeout-seconds <int>`: validation timeout in seconds.
- `--validation-max-retries <int>`: validation structured-output retries.
- `--validation-max-prompt-tokens <int>`: validation max prompt tokens.
Concurrency flags:
- `--total-llm-concurrency <int>`: total concurrent proposal+validation LLM calls.
- `--proposal-llm-concurrency <int>`: concurrent proposal-generation LLM calls.
- `--validation-llm-concurrency <int>`: concurrent validation LLM calls.
- `--llm-concurrency <int>`: alias for `--total-llm-concurrency`.
Chunking and normalization flags:
- `--target-sections <int>`: target number of transcript sections.
- `--max-section-tokens <int>`: maximum section tokens.
- `--min-section-tokens <int>`: minimum section tokens.
- `--normalize-max-segment-gap <float>`: maximum same-speaker merge gap.
- `--normalize-ellipsis-gap <float>`: gap threshold for ellipsis insertion.
- `--normalize-max-segment-duration <float>`: maximum merged segment duration.
- `--normalize-max-segment-tokens <int>`: maximum merged segment token estimate.
Threshold flags:
- `--glossary-confidence-threshold <float>`
- `--homophones-confidence-threshold <float>`
- `--spoken-word-confidence-threshold <float>`
- `--grammar-confidence-threshold <float>`
Context and diagnostics flags:
- `--transcript-description <text>`: background context for prompts; does not override transcript content.
- `--work-dir <path>`: per-run diagnostics work directory.
- `--work-dir-retention <value>`: retention selection; see [configuration](config.md#thresholds-context-and-diagnostics).
Flag descriptions identify their settings; defaults, accepted keys, numeric
constraints, and inheritance rules are owned by [configuration](config.md).
Normalization time flags use numeric seconds; LLM timeout flags use integer seconds.
### `process` Output and Exit Behavior
- With `--output`: stdout is expected to be empty on success.
- Without `--output`: stdout contains transcript JSON only on success.
- `--report-json` writes a file and is never printed to stdout.
- Stderr is human-readable diagnostics/errors.
- Processing failures include the diagnostics directory path when initialization
succeeded. Later report-write or retention errors can omit that path.
- Output files may already exist when a later reporting or cleanup failure occurs.
- Rejected/skipped corrections and warnings can occur in a successful run; inspect
the [report format](integrations/output-files.md#process-report) for details.
- Diagnostic wording is human-readable and is not a machine-stable JSON channel.
Exit behavior:
- `0`: success.
- `1`: runtime failure during processing/reporting/output or retention handling.
- `2`: CLI usage or configuration input error.
Integration references:
- subprocess contract: [`docs/integrations/subprocess.md`](integrations/subprocess.md)
- transcript/glossary file contract: [`docs/integrations/transcript-glossary-files.md`](integrations/transcript-glossary-files.md)
### `process` Examples
Write corrected transcript to a file:
```sh
audita process transcript.json \
--glossary glossary.yaml \
--output corrected.json
```
Emit transcript JSON to stdout:
```sh
audita process transcript.json --glossary glossary.yaml
```
Use explicit config and write report JSON:
```sh
audita process transcript.json \
--glossary glossary.yaml \
--config audita.yml \
--output corrected.json \
--report-json report.json
```
Override the module sequence:
```sh
audita process transcript.json \
--glossary glossary.yaml \
--modules glossary,homophones,grammar \
--output corrected.json
```
## Configuration Commands
### `config validate`
```sh
audita config validate --config examples/minimal-config.yml
audita config validate --config examples/production-config.yml
```
The only command-specific flag is `--config <path>`, which is required.
Positional arguments are rejected. Success prints `config is valid` to stdout
and returns zero; usage, file-read, parse, and validation errors go to stderr and
return `2`. See [applied configuration sources](config.md#effective-precedence)
and [validation boundaries](config.md#examples).
### `config print-effective`
```sh
audita config print-effective --config examples/minimal-config.yml
```
The only command-specific flag is optional `--config <path>`. Positional
arguments are rejected. Without an explicit file, [discovery](config.md#loading-model)
applies. Success prints redacted configuration JSON to stdout and returns zero.
Usage and configuration errors return `2`; serialization or stdout-write failures
return `1`.
The printed JSON reflects the effective in-memory configuration; it is not a YAML
configuration file and is not the process-report schema. Use it to inspect merged
settings, not as a file to pass back to the YAML loader.
## Help
Root invocation without arguments and root help print usage successfully. Use
`--help` or `-h` for subcommand help. Help normally returns zero; process help has
the configuration-loading caveat described above. Unknown commands return `2`.
See the [input files](integrations/transcript-glossary-files.md),
[subprocess contract](integrations/subprocess.md), and
[artifact lifecycle](operations.md) for their respective details.

View File

@@ -1,188 +1,88 @@
# Audita Configuration
This is the canonical reference for discovery, settings, defaults, environment
mapping, and validation. [CLI syntax](cli.md), [artifact lifecycle](operations.md),
and [file formats](integrations/output-files.md) have separate owners.
This reference describes the version-2 configuration loader in `internal/core/config`.
Use the maintained [local](../examples/minimal-config.yml) and
[service](../examples/production-config.yml) examples with their PromptKit
[profiles](../examples/profiles/) and optional [testing overlay](../examples/overlays/testing.yml).
Choose real models for your endpoint; a testing profile makes no cost or quality
guarantee. See the [migration guide](migrations/v2.md) for retired inputs.
## Loading Model
## Loading and precedence
For processing and effective-config inspection, the first applicable source
selects one YAML file:
The application selects one main YAML file: an explicit `--config`, then
`/usr/local/etc/audita/config.yml` if present, then `/etc/audita/config.yml`
if present. Missing system candidates are skipped; an explicit missing or empty
path, inaccessible selected file, or absence of any selected file is an error.
`config validate` requires an explicit file. ConfigKit loads built-in defaults,
the selected file, an optional named overlay from `overlays/<name>.yml` beside the
main file, and only explicitly supplied setting overrides, in that order. Imports
and secret-directory bootstrap are disabled. Unknown fields and missing, null,
or non-integer `version: 2` are errors. Version-1 files require migration; no
compatibility loader or implicit environment configuration remains.
1. Explicit CLI config path.
2. `AUDITA_CONFIG`.
3. `/usr/local/etc/audita/config.yml`, if present.
4. `/etc/audita/config.yml`, if present.
Validation runs after overlays, explicit overrides, and derived settings have
resolved. An override can therefore repair an invalid lower-precedence setting
when that setting can be decoded. ConfigKit retains field origins and derivations
for effective-configuration inspection. Its effective YAML redacts resolved
credentials.
These files are alternatives, not layers merged together. Missing system files
are skipped. An empty or missing explicitly selected path is an error, as is an
inaccessible selected file. Without a selected file, built-in defaults apply.
The validation command requires an explicit file and does not use discovery.
Profile files are resolved against the selected main file's directory, or may
use an absolute `profiles_dir`. Both selected profiles must exist and select a
registered backend. Their effective endpoints must match that backend's endpoint
after surrounding whitespace and trailing slashes are removed. Profiles cannot
supply a competing `api_key_env` lookup. The backend credential supplies the
runtime key when required; inspection makes no provider call. The
[CLI inspection views](cli.md#streams-and-exit-status) show selected controls and
field provenance separately from ConfigKit's redacted settings YAML.
## Effective Precedence
## Settings
| Operation | Applied sources, lowest to highest precedence |
| --- | --- |
| Process | Built-in defaults, selected file, environment overrides, explicit CLI overrides |
| Print effective configuration | Built-in defaults, selected file, environment overrides |
| Validate configuration file | Built-in defaults, explicit file |
File validation does not apply configuration environment overrides, but it still
resolves environment variables named by `api_key_env` fields. Each configuration
layer is validated as it is applied: a later override cannot repair an invalid
earlier layer. [Config commands](cli.md#configuration-commands) provide inspection
and validation without processing a transcript or contacting an LLM.
## File Schema
The file is YAML with required `version: 1`. Unknown fields are rejected.
The remaining sections and fields below are optional. Omitted values preserve
lower-precedence settings, except for the documented inheritance rules.
Complete copyable files are in [examples](#examples).
### Pipeline and Output
| YAML field | Default | Meaning |
| Field | Default | Rule |
| --- | --- | --- |
| `pipeline.modules` | `[glossary, homophones, glossary, spoken_word, grammar]` | Ordered module sequence. Supported keys are `glossary`, `homophones`, `spoken_word`, and `grammar`; repeated keys are allowed. An omitted or empty YAML list leaves the sequence unchanged. |
| `output.schema` | `bare-segments` | Output encoder selection: `bare-segments` or `audita-v1`. See [output formats](integrations/output-files.md#corrected-transcript). |
| `profiles_dir` | `profiles` | Nonempty; profile files are resolved relative to the selected main file. |
| `backends.<id>.endpoint` | None | Required nonempty endpoint for each backend. At least one backend is required. |
| `backends.<id>.concurrency` | `1` | Positive finite capacity; shared by both roles using that backend. |
| `backends.<id>.queue_capacity` | `1024` | Nonnegative; zero disables waiting admission. |
| `backends.<id>.credential` | Unset | Optional environment-variable secret reference; omission or null means unauthenticated use. |
| `llm.proposal_profile` | None | Required nonempty profile ID. |
| `llm.validation_profile` | Final proposal profile | Omission inherits the final proposal selection; explicit empty or null fails. |
| `llm.transient_retries` | `2` | Nonnegative additional attempts per generation. |
| `llm.repair_attempts` | `2` | Additional structural repair generations, from zero through three. |
| `llm.operation_timeout` | `15m` | Positive Go duration string per logical call. |
| `pipeline.modules` | `[glossary, homophones, glossary, spoken_word, grammar]` | Ordered module keys; repeats are meaningful and an explicit empty list runs no correction modules. |
| `output.schema` | `bare-segments` | `bare-segments` or `audita-v1`. See [output formats](integrations/output-files.md). |
| `validation.max_prompt_tokens` | `2048` | Positive validation batching budget. |
| `chunking.target_sections` | Unset | Optional positive target; omission or null selects automatic chunking. |
| `chunking.max_section_tokens` | `8192` | Positive maximum estimate. |
| `chunking.min_section_tokens` | `2048` | Positive and no greater than maximum. |
| `normalization.max_segment_gap` | `4.0` | Nonnegative numeric seconds. |
| `normalization.ellipsis_gap` | `3.5` | Nonnegative numeric seconds. |
| `normalization.max_segment_duration` | `60.0` | Positive numeric seconds. |
| `normalization.max_segment_tokens` | `2048` | Positive token estimate. |
| `thresholds.glossary`, `.homophones`, `.spoken_word`, `.grammar` | `0.8` each | Confidence thresholds in `[0, 1]`. |
| `context.description` | Empty | Trimmed prompt context, at most 500 bytes. |
| `reporting.enabled` | `true` | Write the ordinary three-file report bundle when processing. |
| `reporting.stdout_dir` | `.` | Report directory for stdout transcript output; ignored for file output or disabled reporting. |
| `diagnostics.debug` | `false` | Persist detailed debug artifacts when processing. |
| `diagnostics.work_dir` | `/tmp/audita` | Parent of the retained debug run directory; required only when debug is enabled. |
Module keys select correction stages. Validator chains are built into each module;
there is no configuration field for selecting individual validator keys.
Module keys select correction modules; validator chains remain built into them.
The loader preserves module order and repetitions. Normalization time settings
accept numeric seconds only. The operation timeout uses a duration string.
`reporting.stdout_dir` must be nonblank only for reporting-enabled stdout runs;
`diagnostics.work_dir` must be nonblank only with debug enabled. These path
requirements are checked when choosing process destinations, not during
configuration inspection.
### LLM Settings
## Credentials
The same fields are supported under `llm.proposal` and `llm.validation`.
A backend credential is a ConfigKit secret reference containing an environment
variable name, such as `credential: AUDITA_SERVICE_KEY`. Every reference that
survives overlay and override resolution must name a set, nonempty variable,
including credentials on backends unused by the selected roles. A replacement
reference resolves only the replacement; null removes the reference. General
`AUDITA_*` and `OPENROUTER_API_KEY` variables have no implicit setting effect.
Secret values are available to the runtime through the resolved reference, while
ConfigKit's effective output and provenance contain no resolved secret values.
| Field suffix | Proposal default | Meaning |
| --- | --- | --- |
| `base_url` | `https://openrouter.ai/api/v1` | Endpoint base URL; the [LLM contract](integrations/openai-compatible-llm.md#endpoint-and-authentication) defines how it is used. |
| `model` | `openrouter/google/gemma-4-31b-it` | Model identifier sent to the endpoint. Availability is determined by that endpoint. |
| `api_key_env` | Unset | Name of an environment variable containing the API key; see [secrets](#secrets). |
| `timeout` | `600` seconds | Positive integer seconds or a duration string resolving to whole seconds, such as `120s`. |
| `max_retries` | `3` | Non-negative retry count after the initial attempt; retry conditions belong to the [LLM contract](integrations/openai-compatible-llm.md#retries-and-timeouts). |
Validation settings inherit the effective proposal settings individually. Empty
validation model, base URL, or API key values inherit the proposal value; omitted
validation timeout and retry count inherit, while an explicit retry count of zero
is retained in configuration. The current HTTP adapter interprets a retry count
of zero as its default of three retries, so zero does not disable retries.
No API key is required by local config validation. Model and base-URL
usability are checked during client setup/requests, not by an online config check.
### Concurrency
| YAML field | Default | Meaning |
| --- | --- | --- |
| `concurrency.total_llm` | `1` | Positive total concurrent LLM-call limit. |
| `concurrency.proposal_llm` | `1` | Positive proposal-call limit, no greater than total. |
| `concurrency.validation_llm` | Inherits total | Positive validation-call limit, no greater than total. |
In a YAML file, setting total alone does not change proposal concurrency. At the
environment or CLI layer, setting total without setting proposal at that same
layer also sets proposal to that total, even if a lower layer supplied a proposal
value. Unset validation concurrency inherits the final total. The canonical total
setting wins over its legacy alias when both appear at the same layer.
Validation prompt budgeting is available through environment or CLI only:
`AUDITA_VALIDATION_MAX_PROMPT_TOKENS` has default `2048` and must be positive.
There is no YAML field for it.
### Chunking and Normalization
| YAML field | Default | Meaning and validation |
| --- | --- | --- |
| `chunking.target_sections` | Unset | Optional positive target count. Processing rejects targets impossible for the normalized transcript and token constraints. |
| `chunking.max_section_tokens` | `8192` | Positive maximum section token estimate. |
| `chunking.min_section_tokens` | `2048` | Positive minimum used in chunk balancing; no greater than maximum. |
| `normalization.max_segment_gap` | `4.0` seconds | Non-negative gap limit for same-speaker merging. |
| `normalization.ellipsis_gap` | `3.5` seconds | Non-negative gap threshold for inserting an ellipsis during merging. |
| `normalization.max_segment_duration` | `60.0` seconds | Positive merged-segment duration limit. |
| `normalization.max_segment_tokens` | `2048` | Positive merged-segment token estimate limit. |
The three normalization time fields accept numeric seconds or YAML duration
strings. Their environment and CLI overrides use numeric seconds. Section token
counts are estimates; config validation does not test whether a particular input
can satisfy the selected chunking target.
### Thresholds, Context, and Diagnostics
| YAML field | Default | Meaning and validation |
| --- | --- | --- |
| `thresholds.glossary` | `0.8` | Glossary correction confidence threshold, in `[0, 1]`. |
| `thresholds.homophones` | `0.8` | Homophone correction confidence threshold, in `[0, 1]`. |
| `thresholds.spoken_word` | `0.8` | Spoken-word correction confidence threshold, in `[0, 1]`. |
| `thresholds.grammar` | `0.8` | Grammar correction confidence threshold, in `[0, 1]`. |
| `context.description` | Empty | Trimmed background context for prompts; limited to 500 bytes of UTF-8 text after trimming. It does not replace transcript content. |
| `diagnostics.work_dir` | `/tmp/audita` | Non-empty parent directory for run artifacts. |
| `diagnostics.retention` | `auto` | Accepted values: `auto`, `always`, `never`. Exact lifecycle behavior, including the current behavior of `never`, belongs to [operations](operations.md#retention-behavior). |
## Environment Overrides
Values are read when the variable is set. Empty values are not generally equivalent
to unset values: numeric overrides must parse, and empty strings may clear a
setting or fail validation. Numeric environment values use integers or floating
point seconds as indicated above, not duration strings.
| Environment variable | Setting |
| --- | --- |
| `AUDITA_CONFIG` | Config file selection |
| `AUDITA_MODULES` | Module sequence as comma-separated keys |
| `AUDITA_LLM_API_KEY` | Direct proposal API key |
| `OPENROUTER_API_KEY` | Proposal API key fallback when `AUDITA_LLM_API_KEY` is unset |
| `AUDITA_MODEL` | Proposal model |
| `AUDITA_BASE_URL` | Proposal endpoint |
| `AUDITA_LLM_TIMEOUT_SECONDS` | Proposal timeout |
| `AUDITA_MAX_RETRIES` | Proposal retries |
| `AUDITA_VALIDATION_LLM_API_KEY` | Direct validation API key |
| `AUDITA_VALIDATION_MODEL` | Validation model |
| `AUDITA_VALIDATION_BASE_URL` | Validation endpoint |
| `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS` | Validation timeout |
| `AUDITA_VALIDATION_MAX_RETRIES` | Validation retries |
| `AUDITA_TOTAL_LLM_CONCURRENCY` | Total concurrency |
| `AUDITA_LLM_CONCURRENCY` | Legacy total-concurrency alias |
| `AUDITA_PROPOSAL_LLM_CONCURRENCY` | Proposal concurrency |
| `AUDITA_VALIDATION_LLM_CONCURRENCY` | Validation concurrency |
| `AUDITA_VALIDATION_MAX_PROMPT_TOKENS` | Validation prompt budget |
| `AUDITA_TARGET_SECTIONS` | Section target |
| `AUDITA_MAX_SECTION_TOKENS` | Section maximum |
| `AUDITA_MIN_SECTION_TOKENS` | Section minimum |
| `AUDITA_NORMALIZE_MAX_SEGMENT_GAP` | Merge gap |
| `AUDITA_NORMALIZE_ELLIPSIS_GAP` | Ellipsis gap |
| `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION` | Merged duration limit |
| `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS` | Merged token estimate limit |
| `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD` | Glossary threshold |
| `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD` | Homophone threshold |
| `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD` | Spoken-word threshold |
| `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD` | Grammar threshold |
| `AUDITA_WORK_DIR` | Diagnostics directory |
| `AUDITA_WORK_DIR_RETENTION` | Retention selection |
There is no environment override for output schema or transcript description.
Use file configuration or the corresponding [CLI flags](cli.md#process-flag-reference).
## Secrets
File config accepts environment variable names through `api_key_env`, not literal
API-key fields. Names must be non-empty and match an environment-variable
identifier; an unset named variable resolves to an empty value. Runtime environment
key overrides take precedence over keys resolved from a file. CLI key flags supply
direct values and take precedence over environment settings.
Effective-config inspection redacts API keys. Configured LLM secrets are also
redacted by the runtime's error and diagnostic handling, but transcript content
is still sensitive: see [artifact handling](operations.md#artifact-handling).
## Examples
- [Minimal configuration](../examples/minimal-config.yml): starts from defaults
and obtains the proposal key from the environment.
- [Production-oriented configuration](../examples/production-config.yml): illustrates
both LLM targets, concurrency, chunking, normalization, thresholds, and diagnostics.
Its explicit section target is intended for larger inputs, not the tiny sample.
- [Tiny transcript](../examples/tiny-transcript.json) and
[tiny glossary](../examples/tiny-glossary.yaml): inputs for the README quickstart.
Use the [CLI configuration commands](cli.md#configuration-commands) to validate
files and inspect effective settings. Validation checks local structure and
constraints; it does not verify credentials, model availability, or endpoint access.
Current invocation syntax belongs in [CLI](cli.md).

View File

@@ -20,7 +20,7 @@ ownership. Commit discipline and agent Git restrictions are in the
## Baseline Validation
Use Go 1.24 or newer. From the repository root:
Use Go 1.25.5 or newer. From the repository root:
```sh
go build -o ./bin/audita ./cmd/audita

View File

@@ -1,76 +1,34 @@
# OpenAI-Compatible LLM Integration
This document owns the external protocol used by Audita's structured LLM adapter.
Settings, inheritance, defaults, and credential supply belong to
[configuration](../config.md#llm-settings); implementation and scheduling belong to
[LLM runtime internals](../internal/llm-runtime.md).
Audita uses the pinned [PromptKit](https://gitea.maximumdirect.net/eric/promptkit)
built-in OpenAI-compatible transport. PromptKit owns chat-completions wire framing,
provider envelope parsing, transient retries, per-generation timeouts, and JSON
Schema repair. A backend's configured base endpoint must support the selected
profile's model and structured JSON-schema responses; local configuration
inspection does not contact the provider.
## Endpoint and Authentication
Audita selects a role profile, supplies its registered backend credential, and
passes the embedded response schema as a complete per-call output contract. It
prepares each call once, checks that the prepared target still matches the
inspected backend and endpoint, then executes that same prepared request. A
profile cannot introduce a separate credential environment lookup. An absent
backend credential permits an unauthenticated local service.
Audita sends an HTTP `POST` to the configured base URL with trailing slashes
removed and `/chat/completions` appended. Supply an HTTP or HTTPS base URL that
makes this the intended endpoint; the adapter does not require HTTPS in code.
Requests have `Content-Type: application/json`. A non-empty configured API key
adds `Authorization: Bearer <api_key>`; otherwise that header is omitted.
`llm.transient_retries` counts additional attempts for eligible transient
failures; explicit zero makes one attempt. `llm.repair_attempts` counts
additional generations for schema-invalid model output. The operation timeout
covers preparation, capacity waiting, retries, generation, and repair. Caller
cancellation or an earlier deadline also applies. See [configuration](../config.md#settings)
for defaults and [PromptKit's consumer contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/consumers/pkg-promptkit.md)
for the pinned protocol and retry details.
The endpoint must support the configured model and structured JSON-schema
responses. Local configuration validation does not probe that support.
## Request Shape
The payload contains `model`, an array of `messages` with string `role`/`content`,
and `response_format`. The structured-output portion has this illustrative shape:
```json
{
"type": "json_schema",
"json_schema": {
"name": "correction_set",
"strict": true,
"schema": {"type": "object"}
}
}
```
The actual schema is supplied for the call; the fragment above is not a complete
request schema. Runtime calls use `correction_set` and `validator_decision_set`
identities. The adapter requires a non-empty resolved model, non-empty message
roles/content, a schema name, and valid schema JSON. It sends strict mode to the
provider; local typed decoding and proposal/decision checks still apply afterward.
## Response Handling Contract
A successful response must be a JSON envelope with at least one `choices` item.
Audita reads the first choice's `message.content`, accepting a non-empty string
containing JSON or a non-null raw JSON value that can decode into the expected
output type. Empty content, missing choices, and incompatible payloads fail.
The adapter also reads optional `model` and token-usage metadata.
There is no streaming protocol or provider SDK dependency in this integration.
Audita uses this chat-completions contract, not other provider API shapes.
## Retries and Timeouts
The effective adapter retry count is additional attempts after the initial request;
see [configuration](../config.md#llm-settings) for the current zero-value behavior.
Transport/read errors, malformed response envelopes/content, typed decode failures,
HTTP `429`, and server-error responses are retryable. Other non-success HTTP
responses and a response missing choices are not retried by the adapter.
Cancellation or an expired context stops retrying.
The configured HTTP timeout applies per request; it is not a whole-process time
budget. A parent process can impose its own overall timeout. See
[configuration](../config.md#llm-settings) for timeout and retry values and
[subprocess guidance](subprocess.md) for parent responsibilities.
## Error Behavior
For non-success responses, error detail comes from provider `error.message`, then
a top-level `message`, otherwise the HTTP status and response body. Configured
keys are redacted from surfaced errors and LLM diagnostic payloads.
Malformed structured responses can become section/batch warnings and rejected
proposals rather than fatal run errors, depending on the call site. Other adapter
failures can terminate processing with partial results. Consult the
[report contract](output-files.md#module-outcomes) for those outcomes and
[operations](../operations.md#artifact-handling) for handling diagnostic contents.
A model operation that exhausts structural repair, a live-parent local operation
timeout, capacity rejection, transport failure, malformed provider response,
or provider HTTP failure is recoverable for that operation. Proposal generation
then contributes no candidates from that response; a failed validation batch
marks its candidates unavailable while other batches and later modules can
continue. Caller cancellation, broken assets or configuration, and unknown
internal failures remain fatal and cancel peers. Audita decodes into domain
types only after PromptKit validates the JSON. Ordinary reports use safe reason
codes and counts rather than provider messages. Full request/response content
is written only with `--debug`; see [operations](../operations.md#diagnostics-directory).

View File

@@ -1,186 +1,112 @@
# Output Files and Report Formats
This document owns externally consumed corrected-transcript and report formats,
and describes the correction ledger and utilization diagnostics. It documents
current serialized shapes without making every debugging artifact a versioned
compatibility promise. Physical filenames, placement, retention, and permissions
belong to [operations](../operations.md); invocation and output channels belong to
the [CLI](../cli.md#process-output-and-exit-behavior).
# Output Files and Reports
## Corrected Transcript
The [output setting](../config.md#pipeline-and-output) selects the envelope:
The corrected transcript uses the selected `output.schema`: `bare-segments`
or `audita-v1`. Its schema and text rules are unchanged by resilience
reporting. See [configuration](../config.md#settings) for selection and
[operations](../operations.md#required-output-files) for placement and writes.
| Encoder | JSON shape |
With reporting enabled, Audita produces three ordinary files:
- `errors.yml`: affected work units and run-level failures;
- `recoveries.yml`: successful logical model operations with repair, service
retry, or response filtering;
- `report.md`: a concise overview and complete per-chunk matrices.
The YAML documents have Audita-owned schemas `audita-errors` and
`audita-recoveries`, each with integer `version: 1`. Both contain, in order,
`schema`, `version`, `run_id`, `status`, `started_at`, `completed_at`, `summary`,
and an always-present `errors` or `recoveries` sequence. Status is
`in_progress`, `completed`, or `failed`. Times are UTC RFC3339 strings;
`completed_at` is null until finalization. A recoverable model error can appear
with `status: completed`.
## Summary Counts
Both documents carry the same summary, in this order:
| Count | Meaning |
| --- | --- |
| `bare-segments` | Top-level array of segment objects |
| `audita-v1` | Object with `schema: "audita-v1"`, `version: "v1"`, and `segments` containing the segment array |
| `work_units_total` | Configured module instances × initial chunks. |
| `work_units_started`, `work_units_completed`, `work_units_interrupted`, `work_units_not_started` | Explicit processing states. |
| `work_units_with_errors` | Distinct units with recoverable operation failures. |
| `run_errors` | Fatal processing or output failures outside that unit count. |
| `corrections_applied` | Approved proposals applied, not replacement occurrences. |
| `llm_operations` | Structured-client invocations, including pre-dispatch failures. |
| `proposal_responses` | Structurally usable proposal responses. |
| `recoveries` | Successful operations with one or more recovery reasons. |
| `proposal_responses_with_filtering` | Proposal responses that filtered at least one defective proposal. |
Both emit these segment fields:
Unknown counts before processing are `null`; completed empty counts are `0`.
Retries and repair generations are details of one operation, not additional
operations. Provider token usage, execution timing, and attempt-level evidence
remain in opt-in debug files.
| Field | Type | Meaning |
| --- | --- | --- |
| `id` | Integer | Identifier in the normalized transcript |
| `speaker` | String | Speaker label |
| `start`, `end` | Numbers | Start/end times in seconds |
| `text` | String | Text after normalization and accepted corrections |
| `categories` | Array of strings, optional | Segment categories; omitted when empty |
## Error and Recovery Records
Input IDs and segment boundaries need not survive normalization unchanged;
normalization can merge segments and reassign IDs. The output describes the
resulting transcript, not a lossless copy of the source JSON. Input acceptance
rules belong to the [input-file contract](transcript-glossary-files.md).
An `errors` record with `scope: work_unit` identifies zero-based `stage_index`
and `chunk_index`, module key/instance, initial segment range, applied-correction
count, and a `failures` sequence. Each failure identifies the operation
(`proposal`, `validation`, or `validation_input`), optional validator and
zero-based batch index, safe reason/message, sorted proposal indices when
known, and execution counts when available. Multiple failed batches in one
work unit form one error record. `scope: run` instead records phase, safe reason
and message, optional origin, and output path where relevant. It can describe
a fatal error or secondary output-write failure without inventing a failed
model work unit. Raw provider errors and transcript text are absent.
For `validation_decisions_invalid`, proposal indices identify only candidates
whose required decisions were unavailable. Other candidates in the same batch
may still be approved or rejected normally.
## Process Report
Each recovery identifies its work unit and successful operation, with sorted
`reasons`, a safe message, available execution counts, and sorted
`filtered_proposals` details. One recovery can contain `output_repaired`,
`service_recovered`, and `proposals_filtered` together. Filtering covers
defective response proposals; low confidence, protected-term rejection,
ordinary editorial rejection, and an apply-time conflict caused by an earlier
accepted correction are not filtering recoveries. Failed operations are errors,
not recoveries. The initial chunk identity remains stable even if earlier
modules change its text.
`validation_decisions_filtered` records unexpected validator decision indices
ignored from a successful validation batch. It may accompany repair or service
recovery reasons, but it does not add `filtered_proposals` or increment
`proposal_responses_with_filtering`. A batch with unavailable requested decisions
is an error even when unexpected indices were also ignored.
The report is a JSON object. `report_metadata` identifies its schema separately
from the transcript output envelope:
## Markdown Overview
| Metadata field | Type / current value |
| --- | --- |
| `report_schema_name` | String: `audita-process-report` |
| `report_schema_version` | String: `v1` |
| `output_schema` | String identifying the selected transcript encoder |
| `config_version` | Optional integer, present when file configuration was loaded |
`report.md` includes run ID, status, corrections, work coverage, error and
recovery totals, input/output paths, and full work-unit error and recovery
matrices. Rows show every initial chunk with 1-based chunk number and initial
segment range; columns follow configured module-instance order, including
repeats. Completed error cells are `0` or `1`; recovery cells are operation
counts. `—` means not started and `n*` means interrupted with n facts recorded.
An empty stage or chunk set is identified as having no work units. Short reason breakdowns count
records containing each reason; a recovery can have several reasons, so those
counts need not sum to the recovery total. The report refers to both YAML
files. It does not include transcript excerpts or correction lists.
The remaining top-level fields are:
## Debug-Only Files
| Field | Type and meaning |
| --- | --- |
| `phase` | String; currently `default_pipeline` |
| `status` | String: `success` or `failed` |
| `operation` | String; currently `process` |
| `transcript_path`, `glossary_path` | Input path strings |
| `output_path` | Optional transcript destination string |
| `modules` | Array of configured module keys, including repeats |
| `started_at`, `completed_at` | RFC 3339 timestamps, possibly fractional; completion is an optional field populated by completed process runs |
| `error_phase`, `error_message` | Optional failure classification and human-readable detail |
| `input_segment_count`, `normalized_segment_count` | Optional integer normalization counts |
| `normalization_merges`, `normalization_id_reassignments` | Optional integer transformation counts |
| `normalization_skipped` | Object of optional integer counters: `different_speakers`, `gap_too_large`, `duration_exceeded`, `token_limit_exceeded`; can be empty |
| `chunking` | Optional section summary described below |
| `diagnostics` | Optional artifact-path metadata described below |
| `modules_summary` | Optional aggregate module outcome |
| `module_results` | Optional array of module outcomes |
Failures can preserve partial module results. Missing summaries indicate unavailable
information, not necessarily zero work. Early configuration failures occur before
report construction. Report success describes processing results; a later file-write
or cleanup error can still cause a nonzero CLI exit. Use the exit result and report
together as explained in [subprocess guidance](subprocess.md).
Error phases identify where processing failed, for example `transcript_read`,
`glossary_read`, `transcript_schema`, `glossary_schema`, `chunking`, `runner_setup`,
`runner_execution`, `output_schema`, `serialization`, `output_write`, and
`stdout_write`. This is not an exhaustive enumeration or a promise that every
late failure is represented in a successfully written report. Error messages are
human-readable, not structured fields to parse for control flow.
### Chunking and Diagnostics Metadata
`chunking` contains integer fields `chunk_count`, `min_estimated_chunk_tokens`,
`max_estimated_chunk_tokens`, `total_estimated_transcript_tokens`,
`max_section_tokens`, and `min_section_tokens`, plus optional integer
`target_sections`. Token counts are estimates.
`diagnostics` contains optional path strings:
- `directory_path`
- `source_transcript_path`, `parsed_source_transcript_path`, `normalized_transcript_path`
- `normalization_summary_path`, `chunking_summary_path`
- `utilization_summary_path`, `correction_ledger_path`
- `invocation_metadata_path`, `redacted_effective_config_path`, `error_log_path`
The [operations guide](../operations.md#diagnostics-directory) defines the physical
artifacts and path-lifetime caveats.
### Module Outcomes
`modules_summary` contains integer `module_count`, `total_applied_changes`, and
`total_skipped_changes`, plus optional string `failed_module_instance`.
`total_skipped_changes` counts both apply-time skipped changes and validator
rejections. Their detailed records are stored in separate arrays.
Each `module_results` object contains:
| Field | Type and meaning |
| --- | --- |
| `module_key`, `module_instance` | Strings; the instance distinguishes repeated occurrences of a configured module |
| `replacement_policy` | Optional string describing the module's replacement policy |
| `status` | String describing module outcome (`success` or `failed`) |
| `proposal_count` | Integer |
| `started_at`, `completed_at` | Optional timestamps |
| `error_message` | Optional failure detail |
| `warnings` | Optional array of warning records |
| `validator_decisions` | Optional array of decision records |
| `validator_rejected` | Optional array of rejected-correction records |
| `applied_changes`, `skipped_changes` | Optional arrays of apply outcomes |
A validator decision has `validator_name` (string), `proposal_index` (integer),
`approved` (boolean), `reason_code` and `message` (strings), and optional
`diagnostic_artifact_path` (string).
Rejected corrections include `validator_name`, `proposal_index`, `module_key`,
`module_instance`, `target_segment_id`, `original_text`, `corrected_text`,
`reason_code`, and `message`. Applied changes include the same correction identity
and text fields without validator/reason/message fields, plus integer
`replacement_count`. Skipped changes include correction identity/text fields,
`skip_reason`, and `message`.
A warning has string `scope`, `reason_code`, and `message`; optional integer
`section_index` and `batch_index`; and optional string `validator_name` and
`diagnostic_artifact_path`. Warnings, skipped changes, and rejected corrections
can occur without a failed run. Proposal indexes associate outcomes with their
proposal within a module instance; they are not transcript segment IDs.
## Correction Ledger
The ledger is an array of correction entries; it can be `null` when no module
results are available and empty when modules produced no entries. Unlike the
process report, it has no separate schema-version envelope.
Each entry has required `module_key`, `module_instance`, integer `proposal_index`,
and `disposition`. The current dispositions are:
| Disposition | Meaning |
| --- | --- |
| `applied` | The proposed change was applied |
| `skipped` | The proposal reached application but was skipped |
| `rejected` | A validator rejected the proposal |
| `failed` | Module-level failure record, not necessarily an individual proposal |
Optional fields are `run_id`, integer `segment_id`, `original_text`,
`proposed_corrected_text`, `applied_corrected_text`, `replacement_policy`,
`disposition_reason_code`, and `disposition_message`. Optional validator arrays
are `deterministic_validator_decisions` and `llm_validator_decisions`; each record
contains string `validator_key`, boolean `approved`, string `reason_code`, and
optional string `message`.
Module failure entries can have the zero-value proposal index and no correction
text or segment. Do not interpret them as evidence that proposal zero failed.
Empty optional values are omitted. The ledger records disposition, not a replay
or resume protocol.
## Utilization Diagnostics
This unversioned JSON object describes observations from the runtime, not service
limits or performance guarantees:
- `effective_concurrency`: integer `total_llm`, `proposal_llm`, `validation_llm`.
- `run_timing`: millisecond integers `run_wall_time_ms`, `scheduler_queue_wait_ms`,
`llm_execution_time_ms`, `deterministic_validation_time_ms`; integer
`max_in_flight_llm_calls` and `average_in_flight_llm_calls`.
- `llm_calls`: integer `total_proposal_calls` and `total_validation_calls`.
- Optional `modules`: records with `module_key`, `module_instance`, integer
`proposal_llm_calls` and `validation_llm_calls`, and millisecond integer
`scheduler_queue_wait_ms`, `llm_execution_time_ms`,
`deterministic_validation_time_ms`, and `module_wall_time_ms`.
- Optional `validators`: records with `module_key`, `module_instance`,
`validator_key`, boolean `llm_backed`, integer `calls`, and millisecond integer
`elapsed_ms`.
Aggregated execution/wait durations can overlap in concurrent work; they need
not sum to wall time. The observed average concurrency is serialized as an integer.
Other source snapshots, prompt payloads, and diagnostic files are debugging
material; their placement is documented in operations without defining a separate
public schema for every file.
With `--debug`, `correction-ledger.json` contains proposed candidates and
their application, rejection, or interruption details. `llm-execution.json`
holds logical calls and PromptKit execution events; `utilization-diagnostics.json`
groups backend capacity and timing observations while preserving known and
complete usage semantics. Interaction request/response artifacts, selected
profiles, configuration provenance, and input snapshots are also debug-only.
These files are diagnostic evidence, not an alternate ordinary report schema.
Each candidate-backed ledger entry includes `proposed_corrected_text`, the
literal replacement span, even when it is `""` for a deletion. Applied entries
also include `applied_corrected_text`, likewise `""` for an applied deletion.
Rejected, unavailable, skipped, and interrupted entries omit the applied field; a module
failure without a candidate omits both. These fields are replacement spans,
not full resulting-segment text. Deterministic decision records now identify
the blank-result guard as `non_empty_resulting_segment` (formerly
`non_empty_corrected_text`); its `empty_resulting_segment` reason is unchanged.
Validator decision records carry `disposition`: `approved`, `rejected`, or
`unavailable`. A missing or duplicate required decision is `unavailable`, not
an editorial rejection. The candidate ledger uses the same distinction in its
top-level disposition. Interaction response artifacts retain supplied model
decisions and the reconciliation result, including unavailable and ignored
indices; unexpected indices never create candidate ledger entries.

View File

@@ -7,10 +7,13 @@ The [CLI reference](../cli.md) owns command syntax, streams, and exit semantics;
## Invocation
Use an argument array with the transcript path and required glossary argument,
following the [process examples](../cli.md#process-examples). For machine workflows,
select explicit transcript and report destinations rather than depending on the
working directory or run-directory retention. Supplying a configuration path also
makes file selection predictable; inherited environment overrides still apply.
following the [process examples](../cli.md#commands-and-help). For machine workflows,
select an explicit transcript file destination and a configuration path. Reports
then go beside the transcript. For stdout output, select `--report-dir` or use
the invocation working directory. Debug is independent and off by default.
Supplying a configuration path also
makes file selection predictable; only explicitly referenced credential variables
are read from the environment.
Resolve the executable deliberately and set the child's working directory so
relative input/output paths and reported artifact paths have a known base.
@@ -20,21 +23,26 @@ or cancellation in the parent process when needed.
## Result Handling
Interpret the exit result according to the
[CLI output and exit contract](../cli.md#process-output-and-exit-behavior).
[CLI output and exit contract](../cli.md#streams-and-exit-status).
Consume transcript/report data through the
[output-file contract](output-files.md), and treat stderr as human-readable detail.
The report supplements the exit result: it may be absent after early failures,
and output may have been written before a later reporting or cleanup failure.
The three reports may be absent after early configuration or destination
failures. After initialization, failed reports can describe invalid input or
partial processing. Exit `0` can include recoverable model errors and applied
corrections; inspect `errors.yml` and `recoveries.yml` for those outcomes. A
transcript may have been written before a later required report failure, so
the file alone does not prove completion.
If processing fails, preserve available evidence and follow
[operations recovery](../operations.md#failure-inspection-and-recovery).
For reports retained outside the run directory, account for the
[artifact lifecycle](../operations.md#retention-behavior) before dereferencing
reported paths. Neither the existence of an output file nor a diagnostics path
alone establishes successful completion.
Ordinary reports are independent of optional retained debug evidence. Parent
cancellation reaches the child through its context; Audita joins workers and
attempts failed-status report finalization before exiting. Uncatchable
termination can leave `in_progress` reports. Neither the existence of an output
file nor a debug directory alone establishes successful completion.
## Related Contracts
- [Transcript and glossary inputs](transcript-glossary-files.md)
- [LLM endpoint compatibility](openai-compatible-llm.md)
- [Configuration inspection commands](../cli.md#configuration-commands)
- [Configuration inspection commands](../cli.md#commands-and-help)

View File

@@ -42,7 +42,7 @@ The top-level object contains a `glossary` list with at least one entry.
The parser checks empty strings without trimming them. Unknown YAML fields are
ignored by the glossary parser, unlike the stricter
[configuration-file schema](../config.md#file-schema).
[configuration-file schema](../config.md#loading-and-precedence).
The [tiny glossary](../../examples/tiny-glossary.yaml) is a complete copyable input.
Malformed YAML, an empty/missing glossary, invalid shapes, missing required text,

View File

@@ -7,9 +7,9 @@ CLI wiring, and maintained examples. Definitions of behavior belong to the
## Changing Configuration
1. Update fields, defaults, loading, and validation in `internal/core/config`.
2. Update file, environment, and CLI mapping as applicable, including CLI flag
extraction. Preserve the source-specific behavior defined in the
[configuration contract](../config.md#effective-precedence), including the
2. Update file, selected overlay, credential reference, and CLI mapping as
applicable. Preserve the precedence defined in the
[configuration contract](../config.md#loading-and-precedence), including the
distinct validation and effective-config commands.
3. Update redaction when secrets are involved.
4. Review configuration tests and related CLI tests for precedence, validation,
@@ -35,12 +35,13 @@ From the repository root, validate the maintained configuration examples:
```sh
go run ./cmd/audita config validate --config examples/minimal-config.yml
go run ./cmd/audita config validate --config examples/production-config.yml
go run ./cmd/audita config print-effective --config examples/minimal-config.yml
AUDITA_SERVICE_KEY=synthetic-example go run ./cmd/audita config validate --config examples/production-config.yml
AUDITA_SERVICE_KEY=synthetic-example go run ./cmd/audita config validate --config examples/production-config.yml --overlay testing
go run ./cmd/audita config print-effective --config examples/minimal-config.yml --view profiles
```
The effective-config command incorporates environment settings; use a controlled
environment when comparing its results. Check affected CLI examples against the
Only referenced credentials read environment values; use a controlled synthetic
value when validating the service example. Check affected CLI examples against the
parser and use schema tests for maintained transcript/glossary inputs. Example
validation should not invoke a live LLM merely to check documentation.
@@ -58,7 +59,8 @@ Select checks according to the [testing policy](../policy/testing.md):
- Schema or serialization changes: review [output-schema tests](output-schemas.md#key-tests)
and `internal/core/schema` tests, including supported formats and rejection of
unsupported schemas.
- Changes affecting assembled output: inspect CLI parity and release fixtures.
- Changes affecting assembled output: inspect controlled HTTP subprocess workflows
and retained report/artifact assertions.
Use the [contributor guide](../development.md#baseline-validation) for baseline
commands. These recipes supplement, rather than duplicate, public contracts.

View File

@@ -1,74 +1,63 @@
# Diagnostics and Report Assembly
# Diagnostics and Reporting Internals
## Scope
The [output-file contract](../integrations/output-files.md) owns public
schemas and the [operations guide](../operations.md) owns physical placement.
This document owns implementation flow and collaborators. Physical artifacts and
retention belong to [operations](../operations.md). Serialized reports, ledger
records, and utilization shapes belong to the
[output-file contract](../integrations/output-files.md).
`internal/framework/outcomes` contains processing-owned operation references,
counts, failure reasons, filtering details, and work states. It imports no
reporting, CLI, or provider package. The runner records those facts per initial
section and module instance. `internal/framework/processreport` maps them once
into `internal/core/reporting.Snapshot`, including nullable counts, grouped
work-unit errors, separate run errors, and one recovery per successful logical
operation. Reports order stages, sections, validators, and batches by semantic
identity, never concurrent completion time. The core reporting package owns
Audita's versioned YAML DTOs, fixed Markdown rendering, and brief CLI
completion formatter. YAML redaction walks serialized string nodes so numeric
and boolean values retain their types. Ordinary reports omit raw causes,
transcript text, complete decisions, and token/timing payloads.
Validation operations with unavailable decisions report only affected requested
indices, even when other candidates in the batch continue. Ignored extra
validator entries yield `validation_decisions_filtered` only for a successful
operation; they do not count as filtered correction proposals. Failed operations
retain their execution counts and never produce a recovery record.
## Run Directory and Artifact Writers
`internal/core/diagnostics.OutputSession` owns active destination checks,
transcript preflight, report initialization, regular-file replacement, and direct
stream publication. Transcript input is protected from early report writes but
may be the final transcript destination. The LLM runtime records successful reads
through PromptKit's profile filesystem and supplies those source paths to CLI;
no second filename-based discovery rule is used.
CLI composes it after configuration/profile inspection and before reading
inputs or dispatching model work. It does not create ordinary report parents.
On a final write error it attempts other enabled reports and one failed-status
pass; a published transcript is never erased by a later report failure.
`internal/core/diagnostics` creates run directories, writes snapshots and metadata,
and evaluates retention. CLI orchestration supplies the selected configuration,
paths, run outcome, and correction-presence information. Artifact names are
centralized in diagnostics; their operator-facing inventory is in
[operations](../operations.md#diagnostics-directory).
Snapshot writes are attempted at the relevant processing phases. Several diagnostic
writes are best-effort, so report metadata can identify a location without proving
that a file was written. The CLI distinguishes primary processing errors from
secondary diagnostic/report write errors.
## Process Report Mapping
`internal/framework/processreport.Build` maps invocation facts, normalization and
chunking summaries, and runner output into `reporting.ProcessReport`. Core reporting
owns the Go data structures and JSON writer. The builder attaches diagnostic
metadata when a run-directory path is available.
Module mapping copies outcomes from runner results, collects summary counts, and
preserves available partial module results on failure. The CLI failure path does
not pass normalization or chunking summaries to report construction even when
some earlier processing completed. Consumer interpretation belongs to the
[process-report contract](../integrations/output-files.md#process-report).
## Correction Ledger and Utilization
`BuildCorrectionLedger` flattens module changes, rejections, and failures into
ledger entries. It associates validator records by proposal index within each
module, separates execution classes through validator metadata, and sorts entries
by module instance, proposal index, and disposition.
Runner instrumentation collects utilization; CLI orchestration writes it when
run output is available. Report assembly does not independently measure calls.
For the serialized forms see [ledger](../integrations/output-files.md#correction-ledger)
and [utilization](../integrations/output-files.md#utilization-diagnostics).
## CLI Coordination and Retention
CLI orchestration writes caller-selected reports and run-local artifacts before
asking diagnostics to apply retention. `HasSkippedCorrections` derives the
retention input from runner outcomes. Keep the exact lifecycle definition in
[operations](../operations.md#retention-behavior), not in another internal table.
On a processing failure, secondary report-write errors do not replace the original
error. On the success path, an external report-write failure returns before the
run-local report write; retention errors occur later. This ordering explains why
artifacts and process exit status must be assessed together.
## Redaction
Effective configuration uses config redaction. LLM diagnostic and error paths
use `llm.ConfiguredSecrets` and the runtime's redaction helpers. Keep these separate
mechanisms aligned with the [architecture requirement](../policy/architecture.md#errors-and-secret-handling).
Operational handling of transcript-bearing artifacts belongs to
[operations](../operations.md#artifact-handling).
Debug is independent. Only `diagnostics.debug` creates a unique retained child
under `diagnostics.work_dir`. CLI passes empty diagnostics paths and nil writers
when debug is off. With debug on, source/normalized snapshots, invocation,
effective settings, origins, profiles, initial chunk summary, correction ledger,
indexed execution/utilization, and model interactions are written. Runner
candidate capture includes proposals later rejected or interrupted. Ledger
mapping uses optional string pointers so an empty proposed replacement is
serialized for every candidate and an empty applied replacement appears only
when that candidate applied; see the [public artifact contract](../integrations/output-files.md#debug-only-files).
Debug write failures reach the CLI through an explicit concurrency-safe diagnostic
callback and produce redacted stderr messages without aborting processing.
Completed validator decisions returned with a fatal batch error remain available
to the debug ledger; they never authorize applying an incompletely validated edit.
Runner decision records carry the validator's explicit approved, rejected, or
unavailable disposition into the ledger. Unavailable candidates have a distinct
top-level ledger disposition, while fatal interruptions remain interrupted when
no rejection or unavailable decision was returned. Successful validation
interaction payloads contain the full decoded response, final content, and
reconciliation outcomes; ignored indices do not become ledger candidates.
Every debug writer uses all configured backend secrets for redaction, including
unselected backend credentials and source snapshots. Debug can still contain
sensitive transcript and prompt content.
## Key Tests
- `internal/core/diagnostics/*_test.go`
- `internal/framework/processreport/*_test.go`
- `internal/core/reporting/report_test.go`
- `internal/cli/run_test.go`
- `cmd/audita/main_integration_test.go`
Useful tests: `internal/core/reporting/*_test.go`,
`internal/core/diagnostics/*_test.go`,
`internal/framework/processreport/*_test.go`, and
`cmd/audita/main_integration_test.go`.

View File

@@ -1,68 +1,39 @@
# Audita LLM Runtime
## Scope
This document describes the structured LLM runtime and scheduler behavior.
## Client Boundary
All runtime LLM calls go through `contracts.StructuredLLMClient`.
`llm.NewRuntime` assembles one PromptKit engine for both roles. It loads embedded
prompt and schema assets, registers application backends, and inspects the
selected proposal and validation profiles offline. A role-bound `llm.RoleClient`
implements `structuredoutput.Client`; proposal and validator contracts alias that
interface. Both roles share engine-local backend capacity.
Primary adapter:
- `internal/framework/llm/OpenAICompatibleClient`
For each logical call, the adapter starts an operation context and a diagnostic
record before preparation. It supplies the reserved hardening text, the selected
profile and resolved backend credential, and the complete embedded schema/repair
contract. `PrepareExecution` produces the exact messages and hashes retained in
an Audita snapshot. The adapter checks the prepared backend and endpoint before
`RunPrepared`. It records available final content, usage, model, and profile
metadata and decodes only a validated result. Call hooks expose logical
start/finish records and PromptKit execution events for debug utilization.
## Adapter and Response Schema Registry
The [integration contract](../integrations/openai-compatible-llm.md) describes
wire ownership and failure behavior. `structuredoutput.Failure` has a safe
message, fixed reason code, typed cause, and recoverable/fatal/canceled class.
Counts distinguish provider attempts, transient retries, and structural
repairs, including on errors. Secrets from all configured backends, including unselected
ones, are available to the diagnostics redaction boundary.
The adapter implements the [LLM wire contract](../integrations/openai-compatible-llm.md).
It maps internal structured requests to HTTP messages and decodes provider content
into the supplied Go output target. Endpoint, authentication, response shapes, and
retry conditions are defined in that contract rather than repeated here.
## Orchestration
`internal/framework/responseschema` stores the per-call response schema definitions
and diagnostic metadata. Proposal generation and validation choose their schema
before invoking the client. Provider strict mode does not replace local checks.
## Error Classification
The adapter marks retryable errors internally and uses context state and attempt
count to decide whether to retry. It sanitizes errors before returning them.
`internal/framework/structuredoutput` classifies malformed output for proposal
and validator call sites, which can turn those errors into warnings and rejected
section/batch work. Other errors propagate through the runner.
## Scheduling and Concurrency
`internal/framework/llm/Scheduler` provides FIFO, context-aware permit gating.
CLI orchestration composes global and stage schedulers; runner instrumentation
measures them. Limit definitions and inheritance belong to
[configuration](../config.md#concurrency).
Scheduler release is guarded to avoid permit leaks on cancellation/error. Runner
collects utilization observations; their serialized shape is defined by the
[output-file contract](../integrations/output-files.md#utilization-diagnostics).
## Diagnostics and Redaction
`internal/framework/llm/DiagnosticsWriter` writes request/response/error artifacts.
Configured secrets are derived from `llm.ConfiguredSecrets(cfg)` and redacted from:
- diagnostics payloads;
- surfaced runtime/adapter errors.
## Changing LLM Integration or Scheduling
Review client, scheduler, redaction, and structured-output malformed-response
coverage alongside runner orchestration and utilization tests. Use the
[testing policy](../policy/testing.md) to select sufficient checks; preserve the
[LLM integration contract](../integrations/openai-compatible-llm.md) and
[architecture invariants](../policy/architecture.md#correction-and-concurrency-invariants).
## Key Tests
- `internal/framework/llm/openai_compatible_client_test.go`
- `internal/framework/llm/scheduler_test.go`
- `internal/framework/llm/diagnostics_test.go`
- `internal/framework/responseschema/registry_test.go`
- `internal/framework/structuredoutput/malformed_test.go`
- `internal/framework/runner/*_test.go` (orchestration and utilization)
CLI processing supplies two role clients from the same engine to the runner.
PromptKit's per-backend admission applies to both roles and every generation,
retry, and repair. The runner still processes module instances in order,
collects concurrent section work deterministically, and cancels remaining
section work only on fatal failures. `llm.DiagnosticsWriter` remains the protected
request/response artifact writer. `llm.ObservationCollector` captures metadata
events under a short mutex and freezes them after runner work joins. Its logical
records include preparation failures; summary usage comes from terminal events
once per call and retains knownness and completeness for each field. Runner wall
and validator timings remain separate from provider attempts. Ordinary reports
derive compact runner outcomes; full observations are debug-only.

View File

@@ -3,7 +3,7 @@
## Scope
This document covers internal module contracts and packages. User-selectable keys
and the default sequence belong to [configuration](../config.md#pipeline-and-output).
and the default sequence belong to [configuration](../config.md#settings).
## Module Contract
@@ -35,34 +35,49 @@ Shared proposal-generation plumbing is centralized in:
- `internal/framework/proposal_generation`
Module packages own:
- prompt selection (`internal/prompts` prompt IDs);
- module-specific prompt payload construction.
- prompt selection (`internal/prompts` prompt IDs).
Shared prompt helpers live in `internal/framework/promptcontext`.
`proposal_generation.BuildProposalVariables` encodes the glossary, selected
transcript section, and trimmed description once for all four modules. PromptKit
renders their embedded definitions from those variables and validates the
correction-set schema before Audita maps proposals. Domain validators still
check confidence, applicability, protected glossary terms, and replacement
policy. Model-facing segment IDs are one-based positions within the proposal
request. Generation maps returned references back to the current chunk's actual
segment IDs, preserving response order and multiple proposals per segment.
Unknown references become invalid-target candidates for the existing domain
checks; they are never interpreted as actual transcript IDs. Debug request
metadata records `segment_ids` in local-reference order (reference 1 is the
first entry). Shared context encoding lives in `internal/framework/promptcontext`.
An empty `corrected_text` deletes the identified span when the resulting segment
remains nonblank. Whitespace-only original spans can express spacing corrections;
matching stays literal under each module's replacement policy. No automatic
whitespace cleanup follows application.
## Validator Chain Ownership
Built-in chains are resolved in `internal/validators` per module key.
Module packages call the built-in chain resolver at construction.
## Failure and Warning Behavior
## Failure and Outcome Behavior
Module construction failures propagate to orchestration; execution failures can
preserve partial module results. Malformed structured proposal payloads become
warnings and section-level rejection. Consumer-visible outcomes are defined by
the [report contract](../integrations/output-files.md#module-outcomes).
Module construction failures are fatal. Recoverable proposal model failures
record an unsuccessful operation and yield no candidates for that section;
other sections and module instances continue. Fatal adapter or application
failures cancel peers and retain partial observations. Consumer-visible
outcomes are defined by the [report contract](../integrations/output-files.md).
## Changing Modules
1. Add or update the narrowly scoped package under `internal/modules/<module_key>`.
2. Keep module-specific prompt ownership in the module and `internal/prompts`;
see [prompt rendering and metadata](prompts.md).
2. Keep module prompt selection in the module and its definition in
`internal/prompts`; see [prompt assets](prompts.md).
3. Wire module registration and catalog resolution through framework module
registration and the core module catalog. Update selectable keys in the
[configuration contract](../config.md) when they change.
4. Verify replacement policy and validator-chain selection against the
[architecture requirements](../policy/architecture.md#correction-and-concurrency-invariants).
5. Review module, prompt-rendering, and proposal-generation tests. Inspect parity
5. Review module, prompt-variable, and proposal-generation tests. Inspect parity
and release fixtures when public output changes, choosing sufficient coverage
under the [testing policy](../policy/testing.md).
6. Update this document for implementation changes and the appropriate public

View File

@@ -6,13 +6,13 @@
JSON envelopes and segment fields are defined by the
[output-file contract](../integrations/output-files.md#corrected-transcript);
selectable values and defaults belong to
[configuration](../config.md#pipeline-and-output).
[configuration](../config.md#settings).
## Resolution and Encoding
Configuration validation checks registry support. CLI orchestration resolves the
selected encoder, passes the final transcript to it, and routes the encoded bytes
to the destination defined by the [CLI contract](../cli.md#process-output-and-exit-behavior).
to the destination defined by the [CLI contract](../cli.md#streams-and-exit-status).
An unsupported key fails validation or resolution rather than selecting an
implicit fallback.

View File

@@ -47,15 +47,18 @@ Supporting areas:
## Current Scope
Runtime state is held in memory plus per-run diagnostics artifacts. Audita does
Runtime state is held in memory; optional debug artifacts are retained only
when requested. Audita does
not currently implement a plugin framework, generic workflow engine,
resume/checkpoint/manifest system, remote storage, or multi-process service mode.
Its provider adapter is OpenAI-compatible HTTP behind the structured LLM client
contract. These are current implementation boundaries, not bans on future work.
Its role-bound PromptKit adapter implements the structured LLM client contract.
These are current implementation boundaries, not bans on future work.
## Main Execution Path (`audita process`)
The CLI prepares input and configuration, invokes the runner, and writes results.
The CLI inspects configuration and profiles, preflights and initializes active
outputs, optionally creates debug evidence, invokes the runner, and finalizes
transcript and reports.
The [pipeline](pipeline.md) owns detailed state transitions; the
[operations guide](../operations.md) owns the operator workflow and physical
artifact lifecycle.
@@ -67,8 +70,8 @@ transcript and glossary schemas, normalization and chunking, output encoding,
diagnostics artifact helpers, and public report structures.
Framework components implement module and validator contracts, proposal
generation and application, prompt context, runner orchestration, LLM scheduling
and transport, response schemas, and process-report/correction-ledger assembly.
generation and application, prompt context, runner orchestration, PromptKit
execution and observation mapping, response schemas, and process-report/correction-ledger assembly.
Domain packages provide correction modules, built-in validator chains and
execution metadata, and embedded prompt assets.

View File

@@ -1,66 +1,45 @@
# Audita Internal Pipeline
## Scope
The [input contract](../integrations/transcript-glossary-files.md) owns input
formats; [configuration](../config.md) owns settings; the
[output contract](../integrations/output-files.md) owns serialized results.
This document describes processing collaborators and state transitions. External
input definitions belong to the [input-file contract](../integrations/transcript-glossary-files.md),
settings to [configuration](../config.md), and serialized results to the
[output-file contract](../integrations/output-files.md).
1. CLI filesystem adapters read transcript and glossary files. Schema parsing
produces source objects; transcript segments enter canonical normalization.
2. Normalization sorts and merges segments using configured time and token
limits, reassigning IDs where needed.
3. The runner partitions the normalized transcript exactly once before its
module loop. Initial section identity, segment membership, and token
estimates stay fixed while text evolves.
4. Module instances execute in configured order. Sections may propose
concurrently, but proposal indices follow section order and response order.
5. Deterministic validators precede LLM-backed validators. Every submitted
candidate receives one decision per validator on a normal return. Fatal returns
may preserve a checked subset of completed decisions as diagnostic evidence.
Recoverable failed batches mark their candidates unavailable; successful
survivors continue through all required validators.
6. Only fully approved proposals reach application. Corrections apply in
proposal-index order within their owning section. Existing replacement
policies and apply-time safety checks resolve stale conflicts.
7. CLI encodes the final transcript and publishes active files through the
output session. `framework/processreport` builds one snapshot for YAML,
Markdown, and CLI totals. Debug evidence is captured separately when enabled.
## Inputs
`runner.Run` returns the final working transcript, initial section inventory,
explicit `not_started`/`interrupted`/`completed` work units, compact model
operation outcomes, and per-module results. Candidate payloads are retained
in the returned output only when debug capture is requested. A work unit can
complete with model errors and applied corrections. Recoverable model failures
never trigger shared cancellation. Fatal setup, internal, application, or
parent-cancellation errors cancel peers, join workers, and preserve earlier
committed module text for partial reports; no completed transcript is published.
Pipeline inputs are:
- effective config (`internal/core/config`);
- transcript JSON (`internal/core/schema`);
- glossary YAML (`internal/core/schema`).
Required transcript/report write failures are fatal. Optional debug writes
report safely to stderr without changing decisions or exit status. See
[operations](../operations.md) for file placement and failure handling, and
[architecture policy](../policy/architecture.md#correction-and-concurrency-invariants)
for invariants.
## Pipeline Phases
1. **Input loading:** CLI filesystem adapters read inputs; schema parsing produces
source transcript and glossary objects. Source segments are converted to the
canonical transcript shape before normalization.
2. **Normalization:** the normalizer sorts and merges segments using its configured
time/token rules, reassigns IDs, and produces summary counters.
3. **Chunking:** the chunker partitions the normalized transcript and returns
sections and token estimates, or fails when constraints are impossible.
4. **Proposal generation:** the runner executes configured module instances in
sequence. Sections can generate proposals concurrently; results are collected
for deterministic indexing.
5. **Validation:** deterministic validators run before LLM-backed validators.
LLM validation is batched by prompt budget; candidate decisions feed application.
6. **Application:** `internal/framework/proposals` clones transcript state and
applies approved proposals in index order through replacement-policy checks.
7. **Serialization and reporting:** the CLI encodes the final transcript, maps
runner output into reports, writes artifacts, and delegates retention to
diagnostics. See [report assembly](diagnostics-reporting.md).
## Runner Outputs
`runner.Run` returns:
- final transcript;
- per-module results;
- utilization diagnostics.
CLI/reporting then map this into process report and diagnostics artifacts.
## Failure Behavior
CLI orchestration labels failures at the phase boundary. Runner execution failures
can return completed or partial module results, which report assembly preserves.
[Report fields](../integrations/output-files.md#process-report) define how consumers
see these outcomes; [CLI semantics](../cli.md#process-output-and-exit-behavior)
define streams and exits. Diagnostic write failures do not always abort processing.
## Ordering and State
The runner mutates its working transcript between module instances, after collecting
section results and validating proposals. Proposal indexes follow section order;
reports are derived from actual run results. Global requirements are owned by the
[architecture policy](../policy/architecture.md#correction-and-concurrency-invariants).
## Key Tests
- `internal/framework/runner/runner_test.go`
- `internal/framework/proposal_generation/*_test.go`
- `internal/cli/run_test.go`
- `cmd/audita/main_integration_test.go`
Key tests: `internal/framework/runner/*_test.go`,
`internal/framework/validators/*_test.go`, and
`cmd/audita/main_integration_test.go`.

View File

@@ -1,62 +1,59 @@
# Audita Prompt Registry
## Scope
`internal/prompts` embeds seven PromptKit YAML definitions under
`assets/prompts`. The IDs and selected version remain:
This document describes embedded prompt assets, prompt metadata, and rendering behavior.
- `modules.glossary.proposal` (`v1`)
- `modules.homophones.proposal` (`v1`)
- `modules.spoken_word.proposal` (`v1`)
- `modules.grammar.proposal` (`v1`)
- `validators.spoken_form_plausibility` (`v1`)
- `validators.meaning_reversal_review` (`v1`)
- `validators.editorial_review` (`v1`)
## Registry Ownership
PromptKit reads the embedded definitions and renders their system and user
messages from named string variables. The role-bound adapter supplies
`HardeningPolicy` alongside `ProposalContract` and `ValidationContract` from
embedded fragments under `assets/shared`, after caller variables so callers cannot replace those
instructions. Common literal-span and identity guidance lives in the proposal
fragment; shared candidate-reference, context, and confidence guidance lives in
the validation fragment. Module-specific policies and examples remain in YAML.
Proposal prompts receive the section JSON, glossary JSON where used, and transcript description block.
Validator prompts receive the correction payload and description block.
The prompt assets retain the prior task instructions and output-schema
association. The role-bound runtime supplies the complete JSON Schema output contract
and repair count; the definition does not own an operator-facing repair default.
The correction-set schema requires a string replacement and accepts an empty
string. Proposal guidance uses it for eligible span deletion and permits literal
spacing edits; validator guidance evaluates the full before/after segment under
each validator's existing remit. A blank resulting segment remains a domain
rejection, not a structural schema error.
Prompt registry lives in `internal/prompts` and embeds assets under `internal/prompts/assets/**`.
`internal/framework/responseschema` remains the only embedded source for
`correction_set.v1.json` and `validator_decision_set.v1.json`. It exposes that
filesystem to PromptKit and retains Audita's schema-key lookup. PromptKit's
prepared execution supplies definition and rendered hashes; Audita does not
recompute or promise equality with hashes from the removed template renderer.
The small Audita registry records only prompt ID, version, source, and embedded
path. Prepared runtime metadata belongs in diagnostics rather than the registry.
Registered prompt IDs:
- `modules.glossary.proposal`
- `modules.homophones.proposal`
- `modules.spoken_word.proposal`
- `modules.grammar.proposal`
- `validators.spoken_form_plausibility`
- `validators.meaning_reversal_review`
- `validators.editorial_review`
- `validators.grammar_review`
- `validators.spoken_word_review`
The structured-call contract carries prompt ID/version, string variables,
response-schema key, and logical-call attribution. Module and validator callers
pass these values through the role-bound adapter. See the
[LLM runtime](llm-runtime.md) for runtime assembly and
[configuration](../config.md) for operator-selected profiles.
## Metadata Model
Proposal instructions describe literal matching, each module's replacement policy,
source-based proposal identity, and confidence in whether an edit is warranted.
Proposal `id` and validator `correction_index` values are short, one-based
references scoped to each request, not internal transcript/proposal identities.
Mapping belongs to [proposal generation](modules.md) and
[validator reconciliation](validators.md). Validator instructions distinguish
confidence in the verdict from the verdict itself. Compact output examples
illustrate the supplied schema without requiring fixed JSON formatting or key
order. The grammar article exception and glossary mishearing exception apply
consistently across their instructions.
Each prompt has metadata:
- `prompt_id`
- `prompt_version`
- `prompt_source`
- `embedded_path`
- `sha256`
Current source/version values:
- `prompt_source = builtin`
- `prompt_version = v1`
## Rendering
`prompts.RenderUserSystem(promptID, data)` renders system/user templates.
Template behavior:
- uses Go `text/template`;
- `missingkey=error` is enabled;
- output is trimmed.
A shared hardening fragment is embedded once and referenced by prompt templates.
## Prompt Context Inputs
Shared prompt payload helpers:
- transcript section JSON (`internal/framework/promptcontext/MarshalTranscriptSectionJSON`)
- transcript description block (`TranscriptDescriptionBlock`)
Modules and LLM validators provide typed data maps to render prompt assets.
## Diagnostics Integration
Prompt metadata is attached to proposal/validator diagnostics request metadata using `Metadata.DiagnosticsMap()`.
## Key Tests
- `internal/prompts/registry_test.go`
- `internal/framework/promptcontext/*_test.go`
- module and validator prompt builder tests
Only the three active validator definitions are retained. The unused internal
`grammar_review` and `spoken_word_review` definitions and type selections were
removed; built-in chains and operator configuration are unchanged.

View File

@@ -4,7 +4,7 @@
This document describes internal validator composition, execution order, and
decision handling. These registry keys identify implementation components; they
are not independently selectable through [configuration](../config.md#pipeline-and-output).
are not independently selectable through [configuration](../config.md#settings).
## Ownership
@@ -23,7 +23,8 @@ Deterministic:
- `proposal_shape`
- `confidence_threshold`
- `original_text_presence`
- `non_empty_corrected_text`
- `non_empty_resulting_segment` — rejects edits that would leave the segment
blank, regardless of replacement-span length.
- `no_effect`
- `protected_terms`
@@ -42,24 +43,55 @@ Glossary, homophones, spoken_word, and grammar each resolve a fixed ordered chai
For each module section:
1. run deterministic validators;
2. run LLM-backed validators;
3. record decisions and warnings;
3. record decisions and compact operation outcomes;
4. carry only approved proposals forward.
Decision cardinality is enforced: each candidate proposal must receive exactly one decision per validator.
Decision cardinality is enforced inside Audita: each candidate proposal must
receive exactly one outcome per validator it reaches. The runner still rejects
duplicate, missing, or unknown identities in an internal validator result.
## LLM Validator Batching
LLM validators:
- build canonical validation request payloads;
- batch using the [configured prompt budget](../config.md#concurrency);
- call structured LLM client using response schema registry.
- batch using `validation.max_prompt_tokens` and the existing domain token estimator;
- select an embedded prompt ID and pass payload/description variables to the
role-bound structured client with the validator-decision schema.
Oversized single proposals are rejected with `validator_input_too_large`.
Malformed LLM validator responses are downgraded to warnings and rejected batch decisions.
Oversized single inputs are rejected with a consolidated
`validation_input_too_large` operation, without a provider call. Recoverable
model failures make only their batch candidates unavailable while other
batches continue. PromptKit validates each complete response against the
validator-decision JSON Schema and performs configured structural repair before
Audita receives decoded decisions. Each actual batch presents candidates
numbered from 1, independent of their internal proposal indices. Audita then
reconciles those local references: one usable entry keeps its
approval or editorial rejection; a missing or duplicate requested entry makes
only that candidate unavailable; unexpected indices are ignored. An empty list
makes all requested candidates unavailable. Decisions and unavailable indices
map back to internal proposal identities; results are ordered by request identity
and satisfy strict internal cardinality before the runner advances
approved candidates. An unavailable candidate never advances as approved.
Invalid confidence in a decoded test double also makes only its candidate
unavailable; production schema validation still rejects out-of-range confidence
before reconciliation. A partially unavailable batch records one failed
operation with only those requested indices, while a successful batch with
ignored extras records a decision-filtering recovery. Fatal errors stop the
chain without returning unchecked candidates as approved. A local result builder
retains already-collected decisions and operations on early returns; full
cardinality checks and final sorting remain explicit on completed paths.
Reconciliation makes no semantic retry or extra model call; guided recovery remains
[deferred](../roadmap/future.md#domain-aware-response-recovery).
Debug request metadata records `proposal_indices` in local-reference order
(reference 1 is the first entry). Raw decoded responses and ignored indices use
local references; reconciled decisions and unavailable indices use internal
proposal indices. Batch token estimates use the same locally numbered payload
as prompt encoding.
## Decision and Rejection Reporting
Runner collects decisions, rejected candidates, and warning records. Ledger
Runner collects decisions, rejected candidates, and operation outcomes. Ledger
assembly groups decisions by deterministic versus LLM-backed execution metadata.
The serialized names and fields belong to the
[report and ledger contract](../integrations/output-files.md); implementation mapping
@@ -85,3 +117,40 @@ is covered by [report assembly](diagnostics-reporting.md).
- `internal/framework/validators/*_test.go`
- `internal/framework/processreport/correction_ledger_test.go`
- `internal/cli/run_test.go`
## Supplemental Validation Context
Request construction prepares one segment-position lookup and protected vocabulary
for all candidates. Glossary selection remains candidate-specific, including the
proposed replacement text.
Each candidate includes the immediately preceding and following source segments
available within its fixed chunk, labeled `preceding` or `following` with speaker,
text, and categories, plus glossary entries whose names, aliases, or
recognized plurals appear in its original/resulting text or those neighbors.
Glossary matching reuses the protected vocabulary; entries retain glossary order.
Model-facing candidate and neighboring context omit internal segment IDs.
This supplies evidence such as a troupe definition for a proposed `Jesters`
correction or an adjacent speaker turn explaining deliberate repetition. A correction
already clear from its own segment does not require such evidence. No audio,
out-of-chunk context, or complete transcript is implied.
Context is serialized inside the validation item and included in the existing
payload token estimate. If one enriched item exceeds the configured limit, optional
neighbors are removed from the end first, then glossary entries from the end,
until it fits or only required candidate data remains. Required before/after text
is never truncated. Existing oversized-candidate handling still applies if that
required data does not fit. Context is supplemental, not a promise that every
relevant definition or neighboring utterance always fits. This uses the existing
payload estimator, not a new guarantee about the provider's full context window.
## Protected-Name Repetition
Spoken-word cleanup may reduce repeated protected-name occurrences when every
original protected identity still appears in the resulting segment and surviving
spellings satisfy existing capitalization checks. Full-segment preview supports
both a replacement of the repeated phrase and deletion of a redundant span. This
exception only relaxes the deterministic count guard; editorial and meaning-reversal
validation still decide whether the repetition was accidental and the edit preserves
meaning. Intentional emphasis and substantive references remain protected by those
reviews. Other modules retain their existing protection rules.

135
docs/migrations/v2.md Normal file
View File

@@ -0,0 +1,135 @@
# Migrating to Audita v2
Audita v2 adopts ConfigKit for configuration and PromptKit for LLM execution.
Version-1 configuration files, environment overlays, and retired flags are not
accepted. Copy a maintained [local](../../examples/minimal-config.yml) or
[service](../../examples/production-config.yml) example, then select real models
in the accompanying `examples/profiles/` files. A testing overlay is shown at
`examples/overlays/testing.yml`; its model choice carries no cost, speed, or
quality guarantee.
## Configuration and invocation
Use `version: 2` in a YAML main file. `--config` selects that file; otherwise
`process` and `config print-effective` look at
`/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`.
`config validate` requires `--config`. `AUDITA_CONFIG` no longer selects a file.
Settings resolve from defaults, main file, optional `--overlay <name>` in
`overlays/<name>.yml`, then explicitly supplied setting flags. General `AUDITA_*`
environment overlays no longer apply. Unknown fields and version-1 files fail;
there is no automatic converter.
Keep transcript as the positional path and `--glossary` and `--output` as
invocation paths. The former `--report-json` output has been replaced by the
default `errors.yml`, `recoveries.yml`, and `report.md` bundle; use
`--reporting=false` to disable it or `--report-dir` for stdout output. Keep
`pipeline.modules` / `--modules` and
`output.schema` / `--output-schema`. An explicit `pipeline.modules: []` or
`--modules ''` runs no correction modules; omission uses the default sequence.
Ordered repetitions still matter.
Move `llm.proposal.base_url` and `llm.validation.base_url` to a shared
`backends.<id>.endpoint` when both roles use one service. Move old role models to
PromptKit profile files and select them with `llm.proposal_profile` and
`llm.validation_profile`, or `--proposal-profile` and
`--validation-profile`. Omitted validation selection inherits the final proposal
selection; explicit empty or null is invalid. `profiles_dir` defaults to
`profiles` beside the main config. Complete profiles name a registered backend
and model. Per-role `--base-url`, `--validation-base-url`, `--model`, and
`--validation-model` are retired.
Move old `llm.*.api_key_env` to `backends.<id>.credential`, a scalar name of a
nonempty environment variable, such as `credential: AUDITA_SERVICE_KEY`.
Credentials on every surviving backend must resolve, including unselected
backends. Omit or set null for an unauthenticated local backend. Profiles cannot
perform another key lookup. `--llm-api-key` and `--validation-llm-api-key` are
retired. `AUDITA_LLM_API_KEY`, `OPENROUTER_API_KEY`, and
`AUDITA_VALIDATION_LLM_API_KEY` have no implicit meaning, though any of those
names can be used explicitly as a credential reference.
Replace `concurrency.total_llm`, `.proposal_llm`, and `.validation_llm` plus
`--total-llm-concurrency`, `--llm-concurrency`,
`--proposal-llm-concurrency`, and `--validation-llm-concurrency` with
`backends.<id>.concurrency` (default 1) and optional `queue_capacity` (default
1024; zero means no waiting admission). One backend capacity is shared by both
roles and all retry/repair attempts within this Audita process. It does not
coordinate other processes.
Move old role `timeout` values and `--llm-timeout-seconds` /
`--validation-llm-timeout-seconds` to profile `timeout_seconds`, a per-attempt
limit, and `llm.operation_timeout`, a Go duration such as `15m` covering the
whole logical call. Replace old `max_retries` and `--max-retries` /
`--validation-max-retries` with common `llm.transient_retries` (additional
eligible attempts per generation) and `llm.repair_attempts` (additional
generations for structurally invalid JSON). Explicit zero disables either
feature. Cancellation and the operation deadline cover capacity waiting,
backoff, generation, and repairs. PromptKit owns which failures can retry or
repair; Audita does not retry semantic rejections.
The following domain settings and explicit flags retain their meaning:
| File field | Explicit flag |
| --- | --- |
| `validation.max_prompt_tokens` | `--validation-max-prompt-tokens` |
| `chunking.target_sections` | `--target-sections` |
| `chunking.max_section_tokens`, `.min_section_tokens` | `--max-section-tokens`, `--min-section-tokens` |
| `normalization.max_segment_gap`, `.ellipsis_gap`, `.max_segment_duration`, `.max_segment_tokens` | `--normalize-max-segment-gap`, `--normalize-ellipsis-gap`, `--normalize-max-segment-duration`, `--normalize-max-segment-tokens` |
| `thresholds.glossary`, `.homophones`, `.spoken_word`, `.grammar` | `--glossary-confidence-threshold`, `--homophones-confidence-threshold`, `--spoken-word-confidence-threshold`, `--grammar-confidence-threshold` |
| `context.description` | `--transcript-description` |
| `reporting.enabled`, `.stdout_dir` | `--reporting[=true\|false]`, `--report-dir` |
| `diagnostics.debug`, `.work_dir` | `--debug[=true\|false]`, `--work-dir` |
Normalization gaps and durations are **numeric seconds**, so replace `4s`,
`3.5s`, and `60s` with `4.0`, `3.5`, and `60.0`. The operation timeout remains a
Go duration string. `chunking.target_sections` may be omitted or null for
automatic chunking; zero is invalid. Null is not a general synonym for omission:
nonoptional scalar settings reject it. Empty `context.description` clears
context. Legacy `diagnostics.retention` and `--work-dir-retention` are removed.
Debug is off by default and its directory is retained whenever enabled.
Remove old implicit variables `AUDITA_MODULES`, `AUDITA_MODEL`,
`AUDITA_VALIDATION_MODEL`, `AUDITA_BASE_URL`, `AUDITA_VALIDATION_BASE_URL`,
`AUDITA_LLM_TIMEOUT_SECONDS`, `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS`,
`AUDITA_MAX_RETRIES`, `AUDITA_VALIDATION_MAX_RETRIES`,
`AUDITA_TOTAL_LLM_CONCURRENCY`, `AUDITA_LLM_CONCURRENCY`,
`AUDITA_PROPOSAL_LLM_CONCURRENCY`, `AUDITA_VALIDATION_LLM_CONCURRENCY`,
`AUDITA_VALIDATION_MAX_PROMPT_TOKENS`, `AUDITA_TARGET_SECTIONS`,
`AUDITA_MAX_SECTION_TOKENS`, `AUDITA_MIN_SECTION_TOKENS`,
`AUDITA_NORMALIZE_MAX_SEGMENT_GAP`, `AUDITA_NORMALIZE_ELLIPSIS_GAP`,
`AUDITA_NORMALIZE_MAX_SEGMENT_DURATION`, `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS`,
`AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD`,
`AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD`,
`AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD`,
`AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD`, `AUDITA_WORK_DIR`, and
`AUDITA_WORK_DIR_RETENTION`. Put those values in the named fields above or pass
their surviving explicit flags.
## Inspecting and operating
Run `audita config validate --config <file>` before processing. Use
`audita config print-effective --config <file> --view settings|profiles|explain`
to inspect redacted ConfigKit YAML, selected PromptKit profile controls, or
field origins and derivations. Each view resolves both roles offline without
calling a provider. `--overlay testing` selects the same overlay for validation,
inspection, and processing. The old effective-config JSON format is replaced by
YAML; settings YAML alone does not show profile-owned model controls.
The [ordinary output contract](../integrations/output-files.md) now uses
version-1 `audita-errors` and `audita-recoveries` YAML documents plus a fixed
Markdown overview. Compact outcomes distinguish logical operations from
provider attempts. With debug enabled, utilization remains grouped by backend
capacity and usage fields retain known/complete flags. Debug runs write
`effective-config.redacted.yaml`, `selected-profiles.yaml`,
`config-origins.yaml`, `utilization-diagnostics.json`, and
`llm-execution.json`. Request and response artifacts remain redacted debugging
material. See [operations](../operations.md#diagnostics-directory) for placement
and paths, [configuration](../config.md) for all settings, and the
[CLI reference](../cli.md) for current flags.
Debug ledger consumers should recognize the deterministic validator key
`non_empty_resulting_segment` in place of `non_empty_corrected_text`. The
`empty_resulting_segment` reason is unchanged. Candidate-backed ledger entries
now include `proposed_corrected_text: ""` for an empty replacement, and applied
deletions also include `applied_corrected_text: ""`; omitted fields still mean
there was no candidate or no applied edit, respectively. See the
[debug artifact contract](../integrations/output-files.md#debug-only-files).

View File

@@ -1,110 +1,83 @@
# Audita Operations
This document owns runtime workflow, physical artifact placement, retention, and
recovery. Use the [CLI reference](cli.md) for invocation and stream/exit semantics,
[configuration](config.md) for settings, and [output-file contracts](integrations/output-files.md)
for logical formats.
This guide owns physical output placement and failure handling. The
[CLI reference](cli.md) owns flags and streams, [configuration](config.md) owns
settings, and the [output contract](integrations/output-files.md) owns schemas.
## Normal Workflow
1. Choose a configuration and validate it using the
[configuration commands](cli.md#configuration-commands).
2. Prepare input files conforming to the [input contract](integrations/transcript-glossary-files.md).
3. Run the [processing command](cli.md#process-examples), choosing explicit output
and report destinations when those files must survive run-directory cleanup.
4. Inspect the corrected transcript and process report. Review rejected or skipped
corrections when the report indicates them; successful execution does not mean
every proposed correction was applied.
1. Validate a version-2 configuration and its selected profiles with
`audita config validate --config config.yml`.
2. Prepare a transcript and glossary following the [input contract](integrations/transcript-glossary-files.md).
3. Run `audita process input.json --config config.yml --glossary glossary.yaml --output corrected.json`.
4. Check the exit status, stderr summary, corrected transcript, and reports.
A completed run can contain recoverable model errors and still apply
corrections that passed every required validator.
Processing creates a run directory, loads and normalizes the inputs, chunks the
transcript, executes the correction pipeline, writes results, and applies
retention. Early configuration failures happen before run-directory creation.
Failures after initialization attempt to preserve diagnostic evidence; individual
artifact writes can fail, so a run directory need not contain every listed file.
## Required Output Files
## Output Files
Reporting is enabled by default. For file transcript output, `errors.yml`,
`recoveries.yml`, and `report.md` go in the transcript output's parent directory.
For stdout transcript output, they go in `reporting.stdout_dir` (default `.`),
overridden by `--report-dir`. `--reporting=false` suppresses all three reports
and removes their path requirements. It does not suppress the stderr summary or
fatal errors. Stdout contains only transcript JSON; file output leaves it empty.
The caller chooses transcript and optional external report destinations using the
[CLI](cli.md#process-output-and-exit-behavior). These are separate from the per-run
diagnostic directory unless the caller deliberately places them inside it.
Choose existing, writable parent directories. Output/report files can be overwritten
by subsequent runs using the same destinations.
Existing report directories and parents for regular transcript files must be
writable. Audita prevents report writes from overwriting transcript, glossary,
loaded configuration, or loaded profile sources, and rejects conflicting output
destinations. The final transcript may intentionally replace its own input after
successful processing; other input/configuration/profile sources remain protected.
Preparation never truncates an existing transcript. Device and pipe destinations
(such as `/dev/null` and named pipes) receive ordinary writes at publication time;
they are not opened during preflight or replaced with regular files. A pipe write
can wait for its reader, and stream write errors are fatal. With reporting enabled,
reports still use the output path's parent; disable reporting when that placement
is unsuitable for a stream destination.
A later report or cleanup failure can occur after transcript output was written.
Inspect the process result and available report before treating the files as a
completed run. Retention of the diagnostic directory does not archive caller-selected
output files elsewhere.
With reporting enabled, all three reports are initialized as `in_progress` before
input processing. Output is encoded fully before publication. Each regular output
file is replaced through a temporary file on its destination filesystem:
transcript first, then `errors.yml`,
`recoveries.yml`, and `report.md`. This is per-file replacement, not a
multi-file transaction. New files honor the process umask; replacing an existing
file preserves its permission bits. A late report error may leave the transcript
published; the process exits nonzero and attempts failed-status reports. On fatal processing
or cancellation, no completed transcript is published, but partial reports are
finalized where possible. An uncatchable termination can leave `in_progress`.
## Diagnostics Directory
The [diagnostics configuration](config.md#thresholds-context-and-diagnostics)
selects the parent work directory. Runs create children named like
`run-<unix-nanoseconds>`. The runtime creates the work directory if needed.
Report paths may be relative when the supplied work directory is relative;
resolve them using the invocation's working directory.
`--debug` enables detailed evidence independently of reporting. One directory
named `run-<id>` is created under `diagnostics.work_dir` (default `/tmp/audita`)
and retained on success or failure. Without debug, no directory or payload
files are created. The parent is created when needed only for debug. Failure to
write optional debug evidence prints a redacted stderr diagnostic but does not
change correction decisions or the exit status.
| Artifact in the run directory | Purpose |
| --- | --- |
| `source-transcript.json` | Original transcript input |
| `source-transcript-parsed.json` | Parsed source representation |
| `normalized-transcript.json` | Transcript after normalization |
| `normalization-summary.json` | Normalization counters |
| `chunking-summary.json` | Section/token summary |
| `invocation.json` | Invocation metadata |
| `effective-config.json` | Redacted configuration snapshot |
| `report.json` | Run's [process report](integrations/output-files.md#process-report) |
| `correction-ledger.json` | [Correction dispositions](integrations/output-files.md#correction-ledger) |
| `utilization-diagnostics.json` | [Concurrency and timing observations](integrations/output-files.md#utilization-diagnostics) |
| `error.log` | Failure details when written |
LLM request, response, and error diagnostics are also written within the run
directory. Their nested naming and payloads are debugging details. Prefer paths
provided by the report and its decision/warning records over reconstructing those
paths. The report describes intended artifact locations, not proof that every
write succeeded or that the files still exist after retention.
## Retention Behavior
[Configuration](config.md#thresholds-context-and-diagnostics) owns accepted
retention values and the default. Current lifecycle behavior is:
| Run outcome / selection | Directory outcome |
| --- | --- |
| Failed run | Retained regardless of selection |
| Successful run, `always` | Retained |
| Successful run, `auto` | Retained if corrections were skipped or rejected; otherwise removed |
| Successful run, `never` | Currently retained, the same outcome as `always` |
The name `never` does not currently mean deletion. Warnings alone do not trigger
retention under `auto` unless they also produce skipped or rejected corrections.
An external report outside the run directory survives automatic cleanup, but its
artifact paths can then point to removed files.
## Artifact Handling
Run artifacts can contain full transcripts, glossary-related context, and LLM
prompts/responses. Configured-key redaction does not anonymize those contents.
Choose a work directory whose ownership and permissions fit the transcript data,
and retain or share artifacts intentionally. Credential-supply mechanisms are
owned by [configuration](config.md#secrets).
For a debugging run, select retained diagnostics using the
[retention flag](cli.md#process-flag-reference). Copy needed artifacts before
removing completed run directories manually. Avoid deleting a directory while
its run is active. Audita does not provide a separate archival/backup or remote
artifact storage facility.
Debug evidence includes invocation and effective settings, configuration
origins, selected profiles, redacted source and normalized snapshots,
normalization and initial chunk summaries, all captured proposed candidates
and validator decisions in `correction-ledger.json`, execution/utilization
records, and model interaction files. Individual artifact writes can fail; a
directory's existence does not prove every file was written. These files can
contain transcript text and prompt/response payloads. Configured backend
credentials are redacted, but the transcript itself is not anonymized. Protect
and retain the directory according to the sensitivity of the input.
## Failure Inspection and Recovery
1. Read stderr and the exit classification defined by the [CLI](cli.md#process-output-and-exit-behavior).
2. Locate the external report, or the run directory identified by the error when
available. If a late error omits the path, inspect the selected work directory.
3. Read the available report and error log. Use
[report status and error fields](integrations/output-files.md#process-report)
to identify the failed phase, then inspect referenced artifacts.
4. Correct the concrete input, configuration, endpoint, or filesystem issue and
rerun the command with diagnostics retained if further investigation is needed.
Exit `0` means processing and every enabled required output completed. Exit
`1` means cancellation, fatal processing, or required-output failure. Exit `2`
identifies usage or configuration/destination selection errors. Recoverable
model-operation failures appear in `errors.yml` as one record per affected
work unit; they do not automatically make the run fail. `recoveries.yml`
records successful operations that needed retry, repair, or proposal filtering.
`report.md` shows every initial chunk and configured module instance, including
interrupted and unattempted work. Ordinary reports omit transcript excerpts and
full decision ledgers; enable debug when those are needed.
Each invocation starts a new run. There is no implemented resume/checkpoint API;
recovery is a rerun, not continuation from an artifact. Failures during early
configuration or directory creation may leave no report or diagnostic files.
Read stderr and report status first. If a required output failed after a
transcript was published, do not treat the file alone as proof of a completed
run. Fix the input, service, or destination problem and rerun. Audita has no
resume/checkpoint API.

View File

@@ -66,9 +66,6 @@ Examples:
## Git Commit Policy
This policy applies to all coding agents and agent harnesses working in this
repository, not only Codex.
By default, create a commit after completing each coherent unit of work, using a
concise, plain-English message describing what changed. This policy is standing
authorization to create those local commits; a separate user request is not

View File

@@ -2,7 +2,7 @@
**Base Policy Version:** 1.0
**Last Updated**: 2026-09-23
**Last Updated**: 2026-09-24
## Purpose and System Shape
@@ -22,7 +22,7 @@ and [integration documentation](../integrations/subprocess.md) own external cont
normalization, chunking, output, diagnostics, and reporting contracts. Domain
behavior must not require CLI or provider-transport knowledge.
- **Framework:** owns orchestration contracts and shared runtime mechanics for
modules, validators, proposals, LLM adapters, scheduling, and report assembly.
modules, validators, proposals, LLM adapters, and report assembly.
It composes domain operations through explicit inputs and outputs.
- **Modules:** own narrowly scoped correction behavior and module-specific prompt
choices. Keep module packages separate rather than collapsing their distinct
@@ -45,9 +45,12 @@ hidden global state.
- Validate corrections before final mutation, including local proposal checks,
decision cardinality, and apply-time safety. Provider-side schema enforcement
does not replace these checks.
- Keep concurrency bounded, configurable, context-aware, and observable through
the existing scheduler and utilization diagnostics. Preserve permit release on
success, failure, and cancellation.
- Keep LLM concurrency bounded and context-aware through PromptKit's engine-local
per-backend admission. Proposal and validation calls selecting the same backend
share that capacity, including retries and repairs. Preserve deterministic
collection order. Recoverable model-operation failures stay local to their
proposal response or validation batch; fatal invariant/application failure
and parent cancellation stop peers and join them.
- Keep module and validator identities, reason codes, and execution-class
distinctions stable where exposed as contracts.
@@ -60,12 +63,14 @@ describe how these requirements are implemented.
Keep transcript mutation and output handling deterministic around external
calls. Reports and correction ledgers are derived from processing results.
Preserve the separation between transient processing state and filesystem
adapters; physical artifact placement and retention belong to
adapters; physical report placement and optional debug retention belong to
[operations](../operations.md).
Public CLI, configuration, output-schema, diagnostics, report, prompt, module,
and validator contracts should remain stable unless a change is deliberate and
tested. Prefer additive, compatible report changes. Current implementation
tested. The resilience work intentionally replaced the earlier JSON process
report with versioned errors/recoveries YAML and a fixed Markdown overview;
future changes should treat those current contracts deliberately. Current implementation
limits are documented in the [internal overview](../internal/overview.md#current-scope);
they are not additional prohibitions on future development.
@@ -84,9 +89,10 @@ responsibilities and should remain separate. Implementation details belong in
## Dependencies
Keep Audita dependency-light and prefer the Go standard library for CLI parsing,
HTTP, JSON, filesystems, synchronization, and tests. Third-party dependencies
should be narrow, justified, and preferably established for their purpose.
YAML parsing is the existing direct dependency exception.
filesystems, synchronization, and tests. ConfigKit owns configuration loading,
presence, provenance, and secret resolution; PromptKit owns prompt/profile
loading, OpenAI-compatible transport, retries, repairs, and backend admission.
Audita owns domain meaning, role selection, and deterministic processing.
Do not introduce broad CLI, dependency-injection, workflow, logging, or plugin
frameworks without a concrete implemented need and focused tests.

View File

@@ -7,7 +7,7 @@ source/tag push; binary delivery runs asynchronously.
## Prerequisites
Use Bash, Git, and Go 1.25 or newer (CI uses Go 1.25), with dependencies available
Use Bash, Git, and Go 1.25.5 or newer (CI uses Go 1.25.5 or newer), with dependencies available
locally or downloadable. Tests themselves are offline. Work from a
checkout of `main` with permission to push that branch and tags to `origin`.
For this repository, `origin` is `https://gitea.maximumdirect.net/eric/audita.git`.

View File

@@ -1,5 +1,6 @@
# Release Notes
- [Audita v2.0.0](v2.0.0.md)
- [Audita v1.1.0](v1.1.0.md)
See the [release documentation requirements](../policy/release.md#release-documentation).

38
docs/releases/v2.0.0.md Normal file
View File

@@ -0,0 +1,38 @@
# Audita v2.0.0
Audita v2 adopts ConfigKit for configuration and PromptKit for LLM execution,
adds resilient processing and focused reports, and improves correction prompts
and validation. This is an intentional breaking release; existing installations
must migrate their configuration and invocation settings.
- Configuration uses `version: 2` YAML, named backends, PromptKit model profiles,
and optional named overlays. Offline configuration validation and inspection
expose effective settings, selected profiles, and provenance.
- Proposal and validation calls share per-backend capacity. PromptKit handles
transport retries and structural response repair within bounded operation
timeouts.
- Transcript chunks remain fixed through the LLM stages. Recoverable model
failures preserve progress and allow other work to continue. If every LLM
operation fails recoverably, processing still produces the deterministically
normalized transcript without LLM corrections. Cancellation, application
failures, and required output failures remain fatal.
- Ordinary output includes `errors.yml`, `recoveries.yml`, and `report.md`, plus
a brief completion summary. Detailed correction and LLM diagnostics are opt-in
with `--debug`; `--reporting=false` suppresses all three ordinary reports.
- Empty replacement text supports validated span deletion while preserving a
nonblank segment. Validator responses retain usable decisions despite missing,
duplicate, or extraneous entries; unavailable decisions never approve an edit.
- Prompts use short request-local references, consistent shared instructions,
and relevant neighboring/glossary context for validation. Spoken-word cleanup
can reduce accidental protected-name repetition subject to validation.
Follow the [v2 migration guide](../migrations/v2.md) before upgrading. Version-1
configuration, implicit environment-setting overlays, retired model/credential/
concurrency flags, and the former JSON report interface have no compatibility
shim. Start from the maintained examples linked in that guide, configure profiles
and credential references, and validate the new configuration. Update report and
debug consumers for the [current output contract](../integrations/output-files.md).
Linux amd64 and arm64 remain the production targets; macOS arm64 is best effort
for development. Binary and checksum delivery runs asynchronously after source/tag
publication, as described in the [release procedure](../release.md).

171
docs/roadmap/configkit.md Normal file
View File

@@ -0,0 +1,171 @@
# ConfigKit Adoption Roadmap
**Last updated:** 2026-09-24
**Status:** Production implementation complete. The selected configuration and
overlay model is implemented; [implementation evidence](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md)
records verification. No release has been published.
## Purpose and Direction
Adopt ConfigKit to harmonize Audita's configuration policy with the other
applications in the suite. Consistent configuration conventions are the primary
benefit; reducing application-owned configuration infrastructure is a secondary
benefit and a useful measure of integration quality.
This is an intentional breaking migration. There is no goal of backward
compatibility with existing installations, configuration files, CLI flags,
environment variables, or configuration-inspection output. Provide a migration
guide instead of legacy loaders, aliases, dual operating modes, or an ongoing
compatibility layer. Retain existing syntax only when it independently suits the
new model, not because compatibility requires it.
The initial review and prototype used ConfigKit v0.3.1, then verified v0.4.0
selected overlays. Those findings remain historical; the prototype was removed
after production integration. The application depends on a versioned module,
without a sibling checkout or local module replacement.
The shared boundary/prototype unit is complete. Its [selected contract, inventory,
and evidence](library-boundary-findings.md) define the detailed target alongside
this roadmap. Production configuration migration is complete.
## Target End State
- ConfigKit owns strict YAML loading, input presence, default application, explicit
override mechanics, recorded resolution, secret references, and safe inspection.
- Audita owns its configuration types, field meanings, defaults, schema-version
policy, file selection, CLI parsing, domain validation, and runtime initialization.
- Adopt ConfigKit's precedence and validation lifecycle rather than reproduce
Audita's current per-layer validation and environment-overlay behavior.
- General configuration environment overrides are retired. Credentials use typed
environment-name secret references; supplied references must resolve according
to ConfigKit's contract. File selection follows the application-owned
discovery rules in the selected boundary.
- Standardize duration representations, explicit empty/null/zero handling, and
inheritance on the shared conventions. Express genuine Audita domain rules
through the supported resolution and validation APIs.
- Retained domain-setting flags and proposal/validation profile flags become
explicit ConfigKit overrides according to the migration inventory. Remove old
provider/key/retry/concurrency flags and aliases. Credentials use secret references.
- Use ConfigKit's redacted effective output and provide source/explanation access
through the selected views below. Legacy JSON inspection is replaced.
Diagnostic configuration snapshots use the same new model.
- Delete superseded loading and override machinery. Do not hide the old model
behind adapters that defeat the adoption's consistency benefit.
Keep Audita's domain capabilities, such as ordered and repeated correction modules,
expressible in the new model. Their existing configuration spelling is not binding.
Transcript/glossary formats, processing algorithms, and output schemas are outside
this migration unless later planning identifies a concrete dependency or an
explicitly coordinated change.
## Selected Overlays for Testing
Enable ConfigKit's optional selected-overlay feature in the migration. A testing
configuration that selects cheaper, faster LLMs is the initial use case. ConfigKit
**overlays** modify application settings; PromptKit **execution profiles** still
own model and generation settings. Keep `profiles_dir`, the PromptKit `profiles/`
directory, and proposal/validation profile names unchanged by this terminology
change.
The planned interface is `--overlay <name>` on `process`, `config validate`, and
`config print-effective`. Treat selection as invocation state passed to
`Options.Overlay`, not as another persistent setting or environment overlay.
Follow ConfigKit v0.4.0's selection contract: omission, an empty name, or `default`
selects no overlay; a named selection loads `overlays/<name>.yml` beneath the selected
configuration root. A missing selected overlay is an error. Use the library's
single-overlay semantics without adding stacking, inheritance, or legacy-directory
fallback.
Precedence is application defaults, main configuration, selected overlay, then
explicit setting flags. Perform Audita's role inheritance and semantic validation
on the resolved configuration. ConfigKit's base parsing and schema-version checks
still precede overlay application; overlays cannot replace the schema version or
contain ConfigKit directives. General environment-variable overrides remain retired.
Imports and secret-directory bootstrap remain disabled.
Provide a maintained `overlays/testing.yml` example that selects testing execution
profiles through `llm.proposal_profile` and `llm.validation_profile`. Model names and
generation controls belong in the corresponding PromptKit profile files. Set both
role selections in this example so a base configuration's explicit validation
profile cannot accidentally retain the production model. Normal omission-based
validation-profile inheritance continues to apply to the final effective settings.
The example must make its backend/credential requirements clear and use no real
secrets. Model choice is operator-controlled; “testing” does not promise a price,
performance level, or equivalent correction quality.
Overlays may modify other ordinary application settings supported by the schema;
they are not restricted to profile selection. The resulting configuration must
still satisfy shared backend capacity, credential, and domain rules. Validation,
effective-config inspection, selected-profile inspection, and processing must all
use the same selected overlay. Preserve overlay provenance in configuration
explanations and diagnostics while retaining secret redaction.
The [recorded overlay results](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
prove the representative load, inspection, and HTTP path. Production acceptance
requires the same selection to reach both actual LLM roles, with explicit flags
winning over overlays and surviving secret references resolved/redacted correctly.
The isolated prototype is evidence, not a second supported runtime.
## Inspection and Invocation Contract
`process`, `config validate`, and `config print-effective` share file/overlay
selection and retained setting overrides. Invocation I/O paths remain process-only.
Only explicitly supplied flags override settings. Validation requires an explicit
config file; processing and inspection use the selected discovery order. Help is
available before configuration loading. All three commands validate selected
profiles without requiring a provider call during setup.
`config print-effective --view settings|profiles|explain` exposes three YAML views:
- `settings` (default): ConfigKit's redacted effective configuration.
- `profiles`: an Audita-owned summary of each selected role's profile, backend,
model, temperature, token limit, top-p, timeout, service tier, and reasoning
effort where present. Arbitrary extra provider parameters and credentials are
omitted; this is a selected-controls view, not every execution input.
- `explain`: field paths, sources, and derivation records from ConfigKit, without
resolved secret values.
A ConfigKit snapshot alone does not describe profile-owned controls. Profile
inspection must enforce the registered-backend and credential boundary. Diagnostic
snapshots use these safe views and preserve overlay/explicit-override provenance.
The migration guide explains the intentional replacement of legacy inspection JSON.
## Integration and Completion Criteria
ConfigKit and PromptKit form one intentional breaking refactor. Intermediate
commits need not build or run. The target contains no temporary old-runtime adapter,
compatibility shim, processing gate, dual loader, or automatic legacy conversion.
The [combined feature roadmap](library-adoption.md) owns execution/reporting intent;
[implementation.md](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md) alone owns stage order and implementation tasks.
The final application loads all retained domain settings, assembles both selected
profiles, and processes through PromptKit. Controlled HTTP evidence must connect
configuration selection to actual requests, domain outcomes, safe diagnostics,
and shared finite backend capacity. Maintained examples and one migration guide
cover the complete file/flag/environment transition. Obsolete application-owned
loading, transport, rendering, and scheduler machinery is removed.
The combined target is Audita v2.0.0. Release compliance and publication remain
last and separate from implementing the refactor. Retention and transcript timestamp
changes are not included; the selected retry/repair contract resolves the retry
follow-up in [runtime behavior](runtime-behavior.md).
## Scope Boundaries
- Pin ConfigKit v0.4.0 in production and align Go/toolchain requirements. ConfigKit requires
Go 1.25; the combined PromptKit prototype requires Go 1.25.5. Adopting these libraries
does not automatically remove Audita's existing YAML dependency used for glossary parsing.
- Implement the selected configuration and final PromptKit runtime directly,
without temporary runtime adapters, compatibility shims, or process gates.
- Enable selected overlays as specified above. Keep imports and secret-directory
bootstrap disabled unless a separate concrete requirement justifies them.
- Validation and inspection resolve surviving credential references and fail on
missing or empty values; no credential-free bypass is planned.
- Inspection and diagnostics use the safe selected-profile view and ConfigKit
provenance described above. Changed formats are covered by the migration guide.
The eventual completion test is that Audita follows the suite's ConfigKit
conventions while retaining its intended domain capabilities, with a clear
migration guide and no ongoing obligation to support the previous configuration
model.

View File

@@ -0,0 +1,162 @@
# Partial Success for Validator Decision Sets
**Last updated:** 2026-09-25
**Status:** Implemented. This document owns the target behavior and user intent.
[implementation.md](implementation.md) owns concrete decisions, ordered work,
and verification.
## Purpose and User Intent
Preserve usable validator decisions when other entries in the same model response
have missing, duplicate, or unexpected correction indices. One defective entry
should not prevent unrelated candidates from completing their required validation.
This extends Audita's existing partial-success approach without another model call.
Implement this simpler improvement before reconsidering guided domain recovery.
Missing validation must never authorize applying a correction, but neither should
it invalidate a different candidate's unambiguous approval or editorial rejection.
The existing [resilience design](resilience.md) remains the foundation.
## Historical Baseline
The reviewed application baseline is `d2a6fcf`:
- PromptKit validates the complete model response against Audita's embedded JSON
Schema and performs its configured structural repairs. Only structurally valid,
decoded responses reach domain decision mapping.
- `mapLLMResponseToDecisions` in
[`llm_validators.go`](../../internal/framework/validators/llm_validators.go)
requires exactly one decision for every requested correction index. An empty
decision list, missing index, duplicate index, or unexpected index returns an
error instead of retaining usable decisions.
- `LLMBackedValidator.Validate` consequently marks the entire batch unavailable
with `validation_decisions_invalid`. Other batches continue, but usable decisions
within that batch are lost.
- The runner's cardinality checks enforce an internal contract: each candidate
receives one outcome from each validator it reaches. These checks apply to
Audita-owned results, not directly to raw model entries.
- Deterministic validators already filter individual proposals. Unpreviewable
candidates and oversized validation inputs are isolated; recoverable provider
failures remain batch-local; application skips individual conflicts.
- Ordinary reports distinguish failed operations from successful operations with
recoveries. Several failed operations in one stage/chunk share one work-unit
error record. The [output contract](../integrations/output-files.md) owns the
current counts and serialized representation.
The defect addressed here is the all-or-nothing domain mapping boundary, not
validation itself or the runner's internal consistency checks.
## Target Decision Semantics
Reconcile each structurally valid response against the indices actually requested
in that particular validator batch. Correction indices, rather than segment IDs
or positions in the response array, identify candidates. Several corrections may
legitimately target the same segment.
| Response condition | Target behavior |
| --- | --- |
| Exactly one valid decision for a requested index | Preserve its approval or rejection and associated decision evidence. |
| Requested index omitted | Produce an explicit validation-unavailable outcome for that candidate. |
| Requested index appears more than once | Treat that candidate's validation as unavailable; preserve unrelated decisions. |
| Index was not requested in this batch | Ignore the extraneous entry and record the handled defect. |
| Empty decision list for a nonempty request | Mark all requested candidates unavailable; do not fail the run. |
| Decisions appear in a different order | Match by identity and retain deterministic internal results. |
Duplicate handling does not choose the first or last answer, nor infer a verdict
by voting. Even identical duplicates follow the agreed duplicate-index handling;
accepting them specially is not needed for this feature. Duplicate extraneous
indices remain extraneous entries, not reasons to invalidate requested candidates.
An unavailable candidate remains unapplied and does not proceed as approved to
later validators. It is distinct from a valid `approved: false` decision, which
is ordinary editorial processing. Valid approvals continue through the remaining
required validators and apply-time checks; this feature grants no bypass.
The mapper returns one internal outcome for every requested candidate and no
outcome for an extraneous index. Preserve strict runner cardinality checks so
application bugs are still detected. Do not convert genuine internal invariant
failures, cancellation, or fatal setup failures into tolerated model defects.
## Outcomes and Reporting
Keep existing work-unit aggregation and operation accounting:
- If any requested candidate lacks an unambiguous usable decision, record a
validation-operation error identifying the affected requested indices. Other
candidates' usable decisions from that operation remain effective. An operation
error does not mean every candidate in its batch failed.
- If all requested candidates have usable decisions and extra entries were
ignored, record a successful operation with a recovery for that handled defect.
This remains a recovery even if all legitimate decisions are editorial rejections.
- If missing/duplicate requested decisions and ignored extras coexist, retain the
operation error. Follow the existing rule that failed operations do not also
produce recovery records; retain extra-entry details in debug evidence.
- A valid complete response with no handled defect adds neither error nor recovery.
Valid editorial rejection alone never adds either category.
- Preserve actual structural repair and service-retry counts. On successful
operations, consolidate applicable recovery reasons in one recovery record.
Do not count reconciliation as another LLM operation or structural repair.
Error attribution must distinguish affected candidates from the rest of the batch.
Debug evidence should preserve supplied decisions, ignored entries, and the reason
an expected candidate became unavailable. Do not fabricate candidate ledger entries
for unknown indices or mislabel missing decisions as deliberate editorial rejection.
Ordinary reports must retain safe metadata without raw transcript or response text.
Use a decision-specific recovery reason rather than treating ignored validator
entries as defective correction proposals. Concrete reason names and minimal data
model changes belong in the implementation plan. Keep YAML, Markdown, and CLI
counts consistent, with the existing reporting-disable and independent debug options.
## Ownership and Target Architecture
Audita's validator layer reconciles structurally valid model responses into
complete internal candidate outcomes. The runner preserves strict internal
cardinality and advances only approved candidates. Reporting consumes operation
facts without treating partial validation failure as loss of every decision.
Debug evidence distinguishes unavailable validation from editorial rejection.
The schema/transport boundary remains with PromptKit; reconciliation needs no
additional model call. Concrete helper structures, reason names, mapping changes,
and verification steps belong in the [implementation plan](implementation.md).
## Scope Boundaries
PromptKit's whole-response structural validation and repair remain unchanged.
A malformed entry, including an extraneous entry that violates the schema, may
still invalidate the response and exhaust structural repairs before reconciliation
can run. Ignoring schema-valid extras does not promise salvage from malformed JSON,
missing required fields, wrong types, or out-of-range values.
General per-entry structural salvage is a separate design question, not part of
this fix. Likewise, guided regeneration remains deferred for reconsideration after
this change; no new retries, feedback prompts, budgets, upstream dependency changes,
or operator controls are required here.
Preserve module scope, literal replacement behavior, fixed chunks, stage barriers,
span-deletion safeguards, backend capacity limits, and fatal cancellation behavior.
No additional restrictions on valid corrections or guarantees about model judgment
are introduced. This work does not itself authorize tagging or publishing v2.0.0.
## Acceptance Criteria
- A schema-valid response can retain usable decisions despite missing, duplicate,
or unexpected indices elsewhere in the same batch.
- Only the affected requested candidates become unavailable; extras never create
candidates or attach decisions to a different batch's candidates.
- Valid negative decisions remain editorial outcomes, and every applied correction
completes all required validation.
- Internal validator results satisfy existing strict cardinality checks.
- Missing/duplicate requested decisions yield accurately attributed errors without
losing valid work. Extra-only defects yield a consolidated recovery; mixed
defects follow existing failed-operation reporting semantics.
- Final transcript, debug evidence, YAML reports, Markdown counts, and CLI summary
agree about applied work and failures/recoveries.
- Reconciliation makes no additional provider calls. Existing structural failures,
provider failures, and cancellation retain their documented handling.
## Open Questions
No unresolved product-policy questions block implementation. Guided recovery and
structural salvage remain separate future decisions.

80
docs/roadmap/future.md Normal file
View File

@@ -0,0 +1,80 @@
# Deferred Feature Work
**Last updated:** 2026-09-25
**Status:** Deferred or exploratory work; not implemented or scheduled. These
entries preserve scope outside the [resilience roadmap](resilience.md) and
[span-deletion consistency roadmap](span-deletion.md). Each entry states its
intent; inclusion does not commit us to implementing every capability.
## Domain-Aware Response Recovery
The [partial-success decision reconciliation](decision-set-reconciliation.md)
preserves usable decisions without another model call. Guided recovery remains
deferred pending evidence from failures that remain. General per-entry
salvage from structurally invalid responses is also a separate design question;
the accepted reconciliation feature retains whole-response structural validation.
Consider bounded feedback and regeneration for model responses that satisfy the
structural schema but leave requested decisions unavailable after reconciliation.
The goal would be to recover more usable work before excluding affected
candidates, when doing so materially improves outcomes.
This capability is explicitly deferred. Current behavior retains PromptKit
structural repair and per-candidate reconciliation; unavailable requested
decisions produce errors while other candidates can succeed.
Before implementation, define the Audita/PromptKit ownership boundary, which
checks are eligible for feedback, how feedback identifies defects without exposing
secrets, and how regeneration fits bounded budgets and operation deadlines. Reuse
shared library capabilities where appropriate; do not introduce an uncoordinated
outer retry loop that multiplies existing retry/repair budgets.
Ordinary editorial rejection is a valid decision, not a reason to regenerate until
an edit is approved. Recovery must preserve required validation, candidate identity,
partial-success safety, and truthful error/recovery reporting. Validate with
controlled responses that distinguish recoverable protocol defects from valid
negative decisions. Exact APIs, budgets, and sequencing remain undecided.
## Independent Whole-Pipeline Traversal by Chunk
Explore allowing each fixed chunk to advance through its ordered stages without
waiting for other chunks to finish the current stage. This could reduce waiting
behind a slow chunk and improve utilization while preserving stage order within
each chunk and deterministic final recombination.
The resilience feature deliberately retains stage barriers: finish a stage across
chunks before advancing to the next stage. Fixed boundaries alone do not authorize
independent traversal.
A future design must establish that stages do not depend on other chunks' evolving
state, preserve stable identities and partial-success rules, and continue to share
PromptKit's per-backend capacity across all roles and stages. Reporting must remain
deterministic despite different completion orders, and fatal cancellation must
still stop and join active work. Assess whether the scheduling change materially
improves real workloads before introducing additional orchestration machinery.
No new scheduler, global limiter, or execution mode is selected by this entry.
## Whole-Segment Removal
Potential future work, subject first to evaluating whether it is worth implementing
at all. Empty replacement text and deletion of a span within a surviving segment
are covered by the accepted [span-deletion consistency roadmap](span-deletion.md).
That feature does not remove segment records or permit blank resulting segments.
Before designing segment removal, review representative production examples to
establish whether whole-segment deletion is a meaningful unmet need. An empty
replacement alone does not establish that intent: it can express ordinary filler
or punctuation deletion within a larger segment. Assess how often genuinely
expendable whole segments occur after deterministic normalization, the transcript
quality benefit of removing them, and the risk of losing substantive speech.
Compare those benefits with the implementation and ongoing maintenance costs.
Retaining the segment and declining the edit is a valid outcome if the benefit is
small or uncertain. No numerical adoption threshold is selected.
Only if that evaluation supports proceeding should a separate feature design
choose a representation, authorization and validation rules, and reporting
semantics. It must address meaning preservation and the effects on segment
identity, timestamps, fixed chunks, subsequent stages, recombination, and output
consumers. Do not infer authorization to remove a segment from an empty string
or introduce segment-removal machinery as part of span-deletion consistency.

View File

@@ -0,0 +1,287 @@
# Implementation Plan: Partial Validator Decision Sets
**Last updated:** 2026-09-25
**Status:** Implemented; all stages completed.
**Source baseline:** `60e2fe7`.
**Audience:** gpt-5.6-terra coding agent.
## Scope and Execution
Implement [partial success for validator decision sets](decision-set-reconciliation.md).
That roadmap owns product intent and target semantics; this document selects the
implementation and verification work. Follow Stage 1 through Stage 3 in order and
record completion evidence beneath each stage. Each stage is a focused agent task.
Read [development guidance](../development.md) and all policies under
[docs/policy](../policy/): [agents](../policy/agents.md),
[architecture](../policy/architecture.md), [documentation](../policy/documentation.md),
[testing](../policy/testing.md), and [release](../policy/release.md). Use graph-first
code discovery, preserve unrelated work, and commit completed coherent units with
plain English messages. Do not introduce approval rounds between stages.
Aim directly at the final implementation. No temporary modes, compatibility shims,
new configuration, extra model calls, or upstream dependency work is required.
Keep PromptKit structural validation/repair and the runner's internal cardinality
checks. Implement the agreed outcomes, not extra restrictions suggested by existing
tests or incidental internal structure. Internal helper signatures and layout can
be adjusted idiomatically while preserving the decisions below.
This replaces the completed span-deletion plan, available in Git at
`60e2fe7:docs/roadmap/implementation.md`. It does not reopen that feature. The final
stage includes correcting historical links that otherwise point to this new plan.
No release preparation, tag, push, CI wait, or paid-provider evaluation is included.
## Selected Implementation Decisions
### Reconciliation boundary
Keep reconciliation in `internal/framework/validators`, after successful structured
completion and before converting the batch into runner-facing results. Replace
`mapLLMResponseToDecisions` with a small pure reconciliation helper; use an ordinary
private result struct containing:
- `Decisions []Decision`: one outcome per requested item, in request order;
- `UnavailableIndices []int`: requested indices without a usable verdict;
- `IgnoredIndices []int`: unexpected indices encountered in response order, including
repeated unexpected entries, for debug attribution.
No generic reconciliation framework or new shared abstraction is needed. Requested
identities come from `batch.Items`, not transcript segment IDs, the whole section,
or any other batch. Expected input identities remain an Audita internal contract.
First collect response entries by requested index and count occurrences, separating
unexpected indices. Then walk requested items once to construct outcomes. This
avoids accepting an early duplicate before seeing a conflicting later entry.
Use these candidate-level reason codes in `validators/models.go`:
| Condition | Internal outcome/reason |
| --- | --- |
| Missing requested entry | `DispositionUnavailable`, `missing_validation_decision` |
| More than one entry for a requested index | `DispositionUnavailable`, `duplicate_validation_decision` |
| Single requested entry failing the existing confidence range check | `DispositionUnavailable`, `invalid_validation_decision` |
| Single usable entry | Existing approved or `llm_rejected` decision with its model reason. |
Unavailable outcomes have `Approved: false` and fixed safe explanatory messages.
Duplicate classification takes precedence over inspecting individual duplicate
values; identical duplicates follow the same rule. Unexpected entries are ignored
before inspecting confidence. Retain the existing confidence check as a local
check for decoded callers/test doubles, scoped to that candidate; do not broaden
this into structural salvage, confidence thresholds, or new semantic validation.
The production JSON Schema still rejects out-of-range confidence before mapping.
An empty response list makes every requested item unavailable. Response ordering
must not change verdicts or which candidate is affected. Preserve the existing
handling of valid negative decisions, including reason text. No candidate may
advance merely because another candidate was approved.
### Operation accounting and ordinary reports
Each actual validation batch remains one `outcomes.Operation`. Use existing
`ReasonValidationDecisionsInvalid` (`validation_decisions_invalid`) for a batch
with unavailable requested candidates. Set `Succeeded: false`, set `ProposalIndices`
to only those unavailable indices, retain execution counts, and leave recovery
reasons empty. Still return usable decisions from that batch alongside explicit
unavailable outcomes, with no Go error for the reconciled model defect.
If no requested candidates are unavailable, set `Succeeded: true`. Add
`outcomes.RecoveryValidationDecisionsFiltered` with serialized value
`validation_decisions_filtered` when extras were ignored. Merge it with actual
`output_repaired` and `service_recovered` reasons in the same recovery operation.
Valid negative decisions do not prevent operation success. A clean response creates
no new recovery reason.
When unavailable candidates and extras coexist, record the failed operation only;
keep ignored-entry evidence in debug. Do not create both an error and a recovery
for that operation. Successful survivors must continue even though the operation
itself records a failure. Retain actual execution counts on all paths.
Reuse existing ordinary YAML structures and version 1. The only new public report
value is the decision-specific recovery reason; affected error indices already fit
`Failure.ProposalIndices`. Do not add ignored entries to `Operation.Filtered` or
`filtered_proposals`, which describe correction proposals, and do not increment
`proposal_responses_with_filtering`. The recovery reason and existing operation
identity suffice for ordinary reporting; detailed extra indices belong in debug.
Update `processreport.safeMessage` for `validation_decisions_invalid` to explain
that some required decisions were unavailable without claiming all batch decisions
were discarded. Generalize the recovery message to include response filtering,
rather than only proposal filtering. Exact prose is not a test contract. Preserve
work-unit error grouping, operation counts, and existing YAML/Markdown/CLI totals.
### Debug evidence and runner records
Existing interaction diagnostics already retain the full decoded response and final
content. Add the reconciliation result to the response diagnostic payload for a
successfully decoded validation response. Perform reconciliation before writing
that diagnostic; retain existing call-error diagnostics and diagnostic-write failure
handling. All reconciled decisions, including unavailable outcomes, receive the
returned response artifact path. This records unexpected indices and duplicates
without inventing candidate ledger entries or an additional debug file.
Carry `validators.DecisionDisposition` through `runner.ValidatorDecisionRecord`
and `runner.ValidatorRejectedChange` in an additive `disposition` field. Populate
it from each actual `Decision`, including partial decision evidence returned on
fatal paths. Keep the existing collection names and application flow; no parallel
unavailable-candidate collection is necessary.
In `processreport`, add `disposition` to `LedgerValidatorDecisionRecord` and map
unavailable candidates to a top-level ledger disposition `unavailable`, instead
of `rejected`. Ordinary editorial/deterministic rejections remain `rejected`.
Use the explicit disposition, not a reason-code allowlist, so existing timeout,
provider-failure, and oversized-input outcomes retain the same distinction too.
Update affected test fixtures to supply dispositions where they represent real
validator outputs. Do not invent a fatal check or compatibility subsystem for a
zero-value disposition in handcrafted fixtures.
Retain candidate identity, literal proposed replacements (including explicit empty
strings), and omission of applied replacements for unapplied candidates. Existing
fatal/interrupted/module-failure handling and debug-off behavior remain. Supplied
model decisions remain visible in interaction evidence even when reconciliation
cannot accept them as authoritative verdicts.
## Stage 1 — Reconcile Model Decisions and Preserve Valid Work
**Completed:** 2026-09-25. Reconciliation preserves usable decisions, attributes
unavailable requested indices and ignored extras, and keeps strict runner
cardinality. Validator and runner boundary checks passed with
`go test ./internal/framework/validators ./internal/framework/runner`.
**Work**
1. Read the validator models, `llm_validators.go`, existing partial-batch tests,
`outcomes/outcomes.go`, and `runner.validateSectionCandidates`.
2. Implement the reconciliation result/helper and candidate reason codes selected
above. Replace the whole-batch mapping-error branch with outcome construction
and correct batch accounting. Add the decision-specific recovery constant.
3. Move successful-response diagnostic assembly after reconciliation and include
its result. Preserve raw/decoded response evidence and diagnostic paths; keep
recoverable call errors, fatal errors, and cancellation on their existing paths.
4. Keep `EnforceDecisionCardinality` and `EnforcePartialDecisionCardinality` strict.
Ensure final validator decisions are complete and unique internally. The runner
should continue passing only approved candidates onward without treating an
operation's failure metadata as a reason to discard valid decisions.
5. Replace obsolete batch-rejection expectations in `llm_validators_test.go` and
relevant runner tests. Preserve cardinality tests that exercise broken internal
validators. Update the repaired-but-domain-invalid tests so actual unavailable
cases remain errors with counts retained and no recovery record.
**Focused verification**
Use a small table at the validator boundary for mixed missing/extra indices,
conflicting and identical requested duplicates, extra-only responses, empty lists,
reordered responses, and a single invalid-confidence decoded test-double entry.
Include noncontiguous expected indices and valid negative decisions. Assert semantic
outcomes and affected indices, not helper layout or exact messages. Include one
controlled case combining successful structural/service recovery facts with ignored
extras and one failed reconciliation retaining counts without recovery reasons.
At the runner boundary, use the real LLM-backed validator with a controlled client
and a later validator: prove a survivor reaches later required validation, a later
rejection prevents application, and unavailable candidates never reach that later
validator as approved. Use successful proposal operation facts where accounting is
asserted; do not check an empty operation list. Reuse existing fixtures/tests rather
than creating a second framework or duplicating every table case.
Run `go test ./internal/framework/validators ./internal/framework/runner`.
**End state:** Usable decisions survive model indexing defects, strict internal
cardinality remains satisfied, and operation facts explain partial failure or
successful extra filtering. No extra model generation is introduced.
## Stage 2 — Carry Dispositions Through Debug and Report Partial Outcomes
**Completed:** 2026-09-25. Runner and ledger records carry explicit
dispositions; ordinary reports distinguish unavailable decision errors from
extra-index filtering recoveries. Focused report, ledger, runner, and YAML
checks passed with `go test ./internal/framework/runner
./internal/framework/processreport ./internal/core/reporting`.
**Work**
1. Add and populate disposition fields through the runner and ledger as selected
above. Map unavailable candidates distinctly and preserve empty replacement
serialization. Check both normal and fatal partial-evidence paths.
2. Update `processreport` messages and review `Build`, `mapFailure`, and `mapRecovery`.
Reuse their existing aggregation; change only assumptions that would misreport
the new operation facts. Core YAML structures and Markdown table layout remain.
3. Extend existing report/ledger tests for an operation with surviving work plus
unavailable indices, extra-only recovery, and a mixed defect operation. Verify
error indices exclude survivors, extra filtering does not count as proposal
filtering, and failed operations do not also generate recovery records.
4. Verify debug ledger entries distinguish genuine rejection from unavailability,
and interaction diagnostics preserve ignored entries and reconciliation reasons.
Unknown response indices must not create fake ledger candidates. Retain existing
redaction and optional-debug-failure tests rather than reproducing them all.
5. Update `docs/integrations/output-files.md` for the new recovery reason and debug
disposition, and `docs/internal/diagnostics-reporting.md` for the mapping changes.
Explain that validation failure indices identify unavailable candidates, not
necessarily every candidate submitted to that batch.
**Validation**
Run `go test ./internal/framework/runner ./internal/framework/processreport ./internal/core/reporting`.
Use semantic YAML/summary assertions; no whole-report prose golden, exact message
snapshot, new schema version, or new rendering mode is needed.
**End state:** Reports and debug evidence accurately describe partial validation
success and handled extras through existing reporting mechanisms.
## Stage 3 — Verify the Assembled Workflow and Finalize Documentation
**Completed:** 2026-09-25. Controlled subprocess runs covered a mixed decision
response and an extra-only response through the CLI, transcript, YAML, Markdown,
debug ledger, and interaction evidence. Fixture execution facts showed one
provider attempt and zero structural repairs per logical call. `go test
./cmd/audita` and `go test ./...` passed. Canonical validator guidance and
roadmap status were updated; the two span-deletion plan links now target the
historical plan at `60e2fe7`.
**Work**
1. Extend the existing local-provider subprocess fixtures in
`cmd/audita/main_integration_test.go`. Add a representative mixed-decision run
with an applied survivor, a legitimate rejection, a missing requested decision,
a duplicate requested decision, and an extra index. Return responses by the
requested correction identities; do not freeze incidental validator call order.
Ensure the survivor passes all remaining required validation. Assert successful
CLI completion, correct transcript, one work-unit error with only unavailable
indices, no reconciliation recovery for the failed operation, and debug
evidence distinguishing rejection/unavailability without unknown candidates.
2. Add or adapt a compact extra-only case proving usable work survives and one
`validation_decisions_filtered` recovery appears. Verify YAML, Markdown, and CLI
agree on totals using semantic assertions. Keep proposal-filtering counts separate.
Existing reporting-disable/debug selection tests remain the owners of those options.
3. In these fixtures give PromptKit its normal configured structural repair allowance,
but supply schema-valid responses. Verify reconciliation adds no generations or
structural repairs using fixture requests/execution facts, without encoding today's
total validator count as a permanent contract. Keep structural-malformation tests
and recoverable batch-failure tests; do not relax the schema to make fixtures pass.
4. Update `docs/internal/validators.md` and any affected canonical descriptions found
by searching for whole-batch rejection claims. Explain tolerant external response
reconciliation versus strict internal cardinality. The structural-validation
boundary remains explicit, and guided recovery remains deferred in `future.md`.
5. Mark the feature roadmap implemented only after the assembled checks pass, and
record actual completion evidence in this plan. Repair the two span-deletion
roadmap links to the old plan with the historical URL
`https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md`.
Preserve historical release notes and unrelated roadmaps.
**Validation**
Run `go test ./cmd/audita` while developing the fixture, then `go test ./...` once
for the completed implementation. Run `git diff --check` and verify affected
Markdown links. Do not add live-provider checks, a platform matrix, repeated full
suite runs, or release gates solely for this feature. No code is implemented by
this planning pass; these commands are instructions for the implementation sprint.
**End state:** The accepted partial-success behavior is exercised through the real
CLI/library boundary, canonical documentation matches it, and coherent changes
are committed for subsequent release consideration.
## Open Questions
None required before implementation. The implementation decisions above preserve
the agreed product scope. General structural salvage and guided regeneration remain
separate future decisions, not unresolved prerequisites for this work.

View File

@@ -0,0 +1,129 @@
# Coordinated ConfigKit and PromptKit Adoption
**Last updated:** 2026-09-24
**Status:** Production implementation complete. ConfigKit v0.4.0 and PromptKit
v0.11.0 are adopted; [historical implementation evidence](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md) records the
completed checks. The current `implementation.md` covers span deletion instead.
Release compliance and publication remain separate work.
## Goal and Ownership
Adopt ConfigKit and PromptKit to align Audita with the application suite's shared
configuration and LLM execution conventions. Consistency is the primary benefit;
removing overlapping application infrastructure is a secondary benefit. Preserve
Audita's transcript-correction capabilities and domain protections.
| Owner | Responsibilities |
| --- | --- |
| ConfigKit | Strict application configuration loading, presence/default semantics, explicit overrides, recorded resolution, secret references, and safe inspection |
| PromptKit | Prompt/profile formats and rendering, provider calls, structural output validation and repair, transient retries, engine-local backend admission, and execution observations |
| Audita | Setting meanings, file/CLI selection, role/profile association, normalization/chunking, ordered modules, domain validation, correction application, reports, diagnostic redaction, and artifact lifecycle |
The [configuration roadmap](configkit.md) owns configuration and inspection policy.
The [boundary findings](library-boundary-findings.md) own the selected defaults,
full migration inventory, failure mapping, and prototype evidence. The selected
production versions are ConfigKit v0.4.0 and PromptKit v0.11.0, with Go 1.25.5 as
the minimum. Use published modules, never sibling checkouts or local replacements.
The existing YAML dependency remains while glossary parsing needs it.
## Migration Posture
This is an intentional breaking transition targeting Audita v2.0.0. No backward
compatibility is required for existing installations, config files, CLI flags,
environment overlays, or inspection output. Provide one migration guide, without
legacy loaders, aliases, dual modes, or automatic conversion tooling.
Both libraries belong to one refactor. Intermediate commits need not build or run;
there is no supported intermediate product state. No temporary adapters, compatibility
shims, or processing gates are needed. The complete application must work and pass
its checks before the refactor is finished. Release compliance and publication
come last, with no intermediate public release.
## Configuration and Execution Target
ConfigKit selected overlays enable a testing configuration that chooses cheaper or
faster models. The example selects both proposal and validation profiles. ConfigKit
application overlays and PromptKit execution profiles remain separate concepts.
Complete operator-maintained profiles own model/generation controls; embedded
application prompts and schemas are not operator-selected files. No catalog wiring,
profile-inheritance feature, ConfigKit imports, or secret-directory bootstrap is
part of this migration.
Capacity controls serve two needs: respecting a local service's hard simultaneous-
query limit and bounding concurrent paid API activity. One finite limit per backend
is shared across both roles and every initial, retry, and repair attempt. Limits
are engine-local, not coordinated across processes. No additional global or
per-role limiter or fine-grained scheduling policy is required. Profiles must use
registered backends and ConfigKit-resolved credentials; endpoint-only bypass and
competing profile credential lookup are excluded.
Structural repair is enabled by default for both roles because additional attempts
to obtain valid output are desired. Transient retries and structural repairs have
separate budgets; explicit zero disables the respective behavior. Each generation,
including a repair, receives its own transient-retry budget. The Audita logical-call
deadline covers preparation, waiting, backoff, all generations, and repairs; profile
HTTP timeouts apply per attempt. Exact defaults and bounds are in the
[selected contract](library-boundary-findings.md#configuration-and-profile-contract).
No Audita outer retry loop restores the old broader retry policy.
Audita retains all four correction modules, five LLM-validator prompts,
deterministic validators, confidence/applicability checks, glossary protection,
validation batching/cardinality, deterministic proposal and ledger ordering,
transcript encoders, and artifact retention. PromptKit-valid JSON still passes
Audita's domain checks. Repair exhaustion, provider failure, and domain rejection
follow the [selected failure table](library-boundary-findings.md#execution-and-failure-mapping).
Retention and transcript timestamp changes remain separate work; this migration
resolves the [retry-count follow-up](runtime-behavior.md).
## Reporting and Diagnostic Target
Keep stable Audita-owned serialized records; do not expose raw library structs.
Use process report schema `v2` with the existing domain outcomes and a new
`llm_summary` separating logical calls, dispatched provider attempts, transient
retries, structural repairs, and usage. Transcript schema versions remain unchanged.
The implementation plan specifies the concrete new fields and artifact wiring.
Usage distinguishes known zero, unknown, and incomplete totals. Preserve observations
from failed calls even when no final result exists. Count terminal totals once;
attempt observations provide diagnosis without being added to those totals again.
This is operational evidence, not a complete billing ledger.
Replace global/role scheduler utilization with per-backend configured capacity and
observed queue/generation/retry/logical-call timing. Do not fabricate measurements
that the event API does not provide. Keep domain timing and deterministic correction
ledger ordering. Record prompt definition/rendered hashes and selected profile/model
without promising the old hash values. Callbacks are short, concurrency-safe, and
free of blocking I/O or waits on their own execution.
Effective configuration and selected-profile snapshots remain distinct safe views.
Secret values must not escape through inspection, logs, reports, runtime errors,
or retained diagnostic files. Observation records exclude payloads, URLs, raw
provider messages, prompts, and credentials. Existing artifact lifecycle and
partial-failure handling remain Audita responsibilities.
## Completion Criteria and Release Boundary
- Both libraries own their selected responsibilities in the production application.
Superseded loaders, transport, renderers, schedulers, aliases, and the isolated
prototype are gone once its useful evidence is represented in production tests.
- Offline tests prove configuration-to-execution routing for both roles, selected
overlays and explicit overrides, shared backend capacity, retry/repair outcomes,
cancellation/deadlines, domain decisions, redaction, and truthful partial reports.
Tests exercise real internal collaborators and controlled HTTP boundaries.
- Maintained examples and canonical docs describe the implemented final model.
One migration guide covers files, flags, environment inputs, profiles, credentials,
overlays, inspection, and changed runtime/report/diagnostic semantics.
- The assembled application passes its required checks; Linux amd64 and arm64 build.
macOS arm64 remains best effort. No live-provider or CI-success gate is added.
PromptKit v0.11.0's reviewed retry, observation, and capacity APIs remove the earlier
upstream feature prerequisites; no upstream edit is required for adoption. The
prototype proved representative behavior, not release compliance or model quality.
The selected PromptKit release uses LGPL-3.0-or-later. A later release unit must
review the actual production dependency graph and distribution requirements,
including notices and source/rebuild materials, before binary publication. This
implementation plan does not claim that review is complete. Release publication
uses the existing [procedure](../release.md) and completes on successful source/tag
push, without waiting for CI or artifact delivery.

View File

@@ -0,0 +1,392 @@
# Shared Library Boundary: Decisions and Evidence
**Status:** Boundary and offline prototype completed, 2026-09-23. Selected prototype contract and proposed
production migration target; not the current Audita configuration contract.
The [bounded roadmap](library-boundary.md) owns scope and completion criteria.
**Subsequent scope decision:** The [ConfigKit overlay plan](configkit.md#selected-overlays-for-testing)
adds optional selected overlays, with v0.4.0 as the migration baseline. It supersedes
the original no-overlay decision and adds an input layer between the main file and
explicit setting overrides. The v0.3.1 prototype results below remain historical
evidence; they do not establish overlay integration.
## Selected Boundary
The former prototype is retained in Git history at `590604b`; it was a separate
Go module using published ConfigKit v0.4.0, PromptKit v0.11.0, and
Audita v1.1.0. The Audita dependency lets this consumer reuse actual internal domain
helpers through its Audita module namespace without a local replacement. It does
not make those internal packages a new public API. Remove this dependency along
with the prototype after equivalent production integration tests landed.
The initial prototype results below were recorded with ConfigKit v0.3.1. On
2026-09-24, upgrading only its ConfigKit pin and sums to v0.4.0 required no source
changes. With Go 1.26.8 on linux/amd64, the nested offline `go test ./...` passed
and `go run .` still reported one accepted correction, four attempts, zero
transient retries, and two structural repairs. This verifies the dependency
upgrade, not selected-overlay behavior.
ConfigKit owns loading, presence, explicit overrides, secret resolution, and
redacted effective settings. PromptKit owns profile parsing, rendering, provider
execution, transient retries, structural repair, and backend admission. Audita
owns file selection, setting meanings, role selection, domain acceptance, and
mapping execution facts into reports. Production CLI and runtime remain unchanged.
## Configuration and Profile Contract
Selected for the prototype and recommended for production:
- Main YAML requires integer `version: 2`; absent, null, unknown fields, and older
versions fail. No legacy parser, aliases, environment overlays, or automatic
conversion. Defaults precede the selected file; explicitly supplied CLI settings
become ConfigKit overrides. Validation runs on the resolved result, so an override
may repair an invalid lower-precedence setting when decoding succeeds.
- Production file selection: explicit `--config`, then
`/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`. Select one file;
do not merge them. Missing system files are skipped; explicit missing or empty
paths and inaccessible selected files fail. No selected file fails with a request
to configure backend/profile selection: no hidden provider or model default.
Remove `AUDITA_CONFIG`. The prototype loads its explicit fixture only; discovery
and command parsing belong to the ConfigKit implementation stage.
- `profiles_dir` defaults to `profiles`, resolved relative to the selected main
configuration file, with absolute paths accepted. PromptKit reads that directory.
Prompts and schemas are application-embedded, not operator-selected prompt files.
The prototype uses complete application-owned profile files. Catalog profiles and
profile inheritance are not needed for this migration; do not add catalog wiring.
- `backends.<id>.endpoint` is required. Use one application backend identity per
service capacity pool. Do not split proposal and validation into different backend
IDs when they use the same service. Limits are engine-local, not shared across
multiple Audita processes or other applications.
- `backends.<id>.concurrency` defaults to **1**, must be positive, and is shared by
both roles and all initial/retry/repair attempts. Operators can select **4** for
the described local backend or a suitable finite paid API limit. There is no
additional global or per-role limiter. `queue_capacity` defaults to **1024**,
following PromptKit; non-negative, with zero meaning no waiting admission.
This is an operational escape hatch rather than a second scheduling hierarchy.
- `backends.<id>.credential` is an optional ConfigKit secret reference: a scalar
environment variable name. Omission or null means no credential. A supplied
reference must resolve to a nonempty value, including during validation and
inspection. There is no direct key field or CLI key flag. ConfigKit resolves it
once and Audita supplies `RunRequest.APIKey`. Profiles must not introduce another
`api_key_env` lookup. Backend credentials also support unauthenticated local use.
- Profiles own model, temperature, output token limit, top-p, per-generation timeout,
service tier, reasoning effort, and extra provider parameters using PromptKit's
format. They select the application `backend`; their effective endpoint must
equal that backend's endpoint. Endpoint-only execution is rejected because it
bypasses the requested finite capacity boundary. A named registered backend is
mandatory even if both roles use the same profile.
- `llm.proposal_profile` is required. Omitted `llm.validation_profile` inherits the
final proposal profile as a whole, with ConfigKit recording the derivation.
Explicit empty or null values fail; no individual model/endpoint/key inheritance.
- `llm.transient_retries` defaults to **2** additional attempts per generation;
explicit zero disables retries; negative is invalid. PromptKit owns which failures
are eligible. Do not add an Audita retry loop.
- `llm.repair_attempts` defaults to **2** additional generations for both roles;
explicit zero disables repair when deliberately requested. PromptKit supports
0–3. Positive defaults are exercised in the prototype. Audita owns this common
setting and supplies a complete request output contract with the role's embedded
schema, avoiding separate user-facing repair defaults in each prompt.
- `llm.operation_timeout` defaults to **15m**, must be positive, and uses a Go
duration string. It starts before each logical proposal/assessment call and covers
preparation, capacity waiting, transient backoff, generation, and repairs. It is
separate from whole-process cancellation. Profile `timeout_seconds` defaults to
PromptKit's **600** seconds and is applied afresh to each built-in HTTP attempt.
It does not bound the combined retry/repair sequence. A stricter caller/transport timeout can
also end execution. The two defaults are selected starting points, not throughput
measurements. With the default T=2 and R=2, each logical call dispatches at most
**9** attempts before deadline/cancellation; proposal plus assessment may use 18.
- Omission retains static defaults or invokes the recorded role/backend derivations.
Null is not a general synonym for omission: nonoptional scalars reject it. Optional
credentials and the future optional section target accept null. Explicit empty
module lists will mean an empty correction pipeline, not “restore defaults.”
Repeated modules retain their existing ordered meaning.
- The original prototype disabled ConfigKit imports, selected configuration layers
(then called configuration profiles), and secret-directory bootstrap. Selected
overlays are now verified under the linked overlay decision; the other two
features remain disabled. PromptKit execution profiles retain their meaning.
Configuration validation/inspection must assemble and inspect both selected profiles
without provider calls. ConfigKit's effective YAML describes only its loaded state;
a separate selected-profile view shows effective model controls/backend selection.
Do not imply either snapshot alone represents the full execution configuration.
The prototype exercises the two APIs but does not implement production subcommands.
Do not serialize raw profile/library objects as a permanent Audita report schema.
## Migration Inventory
This inventory covers the current configuration and process flags documented in
[configuration](../config.md) and [CLI](../cli.md), including environment-only fields.
“Retain” below means retain domain meaning in the new schema, not accept old files.
All legacy environment overlays listed afterward are removed.
| Current file field / setting | Current CLI flag(s) | Target and rationale |
| --- | --- | --- |
| File selection | `--config` | Retain explicit invocation flag; discovery described above. |
| Transcript input | positional path | Invocation only; retain one transcript path. |
| Glossary input | `--glossary` | Invocation only; retain required glossary path. |
| Corrected output | `--output` | Invocation only; retain stdout fallback. |
| Report output | `--report-json` | Invocation only; retain separate report file. |
| `pipeline.modules` | `--modules` | Retain sequence/default `[glossary, homophones, glossary, spoken_word, grammar]`; explicit empty clears rather than inherits. |
| `output.schema` | `--output-schema` | Retain encoder selection/default `bare-segments`. |
| `llm.proposal.base_url`, `llm.validation.base_url` | `--base-url`, `--validation-base-url` | Replace with backend endpoint; remove per-role endpoint flags. |
| Both `llm.*.model` | `--model`, `--validation-model` | Replace with profile model; new `--proposal-profile` / `--validation-profile` select whole profiles. |
| Both `llm.*.api_key_env`, direct runtime keys | `--llm-api-key`, `--validation-llm-api-key` | Replace with backend ConfigKit secret reference; remove direct key flags. |
| Both `llm.*.timeout` | `--llm-timeout-seconds`, `--validation-llm-timeout-seconds` | Replace with profile `timeout_seconds`, plus `llm.operation_timeout`; no duplicate timeout flags. |
| Both `llm.*.max_retries` | `--max-retries`, `--validation-max-retries` | Replace with common `llm.transient_retries` and distinct `llm.repair_attempts`; remove misleading old flags. |
| `concurrency.total_llm` | `--total-llm-concurrency`, `--llm-concurrency` | Replace with per-backend concurrency; remove global/alias flags. |
| `concurrency.proposal_llm`, `concurrency.validation_llm` | `--proposal-llm-concurrency`, `--validation-llm-concurrency` | Remove role budgets; shared backend handles service capacity. |
| Validation prompt budget (no old YAML field) | `--validation-max-prompt-tokens` | Add `validation.max_prompt_tokens`, default 2048, positive; retain explicit flag override. Audita owns batching. |
| `chunking.target_sections` | `--target-sections` | Retain optional positive target; omission/null means automatic, zero invalid. |
| `chunking.max_section_tokens` | `--max-section-tokens` | Retain 8192, positive. |
| `chunking.min_section_tokens` | `--min-section-tokens` | Retain 2048, positive, no greater than maximum. |
| `normalization.max_segment_gap` | `--normalize-max-segment-gap` | Retain 4.0 numeric seconds, non-negative; drop alternate YAML duration syntax for these domain numeric fields. |
| `normalization.ellipsis_gap` | `--normalize-ellipsis-gap` | Retain 3.5 numeric seconds, non-negative. |
| `normalization.max_segment_duration` | `--normalize-max-segment-duration` | Retain 60.0 numeric seconds, positive. |
| `normalization.max_segment_tokens` | `--normalize-max-segment-tokens` | Retain 2048, positive. |
| `thresholds.glossary` | `--glossary-confidence-threshold` | Retain 0.8 in [0,1]. |
| `thresholds.homophones` | `--homophones-confidence-threshold` | Retain 0.8 in [0,1]. |
| `thresholds.spoken_word` | `--spoken-word-confidence-threshold` | Retain 0.8 in [0,1]. |
| `thresholds.grammar` | `--grammar-confidence-threshold` | Retain 0.8 in [0,1]. |
| `context.description` | `--transcript-description` | Retain trimming and 500-byte limit; empty clears context. |
| `diagnostics.work_dir` | `--work-dir` | Retain `/tmp/audita`, nonempty; artifact lifecycle remains Audita-owned. |
| `diagnostics.retention` | `--work-dir-retention` | Retain `auto` default and selection; existing `never` follow-up is separate. |
| Help/version/config commands | `--help`, `-h` | Retain command purpose; help should parse before loading configuration. Effective-config output intentionally becomes ConfigKit YAML, not legacy JSON. |
Retained setting flags become explicit ConfigKit overrides only when supplied;
CLI parser defaults must not overwrite file values. Normalization flag values stay
numeric seconds. Domain values convert into existing domain structs after validation;
do not spread ConfigKit or PromptKit types into correction algorithms.
Complete environment removal map (names remain permissible if explicitly named as
secret references; none have implicit configuration meaning):
| Old environment names | Replacement |
| --- | --- |
| `AUDITA_CONFIG` | File selection above. |
| `AUDITA_MODULES` | `pipeline.modules` / explicit flag. |
| `AUDITA_LLM_API_KEY`, `OPENROUTER_API_KEY`, `AUDITA_VALIDATION_LLM_API_KEY` | Explicit backend credential references; no implicit fallback. |
| `AUDITA_MODEL`, `AUDITA_VALIDATION_MODEL` | Profile files and role selection. |
| `AUDITA_BASE_URL`, `AUDITA_VALIDATION_BASE_URL` | Backend endpoints. |
| `AUDITA_LLM_TIMEOUT_SECONDS`, `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS` | Profile timeout and operation deadline. |
| `AUDITA_MAX_RETRIES`, `AUDITA_VALIDATION_MAX_RETRIES` | Common transient/repair settings. |
| `AUDITA_TOTAL_LLM_CONCURRENCY`, `AUDITA_LLM_CONCURRENCY`, `AUDITA_PROPOSAL_LLM_CONCURRENCY`, `AUDITA_VALIDATION_LLM_CONCURRENCY` | Backend concurrency; remove alias/role budgets. |
| `AUDITA_VALIDATION_MAX_PROMPT_TOKENS` | `validation.max_prompt_tokens` / explicit flag. |
| `AUDITA_TARGET_SECTIONS`, `AUDITA_MAX_SECTION_TOKENS`, `AUDITA_MIN_SECTION_TOKENS` | Corresponding chunking fields / flags. |
| `AUDITA_NORMALIZE_MAX_SEGMENT_GAP`, `AUDITA_NORMALIZE_ELLIPSIS_GAP`, `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION`, `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS` | Corresponding normalization fields / flags. |
| `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD`, `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD`, `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD`, `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD` | Corresponding threshold fields / flags. |
| `AUDITA_WORK_DIR`, `AUDITA_WORK_DIR_RETENTION` | Diagnostics fields / flags. |
## Execution and Failure Mapping
The executable uses grammar proposal plus grammar review, current response schemas,
current hardening text, transcript-section/description helpers, proposal preview, and
actual decision-cardinality enforcement from Audita v1.1.0. PromptKit renders the
adapted templates; the custom HTTP transport supplies OpenAI-compatible responses.
The prototype pre-expands Audita's `hardening` template function into the embedded
PromptKit definitions; it passes existing named string variables unchanged. No
provider, paid call, or injected high-level LLM client participates.
Generated JSON is structurally validated/repaired by PromptKit before Audita decodes
it. Audita then checks proposal applicability, confidence, and decision identity and
cardinality. The demonstration previews corrections without mutation. Production
retains every module's full validator chain, glossary protection, batching, and
apply-time checks; the representative prototype does not claim to replace them.
| Outcome | Section/batch effect | Overall run |
| --- | --- | --- |
| Structural repair succeeds | Continue domain validation. | Normal execution. |
| Proposal repair exhausted | No proposals from this section; malformed-output warning. | Continue other work; successful completion may include warnings. |
| Assessment repair exhausted | Reject this batch; malformed-output warning. | Continue other work. |
| Schema-valid invalid/missing/duplicate decision indices | Reject batch and warn; no semantic retry/repair. | Continue other work. |
| Inapplicable/low-confidence proposal or negative assessment | Reject affected proposal; preserve domain reasons. | Continue other work. |
| Provider failure, retry exhaustion, transport or malformed HTTP response | Stop affected operation; retain available observations. | Runtime failure; cancel remaining work, retain existing Audita diagnostic handling. |
| Deadline, caller cancellation, capacity admission failure | Stop affected operation; no outer retry or fabricated output. | Runtime failure with distinct classified cause. |
| Invalid settings/profile/schema setup | Fail before processing where detectable. | Configuration error for user input, runtime error for broken embedded assets. |
Zero transient retries produces one ordinary provider failure, not
`ErrRetriesExhausted`; the latter signals an exhausted **positive** budget.
Provider failures retain a typed cause. Do not expose raw provider messages through
user diagnostics: continue Audita's separate secret-redaction boundary. A failed
logical call may return no result while its observations still record completed
attempts or an earlier successful generation. Already completed domain results can
remain in diagnostics; failure must not be presented as a fully corrected output.
## Observation and Report Handoff
The prototype maps library events into Audita-owned records under a short mutex.
Records contain logical run/role/backend/profile identity, generation and attempt
indices, dispatch state, queue/generation/retry/wall timing, classified outcomes,
HTTP status, exhaustion, repair/retry counts, and each usage field's value,
knownness, and completeness. It omits URLs, credentials, payloads, raw provider
errors, and prompt text. Identifiers themselves are metadata, not a secret channel.
Use terminal records once for logical totals; retain attempt records for diagnosis
without adding them to terminal totals. A known zero is distinct from unknown;
partial totals remain incomplete after unmetered failed attempts. These observations
are not a billing ledger. Callback code must not wait on its own run's progress.
Production report changes must distinguish logical calls from dispatched provider
attempts. Replace scheduler utilization fields based on global/proposal/validation
pools with per-backend admission/active timing. Preserve deterministic ledger and
validator ordering. Map prompt definition/rendered hashes and selected profile/model
into request metadata, but do not promise equality with legacy prompt hashes or
serialize PromptKit's raw event struct as the stable report contract. Update the
output integration reference, report mapper tests, and downstream migration guide
when the production representation is implemented.
## Verification and Remaining Handoff
The focused matrix is implemented in the nested module:
| Roadmap scenario | Evidence |
| --- | --- |
| Successful assembled path | `go run .`; `TestWorkflowBoundaries/repair_both_roles`; actual Audita domain helpers and both selected profiles. |
| Presence, overrides, inheritance | `TestConfigurationContract`: omitted role/backend values, explicit zero, null/empty, version, unknown keys, and explicit overrides. |
| Credentials | Missing/unset and empty references fail; optional null accepted; effective/profile inspection redacted; HTTP authorization verified; no secret/prompt text in observations. `TestProfilesRespectSharedBackendBoundary` rejects competing lookup and capacity bypass. |
| Both roles repaired | `repair_both_roles`: four attempts, two repairs, one accepted correction. |
| Repair exhausted | Proposal and assessment exhaustion subtests: bounded counts and selected warning/rejection outcomes. |
| Semantic boundary | Missing span, missing/duplicate decision indices, negative assessment; no semantic repair loop. |
| Retry/zero/exhaustion | 429 recovery, zero-budget 503, positive-budget exhaustion after completed proposal; typed error assertions. |
| Combined budgets | 503 → invalid JSON → repair 503 → valid proposal → assessment: five attempts, two transient retries, one repair. |
| Shared capacity | `TestSharedCapacityAndRecovery`: caps four and two, both roles, a retry and repair, queue rejection, measured peak equal to cap, later successful call. |
| Waiting/cancellation/deadline | `TestQueuedCancellationAndDeadline`; `TestDeadlineStopsRetryAfter`: no queued dispatch after cancellation/deadline and a 60-second provider retry delay interrupted by the operation deadline. |
| Failed-run observations | `TestPartialUsageOnFailedRun`: valid HTTP output with known-zero usage followed by repair failure; nil result, retained events, incomplete known aggregate, correct counts. |
| Representative assets | `TestRealPromptRenderingParity` compares both rendered message pairs to Audita v1.1.0; `TestSchemaFixturesMatchAudita` compares schema bytes. |
Commands executed from `examples/library-adoption` on Linux amd64 with Go 1.26.8:
```sh
GOWORK=off go mod download all
GOWORK=off GOPROXY=off GOSUMDB=off go mod verify
GOWORK=off GOPROXY=off GOSUMDB=off go run .
GOWORK=off GOPROXY=off GOSUMDB=off go test -race ./...
GOWORK=off GOPROXY=off GOSUMDB=off go vet ./...
```
All passed. The demonstration reports accepted=1, rejected=0, attempts=4,
transient_retries=0, structural_repairs=2. Module verification reports all modules
verified. Published tag availability was checked with `git ls-remote --tags` for
ConfigKit v0.3.1 and PromptKit v0.11.0; successful module acquisition verifies this
is not merely a local-checkout experiment. The module has no replacements.
The minimum Go 1.25.5 toolchain was also exercised with offline tests and the same
successful demonstration. The exact installed-toolchain invocation was:
```sh
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off /home/eric/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.5.linux-amd64/bin/go test ./...
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off /home/eric/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.5.linux-amd64/bin/go run .
```
An initial attempt to select that cached toolchain through `GOTOOLCHAIN=go1.25.5`
with checksum verification disabled was rejected by Go's toolchain verifier. Using
the installed binary with `GOTOOLCHAIN=local` proved offline execution without
weakening dependency checks. The portable README commands assume an installed
sufficiently new Go executable on PATH and disable toolchain switching.
From the Audita root, `go test ./...` and
`go build -o ./bin/audita ./cmd/audita` passed. These check existing production code;
they do not cover the nested module. Affected relative documentation links,
inventory coverage, and `git diff --check` were checked separately.
### Findings and Limits
No adoption-blocking upstream API gap was found. Two integration details deserve
explicit attention: PromptKit's profile GoDoc calls `timeout_seconds` a generation
deadline, but its built-in HTTP path applies it separately to each attempt; the
consumer therefore needs the selected total-operation deadline. Also,
`APIKeyRequired` is an in-memory profile facility, not a profile-YAML field. The
file-profile design uses the presence of ConfigKit's optional credential reference
to require local secret resolution; it does not invent a new PromptKit YAML key.
The chosen defaults are proposed operational policy, not measured model-quality or
cost recommendations. Tests use controlled transport responses and synthetic data;
they do not establish real provider compatibility, model quality, network behavior,
throughput, cross-process coordination, complete billing, or release compliance.
Only the representative LLM-facing subset is implemented in the prototype; the
inventory specifies the remaining domain fields for the ConfigKit migration.
Production file discovery, CLI parsing, full module chains, all nine prompts,
filesystem diagnostics/redaction, and serialized report migration remain their
assigned production tasks. Those are deliberate scope boundaries, not missing
claims of completion for the prototype.
The new profile/credential/capacity and deadline choices are explicit above; no
unresolved upstream requirement prevents starting the ConfigKit implementation
plan. Production report field names and final runtime wiring are selected in the
[implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md), following the outcome and observation
contract here. The coordinated refactor targets both libraries directly; intermediate commits do not require a working legacy runtime.
### Production Implementation Reference
The [implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md) owns ordered production work,
component changes, tests, and acceptance criteria. The [combined roadmap](library-adoption.md)
and [configuration roadmap](configkit.md) own the final target. This evidence does
not require an interim runtime bridge or separately runnable configuration migration.
Retained domain scope includes transcript/glossary schemas, normalization/chunking,
output encoding, proposals and apply-time checks, deterministic validators, ordered
modules, LLM-validator batching/cardinality, ledger ordering, and artifact lifecycle.
The production integration replaces loading, provider execution, prompt rendering,
and scheduler composition while preserving those responsibilities. Retention and
timestamp changes remain separate work.
### Distribution Follow-Up
The pinned graph includes ConfigKit, PromptKit, both PromptKit backend catalogs,
jsonschema/v6, yaml.v3, yaml.v4, x/text, and kr/text (including module/test dependency
edges). Before distribution, review the actual production dependency graph and
its license files, including PromptKit/catalog LGPL-3.0-or-later terms. Decide and
verify the notices, license copies, corresponding library source/modifications,
and rebuild/relink materials needed for Audita's Go binary/container distribution.
This unit does not publish binaries or claim that release compliance is complete.
No upstream relicensing request or library implementation change is required for
this prototype.
## Selected-Overlay Extension: 2026-09-24
The isolated prototype now pins published ConfigKit v0.4.0. ConfigKit loads
`overlays/testing.yml` relative to the selected main file; the overlay selects
complete PromptKit execution profiles from the separate `profiles/` directory.
The original v0.3.1 observations above are historical and do not prove this
extension. No production Audita behavior changed.
With Go 1.26.8 on linux/amd64, from `examples/library-adoption/`, these offline
checks passed with `GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off`:
```sh
go test ./...
go test -race ./...
go vet ./...
go run .
go run . --overlay testing
go run . --overlay testing --proposal-profile proposal
```
Each demonstration reported one accepted correction, four attempts, zero
transient retries, and two structural repairs. The base run selected
`synthetic-grammar`/`synthetic-review`; testing selected
`synthetic-grammar-testing`/`synthetic-review-testing`; the explicit proposal
override restored only `synthetic-grammar`. These are synthetic routing IDs,
not price, latency, quality, or provider-compatibility measurements.
| Requirement | Consumer evidence |
| --- | --- |
| Base, empty/default, testing selection, role overrides, final inheritance, and relative profile resolution | `TestSelectedOverlayConfiguration` loads temporary on-disk layouts and assembles both roles. |
| Actual base/testing/override HTTP models and repairs | `TestDemonstrationSelectsOutboundModels` captures every request body and matches model IDs to the reported selections and domain outcome. |
| Missing/invalid selection, version/directive rejection, and ordinary backend-setting validation | `TestOverlayFailuresAndApplicationValidation` checks ConfigKit diagnostic codes and effective backend limits. |
| Surviving credential resolution, absent/empty failures, null unauthenticated use, and profile-only switching | `TestOverlayCredentialResolutionAndInspection` controls synthetic environment values and observes HTTP Authorization. |
| Redacted effective output, explanation and metadata origin, profile inspection, and observations | `TestOverlayCredentialResolutionAndInspection` and `TestOverlaySelectionProvenance` inspect safe output and overlay/CLI origins. |
| Existing retry, repair, domain, capacity, and concurrent-observation behavior | `TestWorkflowBoundaries`, `TestSharedCapacityAndRecovery`, `TestQueuedCancellationAndDeadline`, `TestDeadlineStopsRetryAfter`, and `TestPartialUsageOnFailedRun` pass, including under `-race`. |
Changing a profile does not remove a backend credential reference from the
effective configuration. ConfigKit validates a surviving reference when only
profile selection changes. Replacing the reference in an
overlay resolves only the replacement; explicit null allows unauthenticated use.
This distinction is part of the selected configuration boundary, not an
application fallback. Imports and secret-directory bootstrap remain disabled.
The tests use a controlled built-in HTTP transport and synthetic credentials;
there were no live provider calls. The prototype still covers one representative
proposal/assessment pair, not all production modules, prompts, CLI paths, or
diagnostic artifacts. Root `go test ./...` does not run this nested module. No
upstream API gap appeared in this extension. The [production implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md)
keeps implementation and release pending.

View File

@@ -0,0 +1,269 @@
# Library Adoption: Shared Boundary and Prototype
**Last updated:** 2026-09-24
**Status:** Original boundary-design and offline-prototype unit completed;
overlay extension and production-planning handoff completed, 2026-09-24.
The work packages below retain the original scope;
[decisions, evidence, limitations, and handoff](library-boundary-findings.md)
record the outcome. The executable prototype was removed after production
integration; Git history at `590604b` retains it. No publication or upstream
changes were made in that historical unit.
## Next Unit: Testing Overlays and Production Migration Plan
**Status:** Complete. The [production implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md)
owns subsequent work; the [overlay findings](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
record this unit's checks. The original work packages below remain
historical scope for the completed first unit.
Extend the existing prototype to prove the [selected-overlay contract](configkit.md#selected-overlays-for-testing)
with ConfigKit v0.4.0, then turn the verified findings into a detailed production
ConfigKit migration plan. Do not repeat the original inventory or redesign the
established library boundary without new evidence.
Required work:
- Upgrade only the nested prototype module from ConfigKit v0.3.1 to v0.4.0.
- Pass an explicit overlay selection through ConfigKit and add `overlays/testing.yml`
plus two PromptKit testing profiles. Select both roles and retain one shared
finite backend. Synthetic model names demonstrate routing; they do not benchmark
real model cost, speed, or quality.
- Make the offline executable accept overlay and explicit role-profile selections,
and show the selected models alongside domain outcomes and retry/repair counts.
- Verify base/overlay/CLI precedence, role inheritance, invalid selections,
surviving credential references, safe inspection, and overlay provenance.
Prove the actual HTTP request models for both roles; loading changed strings
alone is insufficient evidence.
- Record new results separately from the historical v0.3.1 findings. Expand the
existing ConfigKit roadmap into a concrete production implementation handoff,
including final runtime integration and removal of legacy configuration.
Target end state: reviewers can run the prototype with and without the testing
selection and see both LLM roles switch models. The offline test suite establishes
that selection, inspection, and execution agree, and the production ConfigKit work
has bounded tasks with explicit acceptance criteria.
Acceptance criteria:
- The published v0.4.0 dependency is pinned without local replacements; the original
no-overlay path and existing integration tests continue to pass.
- `testing` switches both roles, explicit profile flags override its selections,
and omission/empty/`default` retains base selection according to ConfigKit.
- Invalid overlays fail clearly; inheritance and secret resolution use surviving
effective settings. Inspection records overlay origin without disclosing keys.
- Both selected testing models reach the controlled HTTP transport; structural
repair, shared capacity, observations, and Audita domain checks remain active.
- Updated examples, executable instructions, findings, and a detailed production
migration plan are reviewable. Exact verification commands and limitations are
recorded, including the fact that root tests do not cover the nested module.
- Production code, root dependencies, and upstream libraries remain unchanged.
No release or live provider call is part of this unit.
This unit ends with a production migration **plan**, not production implementation.
The ConfigKit → PromptKit → combined release sequence remains unchanged. Imports
and secret-directory bootstrap remain disabled; PromptKit execution profiles keep
their existing names and meaning.
## Objective
Define and prove the boundary between Audita, ConfigKit, and PromptKit before
replacing Audita's production configuration or LLM runtime. This unit covers the
shared design and representative prototype supporting the
[coordinated adoption roadmap](library-adoption.md). The
[ConfigKit roadmap](configkit.md) continues to own the broader configuration scope.
The deliverable is a concrete target contract supported by an offline executable
prototype, sufficient to implement the ConfigKit migration without subsequently
redesigning its LLM-facing settings. A successful prototype is not completion of
either production migration.
## Established Direction
- This is an intentional breaking migration with a migration guide, not legacy
configuration support, aliases, or a permanent compatibility mode.
- ConfigKit owns shared configuration mechanics; PromptKit owns shared prompt and
generation mechanics; Audita retains domain meaning and transcript correction.
- Structural repair is enabled from the outset for proposal and LLM-validator calls.
Positive default repair budgets must be selected, not silently left at zero.
- Transient retries and structural repair are separate. Explicit zero disables
transient retries; Audita must not add a second automatic retry loop.
- One finite concurrency limit per backend protects local capacity and bounds paid
API activity. Calls from both roles share that limit. Fine-grained role budgets
and an additional global concurrency control are unnecessary.
- Keep the sequence: boundary/prototype, ConfigKit migration, PromptKit migration,
then combined major release. No intermediate public release is planned.
## Scope and Deliverables
| Deliverable | Required result |
| --- | --- |
| Configuration inventory and migration map | Every existing setting and invocation option has a proposed destination, replacement, or removal, with rationale. |
| Boundary specification | One authoritative source for each setting, explicit precedence/defaults, credential flow, runtime conversion, and failure handling. |
| Representative executable prototype | New-style configuration drives one proposal and one assessment through released libraries and a controlled provider. |
| Focused behavioral evidence | Retry, repair, capacity, credential, observation, and domain-boundary scenarios below are demonstrated. |
| Decision and integration handoff | Concrete choices, measured limitations, affected components, removal plan, and next implementation tasks are recorded. |
Do not switch the production CLI to the prototype, migrate every module/prompt,
redesign transcript/glossary formats, change correction algorithms, modify retention
or timestamp behavior, or publish a release in this unit. Do not reproduce the
libraries' full test suites. Upstream changes require a separately scoped task if
an actual gap is found.
## Work Package 1: Inventory and Decide Ownership
Read the current [configuration](../config.md), [CLI](../cli.md),
[LLM runtime](../internal/llm-runtime.md), [prompts](../internal/prompts.md),
[validators](../internal/validators.md), and [output contracts](../integrations/output-files.md).
Use graph-first discovery to verify the relevant code and tests, especially
configuration application, prompt construction, structured responses, scheduling,
and report assembly.
Build a migration table grouped by application/domain settings, invocation state,
backend settings, execution profiles, output contracts, and inspection. Include
current environment settings and aliases without treating their existence as a
requirement to preserve them. Identify report fields whose meanings change when
one logical generation contains multiple provider attempts.
Resolve these design decisions explicitly:
| Decision | Proposed direction to prove |
| --- | --- |
| File selection and version | Audita selects the main configuration and validates a new schema version; specify discovery and the treatment of old versions. |
| Application versus invocation state | ConfigKit loads persistent settings; Audita parses invocation-only input/output paths and translates explicitly supplied setting flags into supported overrides. |
| Backend identity | One registration represents the service capacity shared by proposal and validation calls. Avoid separate identities for the same service merely because roles differ. |
| Endpoint and capacity ownership | Define them once per backend, with finite limits and an explicit policy for waiting capacity/exhaustion. Do not expose endpoint-only unlimited execution accidentally. |
| Execution profiles | Prefer complete application-owned profiles using PromptKit conventions. Decide their source and selection without duplicating model/settings fields in application configuration. Catalog selection remains a deliberate option. |
| Role selection and inheritance | Specify how proposal and validation choose profiles, including omission and any inheritance. Do not inherit old behavior solely for compatibility. |
| Credentials | Prefer one ConfigKit-owned secret-resolution path passed through PromptKit's supported request credential field. Define backend/profile credential association and avoid competing environment lookups. Validate this against profile requirements. |
| Retry, repair, and deadlines | Assign one owner and explicit defaults for each control. Use positive structural-repair defaults and document combined limits and total-operation scope. |
| Optional library features | The original prototype omitted imports, selected configuration layers, and secret-directory bootstrap. Selected overlays are now planned for testing; see the [overlay plan](configkit.md#selected-overlays-for-testing). Imports and secret-directory bootstrap remain out. |
| Inspection | Define effective configuration and selected-profile inspection without implying that one library's snapshot contains the other's complete state. |
Choose concrete field names, example values, defaults, units, and omission/null/zero
semantics for the prototype. In particular, decide what validation and inspection
do when configured credentials are unavailable. Classify choices as proposed,
selected, or unresolved; do not present candidates as implemented contracts.
## Work Package 2: Specify Execution and Reporting Semantics
Document the flow from selected configuration through backend/profile assembly,
prompt inputs, generation, structural validation/repair, typed decoding, and Audita
domain validation. Identify which values cross the boundary and which component
owns each conversion. Keep library types out of unrelated domain code where a
small existing domain value is sufficient.
Use the current correction-set and validator-decision schemas as representative
inputs. Reuse domain checks where practical. Select one real proposal/validator
pair; account for prompt helpers, template variables, shared fragments, and schema
references rather than testing only an unrelated hello-world prompt.
Record the following distinctions:
- Repair seeks output conforming to its structural contract. Success does not
establish semantic correctness or authorize applying a correction.
- A schema-valid decision set still requires Audita's cardinality and proposal
identity checks. Semantic rejection is not automatically another repair request.
- Exhausted structural repair yields failed content validation; provider failures,
retry exhaustion, cancellation, and capacity rejection have separate outcomes.
- Specify each outcome's intended effect on an Audita section/batch and overall run,
including warnings, rejected work, partial domain results, and terminal failures.
Preserve domain intent without preserving obsolete transport behavior by accident.
- For transient retry budget T and structural repair budget R, one logical call can
dispatch up to `(T + 1) * (R + 1)` provider attempts. For example, T=2 and R=2
permits nine attempts; these illustrative values do not select production defaults.
- Choose the total-operation deadline scope for a logical proposal or assessment;
distinguish it from per-attempt timeouts and whole-process cancellation. Provider
retry timing may exceed local backoff, so deadline behavior must be explicit.
Design the Audita observation record and aggregation rules. Include logical-call
and attempt identity, role/backend correlation, queue/generation/retry timing,
repair and retry counts, outcomes, and available usage. Preserve unknown/incomplete
usage rather than inventing zeros or complete costs. Do not add terminal totals to
attempt totals. Collect events safely across concurrent calls, without blocking
callbacks on progress by the same run or persisting raw library events as a stable
Audita schema. Transcript-bearing artifacts remain separately governed by Audita.
## Work Package 3: Build the Bounded Offline Prototype
Use ConfigKit v0.3.1 and PromptKit v0.11.0 as reviewed starting points; verify their
published availability and APIs before pinning the prototype. Recheck the combined
Go minimum (the reviewed PromptKit release requires Go 1.25.5). Do not depend on
sibling checkouts, an active Go workspace, or committed local module replacements.
A proposed home is `examples/library-adoption/` as an explicitly experimental,
standalone Go module, isolating prototype dependencies from the production binary.
Confirm this placement during execution. Its README must give exact offline run
and test commands after dependency acquisition; root `go test ./...` must not be
claimed to cover a nested module. Keep complete proposed configuration and prompt
fixtures with it and clearly label them as prototype-only.
The prototype should:
1. Load representative application settings with ConfigKit, using synthetic secret
values supplied by the test environment and an application-owned version rule.
2. Assemble shared backend registrations and proposal/validation profile selections.
3. Supply representative transcript/proposal inputs to PromptKit definitions and
actual response schemas using supported sources.
4. Use the built-in HTTP generation path with a controlled local endpoint or
transport. An injected LLM client alone cannot prove built-in HTTP retry behavior.
5. Run proposal generation, decode and check candidates, run assessment, and perform
the selected Audita decision checks without modifying real transcript files.
6. Collect selected safe observations and show a compact result understandable to a
reviewer, including repaired output and final domain acceptance/rejection.
No paid calls, real credentials, host environment assumptions, persistent caches,
or general-purpose test framework are needed. Make concurrency and cancellation
checks deterministic through controlled synchronization. Keep delays bounded and
use library-supported controls; do not patch library internals to make tests pass.
## Focused Verification Matrix
| Scenario | Evidence required |
| --- | --- |
| Successful assembled path | ConfigKit-loaded selections drive both real prompt roles through PromptKit and Audita checks. |
| Presence, overrides, and inheritance | Representative omission, explicit values, and role selection follow the proposed contract; unknown fields and unsupported versions fail clearly. |
| Credentials | Required missing/empty references fail as intended; controlled calls receive the chosen synthetic credential; inspection and observations do not expose it. |
| Structural repair succeeds | Invalid proposal output and invalid assessment output are repaired in their respective workflows; positive budgets are actually exercised. |
| Repair exhausted | Repeated invalid output produces the selected Audita outcome and accurate accounting, without an unbounded outer retry. |
| Domain rejection | Structurally valid but unacceptable correction or decision is rejected by Audita, demonstrating the semantic boundary. |
| Transient retry and explicit zero | An eligible temporary status recovers; zero makes one attempt. Exhaustion retains a useful classified cause. |
| Combined repair and retry | A repaired generation encountering a transient failure respects both budgets and reports distinct counts. |
| Shared capacity | Concurrent calls from both roles, including retry/repair activity, stay within one selected backend cap; test a representative local cap of four and a finite API cap. |
| Waiting, cancellation, and deadline | Cancellation/deadline stops pending work and retry waits; capacity exhaustion has the intended application outcome and later calls can still proceed. |
| Failed-run observations | Completed work remains observable after later failure; unknown usage, known zero, and aggregate counts are mapped correctly. |
Tests should protect Audita's integration decisions and important assembled behavior.
Rely on PromptKit's tests for exhaustive HTTP-status, backoff, and scheduler details;
rely on ConfigKit's tests for its parser and presence machinery. Use race testing for
new concurrent observation collection. Keep consumer checks offline and credential-free.
## Completion Criteria and Handoff
This unit is complete only when:
- The settings/ownership table and migration inventory are concrete enough to start
the ConfigKit migration, including file/profile sources, CLI scope, credentials,
defaults, finite backend limits, positive repair budgets, and deadline policy.
- The offline prototype and focused matrix demonstrate the selected boundary with
both libraries together, including both LLM roles and Audita domain checks.
- Exact verification commands, results, limitations, and unresolved upstream gaps
are recorded. Missing evidence is not reported as successful integration.
- No adoption-critical decision remains hidden in prototype code. If a gap prevents
settling the boundary, record it and revise the plan before declaring readiness.
- The existing Audita production behavior remains intact in this unit. Temporary
code is clearly isolated and has a removal/integration destination during the
production migrations; no second supported runtime is introduced.
- A component-level follow-up identifies code to replace, domain code to retain,
tests to update, documentation/migration-guide work, and direct integration of
the final configuration and PromptKit runtime.
- Licensing/distribution follow-up for the selected dependency graph is identified
for the later release stage; this prototype does not claim release compliance.
- Roadmap links and examples resolve, appropriate checks pass, and commits contain
only this unit's work. Significant settled architectural choices may be recorded
in an ADR under the [documentation policy](../policy/documentation.md), while
implementation status stays in the roadmaps.
Retain the overall adoption order unless evidence or the other planned large changes
justifies revising it. Completing this unit enables detailed production migration;
it does not authorize an intermediate release or require unrelated runtime fixes.

161
docs/roadmap/remediation.md Normal file
View File

@@ -0,0 +1,161 @@
# Maintainability Audit: Findings and Recommended Changes
**Reviewed:** 2026-09-25, source commit `011b021`.
**Scope:** Recent partial-decision reconciliation, validation context, prompt
alignment, and request-local reference changes, including related existing code.
**Status:** R27–R30 addressed; see the resolution record below. Previous findings and resolution records remain in Git
history through `011b021`.
## Architecture Assessment
The recent changes fit Audita's existing boundaries. Proposal generation,
validation, prompt assets, and outcome reporting already have suitable owners;
this audit does not recommend a new common package or a broad refactor.
The strongest opportunities are shared prompt instructions and request-scoped
validation context preparation. Two smaller improvements concern recovery-reason
classification and partial-result assembly; both predate the latest commits.
These are bounded maintenance recommendations, not new release gates or changes
to correction policy, response acceptance, or reporting semantics.
## R27 — P2: Consolidate Shared Prompt Instructions
**Category:** Repeated behavioral-contract text; risk of future drift.
**Evidence:** The four [proposal prompts](../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml)
repeat literal source-span matching, request-local identity, replacement-text,
source-based proposal, and confidence instructions. The three active
[validator prompts](../../internal/prompts/assets/prompts/validators.editorial_review.yaml)
repeat candidate-reference, supplemental-context, and verdict-confidence guidance.
Recent contract changes required coordinated edits across these assets. The
[prompt registry](../../internal/prompts/registry.go) already provides a shared
hardening fragment supplied through the role-bound adapter.
**Recommended change:** Extract genuinely common passages into embedded fragments
under `internal/prompts/assets/shared`, following the existing fragment-supply
approach. Keep the fragments owned by `internal/prompts` and use the existing
PromptKit rendering path. Keep module-specific examples, replacement-policy
instructions, protected-name exceptions, and semantic criteria in their individual
prompt definitions. Do not merge similar-looking rules with different meanings.
**Acceptance:** Common instructions have one maintained source and appear in all
intended rendered prompts. The existing module and validator distinctions remain
intact, including glossary replace-all behavior and the spoken-word protected-name
exception. Focused rendering checks cover fragment inclusion and representative
module-specific guidance without requiring exact-prose snapshots or introducing
a separate templating framework.
## R28 — P2: Prepare Validation Context Once Per Request
**Category:** Repeated computation and overlapping segment lookup.
**Evidence:** [BuildLLMValidationRequest](../../internal/framework/validators/llm_validators.go)
already indexes transcript segments, but
[validationContext](../../internal/framework/validators/llm_context.go) scans the
transcript again for each candidate. It also calls
[NewProtectedVocabulary](../../internal/framework/validators/protected_terms.go)
for each candidate, rebuilding the same vocabulary and compiling its matching
expression repeatedly. This is duplicated preparation rather than copied code.
**Recommended change:** Prepare a segment-position lookup and protected vocabulary
once per validation request, then pass that prepared context into the existing
context-selection helper. Keep the implementation within `framework/validators`.
Continue selecting glossary evidence for each candidate individually because its
proposed replacement can mention different terms. A process-wide cache or new
context-management package is unnecessary for this cleanup.
**Acceptance:** Candidates in one validation request reuse the prepared lookup and
vocabulary. Context still consists of the available immediate neighbors within the
current chunk and relevant glossary entries in glossary order. Candidate-specific
replacement text still participates in glossary selection. Existing optional-context
trimming, token estimation, and oversized-input handling retain their behavior.
Verify these outcomes with focused behavioral tests rather than assertions about
private helper call counts or unmeasured performance guarantees.
## R29 — P3: Share Execution-Recovery Classification
**Category:** Duplicate reporting rule; predates the latest commits.
**Evidence:** [Proposal generation](../../internal/framework/proposal_generation/generate.go)
and [LLM validation](../../internal/framework/validators/llm_validators.go)
independently translate structural-repair and transient-retry counts into
`output_repaired` and `service_recovered` reasons. The conversion is the same in
both locations; the surrounding success and domain-outcome decisions differ.
**Recommended change:** Place a small conversion helper in the existing
`framework/outcomes` package and use it from both callers. Keep the decision to
record recoveries at the call sites, along with domain-specific reasons such as
filtered validator decisions. Do not combine proposal and validation orchestration
or move domain-success classification into this helper.
**Acceptance:** Both callers use the same execution-count conversion. Existing
reason ordering, successful-operation reporting, and domain-specific additions
remain intact. A partially unavailable validation batch continues to record its
failure and execution counts without acquiring recovery records merely because a
repair or retry occurred. Existing outcome tests should continue to protect these
distinctions.
## R30 — P3: Centralize Partial Validator Result Assembly
**Category:** Repeated early-return construction; predates the latest commits.
**Evidence:** Several early returns in
[LLMBackedValidator.Validate](../../internal/framework/validators/llm_validators.go)
repeat nested slice appends to combine immediate decisions, oversized-input
decisions, and decisions from completed model batches, together with accumulated
operation outcomes. This makes it easier for a future failure path to omit facts
already collected by the validator.
**Recommended change:** Use a small local result-building helper to assemble the
partial result consistently. Keep error handling, batching, success-path ordering,
and cardinality enforcement explicit. A general-purpose result-aggregation package
is not needed.
**Acceptance:** Cancellation and failure paths retain the same previously collected
decisions and operations. Result ordering and cardinality behavior remain unchanged;
the helper does not enforce complete cardinality on interrupted work. Retain the
existing partial-progress and fatal-failure coverage, adding tests only for a
material gap exposed by the refactor.
## Boundaries to Preserve
- Keep proposal and validator reference mapping separate. Proposals can target the
same segment repeatedly; duplicate validator decisions make that candidate
unavailable. Similar indexing arithmetic alone does not justify a generic
reference-management abstraction.
- Keep distinct protected-term policy checks separate. They already share the
vocabulary implementation and intentionally apply different policies.
- Preserve exact source-span matching, request-local references, batching, and
partial-success behavior. Consolidation is not a reason to strengthen contracts
or broaden the accepted correction scope.
## Review Verification and Limits
- This is a source-level maintainability audit, not a finding of demonstrated
production failures or a measured performance assessment.
- No application tests were rerun for the original read-only audit; its roadmap
update used documentation link and whitespace checks. Implementation validation
is recorded below.
- Future remediation should run focused behavioral tests and the broader suite
appropriate to the actual change, following the testing policy. No live-provider
evaluation or new evaluation framework is required by these recommendations.
- The findings above describe the audited baseline. Implementation and validation
are recorded below; no release publication is included.
## Resolution Record
Implemented as focused changes within the existing packages. Proposal and validator
reference mapping and distinct protected-term policies remain separate.
| Finding | Resolution and evidence |
| --- | --- |
| R27 | Common proposal and validation instructions live in two embedded shared fragments, supplied through the existing role-bound PromptKit adapter. Individual YAML definitions retain policy differences and examples. Rendering tests check fragment inclusion once per applicable prompt and representative module-specific guidance. |
| R28 | Validation request construction builds one segment-position lookup and protected vocabulary, reused by context selection for every candidate. A new test verifies that two replacements of the same source span receive their own relevant glossary entries. Existing neighbor, glossary-order, chunk-boundary, and budget tests pass. |
| R29 | Both proposal generation and validation use `outcomes.ExecutionRecoveries`. Call sites still control success and domain-specific recovery reasons. Existing tests cover combined recoveries and failed validation with repair counts but no recovery record. |
| R30 | A local result builder combines collected decisions and operation outcomes for partial returns and completed validation. Sorting and full cardinality checks remain outside it. Existing fatal-batch, partial-progress, runner, and reporting tests pass. |
Validation: `go test ./...` passed, including PromptKit preparation, partial-decision
reconciliation, subprocess reports, and fatal-progress coverage. Documentation
links and whitespace checks passed. No live model evaluation or performance
measurement was performed; the changes make no new model-quality or latency
promise.

290
docs/roadmap/resilience.md Normal file
View File

@@ -0,0 +1,290 @@
# Resilient Processing and Focused Run Reports
**Last updated:** 2026-09-24
**Status:** Implemented. This document records the selected behavior and user
intent. Concrete design decisions and completion evidence are in
[historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/a83a518/docs/roadmap/implementation.md).
## Purpose and User Intent
Audita should reliably produce a useful transcript even when its LLM services
cannot improve it. Exhausted retries or unusable model responses must not turn
an otherwise processable transcript into a failed production run.
> Failure of every LLM operation across every stage and chunk still produces a
> successful output containing the deterministically normalized transcript, with
> no LLM corrections applied. Partial success preserves as many fully validated,
> applicable corrections as possible.
Deterministic normalization remains intentional and may change segment layout,
identifiers, and text before LLM work begins. Output uses the selected transcript
encoder; byte-for-byte preservation of the original input and mapping corrections
back to pre-normalization segmentation are not requirements.
The overriding goal is to minimize how much useful processing is lost because of
one problematic transcript segment. A stage/chunk work unit is an attribution
boundary, not an all-or-nothing transaction.
Existing reports contain useful information, but there are no compatibility
obligations for reporting or debug output. Reuse existing machinery only when it
makes the implementation simpler and clearer; preservation is not itself a goal.
Internal structures and measurements must serve Audita's processing workflow.
The reporting package adapts those outcomes to structured and human-facing
presentations, rather than reshaping processing around a reporting convenience.
## Historical Baseline and Implemented Change
The [pipeline](../internal/pipeline.md), [LLM integration](../integrations/openai-compatible-llm.md),
[report assembly](../internal/diagnostics-reporting.md), and
[artifact lifecycle](../operations.md) describe current behavior.
Before this work, exhausted structural repairs could become warnings and skipped
proposals or rejected validation batches. Other model runtime failures canceled
peer work, stopped subsequent modules, and prevented final transcript output.
Successful repairs appeared in execution observations rather than a dedicated
recovery classification. Detailed artifacts were written during ordinary runs
and could later be removed by retention. The working transcript was re-chunked
between modules.
This feature changed those behaviors. It builds on the completed
[library adoption](library-adoption.md) and [review remediation](remediation.md).
Fatal cancellation remains necessary; unsuccessful model work must be separated
from failures that actually require cancellation of the run.
## Fixed Chunks and Ordered Stages
Normalize first, then partition the transcript once. Preserve chunk boundaries,
segment membership, segment identity, and ordering throughout LLM processing.
Text evolves as accepted corrections are applied. Recombine chunks in original
order after the LLM stages finish.
A stage is a configured module instance. Repeated instances of the same module
have distinct identities. Proposal generation and each validator are operations
within a stage; validation may use multiple batches within a chunk. A work unit
is one stage instance applied to one chunk.
Preserve the overall stage sequence: finish a stage across its chunks before
advancing to the next stage. Concurrent chunk work shares PromptKit's backend
capacity. Failure in one unit does not prevent other chunks or later stages
from executing.
Retain operator control over chunk size and the current defaults. Production
experience indicates that chunk size affects model latency nonlinearly and that
useful settings are well below backend context limits. Tuning is an empirical
operator decision. Do not add automatic headroom calculations, cross-profile
sizing, adaptive repartitioning, or automatic tuning in this feature. A later
operation that cannot process its input follows the same safe error handling;
it does not silently alter chunk boundaries. Validation batching within a chunk
remains supported.
Independent whole-pipeline traversal by chunks and domain-aware response repair
are explicitly deferred in [future work](future.md). Read-only neighboring
context, cross-chunk edits, and transcript-wide restructuring are outside this
unit, without becoming permanent prohibitions on future capabilities.
## Partial Success and Failure Isolation
- Keep bounded transport retries, structural repairs, operation deadlines, and
shared backend capacity. Completion comes from safe fallback, not unlimited
attempts or an outer retry loop around PromptKit.
- Apply a correction only after it passes every required validator and the
application checks. Failed validation must never be treated as approval.
- If a validation batch fails, exclude its affected candidates while allowing
other candidates to continue through the remaining validators. Preserve fully
validated, applicable corrections from successful batches in the same unit.
- Track candidates by stable proposal identity. Batch membership can change
between validators; equal batch numbers do not establish candidate identity.
- A failed proposal-generation operation leaves its chunk unchanged for that
stage because there are no reliable proposals to salvage. Failures later in
validation need not discard unrelated successful candidates.
- Preserve earlier stages' corrections and other chunks' successful work.
Record partial success explicitly; a unit can apply corrections and have errors.
- Continue despite model-operation failures, including exhausted repairs/retries,
unusable model decisions, provider/transport failures, local operation deadlines,
and admission failures. An operation need not exhaust its attempt budget to fail.
- A valid empty proposal list or valid assessments rejecting all corrections is
successful processing. Correction count alone does not determine failure.
- Keep safe filtering of defective proposals. Do not retry ordinary editorial
rejections. For now, an unusable validator decision set excludes the affected
candidates and records an error after existing permitted recovery. New bounded
feedback/regeneration for domain-invalid responses belongs to future work.
- Retain fatal treatment for invalid input/configuration, internal defects,
explicit run cancellation, and inability to produce/write the transcript.
Caller cancellation is distinct from a local model-operation timeout.
- Completed runs with model errors return success. Required report-write failures
are an additional fatal output condition when reporting is enabled, as below.
PromptKit owns transport, transient retries, structural repair, and admission.
Audita owns domain acceptance, partial outcomes, fallback, and classification.
ConfigKit owns configuration resolution. Debug and reporting choices must not
change correction decisions, retry budgets, or continuation.
## Errors, Recoveries, and Debug
Classify by eventual outcome and impact. These are three reporting categories;
debug is evidence, not a failure severity. Whether execution continues is a
separate property.
| Category | Meaning | Representative cases |
| --- | --- | --- |
| Errors | Intended work could not complete reliably, or a fatal run-level operation failed. Affected candidates are excluded; other corrections can succeed. | Exhausted repair; unavailable provider; unusable validation decisions; operation deadline. |
| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient service failure recovered; defective proposals safely discarded. |
| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompts/responses, individual attempts, execution traces. |
Normal editorial disagreement belongs in debug. A validator correctly rejecting
a meaning-changing edit is expected behavior. A proposal referring to nonexistent
original text is defective; safely filtering it is a recovery. An unresolved
missing/duplicate decision-index response is an error, not a recovery.
Record an operation's eventual outcome once. Several failed attempts followed by
success produce a consolidated recovery. Attempts ending in failure contribute
to error details, not additional recoveries. Earlier operations that recovered
remain recorded if a later operation introduces an error in the same unit.
Separate output-quality recovery, defective-proposal filtering, and service
recovery with structured reasons. Network instability must not look like a prompt
defect. Do not introduce an arbitrary warning threshold or a fourth category.
### Counting and Processing Volume
- Count error-bearing work units once per stage/chunk, consolidating underlying
operation failures as details. A partially successful unit still counts once.
- Count recoveries once per recovered operation. Aggregate defective-proposal
filtering per proposal response, retaining affected-proposal counts as detail.
Attempts and individual discarded proposals are not independent incidents.
- Keep work-unit coverage separate from operation/recovery counts. Show processing
volume with explicitly labeled denominators; do not divide recovered-operation
counts by work-unit counts and call that a recovery rate.
- Report fatal run-level errors separately from the stage/chunk grid when they
cannot be attributed to a work unit. Do not invent a chunk to accommodate them.
- Derive YAML records, Markdown summaries, and CLI counts from actual processing
outcomes with deterministic ordering, rather than independently calculating them.
## Production Outputs, Placement, and Lifecycle
Reporting is enabled by default. The ordinary bundle is:
- `errors.yml`: structured error-bearing units, their underlying failures, and
applicable run-level failures.
- `recoveries.yml`: structured recovered operations and pertinent counts.
- `report.md`: a concise human-facing summary of outcome, corrections, processing
volume, errors, recoveries, and their locations in the pipeline.
Write all three documents, including empty event lists for uneventful completed
runs. Use Audita-owned schemas with safe explanations and stable reason codes;
retain run, stage-instance, chunk, operation, and validator/batch identities where
applicable. The implementation plan specifies the versioned schemas; neither old
report formats nor raw library structures define the new output contract.
Place reports beside the output transcript. Retain transcript-to-stdout support
with an explicit report-directory selection, defaulting to the invocation working
directory. Preserve the existing overwrite convention: a new run can replace files
at its selected destinations. Do not add a mandatory fresh-directory restriction
or compatibility exports.
Before model work, create initial versions of all three reports carrying a common
run identifier and explicit `in_progress` status (rendered as prose in Markdown).
Fail immediately if initialization fails. Also check the transcript destination;
creating neighboring reports does not prove that an existing transcript file is
writable. Preflight must not prematurely destroy an existing transcript.
Finalize all required outputs and check every final write. Initial write success
cannot guarantee later success if space or permissions change. Failure to write
the transcript or finalize a required report returns nonzero and must not erase
an already produced transcript. Where possible, finalize available reports with
an explicit failed status on fatal termination. Abrupt termination may leave
`in_progress`; elaborate crash-recovery machinery is not required.
Provide an option to disable reporting entirely. It suppresses all three files
(`errors.yml`, `recoveries.yml`, and `report.md`), including initialization and
final writes, while retaining the brief CLI summary and independently selectable
debug output. Disabled reporting must not create or require a writable report
destination, and must not remove reports from earlier runs.
Transcript output failures remain fatal regardless of reporting selection.
Ordinary reports survive independently of debug cleanup. Reporting configuration
and finalization must not depend on retaining a debug run directory.
## Debug and Internal Boundaries
Production issue records are compact. Full responses, transcript excerpts,
correction lists, and complete validation decisions are opt-in debug evidence.
Use a single ConfigKit-backed debug boolean with a `--debug` CLI override. Debug
artifacts may use JSON where useful. Preserve configured-secret redaction in every
presentation.
Debug-off prevents detailed artifact persistence; writing and later deleting
those files is insufficient. In-memory domain state needed to process candidates
correctly is not debug output. Debug and ordinary reporting are independent
selections and do not alter processing behavior.
There is no obligation to retain `--report-json`, old artifact layouts, parallel
exports, or old retention settings. Reuse code only where useful to the new
workflow. Enabled debug output is retained on success or failure; disabled debug
creates no debug directory. This supersedes the retention question in the
[runtime follow-up](runtime-behavior.md). The legacy `never` behavior is not the
meaning of the new reporting-disable option. Optional debug-write failures produce
a safe stderr diagnostic without changing correction decisions or the run exit
status; ordinary report and transcript-write failures retain their fatal semantics.
## Deterministic Human-Facing Presentation
Use the CLI style:
> Completed: 18 corrections applied; 3 work units with errors; 7 recoveries.
Do not say that all error-bearing units were skipped: some apply valid corrections.
Include report locations when reporting is enabled and the files exist. Send the
summary to stderr so stdout remains exclusively available for transcript output.
Disabled reporting does not suppress the summary or fatal-error messages.
Keep `report.md` concise and high-level. Use separate error and recovery matrices
with stages as columns and chunks as rows. Include every chunk in fixed order;
retain configured stage order and distinct labels for repeated module instances.
No automatic width detection, thresholds, adaptive omission, or table splitting.
A wide or long table is acceptable.
The error matrix counts error-bearing work units: each attempted cell is 0 or 1.
Recovery cells count recovered operations and can exceed 1. An error cell may
coexist with recoveries and successfully applied corrections in the same unit.
Distinguish completed work with zero events from unattempted work; do not portray
canceled/unavailable processing as clean completion. Include totals, processing
volume, and a short reason breakdown. Detailed inspection belongs in YAML or debug.
## Target Acceptance Criteria
- Total LLM failure still returns success and the encoded normalized transcript,
provided required output writes succeed.
- Mixed-success runs preserve independently validated, applicable corrections
within the same stage/chunk as a failed batch. No candidate bypasses a required
validator; later validators handle survivors by stable proposal identity.
- Subsequent stages and unaffected chunks execute after recoverable failures.
Fatal errors and explicit cancellation stop and join active work appropriately.
- Chunk boundaries and identity remain fixed, with operator-selected sizing and
unchanged defaults; final recombination preserves ordering.
- Valid no-change/editorial-rejection outcomes are successful. Repaired/filtered
output and unresolved failures receive distinct classifications and counts.
- YAML, Markdown, and CLI agree on partial outcomes and totals; error matrices
count affected work units, recovery matrices count recovered operations.
- With reporting enabled, all three reports are initialized before model work,
finalized and retained, including empty lists. Initial or final required-output
write failures produce nonzero exits without destroying completed transcript output.
- Reporting can be disabled without requiring its destination. Debug can be enabled
independently; neither selection changes processing outcomes. Debug-off writes
no detailed debug artifacts, and stdout transcript output remains supported.
- Reports use the fixed deterministic layout without adaptive presentation logic.
- Meaningful controlled-provider tests cover exhausted recovery, failed validation
batches with successful survivors, total failure, fatal cancellation, output
failures, and reporting/debug selections. Preserve domain safeguards, capacity
limits, and redaction.
- Canonical CLI, configuration, operations, integration, internal, and architecture
documentation describes the implemented final behavior.
## Implementation Reference
The product decisions are settled. The [historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/a83a518/docs/roadmap/implementation.md) defines
the selected settings and precedence, YAML schemas, counts and ordering, interrupted
work presentation, destination checks, and output finalization. It also owns the
ordered implementation stages and validation work. This document remains the
reference for the desired behavior and its acceptance criteria.

View File

@@ -1,15 +1,16 @@
# Runtime Behavior Decisions
**Last updated:** 2026-09-23
**Last updated:** 2026-09-24
The documentation migration identified these existing behaviors for separate
product consideration. No behavior change is approved or implemented by this list.
The documentation migration identified existing behaviors for separate product
consideration. The resilience implementation resolved the diagnostics-retention
question; the input-timestamp question remains open.
- Decide whether the `never` retention selection should delete successful-run
diagnostics; current behavior is documented in
[operations](../operations.md#retention-behavior).
- Decide whether an explicit zero retry count should disable retries; current
adapter behavior is documented in [configuration](../config.md).
- Resolved: legacy `auto`/`always`/`never` retention settings were removed.
Debug is explicitly enabled with `diagnostics.debug` or `--debug`, creates no
directory when disabled, and retains its directory on every outcome when enabled.
- Resolved: [configuration](../config.md) now treats explicit zero as disabling
transient retries or structural repair independently; PromptKit performs both.
- Decide whether missing or null input timestamps should be rejected; current
decoding is documented in the [input contract](../integrations/transcript-glossary-files.md).

View File

@@ -0,0 +1,203 @@
# Consistent Span Deletion
**Last updated:** 2026-09-25
**Status:** Implemented. This document owns the desired behavior and user intent.
The [historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md)
records the concrete steps, selected implementation details, and completion
evidence.
## Purpose and User Intent
Align proposal prompts, the LLM response schema, internal validation, and
application behavior so legitimate text-span deletions can be considered and
applied through the ordinary correction workflow. An empty replacement should
not trigger structural repair merely because it is empty, nor invalidate other
usable corrections returned alongside it.
This is a consistency fix within Audita's current editing scope. It does not
authorize whole-segment removal, broaden module responsibilities, or introduce
new recovery machinery. The accepted [resilience behavior](resilience.md) remains
in effect, including partial success, fixed chunks, and validation before mutation.
## Historical Baseline (`0b6cd98`)
The source reviewed at `0b6cd98` contains a mismatch:
- The [spoken-word prompt](../../internal/prompts/assets/prompts/modules.spoken_word.proposal.yaml)
permits removing fillers and asks for the exact affected span and replacement,
without explaining how to represent deletion. Grammar permits punctuation and
spacing changes, which can also naturally involve deletion.
- The shared [correction response schema](../../internal/framework/responseschema/assets/correction_set.v1.json)
requires `corrected_text` to have at least one character. Every built-in proposal
module uses that schema. An empty replacement invalidates the response before
Audita evaluates individual proposals.
- PromptKit performs bounded structural repair. If repair does not produce a
valid response, the proposal operation fails and yields no candidates, including
otherwise valid proposals in the same response. The current resilience path
records the error and continues other work.
- [CorrectionProposal](../../internal/framework/proposals/proposal.go) already
allows empty replacement text. The
[preview/application logic](../../internal/framework/proposals/preview.go)
supports literal span deletion when the resulting segment remains nonblank.
- Domain shape checks currently reject whitespace-only original spans, although
the response schema permits them and grammar prompts permit spacing cleanup.
This is another restriction to align with the target contract below.
- The validator named `non_empty_corrected_text` actually checks the resulting
segment, not the replacement string. Both this check and application prevent
a segment from becoming empty or whitespace-only. Application changes segment
text; it does not remove segment records.
A temporary controlled-provider review reproduced whole-response rejection for
an empty replacement alongside a valid correction, and immediate schema acceptance
for a space-only replacement. Existing proposal tests confirm internal support for
partial deletion and rejection of blank resulting segments. These observations
explain a plausible production failure mechanism; they do not establish the cause
of any particular historical incident without its payload and running version.
## Target Correction Contract
- `original_text` identifies an existing, nonempty literal span in the current
target segment. Whitespace-only spans are eligible; an empty string identifies
no source span and remains invalid. Matching and ambiguity checks still apply
under the module's replacement policy: `require_unique` for grammar, spoken-word,
and homophones; `replace_all` for glossary.
- `corrected_text` remains a required string. An empty string means delete the
identified span; it does not mean remove the segment record or skip validation.
Missing fields and non-string values remain structurally invalid.
- After applying a correction, the segment must contain non-whitespace text.
Enforce this using the source segment and replacement result in Audita, not a
minimum replacement length in the structural schema.
- Every accepted deletion passes the same required validators as other edits in
its module, including confidence, protected terms, editorial review, and meaning
preservation where applicable. Empty replacement is neither automatic approval
nor an automatic rejection.
- Application preserves segment identity, speaker, timing, membership, ordering,
and fixed chunk boundaries. Recheck applicability against text produced by
earlier accepted corrections, retaining existing conflict handling.
- Preserve literal replacement behavior. Prompts should select spans and
replacements that handle adjacent whitespace and punctuation appropriately;
this feature does not add automatic trimming or a second normalization pass.
Neither side of a correction is rejected solely for containing only whitespace.
For example, reducing a uniquely identified run of two spaces to one space may
be appropriate; validity still depends on the resulting text and domain checks.
Preserve the intended safeguards and outcomes, not every detail of the current
implementation. Validator ordering, internal structure, and payload construction
may change when needed to achieve this consistent contract. Existing behavior or
a passing test does not justify retaining a conflicting restriction.
Illustrative cases:
| Segment | Original span | Replacement | Intended handling |
| --- | --- | --- | --- |
| `uh hello` | `uh ` | empty string | Structurally valid; may produce `hello` after required validation. |
| `uh hello` | `uh hello` | `hello` | Remains a valid representation of the same cleanup, subject to validation. |
| `hello world` | two spaces | one space | Eligible spacing correction, subject to unique matching and required validation. |
| `uh` | `uh` | empty string | Structurally valid proposal, then filtered because the resulting segment would be blank. |
| `hello world` | `hello world` | spaces only | Filtered because the resulting segment would be blank. |
These examples describe eligibility and mechanical results, not unconditional
editorial approval. An unchanged segment or valid empty correction list remains
a successful no-change outcome.
## Prompt and Module Alignment
Review relevant prompt definitions for consistency and edit only those needing
clarification. Where needed, explain empty-string deletion, whitespace-only spans,
and the resulting-segment limit with a concise, module-appropriate example. The
model should omit a correction that would leave a blank segment rather than invent
padding or replacement content. Consistent guidance is the goal; adding deletion
language to every prompt is not a requirement.
Retain each module's existing scope:
- Spoken-word cleanup can propose appropriate filler or dysfluency deletion while
preserving substantive content and intentional emphasis.
- Grammar can propose permitted punctuation or spacing deletion; allowing an
empty replacement does not authorize word removal or spoken-word cleanup.
- Glossary and homophone modules retain their existing correction purposes.
Shared deletion syntax does not grant them new editorial responsibilities.
Review the active LLM-validator prompts and their input construction as well as
proposal prompts. Validators must understand the before/after effect of deletion
and assess it under their existing criteria. A correct negative editorial decision
remains ordinary processing, not a defect that should be regenerated until approved.
The resulting-segment validator's name and documented identity should describe
that responsibility accurately, rather than imply that the replacement string
itself must be nonempty. Preserve the existing blank-result failure reason and
safeguard while making that deliberate identity change; current chain position
and preview implementation are not independent product requirements.
## Outcomes and Reporting
A legitimate deletion is an ordinary correction. Empty replacement alone must
produce neither structural repair nor a recovery classification. If another
problem required successful repair or filtering, retain that actual recovery.
A schema-valid proposal that would blank its stage-input segment is a defective
proposal handled through existing safe filtering. Preserve the
`empty_resulting_segment` reason
and consolidated `proposals_filtered` recovery behavior; other usable proposals
in the response continue through required validation. Ordinary editorial rejection
remains debug evidence rather than a model error or filtering recovery.
A stage can propose several edits to one segment in the same response. Each edit
is evaluated against the stage input, then approved edits are applied sequentially.
If a proposal was applicable to that input but an earlier accepted edit makes it
unsafe, skip it as an apply-time conflict. Do not count that conflict as a defective
proposal, model error, or filtering recovery. For example, in `"uh um"`, deleting
`"uh "` and deleting `"um"` each leaves text individually; after the first deletion,
the second must be skipped to prevent a blank result. Retain the skipped decision
in debug evidence. The existing runner already distinguishes this situation.
Debug evidence must accurately retain the empty replacement and the corresponding
validation/application outcome. Ordinary reports retain their existing counts and
purpose; no new report schema, incident category, or deletion-specific counter is
needed. Unrelated structural defects may still exhaust PromptKit repair and produce
an error for the operation, as today.
## Acceptance Criteria
- A controlled valid response containing an empty replacement reaches Audita's
domain validation without structural repair solely due to that empty string.
- A permitted span deletion can pass the required validators and reach the final
transcript while preserving segment metadata and fixed chunk membership.
- A mixed response can apply an approved deletion and another approved correction;
an empty replacement does not invalidate the whole response.
- A proposal that would blank its stage-input segment is filtered independently,
retains the existing reason, and does not prevent unrelated valid candidates
from succeeding. A proposal made unsafe only by an earlier accepted edit is
skipped without adding an error or filtering recovery; a blank result is still
prevented and the conflict remains visible in debug evidence.
- A uniquely matching whitespace-only original span can reach validation and
application. Empty originals remain invalid, ambiguous matches remain subject
to module policy, and whitespace-only resulting segments remain disallowed.
- Protected-term and meaning-changing deletions remain subject to their normal
rejection rules. No candidate bypasses required validation.
- A deletion needing no recovery counts as an ordinary applied correction; actual
repair/filtering and editorial rejection retain their existing classifications.
Debug evidence represents the empty replacement faithfully.
- Schema and domain tests distinguish an empty string from missing/non-string
replacement values and distinguish empty replacement from blank resulting text.
- Prompt guidance and validator naming express the same contract as the code.
Tests protect outcomes rather than exact prompt prose or incidental formatting.
Use deterministic, offline controlled responses for behavioral integration tests;
no live-provider or paid evaluation is a prerequisite. Representative production
examples can inform prompt quality without expanding this into an evaluation
framework or guaranteeing that models always follow the instructions.
## Deferred Work
[Whole-segment removal](future.md#whole-segment-removal) is only potential future
work and first requires deciding whether its benefit warrants implementation.
Domain-aware regeneration and independent whole-pipeline traversal also remain
in [future.md](future.md). This feature changes neither retry budgets nor the
Audita/PromptKit ownership boundary and requires no upstream library feature.
Implementation sequencing and concrete names belong to the
[historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md).
The target remains consistent deletion of spans within surviving segments, with
no new operator controls or report types.

View File

@@ -1,6 +1,12 @@
version: 1
version: 2
backends:
local:
endpoint: http://127.0.0.1:11434/v1
concurrency: 1
llm:
proposal_profile: local-proposal
validation_profile: local-validation
pipeline:
modules: [grammar]
output:
schema: bare-segments
llm:
proposal:
api_key_env: AUDITA_LLM_API_KEY

View File

@@ -0,0 +1,3 @@
llm:
proposal_profile: service-proposal-testing
validation_profile: service-validation-testing

View File

@@ -1,34 +1,29 @@
version: 1
version: 2
backends:
service:
endpoint: https://openrouter.ai/api/v1
credential: AUDITA_SERVICE_KEY
concurrency: 2
queue_capacity: 1024
llm:
proposal_profile: service-proposal
validation_profile: service-validation
transient_retries: 2
repair_attempts: 2
operation_timeout: 15m
pipeline:
modules: [glossary, homophones, glossary, spoken_word, grammar]
output:
schema: audita-v1
llm:
proposal:
base_url: https://openrouter.ai/api/v1
model: openrouter/google/gemma-4-31b-it
api_key_env: AUDITA_LLM_API_KEY
timeout: 120s
max_retries: 3
validation:
base_url: https://openrouter.ai/api/v1
model: openrouter/google/gemma-4-31b-it
api_key_env: AUDITA_VALIDATION_LLM_API_KEY
timeout: 120
max_retries: 3
concurrency:
total_llm: 2
proposal_llm: 2
validation_llm: 1
chunking:
# Requires at least this many normalized segments; omit for automatic chunking.
target_sections: 8
max_section_tokens: 8192
min_section_tokens: 2048
normalization:
max_segment_gap: 4s
ellipsis_gap: 3.5s
max_segment_duration: 60s
max_segment_gap: 4.0
ellipsis_gap: 3.5
max_segment_duration: 60.0
max_segment_tokens: 2048
thresholds:
glossary: 0.8
@@ -37,6 +32,9 @@ thresholds:
grammar: 0.8
context:
description: "General context for domain vocabulary and speaker names."
reporting:
enabled: true
stdout_dir: .
diagnostics:
debug: false
work_dir: /tmp/audita
retention: auto

View File

@@ -0,0 +1,7 @@
id: local-proposal
backend: local
model: replace-with-local-model
temperature: 0
max_tokens: 2048
top_p: 1
timeout_seconds: 120

View File

@@ -0,0 +1,7 @@
id: local-validation
backend: local
model: replace-with-local-model
temperature: 0
max_tokens: 1024
top_p: 1
timeout_seconds: 120

View File

@@ -0,0 +1,7 @@
id: service-proposal-testing
backend: service
model: replace-with-testing-model
temperature: 0
max_tokens: 2048
top_p: 1
timeout_seconds: 120

View File

@@ -0,0 +1,7 @@
id: service-proposal
backend: service
model: replace-with-provider-model
temperature: 0
max_tokens: 2048
top_p: 1
timeout_seconds: 120

View File

@@ -0,0 +1,7 @@
id: service-validation-testing
backend: service
model: replace-with-testing-model
temperature: 0
max_tokens: 1024
top_p: 1
timeout_seconds: 120

View File

@@ -0,0 +1,7 @@
id: service-validation
backend: service
model: replace-with-provider-model
temperature: 0
max_tokens: 1024
top_p: 1
timeout_seconds: 120

15
go.mod
View File

@@ -1,11 +1,20 @@
module gitea.maximumdirect.net/eric/audita
go 1.24.0
require gopkg.in/yaml.v3 v3.0.1
go 1.25.5
require (
gitea.maximumdirect.net/eric/ConfigKit v0.4.0
gitea.maximumdirect.net/eric/promptkit v0.11.0
gopkg.in/yaml.v3 v3.0.1
)
require (
gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v2 v2.20260922.1 // indirect
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/v2 v2.20260922.0 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect
go.yaml.in/yaml/v4 v4.0.0-rc.6 // indirect
golang.org/x/text v0.14.0 // indirect
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
)

16
go.sum
View File

@@ -1,4 +1,14 @@
gitea.maximumdirect.net/eric/ConfigKit v0.4.0 h1:fo+vKff3zTGeWqJIA38NtT84b6OUP//9ZmBH3lFuKgk=
gitea.maximumdirect.net/eric/ConfigKit v0.4.0/go.mod h1:tYWpsrbLyu05r8+1MapWp5fN6ZrX9I8hU2xEPedYmYk=
gitea.maximumdirect.net/eric/promptkit v0.11.0 h1:x7JguOUiInPVXK0GEitjyoRFqqvcuSTjYUvqJ+whtik=
gitea.maximumdirect.net/eric/promptkit v0.11.0/go.mod h1:0zEBVlJJC+ldDdXazAgybYBXNSe74vwFx5dyOuSos/w=
gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v2 v2.20260922.1 h1:S2/Q/JDE+m8Ca3pDhAjVbfYrdlEhs9blj0STgNopHo0=
gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v2 v2.20260922.1/go.mod h1:a3VBbh9oaGXD6ruxhtwHj+AWW6DY/SSyby1QtMmYSUo=
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/v2 v2.20260922.0 h1:Y0EXc3eh12eKW2NrOviVqdk3oBo6UUjopt1YAa1B3Oc=
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/v2 v2.20260922.0/go.mod h1:mtgD1E0SGm0e/I8Ffhz+AYX7GN7RKUWJXehNFIKHs4Y=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
@@ -10,6 +20,12 @@ github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsK
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
go.yaml.in/yaml/v4 v4.0.0-rc.6 h1:1h7H1ohdUh93/FyE4YaDa1Zh64K6VVbjF4K6WUxMtH4=
go.yaml.in/yaml/v4 v4.0.0-rc.6/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=

View File

@@ -0,0 +1,222 @@
package cli
import (
"errors"
"flag"
"fmt"
"io"
"os"
"path/filepath"
"strconv"
"strings"
ck "gitea.maximumdirect.net/eric/ConfigKit"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
)
var systemConfigPaths = []string{"/usr/local/etc/audita/config.yml", "/etc/audita/config.yml"}
type configInput struct {
ConfigPath string
ConfigPathSet bool
Overlay string
View string
TranscriptPath string
GlossaryPath string
OutputPath string
Overrides []ck.Override
}
type selectedConfig struct {
Path string
Source string
Result *ck.Result[config.Config]
}
// parseConfigInput owns the common setting flags and process-only path flags.
// A transcript before flags and one after flags are both accepted.
func parseConfigInput(command string, args []string, output io.Writer) (configInput, bool, error) {
var input configInput
fs := flag.NewFlagSet(command, flag.ContinueOnError)
fs.SetOutput(output)
configPath := fs.String("config", "", "Main configuration YAML file")
overlay := fs.String("overlay", "", "Selected configuration overlay")
if command == "print-effective" {
fs.String("view", "settings", "Effective view: settings, profiles, or explain")
}
settingPaths := map[string]ck.Path{
"proposal-profile": ck.Keys("llm", "proposal_profile"),
"validation-profile": ck.Keys("llm", "validation_profile"),
"output-schema": ck.Keys("output", "schema"),
"validation-max-prompt-tokens": ck.Keys("validation", "max_prompt_tokens"),
"target-sections": ck.Keys("chunking", "target_sections"),
"max-section-tokens": ck.Keys("chunking", "max_section_tokens"),
"min-section-tokens": ck.Keys("chunking", "min_section_tokens"),
"normalize-max-segment-gap": ck.Keys("normalization", "max_segment_gap"),
"normalize-ellipsis-gap": ck.Keys("normalization", "ellipsis_gap"),
"normalize-max-segment-duration": ck.Keys("normalization", "max_segment_duration"),
"normalize-max-segment-tokens": ck.Keys("normalization", "max_segment_tokens"),
"glossary-confidence-threshold": ck.Keys("thresholds", "glossary"),
"homophones-confidence-threshold": ck.Keys("thresholds", "homophones"),
"spoken-word-confidence-threshold": ck.Keys("thresholds", "spoken_word"),
"grammar-confidence-threshold": ck.Keys("thresholds", "grammar"),
"transcript-description": ck.Keys("context", "description"),
"reporting": ck.Keys("reporting", "enabled"),
"report-dir": ck.Keys("reporting", "stdout_dir"),
"debug": ck.Keys("diagnostics", "debug"),
"work-dir": ck.Keys("diagnostics", "work_dir"),
"modules": ck.Keys("pipeline", "modules"),
}
for name := range settingPaths {
switch name {
case "reporting", "debug":
fs.Bool(name, false, "Setting override")
case "validation-max-prompt-tokens", "target-sections", "max-section-tokens", "min-section-tokens", "normalize-max-segment-tokens":
fs.Int(name, 0, "Setting override")
case "normalize-max-segment-gap", "normalize-ellipsis-gap", "normalize-max-segment-duration", "glossary-confidence-threshold", "homophones-confidence-threshold", "spoken-word-confidence-threshold", "grammar-confidence-threshold":
fs.Float64(name, 0, "Setting override")
default:
fs.String(name, "", "Setting override")
}
}
if command == "process" {
fs.String("glossary", "", "Glossary YAML file")
fs.String("output", "", "Corrected transcript output file")
}
parseArgs := args
front := ""
if command == "process" && len(args) > 0 && !strings.HasPrefix(args[0], "-") {
front, parseArgs = args[0], args[1:]
}
if err := fs.Parse(parseArgs); err != nil {
if errors.Is(err, flag.ErrHelp) {
return configInput{}, true, nil
}
return configInput{}, false, err
}
positions := fs.Args()
if front != "" {
positions = append([]string{front}, positions...)
}
if command == "process" {
if len(positions) != 1 {
return configInput{}, false, fmt.Errorf("expected exactly one transcript JSON path")
}
input.TranscriptPath = positions[0]
input.GlossaryPath = fs.Lookup("glossary").Value.String()
input.OutputPath = fs.Lookup("output").Value.String()
if strings.TrimSpace(input.GlossaryPath) == "" {
return configInput{}, false, fmt.Errorf("--glossary is required")
}
} else if len(positions) != 0 {
return configInput{}, false, fmt.Errorf("unexpected positional arguments")
}
input.ConfigPath, input.Overlay = *configPath, *overlay
if command == "print-effective" {
input.View = fs.Lookup("view").Value.String()
switch input.View {
case "settings", "profiles", "explain":
default:
return configInput{}, false, fmt.Errorf("unknown effective view %q", input.View)
}
}
fs.Visit(func(f *flag.Flag) {
if f.Name == "config" {
input.ConfigPathSet = true
}
path, ok := settingPaths[f.Name]
if !ok {
return
}
var value any = f.Value.String()
switch f.Name {
case "reporting", "debug":
value, _ = strconv.ParseBool(f.Value.String())
case "modules":
modules, err := config.ParseModulesCSV(f.Value.String())
if err != nil {
value = err
} else {
values := make([]any, len(modules))
for i, module := range modules {
values[i] = module
}
value = values
}
case "validation-max-prompt-tokens", "target-sections", "max-section-tokens", "min-section-tokens", "normalize-max-segment-tokens":
value, _ = strconv.Atoi(f.Value.String())
case "normalize-max-segment-gap", "normalize-ellipsis-gap", "normalize-max-segment-duration", "glossary-confidence-threshold", "homophones-confidence-threshold", "spoken-word-confidence-threshold", "grammar-confidence-threshold":
value, _ = strconv.ParseFloat(f.Value.String(), 64)
}
input.Overrides = append(input.Overrides, ck.Override{Path: path, Value: value, Flag: "--" + f.Name})
})
for _, override := range input.Overrides {
if err, ok := override.Value.(error); ok {
return configInput{}, false, fmt.Errorf("%s: %w", override.Flag, err)
}
}
return input, false, nil
}
func selectConfigPath(explicit string, set bool, stat func(string) (os.FileInfo, error), candidates []string) (string, string, error) {
if set {
if strings.TrimSpace(explicit) == "" {
return "", "", fmt.Errorf("--config requires a nonempty path")
}
if _, err := stat(explicit); err != nil {
return "", "", fmt.Errorf("config file %q: %w", explicit, err)
}
return explicit, "flag", nil
}
for _, path := range candidates {
if _, err := stat(path); err == nil {
return path, "default", nil
} else if !os.IsNotExist(err) {
return "", "", fmt.Errorf("config file %q: %w", path, err)
}
}
return "", "", fmt.Errorf("no configuration file found; select a config with backend and profile settings")
}
func loadConfigInput(input configInput, command string) (selectedConfig, error) {
if command == "validate" && (!input.ConfigPathSet || strings.TrimSpace(input.ConfigPath) == "") {
return selectedConfig{}, fmt.Errorf("--config is required")
}
path, source, err := selectConfigPath(input.ConfigPath, input.ConfigPathSet, os.Stat, systemConfigPaths)
if err != nil {
return selectedConfig{}, err
}
loaded, err := config.Load(ck.File(path), input.Overlay, input.Overrides...)
if err != nil {
return selectedConfig{}, err
}
return selectedConfig{Path: path, Source: source, Result: loaded}, nil
}
func configInputError(err error) string {
var loadErr *ck.LoadError
if !errors.As(err, &loadErr) {
return err.Error()
}
var messages []string
for _, diagnostic := range loadErr.Diagnostics() {
messages = append(messages, diagnostic.Message)
}
if len(messages) == 0 {
return err.Error()
}
return strings.Join(messages, "; ")
}
func selectedReportDir(cfg config.Config, outputPath string) (string, error) {
if !cfg.Reporting.Enabled {
return "", nil
}
if outputPath != "" {
return filepath.Dir(outputPath), nil
}
if strings.TrimSpace(cfg.Reporting.StdoutDir) == "" {
return "", fmt.Errorf("report directory is required when reporting is enabled and transcript output is stdout")
}
return cfg.Reporting.StdoutDir, nil
}

View File

@@ -0,0 +1,250 @@
package cli
import (
"errors"
"io"
"os"
"path/filepath"
"strings"
"testing"
ck "gitea.maximumdirect.net/eric/ConfigKit"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
)
func TestSharedConfigInput(t *testing.T) {
for _, command := range []string{"process", "validate", "print-effective"} {
t.Run(command, func(t *testing.T) {
args := []string{"--config", "main.yml", "--overlay", "testing", "--proposal-profile", "economy", "--modules", "", "--target-sections", "0"}
if command == "process" {
args = append([]string{"transcript.json", "--glossary", "glossary.yml"}, args...)
}
input, help, err := parseConfigInput(command, args, io.Discard)
if err != nil || help {
t.Fatalf("parse: %+v %v %v", input, help, err)
}
if !input.ConfigPathSet || input.ConfigPath != "main.yml" || input.Overlay != "testing" || len(input.Overrides) != 3 {
t.Fatalf("selection: %+v", input)
}
for _, override := range input.Overrides {
switch override.Path.String() {
case ck.Keys("pipeline", "modules").String():
if values, ok := override.Value.([]any); !ok || len(values) != 0 {
t.Fatalf("empty modules: %#v", override.Value)
}
case ck.Keys("chunking", "target_sections").String():
if override.Value != 0 {
t.Fatalf("zero target: %#v", override.Value)
}
}
}
})
}
}
func TestConfigInputRejectsRemovedAndProcessOnlyFlags(t *testing.T) {
for _, command := range []string{"process", "validate", "print-effective"} {
for _, flag := range []string{"--model", "--llm-api-key", "--max-retries", "--llm-concurrency", "--report-json", "--work-dir-retention"} {
args := []string{flag, "value"}
if command == "process" {
args = append([]string{"transcript.json", "--glossary", "glossary.yml"}, args...)
}
if _, _, err := parseConfigInput(command, args, io.Discard); err == nil {
t.Fatalf("%s accepted %s", command, flag)
}
}
}
for _, command := range []string{"validate", "print-effective"} {
if _, _, err := parseConfigInput(command, []string{"--glossary", "glossary.yml"}, io.Discard); err == nil {
t.Fatalf("%s accepted process path", command)
}
if _, _, err := parseConfigInput(command, []string{"transcript.json"}, io.Discard); err == nil {
t.Fatalf("%s accepted positional", command)
}
}
for _, command := range []string{"process", "validate", "print-effective"} {
_, help, err := parseConfigInput(command, []string{"--help"}, io.Discard)
if err != nil || !help {
t.Fatalf("%s help: %v %v", command, help, err)
}
}
}
func TestConfigSelection(t *testing.T) {
dir := t.TempDir()
first, second := filepath.Join(dir, "first.yml"), filepath.Join(dir, "second.yml")
if err := os.WriteFile(second, []byte("version: 2\n"), 0600); err != nil {
t.Fatal(err)
}
path, source, err := selectConfigPath("", false, os.Stat, []string{first, second})
if err != nil || path != second || source != "default" {
t.Fatalf("discovery: %q %q %v", path, source, err)
}
for _, tc := range []struct {
explicit string
set bool
candidates []string
}{
{"", true, []string{second}}, {first, true, []string{second}}, {"", false, []string{first}},
} {
if _, _, err := selectConfigPath(tc.explicit, tc.set, os.Stat, tc.candidates); err == nil {
t.Fatalf("accepted unavailable selection: %+v", tc)
}
}
statDenied := func(string) (os.FileInfo, error) { return nil, os.ErrPermission }
if _, _, err := selectConfigPath("", false, statDenied, []string{first}); !errors.Is(err, os.ErrPermission) {
t.Fatalf("inaccessible candidate: %v", err)
}
path, source, err = selectConfigPath(second, true, os.Stat, []string{first})
if err != nil || path != second || source != "flag" {
t.Fatalf("explicit path: %q %q %v", path, source, err)
}
}
func TestLegacyEnvironmentDoesNotSelectConfig(t *testing.T) {
t.Setenv("AUDITA_CONFIG", filepath.Join(t.TempDir(), "legacy.yml"))
_, _, err := selectConfigPath("", false, os.Stat, []string{filepath.Join(t.TempDir(), "missing.yml")})
if err == nil || !strings.Contains(err.Error(), "no configuration file") {
t.Fatalf("legacy environment changed discovery: %v", err)
}
}
func TestEffectiveViewParsing(t *testing.T) {
for _, view := range []string{"settings", "profiles", "explain"} {
input, _, err := parseConfigInput("print-effective", []string{"--view", view}, io.Discard)
if err != nil || input.View != view {
t.Fatalf("%q: %+v %v", view, input, err)
}
}
if _, _, err := parseConfigInput("print-effective", []string{"--view", "raw"}, io.Discard); err == nil {
t.Fatal("unknown view accepted")
}
if _, _, err := parseConfigInput("validate", []string{"--view", "profiles"}, io.Discard); err == nil {
t.Fatal("validate accepted inspection-only flag")
}
}
func TestSharedLoadingUsesSelectedFileOverlayAndVisitedOverrides(t *testing.T) {
dir := t.TempDir()
if err := os.Mkdir(filepath.Join(dir, "overlays"), 0700); err != nil {
t.Fatal(err)
}
main := filepath.Join(dir, "config.yml")
base := "version: 2\nbackends:\n local:\n endpoint: https://local.invalid/v1\nllm:\n proposal_profile: base\nchunking:\n max_section_tokens: 900\n"
if err := os.WriteFile(main, []byte(base), 0600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "overlays", "testing.yml"), []byte("llm:\n proposal_profile: testing\n"), 0600); err != nil {
t.Fatal(err)
}
for _, command := range []string{"process", "validate", "print-effective"} {
args := []string{"--config", main, "--overlay", "testing", "--min-section-tokens", "500"}
if command == "process" {
args = append([]string{"input.json", "--glossary", "glossary.yml"}, args...)
}
input, _, err := parseConfigInput(command, args, io.Discard)
if err != nil {
t.Fatal(err)
}
selected, err := loadConfigInput(input, command)
if err != nil {
t.Fatal(err)
}
c := selected.Result.Config
if c.LLM.ProposalProfile != "testing" || c.LLM.ValidationProfile != "testing" || c.Chunking.MinSectionTokens != 500 || c.Chunking.MaxSectionTokens != 900 {
t.Fatalf("%s resolved %+v", command, c)
}
}
input, _, err := parseConfigInput("validate", []string{"--overlay", "testing"}, io.Discard)
if err != nil {
t.Fatal(err)
}
if _, err := loadConfigInput(input, "validate"); err == nil {
t.Fatal("validate accepted discovery")
}
input, _, err = parseConfigInput("validate", []string{"--config", main}, io.Discard)
if err != nil {
t.Fatal(err)
}
if _, err := loadConfigInput(input, "validate"); err == nil {
t.Fatal("unmodified invalid lower-precedence section relation accepted")
}
}
func TestReportingAndDebugOverridesRemainIndependent(t *testing.T) {
dir := t.TempDir()
if err := os.Mkdir(filepath.Join(dir, "overlays"), 0700); err != nil {
t.Fatal(err)
}
main := filepath.Join(dir, "config.yml")
base := "version: 2\nbackends:\n local:\n endpoint: https://local.invalid/v1\nllm:\n proposal_profile: base\n"
if err := os.WriteFile(main, []byte(base), 0600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "overlays", "testing.yml"), []byte("reporting:\n enabled: true\n stdout_dir: reports\ndiagnostics:\n debug: true\n"), 0600); err != nil {
t.Fatal(err)
}
defaultInput, _, err := parseConfigInput("validate", []string{"--config", main}, io.Discard)
if err != nil {
t.Fatal(err)
}
defaults, err := loadConfigInput(defaultInput, "validate")
if err != nil {
t.Fatal(err)
}
if !defaults.Result.Config.Reporting.Enabled || defaults.Result.Config.Reporting.StdoutDir != "." || defaults.Result.Config.Diagnostics.Debug {
t.Fatalf("unexpected defaults: %+v", defaults.Result.Config)
}
for _, command := range []string{"process", "validate", "print-effective"} {
args := []string{"--config", main, "--overlay", "testing", "--reporting=false", "--debug=false"}
if command == "process" {
args = append([]string{"input.json", "--glossary", "glossary.yml"}, args...)
}
input, _, err := parseConfigInput(command, args, io.Discard)
if err != nil {
t.Fatal(err)
}
if len(input.Overrides) != 2 || input.Overrides[0].Value != false || input.Overrides[1].Value != false {
t.Fatalf("boolean overrides must remain typed: %+v", input.Overrides)
}
selected, err := loadConfigInput(input, command)
if err != nil {
t.Fatal(err)
}
if selected.Result.Config.Reporting.Enabled || selected.Result.Config.Diagnostics.Debug || selected.Result.Config.Reporting.StdoutDir != "reports" {
t.Fatalf("%s did not honor explicit false: %+v", command, selected.Result.Config)
}
}
}
func TestReportDirectorySelectionOnlyUsesActiveDestination(t *testing.T) {
cfg := config.Config{Reporting: config.Reporting{Enabled: true, StdoutDir: "reports"}}
if got, err := selectedReportDir(cfg, ""); err != nil || got != "reports" {
t.Fatalf("stdout destination: %q %v", got, err)
}
cfg.Reporting.StdoutDir = ""
if got, err := selectedReportDir(cfg, filepath.Join("out", "corrected.json")); err != nil || got != "out" {
t.Fatalf("file destination ignored stdout setting: %q %v", got, err)
}
if _, err := selectedReportDir(cfg, ""); err == nil {
t.Fatal("enabled stdout reporting accepted blank destination")
}
cfg.Reporting.Enabled = false
if got, err := selectedReportDir(cfg, ""); err != nil || got != "" {
t.Fatalf("disabled reporting probed destination: %q %v", got, err)
}
}
func TestBareBooleanFlagsAndReportDirectoryOverride(t *testing.T) {
input, _, err := parseConfigInput("process", []string{"input.json", "--glossary", "glossary.yml", "--reporting", "--debug", "--report-dir", "reports"}, io.Discard)
if err != nil {
t.Fatal(err)
}
values := map[string]any{}
for _, override := range input.Overrides {
values[override.Path.String()] = override.Value
}
if values[ck.Keys("reporting", "enabled").String()] != true || values[ck.Keys("diagnostics", "debug").String()] != true || values[ck.Keys("reporting", "stdout_dir").String()] != "reports" {
t.Fatalf("incorrect visited overrides: %+v", values)
}
}

View File

@@ -1,36 +0,0 @@
package cli
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestMain(m *testing.M) {
// Tests supply their own overrides; inherited settings must not affect them.
for _, entry := range os.Environ() {
key, _, _ := strings.Cut(entry, "=")
if strings.HasPrefix(key, "AUDITA_") || key == "OPENROUTER_API_KEY" {
if err := os.Unsetenv(key); err != nil {
panic(err)
}
}
}
dir, err := os.MkdirTemp("", "audita-cli-test-config-")
if err != nil {
panic(err)
}
defer os.RemoveAll(dir)
configPath := filepath.Join(dir, "config.yml")
if err := os.WriteFile(configPath, []byte("version: 1\n"), 0600); err != nil {
panic(err)
}
// An explicit environment path prevents discovery of installed system config.
// Individual tests can still override it with t.Setenv or --config.
if err := os.Setenv("AUDITA_CONFIG", configPath); err != nil {
panic(err)
}
m.Run()
}

View File

@@ -1,338 +0,0 @@
package cli
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/reporting"
)
type parityFixtureCase struct {
Name string `json:"name"`
TranscriptFile string `json:"transcript_file"`
GlossaryFile string `json:"glossary_file"`
ModulesCSV string `json:"modules_csv,omitempty"`
ProposalResponsesFile string `json:"proposal_responses_file,omitempty"`
ValidationResponsesFile string `json:"validation_responses_file,omitempty"`
Env map[string]string `json:"env,omitempty"`
Expect parityExpectation `json:"expect"`
}
type parityExpectation struct {
ExitCode int `json:"exit_code"`
Status string `json:"status,omitempty"`
ErrorPhase string `json:"error_phase,omitempty"`
StdoutMode string `json:"stdout_mode,omitempty"` // empty|json
StderrContains string `json:"stderr_contains,omitempty"`
OutputTranscriptFile string `json:"output_transcript_file,omitempty"`
ModuleInstances []string `json:"module_instances,omitempty"`
ModuleCount int `json:"module_count,omitempty"`
TotalAppliedChanges int `json:"total_applied_changes,omitempty"`
TotalSkippedChanges int `json:"total_skipped_changes,omitempty"`
FailedModuleInstance string `json:"failed_module_instance,omitempty"`
ValidatorRejectedReasonCodes []string `json:"validator_rejected_reason_codes,omitempty"`
ApplicationSkipReasonCodes []string `json:"application_skip_reason_codes,omitempty"`
RequireErrorLog bool `json:"require_error_log,omitempty"`
SecretMarkers []string `json:"secret_markers,omitempty"`
ExpectedProposalCalls []string `json:"expected_proposal_calls,omitempty"`
ExpectedValidationCalls []string `json:"expected_validation_calls,omitempty"`
ModuleAppliedCounts []int `json:"module_applied_counts,omitempty"`
ModuleRejectedCounts []int `json:"module_rejected_counts,omitempty"`
ModuleSkipCounts []int `json:"module_skip_counts,omitempty"`
MinResponsePayloadArtifacts int `json:"min_response_payload_artifacts,omitempty"`
}
func TestParityFixtures(t *testing.T) {
casePaths, err := filepath.Glob(parityFixturePath("*.case.json"))
if err != nil {
t.Fatalf("glob parity fixtures: %v", err)
}
if len(casePaths) == 0 {
t.Fatal("expected at least one parity fixture case")
}
for _, casePath := range casePaths {
fx := loadParityFixtureCase(t, casePath)
t.Run(fx.Name, func(t *testing.T) {
runParityFixtureCase(t, filepath.Dir(casePath), fx)
})
}
}
func loadParityFixtureCase(t *testing.T, casePath string) parityFixtureCase {
t.Helper()
var fx parityFixtureCase
raw := readFile(t, casePath)
if err := json.Unmarshal(raw, &fx); err != nil {
t.Fatalf("parse parity case %q: %v", casePath, err)
}
if strings.TrimSpace(fx.Name) == "" {
t.Fatalf("parity case %q missing name", casePath)
}
return fx
}
func runParityFixtureCase(t *testing.T, caseDir string, fx parityFixtureCase) {
t.Helper()
for k, v := range fx.Env {
t.Setenv(k, v)
}
proposalClient := &fakeStructuredLLMClient{}
validationClient := &fakeStructuredLLMClient{}
if strings.TrimSpace(fx.ProposalResponsesFile) != "" {
raw := readFile(t, filepath.Join(caseDir, fx.ProposalResponsesFile))
if err := json.Unmarshal(raw, &proposalClient.proposalResponses); err != nil {
t.Fatalf("parse proposal responses: %v", err)
}
processProposalLLMClient = proposalClient
}
if strings.TrimSpace(fx.ValidationResponsesFile) != "" {
raw := readFile(t, filepath.Join(caseDir, fx.ValidationResponsesFile))
if err := json.Unmarshal(raw, &validationClient.validationResponses); err != nil {
t.Fatalf("parse validation responses: %v", err)
}
processValidationLLMClient = validationClient
}
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
var stdout, stderr bytes.Buffer
workDir := t.TempDir()
outputPath := filepath.Join(t.TempDir(), "out.json")
reportPath := filepath.Join(t.TempDir(), "report.json")
args := []string{
"process",
filepath.Join(caseDir, fx.TranscriptFile),
"--glossary",
filepath.Join(caseDir, fx.GlossaryFile),
"--report-json",
reportPath,
"--work-dir",
workDir,
"--work-dir-retention",
"always",
}
// Keep stdout shape deterministic for parity tests.
if fx.Expect.StdoutMode != "json" {
args = append(args, "--output", outputPath)
}
if strings.TrimSpace(fx.ModulesCSV) != "" {
args = append(args, "--modules", fx.ModulesCSV)
}
exitCode := Run(args, &stdout, &stderr)
if exitCode != fx.Expect.ExitCode {
t.Fatalf("expected exit code %d, got %d stderr=%q", fx.Expect.ExitCode, exitCode, stderr.String())
}
switch fx.Expect.StdoutMode {
case "json":
if _, err := json.Marshal(stdout.String()); err != nil {
t.Fatalf("unexpected stdout marshal error: %v", err)
}
if !json.Valid(stdout.Bytes()) {
t.Fatalf("expected JSON stdout, got %q", stdout.String())
}
default:
if stdout.Len() != 0 {
t.Fatalf("expected empty stdout, got %q", stdout.String())
}
}
if fx.Expect.StderrContains != "" && !strings.Contains(stderr.String(), fx.Expect.StderrContains) {
t.Fatalf("expected stderr to contain %q, got %q", fx.Expect.StderrContains, stderr.String())
}
report := readProcessReport(t, reportPath)
assertParityReport(t, report, fx.Expect)
runDir := onlyRunDir(t, workDir)
runDirReport := readProcessReport(t, filepath.Join(runDir, "report.json"))
assertParityReport(t, runDirReport, fx.Expect)
if fx.Expect.RequireErrorLog {
if _, err := os.Stat(filepath.Join(runDir, "error.log")); err != nil {
t.Fatalf("expected error.log: %v", err)
}
}
if strings.TrimSpace(fx.Expect.OutputTranscriptFile) != "" && fx.Expect.ExitCode == 0 {
got := readFile(t, outputPath)
want := readFile(t, filepath.Join(caseDir, fx.Expect.OutputTranscriptFile))
assertJSONSemanticEqual(t, want, got)
}
if len(fx.Expect.ExpectedProposalCalls) > 0 && !reflect.DeepEqual(proposalClient.calls, fx.Expect.ExpectedProposalCalls) {
t.Fatalf("unexpected proposal calls: got %v want %v", proposalClient.calls, fx.Expect.ExpectedProposalCalls)
}
if len(fx.Expect.ExpectedValidationCalls) > 0 && !reflect.DeepEqual(validationClient.calls, fx.Expect.ExpectedValidationCalls) {
t.Fatalf("unexpected validation calls: got %v want %v", validationClient.calls, fx.Expect.ExpectedValidationCalls)
}
if len(fx.Expect.SecretMarkers) > 0 {
assertNoSecretMarkers(t, reportPath, fx.Expect.SecretMarkers)
assertNoSecretMarkersInTree(t, runDir, fx.Expect.SecretMarkers)
}
if fx.Expect.MinResponsePayloadArtifacts > 0 {
matches, err := filepath.Glob(filepath.Join(runDir, "*", "*response-payload.json"))
if err != nil {
t.Fatalf("glob response payload artifacts: %v", err)
}
if len(matches) < fx.Expect.MinResponsePayloadArtifacts {
t.Fatalf("expected at least %d response payload artifacts, got %d", fx.Expect.MinResponsePayloadArtifacts, len(matches))
}
}
}
func assertParityReport(t *testing.T, report reporting.ProcessReport, exp parityExpectation) {
t.Helper()
if exp.Status != "" && report.Status != exp.Status {
t.Fatalf("expected report status %q, got %q", exp.Status, report.Status)
}
if exp.ErrorPhase != "" && report.ErrorPhase != exp.ErrorPhase {
t.Fatalf("expected report error_phase %q, got %q", exp.ErrorPhase, report.ErrorPhase)
}
if len(exp.ModuleInstances) > 0 {
got := make([]string, 0, len(report.ModuleResults))
for _, mr := range report.ModuleResults {
got = append(got, mr.ModuleInstance)
}
if !reflect.DeepEqual(got, exp.ModuleInstances) {
t.Fatalf("unexpected module instances: got %v want %v", got, exp.ModuleInstances)
}
}
if exp.ModuleCount > 0 {
if report.ModulesSummary == nil || report.ModulesSummary.ModuleCount != exp.ModuleCount {
t.Fatalf("expected module_count=%d, got %+v", exp.ModuleCount, report.ModulesSummary)
}
}
if exp.TotalAppliedChanges > 0 {
if report.ModulesSummary == nil || report.ModulesSummary.TotalAppliedChanges != exp.TotalAppliedChanges {
t.Fatalf("expected total_applied_changes=%d, got %+v", exp.TotalAppliedChanges, report.ModulesSummary)
}
}
if exp.TotalSkippedChanges > 0 {
if report.ModulesSummary == nil || report.ModulesSummary.TotalSkippedChanges != exp.TotalSkippedChanges {
t.Fatalf("expected total_skipped_changes=%d, got %+v", exp.TotalSkippedChanges, report.ModulesSummary)
}
}
if exp.FailedModuleInstance != "" {
if report.ModulesSummary == nil || report.ModulesSummary.FailedModuleInstance != exp.FailedModuleInstance {
t.Fatalf("expected failed_module_instance=%q, got %+v", exp.FailedModuleInstance, report.ModulesSummary)
}
}
if len(exp.ValidatorRejectedReasonCodes) > 0 {
got := collectValidatorRejectedReasonCodes(report.ModuleResults)
if !reflect.DeepEqual(got, exp.ValidatorRejectedReasonCodes) {
t.Fatalf("unexpected validator rejected reason codes: got %v want %v", got, exp.ValidatorRejectedReasonCodes)
}
}
if len(exp.ApplicationSkipReasonCodes) > 0 {
got := collectApplicationSkipReasonCodes(report.ModuleResults)
if !reflect.DeepEqual(got, exp.ApplicationSkipReasonCodes) {
t.Fatalf("unexpected application skip reason codes: got %v want %v", got, exp.ApplicationSkipReasonCodes)
}
}
if len(exp.ModuleAppliedCounts) > 0 {
got := make([]int, 0, len(report.ModuleResults))
for _, mr := range report.ModuleResults {
got = append(got, len(mr.AppliedChanges))
}
if !reflect.DeepEqual(got, exp.ModuleAppliedCounts) {
t.Fatalf("unexpected per-module applied counts: got %v want %v", got, exp.ModuleAppliedCounts)
}
}
if len(exp.ModuleRejectedCounts) > 0 {
got := make([]int, 0, len(report.ModuleResults))
for _, mr := range report.ModuleResults {
got = append(got, len(mr.ValidatorRejected))
}
if !reflect.DeepEqual(got, exp.ModuleRejectedCounts) {
t.Fatalf("unexpected per-module rejected counts: got %v want %v", got, exp.ModuleRejectedCounts)
}
}
if len(exp.ModuleSkipCounts) > 0 {
got := make([]int, 0, len(report.ModuleResults))
for _, mr := range report.ModuleResults {
got = append(got, len(mr.SkippedChanges))
}
if !reflect.DeepEqual(got, exp.ModuleSkipCounts) {
t.Fatalf("unexpected per-module skip counts: got %v want %v", got, exp.ModuleSkipCounts)
}
}
}
func collectValidatorRejectedReasonCodes(results []reporting.ModuleReport) []string {
out := make([]string, 0)
for _, mr := range results {
for _, vr := range mr.ValidatorRejected {
out = append(out, vr.ReasonCode)
}
}
return out
}
func collectApplicationSkipReasonCodes(results []reporting.ModuleReport) []string {
out := make([]string, 0)
for _, mr := range results {
for _, sk := range mr.SkippedChanges {
out = append(out, string(sk.SkipReason))
}
}
return out
}
func assertJSONSemanticEqual(t *testing.T, expected []byte, actual []byte) {
t.Helper()
var exp any
var act any
if err := json.Unmarshal(expected, &exp); err != nil {
t.Fatalf("unmarshal expected json: %v", err)
}
if err := json.Unmarshal(actual, &act); err != nil {
t.Fatalf("unmarshal actual json: %v", err)
}
if !reflect.DeepEqual(exp, act) {
t.Fatalf("JSON mismatch\nexpected=%s\nactual=%s", string(expected), string(actual))
}
}
func assertNoSecretMarkers(t *testing.T, filePath string, markers []string) {
t.Helper()
raw := string(readFile(t, filePath))
for _, marker := range markers {
if marker != "" && strings.Contains(raw, marker) {
t.Fatalf("secret marker %q leaked in %s", marker, filePath)
}
}
}
func assertNoSecretMarkersInTree(t *testing.T, root string, markers []string) {
t.Helper()
_ = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil || d == nil || d.IsDir() {
return nil
}
raw := string(readFile(t, path))
for _, marker := range markers {
if marker != "" && strings.Contains(raw, marker) {
t.Fatalf("secret marker %q leaked in %s", marker, path)
}
}
return nil
})
}
func parityFixturePath(name string) string {
return filepath.Join("testdata", "parity", name)
}

442
internal/cli/process.go Normal file
View File

@@ -0,0 +1,442 @@
package cli
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"io"
"strings"
"sync"
"time"
"gitea.maximumdirect.net/eric/audita/internal/core/chunking"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/diagnostics"
coreio "gitea.maximumdirect.net/eric/audita/internal/core/io"
"gitea.maximumdirect.net/eric/audita/internal/core/normalization"
"gitea.maximumdirect.net/eric/audita/internal/core/outputschema"
"gitea.maximumdirect.net/eric/audita/internal/core/redaction"
"gitea.maximumdirect.net/eric/audita/internal/core/reporting"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/llm"
"gitea.maximumdirect.net/eric/audita/internal/framework/modules"
"gitea.maximumdirect.net/eric/audita/internal/framework/outcomes"
"gitea.maximumdirect.net/eric/audita/internal/framework/processreport"
"gitea.maximumdirect.net/eric/audita/internal/framework/runner"
"gitea.maximumdirect.net/eric/audita/internal/framework/structuredoutput"
)
func runProcess(ctx context.Context, args []string, stdout, stderr io.Writer) int {
input, help, err := parseConfigInput("process", args, stderr)
if err != nil {
fmt.Fprintf(stderr, "audita process: %s\n", configInputError(err))
return 2
}
if help {
writeProcessUsage(stdout)
return 0
}
selected, err := loadConfigInput(input, "process")
if err != nil {
fmt.Fprintf(stderr, "audita process: %s\n", configInputError(err))
return 2
}
cfg := selected.Result.Config
reportDir, err := selectedReportDir(cfg, input.OutputPath)
if err != nil {
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
return 2
}
if cfg.Diagnostics.Debug && strings.TrimSpace(cfg.Diagnostics.WorkDir) == "" {
fmt.Fprintln(stderr, "audita process: debug work directory is required")
return 2
}
if err := ctx.Err(); err != nil {
fmt.Fprintf(stderr, "audita process: %s\n", err)
return 1
}
runtime, err := llm.NewRuntime(ctx, cfg, selected.Path, nil)
if err != nil {
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
if ctx.Err() != nil || llm.IsAssetError(err) {
return 1
}
return 2
}
specs, err := contracts.ResolveModuleRunSpecs(cfg.Pipeline.Modules)
if err != nil {
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
return 2
}
runID, err := generateRunID()
if err != nil {
fmt.Fprintf(stderr, "audita process: run ID: %s\n", llm.RedactError(err, cfg))
return 1
}
started := time.Now().UTC()
state := &processState{cfg: cfg, runtime: runtime, specs: specs, runID: runID, started: started,
transcriptPath: input.TranscriptPath, glossaryPath: input.GlossaryPath, outputPath: input.OutputPath,
selected: selected, overlay: input.Overlay, stderr: stderr, stdout: stdout, ctx: ctx}
protected := []string{input.GlossaryPath}
for _, source := range selected.Result.Sources() {
if source.Path != "" {
protected = append(protected, source.Path)
}
}
protected = append(protected, runtime.ProfilePaths...)
session, err := diagnostics.PrepareOutput(diagnostics.OutputSelection{InputTranscriptPath: input.TranscriptPath, TranscriptPath: input.OutputPath,
ReportDir: reportDir, ReportingEnabled: cfg.Reporting.Enabled, ProtectedPaths: protected})
if err != nil {
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
return 2
}
state.session = session
initial := state.snapshot(reporting.StatusInProgress, nil, nil)
initialBundle, err := state.bundle(initial)
if err != nil {
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
return 1
}
initFailure := []processreport.RunFailure{{Phase: "initialization", Reason: outcomes.ReasonInternalFailure}}
failedInit, err := state.bundle(state.snapshot(reporting.StatusFailed, nil, initFailure))
if err != nil {
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
return 1
}
if err := session.Initialize(initialBundle, failedInit); err != nil {
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
return 1
}
state.startDebug()
return state.execute()
}
type processState struct {
cfg config.Config
runtime *llm.Runtime
specs []contracts.ModuleRunSpec
runID string
started time.Time
transcriptPath string
glossaryPath string
outputPath string
selected selectedConfig
overlay string
ctx context.Context
stdout io.Writer
stderr io.Writer
session *diagnostics.OutputSession
debugDir *diagnostics.RunDirectory
debugWarnings []string
debugMu sync.Mutex
runOutput *runner.RunOutput
}
func generateRunID() (string, error) {
var random [12]byte
if _, err := rand.Read(random[:]); err != nil {
return "", err
}
return "run-" + hex.EncodeToString(random[:]), nil
}
func (s *processState) snapshot(status string, completed *time.Time, failures []processreport.RunFailure) reporting.Snapshot {
return processreport.Build(processreport.BuildInput{RunID: s.runID, Status: status, StartedAt: s.started,
CompletedAt: completed, TranscriptPath: s.transcriptPath, GlossaryPath: s.glossaryPath,
OutputPath: s.outputPath, ModuleSpecs: s.specs, RunOutput: s.runOutput, RunFailures: failures})
}
func (s *processState) bundle(snapshot reporting.Snapshot) (diagnostics.ReportBundle, error) {
if !s.cfg.Reporting.Enabled {
return diagnostics.ReportBundle{}, nil
}
secrets := llm.ConfiguredSecrets(&s.cfg)
errorsYAML, err := reporting.YAML(snapshot.ErrorsDocument(), secrets...)
if err != nil {
return diagnostics.ReportBundle{}, err
}
recoveriesYAML, err := reporting.YAML(snapshot.RecoveriesDocument(), secrets...)
if err != nil {
return diagnostics.ReportBundle{}, err
}
return diagnostics.ReportBundle{Errors: errorsYAML, Recoveries: recoveriesYAML,
Markdown: reporting.Markdown(snapshot, secrets...)}, nil
}
func (s *processState) debugWrite(label string, write func(*diagnostics.RunDirectory) error) {
if s.debugDir == nil {
return
}
if err := write(s.debugDir); err != nil {
s.reportDiagnosticError(fmt.Errorf("%s: %w", label, err))
}
}
func (s *processState) startDebug() {
if !s.cfg.Diagnostics.Debug {
return
}
dir, err := diagnostics.NewRunDirectory(s.cfg.Diagnostics.WorkDir, s.runID, llm.ConfiguredSecrets(&s.cfg)...)
if err != nil {
s.reportDiagnosticError(fmt.Errorf("directory: %w", err))
return
}
s.debugDir = dir
s.debugWrite("invocation", func(d *diagnostics.RunDirectory) error {
return d.WriteInvocationMetadata(diagnostics.InvocationMetadata{
Operation: "process", TranscriptPath: s.transcriptPath, GlossaryPath: s.glossaryPath,
OutputPath: s.outputPath, ConfigPath: s.selected.Path, ConfigSource: s.selected.Source,
Overlay: s.overlay, ConfigVersion: &s.cfg.Version, TranscriptDescription: s.cfg.Context.Description,
Modules: append([]string(nil), s.cfg.Pipeline.Modules...), RunID: s.runID, StartedAt: s.started,
})
})
var effective bytes.Buffer
if err := s.selected.Result.WriteEffective(&effective); err == nil {
s.debugWrite("effective settings", func(d *diagnostics.RunDirectory) error { return d.WriteEffectiveConfig(effective.Bytes()) })
} else {
s.reportDiagnosticError(fmt.Errorf("effective settings: %w", err))
}
if origins, err := llm.ExplanationYAML(s.selected.Result); err == nil {
s.debugWrite("configuration origins", func(d *diagnostics.RunDirectory) error {
return d.WriteYAMLArtifact(diagnostics.ArtifactConfigOrigins, origins)
})
} else {
s.reportDiagnosticError(fmt.Errorf("configuration origins: %w", err))
}
if profiles, err := s.runtime.SelectedProfilesYAML(); err == nil {
s.debugWrite("selected profiles", func(d *diagnostics.RunDirectory) error {
return d.WriteYAMLArtifact(diagnostics.ArtifactSelectedProfiles, profiles)
})
} else {
s.reportDiagnosticError(fmt.Errorf("selected profiles: %w", err))
}
}
func (s *processState) debugPath() string {
if s.debugDir == nil {
return ""
}
return s.debugDir.Path()
}
func (s *processState) execute() int {
transcriptBytes, err := coreio.ReadRequiredFile(s.transcriptPath, "transcript")
if err != nil {
return s.fail("input", outcomes.ReasonInternalFailure, err)
}
glossaryBytes, err := coreio.ReadRequiredFile(s.glossaryPath, "glossary")
if err != nil {
return s.fail("input", outcomes.ReasonInternalFailure, err)
}
source, err := schema.ParseSourceTranscriptJSON(transcriptBytes)
if err != nil {
return s.fail("input", outcomes.ReasonInputInvalid, err)
}
glossary, err := schema.ParseGlossaryYAML(glossaryBytes)
if err != nil {
return s.fail("input", outcomes.ReasonInputInvalid, err)
}
s.debugWrite("source transcript", func(d *diagnostics.RunDirectory) error { return d.WriteSourceTranscript(source, transcriptBytes) })
normalizer := normalization.NewNormalizer(normalization.NormalizationConfig{
MaxSegmentGap: s.cfg.Normalization.MaxSegmentGap, EllipsisGap: s.cfg.Normalization.EllipsisGap,
MaxSegmentDuration: s.cfg.Normalization.MaxSegmentDuration, MaxSegmentTokens: s.cfg.Normalization.MaxSegmentTokens,
})
normalized, normSummary := normalizer.Normalize(sourceToCanonicalTranscript(source))
s.debugWrite("normalized transcript", func(d *diagnostics.RunDirectory) error { return d.WriteNormalizedTranscript(normalized) })
s.debugWrite("normalization summary", func(d *diagnostics.RunDirectory) error { return d.WriteNormalizationSummary(normSummary) })
if err := s.ctx.Err(); err != nil {
return s.fail("processing", outcomes.ReasonCanceled, err)
}
observations := llm.NewObservationCollector()
proposalClient, err := s.runtime.NewRoleClient("proposal", observations.Hooks())
if err != nil {
return s.fail("processing", outcomes.ReasonSetupFailure, err)
}
validationClient, err := s.runtime.NewRoleClient("validation", observations.Hooks())
if err != nil {
return s.fail("processing", outcomes.ReasonSetupFailure, err)
}
factory := modules.NewFactory(modules.Dependencies{Config: &s.cfg, Glossary: glossary, DiagnosticsDir: s.debugPath()})
result, runErr := runner.New(factory).Run(s.ctx, runner.RunInput{
Config: &s.cfg, Transcript: normalized, Glossary: glossary, ModuleSpecs: s.specs,
ProposalLLMClient: proposalClient, ProposalDiagnosticsDir: s.debugPath(),
ValidationLLMClient: validationClient, ValidationDiagnosticsDir: s.debugPath(),
CaptureCandidates: s.debugDir != nil,
ReportDiagnosticError: s.reportDiagnosticError,
})
result.Observations = observations.Snapshot()
result.ObservationsAvailable = true
result.LLMSummary = llm.Summarize(result.Observations)
s.runOutput = &result
s.writeExecutionEvidence()
if len(result.Sections) > 0 || runErr == nil {
s.writeChunkSummary(result.Sections)
}
if runErr != nil {
return s.fail("processing", fatalReason(s.ctx, runErr), runErr)
}
encoder, err := outputschema.Resolve(s.cfg.Output.Schema)
if err != nil {
return s.fail("encoding", outcomes.ReasonSetupFailure, err)
}
outputBytes, err := encoder.Encoder(result.FinalTranscript)
if err != nil {
return s.fail("encoding", outcomes.ReasonInternalFailure, err)
}
return s.finish(outputBytes)
}
func (s *processState) writeChunkSummary(inventory []contracts.SectionMetadata) {
if s.debugDir == nil {
return
}
sections := make([]chunking.Section, len(inventory))
for i, metadata := range inventory {
sections[i] = chunking.Section{Index: metadata.Index, StartSegmentID: metadata.StartSegmentID,
EndSegmentID: metadata.EndSegmentID, EstimatedTokens: metadata.EstimatedTokens,
Segments: make([]schema.Segment, metadata.SegmentCount)}
}
config := chunking.ChunkingConfig{TargetSections: s.cfg.Chunking.TargetSections,
MaxSectionTokens: s.cfg.Chunking.MaxSectionTokens, MinSectionTokens: s.cfg.Chunking.MinSectionTokens}
summary := chunking.ComputeDetailedSummary(sections, config)
s.debugWrite("chunking summary", func(d *diagnostics.RunDirectory) error { return d.WriteChunkingSummary(&summary) })
}
func (s *processState) writeExecutionEvidence() {
if s.debugDir == nil || s.runOutput == nil {
return
}
output := s.runOutput
s.debugWrite("model execution", func(d *diagnostics.RunDirectory) error {
return d.WriteJSONArtifact(diagnostics.ArtifactLLMExecution, processreport.BuildLLMExecution(output.Observations))
})
s.debugWrite("utilization", func(d *diagnostics.RunDirectory) error {
return d.WriteJSONArtifact(diagnostics.ArtifactUtilizationSummary, processreport.BuildUtilization(output, s.cfg.Backends))
})
s.debugWrite("correction ledger", func(d *diagnostics.RunDirectory) error {
return d.WriteJSONArtifact(diagnostics.ArtifactCorrectionLedger, processreport.BuildCorrectionLedger(processreport.CorrectionLedgerInput{
RunDirectoryPath: d.Path(), RunOutput: output}))
})
}
func fatalReason(ctx context.Context, err error) outcomes.ReasonCode {
if ctx.Err() != nil || errors.Is(err, context.Canceled) {
return outcomes.ReasonCanceled
}
var failure *structuredoutput.Failure
if errors.As(err, &failure) && failure.Reason != "" {
return failure.Reason
}
return outcomes.ReasonInternalFailure
}
func (s *processState) fail(phase string, reason outcomes.ReasonCode, cause error) int {
s.debugWrite("error log", func(d *diagnostics.RunDirectory) error {
return d.WriteErrorLog(phase + ": " + llm.RedactError(cause, s.cfg))
})
failures := []processreport.RunFailure{{Phase: phase, Reason: reason}}
completed := time.Now().UTC()
base, err := s.bundle(s.snapshot(reporting.StatusFailed, &completed, failures))
if err != nil {
fmt.Fprintf(s.stderr, "audita process: %s\n", llm.RedactError(err, s.cfg))
}
writeFailures := s.session.Finalize(nil, base, func(extra []diagnostics.FileFailure) diagnostics.ReportBundle {
all := append(append([]processreport.RunFailure(nil), failures...), mapWriteFailures(extra)...)
bundle, renderErr := s.bundle(s.snapshot(reporting.StatusFailed, &completed, all))
if renderErr != nil {
return base
}
return bundle
})
fmt.Fprintf(s.stderr, "audita process: %s\n", llm.RedactError(cause, s.cfg))
s.printFailures(writeFailures)
s.printDebugWarnings()
if s.debugDir != nil {
fmt.Fprintf(s.stderr, "audita process: diagnostics: %s\n", s.safePath(s.debugDir.Path()))
}
return 1
}
func (s *processState) finish(outputBytes []byte) int {
completed := time.Now().UTC()
snapshot := s.snapshot(reporting.StatusCompleted, &completed, nil)
complete, err := s.bundle(snapshot)
if err != nil {
return s.fail("report_write", outcomes.ReasonInternalFailure, err)
}
if s.session.TranscriptPath() == "" {
if _, err := s.stdout.Write(outputBytes); err != nil {
return s.fail("transcript_write", outcomes.ReasonInternalFailure, err)
}
outputBytes = nil
}
writeFailures := s.session.Finalize(outputBytes, complete, func(extra []diagnostics.FileFailure) diagnostics.ReportBundle {
failed, renderErr := s.bundle(s.snapshot(reporting.StatusFailed, &completed, mapWriteFailures(extra)))
if renderErr != nil {
return diagnostics.ReportBundle{}
}
return failed
})
if len(writeFailures) > 0 {
s.printFailures(writeFailures)
s.printDebugWarnings()
if s.debugDir != nil {
fmt.Fprintf(s.stderr, "audita process: diagnostics: %s\n", s.safePath(s.debugDir.Path()))
}
return 1
}
fmt.Fprintln(s.stderr, reporting.CompletionLine(snapshot))
for _, path := range s.session.ReportPaths() {
fmt.Fprintf(s.stderr, "Report: %s\n", s.safePath(path))
}
if s.debugDir != nil {
fmt.Fprintf(s.stderr, "Debug: %s\n", s.safePath(s.debugDir.Path()))
}
s.printDebugWarnings()
return 0
}
func mapWriteFailures(failures []diagnostics.FileFailure) []processreport.RunFailure {
out := make([]processreport.RunFailure, len(failures))
for i, failure := range failures {
phase := "report_write"
if failure.Kind == "transcript write" {
phase = "transcript_write"
}
out[i] = processreport.RunFailure{Phase: phase, Reason: outcomes.ReasonInternalFailure, OutputPath: failure.Path}
}
return out
}
func (s *processState) printFailures(failures []diagnostics.FileFailure) {
for _, failure := range failures {
fmt.Fprintf(s.stderr, "audita process: %s\n", llm.RedactError(failure, s.cfg))
}
}
func (s *processState) printDebugWarnings() {
s.debugMu.Lock()
defer s.debugMu.Unlock()
for _, warning := range s.debugWarnings {
fmt.Fprintf(s.stderr, "audita process: %s\n", warning)
}
}
// safePath sanitizes presentation only; filesystem destinations remain literal.
func (s *processState) safePath(path string) string {
return redaction.New(llm.ConfiguredSecrets(&s.cfg)).String(path)
}
func (s *processState) reportDiagnosticError(err error) {
if err == nil {
return
}
message := "debug " + llm.RedactError(err, s.cfg)
s.debugMu.Lock()
defer s.debugMu.Unlock()
s.debugWarnings = append(s.debugWarnings, message)
}

View File

@@ -1,121 +0,0 @@
package cli
import (
"flag"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
)
type processOverrideBinding func(*config.CLIOverrides, processFlags)
var processOverrideBindings = map[string]processOverrideBinding{
"modules": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.ModulesCSV = flags.modules
},
"output-schema": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.OutputSchema = flags.outputSchema
},
"llm-api-key": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.PrimaryLLMAPIKey = flags.llmAPIKey
},
"validation-llm-api-key": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.ValidationLLMAPIKey = flags.validationLLMAPIKey
},
"model": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.PrimaryModel = flags.model
},
"validation-model": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.ValidationModel = flags.validationModel
},
"base-url": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.PrimaryBaseURL = flags.baseURL
},
"validation-base-url": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.ValidationBaseURL = flags.validationBaseURL
},
"llm-timeout-seconds": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.PrimaryLLMTimeoutSeconds = flags.llmTimeoutSeconds
},
"total-llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.TotalLLMConcurrency = flags.totalLLMConcurrency
},
"proposal-llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.ProposalLLMConcurrency = flags.proposalLLMConcurrency
},
"llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.PrimaryLLMConcurrency = flags.llmConcurrency
},
"validation-llm-timeout-seconds": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.ValidationLLMTimeoutSeconds = flags.validationLLMTimeoutSeconds
},
"max-retries": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.MaxRetries = flags.maxRetries
},
"validation-max-retries": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.ValidationMaxRetries = flags.validationMaxRetries
},
"validation-llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.ValidationLLMConcurrency = flags.validationLLMConcurrency
},
"validation-max-prompt-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.ValidationMaxPromptTokens = flags.validationMaxPromptTokens
},
"max-section-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.MaxSectionTokens = flags.maxSectionTokens
},
"min-section-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.MinSectionTokens = flags.minSectionTokens
},
"target-sections": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.TargetSections = flags.targetSections
},
"glossary-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.GlossaryConfidenceThreshold = flags.glossaryConfidenceThreshold
},
"grammar-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.GrammarConfidenceThreshold = flags.grammarConfidenceThreshold
},
"homophones-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.HomophonesConfidenceThreshold = flags.homophonesConfidenceThreshold
},
"spoken-word-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.SpokenWordConfidenceThreshold = flags.spokenWordConfidenceThreshold
},
"normalize-max-segment-gap": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.NormalizeMaxSegmentGap = flags.normalizeMaxSegmentGap
},
"normalize-ellipsis-gap": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.NormalizeEllipsisGap = flags.normalizeEllipsisGap
},
"normalize-max-segment-duration": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.NormalizeMaxSegmentDuration = flags.normalizeMaxSegmentDuration
},
"normalize-max-segment-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.NormalizeMaxSegmentTokens = flags.normalizeMaxSegmentTokens
},
"transcript-description": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.TranscriptDescription = flags.transcriptDescription
},
"work-dir": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.WorkDir = flags.workDir
},
"work-dir-retention": func(overrides *config.CLIOverrides, flags processFlags) {
overrides.WorkDirRetention = flags.workDirRetention
},
}
func processCLIOverrides(fs *flag.FlagSet, flags processFlags) (config.CLIOverrides, bool) {
overrides := config.CLIOverrides{}
explicitModules := false
fs.Visit(func(f *flag.Flag) {
if f.Name == "modules" {
explicitModules = true
}
binding, ok := processOverrideBindings[f.Name]
if !ok {
return
}
binding(&overrides, flags)
})
return overrides, explicitModules
}

View File

@@ -1,433 +0,0 @@
package cli
import (
"io"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
)
func TestProcessCLIOverridesMapsEveryConfigMutatingFlag(t *testing.T) {
tests := []struct {
name string
flagName string
value string
wantExplicitModules bool
assertOverrideFields func(t *testing.T, overrides config.CLIOverrides)
}{
{
name: "modules",
flagName: "modules",
value: "grammar,glossary",
wantExplicitModules: true,
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertStringOverride(t, "ModulesCSV", overrides.ModulesCSV, "grammar,glossary")
},
},
{
name: "output schema",
flagName: "output-schema",
value: "audita-v1",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertStringOverride(t, "OutputSchema", overrides.OutputSchema, "audita-v1")
},
},
{
name: "primary api key",
flagName: "llm-api-key",
value: "primary-key",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertStringOverride(t, "PrimaryLLMAPIKey", overrides.PrimaryLLMAPIKey, "primary-key")
},
},
{
name: "validation api key",
flagName: "validation-llm-api-key",
value: "validation-key",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertStringOverride(t, "ValidationLLMAPIKey", overrides.ValidationLLMAPIKey, "validation-key")
},
},
{
name: "primary model",
flagName: "model",
value: "primary-model",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertStringOverride(t, "PrimaryModel", overrides.PrimaryModel, "primary-model")
},
},
{
name: "validation model",
flagName: "validation-model",
value: "validation-model",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertStringOverride(t, "ValidationModel", overrides.ValidationModel, "validation-model")
},
},
{
name: "primary base url",
flagName: "base-url",
value: "https://primary.example.test",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertStringOverride(t, "PrimaryBaseURL", overrides.PrimaryBaseURL, "https://primary.example.test")
},
},
{
name: "validation base url",
flagName: "validation-base-url",
value: "https://validation.example.test",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertStringOverride(t, "ValidationBaseURL", overrides.ValidationBaseURL, "https://validation.example.test")
},
},
{
name: "primary timeout",
flagName: "llm-timeout-seconds",
value: "101",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "PrimaryLLMTimeoutSeconds", overrides.PrimaryLLMTimeoutSeconds, 101)
},
},
{
name: "total concurrency",
flagName: "total-llm-concurrency",
value: "5",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "TotalLLMConcurrency", overrides.TotalLLMConcurrency, 5)
},
},
{
name: "proposal concurrency",
flagName: "proposal-llm-concurrency",
value: "3",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "ProposalLLMConcurrency", overrides.ProposalLLMConcurrency, 3)
},
},
{
name: "legacy concurrency alias",
flagName: "llm-concurrency",
value: "4",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "PrimaryLLMConcurrency", overrides.PrimaryLLMConcurrency, 4)
},
},
{
name: "validation timeout",
flagName: "validation-llm-timeout-seconds",
value: "202",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "ValidationLLMTimeoutSeconds", overrides.ValidationLLMTimeoutSeconds, 202)
},
},
{
name: "max retries",
flagName: "max-retries",
value: "6",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "MaxRetries", overrides.MaxRetries, 6)
},
},
{
name: "validation max retries",
flagName: "validation-max-retries",
value: "7",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "ValidationMaxRetries", overrides.ValidationMaxRetries, 7)
},
},
{
name: "validation concurrency",
flagName: "validation-llm-concurrency",
value: "8",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "ValidationLLMConcurrency", overrides.ValidationLLMConcurrency, 8)
},
},
{
name: "validation max prompt tokens",
flagName: "validation-max-prompt-tokens",
value: "4096",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "ValidationMaxPromptTokens", overrides.ValidationMaxPromptTokens, 4096)
},
},
{
name: "max section tokens",
flagName: "max-section-tokens",
value: "9000",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "MaxSectionTokens", overrides.MaxSectionTokens, 9000)
},
},
{
name: "min section tokens",
flagName: "min-section-tokens",
value: "1000",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "MinSectionTokens", overrides.MinSectionTokens, 1000)
},
},
{
name: "target sections",
flagName: "target-sections",
value: "12",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "TargetSections", overrides.TargetSections, 12)
},
},
{
name: "glossary threshold",
flagName: "glossary-confidence-threshold",
value: "0.91",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertFloatOverride(t, "GlossaryConfidenceThreshold", overrides.GlossaryConfidenceThreshold, 0.91)
},
},
{
name: "grammar threshold",
flagName: "grammar-confidence-threshold",
value: "0.92",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertFloatOverride(t, "GrammarConfidenceThreshold", overrides.GrammarConfidenceThreshold, 0.92)
},
},
{
name: "homophones threshold",
flagName: "homophones-confidence-threshold",
value: "0.93",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertFloatOverride(t, "HomophonesConfidenceThreshold", overrides.HomophonesConfidenceThreshold, 0.93)
},
},
{
name: "spoken word threshold",
flagName: "spoken-word-confidence-threshold",
value: "0.94",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertFloatOverride(t, "SpokenWordConfidenceThreshold", overrides.SpokenWordConfidenceThreshold, 0.94)
},
},
{
name: "normalize max segment gap",
flagName: "normalize-max-segment-gap",
value: "1.2",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertFloatOverride(t, "NormalizeMaxSegmentGap", overrides.NormalizeMaxSegmentGap, 1.2)
},
},
{
name: "normalize ellipsis gap",
flagName: "normalize-ellipsis-gap",
value: "2.3",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertFloatOverride(t, "NormalizeEllipsisGap", overrides.NormalizeEllipsisGap, 2.3)
},
},
{
name: "normalize max segment duration",
flagName: "normalize-max-segment-duration",
value: "45.6",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertFloatOverride(t, "NormalizeMaxSegmentDuration", overrides.NormalizeMaxSegmentDuration, 45.6)
},
},
{
name: "normalize max segment tokens",
flagName: "normalize-max-segment-tokens",
value: "321",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertIntOverride(t, "NormalizeMaxSegmentTokens", overrides.NormalizeMaxSegmentTokens, 321)
},
},
{
name: "transcript description",
flagName: "transcript-description",
value: "podcast episode",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertStringOverride(t, "TranscriptDescription", overrides.TranscriptDescription, "podcast episode")
},
},
{
name: "work dir",
flagName: "work-dir",
value: "/tmp/custom-audita",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertStringOverride(t, "WorkDir", overrides.WorkDir, "/tmp/custom-audita")
},
},
{
name: "work dir retention",
flagName: "work-dir-retention",
value: "always",
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
assertStringOverride(t, "WorkDirRetention", overrides.WorkDirRetention, "always")
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
fs, flags := newProcessFlagSet(config.Default(), io.Discard)
if err := fs.Parse([]string{"--" + tc.flagName, tc.value}); err != nil {
t.Fatalf("parse flag: %v", err)
}
overrides, explicitModules := processCLIOverrides(fs, flags)
if explicitModules != tc.wantExplicitModules {
t.Fatalf("explicitModules=%v, want %v", explicitModules, tc.wantExplicitModules)
}
tc.assertOverrideFields(t, overrides)
})
}
}
func TestProcessCLIOverridesIgnoresNonConfigFlags(t *testing.T) {
fs, flags := newProcessFlagSet(config.Default(), io.Discard)
if err := fs.Parse([]string{
"--config", "/tmp/config.yml",
"--glossary", "/tmp/glossary.yml",
"--output", "/tmp/output.json",
"--report-json", "/tmp/report.json",
}); err != nil {
t.Fatalf("parse flags: %v", err)
}
overrides, explicitModules := processCLIOverrides(fs, flags)
if explicitModules {
t.Fatal("non-config flags should not mark modules explicit")
}
assertNoCLIOverrides(t, overrides)
}
func TestNewProcessFlagSetDefaultsReflectEffectiveConfig(t *testing.T) {
cfg := config.Default()
cfg.Modules = []string{"grammar", "glossary"}
cfg.OutputSchema = "audita-v1"
cfg.PrimaryLLM.APIKey = "primary-key"
cfg.ValidationLLM.APIKey = "validation-key"
cfg.PrimaryLLM.Model = "primary-model"
cfg.ValidationLLM.Model = "validation-model"
cfg.PrimaryLLM.BaseURL = "https://primary.example.test"
cfg.ValidationLLM.BaseURL = "https://validation.example.test"
cfg.PrimaryLLM.TimeoutSeconds = 101
cfg.TotalLLMConcurrency = 5
cfg.ProposalLLMConcurrency = 3
cfg.PrimaryLLM.MaxRetries = 6
cfg.ValidationMaxPromptTokens = 4096
cfg.MaxSectionTokens = 9000
cfg.MinSectionTokens = 1000
cfg.Thresholds.Glossary = 0.91
cfg.Thresholds.Grammar = 0.92
cfg.Thresholds.Homophones = 0.93
cfg.Thresholds.SpokenWord = 0.94
cfg.Normalization.MaxSegmentGap = 1.2
cfg.Normalization.EllipsisGap = 2.3
cfg.Normalization.MaxSegmentDuration = 45.6
cfg.Normalization.MaxSegmentTokens = 321
cfg.TranscriptDescription = "podcast episode"
cfg.WorkDir = "/tmp/custom-audita"
cfg.WorkDirRetention = config.WorkDirRetentionAlways
validationTimeout := 202
validationRetries := 7
validationConcurrency := 8
targetSections := 12
cfg.ValidationLLM.TimeoutSeconds = &validationTimeout
cfg.ValidationLLM.MaxRetries = &validationRetries
cfg.ValidationLLMConcurrency = &validationConcurrency
cfg.TargetSections = &targetSections
_, flags := newProcessFlagSet(cfg, io.Discard)
assertStringOverride(t, "modules default", flags.modules, "grammar,glossary")
assertStringOverride(t, "output schema default", flags.outputSchema, "audita-v1")
assertStringOverride(t, "primary api key default", flags.llmAPIKey, "primary-key")
assertStringOverride(t, "validation api key default", flags.validationLLMAPIKey, "validation-key")
assertStringOverride(t, "primary model default", flags.model, "primary-model")
assertStringOverride(t, "validation model default", flags.validationModel, "validation-model")
assertStringOverride(t, "primary base url default", flags.baseURL, "https://primary.example.test")
assertStringOverride(t, "validation base url default", flags.validationBaseURL, "https://validation.example.test")
assertIntOverride(t, "primary timeout default", flags.llmTimeoutSeconds, 101)
assertIntOverride(t, "total concurrency default", flags.totalLLMConcurrency, 5)
assertIntOverride(t, "proposal concurrency default", flags.proposalLLMConcurrency, 3)
assertIntOverride(t, "legacy concurrency alias default", flags.llmConcurrency, 5)
assertIntOverride(t, "validation timeout default", flags.validationLLMTimeoutSeconds, validationTimeout)
assertIntOverride(t, "max retries default", flags.maxRetries, 6)
assertIntOverride(t, "validation max retries default", flags.validationMaxRetries, validationRetries)
assertIntOverride(t, "validation concurrency default", flags.validationLLMConcurrency, validationConcurrency)
assertIntOverride(t, "validation max prompt tokens default", flags.validationMaxPromptTokens, 4096)
assertIntOverride(t, "max section tokens default", flags.maxSectionTokens, 9000)
assertIntOverride(t, "min section tokens default", flags.minSectionTokens, 1000)
assertIntOverride(t, "target sections default", flags.targetSections, targetSections)
assertFloatOverride(t, "glossary threshold default", flags.glossaryConfidenceThreshold, 0.91)
assertFloatOverride(t, "grammar threshold default", flags.grammarConfidenceThreshold, 0.92)
assertFloatOverride(t, "homophones threshold default", flags.homophonesConfidenceThreshold, 0.93)
assertFloatOverride(t, "spoken word threshold default", flags.spokenWordConfidenceThreshold, 0.94)
assertFloatOverride(t, "normalize max segment gap default", flags.normalizeMaxSegmentGap, 1.2)
assertFloatOverride(t, "normalize ellipsis gap default", flags.normalizeEllipsisGap, 2.3)
assertFloatOverride(t, "normalize max segment duration default", flags.normalizeMaxSegmentDuration, 45.6)
assertIntOverride(t, "normalize max segment tokens default", flags.normalizeMaxSegmentTokens, 321)
assertStringOverride(t, "transcript description default", flags.transcriptDescription, "podcast episode")
assertStringOverride(t, "work dir default", flags.workDir, "/tmp/custom-audita")
assertStringOverride(t, "work dir retention default", flags.workDirRetention, "always")
}
func TestNewProcessFlagSetUsesFallbackDefaultsForUnsetOptionalConfig(t *testing.T) {
cfg := config.Default()
_, flags := newProcessFlagSet(cfg, io.Discard)
assertIntOverride(t, "validation timeout fallback", flags.validationLLMTimeoutSeconds, cfg.PrimaryLLM.TimeoutSeconds)
assertIntOverride(t, "validation retries fallback", flags.validationMaxRetries, cfg.PrimaryLLM.MaxRetries)
assertIntOverride(t, "validation concurrency fallback", flags.validationLLMConcurrency, cfg.TotalLLMConcurrency)
assertIntOverride(t, "target sections fallback", flags.targetSections, 0)
}
func assertStringOverride(t *testing.T, name string, got *string, want string) {
t.Helper()
if got == nil || *got != want {
t.Fatalf("%s=%v, want %q", name, pointerValue(got), want)
}
}
func assertIntOverride(t *testing.T, name string, got *int, want int) {
t.Helper()
if got == nil || *got != want {
t.Fatalf("%s=%v, want %d", name, pointerValue(got), want)
}
}
func assertFloatOverride(t *testing.T, name string, got *float64, want float64) {
t.Helper()
if got == nil || *got != want {
t.Fatalf("%s=%v, want %v", name, pointerValue(got), want)
}
}
func assertNoCLIOverrides(t *testing.T, overrides config.CLIOverrides) {
t.Helper()
value := reflect.ValueOf(overrides)
typ := value.Type()
for i := 0; i < value.NumField(); i++ {
field := value.Field(i)
if field.Kind() != reflect.Ptr {
t.Fatalf("unexpected non-pointer CLIOverrides field %s", typ.Field(i).Name)
}
if !field.IsNil() {
t.Fatalf("expected no CLI overrides, field %s was set", typ.Field(i).Name)
}
}
}
func pointerValue[T any](ptr *T) any {
if ptr == nil {
return "<nil>"
}
if stringer, ok := any(*ptr).(interface{ String() string }); ok {
return strings.TrimSpace(stringer.String())
}
return *ptr
}

View File

@@ -1,469 +0,0 @@
package cli
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/audita/internal/core/reporting"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
)
type releaseFixtureExpectations struct {
MustApplyTexts []string `json:"must_apply_texts"`
MustNotApplyTexts []string `json:"must_not_apply_texts"`
ProtectedTerms []string `json:"protected_terms"`
ExpectedModuleInstance []string `json:"expected_module_instances"`
MinimumCounts struct {
Applied int `json:"applied"`
Rejected int `json:"rejected"`
Skipped int `json:"skipped"`
} `json:"minimum_counts"`
}
func TestReleaseFixtureDefaultPipelineReadiness(t *testing.T) {
base := fixturePath(filepath.Join("release", "default-release"))
var expectations releaseFixtureExpectations
if err := json.Unmarshal(readFile(t, base+".expectations.json"), &expectations); err != nil {
t.Fatalf("unmarshal release expectations: %v", err)
}
proposalResponses := readProposalResponses(t, base+".proposals.json")
validationResponses := readValidationResponses(t, base+".validations.json")
// First pass: default full pipeline with deterministic fake LLM responses.
first := runReleaseFixturePass(t, releaseRunConfig{
transcriptPath: base + ".transcript.json",
glossaryPath: base + ".glossary.yaml",
outputSchema: "bare-segments",
proposalResponses: proposalResponses,
validationResponses: validationResponses,
expectedProposalCalls: []string{"glossary_1:proposal", "homophones:proposal", "glossary_2:proposal", "spoken_word:proposal", "grammar:proposal"},
reportSchemaName: reporting.DefaultProcessReportSchemaName,
reportSchemaVersion: reporting.DefaultProcessReportSchemaVersion,
expectedOutputSchema: "bare-segments",
expectModuleInstances: expectations.ExpectedModuleInstance,
expectUtilizationPaths: true,
})
gotTranscript := mustReadTranscript(t, first.outputPath)
expectFinalTranscriptContains(t, gotTranscript, expectations.MustApplyTexts)
expectFinalTranscriptDoesNotContain(t, gotTranscript, expectations.MustNotApplyTexts)
expectFinalTranscriptContains(t, gotTranscript, expectations.ProtectedTerms)
assertReleaseCounts(t, first.report, expectations)
assertPromptAndSchemaMetadataPresent(t, first.runDir)
assertReleaseLedgerShape(t, first.report)
assertReleaseUtilizationShape(t, first.report)
assertStableValidatorKeysPresent(t, first.report)
assertStdoutStderrContract(t, first.stdout, first.stderr)
assertNoSecretMarkersInTree(t, first.runDir, []string{"release-secret"})
assertNoSecretMarkers(t, first.reportPath, []string{"release-secret"})
// Output schema check: audita-v1 object payload.
auditaV1 := runReleaseFixturePass(t, releaseRunConfig{
transcriptPath: base + ".transcript.json",
glossaryPath: base + ".glossary.yaml",
outputSchema: "audita-v1",
proposalResponses: proposalResponses,
validationResponses: validationResponses,
expectedProposalCalls: []string{"glossary_1:proposal", "homophones:proposal", "glossary_2:proposal", "spoken_word:proposal", "grammar:proposal"},
reportSchemaName: reporting.DefaultProcessReportSchemaName,
reportSchemaVersion: reporting.DefaultProcessReportSchemaVersion,
expectedOutputSchema: "audita-v1",
expectModuleInstances: expectations.ExpectedModuleInstance,
expectUtilizationPaths: true,
})
assertAuditaV1OutputShape(t, auditaV1.outputPath)
// Idempotence-oriented second pass:
// run again on first output with deterministic no-op responses.
noOpProposals := make([]proposal_generation.StructuredCorrectionSet, 5)
for i := range noOpProposals {
noOpProposals[i] = proposal_generation.StructuredCorrectionSet{Corrections: nil}
}
second := runReleaseFixturePass(t, releaseRunConfig{
transcriptPath: first.outputPath,
glossaryPath: base + ".glossary.yaml",
outputSchema: "bare-segments",
proposalResponses: noOpProposals,
validationResponses: nil,
expectedProposalCalls: []string{"glossary_1:proposal", "homophones:proposal", "glossary_2:proposal", "spoken_word:proposal", "grammar:proposal"},
reportSchemaName: reporting.DefaultProcessReportSchemaName,
reportSchemaVersion: reporting.DefaultProcessReportSchemaVersion,
expectedOutputSchema: "bare-segments",
expectModuleInstances: expectations.ExpectedModuleInstance,
expectUtilizationPaths: true,
})
firstSegments := mustReadTranscript(t, first.outputPath)
secondSegments := mustReadTranscript(t, second.outputPath)
if !reflect.DeepEqual(firstSegments, secondSegments) {
t.Fatalf("expected idempotent second pass transcript; first=%+v second=%+v", firstSegments, secondSegments)
}
if second.report.ModulesSummary == nil {
t.Fatalf("expected modules summary on second pass")
}
if second.report.ModulesSummary.TotalAppliedChanges != 0 {
t.Fatalf("expected no-op second pass (0 applied), got %+v", second.report.ModulesSummary)
}
}
type releaseRunConfig struct {
transcriptPath string
glossaryPath string
outputSchema string
proposalResponses []proposal_generation.StructuredCorrectionSet
validationResponses []validators.LLMValidationResponse
expectedProposalCalls []string
reportSchemaName string
reportSchemaVersion string
expectedOutputSchema string
expectModuleInstances []string
expectUtilizationPaths bool
}
type releaseRunResult struct {
stdout string
stderr string
outputPath string
reportPath string
report reporting.ProcessReport
runDir string
}
func runReleaseFixturePass(t *testing.T, cfg releaseRunConfig) releaseRunResult {
t.Helper()
processProposalLLMClient = &fakeStructuredLLMClient{proposalResponses: append([]proposal_generation.StructuredCorrectionSet(nil), cfg.proposalResponses...)}
processValidationLLMClient = &fakeStructuredLLMClient{validationResponses: append([]validators.LLMValidationResponse(nil), cfg.validationResponses...)}
t.Cleanup(func() {
processProposalLLMClient = nil
processValidationLLMClient = nil
})
workDir := t.TempDir()
reportPath := filepath.Join(t.TempDir(), "report.json")
outputPath := filepath.Join(t.TempDir(), "out.json")
configPath := writeFile(t, "release-config.yml", "version: 1\n")
args := []string{
"process",
cfg.transcriptPath,
"--glossary",
cfg.glossaryPath,
"--config",
configPath,
"--output",
outputPath,
"--output-schema",
cfg.outputSchema,
"--report-json",
reportPath,
"--work-dir",
workDir,
"--work-dir-retention",
"always",
}
var stdout, stderr bytes.Buffer
exitCode := Run(args, &stdout, &stderr)
if exitCode != 0 {
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
}
report := readProcessReport(t, reportPath)
if report.ReportMetadata.ReportSchemaName != cfg.reportSchemaName || report.ReportMetadata.ReportSchemaVersion != cfg.reportSchemaVersion {
t.Fatalf("unexpected report schema metadata: %+v", report.ReportMetadata)
}
if report.ReportMetadata.OutputSchema != cfg.expectedOutputSchema {
t.Fatalf("unexpected output schema metadata: got %q want %q", report.ReportMetadata.OutputSchema, cfg.expectedOutputSchema)
}
if len(cfg.expectModuleInstances) > 0 {
got := make([]string, 0, len(report.ModuleResults))
for _, mr := range report.ModuleResults {
got = append(got, mr.ModuleInstance)
}
if !reflect.DeepEqual(got, cfg.expectModuleInstances) {
t.Fatalf("unexpected module instances: got %v want %v", got, cfg.expectModuleInstances)
}
}
if report.Diagnostics == nil {
t.Fatalf("expected diagnostics metadata")
}
if cfg.expectUtilizationPaths {
if report.Diagnostics.UtilizationSummaryPath == "" || report.Diagnostics.CorrectionLedgerPath == "" {
t.Fatalf("expected utilization/ledger artifact paths in report diagnostics: %+v", report.Diagnostics)
}
}
runDir := onlyRunDir(t, workDir)
if _, err := os.Stat(filepath.Join(runDir, "report.json")); err != nil {
t.Fatalf("expected run-dir report: %v", err)
}
if c, ok := processProposalLLMClient.(*fakeStructuredLLMClient); ok {
if !reflect.DeepEqual(c.calls, cfg.expectedProposalCalls) {
t.Fatalf("unexpected proposal call order: got %v want %v", c.calls, cfg.expectedProposalCalls)
}
}
return releaseRunResult{
stdout: stdout.String(),
stderr: stderr.String(),
outputPath: outputPath,
reportPath: reportPath,
report: report,
runDir: runDir,
}
}
func readProposalResponses(t *testing.T, path string) []proposal_generation.StructuredCorrectionSet {
t.Helper()
var out []proposal_generation.StructuredCorrectionSet
if err := json.Unmarshal(readFile(t, path), &out); err != nil {
t.Fatalf("unmarshal proposal responses: %v", err)
}
return out
}
func readValidationResponses(t *testing.T, path string) []validators.LLMValidationResponse {
t.Helper()
var out []validators.LLMValidationResponse
if err := json.Unmarshal(readFile(t, path), &out); err != nil {
t.Fatalf("unmarshal validation responses: %v", err)
}
return out
}
func mustReadTranscript(t *testing.T, path string) []schema.Segment {
t.Helper()
transcript, err := schema.ParseTranscriptJSON(readFile(t, path))
if err != nil {
t.Fatalf("parse transcript output: %v", err)
}
return transcript.Segments
}
func expectFinalTranscriptContains(t *testing.T, segments []schema.Segment, needles []string) {
t.Helper()
joined := flattenTranscriptText(segments)
for _, needle := range needles {
if !strings.Contains(joined, needle) {
t.Fatalf("expected transcript to contain %q, got %q", needle, joined)
}
}
}
func expectFinalTranscriptDoesNotContain(t *testing.T, segments []schema.Segment, needles []string) {
t.Helper()
joined := flattenTranscriptText(segments)
for _, needle := range needles {
if strings.Contains(joined, needle) {
t.Fatalf("expected transcript to not contain %q, got %q", needle, joined)
}
}
}
func flattenTranscriptText(segments []schema.Segment) string {
parts := make([]string, 0, len(segments))
for _, s := range segments {
parts = append(parts, s.Text)
}
return strings.Join(parts, "\n")
}
func assertReleaseCounts(t *testing.T, report reporting.ProcessReport, exp releaseFixtureExpectations) {
t.Helper()
if report.ModulesSummary == nil {
t.Fatalf("expected modules_summary")
}
if report.ModulesSummary.TotalAppliedChanges < exp.MinimumCounts.Applied {
t.Fatalf("expected at least %d applied changes, got %+v", exp.MinimumCounts.Applied, report.ModulesSummary)
}
validatorRejected := 0
skipped := 0
for _, mr := range report.ModuleResults {
validatorRejected += len(mr.ValidatorRejected)
skipped += len(mr.SkippedChanges)
}
if validatorRejected < exp.MinimumCounts.Rejected {
t.Fatalf("expected at least %d validator rejections, got %d", exp.MinimumCounts.Rejected, validatorRejected)
}
if skipped < exp.MinimumCounts.Skipped {
t.Fatalf("expected at least %d application skips, got %d", exp.MinimumCounts.Skipped, skipped)
}
}
func assertReleaseUtilizationShape(t *testing.T, report reporting.ProcessReport) {
t.Helper()
var payload struct {
EffectiveConcurrency struct {
TotalLLM int `json:"total_llm"`
} `json:"effective_concurrency"`
RunTiming struct {
SchedulerQueueWaitMS int64 `json:"scheduler_queue_wait_ms"`
LLMExecutionTimeMS int64 `json:"llm_execution_time_ms"`
DeterministicValidationMS int64 `json:"deterministic_validation_time_ms"`
} `json:"run_timing"`
Modules []map[string]any `json:"modules"`
Validators []map[string]any `json:"validators"`
}
if err := json.Unmarshal(readFile(t, report.Diagnostics.UtilizationSummaryPath), &payload); err != nil {
t.Fatalf("unmarshal utilization diagnostics: %v", err)
}
if payload.EffectiveConcurrency.TotalLLM <= 0 {
t.Fatalf("expected positive total llm concurrency, got %+v", payload.EffectiveConcurrency)
}
if payload.RunTiming.SchedulerQueueWaitMS < 0 || payload.RunTiming.LLMExecutionTimeMS < 0 || payload.RunTiming.DeterministicValidationMS < 0 {
t.Fatalf("expected non-negative run timing values, got %+v", payload.RunTiming)
}
if len(payload.Modules) == 0 {
t.Fatalf("expected module timing summaries")
}
if len(payload.Validators) == 0 {
t.Fatalf("expected validator timing summaries")
}
}
func assertReleaseLedgerShape(t *testing.T, report reporting.ProcessReport) {
t.Helper()
var entries []struct {
ModuleKey string `json:"module_key"`
ModuleInstance string `json:"module_instance"`
ProposalIndex int `json:"proposal_index"`
Disposition string `json:"disposition"`
DispositionReason string `json:"disposition_reason_code"`
OriginalText string `json:"original_text"`
ProposedCorrected string `json:"proposed_corrected_text"`
ReplacementPolicy string `json:"replacement_policy"`
DeterministicResults []struct {
ValidatorKey string `json:"validator_key"`
} `json:"deterministic_validator_decisions"`
LLMResults []struct {
ValidatorKey string `json:"validator_key"`
} `json:"llm_validator_decisions"`
}
if err := json.Unmarshal(readFile(t, report.Diagnostics.CorrectionLedgerPath), &entries); err != nil {
t.Fatalf("unmarshal correction ledger: %v", err)
}
if len(entries) == 0 {
t.Fatalf("expected correction ledger entries")
}
hasApplied := false
hasRejected := false
hasSkipped := false
for _, entry := range entries {
if entry.ModuleInstance == "" || entry.ModuleKey == "" {
t.Fatalf("expected module identity in ledger entry: %+v", entry)
}
switch entry.Disposition {
case "applied":
hasApplied = true
case "rejected":
hasRejected = true
case "skipped":
hasSkipped = true
}
}
if !hasApplied || !hasRejected {
t.Fatalf("expected applied and rejected entries in correction ledger, got %+v", entries)
}
if !hasSkipped {
// Some deterministic fixture paths do not trigger apply-time skips;
// rejections are still captured separately from application skips.
}
}
func assertPromptAndSchemaMetadataPresent(t *testing.T, runDir string) {
t.Helper()
metadataPaths, err := filepath.Glob(filepath.Join(runDir, "*", "*request-metadata.json"))
if err != nil {
t.Fatalf("glob request metadata artifacts: %v", err)
}
if len(metadataPaths) == 0 {
t.Fatalf("expected request metadata artifacts with prompt metadata")
}
foundPromptMetadata := false
foundSchemaMetadata := false
for _, path := range metadataPaths {
var payload map[string]any
if err := json.Unmarshal(readFile(t, path), &payload); err != nil {
t.Fatalf("unmarshal request metadata artifact %q: %v", path, err)
}
if pm, ok := payload["prompt_metadata"].(map[string]any); ok {
if pm["prompt_id"] != nil && pm["prompt_version"] != nil && pm["sha256"] != nil {
foundPromptMetadata = true
}
}
if sm, ok := payload["response_schema"].(map[string]any); ok {
if sm["id"] != nil && sm["version"] != nil && sm["name"] != nil && sm["sha256"] != nil {
foundSchemaMetadata = true
}
}
}
if !foundPromptMetadata {
t.Fatalf("expected prompt metadata in request metadata artifacts")
}
if !foundSchemaMetadata {
t.Fatalf("expected structured response schema metadata in request metadata artifacts")
}
}
func assertStableValidatorKeysPresent(t *testing.T, report reporting.ProcessReport) {
t.Helper()
seen := map[string]bool{}
for _, module := range report.ModuleResults {
for _, decision := range module.ValidatorDecisions {
seen[decision.ValidatorName] = true
}
for _, rejected := range module.ValidatorRejected {
seen[rejected.ValidatorName] = true
}
}
expectedAny := []string{
"confidence_threshold",
"original_text_presence",
"no_effect",
}
for _, key := range expectedAny {
if !seen[key] {
t.Fatalf("expected stable validator key %q in report decisions/rejections; seen=%v", key, seen)
}
}
}
func assertStdoutStderrContract(t *testing.T, stdout, stderr string) {
t.Helper()
if stdout != "" {
t.Fatalf("expected empty stdout with --output, got %q", stdout)
}
if strings.Contains(stderr, `"module_results"`) || strings.Contains(stderr, `"report_metadata"`) {
t.Fatalf("stderr should remain human-readable, not report JSON: %q", stderr)
}
}
func assertAuditaV1OutputShape(t *testing.T, outputPath string) {
t.Helper()
var payload struct {
Schema string `json:"schema"`
Version string `json:"version"`
Segments []schema.Segment `json:"segments"`
}
if err := json.Unmarshal(readFile(t, outputPath), &payload); err != nil {
t.Fatalf("unmarshal audita-v1 output: %v", err)
}
if payload.Schema != "audita-v1" || payload.Version != "v1" {
t.Fatalf("unexpected audita-v1 metadata: %+v", payload)
}
if len(payload.Segments) == 0 {
t.Fatalf("expected non-empty audita-v1 segments")
}
}

View File

@@ -2,292 +2,13 @@ package cli
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"time"
"gitea.maximumdirect.net/eric/audita/internal/core/chunking"
"gitea.maximumdirect.net/eric/audita/internal/core/config"
"gitea.maximumdirect.net/eric/audita/internal/core/diagnostics"
coreio "gitea.maximumdirect.net/eric/audita/internal/core/io"
"gitea.maximumdirect.net/eric/audita/internal/core/normalization"
"gitea.maximumdirect.net/eric/audita/internal/core/outputschema"
"gitea.maximumdirect.net/eric/audita/internal/core/reporting"
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/llm"
"gitea.maximumdirect.net/eric/audita/internal/framework/modules"
"gitea.maximumdirect.net/eric/audita/internal/framework/processreport"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/runner"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
)
type noOpStructuredLLMClient struct{}
func (c noOpStructuredLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
_ = ctx
_ = req
switch target := out.(type) {
case *validators.LLMValidationResponse:
*target = validators.LLMValidationResponse{Validations: []validators.LLMValidationDecision{}}
case *proposal_generation.StructuredCorrectionSet:
*target = proposal_generation.StructuredCorrectionSet{Corrections: nil}
}
return contracts.StructuredCompletionResponse{}, nil
}
func shouldUseNoOpLLMClientForTests() bool {
return strings.HasSuffix(filepath.Base(os.Args[0]), ".test") || os.Getenv("GO_WANT_HELPER_PROCESS") == "1"
}
type processInvocation struct {
TranscriptPath string
GlossaryPath string
OutputPath string
ReportJSONPath string
Config config.Config
ConfigPath string
ConfigSource string
ConfigVersion *int
ExplicitModules bool
}
var processModuleFactory runner.ModuleFactory
var processProposalLLMClient contracts.StructuredLLMClient
var processProposalLLMScheduler runner.ValidationScheduler
var processValidationLLMClient contracts.StructuredLLMClient
var processValidationLLMScheduler runner.ValidationScheduler
var processRunnerContext = func() (context.Context, context.CancelFunc) {
return context.Background(), func() {}
}
var processRunner = func(inv processInvocation, stdout io.Writer) (*normalization.NormalizationSummary, *chunking.Summary, *runner.RunOutput, *diagnostics.RunDirectory, error) {
runDir, err := diagnostics.NewRunDirectory(inv.Config.WorkDir, string(inv.Config.WorkDirRetention))
if err != nil {
return nil, nil, nil, nil, fmt.Errorf("run_dir_creation: %w", err)
}
fail := func(phase string, err error, runOutput *runner.RunOutput) (*normalization.NormalizationSummary, *chunking.Summary, *runner.RunOutput, *diagnostics.RunDirectory, error) {
_ = runDir.WriteErrorLog(fmt.Sprintf("%s: %v", phase, err))
return nil, nil, runOutput, runDir, fmt.Errorf("%s: %w", phase, err)
}
if err := runDir.WriteInvocationMetadata(diagnostics.InvocationMetadata{
Operation: "process",
TranscriptPath: inv.TranscriptPath,
GlossaryPath: inv.GlossaryPath,
OutputPath: inv.OutputPath,
ReportJSONPath: inv.ReportJSONPath,
ConfigPath: inv.ConfigPath,
ConfigSource: inv.ConfigSource,
ConfigVersion: inv.ConfigVersion,
TranscriptDescription: inv.Config.TranscriptDescription,
Modules: append([]string(nil), inv.Config.Modules...),
}); err != nil {
_ = runDir.WriteErrorLog(fmt.Sprintf("invocation_metadata: %v", err))
}
if err := runDir.WriteEffectiveConfig(inv.Config); err != nil {
_ = runDir.WriteErrorLog(fmt.Sprintf("effective_config: %v", err))
}
transcriptBytes, err := coreio.ReadRequiredFile(inv.TranscriptPath, "transcript")
if err != nil {
return fail("transcript_read", err, nil)
}
glossaryBytes, err := coreio.ReadRequiredFile(inv.GlossaryPath, "glossary")
if err != nil {
return fail("glossary_read", err, nil)
}
sourceTranscript, err := schema.ParseSourceTranscriptJSON(transcriptBytes)
if err != nil {
return fail("transcript_schema", err, nil)
}
glossary, err := schema.ParseGlossaryYAML(glossaryBytes)
if err != nil {
return fail("glossary_schema", err, nil)
}
if err := runDir.WriteSourceTranscript(sourceTranscript, transcriptBytes); err != nil {
_ = runDir.WriteErrorLog(fmt.Sprintf("source_artifact: %v", err))
}
canonical := sourceToCanonicalTranscript(sourceTranscript)
normalizer := normalization.NewNormalizer(normalization.NormalizationConfig{
MaxSegmentGap: inv.Config.Normalization.MaxSegmentGap,
EllipsisGap: inv.Config.Normalization.EllipsisGap,
MaxSegmentDuration: inv.Config.Normalization.MaxSegmentDuration,
MaxSegmentTokens: inv.Config.Normalization.MaxSegmentTokens,
})
normalizedTranscript, normSummary := normalizer.Normalize(canonical)
if err := runDir.WriteNormalizedTranscript(normalizedTranscript); err != nil {
_ = runDir.WriteErrorLog(fmt.Sprintf("normalized_artifact: %v", err))
}
if err := runDir.WriteNormalizationSummary(normSummary); err != nil {
_ = runDir.WriteErrorLog(fmt.Sprintf("normalization_summary: %v", err))
}
// Compute chunks after normalization
chunker := chunking.NewChunker(chunking.ChunkingConfig{
MaxSectionTokens: inv.Config.MaxSectionTokens,
MinSectionTokens: inv.Config.MinSectionTokens,
TargetSections: inv.Config.TargetSections,
})
sections, chunkErr := chunker.ChunkTranscript(normalizedTranscript)
if chunkErr != nil {
return fail("chunking", chunkErr, nil)
}
chunkConfig := chunking.ChunkingConfig{
MaxSectionTokens: inv.Config.MaxSectionTokens,
MinSectionTokens: inv.Config.MinSectionTokens,
TargetSections: inv.Config.TargetSections,
}
chunkSummary := chunking.ComputeSummary(sections, chunkConfig)
chunkDetailedSummary := chunking.ComputeDetailedSummary(sections, chunkConfig)
if err := runDir.WriteChunkingSummary(&chunkDetailedSummary); err != nil {
_ = runDir.WriteErrorLog(fmt.Sprintf("chunking_summary: %v", err))
}
workingTranscript := normalizedTranscript
var runOutput *runner.RunOutput
moduleFactory := processModuleFactory
if moduleFactory == nil {
moduleFactory = modules.NewFactory(modules.Dependencies{
Config: &inv.Config,
Glossary: glossary,
DiagnosticsDir: runDir.Path(),
})
}
if moduleFactory != nil {
proposalLLMClient := processProposalLLMClient
validationLLMClient := processValidationLLMClient
proposalScheduler := processProposalLLMScheduler
validationScheduler := processValidationLLMScheduler
if processModuleFactory == nil {
// Production runtime path: construct clients/schedulers from config.
if proposalLLMClient == nil {
if shouldUseNoOpLLMClientForTests() {
proposalLLMClient = noOpStructuredLLMClient{}
} else {
primaryCfg := llm.ResolvePrimaryConfig(inv.Config)
client, clientErr := llm.NewOpenAICompatibleClient(primaryCfg.ToOpenAICompatibleClientConfig(nil))
if clientErr != nil {
return fail("runner_setup", clientErr, nil)
}
proposalLLMClient = client
}
}
if validationLLMClient == nil {
if shouldUseNoOpLLMClientForTests() {
validationLLMClient = noOpStructuredLLMClient{}
} else {
validationCfg := llm.ResolveValidationConfig(inv.Config)
client, clientErr := llm.NewOpenAICompatibleClient(validationCfg.ToOpenAICompatibleClientConfig(nil))
if clientErr != nil {
return fail("runner_setup", clientErr, nil)
}
validationLLMClient = client
}
}
globalScheduler := proposalScheduler
if globalScheduler == nil {
s, sErr := llm.NewScheduler(inv.Config.TotalLLMConcurrency)
if sErr != nil {
return fail("runner_setup", sErr, nil)
}
globalScheduler = s
}
if proposalScheduler == nil {
proposalScheduler = globalScheduler
if inv.Config.EffectiveProposalLLMConcurrency() < inv.Config.TotalLLMConcurrency {
s, sErr := llm.NewScheduler(inv.Config.EffectiveProposalLLMConcurrency())
if sErr != nil {
return fail("runner_setup", sErr, nil)
}
proposalScheduler = composeSchedulers(globalScheduler, s)
}
}
if validationScheduler == nil {
validationScheduler = globalScheduler
if inv.Config.ValidationLLMConcurrency != nil && inv.Config.EffectiveValidationLLMConcurrency() < inv.Config.TotalLLMConcurrency {
s, sErr := llm.NewScheduler(inv.Config.EffectiveValidationLLMConcurrency())
if sErr != nil {
return fail("runner_setup", sErr, nil)
}
validationScheduler = composeSchedulers(globalScheduler, s)
}
}
}
moduleSpecs, err := contracts.ResolveModuleRunSpecs(inv.Config.Modules)
if err != nil {
return fail("runner_setup", err, nil)
}
runCtx, cancelRun := processRunnerContext()
defer cancelRun()
runnerResult, runErr := runner.New(moduleFactory).Run(runCtx, runner.RunInput{
Config: &inv.Config,
Transcript: normalizedTranscript,
Glossary: glossary,
ModuleSpecs: moduleSpecs,
EffectiveConcurrency: runner.EffectiveConcurrencyLimits{
TotalLLM: inv.Config.TotalLLMConcurrency,
ProposalLLM: inv.Config.EffectiveProposalLLMConcurrency(),
ValidationLLM: inv.Config.EffectiveValidationLLMConcurrency(),
},
ProposalLLMClient: proposalLLMClient,
ProposalLLMScheduler: proposalScheduler,
ProposalDiagnosticsDir: runDir.Path(),
ValidationLLMClient: validationLLMClient,
ValidationLLMScheduler: validationScheduler,
ValidationDiagnosticsDir: runDir.Path(),
})
runOutput = &runnerResult
if runErr != nil {
return fail("runner_execution", runErr, runOutput)
}
workingTranscript = runnerResult.FinalTranscript
}
encoderDef, err := outputschema.Resolve(inv.Config.OutputSchema)
if err != nil {
return fail("output_schema", err, runOutput)
}
outputBytes, err := encoderDef.Encoder(workingTranscript)
if err != nil {
return fail("serialization", err, runOutput)
}
if strings.TrimSpace(inv.OutputPath) != "" {
if err := coreio.WriteFile(inv.OutputPath, outputBytes); err != nil {
return fail("output_write", err, runOutput)
}
return normSummary, &chunkSummary, runOutput, runDir, nil
}
if _, err := stdout.Write(outputBytes); err != nil {
return fail("stdout_write", err, runOutput)
}
return normSummary, &chunkSummary, runOutput, runDir, nil
}
func sourceToCanonicalTranscript(source *schema.SourceTranscript) *schema.Transcript {
segments := make([]schema.Segment, len(source.Segments))
for i, s := range source.Segments {
@@ -307,48 +28,22 @@ func sourceToCanonicalTranscript(source *schema.SourceTranscript) *schema.Transc
return &schema.Transcript{Segments: segments}
}
type chainedScheduler struct {
schedulers []runner.ValidationScheduler
}
func (s chainedScheduler) Run(ctx context.Context, fn func(context.Context) error) error {
if len(s.schedulers) == 0 {
return fn(ctx)
}
run := fn
for i := len(s.schedulers) - 1; i >= 0; i-- {
scheduler := s.schedulers[i]
next := run
run = func(callCtx context.Context) error {
return scheduler.Run(callCtx, next)
}
}
return run(ctx)
}
func composeSchedulers(schedulers ...runner.ValidationScheduler) runner.ValidationScheduler {
filtered := make([]runner.ValidationScheduler, 0, len(schedulers))
for _, scheduler := range schedulers {
if scheduler != nil {
filtered = append(filtered, scheduler)
}
}
switch len(filtered) {
case 0:
return nil
case 1:
return filtered[0]
default:
return chainedScheduler{schedulers: filtered}
}
}
// Run executes the Audita CLI with the provided arguments and streams.
func Run(args []string, stdout, stderr io.Writer) int {
func Run(ctx context.Context, args []string, stdout, stderr io.Writer) int {
if len(args) == 0 {
writeRootUsage(stdout)
return 0
}
if len(args) == 2 && args[0] == "help" {
switch args[1] {
case "process":
writeProcessUsage(stdout)
return 0
case "config":
writeConfigUsage(stdout)
return 0
}
}
if isHelpCommand(args) {
writeRootUsage(stdout)
@@ -356,10 +51,10 @@ func Run(args []string, stdout, stderr io.Writer) int {
}
if args[0] == "process" {
return runProcess(args[1:], stdout, stderr)
return runProcess(ctx, args[1:], stdout, stderr)
}
if args[0] == "config" {
return runConfig(args[1:], stdout, stderr)
return runConfig(ctx, args[1:], stdout, stderr)
}
fmt.Fprintf(stderr, "audita: unknown command %q\n\n", args[0])
@@ -367,180 +62,17 @@ func Run(args []string, stdout, stderr io.Writer) int {
return 2
}
func runProcess(args []string, stdout, stderr io.Writer) int {
startedAt := time.Now().UTC()
configPathOverride, configPathOverrideSet, err := findConfigPathOverride(args)
if err != nil {
fmt.Fprintf(stderr, "audita process: invalid CLI configuration: %v\n", err)
return 2
}
effectiveConfig, err := config.LoadEffectiveConfig(configPathOverride, configPathOverrideSet)
if err != nil {
var effectiveConfigErr *config.EffectiveConfigError
if errors.As(err, &effectiveConfigErr) {
switch effectiveConfigErr.Kind {
case config.EffectiveConfigErrorLoadFile, config.EffectiveConfigErrorApplyFile:
fmt.Fprintf(stderr, "audita process: invalid config file: %v\n", effectiveConfigErr)
case config.EffectiveConfigErrorApplyEnv:
fmt.Fprintf(stderr, "audita process: invalid environment configuration: %v\n", effectiveConfigErr)
default:
fmt.Fprintf(stderr, "audita process: %v\n", effectiveConfigErr)
}
} else {
fmt.Fprintf(stderr, "audita process: %v\n", err)
}
return 2
}
cfg := effectiveConfig.Config
configPath := effectiveConfig.ConfigPath
configSource := effectiveConfig.ConfigSource
configVersion := effectiveConfig.ConfigVersion
fs, pFlags := newProcessFlagSet(cfg, stderr)
if isHelpCommand(args) || hasHelpFlag(args) {
writeProcessUsage(stdout, fs)
return 0
}
parseArgs := args
transcriptFromFront := ""
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
transcriptFromFront = args[0]
parseArgs = args[1:]
}
if err := fs.Parse(parseArgs); err != nil {
if errors.Is(err, flag.ErrHelp) {
writeProcessUsage(stdout, fs)
return 0
}
return 2
}
overrides, explicitModules := processCLIOverrides(fs, pFlags)
if err := cfg.ApplyCLIOverrides(overrides); err != nil {
fmt.Fprintf(stderr, "audita process: invalid CLI configuration: %v\n", err)
return 2
}
remaining := fs.Args()
positional := make([]string, 0, len(remaining)+1)
if transcriptFromFront != "" {
positional = append(positional, transcriptFromFront)
}
positional = append(positional, remaining...)
if len(positional) != 1 {
fmt.Fprintln(stderr, "audita process: expected exactly 1 transcript JSON path argument")
return 2
}
if strings.TrimSpace(*pFlags.glossaryPath) == "" {
fmt.Fprintln(stderr, "audita process: --glossary is required")
return 2
}
inv := processInvocation{
TranscriptPath: positional[0],
GlossaryPath: *pFlags.glossaryPath,
OutputPath: *pFlags.outputPath,
ReportJSONPath: *pFlags.reportJSONPath,
Config: cfg,
ConfigPath: configPath,
ConfigSource: configSource,
ConfigVersion: configVersion,
ExplicitModules: explicitModules,
}
normSummary, chunkSummary, runOutput, runDir, runErr := processRunner(inv, stdout)
completedAt := time.Now().UTC()
if runErr != nil {
if runDir != nil && runOutput != nil {
if runOutput.Utilization != nil {
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactUtilizationSummary, runOutput.Utilization)
}
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactCorrectionLedger, processreport.BuildCorrectionLedger(processreport.CorrectionLedgerInput{
RunDirectoryPath: runDir.Path(),
RunOutput: runOutput,
}))
}
errorPhase, errorMessage := extractErrorPhase(runErr)
report := processreport.Build(processReportInput("failed", inv, runDir, startedAt, completedAt, errorMessage, errorPhase, nil, nil, runOutput))
if strings.TrimSpace(inv.ReportJSONPath) != "" {
if err := reporting.WriteProcessReport(inv.ReportJSONPath, report); err != nil {
fmt.Fprintf(stderr, "audita process: %v\n", err)
}
}
if runDir != nil {
_ = runDir.WriteReport(report)
_ = runDir.ApplyRetention(diagnostics.RetentionDecisionInput{
RunSucceeded: false,
})
}
fmt.Fprintf(stderr, "audita process: %v\n", runErr)
if runDir != nil {
fmt.Fprintf(stderr, "audita process: diagnostics: %s\n", runDir.Path())
}
return 1
}
if runDir != nil && runOutput != nil {
if runOutput.Utilization != nil {
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactUtilizationSummary, runOutput.Utilization)
}
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactCorrectionLedger, processreport.BuildCorrectionLedger(processreport.CorrectionLedgerInput{
RunDirectoryPath: runDir.Path(),
RunOutput: runOutput,
}))
}
report := processreport.Build(processReportInput("success", inv, runDir, startedAt, completedAt, "", "", normSummary, chunkSummary, runOutput))
if strings.TrimSpace(inv.ReportJSONPath) != "" {
if err := reporting.WriteProcessReport(inv.ReportJSONPath, report); err != nil {
if runDir != nil {
_ = runDir.WriteErrorLog(fmt.Sprintf("report_write: %v", err))
_ = runDir.ApplyRetention(diagnostics.RetentionDecisionInput{
RunSucceeded: false,
})
}
fmt.Fprintf(stderr, "audita process: %v\n", err)
return 1
}
}
if runDir != nil {
_ = runDir.WriteReport(report)
if err := runDir.ApplyRetention(diagnostics.RetentionDecisionInput{
RunSucceeded: true,
HasSkippedCorrections: processreport.HasSkippedCorrections(runOutput),
}); err != nil {
fmt.Fprintf(stderr, "audita process: failed to apply work-dir retention: %v\n", err)
return 1
}
}
return 0
}
func runConfig(args []string, stdout, stderr io.Writer) int {
if len(args) == 0 || isHelpCommand(args) || hasHelpFlag(args) {
func runConfig(ctx context.Context, args []string, stdout, stderr io.Writer) int {
if len(args) == 0 || isHelpCommand(args) || args[0] == "--help" || args[0] == "-h" {
writeConfigUsage(stdout)
return 0
}
switch args[0] {
case "validate":
return runConfigValidate(args[1:], stdout, stderr)
return runConfigValidate(ctx, args[1:], stdout, stderr)
case "print-effective":
return runConfigPrintEffective(args[1:], stdout, stderr)
return runConfigPrintEffective(ctx, args[1:], stdout, stderr)
default:
fmt.Fprintf(stderr, "audita config: unknown command %q\n\n", args[0])
writeConfigUsage(stderr)
@@ -548,242 +80,75 @@ func runConfig(args []string, stdout, stderr io.Writer) int {
}
}
func runConfigValidate(args []string, stdout, stderr io.Writer) int {
fs := flag.NewFlagSet("config validate", flag.ContinueOnError)
fs.SetOutput(stderr)
configPath := fs.String("config", "", "Path to versioned YAML config file")
if err := fs.Parse(args); err != nil {
if errors.Is(err, flag.ErrHelp) {
writeConfigValidateUsage(stdout)
return 0
}
return 2
}
if strings.TrimSpace(*configPath) == "" {
fmt.Fprintln(stderr, "audita config validate: --config is required")
return 2
}
if len(fs.Args()) != 0 {
fmt.Fprintln(stderr, "audita config validate: unexpected positional arguments")
return 2
}
fileCfg, err := config.LoadFileConfig(strings.TrimSpace(*configPath))
func runConfigValidate(ctx context.Context, args []string, stdout, stderr io.Writer) int {
input, help, err := parseConfigInput("validate", args, stderr)
if err != nil {
fmt.Fprintf(stderr, "audita config validate: %v\n", err)
fmt.Fprintf(stderr, "audita config validate: %s\n", configInputError(err))
return 2
}
cfg := config.Default()
if err := cfg.ApplyFileConfig(fileCfg); err != nil {
fmt.Fprintf(stderr, "audita config validate: %v\n", err)
if help {
writeConfigValidateUsage(stdout)
return 0
}
selected, err := loadConfigInput(input, "validate")
if err != nil {
fmt.Fprintf(stderr, "audita config validate: %s\n", configInputError(err))
return 2
}
if err := cfg.Validate(); err != nil {
fmt.Fprintf(stderr, "audita config validate: %v\n", err)
if _, err := llm.NewRuntime(ctx, selected.Result.Config, selected.Path, nil); err != nil {
fmt.Fprintf(stderr, "audita config validate: %s\n", llm.RedactError(err, selected.Result.Config))
if llm.IsAssetError(err) {
return 1
}
return 2
}
fmt.Fprintln(stdout, "config is valid")
return 0
}
func runConfigPrintEffective(args []string, stdout, stderr io.Writer) int {
fs := flag.NewFlagSet("config print-effective", flag.ContinueOnError)
fs.SetOutput(stderr)
configPath := fs.String("config", "", "Path to versioned YAML config file")
if err := fs.Parse(args); err != nil {
if errors.Is(err, flag.ErrHelp) {
writeConfigPrintEffectiveUsage(stdout)
return 0
func runConfigPrintEffective(ctx context.Context, args []string, stdout, stderr io.Writer) int {
input, help, err := parseConfigInput("print-effective", args, stderr)
if err != nil {
fmt.Fprintf(stderr, "audita config print-effective: %s\n", configInputError(err))
return 2
}
if help {
writeConfigPrintEffectiveUsage(stdout)
return 0
}
selected, err := loadConfigInput(input, "print-effective")
if err != nil {
fmt.Fprintf(stderr, "audita config print-effective: %s\n", configInputError(err))
return 2
}
llmRuntime, err := llm.NewRuntime(ctx, selected.Result.Config, selected.Path, nil)
if err != nil {
fmt.Fprintf(stderr, "audita config print-effective: %s\n", llm.RedactError(err, selected.Result.Config))
if llm.IsAssetError(err) {
return 1
}
return 2
}
if len(fs.Args()) != 0 {
fmt.Fprintln(stderr, "audita config print-effective: unexpected positional arguments")
return 2
if input.View == "settings" {
err = selected.Result.WriteEffective(stdout)
} else {
var output []byte
if input.View == "profiles" {
output, err = llmRuntime.SelectedProfilesYAML()
} else {
output, err = llm.ExplanationYAML(selected.Result)
}
if err == nil {
_, err = stdout.Write(output)
}
}
configPathValue := strings.TrimSpace(*configPath)
configPathSet := configPathValue != ""
effectiveConfig, err := config.LoadEffectiveConfig(configPathValue, configPathSet)
if err != nil {
fmt.Fprintf(stderr, "audita config print-effective: %v\n", err)
return 2
}
cfg := effectiveConfig.Config
redacted := cfg.Redacted()
out, err := json.MarshalIndent(redacted, "", " ")
if err != nil {
fmt.Fprintf(stderr, "audita config print-effective: %v\n", err)
return 1
}
out = append(out, '\n')
if _, err := stdout.Write(out); err != nil {
fmt.Fprintf(stderr, "audita config print-effective: %v\n", err)
return 1
}
return 0
}
func extractErrorPhase(err error) (phase string, message string) {
msg := err.Error()
if strings.Contains(msg, ": ") {
parts := strings.SplitN(msg, ": ", 2)
if len(parts) == 2 {
return parts[0], parts[1]
}
}
return "", msg
}
func processReportInput(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) processreport.BuildInput {
runDirectoryPath := ""
if runDir != nil {
runDirectoryPath = runDir.Path()
}
return processreport.BuildInput{
Status: status,
TranscriptPath: inv.TranscriptPath,
GlossaryPath: inv.GlossaryPath,
OutputPath: inv.OutputPath,
Modules: inv.Config.Modules,
OutputSchema: inv.Config.OutputSchema,
ConfigVersion: inv.ConfigVersion,
StartedAt: startedAt,
CompletedAt: completedAt,
ErrorMessage: errorMessage,
ErrorPhase: errorPhase,
RunDirectoryPath: runDirectoryPath,
NormalizationSummary: normalizationSummary,
ChunkingSummary: chunkingSummary,
RunOutput: runOutput,
}
}
type processFlags struct {
configPath *string
glossaryPath *string
outputPath *string
reportJSONPath *string
modules *string
outputSchema *string
llmAPIKey *string
validationLLMAPIKey *string
model *string
validationModel *string
baseURL *string
validationBaseURL *string
llmTimeoutSeconds *int
totalLLMConcurrency *int
proposalLLMConcurrency *int
llmConcurrency *int
validationLLMTimeoutSeconds *int
validationMaxPromptTokens *int
targetSections *int
maxRetries *int
validationMaxRetries *int
validationLLMConcurrency *int
maxSectionTokens *int
minSectionTokens *int
glossaryConfidenceThreshold *float64
grammarConfidenceThreshold *float64
homophonesConfidenceThreshold *float64
spokenWordConfidenceThreshold *float64
normalizeMaxSegmentGap *float64
normalizeEllipsisGap *float64
normalizeMaxSegmentDuration *float64
normalizeMaxSegmentTokens *int
transcriptDescription *string
workDir *string
workDirRetention *string
}
func newProcessFlagSet(cfg config.Config, stderr io.Writer) (*flag.FlagSet, processFlags) {
fs := flag.NewFlagSet("process", flag.ContinueOnError)
fs.SetOutput(stderr)
validationTimeoutSecondsDefault := cfg.PrimaryLLM.TimeoutSeconds
if cfg.ValidationLLM.TimeoutSeconds != nil {
validationTimeoutSecondsDefault = *cfg.ValidationLLM.TimeoutSeconds
}
validationMaxRetriesDefault := cfg.PrimaryLLM.MaxRetries
if cfg.ValidationLLM.MaxRetries != nil {
validationMaxRetriesDefault = *cfg.ValidationLLM.MaxRetries
}
validationLLMConcurrencyDefault := cfg.TotalLLMConcurrency
if cfg.ValidationLLMConcurrency != nil {
validationLLMConcurrencyDefault = *cfg.ValidationLLMConcurrency
}
targetSectionsDefault := 0
if cfg.TargetSections != nil {
targetSectionsDefault = *cfg.TargetSections
}
pFlags := processFlags{
configPath: fs.String("config", "", "Path to versioned YAML config file"),
glossaryPath: fs.String("glossary", "", "Path to glossary YAML file"),
outputPath: fs.String("output", "", "Path to corrected transcript JSON output file"),
reportJSONPath: fs.String("report-json", "", "Path to machine-readable report JSON output file"),
modules: fs.String("modules", strings.Join(cfg.Modules, ","), "Comma-separated module sequence override"),
outputSchema: fs.String("output-schema", cfg.OutputSchema, "Output schema: bare-segments|audita-v1"),
llmAPIKey: fs.String("llm-api-key", cfg.PrimaryLLM.APIKey, "Primary LLM API key"),
validationLLMAPIKey: fs.String("validation-llm-api-key", cfg.ValidationLLM.APIKey, "Validation LLM API key"),
model: fs.String("model", cfg.PrimaryLLM.Model, "Primary LLM model name"),
validationModel: fs.String("validation-model", cfg.ValidationLLM.Model, "Validation LLM model name"),
baseURL: fs.String("base-url", cfg.PrimaryLLM.BaseURL, "Primary OpenAI-compatible base URL"),
validationBaseURL: fs.String("validation-base-url", cfg.ValidationLLM.BaseURL, "Validation OpenAI-compatible base URL"),
llmTimeoutSeconds: fs.Int("llm-timeout-seconds", cfg.PrimaryLLM.TimeoutSeconds, "Primary LLM timeout in seconds"),
totalLLMConcurrency: fs.Int("total-llm-concurrency", cfg.TotalLLMConcurrency, "Total concurrent LLM calls across proposal and validation"),
proposalLLMConcurrency: fs.Int("proposal-llm-concurrency", cfg.EffectiveProposalLLMConcurrency(), "Concurrent proposal-generation LLM calls"),
llmConcurrency: fs.Int("llm-concurrency", cfg.TotalLLMConcurrency, "Alias for --total-llm-concurrency"),
validationLLMTimeoutSeconds: fs.Int("validation-llm-timeout-seconds", validationTimeoutSecondsDefault, "Validation LLM timeout in seconds"),
validationMaxPromptTokens: fs.Int("validation-max-prompt-tokens", cfg.ValidationMaxPromptTokens, "Validation max prompt tokens"),
targetSections: fs.Int("target-sections", targetSectionsDefault, "Target number of transcript sections"),
maxRetries: fs.Int("max-retries", cfg.PrimaryLLM.MaxRetries, "Maximum structured-output retries"),
validationMaxRetries: fs.Int("validation-max-retries", validationMaxRetriesDefault, "Validation structured-output retries"),
validationLLMConcurrency: fs.Int("validation-llm-concurrency", validationLLMConcurrencyDefault, "Concurrent validation LLM calls (inherits total when unset)"),
maxSectionTokens: fs.Int("max-section-tokens", cfg.MaxSectionTokens, "Maximum section tokens"),
minSectionTokens: fs.Int("min-section-tokens", cfg.MinSectionTokens, "Minimum section tokens"),
glossaryConfidenceThreshold: fs.Float64("glossary-confidence-threshold", cfg.Thresholds.Glossary, "Glossary confidence threshold"),
grammarConfidenceThreshold: fs.Float64("grammar-confidence-threshold", cfg.Thresholds.Grammar, "Grammar confidence threshold"),
homophonesConfidenceThreshold: fs.Float64("homophones-confidence-threshold", cfg.Thresholds.Homophones, "Homophones confidence threshold"),
spokenWordConfidenceThreshold: fs.Float64("spoken-word-confidence-threshold", cfg.Thresholds.SpokenWord, "Spoken-word confidence threshold"),
normalizeMaxSegmentGap: fs.Float64("normalize-max-segment-gap", cfg.Normalization.MaxSegmentGap, "Maximum same-speaker merge gap"),
normalizeEllipsisGap: fs.Float64("normalize-ellipsis-gap", cfg.Normalization.EllipsisGap, "Gap threshold for ellipsis insertion"),
normalizeMaxSegmentDuration: fs.Float64("normalize-max-segment-duration", cfg.Normalization.MaxSegmentDuration, "Maximum merged segment duration"),
normalizeMaxSegmentTokens: fs.Int("normalize-max-segment-tokens", cfg.Normalization.MaxSegmentTokens, "Maximum merged segment token estimate"),
transcriptDescription: fs.String("transcript-description", cfg.TranscriptDescription, "Brief background context for LLM prompts; does not override transcript content"),
workDir: fs.String("work-dir", cfg.WorkDir, "Per-run work directory"),
workDirRetention: fs.String("work-dir-retention", string(cfg.WorkDirRetention), "Work-dir retention policy: auto|always|never"),
}
return fs, pFlags
}
func findConfigPathOverride(args []string) (path string, set bool, err error) {
for i := 0; i < len(args); i++ {
arg := strings.TrimSpace(args[i])
if arg == "" {
continue
}
if arg == "--config" {
if i+1 >= len(args) {
return "", false, fmt.Errorf("--config requires a path")
}
return strings.TrimSpace(args[i+1]), true, nil
}
if strings.HasPrefix(arg, "--config=") {
return strings.TrimSpace(strings.TrimPrefix(arg, "--config=")), true, nil
}
}
return "", false, nil
}
func isHelpCommand(args []string) bool {
if len(args) == 0 {
return false
@@ -803,15 +168,6 @@ func isHelpCommand(args []string) bool {
return false
}
func hasHelpFlag(args []string) bool {
for _, arg := range args {
if arg == "-h" || arg == "--help" {
return true
}
}
return false
}
func writeRootUsage(w io.Writer) {
fmt.Fprintln(w, "Audita is a transcript processing CLI.")
fmt.Fprintln(w)
@@ -834,33 +190,43 @@ func writeConfigUsage(w io.Writer) {
fmt.Fprintln(w)
fmt.Fprintln(w, "Commands:")
fmt.Fprintln(w, " validate Validate a versioned YAML config file")
fmt.Fprintln(w, " print-effective Print redacted effective config JSON (defaults + config file + env)")
fmt.Fprintln(w, " print-effective Print redacted effective settings YAML")
}
func writeConfigValidateUsage(w io.Writer) {
fmt.Fprintln(w, "Validate a versioned YAML config file.")
fmt.Fprintln(w)
fmt.Fprintln(w, "Usage:")
fmt.Fprintln(w, " audita config validate --config <path>")
fmt.Fprintln(w, " audita config validate --config <path> [--overlay <name>] [setting flags]")
}
func writeConfigPrintEffectiveUsage(w io.Writer) {
fmt.Fprintln(w, "Print redacted effective config JSON.")
fmt.Fprintln(w, "Print redacted effective settings YAML.")
fmt.Fprintln(w)
fmt.Fprintln(w, "Usage:")
fmt.Fprintln(w, " audita config print-effective [--config <path>]")
fmt.Fprintln(w, " audita config print-effective [--config <path>] [--overlay <name>] [setting flags]")
}
func writeProcessUsage(w io.Writer, fs *flag.FlagSet) {
func writeProcessUsage(w io.Writer) {
fmt.Fprintln(w, "Process a transcript JSON file.")
fmt.Fprintln(w)
fmt.Fprintln(w, "Usage:")
fmt.Fprintln(w, " audita process <transcript.json> [flags]")
fmt.Fprintln(w)
fmt.Fprintln(w, "Flags:")
fs.VisitAll(func(f *flag.Flag) {
fmt.Fprintf(w, " --%s\n", f.Name)
})
for _, name := range []string{
"config", "overlay", "glossary", "output",
"proposal-profile", "validation-profile", "modules", "output-schema",
"validation-max-prompt-tokens", "target-sections", "max-section-tokens", "min-section-tokens",
"normalize-max-segment-gap", "normalize-ellipsis-gap", "normalize-max-segment-duration", "normalize-max-segment-tokens",
"glossary-confidence-threshold", "homophones-confidence-threshold", "spoken-word-confidence-threshold", "grammar-confidence-threshold",
"transcript-description", "work-dir",
} {
fmt.Fprintf(w, " --%s\n", name)
}
fmt.Fprintln(w, " --reporting[=true|false] Write the three-file report bundle (default true)")
fmt.Fprintln(w, " --report-dir DIR Report directory for stdout output only; ignored with --output or --reporting=false")
fmt.Fprintln(w, " --debug[=true|false] Persist detailed diagnostics (default false)")
fmt.Fprintln(w)
fmt.Fprintln(w, "Example:")
fmt.Fprintln(w, " audita process transcript.json --glossary glossary.yaml --output corrected.json")

File diff suppressed because it is too large Load Diff

View File

@@ -1,110 +0,0 @@
package cli
import (
"context"
"errors"
"os"
"strconv"
"strings"
"sync"
"time"
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
)
const (
subprocessTestLLMModeEnv = "AUDITA_SUBPROCESS_TEST_LLM_MODE"
subprocessTestRunTimeoutMSEnv = "AUDITA_SUBPROCESS_TEST_RUN_TIMEOUT_MS"
)
// ConfigureSubprocessTestHooksFromEnv enables deterministic test-only hooks for
// subprocess integration tests that run through the Go test binary helper path.
func ConfigureSubprocessTestHooksFromEnv() {
mode := strings.TrimSpace(os.Getenv(subprocessTestLLMModeEnv))
timeoutMSRaw := strings.TrimSpace(os.Getenv(subprocessTestRunTimeoutMSEnv))
// Only activate in explicit subprocess test mode.
if mode == "" && timeoutMSRaw == "" {
return
}
if mode != "" {
client := &subprocessTestLLMClient{mode: mode}
processProposalLLMClient = client
processValidationLLMClient = client
}
if timeoutMSRaw == "" {
return
}
timeoutMS, err := strconv.Atoi(timeoutMSRaw)
if err != nil || timeoutMS <= 0 {
return
}
processRunnerContext = func() (context.Context, context.CancelFunc) {
return context.WithTimeout(context.Background(), time.Duration(timeoutMS)*time.Millisecond)
}
}
type subprocessTestLLMClient struct {
mode string
mu sync.Mutex
proposals int
}
func (c *subprocessTestLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
_ = req
switch c.mode {
case "backend_error":
return contracts.StructuredCompletionResponse{}, errors.New("synthetic backend failure")
case "block_until_cancel":
<-ctx.Done()
return contracts.StructuredCompletionResponse{}, ctx.Err()
case "malformed_structured":
switch target := out.(type) {
case *proposal_generation.StructuredCorrectionSet:
*target = proposal_generation.StructuredCorrectionSet{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 0, OriginalText: "x", CorrectedText: "y", Confidence: 0.99},
},
}
case *validators.LLMValidationResponse:
*target = validators.LLMValidationResponse{
Validations: []validators.LLMValidationDecision{
{CorrectionIndex: 999, Approved: true, Confidence: 0.9, Reason: "bad index"},
},
}
}
case "mid_pipeline_fail":
if _, ok := out.(*proposal_generation.StructuredCorrectionSet); ok {
c.mu.Lock()
c.proposals++
proposalCall := c.proposals
c.mu.Unlock()
if proposalCall >= 3 {
return contracts.StructuredCompletionResponse{}, errors.New("synthetic mid-pipeline failure")
}
}
switch target := out.(type) {
case *proposal_generation.StructuredCorrectionSet:
*target = proposal_generation.StructuredCorrectionSet{
Corrections: []proposal_generation.StructuredCorrectionProposal{
{TargetSegmentID: 1, OriginalText: "Segment", CorrectedText: "Segment", Confidence: 0.99},
},
}
case *validators.LLMValidationResponse:
*target = validators.LLMValidationResponse{Validations: nil}
}
default:
switch target := out.(type) {
case *proposal_generation.StructuredCorrectionSet:
*target = proposal_generation.StructuredCorrectionSet{Corrections: nil}
case *validators.LLMValidationResponse:
*target = validators.LLMValidationResponse{Validations: nil}
}
}
return contracts.StructuredCompletionResponse{}, nil
}

View File

@@ -1,22 +0,0 @@
{
"name": "ambiguous_match_rejected_before_application",
"transcript_file": "application-skip-ambiguous.transcript.json",
"glossary_file": "default-handoff.glossary.yaml",
"modules_csv": "homophones",
"proposal_responses_file": "application-skip-ambiguous.proposals.json",
"validation_responses_file": "application-skip-ambiguous.validations.json",
"expect": {
"exit_code": 0,
"status": "success",
"output_transcript_file": "application-skip-ambiguous.expected-transcript.json",
"module_instances": ["homophones"],
"module_count": 1,
"total_applied_changes": 0,
"total_skipped_changes": 1,
"module_applied_counts": [0],
"module_rejected_counts": [1],
"module_skip_counts": [0],
"validator_rejected_reason_codes": ["ambiguous_original_text"],
"expected_proposal_calls": ["homophones:proposal"]
}
}

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"Alice","start":0,"end":1,"text":"the site near another site"}
]

View File

@@ -1,3 +0,0 @@
[
{"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]}
]

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"the site near another site"}
]

View File

@@ -1,4 +0,0 @@
[
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
]

View File

@@ -1,28 +0,0 @@
{
"name": "default_full_pipeline_shape_and_reports",
"transcript_file": "default-full-pipeline.transcript.json",
"glossary_file": "default-full-pipeline.glossary.yaml",
"proposal_responses_file": "default-full-pipeline.proposals.json",
"validation_responses_file": "default-full-pipeline.validations.json",
"env": {
"AUDITA_LLM_API_KEY": "parity-secret",
"AUDITA_VALIDATION_LLM_API_KEY": "parity-secret"
},
"expect": {
"exit_code": 0,
"status": "success",
"output_transcript_file": "default-full-pipeline.expected-transcript.json",
"module_instances": ["glossary_1", "homophones", "glossary_2", "spoken_word", "grammar"],
"module_count": 5,
"total_applied_changes": 3,
"total_skipped_changes": 3,
"secret_markers": ["parity-secret"],
"expected_proposal_calls": [
"glossary_1:proposal",
"homophones:proposal",
"glossary_2:proposal",
"spoken_word:proposal",
"grammar:proposal"
]
}
}

View File

@@ -1,9 +0,0 @@
[
{
"id": 1,
"speaker": "Alice",
"start": 0,
"end": 1,
"text": "Hello, there were Jesters hmm"
}
]

View File

@@ -1,7 +0,0 @@
glossary:
- name: Jesters
aliases:
- jester
plural: jesters
category: faction
summary: A protected in-world faction term.

View File

@@ -1,28 +0,0 @@
[
{
"corrections": [
{"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99}
]
},
{
"corrections": [
{"id": 1, "original_text": "Jesters", "corrected_text": "jesters", "confidence": 0.99},
{"id": 1, "original_text": "Jesters", "corrected_text": "JESTERX", "confidence": 0.99}
]
},
{
"corrections": [
{"id": 1, "original_text": "jesters", "corrected_text": "JESTERS", "confidence": 0.99}
]
},
{
"corrections": [
{"id": 1, "original_text": "uh", "corrected_text": "hmm", "confidence": 0.99}
]
},
{
"corrections": [
{"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99}
]
}
]

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , there were gestures uh"}
]

View File

@@ -1,12 +0,0 @@
[
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": false, "confidence": 0.99, "reason": "reject cleanup"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "parity-secret"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
]

View File

@@ -1,27 +0,0 @@
{
"name": "default_pipeline_handoff_and_module_order",
"transcript_file": "default-handoff.transcript.json",
"glossary_file": "default-handoff.glossary.yaml",
"proposal_responses_file": "default-handoff.proposals.json",
"validation_responses_file": "default-handoff.validations.json",
"expect": {
"exit_code": 0,
"status": "success",
"output_transcript_file": "default-handoff.expected-transcript.json",
"module_instances": ["glossary_1", "homophones", "glossary_2", "spoken_word", "grammar"],
"module_count": 5,
"total_applied_changes": 5,
"total_skipped_changes": 0,
"module_applied_counts": [1, 1, 1, 1, 1],
"module_rejected_counts": [0, 0, 0, 0, 0],
"module_skip_counts": [0, 0, 0, 0, 0],
"expected_proposal_calls": [
"glossary_1:proposal",
"homophones:proposal",
"glossary_2:proposal",
"spoken_word:proposal",
"grammar:proposal"
],
"min_response_payload_artifacts": 15
}
}

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"Alice","start":0,"end":1,"text":"Hello, there were Jesters at the Sight um"}
]

View File

@@ -1,6 +0,0 @@
glossary:
- name: Jesters
aliases: [jester]
plural: jesters
category: faction
summary: A protected in-world faction term.

View File

@@ -1,7 +0,0 @@
[
{"corrections": [{"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99}]},
{"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]},
{"corrections": [{"id": 1, "original_text": "sight", "corrected_text": "Sight", "confidence": 0.99}]},
{"corrections": [{"id": 1, "original_text": "um um", "corrected_text": "um", "confidence": 0.99}]},
{"corrections": [{"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99}]}
]

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , there were gestures at the site um um"}
]

View File

@@ -1,12 +0,0 @@
[
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
]

View File

@@ -1,17 +0,0 @@
{
"name": "deterministic_validator_low_confidence",
"transcript_file": "deterministic-validator-low-confidence.transcript.json",
"glossary_file": "default-full-pipeline.glossary.yaml",
"modules_csv": "grammar",
"proposal_responses_file": "deterministic-validator-low-confidence.proposals.json",
"expect": {
"exit_code": 0,
"status": "success",
"output_transcript_file": "deterministic-validator-low-confidence.expected-transcript.json",
"module_instances": ["grammar"],
"module_count": 1,
"total_skipped_changes": 1,
"validator_rejected_reason_codes": ["low_confidence"],
"expected_proposal_calls": ["grammar:proposal"]
}
}

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"Alice","start":0,"end":1,"text":"hello , world"}
]

View File

@@ -1,7 +0,0 @@
[
{
"corrections": [
{"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.1}
]
}
]

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , world"}
]

View File

@@ -1,12 +0,0 @@
{
"name": "glossary_schema_handling",
"transcript_file": "default-full-pipeline.transcript.json",
"glossary_file": "glossary-schema-error.glossary.yaml",
"expect": {
"exit_code": 1,
"status": "failed",
"error_phase": "glossary_schema",
"stderr_contains": "glossary_schema",
"require_error_log": true
}
}

View File

@@ -1,2 +0,0 @@
glossary:
- name: MissingCategoryAndSummary

View File

@@ -1,19 +0,0 @@
{
"name": "llm_validator_decision_handling",
"transcript_file": "llm-validator-rejection.transcript.json",
"glossary_file": "default-full-pipeline.glossary.yaml",
"modules_csv": "grammar",
"proposal_responses_file": "llm-validator-rejection.proposals.json",
"validation_responses_file": "llm-validator-rejection.validations.json",
"expect": {
"exit_code": 0,
"status": "success",
"output_transcript_file": "llm-validator-rejection.expected-transcript.json",
"module_instances": ["grammar"],
"module_count": 1,
"total_skipped_changes": 1,
"validator_rejected_reason_codes": ["llm_rejected"],
"expected_proposal_calls": ["grammar:proposal"],
"expected_validation_calls": ["grammar:section-0000:editorial_review:batch-0000"]
}
}

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"Alice","start":0,"end":1,"text":"hello , world"}
]

View File

@@ -1,7 +0,0 @@
[
{
"corrections": [
{"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99}
]
}
]

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , world"}
]

View File

@@ -1,7 +0,0 @@
[
{
"validations": [
{"correction_index": 0, "approved": false, "confidence": 0.99, "reason": "reject stylistic overreach"}
]
}
]

View File

@@ -1,23 +0,0 @@
{
"name": "mid_pipeline_failure_partial_progress",
"transcript_file": "default-handoff.transcript.json",
"glossary_file": "default-handoff.glossary.yaml",
"proposal_responses_file": "mid-pipeline-failure.proposals.json",
"validation_responses_file": "mid-pipeline-failure.validations.json",
"expect": {
"exit_code": 1,
"status": "failed",
"error_phase": "runner_execution",
"stderr_contains": "runner_execution",
"module_instances": ["glossary_1", "homophones", "glossary_2"],
"module_count": 3,
"total_applied_changes": 2,
"total_skipped_changes": 0,
"failed_module_instance": "glossary_2",
"module_applied_counts": [1, 1, 0],
"module_rejected_counts": [0, 0, 0],
"module_skip_counts": [0, 0, 0],
"require_error_log": true,
"expected_proposal_calls": ["glossary_1:proposal", "homophones:proposal", "glossary_2:proposal"]
}
}

View File

@@ -1,4 +0,0 @@
[
{"corrections": [{"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99}]},
{"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]}
]

View File

@@ -1,6 +0,0 @@
[
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
]

View File

@@ -1,21 +0,0 @@
{
"name": "protected_glossary_term_behavior",
"transcript_file": "protected-term-rejection.transcript.json",
"glossary_file": "default-handoff.glossary.yaml",
"modules_csv": "homophones",
"proposal_responses_file": "protected-term-rejection.proposals.json",
"expect": {
"exit_code": 0,
"status": "success",
"output_transcript_file": "protected-term-rejection.expected-transcript.json",
"module_instances": ["homophones"],
"module_count": 1,
"total_applied_changes": 0,
"total_skipped_changes": 1,
"module_applied_counts": [0],
"module_rejected_counts": [1],
"module_skip_counts": [0],
"validator_rejected_reason_codes": ["protected_glossary_term"],
"expected_proposal_calls": ["homophones:proposal"]
}
}

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"Alice","start":0,"end":1,"text":"The Jesters entered the hall."}
]

View File

@@ -1,3 +0,0 @@
[
{"corrections": [{"id": 1, "original_text": "Jesters", "corrected_text": "Gestures", "confidence": 0.99}]}
]

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"The Jesters entered the hall."}
]

View File

@@ -1,12 +0,0 @@
{
"name": "transcript_schema_handling",
"transcript_file": "transcript-schema-error.transcript.json",
"glossary_file": "default-full-pipeline.glossary.yaml",
"expect": {
"exit_code": 1,
"status": "failed",
"error_phase": "transcript_schema",
"stderr_contains": "transcript_schema",
"require_error_log": true
}
}

View File

@@ -1,3 +0,0 @@
[
{"id":1,"speaker":"","start":0.0,"end":1.0,"text":"bad"}
]

View File

@@ -1,24 +0,0 @@
{
"must_apply_texts": [
"Hello, there were Jesters hmm"
],
"must_not_apply_texts": [
"JESTERX",
"there were gestures"
],
"protected_terms": [
"Jesters"
],
"expected_module_instances": [
"glossary_1",
"homophones",
"glossary_2",
"spoken_word",
"grammar"
],
"minimum_counts": {
"applied": 1,
"rejected": 1,
"skipped": 0
}
}

View File

@@ -1,9 +0,0 @@
[
{
"id": 1,
"speaker": "Alice",
"start": 0,
"end": 1,
"text": "Hello, there were Jesters hmm"
}
]

View File

@@ -1,7 +0,0 @@
glossary:
- name: Jesters
aliases:
- jester
plural: jesters
category: faction
summary: A protected in-world faction term.

Some files were not shown because too many files have changed in this diff Show More