Compare commits
72 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 4cf1b6fbca | |||
| 07a4b65570 | |||
| 5feec1bf14 | |||
| 011b021d47 | |||
| 325c4ea7d3 | |||
| 3eb73e70fc | |||
| b8089f4a50 | |||
| 22f0932389 | |||
| d726ea8f6d | |||
| 7199cb565f | |||
| 60e2fe74b5 | |||
| d2a6fcfd41 | |||
| 5e2e863fbd | |||
| 24b899281c | |||
| 2a58db04c2 | |||
| 93f62c9095 | |||
| 640f663ace | |||
| 31979d4081 | |||
| f219309617 | |||
| a83a51848b | |||
| 0b6cd98262 | |||
| efaefdbf3d | |||
| ec78b5a5db | |||
| 9fc4de22b7 | |||
| 6523d7ac74 | |||
| a770c77a23 | |||
| 2f036b0356 | |||
| 87b8d9ad5c | |||
| a80eed4745 | |||
| 989b28e8c5 | |||
| 61af29ecb8 | |||
| 63fcfc487d | |||
| b98604e5ca | |||
| bb21a81805 | |||
| 0a19f163c6 | |||
| 42bde59827 | |||
| 51169ff9e9 | |||
| 7dc1be5a08 | |||
| 8ffbf41b89 | |||
| cb405f6fc7 | |||
| 053b714ce6 | |||
| aa2f486845 | |||
| 665ee85215 | |||
| c42bc01052 | |||
| 7cefb3fb04 | |||
| 80804a038f | |||
| 9561e80efb | |||
| 4bfc5f3456 | |||
| 4812032c9e | |||
| a4a72168d5 | |||
| f577847ca2 | |||
| c8a78bf024 | |||
| 96d3d03bee | |||
| f8c18247a0 | |||
| 51e370b4c8 | |||
| 9fb89057ce | |||
| 1ad2119283 | |||
| 116648fa78 | |||
| 5b7d2e6e81 | |||
| 590604bf38 | |||
| 7b809878d6 | |||
| 228ac4e26d | |||
| 2d8b441802 | |||
| 2c55bf5f1e | |||
| c97a6fa70a | |||
| 187dc4cb1c | |||
| 228087dec8 | |||
| f70bb27636 | |||
| a379cc915e | |||
| 837efc78a7 | |||
| 6b33254a82 | |||
| e8bc005040 |
@@ -3,7 +3,7 @@ when:
|
||||
|
||||
steps:
|
||||
- name: build-release-assets
|
||||
image: golang:1.25
|
||||
image: golang:1.25.5
|
||||
commands:
|
||||
- bash scripts/build-release.sh "$CI_COMMIT_TAG" dist
|
||||
|
||||
|
||||
32
README.md
32
README.md
@@ -6,19 +6,27 @@ reports for review and subprocess integration.
|
||||
|
||||
## Quickstart
|
||||
|
||||
From a checkout, build with Go 1.24 or newer:
|
||||
From a checkout, build with Go 1.25.5 or newer:
|
||||
|
||||
```sh
|
||||
go build -o ./bin/audita ./cmd/audita
|
||||
```
|
||||
|
||||
Processing requires a reachable [compatible LLM endpoint](docs/integrations/openai-compatible-llm.md)
|
||||
and access to its configured model. Supply its credentials through the
|
||||
[configuration mechanisms](docs/config.md#secrets); the
|
||||
[minimal config](examples/minimal-config.yml) reads `AUDITA_LLM_API_KEY` and uses
|
||||
the [default model and endpoint](docs/config.md#llm-settings). Set that variable
|
||||
in your environment before using an endpoint that requires authentication.
|
||||
For another endpoint or model, edit a copy of the config.
|
||||
and access to a model you select. The [minimal config](examples/minimal-config.yml)
|
||||
targets an unauthenticated local endpoint; edit its endpoint and
|
||||
[local profiles](examples/profiles/) to match your service and models. The
|
||||
[service example](examples/production-config.yml) demonstrates a named credential
|
||||
reference and a shared backend. Set its referenced environment variable before
|
||||
validating it. The [testing overlay](examples/overlays/testing.yml) selects both
|
||||
role profiles, but model cost, speed, and quality depend on your provider choices.
|
||||
|
||||
Check configuration and profile selection offline:
|
||||
|
||||
```sh
|
||||
./bin/audita config validate --config examples/minimal-config.yml
|
||||
./bin/audita config print-effective --config examples/minimal-config.yml --view profiles
|
||||
```
|
||||
|
||||
Run the included small input with the minimal config:
|
||||
|
||||
@@ -29,8 +37,14 @@ Run the included small input with the minimal config:
|
||||
--output corrected.json
|
||||
```
|
||||
|
||||
This makes LLM requests. Inherited configuration environment overrides still
|
||||
apply; see [precedence](docs/config.md#effective-precedence).
|
||||
This makes LLM requests. Explicit setting flags override the selected file and
|
||||
optional overlay; general environment setting overrides are no longer read.
|
||||
By default the command writes `errors.yml`, `recoveries.yml`, and `report.md`
|
||||
beside `corrected.json`. Use `--reporting=false` to suppress those files, or
|
||||
`--debug` to retain detailed run artifacts under the configured debug work
|
||||
directory. Model failures affecting some operations can still yield a completed
|
||||
transcript with applied corrections; the summary and reports identify them.
|
||||
See the [v2 migration guide](docs/migrations/v2.md) for old configurations.
|
||||
|
||||
## Documentation
|
||||
|
||||
|
||||
@@ -1,11 +1,20 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/cli"
|
||||
)
|
||||
|
||||
func main() {
|
||||
os.Exit(cli.Run(os.Args[1:], os.Stdout, os.Stderr))
|
||||
os.Exit(run())
|
||||
}
|
||||
|
||||
func run() int {
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
return cli.Run(ctx, os.Args[1:], os.Stdout, os.Stderr)
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
261
docs/cli.md
261
docs/cli.md
@@ -1,198 +1,95 @@
|
||||
# Audita CLI Reference
|
||||
|
||||
## Shortest Useful Command
|
||||
The shared version-2 configuration parser and offline profile inspection serve
|
||||
`process`, `config validate`, and `config print-effective`.
|
||||
The [configuration reference](config.md) owns fields, defaults, and validation;
|
||||
the [migration guide](migrations/v2.md) maps retired inputs.
|
||||
|
||||
## Commands and help
|
||||
|
||||
```sh
|
||||
audita process transcript.json --glossary glossary.yaml --output corrected.json
|
||||
audita process <transcript.json> --glossary <glossary.yml> [flags]
|
||||
audita config validate --config <config.yml> [flags]
|
||||
audita config print-effective [flags]
|
||||
```
|
||||
|
||||
This command runs the core workflow with your input files. For a copyable example
|
||||
using maintained inputs and explicit prerequisites, see the [README](../README.md).
|
||||
Commands here assume `audita` is on `PATH`; use `./bin/audita` after a local build.
|
||||
Root invocation, `--help`, `-h`, and command help succeed without loading a config
|
||||
file or resolving secrets. Unknown commands and flags return exit code `2`.
|
||||
`process` accepts exactly one transcript path, either before the flags or after
|
||||
them. With flags first, put the transcript last. Both config commands reject
|
||||
positional arguments and process-only flags.
|
||||
|
||||
## Command Overview
|
||||
All three commands accept `--config <path>` and `--overlay <name>`. `config validate`
|
||||
requires an explicit nonempty `--config`; the other commands select the explicit
|
||||
file, then `/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`.
|
||||
Missing system candidates are skipped; inaccessible selected files are errors.
|
||||
No file means a configuration error. `AUDITA_CONFIG` is ignored. The selected
|
||||
overlay is loaded beside the main file, not used to locate profile files.
|
||||
|
||||
- `audita process`: process one transcript JSON file.
|
||||
- `audita config validate`: validate a versioned YAML config file.
|
||||
- `audita config print-effective`: print redacted effective config JSON.
|
||||
## Setting overrides
|
||||
|
||||
General help:
|
||||
Only flags actually supplied override file or overlay settings. `--modules ''`
|
||||
clears the module pipeline, and an explicit numeric zero reaches configuration
|
||||
validation. The shared setting flags are:
|
||||
|
||||
```sh
|
||||
audita --help
|
||||
audita process --help
|
||||
audita config --help
|
||||
```
|
||||
| Setting | Flags |
|
||||
| --- | --- |
|
||||
| Profile selection | `--proposal-profile`, `--validation-profile` |
|
||||
| Pipeline and output | `--modules`, `--output-schema` |
|
||||
| Validation budget | `--validation-max-prompt-tokens` |
|
||||
| Chunking | `--target-sections`, `--max-section-tokens`, `--min-section-tokens` |
|
||||
| Normalization (numeric seconds) | `--normalize-max-segment-gap`, `--normalize-ellipsis-gap`, `--normalize-max-segment-duration`, `--normalize-max-segment-tokens` |
|
||||
| Confidence thresholds | `--glossary-confidence-threshold`, `--homophones-confidence-threshold`, `--spoken-word-confidence-threshold`, `--grammar-confidence-threshold` |
|
||||
| Context | `--transcript-description` |
|
||||
| Reporting | `--reporting[=true\|false]`, `--report-dir DIR` |
|
||||
| Debug | `--debug[=true\|false]`, `--work-dir DIR` |
|
||||
|
||||
## `process`
|
||||
`process` alone accepts `--glossary` (required) and `--output`.
|
||||
An omitted or empty output value selects stdout. Nonempty output filenames are
|
||||
used literally, including whitespace.
|
||||
Reporting defaults to enabled. `--reporting=false` disables all three ordinary
|
||||
reports; an explicit false overrides a true file or overlay setting. With file
|
||||
output, reports go beside the transcript and `--report-dir` has no effect. With
|
||||
stdout output, `--report-dir` selects the report directory, defaulting to the
|
||||
invocation working directory. `--debug` separately enables retained detailed
|
||||
artifacts under `--work-dir`. Configuration inspection does not create or probe
|
||||
these output directories.
|
||||
Per-role model, base URL, key, timeout, retry, and concurrency flags have been
|
||||
removed. Models and generation controls belong to PromptKit profiles, backend
|
||||
credentials and capacity to configuration, and retry/deadline settings to the
|
||||
shared LLM section. Legacy configuration environment overrides are ignored.
|
||||
|
||||
Usage:
|
||||
## Streams and exit status
|
||||
|
||||
```sh
|
||||
audita process <transcript.json> [flags]
|
||||
```
|
||||
`config validate` loads settings and both selected PromptKit profiles offline,
|
||||
then prints `config is valid` on success. `config print-effective` accepts
|
||||
`--view settings|profiles|explain` and defaults to `settings`. Settings writes
|
||||
ConfigKit's redacted effective-settings YAML. Profiles writes an Audita-owned
|
||||
YAML summary of selected ID, backend, model, sampling controls, per-attempt
|
||||
timeout, service tier, and reasoning effort for each role. It omits credentials,
|
||||
endpoints, and arbitrary extra provider parameters. Explain writes a YAML
|
||||
sequence of field paths, source origins, and recorded derivations without
|
||||
resolved setting values. These views share the same settings and profile
|
||||
validation path as `process`. Unknown views are usage errors. The effective
|
||||
settings YAML is for inspection and is not guaranteed reloadable.
|
||||
|
||||
Input requirements:
|
||||
- exactly one transcript JSON positional argument is required;
|
||||
- `--glossary <path>` is required.
|
||||
For `process`, `--output` writes the corrected transcript to a file and leaves
|
||||
stdout empty on success. Without it, stdout carries transcript JSON only on
|
||||
success. Enabled reporting creates `errors.yml`, `recoveries.yml`, and
|
||||
`report.md` in the selected directory. Successful stderr begins with
|
||||
`Completed: N corrections applied; E work units with errors; R recoveries.`
|
||||
Report and debug locations follow only when created. Model-operation failures
|
||||
can coexist with a completed run and applied corrections; read the YAML reports
|
||||
for details. Debug artifacts are created only with `--debug`. See the
|
||||
[report contract](integrations/output-files.md).
|
||||
|
||||
Place the transcript path first, followed by flags, as shown above. Flag values
|
||||
may use `--name value` or `--name=value`. Unknown flags or extra positional
|
||||
arguments are errors. Flags before a final transcript path are also accepted;
|
||||
do not interleave further flags after a positional argument consumed by the flag
|
||||
parser.
|
||||
Exit codes are `0` for success, `1` for runtime or output failure, and `2` for
|
||||
usage or configuration input errors. Invalid settings and profiles return `2`;
|
||||
broken embedded assets, caller cancellation, fatal processing, and required
|
||||
output failures return `1`. A late report-write failure can follow a published
|
||||
transcript; exit status and report status remain authoritative.
|
||||
Configuration commands inspect profiles without contacting a provider.
|
||||
|
||||
[Configuration discovery and precedence](config.md#loading-model) determine the
|
||||
initial settings; explicit process flags supply overrides. Configuration is loaded
|
||||
before process help is printed, so invalid discovered configuration can also make
|
||||
process help fail. Input formats belong to the
|
||||
[input-file contract](integrations/transcript-glossary-files.md).
|
||||
|
||||
### `process` Flag Reference
|
||||
|
||||
Core I/O flags:
|
||||
- `--config <path>`: path to versioned YAML config file.
|
||||
- `--glossary <path>`: glossary YAML input path (required).
|
||||
- `--output <path>`: corrected transcript JSON output file path.
|
||||
- `--report-json <path>`: machine-readable report JSON output path.
|
||||
- `--output-schema <key>`: output encoder selection; see [configuration](config.md#pipeline-and-output).
|
||||
- `--modules <csv>`: comma-separated module sequence override.
|
||||
|
||||
Primary LLM flags:
|
||||
- `--llm-api-key <value>`: primary LLM API key.
|
||||
- `--model <name>`: primary LLM model name.
|
||||
- `--base-url <url>`: primary OpenAI-compatible base URL.
|
||||
- `--llm-timeout-seconds <int>`: primary timeout in seconds.
|
||||
- `--max-retries <int>`: primary structured-output retries.
|
||||
|
||||
Validation LLM flags:
|
||||
- `--validation-llm-api-key <value>`: validation LLM API key.
|
||||
- `--validation-model <name>`: validation LLM model name.
|
||||
- `--validation-base-url <url>`: validation OpenAI-compatible base URL.
|
||||
- `--validation-llm-timeout-seconds <int>`: validation timeout in seconds.
|
||||
- `--validation-max-retries <int>`: validation structured-output retries.
|
||||
- `--validation-max-prompt-tokens <int>`: validation max prompt tokens.
|
||||
|
||||
Concurrency flags:
|
||||
- `--total-llm-concurrency <int>`: total concurrent proposal+validation LLM calls.
|
||||
- `--proposal-llm-concurrency <int>`: concurrent proposal-generation LLM calls.
|
||||
- `--validation-llm-concurrency <int>`: concurrent validation LLM calls.
|
||||
- `--llm-concurrency <int>`: alias for `--total-llm-concurrency`.
|
||||
|
||||
Chunking and normalization flags:
|
||||
- `--target-sections <int>`: target number of transcript sections.
|
||||
- `--max-section-tokens <int>`: maximum section tokens.
|
||||
- `--min-section-tokens <int>`: minimum section tokens.
|
||||
- `--normalize-max-segment-gap <float>`: maximum same-speaker merge gap.
|
||||
- `--normalize-ellipsis-gap <float>`: gap threshold for ellipsis insertion.
|
||||
- `--normalize-max-segment-duration <float>`: maximum merged segment duration.
|
||||
- `--normalize-max-segment-tokens <int>`: maximum merged segment token estimate.
|
||||
|
||||
Threshold flags:
|
||||
- `--glossary-confidence-threshold <float>`
|
||||
- `--homophones-confidence-threshold <float>`
|
||||
- `--spoken-word-confidence-threshold <float>`
|
||||
- `--grammar-confidence-threshold <float>`
|
||||
|
||||
Context and diagnostics flags:
|
||||
- `--transcript-description <text>`: background context for prompts; does not override transcript content.
|
||||
- `--work-dir <path>`: per-run diagnostics work directory.
|
||||
- `--work-dir-retention <value>`: retention selection; see [configuration](config.md#thresholds-context-and-diagnostics).
|
||||
|
||||
Flag descriptions identify their settings; defaults, accepted keys, numeric
|
||||
constraints, and inheritance rules are owned by [configuration](config.md).
|
||||
Normalization time flags use numeric seconds; LLM timeout flags use integer seconds.
|
||||
|
||||
### `process` Output and Exit Behavior
|
||||
|
||||
- With `--output`: stdout is expected to be empty on success.
|
||||
- Without `--output`: stdout contains transcript JSON only on success.
|
||||
- `--report-json` writes a file and is never printed to stdout.
|
||||
- Stderr is human-readable diagnostics/errors.
|
||||
- Processing failures include the diagnostics directory path when initialization
|
||||
succeeded. Later report-write or retention errors can omit that path.
|
||||
- Output files may already exist when a later reporting or cleanup failure occurs.
|
||||
- Rejected/skipped corrections and warnings can occur in a successful run; inspect
|
||||
the [report format](integrations/output-files.md#process-report) for details.
|
||||
- Diagnostic wording is human-readable and is not a machine-stable JSON channel.
|
||||
|
||||
Exit behavior:
|
||||
- `0`: success.
|
||||
- `1`: runtime failure during processing/reporting/output or retention handling.
|
||||
- `2`: CLI usage or configuration input error.
|
||||
|
||||
Integration references:
|
||||
- subprocess contract: [`docs/integrations/subprocess.md`](integrations/subprocess.md)
|
||||
- transcript/glossary file contract: [`docs/integrations/transcript-glossary-files.md`](integrations/transcript-glossary-files.md)
|
||||
|
||||
### `process` Examples
|
||||
|
||||
Write corrected transcript to a file:
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--output corrected.json
|
||||
```
|
||||
|
||||
Emit transcript JSON to stdout:
|
||||
|
||||
```sh
|
||||
audita process transcript.json --glossary glossary.yaml
|
||||
```
|
||||
|
||||
Use explicit config and write report JSON:
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--config audita.yml \
|
||||
--output corrected.json \
|
||||
--report-json report.json
|
||||
```
|
||||
|
||||
Override the module sequence:
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--modules glossary,homophones,grammar \
|
||||
--output corrected.json
|
||||
```
|
||||
|
||||
## Configuration Commands
|
||||
|
||||
### `config validate`
|
||||
|
||||
```sh
|
||||
audita config validate --config examples/minimal-config.yml
|
||||
audita config validate --config examples/production-config.yml
|
||||
```
|
||||
|
||||
The only command-specific flag is `--config <path>`, which is required.
|
||||
Positional arguments are rejected. Success prints `config is valid` to stdout
|
||||
and returns zero; usage, file-read, parse, and validation errors go to stderr and
|
||||
return `2`. See [applied configuration sources](config.md#effective-precedence)
|
||||
and [validation boundaries](config.md#examples).
|
||||
|
||||
### `config print-effective`
|
||||
|
||||
```sh
|
||||
audita config print-effective --config examples/minimal-config.yml
|
||||
```
|
||||
|
||||
The only command-specific flag is optional `--config <path>`. Positional
|
||||
arguments are rejected. Without an explicit file, [discovery](config.md#loading-model)
|
||||
applies. Success prints redacted configuration JSON to stdout and returns zero.
|
||||
Usage and configuration errors return `2`; serialization or stdout-write failures
|
||||
return `1`.
|
||||
|
||||
The printed JSON reflects the effective in-memory configuration; it is not a YAML
|
||||
configuration file and is not the process-report schema. Use it to inspect merged
|
||||
settings, not as a file to pass back to the YAML loader.
|
||||
|
||||
## Help
|
||||
|
||||
Root invocation without arguments and root help print usage successfully. Use
|
||||
`--help` or `-h` for subcommand help. Help normally returns zero; process help has
|
||||
the configuration-loading caveat described above. Unknown commands return `2`.
|
||||
See the [input files](integrations/transcript-glossary-files.md),
|
||||
[subprocess contract](integrations/subprocess.md), and
|
||||
[artifact lifecycle](operations.md) for their respective details.
|
||||
|
||||
250
docs/config.md
250
docs/config.md
@@ -1,188 +1,88 @@
|
||||
# Audita Configuration
|
||||
|
||||
This is the canonical reference for discovery, settings, defaults, environment
|
||||
mapping, and validation. [CLI syntax](cli.md), [artifact lifecycle](operations.md),
|
||||
and [file formats](integrations/output-files.md) have separate owners.
|
||||
This reference describes the version-2 configuration loader in `internal/core/config`.
|
||||
Use the maintained [local](../examples/minimal-config.yml) and
|
||||
[service](../examples/production-config.yml) examples with their PromptKit
|
||||
[profiles](../examples/profiles/) and optional [testing overlay](../examples/overlays/testing.yml).
|
||||
Choose real models for your endpoint; a testing profile makes no cost or quality
|
||||
guarantee. See the [migration guide](migrations/v2.md) for retired inputs.
|
||||
|
||||
## Loading Model
|
||||
## Loading and precedence
|
||||
|
||||
For processing and effective-config inspection, the first applicable source
|
||||
selects one YAML file:
|
||||
The application selects one main YAML file: an explicit `--config`, then
|
||||
`/usr/local/etc/audita/config.yml` if present, then `/etc/audita/config.yml`
|
||||
if present. Missing system candidates are skipped; an explicit missing or empty
|
||||
path, inaccessible selected file, or absence of any selected file is an error.
|
||||
`config validate` requires an explicit file. ConfigKit loads built-in defaults,
|
||||
the selected file, an optional named overlay from `overlays/<name>.yml` beside the
|
||||
main file, and only explicitly supplied setting overrides, in that order. Imports
|
||||
and secret-directory bootstrap are disabled. Unknown fields and missing, null,
|
||||
or non-integer `version: 2` are errors. Version-1 files require migration; no
|
||||
compatibility loader or implicit environment configuration remains.
|
||||
|
||||
1. Explicit CLI config path.
|
||||
2. `AUDITA_CONFIG`.
|
||||
3. `/usr/local/etc/audita/config.yml`, if present.
|
||||
4. `/etc/audita/config.yml`, if present.
|
||||
Validation runs after overlays, explicit overrides, and derived settings have
|
||||
resolved. An override can therefore repair an invalid lower-precedence setting
|
||||
when that setting can be decoded. ConfigKit retains field origins and derivations
|
||||
for effective-configuration inspection. Its effective YAML redacts resolved
|
||||
credentials.
|
||||
|
||||
These files are alternatives, not layers merged together. Missing system files
|
||||
are skipped. An empty or missing explicitly selected path is an error, as is an
|
||||
inaccessible selected file. Without a selected file, built-in defaults apply.
|
||||
The validation command requires an explicit file and does not use discovery.
|
||||
Profile files are resolved against the selected main file's directory, or may
|
||||
use an absolute `profiles_dir`. Both selected profiles must exist and select a
|
||||
registered backend. Their effective endpoints must match that backend's endpoint
|
||||
after surrounding whitespace and trailing slashes are removed. Profiles cannot
|
||||
supply a competing `api_key_env` lookup. The backend credential supplies the
|
||||
runtime key when required; inspection makes no provider call. The
|
||||
[CLI inspection views](cli.md#streams-and-exit-status) show selected controls and
|
||||
field provenance separately from ConfigKit's redacted settings YAML.
|
||||
|
||||
## Effective Precedence
|
||||
## Settings
|
||||
|
||||
| Operation | Applied sources, lowest to highest precedence |
|
||||
| --- | --- |
|
||||
| Process | Built-in defaults, selected file, environment overrides, explicit CLI overrides |
|
||||
| Print effective configuration | Built-in defaults, selected file, environment overrides |
|
||||
| Validate configuration file | Built-in defaults, explicit file |
|
||||
|
||||
File validation does not apply configuration environment overrides, but it still
|
||||
resolves environment variables named by `api_key_env` fields. Each configuration
|
||||
layer is validated as it is applied: a later override cannot repair an invalid
|
||||
earlier layer. [Config commands](cli.md#configuration-commands) provide inspection
|
||||
and validation without processing a transcript or contacting an LLM.
|
||||
|
||||
## File Schema
|
||||
|
||||
The file is YAML with required `version: 1`. Unknown fields are rejected.
|
||||
The remaining sections and fields below are optional. Omitted values preserve
|
||||
lower-precedence settings, except for the documented inheritance rules.
|
||||
Complete copyable files are in [examples](#examples).
|
||||
|
||||
### Pipeline and Output
|
||||
|
||||
| YAML field | Default | Meaning |
|
||||
| Field | Default | Rule |
|
||||
| --- | --- | --- |
|
||||
| `pipeline.modules` | `[glossary, homophones, glossary, spoken_word, grammar]` | Ordered module sequence. Supported keys are `glossary`, `homophones`, `spoken_word`, and `grammar`; repeated keys are allowed. An omitted or empty YAML list leaves the sequence unchanged. |
|
||||
| `output.schema` | `bare-segments` | Output encoder selection: `bare-segments` or `audita-v1`. See [output formats](integrations/output-files.md#corrected-transcript). |
|
||||
| `profiles_dir` | `profiles` | Nonempty; profile files are resolved relative to the selected main file. |
|
||||
| `backends.<id>.endpoint` | None | Required nonempty endpoint for each backend. At least one backend is required. |
|
||||
| `backends.<id>.concurrency` | `1` | Positive finite capacity; shared by both roles using that backend. |
|
||||
| `backends.<id>.queue_capacity` | `1024` | Nonnegative; zero disables waiting admission. |
|
||||
| `backends.<id>.credential` | Unset | Optional environment-variable secret reference; omission or null means unauthenticated use. |
|
||||
| `llm.proposal_profile` | None | Required nonempty profile ID. |
|
||||
| `llm.validation_profile` | Final proposal profile | Omission inherits the final proposal selection; explicit empty or null fails. |
|
||||
| `llm.transient_retries` | `2` | Nonnegative additional attempts per generation. |
|
||||
| `llm.repair_attempts` | `2` | Additional structural repair generations, from zero through three. |
|
||||
| `llm.operation_timeout` | `15m` | Positive Go duration string per logical call. |
|
||||
| `pipeline.modules` | `[glossary, homophones, glossary, spoken_word, grammar]` | Ordered module keys; repeats are meaningful and an explicit empty list runs no correction modules. |
|
||||
| `output.schema` | `bare-segments` | `bare-segments` or `audita-v1`. See [output formats](integrations/output-files.md). |
|
||||
| `validation.max_prompt_tokens` | `2048` | Positive validation batching budget. |
|
||||
| `chunking.target_sections` | Unset | Optional positive target; omission or null selects automatic chunking. |
|
||||
| `chunking.max_section_tokens` | `8192` | Positive maximum estimate. |
|
||||
| `chunking.min_section_tokens` | `2048` | Positive and no greater than maximum. |
|
||||
| `normalization.max_segment_gap` | `4.0` | Nonnegative numeric seconds. |
|
||||
| `normalization.ellipsis_gap` | `3.5` | Nonnegative numeric seconds. |
|
||||
| `normalization.max_segment_duration` | `60.0` | Positive numeric seconds. |
|
||||
| `normalization.max_segment_tokens` | `2048` | Positive token estimate. |
|
||||
| `thresholds.glossary`, `.homophones`, `.spoken_word`, `.grammar` | `0.8` each | Confidence thresholds in `[0, 1]`. |
|
||||
| `context.description` | Empty | Trimmed prompt context, at most 500 bytes. |
|
||||
| `reporting.enabled` | `true` | Write the ordinary three-file report bundle when processing. |
|
||||
| `reporting.stdout_dir` | `.` | Report directory for stdout transcript output; ignored for file output or disabled reporting. |
|
||||
| `diagnostics.debug` | `false` | Persist detailed debug artifacts when processing. |
|
||||
| `diagnostics.work_dir` | `/tmp/audita` | Parent of the retained debug run directory; required only when debug is enabled. |
|
||||
|
||||
Module keys select correction stages. Validator chains are built into each module;
|
||||
there is no configuration field for selecting individual validator keys.
|
||||
Module keys select correction modules; validator chains remain built into them.
|
||||
The loader preserves module order and repetitions. Normalization time settings
|
||||
accept numeric seconds only. The operation timeout uses a duration string.
|
||||
`reporting.stdout_dir` must be nonblank only for reporting-enabled stdout runs;
|
||||
`diagnostics.work_dir` must be nonblank only with debug enabled. These path
|
||||
requirements are checked when choosing process destinations, not during
|
||||
configuration inspection.
|
||||
|
||||
### LLM Settings
|
||||
## Credentials
|
||||
|
||||
The same fields are supported under `llm.proposal` and `llm.validation`.
|
||||
A backend credential is a ConfigKit secret reference containing an environment
|
||||
variable name, such as `credential: AUDITA_SERVICE_KEY`. Every reference that
|
||||
survives overlay and override resolution must name a set, nonempty variable,
|
||||
including credentials on backends unused by the selected roles. A replacement
|
||||
reference resolves only the replacement; null removes the reference. General
|
||||
`AUDITA_*` and `OPENROUTER_API_KEY` variables have no implicit setting effect.
|
||||
Secret values are available to the runtime through the resolved reference, while
|
||||
ConfigKit's effective output and provenance contain no resolved secret values.
|
||||
|
||||
| Field suffix | Proposal default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `base_url` | `https://openrouter.ai/api/v1` | Endpoint base URL; the [LLM contract](integrations/openai-compatible-llm.md#endpoint-and-authentication) defines how it is used. |
|
||||
| `model` | `openrouter/google/gemma-4-31b-it` | Model identifier sent to the endpoint. Availability is determined by that endpoint. |
|
||||
| `api_key_env` | Unset | Name of an environment variable containing the API key; see [secrets](#secrets). |
|
||||
| `timeout` | `600` seconds | Positive integer seconds or a duration string resolving to whole seconds, such as `120s`. |
|
||||
| `max_retries` | `3` | Non-negative retry count after the initial attempt; retry conditions belong to the [LLM contract](integrations/openai-compatible-llm.md#retries-and-timeouts). |
|
||||
|
||||
Validation settings inherit the effective proposal settings individually. Empty
|
||||
validation model, base URL, or API key values inherit the proposal value; omitted
|
||||
validation timeout and retry count inherit, while an explicit retry count of zero
|
||||
is retained in configuration. The current HTTP adapter interprets a retry count
|
||||
of zero as its default of three retries, so zero does not disable retries.
|
||||
No API key is required by local config validation. Model and base-URL
|
||||
usability are checked during client setup/requests, not by an online config check.
|
||||
|
||||
### Concurrency
|
||||
|
||||
| YAML field | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `concurrency.total_llm` | `1` | Positive total concurrent LLM-call limit. |
|
||||
| `concurrency.proposal_llm` | `1` | Positive proposal-call limit, no greater than total. |
|
||||
| `concurrency.validation_llm` | Inherits total | Positive validation-call limit, no greater than total. |
|
||||
|
||||
In a YAML file, setting total alone does not change proposal concurrency. At the
|
||||
environment or CLI layer, setting total without setting proposal at that same
|
||||
layer also sets proposal to that total, even if a lower layer supplied a proposal
|
||||
value. Unset validation concurrency inherits the final total. The canonical total
|
||||
setting wins over its legacy alias when both appear at the same layer.
|
||||
|
||||
Validation prompt budgeting is available through environment or CLI only:
|
||||
`AUDITA_VALIDATION_MAX_PROMPT_TOKENS` has default `2048` and must be positive.
|
||||
There is no YAML field for it.
|
||||
|
||||
### Chunking and Normalization
|
||||
|
||||
| YAML field | Default | Meaning and validation |
|
||||
| --- | --- | --- |
|
||||
| `chunking.target_sections` | Unset | Optional positive target count. Processing rejects targets impossible for the normalized transcript and token constraints. |
|
||||
| `chunking.max_section_tokens` | `8192` | Positive maximum section token estimate. |
|
||||
| `chunking.min_section_tokens` | `2048` | Positive minimum used in chunk balancing; no greater than maximum. |
|
||||
| `normalization.max_segment_gap` | `4.0` seconds | Non-negative gap limit for same-speaker merging. |
|
||||
| `normalization.ellipsis_gap` | `3.5` seconds | Non-negative gap threshold for inserting an ellipsis during merging. |
|
||||
| `normalization.max_segment_duration` | `60.0` seconds | Positive merged-segment duration limit. |
|
||||
| `normalization.max_segment_tokens` | `2048` | Positive merged-segment token estimate limit. |
|
||||
|
||||
The three normalization time fields accept numeric seconds or YAML duration
|
||||
strings. Their environment and CLI overrides use numeric seconds. Section token
|
||||
counts are estimates; config validation does not test whether a particular input
|
||||
can satisfy the selected chunking target.
|
||||
|
||||
### Thresholds, Context, and Diagnostics
|
||||
|
||||
| YAML field | Default | Meaning and validation |
|
||||
| --- | --- | --- |
|
||||
| `thresholds.glossary` | `0.8` | Glossary correction confidence threshold, in `[0, 1]`. |
|
||||
| `thresholds.homophones` | `0.8` | Homophone correction confidence threshold, in `[0, 1]`. |
|
||||
| `thresholds.spoken_word` | `0.8` | Spoken-word correction confidence threshold, in `[0, 1]`. |
|
||||
| `thresholds.grammar` | `0.8` | Grammar correction confidence threshold, in `[0, 1]`. |
|
||||
| `context.description` | Empty | Trimmed background context for prompts; limited to 500 bytes of UTF-8 text after trimming. It does not replace transcript content. |
|
||||
| `diagnostics.work_dir` | `/tmp/audita` | Non-empty parent directory for run artifacts. |
|
||||
| `diagnostics.retention` | `auto` | Accepted values: `auto`, `always`, `never`. Exact lifecycle behavior, including the current behavior of `never`, belongs to [operations](operations.md#retention-behavior). |
|
||||
|
||||
## Environment Overrides
|
||||
|
||||
Values are read when the variable is set. Empty values are not generally equivalent
|
||||
to unset values: numeric overrides must parse, and empty strings may clear a
|
||||
setting or fail validation. Numeric environment values use integers or floating
|
||||
point seconds as indicated above, not duration strings.
|
||||
|
||||
| Environment variable | Setting |
|
||||
| --- | --- |
|
||||
| `AUDITA_CONFIG` | Config file selection |
|
||||
| `AUDITA_MODULES` | Module sequence as comma-separated keys |
|
||||
| `AUDITA_LLM_API_KEY` | Direct proposal API key |
|
||||
| `OPENROUTER_API_KEY` | Proposal API key fallback when `AUDITA_LLM_API_KEY` is unset |
|
||||
| `AUDITA_MODEL` | Proposal model |
|
||||
| `AUDITA_BASE_URL` | Proposal endpoint |
|
||||
| `AUDITA_LLM_TIMEOUT_SECONDS` | Proposal timeout |
|
||||
| `AUDITA_MAX_RETRIES` | Proposal retries |
|
||||
| `AUDITA_VALIDATION_LLM_API_KEY` | Direct validation API key |
|
||||
| `AUDITA_VALIDATION_MODEL` | Validation model |
|
||||
| `AUDITA_VALIDATION_BASE_URL` | Validation endpoint |
|
||||
| `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS` | Validation timeout |
|
||||
| `AUDITA_VALIDATION_MAX_RETRIES` | Validation retries |
|
||||
| `AUDITA_TOTAL_LLM_CONCURRENCY` | Total concurrency |
|
||||
| `AUDITA_LLM_CONCURRENCY` | Legacy total-concurrency alias |
|
||||
| `AUDITA_PROPOSAL_LLM_CONCURRENCY` | Proposal concurrency |
|
||||
| `AUDITA_VALIDATION_LLM_CONCURRENCY` | Validation concurrency |
|
||||
| `AUDITA_VALIDATION_MAX_PROMPT_TOKENS` | Validation prompt budget |
|
||||
| `AUDITA_TARGET_SECTIONS` | Section target |
|
||||
| `AUDITA_MAX_SECTION_TOKENS` | Section maximum |
|
||||
| `AUDITA_MIN_SECTION_TOKENS` | Section minimum |
|
||||
| `AUDITA_NORMALIZE_MAX_SEGMENT_GAP` | Merge gap |
|
||||
| `AUDITA_NORMALIZE_ELLIPSIS_GAP` | Ellipsis gap |
|
||||
| `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION` | Merged duration limit |
|
||||
| `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS` | Merged token estimate limit |
|
||||
| `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD` | Glossary threshold |
|
||||
| `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD` | Homophone threshold |
|
||||
| `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD` | Spoken-word threshold |
|
||||
| `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD` | Grammar threshold |
|
||||
| `AUDITA_WORK_DIR` | Diagnostics directory |
|
||||
| `AUDITA_WORK_DIR_RETENTION` | Retention selection |
|
||||
|
||||
There is no environment override for output schema or transcript description.
|
||||
Use file configuration or the corresponding [CLI flags](cli.md#process-flag-reference).
|
||||
|
||||
## Secrets
|
||||
|
||||
File config accepts environment variable names through `api_key_env`, not literal
|
||||
API-key fields. Names must be non-empty and match an environment-variable
|
||||
identifier; an unset named variable resolves to an empty value. Runtime environment
|
||||
key overrides take precedence over keys resolved from a file. CLI key flags supply
|
||||
direct values and take precedence over environment settings.
|
||||
|
||||
Effective-config inspection redacts API keys. Configured LLM secrets are also
|
||||
redacted by the runtime's error and diagnostic handling, but transcript content
|
||||
is still sensitive: see [artifact handling](operations.md#artifact-handling).
|
||||
|
||||
## Examples
|
||||
|
||||
- [Minimal configuration](../examples/minimal-config.yml): starts from defaults
|
||||
and obtains the proposal key from the environment.
|
||||
- [Production-oriented configuration](../examples/production-config.yml): illustrates
|
||||
both LLM targets, concurrency, chunking, normalization, thresholds, and diagnostics.
|
||||
Its explicit section target is intended for larger inputs, not the tiny sample.
|
||||
- [Tiny transcript](../examples/tiny-transcript.json) and
|
||||
[tiny glossary](../examples/tiny-glossary.yaml): inputs for the README quickstart.
|
||||
|
||||
Use the [CLI configuration commands](cli.md#configuration-commands) to validate
|
||||
files and inspect effective settings. Validation checks local structure and
|
||||
constraints; it does not verify credentials, model availability, or endpoint access.
|
||||
Current invocation syntax belongs in [CLI](cli.md).
|
||||
|
||||
@@ -20,7 +20,7 @@ ownership. Commit discipline and agent Git restrictions are in the
|
||||
|
||||
## Baseline Validation
|
||||
|
||||
Use Go 1.24 or newer. From the repository root:
|
||||
Use Go 1.25.5 or newer. From the repository root:
|
||||
|
||||
```sh
|
||||
go build -o ./bin/audita ./cmd/audita
|
||||
|
||||
@@ -1,76 +1,34 @@
|
||||
# OpenAI-Compatible LLM Integration
|
||||
|
||||
This document owns the external protocol used by Audita's structured LLM adapter.
|
||||
Settings, inheritance, defaults, and credential supply belong to
|
||||
[configuration](../config.md#llm-settings); implementation and scheduling belong to
|
||||
[LLM runtime internals](../internal/llm-runtime.md).
|
||||
Audita uses the pinned [PromptKit](https://gitea.maximumdirect.net/eric/promptkit)
|
||||
built-in OpenAI-compatible transport. PromptKit owns chat-completions wire framing,
|
||||
provider envelope parsing, transient retries, per-generation timeouts, and JSON
|
||||
Schema repair. A backend's configured base endpoint must support the selected
|
||||
profile's model and structured JSON-schema responses; local configuration
|
||||
inspection does not contact the provider.
|
||||
|
||||
## Endpoint and Authentication
|
||||
Audita selects a role profile, supplies its registered backend credential, and
|
||||
passes the embedded response schema as a complete per-call output contract. It
|
||||
prepares each call once, checks that the prepared target still matches the
|
||||
inspected backend and endpoint, then executes that same prepared request. A
|
||||
profile cannot introduce a separate credential environment lookup. An absent
|
||||
backend credential permits an unauthenticated local service.
|
||||
|
||||
Audita sends an HTTP `POST` to the configured base URL with trailing slashes
|
||||
removed and `/chat/completions` appended. Supply an HTTP or HTTPS base URL that
|
||||
makes this the intended endpoint; the adapter does not require HTTPS in code.
|
||||
Requests have `Content-Type: application/json`. A non-empty configured API key
|
||||
adds `Authorization: Bearer <api_key>`; otherwise that header is omitted.
|
||||
`llm.transient_retries` counts additional attempts for eligible transient
|
||||
failures; explicit zero makes one attempt. `llm.repair_attempts` counts
|
||||
additional generations for schema-invalid model output. The operation timeout
|
||||
covers preparation, capacity waiting, retries, generation, and repair. Caller
|
||||
cancellation or an earlier deadline also applies. See [configuration](../config.md#settings)
|
||||
for defaults and [PromptKit's consumer contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/consumers/pkg-promptkit.md)
|
||||
for the pinned protocol and retry details.
|
||||
|
||||
The endpoint must support the configured model and structured JSON-schema
|
||||
responses. Local configuration validation does not probe that support.
|
||||
|
||||
## Request Shape
|
||||
|
||||
The payload contains `model`, an array of `messages` with string `role`/`content`,
|
||||
and `response_format`. The structured-output portion has this illustrative shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "json_schema",
|
||||
"json_schema": {
|
||||
"name": "correction_set",
|
||||
"strict": true,
|
||||
"schema": {"type": "object"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The actual schema is supplied for the call; the fragment above is not a complete
|
||||
request schema. Runtime calls use `correction_set` and `validator_decision_set`
|
||||
identities. The adapter requires a non-empty resolved model, non-empty message
|
||||
roles/content, a schema name, and valid schema JSON. It sends strict mode to the
|
||||
provider; local typed decoding and proposal/decision checks still apply afterward.
|
||||
|
||||
## Response Handling Contract
|
||||
|
||||
A successful response must be a JSON envelope with at least one `choices` item.
|
||||
Audita reads the first choice's `message.content`, accepting a non-empty string
|
||||
containing JSON or a non-null raw JSON value that can decode into the expected
|
||||
output type. Empty content, missing choices, and incompatible payloads fail.
|
||||
The adapter also reads optional `model` and token-usage metadata.
|
||||
|
||||
There is no streaming protocol or provider SDK dependency in this integration.
|
||||
Audita uses this chat-completions contract, not other provider API shapes.
|
||||
|
||||
## Retries and Timeouts
|
||||
|
||||
The effective adapter retry count is additional attempts after the initial request;
|
||||
see [configuration](../config.md#llm-settings) for the current zero-value behavior.
|
||||
Transport/read errors, malformed response envelopes/content, typed decode failures,
|
||||
HTTP `429`, and server-error responses are retryable. Other non-success HTTP
|
||||
responses and a response missing choices are not retried by the adapter.
|
||||
Cancellation or an expired context stops retrying.
|
||||
|
||||
The configured HTTP timeout applies per request; it is not a whole-process time
|
||||
budget. A parent process can impose its own overall timeout. See
|
||||
[configuration](../config.md#llm-settings) for timeout and retry values and
|
||||
[subprocess guidance](subprocess.md) for parent responsibilities.
|
||||
|
||||
## Error Behavior
|
||||
|
||||
For non-success responses, error detail comes from provider `error.message`, then
|
||||
a top-level `message`, otherwise the HTTP status and response body. Configured
|
||||
keys are redacted from surfaced errors and LLM diagnostic payloads.
|
||||
|
||||
Malformed structured responses can become section/batch warnings and rejected
|
||||
proposals rather than fatal run errors, depending on the call site. Other adapter
|
||||
failures can terminate processing with partial results. Consult the
|
||||
[report contract](output-files.md#module-outcomes) for those outcomes and
|
||||
[operations](../operations.md#artifact-handling) for handling diagnostic contents.
|
||||
A model operation that exhausts structural repair, a live-parent local operation
|
||||
timeout, capacity rejection, transport failure, malformed provider response,
|
||||
or provider HTTP failure is recoverable for that operation. Proposal generation
|
||||
then contributes no candidates from that response; a failed validation batch
|
||||
marks its candidates unavailable while other batches and later modules can
|
||||
continue. Caller cancellation, broken assets or configuration, and unknown
|
||||
internal failures remain fatal and cancel peers. Audita decodes into domain
|
||||
types only after PromptKit validates the JSON. Ordinary reports use safe reason
|
||||
codes and counts rather than provider messages. Full request/response content
|
||||
is written only with `--debug`; see [operations](../operations.md#diagnostics-directory).
|
||||
|
||||
@@ -1,186 +1,112 @@
|
||||
# Output Files and Report Formats
|
||||
|
||||
This document owns externally consumed corrected-transcript and report formats,
|
||||
and describes the correction ledger and utilization diagnostics. It documents
|
||||
current serialized shapes without making every debugging artifact a versioned
|
||||
compatibility promise. Physical filenames, placement, retention, and permissions
|
||||
belong to [operations](../operations.md); invocation and output channels belong to
|
||||
the [CLI](../cli.md#process-output-and-exit-behavior).
|
||||
# Output Files and Reports
|
||||
|
||||
## Corrected Transcript
|
||||
|
||||
The [output setting](../config.md#pipeline-and-output) selects the envelope:
|
||||
The corrected transcript uses the selected `output.schema`: `bare-segments`
|
||||
or `audita-v1`. Its schema and text rules are unchanged by resilience
|
||||
reporting. See [configuration](../config.md#settings) for selection and
|
||||
[operations](../operations.md#required-output-files) for placement and writes.
|
||||
|
||||
| Encoder | JSON shape |
|
||||
With reporting enabled, Audita produces three ordinary files:
|
||||
|
||||
- `errors.yml`: affected work units and run-level failures;
|
||||
- `recoveries.yml`: successful logical model operations with repair, service
|
||||
retry, or response filtering;
|
||||
- `report.md`: a concise overview and complete per-chunk matrices.
|
||||
|
||||
The YAML documents have Audita-owned schemas `audita-errors` and
|
||||
`audita-recoveries`, each with integer `version: 1`. Both contain, in order,
|
||||
`schema`, `version`, `run_id`, `status`, `started_at`, `completed_at`, `summary`,
|
||||
and an always-present `errors` or `recoveries` sequence. Status is
|
||||
`in_progress`, `completed`, or `failed`. Times are UTC RFC3339 strings;
|
||||
`completed_at` is null until finalization. A recoverable model error can appear
|
||||
with `status: completed`.
|
||||
|
||||
## Summary Counts
|
||||
|
||||
Both documents carry the same summary, in this order:
|
||||
|
||||
| Count | Meaning |
|
||||
| --- | --- |
|
||||
| `bare-segments` | Top-level array of segment objects |
|
||||
| `audita-v1` | Object with `schema: "audita-v1"`, `version: "v1"`, and `segments` containing the segment array |
|
||||
| `work_units_total` | Configured module instances × initial chunks. |
|
||||
| `work_units_started`, `work_units_completed`, `work_units_interrupted`, `work_units_not_started` | Explicit processing states. |
|
||||
| `work_units_with_errors` | Distinct units with recoverable operation failures. |
|
||||
| `run_errors` | Fatal processing or output failures outside that unit count. |
|
||||
| `corrections_applied` | Approved proposals applied, not replacement occurrences. |
|
||||
| `llm_operations` | Structured-client invocations, including pre-dispatch failures. |
|
||||
| `proposal_responses` | Structurally usable proposal responses. |
|
||||
| `recoveries` | Successful operations with one or more recovery reasons. |
|
||||
| `proposal_responses_with_filtering` | Proposal responses that filtered at least one defective proposal. |
|
||||
|
||||
Both emit these segment fields:
|
||||
Unknown counts before processing are `null`; completed empty counts are `0`.
|
||||
Retries and repair generations are details of one operation, not additional
|
||||
operations. Provider token usage, execution timing, and attempt-level evidence
|
||||
remain in opt-in debug files.
|
||||
|
||||
| Field | Type | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `id` | Integer | Identifier in the normalized transcript |
|
||||
| `speaker` | String | Speaker label |
|
||||
| `start`, `end` | Numbers | Start/end times in seconds |
|
||||
| `text` | String | Text after normalization and accepted corrections |
|
||||
| `categories` | Array of strings, optional | Segment categories; omitted when empty |
|
||||
## Error and Recovery Records
|
||||
|
||||
Input IDs and segment boundaries need not survive normalization unchanged;
|
||||
normalization can merge segments and reassign IDs. The output describes the
|
||||
resulting transcript, not a lossless copy of the source JSON. Input acceptance
|
||||
rules belong to the [input-file contract](transcript-glossary-files.md).
|
||||
An `errors` record with `scope: work_unit` identifies zero-based `stage_index`
|
||||
and `chunk_index`, module key/instance, initial segment range, applied-correction
|
||||
count, and a `failures` sequence. Each failure identifies the operation
|
||||
(`proposal`, `validation`, or `validation_input`), optional validator and
|
||||
zero-based batch index, safe reason/message, sorted proposal indices when
|
||||
known, and execution counts when available. Multiple failed batches in one
|
||||
work unit form one error record. `scope: run` instead records phase, safe reason
|
||||
and message, optional origin, and output path where relevant. It can describe
|
||||
a fatal error or secondary output-write failure without inventing a failed
|
||||
model work unit. Raw provider errors and transcript text are absent.
|
||||
For `validation_decisions_invalid`, proposal indices identify only candidates
|
||||
whose required decisions were unavailable. Other candidates in the same batch
|
||||
may still be approved or rejected normally.
|
||||
|
||||
## Process Report
|
||||
Each recovery identifies its work unit and successful operation, with sorted
|
||||
`reasons`, a safe message, available execution counts, and sorted
|
||||
`filtered_proposals` details. One recovery can contain `output_repaired`,
|
||||
`service_recovered`, and `proposals_filtered` together. Filtering covers
|
||||
defective response proposals; low confidence, protected-term rejection,
|
||||
ordinary editorial rejection, and an apply-time conflict caused by an earlier
|
||||
accepted correction are not filtering recoveries. Failed operations are errors,
|
||||
not recoveries. The initial chunk identity remains stable even if earlier
|
||||
modules change its text.
|
||||
`validation_decisions_filtered` records unexpected validator decision indices
|
||||
ignored from a successful validation batch. It may accompany repair or service
|
||||
recovery reasons, but it does not add `filtered_proposals` or increment
|
||||
`proposal_responses_with_filtering`. A batch with unavailable requested decisions
|
||||
is an error even when unexpected indices were also ignored.
|
||||
|
||||
The report is a JSON object. `report_metadata` identifies its schema separately
|
||||
from the transcript output envelope:
|
||||
## Markdown Overview
|
||||
|
||||
| Metadata field | Type / current value |
|
||||
| --- | --- |
|
||||
| `report_schema_name` | String: `audita-process-report` |
|
||||
| `report_schema_version` | String: `v1` |
|
||||
| `output_schema` | String identifying the selected transcript encoder |
|
||||
| `config_version` | Optional integer, present when file configuration was loaded |
|
||||
`report.md` includes run ID, status, corrections, work coverage, error and
|
||||
recovery totals, input/output paths, and full work-unit error and recovery
|
||||
matrices. Rows show every initial chunk with 1-based chunk number and initial
|
||||
segment range; columns follow configured module-instance order, including
|
||||
repeats. Completed error cells are `0` or `1`; recovery cells are operation
|
||||
counts. `—` means not started and `n*` means interrupted with n facts recorded.
|
||||
An empty stage or chunk set is identified as having no work units. Short reason breakdowns count
|
||||
records containing each reason; a recovery can have several reasons, so those
|
||||
counts need not sum to the recovery total. The report refers to both YAML
|
||||
files. It does not include transcript excerpts or correction lists.
|
||||
|
||||
The remaining top-level fields are:
|
||||
## Debug-Only Files
|
||||
|
||||
| Field | Type and meaning |
|
||||
| --- | --- |
|
||||
| `phase` | String; currently `default_pipeline` |
|
||||
| `status` | String: `success` or `failed` |
|
||||
| `operation` | String; currently `process` |
|
||||
| `transcript_path`, `glossary_path` | Input path strings |
|
||||
| `output_path` | Optional transcript destination string |
|
||||
| `modules` | Array of configured module keys, including repeats |
|
||||
| `started_at`, `completed_at` | RFC 3339 timestamps, possibly fractional; completion is an optional field populated by completed process runs |
|
||||
| `error_phase`, `error_message` | Optional failure classification and human-readable detail |
|
||||
| `input_segment_count`, `normalized_segment_count` | Optional integer normalization counts |
|
||||
| `normalization_merges`, `normalization_id_reassignments` | Optional integer transformation counts |
|
||||
| `normalization_skipped` | Object of optional integer counters: `different_speakers`, `gap_too_large`, `duration_exceeded`, `token_limit_exceeded`; can be empty |
|
||||
| `chunking` | Optional section summary described below |
|
||||
| `diagnostics` | Optional artifact-path metadata described below |
|
||||
| `modules_summary` | Optional aggregate module outcome |
|
||||
| `module_results` | Optional array of module outcomes |
|
||||
|
||||
Failures can preserve partial module results. Missing summaries indicate unavailable
|
||||
information, not necessarily zero work. Early configuration failures occur before
|
||||
report construction. Report success describes processing results; a later file-write
|
||||
or cleanup error can still cause a nonzero CLI exit. Use the exit result and report
|
||||
together as explained in [subprocess guidance](subprocess.md).
|
||||
|
||||
Error phases identify where processing failed, for example `transcript_read`,
|
||||
`glossary_read`, `transcript_schema`, `glossary_schema`, `chunking`, `runner_setup`,
|
||||
`runner_execution`, `output_schema`, `serialization`, `output_write`, and
|
||||
`stdout_write`. This is not an exhaustive enumeration or a promise that every
|
||||
late failure is represented in a successfully written report. Error messages are
|
||||
human-readable, not structured fields to parse for control flow.
|
||||
|
||||
### Chunking and Diagnostics Metadata
|
||||
|
||||
`chunking` contains integer fields `chunk_count`, `min_estimated_chunk_tokens`,
|
||||
`max_estimated_chunk_tokens`, `total_estimated_transcript_tokens`,
|
||||
`max_section_tokens`, and `min_section_tokens`, plus optional integer
|
||||
`target_sections`. Token counts are estimates.
|
||||
|
||||
`diagnostics` contains optional path strings:
|
||||
|
||||
- `directory_path`
|
||||
- `source_transcript_path`, `parsed_source_transcript_path`, `normalized_transcript_path`
|
||||
- `normalization_summary_path`, `chunking_summary_path`
|
||||
- `utilization_summary_path`, `correction_ledger_path`
|
||||
- `invocation_metadata_path`, `redacted_effective_config_path`, `error_log_path`
|
||||
|
||||
The [operations guide](../operations.md#diagnostics-directory) defines the physical
|
||||
artifacts and path-lifetime caveats.
|
||||
|
||||
### Module Outcomes
|
||||
|
||||
`modules_summary` contains integer `module_count`, `total_applied_changes`, and
|
||||
`total_skipped_changes`, plus optional string `failed_module_instance`.
|
||||
`total_skipped_changes` counts both apply-time skipped changes and validator
|
||||
rejections. Their detailed records are stored in separate arrays.
|
||||
|
||||
Each `module_results` object contains:
|
||||
|
||||
| Field | Type and meaning |
|
||||
| --- | --- |
|
||||
| `module_key`, `module_instance` | Strings; the instance distinguishes repeated occurrences of a configured module |
|
||||
| `replacement_policy` | Optional string describing the module's replacement policy |
|
||||
| `status` | String describing module outcome (`success` or `failed`) |
|
||||
| `proposal_count` | Integer |
|
||||
| `started_at`, `completed_at` | Optional timestamps |
|
||||
| `error_message` | Optional failure detail |
|
||||
| `warnings` | Optional array of warning records |
|
||||
| `validator_decisions` | Optional array of decision records |
|
||||
| `validator_rejected` | Optional array of rejected-correction records |
|
||||
| `applied_changes`, `skipped_changes` | Optional arrays of apply outcomes |
|
||||
|
||||
A validator decision has `validator_name` (string), `proposal_index` (integer),
|
||||
`approved` (boolean), `reason_code` and `message` (strings), and optional
|
||||
`diagnostic_artifact_path` (string).
|
||||
|
||||
Rejected corrections include `validator_name`, `proposal_index`, `module_key`,
|
||||
`module_instance`, `target_segment_id`, `original_text`, `corrected_text`,
|
||||
`reason_code`, and `message`. Applied changes include the same correction identity
|
||||
and text fields without validator/reason/message fields, plus integer
|
||||
`replacement_count`. Skipped changes include correction identity/text fields,
|
||||
`skip_reason`, and `message`.
|
||||
|
||||
A warning has string `scope`, `reason_code`, and `message`; optional integer
|
||||
`section_index` and `batch_index`; and optional string `validator_name` and
|
||||
`diagnostic_artifact_path`. Warnings, skipped changes, and rejected corrections
|
||||
can occur without a failed run. Proposal indexes associate outcomes with their
|
||||
proposal within a module instance; they are not transcript segment IDs.
|
||||
|
||||
## Correction Ledger
|
||||
|
||||
The ledger is an array of correction entries; it can be `null` when no module
|
||||
results are available and empty when modules produced no entries. Unlike the
|
||||
process report, it has no separate schema-version envelope.
|
||||
|
||||
Each entry has required `module_key`, `module_instance`, integer `proposal_index`,
|
||||
and `disposition`. The current dispositions are:
|
||||
|
||||
| Disposition | Meaning |
|
||||
| --- | --- |
|
||||
| `applied` | The proposed change was applied |
|
||||
| `skipped` | The proposal reached application but was skipped |
|
||||
| `rejected` | A validator rejected the proposal |
|
||||
| `failed` | Module-level failure record, not necessarily an individual proposal |
|
||||
|
||||
Optional fields are `run_id`, integer `segment_id`, `original_text`,
|
||||
`proposed_corrected_text`, `applied_corrected_text`, `replacement_policy`,
|
||||
`disposition_reason_code`, and `disposition_message`. Optional validator arrays
|
||||
are `deterministic_validator_decisions` and `llm_validator_decisions`; each record
|
||||
contains string `validator_key`, boolean `approved`, string `reason_code`, and
|
||||
optional string `message`.
|
||||
|
||||
Module failure entries can have the zero-value proposal index and no correction
|
||||
text or segment. Do not interpret them as evidence that proposal zero failed.
|
||||
Empty optional values are omitted. The ledger records disposition, not a replay
|
||||
or resume protocol.
|
||||
|
||||
## Utilization Diagnostics
|
||||
|
||||
This unversioned JSON object describes observations from the runtime, not service
|
||||
limits or performance guarantees:
|
||||
|
||||
- `effective_concurrency`: integer `total_llm`, `proposal_llm`, `validation_llm`.
|
||||
- `run_timing`: millisecond integers `run_wall_time_ms`, `scheduler_queue_wait_ms`,
|
||||
`llm_execution_time_ms`, `deterministic_validation_time_ms`; integer
|
||||
`max_in_flight_llm_calls` and `average_in_flight_llm_calls`.
|
||||
- `llm_calls`: integer `total_proposal_calls` and `total_validation_calls`.
|
||||
- Optional `modules`: records with `module_key`, `module_instance`, integer
|
||||
`proposal_llm_calls` and `validation_llm_calls`, and millisecond integer
|
||||
`scheduler_queue_wait_ms`, `llm_execution_time_ms`,
|
||||
`deterministic_validation_time_ms`, and `module_wall_time_ms`.
|
||||
- Optional `validators`: records with `module_key`, `module_instance`,
|
||||
`validator_key`, boolean `llm_backed`, integer `calls`, and millisecond integer
|
||||
`elapsed_ms`.
|
||||
|
||||
Aggregated execution/wait durations can overlap in concurrent work; they need
|
||||
not sum to wall time. The observed average concurrency is serialized as an integer.
|
||||
Other source snapshots, prompt payloads, and diagnostic files are debugging
|
||||
material; their placement is documented in operations without defining a separate
|
||||
public schema for every file.
|
||||
With `--debug`, `correction-ledger.json` contains proposed candidates and
|
||||
their application, rejection, or interruption details. `llm-execution.json`
|
||||
holds logical calls and PromptKit execution events; `utilization-diagnostics.json`
|
||||
groups backend capacity and timing observations while preserving known and
|
||||
complete usage semantics. Interaction request/response artifacts, selected
|
||||
profiles, configuration provenance, and input snapshots are also debug-only.
|
||||
These files are diagnostic evidence, not an alternate ordinary report schema.
|
||||
Each candidate-backed ledger entry includes `proposed_corrected_text`, the
|
||||
literal replacement span, even when it is `""` for a deletion. Applied entries
|
||||
also include `applied_corrected_text`, likewise `""` for an applied deletion.
|
||||
Rejected, unavailable, skipped, and interrupted entries omit the applied field; a module
|
||||
failure without a candidate omits both. These fields are replacement spans,
|
||||
not full resulting-segment text. Deterministic decision records now identify
|
||||
the blank-result guard as `non_empty_resulting_segment` (formerly
|
||||
`non_empty_corrected_text`); its `empty_resulting_segment` reason is unchanged.
|
||||
Validator decision records carry `disposition`: `approved`, `rejected`, or
|
||||
`unavailable`. A missing or duplicate required decision is `unavailable`, not
|
||||
an editorial rejection. The candidate ledger uses the same distinction in its
|
||||
top-level disposition. Interaction response artifacts retain supplied model
|
||||
decisions and the reconciliation result, including unavailable and ignored
|
||||
indices; unexpected indices never create candidate ledger entries.
|
||||
|
||||
@@ -7,10 +7,13 @@ The [CLI reference](../cli.md) owns command syntax, streams, and exit semantics;
|
||||
## Invocation
|
||||
|
||||
Use an argument array with the transcript path and required glossary argument,
|
||||
following the [process examples](../cli.md#process-examples). For machine workflows,
|
||||
select explicit transcript and report destinations rather than depending on the
|
||||
working directory or run-directory retention. Supplying a configuration path also
|
||||
makes file selection predictable; inherited environment overrides still apply.
|
||||
following the [process examples](../cli.md#commands-and-help). For machine workflows,
|
||||
select an explicit transcript file destination and a configuration path. Reports
|
||||
then go beside the transcript. For stdout output, select `--report-dir` or use
|
||||
the invocation working directory. Debug is independent and off by default.
|
||||
Supplying a configuration path also
|
||||
makes file selection predictable; only explicitly referenced credential variables
|
||||
are read from the environment.
|
||||
|
||||
Resolve the executable deliberately and set the child's working directory so
|
||||
relative input/output paths and reported artifact paths have a known base.
|
||||
@@ -20,21 +23,26 @@ or cancellation in the parent process when needed.
|
||||
## Result Handling
|
||||
|
||||
Interpret the exit result according to the
|
||||
[CLI output and exit contract](../cli.md#process-output-and-exit-behavior).
|
||||
[CLI output and exit contract](../cli.md#streams-and-exit-status).
|
||||
Consume transcript/report data through the
|
||||
[output-file contract](output-files.md), and treat stderr as human-readable detail.
|
||||
The report supplements the exit result: it may be absent after early failures,
|
||||
and output may have been written before a later reporting or cleanup failure.
|
||||
The three reports may be absent after early configuration or destination
|
||||
failures. After initialization, failed reports can describe invalid input or
|
||||
partial processing. Exit `0` can include recoverable model errors and applied
|
||||
corrections; inspect `errors.yml` and `recoveries.yml` for those outcomes. A
|
||||
transcript may have been written before a later required report failure, so
|
||||
the file alone does not prove completion.
|
||||
|
||||
If processing fails, preserve available evidence and follow
|
||||
[operations recovery](../operations.md#failure-inspection-and-recovery).
|
||||
For reports retained outside the run directory, account for the
|
||||
[artifact lifecycle](../operations.md#retention-behavior) before dereferencing
|
||||
reported paths. Neither the existence of an output file nor a diagnostics path
|
||||
alone establishes successful completion.
|
||||
Ordinary reports are independent of optional retained debug evidence. Parent
|
||||
cancellation reaches the child through its context; Audita joins workers and
|
||||
attempts failed-status report finalization before exiting. Uncatchable
|
||||
termination can leave `in_progress` reports. Neither the existence of an output
|
||||
file nor a debug directory alone establishes successful completion.
|
||||
|
||||
## Related Contracts
|
||||
|
||||
- [Transcript and glossary inputs](transcript-glossary-files.md)
|
||||
- [LLM endpoint compatibility](openai-compatible-llm.md)
|
||||
- [Configuration inspection commands](../cli.md#configuration-commands)
|
||||
- [Configuration inspection commands](../cli.md#commands-and-help)
|
||||
|
||||
@@ -42,7 +42,7 @@ The top-level object contains a `glossary` list with at least one entry.
|
||||
|
||||
The parser checks empty strings without trimming them. Unknown YAML fields are
|
||||
ignored by the glossary parser, unlike the stricter
|
||||
[configuration-file schema](../config.md#file-schema).
|
||||
[configuration-file schema](../config.md#loading-and-precedence).
|
||||
|
||||
The [tiny glossary](../../examples/tiny-glossary.yaml) is a complete copyable input.
|
||||
Malformed YAML, an empty/missing glossary, invalid shapes, missing required text,
|
||||
|
||||
@@ -7,9 +7,9 @@ CLI wiring, and maintained examples. Definitions of behavior belong to the
|
||||
## Changing Configuration
|
||||
|
||||
1. Update fields, defaults, loading, and validation in `internal/core/config`.
|
||||
2. Update file, environment, and CLI mapping as applicable, including CLI flag
|
||||
extraction. Preserve the source-specific behavior defined in the
|
||||
[configuration contract](../config.md#effective-precedence), including the
|
||||
2. Update file, selected overlay, credential reference, and CLI mapping as
|
||||
applicable. Preserve the precedence defined in the
|
||||
[configuration contract](../config.md#loading-and-precedence), including the
|
||||
distinct validation and effective-config commands.
|
||||
3. Update redaction when secrets are involved.
|
||||
4. Review configuration tests and related CLI tests for precedence, validation,
|
||||
@@ -35,12 +35,13 @@ From the repository root, validate the maintained configuration examples:
|
||||
|
||||
```sh
|
||||
go run ./cmd/audita config validate --config examples/minimal-config.yml
|
||||
go run ./cmd/audita config validate --config examples/production-config.yml
|
||||
go run ./cmd/audita config print-effective --config examples/minimal-config.yml
|
||||
AUDITA_SERVICE_KEY=synthetic-example go run ./cmd/audita config validate --config examples/production-config.yml
|
||||
AUDITA_SERVICE_KEY=synthetic-example go run ./cmd/audita config validate --config examples/production-config.yml --overlay testing
|
||||
go run ./cmd/audita config print-effective --config examples/minimal-config.yml --view profiles
|
||||
```
|
||||
|
||||
The effective-config command incorporates environment settings; use a controlled
|
||||
environment when comparing its results. Check affected CLI examples against the
|
||||
Only referenced credentials read environment values; use a controlled synthetic
|
||||
value when validating the service example. Check affected CLI examples against the
|
||||
parser and use schema tests for maintained transcript/glossary inputs. Example
|
||||
validation should not invoke a live LLM merely to check documentation.
|
||||
|
||||
@@ -58,7 +59,8 @@ Select checks according to the [testing policy](../policy/testing.md):
|
||||
- Schema or serialization changes: review [output-schema tests](output-schemas.md#key-tests)
|
||||
and `internal/core/schema` tests, including supported formats and rejection of
|
||||
unsupported schemas.
|
||||
- Changes affecting assembled output: inspect CLI parity and release fixtures.
|
||||
- Changes affecting assembled output: inspect controlled HTTP subprocess workflows
|
||||
and retained report/artifact assertions.
|
||||
|
||||
Use the [contributor guide](../development.md#baseline-validation) for baseline
|
||||
commands. These recipes supplement, rather than duplicate, public contracts.
|
||||
|
||||
@@ -1,74 +1,63 @@
|
||||
# Diagnostics and Report Assembly
|
||||
# Diagnostics and Reporting Internals
|
||||
|
||||
## Scope
|
||||
The [output-file contract](../integrations/output-files.md) owns public
|
||||
schemas and the [operations guide](../operations.md) owns physical placement.
|
||||
|
||||
This document owns implementation flow and collaborators. Physical artifacts and
|
||||
retention belong to [operations](../operations.md). Serialized reports, ledger
|
||||
records, and utilization shapes belong to the
|
||||
[output-file contract](../integrations/output-files.md).
|
||||
`internal/framework/outcomes` contains processing-owned operation references,
|
||||
counts, failure reasons, filtering details, and work states. It imports no
|
||||
reporting, CLI, or provider package. The runner records those facts per initial
|
||||
section and module instance. `internal/framework/processreport` maps them once
|
||||
into `internal/core/reporting.Snapshot`, including nullable counts, grouped
|
||||
work-unit errors, separate run errors, and one recovery per successful logical
|
||||
operation. Reports order stages, sections, validators, and batches by semantic
|
||||
identity, never concurrent completion time. The core reporting package owns
|
||||
Audita's versioned YAML DTOs, fixed Markdown rendering, and brief CLI
|
||||
completion formatter. YAML redaction walks serialized string nodes so numeric
|
||||
and boolean values retain their types. Ordinary reports omit raw causes,
|
||||
transcript text, complete decisions, and token/timing payloads.
|
||||
Validation operations with unavailable decisions report only affected requested
|
||||
indices, even when other candidates in the batch continue. Ignored extra
|
||||
validator entries yield `validation_decisions_filtered` only for a successful
|
||||
operation; they do not count as filtered correction proposals. Failed operations
|
||||
retain their execution counts and never produce a recovery record.
|
||||
|
||||
## Run Directory and Artifact Writers
|
||||
`internal/core/diagnostics.OutputSession` owns active destination checks,
|
||||
transcript preflight, report initialization, regular-file replacement, and direct
|
||||
stream publication. Transcript input is protected from early report writes but
|
||||
may be the final transcript destination. The LLM runtime records successful reads
|
||||
through PromptKit's profile filesystem and supplies those source paths to CLI;
|
||||
no second filename-based discovery rule is used.
|
||||
CLI composes it after configuration/profile inspection and before reading
|
||||
inputs or dispatching model work. It does not create ordinary report parents.
|
||||
On a final write error it attempts other enabled reports and one failed-status
|
||||
pass; a published transcript is never erased by a later report failure.
|
||||
|
||||
`internal/core/diagnostics` creates run directories, writes snapshots and metadata,
|
||||
and evaluates retention. CLI orchestration supplies the selected configuration,
|
||||
paths, run outcome, and correction-presence information. Artifact names are
|
||||
centralized in diagnostics; their operator-facing inventory is in
|
||||
[operations](../operations.md#diagnostics-directory).
|
||||
|
||||
Snapshot writes are attempted at the relevant processing phases. Several diagnostic
|
||||
writes are best-effort, so report metadata can identify a location without proving
|
||||
that a file was written. The CLI distinguishes primary processing errors from
|
||||
secondary diagnostic/report write errors.
|
||||
|
||||
## Process Report Mapping
|
||||
|
||||
`internal/framework/processreport.Build` maps invocation facts, normalization and
|
||||
chunking summaries, and runner output into `reporting.ProcessReport`. Core reporting
|
||||
owns the Go data structures and JSON writer. The builder attaches diagnostic
|
||||
metadata when a run-directory path is available.
|
||||
|
||||
Module mapping copies outcomes from runner results, collects summary counts, and
|
||||
preserves available partial module results on failure. The CLI failure path does
|
||||
not pass normalization or chunking summaries to report construction even when
|
||||
some earlier processing completed. Consumer interpretation belongs to the
|
||||
[process-report contract](../integrations/output-files.md#process-report).
|
||||
|
||||
## Correction Ledger and Utilization
|
||||
|
||||
`BuildCorrectionLedger` flattens module changes, rejections, and failures into
|
||||
ledger entries. It associates validator records by proposal index within each
|
||||
module, separates execution classes through validator metadata, and sorts entries
|
||||
by module instance, proposal index, and disposition.
|
||||
|
||||
Runner instrumentation collects utilization; CLI orchestration writes it when
|
||||
run output is available. Report assembly does not independently measure calls.
|
||||
For the serialized forms see [ledger](../integrations/output-files.md#correction-ledger)
|
||||
and [utilization](../integrations/output-files.md#utilization-diagnostics).
|
||||
|
||||
## CLI Coordination and Retention
|
||||
|
||||
CLI orchestration writes caller-selected reports and run-local artifacts before
|
||||
asking diagnostics to apply retention. `HasSkippedCorrections` derives the
|
||||
retention input from runner outcomes. Keep the exact lifecycle definition in
|
||||
[operations](../operations.md#retention-behavior), not in another internal table.
|
||||
|
||||
On a processing failure, secondary report-write errors do not replace the original
|
||||
error. On the success path, an external report-write failure returns before the
|
||||
run-local report write; retention errors occur later. This ordering explains why
|
||||
artifacts and process exit status must be assessed together.
|
||||
|
||||
## Redaction
|
||||
|
||||
Effective configuration uses config redaction. LLM diagnostic and error paths
|
||||
use `llm.ConfiguredSecrets` and the runtime's redaction helpers. Keep these separate
|
||||
mechanisms aligned with the [architecture requirement](../policy/architecture.md#errors-and-secret-handling).
|
||||
Operational handling of transcript-bearing artifacts belongs to
|
||||
[operations](../operations.md#artifact-handling).
|
||||
Debug is independent. Only `diagnostics.debug` creates a unique retained child
|
||||
under `diagnostics.work_dir`. CLI passes empty diagnostics paths and nil writers
|
||||
when debug is off. With debug on, source/normalized snapshots, invocation,
|
||||
effective settings, origins, profiles, initial chunk summary, correction ledger,
|
||||
indexed execution/utilization, and model interactions are written. Runner
|
||||
candidate capture includes proposals later rejected or interrupted. Ledger
|
||||
mapping uses optional string pointers so an empty proposed replacement is
|
||||
serialized for every candidate and an empty applied replacement appears only
|
||||
when that candidate applied; see the [public artifact contract](../integrations/output-files.md#debug-only-files).
|
||||
Debug write failures reach the CLI through an explicit concurrency-safe diagnostic
|
||||
callback and produce redacted stderr messages without aborting processing.
|
||||
Completed validator decisions returned with a fatal batch error remain available
|
||||
to the debug ledger; they never authorize applying an incompletely validated edit.
|
||||
Runner decision records carry the validator's explicit approved, rejected, or
|
||||
unavailable disposition into the ledger. Unavailable candidates have a distinct
|
||||
top-level ledger disposition, while fatal interruptions remain interrupted when
|
||||
no rejection or unavailable decision was returned. Successful validation
|
||||
interaction payloads contain the full decoded response, final content, and
|
||||
reconciliation outcomes; ignored indices do not become ledger candidates.
|
||||
Every debug writer uses all configured backend secrets for redaction, including
|
||||
unselected backend credentials and source snapshots. Debug can still contain
|
||||
sensitive transcript and prompt content.
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/core/diagnostics/*_test.go`
|
||||
- `internal/framework/processreport/*_test.go`
|
||||
- `internal/core/reporting/report_test.go`
|
||||
- `internal/cli/run_test.go`
|
||||
- `cmd/audita/main_integration_test.go`
|
||||
Useful tests: `internal/core/reporting/*_test.go`,
|
||||
`internal/core/diagnostics/*_test.go`,
|
||||
`internal/framework/processreport/*_test.go`, and
|
||||
`cmd/audita/main_integration_test.go`.
|
||||
|
||||
@@ -1,68 +1,39 @@
|
||||
# Audita LLM Runtime
|
||||
|
||||
## Scope
|
||||
|
||||
This document describes the structured LLM runtime and scheduler behavior.
|
||||
|
||||
## Client Boundary
|
||||
|
||||
All runtime LLM calls go through `contracts.StructuredLLMClient`.
|
||||
`llm.NewRuntime` assembles one PromptKit engine for both roles. It loads embedded
|
||||
prompt and schema assets, registers application backends, and inspects the
|
||||
selected proposal and validation profiles offline. A role-bound `llm.RoleClient`
|
||||
implements `structuredoutput.Client`; proposal and validator contracts alias that
|
||||
interface. Both roles share engine-local backend capacity.
|
||||
|
||||
Primary adapter:
|
||||
- `internal/framework/llm/OpenAICompatibleClient`
|
||||
For each logical call, the adapter starts an operation context and a diagnostic
|
||||
record before preparation. It supplies the reserved hardening text, the selected
|
||||
profile and resolved backend credential, and the complete embedded schema/repair
|
||||
contract. `PrepareExecution` produces the exact messages and hashes retained in
|
||||
an Audita snapshot. The adapter checks the prepared backend and endpoint before
|
||||
`RunPrepared`. It records available final content, usage, model, and profile
|
||||
metadata and decodes only a validated result. Call hooks expose logical
|
||||
start/finish records and PromptKit execution events for debug utilization.
|
||||
|
||||
## Adapter and Response Schema Registry
|
||||
The [integration contract](../integrations/openai-compatible-llm.md) describes
|
||||
wire ownership and failure behavior. `structuredoutput.Failure` has a safe
|
||||
message, fixed reason code, typed cause, and recoverable/fatal/canceled class.
|
||||
Counts distinguish provider attempts, transient retries, and structural
|
||||
repairs, including on errors. Secrets from all configured backends, including unselected
|
||||
ones, are available to the diagnostics redaction boundary.
|
||||
|
||||
The adapter implements the [LLM wire contract](../integrations/openai-compatible-llm.md).
|
||||
It maps internal structured requests to HTTP messages and decodes provider content
|
||||
into the supplied Go output target. Endpoint, authentication, response shapes, and
|
||||
retry conditions are defined in that contract rather than repeated here.
|
||||
## Orchestration
|
||||
|
||||
`internal/framework/responseschema` stores the per-call response schema definitions
|
||||
and diagnostic metadata. Proposal generation and validation choose their schema
|
||||
before invoking the client. Provider strict mode does not replace local checks.
|
||||
|
||||
## Error Classification
|
||||
|
||||
The adapter marks retryable errors internally and uses context state and attempt
|
||||
count to decide whether to retry. It sanitizes errors before returning them.
|
||||
`internal/framework/structuredoutput` classifies malformed output for proposal
|
||||
and validator call sites, which can turn those errors into warnings and rejected
|
||||
section/batch work. Other errors propagate through the runner.
|
||||
|
||||
## Scheduling and Concurrency
|
||||
|
||||
`internal/framework/llm/Scheduler` provides FIFO, context-aware permit gating.
|
||||
|
||||
CLI orchestration composes global and stage schedulers; runner instrumentation
|
||||
measures them. Limit definitions and inheritance belong to
|
||||
[configuration](../config.md#concurrency).
|
||||
|
||||
Scheduler release is guarded to avoid permit leaks on cancellation/error. Runner
|
||||
collects utilization observations; their serialized shape is defined by the
|
||||
[output-file contract](../integrations/output-files.md#utilization-diagnostics).
|
||||
|
||||
## Diagnostics and Redaction
|
||||
|
||||
`internal/framework/llm/DiagnosticsWriter` writes request/response/error artifacts.
|
||||
|
||||
Configured secrets are derived from `llm.ConfiguredSecrets(cfg)` and redacted from:
|
||||
- diagnostics payloads;
|
||||
- surfaced runtime/adapter errors.
|
||||
|
||||
## Changing LLM Integration or Scheduling
|
||||
|
||||
Review client, scheduler, redaction, and structured-output malformed-response
|
||||
coverage alongside runner orchestration and utilization tests. Use the
|
||||
[testing policy](../policy/testing.md) to select sufficient checks; preserve the
|
||||
[LLM integration contract](../integrations/openai-compatible-llm.md) and
|
||||
[architecture invariants](../policy/architecture.md#correction-and-concurrency-invariants).
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/framework/llm/openai_compatible_client_test.go`
|
||||
- `internal/framework/llm/scheduler_test.go`
|
||||
- `internal/framework/llm/diagnostics_test.go`
|
||||
- `internal/framework/responseschema/registry_test.go`
|
||||
- `internal/framework/structuredoutput/malformed_test.go`
|
||||
- `internal/framework/runner/*_test.go` (orchestration and utilization)
|
||||
CLI processing supplies two role clients from the same engine to the runner.
|
||||
PromptKit's per-backend admission applies to both roles and every generation,
|
||||
retry, and repair. The runner still processes module instances in order,
|
||||
collects concurrent section work deterministically, and cancels remaining
|
||||
section work only on fatal failures. `llm.DiagnosticsWriter` remains the protected
|
||||
request/response artifact writer. `llm.ObservationCollector` captures metadata
|
||||
events under a short mutex and freezes them after runner work joins. Its logical
|
||||
records include preparation failures; summary usage comes from terminal events
|
||||
once per call and retains knownness and completeness for each field. Runner wall
|
||||
and validator timings remain separate from provider attempts. Ordinary reports
|
||||
derive compact runner outcomes; full observations are debug-only.
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
## Scope
|
||||
|
||||
This document covers internal module contracts and packages. User-selectable keys
|
||||
and the default sequence belong to [configuration](../config.md#pipeline-and-output).
|
||||
and the default sequence belong to [configuration](../config.md#settings).
|
||||
|
||||
## Module Contract
|
||||
|
||||
@@ -35,34 +35,49 @@ Shared proposal-generation plumbing is centralized in:
|
||||
- `internal/framework/proposal_generation`
|
||||
|
||||
Module packages own:
|
||||
- prompt selection (`internal/prompts` prompt IDs);
|
||||
- module-specific prompt payload construction.
|
||||
- prompt selection (`internal/prompts` prompt IDs).
|
||||
|
||||
Shared prompt helpers live in `internal/framework/promptcontext`.
|
||||
`proposal_generation.BuildProposalVariables` encodes the glossary, selected
|
||||
transcript section, and trimmed description once for all four modules. PromptKit
|
||||
renders their embedded definitions from those variables and validates the
|
||||
correction-set schema before Audita maps proposals. Domain validators still
|
||||
check confidence, applicability, protected glossary terms, and replacement
|
||||
policy. Model-facing segment IDs are one-based positions within the proposal
|
||||
request. Generation maps returned references back to the current chunk's actual
|
||||
segment IDs, preserving response order and multiple proposals per segment.
|
||||
Unknown references become invalid-target candidates for the existing domain
|
||||
checks; they are never interpreted as actual transcript IDs. Debug request
|
||||
metadata records `segment_ids` in local-reference order (reference 1 is the
|
||||
first entry). Shared context encoding lives in `internal/framework/promptcontext`.
|
||||
An empty `corrected_text` deletes the identified span when the resulting segment
|
||||
remains nonblank. Whitespace-only original spans can express spacing corrections;
|
||||
matching stays literal under each module's replacement policy. No automatic
|
||||
whitespace cleanup follows application.
|
||||
|
||||
## Validator Chain Ownership
|
||||
|
||||
Built-in chains are resolved in `internal/validators` per module key.
|
||||
Module packages call the built-in chain resolver at construction.
|
||||
|
||||
## Failure and Warning Behavior
|
||||
## Failure and Outcome Behavior
|
||||
|
||||
Module construction failures propagate to orchestration; execution failures can
|
||||
preserve partial module results. Malformed structured proposal payloads become
|
||||
warnings and section-level rejection. Consumer-visible outcomes are defined by
|
||||
the [report contract](../integrations/output-files.md#module-outcomes).
|
||||
Module construction failures are fatal. Recoverable proposal model failures
|
||||
record an unsuccessful operation and yield no candidates for that section;
|
||||
other sections and module instances continue. Fatal adapter or application
|
||||
failures cancel peers and retain partial observations. Consumer-visible
|
||||
outcomes are defined by the [report contract](../integrations/output-files.md).
|
||||
|
||||
## Changing Modules
|
||||
|
||||
1. Add or update the narrowly scoped package under `internal/modules/<module_key>`.
|
||||
2. Keep module-specific prompt ownership in the module and `internal/prompts`;
|
||||
see [prompt rendering and metadata](prompts.md).
|
||||
2. Keep module prompt selection in the module and its definition in
|
||||
`internal/prompts`; see [prompt assets](prompts.md).
|
||||
3. Wire module registration and catalog resolution through framework module
|
||||
registration and the core module catalog. Update selectable keys in the
|
||||
[configuration contract](../config.md) when they change.
|
||||
4. Verify replacement policy and validator-chain selection against the
|
||||
[architecture requirements](../policy/architecture.md#correction-and-concurrency-invariants).
|
||||
5. Review module, prompt-rendering, and proposal-generation tests. Inspect parity
|
||||
5. Review module, prompt-variable, and proposal-generation tests. Inspect parity
|
||||
and release fixtures when public output changes, choosing sufficient coverage
|
||||
under the [testing policy](../policy/testing.md).
|
||||
6. Update this document for implementation changes and the appropriate public
|
||||
|
||||
@@ -6,13 +6,13 @@
|
||||
JSON envelopes and segment fields are defined by the
|
||||
[output-file contract](../integrations/output-files.md#corrected-transcript);
|
||||
selectable values and defaults belong to
|
||||
[configuration](../config.md#pipeline-and-output).
|
||||
[configuration](../config.md#settings).
|
||||
|
||||
## Resolution and Encoding
|
||||
|
||||
Configuration validation checks registry support. CLI orchestration resolves the
|
||||
selected encoder, passes the final transcript to it, and routes the encoded bytes
|
||||
to the destination defined by the [CLI contract](../cli.md#process-output-and-exit-behavior).
|
||||
to the destination defined by the [CLI contract](../cli.md#streams-and-exit-status).
|
||||
An unsupported key fails validation or resolution rather than selecting an
|
||||
implicit fallback.
|
||||
|
||||
|
||||
@@ -47,15 +47,18 @@ Supporting areas:
|
||||
|
||||
## Current Scope
|
||||
|
||||
Runtime state is held in memory plus per-run diagnostics artifacts. Audita does
|
||||
Runtime state is held in memory; optional debug artifacts are retained only
|
||||
when requested. Audita does
|
||||
not currently implement a plugin framework, generic workflow engine,
|
||||
resume/checkpoint/manifest system, remote storage, or multi-process service mode.
|
||||
Its provider adapter is OpenAI-compatible HTTP behind the structured LLM client
|
||||
contract. These are current implementation boundaries, not bans on future work.
|
||||
Its role-bound PromptKit adapter implements the structured LLM client contract.
|
||||
These are current implementation boundaries, not bans on future work.
|
||||
|
||||
## Main Execution Path (`audita process`)
|
||||
|
||||
The CLI prepares input and configuration, invokes the runner, and writes results.
|
||||
The CLI inspects configuration and profiles, preflights and initializes active
|
||||
outputs, optionally creates debug evidence, invokes the runner, and finalizes
|
||||
transcript and reports.
|
||||
The [pipeline](pipeline.md) owns detailed state transitions; the
|
||||
[operations guide](../operations.md) owns the operator workflow and physical
|
||||
artifact lifecycle.
|
||||
@@ -67,8 +70,8 @@ transcript and glossary schemas, normalization and chunking, output encoding,
|
||||
diagnostics artifact helpers, and public report structures.
|
||||
|
||||
Framework components implement module and validator contracts, proposal
|
||||
generation and application, prompt context, runner orchestration, LLM scheduling
|
||||
and transport, response schemas, and process-report/correction-ledger assembly.
|
||||
generation and application, prompt context, runner orchestration, PromptKit
|
||||
execution and observation mapping, response schemas, and process-report/correction-ledger assembly.
|
||||
Domain packages provide correction modules, built-in validator chains and
|
||||
execution metadata, and embedded prompt assets.
|
||||
|
||||
|
||||
@@ -1,66 +1,45 @@
|
||||
# Audita Internal Pipeline
|
||||
|
||||
## Scope
|
||||
The [input contract](../integrations/transcript-glossary-files.md) owns input
|
||||
formats; [configuration](../config.md) owns settings; the
|
||||
[output contract](../integrations/output-files.md) owns serialized results.
|
||||
|
||||
This document describes processing collaborators and state transitions. External
|
||||
input definitions belong to the [input-file contract](../integrations/transcript-glossary-files.md),
|
||||
settings to [configuration](../config.md), and serialized results to the
|
||||
[output-file contract](../integrations/output-files.md).
|
||||
1. CLI filesystem adapters read transcript and glossary files. Schema parsing
|
||||
produces source objects; transcript segments enter canonical normalization.
|
||||
2. Normalization sorts and merges segments using configured time and token
|
||||
limits, reassigning IDs where needed.
|
||||
3. The runner partitions the normalized transcript exactly once before its
|
||||
module loop. Initial section identity, segment membership, and token
|
||||
estimates stay fixed while text evolves.
|
||||
4. Module instances execute in configured order. Sections may propose
|
||||
concurrently, but proposal indices follow section order and response order.
|
||||
5. Deterministic validators precede LLM-backed validators. Every submitted
|
||||
candidate receives one decision per validator on a normal return. Fatal returns
|
||||
may preserve a checked subset of completed decisions as diagnostic evidence.
|
||||
Recoverable failed batches mark their candidates unavailable; successful
|
||||
survivors continue through all required validators.
|
||||
6. Only fully approved proposals reach application. Corrections apply in
|
||||
proposal-index order within their owning section. Existing replacement
|
||||
policies and apply-time safety checks resolve stale conflicts.
|
||||
7. CLI encodes the final transcript and publishes active files through the
|
||||
output session. `framework/processreport` builds one snapshot for YAML,
|
||||
Markdown, and CLI totals. Debug evidence is captured separately when enabled.
|
||||
|
||||
## Inputs
|
||||
`runner.Run` returns the final working transcript, initial section inventory,
|
||||
explicit `not_started`/`interrupted`/`completed` work units, compact model
|
||||
operation outcomes, and per-module results. Candidate payloads are retained
|
||||
in the returned output only when debug capture is requested. A work unit can
|
||||
complete with model errors and applied corrections. Recoverable model failures
|
||||
never trigger shared cancellation. Fatal setup, internal, application, or
|
||||
parent-cancellation errors cancel peers, join workers, and preserve earlier
|
||||
committed module text for partial reports; no completed transcript is published.
|
||||
|
||||
Pipeline inputs are:
|
||||
- effective config (`internal/core/config`);
|
||||
- transcript JSON (`internal/core/schema`);
|
||||
- glossary YAML (`internal/core/schema`).
|
||||
Required transcript/report write failures are fatal. Optional debug writes
|
||||
report safely to stderr without changing decisions or exit status. See
|
||||
[operations](../operations.md) for file placement and failure handling, and
|
||||
[architecture policy](../policy/architecture.md#correction-and-concurrency-invariants)
|
||||
for invariants.
|
||||
|
||||
## Pipeline Phases
|
||||
|
||||
1. **Input loading:** CLI filesystem adapters read inputs; schema parsing produces
|
||||
source transcript and glossary objects. Source segments are converted to the
|
||||
canonical transcript shape before normalization.
|
||||
2. **Normalization:** the normalizer sorts and merges segments using its configured
|
||||
time/token rules, reassigns IDs, and produces summary counters.
|
||||
3. **Chunking:** the chunker partitions the normalized transcript and returns
|
||||
sections and token estimates, or fails when constraints are impossible.
|
||||
4. **Proposal generation:** the runner executes configured module instances in
|
||||
sequence. Sections can generate proposals concurrently; results are collected
|
||||
for deterministic indexing.
|
||||
5. **Validation:** deterministic validators run before LLM-backed validators.
|
||||
LLM validation is batched by prompt budget; candidate decisions feed application.
|
||||
6. **Application:** `internal/framework/proposals` clones transcript state and
|
||||
applies approved proposals in index order through replacement-policy checks.
|
||||
7. **Serialization and reporting:** the CLI encodes the final transcript, maps
|
||||
runner output into reports, writes artifacts, and delegates retention to
|
||||
diagnostics. See [report assembly](diagnostics-reporting.md).
|
||||
|
||||
## Runner Outputs
|
||||
|
||||
`runner.Run` returns:
|
||||
- final transcript;
|
||||
- per-module results;
|
||||
- utilization diagnostics.
|
||||
|
||||
CLI/reporting then map this into process report and diagnostics artifacts.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
CLI orchestration labels failures at the phase boundary. Runner execution failures
|
||||
can return completed or partial module results, which report assembly preserves.
|
||||
[Report fields](../integrations/output-files.md#process-report) define how consumers
|
||||
see these outcomes; [CLI semantics](../cli.md#process-output-and-exit-behavior)
|
||||
define streams and exits. Diagnostic write failures do not always abort processing.
|
||||
|
||||
## Ordering and State
|
||||
|
||||
The runner mutates its working transcript between module instances, after collecting
|
||||
section results and validating proposals. Proposal indexes follow section order;
|
||||
reports are derived from actual run results. Global requirements are owned by the
|
||||
[architecture policy](../policy/architecture.md#correction-and-concurrency-invariants).
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/framework/runner/runner_test.go`
|
||||
- `internal/framework/proposal_generation/*_test.go`
|
||||
- `internal/cli/run_test.go`
|
||||
- `cmd/audita/main_integration_test.go`
|
||||
Key tests: `internal/framework/runner/*_test.go`,
|
||||
`internal/framework/validators/*_test.go`, and
|
||||
`cmd/audita/main_integration_test.go`.
|
||||
|
||||
@@ -1,62 +1,59 @@
|
||||
# Audita Prompt Registry
|
||||
|
||||
## Scope
|
||||
`internal/prompts` embeds seven PromptKit YAML definitions under
|
||||
`assets/prompts`. The IDs and selected version remain:
|
||||
|
||||
This document describes embedded prompt assets, prompt metadata, and rendering behavior.
|
||||
- `modules.glossary.proposal` (`v1`)
|
||||
- `modules.homophones.proposal` (`v1`)
|
||||
- `modules.spoken_word.proposal` (`v1`)
|
||||
- `modules.grammar.proposal` (`v1`)
|
||||
- `validators.spoken_form_plausibility` (`v1`)
|
||||
- `validators.meaning_reversal_review` (`v1`)
|
||||
- `validators.editorial_review` (`v1`)
|
||||
|
||||
## Registry Ownership
|
||||
PromptKit reads the embedded definitions and renders their system and user
|
||||
messages from named string variables. The role-bound adapter supplies
|
||||
`HardeningPolicy` alongside `ProposalContract` and `ValidationContract` from
|
||||
embedded fragments under `assets/shared`, after caller variables so callers cannot replace those
|
||||
instructions. Common literal-span and identity guidance lives in the proposal
|
||||
fragment; shared candidate-reference, context, and confidence guidance lives in
|
||||
the validation fragment. Module-specific policies and examples remain in YAML.
|
||||
Proposal prompts receive the section JSON, glossary JSON where used, and transcript description block.
|
||||
Validator prompts receive the correction payload and description block.
|
||||
The prompt assets retain the prior task instructions and output-schema
|
||||
association. The role-bound runtime supplies the complete JSON Schema output contract
|
||||
and repair count; the definition does not own an operator-facing repair default.
|
||||
The correction-set schema requires a string replacement and accepts an empty
|
||||
string. Proposal guidance uses it for eligible span deletion and permits literal
|
||||
spacing edits; validator guidance evaluates the full before/after segment under
|
||||
each validator's existing remit. A blank resulting segment remains a domain
|
||||
rejection, not a structural schema error.
|
||||
|
||||
Prompt registry lives in `internal/prompts` and embeds assets under `internal/prompts/assets/**`.
|
||||
`internal/framework/responseschema` remains the only embedded source for
|
||||
`correction_set.v1.json` and `validator_decision_set.v1.json`. It exposes that
|
||||
filesystem to PromptKit and retains Audita's schema-key lookup. PromptKit's
|
||||
prepared execution supplies definition and rendered hashes; Audita does not
|
||||
recompute or promise equality with hashes from the removed template renderer.
|
||||
The small Audita registry records only prompt ID, version, source, and embedded
|
||||
path. Prepared runtime metadata belongs in diagnostics rather than the registry.
|
||||
|
||||
Registered prompt IDs:
|
||||
- `modules.glossary.proposal`
|
||||
- `modules.homophones.proposal`
|
||||
- `modules.spoken_word.proposal`
|
||||
- `modules.grammar.proposal`
|
||||
- `validators.spoken_form_plausibility`
|
||||
- `validators.meaning_reversal_review`
|
||||
- `validators.editorial_review`
|
||||
- `validators.grammar_review`
|
||||
- `validators.spoken_word_review`
|
||||
The structured-call contract carries prompt ID/version, string variables,
|
||||
response-schema key, and logical-call attribution. Module and validator callers
|
||||
pass these values through the role-bound adapter. See the
|
||||
[LLM runtime](llm-runtime.md) for runtime assembly and
|
||||
[configuration](../config.md) for operator-selected profiles.
|
||||
|
||||
## Metadata Model
|
||||
Proposal instructions describe literal matching, each module's replacement policy,
|
||||
source-based proposal identity, and confidence in whether an edit is warranted.
|
||||
Proposal `id` and validator `correction_index` values are short, one-based
|
||||
references scoped to each request, not internal transcript/proposal identities.
|
||||
Mapping belongs to [proposal generation](modules.md) and
|
||||
[validator reconciliation](validators.md). Validator instructions distinguish
|
||||
confidence in the verdict from the verdict itself. Compact output examples
|
||||
illustrate the supplied schema without requiring fixed JSON formatting or key
|
||||
order. The grammar article exception and glossary mishearing exception apply
|
||||
consistently across their instructions.
|
||||
|
||||
Each prompt has metadata:
|
||||
- `prompt_id`
|
||||
- `prompt_version`
|
||||
- `prompt_source`
|
||||
- `embedded_path`
|
||||
- `sha256`
|
||||
|
||||
Current source/version values:
|
||||
- `prompt_source = builtin`
|
||||
- `prompt_version = v1`
|
||||
|
||||
## Rendering
|
||||
|
||||
`prompts.RenderUserSystem(promptID, data)` renders system/user templates.
|
||||
|
||||
Template behavior:
|
||||
- uses Go `text/template`;
|
||||
- `missingkey=error` is enabled;
|
||||
- output is trimmed.
|
||||
|
||||
A shared hardening fragment is embedded once and referenced by prompt templates.
|
||||
|
||||
## Prompt Context Inputs
|
||||
|
||||
Shared prompt payload helpers:
|
||||
- transcript section JSON (`internal/framework/promptcontext/MarshalTranscriptSectionJSON`)
|
||||
- transcript description block (`TranscriptDescriptionBlock`)
|
||||
|
||||
Modules and LLM validators provide typed data maps to render prompt assets.
|
||||
|
||||
## Diagnostics Integration
|
||||
|
||||
Prompt metadata is attached to proposal/validator diagnostics request metadata using `Metadata.DiagnosticsMap()`.
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/prompts/registry_test.go`
|
||||
- `internal/framework/promptcontext/*_test.go`
|
||||
- module and validator prompt builder tests
|
||||
Only the three active validator definitions are retained. The unused internal
|
||||
`grammar_review` and `spoken_word_review` definitions and type selections were
|
||||
removed; built-in chains and operator configuration are unchanged.
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
This document describes internal validator composition, execution order, and
|
||||
decision handling. These registry keys identify implementation components; they
|
||||
are not independently selectable through [configuration](../config.md#pipeline-and-output).
|
||||
are not independently selectable through [configuration](../config.md#settings).
|
||||
|
||||
## Ownership
|
||||
|
||||
@@ -23,7 +23,8 @@ Deterministic:
|
||||
- `proposal_shape`
|
||||
- `confidence_threshold`
|
||||
- `original_text_presence`
|
||||
- `non_empty_corrected_text`
|
||||
- `non_empty_resulting_segment` — rejects edits that would leave the segment
|
||||
blank, regardless of replacement-span length.
|
||||
- `no_effect`
|
||||
- `protected_terms`
|
||||
|
||||
@@ -42,24 +43,55 @@ Glossary, homophones, spoken_word, and grammar each resolve a fixed ordered chai
|
||||
For each module section:
|
||||
1. run deterministic validators;
|
||||
2. run LLM-backed validators;
|
||||
3. record decisions and warnings;
|
||||
3. record decisions and compact operation outcomes;
|
||||
4. carry only approved proposals forward.
|
||||
|
||||
Decision cardinality is enforced: each candidate proposal must receive exactly one decision per validator.
|
||||
Decision cardinality is enforced inside Audita: each candidate proposal must
|
||||
receive exactly one outcome per validator it reaches. The runner still rejects
|
||||
duplicate, missing, or unknown identities in an internal validator result.
|
||||
|
||||
## LLM Validator Batching
|
||||
|
||||
LLM validators:
|
||||
- build canonical validation request payloads;
|
||||
- batch using the [configured prompt budget](../config.md#concurrency);
|
||||
- call structured LLM client using response schema registry.
|
||||
- batch using `validation.max_prompt_tokens` and the existing domain token estimator;
|
||||
- select an embedded prompt ID and pass payload/description variables to the
|
||||
role-bound structured client with the validator-decision schema.
|
||||
|
||||
Oversized single proposals are rejected with `validator_input_too_large`.
|
||||
Malformed LLM validator responses are downgraded to warnings and rejected batch decisions.
|
||||
Oversized single inputs are rejected with a consolidated
|
||||
`validation_input_too_large` operation, without a provider call. Recoverable
|
||||
model failures make only their batch candidates unavailable while other
|
||||
batches continue. PromptKit validates each complete response against the
|
||||
validator-decision JSON Schema and performs configured structural repair before
|
||||
Audita receives decoded decisions. Each actual batch presents candidates
|
||||
numbered from 1, independent of their internal proposal indices. Audita then
|
||||
reconciles those local references: one usable entry keeps its
|
||||
approval or editorial rejection; a missing or duplicate requested entry makes
|
||||
only that candidate unavailable; unexpected indices are ignored. An empty list
|
||||
makes all requested candidates unavailable. Decisions and unavailable indices
|
||||
map back to internal proposal identities; results are ordered by request identity
|
||||
and satisfy strict internal cardinality before the runner advances
|
||||
approved candidates. An unavailable candidate never advances as approved.
|
||||
Invalid confidence in a decoded test double also makes only its candidate
|
||||
unavailable; production schema validation still rejects out-of-range confidence
|
||||
before reconciliation. A partially unavailable batch records one failed
|
||||
operation with only those requested indices, while a successful batch with
|
||||
ignored extras records a decision-filtering recovery. Fatal errors stop the
|
||||
chain without returning unchecked candidates as approved. A local result builder
|
||||
retains already-collected decisions and operations on early returns; full
|
||||
cardinality checks and final sorting remain explicit on completed paths.
|
||||
Reconciliation makes no semantic retry or extra model call; guided recovery remains
|
||||
[deferred](../roadmap/future.md#domain-aware-response-recovery).
|
||||
|
||||
Debug request metadata records `proposal_indices` in local-reference order
|
||||
(reference 1 is the first entry). Raw decoded responses and ignored indices use
|
||||
local references; reconciled decisions and unavailable indices use internal
|
||||
proposal indices. Batch token estimates use the same locally numbered payload
|
||||
as prompt encoding.
|
||||
|
||||
## Decision and Rejection Reporting
|
||||
|
||||
Runner collects decisions, rejected candidates, and warning records. Ledger
|
||||
Runner collects decisions, rejected candidates, and operation outcomes. Ledger
|
||||
assembly groups decisions by deterministic versus LLM-backed execution metadata.
|
||||
The serialized names and fields belong to the
|
||||
[report and ledger contract](../integrations/output-files.md); implementation mapping
|
||||
@@ -85,3 +117,40 @@ is covered by [report assembly](diagnostics-reporting.md).
|
||||
- `internal/framework/validators/*_test.go`
|
||||
- `internal/framework/processreport/correction_ledger_test.go`
|
||||
- `internal/cli/run_test.go`
|
||||
|
||||
## Supplemental Validation Context
|
||||
|
||||
Request construction prepares one segment-position lookup and protected vocabulary
|
||||
for all candidates. Glossary selection remains candidate-specific, including the
|
||||
proposed replacement text.
|
||||
|
||||
Each candidate includes the immediately preceding and following source segments
|
||||
available within its fixed chunk, labeled `preceding` or `following` with speaker,
|
||||
text, and categories, plus glossary entries whose names, aliases, or
|
||||
recognized plurals appear in its original/resulting text or those neighbors.
|
||||
Glossary matching reuses the protected vocabulary; entries retain glossary order.
|
||||
Model-facing candidate and neighboring context omit internal segment IDs.
|
||||
This supplies evidence such as a troupe definition for a proposed `Jesters`
|
||||
correction or an adjacent speaker turn explaining deliberate repetition. A correction
|
||||
already clear from its own segment does not require such evidence. No audio,
|
||||
out-of-chunk context, or complete transcript is implied.
|
||||
|
||||
Context is serialized inside the validation item and included in the existing
|
||||
payload token estimate. If one enriched item exceeds the configured limit, optional
|
||||
neighbors are removed from the end first, then glossary entries from the end,
|
||||
until it fits or only required candidate data remains. Required before/after text
|
||||
is never truncated. Existing oversized-candidate handling still applies if that
|
||||
required data does not fit. Context is supplemental, not a promise that every
|
||||
relevant definition or neighboring utterance always fits. This uses the existing
|
||||
payload estimator, not a new guarantee about the provider's full context window.
|
||||
|
||||
## Protected-Name Repetition
|
||||
|
||||
Spoken-word cleanup may reduce repeated protected-name occurrences when every
|
||||
original protected identity still appears in the resulting segment and surviving
|
||||
spellings satisfy existing capitalization checks. Full-segment preview supports
|
||||
both a replacement of the repeated phrase and deletion of a redundant span. This
|
||||
exception only relaxes the deterministic count guard; editorial and meaning-reversal
|
||||
validation still decide whether the repetition was accidental and the edit preserves
|
||||
meaning. Intentional emphasis and substantive references remain protected by those
|
||||
reviews. Other modules retain their existing protection rules.
|
||||
|
||||
135
docs/migrations/v2.md
Normal file
135
docs/migrations/v2.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# Migrating to Audita v2
|
||||
|
||||
Audita v2 adopts ConfigKit for configuration and PromptKit for LLM execution.
|
||||
Version-1 configuration files, environment overlays, and retired flags are not
|
||||
accepted. Copy a maintained [local](../../examples/minimal-config.yml) or
|
||||
[service](../../examples/production-config.yml) example, then select real models
|
||||
in the accompanying `examples/profiles/` files. A testing overlay is shown at
|
||||
`examples/overlays/testing.yml`; its model choice carries no cost, speed, or
|
||||
quality guarantee.
|
||||
|
||||
## Configuration and invocation
|
||||
|
||||
Use `version: 2` in a YAML main file. `--config` selects that file; otherwise
|
||||
`process` and `config print-effective` look at
|
||||
`/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`.
|
||||
`config validate` requires `--config`. `AUDITA_CONFIG` no longer selects a file.
|
||||
Settings resolve from defaults, main file, optional `--overlay <name>` in
|
||||
`overlays/<name>.yml`, then explicitly supplied setting flags. General `AUDITA_*`
|
||||
environment overlays no longer apply. Unknown fields and version-1 files fail;
|
||||
there is no automatic converter.
|
||||
|
||||
Keep transcript as the positional path and `--glossary` and `--output` as
|
||||
invocation paths. The former `--report-json` output has been replaced by the
|
||||
default `errors.yml`, `recoveries.yml`, and `report.md` bundle; use
|
||||
`--reporting=false` to disable it or `--report-dir` for stdout output. Keep
|
||||
`pipeline.modules` / `--modules` and
|
||||
`output.schema` / `--output-schema`. An explicit `pipeline.modules: []` or
|
||||
`--modules ''` runs no correction modules; omission uses the default sequence.
|
||||
Ordered repetitions still matter.
|
||||
|
||||
Move `llm.proposal.base_url` and `llm.validation.base_url` to a shared
|
||||
`backends.<id>.endpoint` when both roles use one service. Move old role models to
|
||||
PromptKit profile files and select them with `llm.proposal_profile` and
|
||||
`llm.validation_profile`, or `--proposal-profile` and
|
||||
`--validation-profile`. Omitted validation selection inherits the final proposal
|
||||
selection; explicit empty or null is invalid. `profiles_dir` defaults to
|
||||
`profiles` beside the main config. Complete profiles name a registered backend
|
||||
and model. Per-role `--base-url`, `--validation-base-url`, `--model`, and
|
||||
`--validation-model` are retired.
|
||||
|
||||
Move old `llm.*.api_key_env` to `backends.<id>.credential`, a scalar name of a
|
||||
nonempty environment variable, such as `credential: AUDITA_SERVICE_KEY`.
|
||||
Credentials on every surviving backend must resolve, including unselected
|
||||
backends. Omit or set null for an unauthenticated local backend. Profiles cannot
|
||||
perform another key lookup. `--llm-api-key` and `--validation-llm-api-key` are
|
||||
retired. `AUDITA_LLM_API_KEY`, `OPENROUTER_API_KEY`, and
|
||||
`AUDITA_VALIDATION_LLM_API_KEY` have no implicit meaning, though any of those
|
||||
names can be used explicitly as a credential reference.
|
||||
|
||||
Replace `concurrency.total_llm`, `.proposal_llm`, and `.validation_llm` plus
|
||||
`--total-llm-concurrency`, `--llm-concurrency`,
|
||||
`--proposal-llm-concurrency`, and `--validation-llm-concurrency` with
|
||||
`backends.<id>.concurrency` (default 1) and optional `queue_capacity` (default
|
||||
1024; zero means no waiting admission). One backend capacity is shared by both
|
||||
roles and all retry/repair attempts within this Audita process. It does not
|
||||
coordinate other processes.
|
||||
|
||||
Move old role `timeout` values and `--llm-timeout-seconds` /
|
||||
`--validation-llm-timeout-seconds` to profile `timeout_seconds`, a per-attempt
|
||||
limit, and `llm.operation_timeout`, a Go duration such as `15m` covering the
|
||||
whole logical call. Replace old `max_retries` and `--max-retries` /
|
||||
`--validation-max-retries` with common `llm.transient_retries` (additional
|
||||
eligible attempts per generation) and `llm.repair_attempts` (additional
|
||||
generations for structurally invalid JSON). Explicit zero disables either
|
||||
feature. Cancellation and the operation deadline cover capacity waiting,
|
||||
backoff, generation, and repairs. PromptKit owns which failures can retry or
|
||||
repair; Audita does not retry semantic rejections.
|
||||
|
||||
The following domain settings and explicit flags retain their meaning:
|
||||
|
||||
| File field | Explicit flag |
|
||||
| --- | --- |
|
||||
| `validation.max_prompt_tokens` | `--validation-max-prompt-tokens` |
|
||||
| `chunking.target_sections` | `--target-sections` |
|
||||
| `chunking.max_section_tokens`, `.min_section_tokens` | `--max-section-tokens`, `--min-section-tokens` |
|
||||
| `normalization.max_segment_gap`, `.ellipsis_gap`, `.max_segment_duration`, `.max_segment_tokens` | `--normalize-max-segment-gap`, `--normalize-ellipsis-gap`, `--normalize-max-segment-duration`, `--normalize-max-segment-tokens` |
|
||||
| `thresholds.glossary`, `.homophones`, `.spoken_word`, `.grammar` | `--glossary-confidence-threshold`, `--homophones-confidence-threshold`, `--spoken-word-confidence-threshold`, `--grammar-confidence-threshold` |
|
||||
| `context.description` | `--transcript-description` |
|
||||
| `reporting.enabled`, `.stdout_dir` | `--reporting[=true\|false]`, `--report-dir` |
|
||||
| `diagnostics.debug`, `.work_dir` | `--debug[=true\|false]`, `--work-dir` |
|
||||
|
||||
Normalization gaps and durations are **numeric seconds**, so replace `4s`,
|
||||
`3.5s`, and `60s` with `4.0`, `3.5`, and `60.0`. The operation timeout remains a
|
||||
Go duration string. `chunking.target_sections` may be omitted or null for
|
||||
automatic chunking; zero is invalid. Null is not a general synonym for omission:
|
||||
nonoptional scalar settings reject it. Empty `context.description` clears
|
||||
context. Legacy `diagnostics.retention` and `--work-dir-retention` are removed.
|
||||
Debug is off by default and its directory is retained whenever enabled.
|
||||
|
||||
Remove old implicit variables `AUDITA_MODULES`, `AUDITA_MODEL`,
|
||||
`AUDITA_VALIDATION_MODEL`, `AUDITA_BASE_URL`, `AUDITA_VALIDATION_BASE_URL`,
|
||||
`AUDITA_LLM_TIMEOUT_SECONDS`, `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS`,
|
||||
`AUDITA_MAX_RETRIES`, `AUDITA_VALIDATION_MAX_RETRIES`,
|
||||
`AUDITA_TOTAL_LLM_CONCURRENCY`, `AUDITA_LLM_CONCURRENCY`,
|
||||
`AUDITA_PROPOSAL_LLM_CONCURRENCY`, `AUDITA_VALIDATION_LLM_CONCURRENCY`,
|
||||
`AUDITA_VALIDATION_MAX_PROMPT_TOKENS`, `AUDITA_TARGET_SECTIONS`,
|
||||
`AUDITA_MAX_SECTION_TOKENS`, `AUDITA_MIN_SECTION_TOKENS`,
|
||||
`AUDITA_NORMALIZE_MAX_SEGMENT_GAP`, `AUDITA_NORMALIZE_ELLIPSIS_GAP`,
|
||||
`AUDITA_NORMALIZE_MAX_SEGMENT_DURATION`, `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS`,
|
||||
`AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD`,
|
||||
`AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD`,
|
||||
`AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD`,
|
||||
`AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD`, `AUDITA_WORK_DIR`, and
|
||||
`AUDITA_WORK_DIR_RETENTION`. Put those values in the named fields above or pass
|
||||
their surviving explicit flags.
|
||||
|
||||
## Inspecting and operating
|
||||
|
||||
Run `audita config validate --config <file>` before processing. Use
|
||||
`audita config print-effective --config <file> --view settings|profiles|explain`
|
||||
to inspect redacted ConfigKit YAML, selected PromptKit profile controls, or
|
||||
field origins and derivations. Each view resolves both roles offline without
|
||||
calling a provider. `--overlay testing` selects the same overlay for validation,
|
||||
inspection, and processing. The old effective-config JSON format is replaced by
|
||||
YAML; settings YAML alone does not show profile-owned model controls.
|
||||
|
||||
The [ordinary output contract](../integrations/output-files.md) now uses
|
||||
version-1 `audita-errors` and `audita-recoveries` YAML documents plus a fixed
|
||||
Markdown overview. Compact outcomes distinguish logical operations from
|
||||
provider attempts. With debug enabled, utilization remains grouped by backend
|
||||
capacity and usage fields retain known/complete flags. Debug runs write
|
||||
`effective-config.redacted.yaml`, `selected-profiles.yaml`,
|
||||
`config-origins.yaml`, `utilization-diagnostics.json`, and
|
||||
`llm-execution.json`. Request and response artifacts remain redacted debugging
|
||||
material. See [operations](../operations.md#diagnostics-directory) for placement
|
||||
and paths, [configuration](../config.md) for all settings, and the
|
||||
[CLI reference](../cli.md) for current flags.
|
||||
|
||||
Debug ledger consumers should recognize the deterministic validator key
|
||||
`non_empty_resulting_segment` in place of `non_empty_corrected_text`. The
|
||||
`empty_resulting_segment` reason is unchanged. Candidate-backed ledger entries
|
||||
now include `proposed_corrected_text: ""` for an empty replacement, and applied
|
||||
deletions also include `applied_corrected_text: ""`; omitted fields still mean
|
||||
there was no candidate or no applied edit, respectively. See the
|
||||
[debug artifact contract](../integrations/output-files.md#debug-only-files).
|
||||
@@ -1,110 +1,83 @@
|
||||
# Audita Operations
|
||||
|
||||
This document owns runtime workflow, physical artifact placement, retention, and
|
||||
recovery. Use the [CLI reference](cli.md) for invocation and stream/exit semantics,
|
||||
[configuration](config.md) for settings, and [output-file contracts](integrations/output-files.md)
|
||||
for logical formats.
|
||||
This guide owns physical output placement and failure handling. The
|
||||
[CLI reference](cli.md) owns flags and streams, [configuration](config.md) owns
|
||||
settings, and the [output contract](integrations/output-files.md) owns schemas.
|
||||
|
||||
## Normal Workflow
|
||||
|
||||
1. Choose a configuration and validate it using the
|
||||
[configuration commands](cli.md#configuration-commands).
|
||||
2. Prepare input files conforming to the [input contract](integrations/transcript-glossary-files.md).
|
||||
3. Run the [processing command](cli.md#process-examples), choosing explicit output
|
||||
and report destinations when those files must survive run-directory cleanup.
|
||||
4. Inspect the corrected transcript and process report. Review rejected or skipped
|
||||
corrections when the report indicates them; successful execution does not mean
|
||||
every proposed correction was applied.
|
||||
1. Validate a version-2 configuration and its selected profiles with
|
||||
`audita config validate --config config.yml`.
|
||||
2. Prepare a transcript and glossary following the [input contract](integrations/transcript-glossary-files.md).
|
||||
3. Run `audita process input.json --config config.yml --glossary glossary.yaml --output corrected.json`.
|
||||
4. Check the exit status, stderr summary, corrected transcript, and reports.
|
||||
A completed run can contain recoverable model errors and still apply
|
||||
corrections that passed every required validator.
|
||||
|
||||
Processing creates a run directory, loads and normalizes the inputs, chunks the
|
||||
transcript, executes the correction pipeline, writes results, and applies
|
||||
retention. Early configuration failures happen before run-directory creation.
|
||||
Failures after initialization attempt to preserve diagnostic evidence; individual
|
||||
artifact writes can fail, so a run directory need not contain every listed file.
|
||||
## Required Output Files
|
||||
|
||||
## Output Files
|
||||
Reporting is enabled by default. For file transcript output, `errors.yml`,
|
||||
`recoveries.yml`, and `report.md` go in the transcript output's parent directory.
|
||||
For stdout transcript output, they go in `reporting.stdout_dir` (default `.`),
|
||||
overridden by `--report-dir`. `--reporting=false` suppresses all three reports
|
||||
and removes their path requirements. It does not suppress the stderr summary or
|
||||
fatal errors. Stdout contains only transcript JSON; file output leaves it empty.
|
||||
|
||||
The caller chooses transcript and optional external report destinations using the
|
||||
[CLI](cli.md#process-output-and-exit-behavior). These are separate from the per-run
|
||||
diagnostic directory unless the caller deliberately places them inside it.
|
||||
Choose existing, writable parent directories. Output/report files can be overwritten
|
||||
by subsequent runs using the same destinations.
|
||||
Existing report directories and parents for regular transcript files must be
|
||||
writable. Audita prevents report writes from overwriting transcript, glossary,
|
||||
loaded configuration, or loaded profile sources, and rejects conflicting output
|
||||
destinations. The final transcript may intentionally replace its own input after
|
||||
successful processing; other input/configuration/profile sources remain protected.
|
||||
Preparation never truncates an existing transcript. Device and pipe destinations
|
||||
(such as `/dev/null` and named pipes) receive ordinary writes at publication time;
|
||||
they are not opened during preflight or replaced with regular files. A pipe write
|
||||
can wait for its reader, and stream write errors are fatal. With reporting enabled,
|
||||
reports still use the output path's parent; disable reporting when that placement
|
||||
is unsuitable for a stream destination.
|
||||
|
||||
A later report or cleanup failure can occur after transcript output was written.
|
||||
Inspect the process result and available report before treating the files as a
|
||||
completed run. Retention of the diagnostic directory does not archive caller-selected
|
||||
output files elsewhere.
|
||||
With reporting enabled, all three reports are initialized as `in_progress` before
|
||||
input processing. Output is encoded fully before publication. Each regular output
|
||||
file is replaced through a temporary file on its destination filesystem:
|
||||
transcript first, then `errors.yml`,
|
||||
`recoveries.yml`, and `report.md`. This is per-file replacement, not a
|
||||
multi-file transaction. New files honor the process umask; replacing an existing
|
||||
file preserves its permission bits. A late report error may leave the transcript
|
||||
published; the process exits nonzero and attempts failed-status reports. On fatal processing
|
||||
or cancellation, no completed transcript is published, but partial reports are
|
||||
finalized where possible. An uncatchable termination can leave `in_progress`.
|
||||
|
||||
## Diagnostics Directory
|
||||
|
||||
The [diagnostics configuration](config.md#thresholds-context-and-diagnostics)
|
||||
selects the parent work directory. Runs create children named like
|
||||
`run-<unix-nanoseconds>`. The runtime creates the work directory if needed.
|
||||
Report paths may be relative when the supplied work directory is relative;
|
||||
resolve them using the invocation's working directory.
|
||||
`--debug` enables detailed evidence independently of reporting. One directory
|
||||
named `run-<id>` is created under `diagnostics.work_dir` (default `/tmp/audita`)
|
||||
and retained on success or failure. Without debug, no directory or payload
|
||||
files are created. The parent is created when needed only for debug. Failure to
|
||||
write optional debug evidence prints a redacted stderr diagnostic but does not
|
||||
change correction decisions or the exit status.
|
||||
|
||||
| Artifact in the run directory | Purpose |
|
||||
| --- | --- |
|
||||
| `source-transcript.json` | Original transcript input |
|
||||
| `source-transcript-parsed.json` | Parsed source representation |
|
||||
| `normalized-transcript.json` | Transcript after normalization |
|
||||
| `normalization-summary.json` | Normalization counters |
|
||||
| `chunking-summary.json` | Section/token summary |
|
||||
| `invocation.json` | Invocation metadata |
|
||||
| `effective-config.json` | Redacted configuration snapshot |
|
||||
| `report.json` | Run's [process report](integrations/output-files.md#process-report) |
|
||||
| `correction-ledger.json` | [Correction dispositions](integrations/output-files.md#correction-ledger) |
|
||||
| `utilization-diagnostics.json` | [Concurrency and timing observations](integrations/output-files.md#utilization-diagnostics) |
|
||||
| `error.log` | Failure details when written |
|
||||
|
||||
LLM request, response, and error diagnostics are also written within the run
|
||||
directory. Their nested naming and payloads are debugging details. Prefer paths
|
||||
provided by the report and its decision/warning records over reconstructing those
|
||||
paths. The report describes intended artifact locations, not proof that every
|
||||
write succeeded or that the files still exist after retention.
|
||||
|
||||
## Retention Behavior
|
||||
|
||||
[Configuration](config.md#thresholds-context-and-diagnostics) owns accepted
|
||||
retention values and the default. Current lifecycle behavior is:
|
||||
|
||||
| Run outcome / selection | Directory outcome |
|
||||
| --- | --- |
|
||||
| Failed run | Retained regardless of selection |
|
||||
| Successful run, `always` | Retained |
|
||||
| Successful run, `auto` | Retained if corrections were skipped or rejected; otherwise removed |
|
||||
| Successful run, `never` | Currently retained, the same outcome as `always` |
|
||||
|
||||
The name `never` does not currently mean deletion. Warnings alone do not trigger
|
||||
retention under `auto` unless they also produce skipped or rejected corrections.
|
||||
An external report outside the run directory survives automatic cleanup, but its
|
||||
artifact paths can then point to removed files.
|
||||
|
||||
## Artifact Handling
|
||||
|
||||
Run artifacts can contain full transcripts, glossary-related context, and LLM
|
||||
prompts/responses. Configured-key redaction does not anonymize those contents.
|
||||
Choose a work directory whose ownership and permissions fit the transcript data,
|
||||
and retain or share artifacts intentionally. Credential-supply mechanisms are
|
||||
owned by [configuration](config.md#secrets).
|
||||
|
||||
For a debugging run, select retained diagnostics using the
|
||||
[retention flag](cli.md#process-flag-reference). Copy needed artifacts before
|
||||
removing completed run directories manually. Avoid deleting a directory while
|
||||
its run is active. Audita does not provide a separate archival/backup or remote
|
||||
artifact storage facility.
|
||||
Debug evidence includes invocation and effective settings, configuration
|
||||
origins, selected profiles, redacted source and normalized snapshots,
|
||||
normalization and initial chunk summaries, all captured proposed candidates
|
||||
and validator decisions in `correction-ledger.json`, execution/utilization
|
||||
records, and model interaction files. Individual artifact writes can fail; a
|
||||
directory's existence does not prove every file was written. These files can
|
||||
contain transcript text and prompt/response payloads. Configured backend
|
||||
credentials are redacted, but the transcript itself is not anonymized. Protect
|
||||
and retain the directory according to the sensitivity of the input.
|
||||
|
||||
## Failure Inspection and Recovery
|
||||
|
||||
1. Read stderr and the exit classification defined by the [CLI](cli.md#process-output-and-exit-behavior).
|
||||
2. Locate the external report, or the run directory identified by the error when
|
||||
available. If a late error omits the path, inspect the selected work directory.
|
||||
3. Read the available report and error log. Use
|
||||
[report status and error fields](integrations/output-files.md#process-report)
|
||||
to identify the failed phase, then inspect referenced artifacts.
|
||||
4. Correct the concrete input, configuration, endpoint, or filesystem issue and
|
||||
rerun the command with diagnostics retained if further investigation is needed.
|
||||
Exit `0` means processing and every enabled required output completed. Exit
|
||||
`1` means cancellation, fatal processing, or required-output failure. Exit `2`
|
||||
identifies usage or configuration/destination selection errors. Recoverable
|
||||
model-operation failures appear in `errors.yml` as one record per affected
|
||||
work unit; they do not automatically make the run fail. `recoveries.yml`
|
||||
records successful operations that needed retry, repair, or proposal filtering.
|
||||
`report.md` shows every initial chunk and configured module instance, including
|
||||
interrupted and unattempted work. Ordinary reports omit transcript excerpts and
|
||||
full decision ledgers; enable debug when those are needed.
|
||||
|
||||
Each invocation starts a new run. There is no implemented resume/checkpoint API;
|
||||
recovery is a rerun, not continuation from an artifact. Failures during early
|
||||
configuration or directory creation may leave no report or diagnostic files.
|
||||
Read stderr and report status first. If a required output failed after a
|
||||
transcript was published, do not treat the file alone as proof of a completed
|
||||
run. Fix the input, service, or destination problem and rerun. Audita has no
|
||||
resume/checkpoint API.
|
||||
|
||||
@@ -66,9 +66,6 @@ Examples:
|
||||
|
||||
## Git Commit Policy
|
||||
|
||||
This policy applies to all coding agents and agent harnesses working in this
|
||||
repository, not only Codex.
|
||||
|
||||
By default, create a commit after completing each coherent unit of work, using a
|
||||
concise, plain-English message describing what changed. This policy is standing
|
||||
authorization to create those local commits; a separate user request is not
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Base Policy Version:** 1.0
|
||||
|
||||
**Last Updated**: 2026-09-23
|
||||
**Last Updated**: 2026-09-24
|
||||
|
||||
## Purpose and System Shape
|
||||
|
||||
@@ -22,7 +22,7 @@ and [integration documentation](../integrations/subprocess.md) own external cont
|
||||
normalization, chunking, output, diagnostics, and reporting contracts. Domain
|
||||
behavior must not require CLI or provider-transport knowledge.
|
||||
- **Framework:** owns orchestration contracts and shared runtime mechanics for
|
||||
modules, validators, proposals, LLM adapters, scheduling, and report assembly.
|
||||
modules, validators, proposals, LLM adapters, and report assembly.
|
||||
It composes domain operations through explicit inputs and outputs.
|
||||
- **Modules:** own narrowly scoped correction behavior and module-specific prompt
|
||||
choices. Keep module packages separate rather than collapsing their distinct
|
||||
@@ -45,9 +45,12 @@ hidden global state.
|
||||
- Validate corrections before final mutation, including local proposal checks,
|
||||
decision cardinality, and apply-time safety. Provider-side schema enforcement
|
||||
does not replace these checks.
|
||||
- Keep concurrency bounded, configurable, context-aware, and observable through
|
||||
the existing scheduler and utilization diagnostics. Preserve permit release on
|
||||
success, failure, and cancellation.
|
||||
- Keep LLM concurrency bounded and context-aware through PromptKit's engine-local
|
||||
per-backend admission. Proposal and validation calls selecting the same backend
|
||||
share that capacity, including retries and repairs. Preserve deterministic
|
||||
collection order. Recoverable model-operation failures stay local to their
|
||||
proposal response or validation batch; fatal invariant/application failure
|
||||
and parent cancellation stop peers and join them.
|
||||
- Keep module and validator identities, reason codes, and execution-class
|
||||
distinctions stable where exposed as contracts.
|
||||
|
||||
@@ -60,12 +63,14 @@ describe how these requirements are implemented.
|
||||
Keep transcript mutation and output handling deterministic around external
|
||||
calls. Reports and correction ledgers are derived from processing results.
|
||||
Preserve the separation between transient processing state and filesystem
|
||||
adapters; physical artifact placement and retention belong to
|
||||
adapters; physical report placement and optional debug retention belong to
|
||||
[operations](../operations.md).
|
||||
|
||||
Public CLI, configuration, output-schema, diagnostics, report, prompt, module,
|
||||
and validator contracts should remain stable unless a change is deliberate and
|
||||
tested. Prefer additive, compatible report changes. Current implementation
|
||||
tested. The resilience work intentionally replaced the earlier JSON process
|
||||
report with versioned errors/recoveries YAML and a fixed Markdown overview;
|
||||
future changes should treat those current contracts deliberately. Current implementation
|
||||
limits are documented in the [internal overview](../internal/overview.md#current-scope);
|
||||
they are not additional prohibitions on future development.
|
||||
|
||||
@@ -84,9 +89,10 @@ responsibilities and should remain separate. Implementation details belong in
|
||||
## Dependencies
|
||||
|
||||
Keep Audita dependency-light and prefer the Go standard library for CLI parsing,
|
||||
HTTP, JSON, filesystems, synchronization, and tests. Third-party dependencies
|
||||
should be narrow, justified, and preferably established for their purpose.
|
||||
YAML parsing is the existing direct dependency exception.
|
||||
filesystems, synchronization, and tests. ConfigKit owns configuration loading,
|
||||
presence, provenance, and secret resolution; PromptKit owns prompt/profile
|
||||
loading, OpenAI-compatible transport, retries, repairs, and backend admission.
|
||||
Audita owns domain meaning, role selection, and deterministic processing.
|
||||
|
||||
Do not introduce broad CLI, dependency-injection, workflow, logging, or plugin
|
||||
frameworks without a concrete implemented need and focused tests.
|
||||
|
||||
@@ -7,7 +7,7 @@ source/tag push; binary delivery runs asynchronously.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Use Bash, Git, and Go 1.25 or newer (CI uses Go 1.25), with dependencies available
|
||||
Use Bash, Git, and Go 1.25.5 or newer (CI uses Go 1.25.5 or newer), with dependencies available
|
||||
locally or downloadable. Tests themselves are offline. Work from a
|
||||
checkout of `main` with permission to push that branch and tags to `origin`.
|
||||
For this repository, `origin` is `https://gitea.maximumdirect.net/eric/audita.git`.
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
# Release Notes
|
||||
|
||||
- [Audita v2.0.0](v2.0.0.md)
|
||||
- [Audita v1.1.0](v1.1.0.md)
|
||||
|
||||
See the [release documentation requirements](../policy/release.md#release-documentation).
|
||||
|
||||
38
docs/releases/v2.0.0.md
Normal file
38
docs/releases/v2.0.0.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Audita v2.0.0
|
||||
|
||||
Audita v2 adopts ConfigKit for configuration and PromptKit for LLM execution,
|
||||
adds resilient processing and focused reports, and improves correction prompts
|
||||
and validation. This is an intentional breaking release; existing installations
|
||||
must migrate their configuration and invocation settings.
|
||||
|
||||
- Configuration uses `version: 2` YAML, named backends, PromptKit model profiles,
|
||||
and optional named overlays. Offline configuration validation and inspection
|
||||
expose effective settings, selected profiles, and provenance.
|
||||
- Proposal and validation calls share per-backend capacity. PromptKit handles
|
||||
transport retries and structural response repair within bounded operation
|
||||
timeouts.
|
||||
- Transcript chunks remain fixed through the LLM stages. Recoverable model
|
||||
failures preserve progress and allow other work to continue. If every LLM
|
||||
operation fails recoverably, processing still produces the deterministically
|
||||
normalized transcript without LLM corrections. Cancellation, application
|
||||
failures, and required output failures remain fatal.
|
||||
- Ordinary output includes `errors.yml`, `recoveries.yml`, and `report.md`, plus
|
||||
a brief completion summary. Detailed correction and LLM diagnostics are opt-in
|
||||
with `--debug`; `--reporting=false` suppresses all three ordinary reports.
|
||||
- Empty replacement text supports validated span deletion while preserving a
|
||||
nonblank segment. Validator responses retain usable decisions despite missing,
|
||||
duplicate, or extraneous entries; unavailable decisions never approve an edit.
|
||||
- Prompts use short request-local references, consistent shared instructions,
|
||||
and relevant neighboring/glossary context for validation. Spoken-word cleanup
|
||||
can reduce accidental protected-name repetition subject to validation.
|
||||
|
||||
Follow the [v2 migration guide](../migrations/v2.md) before upgrading. Version-1
|
||||
configuration, implicit environment-setting overlays, retired model/credential/
|
||||
concurrency flags, and the former JSON report interface have no compatibility
|
||||
shim. Start from the maintained examples linked in that guide, configure profiles
|
||||
and credential references, and validate the new configuration. Update report and
|
||||
debug consumers for the [current output contract](../integrations/output-files.md).
|
||||
|
||||
Linux amd64 and arm64 remain the production targets; macOS arm64 is best effort
|
||||
for development. Binary and checksum delivery runs asynchronously after source/tag
|
||||
publication, as described in the [release procedure](../release.md).
|
||||
171
docs/roadmap/configkit.md
Normal file
171
docs/roadmap/configkit.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# ConfigKit Adoption Roadmap
|
||||
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Production implementation complete. The selected configuration and
|
||||
overlay model is implemented; [implementation evidence](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md)
|
||||
records verification. No release has been published.
|
||||
|
||||
## Purpose and Direction
|
||||
|
||||
Adopt ConfigKit to harmonize Audita's configuration policy with the other
|
||||
applications in the suite. Consistent configuration conventions are the primary
|
||||
benefit; reducing application-owned configuration infrastructure is a secondary
|
||||
benefit and a useful measure of integration quality.
|
||||
|
||||
This is an intentional breaking migration. There is no goal of backward
|
||||
compatibility with existing installations, configuration files, CLI flags,
|
||||
environment variables, or configuration-inspection output. Provide a migration
|
||||
guide instead of legacy loaders, aliases, dual operating modes, or an ongoing
|
||||
compatibility layer. Retain existing syntax only when it independently suits the
|
||||
new model, not because compatibility requires it.
|
||||
|
||||
The initial review and prototype used ConfigKit v0.3.1, then verified v0.4.0
|
||||
selected overlays. Those findings remain historical; the prototype was removed
|
||||
after production integration. The application depends on a versioned module,
|
||||
without a sibling checkout or local module replacement.
|
||||
|
||||
The shared boundary/prototype unit is complete. Its [selected contract, inventory,
|
||||
and evidence](library-boundary-findings.md) define the detailed target alongside
|
||||
this roadmap. Production configuration migration is complete.
|
||||
|
||||
## Target End State
|
||||
|
||||
- ConfigKit owns strict YAML loading, input presence, default application, explicit
|
||||
override mechanics, recorded resolution, secret references, and safe inspection.
|
||||
- Audita owns its configuration types, field meanings, defaults, schema-version
|
||||
policy, file selection, CLI parsing, domain validation, and runtime initialization.
|
||||
- Adopt ConfigKit's precedence and validation lifecycle rather than reproduce
|
||||
Audita's current per-layer validation and environment-overlay behavior.
|
||||
- General configuration environment overrides are retired. Credentials use typed
|
||||
environment-name secret references; supplied references must resolve according
|
||||
to ConfigKit's contract. File selection follows the application-owned
|
||||
discovery rules in the selected boundary.
|
||||
- Standardize duration representations, explicit empty/null/zero handling, and
|
||||
inheritance on the shared conventions. Express genuine Audita domain rules
|
||||
through the supported resolution and validation APIs.
|
||||
- Retained domain-setting flags and proposal/validation profile flags become
|
||||
explicit ConfigKit overrides according to the migration inventory. Remove old
|
||||
provider/key/retry/concurrency flags and aliases. Credentials use secret references.
|
||||
- Use ConfigKit's redacted effective output and provide source/explanation access
|
||||
through the selected views below. Legacy JSON inspection is replaced.
|
||||
Diagnostic configuration snapshots use the same new model.
|
||||
- Delete superseded loading and override machinery. Do not hide the old model
|
||||
behind adapters that defeat the adoption's consistency benefit.
|
||||
|
||||
Keep Audita's domain capabilities, such as ordered and repeated correction modules,
|
||||
expressible in the new model. Their existing configuration spelling is not binding.
|
||||
Transcript/glossary formats, processing algorithms, and output schemas are outside
|
||||
this migration unless later planning identifies a concrete dependency or an
|
||||
explicitly coordinated change.
|
||||
|
||||
## Selected Overlays for Testing
|
||||
|
||||
Enable ConfigKit's optional selected-overlay feature in the migration. A testing
|
||||
configuration that selects cheaper, faster LLMs is the initial use case. ConfigKit
|
||||
**overlays** modify application settings; PromptKit **execution profiles** still
|
||||
own model and generation settings. Keep `profiles_dir`, the PromptKit `profiles/`
|
||||
directory, and proposal/validation profile names unchanged by this terminology
|
||||
change.
|
||||
|
||||
The planned interface is `--overlay <name>` on `process`, `config validate`, and
|
||||
`config print-effective`. Treat selection as invocation state passed to
|
||||
`Options.Overlay`, not as another persistent setting or environment overlay.
|
||||
Follow ConfigKit v0.4.0's selection contract: omission, an empty name, or `default`
|
||||
selects no overlay; a named selection loads `overlays/<name>.yml` beneath the selected
|
||||
configuration root. A missing selected overlay is an error. Use the library's
|
||||
single-overlay semantics without adding stacking, inheritance, or legacy-directory
|
||||
fallback.
|
||||
|
||||
Precedence is application defaults, main configuration, selected overlay, then
|
||||
explicit setting flags. Perform Audita's role inheritance and semantic validation
|
||||
on the resolved configuration. ConfigKit's base parsing and schema-version checks
|
||||
still precede overlay application; overlays cannot replace the schema version or
|
||||
contain ConfigKit directives. General environment-variable overrides remain retired.
|
||||
Imports and secret-directory bootstrap remain disabled.
|
||||
|
||||
Provide a maintained `overlays/testing.yml` example that selects testing execution
|
||||
profiles through `llm.proposal_profile` and `llm.validation_profile`. Model names and
|
||||
generation controls belong in the corresponding PromptKit profile files. Set both
|
||||
role selections in this example so a base configuration's explicit validation
|
||||
profile cannot accidentally retain the production model. Normal omission-based
|
||||
validation-profile inheritance continues to apply to the final effective settings.
|
||||
The example must make its backend/credential requirements clear and use no real
|
||||
secrets. Model choice is operator-controlled; “testing” does not promise a price,
|
||||
performance level, or equivalent correction quality.
|
||||
|
||||
Overlays may modify other ordinary application settings supported by the schema;
|
||||
they are not restricted to profile selection. The resulting configuration must
|
||||
still satisfy shared backend capacity, credential, and domain rules. Validation,
|
||||
effective-config inspection, selected-profile inspection, and processing must all
|
||||
use the same selected overlay. Preserve overlay provenance in configuration
|
||||
explanations and diagnostics while retaining secret redaction.
|
||||
|
||||
The [recorded overlay results](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
|
||||
prove the representative load, inspection, and HTTP path. Production acceptance
|
||||
requires the same selection to reach both actual LLM roles, with explicit flags
|
||||
winning over overlays and surviving secret references resolved/redacted correctly.
|
||||
The isolated prototype is evidence, not a second supported runtime.
|
||||
|
||||
## Inspection and Invocation Contract
|
||||
|
||||
`process`, `config validate`, and `config print-effective` share file/overlay
|
||||
selection and retained setting overrides. Invocation I/O paths remain process-only.
|
||||
Only explicitly supplied flags override settings. Validation requires an explicit
|
||||
config file; processing and inspection use the selected discovery order. Help is
|
||||
available before configuration loading. All three commands validate selected
|
||||
profiles without requiring a provider call during setup.
|
||||
|
||||
`config print-effective --view settings|profiles|explain` exposes three YAML views:
|
||||
|
||||
- `settings` (default): ConfigKit's redacted effective configuration.
|
||||
- `profiles`: an Audita-owned summary of each selected role's profile, backend,
|
||||
model, temperature, token limit, top-p, timeout, service tier, and reasoning
|
||||
effort where present. Arbitrary extra provider parameters and credentials are
|
||||
omitted; this is a selected-controls view, not every execution input.
|
||||
- `explain`: field paths, sources, and derivation records from ConfigKit, without
|
||||
resolved secret values.
|
||||
|
||||
A ConfigKit snapshot alone does not describe profile-owned controls. Profile
|
||||
inspection must enforce the registered-backend and credential boundary. Diagnostic
|
||||
snapshots use these safe views and preserve overlay/explicit-override provenance.
|
||||
The migration guide explains the intentional replacement of legacy inspection JSON.
|
||||
|
||||
## Integration and Completion Criteria
|
||||
|
||||
ConfigKit and PromptKit form one intentional breaking refactor. Intermediate
|
||||
commits need not build or run. The target contains no temporary old-runtime adapter,
|
||||
compatibility shim, processing gate, dual loader, or automatic legacy conversion.
|
||||
The [combined feature roadmap](library-adoption.md) owns execution/reporting intent;
|
||||
[implementation.md](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md) alone owns stage order and implementation tasks.
|
||||
|
||||
The final application loads all retained domain settings, assembles both selected
|
||||
profiles, and processes through PromptKit. Controlled HTTP evidence must connect
|
||||
configuration selection to actual requests, domain outcomes, safe diagnostics,
|
||||
and shared finite backend capacity. Maintained examples and one migration guide
|
||||
cover the complete file/flag/environment transition. Obsolete application-owned
|
||||
loading, transport, rendering, and scheduler machinery is removed.
|
||||
|
||||
The combined target is Audita v2.0.0. Release compliance and publication remain
|
||||
last and separate from implementing the refactor. Retention and transcript timestamp
|
||||
changes are not included; the selected retry/repair contract resolves the retry
|
||||
follow-up in [runtime behavior](runtime-behavior.md).
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
- Pin ConfigKit v0.4.0 in production and align Go/toolchain requirements. ConfigKit requires
|
||||
Go 1.25; the combined PromptKit prototype requires Go 1.25.5. Adopting these libraries
|
||||
does not automatically remove Audita's existing YAML dependency used for glossary parsing.
|
||||
- Implement the selected configuration and final PromptKit runtime directly,
|
||||
without temporary runtime adapters, compatibility shims, or process gates.
|
||||
- Enable selected overlays as specified above. Keep imports and secret-directory
|
||||
bootstrap disabled unless a separate concrete requirement justifies them.
|
||||
- Validation and inspection resolve surviving credential references and fail on
|
||||
missing or empty values; no credential-free bypass is planned.
|
||||
- Inspection and diagnostics use the safe selected-profile view and ConfigKit
|
||||
provenance described above. Changed formats are covered by the migration guide.
|
||||
|
||||
The eventual completion test is that Audita follows the suite's ConfigKit
|
||||
conventions while retaining its intended domain capabilities, with a clear
|
||||
migration guide and no ongoing obligation to support the previous configuration
|
||||
model.
|
||||
162
docs/roadmap/decision-set-reconciliation.md
Normal file
162
docs/roadmap/decision-set-reconciliation.md
Normal file
@@ -0,0 +1,162 @@
|
||||
# Partial Success for Validator Decision Sets
|
||||
|
||||
**Last updated:** 2026-09-25
|
||||
|
||||
**Status:** Implemented. This document owns the target behavior and user intent.
|
||||
[implementation.md](implementation.md) owns concrete decisions, ordered work,
|
||||
and verification.
|
||||
|
||||
## Purpose and User Intent
|
||||
|
||||
Preserve usable validator decisions when other entries in the same model response
|
||||
have missing, duplicate, or unexpected correction indices. One defective entry
|
||||
should not prevent unrelated candidates from completing their required validation.
|
||||
This extends Audita's existing partial-success approach without another model call.
|
||||
|
||||
Implement this simpler improvement before reconsidering guided domain recovery.
|
||||
Missing validation must never authorize applying a correction, but neither should
|
||||
it invalidate a different candidate's unambiguous approval or editorial rejection.
|
||||
The existing [resilience design](resilience.md) remains the foundation.
|
||||
|
||||
## Historical Baseline
|
||||
|
||||
The reviewed application baseline is `d2a6fcf`:
|
||||
|
||||
- PromptKit validates the complete model response against Audita's embedded JSON
|
||||
Schema and performs its configured structural repairs. Only structurally valid,
|
||||
decoded responses reach domain decision mapping.
|
||||
- `mapLLMResponseToDecisions` in
|
||||
[`llm_validators.go`](../../internal/framework/validators/llm_validators.go)
|
||||
requires exactly one decision for every requested correction index. An empty
|
||||
decision list, missing index, duplicate index, or unexpected index returns an
|
||||
error instead of retaining usable decisions.
|
||||
- `LLMBackedValidator.Validate` consequently marks the entire batch unavailable
|
||||
with `validation_decisions_invalid`. Other batches continue, but usable decisions
|
||||
within that batch are lost.
|
||||
- The runner's cardinality checks enforce an internal contract: each candidate
|
||||
receives one outcome from each validator it reaches. These checks apply to
|
||||
Audita-owned results, not directly to raw model entries.
|
||||
- Deterministic validators already filter individual proposals. Unpreviewable
|
||||
candidates and oversized validation inputs are isolated; recoverable provider
|
||||
failures remain batch-local; application skips individual conflicts.
|
||||
- Ordinary reports distinguish failed operations from successful operations with
|
||||
recoveries. Several failed operations in one stage/chunk share one work-unit
|
||||
error record. The [output contract](../integrations/output-files.md) owns the
|
||||
current counts and serialized representation.
|
||||
|
||||
The defect addressed here is the all-or-nothing domain mapping boundary, not
|
||||
validation itself or the runner's internal consistency checks.
|
||||
|
||||
## Target Decision Semantics
|
||||
|
||||
Reconcile each structurally valid response against the indices actually requested
|
||||
in that particular validator batch. Correction indices, rather than segment IDs
|
||||
or positions in the response array, identify candidates. Several corrections may
|
||||
legitimately target the same segment.
|
||||
|
||||
| Response condition | Target behavior |
|
||||
| --- | --- |
|
||||
| Exactly one valid decision for a requested index | Preserve its approval or rejection and associated decision evidence. |
|
||||
| Requested index omitted | Produce an explicit validation-unavailable outcome for that candidate. |
|
||||
| Requested index appears more than once | Treat that candidate's validation as unavailable; preserve unrelated decisions. |
|
||||
| Index was not requested in this batch | Ignore the extraneous entry and record the handled defect. |
|
||||
| Empty decision list for a nonempty request | Mark all requested candidates unavailable; do not fail the run. |
|
||||
| Decisions appear in a different order | Match by identity and retain deterministic internal results. |
|
||||
|
||||
Duplicate handling does not choose the first or last answer, nor infer a verdict
|
||||
by voting. Even identical duplicates follow the agreed duplicate-index handling;
|
||||
accepting them specially is not needed for this feature. Duplicate extraneous
|
||||
indices remain extraneous entries, not reasons to invalidate requested candidates.
|
||||
|
||||
An unavailable candidate remains unapplied and does not proceed as approved to
|
||||
later validators. It is distinct from a valid `approved: false` decision, which
|
||||
is ordinary editorial processing. Valid approvals continue through the remaining
|
||||
required validators and apply-time checks; this feature grants no bypass.
|
||||
|
||||
The mapper returns one internal outcome for every requested candidate and no
|
||||
outcome for an extraneous index. Preserve strict runner cardinality checks so
|
||||
application bugs are still detected. Do not convert genuine internal invariant
|
||||
failures, cancellation, or fatal setup failures into tolerated model defects.
|
||||
|
||||
## Outcomes and Reporting
|
||||
|
||||
Keep existing work-unit aggregation and operation accounting:
|
||||
|
||||
- If any requested candidate lacks an unambiguous usable decision, record a
|
||||
validation-operation error identifying the affected requested indices. Other
|
||||
candidates' usable decisions from that operation remain effective. An operation
|
||||
error does not mean every candidate in its batch failed.
|
||||
- If all requested candidates have usable decisions and extra entries were
|
||||
ignored, record a successful operation with a recovery for that handled defect.
|
||||
This remains a recovery even if all legitimate decisions are editorial rejections.
|
||||
- If missing/duplicate requested decisions and ignored extras coexist, retain the
|
||||
operation error. Follow the existing rule that failed operations do not also
|
||||
produce recovery records; retain extra-entry details in debug evidence.
|
||||
- A valid complete response with no handled defect adds neither error nor recovery.
|
||||
Valid editorial rejection alone never adds either category.
|
||||
- Preserve actual structural repair and service-retry counts. On successful
|
||||
operations, consolidate applicable recovery reasons in one recovery record.
|
||||
Do not count reconciliation as another LLM operation or structural repair.
|
||||
|
||||
Error attribution must distinguish affected candidates from the rest of the batch.
|
||||
Debug evidence should preserve supplied decisions, ignored entries, and the reason
|
||||
an expected candidate became unavailable. Do not fabricate candidate ledger entries
|
||||
for unknown indices or mislabel missing decisions as deliberate editorial rejection.
|
||||
Ordinary reports must retain safe metadata without raw transcript or response text.
|
||||
|
||||
Use a decision-specific recovery reason rather than treating ignored validator
|
||||
entries as defective correction proposals. Concrete reason names and minimal data
|
||||
model changes belong in the implementation plan. Keep YAML, Markdown, and CLI
|
||||
counts consistent, with the existing reporting-disable and independent debug options.
|
||||
|
||||
## Ownership and Target Architecture
|
||||
|
||||
Audita's validator layer reconciles structurally valid model responses into
|
||||
complete internal candidate outcomes. The runner preserves strict internal
|
||||
cardinality and advances only approved candidates. Reporting consumes operation
|
||||
facts without treating partial validation failure as loss of every decision.
|
||||
Debug evidence distinguishes unavailable validation from editorial rejection.
|
||||
|
||||
The schema/transport boundary remains with PromptKit; reconciliation needs no
|
||||
additional model call. Concrete helper structures, reason names, mapping changes,
|
||||
and verification steps belong in the [implementation plan](implementation.md).
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
PromptKit's whole-response structural validation and repair remain unchanged.
|
||||
A malformed entry, including an extraneous entry that violates the schema, may
|
||||
still invalidate the response and exhaust structural repairs before reconciliation
|
||||
can run. Ignoring schema-valid extras does not promise salvage from malformed JSON,
|
||||
missing required fields, wrong types, or out-of-range values.
|
||||
|
||||
General per-entry structural salvage is a separate design question, not part of
|
||||
this fix. Likewise, guided regeneration remains deferred for reconsideration after
|
||||
this change; no new retries, feedback prompts, budgets, upstream dependency changes,
|
||||
or operator controls are required here.
|
||||
|
||||
Preserve module scope, literal replacement behavior, fixed chunks, stage barriers,
|
||||
span-deletion safeguards, backend capacity limits, and fatal cancellation behavior.
|
||||
No additional restrictions on valid corrections or guarantees about model judgment
|
||||
are introduced. This work does not itself authorize tagging or publishing v2.0.0.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A schema-valid response can retain usable decisions despite missing, duplicate,
|
||||
or unexpected indices elsewhere in the same batch.
|
||||
- Only the affected requested candidates become unavailable; extras never create
|
||||
candidates or attach decisions to a different batch's candidates.
|
||||
- Valid negative decisions remain editorial outcomes, and every applied correction
|
||||
completes all required validation.
|
||||
- Internal validator results satisfy existing strict cardinality checks.
|
||||
- Missing/duplicate requested decisions yield accurately attributed errors without
|
||||
losing valid work. Extra-only defects yield a consolidated recovery; mixed
|
||||
defects follow existing failed-operation reporting semantics.
|
||||
- Final transcript, debug evidence, YAML reports, Markdown counts, and CLI summary
|
||||
agree about applied work and failures/recoveries.
|
||||
- Reconciliation makes no additional provider calls. Existing structural failures,
|
||||
provider failures, and cancellation retain their documented handling.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No unresolved product-policy questions block implementation. Guided recovery and
|
||||
structural salvage remain separate future decisions.
|
||||
80
docs/roadmap/future.md
Normal file
80
docs/roadmap/future.md
Normal file
@@ -0,0 +1,80 @@
|
||||
# Deferred Feature Work
|
||||
|
||||
**Last updated:** 2026-09-25
|
||||
|
||||
**Status:** Deferred or exploratory work; not implemented or scheduled. These
|
||||
entries preserve scope outside the [resilience roadmap](resilience.md) and
|
||||
[span-deletion consistency roadmap](span-deletion.md). Each entry states its
|
||||
intent; inclusion does not commit us to implementing every capability.
|
||||
|
||||
## Domain-Aware Response Recovery
|
||||
|
||||
The [partial-success decision reconciliation](decision-set-reconciliation.md)
|
||||
preserves usable decisions without another model call. Guided recovery remains
|
||||
deferred pending evidence from failures that remain. General per-entry
|
||||
salvage from structurally invalid responses is also a separate design question;
|
||||
the accepted reconciliation feature retains whole-response structural validation.
|
||||
|
||||
Consider bounded feedback and regeneration for model responses that satisfy the
|
||||
structural schema but leave requested decisions unavailable after reconciliation.
|
||||
The goal would be to recover more usable work before excluding affected
|
||||
candidates, when doing so materially improves outcomes.
|
||||
|
||||
This capability is explicitly deferred. Current behavior retains PromptKit
|
||||
structural repair and per-candidate reconciliation; unavailable requested
|
||||
decisions produce errors while other candidates can succeed.
|
||||
|
||||
Before implementation, define the Audita/PromptKit ownership boundary, which
|
||||
checks are eligible for feedback, how feedback identifies defects without exposing
|
||||
secrets, and how regeneration fits bounded budgets and operation deadlines. Reuse
|
||||
shared library capabilities where appropriate; do not introduce an uncoordinated
|
||||
outer retry loop that multiplies existing retry/repair budgets.
|
||||
|
||||
Ordinary editorial rejection is a valid decision, not a reason to regenerate until
|
||||
an edit is approved. Recovery must preserve required validation, candidate identity,
|
||||
partial-success safety, and truthful error/recovery reporting. Validate with
|
||||
controlled responses that distinguish recoverable protocol defects from valid
|
||||
negative decisions. Exact APIs, budgets, and sequencing remain undecided.
|
||||
|
||||
## Independent Whole-Pipeline Traversal by Chunk
|
||||
|
||||
Explore allowing each fixed chunk to advance through its ordered stages without
|
||||
waiting for other chunks to finish the current stage. This could reduce waiting
|
||||
behind a slow chunk and improve utilization while preserving stage order within
|
||||
each chunk and deterministic final recombination.
|
||||
|
||||
The resilience feature deliberately retains stage barriers: finish a stage across
|
||||
chunks before advancing to the next stage. Fixed boundaries alone do not authorize
|
||||
independent traversal.
|
||||
|
||||
A future design must establish that stages do not depend on other chunks' evolving
|
||||
state, preserve stable identities and partial-success rules, and continue to share
|
||||
PromptKit's per-backend capacity across all roles and stages. Reporting must remain
|
||||
deterministic despite different completion orders, and fatal cancellation must
|
||||
still stop and join active work. Assess whether the scheduling change materially
|
||||
improves real workloads before introducing additional orchestration machinery.
|
||||
No new scheduler, global limiter, or execution mode is selected by this entry.
|
||||
|
||||
## Whole-Segment Removal
|
||||
|
||||
Potential future work, subject first to evaluating whether it is worth implementing
|
||||
at all. Empty replacement text and deletion of a span within a surviving segment
|
||||
are covered by the accepted [span-deletion consistency roadmap](span-deletion.md).
|
||||
That feature does not remove segment records or permit blank resulting segments.
|
||||
|
||||
Before designing segment removal, review representative production examples to
|
||||
establish whether whole-segment deletion is a meaningful unmet need. An empty
|
||||
replacement alone does not establish that intent: it can express ordinary filler
|
||||
or punctuation deletion within a larger segment. Assess how often genuinely
|
||||
expendable whole segments occur after deterministic normalization, the transcript
|
||||
quality benefit of removing them, and the risk of losing substantive speech.
|
||||
Compare those benefits with the implementation and ongoing maintenance costs.
|
||||
Retaining the segment and declining the edit is a valid outcome if the benefit is
|
||||
small or uncertain. No numerical adoption threshold is selected.
|
||||
|
||||
Only if that evaluation supports proceeding should a separate feature design
|
||||
choose a representation, authorization and validation rules, and reporting
|
||||
semantics. It must address meaning preservation and the effects on segment
|
||||
identity, timestamps, fixed chunks, subsequent stages, recombination, and output
|
||||
consumers. Do not infer authorization to remove a segment from an empty string
|
||||
or introduce segment-removal machinery as part of span-deletion consistency.
|
||||
287
docs/roadmap/implementation.md
Normal file
287
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,287 @@
|
||||
# Implementation Plan: Partial Validator Decision Sets
|
||||
|
||||
**Last updated:** 2026-09-25
|
||||
|
||||
**Status:** Implemented; all stages completed.
|
||||
**Source baseline:** `60e2fe7`.
|
||||
**Audience:** gpt-5.6-terra coding agent.
|
||||
|
||||
## Scope and Execution
|
||||
|
||||
Implement [partial success for validator decision sets](decision-set-reconciliation.md).
|
||||
That roadmap owns product intent and target semantics; this document selects the
|
||||
implementation and verification work. Follow Stage 1 through Stage 3 in order and
|
||||
record completion evidence beneath each stage. Each stage is a focused agent task.
|
||||
|
||||
Read [development guidance](../development.md) and all policies under
|
||||
[docs/policy](../policy/): [agents](../policy/agents.md),
|
||||
[architecture](../policy/architecture.md), [documentation](../policy/documentation.md),
|
||||
[testing](../policy/testing.md), and [release](../policy/release.md). Use graph-first
|
||||
code discovery, preserve unrelated work, and commit completed coherent units with
|
||||
plain English messages. Do not introduce approval rounds between stages.
|
||||
|
||||
Aim directly at the final implementation. No temporary modes, compatibility shims,
|
||||
new configuration, extra model calls, or upstream dependency work is required.
|
||||
Keep PromptKit structural validation/repair and the runner's internal cardinality
|
||||
checks. Implement the agreed outcomes, not extra restrictions suggested by existing
|
||||
tests or incidental internal structure. Internal helper signatures and layout can
|
||||
be adjusted idiomatically while preserving the decisions below.
|
||||
|
||||
This replaces the completed span-deletion plan, available in Git at
|
||||
`60e2fe7:docs/roadmap/implementation.md`. It does not reopen that feature. The final
|
||||
stage includes correcting historical links that otherwise point to this new plan.
|
||||
No release preparation, tag, push, CI wait, or paid-provider evaluation is included.
|
||||
|
||||
## Selected Implementation Decisions
|
||||
|
||||
### Reconciliation boundary
|
||||
|
||||
Keep reconciliation in `internal/framework/validators`, after successful structured
|
||||
completion and before converting the batch into runner-facing results. Replace
|
||||
`mapLLMResponseToDecisions` with a small pure reconciliation helper; use an ordinary
|
||||
private result struct containing:
|
||||
|
||||
- `Decisions []Decision`: one outcome per requested item, in request order;
|
||||
- `UnavailableIndices []int`: requested indices without a usable verdict;
|
||||
- `IgnoredIndices []int`: unexpected indices encountered in response order, including
|
||||
repeated unexpected entries, for debug attribution.
|
||||
|
||||
No generic reconciliation framework or new shared abstraction is needed. Requested
|
||||
identities come from `batch.Items`, not transcript segment IDs, the whole section,
|
||||
or any other batch. Expected input identities remain an Audita internal contract.
|
||||
|
||||
First collect response entries by requested index and count occurrences, separating
|
||||
unexpected indices. Then walk requested items once to construct outcomes. This
|
||||
avoids accepting an early duplicate before seeing a conflicting later entry.
|
||||
|
||||
Use these candidate-level reason codes in `validators/models.go`:
|
||||
|
||||
| Condition | Internal outcome/reason |
|
||||
| --- | --- |
|
||||
| Missing requested entry | `DispositionUnavailable`, `missing_validation_decision` |
|
||||
| More than one entry for a requested index | `DispositionUnavailable`, `duplicate_validation_decision` |
|
||||
| Single requested entry failing the existing confidence range check | `DispositionUnavailable`, `invalid_validation_decision` |
|
||||
| Single usable entry | Existing approved or `llm_rejected` decision with its model reason. |
|
||||
|
||||
Unavailable outcomes have `Approved: false` and fixed safe explanatory messages.
|
||||
Duplicate classification takes precedence over inspecting individual duplicate
|
||||
values; identical duplicates follow the same rule. Unexpected entries are ignored
|
||||
before inspecting confidence. Retain the existing confidence check as a local
|
||||
check for decoded callers/test doubles, scoped to that candidate; do not broaden
|
||||
this into structural salvage, confidence thresholds, or new semantic validation.
|
||||
The production JSON Schema still rejects out-of-range confidence before mapping.
|
||||
|
||||
An empty response list makes every requested item unavailable. Response ordering
|
||||
must not change verdicts or which candidate is affected. Preserve the existing
|
||||
handling of valid negative decisions, including reason text. No candidate may
|
||||
advance merely because another candidate was approved.
|
||||
|
||||
### Operation accounting and ordinary reports
|
||||
|
||||
Each actual validation batch remains one `outcomes.Operation`. Use existing
|
||||
`ReasonValidationDecisionsInvalid` (`validation_decisions_invalid`) for a batch
|
||||
with unavailable requested candidates. Set `Succeeded: false`, set `ProposalIndices`
|
||||
to only those unavailable indices, retain execution counts, and leave recovery
|
||||
reasons empty. Still return usable decisions from that batch alongside explicit
|
||||
unavailable outcomes, with no Go error for the reconciled model defect.
|
||||
|
||||
If no requested candidates are unavailable, set `Succeeded: true`. Add
|
||||
`outcomes.RecoveryValidationDecisionsFiltered` with serialized value
|
||||
`validation_decisions_filtered` when extras were ignored. Merge it with actual
|
||||
`output_repaired` and `service_recovered` reasons in the same recovery operation.
|
||||
Valid negative decisions do not prevent operation success. A clean response creates
|
||||
no new recovery reason.
|
||||
|
||||
When unavailable candidates and extras coexist, record the failed operation only;
|
||||
keep ignored-entry evidence in debug. Do not create both an error and a recovery
|
||||
for that operation. Successful survivors must continue even though the operation
|
||||
itself records a failure. Retain actual execution counts on all paths.
|
||||
|
||||
Reuse existing ordinary YAML structures and version 1. The only new public report
|
||||
value is the decision-specific recovery reason; affected error indices already fit
|
||||
`Failure.ProposalIndices`. Do not add ignored entries to `Operation.Filtered` or
|
||||
`filtered_proposals`, which describe correction proposals, and do not increment
|
||||
`proposal_responses_with_filtering`. The recovery reason and existing operation
|
||||
identity suffice for ordinary reporting; detailed extra indices belong in debug.
|
||||
|
||||
Update `processreport.safeMessage` for `validation_decisions_invalid` to explain
|
||||
that some required decisions were unavailable without claiming all batch decisions
|
||||
were discarded. Generalize the recovery message to include response filtering,
|
||||
rather than only proposal filtering. Exact prose is not a test contract. Preserve
|
||||
work-unit error grouping, operation counts, and existing YAML/Markdown/CLI totals.
|
||||
|
||||
### Debug evidence and runner records
|
||||
|
||||
Existing interaction diagnostics already retain the full decoded response and final
|
||||
content. Add the reconciliation result to the response diagnostic payload for a
|
||||
successfully decoded validation response. Perform reconciliation before writing
|
||||
that diagnostic; retain existing call-error diagnostics and diagnostic-write failure
|
||||
handling. All reconciled decisions, including unavailable outcomes, receive the
|
||||
returned response artifact path. This records unexpected indices and duplicates
|
||||
without inventing candidate ledger entries or an additional debug file.
|
||||
|
||||
Carry `validators.DecisionDisposition` through `runner.ValidatorDecisionRecord`
|
||||
and `runner.ValidatorRejectedChange` in an additive `disposition` field. Populate
|
||||
it from each actual `Decision`, including partial decision evidence returned on
|
||||
fatal paths. Keep the existing collection names and application flow; no parallel
|
||||
unavailable-candidate collection is necessary.
|
||||
|
||||
In `processreport`, add `disposition` to `LedgerValidatorDecisionRecord` and map
|
||||
unavailable candidates to a top-level ledger disposition `unavailable`, instead
|
||||
of `rejected`. Ordinary editorial/deterministic rejections remain `rejected`.
|
||||
Use the explicit disposition, not a reason-code allowlist, so existing timeout,
|
||||
provider-failure, and oversized-input outcomes retain the same distinction too.
|
||||
Update affected test fixtures to supply dispositions where they represent real
|
||||
validator outputs. Do not invent a fatal check or compatibility subsystem for a
|
||||
zero-value disposition in handcrafted fixtures.
|
||||
|
||||
Retain candidate identity, literal proposed replacements (including explicit empty
|
||||
strings), and omission of applied replacements for unapplied candidates. Existing
|
||||
fatal/interrupted/module-failure handling and debug-off behavior remain. Supplied
|
||||
model decisions remain visible in interaction evidence even when reconciliation
|
||||
cannot accept them as authoritative verdicts.
|
||||
|
||||
## Stage 1 — Reconcile Model Decisions and Preserve Valid Work
|
||||
|
||||
**Completed:** 2026-09-25. Reconciliation preserves usable decisions, attributes
|
||||
unavailable requested indices and ignored extras, and keeps strict runner
|
||||
cardinality. Validator and runner boundary checks passed with
|
||||
`go test ./internal/framework/validators ./internal/framework/runner`.
|
||||
|
||||
**Work**
|
||||
|
||||
1. Read the validator models, `llm_validators.go`, existing partial-batch tests,
|
||||
`outcomes/outcomes.go`, and `runner.validateSectionCandidates`.
|
||||
2. Implement the reconciliation result/helper and candidate reason codes selected
|
||||
above. Replace the whole-batch mapping-error branch with outcome construction
|
||||
and correct batch accounting. Add the decision-specific recovery constant.
|
||||
3. Move successful-response diagnostic assembly after reconciliation and include
|
||||
its result. Preserve raw/decoded response evidence and diagnostic paths; keep
|
||||
recoverable call errors, fatal errors, and cancellation on their existing paths.
|
||||
4. Keep `EnforceDecisionCardinality` and `EnforcePartialDecisionCardinality` strict.
|
||||
Ensure final validator decisions are complete and unique internally. The runner
|
||||
should continue passing only approved candidates onward without treating an
|
||||
operation's failure metadata as a reason to discard valid decisions.
|
||||
5. Replace obsolete batch-rejection expectations in `llm_validators_test.go` and
|
||||
relevant runner tests. Preserve cardinality tests that exercise broken internal
|
||||
validators. Update the repaired-but-domain-invalid tests so actual unavailable
|
||||
cases remain errors with counts retained and no recovery record.
|
||||
|
||||
**Focused verification**
|
||||
|
||||
Use a small table at the validator boundary for mixed missing/extra indices,
|
||||
conflicting and identical requested duplicates, extra-only responses, empty lists,
|
||||
reordered responses, and a single invalid-confidence decoded test-double entry.
|
||||
Include noncontiguous expected indices and valid negative decisions. Assert semantic
|
||||
outcomes and affected indices, not helper layout or exact messages. Include one
|
||||
controlled case combining successful structural/service recovery facts with ignored
|
||||
extras and one failed reconciliation retaining counts without recovery reasons.
|
||||
|
||||
At the runner boundary, use the real LLM-backed validator with a controlled client
|
||||
and a later validator: prove a survivor reaches later required validation, a later
|
||||
rejection prevents application, and unavailable candidates never reach that later
|
||||
validator as approved. Use successful proposal operation facts where accounting is
|
||||
asserted; do not check an empty operation list. Reuse existing fixtures/tests rather
|
||||
than creating a second framework or duplicating every table case.
|
||||
|
||||
Run `go test ./internal/framework/validators ./internal/framework/runner`.
|
||||
|
||||
**End state:** Usable decisions survive model indexing defects, strict internal
|
||||
cardinality remains satisfied, and operation facts explain partial failure or
|
||||
successful extra filtering. No extra model generation is introduced.
|
||||
|
||||
## Stage 2 — Carry Dispositions Through Debug and Report Partial Outcomes
|
||||
|
||||
**Completed:** 2026-09-25. Runner and ledger records carry explicit
|
||||
dispositions; ordinary reports distinguish unavailable decision errors from
|
||||
extra-index filtering recoveries. Focused report, ledger, runner, and YAML
|
||||
checks passed with `go test ./internal/framework/runner
|
||||
./internal/framework/processreport ./internal/core/reporting`.
|
||||
|
||||
**Work**
|
||||
|
||||
1. Add and populate disposition fields through the runner and ledger as selected
|
||||
above. Map unavailable candidates distinctly and preserve empty replacement
|
||||
serialization. Check both normal and fatal partial-evidence paths.
|
||||
2. Update `processreport` messages and review `Build`, `mapFailure`, and `mapRecovery`.
|
||||
Reuse their existing aggregation; change only assumptions that would misreport
|
||||
the new operation facts. Core YAML structures and Markdown table layout remain.
|
||||
3. Extend existing report/ledger tests for an operation with surviving work plus
|
||||
unavailable indices, extra-only recovery, and a mixed defect operation. Verify
|
||||
error indices exclude survivors, extra filtering does not count as proposal
|
||||
filtering, and failed operations do not also generate recovery records.
|
||||
4. Verify debug ledger entries distinguish genuine rejection from unavailability,
|
||||
and interaction diagnostics preserve ignored entries and reconciliation reasons.
|
||||
Unknown response indices must not create fake ledger candidates. Retain existing
|
||||
redaction and optional-debug-failure tests rather than reproducing them all.
|
||||
5. Update `docs/integrations/output-files.md` for the new recovery reason and debug
|
||||
disposition, and `docs/internal/diagnostics-reporting.md` for the mapping changes.
|
||||
Explain that validation failure indices identify unavailable candidates, not
|
||||
necessarily every candidate submitted to that batch.
|
||||
|
||||
**Validation**
|
||||
|
||||
Run `go test ./internal/framework/runner ./internal/framework/processreport ./internal/core/reporting`.
|
||||
Use semantic YAML/summary assertions; no whole-report prose golden, exact message
|
||||
snapshot, new schema version, or new rendering mode is needed.
|
||||
|
||||
**End state:** Reports and debug evidence accurately describe partial validation
|
||||
success and handled extras through existing reporting mechanisms.
|
||||
|
||||
## Stage 3 — Verify the Assembled Workflow and Finalize Documentation
|
||||
|
||||
**Completed:** 2026-09-25. Controlled subprocess runs covered a mixed decision
|
||||
response and an extra-only response through the CLI, transcript, YAML, Markdown,
|
||||
debug ledger, and interaction evidence. Fixture execution facts showed one
|
||||
provider attempt and zero structural repairs per logical call. `go test
|
||||
./cmd/audita` and `go test ./...` passed. Canonical validator guidance and
|
||||
roadmap status were updated; the two span-deletion plan links now target the
|
||||
historical plan at `60e2fe7`.
|
||||
|
||||
**Work**
|
||||
|
||||
1. Extend the existing local-provider subprocess fixtures in
|
||||
`cmd/audita/main_integration_test.go`. Add a representative mixed-decision run
|
||||
with an applied survivor, a legitimate rejection, a missing requested decision,
|
||||
a duplicate requested decision, and an extra index. Return responses by the
|
||||
requested correction identities; do not freeze incidental validator call order.
|
||||
Ensure the survivor passes all remaining required validation. Assert successful
|
||||
CLI completion, correct transcript, one work-unit error with only unavailable
|
||||
indices, no reconciliation recovery for the failed operation, and debug
|
||||
evidence distinguishing rejection/unavailability without unknown candidates.
|
||||
2. Add or adapt a compact extra-only case proving usable work survives and one
|
||||
`validation_decisions_filtered` recovery appears. Verify YAML, Markdown, and CLI
|
||||
agree on totals using semantic assertions. Keep proposal-filtering counts separate.
|
||||
Existing reporting-disable/debug selection tests remain the owners of those options.
|
||||
3. In these fixtures give PromptKit its normal configured structural repair allowance,
|
||||
but supply schema-valid responses. Verify reconciliation adds no generations or
|
||||
structural repairs using fixture requests/execution facts, without encoding today's
|
||||
total validator count as a permanent contract. Keep structural-malformation tests
|
||||
and recoverable batch-failure tests; do not relax the schema to make fixtures pass.
|
||||
4. Update `docs/internal/validators.md` and any affected canonical descriptions found
|
||||
by searching for whole-batch rejection claims. Explain tolerant external response
|
||||
reconciliation versus strict internal cardinality. The structural-validation
|
||||
boundary remains explicit, and guided recovery remains deferred in `future.md`.
|
||||
5. Mark the feature roadmap implemented only after the assembled checks pass, and
|
||||
record actual completion evidence in this plan. Repair the two span-deletion
|
||||
roadmap links to the old plan with the historical URL
|
||||
`https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md`.
|
||||
Preserve historical release notes and unrelated roadmaps.
|
||||
|
||||
**Validation**
|
||||
|
||||
Run `go test ./cmd/audita` while developing the fixture, then `go test ./...` once
|
||||
for the completed implementation. Run `git diff --check` and verify affected
|
||||
Markdown links. Do not add live-provider checks, a platform matrix, repeated full
|
||||
suite runs, or release gates solely for this feature. No code is implemented by
|
||||
this planning pass; these commands are instructions for the implementation sprint.
|
||||
|
||||
**End state:** The accepted partial-success behavior is exercised through the real
|
||||
CLI/library boundary, canonical documentation matches it, and coherent changes
|
||||
are committed for subsequent release consideration.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None required before implementation. The implementation decisions above preserve
|
||||
the agreed product scope. General structural salvage and guided regeneration remain
|
||||
separate future decisions, not unresolved prerequisites for this work.
|
||||
129
docs/roadmap/library-adoption.md
Normal file
129
docs/roadmap/library-adoption.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# Coordinated ConfigKit and PromptKit Adoption
|
||||
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Production implementation complete. ConfigKit v0.4.0 and PromptKit
|
||||
v0.11.0 are adopted; [historical implementation evidence](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md) records the
|
||||
completed checks. The current `implementation.md` covers span deletion instead.
|
||||
Release compliance and publication remain separate work.
|
||||
|
||||
## Goal and Ownership
|
||||
|
||||
Adopt ConfigKit and PromptKit to align Audita with the application suite's shared
|
||||
configuration and LLM execution conventions. Consistency is the primary benefit;
|
||||
removing overlapping application infrastructure is a secondary benefit. Preserve
|
||||
Audita's transcript-correction capabilities and domain protections.
|
||||
|
||||
| Owner | Responsibilities |
|
||||
| --- | --- |
|
||||
| ConfigKit | Strict application configuration loading, presence/default semantics, explicit overrides, recorded resolution, secret references, and safe inspection |
|
||||
| PromptKit | Prompt/profile formats and rendering, provider calls, structural output validation and repair, transient retries, engine-local backend admission, and execution observations |
|
||||
| Audita | Setting meanings, file/CLI selection, role/profile association, normalization/chunking, ordered modules, domain validation, correction application, reports, diagnostic redaction, and artifact lifecycle |
|
||||
|
||||
The [configuration roadmap](configkit.md) owns configuration and inspection policy.
|
||||
The [boundary findings](library-boundary-findings.md) own the selected defaults,
|
||||
full migration inventory, failure mapping, and prototype evidence. The selected
|
||||
production versions are ConfigKit v0.4.0 and PromptKit v0.11.0, with Go 1.25.5 as
|
||||
the minimum. Use published modules, never sibling checkouts or local replacements.
|
||||
The existing YAML dependency remains while glossary parsing needs it.
|
||||
|
||||
## Migration Posture
|
||||
|
||||
This is an intentional breaking transition targeting Audita v2.0.0. No backward
|
||||
compatibility is required for existing installations, config files, CLI flags,
|
||||
environment overlays, or inspection output. Provide one migration guide, without
|
||||
legacy loaders, aliases, dual modes, or automatic conversion tooling.
|
||||
|
||||
Both libraries belong to one refactor. Intermediate commits need not build or run;
|
||||
there is no supported intermediate product state. No temporary adapters, compatibility
|
||||
shims, or processing gates are needed. The complete application must work and pass
|
||||
its checks before the refactor is finished. Release compliance and publication
|
||||
come last, with no intermediate public release.
|
||||
|
||||
## Configuration and Execution Target
|
||||
|
||||
ConfigKit selected overlays enable a testing configuration that chooses cheaper or
|
||||
faster models. The example selects both proposal and validation profiles. ConfigKit
|
||||
application overlays and PromptKit execution profiles remain separate concepts.
|
||||
Complete operator-maintained profiles own model/generation controls; embedded
|
||||
application prompts and schemas are not operator-selected files. No catalog wiring,
|
||||
profile-inheritance feature, ConfigKit imports, or secret-directory bootstrap is
|
||||
part of this migration.
|
||||
|
||||
Capacity controls serve two needs: respecting a local service's hard simultaneous-
|
||||
query limit and bounding concurrent paid API activity. One finite limit per backend
|
||||
is shared across both roles and every initial, retry, and repair attempt. Limits
|
||||
are engine-local, not coordinated across processes. No additional global or
|
||||
per-role limiter or fine-grained scheduling policy is required. Profiles must use
|
||||
registered backends and ConfigKit-resolved credentials; endpoint-only bypass and
|
||||
competing profile credential lookup are excluded.
|
||||
|
||||
Structural repair is enabled by default for both roles because additional attempts
|
||||
to obtain valid output are desired. Transient retries and structural repairs have
|
||||
separate budgets; explicit zero disables the respective behavior. Each generation,
|
||||
including a repair, receives its own transient-retry budget. The Audita logical-call
|
||||
deadline covers preparation, waiting, backoff, all generations, and repairs; profile
|
||||
HTTP timeouts apply per attempt. Exact defaults and bounds are in the
|
||||
[selected contract](library-boundary-findings.md#configuration-and-profile-contract).
|
||||
No Audita outer retry loop restores the old broader retry policy.
|
||||
|
||||
Audita retains all four correction modules, five LLM-validator prompts,
|
||||
deterministic validators, confidence/applicability checks, glossary protection,
|
||||
validation batching/cardinality, deterministic proposal and ledger ordering,
|
||||
transcript encoders, and artifact retention. PromptKit-valid JSON still passes
|
||||
Audita's domain checks. Repair exhaustion, provider failure, and domain rejection
|
||||
follow the [selected failure table](library-boundary-findings.md#execution-and-failure-mapping).
|
||||
Retention and transcript timestamp changes remain separate work; this migration
|
||||
resolves the [retry-count follow-up](runtime-behavior.md).
|
||||
|
||||
## Reporting and Diagnostic Target
|
||||
|
||||
Keep stable Audita-owned serialized records; do not expose raw library structs.
|
||||
Use process report schema `v2` with the existing domain outcomes and a new
|
||||
`llm_summary` separating logical calls, dispatched provider attempts, transient
|
||||
retries, structural repairs, and usage. Transcript schema versions remain unchanged.
|
||||
The implementation plan specifies the concrete new fields and artifact wiring.
|
||||
|
||||
Usage distinguishes known zero, unknown, and incomplete totals. Preserve observations
|
||||
from failed calls even when no final result exists. Count terminal totals once;
|
||||
attempt observations provide diagnosis without being added to those totals again.
|
||||
This is operational evidence, not a complete billing ledger.
|
||||
|
||||
Replace global/role scheduler utilization with per-backend configured capacity and
|
||||
observed queue/generation/retry/logical-call timing. Do not fabricate measurements
|
||||
that the event API does not provide. Keep domain timing and deterministic correction
|
||||
ledger ordering. Record prompt definition/rendered hashes and selected profile/model
|
||||
without promising the old hash values. Callbacks are short, concurrency-safe, and
|
||||
free of blocking I/O or waits on their own execution.
|
||||
|
||||
Effective configuration and selected-profile snapshots remain distinct safe views.
|
||||
Secret values must not escape through inspection, logs, reports, runtime errors,
|
||||
or retained diagnostic files. Observation records exclude payloads, URLs, raw
|
||||
provider messages, prompts, and credentials. Existing artifact lifecycle and
|
||||
partial-failure handling remain Audita responsibilities.
|
||||
|
||||
## Completion Criteria and Release Boundary
|
||||
|
||||
- Both libraries own their selected responsibilities in the production application.
|
||||
Superseded loaders, transport, renderers, schedulers, aliases, and the isolated
|
||||
prototype are gone once its useful evidence is represented in production tests.
|
||||
- Offline tests prove configuration-to-execution routing for both roles, selected
|
||||
overlays and explicit overrides, shared backend capacity, retry/repair outcomes,
|
||||
cancellation/deadlines, domain decisions, redaction, and truthful partial reports.
|
||||
Tests exercise real internal collaborators and controlled HTTP boundaries.
|
||||
- Maintained examples and canonical docs describe the implemented final model.
|
||||
One migration guide covers files, flags, environment inputs, profiles, credentials,
|
||||
overlays, inspection, and changed runtime/report/diagnostic semantics.
|
||||
- The assembled application passes its required checks; Linux amd64 and arm64 build.
|
||||
macOS arm64 remains best effort. No live-provider or CI-success gate is added.
|
||||
|
||||
PromptKit v0.11.0's reviewed retry, observation, and capacity APIs remove the earlier
|
||||
upstream feature prerequisites; no upstream edit is required for adoption. The
|
||||
prototype proved representative behavior, not release compliance or model quality.
|
||||
|
||||
The selected PromptKit release uses LGPL-3.0-or-later. A later release unit must
|
||||
review the actual production dependency graph and distribution requirements,
|
||||
including notices and source/rebuild materials, before binary publication. This
|
||||
implementation plan does not claim that review is complete. Release publication
|
||||
uses the existing [procedure](../release.md) and completes on successful source/tag
|
||||
push, without waiting for CI or artifact delivery.
|
||||
392
docs/roadmap/library-boundary-findings.md
Normal file
392
docs/roadmap/library-boundary-findings.md
Normal file
@@ -0,0 +1,392 @@
|
||||
# Shared Library Boundary: Decisions and Evidence
|
||||
|
||||
**Status:** Boundary and offline prototype completed, 2026-09-23. Selected prototype contract and proposed
|
||||
production migration target; not the current Audita configuration contract.
|
||||
The [bounded roadmap](library-boundary.md) owns scope and completion criteria.
|
||||
|
||||
**Subsequent scope decision:** The [ConfigKit overlay plan](configkit.md#selected-overlays-for-testing)
|
||||
adds optional selected overlays, with v0.4.0 as the migration baseline. It supersedes
|
||||
the original no-overlay decision and adds an input layer between the main file and
|
||||
explicit setting overrides. The v0.3.1 prototype results below remain historical
|
||||
evidence; they do not establish overlay integration.
|
||||
|
||||
## Selected Boundary
|
||||
|
||||
The former prototype is retained in Git history at `590604b`; it was a separate
|
||||
Go module using published ConfigKit v0.4.0, PromptKit v0.11.0, and
|
||||
Audita v1.1.0. The Audita dependency lets this consumer reuse actual internal domain
|
||||
helpers through its Audita module namespace without a local replacement. It does
|
||||
not make those internal packages a new public API. Remove this dependency along
|
||||
with the prototype after equivalent production integration tests landed.
|
||||
|
||||
The initial prototype results below were recorded with ConfigKit v0.3.1. On
|
||||
2026-09-24, upgrading only its ConfigKit pin and sums to v0.4.0 required no source
|
||||
changes. With Go 1.26.8 on linux/amd64, the nested offline `go test ./...` passed
|
||||
and `go run .` still reported one accepted correction, four attempts, zero
|
||||
transient retries, and two structural repairs. This verifies the dependency
|
||||
upgrade, not selected-overlay behavior.
|
||||
|
||||
ConfigKit owns loading, presence, explicit overrides, secret resolution, and
|
||||
redacted effective settings. PromptKit owns profile parsing, rendering, provider
|
||||
execution, transient retries, structural repair, and backend admission. Audita
|
||||
owns file selection, setting meanings, role selection, domain acceptance, and
|
||||
mapping execution facts into reports. Production CLI and runtime remain unchanged.
|
||||
|
||||
## Configuration and Profile Contract
|
||||
|
||||
Selected for the prototype and recommended for production:
|
||||
|
||||
- Main YAML requires integer `version: 2`; absent, null, unknown fields, and older
|
||||
versions fail. No legacy parser, aliases, environment overlays, or automatic
|
||||
conversion. Defaults precede the selected file; explicitly supplied CLI settings
|
||||
become ConfigKit overrides. Validation runs on the resolved result, so an override
|
||||
may repair an invalid lower-precedence setting when decoding succeeds.
|
||||
- Production file selection: explicit `--config`, then
|
||||
`/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`. Select one file;
|
||||
do not merge them. Missing system files are skipped; explicit missing or empty
|
||||
paths and inaccessible selected files fail. No selected file fails with a request
|
||||
to configure backend/profile selection: no hidden provider or model default.
|
||||
Remove `AUDITA_CONFIG`. The prototype loads its explicit fixture only; discovery
|
||||
and command parsing belong to the ConfigKit implementation stage.
|
||||
- `profiles_dir` defaults to `profiles`, resolved relative to the selected main
|
||||
configuration file, with absolute paths accepted. PromptKit reads that directory.
|
||||
Prompts and schemas are application-embedded, not operator-selected prompt files.
|
||||
The prototype uses complete application-owned profile files. Catalog profiles and
|
||||
profile inheritance are not needed for this migration; do not add catalog wiring.
|
||||
- `backends.<id>.endpoint` is required. Use one application backend identity per
|
||||
service capacity pool. Do not split proposal and validation into different backend
|
||||
IDs when they use the same service. Limits are engine-local, not shared across
|
||||
multiple Audita processes or other applications.
|
||||
- `backends.<id>.concurrency` defaults to **1**, must be positive, and is shared by
|
||||
both roles and all initial/retry/repair attempts. Operators can select **4** for
|
||||
the described local backend or a suitable finite paid API limit. There is no
|
||||
additional global or per-role limiter. `queue_capacity` defaults to **1024**,
|
||||
following PromptKit; non-negative, with zero meaning no waiting admission.
|
||||
This is an operational escape hatch rather than a second scheduling hierarchy.
|
||||
- `backends.<id>.credential` is an optional ConfigKit secret reference: a scalar
|
||||
environment variable name. Omission or null means no credential. A supplied
|
||||
reference must resolve to a nonempty value, including during validation and
|
||||
inspection. There is no direct key field or CLI key flag. ConfigKit resolves it
|
||||
once and Audita supplies `RunRequest.APIKey`. Profiles must not introduce another
|
||||
`api_key_env` lookup. Backend credentials also support unauthenticated local use.
|
||||
- Profiles own model, temperature, output token limit, top-p, per-generation timeout,
|
||||
service tier, reasoning effort, and extra provider parameters using PromptKit's
|
||||
format. They select the application `backend`; their effective endpoint must
|
||||
equal that backend's endpoint. Endpoint-only execution is rejected because it
|
||||
bypasses the requested finite capacity boundary. A named registered backend is
|
||||
mandatory even if both roles use the same profile.
|
||||
- `llm.proposal_profile` is required. Omitted `llm.validation_profile` inherits the
|
||||
final proposal profile as a whole, with ConfigKit recording the derivation.
|
||||
Explicit empty or null values fail; no individual model/endpoint/key inheritance.
|
||||
- `llm.transient_retries` defaults to **2** additional attempts per generation;
|
||||
explicit zero disables retries; negative is invalid. PromptKit owns which failures
|
||||
are eligible. Do not add an Audita retry loop.
|
||||
- `llm.repair_attempts` defaults to **2** additional generations for both roles;
|
||||
explicit zero disables repair when deliberately requested. PromptKit supports
|
||||
0–3. Positive defaults are exercised in the prototype. Audita owns this common
|
||||
setting and supplies a complete request output contract with the role's embedded
|
||||
schema, avoiding separate user-facing repair defaults in each prompt.
|
||||
- `llm.operation_timeout` defaults to **15m**, must be positive, and uses a Go
|
||||
duration string. It starts before each logical proposal/assessment call and covers
|
||||
preparation, capacity waiting, transient backoff, generation, and repairs. It is
|
||||
separate from whole-process cancellation. Profile `timeout_seconds` defaults to
|
||||
PromptKit's **600** seconds and is applied afresh to each built-in HTTP attempt.
|
||||
It does not bound the combined retry/repair sequence. A stricter caller/transport timeout can
|
||||
also end execution. The two defaults are selected starting points, not throughput
|
||||
measurements. With the default T=2 and R=2, each logical call dispatches at most
|
||||
**9** attempts before deadline/cancellation; proposal plus assessment may use 18.
|
||||
- Omission retains static defaults or invokes the recorded role/backend derivations.
|
||||
Null is not a general synonym for omission: nonoptional scalars reject it. Optional
|
||||
credentials and the future optional section target accept null. Explicit empty
|
||||
module lists will mean an empty correction pipeline, not “restore defaults.”
|
||||
Repeated modules retain their existing ordered meaning.
|
||||
- The original prototype disabled ConfigKit imports, selected configuration layers
|
||||
(then called configuration profiles), and secret-directory bootstrap. Selected
|
||||
overlays are now verified under the linked overlay decision; the other two
|
||||
features remain disabled. PromptKit execution profiles retain their meaning.
|
||||
|
||||
Configuration validation/inspection must assemble and inspect both selected profiles
|
||||
without provider calls. ConfigKit's effective YAML describes only its loaded state;
|
||||
a separate selected-profile view shows effective model controls/backend selection.
|
||||
Do not imply either snapshot alone represents the full execution configuration.
|
||||
The prototype exercises the two APIs but does not implement production subcommands.
|
||||
Do not serialize raw profile/library objects as a permanent Audita report schema.
|
||||
|
||||
## Migration Inventory
|
||||
|
||||
This inventory covers the current configuration and process flags documented in
|
||||
[configuration](../config.md) and [CLI](../cli.md), including environment-only fields.
|
||||
“Retain” below means retain domain meaning in the new schema, not accept old files.
|
||||
All legacy environment overlays listed afterward are removed.
|
||||
|
||||
| Current file field / setting | Current CLI flag(s) | Target and rationale |
|
||||
| --- | --- | --- |
|
||||
| File selection | `--config` | Retain explicit invocation flag; discovery described above. |
|
||||
| Transcript input | positional path | Invocation only; retain one transcript path. |
|
||||
| Glossary input | `--glossary` | Invocation only; retain required glossary path. |
|
||||
| Corrected output | `--output` | Invocation only; retain stdout fallback. |
|
||||
| Report output | `--report-json` | Invocation only; retain separate report file. |
|
||||
| `pipeline.modules` | `--modules` | Retain sequence/default `[glossary, homophones, glossary, spoken_word, grammar]`; explicit empty clears rather than inherits. |
|
||||
| `output.schema` | `--output-schema` | Retain encoder selection/default `bare-segments`. |
|
||||
| `llm.proposal.base_url`, `llm.validation.base_url` | `--base-url`, `--validation-base-url` | Replace with backend endpoint; remove per-role endpoint flags. |
|
||||
| Both `llm.*.model` | `--model`, `--validation-model` | Replace with profile model; new `--proposal-profile` / `--validation-profile` select whole profiles. |
|
||||
| Both `llm.*.api_key_env`, direct runtime keys | `--llm-api-key`, `--validation-llm-api-key` | Replace with backend ConfigKit secret reference; remove direct key flags. |
|
||||
| Both `llm.*.timeout` | `--llm-timeout-seconds`, `--validation-llm-timeout-seconds` | Replace with profile `timeout_seconds`, plus `llm.operation_timeout`; no duplicate timeout flags. |
|
||||
| Both `llm.*.max_retries` | `--max-retries`, `--validation-max-retries` | Replace with common `llm.transient_retries` and distinct `llm.repair_attempts`; remove misleading old flags. |
|
||||
| `concurrency.total_llm` | `--total-llm-concurrency`, `--llm-concurrency` | Replace with per-backend concurrency; remove global/alias flags. |
|
||||
| `concurrency.proposal_llm`, `concurrency.validation_llm` | `--proposal-llm-concurrency`, `--validation-llm-concurrency` | Remove role budgets; shared backend handles service capacity. |
|
||||
| Validation prompt budget (no old YAML field) | `--validation-max-prompt-tokens` | Add `validation.max_prompt_tokens`, default 2048, positive; retain explicit flag override. Audita owns batching. |
|
||||
| `chunking.target_sections` | `--target-sections` | Retain optional positive target; omission/null means automatic, zero invalid. |
|
||||
| `chunking.max_section_tokens` | `--max-section-tokens` | Retain 8192, positive. |
|
||||
| `chunking.min_section_tokens` | `--min-section-tokens` | Retain 2048, positive, no greater than maximum. |
|
||||
| `normalization.max_segment_gap` | `--normalize-max-segment-gap` | Retain 4.0 numeric seconds, non-negative; drop alternate YAML duration syntax for these domain numeric fields. |
|
||||
| `normalization.ellipsis_gap` | `--normalize-ellipsis-gap` | Retain 3.5 numeric seconds, non-negative. |
|
||||
| `normalization.max_segment_duration` | `--normalize-max-segment-duration` | Retain 60.0 numeric seconds, positive. |
|
||||
| `normalization.max_segment_tokens` | `--normalize-max-segment-tokens` | Retain 2048, positive. |
|
||||
| `thresholds.glossary` | `--glossary-confidence-threshold` | Retain 0.8 in [0,1]. |
|
||||
| `thresholds.homophones` | `--homophones-confidence-threshold` | Retain 0.8 in [0,1]. |
|
||||
| `thresholds.spoken_word` | `--spoken-word-confidence-threshold` | Retain 0.8 in [0,1]. |
|
||||
| `thresholds.grammar` | `--grammar-confidence-threshold` | Retain 0.8 in [0,1]. |
|
||||
| `context.description` | `--transcript-description` | Retain trimming and 500-byte limit; empty clears context. |
|
||||
| `diagnostics.work_dir` | `--work-dir` | Retain `/tmp/audita`, nonempty; artifact lifecycle remains Audita-owned. |
|
||||
| `diagnostics.retention` | `--work-dir-retention` | Retain `auto` default and selection; existing `never` follow-up is separate. |
|
||||
| Help/version/config commands | `--help`, `-h` | Retain command purpose; help should parse before loading configuration. Effective-config output intentionally becomes ConfigKit YAML, not legacy JSON. |
|
||||
|
||||
Retained setting flags become explicit ConfigKit overrides only when supplied;
|
||||
CLI parser defaults must not overwrite file values. Normalization flag values stay
|
||||
numeric seconds. Domain values convert into existing domain structs after validation;
|
||||
do not spread ConfigKit or PromptKit types into correction algorithms.
|
||||
|
||||
Complete environment removal map (names remain permissible if explicitly named as
|
||||
secret references; none have implicit configuration meaning):
|
||||
|
||||
| Old environment names | Replacement |
|
||||
| --- | --- |
|
||||
| `AUDITA_CONFIG` | File selection above. |
|
||||
| `AUDITA_MODULES` | `pipeline.modules` / explicit flag. |
|
||||
| `AUDITA_LLM_API_KEY`, `OPENROUTER_API_KEY`, `AUDITA_VALIDATION_LLM_API_KEY` | Explicit backend credential references; no implicit fallback. |
|
||||
| `AUDITA_MODEL`, `AUDITA_VALIDATION_MODEL` | Profile files and role selection. |
|
||||
| `AUDITA_BASE_URL`, `AUDITA_VALIDATION_BASE_URL` | Backend endpoints. |
|
||||
| `AUDITA_LLM_TIMEOUT_SECONDS`, `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS` | Profile timeout and operation deadline. |
|
||||
| `AUDITA_MAX_RETRIES`, `AUDITA_VALIDATION_MAX_RETRIES` | Common transient/repair settings. |
|
||||
| `AUDITA_TOTAL_LLM_CONCURRENCY`, `AUDITA_LLM_CONCURRENCY`, `AUDITA_PROPOSAL_LLM_CONCURRENCY`, `AUDITA_VALIDATION_LLM_CONCURRENCY` | Backend concurrency; remove alias/role budgets. |
|
||||
| `AUDITA_VALIDATION_MAX_PROMPT_TOKENS` | `validation.max_prompt_tokens` / explicit flag. |
|
||||
| `AUDITA_TARGET_SECTIONS`, `AUDITA_MAX_SECTION_TOKENS`, `AUDITA_MIN_SECTION_TOKENS` | Corresponding chunking fields / flags. |
|
||||
| `AUDITA_NORMALIZE_MAX_SEGMENT_GAP`, `AUDITA_NORMALIZE_ELLIPSIS_GAP`, `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION`, `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS` | Corresponding normalization fields / flags. |
|
||||
| `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD`, `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD`, `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD`, `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD` | Corresponding threshold fields / flags. |
|
||||
| `AUDITA_WORK_DIR`, `AUDITA_WORK_DIR_RETENTION` | Diagnostics fields / flags. |
|
||||
|
||||
## Execution and Failure Mapping
|
||||
|
||||
The executable uses grammar proposal plus grammar review, current response schemas,
|
||||
current hardening text, transcript-section/description helpers, proposal preview, and
|
||||
actual decision-cardinality enforcement from Audita v1.1.0. PromptKit renders the
|
||||
adapted templates; the custom HTTP transport supplies OpenAI-compatible responses.
|
||||
The prototype pre-expands Audita's `hardening` template function into the embedded
|
||||
PromptKit definitions; it passes existing named string variables unchanged. No
|
||||
provider, paid call, or injected high-level LLM client participates.
|
||||
|
||||
Generated JSON is structurally validated/repaired by PromptKit before Audita decodes
|
||||
it. Audita then checks proposal applicability, confidence, and decision identity and
|
||||
cardinality. The demonstration previews corrections without mutation. Production
|
||||
retains every module's full validator chain, glossary protection, batching, and
|
||||
apply-time checks; the representative prototype does not claim to replace them.
|
||||
|
||||
| Outcome | Section/batch effect | Overall run |
|
||||
| --- | --- | --- |
|
||||
| Structural repair succeeds | Continue domain validation. | Normal execution. |
|
||||
| Proposal repair exhausted | No proposals from this section; malformed-output warning. | Continue other work; successful completion may include warnings. |
|
||||
| Assessment repair exhausted | Reject this batch; malformed-output warning. | Continue other work. |
|
||||
| Schema-valid invalid/missing/duplicate decision indices | Reject batch and warn; no semantic retry/repair. | Continue other work. |
|
||||
| Inapplicable/low-confidence proposal or negative assessment | Reject affected proposal; preserve domain reasons. | Continue other work. |
|
||||
| Provider failure, retry exhaustion, transport or malformed HTTP response | Stop affected operation; retain available observations. | Runtime failure; cancel remaining work, retain existing Audita diagnostic handling. |
|
||||
| Deadline, caller cancellation, capacity admission failure | Stop affected operation; no outer retry or fabricated output. | Runtime failure with distinct classified cause. |
|
||||
| Invalid settings/profile/schema setup | Fail before processing where detectable. | Configuration error for user input, runtime error for broken embedded assets. |
|
||||
|
||||
Zero transient retries produces one ordinary provider failure, not
|
||||
`ErrRetriesExhausted`; the latter signals an exhausted **positive** budget.
|
||||
Provider failures retain a typed cause. Do not expose raw provider messages through
|
||||
user diagnostics: continue Audita's separate secret-redaction boundary. A failed
|
||||
logical call may return no result while its observations still record completed
|
||||
attempts or an earlier successful generation. Already completed domain results can
|
||||
remain in diagnostics; failure must not be presented as a fully corrected output.
|
||||
|
||||
## Observation and Report Handoff
|
||||
|
||||
The prototype maps library events into Audita-owned records under a short mutex.
|
||||
Records contain logical run/role/backend/profile identity, generation and attempt
|
||||
indices, dispatch state, queue/generation/retry/wall timing, classified outcomes,
|
||||
HTTP status, exhaustion, repair/retry counts, and each usage field's value,
|
||||
knownness, and completeness. It omits URLs, credentials, payloads, raw provider
|
||||
errors, and prompt text. Identifiers themselves are metadata, not a secret channel.
|
||||
|
||||
Use terminal records once for logical totals; retain attempt records for diagnosis
|
||||
without adding them to terminal totals. A known zero is distinct from unknown;
|
||||
partial totals remain incomplete after unmetered failed attempts. These observations
|
||||
are not a billing ledger. Callback code must not wait on its own run's progress.
|
||||
|
||||
Production report changes must distinguish logical calls from dispatched provider
|
||||
attempts. Replace scheduler utilization fields based on global/proposal/validation
|
||||
pools with per-backend admission/active timing. Preserve deterministic ledger and
|
||||
validator ordering. Map prompt definition/rendered hashes and selected profile/model
|
||||
into request metadata, but do not promise equality with legacy prompt hashes or
|
||||
serialize PromptKit's raw event struct as the stable report contract. Update the
|
||||
output integration reference, report mapper tests, and downstream migration guide
|
||||
when the production representation is implemented.
|
||||
|
||||
## Verification and Remaining Handoff
|
||||
|
||||
The focused matrix is implemented in the nested module:
|
||||
|
||||
| Roadmap scenario | Evidence |
|
||||
| --- | --- |
|
||||
| Successful assembled path | `go run .`; `TestWorkflowBoundaries/repair_both_roles`; actual Audita domain helpers and both selected profiles. |
|
||||
| Presence, overrides, inheritance | `TestConfigurationContract`: omitted role/backend values, explicit zero, null/empty, version, unknown keys, and explicit overrides. |
|
||||
| Credentials | Missing/unset and empty references fail; optional null accepted; effective/profile inspection redacted; HTTP authorization verified; no secret/prompt text in observations. `TestProfilesRespectSharedBackendBoundary` rejects competing lookup and capacity bypass. |
|
||||
| Both roles repaired | `repair_both_roles`: four attempts, two repairs, one accepted correction. |
|
||||
| Repair exhausted | Proposal and assessment exhaustion subtests: bounded counts and selected warning/rejection outcomes. |
|
||||
| Semantic boundary | Missing span, missing/duplicate decision indices, negative assessment; no semantic repair loop. |
|
||||
| Retry/zero/exhaustion | 429 recovery, zero-budget 503, positive-budget exhaustion after completed proposal; typed error assertions. |
|
||||
| Combined budgets | 503 → invalid JSON → repair 503 → valid proposal → assessment: five attempts, two transient retries, one repair. |
|
||||
| Shared capacity | `TestSharedCapacityAndRecovery`: caps four and two, both roles, a retry and repair, queue rejection, measured peak equal to cap, later successful call. |
|
||||
| Waiting/cancellation/deadline | `TestQueuedCancellationAndDeadline`; `TestDeadlineStopsRetryAfter`: no queued dispatch after cancellation/deadline and a 60-second provider retry delay interrupted by the operation deadline. |
|
||||
| Failed-run observations | `TestPartialUsageOnFailedRun`: valid HTTP output with known-zero usage followed by repair failure; nil result, retained events, incomplete known aggregate, correct counts. |
|
||||
| Representative assets | `TestRealPromptRenderingParity` compares both rendered message pairs to Audita v1.1.0; `TestSchemaFixturesMatchAudita` compares schema bytes. |
|
||||
|
||||
Commands executed from `examples/library-adoption` on Linux amd64 with Go 1.26.8:
|
||||
|
||||
```sh
|
||||
GOWORK=off go mod download all
|
||||
GOWORK=off GOPROXY=off GOSUMDB=off go mod verify
|
||||
GOWORK=off GOPROXY=off GOSUMDB=off go run .
|
||||
GOWORK=off GOPROXY=off GOSUMDB=off go test -race ./...
|
||||
GOWORK=off GOPROXY=off GOSUMDB=off go vet ./...
|
||||
```
|
||||
|
||||
All passed. The demonstration reports accepted=1, rejected=0, attempts=4,
|
||||
transient_retries=0, structural_repairs=2. Module verification reports all modules
|
||||
verified. Published tag availability was checked with `git ls-remote --tags` for
|
||||
ConfigKit v0.3.1 and PromptKit v0.11.0; successful module acquisition verifies this
|
||||
is not merely a local-checkout experiment. The module has no replacements.
|
||||
|
||||
The minimum Go 1.25.5 toolchain was also exercised with offline tests and the same
|
||||
successful demonstration. The exact installed-toolchain invocation was:
|
||||
|
||||
```sh
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off /home/eric/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.5.linux-amd64/bin/go test ./...
|
||||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off /home/eric/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.5.linux-amd64/bin/go run .
|
||||
```
|
||||
|
||||
An initial attempt to select that cached toolchain through `GOTOOLCHAIN=go1.25.5`
|
||||
with checksum verification disabled was rejected by Go's toolchain verifier. Using
|
||||
the installed binary with `GOTOOLCHAIN=local` proved offline execution without
|
||||
weakening dependency checks. The portable README commands assume an installed
|
||||
sufficiently new Go executable on PATH and disable toolchain switching.
|
||||
|
||||
From the Audita root, `go test ./...` and
|
||||
`go build -o ./bin/audita ./cmd/audita` passed. These check existing production code;
|
||||
they do not cover the nested module. Affected relative documentation links,
|
||||
inventory coverage, and `git diff --check` were checked separately.
|
||||
|
||||
### Findings and Limits
|
||||
|
||||
No adoption-blocking upstream API gap was found. Two integration details deserve
|
||||
explicit attention: PromptKit's profile GoDoc calls `timeout_seconds` a generation
|
||||
deadline, but its built-in HTTP path applies it separately to each attempt; the
|
||||
consumer therefore needs the selected total-operation deadline. Also,
|
||||
`APIKeyRequired` is an in-memory profile facility, not a profile-YAML field. The
|
||||
file-profile design uses the presence of ConfigKit's optional credential reference
|
||||
to require local secret resolution; it does not invent a new PromptKit YAML key.
|
||||
|
||||
The chosen defaults are proposed operational policy, not measured model-quality or
|
||||
cost recommendations. Tests use controlled transport responses and synthetic data;
|
||||
they do not establish real provider compatibility, model quality, network behavior,
|
||||
throughput, cross-process coordination, complete billing, or release compliance.
|
||||
Only the representative LLM-facing subset is implemented in the prototype; the
|
||||
inventory specifies the remaining domain fields for the ConfigKit migration.
|
||||
Production file discovery, CLI parsing, full module chains, all nine prompts,
|
||||
filesystem diagnostics/redaction, and serialized report migration remain their
|
||||
assigned production tasks. Those are deliberate scope boundaries, not missing
|
||||
claims of completion for the prototype.
|
||||
|
||||
The new profile/credential/capacity and deadline choices are explicit above; no
|
||||
unresolved upstream requirement prevents starting the ConfigKit implementation
|
||||
plan. Production report field names and final runtime wiring are selected in the
|
||||
[implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md), following the outcome and observation
|
||||
contract here. The coordinated refactor targets both libraries directly; intermediate commits do not require a working legacy runtime.
|
||||
|
||||
### Production Implementation Reference
|
||||
|
||||
The [implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md) owns ordered production work,
|
||||
component changes, tests, and acceptance criteria. The [combined roadmap](library-adoption.md)
|
||||
and [configuration roadmap](configkit.md) own the final target. This evidence does
|
||||
not require an interim runtime bridge or separately runnable configuration migration.
|
||||
|
||||
Retained domain scope includes transcript/glossary schemas, normalization/chunking,
|
||||
output encoding, proposals and apply-time checks, deterministic validators, ordered
|
||||
modules, LLM-validator batching/cardinality, ledger ordering, and artifact lifecycle.
|
||||
The production integration replaces loading, provider execution, prompt rendering,
|
||||
and scheduler composition while preserving those responsibilities. Retention and
|
||||
timestamp changes remain separate work.
|
||||
|
||||
### Distribution Follow-Up
|
||||
|
||||
The pinned graph includes ConfigKit, PromptKit, both PromptKit backend catalogs,
|
||||
jsonschema/v6, yaml.v3, yaml.v4, x/text, and kr/text (including module/test dependency
|
||||
edges). Before distribution, review the actual production dependency graph and
|
||||
its license files, including PromptKit/catalog LGPL-3.0-or-later terms. Decide and
|
||||
verify the notices, license copies, corresponding library source/modifications,
|
||||
and rebuild/relink materials needed for Audita's Go binary/container distribution.
|
||||
This unit does not publish binaries or claim that release compliance is complete.
|
||||
No upstream relicensing request or library implementation change is required for
|
||||
this prototype.
|
||||
|
||||
## Selected-Overlay Extension: 2026-09-24
|
||||
|
||||
The isolated prototype now pins published ConfigKit v0.4.0. ConfigKit loads
|
||||
`overlays/testing.yml` relative to the selected main file; the overlay selects
|
||||
complete PromptKit execution profiles from the separate `profiles/` directory.
|
||||
The original v0.3.1 observations above are historical and do not prove this
|
||||
extension. No production Audita behavior changed.
|
||||
|
||||
With Go 1.26.8 on linux/amd64, from `examples/library-adoption/`, these offline
|
||||
checks passed with `GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOSUMDB=off`:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go run .
|
||||
go run . --overlay testing
|
||||
go run . --overlay testing --proposal-profile proposal
|
||||
```
|
||||
|
||||
Each demonstration reported one accepted correction, four attempts, zero
|
||||
transient retries, and two structural repairs. The base run selected
|
||||
`synthetic-grammar`/`synthetic-review`; testing selected
|
||||
`synthetic-grammar-testing`/`synthetic-review-testing`; the explicit proposal
|
||||
override restored only `synthetic-grammar`. These are synthetic routing IDs,
|
||||
not price, latency, quality, or provider-compatibility measurements.
|
||||
|
||||
| Requirement | Consumer evidence |
|
||||
| --- | --- |
|
||||
| Base, empty/default, testing selection, role overrides, final inheritance, and relative profile resolution | `TestSelectedOverlayConfiguration` loads temporary on-disk layouts and assembles both roles. |
|
||||
| Actual base/testing/override HTTP models and repairs | `TestDemonstrationSelectsOutboundModels` captures every request body and matches model IDs to the reported selections and domain outcome. |
|
||||
| Missing/invalid selection, version/directive rejection, and ordinary backend-setting validation | `TestOverlayFailuresAndApplicationValidation` checks ConfigKit diagnostic codes and effective backend limits. |
|
||||
| Surviving credential resolution, absent/empty failures, null unauthenticated use, and profile-only switching | `TestOverlayCredentialResolutionAndInspection` controls synthetic environment values and observes HTTP Authorization. |
|
||||
| Redacted effective output, explanation and metadata origin, profile inspection, and observations | `TestOverlayCredentialResolutionAndInspection` and `TestOverlaySelectionProvenance` inspect safe output and overlay/CLI origins. |
|
||||
| Existing retry, repair, domain, capacity, and concurrent-observation behavior | `TestWorkflowBoundaries`, `TestSharedCapacityAndRecovery`, `TestQueuedCancellationAndDeadline`, `TestDeadlineStopsRetryAfter`, and `TestPartialUsageOnFailedRun` pass, including under `-race`. |
|
||||
|
||||
Changing a profile does not remove a backend credential reference from the
|
||||
effective configuration. ConfigKit validates a surviving reference when only
|
||||
profile selection changes. Replacing the reference in an
|
||||
overlay resolves only the replacement; explicit null allows unauthenticated use.
|
||||
This distinction is part of the selected configuration boundary, not an
|
||||
application fallback. Imports and secret-directory bootstrap remain disabled.
|
||||
|
||||
The tests use a controlled built-in HTTP transport and synthetic credentials;
|
||||
there were no live provider calls. The prototype still covers one representative
|
||||
proposal/assessment pair, not all production modules, prompts, CLI paths, or
|
||||
diagnostic artifacts. Root `go test ./...` does not run this nested module. No
|
||||
upstream API gap appeared in this extension. The [production implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md)
|
||||
keeps implementation and release pending.
|
||||
269
docs/roadmap/library-boundary.md
Normal file
269
docs/roadmap/library-boundary.md
Normal file
@@ -0,0 +1,269 @@
|
||||
# Library Adoption: Shared Boundary and Prototype
|
||||
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Original boundary-design and offline-prototype unit completed;
|
||||
overlay extension and production-planning handoff completed, 2026-09-24.
|
||||
The work packages below retain the original scope;
|
||||
[decisions, evidence, limitations, and handoff](library-boundary-findings.md)
|
||||
record the outcome. The executable prototype was removed after production
|
||||
integration; Git history at `590604b` retains it. No publication or upstream
|
||||
changes were made in that historical unit.
|
||||
|
||||
## Next Unit: Testing Overlays and Production Migration Plan
|
||||
|
||||
**Status:** Complete. The [production implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md)
|
||||
owns subsequent work; the [overlay findings](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
|
||||
record this unit's checks. The original work packages below remain
|
||||
historical scope for the completed first unit.
|
||||
|
||||
Extend the existing prototype to prove the [selected-overlay contract](configkit.md#selected-overlays-for-testing)
|
||||
with ConfigKit v0.4.0, then turn the verified findings into a detailed production
|
||||
ConfigKit migration plan. Do not repeat the original inventory or redesign the
|
||||
established library boundary without new evidence.
|
||||
|
||||
Required work:
|
||||
|
||||
- Upgrade only the nested prototype module from ConfigKit v0.3.1 to v0.4.0.
|
||||
- Pass an explicit overlay selection through ConfigKit and add `overlays/testing.yml`
|
||||
plus two PromptKit testing profiles. Select both roles and retain one shared
|
||||
finite backend. Synthetic model names demonstrate routing; they do not benchmark
|
||||
real model cost, speed, or quality.
|
||||
- Make the offline executable accept overlay and explicit role-profile selections,
|
||||
and show the selected models alongside domain outcomes and retry/repair counts.
|
||||
- Verify base/overlay/CLI precedence, role inheritance, invalid selections,
|
||||
surviving credential references, safe inspection, and overlay provenance.
|
||||
Prove the actual HTTP request models for both roles; loading changed strings
|
||||
alone is insufficient evidence.
|
||||
- Record new results separately from the historical v0.3.1 findings. Expand the
|
||||
existing ConfigKit roadmap into a concrete production implementation handoff,
|
||||
including final runtime integration and removal of legacy configuration.
|
||||
|
||||
Target end state: reviewers can run the prototype with and without the testing
|
||||
selection and see both LLM roles switch models. The offline test suite establishes
|
||||
that selection, inspection, and execution agree, and the production ConfigKit work
|
||||
has bounded tasks with explicit acceptance criteria.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- The published v0.4.0 dependency is pinned without local replacements; the original
|
||||
no-overlay path and existing integration tests continue to pass.
|
||||
- `testing` switches both roles, explicit profile flags override its selections,
|
||||
and omission/empty/`default` retains base selection according to ConfigKit.
|
||||
- Invalid overlays fail clearly; inheritance and secret resolution use surviving
|
||||
effective settings. Inspection records overlay origin without disclosing keys.
|
||||
- Both selected testing models reach the controlled HTTP transport; structural
|
||||
repair, shared capacity, observations, and Audita domain checks remain active.
|
||||
- Updated examples, executable instructions, findings, and a detailed production
|
||||
migration plan are reviewable. Exact verification commands and limitations are
|
||||
recorded, including the fact that root tests do not cover the nested module.
|
||||
- Production code, root dependencies, and upstream libraries remain unchanged.
|
||||
No release or live provider call is part of this unit.
|
||||
|
||||
This unit ends with a production migration **plan**, not production implementation.
|
||||
The ConfigKit → PromptKit → combined release sequence remains unchanged. Imports
|
||||
and secret-directory bootstrap remain disabled; PromptKit execution profiles keep
|
||||
their existing names and meaning.
|
||||
|
||||
## Objective
|
||||
|
||||
Define and prove the boundary between Audita, ConfigKit, and PromptKit before
|
||||
replacing Audita's production configuration or LLM runtime. This unit covers the
|
||||
shared design and representative prototype supporting the
|
||||
[coordinated adoption roadmap](library-adoption.md). The
|
||||
[ConfigKit roadmap](configkit.md) continues to own the broader configuration scope.
|
||||
|
||||
The deliverable is a concrete target contract supported by an offline executable
|
||||
prototype, sufficient to implement the ConfigKit migration without subsequently
|
||||
redesigning its LLM-facing settings. A successful prototype is not completion of
|
||||
either production migration.
|
||||
|
||||
## Established Direction
|
||||
|
||||
- This is an intentional breaking migration with a migration guide, not legacy
|
||||
configuration support, aliases, or a permanent compatibility mode.
|
||||
- ConfigKit owns shared configuration mechanics; PromptKit owns shared prompt and
|
||||
generation mechanics; Audita retains domain meaning and transcript correction.
|
||||
- Structural repair is enabled from the outset for proposal and LLM-validator calls.
|
||||
Positive default repair budgets must be selected, not silently left at zero.
|
||||
- Transient retries and structural repair are separate. Explicit zero disables
|
||||
transient retries; Audita must not add a second automatic retry loop.
|
||||
- One finite concurrency limit per backend protects local capacity and bounds paid
|
||||
API activity. Calls from both roles share that limit. Fine-grained role budgets
|
||||
and an additional global concurrency control are unnecessary.
|
||||
- Keep the sequence: boundary/prototype, ConfigKit migration, PromptKit migration,
|
||||
then combined major release. No intermediate public release is planned.
|
||||
|
||||
## Scope and Deliverables
|
||||
|
||||
| Deliverable | Required result |
|
||||
| --- | --- |
|
||||
| Configuration inventory and migration map | Every existing setting and invocation option has a proposed destination, replacement, or removal, with rationale. |
|
||||
| Boundary specification | One authoritative source for each setting, explicit precedence/defaults, credential flow, runtime conversion, and failure handling. |
|
||||
| Representative executable prototype | New-style configuration drives one proposal and one assessment through released libraries and a controlled provider. |
|
||||
| Focused behavioral evidence | Retry, repair, capacity, credential, observation, and domain-boundary scenarios below are demonstrated. |
|
||||
| Decision and integration handoff | Concrete choices, measured limitations, affected components, removal plan, and next implementation tasks are recorded. |
|
||||
|
||||
Do not switch the production CLI to the prototype, migrate every module/prompt,
|
||||
redesign transcript/glossary formats, change correction algorithms, modify retention
|
||||
or timestamp behavior, or publish a release in this unit. Do not reproduce the
|
||||
libraries' full test suites. Upstream changes require a separately scoped task if
|
||||
an actual gap is found.
|
||||
|
||||
## Work Package 1: Inventory and Decide Ownership
|
||||
|
||||
Read the current [configuration](../config.md), [CLI](../cli.md),
|
||||
[LLM runtime](../internal/llm-runtime.md), [prompts](../internal/prompts.md),
|
||||
[validators](../internal/validators.md), and [output contracts](../integrations/output-files.md).
|
||||
Use graph-first discovery to verify the relevant code and tests, especially
|
||||
configuration application, prompt construction, structured responses, scheduling,
|
||||
and report assembly.
|
||||
|
||||
Build a migration table grouped by application/domain settings, invocation state,
|
||||
backend settings, execution profiles, output contracts, and inspection. Include
|
||||
current environment settings and aliases without treating their existence as a
|
||||
requirement to preserve them. Identify report fields whose meanings change when
|
||||
one logical generation contains multiple provider attempts.
|
||||
|
||||
Resolve these design decisions explicitly:
|
||||
|
||||
| Decision | Proposed direction to prove |
|
||||
| --- | --- |
|
||||
| File selection and version | Audita selects the main configuration and validates a new schema version; specify discovery and the treatment of old versions. |
|
||||
| Application versus invocation state | ConfigKit loads persistent settings; Audita parses invocation-only input/output paths and translates explicitly supplied setting flags into supported overrides. |
|
||||
| Backend identity | One registration represents the service capacity shared by proposal and validation calls. Avoid separate identities for the same service merely because roles differ. |
|
||||
| Endpoint and capacity ownership | Define them once per backend, with finite limits and an explicit policy for waiting capacity/exhaustion. Do not expose endpoint-only unlimited execution accidentally. |
|
||||
| Execution profiles | Prefer complete application-owned profiles using PromptKit conventions. Decide their source and selection without duplicating model/settings fields in application configuration. Catalog selection remains a deliberate option. |
|
||||
| Role selection and inheritance | Specify how proposal and validation choose profiles, including omission and any inheritance. Do not inherit old behavior solely for compatibility. |
|
||||
| Credentials | Prefer one ConfigKit-owned secret-resolution path passed through PromptKit's supported request credential field. Define backend/profile credential association and avoid competing environment lookups. Validate this against profile requirements. |
|
||||
| Retry, repair, and deadlines | Assign one owner and explicit defaults for each control. Use positive structural-repair defaults and document combined limits and total-operation scope. |
|
||||
| Optional library features | The original prototype omitted imports, selected configuration layers, and secret-directory bootstrap. Selected overlays are now planned for testing; see the [overlay plan](configkit.md#selected-overlays-for-testing). Imports and secret-directory bootstrap remain out. |
|
||||
| Inspection | Define effective configuration and selected-profile inspection without implying that one library's snapshot contains the other's complete state. |
|
||||
|
||||
Choose concrete field names, example values, defaults, units, and omission/null/zero
|
||||
semantics for the prototype. In particular, decide what validation and inspection
|
||||
do when configured credentials are unavailable. Classify choices as proposed,
|
||||
selected, or unresolved; do not present candidates as implemented contracts.
|
||||
|
||||
## Work Package 2: Specify Execution and Reporting Semantics
|
||||
|
||||
Document the flow from selected configuration through backend/profile assembly,
|
||||
prompt inputs, generation, structural validation/repair, typed decoding, and Audita
|
||||
domain validation. Identify which values cross the boundary and which component
|
||||
owns each conversion. Keep library types out of unrelated domain code where a
|
||||
small existing domain value is sufficient.
|
||||
|
||||
Use the current correction-set and validator-decision schemas as representative
|
||||
inputs. Reuse domain checks where practical. Select one real proposal/validator
|
||||
pair; account for prompt helpers, template variables, shared fragments, and schema
|
||||
references rather than testing only an unrelated hello-world prompt.
|
||||
|
||||
Record the following distinctions:
|
||||
|
||||
- Repair seeks output conforming to its structural contract. Success does not
|
||||
establish semantic correctness or authorize applying a correction.
|
||||
- A schema-valid decision set still requires Audita's cardinality and proposal
|
||||
identity checks. Semantic rejection is not automatically another repair request.
|
||||
- Exhausted structural repair yields failed content validation; provider failures,
|
||||
retry exhaustion, cancellation, and capacity rejection have separate outcomes.
|
||||
- Specify each outcome's intended effect on an Audita section/batch and overall run,
|
||||
including warnings, rejected work, partial domain results, and terminal failures.
|
||||
Preserve domain intent without preserving obsolete transport behavior by accident.
|
||||
- For transient retry budget T and structural repair budget R, one logical call can
|
||||
dispatch up to `(T + 1) * (R + 1)` provider attempts. For example, T=2 and R=2
|
||||
permits nine attempts; these illustrative values do not select production defaults.
|
||||
- Choose the total-operation deadline scope for a logical proposal or assessment;
|
||||
distinguish it from per-attempt timeouts and whole-process cancellation. Provider
|
||||
retry timing may exceed local backoff, so deadline behavior must be explicit.
|
||||
|
||||
Design the Audita observation record and aggregation rules. Include logical-call
|
||||
and attempt identity, role/backend correlation, queue/generation/retry timing,
|
||||
repair and retry counts, outcomes, and available usage. Preserve unknown/incomplete
|
||||
usage rather than inventing zeros or complete costs. Do not add terminal totals to
|
||||
attempt totals. Collect events safely across concurrent calls, without blocking
|
||||
callbacks on progress by the same run or persisting raw library events as a stable
|
||||
Audita schema. Transcript-bearing artifacts remain separately governed by Audita.
|
||||
|
||||
## Work Package 3: Build the Bounded Offline Prototype
|
||||
|
||||
Use ConfigKit v0.3.1 and PromptKit v0.11.0 as reviewed starting points; verify their
|
||||
published availability and APIs before pinning the prototype. Recheck the combined
|
||||
Go minimum (the reviewed PromptKit release requires Go 1.25.5). Do not depend on
|
||||
sibling checkouts, an active Go workspace, or committed local module replacements.
|
||||
|
||||
A proposed home is `examples/library-adoption/` as an explicitly experimental,
|
||||
standalone Go module, isolating prototype dependencies from the production binary.
|
||||
Confirm this placement during execution. Its README must give exact offline run
|
||||
and test commands after dependency acquisition; root `go test ./...` must not be
|
||||
claimed to cover a nested module. Keep complete proposed configuration and prompt
|
||||
fixtures with it and clearly label them as prototype-only.
|
||||
|
||||
The prototype should:
|
||||
|
||||
1. Load representative application settings with ConfigKit, using synthetic secret
|
||||
values supplied by the test environment and an application-owned version rule.
|
||||
2. Assemble shared backend registrations and proposal/validation profile selections.
|
||||
3. Supply representative transcript/proposal inputs to PromptKit definitions and
|
||||
actual response schemas using supported sources.
|
||||
4. Use the built-in HTTP generation path with a controlled local endpoint or
|
||||
transport. An injected LLM client alone cannot prove built-in HTTP retry behavior.
|
||||
5. Run proposal generation, decode and check candidates, run assessment, and perform
|
||||
the selected Audita decision checks without modifying real transcript files.
|
||||
6. Collect selected safe observations and show a compact result understandable to a
|
||||
reviewer, including repaired output and final domain acceptance/rejection.
|
||||
|
||||
No paid calls, real credentials, host environment assumptions, persistent caches,
|
||||
or general-purpose test framework are needed. Make concurrency and cancellation
|
||||
checks deterministic through controlled synchronization. Keep delays bounded and
|
||||
use library-supported controls; do not patch library internals to make tests pass.
|
||||
|
||||
## Focused Verification Matrix
|
||||
|
||||
| Scenario | Evidence required |
|
||||
| --- | --- |
|
||||
| Successful assembled path | ConfigKit-loaded selections drive both real prompt roles through PromptKit and Audita checks. |
|
||||
| Presence, overrides, and inheritance | Representative omission, explicit values, and role selection follow the proposed contract; unknown fields and unsupported versions fail clearly. |
|
||||
| Credentials | Required missing/empty references fail as intended; controlled calls receive the chosen synthetic credential; inspection and observations do not expose it. |
|
||||
| Structural repair succeeds | Invalid proposal output and invalid assessment output are repaired in their respective workflows; positive budgets are actually exercised. |
|
||||
| Repair exhausted | Repeated invalid output produces the selected Audita outcome and accurate accounting, without an unbounded outer retry. |
|
||||
| Domain rejection | Structurally valid but unacceptable correction or decision is rejected by Audita, demonstrating the semantic boundary. |
|
||||
| Transient retry and explicit zero | An eligible temporary status recovers; zero makes one attempt. Exhaustion retains a useful classified cause. |
|
||||
| Combined repair and retry | A repaired generation encountering a transient failure respects both budgets and reports distinct counts. |
|
||||
| Shared capacity | Concurrent calls from both roles, including retry/repair activity, stay within one selected backend cap; test a representative local cap of four and a finite API cap. |
|
||||
| Waiting, cancellation, and deadline | Cancellation/deadline stops pending work and retry waits; capacity exhaustion has the intended application outcome and later calls can still proceed. |
|
||||
| Failed-run observations | Completed work remains observable after later failure; unknown usage, known zero, and aggregate counts are mapped correctly. |
|
||||
|
||||
Tests should protect Audita's integration decisions and important assembled behavior.
|
||||
Rely on PromptKit's tests for exhaustive HTTP-status, backoff, and scheduler details;
|
||||
rely on ConfigKit's tests for its parser and presence machinery. Use race testing for
|
||||
new concurrent observation collection. Keep consumer checks offline and credential-free.
|
||||
|
||||
## Completion Criteria and Handoff
|
||||
|
||||
This unit is complete only when:
|
||||
|
||||
- The settings/ownership table and migration inventory are concrete enough to start
|
||||
the ConfigKit migration, including file/profile sources, CLI scope, credentials,
|
||||
defaults, finite backend limits, positive repair budgets, and deadline policy.
|
||||
- The offline prototype and focused matrix demonstrate the selected boundary with
|
||||
both libraries together, including both LLM roles and Audita domain checks.
|
||||
- Exact verification commands, results, limitations, and unresolved upstream gaps
|
||||
are recorded. Missing evidence is not reported as successful integration.
|
||||
- No adoption-critical decision remains hidden in prototype code. If a gap prevents
|
||||
settling the boundary, record it and revise the plan before declaring readiness.
|
||||
- The existing Audita production behavior remains intact in this unit. Temporary
|
||||
code is clearly isolated and has a removal/integration destination during the
|
||||
production migrations; no second supported runtime is introduced.
|
||||
- A component-level follow-up identifies code to replace, domain code to retain,
|
||||
tests to update, documentation/migration-guide work, and direct integration of
|
||||
the final configuration and PromptKit runtime.
|
||||
- Licensing/distribution follow-up for the selected dependency graph is identified
|
||||
for the later release stage; this prototype does not claim release compliance.
|
||||
- Roadmap links and examples resolve, appropriate checks pass, and commits contain
|
||||
only this unit's work. Significant settled architectural choices may be recorded
|
||||
in an ADR under the [documentation policy](../policy/documentation.md), while
|
||||
implementation status stays in the roadmaps.
|
||||
|
||||
Retain the overall adoption order unless evidence or the other planned large changes
|
||||
justifies revising it. Completing this unit enables detailed production migration;
|
||||
it does not authorize an intermediate release or require unrelated runtime fixes.
|
||||
161
docs/roadmap/remediation.md
Normal file
161
docs/roadmap/remediation.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Maintainability Audit: Findings and Recommended Changes
|
||||
|
||||
**Reviewed:** 2026-09-25, source commit `011b021`.
|
||||
**Scope:** Recent partial-decision reconciliation, validation context, prompt
|
||||
alignment, and request-local reference changes, including related existing code.
|
||||
**Status:** R27–R30 addressed; see the resolution record below. Previous findings and resolution records remain in Git
|
||||
history through `011b021`.
|
||||
|
||||
## Architecture Assessment
|
||||
|
||||
The recent changes fit Audita's existing boundaries. Proposal generation,
|
||||
validation, prompt assets, and outcome reporting already have suitable owners;
|
||||
this audit does not recommend a new common package or a broad refactor.
|
||||
|
||||
The strongest opportunities are shared prompt instructions and request-scoped
|
||||
validation context preparation. Two smaller improvements concern recovery-reason
|
||||
classification and partial-result assembly; both predate the latest commits.
|
||||
These are bounded maintenance recommendations, not new release gates or changes
|
||||
to correction policy, response acceptance, or reporting semantics.
|
||||
|
||||
## R27 — P2: Consolidate Shared Prompt Instructions
|
||||
|
||||
**Category:** Repeated behavioral-contract text; risk of future drift.
|
||||
|
||||
**Evidence:** The four [proposal prompts](../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml)
|
||||
repeat literal source-span matching, request-local identity, replacement-text,
|
||||
source-based proposal, and confidence instructions. The three active
|
||||
[validator prompts](../../internal/prompts/assets/prompts/validators.editorial_review.yaml)
|
||||
repeat candidate-reference, supplemental-context, and verdict-confidence guidance.
|
||||
Recent contract changes required coordinated edits across these assets. The
|
||||
[prompt registry](../../internal/prompts/registry.go) already provides a shared
|
||||
hardening fragment supplied through the role-bound adapter.
|
||||
|
||||
**Recommended change:** Extract genuinely common passages into embedded fragments
|
||||
under `internal/prompts/assets/shared`, following the existing fragment-supply
|
||||
approach. Keep the fragments owned by `internal/prompts` and use the existing
|
||||
PromptKit rendering path. Keep module-specific examples, replacement-policy
|
||||
instructions, protected-name exceptions, and semantic criteria in their individual
|
||||
prompt definitions. Do not merge similar-looking rules with different meanings.
|
||||
|
||||
**Acceptance:** Common instructions have one maintained source and appear in all
|
||||
intended rendered prompts. The existing module and validator distinctions remain
|
||||
intact, including glossary replace-all behavior and the spoken-word protected-name
|
||||
exception. Focused rendering checks cover fragment inclusion and representative
|
||||
module-specific guidance without requiring exact-prose snapshots or introducing
|
||||
a separate templating framework.
|
||||
|
||||
## R28 — P2: Prepare Validation Context Once Per Request
|
||||
|
||||
**Category:** Repeated computation and overlapping segment lookup.
|
||||
|
||||
**Evidence:** [BuildLLMValidationRequest](../../internal/framework/validators/llm_validators.go)
|
||||
already indexes transcript segments, but
|
||||
[validationContext](../../internal/framework/validators/llm_context.go) scans the
|
||||
transcript again for each candidate. It also calls
|
||||
[NewProtectedVocabulary](../../internal/framework/validators/protected_terms.go)
|
||||
for each candidate, rebuilding the same vocabulary and compiling its matching
|
||||
expression repeatedly. This is duplicated preparation rather than copied code.
|
||||
|
||||
**Recommended change:** Prepare a segment-position lookup and protected vocabulary
|
||||
once per validation request, then pass that prepared context into the existing
|
||||
context-selection helper. Keep the implementation within `framework/validators`.
|
||||
Continue selecting glossary evidence for each candidate individually because its
|
||||
proposed replacement can mention different terms. A process-wide cache or new
|
||||
context-management package is unnecessary for this cleanup.
|
||||
|
||||
**Acceptance:** Candidates in one validation request reuse the prepared lookup and
|
||||
vocabulary. Context still consists of the available immediate neighbors within the
|
||||
current chunk and relevant glossary entries in glossary order. Candidate-specific
|
||||
replacement text still participates in glossary selection. Existing optional-context
|
||||
trimming, token estimation, and oversized-input handling retain their behavior.
|
||||
Verify these outcomes with focused behavioral tests rather than assertions about
|
||||
private helper call counts or unmeasured performance guarantees.
|
||||
|
||||
## R29 — P3: Share Execution-Recovery Classification
|
||||
|
||||
**Category:** Duplicate reporting rule; predates the latest commits.
|
||||
|
||||
**Evidence:** [Proposal generation](../../internal/framework/proposal_generation/generate.go)
|
||||
and [LLM validation](../../internal/framework/validators/llm_validators.go)
|
||||
independently translate structural-repair and transient-retry counts into
|
||||
`output_repaired` and `service_recovered` reasons. The conversion is the same in
|
||||
both locations; the surrounding success and domain-outcome decisions differ.
|
||||
|
||||
**Recommended change:** Place a small conversion helper in the existing
|
||||
`framework/outcomes` package and use it from both callers. Keep the decision to
|
||||
record recoveries at the call sites, along with domain-specific reasons such as
|
||||
filtered validator decisions. Do not combine proposal and validation orchestration
|
||||
or move domain-success classification into this helper.
|
||||
|
||||
**Acceptance:** Both callers use the same execution-count conversion. Existing
|
||||
reason ordering, successful-operation reporting, and domain-specific additions
|
||||
remain intact. A partially unavailable validation batch continues to record its
|
||||
failure and execution counts without acquiring recovery records merely because a
|
||||
repair or retry occurred. Existing outcome tests should continue to protect these
|
||||
distinctions.
|
||||
|
||||
## R30 — P3: Centralize Partial Validator Result Assembly
|
||||
|
||||
**Category:** Repeated early-return construction; predates the latest commits.
|
||||
|
||||
**Evidence:** Several early returns in
|
||||
[LLMBackedValidator.Validate](../../internal/framework/validators/llm_validators.go)
|
||||
repeat nested slice appends to combine immediate decisions, oversized-input
|
||||
decisions, and decisions from completed model batches, together with accumulated
|
||||
operation outcomes. This makes it easier for a future failure path to omit facts
|
||||
already collected by the validator.
|
||||
|
||||
**Recommended change:** Use a small local result-building helper to assemble the
|
||||
partial result consistently. Keep error handling, batching, success-path ordering,
|
||||
and cardinality enforcement explicit. A general-purpose result-aggregation package
|
||||
is not needed.
|
||||
|
||||
**Acceptance:** Cancellation and failure paths retain the same previously collected
|
||||
decisions and operations. Result ordering and cardinality behavior remain unchanged;
|
||||
the helper does not enforce complete cardinality on interrupted work. Retain the
|
||||
existing partial-progress and fatal-failure coverage, adding tests only for a
|
||||
material gap exposed by the refactor.
|
||||
|
||||
## Boundaries to Preserve
|
||||
|
||||
- Keep proposal and validator reference mapping separate. Proposals can target the
|
||||
same segment repeatedly; duplicate validator decisions make that candidate
|
||||
unavailable. Similar indexing arithmetic alone does not justify a generic
|
||||
reference-management abstraction.
|
||||
- Keep distinct protected-term policy checks separate. They already share the
|
||||
vocabulary implementation and intentionally apply different policies.
|
||||
- Preserve exact source-span matching, request-local references, batching, and
|
||||
partial-success behavior. Consolidation is not a reason to strengthen contracts
|
||||
or broaden the accepted correction scope.
|
||||
|
||||
## Review Verification and Limits
|
||||
|
||||
- This is a source-level maintainability audit, not a finding of demonstrated
|
||||
production failures or a measured performance assessment.
|
||||
- No application tests were rerun for the original read-only audit; its roadmap
|
||||
update used documentation link and whitespace checks. Implementation validation
|
||||
is recorded below.
|
||||
- Future remediation should run focused behavioral tests and the broader suite
|
||||
appropriate to the actual change, following the testing policy. No live-provider
|
||||
evaluation or new evaluation framework is required by these recommendations.
|
||||
- The findings above describe the audited baseline. Implementation and validation
|
||||
are recorded below; no release publication is included.
|
||||
|
||||
## Resolution Record
|
||||
|
||||
Implemented as focused changes within the existing packages. Proposal and validator
|
||||
reference mapping and distinct protected-term policies remain separate.
|
||||
|
||||
| Finding | Resolution and evidence |
|
||||
| --- | --- |
|
||||
| R27 | Common proposal and validation instructions live in two embedded shared fragments, supplied through the existing role-bound PromptKit adapter. Individual YAML definitions retain policy differences and examples. Rendering tests check fragment inclusion once per applicable prompt and representative module-specific guidance. |
|
||||
| R28 | Validation request construction builds one segment-position lookup and protected vocabulary, reused by context selection for every candidate. A new test verifies that two replacements of the same source span receive their own relevant glossary entries. Existing neighbor, glossary-order, chunk-boundary, and budget tests pass. |
|
||||
| R29 | Both proposal generation and validation use `outcomes.ExecutionRecoveries`. Call sites still control success and domain-specific recovery reasons. Existing tests cover combined recoveries and failed validation with repair counts but no recovery record. |
|
||||
| R30 | A local result builder combines collected decisions and operation outcomes for partial returns and completed validation. Sorting and full cardinality checks remain outside it. Existing fatal-batch, partial-progress, runner, and reporting tests pass. |
|
||||
|
||||
Validation: `go test ./...` passed, including PromptKit preparation, partial-decision
|
||||
reconciliation, subprocess reports, and fatal-progress coverage. Documentation
|
||||
links and whitespace checks passed. No live model evaluation or performance
|
||||
measurement was performed; the changes make no new model-quality or latency
|
||||
promise.
|
||||
290
docs/roadmap/resilience.md
Normal file
290
docs/roadmap/resilience.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Resilient Processing and Focused Run Reports
|
||||
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Implemented. This document records the selected behavior and user
|
||||
intent. Concrete design decisions and completion evidence are in
|
||||
[historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/a83a518/docs/roadmap/implementation.md).
|
||||
|
||||
## Purpose and User Intent
|
||||
|
||||
Audita should reliably produce a useful transcript even when its LLM services
|
||||
cannot improve it. Exhausted retries or unusable model responses must not turn
|
||||
an otherwise processable transcript into a failed production run.
|
||||
|
||||
> Failure of every LLM operation across every stage and chunk still produces a
|
||||
> successful output containing the deterministically normalized transcript, with
|
||||
> no LLM corrections applied. Partial success preserves as many fully validated,
|
||||
> applicable corrections as possible.
|
||||
|
||||
Deterministic normalization remains intentional and may change segment layout,
|
||||
identifiers, and text before LLM work begins. Output uses the selected transcript
|
||||
encoder; byte-for-byte preservation of the original input and mapping corrections
|
||||
back to pre-normalization segmentation are not requirements.
|
||||
|
||||
The overriding goal is to minimize how much useful processing is lost because of
|
||||
one problematic transcript segment. A stage/chunk work unit is an attribution
|
||||
boundary, not an all-or-nothing transaction.
|
||||
|
||||
Existing reports contain useful information, but there are no compatibility
|
||||
obligations for reporting or debug output. Reuse existing machinery only when it
|
||||
makes the implementation simpler and clearer; preservation is not itself a goal.
|
||||
Internal structures and measurements must serve Audita's processing workflow.
|
||||
The reporting package adapts those outcomes to structured and human-facing
|
||||
presentations, rather than reshaping processing around a reporting convenience.
|
||||
|
||||
## Historical Baseline and Implemented Change
|
||||
|
||||
The [pipeline](../internal/pipeline.md), [LLM integration](../integrations/openai-compatible-llm.md),
|
||||
[report assembly](../internal/diagnostics-reporting.md), and
|
||||
[artifact lifecycle](../operations.md) describe current behavior.
|
||||
|
||||
Before this work, exhausted structural repairs could become warnings and skipped
|
||||
proposals or rejected validation batches. Other model runtime failures canceled
|
||||
peer work, stopped subsequent modules, and prevented final transcript output.
|
||||
Successful repairs appeared in execution observations rather than a dedicated
|
||||
recovery classification. Detailed artifacts were written during ordinary runs
|
||||
and could later be removed by retention. The working transcript was re-chunked
|
||||
between modules.
|
||||
|
||||
This feature changed those behaviors. It builds on the completed
|
||||
[library adoption](library-adoption.md) and [review remediation](remediation.md).
|
||||
Fatal cancellation remains necessary; unsuccessful model work must be separated
|
||||
from failures that actually require cancellation of the run.
|
||||
|
||||
## Fixed Chunks and Ordered Stages
|
||||
|
||||
Normalize first, then partition the transcript once. Preserve chunk boundaries,
|
||||
segment membership, segment identity, and ordering throughout LLM processing.
|
||||
Text evolves as accepted corrections are applied. Recombine chunks in original
|
||||
order after the LLM stages finish.
|
||||
|
||||
A stage is a configured module instance. Repeated instances of the same module
|
||||
have distinct identities. Proposal generation and each validator are operations
|
||||
within a stage; validation may use multiple batches within a chunk. A work unit
|
||||
is one stage instance applied to one chunk.
|
||||
|
||||
Preserve the overall stage sequence: finish a stage across its chunks before
|
||||
advancing to the next stage. Concurrent chunk work shares PromptKit's backend
|
||||
capacity. Failure in one unit does not prevent other chunks or later stages
|
||||
from executing.
|
||||
|
||||
Retain operator control over chunk size and the current defaults. Production
|
||||
experience indicates that chunk size affects model latency nonlinearly and that
|
||||
useful settings are well below backend context limits. Tuning is an empirical
|
||||
operator decision. Do not add automatic headroom calculations, cross-profile
|
||||
sizing, adaptive repartitioning, or automatic tuning in this feature. A later
|
||||
operation that cannot process its input follows the same safe error handling;
|
||||
it does not silently alter chunk boundaries. Validation batching within a chunk
|
||||
remains supported.
|
||||
|
||||
Independent whole-pipeline traversal by chunks and domain-aware response repair
|
||||
are explicitly deferred in [future work](future.md). Read-only neighboring
|
||||
context, cross-chunk edits, and transcript-wide restructuring are outside this
|
||||
unit, without becoming permanent prohibitions on future capabilities.
|
||||
|
||||
## Partial Success and Failure Isolation
|
||||
|
||||
- Keep bounded transport retries, structural repairs, operation deadlines, and
|
||||
shared backend capacity. Completion comes from safe fallback, not unlimited
|
||||
attempts or an outer retry loop around PromptKit.
|
||||
- Apply a correction only after it passes every required validator and the
|
||||
application checks. Failed validation must never be treated as approval.
|
||||
- If a validation batch fails, exclude its affected candidates while allowing
|
||||
other candidates to continue through the remaining validators. Preserve fully
|
||||
validated, applicable corrections from successful batches in the same unit.
|
||||
- Track candidates by stable proposal identity. Batch membership can change
|
||||
between validators; equal batch numbers do not establish candidate identity.
|
||||
- A failed proposal-generation operation leaves its chunk unchanged for that
|
||||
stage because there are no reliable proposals to salvage. Failures later in
|
||||
validation need not discard unrelated successful candidates.
|
||||
- Preserve earlier stages' corrections and other chunks' successful work.
|
||||
Record partial success explicitly; a unit can apply corrections and have errors.
|
||||
- Continue despite model-operation failures, including exhausted repairs/retries,
|
||||
unusable model decisions, provider/transport failures, local operation deadlines,
|
||||
and admission failures. An operation need not exhaust its attempt budget to fail.
|
||||
- A valid empty proposal list or valid assessments rejecting all corrections is
|
||||
successful processing. Correction count alone does not determine failure.
|
||||
- Keep safe filtering of defective proposals. Do not retry ordinary editorial
|
||||
rejections. For now, an unusable validator decision set excludes the affected
|
||||
candidates and records an error after existing permitted recovery. New bounded
|
||||
feedback/regeneration for domain-invalid responses belongs to future work.
|
||||
- Retain fatal treatment for invalid input/configuration, internal defects,
|
||||
explicit run cancellation, and inability to produce/write the transcript.
|
||||
Caller cancellation is distinct from a local model-operation timeout.
|
||||
- Completed runs with model errors return success. Required report-write failures
|
||||
are an additional fatal output condition when reporting is enabled, as below.
|
||||
|
||||
PromptKit owns transport, transient retries, structural repair, and admission.
|
||||
Audita owns domain acceptance, partial outcomes, fallback, and classification.
|
||||
ConfigKit owns configuration resolution. Debug and reporting choices must not
|
||||
change correction decisions, retry budgets, or continuation.
|
||||
|
||||
## Errors, Recoveries, and Debug
|
||||
|
||||
Classify by eventual outcome and impact. These are three reporting categories;
|
||||
debug is evidence, not a failure severity. Whether execution continues is a
|
||||
separate property.
|
||||
|
||||
| Category | Meaning | Representative cases |
|
||||
| --- | --- | --- |
|
||||
| Errors | Intended work could not complete reliably, or a fatal run-level operation failed. Affected candidates are excluded; other corrections can succeed. | Exhausted repair; unavailable provider; unusable validation decisions; operation deadline. |
|
||||
| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient service failure recovered; defective proposals safely discarded. |
|
||||
| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompts/responses, individual attempts, execution traces. |
|
||||
|
||||
Normal editorial disagreement belongs in debug. A validator correctly rejecting
|
||||
a meaning-changing edit is expected behavior. A proposal referring to nonexistent
|
||||
original text is defective; safely filtering it is a recovery. An unresolved
|
||||
missing/duplicate decision-index response is an error, not a recovery.
|
||||
|
||||
Record an operation's eventual outcome once. Several failed attempts followed by
|
||||
success produce a consolidated recovery. Attempts ending in failure contribute
|
||||
to error details, not additional recoveries. Earlier operations that recovered
|
||||
remain recorded if a later operation introduces an error in the same unit.
|
||||
|
||||
Separate output-quality recovery, defective-proposal filtering, and service
|
||||
recovery with structured reasons. Network instability must not look like a prompt
|
||||
defect. Do not introduce an arbitrary warning threshold or a fourth category.
|
||||
|
||||
### Counting and Processing Volume
|
||||
|
||||
- Count error-bearing work units once per stage/chunk, consolidating underlying
|
||||
operation failures as details. A partially successful unit still counts once.
|
||||
- Count recoveries once per recovered operation. Aggregate defective-proposal
|
||||
filtering per proposal response, retaining affected-proposal counts as detail.
|
||||
Attempts and individual discarded proposals are not independent incidents.
|
||||
- Keep work-unit coverage separate from operation/recovery counts. Show processing
|
||||
volume with explicitly labeled denominators; do not divide recovered-operation
|
||||
counts by work-unit counts and call that a recovery rate.
|
||||
- Report fatal run-level errors separately from the stage/chunk grid when they
|
||||
cannot be attributed to a work unit. Do not invent a chunk to accommodate them.
|
||||
- Derive YAML records, Markdown summaries, and CLI counts from actual processing
|
||||
outcomes with deterministic ordering, rather than independently calculating them.
|
||||
|
||||
## Production Outputs, Placement, and Lifecycle
|
||||
|
||||
Reporting is enabled by default. The ordinary bundle is:
|
||||
|
||||
- `errors.yml`: structured error-bearing units, their underlying failures, and
|
||||
applicable run-level failures.
|
||||
- `recoveries.yml`: structured recovered operations and pertinent counts.
|
||||
- `report.md`: a concise human-facing summary of outcome, corrections, processing
|
||||
volume, errors, recoveries, and their locations in the pipeline.
|
||||
|
||||
Write all three documents, including empty event lists for uneventful completed
|
||||
runs. Use Audita-owned schemas with safe explanations and stable reason codes;
|
||||
retain run, stage-instance, chunk, operation, and validator/batch identities where
|
||||
applicable. The implementation plan specifies the versioned schemas; neither old
|
||||
report formats nor raw library structures define the new output contract.
|
||||
|
||||
Place reports beside the output transcript. Retain transcript-to-stdout support
|
||||
with an explicit report-directory selection, defaulting to the invocation working
|
||||
directory. Preserve the existing overwrite convention: a new run can replace files
|
||||
at its selected destinations. Do not add a mandatory fresh-directory restriction
|
||||
or compatibility exports.
|
||||
|
||||
Before model work, create initial versions of all three reports carrying a common
|
||||
run identifier and explicit `in_progress` status (rendered as prose in Markdown).
|
||||
Fail immediately if initialization fails. Also check the transcript destination;
|
||||
creating neighboring reports does not prove that an existing transcript file is
|
||||
writable. Preflight must not prematurely destroy an existing transcript.
|
||||
|
||||
Finalize all required outputs and check every final write. Initial write success
|
||||
cannot guarantee later success if space or permissions change. Failure to write
|
||||
the transcript or finalize a required report returns nonzero and must not erase
|
||||
an already produced transcript. Where possible, finalize available reports with
|
||||
an explicit failed status on fatal termination. Abrupt termination may leave
|
||||
`in_progress`; elaborate crash-recovery machinery is not required.
|
||||
|
||||
Provide an option to disable reporting entirely. It suppresses all three files
|
||||
(`errors.yml`, `recoveries.yml`, and `report.md`), including initialization and
|
||||
final writes, while retaining the brief CLI summary and independently selectable
|
||||
debug output. Disabled reporting must not create or require a writable report
|
||||
destination, and must not remove reports from earlier runs.
|
||||
Transcript output failures remain fatal regardless of reporting selection.
|
||||
|
||||
Ordinary reports survive independently of debug cleanup. Reporting configuration
|
||||
and finalization must not depend on retaining a debug run directory.
|
||||
|
||||
## Debug and Internal Boundaries
|
||||
|
||||
Production issue records are compact. Full responses, transcript excerpts,
|
||||
correction lists, and complete validation decisions are opt-in debug evidence.
|
||||
Use a single ConfigKit-backed debug boolean with a `--debug` CLI override. Debug
|
||||
artifacts may use JSON where useful. Preserve configured-secret redaction in every
|
||||
presentation.
|
||||
|
||||
Debug-off prevents detailed artifact persistence; writing and later deleting
|
||||
those files is insufficient. In-memory domain state needed to process candidates
|
||||
correctly is not debug output. Debug and ordinary reporting are independent
|
||||
selections and do not alter processing behavior.
|
||||
|
||||
There is no obligation to retain `--report-json`, old artifact layouts, parallel
|
||||
exports, or old retention settings. Reuse code only where useful to the new
|
||||
workflow. Enabled debug output is retained on success or failure; disabled debug
|
||||
creates no debug directory. This supersedes the retention question in the
|
||||
[runtime follow-up](runtime-behavior.md). The legacy `never` behavior is not the
|
||||
meaning of the new reporting-disable option. Optional debug-write failures produce
|
||||
a safe stderr diagnostic without changing correction decisions or the run exit
|
||||
status; ordinary report and transcript-write failures retain their fatal semantics.
|
||||
|
||||
## Deterministic Human-Facing Presentation
|
||||
|
||||
Use the CLI style:
|
||||
|
||||
> Completed: 18 corrections applied; 3 work units with errors; 7 recoveries.
|
||||
|
||||
Do not say that all error-bearing units were skipped: some apply valid corrections.
|
||||
Include report locations when reporting is enabled and the files exist. Send the
|
||||
summary to stderr so stdout remains exclusively available for transcript output.
|
||||
Disabled reporting does not suppress the summary or fatal-error messages.
|
||||
|
||||
Keep `report.md` concise and high-level. Use separate error and recovery matrices
|
||||
with stages as columns and chunks as rows. Include every chunk in fixed order;
|
||||
retain configured stage order and distinct labels for repeated module instances.
|
||||
No automatic width detection, thresholds, adaptive omission, or table splitting.
|
||||
A wide or long table is acceptable.
|
||||
|
||||
The error matrix counts error-bearing work units: each attempted cell is 0 or 1.
|
||||
Recovery cells count recovered operations and can exceed 1. An error cell may
|
||||
coexist with recoveries and successfully applied corrections in the same unit.
|
||||
Distinguish completed work with zero events from unattempted work; do not portray
|
||||
canceled/unavailable processing as clean completion. Include totals, processing
|
||||
volume, and a short reason breakdown. Detailed inspection belongs in YAML or debug.
|
||||
|
||||
## Target Acceptance Criteria
|
||||
|
||||
- Total LLM failure still returns success and the encoded normalized transcript,
|
||||
provided required output writes succeed.
|
||||
- Mixed-success runs preserve independently validated, applicable corrections
|
||||
within the same stage/chunk as a failed batch. No candidate bypasses a required
|
||||
validator; later validators handle survivors by stable proposal identity.
|
||||
- Subsequent stages and unaffected chunks execute after recoverable failures.
|
||||
Fatal errors and explicit cancellation stop and join active work appropriately.
|
||||
- Chunk boundaries and identity remain fixed, with operator-selected sizing and
|
||||
unchanged defaults; final recombination preserves ordering.
|
||||
- Valid no-change/editorial-rejection outcomes are successful. Repaired/filtered
|
||||
output and unresolved failures receive distinct classifications and counts.
|
||||
- YAML, Markdown, and CLI agree on partial outcomes and totals; error matrices
|
||||
count affected work units, recovery matrices count recovered operations.
|
||||
- With reporting enabled, all three reports are initialized before model work,
|
||||
finalized and retained, including empty lists. Initial or final required-output
|
||||
write failures produce nonzero exits without destroying completed transcript output.
|
||||
- Reporting can be disabled without requiring its destination. Debug can be enabled
|
||||
independently; neither selection changes processing outcomes. Debug-off writes
|
||||
no detailed debug artifacts, and stdout transcript output remains supported.
|
||||
- Reports use the fixed deterministic layout without adaptive presentation logic.
|
||||
- Meaningful controlled-provider tests cover exhausted recovery, failed validation
|
||||
batches with successful survivors, total failure, fatal cancellation, output
|
||||
failures, and reporting/debug selections. Preserve domain safeguards, capacity
|
||||
limits, and redaction.
|
||||
- Canonical CLI, configuration, operations, integration, internal, and architecture
|
||||
documentation describes the implemented final behavior.
|
||||
|
||||
## Implementation Reference
|
||||
|
||||
The product decisions are settled. The [historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/a83a518/docs/roadmap/implementation.md) defines
|
||||
the selected settings and precedence, YAML schemas, counts and ordering, interrupted
|
||||
work presentation, destination checks, and output finalization. It also owns the
|
||||
ordered implementation stages and validation work. This document remains the
|
||||
reference for the desired behavior and its acceptance criteria.
|
||||
@@ -1,15 +1,16 @@
|
||||
# Runtime Behavior Decisions
|
||||
|
||||
**Last updated:** 2026-09-23
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
The documentation migration identified these existing behaviors for separate
|
||||
product consideration. No behavior change is approved or implemented by this list.
|
||||
The documentation migration identified existing behaviors for separate product
|
||||
consideration. The resilience implementation resolved the diagnostics-retention
|
||||
question; the input-timestamp question remains open.
|
||||
|
||||
- Decide whether the `never` retention selection should delete successful-run
|
||||
diagnostics; current behavior is documented in
|
||||
[operations](../operations.md#retention-behavior).
|
||||
- Decide whether an explicit zero retry count should disable retries; current
|
||||
adapter behavior is documented in [configuration](../config.md).
|
||||
- Resolved: legacy `auto`/`always`/`never` retention settings were removed.
|
||||
Debug is explicitly enabled with `diagnostics.debug` or `--debug`, creates no
|
||||
directory when disabled, and retains its directory on every outcome when enabled.
|
||||
- Resolved: [configuration](../config.md) now treats explicit zero as disabling
|
||||
transient retries or structural repair independently; PromptKit performs both.
|
||||
- Decide whether missing or null input timestamps should be rejected; current
|
||||
decoding is documented in the [input contract](../integrations/transcript-glossary-files.md).
|
||||
|
||||
|
||||
203
docs/roadmap/span-deletion.md
Normal file
203
docs/roadmap/span-deletion.md
Normal file
@@ -0,0 +1,203 @@
|
||||
# Consistent Span Deletion
|
||||
|
||||
**Last updated:** 2026-09-25
|
||||
|
||||
**Status:** Implemented. This document owns the desired behavior and user intent.
|
||||
The [historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md)
|
||||
records the concrete steps, selected implementation details, and completion
|
||||
evidence.
|
||||
|
||||
## Purpose and User Intent
|
||||
|
||||
Align proposal prompts, the LLM response schema, internal validation, and
|
||||
application behavior so legitimate text-span deletions can be considered and
|
||||
applied through the ordinary correction workflow. An empty replacement should
|
||||
not trigger structural repair merely because it is empty, nor invalidate other
|
||||
usable corrections returned alongside it.
|
||||
|
||||
This is a consistency fix within Audita's current editing scope. It does not
|
||||
authorize whole-segment removal, broaden module responsibilities, or introduce
|
||||
new recovery machinery. The accepted [resilience behavior](resilience.md) remains
|
||||
in effect, including partial success, fixed chunks, and validation before mutation.
|
||||
|
||||
## Historical Baseline (`0b6cd98`)
|
||||
|
||||
The source reviewed at `0b6cd98` contains a mismatch:
|
||||
|
||||
- The [spoken-word prompt](../../internal/prompts/assets/prompts/modules.spoken_word.proposal.yaml)
|
||||
permits removing fillers and asks for the exact affected span and replacement,
|
||||
without explaining how to represent deletion. Grammar permits punctuation and
|
||||
spacing changes, which can also naturally involve deletion.
|
||||
- The shared [correction response schema](../../internal/framework/responseschema/assets/correction_set.v1.json)
|
||||
requires `corrected_text` to have at least one character. Every built-in proposal
|
||||
module uses that schema. An empty replacement invalidates the response before
|
||||
Audita evaluates individual proposals.
|
||||
- PromptKit performs bounded structural repair. If repair does not produce a
|
||||
valid response, the proposal operation fails and yields no candidates, including
|
||||
otherwise valid proposals in the same response. The current resilience path
|
||||
records the error and continues other work.
|
||||
- [CorrectionProposal](../../internal/framework/proposals/proposal.go) already
|
||||
allows empty replacement text. The
|
||||
[preview/application logic](../../internal/framework/proposals/preview.go)
|
||||
supports literal span deletion when the resulting segment remains nonblank.
|
||||
- Domain shape checks currently reject whitespace-only original spans, although
|
||||
the response schema permits them and grammar prompts permit spacing cleanup.
|
||||
This is another restriction to align with the target contract below.
|
||||
- The validator named `non_empty_corrected_text` actually checks the resulting
|
||||
segment, not the replacement string. Both this check and application prevent
|
||||
a segment from becoming empty or whitespace-only. Application changes segment
|
||||
text; it does not remove segment records.
|
||||
|
||||
A temporary controlled-provider review reproduced whole-response rejection for
|
||||
an empty replacement alongside a valid correction, and immediate schema acceptance
|
||||
for a space-only replacement. Existing proposal tests confirm internal support for
|
||||
partial deletion and rejection of blank resulting segments. These observations
|
||||
explain a plausible production failure mechanism; they do not establish the cause
|
||||
of any particular historical incident without its payload and running version.
|
||||
|
||||
## Target Correction Contract
|
||||
|
||||
- `original_text` identifies an existing, nonempty literal span in the current
|
||||
target segment. Whitespace-only spans are eligible; an empty string identifies
|
||||
no source span and remains invalid. Matching and ambiguity checks still apply
|
||||
under the module's replacement policy: `require_unique` for grammar, spoken-word,
|
||||
and homophones; `replace_all` for glossary.
|
||||
- `corrected_text` remains a required string. An empty string means delete the
|
||||
identified span; it does not mean remove the segment record or skip validation.
|
||||
Missing fields and non-string values remain structurally invalid.
|
||||
- After applying a correction, the segment must contain non-whitespace text.
|
||||
Enforce this using the source segment and replacement result in Audita, not a
|
||||
minimum replacement length in the structural schema.
|
||||
- Every accepted deletion passes the same required validators as other edits in
|
||||
its module, including confidence, protected terms, editorial review, and meaning
|
||||
preservation where applicable. Empty replacement is neither automatic approval
|
||||
nor an automatic rejection.
|
||||
- Application preserves segment identity, speaker, timing, membership, ordering,
|
||||
and fixed chunk boundaries. Recheck applicability against text produced by
|
||||
earlier accepted corrections, retaining existing conflict handling.
|
||||
- Preserve literal replacement behavior. Prompts should select spans and
|
||||
replacements that handle adjacent whitespace and punctuation appropriately;
|
||||
this feature does not add automatic trimming or a second normalization pass.
|
||||
Neither side of a correction is rejected solely for containing only whitespace.
|
||||
For example, reducing a uniquely identified run of two spaces to one space may
|
||||
be appropriate; validity still depends on the resulting text and domain checks.
|
||||
|
||||
Preserve the intended safeguards and outcomes, not every detail of the current
|
||||
implementation. Validator ordering, internal structure, and payload construction
|
||||
may change when needed to achieve this consistent contract. Existing behavior or
|
||||
a passing test does not justify retaining a conflicting restriction.
|
||||
|
||||
Illustrative cases:
|
||||
|
||||
| Segment | Original span | Replacement | Intended handling |
|
||||
| --- | --- | --- | --- |
|
||||
| `uh hello` | `uh ` | empty string | Structurally valid; may produce `hello` after required validation. |
|
||||
| `uh hello` | `uh hello` | `hello` | Remains a valid representation of the same cleanup, subject to validation. |
|
||||
| `hello world` | two spaces | one space | Eligible spacing correction, subject to unique matching and required validation. |
|
||||
| `uh` | `uh` | empty string | Structurally valid proposal, then filtered because the resulting segment would be blank. |
|
||||
| `hello world` | `hello world` | spaces only | Filtered because the resulting segment would be blank. |
|
||||
|
||||
These examples describe eligibility and mechanical results, not unconditional
|
||||
editorial approval. An unchanged segment or valid empty correction list remains
|
||||
a successful no-change outcome.
|
||||
|
||||
## Prompt and Module Alignment
|
||||
|
||||
Review relevant prompt definitions for consistency and edit only those needing
|
||||
clarification. Where needed, explain empty-string deletion, whitespace-only spans,
|
||||
and the resulting-segment limit with a concise, module-appropriate example. The
|
||||
model should omit a correction that would leave a blank segment rather than invent
|
||||
padding or replacement content. Consistent guidance is the goal; adding deletion
|
||||
language to every prompt is not a requirement.
|
||||
|
||||
Retain each module's existing scope:
|
||||
|
||||
- Spoken-word cleanup can propose appropriate filler or dysfluency deletion while
|
||||
preserving substantive content and intentional emphasis.
|
||||
- Grammar can propose permitted punctuation or spacing deletion; allowing an
|
||||
empty replacement does not authorize word removal or spoken-word cleanup.
|
||||
- Glossary and homophone modules retain their existing correction purposes.
|
||||
Shared deletion syntax does not grant them new editorial responsibilities.
|
||||
|
||||
Review the active LLM-validator prompts and their input construction as well as
|
||||
proposal prompts. Validators must understand the before/after effect of deletion
|
||||
and assess it under their existing criteria. A correct negative editorial decision
|
||||
remains ordinary processing, not a defect that should be regenerated until approved.
|
||||
|
||||
The resulting-segment validator's name and documented identity should describe
|
||||
that responsibility accurately, rather than imply that the replacement string
|
||||
itself must be nonempty. Preserve the existing blank-result failure reason and
|
||||
safeguard while making that deliberate identity change; current chain position
|
||||
and preview implementation are not independent product requirements.
|
||||
|
||||
## Outcomes and Reporting
|
||||
|
||||
A legitimate deletion is an ordinary correction. Empty replacement alone must
|
||||
produce neither structural repair nor a recovery classification. If another
|
||||
problem required successful repair or filtering, retain that actual recovery.
|
||||
|
||||
A schema-valid proposal that would blank its stage-input segment is a defective
|
||||
proposal handled through existing safe filtering. Preserve the
|
||||
`empty_resulting_segment` reason
|
||||
and consolidated `proposals_filtered` recovery behavior; other usable proposals
|
||||
in the response continue through required validation. Ordinary editorial rejection
|
||||
remains debug evidence rather than a model error or filtering recovery.
|
||||
|
||||
A stage can propose several edits to one segment in the same response. Each edit
|
||||
is evaluated against the stage input, then approved edits are applied sequentially.
|
||||
If a proposal was applicable to that input but an earlier accepted edit makes it
|
||||
unsafe, skip it as an apply-time conflict. Do not count that conflict as a defective
|
||||
proposal, model error, or filtering recovery. For example, in `"uh um"`, deleting
|
||||
`"uh "` and deleting `"um"` each leaves text individually; after the first deletion,
|
||||
the second must be skipped to prevent a blank result. Retain the skipped decision
|
||||
in debug evidence. The existing runner already distinguishes this situation.
|
||||
|
||||
Debug evidence must accurately retain the empty replacement and the corresponding
|
||||
validation/application outcome. Ordinary reports retain their existing counts and
|
||||
purpose; no new report schema, incident category, or deletion-specific counter is
|
||||
needed. Unrelated structural defects may still exhaust PromptKit repair and produce
|
||||
an error for the operation, as today.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A controlled valid response containing an empty replacement reaches Audita's
|
||||
domain validation without structural repair solely due to that empty string.
|
||||
- A permitted span deletion can pass the required validators and reach the final
|
||||
transcript while preserving segment metadata and fixed chunk membership.
|
||||
- A mixed response can apply an approved deletion and another approved correction;
|
||||
an empty replacement does not invalidate the whole response.
|
||||
- A proposal that would blank its stage-input segment is filtered independently,
|
||||
retains the existing reason, and does not prevent unrelated valid candidates
|
||||
from succeeding. A proposal made unsafe only by an earlier accepted edit is
|
||||
skipped without adding an error or filtering recovery; a blank result is still
|
||||
prevented and the conflict remains visible in debug evidence.
|
||||
- A uniquely matching whitespace-only original span can reach validation and
|
||||
application. Empty originals remain invalid, ambiguous matches remain subject
|
||||
to module policy, and whitespace-only resulting segments remain disallowed.
|
||||
- Protected-term and meaning-changing deletions remain subject to their normal
|
||||
rejection rules. No candidate bypasses required validation.
|
||||
- A deletion needing no recovery counts as an ordinary applied correction; actual
|
||||
repair/filtering and editorial rejection retain their existing classifications.
|
||||
Debug evidence represents the empty replacement faithfully.
|
||||
- Schema and domain tests distinguish an empty string from missing/non-string
|
||||
replacement values and distinguish empty replacement from blank resulting text.
|
||||
- Prompt guidance and validator naming express the same contract as the code.
|
||||
Tests protect outcomes rather than exact prompt prose or incidental formatting.
|
||||
|
||||
Use deterministic, offline controlled responses for behavioral integration tests;
|
||||
no live-provider or paid evaluation is a prerequisite. Representative production
|
||||
examples can inform prompt quality without expanding this into an evaluation
|
||||
framework or guaranteeing that models always follow the instructions.
|
||||
|
||||
## Deferred Work
|
||||
|
||||
[Whole-segment removal](future.md#whole-segment-removal) is only potential future
|
||||
work and first requires deciding whether its benefit warrants implementation.
|
||||
Domain-aware regeneration and independent whole-pipeline traversal also remain
|
||||
in [future.md](future.md). This feature changes neither retry budgets nor the
|
||||
Audita/PromptKit ownership boundary and requires no upstream library feature.
|
||||
|
||||
Implementation sequencing and concrete names belong to the
|
||||
[historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/60e2fe7/docs/roadmap/implementation.md).
|
||||
The target remains consistent deletion of spans within surviving segments, with
|
||||
no new operator controls or report types.
|
||||
@@ -1,6 +1,12 @@
|
||||
version: 1
|
||||
version: 2
|
||||
backends:
|
||||
local:
|
||||
endpoint: http://127.0.0.1:11434/v1
|
||||
concurrency: 1
|
||||
llm:
|
||||
proposal_profile: local-proposal
|
||||
validation_profile: local-validation
|
||||
pipeline:
|
||||
modules: [grammar]
|
||||
output:
|
||||
schema: bare-segments
|
||||
llm:
|
||||
proposal:
|
||||
api_key_env: AUDITA_LLM_API_KEY
|
||||
|
||||
3
examples/overlays/testing.yml
Normal file
3
examples/overlays/testing.yml
Normal file
@@ -0,0 +1,3 @@
|
||||
llm:
|
||||
proposal_profile: service-proposal-testing
|
||||
validation_profile: service-validation-testing
|
||||
@@ -1,34 +1,29 @@
|
||||
version: 1
|
||||
version: 2
|
||||
backends:
|
||||
service:
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
credential: AUDITA_SERVICE_KEY
|
||||
concurrency: 2
|
||||
queue_capacity: 1024
|
||||
llm:
|
||||
proposal_profile: service-proposal
|
||||
validation_profile: service-validation
|
||||
transient_retries: 2
|
||||
repair_attempts: 2
|
||||
operation_timeout: 15m
|
||||
pipeline:
|
||||
modules: [glossary, homophones, glossary, spoken_word, grammar]
|
||||
output:
|
||||
schema: audita-v1
|
||||
llm:
|
||||
proposal:
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
model: openrouter/google/gemma-4-31b-it
|
||||
api_key_env: AUDITA_LLM_API_KEY
|
||||
timeout: 120s
|
||||
max_retries: 3
|
||||
validation:
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
model: openrouter/google/gemma-4-31b-it
|
||||
api_key_env: AUDITA_VALIDATION_LLM_API_KEY
|
||||
timeout: 120
|
||||
max_retries: 3
|
||||
concurrency:
|
||||
total_llm: 2
|
||||
proposal_llm: 2
|
||||
validation_llm: 1
|
||||
chunking:
|
||||
# Requires at least this many normalized segments; omit for automatic chunking.
|
||||
target_sections: 8
|
||||
max_section_tokens: 8192
|
||||
min_section_tokens: 2048
|
||||
normalization:
|
||||
max_segment_gap: 4s
|
||||
ellipsis_gap: 3.5s
|
||||
max_segment_duration: 60s
|
||||
max_segment_gap: 4.0
|
||||
ellipsis_gap: 3.5
|
||||
max_segment_duration: 60.0
|
||||
max_segment_tokens: 2048
|
||||
thresholds:
|
||||
glossary: 0.8
|
||||
@@ -37,6 +32,9 @@ thresholds:
|
||||
grammar: 0.8
|
||||
context:
|
||||
description: "General context for domain vocabulary and speaker names."
|
||||
reporting:
|
||||
enabled: true
|
||||
stdout_dir: .
|
||||
diagnostics:
|
||||
debug: false
|
||||
work_dir: /tmp/audita
|
||||
retention: auto
|
||||
|
||||
7
examples/profiles/local-proposal.yaml
Normal file
7
examples/profiles/local-proposal.yaml
Normal file
@@ -0,0 +1,7 @@
|
||||
id: local-proposal
|
||||
backend: local
|
||||
model: replace-with-local-model
|
||||
temperature: 0
|
||||
max_tokens: 2048
|
||||
top_p: 1
|
||||
timeout_seconds: 120
|
||||
7
examples/profiles/local-validation.yaml
Normal file
7
examples/profiles/local-validation.yaml
Normal file
@@ -0,0 +1,7 @@
|
||||
id: local-validation
|
||||
backend: local
|
||||
model: replace-with-local-model
|
||||
temperature: 0
|
||||
max_tokens: 1024
|
||||
top_p: 1
|
||||
timeout_seconds: 120
|
||||
7
examples/profiles/service-proposal-testing.yaml
Normal file
7
examples/profiles/service-proposal-testing.yaml
Normal file
@@ -0,0 +1,7 @@
|
||||
id: service-proposal-testing
|
||||
backend: service
|
||||
model: replace-with-testing-model
|
||||
temperature: 0
|
||||
max_tokens: 2048
|
||||
top_p: 1
|
||||
timeout_seconds: 120
|
||||
7
examples/profiles/service-proposal.yaml
Normal file
7
examples/profiles/service-proposal.yaml
Normal file
@@ -0,0 +1,7 @@
|
||||
id: service-proposal
|
||||
backend: service
|
||||
model: replace-with-provider-model
|
||||
temperature: 0
|
||||
max_tokens: 2048
|
||||
top_p: 1
|
||||
timeout_seconds: 120
|
||||
7
examples/profiles/service-validation-testing.yaml
Normal file
7
examples/profiles/service-validation-testing.yaml
Normal file
@@ -0,0 +1,7 @@
|
||||
id: service-validation-testing
|
||||
backend: service
|
||||
model: replace-with-testing-model
|
||||
temperature: 0
|
||||
max_tokens: 1024
|
||||
top_p: 1
|
||||
timeout_seconds: 120
|
||||
7
examples/profiles/service-validation.yaml
Normal file
7
examples/profiles/service-validation.yaml
Normal file
@@ -0,0 +1,7 @@
|
||||
id: service-validation
|
||||
backend: service
|
||||
model: replace-with-provider-model
|
||||
temperature: 0
|
||||
max_tokens: 1024
|
||||
top_p: 1
|
||||
timeout_seconds: 120
|
||||
15
go.mod
15
go.mod
@@ -1,11 +1,20 @@
|
||||
module gitea.maximumdirect.net/eric/audita
|
||||
|
||||
go 1.24.0
|
||||
|
||||
require gopkg.in/yaml.v3 v3.0.1
|
||||
go 1.25.5
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.4.0
|
||||
gitea.maximumdirect.net/eric/promptkit v0.11.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v2 v2.20260922.1 // indirect
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/v2 v2.20260922.0 // indirect
|
||||
github.com/kr/pretty v0.3.1 // indirect
|
||||
github.com/rogpeppe/go-internal v1.14.1 // indirect
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.6 // indirect
|
||||
golang.org/x/text v0.14.0 // indirect
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
|
||||
)
|
||||
|
||||
16
go.sum
16
go.sum
@@ -1,4 +1,14 @@
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.4.0 h1:fo+vKff3zTGeWqJIA38NtT84b6OUP//9ZmBH3lFuKgk=
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.4.0/go.mod h1:tYWpsrbLyu05r8+1MapWp5fN6ZrX9I8hU2xEPedYmYk=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.11.0 h1:x7JguOUiInPVXK0GEitjyoRFqqvcuSTjYUvqJ+whtik=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.11.0/go.mod h1:0zEBVlJJC+ldDdXazAgybYBXNSe74vwFx5dyOuSos/w=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v2 v2.20260922.1 h1:S2/Q/JDE+m8Ca3pDhAjVbfYrdlEhs9blj0STgNopHo0=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v2 v2.20260922.1/go.mod h1:a3VBbh9oaGXD6ruxhtwHj+AWW6DY/SSyby1QtMmYSUo=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/v2 v2.20260922.0 h1:Y0EXc3eh12eKW2NrOviVqdk3oBo6UUjopt1YAa1B3Oc=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/v2 v2.20260922.0/go.mod h1:mtgD1E0SGm0e/I8Ffhz+AYX7GN7RKUWJXehNFIKHs4Y=
|
||||
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
|
||||
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
|
||||
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
|
||||
@@ -10,6 +20,12 @@ github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsK
|
||||
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
|
||||
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
|
||||
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.6 h1:1h7H1ohdUh93/FyE4YaDa1Zh64K6VVbjF4K6WUxMtH4=
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.6/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
|
||||
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
|
||||
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||
|
||||
222
internal/cli/config_input.go
Normal file
222
internal/cli/config_input.go
Normal file
@@ -0,0 +1,222 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
ck "gitea.maximumdirect.net/eric/ConfigKit"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||
)
|
||||
|
||||
var systemConfigPaths = []string{"/usr/local/etc/audita/config.yml", "/etc/audita/config.yml"}
|
||||
|
||||
type configInput struct {
|
||||
ConfigPath string
|
||||
ConfigPathSet bool
|
||||
Overlay string
|
||||
View string
|
||||
TranscriptPath string
|
||||
GlossaryPath string
|
||||
OutputPath string
|
||||
Overrides []ck.Override
|
||||
}
|
||||
|
||||
type selectedConfig struct {
|
||||
Path string
|
||||
Source string
|
||||
Result *ck.Result[config.Config]
|
||||
}
|
||||
|
||||
// parseConfigInput owns the common setting flags and process-only path flags.
|
||||
// A transcript before flags and one after flags are both accepted.
|
||||
func parseConfigInput(command string, args []string, output io.Writer) (configInput, bool, error) {
|
||||
var input configInput
|
||||
fs := flag.NewFlagSet(command, flag.ContinueOnError)
|
||||
fs.SetOutput(output)
|
||||
configPath := fs.String("config", "", "Main configuration YAML file")
|
||||
overlay := fs.String("overlay", "", "Selected configuration overlay")
|
||||
if command == "print-effective" {
|
||||
fs.String("view", "settings", "Effective view: settings, profiles, or explain")
|
||||
}
|
||||
settingPaths := map[string]ck.Path{
|
||||
"proposal-profile": ck.Keys("llm", "proposal_profile"),
|
||||
"validation-profile": ck.Keys("llm", "validation_profile"),
|
||||
"output-schema": ck.Keys("output", "schema"),
|
||||
"validation-max-prompt-tokens": ck.Keys("validation", "max_prompt_tokens"),
|
||||
"target-sections": ck.Keys("chunking", "target_sections"),
|
||||
"max-section-tokens": ck.Keys("chunking", "max_section_tokens"),
|
||||
"min-section-tokens": ck.Keys("chunking", "min_section_tokens"),
|
||||
"normalize-max-segment-gap": ck.Keys("normalization", "max_segment_gap"),
|
||||
"normalize-ellipsis-gap": ck.Keys("normalization", "ellipsis_gap"),
|
||||
"normalize-max-segment-duration": ck.Keys("normalization", "max_segment_duration"),
|
||||
"normalize-max-segment-tokens": ck.Keys("normalization", "max_segment_tokens"),
|
||||
"glossary-confidence-threshold": ck.Keys("thresholds", "glossary"),
|
||||
"homophones-confidence-threshold": ck.Keys("thresholds", "homophones"),
|
||||
"spoken-word-confidence-threshold": ck.Keys("thresholds", "spoken_word"),
|
||||
"grammar-confidence-threshold": ck.Keys("thresholds", "grammar"),
|
||||
"transcript-description": ck.Keys("context", "description"),
|
||||
"reporting": ck.Keys("reporting", "enabled"),
|
||||
"report-dir": ck.Keys("reporting", "stdout_dir"),
|
||||
"debug": ck.Keys("diagnostics", "debug"),
|
||||
"work-dir": ck.Keys("diagnostics", "work_dir"),
|
||||
"modules": ck.Keys("pipeline", "modules"),
|
||||
}
|
||||
for name := range settingPaths {
|
||||
switch name {
|
||||
case "reporting", "debug":
|
||||
fs.Bool(name, false, "Setting override")
|
||||
case "validation-max-prompt-tokens", "target-sections", "max-section-tokens", "min-section-tokens", "normalize-max-segment-tokens":
|
||||
fs.Int(name, 0, "Setting override")
|
||||
case "normalize-max-segment-gap", "normalize-ellipsis-gap", "normalize-max-segment-duration", "glossary-confidence-threshold", "homophones-confidence-threshold", "spoken-word-confidence-threshold", "grammar-confidence-threshold":
|
||||
fs.Float64(name, 0, "Setting override")
|
||||
default:
|
||||
fs.String(name, "", "Setting override")
|
||||
}
|
||||
}
|
||||
if command == "process" {
|
||||
fs.String("glossary", "", "Glossary YAML file")
|
||||
fs.String("output", "", "Corrected transcript output file")
|
||||
}
|
||||
parseArgs := args
|
||||
front := ""
|
||||
if command == "process" && len(args) > 0 && !strings.HasPrefix(args[0], "-") {
|
||||
front, parseArgs = args[0], args[1:]
|
||||
}
|
||||
if err := fs.Parse(parseArgs); err != nil {
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
return configInput{}, true, nil
|
||||
}
|
||||
return configInput{}, false, err
|
||||
}
|
||||
positions := fs.Args()
|
||||
if front != "" {
|
||||
positions = append([]string{front}, positions...)
|
||||
}
|
||||
if command == "process" {
|
||||
if len(positions) != 1 {
|
||||
return configInput{}, false, fmt.Errorf("expected exactly one transcript JSON path")
|
||||
}
|
||||
input.TranscriptPath = positions[0]
|
||||
input.GlossaryPath = fs.Lookup("glossary").Value.String()
|
||||
input.OutputPath = fs.Lookup("output").Value.String()
|
||||
if strings.TrimSpace(input.GlossaryPath) == "" {
|
||||
return configInput{}, false, fmt.Errorf("--glossary is required")
|
||||
}
|
||||
} else if len(positions) != 0 {
|
||||
return configInput{}, false, fmt.Errorf("unexpected positional arguments")
|
||||
}
|
||||
input.ConfigPath, input.Overlay = *configPath, *overlay
|
||||
if command == "print-effective" {
|
||||
input.View = fs.Lookup("view").Value.String()
|
||||
switch input.View {
|
||||
case "settings", "profiles", "explain":
|
||||
default:
|
||||
return configInput{}, false, fmt.Errorf("unknown effective view %q", input.View)
|
||||
}
|
||||
}
|
||||
fs.Visit(func(f *flag.Flag) {
|
||||
if f.Name == "config" {
|
||||
input.ConfigPathSet = true
|
||||
}
|
||||
path, ok := settingPaths[f.Name]
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var value any = f.Value.String()
|
||||
switch f.Name {
|
||||
case "reporting", "debug":
|
||||
value, _ = strconv.ParseBool(f.Value.String())
|
||||
case "modules":
|
||||
modules, err := config.ParseModulesCSV(f.Value.String())
|
||||
if err != nil {
|
||||
value = err
|
||||
} else {
|
||||
values := make([]any, len(modules))
|
||||
for i, module := range modules {
|
||||
values[i] = module
|
||||
}
|
||||
value = values
|
||||
}
|
||||
case "validation-max-prompt-tokens", "target-sections", "max-section-tokens", "min-section-tokens", "normalize-max-segment-tokens":
|
||||
value, _ = strconv.Atoi(f.Value.String())
|
||||
case "normalize-max-segment-gap", "normalize-ellipsis-gap", "normalize-max-segment-duration", "glossary-confidence-threshold", "homophones-confidence-threshold", "spoken-word-confidence-threshold", "grammar-confidence-threshold":
|
||||
value, _ = strconv.ParseFloat(f.Value.String(), 64)
|
||||
}
|
||||
input.Overrides = append(input.Overrides, ck.Override{Path: path, Value: value, Flag: "--" + f.Name})
|
||||
})
|
||||
for _, override := range input.Overrides {
|
||||
if err, ok := override.Value.(error); ok {
|
||||
return configInput{}, false, fmt.Errorf("%s: %w", override.Flag, err)
|
||||
}
|
||||
}
|
||||
return input, false, nil
|
||||
}
|
||||
|
||||
func selectConfigPath(explicit string, set bool, stat func(string) (os.FileInfo, error), candidates []string) (string, string, error) {
|
||||
if set {
|
||||
if strings.TrimSpace(explicit) == "" {
|
||||
return "", "", fmt.Errorf("--config requires a nonempty path")
|
||||
}
|
||||
if _, err := stat(explicit); err != nil {
|
||||
return "", "", fmt.Errorf("config file %q: %w", explicit, err)
|
||||
}
|
||||
return explicit, "flag", nil
|
||||
}
|
||||
for _, path := range candidates {
|
||||
if _, err := stat(path); err == nil {
|
||||
return path, "default", nil
|
||||
} else if !os.IsNotExist(err) {
|
||||
return "", "", fmt.Errorf("config file %q: %w", path, err)
|
||||
}
|
||||
}
|
||||
return "", "", fmt.Errorf("no configuration file found; select a config with backend and profile settings")
|
||||
}
|
||||
|
||||
func loadConfigInput(input configInput, command string) (selectedConfig, error) {
|
||||
if command == "validate" && (!input.ConfigPathSet || strings.TrimSpace(input.ConfigPath) == "") {
|
||||
return selectedConfig{}, fmt.Errorf("--config is required")
|
||||
}
|
||||
path, source, err := selectConfigPath(input.ConfigPath, input.ConfigPathSet, os.Stat, systemConfigPaths)
|
||||
if err != nil {
|
||||
return selectedConfig{}, err
|
||||
}
|
||||
loaded, err := config.Load(ck.File(path), input.Overlay, input.Overrides...)
|
||||
if err != nil {
|
||||
return selectedConfig{}, err
|
||||
}
|
||||
return selectedConfig{Path: path, Source: source, Result: loaded}, nil
|
||||
}
|
||||
|
||||
func configInputError(err error) string {
|
||||
var loadErr *ck.LoadError
|
||||
if !errors.As(err, &loadErr) {
|
||||
return err.Error()
|
||||
}
|
||||
var messages []string
|
||||
for _, diagnostic := range loadErr.Diagnostics() {
|
||||
messages = append(messages, diagnostic.Message)
|
||||
}
|
||||
if len(messages) == 0 {
|
||||
return err.Error()
|
||||
}
|
||||
return strings.Join(messages, "; ")
|
||||
}
|
||||
|
||||
func selectedReportDir(cfg config.Config, outputPath string) (string, error) {
|
||||
if !cfg.Reporting.Enabled {
|
||||
return "", nil
|
||||
}
|
||||
if outputPath != "" {
|
||||
return filepath.Dir(outputPath), nil
|
||||
}
|
||||
if strings.TrimSpace(cfg.Reporting.StdoutDir) == "" {
|
||||
return "", fmt.Errorf("report directory is required when reporting is enabled and transcript output is stdout")
|
||||
}
|
||||
return cfg.Reporting.StdoutDir, nil
|
||||
}
|
||||
250
internal/cli/config_input_test.go
Normal file
250
internal/cli/config_input_test.go
Normal file
@@ -0,0 +1,250 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
ck "gitea.maximumdirect.net/eric/ConfigKit"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||
)
|
||||
|
||||
func TestSharedConfigInput(t *testing.T) {
|
||||
for _, command := range []string{"process", "validate", "print-effective"} {
|
||||
t.Run(command, func(t *testing.T) {
|
||||
args := []string{"--config", "main.yml", "--overlay", "testing", "--proposal-profile", "economy", "--modules", "", "--target-sections", "0"}
|
||||
if command == "process" {
|
||||
args = append([]string{"transcript.json", "--glossary", "glossary.yml"}, args...)
|
||||
}
|
||||
input, help, err := parseConfigInput(command, args, io.Discard)
|
||||
if err != nil || help {
|
||||
t.Fatalf("parse: %+v %v %v", input, help, err)
|
||||
}
|
||||
if !input.ConfigPathSet || input.ConfigPath != "main.yml" || input.Overlay != "testing" || len(input.Overrides) != 3 {
|
||||
t.Fatalf("selection: %+v", input)
|
||||
}
|
||||
for _, override := range input.Overrides {
|
||||
switch override.Path.String() {
|
||||
case ck.Keys("pipeline", "modules").String():
|
||||
if values, ok := override.Value.([]any); !ok || len(values) != 0 {
|
||||
t.Fatalf("empty modules: %#v", override.Value)
|
||||
}
|
||||
case ck.Keys("chunking", "target_sections").String():
|
||||
if override.Value != 0 {
|
||||
t.Fatalf("zero target: %#v", override.Value)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigInputRejectsRemovedAndProcessOnlyFlags(t *testing.T) {
|
||||
for _, command := range []string{"process", "validate", "print-effective"} {
|
||||
for _, flag := range []string{"--model", "--llm-api-key", "--max-retries", "--llm-concurrency", "--report-json", "--work-dir-retention"} {
|
||||
args := []string{flag, "value"}
|
||||
if command == "process" {
|
||||
args = append([]string{"transcript.json", "--glossary", "glossary.yml"}, args...)
|
||||
}
|
||||
if _, _, err := parseConfigInput(command, args, io.Discard); err == nil {
|
||||
t.Fatalf("%s accepted %s", command, flag)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, command := range []string{"validate", "print-effective"} {
|
||||
if _, _, err := parseConfigInput(command, []string{"--glossary", "glossary.yml"}, io.Discard); err == nil {
|
||||
t.Fatalf("%s accepted process path", command)
|
||||
}
|
||||
if _, _, err := parseConfigInput(command, []string{"transcript.json"}, io.Discard); err == nil {
|
||||
t.Fatalf("%s accepted positional", command)
|
||||
}
|
||||
}
|
||||
for _, command := range []string{"process", "validate", "print-effective"} {
|
||||
_, help, err := parseConfigInput(command, []string{"--help"}, io.Discard)
|
||||
if err != nil || !help {
|
||||
t.Fatalf("%s help: %v %v", command, help, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigSelection(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
first, second := filepath.Join(dir, "first.yml"), filepath.Join(dir, "second.yml")
|
||||
if err := os.WriteFile(second, []byte("version: 2\n"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
path, source, err := selectConfigPath("", false, os.Stat, []string{first, second})
|
||||
if err != nil || path != second || source != "default" {
|
||||
t.Fatalf("discovery: %q %q %v", path, source, err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
explicit string
|
||||
set bool
|
||||
candidates []string
|
||||
}{
|
||||
{"", true, []string{second}}, {first, true, []string{second}}, {"", false, []string{first}},
|
||||
} {
|
||||
if _, _, err := selectConfigPath(tc.explicit, tc.set, os.Stat, tc.candidates); err == nil {
|
||||
t.Fatalf("accepted unavailable selection: %+v", tc)
|
||||
}
|
||||
}
|
||||
statDenied := func(string) (os.FileInfo, error) { return nil, os.ErrPermission }
|
||||
if _, _, err := selectConfigPath("", false, statDenied, []string{first}); !errors.Is(err, os.ErrPermission) {
|
||||
t.Fatalf("inaccessible candidate: %v", err)
|
||||
}
|
||||
path, source, err = selectConfigPath(second, true, os.Stat, []string{first})
|
||||
if err != nil || path != second || source != "flag" {
|
||||
t.Fatalf("explicit path: %q %q %v", path, source, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLegacyEnvironmentDoesNotSelectConfig(t *testing.T) {
|
||||
t.Setenv("AUDITA_CONFIG", filepath.Join(t.TempDir(), "legacy.yml"))
|
||||
_, _, err := selectConfigPath("", false, os.Stat, []string{filepath.Join(t.TempDir(), "missing.yml")})
|
||||
if err == nil || !strings.Contains(err.Error(), "no configuration file") {
|
||||
t.Fatalf("legacy environment changed discovery: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEffectiveViewParsing(t *testing.T) {
|
||||
for _, view := range []string{"settings", "profiles", "explain"} {
|
||||
input, _, err := parseConfigInput("print-effective", []string{"--view", view}, io.Discard)
|
||||
if err != nil || input.View != view {
|
||||
t.Fatalf("%q: %+v %v", view, input, err)
|
||||
}
|
||||
}
|
||||
if _, _, err := parseConfigInput("print-effective", []string{"--view", "raw"}, io.Discard); err == nil {
|
||||
t.Fatal("unknown view accepted")
|
||||
}
|
||||
if _, _, err := parseConfigInput("validate", []string{"--view", "profiles"}, io.Discard); err == nil {
|
||||
t.Fatal("validate accepted inspection-only flag")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSharedLoadingUsesSelectedFileOverlayAndVisitedOverrides(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.Mkdir(filepath.Join(dir, "overlays"), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
main := filepath.Join(dir, "config.yml")
|
||||
base := "version: 2\nbackends:\n local:\n endpoint: https://local.invalid/v1\nllm:\n proposal_profile: base\nchunking:\n max_section_tokens: 900\n"
|
||||
if err := os.WriteFile(main, []byte(base), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "overlays", "testing.yml"), []byte("llm:\n proposal_profile: testing\n"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, command := range []string{"process", "validate", "print-effective"} {
|
||||
args := []string{"--config", main, "--overlay", "testing", "--min-section-tokens", "500"}
|
||||
if command == "process" {
|
||||
args = append([]string{"input.json", "--glossary", "glossary.yml"}, args...)
|
||||
}
|
||||
input, _, err := parseConfigInput(command, args, io.Discard)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
selected, err := loadConfigInput(input, command)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c := selected.Result.Config
|
||||
if c.LLM.ProposalProfile != "testing" || c.LLM.ValidationProfile != "testing" || c.Chunking.MinSectionTokens != 500 || c.Chunking.MaxSectionTokens != 900 {
|
||||
t.Fatalf("%s resolved %+v", command, c)
|
||||
}
|
||||
}
|
||||
input, _, err := parseConfigInput("validate", []string{"--overlay", "testing"}, io.Discard)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := loadConfigInput(input, "validate"); err == nil {
|
||||
t.Fatal("validate accepted discovery")
|
||||
}
|
||||
input, _, err = parseConfigInput("validate", []string{"--config", main}, io.Discard)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := loadConfigInput(input, "validate"); err == nil {
|
||||
t.Fatal("unmodified invalid lower-precedence section relation accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestReportingAndDebugOverridesRemainIndependent(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.Mkdir(filepath.Join(dir, "overlays"), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
main := filepath.Join(dir, "config.yml")
|
||||
base := "version: 2\nbackends:\n local:\n endpoint: https://local.invalid/v1\nllm:\n proposal_profile: base\n"
|
||||
if err := os.WriteFile(main, []byte(base), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "overlays", "testing.yml"), []byte("reporting:\n enabled: true\n stdout_dir: reports\ndiagnostics:\n debug: true\n"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defaultInput, _, err := parseConfigInput("validate", []string{"--config", main}, io.Discard)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defaults, err := loadConfigInput(defaultInput, "validate")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !defaults.Result.Config.Reporting.Enabled || defaults.Result.Config.Reporting.StdoutDir != "." || defaults.Result.Config.Diagnostics.Debug {
|
||||
t.Fatalf("unexpected defaults: %+v", defaults.Result.Config)
|
||||
}
|
||||
for _, command := range []string{"process", "validate", "print-effective"} {
|
||||
args := []string{"--config", main, "--overlay", "testing", "--reporting=false", "--debug=false"}
|
||||
if command == "process" {
|
||||
args = append([]string{"input.json", "--glossary", "glossary.yml"}, args...)
|
||||
}
|
||||
input, _, err := parseConfigInput(command, args, io.Discard)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(input.Overrides) != 2 || input.Overrides[0].Value != false || input.Overrides[1].Value != false {
|
||||
t.Fatalf("boolean overrides must remain typed: %+v", input.Overrides)
|
||||
}
|
||||
selected, err := loadConfigInput(input, command)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if selected.Result.Config.Reporting.Enabled || selected.Result.Config.Diagnostics.Debug || selected.Result.Config.Reporting.StdoutDir != "reports" {
|
||||
t.Fatalf("%s did not honor explicit false: %+v", command, selected.Result.Config)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestReportDirectorySelectionOnlyUsesActiveDestination(t *testing.T) {
|
||||
cfg := config.Config{Reporting: config.Reporting{Enabled: true, StdoutDir: "reports"}}
|
||||
if got, err := selectedReportDir(cfg, ""); err != nil || got != "reports" {
|
||||
t.Fatalf("stdout destination: %q %v", got, err)
|
||||
}
|
||||
cfg.Reporting.StdoutDir = ""
|
||||
if got, err := selectedReportDir(cfg, filepath.Join("out", "corrected.json")); err != nil || got != "out" {
|
||||
t.Fatalf("file destination ignored stdout setting: %q %v", got, err)
|
||||
}
|
||||
if _, err := selectedReportDir(cfg, ""); err == nil {
|
||||
t.Fatal("enabled stdout reporting accepted blank destination")
|
||||
}
|
||||
cfg.Reporting.Enabled = false
|
||||
if got, err := selectedReportDir(cfg, ""); err != nil || got != "" {
|
||||
t.Fatalf("disabled reporting probed destination: %q %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBareBooleanFlagsAndReportDirectoryOverride(t *testing.T) {
|
||||
input, _, err := parseConfigInput("process", []string{"input.json", "--glossary", "glossary.yml", "--reporting", "--debug", "--report-dir", "reports"}, io.Discard)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
values := map[string]any{}
|
||||
for _, override := range input.Overrides {
|
||||
values[override.Path.String()] = override.Value
|
||||
}
|
||||
if values[ck.Keys("reporting", "enabled").String()] != true || values[ck.Keys("diagnostics", "debug").String()] != true || values[ck.Keys("reporting", "stdout_dir").String()] != "reports" {
|
||||
t.Fatalf("incorrect visited overrides: %+v", values)
|
||||
}
|
||||
}
|
||||
@@ -1,36 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
// Tests supply their own overrides; inherited settings must not affect them.
|
||||
for _, entry := range os.Environ() {
|
||||
key, _, _ := strings.Cut(entry, "=")
|
||||
if strings.HasPrefix(key, "AUDITA_") || key == "OPENROUTER_API_KEY" {
|
||||
if err := os.Unsetenv(key); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
dir, err := os.MkdirTemp("", "audita-cli-test-config-")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
defer os.RemoveAll(dir)
|
||||
configPath := filepath.Join(dir, "config.yml")
|
||||
if err := os.WriteFile(configPath, []byte("version: 1\n"), 0600); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
// An explicit environment path prevents discovery of installed system config.
|
||||
// Individual tests can still override it with t.Setenv or --config.
|
||||
if err := os.Setenv("AUDITA_CONFIG", configPath); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
m.Run()
|
||||
}
|
||||
@@ -1,338 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/reporting"
|
||||
)
|
||||
|
||||
type parityFixtureCase struct {
|
||||
Name string `json:"name"`
|
||||
TranscriptFile string `json:"transcript_file"`
|
||||
GlossaryFile string `json:"glossary_file"`
|
||||
ModulesCSV string `json:"modules_csv,omitempty"`
|
||||
ProposalResponsesFile string `json:"proposal_responses_file,omitempty"`
|
||||
ValidationResponsesFile string `json:"validation_responses_file,omitempty"`
|
||||
Env map[string]string `json:"env,omitempty"`
|
||||
Expect parityExpectation `json:"expect"`
|
||||
}
|
||||
|
||||
type parityExpectation struct {
|
||||
ExitCode int `json:"exit_code"`
|
||||
Status string `json:"status,omitempty"`
|
||||
ErrorPhase string `json:"error_phase,omitempty"`
|
||||
StdoutMode string `json:"stdout_mode,omitempty"` // empty|json
|
||||
StderrContains string `json:"stderr_contains,omitempty"`
|
||||
OutputTranscriptFile string `json:"output_transcript_file,omitempty"`
|
||||
ModuleInstances []string `json:"module_instances,omitempty"`
|
||||
ModuleCount int `json:"module_count,omitempty"`
|
||||
TotalAppliedChanges int `json:"total_applied_changes,omitempty"`
|
||||
TotalSkippedChanges int `json:"total_skipped_changes,omitempty"`
|
||||
FailedModuleInstance string `json:"failed_module_instance,omitempty"`
|
||||
ValidatorRejectedReasonCodes []string `json:"validator_rejected_reason_codes,omitempty"`
|
||||
ApplicationSkipReasonCodes []string `json:"application_skip_reason_codes,omitempty"`
|
||||
RequireErrorLog bool `json:"require_error_log,omitempty"`
|
||||
SecretMarkers []string `json:"secret_markers,omitempty"`
|
||||
ExpectedProposalCalls []string `json:"expected_proposal_calls,omitempty"`
|
||||
ExpectedValidationCalls []string `json:"expected_validation_calls,omitempty"`
|
||||
ModuleAppliedCounts []int `json:"module_applied_counts,omitempty"`
|
||||
ModuleRejectedCounts []int `json:"module_rejected_counts,omitempty"`
|
||||
ModuleSkipCounts []int `json:"module_skip_counts,omitempty"`
|
||||
MinResponsePayloadArtifacts int `json:"min_response_payload_artifacts,omitempty"`
|
||||
}
|
||||
|
||||
func TestParityFixtures(t *testing.T) {
|
||||
casePaths, err := filepath.Glob(parityFixturePath("*.case.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("glob parity fixtures: %v", err)
|
||||
}
|
||||
if len(casePaths) == 0 {
|
||||
t.Fatal("expected at least one parity fixture case")
|
||||
}
|
||||
|
||||
for _, casePath := range casePaths {
|
||||
fx := loadParityFixtureCase(t, casePath)
|
||||
t.Run(fx.Name, func(t *testing.T) {
|
||||
runParityFixtureCase(t, filepath.Dir(casePath), fx)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func loadParityFixtureCase(t *testing.T, casePath string) parityFixtureCase {
|
||||
t.Helper()
|
||||
var fx parityFixtureCase
|
||||
raw := readFile(t, casePath)
|
||||
if err := json.Unmarshal(raw, &fx); err != nil {
|
||||
t.Fatalf("parse parity case %q: %v", casePath, err)
|
||||
}
|
||||
if strings.TrimSpace(fx.Name) == "" {
|
||||
t.Fatalf("parity case %q missing name", casePath)
|
||||
}
|
||||
return fx
|
||||
}
|
||||
|
||||
func runParityFixtureCase(t *testing.T, caseDir string, fx parityFixtureCase) {
|
||||
t.Helper()
|
||||
for k, v := range fx.Env {
|
||||
t.Setenv(k, v)
|
||||
}
|
||||
|
||||
proposalClient := &fakeStructuredLLMClient{}
|
||||
validationClient := &fakeStructuredLLMClient{}
|
||||
|
||||
if strings.TrimSpace(fx.ProposalResponsesFile) != "" {
|
||||
raw := readFile(t, filepath.Join(caseDir, fx.ProposalResponsesFile))
|
||||
if err := json.Unmarshal(raw, &proposalClient.proposalResponses); err != nil {
|
||||
t.Fatalf("parse proposal responses: %v", err)
|
||||
}
|
||||
processProposalLLMClient = proposalClient
|
||||
}
|
||||
if strings.TrimSpace(fx.ValidationResponsesFile) != "" {
|
||||
raw := readFile(t, filepath.Join(caseDir, fx.ValidationResponsesFile))
|
||||
if err := json.Unmarshal(raw, &validationClient.validationResponses); err != nil {
|
||||
t.Fatalf("parse validation responses: %v", err)
|
||||
}
|
||||
processValidationLLMClient = validationClient
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
processProposalLLMClient = nil
|
||||
processValidationLLMClient = nil
|
||||
})
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
workDir := t.TempDir()
|
||||
outputPath := filepath.Join(t.TempDir(), "out.json")
|
||||
reportPath := filepath.Join(t.TempDir(), "report.json")
|
||||
args := []string{
|
||||
"process",
|
||||
filepath.Join(caseDir, fx.TranscriptFile),
|
||||
"--glossary",
|
||||
filepath.Join(caseDir, fx.GlossaryFile),
|
||||
"--report-json",
|
||||
reportPath,
|
||||
"--work-dir",
|
||||
workDir,
|
||||
"--work-dir-retention",
|
||||
"always",
|
||||
}
|
||||
// Keep stdout shape deterministic for parity tests.
|
||||
if fx.Expect.StdoutMode != "json" {
|
||||
args = append(args, "--output", outputPath)
|
||||
}
|
||||
if strings.TrimSpace(fx.ModulesCSV) != "" {
|
||||
args = append(args, "--modules", fx.ModulesCSV)
|
||||
}
|
||||
|
||||
exitCode := Run(args, &stdout, &stderr)
|
||||
if exitCode != fx.Expect.ExitCode {
|
||||
t.Fatalf("expected exit code %d, got %d stderr=%q", fx.Expect.ExitCode, exitCode, stderr.String())
|
||||
}
|
||||
|
||||
switch fx.Expect.StdoutMode {
|
||||
case "json":
|
||||
if _, err := json.Marshal(stdout.String()); err != nil {
|
||||
t.Fatalf("unexpected stdout marshal error: %v", err)
|
||||
}
|
||||
if !json.Valid(stdout.Bytes()) {
|
||||
t.Fatalf("expected JSON stdout, got %q", stdout.String())
|
||||
}
|
||||
default:
|
||||
if stdout.Len() != 0 {
|
||||
t.Fatalf("expected empty stdout, got %q", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
if fx.Expect.StderrContains != "" && !strings.Contains(stderr.String(), fx.Expect.StderrContains) {
|
||||
t.Fatalf("expected stderr to contain %q, got %q", fx.Expect.StderrContains, stderr.String())
|
||||
}
|
||||
|
||||
report := readProcessReport(t, reportPath)
|
||||
assertParityReport(t, report, fx.Expect)
|
||||
|
||||
runDir := onlyRunDir(t, workDir)
|
||||
runDirReport := readProcessReport(t, filepath.Join(runDir, "report.json"))
|
||||
assertParityReport(t, runDirReport, fx.Expect)
|
||||
|
||||
if fx.Expect.RequireErrorLog {
|
||||
if _, err := os.Stat(filepath.Join(runDir, "error.log")); err != nil {
|
||||
t.Fatalf("expected error.log: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
if strings.TrimSpace(fx.Expect.OutputTranscriptFile) != "" && fx.Expect.ExitCode == 0 {
|
||||
got := readFile(t, outputPath)
|
||||
want := readFile(t, filepath.Join(caseDir, fx.Expect.OutputTranscriptFile))
|
||||
assertJSONSemanticEqual(t, want, got)
|
||||
}
|
||||
|
||||
if len(fx.Expect.ExpectedProposalCalls) > 0 && !reflect.DeepEqual(proposalClient.calls, fx.Expect.ExpectedProposalCalls) {
|
||||
t.Fatalf("unexpected proposal calls: got %v want %v", proposalClient.calls, fx.Expect.ExpectedProposalCalls)
|
||||
}
|
||||
if len(fx.Expect.ExpectedValidationCalls) > 0 && !reflect.DeepEqual(validationClient.calls, fx.Expect.ExpectedValidationCalls) {
|
||||
t.Fatalf("unexpected validation calls: got %v want %v", validationClient.calls, fx.Expect.ExpectedValidationCalls)
|
||||
}
|
||||
|
||||
if len(fx.Expect.SecretMarkers) > 0 {
|
||||
assertNoSecretMarkers(t, reportPath, fx.Expect.SecretMarkers)
|
||||
assertNoSecretMarkersInTree(t, runDir, fx.Expect.SecretMarkers)
|
||||
}
|
||||
if fx.Expect.MinResponsePayloadArtifacts > 0 {
|
||||
matches, err := filepath.Glob(filepath.Join(runDir, "*", "*response-payload.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("glob response payload artifacts: %v", err)
|
||||
}
|
||||
if len(matches) < fx.Expect.MinResponsePayloadArtifacts {
|
||||
t.Fatalf("expected at least %d response payload artifacts, got %d", fx.Expect.MinResponsePayloadArtifacts, len(matches))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func assertParityReport(t *testing.T, report reporting.ProcessReport, exp parityExpectation) {
|
||||
t.Helper()
|
||||
if exp.Status != "" && report.Status != exp.Status {
|
||||
t.Fatalf("expected report status %q, got %q", exp.Status, report.Status)
|
||||
}
|
||||
if exp.ErrorPhase != "" && report.ErrorPhase != exp.ErrorPhase {
|
||||
t.Fatalf("expected report error_phase %q, got %q", exp.ErrorPhase, report.ErrorPhase)
|
||||
}
|
||||
if len(exp.ModuleInstances) > 0 {
|
||||
got := make([]string, 0, len(report.ModuleResults))
|
||||
for _, mr := range report.ModuleResults {
|
||||
got = append(got, mr.ModuleInstance)
|
||||
}
|
||||
if !reflect.DeepEqual(got, exp.ModuleInstances) {
|
||||
t.Fatalf("unexpected module instances: got %v want %v", got, exp.ModuleInstances)
|
||||
}
|
||||
}
|
||||
if exp.ModuleCount > 0 {
|
||||
if report.ModulesSummary == nil || report.ModulesSummary.ModuleCount != exp.ModuleCount {
|
||||
t.Fatalf("expected module_count=%d, got %+v", exp.ModuleCount, report.ModulesSummary)
|
||||
}
|
||||
}
|
||||
if exp.TotalAppliedChanges > 0 {
|
||||
if report.ModulesSummary == nil || report.ModulesSummary.TotalAppliedChanges != exp.TotalAppliedChanges {
|
||||
t.Fatalf("expected total_applied_changes=%d, got %+v", exp.TotalAppliedChanges, report.ModulesSummary)
|
||||
}
|
||||
}
|
||||
if exp.TotalSkippedChanges > 0 {
|
||||
if report.ModulesSummary == nil || report.ModulesSummary.TotalSkippedChanges != exp.TotalSkippedChanges {
|
||||
t.Fatalf("expected total_skipped_changes=%d, got %+v", exp.TotalSkippedChanges, report.ModulesSummary)
|
||||
}
|
||||
}
|
||||
if exp.FailedModuleInstance != "" {
|
||||
if report.ModulesSummary == nil || report.ModulesSummary.FailedModuleInstance != exp.FailedModuleInstance {
|
||||
t.Fatalf("expected failed_module_instance=%q, got %+v", exp.FailedModuleInstance, report.ModulesSummary)
|
||||
}
|
||||
}
|
||||
|
||||
if len(exp.ValidatorRejectedReasonCodes) > 0 {
|
||||
got := collectValidatorRejectedReasonCodes(report.ModuleResults)
|
||||
if !reflect.DeepEqual(got, exp.ValidatorRejectedReasonCodes) {
|
||||
t.Fatalf("unexpected validator rejected reason codes: got %v want %v", got, exp.ValidatorRejectedReasonCodes)
|
||||
}
|
||||
}
|
||||
if len(exp.ApplicationSkipReasonCodes) > 0 {
|
||||
got := collectApplicationSkipReasonCodes(report.ModuleResults)
|
||||
if !reflect.DeepEqual(got, exp.ApplicationSkipReasonCodes) {
|
||||
t.Fatalf("unexpected application skip reason codes: got %v want %v", got, exp.ApplicationSkipReasonCodes)
|
||||
}
|
||||
}
|
||||
if len(exp.ModuleAppliedCounts) > 0 {
|
||||
got := make([]int, 0, len(report.ModuleResults))
|
||||
for _, mr := range report.ModuleResults {
|
||||
got = append(got, len(mr.AppliedChanges))
|
||||
}
|
||||
if !reflect.DeepEqual(got, exp.ModuleAppliedCounts) {
|
||||
t.Fatalf("unexpected per-module applied counts: got %v want %v", got, exp.ModuleAppliedCounts)
|
||||
}
|
||||
}
|
||||
if len(exp.ModuleRejectedCounts) > 0 {
|
||||
got := make([]int, 0, len(report.ModuleResults))
|
||||
for _, mr := range report.ModuleResults {
|
||||
got = append(got, len(mr.ValidatorRejected))
|
||||
}
|
||||
if !reflect.DeepEqual(got, exp.ModuleRejectedCounts) {
|
||||
t.Fatalf("unexpected per-module rejected counts: got %v want %v", got, exp.ModuleRejectedCounts)
|
||||
}
|
||||
}
|
||||
if len(exp.ModuleSkipCounts) > 0 {
|
||||
got := make([]int, 0, len(report.ModuleResults))
|
||||
for _, mr := range report.ModuleResults {
|
||||
got = append(got, len(mr.SkippedChanges))
|
||||
}
|
||||
if !reflect.DeepEqual(got, exp.ModuleSkipCounts) {
|
||||
t.Fatalf("unexpected per-module skip counts: got %v want %v", got, exp.ModuleSkipCounts)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func collectValidatorRejectedReasonCodes(results []reporting.ModuleReport) []string {
|
||||
out := make([]string, 0)
|
||||
for _, mr := range results {
|
||||
for _, vr := range mr.ValidatorRejected {
|
||||
out = append(out, vr.ReasonCode)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func collectApplicationSkipReasonCodes(results []reporting.ModuleReport) []string {
|
||||
out := make([]string, 0)
|
||||
for _, mr := range results {
|
||||
for _, sk := range mr.SkippedChanges {
|
||||
out = append(out, string(sk.SkipReason))
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func assertJSONSemanticEqual(t *testing.T, expected []byte, actual []byte) {
|
||||
t.Helper()
|
||||
var exp any
|
||||
var act any
|
||||
if err := json.Unmarshal(expected, &exp); err != nil {
|
||||
t.Fatalf("unmarshal expected json: %v", err)
|
||||
}
|
||||
if err := json.Unmarshal(actual, &act); err != nil {
|
||||
t.Fatalf("unmarshal actual json: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(exp, act) {
|
||||
t.Fatalf("JSON mismatch\nexpected=%s\nactual=%s", string(expected), string(actual))
|
||||
}
|
||||
}
|
||||
|
||||
func assertNoSecretMarkers(t *testing.T, filePath string, markers []string) {
|
||||
t.Helper()
|
||||
raw := string(readFile(t, filePath))
|
||||
for _, marker := range markers {
|
||||
if marker != "" && strings.Contains(raw, marker) {
|
||||
t.Fatalf("secret marker %q leaked in %s", marker, filePath)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func assertNoSecretMarkersInTree(t *testing.T, root string, markers []string) {
|
||||
t.Helper()
|
||||
_ = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil || d == nil || d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
raw := string(readFile(t, path))
|
||||
for _, marker := range markers {
|
||||
if marker != "" && strings.Contains(raw, marker) {
|
||||
t.Fatalf("secret marker %q leaked in %s", marker, path)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func parityFixturePath(name string) string {
|
||||
return filepath.Join("testdata", "parity", name)
|
||||
}
|
||||
442
internal/cli/process.go
Normal file
442
internal/cli/process.go
Normal file
@@ -0,0 +1,442 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/chunking"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/diagnostics"
|
||||
coreio "gitea.maximumdirect.net/eric/audita/internal/core/io"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/normalization"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/outputschema"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/redaction"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/reporting"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/modules"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/outcomes"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/processreport"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/runner"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/structuredoutput"
|
||||
)
|
||||
|
||||
func runProcess(ctx context.Context, args []string, stdout, stderr io.Writer) int {
|
||||
input, help, err := parseConfigInput("process", args, stderr)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: %s\n", configInputError(err))
|
||||
return 2
|
||||
}
|
||||
if help {
|
||||
writeProcessUsage(stdout)
|
||||
return 0
|
||||
}
|
||||
selected, err := loadConfigInput(input, "process")
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: %s\n", configInputError(err))
|
||||
return 2
|
||||
}
|
||||
cfg := selected.Result.Config
|
||||
reportDir, err := selectedReportDir(cfg, input.OutputPath)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
|
||||
return 2
|
||||
}
|
||||
if cfg.Diagnostics.Debug && strings.TrimSpace(cfg.Diagnostics.WorkDir) == "" {
|
||||
fmt.Fprintln(stderr, "audita process: debug work directory is required")
|
||||
return 2
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: %s\n", err)
|
||||
return 1
|
||||
}
|
||||
runtime, err := llm.NewRuntime(ctx, cfg, selected.Path, nil)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
|
||||
if ctx.Err() != nil || llm.IsAssetError(err) {
|
||||
return 1
|
||||
}
|
||||
return 2
|
||||
}
|
||||
specs, err := contracts.ResolveModuleRunSpecs(cfg.Pipeline.Modules)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
|
||||
return 2
|
||||
}
|
||||
runID, err := generateRunID()
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: run ID: %s\n", llm.RedactError(err, cfg))
|
||||
return 1
|
||||
}
|
||||
started := time.Now().UTC()
|
||||
state := &processState{cfg: cfg, runtime: runtime, specs: specs, runID: runID, started: started,
|
||||
transcriptPath: input.TranscriptPath, glossaryPath: input.GlossaryPath, outputPath: input.OutputPath,
|
||||
selected: selected, overlay: input.Overlay, stderr: stderr, stdout: stdout, ctx: ctx}
|
||||
protected := []string{input.GlossaryPath}
|
||||
for _, source := range selected.Result.Sources() {
|
||||
if source.Path != "" {
|
||||
protected = append(protected, source.Path)
|
||||
}
|
||||
}
|
||||
protected = append(protected, runtime.ProfilePaths...)
|
||||
session, err := diagnostics.PrepareOutput(diagnostics.OutputSelection{InputTranscriptPath: input.TranscriptPath, TranscriptPath: input.OutputPath,
|
||||
ReportDir: reportDir, ReportingEnabled: cfg.Reporting.Enabled, ProtectedPaths: protected})
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
|
||||
return 2
|
||||
}
|
||||
state.session = session
|
||||
initial := state.snapshot(reporting.StatusInProgress, nil, nil)
|
||||
initialBundle, err := state.bundle(initial)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
|
||||
return 1
|
||||
}
|
||||
initFailure := []processreport.RunFailure{{Phase: "initialization", Reason: outcomes.ReasonInternalFailure}}
|
||||
failedInit, err := state.bundle(state.snapshot(reporting.StatusFailed, nil, initFailure))
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
|
||||
return 1
|
||||
}
|
||||
if err := session.Initialize(initialBundle, failedInit); err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: %s\n", llm.RedactError(err, cfg))
|
||||
return 1
|
||||
}
|
||||
state.startDebug()
|
||||
return state.execute()
|
||||
}
|
||||
|
||||
type processState struct {
|
||||
cfg config.Config
|
||||
runtime *llm.Runtime
|
||||
specs []contracts.ModuleRunSpec
|
||||
runID string
|
||||
started time.Time
|
||||
transcriptPath string
|
||||
glossaryPath string
|
||||
outputPath string
|
||||
selected selectedConfig
|
||||
overlay string
|
||||
ctx context.Context
|
||||
stdout io.Writer
|
||||
stderr io.Writer
|
||||
session *diagnostics.OutputSession
|
||||
debugDir *diagnostics.RunDirectory
|
||||
debugWarnings []string
|
||||
debugMu sync.Mutex
|
||||
runOutput *runner.RunOutput
|
||||
}
|
||||
|
||||
func generateRunID() (string, error) {
|
||||
var random [12]byte
|
||||
if _, err := rand.Read(random[:]); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "run-" + hex.EncodeToString(random[:]), nil
|
||||
}
|
||||
|
||||
func (s *processState) snapshot(status string, completed *time.Time, failures []processreport.RunFailure) reporting.Snapshot {
|
||||
return processreport.Build(processreport.BuildInput{RunID: s.runID, Status: status, StartedAt: s.started,
|
||||
CompletedAt: completed, TranscriptPath: s.transcriptPath, GlossaryPath: s.glossaryPath,
|
||||
OutputPath: s.outputPath, ModuleSpecs: s.specs, RunOutput: s.runOutput, RunFailures: failures})
|
||||
}
|
||||
|
||||
func (s *processState) bundle(snapshot reporting.Snapshot) (diagnostics.ReportBundle, error) {
|
||||
if !s.cfg.Reporting.Enabled {
|
||||
return diagnostics.ReportBundle{}, nil
|
||||
}
|
||||
secrets := llm.ConfiguredSecrets(&s.cfg)
|
||||
errorsYAML, err := reporting.YAML(snapshot.ErrorsDocument(), secrets...)
|
||||
if err != nil {
|
||||
return diagnostics.ReportBundle{}, err
|
||||
}
|
||||
recoveriesYAML, err := reporting.YAML(snapshot.RecoveriesDocument(), secrets...)
|
||||
if err != nil {
|
||||
return diagnostics.ReportBundle{}, err
|
||||
}
|
||||
return diagnostics.ReportBundle{Errors: errorsYAML, Recoveries: recoveriesYAML,
|
||||
Markdown: reporting.Markdown(snapshot, secrets...)}, nil
|
||||
}
|
||||
|
||||
func (s *processState) debugWrite(label string, write func(*diagnostics.RunDirectory) error) {
|
||||
if s.debugDir == nil {
|
||||
return
|
||||
}
|
||||
if err := write(s.debugDir); err != nil {
|
||||
s.reportDiagnosticError(fmt.Errorf("%s: %w", label, err))
|
||||
}
|
||||
}
|
||||
|
||||
func (s *processState) startDebug() {
|
||||
if !s.cfg.Diagnostics.Debug {
|
||||
return
|
||||
}
|
||||
dir, err := diagnostics.NewRunDirectory(s.cfg.Diagnostics.WorkDir, s.runID, llm.ConfiguredSecrets(&s.cfg)...)
|
||||
if err != nil {
|
||||
s.reportDiagnosticError(fmt.Errorf("directory: %w", err))
|
||||
return
|
||||
}
|
||||
s.debugDir = dir
|
||||
s.debugWrite("invocation", func(d *diagnostics.RunDirectory) error {
|
||||
return d.WriteInvocationMetadata(diagnostics.InvocationMetadata{
|
||||
Operation: "process", TranscriptPath: s.transcriptPath, GlossaryPath: s.glossaryPath,
|
||||
OutputPath: s.outputPath, ConfigPath: s.selected.Path, ConfigSource: s.selected.Source,
|
||||
Overlay: s.overlay, ConfigVersion: &s.cfg.Version, TranscriptDescription: s.cfg.Context.Description,
|
||||
Modules: append([]string(nil), s.cfg.Pipeline.Modules...), RunID: s.runID, StartedAt: s.started,
|
||||
})
|
||||
})
|
||||
var effective bytes.Buffer
|
||||
if err := s.selected.Result.WriteEffective(&effective); err == nil {
|
||||
s.debugWrite("effective settings", func(d *diagnostics.RunDirectory) error { return d.WriteEffectiveConfig(effective.Bytes()) })
|
||||
} else {
|
||||
s.reportDiagnosticError(fmt.Errorf("effective settings: %w", err))
|
||||
}
|
||||
if origins, err := llm.ExplanationYAML(s.selected.Result); err == nil {
|
||||
s.debugWrite("configuration origins", func(d *diagnostics.RunDirectory) error {
|
||||
return d.WriteYAMLArtifact(diagnostics.ArtifactConfigOrigins, origins)
|
||||
})
|
||||
} else {
|
||||
s.reportDiagnosticError(fmt.Errorf("configuration origins: %w", err))
|
||||
}
|
||||
if profiles, err := s.runtime.SelectedProfilesYAML(); err == nil {
|
||||
s.debugWrite("selected profiles", func(d *diagnostics.RunDirectory) error {
|
||||
return d.WriteYAMLArtifact(diagnostics.ArtifactSelectedProfiles, profiles)
|
||||
})
|
||||
} else {
|
||||
s.reportDiagnosticError(fmt.Errorf("selected profiles: %w", err))
|
||||
}
|
||||
}
|
||||
|
||||
func (s *processState) debugPath() string {
|
||||
if s.debugDir == nil {
|
||||
return ""
|
||||
}
|
||||
return s.debugDir.Path()
|
||||
}
|
||||
|
||||
func (s *processState) execute() int {
|
||||
transcriptBytes, err := coreio.ReadRequiredFile(s.transcriptPath, "transcript")
|
||||
if err != nil {
|
||||
return s.fail("input", outcomes.ReasonInternalFailure, err)
|
||||
}
|
||||
glossaryBytes, err := coreio.ReadRequiredFile(s.glossaryPath, "glossary")
|
||||
if err != nil {
|
||||
return s.fail("input", outcomes.ReasonInternalFailure, err)
|
||||
}
|
||||
source, err := schema.ParseSourceTranscriptJSON(transcriptBytes)
|
||||
if err != nil {
|
||||
return s.fail("input", outcomes.ReasonInputInvalid, err)
|
||||
}
|
||||
glossary, err := schema.ParseGlossaryYAML(glossaryBytes)
|
||||
if err != nil {
|
||||
return s.fail("input", outcomes.ReasonInputInvalid, err)
|
||||
}
|
||||
s.debugWrite("source transcript", func(d *diagnostics.RunDirectory) error { return d.WriteSourceTranscript(source, transcriptBytes) })
|
||||
normalizer := normalization.NewNormalizer(normalization.NormalizationConfig{
|
||||
MaxSegmentGap: s.cfg.Normalization.MaxSegmentGap, EllipsisGap: s.cfg.Normalization.EllipsisGap,
|
||||
MaxSegmentDuration: s.cfg.Normalization.MaxSegmentDuration, MaxSegmentTokens: s.cfg.Normalization.MaxSegmentTokens,
|
||||
})
|
||||
normalized, normSummary := normalizer.Normalize(sourceToCanonicalTranscript(source))
|
||||
s.debugWrite("normalized transcript", func(d *diagnostics.RunDirectory) error { return d.WriteNormalizedTranscript(normalized) })
|
||||
s.debugWrite("normalization summary", func(d *diagnostics.RunDirectory) error { return d.WriteNormalizationSummary(normSummary) })
|
||||
if err := s.ctx.Err(); err != nil {
|
||||
return s.fail("processing", outcomes.ReasonCanceled, err)
|
||||
}
|
||||
observations := llm.NewObservationCollector()
|
||||
proposalClient, err := s.runtime.NewRoleClient("proposal", observations.Hooks())
|
||||
if err != nil {
|
||||
return s.fail("processing", outcomes.ReasonSetupFailure, err)
|
||||
}
|
||||
validationClient, err := s.runtime.NewRoleClient("validation", observations.Hooks())
|
||||
if err != nil {
|
||||
return s.fail("processing", outcomes.ReasonSetupFailure, err)
|
||||
}
|
||||
factory := modules.NewFactory(modules.Dependencies{Config: &s.cfg, Glossary: glossary, DiagnosticsDir: s.debugPath()})
|
||||
result, runErr := runner.New(factory).Run(s.ctx, runner.RunInput{
|
||||
Config: &s.cfg, Transcript: normalized, Glossary: glossary, ModuleSpecs: s.specs,
|
||||
ProposalLLMClient: proposalClient, ProposalDiagnosticsDir: s.debugPath(),
|
||||
ValidationLLMClient: validationClient, ValidationDiagnosticsDir: s.debugPath(),
|
||||
CaptureCandidates: s.debugDir != nil,
|
||||
ReportDiagnosticError: s.reportDiagnosticError,
|
||||
})
|
||||
result.Observations = observations.Snapshot()
|
||||
result.ObservationsAvailable = true
|
||||
result.LLMSummary = llm.Summarize(result.Observations)
|
||||
s.runOutput = &result
|
||||
s.writeExecutionEvidence()
|
||||
if len(result.Sections) > 0 || runErr == nil {
|
||||
s.writeChunkSummary(result.Sections)
|
||||
}
|
||||
if runErr != nil {
|
||||
return s.fail("processing", fatalReason(s.ctx, runErr), runErr)
|
||||
}
|
||||
encoder, err := outputschema.Resolve(s.cfg.Output.Schema)
|
||||
if err != nil {
|
||||
return s.fail("encoding", outcomes.ReasonSetupFailure, err)
|
||||
}
|
||||
outputBytes, err := encoder.Encoder(result.FinalTranscript)
|
||||
if err != nil {
|
||||
return s.fail("encoding", outcomes.ReasonInternalFailure, err)
|
||||
}
|
||||
return s.finish(outputBytes)
|
||||
}
|
||||
|
||||
func (s *processState) writeChunkSummary(inventory []contracts.SectionMetadata) {
|
||||
if s.debugDir == nil {
|
||||
return
|
||||
}
|
||||
sections := make([]chunking.Section, len(inventory))
|
||||
for i, metadata := range inventory {
|
||||
sections[i] = chunking.Section{Index: metadata.Index, StartSegmentID: metadata.StartSegmentID,
|
||||
EndSegmentID: metadata.EndSegmentID, EstimatedTokens: metadata.EstimatedTokens,
|
||||
Segments: make([]schema.Segment, metadata.SegmentCount)}
|
||||
}
|
||||
config := chunking.ChunkingConfig{TargetSections: s.cfg.Chunking.TargetSections,
|
||||
MaxSectionTokens: s.cfg.Chunking.MaxSectionTokens, MinSectionTokens: s.cfg.Chunking.MinSectionTokens}
|
||||
summary := chunking.ComputeDetailedSummary(sections, config)
|
||||
s.debugWrite("chunking summary", func(d *diagnostics.RunDirectory) error { return d.WriteChunkingSummary(&summary) })
|
||||
}
|
||||
|
||||
func (s *processState) writeExecutionEvidence() {
|
||||
if s.debugDir == nil || s.runOutput == nil {
|
||||
return
|
||||
}
|
||||
output := s.runOutput
|
||||
s.debugWrite("model execution", func(d *diagnostics.RunDirectory) error {
|
||||
return d.WriteJSONArtifact(diagnostics.ArtifactLLMExecution, processreport.BuildLLMExecution(output.Observations))
|
||||
})
|
||||
s.debugWrite("utilization", func(d *diagnostics.RunDirectory) error {
|
||||
return d.WriteJSONArtifact(diagnostics.ArtifactUtilizationSummary, processreport.BuildUtilization(output, s.cfg.Backends))
|
||||
})
|
||||
s.debugWrite("correction ledger", func(d *diagnostics.RunDirectory) error {
|
||||
return d.WriteJSONArtifact(diagnostics.ArtifactCorrectionLedger, processreport.BuildCorrectionLedger(processreport.CorrectionLedgerInput{
|
||||
RunDirectoryPath: d.Path(), RunOutput: output}))
|
||||
})
|
||||
}
|
||||
|
||||
func fatalReason(ctx context.Context, err error) outcomes.ReasonCode {
|
||||
if ctx.Err() != nil || errors.Is(err, context.Canceled) {
|
||||
return outcomes.ReasonCanceled
|
||||
}
|
||||
var failure *structuredoutput.Failure
|
||||
if errors.As(err, &failure) && failure.Reason != "" {
|
||||
return failure.Reason
|
||||
}
|
||||
return outcomes.ReasonInternalFailure
|
||||
}
|
||||
|
||||
func (s *processState) fail(phase string, reason outcomes.ReasonCode, cause error) int {
|
||||
s.debugWrite("error log", func(d *diagnostics.RunDirectory) error {
|
||||
return d.WriteErrorLog(phase + ": " + llm.RedactError(cause, s.cfg))
|
||||
})
|
||||
failures := []processreport.RunFailure{{Phase: phase, Reason: reason}}
|
||||
completed := time.Now().UTC()
|
||||
base, err := s.bundle(s.snapshot(reporting.StatusFailed, &completed, failures))
|
||||
if err != nil {
|
||||
fmt.Fprintf(s.stderr, "audita process: %s\n", llm.RedactError(err, s.cfg))
|
||||
}
|
||||
writeFailures := s.session.Finalize(nil, base, func(extra []diagnostics.FileFailure) diagnostics.ReportBundle {
|
||||
all := append(append([]processreport.RunFailure(nil), failures...), mapWriteFailures(extra)...)
|
||||
bundle, renderErr := s.bundle(s.snapshot(reporting.StatusFailed, &completed, all))
|
||||
if renderErr != nil {
|
||||
return base
|
||||
}
|
||||
return bundle
|
||||
})
|
||||
fmt.Fprintf(s.stderr, "audita process: %s\n", llm.RedactError(cause, s.cfg))
|
||||
s.printFailures(writeFailures)
|
||||
s.printDebugWarnings()
|
||||
if s.debugDir != nil {
|
||||
fmt.Fprintf(s.stderr, "audita process: diagnostics: %s\n", s.safePath(s.debugDir.Path()))
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
func (s *processState) finish(outputBytes []byte) int {
|
||||
completed := time.Now().UTC()
|
||||
snapshot := s.snapshot(reporting.StatusCompleted, &completed, nil)
|
||||
complete, err := s.bundle(snapshot)
|
||||
if err != nil {
|
||||
return s.fail("report_write", outcomes.ReasonInternalFailure, err)
|
||||
}
|
||||
if s.session.TranscriptPath() == "" {
|
||||
if _, err := s.stdout.Write(outputBytes); err != nil {
|
||||
return s.fail("transcript_write", outcomes.ReasonInternalFailure, err)
|
||||
}
|
||||
outputBytes = nil
|
||||
}
|
||||
writeFailures := s.session.Finalize(outputBytes, complete, func(extra []diagnostics.FileFailure) diagnostics.ReportBundle {
|
||||
failed, renderErr := s.bundle(s.snapshot(reporting.StatusFailed, &completed, mapWriteFailures(extra)))
|
||||
if renderErr != nil {
|
||||
return diagnostics.ReportBundle{}
|
||||
}
|
||||
return failed
|
||||
})
|
||||
if len(writeFailures) > 0 {
|
||||
s.printFailures(writeFailures)
|
||||
s.printDebugWarnings()
|
||||
if s.debugDir != nil {
|
||||
fmt.Fprintf(s.stderr, "audita process: diagnostics: %s\n", s.safePath(s.debugDir.Path()))
|
||||
}
|
||||
return 1
|
||||
}
|
||||
fmt.Fprintln(s.stderr, reporting.CompletionLine(snapshot))
|
||||
for _, path := range s.session.ReportPaths() {
|
||||
fmt.Fprintf(s.stderr, "Report: %s\n", s.safePath(path))
|
||||
}
|
||||
if s.debugDir != nil {
|
||||
fmt.Fprintf(s.stderr, "Debug: %s\n", s.safePath(s.debugDir.Path()))
|
||||
}
|
||||
s.printDebugWarnings()
|
||||
return 0
|
||||
}
|
||||
|
||||
func mapWriteFailures(failures []diagnostics.FileFailure) []processreport.RunFailure {
|
||||
out := make([]processreport.RunFailure, len(failures))
|
||||
for i, failure := range failures {
|
||||
phase := "report_write"
|
||||
if failure.Kind == "transcript write" {
|
||||
phase = "transcript_write"
|
||||
}
|
||||
out[i] = processreport.RunFailure{Phase: phase, Reason: outcomes.ReasonInternalFailure, OutputPath: failure.Path}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *processState) printFailures(failures []diagnostics.FileFailure) {
|
||||
for _, failure := range failures {
|
||||
fmt.Fprintf(s.stderr, "audita process: %s\n", llm.RedactError(failure, s.cfg))
|
||||
}
|
||||
}
|
||||
|
||||
func (s *processState) printDebugWarnings() {
|
||||
s.debugMu.Lock()
|
||||
defer s.debugMu.Unlock()
|
||||
for _, warning := range s.debugWarnings {
|
||||
fmt.Fprintf(s.stderr, "audita process: %s\n", warning)
|
||||
}
|
||||
}
|
||||
|
||||
// safePath sanitizes presentation only; filesystem destinations remain literal.
|
||||
func (s *processState) safePath(path string) string {
|
||||
return redaction.New(llm.ConfiguredSecrets(&s.cfg)).String(path)
|
||||
}
|
||||
|
||||
func (s *processState) reportDiagnosticError(err error) {
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
message := "debug " + llm.RedactError(err, s.cfg)
|
||||
s.debugMu.Lock()
|
||||
defer s.debugMu.Unlock()
|
||||
s.debugWarnings = append(s.debugWarnings, message)
|
||||
}
|
||||
@@ -1,121 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"flag"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||
)
|
||||
|
||||
type processOverrideBinding func(*config.CLIOverrides, processFlags)
|
||||
|
||||
var processOverrideBindings = map[string]processOverrideBinding{
|
||||
"modules": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ModulesCSV = flags.modules
|
||||
},
|
||||
"output-schema": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.OutputSchema = flags.outputSchema
|
||||
},
|
||||
"llm-api-key": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.PrimaryLLMAPIKey = flags.llmAPIKey
|
||||
},
|
||||
"validation-llm-api-key": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationLLMAPIKey = flags.validationLLMAPIKey
|
||||
},
|
||||
"model": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.PrimaryModel = flags.model
|
||||
},
|
||||
"validation-model": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationModel = flags.validationModel
|
||||
},
|
||||
"base-url": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.PrimaryBaseURL = flags.baseURL
|
||||
},
|
||||
"validation-base-url": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationBaseURL = flags.validationBaseURL
|
||||
},
|
||||
"llm-timeout-seconds": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.PrimaryLLMTimeoutSeconds = flags.llmTimeoutSeconds
|
||||
},
|
||||
"total-llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.TotalLLMConcurrency = flags.totalLLMConcurrency
|
||||
},
|
||||
"proposal-llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ProposalLLMConcurrency = flags.proposalLLMConcurrency
|
||||
},
|
||||
"llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.PrimaryLLMConcurrency = flags.llmConcurrency
|
||||
},
|
||||
"validation-llm-timeout-seconds": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationLLMTimeoutSeconds = flags.validationLLMTimeoutSeconds
|
||||
},
|
||||
"max-retries": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.MaxRetries = flags.maxRetries
|
||||
},
|
||||
"validation-max-retries": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationMaxRetries = flags.validationMaxRetries
|
||||
},
|
||||
"validation-llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationLLMConcurrency = flags.validationLLMConcurrency
|
||||
},
|
||||
"validation-max-prompt-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationMaxPromptTokens = flags.validationMaxPromptTokens
|
||||
},
|
||||
"max-section-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.MaxSectionTokens = flags.maxSectionTokens
|
||||
},
|
||||
"min-section-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.MinSectionTokens = flags.minSectionTokens
|
||||
},
|
||||
"target-sections": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.TargetSections = flags.targetSections
|
||||
},
|
||||
"glossary-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.GlossaryConfidenceThreshold = flags.glossaryConfidenceThreshold
|
||||
},
|
||||
"grammar-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.GrammarConfidenceThreshold = flags.grammarConfidenceThreshold
|
||||
},
|
||||
"homophones-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.HomophonesConfidenceThreshold = flags.homophonesConfidenceThreshold
|
||||
},
|
||||
"spoken-word-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.SpokenWordConfidenceThreshold = flags.spokenWordConfidenceThreshold
|
||||
},
|
||||
"normalize-max-segment-gap": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.NormalizeMaxSegmentGap = flags.normalizeMaxSegmentGap
|
||||
},
|
||||
"normalize-ellipsis-gap": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.NormalizeEllipsisGap = flags.normalizeEllipsisGap
|
||||
},
|
||||
"normalize-max-segment-duration": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.NormalizeMaxSegmentDuration = flags.normalizeMaxSegmentDuration
|
||||
},
|
||||
"normalize-max-segment-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.NormalizeMaxSegmentTokens = flags.normalizeMaxSegmentTokens
|
||||
},
|
||||
"transcript-description": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.TranscriptDescription = flags.transcriptDescription
|
||||
},
|
||||
"work-dir": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.WorkDir = flags.workDir
|
||||
},
|
||||
"work-dir-retention": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.WorkDirRetention = flags.workDirRetention
|
||||
},
|
||||
}
|
||||
|
||||
func processCLIOverrides(fs *flag.FlagSet, flags processFlags) (config.CLIOverrides, bool) {
|
||||
overrides := config.CLIOverrides{}
|
||||
explicitModules := false
|
||||
fs.Visit(func(f *flag.Flag) {
|
||||
if f.Name == "modules" {
|
||||
explicitModules = true
|
||||
}
|
||||
binding, ok := processOverrideBindings[f.Name]
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
binding(&overrides, flags)
|
||||
})
|
||||
return overrides, explicitModules
|
||||
}
|
||||
@@ -1,433 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"io"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||
)
|
||||
|
||||
func TestProcessCLIOverridesMapsEveryConfigMutatingFlag(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
flagName string
|
||||
value string
|
||||
wantExplicitModules bool
|
||||
assertOverrideFields func(t *testing.T, overrides config.CLIOverrides)
|
||||
}{
|
||||
{
|
||||
name: "modules",
|
||||
flagName: "modules",
|
||||
value: "grammar,glossary",
|
||||
wantExplicitModules: true,
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "ModulesCSV", overrides.ModulesCSV, "grammar,glossary")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "output schema",
|
||||
flagName: "output-schema",
|
||||
value: "audita-v1",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "OutputSchema", overrides.OutputSchema, "audita-v1")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "primary api key",
|
||||
flagName: "llm-api-key",
|
||||
value: "primary-key",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "PrimaryLLMAPIKey", overrides.PrimaryLLMAPIKey, "primary-key")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation api key",
|
||||
flagName: "validation-llm-api-key",
|
||||
value: "validation-key",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "ValidationLLMAPIKey", overrides.ValidationLLMAPIKey, "validation-key")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "primary model",
|
||||
flagName: "model",
|
||||
value: "primary-model",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "PrimaryModel", overrides.PrimaryModel, "primary-model")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation model",
|
||||
flagName: "validation-model",
|
||||
value: "validation-model",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "ValidationModel", overrides.ValidationModel, "validation-model")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "primary base url",
|
||||
flagName: "base-url",
|
||||
value: "https://primary.example.test",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "PrimaryBaseURL", overrides.PrimaryBaseURL, "https://primary.example.test")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation base url",
|
||||
flagName: "validation-base-url",
|
||||
value: "https://validation.example.test",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "ValidationBaseURL", overrides.ValidationBaseURL, "https://validation.example.test")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "primary timeout",
|
||||
flagName: "llm-timeout-seconds",
|
||||
value: "101",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "PrimaryLLMTimeoutSeconds", overrides.PrimaryLLMTimeoutSeconds, 101)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "total concurrency",
|
||||
flagName: "total-llm-concurrency",
|
||||
value: "5",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "TotalLLMConcurrency", overrides.TotalLLMConcurrency, 5)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "proposal concurrency",
|
||||
flagName: "proposal-llm-concurrency",
|
||||
value: "3",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "ProposalLLMConcurrency", overrides.ProposalLLMConcurrency, 3)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "legacy concurrency alias",
|
||||
flagName: "llm-concurrency",
|
||||
value: "4",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "PrimaryLLMConcurrency", overrides.PrimaryLLMConcurrency, 4)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation timeout",
|
||||
flagName: "validation-llm-timeout-seconds",
|
||||
value: "202",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "ValidationLLMTimeoutSeconds", overrides.ValidationLLMTimeoutSeconds, 202)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "max retries",
|
||||
flagName: "max-retries",
|
||||
value: "6",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "MaxRetries", overrides.MaxRetries, 6)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation max retries",
|
||||
flagName: "validation-max-retries",
|
||||
value: "7",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "ValidationMaxRetries", overrides.ValidationMaxRetries, 7)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation concurrency",
|
||||
flagName: "validation-llm-concurrency",
|
||||
value: "8",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "ValidationLLMConcurrency", overrides.ValidationLLMConcurrency, 8)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation max prompt tokens",
|
||||
flagName: "validation-max-prompt-tokens",
|
||||
value: "4096",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "ValidationMaxPromptTokens", overrides.ValidationMaxPromptTokens, 4096)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "max section tokens",
|
||||
flagName: "max-section-tokens",
|
||||
value: "9000",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "MaxSectionTokens", overrides.MaxSectionTokens, 9000)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "min section tokens",
|
||||
flagName: "min-section-tokens",
|
||||
value: "1000",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "MinSectionTokens", overrides.MinSectionTokens, 1000)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "target sections",
|
||||
flagName: "target-sections",
|
||||
value: "12",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "TargetSections", overrides.TargetSections, 12)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "glossary threshold",
|
||||
flagName: "glossary-confidence-threshold",
|
||||
value: "0.91",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "GlossaryConfidenceThreshold", overrides.GlossaryConfidenceThreshold, 0.91)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "grammar threshold",
|
||||
flagName: "grammar-confidence-threshold",
|
||||
value: "0.92",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "GrammarConfidenceThreshold", overrides.GrammarConfidenceThreshold, 0.92)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "homophones threshold",
|
||||
flagName: "homophones-confidence-threshold",
|
||||
value: "0.93",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "HomophonesConfidenceThreshold", overrides.HomophonesConfidenceThreshold, 0.93)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "spoken word threshold",
|
||||
flagName: "spoken-word-confidence-threshold",
|
||||
value: "0.94",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "SpokenWordConfidenceThreshold", overrides.SpokenWordConfidenceThreshold, 0.94)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "normalize max segment gap",
|
||||
flagName: "normalize-max-segment-gap",
|
||||
value: "1.2",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "NormalizeMaxSegmentGap", overrides.NormalizeMaxSegmentGap, 1.2)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "normalize ellipsis gap",
|
||||
flagName: "normalize-ellipsis-gap",
|
||||
value: "2.3",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "NormalizeEllipsisGap", overrides.NormalizeEllipsisGap, 2.3)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "normalize max segment duration",
|
||||
flagName: "normalize-max-segment-duration",
|
||||
value: "45.6",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "NormalizeMaxSegmentDuration", overrides.NormalizeMaxSegmentDuration, 45.6)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "normalize max segment tokens",
|
||||
flagName: "normalize-max-segment-tokens",
|
||||
value: "321",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "NormalizeMaxSegmentTokens", overrides.NormalizeMaxSegmentTokens, 321)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "transcript description",
|
||||
flagName: "transcript-description",
|
||||
value: "podcast episode",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "TranscriptDescription", overrides.TranscriptDescription, "podcast episode")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "work dir",
|
||||
flagName: "work-dir",
|
||||
value: "/tmp/custom-audita",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "WorkDir", overrides.WorkDir, "/tmp/custom-audita")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "work dir retention",
|
||||
flagName: "work-dir-retention",
|
||||
value: "always",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "WorkDirRetention", overrides.WorkDirRetention, "always")
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
fs, flags := newProcessFlagSet(config.Default(), io.Discard)
|
||||
if err := fs.Parse([]string{"--" + tc.flagName, tc.value}); err != nil {
|
||||
t.Fatalf("parse flag: %v", err)
|
||||
}
|
||||
|
||||
overrides, explicitModules := processCLIOverrides(fs, flags)
|
||||
if explicitModules != tc.wantExplicitModules {
|
||||
t.Fatalf("explicitModules=%v, want %v", explicitModules, tc.wantExplicitModules)
|
||||
}
|
||||
tc.assertOverrideFields(t, overrides)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessCLIOverridesIgnoresNonConfigFlags(t *testing.T) {
|
||||
fs, flags := newProcessFlagSet(config.Default(), io.Discard)
|
||||
if err := fs.Parse([]string{
|
||||
"--config", "/tmp/config.yml",
|
||||
"--glossary", "/tmp/glossary.yml",
|
||||
"--output", "/tmp/output.json",
|
||||
"--report-json", "/tmp/report.json",
|
||||
}); err != nil {
|
||||
t.Fatalf("parse flags: %v", err)
|
||||
}
|
||||
|
||||
overrides, explicitModules := processCLIOverrides(fs, flags)
|
||||
if explicitModules {
|
||||
t.Fatal("non-config flags should not mark modules explicit")
|
||||
}
|
||||
assertNoCLIOverrides(t, overrides)
|
||||
}
|
||||
|
||||
func TestNewProcessFlagSetDefaultsReflectEffectiveConfig(t *testing.T) {
|
||||
cfg := config.Default()
|
||||
cfg.Modules = []string{"grammar", "glossary"}
|
||||
cfg.OutputSchema = "audita-v1"
|
||||
cfg.PrimaryLLM.APIKey = "primary-key"
|
||||
cfg.ValidationLLM.APIKey = "validation-key"
|
||||
cfg.PrimaryLLM.Model = "primary-model"
|
||||
cfg.ValidationLLM.Model = "validation-model"
|
||||
cfg.PrimaryLLM.BaseURL = "https://primary.example.test"
|
||||
cfg.ValidationLLM.BaseURL = "https://validation.example.test"
|
||||
cfg.PrimaryLLM.TimeoutSeconds = 101
|
||||
cfg.TotalLLMConcurrency = 5
|
||||
cfg.ProposalLLMConcurrency = 3
|
||||
cfg.PrimaryLLM.MaxRetries = 6
|
||||
cfg.ValidationMaxPromptTokens = 4096
|
||||
cfg.MaxSectionTokens = 9000
|
||||
cfg.MinSectionTokens = 1000
|
||||
cfg.Thresholds.Glossary = 0.91
|
||||
cfg.Thresholds.Grammar = 0.92
|
||||
cfg.Thresholds.Homophones = 0.93
|
||||
cfg.Thresholds.SpokenWord = 0.94
|
||||
cfg.Normalization.MaxSegmentGap = 1.2
|
||||
cfg.Normalization.EllipsisGap = 2.3
|
||||
cfg.Normalization.MaxSegmentDuration = 45.6
|
||||
cfg.Normalization.MaxSegmentTokens = 321
|
||||
cfg.TranscriptDescription = "podcast episode"
|
||||
cfg.WorkDir = "/tmp/custom-audita"
|
||||
cfg.WorkDirRetention = config.WorkDirRetentionAlways
|
||||
|
||||
validationTimeout := 202
|
||||
validationRetries := 7
|
||||
validationConcurrency := 8
|
||||
targetSections := 12
|
||||
cfg.ValidationLLM.TimeoutSeconds = &validationTimeout
|
||||
cfg.ValidationLLM.MaxRetries = &validationRetries
|
||||
cfg.ValidationLLMConcurrency = &validationConcurrency
|
||||
cfg.TargetSections = &targetSections
|
||||
|
||||
_, flags := newProcessFlagSet(cfg, io.Discard)
|
||||
|
||||
assertStringOverride(t, "modules default", flags.modules, "grammar,glossary")
|
||||
assertStringOverride(t, "output schema default", flags.outputSchema, "audita-v1")
|
||||
assertStringOverride(t, "primary api key default", flags.llmAPIKey, "primary-key")
|
||||
assertStringOverride(t, "validation api key default", flags.validationLLMAPIKey, "validation-key")
|
||||
assertStringOverride(t, "primary model default", flags.model, "primary-model")
|
||||
assertStringOverride(t, "validation model default", flags.validationModel, "validation-model")
|
||||
assertStringOverride(t, "primary base url default", flags.baseURL, "https://primary.example.test")
|
||||
assertStringOverride(t, "validation base url default", flags.validationBaseURL, "https://validation.example.test")
|
||||
assertIntOverride(t, "primary timeout default", flags.llmTimeoutSeconds, 101)
|
||||
assertIntOverride(t, "total concurrency default", flags.totalLLMConcurrency, 5)
|
||||
assertIntOverride(t, "proposal concurrency default", flags.proposalLLMConcurrency, 3)
|
||||
assertIntOverride(t, "legacy concurrency alias default", flags.llmConcurrency, 5)
|
||||
assertIntOverride(t, "validation timeout default", flags.validationLLMTimeoutSeconds, validationTimeout)
|
||||
assertIntOverride(t, "max retries default", flags.maxRetries, 6)
|
||||
assertIntOverride(t, "validation max retries default", flags.validationMaxRetries, validationRetries)
|
||||
assertIntOverride(t, "validation concurrency default", flags.validationLLMConcurrency, validationConcurrency)
|
||||
assertIntOverride(t, "validation max prompt tokens default", flags.validationMaxPromptTokens, 4096)
|
||||
assertIntOverride(t, "max section tokens default", flags.maxSectionTokens, 9000)
|
||||
assertIntOverride(t, "min section tokens default", flags.minSectionTokens, 1000)
|
||||
assertIntOverride(t, "target sections default", flags.targetSections, targetSections)
|
||||
assertFloatOverride(t, "glossary threshold default", flags.glossaryConfidenceThreshold, 0.91)
|
||||
assertFloatOverride(t, "grammar threshold default", flags.grammarConfidenceThreshold, 0.92)
|
||||
assertFloatOverride(t, "homophones threshold default", flags.homophonesConfidenceThreshold, 0.93)
|
||||
assertFloatOverride(t, "spoken word threshold default", flags.spokenWordConfidenceThreshold, 0.94)
|
||||
assertFloatOverride(t, "normalize max segment gap default", flags.normalizeMaxSegmentGap, 1.2)
|
||||
assertFloatOverride(t, "normalize ellipsis gap default", flags.normalizeEllipsisGap, 2.3)
|
||||
assertFloatOverride(t, "normalize max segment duration default", flags.normalizeMaxSegmentDuration, 45.6)
|
||||
assertIntOverride(t, "normalize max segment tokens default", flags.normalizeMaxSegmentTokens, 321)
|
||||
assertStringOverride(t, "transcript description default", flags.transcriptDescription, "podcast episode")
|
||||
assertStringOverride(t, "work dir default", flags.workDir, "/tmp/custom-audita")
|
||||
assertStringOverride(t, "work dir retention default", flags.workDirRetention, "always")
|
||||
}
|
||||
|
||||
func TestNewProcessFlagSetUsesFallbackDefaultsForUnsetOptionalConfig(t *testing.T) {
|
||||
cfg := config.Default()
|
||||
|
||||
_, flags := newProcessFlagSet(cfg, io.Discard)
|
||||
|
||||
assertIntOverride(t, "validation timeout fallback", flags.validationLLMTimeoutSeconds, cfg.PrimaryLLM.TimeoutSeconds)
|
||||
assertIntOverride(t, "validation retries fallback", flags.validationMaxRetries, cfg.PrimaryLLM.MaxRetries)
|
||||
assertIntOverride(t, "validation concurrency fallback", flags.validationLLMConcurrency, cfg.TotalLLMConcurrency)
|
||||
assertIntOverride(t, "target sections fallback", flags.targetSections, 0)
|
||||
}
|
||||
|
||||
func assertStringOverride(t *testing.T, name string, got *string, want string) {
|
||||
t.Helper()
|
||||
if got == nil || *got != want {
|
||||
t.Fatalf("%s=%v, want %q", name, pointerValue(got), want)
|
||||
}
|
||||
}
|
||||
|
||||
func assertIntOverride(t *testing.T, name string, got *int, want int) {
|
||||
t.Helper()
|
||||
if got == nil || *got != want {
|
||||
t.Fatalf("%s=%v, want %d", name, pointerValue(got), want)
|
||||
}
|
||||
}
|
||||
|
||||
func assertFloatOverride(t *testing.T, name string, got *float64, want float64) {
|
||||
t.Helper()
|
||||
if got == nil || *got != want {
|
||||
t.Fatalf("%s=%v, want %v", name, pointerValue(got), want)
|
||||
}
|
||||
}
|
||||
|
||||
func assertNoCLIOverrides(t *testing.T, overrides config.CLIOverrides) {
|
||||
t.Helper()
|
||||
value := reflect.ValueOf(overrides)
|
||||
typ := value.Type()
|
||||
for i := 0; i < value.NumField(); i++ {
|
||||
field := value.Field(i)
|
||||
if field.Kind() != reflect.Ptr {
|
||||
t.Fatalf("unexpected non-pointer CLIOverrides field %s", typ.Field(i).Name)
|
||||
}
|
||||
if !field.IsNil() {
|
||||
t.Fatalf("expected no CLI overrides, field %s was set", typ.Field(i).Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func pointerValue[T any](ptr *T) any {
|
||||
if ptr == nil {
|
||||
return "<nil>"
|
||||
}
|
||||
if stringer, ok := any(*ptr).(interface{ String() string }); ok {
|
||||
return strings.TrimSpace(stringer.String())
|
||||
}
|
||||
return *ptr
|
||||
}
|
||||
@@ -1,469 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/reporting"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||
)
|
||||
|
||||
type releaseFixtureExpectations struct {
|
||||
MustApplyTexts []string `json:"must_apply_texts"`
|
||||
MustNotApplyTexts []string `json:"must_not_apply_texts"`
|
||||
ProtectedTerms []string `json:"protected_terms"`
|
||||
ExpectedModuleInstance []string `json:"expected_module_instances"`
|
||||
MinimumCounts struct {
|
||||
Applied int `json:"applied"`
|
||||
Rejected int `json:"rejected"`
|
||||
Skipped int `json:"skipped"`
|
||||
} `json:"minimum_counts"`
|
||||
}
|
||||
|
||||
func TestReleaseFixtureDefaultPipelineReadiness(t *testing.T) {
|
||||
base := fixturePath(filepath.Join("release", "default-release"))
|
||||
|
||||
var expectations releaseFixtureExpectations
|
||||
if err := json.Unmarshal(readFile(t, base+".expectations.json"), &expectations); err != nil {
|
||||
t.Fatalf("unmarshal release expectations: %v", err)
|
||||
}
|
||||
|
||||
proposalResponses := readProposalResponses(t, base+".proposals.json")
|
||||
validationResponses := readValidationResponses(t, base+".validations.json")
|
||||
|
||||
// First pass: default full pipeline with deterministic fake LLM responses.
|
||||
first := runReleaseFixturePass(t, releaseRunConfig{
|
||||
transcriptPath: base + ".transcript.json",
|
||||
glossaryPath: base + ".glossary.yaml",
|
||||
outputSchema: "bare-segments",
|
||||
proposalResponses: proposalResponses,
|
||||
validationResponses: validationResponses,
|
||||
expectedProposalCalls: []string{"glossary_1:proposal", "homophones:proposal", "glossary_2:proposal", "spoken_word:proposal", "grammar:proposal"},
|
||||
reportSchemaName: reporting.DefaultProcessReportSchemaName,
|
||||
reportSchemaVersion: reporting.DefaultProcessReportSchemaVersion,
|
||||
expectedOutputSchema: "bare-segments",
|
||||
expectModuleInstances: expectations.ExpectedModuleInstance,
|
||||
expectUtilizationPaths: true,
|
||||
})
|
||||
|
||||
gotTranscript := mustReadTranscript(t, first.outputPath)
|
||||
expectFinalTranscriptContains(t, gotTranscript, expectations.MustApplyTexts)
|
||||
expectFinalTranscriptDoesNotContain(t, gotTranscript, expectations.MustNotApplyTexts)
|
||||
expectFinalTranscriptContains(t, gotTranscript, expectations.ProtectedTerms)
|
||||
|
||||
assertReleaseCounts(t, first.report, expectations)
|
||||
assertPromptAndSchemaMetadataPresent(t, first.runDir)
|
||||
assertReleaseLedgerShape(t, first.report)
|
||||
assertReleaseUtilizationShape(t, first.report)
|
||||
assertStableValidatorKeysPresent(t, first.report)
|
||||
assertStdoutStderrContract(t, first.stdout, first.stderr)
|
||||
assertNoSecretMarkersInTree(t, first.runDir, []string{"release-secret"})
|
||||
assertNoSecretMarkers(t, first.reportPath, []string{"release-secret"})
|
||||
|
||||
// Output schema check: audita-v1 object payload.
|
||||
auditaV1 := runReleaseFixturePass(t, releaseRunConfig{
|
||||
transcriptPath: base + ".transcript.json",
|
||||
glossaryPath: base + ".glossary.yaml",
|
||||
outputSchema: "audita-v1",
|
||||
proposalResponses: proposalResponses,
|
||||
validationResponses: validationResponses,
|
||||
expectedProposalCalls: []string{"glossary_1:proposal", "homophones:proposal", "glossary_2:proposal", "spoken_word:proposal", "grammar:proposal"},
|
||||
reportSchemaName: reporting.DefaultProcessReportSchemaName,
|
||||
reportSchemaVersion: reporting.DefaultProcessReportSchemaVersion,
|
||||
expectedOutputSchema: "audita-v1",
|
||||
expectModuleInstances: expectations.ExpectedModuleInstance,
|
||||
expectUtilizationPaths: true,
|
||||
})
|
||||
assertAuditaV1OutputShape(t, auditaV1.outputPath)
|
||||
|
||||
// Idempotence-oriented second pass:
|
||||
// run again on first output with deterministic no-op responses.
|
||||
noOpProposals := make([]proposal_generation.StructuredCorrectionSet, 5)
|
||||
for i := range noOpProposals {
|
||||
noOpProposals[i] = proposal_generation.StructuredCorrectionSet{Corrections: nil}
|
||||
}
|
||||
second := runReleaseFixturePass(t, releaseRunConfig{
|
||||
transcriptPath: first.outputPath,
|
||||
glossaryPath: base + ".glossary.yaml",
|
||||
outputSchema: "bare-segments",
|
||||
proposalResponses: noOpProposals,
|
||||
validationResponses: nil,
|
||||
expectedProposalCalls: []string{"glossary_1:proposal", "homophones:proposal", "glossary_2:proposal", "spoken_word:proposal", "grammar:proposal"},
|
||||
reportSchemaName: reporting.DefaultProcessReportSchemaName,
|
||||
reportSchemaVersion: reporting.DefaultProcessReportSchemaVersion,
|
||||
expectedOutputSchema: "bare-segments",
|
||||
expectModuleInstances: expectations.ExpectedModuleInstance,
|
||||
expectUtilizationPaths: true,
|
||||
})
|
||||
firstSegments := mustReadTranscript(t, first.outputPath)
|
||||
secondSegments := mustReadTranscript(t, second.outputPath)
|
||||
if !reflect.DeepEqual(firstSegments, secondSegments) {
|
||||
t.Fatalf("expected idempotent second pass transcript; first=%+v second=%+v", firstSegments, secondSegments)
|
||||
}
|
||||
if second.report.ModulesSummary == nil {
|
||||
t.Fatalf("expected modules summary on second pass")
|
||||
}
|
||||
if second.report.ModulesSummary.TotalAppliedChanges != 0 {
|
||||
t.Fatalf("expected no-op second pass (0 applied), got %+v", second.report.ModulesSummary)
|
||||
}
|
||||
}
|
||||
|
||||
type releaseRunConfig struct {
|
||||
transcriptPath string
|
||||
glossaryPath string
|
||||
outputSchema string
|
||||
proposalResponses []proposal_generation.StructuredCorrectionSet
|
||||
validationResponses []validators.LLMValidationResponse
|
||||
expectedProposalCalls []string
|
||||
reportSchemaName string
|
||||
reportSchemaVersion string
|
||||
expectedOutputSchema string
|
||||
expectModuleInstances []string
|
||||
expectUtilizationPaths bool
|
||||
}
|
||||
|
||||
type releaseRunResult struct {
|
||||
stdout string
|
||||
stderr string
|
||||
outputPath string
|
||||
reportPath string
|
||||
report reporting.ProcessReport
|
||||
runDir string
|
||||
}
|
||||
|
||||
func runReleaseFixturePass(t *testing.T, cfg releaseRunConfig) releaseRunResult {
|
||||
t.Helper()
|
||||
|
||||
processProposalLLMClient = &fakeStructuredLLMClient{proposalResponses: append([]proposal_generation.StructuredCorrectionSet(nil), cfg.proposalResponses...)}
|
||||
processValidationLLMClient = &fakeStructuredLLMClient{validationResponses: append([]validators.LLMValidationResponse(nil), cfg.validationResponses...)}
|
||||
t.Cleanup(func() {
|
||||
processProposalLLMClient = nil
|
||||
processValidationLLMClient = nil
|
||||
})
|
||||
|
||||
workDir := t.TempDir()
|
||||
reportPath := filepath.Join(t.TempDir(), "report.json")
|
||||
outputPath := filepath.Join(t.TempDir(), "out.json")
|
||||
configPath := writeFile(t, "release-config.yml", "version: 1\n")
|
||||
|
||||
args := []string{
|
||||
"process",
|
||||
cfg.transcriptPath,
|
||||
"--glossary",
|
||||
cfg.glossaryPath,
|
||||
"--config",
|
||||
configPath,
|
||||
"--output",
|
||||
outputPath,
|
||||
"--output-schema",
|
||||
cfg.outputSchema,
|
||||
"--report-json",
|
||||
reportPath,
|
||||
"--work-dir",
|
||||
workDir,
|
||||
"--work-dir-retention",
|
||||
"always",
|
||||
}
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
exitCode := Run(args, &stdout, &stderr)
|
||||
if exitCode != 0 {
|
||||
t.Fatalf("expected success, got %d stderr=%q", exitCode, stderr.String())
|
||||
}
|
||||
|
||||
report := readProcessReport(t, reportPath)
|
||||
if report.ReportMetadata.ReportSchemaName != cfg.reportSchemaName || report.ReportMetadata.ReportSchemaVersion != cfg.reportSchemaVersion {
|
||||
t.Fatalf("unexpected report schema metadata: %+v", report.ReportMetadata)
|
||||
}
|
||||
if report.ReportMetadata.OutputSchema != cfg.expectedOutputSchema {
|
||||
t.Fatalf("unexpected output schema metadata: got %q want %q", report.ReportMetadata.OutputSchema, cfg.expectedOutputSchema)
|
||||
}
|
||||
if len(cfg.expectModuleInstances) > 0 {
|
||||
got := make([]string, 0, len(report.ModuleResults))
|
||||
for _, mr := range report.ModuleResults {
|
||||
got = append(got, mr.ModuleInstance)
|
||||
}
|
||||
if !reflect.DeepEqual(got, cfg.expectModuleInstances) {
|
||||
t.Fatalf("unexpected module instances: got %v want %v", got, cfg.expectModuleInstances)
|
||||
}
|
||||
}
|
||||
if report.Diagnostics == nil {
|
||||
t.Fatalf("expected diagnostics metadata")
|
||||
}
|
||||
if cfg.expectUtilizationPaths {
|
||||
if report.Diagnostics.UtilizationSummaryPath == "" || report.Diagnostics.CorrectionLedgerPath == "" {
|
||||
t.Fatalf("expected utilization/ledger artifact paths in report diagnostics: %+v", report.Diagnostics)
|
||||
}
|
||||
}
|
||||
|
||||
runDir := onlyRunDir(t, workDir)
|
||||
if _, err := os.Stat(filepath.Join(runDir, "report.json")); err != nil {
|
||||
t.Fatalf("expected run-dir report: %v", err)
|
||||
}
|
||||
|
||||
if c, ok := processProposalLLMClient.(*fakeStructuredLLMClient); ok {
|
||||
if !reflect.DeepEqual(c.calls, cfg.expectedProposalCalls) {
|
||||
t.Fatalf("unexpected proposal call order: got %v want %v", c.calls, cfg.expectedProposalCalls)
|
||||
}
|
||||
}
|
||||
|
||||
return releaseRunResult{
|
||||
stdout: stdout.String(),
|
||||
stderr: stderr.String(),
|
||||
outputPath: outputPath,
|
||||
reportPath: reportPath,
|
||||
report: report,
|
||||
runDir: runDir,
|
||||
}
|
||||
}
|
||||
|
||||
func readProposalResponses(t *testing.T, path string) []proposal_generation.StructuredCorrectionSet {
|
||||
t.Helper()
|
||||
var out []proposal_generation.StructuredCorrectionSet
|
||||
if err := json.Unmarshal(readFile(t, path), &out); err != nil {
|
||||
t.Fatalf("unmarshal proposal responses: %v", err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func readValidationResponses(t *testing.T, path string) []validators.LLMValidationResponse {
|
||||
t.Helper()
|
||||
var out []validators.LLMValidationResponse
|
||||
if err := json.Unmarshal(readFile(t, path), &out); err != nil {
|
||||
t.Fatalf("unmarshal validation responses: %v", err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func mustReadTranscript(t *testing.T, path string) []schema.Segment {
|
||||
t.Helper()
|
||||
transcript, err := schema.ParseTranscriptJSON(readFile(t, path))
|
||||
if err != nil {
|
||||
t.Fatalf("parse transcript output: %v", err)
|
||||
}
|
||||
return transcript.Segments
|
||||
}
|
||||
|
||||
func expectFinalTranscriptContains(t *testing.T, segments []schema.Segment, needles []string) {
|
||||
t.Helper()
|
||||
joined := flattenTranscriptText(segments)
|
||||
for _, needle := range needles {
|
||||
if !strings.Contains(joined, needle) {
|
||||
t.Fatalf("expected transcript to contain %q, got %q", needle, joined)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func expectFinalTranscriptDoesNotContain(t *testing.T, segments []schema.Segment, needles []string) {
|
||||
t.Helper()
|
||||
joined := flattenTranscriptText(segments)
|
||||
for _, needle := range needles {
|
||||
if strings.Contains(joined, needle) {
|
||||
t.Fatalf("expected transcript to not contain %q, got %q", needle, joined)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func flattenTranscriptText(segments []schema.Segment) string {
|
||||
parts := make([]string, 0, len(segments))
|
||||
for _, s := range segments {
|
||||
parts = append(parts, s.Text)
|
||||
}
|
||||
return strings.Join(parts, "\n")
|
||||
}
|
||||
|
||||
func assertReleaseCounts(t *testing.T, report reporting.ProcessReport, exp releaseFixtureExpectations) {
|
||||
t.Helper()
|
||||
if report.ModulesSummary == nil {
|
||||
t.Fatalf("expected modules_summary")
|
||||
}
|
||||
if report.ModulesSummary.TotalAppliedChanges < exp.MinimumCounts.Applied {
|
||||
t.Fatalf("expected at least %d applied changes, got %+v", exp.MinimumCounts.Applied, report.ModulesSummary)
|
||||
}
|
||||
validatorRejected := 0
|
||||
skipped := 0
|
||||
for _, mr := range report.ModuleResults {
|
||||
validatorRejected += len(mr.ValidatorRejected)
|
||||
skipped += len(mr.SkippedChanges)
|
||||
}
|
||||
if validatorRejected < exp.MinimumCounts.Rejected {
|
||||
t.Fatalf("expected at least %d validator rejections, got %d", exp.MinimumCounts.Rejected, validatorRejected)
|
||||
}
|
||||
if skipped < exp.MinimumCounts.Skipped {
|
||||
t.Fatalf("expected at least %d application skips, got %d", exp.MinimumCounts.Skipped, skipped)
|
||||
}
|
||||
}
|
||||
|
||||
func assertReleaseUtilizationShape(t *testing.T, report reporting.ProcessReport) {
|
||||
t.Helper()
|
||||
var payload struct {
|
||||
EffectiveConcurrency struct {
|
||||
TotalLLM int `json:"total_llm"`
|
||||
} `json:"effective_concurrency"`
|
||||
RunTiming struct {
|
||||
SchedulerQueueWaitMS int64 `json:"scheduler_queue_wait_ms"`
|
||||
LLMExecutionTimeMS int64 `json:"llm_execution_time_ms"`
|
||||
DeterministicValidationMS int64 `json:"deterministic_validation_time_ms"`
|
||||
} `json:"run_timing"`
|
||||
Modules []map[string]any `json:"modules"`
|
||||
Validators []map[string]any `json:"validators"`
|
||||
}
|
||||
if err := json.Unmarshal(readFile(t, report.Diagnostics.UtilizationSummaryPath), &payload); err != nil {
|
||||
t.Fatalf("unmarshal utilization diagnostics: %v", err)
|
||||
}
|
||||
if payload.EffectiveConcurrency.TotalLLM <= 0 {
|
||||
t.Fatalf("expected positive total llm concurrency, got %+v", payload.EffectiveConcurrency)
|
||||
}
|
||||
if payload.RunTiming.SchedulerQueueWaitMS < 0 || payload.RunTiming.LLMExecutionTimeMS < 0 || payload.RunTiming.DeterministicValidationMS < 0 {
|
||||
t.Fatalf("expected non-negative run timing values, got %+v", payload.RunTiming)
|
||||
}
|
||||
if len(payload.Modules) == 0 {
|
||||
t.Fatalf("expected module timing summaries")
|
||||
}
|
||||
if len(payload.Validators) == 0 {
|
||||
t.Fatalf("expected validator timing summaries")
|
||||
}
|
||||
}
|
||||
|
||||
func assertReleaseLedgerShape(t *testing.T, report reporting.ProcessReport) {
|
||||
t.Helper()
|
||||
var entries []struct {
|
||||
ModuleKey string `json:"module_key"`
|
||||
ModuleInstance string `json:"module_instance"`
|
||||
ProposalIndex int `json:"proposal_index"`
|
||||
Disposition string `json:"disposition"`
|
||||
DispositionReason string `json:"disposition_reason_code"`
|
||||
OriginalText string `json:"original_text"`
|
||||
ProposedCorrected string `json:"proposed_corrected_text"`
|
||||
ReplacementPolicy string `json:"replacement_policy"`
|
||||
DeterministicResults []struct {
|
||||
ValidatorKey string `json:"validator_key"`
|
||||
} `json:"deterministic_validator_decisions"`
|
||||
LLMResults []struct {
|
||||
ValidatorKey string `json:"validator_key"`
|
||||
} `json:"llm_validator_decisions"`
|
||||
}
|
||||
if err := json.Unmarshal(readFile(t, report.Diagnostics.CorrectionLedgerPath), &entries); err != nil {
|
||||
t.Fatalf("unmarshal correction ledger: %v", err)
|
||||
}
|
||||
if len(entries) == 0 {
|
||||
t.Fatalf("expected correction ledger entries")
|
||||
}
|
||||
hasApplied := false
|
||||
hasRejected := false
|
||||
hasSkipped := false
|
||||
for _, entry := range entries {
|
||||
if entry.ModuleInstance == "" || entry.ModuleKey == "" {
|
||||
t.Fatalf("expected module identity in ledger entry: %+v", entry)
|
||||
}
|
||||
switch entry.Disposition {
|
||||
case "applied":
|
||||
hasApplied = true
|
||||
case "rejected":
|
||||
hasRejected = true
|
||||
case "skipped":
|
||||
hasSkipped = true
|
||||
}
|
||||
}
|
||||
if !hasApplied || !hasRejected {
|
||||
t.Fatalf("expected applied and rejected entries in correction ledger, got %+v", entries)
|
||||
}
|
||||
if !hasSkipped {
|
||||
// Some deterministic fixture paths do not trigger apply-time skips;
|
||||
// rejections are still captured separately from application skips.
|
||||
}
|
||||
}
|
||||
|
||||
func assertPromptAndSchemaMetadataPresent(t *testing.T, runDir string) {
|
||||
t.Helper()
|
||||
metadataPaths, err := filepath.Glob(filepath.Join(runDir, "*", "*request-metadata.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("glob request metadata artifacts: %v", err)
|
||||
}
|
||||
if len(metadataPaths) == 0 {
|
||||
t.Fatalf("expected request metadata artifacts with prompt metadata")
|
||||
}
|
||||
|
||||
foundPromptMetadata := false
|
||||
foundSchemaMetadata := false
|
||||
for _, path := range metadataPaths {
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(readFile(t, path), &payload); err != nil {
|
||||
t.Fatalf("unmarshal request metadata artifact %q: %v", path, err)
|
||||
}
|
||||
if pm, ok := payload["prompt_metadata"].(map[string]any); ok {
|
||||
if pm["prompt_id"] != nil && pm["prompt_version"] != nil && pm["sha256"] != nil {
|
||||
foundPromptMetadata = true
|
||||
}
|
||||
}
|
||||
if sm, ok := payload["response_schema"].(map[string]any); ok {
|
||||
if sm["id"] != nil && sm["version"] != nil && sm["name"] != nil && sm["sha256"] != nil {
|
||||
foundSchemaMetadata = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if !foundPromptMetadata {
|
||||
t.Fatalf("expected prompt metadata in request metadata artifacts")
|
||||
}
|
||||
if !foundSchemaMetadata {
|
||||
t.Fatalf("expected structured response schema metadata in request metadata artifacts")
|
||||
}
|
||||
}
|
||||
|
||||
func assertStableValidatorKeysPresent(t *testing.T, report reporting.ProcessReport) {
|
||||
t.Helper()
|
||||
seen := map[string]bool{}
|
||||
for _, module := range report.ModuleResults {
|
||||
for _, decision := range module.ValidatorDecisions {
|
||||
seen[decision.ValidatorName] = true
|
||||
}
|
||||
for _, rejected := range module.ValidatorRejected {
|
||||
seen[rejected.ValidatorName] = true
|
||||
}
|
||||
}
|
||||
expectedAny := []string{
|
||||
"confidence_threshold",
|
||||
"original_text_presence",
|
||||
"no_effect",
|
||||
}
|
||||
for _, key := range expectedAny {
|
||||
if !seen[key] {
|
||||
t.Fatalf("expected stable validator key %q in report decisions/rejections; seen=%v", key, seen)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func assertStdoutStderrContract(t *testing.T, stdout, stderr string) {
|
||||
t.Helper()
|
||||
if stdout != "" {
|
||||
t.Fatalf("expected empty stdout with --output, got %q", stdout)
|
||||
}
|
||||
if strings.Contains(stderr, `"module_results"`) || strings.Contains(stderr, `"report_metadata"`) {
|
||||
t.Fatalf("stderr should remain human-readable, not report JSON: %q", stderr)
|
||||
}
|
||||
}
|
||||
|
||||
func assertAuditaV1OutputShape(t *testing.T, outputPath string) {
|
||||
t.Helper()
|
||||
var payload struct {
|
||||
Schema string `json:"schema"`
|
||||
Version string `json:"version"`
|
||||
Segments []schema.Segment `json:"segments"`
|
||||
}
|
||||
if err := json.Unmarshal(readFile(t, outputPath), &payload); err != nil {
|
||||
t.Fatalf("unmarshal audita-v1 output: %v", err)
|
||||
}
|
||||
if payload.Schema != "audita-v1" || payload.Version != "v1" {
|
||||
t.Fatalf("unexpected audita-v1 metadata: %+v", payload)
|
||||
}
|
||||
if len(payload.Segments) == 0 {
|
||||
t.Fatalf("expected non-empty audita-v1 segments")
|
||||
}
|
||||
}
|
||||
@@ -2,292 +2,13 @@ package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/chunking"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/diagnostics"
|
||||
coreio "gitea.maximumdirect.net/eric/audita/internal/core/io"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/normalization"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/outputschema"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/reporting"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/modules"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/processreport"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/runner"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||
)
|
||||
|
||||
type noOpStructuredLLMClient struct{}
|
||||
|
||||
func (c noOpStructuredLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
_ = ctx
|
||||
_ = req
|
||||
switch target := out.(type) {
|
||||
case *validators.LLMValidationResponse:
|
||||
*target = validators.LLMValidationResponse{Validations: []validators.LLMValidationDecision{}}
|
||||
case *proposal_generation.StructuredCorrectionSet:
|
||||
*target = proposal_generation.StructuredCorrectionSet{Corrections: nil}
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{}, nil
|
||||
}
|
||||
|
||||
func shouldUseNoOpLLMClientForTests() bool {
|
||||
return strings.HasSuffix(filepath.Base(os.Args[0]), ".test") || os.Getenv("GO_WANT_HELPER_PROCESS") == "1"
|
||||
}
|
||||
|
||||
type processInvocation struct {
|
||||
TranscriptPath string
|
||||
GlossaryPath string
|
||||
OutputPath string
|
||||
ReportJSONPath string
|
||||
Config config.Config
|
||||
ConfigPath string
|
||||
ConfigSource string
|
||||
ConfigVersion *int
|
||||
ExplicitModules bool
|
||||
}
|
||||
|
||||
var processModuleFactory runner.ModuleFactory
|
||||
var processProposalLLMClient contracts.StructuredLLMClient
|
||||
var processProposalLLMScheduler runner.ValidationScheduler
|
||||
var processValidationLLMClient contracts.StructuredLLMClient
|
||||
var processValidationLLMScheduler runner.ValidationScheduler
|
||||
var processRunnerContext = func() (context.Context, context.CancelFunc) {
|
||||
return context.Background(), func() {}
|
||||
}
|
||||
|
||||
var processRunner = func(inv processInvocation, stdout io.Writer) (*normalization.NormalizationSummary, *chunking.Summary, *runner.RunOutput, *diagnostics.RunDirectory, error) {
|
||||
runDir, err := diagnostics.NewRunDirectory(inv.Config.WorkDir, string(inv.Config.WorkDirRetention))
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, fmt.Errorf("run_dir_creation: %w", err)
|
||||
}
|
||||
|
||||
fail := func(phase string, err error, runOutput *runner.RunOutput) (*normalization.NormalizationSummary, *chunking.Summary, *runner.RunOutput, *diagnostics.RunDirectory, error) {
|
||||
_ = runDir.WriteErrorLog(fmt.Sprintf("%s: %v", phase, err))
|
||||
return nil, nil, runOutput, runDir, fmt.Errorf("%s: %w", phase, err)
|
||||
}
|
||||
|
||||
if err := runDir.WriteInvocationMetadata(diagnostics.InvocationMetadata{
|
||||
Operation: "process",
|
||||
TranscriptPath: inv.TranscriptPath,
|
||||
GlossaryPath: inv.GlossaryPath,
|
||||
OutputPath: inv.OutputPath,
|
||||
ReportJSONPath: inv.ReportJSONPath,
|
||||
ConfigPath: inv.ConfigPath,
|
||||
ConfigSource: inv.ConfigSource,
|
||||
ConfigVersion: inv.ConfigVersion,
|
||||
TranscriptDescription: inv.Config.TranscriptDescription,
|
||||
Modules: append([]string(nil), inv.Config.Modules...),
|
||||
}); err != nil {
|
||||
_ = runDir.WriteErrorLog(fmt.Sprintf("invocation_metadata: %v", err))
|
||||
}
|
||||
if err := runDir.WriteEffectiveConfig(inv.Config); err != nil {
|
||||
_ = runDir.WriteErrorLog(fmt.Sprintf("effective_config: %v", err))
|
||||
}
|
||||
|
||||
transcriptBytes, err := coreio.ReadRequiredFile(inv.TranscriptPath, "transcript")
|
||||
if err != nil {
|
||||
return fail("transcript_read", err, nil)
|
||||
}
|
||||
|
||||
glossaryBytes, err := coreio.ReadRequiredFile(inv.GlossaryPath, "glossary")
|
||||
if err != nil {
|
||||
return fail("glossary_read", err, nil)
|
||||
}
|
||||
|
||||
sourceTranscript, err := schema.ParseSourceTranscriptJSON(transcriptBytes)
|
||||
if err != nil {
|
||||
return fail("transcript_schema", err, nil)
|
||||
}
|
||||
|
||||
glossary, err := schema.ParseGlossaryYAML(glossaryBytes)
|
||||
if err != nil {
|
||||
return fail("glossary_schema", err, nil)
|
||||
}
|
||||
|
||||
if err := runDir.WriteSourceTranscript(sourceTranscript, transcriptBytes); err != nil {
|
||||
_ = runDir.WriteErrorLog(fmt.Sprintf("source_artifact: %v", err))
|
||||
}
|
||||
|
||||
canonical := sourceToCanonicalTranscript(sourceTranscript)
|
||||
normalizer := normalization.NewNormalizer(normalization.NormalizationConfig{
|
||||
MaxSegmentGap: inv.Config.Normalization.MaxSegmentGap,
|
||||
EllipsisGap: inv.Config.Normalization.EllipsisGap,
|
||||
MaxSegmentDuration: inv.Config.Normalization.MaxSegmentDuration,
|
||||
MaxSegmentTokens: inv.Config.Normalization.MaxSegmentTokens,
|
||||
})
|
||||
|
||||
normalizedTranscript, normSummary := normalizer.Normalize(canonical)
|
||||
|
||||
if err := runDir.WriteNormalizedTranscript(normalizedTranscript); err != nil {
|
||||
_ = runDir.WriteErrorLog(fmt.Sprintf("normalized_artifact: %v", err))
|
||||
}
|
||||
if err := runDir.WriteNormalizationSummary(normSummary); err != nil {
|
||||
_ = runDir.WriteErrorLog(fmt.Sprintf("normalization_summary: %v", err))
|
||||
}
|
||||
|
||||
// Compute chunks after normalization
|
||||
chunker := chunking.NewChunker(chunking.ChunkingConfig{
|
||||
MaxSectionTokens: inv.Config.MaxSectionTokens,
|
||||
MinSectionTokens: inv.Config.MinSectionTokens,
|
||||
TargetSections: inv.Config.TargetSections,
|
||||
})
|
||||
|
||||
sections, chunkErr := chunker.ChunkTranscript(normalizedTranscript)
|
||||
if chunkErr != nil {
|
||||
return fail("chunking", chunkErr, nil)
|
||||
}
|
||||
|
||||
chunkConfig := chunking.ChunkingConfig{
|
||||
MaxSectionTokens: inv.Config.MaxSectionTokens,
|
||||
MinSectionTokens: inv.Config.MinSectionTokens,
|
||||
TargetSections: inv.Config.TargetSections,
|
||||
}
|
||||
chunkSummary := chunking.ComputeSummary(sections, chunkConfig)
|
||||
chunkDetailedSummary := chunking.ComputeDetailedSummary(sections, chunkConfig)
|
||||
|
||||
if err := runDir.WriteChunkingSummary(&chunkDetailedSummary); err != nil {
|
||||
_ = runDir.WriteErrorLog(fmt.Sprintf("chunking_summary: %v", err))
|
||||
}
|
||||
|
||||
workingTranscript := normalizedTranscript
|
||||
var runOutput *runner.RunOutput
|
||||
moduleFactory := processModuleFactory
|
||||
if moduleFactory == nil {
|
||||
moduleFactory = modules.NewFactory(modules.Dependencies{
|
||||
Config: &inv.Config,
|
||||
Glossary: glossary,
|
||||
DiagnosticsDir: runDir.Path(),
|
||||
})
|
||||
}
|
||||
if moduleFactory != nil {
|
||||
proposalLLMClient := processProposalLLMClient
|
||||
validationLLMClient := processValidationLLMClient
|
||||
proposalScheduler := processProposalLLMScheduler
|
||||
validationScheduler := processValidationLLMScheduler
|
||||
|
||||
if processModuleFactory == nil {
|
||||
// Production runtime path: construct clients/schedulers from config.
|
||||
if proposalLLMClient == nil {
|
||||
if shouldUseNoOpLLMClientForTests() {
|
||||
proposalLLMClient = noOpStructuredLLMClient{}
|
||||
} else {
|
||||
primaryCfg := llm.ResolvePrimaryConfig(inv.Config)
|
||||
client, clientErr := llm.NewOpenAICompatibleClient(primaryCfg.ToOpenAICompatibleClientConfig(nil))
|
||||
if clientErr != nil {
|
||||
return fail("runner_setup", clientErr, nil)
|
||||
}
|
||||
proposalLLMClient = client
|
||||
}
|
||||
}
|
||||
if validationLLMClient == nil {
|
||||
if shouldUseNoOpLLMClientForTests() {
|
||||
validationLLMClient = noOpStructuredLLMClient{}
|
||||
} else {
|
||||
validationCfg := llm.ResolveValidationConfig(inv.Config)
|
||||
client, clientErr := llm.NewOpenAICompatibleClient(validationCfg.ToOpenAICompatibleClientConfig(nil))
|
||||
if clientErr != nil {
|
||||
return fail("runner_setup", clientErr, nil)
|
||||
}
|
||||
validationLLMClient = client
|
||||
}
|
||||
}
|
||||
globalScheduler := proposalScheduler
|
||||
if globalScheduler == nil {
|
||||
s, sErr := llm.NewScheduler(inv.Config.TotalLLMConcurrency)
|
||||
if sErr != nil {
|
||||
return fail("runner_setup", sErr, nil)
|
||||
}
|
||||
globalScheduler = s
|
||||
}
|
||||
if proposalScheduler == nil {
|
||||
proposalScheduler = globalScheduler
|
||||
if inv.Config.EffectiveProposalLLMConcurrency() < inv.Config.TotalLLMConcurrency {
|
||||
s, sErr := llm.NewScheduler(inv.Config.EffectiveProposalLLMConcurrency())
|
||||
if sErr != nil {
|
||||
return fail("runner_setup", sErr, nil)
|
||||
}
|
||||
proposalScheduler = composeSchedulers(globalScheduler, s)
|
||||
}
|
||||
}
|
||||
if validationScheduler == nil {
|
||||
validationScheduler = globalScheduler
|
||||
if inv.Config.ValidationLLMConcurrency != nil && inv.Config.EffectiveValidationLLMConcurrency() < inv.Config.TotalLLMConcurrency {
|
||||
s, sErr := llm.NewScheduler(inv.Config.EffectiveValidationLLMConcurrency())
|
||||
if sErr != nil {
|
||||
return fail("runner_setup", sErr, nil)
|
||||
}
|
||||
validationScheduler = composeSchedulers(globalScheduler, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
moduleSpecs, err := contracts.ResolveModuleRunSpecs(inv.Config.Modules)
|
||||
if err != nil {
|
||||
return fail("runner_setup", err, nil)
|
||||
}
|
||||
|
||||
runCtx, cancelRun := processRunnerContext()
|
||||
defer cancelRun()
|
||||
runnerResult, runErr := runner.New(moduleFactory).Run(runCtx, runner.RunInput{
|
||||
Config: &inv.Config,
|
||||
Transcript: normalizedTranscript,
|
||||
Glossary: glossary,
|
||||
ModuleSpecs: moduleSpecs,
|
||||
EffectiveConcurrency: runner.EffectiveConcurrencyLimits{
|
||||
TotalLLM: inv.Config.TotalLLMConcurrency,
|
||||
ProposalLLM: inv.Config.EffectiveProposalLLMConcurrency(),
|
||||
ValidationLLM: inv.Config.EffectiveValidationLLMConcurrency(),
|
||||
},
|
||||
ProposalLLMClient: proposalLLMClient,
|
||||
ProposalLLMScheduler: proposalScheduler,
|
||||
ProposalDiagnosticsDir: runDir.Path(),
|
||||
ValidationLLMClient: validationLLMClient,
|
||||
ValidationLLMScheduler: validationScheduler,
|
||||
ValidationDiagnosticsDir: runDir.Path(),
|
||||
})
|
||||
runOutput = &runnerResult
|
||||
if runErr != nil {
|
||||
return fail("runner_execution", runErr, runOutput)
|
||||
}
|
||||
workingTranscript = runnerResult.FinalTranscript
|
||||
}
|
||||
|
||||
encoderDef, err := outputschema.Resolve(inv.Config.OutputSchema)
|
||||
if err != nil {
|
||||
return fail("output_schema", err, runOutput)
|
||||
}
|
||||
outputBytes, err := encoderDef.Encoder(workingTranscript)
|
||||
if err != nil {
|
||||
return fail("serialization", err, runOutput)
|
||||
}
|
||||
|
||||
if strings.TrimSpace(inv.OutputPath) != "" {
|
||||
if err := coreio.WriteFile(inv.OutputPath, outputBytes); err != nil {
|
||||
return fail("output_write", err, runOutput)
|
||||
}
|
||||
return normSummary, &chunkSummary, runOutput, runDir, nil
|
||||
}
|
||||
|
||||
if _, err := stdout.Write(outputBytes); err != nil {
|
||||
return fail("stdout_write", err, runOutput)
|
||||
}
|
||||
|
||||
return normSummary, &chunkSummary, runOutput, runDir, nil
|
||||
}
|
||||
|
||||
func sourceToCanonicalTranscript(source *schema.SourceTranscript) *schema.Transcript {
|
||||
segments := make([]schema.Segment, len(source.Segments))
|
||||
for i, s := range source.Segments {
|
||||
@@ -307,48 +28,22 @@ func sourceToCanonicalTranscript(source *schema.SourceTranscript) *schema.Transc
|
||||
return &schema.Transcript{Segments: segments}
|
||||
}
|
||||
|
||||
type chainedScheduler struct {
|
||||
schedulers []runner.ValidationScheduler
|
||||
}
|
||||
|
||||
func (s chainedScheduler) Run(ctx context.Context, fn func(context.Context) error) error {
|
||||
if len(s.schedulers) == 0 {
|
||||
return fn(ctx)
|
||||
}
|
||||
run := fn
|
||||
for i := len(s.schedulers) - 1; i >= 0; i-- {
|
||||
scheduler := s.schedulers[i]
|
||||
next := run
|
||||
run = func(callCtx context.Context) error {
|
||||
return scheduler.Run(callCtx, next)
|
||||
}
|
||||
}
|
||||
return run(ctx)
|
||||
}
|
||||
|
||||
func composeSchedulers(schedulers ...runner.ValidationScheduler) runner.ValidationScheduler {
|
||||
filtered := make([]runner.ValidationScheduler, 0, len(schedulers))
|
||||
for _, scheduler := range schedulers {
|
||||
if scheduler != nil {
|
||||
filtered = append(filtered, scheduler)
|
||||
}
|
||||
}
|
||||
switch len(filtered) {
|
||||
case 0:
|
||||
return nil
|
||||
case 1:
|
||||
return filtered[0]
|
||||
default:
|
||||
return chainedScheduler{schedulers: filtered}
|
||||
}
|
||||
}
|
||||
|
||||
// Run executes the Audita CLI with the provided arguments and streams.
|
||||
func Run(args []string, stdout, stderr io.Writer) int {
|
||||
func Run(ctx context.Context, args []string, stdout, stderr io.Writer) int {
|
||||
if len(args) == 0 {
|
||||
writeRootUsage(stdout)
|
||||
return 0
|
||||
}
|
||||
if len(args) == 2 && args[0] == "help" {
|
||||
switch args[1] {
|
||||
case "process":
|
||||
writeProcessUsage(stdout)
|
||||
return 0
|
||||
case "config":
|
||||
writeConfigUsage(stdout)
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
if isHelpCommand(args) {
|
||||
writeRootUsage(stdout)
|
||||
@@ -356,10 +51,10 @@ func Run(args []string, stdout, stderr io.Writer) int {
|
||||
}
|
||||
|
||||
if args[0] == "process" {
|
||||
return runProcess(args[1:], stdout, stderr)
|
||||
return runProcess(ctx, args[1:], stdout, stderr)
|
||||
}
|
||||
if args[0] == "config" {
|
||||
return runConfig(args[1:], stdout, stderr)
|
||||
return runConfig(ctx, args[1:], stdout, stderr)
|
||||
}
|
||||
|
||||
fmt.Fprintf(stderr, "audita: unknown command %q\n\n", args[0])
|
||||
@@ -367,180 +62,17 @@ func Run(args []string, stdout, stderr io.Writer) int {
|
||||
return 2
|
||||
}
|
||||
|
||||
func runProcess(args []string, stdout, stderr io.Writer) int {
|
||||
startedAt := time.Now().UTC()
|
||||
|
||||
configPathOverride, configPathOverrideSet, err := findConfigPathOverride(args)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: invalid CLI configuration: %v\n", err)
|
||||
return 2
|
||||
}
|
||||
effectiveConfig, err := config.LoadEffectiveConfig(configPathOverride, configPathOverrideSet)
|
||||
if err != nil {
|
||||
var effectiveConfigErr *config.EffectiveConfigError
|
||||
if errors.As(err, &effectiveConfigErr) {
|
||||
switch effectiveConfigErr.Kind {
|
||||
case config.EffectiveConfigErrorLoadFile, config.EffectiveConfigErrorApplyFile:
|
||||
fmt.Fprintf(stderr, "audita process: invalid config file: %v\n", effectiveConfigErr)
|
||||
case config.EffectiveConfigErrorApplyEnv:
|
||||
fmt.Fprintf(stderr, "audita process: invalid environment configuration: %v\n", effectiveConfigErr)
|
||||
default:
|
||||
fmt.Fprintf(stderr, "audita process: %v\n", effectiveConfigErr)
|
||||
}
|
||||
} else {
|
||||
fmt.Fprintf(stderr, "audita process: %v\n", err)
|
||||
}
|
||||
return 2
|
||||
}
|
||||
|
||||
cfg := effectiveConfig.Config
|
||||
configPath := effectiveConfig.ConfigPath
|
||||
configSource := effectiveConfig.ConfigSource
|
||||
configVersion := effectiveConfig.ConfigVersion
|
||||
|
||||
fs, pFlags := newProcessFlagSet(cfg, stderr)
|
||||
|
||||
if isHelpCommand(args) || hasHelpFlag(args) {
|
||||
writeProcessUsage(stdout, fs)
|
||||
return 0
|
||||
}
|
||||
|
||||
parseArgs := args
|
||||
transcriptFromFront := ""
|
||||
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
|
||||
transcriptFromFront = args[0]
|
||||
parseArgs = args[1:]
|
||||
}
|
||||
|
||||
if err := fs.Parse(parseArgs); err != nil {
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
writeProcessUsage(stdout, fs)
|
||||
return 0
|
||||
}
|
||||
return 2
|
||||
}
|
||||
|
||||
overrides, explicitModules := processCLIOverrides(fs, pFlags)
|
||||
|
||||
if err := cfg.ApplyCLIOverrides(overrides); err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: invalid CLI configuration: %v\n", err)
|
||||
return 2
|
||||
}
|
||||
|
||||
remaining := fs.Args()
|
||||
positional := make([]string, 0, len(remaining)+1)
|
||||
if transcriptFromFront != "" {
|
||||
positional = append(positional, transcriptFromFront)
|
||||
}
|
||||
positional = append(positional, remaining...)
|
||||
|
||||
if len(positional) != 1 {
|
||||
fmt.Fprintln(stderr, "audita process: expected exactly 1 transcript JSON path argument")
|
||||
return 2
|
||||
}
|
||||
if strings.TrimSpace(*pFlags.glossaryPath) == "" {
|
||||
fmt.Fprintln(stderr, "audita process: --glossary is required")
|
||||
return 2
|
||||
}
|
||||
|
||||
inv := processInvocation{
|
||||
TranscriptPath: positional[0],
|
||||
GlossaryPath: *pFlags.glossaryPath,
|
||||
OutputPath: *pFlags.outputPath,
|
||||
ReportJSONPath: *pFlags.reportJSONPath,
|
||||
Config: cfg,
|
||||
ConfigPath: configPath,
|
||||
ConfigSource: configSource,
|
||||
ConfigVersion: configVersion,
|
||||
ExplicitModules: explicitModules,
|
||||
}
|
||||
|
||||
normSummary, chunkSummary, runOutput, runDir, runErr := processRunner(inv, stdout)
|
||||
completedAt := time.Now().UTC()
|
||||
|
||||
if runErr != nil {
|
||||
if runDir != nil && runOutput != nil {
|
||||
if runOutput.Utilization != nil {
|
||||
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactUtilizationSummary, runOutput.Utilization)
|
||||
}
|
||||
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactCorrectionLedger, processreport.BuildCorrectionLedger(processreport.CorrectionLedgerInput{
|
||||
RunDirectoryPath: runDir.Path(),
|
||||
RunOutput: runOutput,
|
||||
}))
|
||||
}
|
||||
errorPhase, errorMessage := extractErrorPhase(runErr)
|
||||
report := processreport.Build(processReportInput("failed", inv, runDir, startedAt, completedAt, errorMessage, errorPhase, nil, nil, runOutput))
|
||||
|
||||
if strings.TrimSpace(inv.ReportJSONPath) != "" {
|
||||
if err := reporting.WriteProcessReport(inv.ReportJSONPath, report); err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: %v\n", err)
|
||||
}
|
||||
}
|
||||
|
||||
if runDir != nil {
|
||||
_ = runDir.WriteReport(report)
|
||||
_ = runDir.ApplyRetention(diagnostics.RetentionDecisionInput{
|
||||
RunSucceeded: false,
|
||||
})
|
||||
}
|
||||
|
||||
fmt.Fprintf(stderr, "audita process: %v\n", runErr)
|
||||
if runDir != nil {
|
||||
fmt.Fprintf(stderr, "audita process: diagnostics: %s\n", runDir.Path())
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
if runDir != nil && runOutput != nil {
|
||||
if runOutput.Utilization != nil {
|
||||
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactUtilizationSummary, runOutput.Utilization)
|
||||
}
|
||||
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactCorrectionLedger, processreport.BuildCorrectionLedger(processreport.CorrectionLedgerInput{
|
||||
RunDirectoryPath: runDir.Path(),
|
||||
RunOutput: runOutput,
|
||||
}))
|
||||
}
|
||||
|
||||
report := processreport.Build(processReportInput("success", inv, runDir, startedAt, completedAt, "", "", normSummary, chunkSummary, runOutput))
|
||||
|
||||
if strings.TrimSpace(inv.ReportJSONPath) != "" {
|
||||
if err := reporting.WriteProcessReport(inv.ReportJSONPath, report); err != nil {
|
||||
if runDir != nil {
|
||||
_ = runDir.WriteErrorLog(fmt.Sprintf("report_write: %v", err))
|
||||
_ = runDir.ApplyRetention(diagnostics.RetentionDecisionInput{
|
||||
RunSucceeded: false,
|
||||
})
|
||||
}
|
||||
fmt.Fprintf(stderr, "audita process: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
if runDir != nil {
|
||||
_ = runDir.WriteReport(report)
|
||||
if err := runDir.ApplyRetention(diagnostics.RetentionDecisionInput{
|
||||
RunSucceeded: true,
|
||||
HasSkippedCorrections: processreport.HasSkippedCorrections(runOutput),
|
||||
}); err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: failed to apply work-dir retention: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
func runConfig(args []string, stdout, stderr io.Writer) int {
|
||||
if len(args) == 0 || isHelpCommand(args) || hasHelpFlag(args) {
|
||||
func runConfig(ctx context.Context, args []string, stdout, stderr io.Writer) int {
|
||||
if len(args) == 0 || isHelpCommand(args) || args[0] == "--help" || args[0] == "-h" {
|
||||
writeConfigUsage(stdout)
|
||||
return 0
|
||||
}
|
||||
|
||||
switch args[0] {
|
||||
case "validate":
|
||||
return runConfigValidate(args[1:], stdout, stderr)
|
||||
return runConfigValidate(ctx, args[1:], stdout, stderr)
|
||||
case "print-effective":
|
||||
return runConfigPrintEffective(args[1:], stdout, stderr)
|
||||
return runConfigPrintEffective(ctx, args[1:], stdout, stderr)
|
||||
default:
|
||||
fmt.Fprintf(stderr, "audita config: unknown command %q\n\n", args[0])
|
||||
writeConfigUsage(stderr)
|
||||
@@ -548,242 +80,75 @@ func runConfig(args []string, stdout, stderr io.Writer) int {
|
||||
}
|
||||
}
|
||||
|
||||
func runConfigValidate(args []string, stdout, stderr io.Writer) int {
|
||||
fs := flag.NewFlagSet("config validate", flag.ContinueOnError)
|
||||
fs.SetOutput(stderr)
|
||||
configPath := fs.String("config", "", "Path to versioned YAML config file")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
writeConfigValidateUsage(stdout)
|
||||
return 0
|
||||
}
|
||||
return 2
|
||||
}
|
||||
if strings.TrimSpace(*configPath) == "" {
|
||||
fmt.Fprintln(stderr, "audita config validate: --config is required")
|
||||
return 2
|
||||
}
|
||||
if len(fs.Args()) != 0 {
|
||||
fmt.Fprintln(stderr, "audita config validate: unexpected positional arguments")
|
||||
return 2
|
||||
}
|
||||
|
||||
fileCfg, err := config.LoadFileConfig(strings.TrimSpace(*configPath))
|
||||
func runConfigValidate(ctx context.Context, args []string, stdout, stderr io.Writer) int {
|
||||
input, help, err := parseConfigInput("validate", args, stderr)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita config validate: %v\n", err)
|
||||
fmt.Fprintf(stderr, "audita config validate: %s\n", configInputError(err))
|
||||
return 2
|
||||
}
|
||||
cfg := config.Default()
|
||||
if err := cfg.ApplyFileConfig(fileCfg); err != nil {
|
||||
fmt.Fprintf(stderr, "audita config validate: %v\n", err)
|
||||
if help {
|
||||
writeConfigValidateUsage(stdout)
|
||||
return 0
|
||||
}
|
||||
selected, err := loadConfigInput(input, "validate")
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita config validate: %s\n", configInputError(err))
|
||||
return 2
|
||||
}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
fmt.Fprintf(stderr, "audita config validate: %v\n", err)
|
||||
if _, err := llm.NewRuntime(ctx, selected.Result.Config, selected.Path, nil); err != nil {
|
||||
fmt.Fprintf(stderr, "audita config validate: %s\n", llm.RedactError(err, selected.Result.Config))
|
||||
if llm.IsAssetError(err) {
|
||||
return 1
|
||||
}
|
||||
return 2
|
||||
}
|
||||
fmt.Fprintln(stdout, "config is valid")
|
||||
return 0
|
||||
}
|
||||
|
||||
func runConfigPrintEffective(args []string, stdout, stderr io.Writer) int {
|
||||
fs := flag.NewFlagSet("config print-effective", flag.ContinueOnError)
|
||||
fs.SetOutput(stderr)
|
||||
configPath := fs.String("config", "", "Path to versioned YAML config file")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
writeConfigPrintEffectiveUsage(stdout)
|
||||
return 0
|
||||
func runConfigPrintEffective(ctx context.Context, args []string, stdout, stderr io.Writer) int {
|
||||
input, help, err := parseConfigInput("print-effective", args, stderr)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita config print-effective: %s\n", configInputError(err))
|
||||
return 2
|
||||
}
|
||||
if help {
|
||||
writeConfigPrintEffectiveUsage(stdout)
|
||||
return 0
|
||||
}
|
||||
selected, err := loadConfigInput(input, "print-effective")
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita config print-effective: %s\n", configInputError(err))
|
||||
return 2
|
||||
}
|
||||
llmRuntime, err := llm.NewRuntime(ctx, selected.Result.Config, selected.Path, nil)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita config print-effective: %s\n", llm.RedactError(err, selected.Result.Config))
|
||||
if llm.IsAssetError(err) {
|
||||
return 1
|
||||
}
|
||||
return 2
|
||||
}
|
||||
if len(fs.Args()) != 0 {
|
||||
fmt.Fprintln(stderr, "audita config print-effective: unexpected positional arguments")
|
||||
return 2
|
||||
if input.View == "settings" {
|
||||
err = selected.Result.WriteEffective(stdout)
|
||||
} else {
|
||||
var output []byte
|
||||
if input.View == "profiles" {
|
||||
output, err = llmRuntime.SelectedProfilesYAML()
|
||||
} else {
|
||||
output, err = llm.ExplanationYAML(selected.Result)
|
||||
}
|
||||
if err == nil {
|
||||
_, err = stdout.Write(output)
|
||||
}
|
||||
}
|
||||
|
||||
configPathValue := strings.TrimSpace(*configPath)
|
||||
configPathSet := configPathValue != ""
|
||||
effectiveConfig, err := config.LoadEffectiveConfig(configPathValue, configPathSet)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita config print-effective: %v\n", err)
|
||||
return 2
|
||||
}
|
||||
|
||||
cfg := effectiveConfig.Config
|
||||
|
||||
redacted := cfg.Redacted()
|
||||
out, err := json.MarshalIndent(redacted, "", " ")
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "audita config print-effective: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
out = append(out, '\n')
|
||||
if _, err := stdout.Write(out); err != nil {
|
||||
fmt.Fprintf(stderr, "audita config print-effective: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func extractErrorPhase(err error) (phase string, message string) {
|
||||
msg := err.Error()
|
||||
if strings.Contains(msg, ": ") {
|
||||
parts := strings.SplitN(msg, ": ", 2)
|
||||
if len(parts) == 2 {
|
||||
return parts[0], parts[1]
|
||||
}
|
||||
}
|
||||
return "", msg
|
||||
}
|
||||
|
||||
func processReportInput(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) processreport.BuildInput {
|
||||
runDirectoryPath := ""
|
||||
if runDir != nil {
|
||||
runDirectoryPath = runDir.Path()
|
||||
}
|
||||
return processreport.BuildInput{
|
||||
Status: status,
|
||||
TranscriptPath: inv.TranscriptPath,
|
||||
GlossaryPath: inv.GlossaryPath,
|
||||
OutputPath: inv.OutputPath,
|
||||
Modules: inv.Config.Modules,
|
||||
OutputSchema: inv.Config.OutputSchema,
|
||||
ConfigVersion: inv.ConfigVersion,
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: completedAt,
|
||||
ErrorMessage: errorMessage,
|
||||
ErrorPhase: errorPhase,
|
||||
RunDirectoryPath: runDirectoryPath,
|
||||
NormalizationSummary: normalizationSummary,
|
||||
ChunkingSummary: chunkingSummary,
|
||||
RunOutput: runOutput,
|
||||
}
|
||||
}
|
||||
|
||||
type processFlags struct {
|
||||
configPath *string
|
||||
glossaryPath *string
|
||||
outputPath *string
|
||||
reportJSONPath *string
|
||||
modules *string
|
||||
outputSchema *string
|
||||
llmAPIKey *string
|
||||
validationLLMAPIKey *string
|
||||
model *string
|
||||
validationModel *string
|
||||
baseURL *string
|
||||
validationBaseURL *string
|
||||
llmTimeoutSeconds *int
|
||||
totalLLMConcurrency *int
|
||||
proposalLLMConcurrency *int
|
||||
llmConcurrency *int
|
||||
validationLLMTimeoutSeconds *int
|
||||
validationMaxPromptTokens *int
|
||||
targetSections *int
|
||||
maxRetries *int
|
||||
validationMaxRetries *int
|
||||
validationLLMConcurrency *int
|
||||
maxSectionTokens *int
|
||||
minSectionTokens *int
|
||||
glossaryConfidenceThreshold *float64
|
||||
grammarConfidenceThreshold *float64
|
||||
homophonesConfidenceThreshold *float64
|
||||
spokenWordConfidenceThreshold *float64
|
||||
normalizeMaxSegmentGap *float64
|
||||
normalizeEllipsisGap *float64
|
||||
normalizeMaxSegmentDuration *float64
|
||||
normalizeMaxSegmentTokens *int
|
||||
transcriptDescription *string
|
||||
workDir *string
|
||||
workDirRetention *string
|
||||
}
|
||||
|
||||
func newProcessFlagSet(cfg config.Config, stderr io.Writer) (*flag.FlagSet, processFlags) {
|
||||
fs := flag.NewFlagSet("process", flag.ContinueOnError)
|
||||
fs.SetOutput(stderr)
|
||||
|
||||
validationTimeoutSecondsDefault := cfg.PrimaryLLM.TimeoutSeconds
|
||||
if cfg.ValidationLLM.TimeoutSeconds != nil {
|
||||
validationTimeoutSecondsDefault = *cfg.ValidationLLM.TimeoutSeconds
|
||||
}
|
||||
|
||||
validationMaxRetriesDefault := cfg.PrimaryLLM.MaxRetries
|
||||
if cfg.ValidationLLM.MaxRetries != nil {
|
||||
validationMaxRetriesDefault = *cfg.ValidationLLM.MaxRetries
|
||||
}
|
||||
|
||||
validationLLMConcurrencyDefault := cfg.TotalLLMConcurrency
|
||||
if cfg.ValidationLLMConcurrency != nil {
|
||||
validationLLMConcurrencyDefault = *cfg.ValidationLLMConcurrency
|
||||
}
|
||||
|
||||
targetSectionsDefault := 0
|
||||
if cfg.TargetSections != nil {
|
||||
targetSectionsDefault = *cfg.TargetSections
|
||||
}
|
||||
|
||||
pFlags := processFlags{
|
||||
configPath: fs.String("config", "", "Path to versioned YAML config file"),
|
||||
glossaryPath: fs.String("glossary", "", "Path to glossary YAML file"),
|
||||
outputPath: fs.String("output", "", "Path to corrected transcript JSON output file"),
|
||||
reportJSONPath: fs.String("report-json", "", "Path to machine-readable report JSON output file"),
|
||||
modules: fs.String("modules", strings.Join(cfg.Modules, ","), "Comma-separated module sequence override"),
|
||||
outputSchema: fs.String("output-schema", cfg.OutputSchema, "Output schema: bare-segments|audita-v1"),
|
||||
llmAPIKey: fs.String("llm-api-key", cfg.PrimaryLLM.APIKey, "Primary LLM API key"),
|
||||
validationLLMAPIKey: fs.String("validation-llm-api-key", cfg.ValidationLLM.APIKey, "Validation LLM API key"),
|
||||
model: fs.String("model", cfg.PrimaryLLM.Model, "Primary LLM model name"),
|
||||
validationModel: fs.String("validation-model", cfg.ValidationLLM.Model, "Validation LLM model name"),
|
||||
baseURL: fs.String("base-url", cfg.PrimaryLLM.BaseURL, "Primary OpenAI-compatible base URL"),
|
||||
validationBaseURL: fs.String("validation-base-url", cfg.ValidationLLM.BaseURL, "Validation OpenAI-compatible base URL"),
|
||||
llmTimeoutSeconds: fs.Int("llm-timeout-seconds", cfg.PrimaryLLM.TimeoutSeconds, "Primary LLM timeout in seconds"),
|
||||
totalLLMConcurrency: fs.Int("total-llm-concurrency", cfg.TotalLLMConcurrency, "Total concurrent LLM calls across proposal and validation"),
|
||||
proposalLLMConcurrency: fs.Int("proposal-llm-concurrency", cfg.EffectiveProposalLLMConcurrency(), "Concurrent proposal-generation LLM calls"),
|
||||
llmConcurrency: fs.Int("llm-concurrency", cfg.TotalLLMConcurrency, "Alias for --total-llm-concurrency"),
|
||||
validationLLMTimeoutSeconds: fs.Int("validation-llm-timeout-seconds", validationTimeoutSecondsDefault, "Validation LLM timeout in seconds"),
|
||||
validationMaxPromptTokens: fs.Int("validation-max-prompt-tokens", cfg.ValidationMaxPromptTokens, "Validation max prompt tokens"),
|
||||
targetSections: fs.Int("target-sections", targetSectionsDefault, "Target number of transcript sections"),
|
||||
maxRetries: fs.Int("max-retries", cfg.PrimaryLLM.MaxRetries, "Maximum structured-output retries"),
|
||||
validationMaxRetries: fs.Int("validation-max-retries", validationMaxRetriesDefault, "Validation structured-output retries"),
|
||||
validationLLMConcurrency: fs.Int("validation-llm-concurrency", validationLLMConcurrencyDefault, "Concurrent validation LLM calls (inherits total when unset)"),
|
||||
maxSectionTokens: fs.Int("max-section-tokens", cfg.MaxSectionTokens, "Maximum section tokens"),
|
||||
minSectionTokens: fs.Int("min-section-tokens", cfg.MinSectionTokens, "Minimum section tokens"),
|
||||
glossaryConfidenceThreshold: fs.Float64("glossary-confidence-threshold", cfg.Thresholds.Glossary, "Glossary confidence threshold"),
|
||||
grammarConfidenceThreshold: fs.Float64("grammar-confidence-threshold", cfg.Thresholds.Grammar, "Grammar confidence threshold"),
|
||||
homophonesConfidenceThreshold: fs.Float64("homophones-confidence-threshold", cfg.Thresholds.Homophones, "Homophones confidence threshold"),
|
||||
spokenWordConfidenceThreshold: fs.Float64("spoken-word-confidence-threshold", cfg.Thresholds.SpokenWord, "Spoken-word confidence threshold"),
|
||||
normalizeMaxSegmentGap: fs.Float64("normalize-max-segment-gap", cfg.Normalization.MaxSegmentGap, "Maximum same-speaker merge gap"),
|
||||
normalizeEllipsisGap: fs.Float64("normalize-ellipsis-gap", cfg.Normalization.EllipsisGap, "Gap threshold for ellipsis insertion"),
|
||||
normalizeMaxSegmentDuration: fs.Float64("normalize-max-segment-duration", cfg.Normalization.MaxSegmentDuration, "Maximum merged segment duration"),
|
||||
normalizeMaxSegmentTokens: fs.Int("normalize-max-segment-tokens", cfg.Normalization.MaxSegmentTokens, "Maximum merged segment token estimate"),
|
||||
transcriptDescription: fs.String("transcript-description", cfg.TranscriptDescription, "Brief background context for LLM prompts; does not override transcript content"),
|
||||
workDir: fs.String("work-dir", cfg.WorkDir, "Per-run work directory"),
|
||||
workDirRetention: fs.String("work-dir-retention", string(cfg.WorkDirRetention), "Work-dir retention policy: auto|always|never"),
|
||||
}
|
||||
|
||||
return fs, pFlags
|
||||
}
|
||||
|
||||
func findConfigPathOverride(args []string) (path string, set bool, err error) {
|
||||
for i := 0; i < len(args); i++ {
|
||||
arg := strings.TrimSpace(args[i])
|
||||
if arg == "" {
|
||||
continue
|
||||
}
|
||||
if arg == "--config" {
|
||||
if i+1 >= len(args) {
|
||||
return "", false, fmt.Errorf("--config requires a path")
|
||||
}
|
||||
return strings.TrimSpace(args[i+1]), true, nil
|
||||
}
|
||||
if strings.HasPrefix(arg, "--config=") {
|
||||
return strings.TrimSpace(strings.TrimPrefix(arg, "--config=")), true, nil
|
||||
}
|
||||
}
|
||||
return "", false, nil
|
||||
}
|
||||
|
||||
func isHelpCommand(args []string) bool {
|
||||
if len(args) == 0 {
|
||||
return false
|
||||
@@ -803,15 +168,6 @@ func isHelpCommand(args []string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func hasHelpFlag(args []string) bool {
|
||||
for _, arg := range args {
|
||||
if arg == "-h" || arg == "--help" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func writeRootUsage(w io.Writer) {
|
||||
fmt.Fprintln(w, "Audita is a transcript processing CLI.")
|
||||
fmt.Fprintln(w)
|
||||
@@ -834,33 +190,43 @@ func writeConfigUsage(w io.Writer) {
|
||||
fmt.Fprintln(w)
|
||||
fmt.Fprintln(w, "Commands:")
|
||||
fmt.Fprintln(w, " validate Validate a versioned YAML config file")
|
||||
fmt.Fprintln(w, " print-effective Print redacted effective config JSON (defaults + config file + env)")
|
||||
fmt.Fprintln(w, " print-effective Print redacted effective settings YAML")
|
||||
}
|
||||
|
||||
func writeConfigValidateUsage(w io.Writer) {
|
||||
fmt.Fprintln(w, "Validate a versioned YAML config file.")
|
||||
fmt.Fprintln(w)
|
||||
fmt.Fprintln(w, "Usage:")
|
||||
fmt.Fprintln(w, " audita config validate --config <path>")
|
||||
fmt.Fprintln(w, " audita config validate --config <path> [--overlay <name>] [setting flags]")
|
||||
}
|
||||
|
||||
func writeConfigPrintEffectiveUsage(w io.Writer) {
|
||||
fmt.Fprintln(w, "Print redacted effective config JSON.")
|
||||
fmt.Fprintln(w, "Print redacted effective settings YAML.")
|
||||
fmt.Fprintln(w)
|
||||
fmt.Fprintln(w, "Usage:")
|
||||
fmt.Fprintln(w, " audita config print-effective [--config <path>]")
|
||||
fmt.Fprintln(w, " audita config print-effective [--config <path>] [--overlay <name>] [setting flags]")
|
||||
}
|
||||
|
||||
func writeProcessUsage(w io.Writer, fs *flag.FlagSet) {
|
||||
func writeProcessUsage(w io.Writer) {
|
||||
fmt.Fprintln(w, "Process a transcript JSON file.")
|
||||
fmt.Fprintln(w)
|
||||
fmt.Fprintln(w, "Usage:")
|
||||
fmt.Fprintln(w, " audita process <transcript.json> [flags]")
|
||||
fmt.Fprintln(w)
|
||||
fmt.Fprintln(w, "Flags:")
|
||||
fs.VisitAll(func(f *flag.Flag) {
|
||||
fmt.Fprintf(w, " --%s\n", f.Name)
|
||||
})
|
||||
for _, name := range []string{
|
||||
"config", "overlay", "glossary", "output",
|
||||
"proposal-profile", "validation-profile", "modules", "output-schema",
|
||||
"validation-max-prompt-tokens", "target-sections", "max-section-tokens", "min-section-tokens",
|
||||
"normalize-max-segment-gap", "normalize-ellipsis-gap", "normalize-max-segment-duration", "normalize-max-segment-tokens",
|
||||
"glossary-confidence-threshold", "homophones-confidence-threshold", "spoken-word-confidence-threshold", "grammar-confidence-threshold",
|
||||
"transcript-description", "work-dir",
|
||||
} {
|
||||
fmt.Fprintf(w, " --%s\n", name)
|
||||
}
|
||||
fmt.Fprintln(w, " --reporting[=true|false] Write the three-file report bundle (default true)")
|
||||
fmt.Fprintln(w, " --report-dir DIR Report directory for stdout output only; ignored with --output or --reporting=false")
|
||||
fmt.Fprintln(w, " --debug[=true|false] Persist detailed diagnostics (default false)")
|
||||
fmt.Fprintln(w)
|
||||
fmt.Fprintln(w, "Example:")
|
||||
fmt.Fprintln(w, " audita process transcript.json --glossary glossary.yaml --output corrected.json")
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,110 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||
)
|
||||
|
||||
const (
|
||||
subprocessTestLLMModeEnv = "AUDITA_SUBPROCESS_TEST_LLM_MODE"
|
||||
subprocessTestRunTimeoutMSEnv = "AUDITA_SUBPROCESS_TEST_RUN_TIMEOUT_MS"
|
||||
)
|
||||
|
||||
// ConfigureSubprocessTestHooksFromEnv enables deterministic test-only hooks for
|
||||
// subprocess integration tests that run through the Go test binary helper path.
|
||||
func ConfigureSubprocessTestHooksFromEnv() {
|
||||
mode := strings.TrimSpace(os.Getenv(subprocessTestLLMModeEnv))
|
||||
timeoutMSRaw := strings.TrimSpace(os.Getenv(subprocessTestRunTimeoutMSEnv))
|
||||
// Only activate in explicit subprocess test mode.
|
||||
if mode == "" && timeoutMSRaw == "" {
|
||||
return
|
||||
}
|
||||
|
||||
if mode != "" {
|
||||
client := &subprocessTestLLMClient{mode: mode}
|
||||
processProposalLLMClient = client
|
||||
processValidationLLMClient = client
|
||||
}
|
||||
|
||||
if timeoutMSRaw == "" {
|
||||
return
|
||||
}
|
||||
timeoutMS, err := strconv.Atoi(timeoutMSRaw)
|
||||
if err != nil || timeoutMS <= 0 {
|
||||
return
|
||||
}
|
||||
processRunnerContext = func() (context.Context, context.CancelFunc) {
|
||||
return context.WithTimeout(context.Background(), time.Duration(timeoutMS)*time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
type subprocessTestLLMClient struct {
|
||||
mode string
|
||||
mu sync.Mutex
|
||||
proposals int
|
||||
}
|
||||
|
||||
func (c *subprocessTestLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
_ = req
|
||||
|
||||
switch c.mode {
|
||||
case "backend_error":
|
||||
return contracts.StructuredCompletionResponse{}, errors.New("synthetic backend failure")
|
||||
case "block_until_cancel":
|
||||
<-ctx.Done()
|
||||
return contracts.StructuredCompletionResponse{}, ctx.Err()
|
||||
case "malformed_structured":
|
||||
switch target := out.(type) {
|
||||
case *proposal_generation.StructuredCorrectionSet:
|
||||
*target = proposal_generation.StructuredCorrectionSet{
|
||||
Corrections: []proposal_generation.StructuredCorrectionProposal{
|
||||
{TargetSegmentID: 0, OriginalText: "x", CorrectedText: "y", Confidence: 0.99},
|
||||
},
|
||||
}
|
||||
case *validators.LLMValidationResponse:
|
||||
*target = validators.LLMValidationResponse{
|
||||
Validations: []validators.LLMValidationDecision{
|
||||
{CorrectionIndex: 999, Approved: true, Confidence: 0.9, Reason: "bad index"},
|
||||
},
|
||||
}
|
||||
}
|
||||
case "mid_pipeline_fail":
|
||||
if _, ok := out.(*proposal_generation.StructuredCorrectionSet); ok {
|
||||
c.mu.Lock()
|
||||
c.proposals++
|
||||
proposalCall := c.proposals
|
||||
c.mu.Unlock()
|
||||
|
||||
if proposalCall >= 3 {
|
||||
return contracts.StructuredCompletionResponse{}, errors.New("synthetic mid-pipeline failure")
|
||||
}
|
||||
}
|
||||
switch target := out.(type) {
|
||||
case *proposal_generation.StructuredCorrectionSet:
|
||||
*target = proposal_generation.StructuredCorrectionSet{
|
||||
Corrections: []proposal_generation.StructuredCorrectionProposal{
|
||||
{TargetSegmentID: 1, OriginalText: "Segment", CorrectedText: "Segment", Confidence: 0.99},
|
||||
},
|
||||
}
|
||||
case *validators.LLMValidationResponse:
|
||||
*target = validators.LLMValidationResponse{Validations: nil}
|
||||
}
|
||||
default:
|
||||
switch target := out.(type) {
|
||||
case *proposal_generation.StructuredCorrectionSet:
|
||||
*target = proposal_generation.StructuredCorrectionSet{Corrections: nil}
|
||||
case *validators.LLMValidationResponse:
|
||||
*target = validators.LLMValidationResponse{Validations: nil}
|
||||
}
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{}, nil
|
||||
}
|
||||
@@ -1,22 +0,0 @@
|
||||
{
|
||||
"name": "ambiguous_match_rejected_before_application",
|
||||
"transcript_file": "application-skip-ambiguous.transcript.json",
|
||||
"glossary_file": "default-handoff.glossary.yaml",
|
||||
"modules_csv": "homophones",
|
||||
"proposal_responses_file": "application-skip-ambiguous.proposals.json",
|
||||
"validation_responses_file": "application-skip-ambiguous.validations.json",
|
||||
"expect": {
|
||||
"exit_code": 0,
|
||||
"status": "success",
|
||||
"output_transcript_file": "application-skip-ambiguous.expected-transcript.json",
|
||||
"module_instances": ["homophones"],
|
||||
"module_count": 1,
|
||||
"total_applied_changes": 0,
|
||||
"total_skipped_changes": 1,
|
||||
"module_applied_counts": [0],
|
||||
"module_rejected_counts": [1],
|
||||
"module_skip_counts": [0],
|
||||
"validator_rejected_reason_codes": ["ambiguous_original_text"],
|
||||
"expected_proposal_calls": ["homophones:proposal"]
|
||||
}
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"Alice","start":0,"end":1,"text":"the site near another site"}
|
||||
]
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]}
|
||||
]
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"the site near another site"}
|
||||
]
|
||||
@@ -1,4 +0,0 @@
|
||||
[
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
|
||||
]
|
||||
@@ -1,28 +0,0 @@
|
||||
{
|
||||
"name": "default_full_pipeline_shape_and_reports",
|
||||
"transcript_file": "default-full-pipeline.transcript.json",
|
||||
"glossary_file": "default-full-pipeline.glossary.yaml",
|
||||
"proposal_responses_file": "default-full-pipeline.proposals.json",
|
||||
"validation_responses_file": "default-full-pipeline.validations.json",
|
||||
"env": {
|
||||
"AUDITA_LLM_API_KEY": "parity-secret",
|
||||
"AUDITA_VALIDATION_LLM_API_KEY": "parity-secret"
|
||||
},
|
||||
"expect": {
|
||||
"exit_code": 0,
|
||||
"status": "success",
|
||||
"output_transcript_file": "default-full-pipeline.expected-transcript.json",
|
||||
"module_instances": ["glossary_1", "homophones", "glossary_2", "spoken_word", "grammar"],
|
||||
"module_count": 5,
|
||||
"total_applied_changes": 3,
|
||||
"total_skipped_changes": 3,
|
||||
"secret_markers": ["parity-secret"],
|
||||
"expected_proposal_calls": [
|
||||
"glossary_1:proposal",
|
||||
"homophones:proposal",
|
||||
"glossary_2:proposal",
|
||||
"spoken_word:proposal",
|
||||
"grammar:proposal"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -1,9 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": 1,
|
||||
"speaker": "Alice",
|
||||
"start": 0,
|
||||
"end": 1,
|
||||
"text": "Hello, there were Jesters hmm"
|
||||
}
|
||||
]
|
||||
@@ -1,7 +0,0 @@
|
||||
glossary:
|
||||
- name: Jesters
|
||||
aliases:
|
||||
- jester
|
||||
plural: jesters
|
||||
category: faction
|
||||
summary: A protected in-world faction term.
|
||||
@@ -1,28 +0,0 @@
|
||||
[
|
||||
{
|
||||
"corrections": [
|
||||
{"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99}
|
||||
]
|
||||
},
|
||||
{
|
||||
"corrections": [
|
||||
{"id": 1, "original_text": "Jesters", "corrected_text": "jesters", "confidence": 0.99},
|
||||
{"id": 1, "original_text": "Jesters", "corrected_text": "JESTERX", "confidence": 0.99}
|
||||
]
|
||||
},
|
||||
{
|
||||
"corrections": [
|
||||
{"id": 1, "original_text": "jesters", "corrected_text": "JESTERS", "confidence": 0.99}
|
||||
]
|
||||
},
|
||||
{
|
||||
"corrections": [
|
||||
{"id": 1, "original_text": "uh", "corrected_text": "hmm", "confidence": 0.99}
|
||||
]
|
||||
},
|
||||
{
|
||||
"corrections": [
|
||||
{"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , there were gestures uh"}
|
||||
]
|
||||
@@ -1,12 +0,0 @@
|
||||
[
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": false, "confidence": 0.99, "reason": "reject cleanup"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "parity-secret"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
|
||||
]
|
||||
@@ -1,27 +0,0 @@
|
||||
{
|
||||
"name": "default_pipeline_handoff_and_module_order",
|
||||
"transcript_file": "default-handoff.transcript.json",
|
||||
"glossary_file": "default-handoff.glossary.yaml",
|
||||
"proposal_responses_file": "default-handoff.proposals.json",
|
||||
"validation_responses_file": "default-handoff.validations.json",
|
||||
"expect": {
|
||||
"exit_code": 0,
|
||||
"status": "success",
|
||||
"output_transcript_file": "default-handoff.expected-transcript.json",
|
||||
"module_instances": ["glossary_1", "homophones", "glossary_2", "spoken_word", "grammar"],
|
||||
"module_count": 5,
|
||||
"total_applied_changes": 5,
|
||||
"total_skipped_changes": 0,
|
||||
"module_applied_counts": [1, 1, 1, 1, 1],
|
||||
"module_rejected_counts": [0, 0, 0, 0, 0],
|
||||
"module_skip_counts": [0, 0, 0, 0, 0],
|
||||
"expected_proposal_calls": [
|
||||
"glossary_1:proposal",
|
||||
"homophones:proposal",
|
||||
"glossary_2:proposal",
|
||||
"spoken_word:proposal",
|
||||
"grammar:proposal"
|
||||
],
|
||||
"min_response_payload_artifacts": 15
|
||||
}
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"Alice","start":0,"end":1,"text":"Hello, there were Jesters at the Sight um"}
|
||||
]
|
||||
@@ -1,6 +0,0 @@
|
||||
glossary:
|
||||
- name: Jesters
|
||||
aliases: [jester]
|
||||
plural: jesters
|
||||
category: faction
|
||||
summary: A protected in-world faction term.
|
||||
@@ -1,7 +0,0 @@
|
||||
[
|
||||
{"corrections": [{"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99}]},
|
||||
{"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]},
|
||||
{"corrections": [{"id": 1, "original_text": "sight", "corrected_text": "Sight", "confidence": 0.99}]},
|
||||
{"corrections": [{"id": 1, "original_text": "um um", "corrected_text": "um", "confidence": 0.99}]},
|
||||
{"corrections": [{"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99}]}
|
||||
]
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , there were gestures at the site um um"}
|
||||
]
|
||||
@@ -1,12 +0,0 @@
|
||||
[
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
|
||||
]
|
||||
@@ -1,17 +0,0 @@
|
||||
{
|
||||
"name": "deterministic_validator_low_confidence",
|
||||
"transcript_file": "deterministic-validator-low-confidence.transcript.json",
|
||||
"glossary_file": "default-full-pipeline.glossary.yaml",
|
||||
"modules_csv": "grammar",
|
||||
"proposal_responses_file": "deterministic-validator-low-confidence.proposals.json",
|
||||
"expect": {
|
||||
"exit_code": 0,
|
||||
"status": "success",
|
||||
"output_transcript_file": "deterministic-validator-low-confidence.expected-transcript.json",
|
||||
"module_instances": ["grammar"],
|
||||
"module_count": 1,
|
||||
"total_skipped_changes": 1,
|
||||
"validator_rejected_reason_codes": ["low_confidence"],
|
||||
"expected_proposal_calls": ["grammar:proposal"]
|
||||
}
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"Alice","start":0,"end":1,"text":"hello , world"}
|
||||
]
|
||||
@@ -1,7 +0,0 @@
|
||||
[
|
||||
{
|
||||
"corrections": [
|
||||
{"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.1}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , world"}
|
||||
]
|
||||
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"name": "glossary_schema_handling",
|
||||
"transcript_file": "default-full-pipeline.transcript.json",
|
||||
"glossary_file": "glossary-schema-error.glossary.yaml",
|
||||
"expect": {
|
||||
"exit_code": 1,
|
||||
"status": "failed",
|
||||
"error_phase": "glossary_schema",
|
||||
"stderr_contains": "glossary_schema",
|
||||
"require_error_log": true
|
||||
}
|
||||
}
|
||||
@@ -1,2 +0,0 @@
|
||||
glossary:
|
||||
- name: MissingCategoryAndSummary
|
||||
@@ -1,19 +0,0 @@
|
||||
{
|
||||
"name": "llm_validator_decision_handling",
|
||||
"transcript_file": "llm-validator-rejection.transcript.json",
|
||||
"glossary_file": "default-full-pipeline.glossary.yaml",
|
||||
"modules_csv": "grammar",
|
||||
"proposal_responses_file": "llm-validator-rejection.proposals.json",
|
||||
"validation_responses_file": "llm-validator-rejection.validations.json",
|
||||
"expect": {
|
||||
"exit_code": 0,
|
||||
"status": "success",
|
||||
"output_transcript_file": "llm-validator-rejection.expected-transcript.json",
|
||||
"module_instances": ["grammar"],
|
||||
"module_count": 1,
|
||||
"total_skipped_changes": 1,
|
||||
"validator_rejected_reason_codes": ["llm_rejected"],
|
||||
"expected_proposal_calls": ["grammar:proposal"],
|
||||
"expected_validation_calls": ["grammar:section-0000:editorial_review:batch-0000"]
|
||||
}
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"Alice","start":0,"end":1,"text":"hello , world"}
|
||||
]
|
||||
@@ -1,7 +0,0 @@
|
||||
[
|
||||
{
|
||||
"corrections": [
|
||||
{"id": 1, "original_text": "hello ,", "corrected_text": "Hello,", "confidence": 0.99}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"hello , world"}
|
||||
]
|
||||
@@ -1,7 +0,0 @@
|
||||
[
|
||||
{
|
||||
"validations": [
|
||||
{"correction_index": 0, "approved": false, "confidence": 0.99, "reason": "reject stylistic overreach"}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -1,23 +0,0 @@
|
||||
{
|
||||
"name": "mid_pipeline_failure_partial_progress",
|
||||
"transcript_file": "default-handoff.transcript.json",
|
||||
"glossary_file": "default-handoff.glossary.yaml",
|
||||
"proposal_responses_file": "mid-pipeline-failure.proposals.json",
|
||||
"validation_responses_file": "mid-pipeline-failure.validations.json",
|
||||
"expect": {
|
||||
"exit_code": 1,
|
||||
"status": "failed",
|
||||
"error_phase": "runner_execution",
|
||||
"stderr_contains": "runner_execution",
|
||||
"module_instances": ["glossary_1", "homophones", "glossary_2"],
|
||||
"module_count": 3,
|
||||
"total_applied_changes": 2,
|
||||
"total_skipped_changes": 0,
|
||||
"failed_module_instance": "glossary_2",
|
||||
"module_applied_counts": [1, 1, 0],
|
||||
"module_rejected_counts": [0, 0, 0],
|
||||
"module_skip_counts": [0, 0, 0],
|
||||
"require_error_log": true,
|
||||
"expected_proposal_calls": ["glossary_1:proposal", "homophones:proposal", "glossary_2:proposal"]
|
||||
}
|
||||
}
|
||||
@@ -1,4 +0,0 @@
|
||||
[
|
||||
{"corrections": [{"id": 1, "original_text": "gestures", "corrected_text": "Jesters", "confidence": 0.99}]},
|
||||
{"corrections": [{"id": 1, "original_text": "site", "corrected_text": "sight", "confidence": 0.99}]}
|
||||
]
|
||||
@@ -1,6 +0,0 @@
|
||||
[
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]},
|
||||
{"validations": [{"correction_index": 0, "approved": true, "confidence": 0.99, "reason": "ok"}]}
|
||||
]
|
||||
@@ -1,21 +0,0 @@
|
||||
{
|
||||
"name": "protected_glossary_term_behavior",
|
||||
"transcript_file": "protected-term-rejection.transcript.json",
|
||||
"glossary_file": "default-handoff.glossary.yaml",
|
||||
"modules_csv": "homophones",
|
||||
"proposal_responses_file": "protected-term-rejection.proposals.json",
|
||||
"expect": {
|
||||
"exit_code": 0,
|
||||
"status": "success",
|
||||
"output_transcript_file": "protected-term-rejection.expected-transcript.json",
|
||||
"module_instances": ["homophones"],
|
||||
"module_count": 1,
|
||||
"total_applied_changes": 0,
|
||||
"total_skipped_changes": 1,
|
||||
"module_applied_counts": [0],
|
||||
"module_rejected_counts": [1],
|
||||
"module_skip_counts": [0],
|
||||
"validator_rejected_reason_codes": ["protected_glossary_term"],
|
||||
"expected_proposal_calls": ["homophones:proposal"]
|
||||
}
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"Alice","start":0,"end":1,"text":"The Jesters entered the hall."}
|
||||
]
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"corrections": [{"id": 1, "original_text": "Jesters", "corrected_text": "Gestures", "confidence": 0.99}]}
|
||||
]
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"Alice","start":0.0,"end":1.0,"text":"The Jesters entered the hall."}
|
||||
]
|
||||
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"name": "transcript_schema_handling",
|
||||
"transcript_file": "transcript-schema-error.transcript.json",
|
||||
"glossary_file": "default-full-pipeline.glossary.yaml",
|
||||
"expect": {
|
||||
"exit_code": 1,
|
||||
"status": "failed",
|
||||
"error_phase": "transcript_schema",
|
||||
"stderr_contains": "transcript_schema",
|
||||
"require_error_log": true
|
||||
}
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
[
|
||||
{"id":1,"speaker":"","start":0.0,"end":1.0,"text":"bad"}
|
||||
]
|
||||
@@ -1,24 +0,0 @@
|
||||
{
|
||||
"must_apply_texts": [
|
||||
"Hello, there were Jesters hmm"
|
||||
],
|
||||
"must_not_apply_texts": [
|
||||
"JESTERX",
|
||||
"there were gestures"
|
||||
],
|
||||
"protected_terms": [
|
||||
"Jesters"
|
||||
],
|
||||
"expected_module_instances": [
|
||||
"glossary_1",
|
||||
"homophones",
|
||||
"glossary_2",
|
||||
"spoken_word",
|
||||
"grammar"
|
||||
],
|
||||
"minimum_counts": {
|
||||
"applied": 1,
|
||||
"rejected": 1,
|
||||
"skipped": 0
|
||||
}
|
||||
}
|
||||
@@ -1,9 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": 1,
|
||||
"speaker": "Alice",
|
||||
"start": 0,
|
||||
"end": 1,
|
||||
"text": "Hello, there were Jesters hmm"
|
||||
}
|
||||
]
|
||||
@@ -1,7 +0,0 @@
|
||||
glossary:
|
||||
- name: Jesters
|
||||
aliases:
|
||||
- jester
|
||||
plural: jesters
|
||||
category: faction
|
||||
summary: A protected in-world faction term.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user