Compare commits
15 Commits
f790c1441c
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 018d08fd4c | |||
| 3a0a0b3940 | |||
| 99ab2f181b | |||
| 4e4801dc98 | |||
| 900ad74958 | |||
| 28d5201a69 | |||
| e5944f9875 | |||
| 6344fc91ba | |||
| 7f3a91cc9e | |||
| 72fb021453 | |||
| 40e8b54d3b | |||
| 76651333b1 | |||
| 0b01c3a83d | |||
| 0630d36734 | |||
| 52c2697040 |
2
LICENSE
2
LICENSE
@@ -1,4 +1,4 @@
|
||||
Copyright (c) 2026 eric.
|
||||
Copyright (c) 2026 Eric Rakestraw.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
|
||||
|
||||
|
||||
305
README.md
305
README.md
@@ -1,305 +1,46 @@
|
||||
# Audita
|
||||
|
||||
Audita is a transcript polishing CLI.
|
||||
Audita is a CLI that polishes transcript JSON using glossary-aware and LLM-backed correction modules.
|
||||
|
||||
`audita process` validates transcript/glossary input, normalizes and chunks transcript segments, runs the default correction pipeline, and emits corrected transcript output plus machine-readable diagnostics and reports.
|
||||
## Quickstart
|
||||
|
||||
## What Audita Does
|
||||
|
||||
Default module sequence:
|
||||
- `glossary`
|
||||
- `homophones`
|
||||
- `glossary`
|
||||
- `spoken_word`
|
||||
- `grammar`
|
||||
|
||||
Pipeline behavior includes:
|
||||
- glossary-backed domain/acoustic corrections
|
||||
- conservative homophone and mistranscription corrections
|
||||
- conservative spoken-word dysfluency cleanup with semantic guardrails
|
||||
- grammar/punctuation/capitalization/formatting cleanup
|
||||
- validator-chain enforcement before application
|
||||
- malformed module-stage LLM payloads degrade to warnings/rejections instead of failing the run
|
||||
- run reports and diagnostics artifacts with secret redaction
|
||||
|
||||
## Build and Install
|
||||
|
||||
Build a local binary:
|
||||
Build:
|
||||
|
||||
```sh
|
||||
go build -o ./bin/audita ./cmd/audita
|
||||
```
|
||||
|
||||
Install into your Go bin directory:
|
||||
Run the shortest useful command:
|
||||
|
||||
```sh
|
||||
go install ./cmd/audita
|
||||
audita process ./transcript.json --glossary ./glossary.yaml --output ./corrected.json
|
||||
```
|
||||
|
||||
CLI help:
|
||||
|
||||
```sh
|
||||
audita --help
|
||||
audita process --help
|
||||
audita config --help
|
||||
```
|
||||
|
||||
## Test
|
||||
|
||||
Run all tests:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Basic Usage
|
||||
|
||||
Required inputs:
|
||||
- transcript JSON path (positional argument)
|
||||
- `--glossary <glossary.yaml>`
|
||||
|
||||
Recommended run:
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--output corrected.json \
|
||||
--report-json report.json
|
||||
```
|
||||
|
||||
Select an explicit output schema (default is `bare-segments`):
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--output-schema audita-v1 \
|
||||
--output corrected.json \
|
||||
--report-json report.json
|
||||
```
|
||||
|
||||
Recommended config-based run:
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--config audita.yml \
|
||||
--output corrected.json \
|
||||
--report-json report.json
|
||||
```
|
||||
|
||||
Explicit module override:
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--modules glossary,homophones,grammar \
|
||||
--output corrected.json \
|
||||
--report-json report.json
|
||||
```
|
||||
|
||||
Optional transcript background context:
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--transcript-description "Brief context that may help resolve ambiguous terms." \
|
||||
--output corrected.json
|
||||
```
|
||||
|
||||
The transcript description is background context only and does not override transcript content.
|
||||
|
||||
Write transcript JSON to stdout (no `--output`):
|
||||
|
||||
```sh
|
||||
audita process transcript.json --glossary glossary.yaml
|
||||
```
|
||||
|
||||
Control diagnostics location/retention:
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--work-dir /tmp/audita \
|
||||
--work-dir-retention auto \
|
||||
--output corrected.json \
|
||||
--report-json report.json
|
||||
```
|
||||
|
||||
## Stdout/Stderr Contract
|
||||
|
||||
- With `--output`, stdout is expected to be empty on success.
|
||||
- Without `--output`, stdout contains transcript JSON only on success.
|
||||
- `--report-json` writes a file and is never printed to stdout.
|
||||
- stderr is human-readable diagnostics/errors.
|
||||
- successful runs remain quiet on stderr even when module warnings are recorded in report/diagnostics artifacts.
|
||||
|
||||
For subprocess orchestration guidance, see [`docs/subprocess-operations.md`](docs/subprocess-operations.md).
|
||||
Notes:
|
||||
- the transcript JSON path is required as a positional argument;
|
||||
- `--glossary` is required;
|
||||
- without `--output`, corrected transcript JSON is written to stdout.
|
||||
|
||||
## Configuration
|
||||
|
||||
Precedence:
|
||||
1. defaults
|
||||
2. config file (`--config`, `AUDITA_CONFIG`, or default search paths when present: `/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`)
|
||||
3. environment (`AUDITA_*`)
|
||||
4. CLI flags
|
||||
Audita loads defaults, optional file config, environment overrides, then CLI overrides.
|
||||
|
||||
Config commands:
|
||||
Use these commands to validate and inspect config:
|
||||
|
||||
```sh
|
||||
audita config validate --config audita.yml
|
||||
audita config print-effective --config audita.yml
|
||||
audita config validate --config ./audita.yml
|
||||
audita config print-effective --config ./audita.yml
|
||||
```
|
||||
|
||||
For full config-file schema and examples, see [`docs/configuration.md`](docs/configuration.md).
|
||||
For output-schema details, see [`docs/architecture/output-schemas.md`](docs/architecture/output-schemas.md).
|
||||
For built-in validator keys and chain definitions, see [`docs/architecture/validators.md`](docs/architecture/validators.md).
|
||||
For embedded prompt assets and prompt metadata behavior, see [`docs/architecture/prompts.md`](docs/architecture/prompts.md).
|
||||
For CLI/process compatibility guarantees, see [`docs/architecture/public-contract.md`](docs/architecture/public-contract.md).
|
||||
|
||||
### Modules
|
||||
|
||||
- `AUDITA_MODULES` (CSV)
|
||||
- CLI: `--modules`
|
||||
|
||||
### Transcript Description
|
||||
|
||||
CLI:
|
||||
- `--transcript-description`
|
||||
|
||||
Behavior:
|
||||
- optional background context for proposal and LLM-validator prompts;
|
||||
- trimmed and length-limited by CLI validation;
|
||||
- does not override transcript content;
|
||||
- no `AUDITA_*` environment variable is currently defined for this setting.
|
||||
|
||||
### Primary LLM
|
||||
|
||||
Environment:
|
||||
- `AUDITA_LLM_API_KEY` (or `OPENROUTER_API_KEY` fallback)
|
||||
- `AUDITA_MODEL`
|
||||
- `AUDITA_BASE_URL`
|
||||
- `AUDITA_LLM_TIMEOUT_SECONDS`
|
||||
- `AUDITA_MAX_RETRIES`
|
||||
|
||||
CLI:
|
||||
- `--llm-api-key`
|
||||
- `--model`
|
||||
- `--base-url`
|
||||
- `--llm-timeout-seconds`
|
||||
- `--max-retries`
|
||||
|
||||
### Validation LLM
|
||||
|
||||
Environment:
|
||||
- `AUDITA_VALIDATION_LLM_API_KEY`
|
||||
- `AUDITA_VALIDATION_MODEL`
|
||||
- `AUDITA_VALIDATION_BASE_URL`
|
||||
- `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS`
|
||||
- `AUDITA_VALIDATION_MAX_RETRIES`
|
||||
- `AUDITA_VALIDATION_LLM_CONCURRENCY`
|
||||
- `AUDITA_VALIDATION_MAX_PROMPT_TOKENS`
|
||||
|
||||
CLI:
|
||||
- `--validation-llm-api-key`
|
||||
- `--validation-model`
|
||||
- `--validation-base-url`
|
||||
- `--validation-llm-timeout-seconds`
|
||||
- `--validation-max-retries`
|
||||
- `--validation-llm-concurrency`
|
||||
- `--validation-max-prompt-tokens`
|
||||
|
||||
### LLM Concurrency
|
||||
|
||||
Environment:
|
||||
- `AUDITA_TOTAL_LLM_CONCURRENCY`
|
||||
- `AUDITA_PROPOSAL_LLM_CONCURRENCY`
|
||||
- `AUDITA_VALIDATION_LLM_CONCURRENCY`
|
||||
- `AUDITA_LLM_CONCURRENCY` (legacy alias for `AUDITA_TOTAL_LLM_CONCURRENCY`)
|
||||
|
||||
CLI:
|
||||
- `--total-llm-concurrency`
|
||||
- `--proposal-llm-concurrency`
|
||||
- `--validation-llm-concurrency`
|
||||
- `--llm-concurrency` (legacy alias for `--total-llm-concurrency`)
|
||||
|
||||
Behavior:
|
||||
- all proposal and validation LLM calls are bounded by total LLM concurrency
|
||||
- proposal LLM calls are additionally bounded by proposal LLM concurrency
|
||||
- when validation concurrency is unset, it inherits total LLM concurrency
|
||||
- when explicitly set, proposal and validation concurrency must each be `<= total-llm-concurrency`
|
||||
- canonical total settings win when both canonical and legacy alias settings are provided at the same precedence layer
|
||||
|
||||
### Confidence Thresholds
|
||||
|
||||
Environment:
|
||||
- `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD`
|
||||
- `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD`
|
||||
- `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD`
|
||||
- `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD`
|
||||
|
||||
CLI:
|
||||
- `--glossary-confidence-threshold`
|
||||
- `--homophones-confidence-threshold`
|
||||
- `--spoken-word-confidence-threshold`
|
||||
- `--grammar-confidence-threshold`
|
||||
|
||||
### Normalization and Chunking
|
||||
|
||||
Environment:
|
||||
- `AUDITA_NORMALIZE_MAX_SEGMENT_GAP`
|
||||
- `AUDITA_NORMALIZE_ELLIPSIS_GAP`
|
||||
- `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION`
|
||||
- `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS`
|
||||
- `AUDITA_MAX_SECTION_TOKENS`
|
||||
- `AUDITA_MIN_SECTION_TOKENS`
|
||||
- `AUDITA_TARGET_SECTIONS`
|
||||
|
||||
CLI:
|
||||
- `--normalize-max-segment-gap`
|
||||
- `--normalize-ellipsis-gap`
|
||||
- `--normalize-max-segment-duration`
|
||||
- `--normalize-max-segment-tokens`
|
||||
- `--max-section-tokens`
|
||||
- `--min-section-tokens`
|
||||
- `--target-sections`
|
||||
|
||||
### Work Directory
|
||||
|
||||
Environment:
|
||||
- `AUDITA_WORK_DIR`
|
||||
- `AUDITA_WORK_DIR_RETENTION` (`auto`, `always`, `never`)
|
||||
|
||||
CLI:
|
||||
- `--work-dir`
|
||||
- `--work-dir-retention`
|
||||
|
||||
Retention behavior:
|
||||
- `always`: keep all run directories
|
||||
- `never`: keep successful run directories
|
||||
- `auto`: keep failed runs and successful runs with skipped/rejected corrections
|
||||
|
||||
## Reports and Diagnostics
|
||||
|
||||
Per-run diagnostics include:
|
||||
- source transcript artifacts
|
||||
- normalized transcript artifact
|
||||
- normalization summary
|
||||
- chunking summary
|
||||
- utilization diagnostics summary
|
||||
- correction ledger
|
||||
- invocation metadata
|
||||
- redacted effective config
|
||||
- module/validator prompt-response diagnostics
|
||||
- `report.json`
|
||||
- `error.log` on failure
|
||||
|
||||
Optional external report output:
|
||||
- `--report-json <path>`
|
||||
|
||||
## Documentation
|
||||
|
||||
- Architecture: [`docs/architecture.md`](docs/architecture.md)
|
||||
- Diagnostics: [`docs/diagnostics.md`](docs/diagnostics.md)
|
||||
- Structured LLM adapter: [`docs/structured-llm.md`](docs/structured-llm.md)
|
||||
- Subprocess operations: [`docs/subprocess-operations.md`](docs/subprocess-operations.md)
|
||||
- Release checklist: [`docs/release-checklist.md`](docs/release-checklist.md)
|
||||
- CLI reference: [`docs/cli.md`](docs/cli.md)
|
||||
- Configuration reference: [`docs/config.md`](docs/config.md)
|
||||
- Operations guide: [`docs/operations.md`](docs/operations.md)
|
||||
- Troubleshooting: [`docs/troubleshooting.md`](docs/troubleshooting.md)
|
||||
- Subprocess integration: [`docs/integrations/subprocess.md`](docs/integrations/subprocess.md)
|
||||
- OpenAI-compatible LLM integration: [`docs/integrations/openai-compatible-llm.md`](docs/integrations/openai-compatible-llm.md)
|
||||
- Transcript and glossary file integration: [`docs/integrations/transcript-glossary-files.md`](docs/integrations/transcript-glossary-files.md)
|
||||
- Development workflow: [`docs/policy/development.md`](docs/policy/development.md)
|
||||
- Architecture policy: [`docs/policy/architecture.md`](docs/policy/architecture.md)
|
||||
- Documentation policy: [`docs/policy/documentation.md`](docs/policy/documentation.md)
|
||||
|
||||
@@ -1,14 +0,0 @@
|
||||
# Audita Architecture Index
|
||||
|
||||
This file is the entrypoint for architecture documentation.
|
||||
|
||||
Core architecture overview:
|
||||
- [Architecture Overview](./architecture/architecture.md)
|
||||
|
||||
Focused architecture contracts:
|
||||
- [Public Contract](./architecture/public-contract.md)
|
||||
- [Diagnostics](./architecture/diagnostics.md)
|
||||
- [Structured LLM](./architecture/structured-llm.md)
|
||||
- [Validators](./architecture/validators.md)
|
||||
- [Prompts](./architecture/prompts.md)
|
||||
- [Output Schemas](./architecture/output-schemas.md)
|
||||
@@ -1,153 +0,0 @@
|
||||
# Audita Architecture
|
||||
|
||||
## Scope
|
||||
This document describes the production architecture implemented in this repository today.
|
||||
|
||||
Audita is a single-process Go CLI that:
|
||||
- loads effective runtime configuration;
|
||||
- reads transcript and glossary inputs;
|
||||
- normalizes and sections transcripts;
|
||||
- runs a built-in module pipeline with validator chains;
|
||||
- writes transcript output and run diagnostics.
|
||||
|
||||
## Runtime entrypoints
|
||||
Primary CLI commands:
|
||||
- `audita process <transcript.json> --glossary <glossary.yaml> [flags]`
|
||||
- `audita config validate --config <config.yml>`
|
||||
- `audita config print-effective [--config <config.yml>]`
|
||||
|
||||
Command ownership lives in `internal/cli/run.go`.
|
||||
|
||||
## Configuration model
|
||||
`internal/core/config` owns defaults, file parsing, environment overrides, CLI overrides, and validation.
|
||||
|
||||
Effective-config loading for `process` and `config print-effective` is centralized in:
|
||||
- `ResolveConfigPath`
|
||||
- `LoadEffectiveConfig`
|
||||
|
||||
Effective precedence for `audita process`:
|
||||
1. defaults
|
||||
2. config file
|
||||
3. environment overrides
|
||||
4. CLI overrides
|
||||
|
||||
`audita config validate` is intentionally file-only validation:
|
||||
- load versioned file;
|
||||
- apply onto defaults;
|
||||
- validate;
|
||||
- do not apply environment overrides.
|
||||
|
||||
Supported module and output-schema keys are validated through shared catalogs:
|
||||
- module keys: `internal/core/modulecatalog`
|
||||
- output schemas: `internal/core/outputschema`
|
||||
|
||||
## Pipeline and module orchestration
|
||||
The built-in module sequence is configured in runtime config and executed by `internal/framework/runner` through resolved module specs.
|
||||
|
||||
Current default sequence:
|
||||
- `glossary`
|
||||
- `homophones`
|
||||
- `glossary`
|
||||
- `spoken_word`
|
||||
- `grammar`
|
||||
|
||||
Execution behavior:
|
||||
- modules execute serially over the working transcript;
|
||||
- section proposal work can run concurrently within a module;
|
||||
- validator execution happens on generated proposals before application;
|
||||
- approved proposals are applied once per module in deterministic proposal-index order.
|
||||
|
||||
Production modules remain separate packages:
|
||||
- `internal/modules/glossary`
|
||||
- `internal/modules/homophones`
|
||||
- `internal/modules/spoken_word`
|
||||
- `internal/modules/grammar`
|
||||
|
||||
## Proposal generation and prompt context
|
||||
Shared proposal plumbing is centralized in `internal/framework/proposal_generation`.
|
||||
|
||||
Module packages provide:
|
||||
- module identity and replacement policy;
|
||||
- module-specific prompt message building;
|
||||
- built-in validator chain selection.
|
||||
|
||||
Shared prompt payload helpers are in `internal/framework/promptcontext`.
|
||||
|
||||
## Validator architecture
|
||||
Built-in validator construction and chain composition live in `internal/validators`.
|
||||
|
||||
Shared validator runtime mechanics live in `internal/framework/validators`.
|
||||
|
||||
Execution class metadata (deterministic vs LLM-backed) is centralized in `internal/validators/metadata` and used for ordering and reporting classification.
|
||||
|
||||
## Structured LLM boundary
|
||||
All production LLM calls go through the internal contract:
|
||||
- `contracts.StructuredLLMClient`
|
||||
- `CompleteStructured(ctx, req, out)`
|
||||
|
||||
The OpenAI-compatible HTTP adapter is implemented in `internal/framework/llm`.
|
||||
|
||||
Structured response schemas are registered in `internal/framework/responseschema` and attached to requests via `response_format` metadata.
|
||||
|
||||
Malformed structured-output detection is centralized in `internal/framework/structuredoutput` and reused by proposal generation and validator execution so downgrade behavior stays consistent.
|
||||
|
||||
## Stage naming and diagnostics metadata
|
||||
Diagnostics stage naming is centralized in `internal/framework/stagename`:
|
||||
- module proposal stage names;
|
||||
- proposal-generation stage names;
|
||||
- validator batch stage names.
|
||||
|
||||
Prompt metadata and response-schema metadata each expose canonical diagnostics maps via:
|
||||
- `prompts.Metadata.DiagnosticsMap()`
|
||||
- `responseschema.Schema.DiagnosticsMap()`
|
||||
|
||||
## Diagnostics and reporting
|
||||
Run-directory artifacts are owned by `internal/core/diagnostics`.
|
||||
|
||||
Stable artifact names are centralized constants (for example transcript artifacts, `invocation.json`, `effective-config.json`, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, `error.log`).
|
||||
|
||||
Report diagnostics path metadata is constructed through `BuildDiagnosticsMetadata`, which keeps run-directory artifact references consistent between success and failure reports.
|
||||
|
||||
## Secret redaction
|
||||
Redaction responsibilities are split by concern:
|
||||
- structural config redaction: `config.Config.Redacted()`
|
||||
- byte/string payload redaction for diagnostics and surfaced errors: framework redaction utilities.
|
||||
|
||||
Configured LLM secret extraction is centralized in `llm.ConfiguredSecrets(cfg)` and reused across proposal and validator diagnostics paths.
|
||||
|
||||
## Output contracts
|
||||
Transcript output schema selection is owned by `internal/core/outputschema`.
|
||||
|
||||
Supported schemas:
|
||||
- `bare-segments`
|
||||
- `audita-v1`
|
||||
|
||||
Unknown schema keys fail validation and runtime resolution.
|
||||
|
||||
## Key package map
|
||||
Core packages:
|
||||
- `internal/core/config`
|
||||
- `internal/core/schema`
|
||||
- `internal/core/normalization`
|
||||
- `internal/core/chunking`
|
||||
- `internal/core/diagnostics`
|
||||
- `internal/core/reporting`
|
||||
- `internal/core/modulecatalog`
|
||||
- `internal/core/outputschema`
|
||||
|
||||
Framework packages:
|
||||
- `internal/framework/contracts`
|
||||
- `internal/framework/proposals`
|
||||
- `internal/framework/proposal_generation`
|
||||
- `internal/framework/promptcontext`
|
||||
- `internal/framework/runner`
|
||||
- `internal/framework/validators`
|
||||
- `internal/framework/llm`
|
||||
- `internal/framework/responseschema`
|
||||
- `internal/framework/stagename`
|
||||
- `internal/framework/structuredoutput`
|
||||
|
||||
Domain packages:
|
||||
- `internal/modules/*`
|
||||
- `internal/validators/*`
|
||||
- `internal/prompts`
|
||||
@@ -1,104 +0,0 @@
|
||||
# Audita Diagnostics
|
||||
|
||||
This document describes the run-directory diagnostics artifacts produced by `audita process`.
|
||||
|
||||
## Purpose
|
||||
|
||||
Diagnostics provide machine-readable run context and execution artifacts for:
|
||||
- failure debugging;
|
||||
- validator/correction review;
|
||||
- post-run performance analysis.
|
||||
|
||||
Diagnostics are written under the configured work directory (`--work-dir`) when run-directory initialization succeeds.
|
||||
|
||||
## Core artifacts
|
||||
|
||||
Typical artifacts in each run directory:
|
||||
- `source-transcript.json`
|
||||
- `source-transcript-parsed.json`
|
||||
- `normalized-transcript.json`
|
||||
- `normalization-summary.json`
|
||||
- `chunking-summary.json`
|
||||
- `invocation.json`
|
||||
- `effective-config.json` (redacted)
|
||||
- module/validator LLM interaction artifacts
|
||||
- `report.json`
|
||||
- `error.log` on failure
|
||||
|
||||
## Utilization diagnostics artifact
|
||||
|
||||
Artifact:
|
||||
- `utilization-diagnostics.json`
|
||||
|
||||
High-level fields:
|
||||
- `effective_concurrency`:
|
||||
- total/proposal/validation LLM concurrency limits in effect.
|
||||
- `run_timing`:
|
||||
- run wall time;
|
||||
- scheduler queue wait time;
|
||||
- LLM execution time;
|
||||
- deterministic validator time;
|
||||
- max/average in-flight LLM calls.
|
||||
- `llm_calls`:
|
||||
- total proposal and validation LLM call counts.
|
||||
- `modules`:
|
||||
- module-level timing summaries.
|
||||
- `validators`:
|
||||
- per-validator timing summaries keyed by stable validator key.
|
||||
|
||||
## Correction ledger artifact
|
||||
|
||||
Artifact:
|
||||
- `correction-ledger.json`
|
||||
|
||||
Ledger records are flattened review entries derived from module results and include:
|
||||
- module/proposal identity (`module_key`, `module_instance`, `proposal_index`, `segment_id`);
|
||||
- correction text fields and replacement policy when available;
|
||||
- disposition:
|
||||
- `applied`
|
||||
- `rejected`
|
||||
- `skipped`
|
||||
- `failed`
|
||||
- stable reason codes/messages;
|
||||
- deterministic and LLM validator decision snapshots using stable validator keys.
|
||||
|
||||
Validator rejection and proposal-application skip are distinct dispositions.
|
||||
Module warnings are reported in module results and diagnostics metadata, but do not create standalone correction-ledger rows.
|
||||
|
||||
## Report references
|
||||
|
||||
`report.json` and optional `--report-json` output include diagnostics metadata paths for:
|
||||
- utilization diagnostics artifact;
|
||||
- correction ledger artifact;
|
||||
- existing transcript/normalization/chunking/invocation/effective-config artifacts.
|
||||
|
||||
Module report entries also include warning records for malformed proposal-generation payloads and malformed validator batches.
|
||||
|
||||
## Retention behavior
|
||||
|
||||
Run-directory retention follows configured policy:
|
||||
- `always`: keep all run directories;
|
||||
- `never`: keep successful run directories;
|
||||
- `auto`: keep failed runs and successful runs with skipped/rejected corrections.
|
||||
|
||||
## Redaction guarantees
|
||||
|
||||
API keys and other configured secrets are redacted from:
|
||||
- `effective-config.json`;
|
||||
- LLM interaction diagnostics artifacts;
|
||||
- reports and surfaced errors.
|
||||
|
||||
## Debugging guide
|
||||
|
||||
When debugging:
|
||||
- slow runs:
|
||||
- inspect `utilization-diagnostics.json` (`run_timing`, `modules`, `validators`, in-flight metrics).
|
||||
- validator rejections:
|
||||
- inspect `correction-ledger.json` rejected entries and matching validator decisions;
|
||||
- inspect validator response diagnostics payloads.
|
||||
- module warnings:
|
||||
- inspect module `warnings` entries in `report.json` or `--report-json`;
|
||||
- follow any diagnostic artifact path on the warning to the recorded error/response payload.
|
||||
- application skips:
|
||||
- inspect `correction-ledger.json` skipped entries and skip reason codes;
|
||||
- compare with validator decisions to distinguish validation rejection vs apply-time skip.
|
||||
@@ -1,88 +0,0 @@
|
||||
# Audita Output Schemas
|
||||
|
||||
This document describes the built-in transcript output schema registry used by `audita process`.
|
||||
|
||||
## Supported schema names
|
||||
|
||||
### `bare-segments`
|
||||
|
||||
Status:
|
||||
- implemented
|
||||
- default output schema
|
||||
|
||||
Shape:
|
||||
- top-level JSON array of transcript segments
|
||||
|
||||
Segment fields:
|
||||
- `id`
|
||||
- `speaker`
|
||||
- `start`
|
||||
- `end`
|
||||
- `text`
|
||||
- optional `categories`
|
||||
|
||||
Compatibility:
|
||||
- this preserves the long-standing output shape used by existing consumers.
|
||||
|
||||
### `audita-v1`
|
||||
|
||||
Status:
|
||||
- implemented
|
||||
|
||||
Shape:
|
||||
- top-level JSON object:
|
||||
- `schema`: `"audita-v1"`
|
||||
- `version`: `"v1"`
|
||||
- `segments`: transcript segment array
|
||||
|
||||
Segment fields inside `segments` match `bare-segments` segment fields.
|
||||
|
||||
Compatibility:
|
||||
- this is the Audita-native object format with explicit schema/version metadata.
|
||||
|
||||
### `seriatim-intermediate`
|
||||
|
||||
Status:
|
||||
- deferred / not implemented
|
||||
|
||||
Current behavior:
|
||||
- selecting `seriatim-intermediate` fails clearly as an unsupported output schema.
|
||||
|
||||
Reason:
|
||||
- a concrete, repository-backed contract for this schema has not been finalized yet.
|
||||
|
||||
## Selection
|
||||
|
||||
Choose output schema with CLI:
|
||||
|
||||
```sh
|
||||
audita process <transcript.json> --glossary <glossary.yaml> --output-schema audita-v1
|
||||
```
|
||||
|
||||
Or in file config:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
output:
|
||||
schema: audita-v1
|
||||
```
|
||||
|
||||
Precedence remains:
|
||||
1. defaults
|
||||
2. file config
|
||||
3. environment overrides
|
||||
4. CLI overrides
|
||||
|
||||
`--output-schema` overrides `output.schema` when both are supplied.
|
||||
|
||||
## Output routing behavior
|
||||
|
||||
- With `--output`, transcript JSON is written to file using the selected schema and stdout stays empty on success.
|
||||
- Without `--output`, stdout contains transcript JSON only, using the selected schema.
|
||||
- `--report-json` writes report JSON to file and does not write report payloads to stdout.
|
||||
|
||||
## Backward-compatibility expectations
|
||||
|
||||
- default schema stays `bare-segments` for compatibility unless explicitly changed in a future breaking release;
|
||||
- supported schema names are treated as stable public contract values;
|
||||
- unsupported schema names fail before output write.
|
||||
@@ -1,118 +0,0 @@
|
||||
# Audita Prompts
|
||||
|
||||
This document describes Audita's built-in embedded prompt assets and prompt registry behavior.
|
||||
|
||||
## Why embedded prompt assets
|
||||
|
||||
Audita embeds production prompt text into the binary so runtime behavior is:
|
||||
- deterministic;
|
||||
- auditable;
|
||||
- dependency-light;
|
||||
- not dependent on external prompt files at execution time.
|
||||
|
||||
Prompt text is authored as Markdown assets and rendered by Go code using typed template data.
|
||||
|
||||
## Built-in prompt registry
|
||||
|
||||
The prompt registry lives in `internal/prompts` and is responsible for:
|
||||
- loading embedded prompt assets;
|
||||
- registering stable prompt IDs and versions;
|
||||
- recording prompt source metadata;
|
||||
- computing deterministic SHA-256 source hashes;
|
||||
- rendering system/user prompts with strict missing-key failures.
|
||||
|
||||
Current prompt source behavior:
|
||||
- built-in embedded prompts only (`prompt_source = builtin`).
|
||||
- filesystem prompt overrides are not supported.
|
||||
|
||||
## Built-in prompt IDs
|
||||
|
||||
Module proposal prompts:
|
||||
- `modules.glossary.proposal`
|
||||
- `modules.homophones.proposal`
|
||||
- `modules.spoken_word.proposal`
|
||||
- `modules.grammar.proposal`
|
||||
|
||||
LLM-backed validator prompts:
|
||||
- `validators.spoken_form_plausibility`
|
||||
- `validators.meaning_reversal_review`
|
||||
- `validators.editorial_review`
|
||||
- `validators.grammar_review`
|
||||
- `validators.spoken_word_review`
|
||||
|
||||
## Prompt version semantics
|
||||
|
||||
Current built-in prompt version value is `v1`.
|
||||
|
||||
Version is a stable metadata identifier for diagnostics and debugging. It is not a dynamic prompt-selection mechanism.
|
||||
|
||||
## Prompt hash semantics
|
||||
|
||||
Each registered prompt includes a deterministic SHA-256 hash of embedded source text.
|
||||
|
||||
Hash purpose:
|
||||
- identify exact prompt source used in a run;
|
||||
- support diagnostics reproducibility and change auditing.
|
||||
|
||||
Current hash scope:
|
||||
- source prompt text (system + user assets for a registered prompt), not a runtime secret-bearing payload.
|
||||
|
||||
## Template rendering behavior
|
||||
|
||||
Prompt rendering uses Go `text/template` with typed template data from module/validator builders.
|
||||
|
||||
Missing-key behavior:
|
||||
- rendering uses missing-key errors;
|
||||
- missing/renamed template fields fail quickly instead of silently producing incomplete prompts.
|
||||
|
||||
Go code still owns:
|
||||
- structured request/response models;
|
||||
- response schema selection;
|
||||
- transcript/glossary/payload formatting;
|
||||
- module and validator selection;
|
||||
- diagnostics wiring.
|
||||
|
||||
## Shared prompt hardening policy
|
||||
|
||||
A shared hardening fragment is embedded once and included in every module proposal prompt and every LLM-validator prompt.
|
||||
|
||||
Hardening policy includes:
|
||||
- transcript text is untrusted data;
|
||||
- glossary entries and transcript descriptions are reference data, not instructions;
|
||||
- instructions found inside transcript text must not be obeyed;
|
||||
- model must perform only the requested correction/validation task;
|
||||
- no invention of facts, names, events, motivations, speaker intent, or corrections;
|
||||
- transcript remains the source of truth.
|
||||
|
||||
## Transcript description behavior
|
||||
|
||||
Transcript description remains background-only prompt context:
|
||||
- it may help interpret ambiguous terms;
|
||||
- it is explicitly non-authoritative and must not override transcript content;
|
||||
- empty descriptions do not render awkward blank context sections.
|
||||
|
||||
Generated transcript descriptions are not implemented in this workstream.
|
||||
|
||||
## Diagnostics and report metadata boundaries
|
||||
|
||||
Current metadata flow:
|
||||
- proposal-generation diagnostics request metadata includes prompt metadata;
|
||||
- LLM-validator diagnostics request metadata includes prompt metadata.
|
||||
|
||||
Prompt metadata fields used in diagnostics:
|
||||
- `prompt_id`
|
||||
- `prompt_version`
|
||||
- `prompt_source`
|
||||
- `embedded_path`
|
||||
- `sha256`
|
||||
|
||||
Current boundary:
|
||||
- detailed prompt metadata is diagnostics-first;
|
||||
- broad report-level prompt registries/ledgers are deferred.
|
||||
|
||||
## 1.0 boundary
|
||||
|
||||
Not implemented for 1.0 in this workstream:
|
||||
- filesystem prompt overrides;
|
||||
- user-configurable prompt selection;
|
||||
- external prompt directories.
|
||||
@@ -1,121 +0,0 @@
|
||||
# Audita Public Contract
|
||||
|
||||
## Scope
|
||||
This document defines stability expectations for Audita's external runtime interfaces.
|
||||
|
||||
Covered interfaces:
|
||||
- CLI commands and major flags;
|
||||
- versioned config behavior and precedence;
|
||||
- transcript/glossary input forms;
|
||||
- output schema selection;
|
||||
- report schema metadata;
|
||||
- diagnostics artifact path metadata;
|
||||
- stdout/stderr and exit-code behavior;
|
||||
- redaction guarantees.
|
||||
|
||||
## CLI contract
|
||||
Stable commands:
|
||||
- `audita process`
|
||||
- `audita config validate`
|
||||
- `audita config print-effective`
|
||||
|
||||
Stable high-value `process` flags:
|
||||
- `--config`
|
||||
- `--glossary`
|
||||
- `--output`
|
||||
- `--report-json`
|
||||
- `--modules`
|
||||
- `--output-schema`
|
||||
|
||||
## Config contract
|
||||
Supported config format:
|
||||
- YAML;
|
||||
- `version: 1`;
|
||||
- strict unknown-field rejection.
|
||||
|
||||
Path resolution for `process` and `config print-effective`:
|
||||
1. `--config`
|
||||
2. `AUDITA_CONFIG`
|
||||
3. `/usr/local/etc/audita/config.yml`
|
||||
4. `/etc/audita/config.yml`
|
||||
|
||||
Missing explicit path is an error. Missing default paths is non-fatal.
|
||||
|
||||
Precedence for `process`:
|
||||
1. defaults
|
||||
2. file config
|
||||
3. environment overrides
|
||||
4. CLI overrides
|
||||
|
||||
`config validate` remains file-only validation (defaults + file config; no env overrides).
|
||||
|
||||
Module and output-schema keys are validated against built-in catalogs. Unknown keys fail validation.
|
||||
|
||||
## Input contract
|
||||
Supported transcript JSON top-level forms:
|
||||
- array of segments
|
||||
- object with `segments` array
|
||||
|
||||
Supported glossary YAML form:
|
||||
- top-level `glossary` list with required entry fields validated by schema parsing.
|
||||
|
||||
## Output schema contract
|
||||
Supported transcript output schemas:
|
||||
- `bare-segments` (default)
|
||||
- `audita-v1`
|
||||
|
||||
Unknown schema keys fail before output write.
|
||||
|
||||
## Report metadata contract
|
||||
Process reports include stable report metadata fields:
|
||||
- `report_schema_name`
|
||||
- `report_schema_version`
|
||||
- `output_schema`
|
||||
- `config_version` (when file config is loaded)
|
||||
|
||||
Current values:
|
||||
- `report_schema_name = audita-process-report`
|
||||
- `report_schema_version = v1`
|
||||
|
||||
`--report-json` output and run-directory `report.json` use the same report schema metadata.
|
||||
|
||||
Validator decision/rejection records use stable validator keys via `validator_name`.
|
||||
|
||||
## Diagnostics metadata contract
|
||||
When run-directory initialization succeeds, diagnostics metadata paths reference stable artifacts, including:
|
||||
- transcript and normalization artifacts;
|
||||
- chunking summary;
|
||||
- invocation metadata;
|
||||
- redacted effective config;
|
||||
- utilization diagnostics;
|
||||
- correction ledger;
|
||||
- `error.log` on failures.
|
||||
|
||||
LLM interaction diagnostics include stable prompt and structured-schema identifiers where applicable.
|
||||
|
||||
## Stdout/stderr and exit codes
|
||||
Success:
|
||||
- with `--output`, stdout is empty;
|
||||
- without `--output`, stdout contains transcript JSON only;
|
||||
- report JSON is not written to stdout.
|
||||
|
||||
Failures:
|
||||
- nonzero exit;
|
||||
- human-readable stderr summary;
|
||||
- diagnostics directory path on stderr when available.
|
||||
|
||||
Exit codes:
|
||||
- `0` success
|
||||
- nonzero failure
|
||||
|
||||
## Redaction contract
|
||||
Configured secrets are redacted from:
|
||||
- effective config outputs;
|
||||
- diagnostics artifacts;
|
||||
- report artifacts;
|
||||
- surfaced adapter/runtime errors.
|
||||
|
||||
## Compatibility policy
|
||||
Stable command behavior, schema names, report metadata keys, diagnostics-path field semantics, and validator key identities are treated as public contract.
|
||||
|
||||
Additive fields are acceptable when existing fields and behavior remain compatible.
|
||||
@@ -1,72 +0,0 @@
|
||||
# Structured LLM Architecture
|
||||
|
||||
## Scope
|
||||
This document describes Audita's structured LLM runtime boundary and adapter behavior.
|
||||
|
||||
## Runtime boundary
|
||||
Production LLM integration depends on the internal contract only:
|
||||
- `contracts.StructuredLLMClient`
|
||||
- `CompleteStructured(ctx, req, out)`
|
||||
|
||||
Provider SDK types do not leak past this boundary.
|
||||
|
||||
## Adapter ownership
|
||||
`internal/framework/llm` owns the OpenAI-compatible HTTP adapter and shared LLM runtime utilities.
|
||||
|
||||
Key responsibilities:
|
||||
- request assembly;
|
||||
- timeout/cancellation propagation;
|
||||
- bounded retry behavior;
|
||||
- scheduler integration;
|
||||
- provider response decoding;
|
||||
- error redaction.
|
||||
|
||||
## Structured schema registry
|
||||
Structured response schemas are registered in `internal/framework/responseschema` and include stable metadata:
|
||||
- `id`
|
||||
- `version`
|
||||
- `name`
|
||||
- `json_schema`
|
||||
- `sha256`
|
||||
|
||||
Current schema keys:
|
||||
- `correction_set`
|
||||
- `validator_decision_set`
|
||||
|
||||
Schema metadata is attached to diagnostics through `Schema.DiagnosticsMap()`.
|
||||
|
||||
## Request shape assumptions
|
||||
Audita targets OpenAI-compatible chat-completions endpoints and sends structured requests with:
|
||||
- model;
|
||||
- chat messages;
|
||||
- `response_format.type = json_schema`;
|
||||
- schema name and JSON schema payload.
|
||||
|
||||
## Local validation remains mandatory
|
||||
Provider schema enforcement is treated as transport-level guardrails.
|
||||
|
||||
Audita still validates output locally before applying behavior changes:
|
||||
- proposal decoding and proposal invariants;
|
||||
- validator decision decoding and cardinality checks;
|
||||
- deterministic validation and apply-time rules.
|
||||
|
||||
## Shared malformed-output policy
|
||||
Malformed structured-output classification is centralized in `internal/framework/structuredoutput`.
|
||||
|
||||
Proposal generation and validator execution both use this shared classifier so downgrade behavior cannot drift between the two paths.
|
||||
|
||||
## Secrets and redaction
|
||||
Secret extraction for LLM redaction is centralized in `llm.ConfiguredSecrets(cfg)` and reused by proposal and validator diagnostics writers.
|
||||
|
||||
Secrets are redacted from:
|
||||
- diagnostics artifacts;
|
||||
- report artifacts;
|
||||
- surfaced adapter/runtime errors.
|
||||
|
||||
## Concurrency and scheduling
|
||||
LLM execution is constrained by composed scheduler limits:
|
||||
- total LLM concurrency;
|
||||
- proposal LLM concurrency;
|
||||
- validation LLM concurrency.
|
||||
|
||||
The scheduler is FIFO and context-aware so permits are released on success, failure, and cancellation.
|
||||
@@ -1,96 +0,0 @@
|
||||
# Audita Validators
|
||||
|
||||
## Scope
|
||||
This document defines the built-in validator system used by production module runs.
|
||||
|
||||
## Ownership boundaries
|
||||
Built-in validator keys, constructors, and module chains are owned by `internal/validators`.
|
||||
|
||||
Shared runtime execution mechanics are owned by `internal/framework/validators`, including:
|
||||
- validator request/result models;
|
||||
- deterministic proposal checks;
|
||||
- LLM validator batching and execution;
|
||||
- decision-cardinality enforcement;
|
||||
- diagnostics integration.
|
||||
|
||||
Execution class metadata is owned by `internal/validators/metadata`.
|
||||
|
||||
## Stable validator keys
|
||||
Deterministic:
|
||||
- `proposal_shape`
|
||||
- `confidence_threshold`
|
||||
- `original_text_presence`
|
||||
- `non_empty_corrected_text`
|
||||
- `no_effect`
|
||||
- `protected_terms`
|
||||
|
||||
LLM-backed:
|
||||
- `spoken_form_plausibility`
|
||||
- `meaning_reversal_review`
|
||||
- `editorial_review`
|
||||
|
||||
## Built-in module chains
|
||||
`glossary`:
|
||||
- `proposal_shape`
|
||||
- `no_effect`
|
||||
- `original_text_presence`
|
||||
- `confidence_threshold`
|
||||
- `protected_terms`
|
||||
- `non_empty_corrected_text`
|
||||
- `spoken_form_plausibility`
|
||||
- `meaning_reversal_review`
|
||||
|
||||
`homophones`:
|
||||
- `proposal_shape`
|
||||
- `no_effect`
|
||||
- `original_text_presence`
|
||||
- `confidence_threshold`
|
||||
- `protected_terms`
|
||||
- `non_empty_corrected_text`
|
||||
- `spoken_form_plausibility`
|
||||
- `meaning_reversal_review`
|
||||
|
||||
`spoken_word`:
|
||||
- `proposal_shape`
|
||||
- `no_effect`
|
||||
- `original_text_presence`
|
||||
- `confidence_threshold`
|
||||
- `protected_terms`
|
||||
- `non_empty_corrected_text`
|
||||
- `editorial_review`
|
||||
- `meaning_reversal_review`
|
||||
|
||||
`grammar`:
|
||||
- `proposal_shape`
|
||||
- `no_effect`
|
||||
- `original_text_presence`
|
||||
- `confidence_threshold`
|
||||
- `protected_terms`
|
||||
- `non_empty_corrected_text`
|
||||
- `editorial_review`
|
||||
- `meaning_reversal_review`
|
||||
|
||||
## Ordering and execution semantics
|
||||
Validator ordering is based on canonical metadata:
|
||||
- deterministic validators run before LLM-backed validators.
|
||||
|
||||
Within each module stage:
|
||||
- proposals are generated per section;
|
||||
- validator chains execute on those proposals;
|
||||
- approved proposals are applied once after section work settles.
|
||||
|
||||
## Malformed payload behavior
|
||||
Malformed structured-output from proposal generation and LLM validator calls is downgraded, not treated as a process-fatal transport error.
|
||||
|
||||
Current outcomes:
|
||||
- malformed proposal-generation payloads produce section/module warnings and zero proposals for the affected section;
|
||||
- malformed validator decision payloads reject the affected validator batch with warnings;
|
||||
- deterministic validator behavior and runner order remain unchanged.
|
||||
|
||||
## Reporting identity
|
||||
Reports and diagnostics use stable validator keys as identifiers.
|
||||
|
||||
Correction-ledger deterministic-vs-LLM classification is derived from canonical validator metadata, not package-local hardcoded maps.
|
||||
|
||||
## Prompt assets
|
||||
LLM validator prompt assets and prompt metadata are documented in [Prompts](./prompts.md).
|
||||
186
docs/cli.md
Normal file
186
docs/cli.md
Normal file
@@ -0,0 +1,186 @@
|
||||
# Audita CLI Reference
|
||||
|
||||
## Shortest Useful Command
|
||||
|
||||
```sh
|
||||
audita process <transcript.json> --glossary <glossary.yaml> --output <corrected.json>
|
||||
```
|
||||
|
||||
This command validates input files, runs the configured correction pipeline, and writes corrected transcript JSON.
|
||||
|
||||
## Command Overview
|
||||
|
||||
- `audita process`: process one transcript JSON file.
|
||||
- `audita config validate`: validate a versioned YAML config file.
|
||||
- `audita config print-effective`: print redacted effective config JSON.
|
||||
|
||||
General help:
|
||||
|
||||
```sh
|
||||
audita --help
|
||||
audita process --help
|
||||
audita config --help
|
||||
```
|
||||
|
||||
## `process`
|
||||
|
||||
Usage:
|
||||
|
||||
```sh
|
||||
audita process <transcript.json> [flags]
|
||||
```
|
||||
|
||||
Input requirements:
|
||||
- exactly one transcript JSON positional argument is required;
|
||||
- `--glossary <path>` is required.
|
||||
|
||||
Config path selection for `process`:
|
||||
1. `--config <path>`
|
||||
2. `AUDITA_CONFIG`
|
||||
3. `/usr/local/etc/audita/config.yml` (if present)
|
||||
4. `/etc/audita/config.yml` (if present)
|
||||
|
||||
For precedence and full config schema, see [`docs/config.md`](config.md).
|
||||
|
||||
### `process` Flag Reference
|
||||
|
||||
Core I/O flags:
|
||||
- `--config <path>`: path to versioned YAML config file.
|
||||
- `--glossary <path>`: glossary YAML input path (required).
|
||||
- `--output <path>`: corrected transcript JSON output file path.
|
||||
- `--report-json <path>`: machine-readable report JSON output path.
|
||||
- `--output-schema <key>`: output schema key (`bare-segments` or `audita-v1`).
|
||||
- `--modules <csv>`: comma-separated module sequence override.
|
||||
|
||||
Primary LLM flags:
|
||||
- `--llm-api-key <value>`: primary LLM API key.
|
||||
- `--model <name>`: primary LLM model name.
|
||||
- `--base-url <url>`: primary OpenAI-compatible base URL.
|
||||
- `--llm-timeout-seconds <int>`: primary timeout in seconds.
|
||||
- `--max-retries <int>`: primary structured-output retries.
|
||||
|
||||
Validation LLM flags:
|
||||
- `--validation-llm-api-key <value>`: validation LLM API key.
|
||||
- `--validation-model <name>`: validation LLM model name.
|
||||
- `--validation-base-url <url>`: validation OpenAI-compatible base URL.
|
||||
- `--validation-llm-timeout-seconds <int>`: validation timeout in seconds.
|
||||
- `--validation-max-retries <int>`: validation structured-output retries.
|
||||
- `--validation-max-prompt-tokens <int>`: validation max prompt tokens.
|
||||
|
||||
Concurrency flags:
|
||||
- `--total-llm-concurrency <int>`: total concurrent proposal+validation LLM calls.
|
||||
- `--proposal-llm-concurrency <int>`: concurrent proposal-generation LLM calls.
|
||||
- `--validation-llm-concurrency <int>`: concurrent validation LLM calls.
|
||||
- `--llm-concurrency <int>`: alias for `--total-llm-concurrency`.
|
||||
|
||||
Chunking and normalization flags:
|
||||
- `--target-sections <int>`: target number of transcript sections.
|
||||
- `--max-section-tokens <int>`: maximum section tokens.
|
||||
- `--min-section-tokens <int>`: minimum section tokens.
|
||||
- `--normalize-max-segment-gap <float>`: maximum same-speaker merge gap.
|
||||
- `--normalize-ellipsis-gap <float>`: gap threshold for ellipsis insertion.
|
||||
- `--normalize-max-segment-duration <float>`: maximum merged segment duration.
|
||||
- `--normalize-max-segment-tokens <int>`: maximum merged segment token estimate.
|
||||
|
||||
Threshold flags:
|
||||
- `--glossary-confidence-threshold <float>`
|
||||
- `--homophones-confidence-threshold <float>`
|
||||
- `--spoken-word-confidence-threshold <float>`
|
||||
- `--grammar-confidence-threshold <float>`
|
||||
|
||||
Context and diagnostics flags:
|
||||
- `--transcript-description <text>`: background context for prompts; does not override transcript content.
|
||||
- `--work-dir <path>`: per-run diagnostics work directory.
|
||||
- `--work-dir-retention <auto|always|never>`: run-directory retention policy.
|
||||
|
||||
### `process` Output and Exit Behavior
|
||||
|
||||
- With `--output`: stdout is expected to be empty on success.
|
||||
- Without `--output`: stdout contains transcript JSON only on success.
|
||||
- `--report-json` writes a file and is never printed to stdout.
|
||||
- Stderr is human-readable diagnostics/errors.
|
||||
- On failures after diagnostics initialization, stderr includes the diagnostics directory path.
|
||||
|
||||
Exit behavior:
|
||||
- `0`: success.
|
||||
- `1`: runtime failure during processing/reporting/output paths.
|
||||
- `2`: CLI usage or configuration input error.
|
||||
|
||||
Integration references:
|
||||
- subprocess contract: [`docs/integrations/subprocess.md`](integrations/subprocess.md)
|
||||
- transcript/glossary file contract: [`docs/integrations/transcript-glossary-files.md`](integrations/transcript-glossary-files.md)
|
||||
|
||||
### `process` Examples
|
||||
|
||||
Write corrected transcript to a file:
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--output corrected.json
|
||||
```
|
||||
|
||||
Emit transcript JSON to stdout:
|
||||
|
||||
```sh
|
||||
audita process transcript.json --glossary glossary.yaml
|
||||
```
|
||||
|
||||
Use explicit config and write report JSON:
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--config audita.yml \
|
||||
--output corrected.json \
|
||||
--report-json report.json
|
||||
```
|
||||
|
||||
Override the module sequence:
|
||||
|
||||
```sh
|
||||
audita process transcript.json \
|
||||
--glossary glossary.yaml \
|
||||
--modules glossary,homophones,grammar \
|
||||
--output corrected.json
|
||||
```
|
||||
|
||||
## `config validate`
|
||||
|
||||
Usage:
|
||||
|
||||
```sh
|
||||
audita config validate --config <path>
|
||||
```
|
||||
|
||||
Behavior:
|
||||
- validates defaults merged with file config;
|
||||
- does not apply environment overrides;
|
||||
- prints `config is valid` on success.
|
||||
|
||||
Errors:
|
||||
- `--config` is required;
|
||||
- positional arguments are rejected;
|
||||
- validation failures are printed to stderr.
|
||||
|
||||
## `config print-effective`
|
||||
|
||||
Usage:
|
||||
|
||||
```sh
|
||||
audita config print-effective [--config <path>]
|
||||
```
|
||||
|
||||
Config path selection:
|
||||
1. `--config <path>` when provided
|
||||
2. `AUDITA_CONFIG`
|
||||
3. `/usr/local/etc/audita/config.yml` (if present)
|
||||
4. `/etc/audita/config.yml` (if present)
|
||||
|
||||
Behavior:
|
||||
- merges defaults, optional config file, and environment overrides;
|
||||
- prints redacted JSON to stdout.
|
||||
|
||||
Errors:
|
||||
- positional arguments are rejected;
|
||||
- resolution or parse failures are printed to stderr.
|
||||
238
docs/config.md
Normal file
238
docs/config.md
Normal file
@@ -0,0 +1,238 @@
|
||||
# Audita Configuration
|
||||
|
||||
## Scope
|
||||
|
||||
This is the canonical configuration reference for Audita.
|
||||
|
||||
It documents:
|
||||
- config path resolution;
|
||||
- effective precedence across defaults, file config, environment, and CLI;
|
||||
- supported `version: 1` YAML schema;
|
||||
- environment overrides;
|
||||
- CLI override relationship;
|
||||
- validation and secrets behavior.
|
||||
|
||||
For CLI command syntax, see [`docs/cli.md`](cli.md).
|
||||
For OpenAI-compatible endpoint behavior, see [`docs/integrations/openai-compatible-llm.md`](integrations/openai-compatible-llm.md).
|
||||
For transcript/glossary input file contracts, see [`docs/integrations/transcript-glossary-files.md`](integrations/transcript-glossary-files.md).
|
||||
|
||||
## Loading Model
|
||||
|
||||
Path resolution for `audita process` and `audita config print-effective`:
|
||||
1. `--config <path>`
|
||||
2. `AUDITA_CONFIG`
|
||||
3. `/usr/local/etc/audita/config.yml` (if present)
|
||||
4. `/etc/audita/config.yml` (if present)
|
||||
|
||||
Missing explicit path behavior:
|
||||
- missing `--config` target is an error;
|
||||
- missing `AUDITA_CONFIG` target is an error.
|
||||
|
||||
Missing default-path files are non-fatal.
|
||||
|
||||
## Effective Precedence
|
||||
|
||||
`audita process`:
|
||||
1. defaults
|
||||
2. file config
|
||||
3. environment overrides
|
||||
4. CLI overrides
|
||||
|
||||
`audita config print-effective`:
|
||||
1. defaults
|
||||
2. file config
|
||||
3. environment overrides
|
||||
|
||||
`audita config validate`:
|
||||
1. defaults
|
||||
2. file config
|
||||
|
||||
`config validate` is intentionally file-only (no environment overrides).
|
||||
|
||||
## Defaults
|
||||
|
||||
Current defaults:
|
||||
- modules: `glossary,homophones,glossary,spoken_word,grammar`
|
||||
- output schema: `bare-segments`
|
||||
- primary model: `openrouter/google/gemma-4-31b-it`
|
||||
- primary base URL: `https://openrouter.ai/api/v1`
|
||||
- primary timeout: `600` seconds
|
||||
- max retries: `3`
|
||||
- total/proposal LLM concurrency: `1`
|
||||
- validation max prompt tokens: `2048`
|
||||
- max section tokens: `8192`
|
||||
- min section tokens: `2048`
|
||||
- confidence thresholds: `0.8`
|
||||
- normalization max segment gap: `4.0`
|
||||
- normalization ellipsis gap: `3.5`
|
||||
- normalization max segment duration: `60.0`
|
||||
- normalization max segment tokens: `2048`
|
||||
- transcript description: empty
|
||||
- work dir: `/tmp/audita`
|
||||
- work dir retention: `auto`
|
||||
|
||||
## YAML Schema (`version: 1`)
|
||||
|
||||
Supported file version:
|
||||
- `version: 1` (required)
|
||||
|
||||
Unknown YAML fields are rejected.
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
|
||||
pipeline:
|
||||
modules: [glossary, homophones, glossary, spoken_word, grammar]
|
||||
|
||||
output:
|
||||
schema: bare-segments
|
||||
|
||||
llm:
|
||||
proposal:
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
model: openrouter/google/gemma-4-31b-it
|
||||
api_key_env: AUDITA_LLM_API_KEY
|
||||
timeout: 600s
|
||||
max_retries: 3
|
||||
validation:
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
model: openrouter/google/gemma-4-31b-it
|
||||
api_key_env: AUDITA_VALIDATION_LLM_API_KEY
|
||||
timeout: 600
|
||||
max_retries: 3
|
||||
|
||||
concurrency:
|
||||
total_llm: 1
|
||||
proposal_llm: 1
|
||||
validation_llm: 1
|
||||
|
||||
chunking:
|
||||
target_sections: 8
|
||||
max_section_tokens: 8192
|
||||
min_section_tokens: 2048
|
||||
|
||||
normalization:
|
||||
max_segment_gap: 4s
|
||||
ellipsis_gap: 3.5s
|
||||
max_segment_duration: 60s
|
||||
max_segment_tokens: 2048
|
||||
|
||||
thresholds:
|
||||
glossary: 0.8
|
||||
homophones: 0.8
|
||||
spoken_word: 0.8
|
||||
grammar: 0.8
|
||||
|
||||
context:
|
||||
description: optional background context
|
||||
|
||||
diagnostics:
|
||||
work_dir: /tmp/audita
|
||||
retention: auto
|
||||
```
|
||||
|
||||
Duration-parsing behavior:
|
||||
- `llm.*.timeout`: integer seconds or duration string; duration strings must resolve to whole seconds.
|
||||
- `normalization.*` duration-like fields: numeric seconds or duration string.
|
||||
|
||||
## Environment Overrides
|
||||
|
||||
Modules:
|
||||
- `AUDITA_MODULES`
|
||||
|
||||
Config path:
|
||||
- `AUDITA_CONFIG`
|
||||
|
||||
Primary LLM:
|
||||
- `AUDITA_LLM_API_KEY` (falls back to `OPENROUTER_API_KEY` when unset)
|
||||
- `AUDITA_MODEL`
|
||||
- `AUDITA_BASE_URL`
|
||||
- `AUDITA_LLM_TIMEOUT_SECONDS`
|
||||
- `AUDITA_MAX_RETRIES`
|
||||
|
||||
Validation LLM:
|
||||
- `AUDITA_VALIDATION_LLM_API_KEY`
|
||||
- `AUDITA_VALIDATION_MODEL`
|
||||
- `AUDITA_VALIDATION_BASE_URL`
|
||||
- `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS`
|
||||
- `AUDITA_VALIDATION_MAX_RETRIES`
|
||||
- `AUDITA_VALIDATION_MAX_PROMPT_TOKENS`
|
||||
|
||||
Concurrency:
|
||||
- `AUDITA_TOTAL_LLM_CONCURRENCY`
|
||||
- `AUDITA_PROPOSAL_LLM_CONCURRENCY`
|
||||
- `AUDITA_VALIDATION_LLM_CONCURRENCY`
|
||||
- `AUDITA_LLM_CONCURRENCY` (legacy alias for total)
|
||||
|
||||
Chunking:
|
||||
- `AUDITA_MAX_SECTION_TOKENS`
|
||||
- `AUDITA_MIN_SECTION_TOKENS`
|
||||
- `AUDITA_TARGET_SECTIONS`
|
||||
|
||||
Thresholds:
|
||||
- `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD`
|
||||
- `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD`
|
||||
- `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD`
|
||||
- `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD`
|
||||
|
||||
Normalization:
|
||||
- `AUDITA_NORMALIZE_MAX_SEGMENT_GAP`
|
||||
- `AUDITA_NORMALIZE_ELLIPSIS_GAP`
|
||||
- `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION`
|
||||
- `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS`
|
||||
|
||||
Diagnostics:
|
||||
- `AUDITA_WORK_DIR`
|
||||
- `AUDITA_WORK_DIR_RETENTION` (`auto`, `always`, `never`)
|
||||
|
||||
Transcript description:
|
||||
- no `AUDITA_*` environment variable is currently defined.
|
||||
|
||||
## CLI Override Relationship
|
||||
|
||||
CLI flags override file and environment values for `audita process`.
|
||||
|
||||
The CLI supports canonical total concurrency (`--total-llm-concurrency`) and legacy alias (`--llm-concurrency`):
|
||||
- when both are provided at the same precedence layer, canonical total wins;
|
||||
- if proposal concurrency is not explicitly set and total is set via environment or CLI, proposal concurrency inherits that total;
|
||||
- validation concurrency inherits total only when validation concurrency is unset.
|
||||
|
||||
For full flag syntax, see [`docs/cli.md`](cli.md).
|
||||
|
||||
## Validation Rules
|
||||
|
||||
Validation includes:
|
||||
- supported module keys only;
|
||||
- supported output schema keys only (`bare-segments`, `audita-v1`);
|
||||
- positive timeout/concurrency/token constraints;
|
||||
- `proposal_llm <= total_llm` and `validation_llm <= total_llm` when validation is set;
|
||||
- confidence thresholds in `[0.0, 1.0]`;
|
||||
- transcript description length `<= 500` characters;
|
||||
- non-empty work dir;
|
||||
- work-dir retention in `auto|always|never`.
|
||||
|
||||
## Secrets
|
||||
|
||||
Recommended secret handling:
|
||||
- use `llm.proposal.api_key_env` and `llm.validation.api_key_env` in file config;
|
||||
- use `AUDITA_*_API_KEY` environment overrides or CLI key flags when needed.
|
||||
|
||||
`api_key_env` fields contain environment variable names, not secret values.
|
||||
|
||||
Redaction behavior:
|
||||
- `audita config print-effective` redacts resolved API keys.
|
||||
- diagnostics and report paths redact configured secret values.
|
||||
|
||||
## Examples
|
||||
|
||||
- Minimal config: [`examples/minimal-config.yml`](../examples/minimal-config.yml)
|
||||
- Production-style config: [`examples/production-config.yml`](../examples/production-config.yml)
|
||||
- Tiny transcript input: [`examples/tiny-transcript.json`](../examples/tiny-transcript.json)
|
||||
- Tiny glossary input: [`examples/tiny-glossary.yaml`](../examples/tiny-glossary.yaml)
|
||||
|
||||
Validate the config examples:
|
||||
|
||||
```sh
|
||||
audita config validate --config examples/minimal-config.yml
|
||||
audita config validate --config examples/production-config.yml
|
||||
```
|
||||
@@ -1,149 +0,0 @@
|
||||
# Audita Configuration
|
||||
|
||||
## Scope
|
||||
This document defines the supported versioned YAML configuration model and runtime precedence behavior.
|
||||
|
||||
## Supported file version
|
||||
Current supported config file version:
|
||||
- `version: 1`
|
||||
|
||||
Validation rules:
|
||||
- missing `version` fails;
|
||||
- unsupported version fails;
|
||||
- unknown YAML fields fail (strict decoding).
|
||||
|
||||
## Config path resolution
|
||||
For `audita process` and `audita config print-effective`, path resolution order is:
|
||||
1. `--config <path>`
|
||||
2. `AUDITA_CONFIG`
|
||||
3. `/usr/local/etc/audita/config.yml` (if present)
|
||||
4. `/etc/audita/config.yml` (if present)
|
||||
|
||||
Missing-path behavior:
|
||||
- missing `--config` path is an error;
|
||||
- missing `AUDITA_CONFIG` path is an error;
|
||||
- missing both default paths is non-fatal.
|
||||
|
||||
## Effective precedence
|
||||
`audita process` effective precedence:
|
||||
1. defaults
|
||||
2. file config
|
||||
3. environment overrides
|
||||
4. CLI overrides
|
||||
|
||||
`audita config print-effective` uses:
|
||||
1. defaults
|
||||
2. file config
|
||||
3. environment overrides
|
||||
|
||||
`audita config validate` intentionally uses file-only validation:
|
||||
1. defaults
|
||||
2. file config
|
||||
|
||||
Environment overrides are not applied in `config validate`.
|
||||
|
||||
## Supported top-level YAML fields
|
||||
```yaml
|
||||
version: 1
|
||||
|
||||
pipeline:
|
||||
modules: [glossary, homophones, glossary, spoken_word, grammar]
|
||||
|
||||
output:
|
||||
schema: bare-segments
|
||||
|
||||
llm:
|
||||
proposal:
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
model: openrouter/google/gemma-4-31b-it
|
||||
api_key_env: AUDITA_LLM_API_KEY
|
||||
timeout: 120s
|
||||
max_retries: 3
|
||||
validation:
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
model: openrouter/google/gemma-4-31b-it
|
||||
api_key_env: AUDITA_VALIDATION_LLM_API_KEY
|
||||
timeout: 120s
|
||||
max_retries: 3
|
||||
|
||||
concurrency:
|
||||
total_llm: 2
|
||||
proposal_llm: 2
|
||||
validation_llm: 1
|
||||
|
||||
chunking:
|
||||
target_sections: 8
|
||||
max_section_tokens: 8192
|
||||
min_section_tokens: 2048
|
||||
|
||||
normalization:
|
||||
max_segment_gap: 4s
|
||||
ellipsis_gap: 3.5s
|
||||
max_segment_duration: 60s
|
||||
max_segment_tokens: 2048
|
||||
|
||||
thresholds:
|
||||
glossary: 0.8
|
||||
homophones: 0.8
|
||||
spoken_word: 0.8
|
||||
grammar: 0.8
|
||||
|
||||
context:
|
||||
description: "optional transcript background context"
|
||||
|
||||
diagnostics:
|
||||
work_dir: /tmp/audita
|
||||
retention: auto
|
||||
```
|
||||
|
||||
## Module and output-schema validation
|
||||
`pipeline.modules` keys are validated against the built-in supported module catalog.
|
||||
|
||||
Supported module keys:
|
||||
- `glossary`
|
||||
- `homophones`
|
||||
- `spoken_word`
|
||||
- `grammar`
|
||||
|
||||
Repeated supported module keys are allowed.
|
||||
|
||||
`output.schema` is validated against the built-in output schema catalog.
|
||||
|
||||
Supported output schema keys:
|
||||
- `bare-segments`
|
||||
- `audita-v1`
|
||||
|
||||
Unknown module keys and unknown output schema keys fail validation.
|
||||
|
||||
## Duration field parsing
|
||||
Duration-like fields support:
|
||||
- numeric seconds (for example `120`, `3.5`)
|
||||
- duration strings (for example `120s`, `2m`)
|
||||
|
||||
LLM timeout duration strings must resolve to whole seconds.
|
||||
|
||||
## Secret handling
|
||||
Use `api_key_env` fields for secrets:
|
||||
- `llm.proposal.api_key_env`
|
||||
- `llm.validation.api_key_env`
|
||||
|
||||
These fields store environment variable names, not secret values.
|
||||
|
||||
Resolved secret values are redacted from:
|
||||
- `audita config print-effective` output;
|
||||
- diagnostics `effective-config.json`;
|
||||
- report and diagnostics payloads.
|
||||
|
||||
## Commands
|
||||
Validate a file config:
|
||||
```sh
|
||||
audita config validate --config ./audita.yml
|
||||
```
|
||||
|
||||
Print redacted effective config:
|
||||
```sh
|
||||
audita config print-effective --config ./audita.yml
|
||||
```
|
||||
|
||||
## Compatibility notes
|
||||
Legacy compatibility flags and environment aliases remain available where implemented, but the stable configuration surface is the versioned YAML model described above.
|
||||
@@ -1,33 +0,0 @@
|
||||
# Audita Development Workflow
|
||||
|
||||
## Scope
|
||||
This document defines the canonical contributor workflow and engineering conventions for this repository.
|
||||
|
||||
## Workflow
|
||||
1. Start from a clean understanding of scope and constraints.
|
||||
2. Make focused changes that preserve existing public behavior unless behavior change is explicitly intended.
|
||||
3. Run targeted tests for touched packages.
|
||||
4. Run `go test ./...` before finalizing substantial changes.
|
||||
5. Update affected documentation so it describes current behavior only.
|
||||
|
||||
## Engineering conventions
|
||||
- Keep module packages separate: `glossary`, `homophones`, `spoken_word`, `grammar`.
|
||||
- Prefer narrow shared helpers and catalogs over broad abstractions.
|
||||
- Preserve diagnostics artifact naming and report field contracts unless intentionally changed.
|
||||
- Preserve CLI/config precedence semantics unless intentionally changed.
|
||||
- Treat stable validator keys, prompt identifiers, and output-schema keys as contract surfaces.
|
||||
|
||||
## Configuration and runtime expectations
|
||||
- `audita process` precedence is defaults -> file -> env -> CLI.
|
||||
- `audita config validate` validates file config merged onto defaults only.
|
||||
- `audita config print-effective` includes environment overrides and prints redacted JSON.
|
||||
|
||||
## Testing expectations
|
||||
- Add tests for new behavior and for bug fixes.
|
||||
- Keep deterministic fixtures stable.
|
||||
- Do not reduce existing parity, release-fixture, subprocess, or module-specific coverage without equivalent replacement.
|
||||
|
||||
## Commit discipline
|
||||
- Keep commits scoped and reviewable.
|
||||
- Avoid mixing unrelated refactors with behavior changes.
|
||||
- Use clear plain-English commit messages.
|
||||
@@ -1,27 +0,0 @@
|
||||
# Documentation Policy
|
||||
|
||||
## Scope
|
||||
This policy defines how project documentation should be authored and maintained.
|
||||
|
||||
## Core rules
|
||||
- Document the current behavior of the codebase.
|
||||
- Remove stale behavior descriptions promptly when code changes.
|
||||
- Do not describe development history in architecture or behavior docs unless a document is explicitly historical.
|
||||
- Do not use architecture or behavior docs as changelogs.
|
||||
- Prefer rewriting stale sections from scratch when substantial behavior or ownership changes occur.
|
||||
|
||||
## Consistency requirements
|
||||
- Keep command examples aligned with current CLI surfaces.
|
||||
- Keep configuration examples aligned with supported fields and precedence.
|
||||
- Keep architecture package ownership descriptions aligned with current code layout.
|
||||
- Keep stable contract identifiers accurate (module keys, validator keys, output-schema keys, report metadata fields).
|
||||
|
||||
## Cross-document expectations
|
||||
- `docs/architecture/*` documents runtime behavior and package ownership.
|
||||
- `docs/configuration.md` documents config schema and precedence.
|
||||
- `docs/development.md` documents contributor workflow and engineering conventions.
|
||||
|
||||
## Review expectations for documentation changes
|
||||
- Verify referenced files and links exist.
|
||||
- Verify examples match current behavior.
|
||||
- Prefer concise, direct language and avoid speculative future claims.
|
||||
@@ -1,96 +0,0 @@
|
||||
# Audita Subprocess Operations
|
||||
|
||||
This document describes how parent processes should invoke `audita process` safely in production orchestration.
|
||||
|
||||
## Recommended command form
|
||||
|
||||
Use explicit file outputs for orchestrated runs:
|
||||
|
||||
```sh
|
||||
audita process <transcript.json> \
|
||||
--transcript-description "Brief context that may help resolve ambiguous terms." \
|
||||
--glossary <glossary.yaml> \
|
||||
--output <output-transcript.json> \
|
||||
--report-json <report.json>
|
||||
```
|
||||
|
||||
Additional flags that may be situationally appropriate:
|
||||
- `--config <path>` to select an explicit versioned config file.
|
||||
- `--output-schema <bare-segments|audita-v1>` to select transcript output shape.
|
||||
- `--work-dir <dir>` to control diagnostics location.
|
||||
- `--work-dir-retention <always|auto|never>` to control retained run directories.
|
||||
- `--total-llm-concurrency`, `--proposal-llm-concurrency`, and `--validation-llm-concurrency` when orchestration needs to set explicit LLM throughput controls.
|
||||
- `--modules ...` only when intentionally overriding the default sequence.
|
||||
|
||||
For config-driven orchestration, validate config files in CI/preflight:
|
||||
|
||||
```sh
|
||||
audita config validate --config <path>
|
||||
```
|
||||
|
||||
## Stdout behavior
|
||||
|
||||
- With `--output`: stdout is expected to be empty on success.
|
||||
- Without `--output`: stdout contains transcript JSON only on success.
|
||||
- Report JSON is never written to stdout.
|
||||
|
||||
## Stderr behavior
|
||||
|
||||
- Success path should be quiet or minimal human-readable logs.
|
||||
- Failure path writes concise human-readable errors.
|
||||
- When a diagnostics run directory exists, failure stderr includes its path.
|
||||
- Prompt/response diagnostic payloads are not streamed to stderr.
|
||||
|
||||
## Output file behavior
|
||||
|
||||
- `--output` writes transcript JSON in the selected output schema to the provided path.
|
||||
- Output write failures return nonzero and surface actionable errors.
|
||||
- The command does not silently ignore output write errors.
|
||||
|
||||
## Report JSON behavior
|
||||
|
||||
- `--report-json` writes a machine-readable process report to the requested path.
|
||||
- Run-directory `report.json` is written independently under diagnostics.
|
||||
- Best-effort failure reports are emitted when possible without masking the primary failure.
|
||||
- Report write failures return nonzero with clear stderr messaging.
|
||||
- Report diagnostics metadata references run-directory artifacts including utilization diagnostics and correction ledger paths when available.
|
||||
|
||||
## Diagnostics directory behavior
|
||||
|
||||
- Each run creates (when possible) a per-run diagnostics directory.
|
||||
- Typical artifacts include transcript, normalization, chunking, invocation, effective config, LLM diagnostics, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, and `error.log` on failure.
|
||||
- Failed runs retain diagnostics.
|
||||
- Under `auto` retention, successful runs with skipped/rejected corrections are retained; clean successful runs may be removed.
|
||||
|
||||
## Exit codes
|
||||
|
||||
- `0`: success.
|
||||
- Nonzero: failure (input/schema/config/module/LLM/runtime/output/report/diagnostics errors).
|
||||
|
||||
Treat any nonzero as a failed subprocess invocation.
|
||||
|
||||
## Timeout and cancellation
|
||||
|
||||
- Runtime operations propagate context cancellation and request timeouts through LLM/scheduler paths.
|
||||
- On cancellation or timeout, the process exits nonzero and should not hang.
|
||||
- If diagnostics were initialized before failure, failure artifacts remain available for debugging.
|
||||
|
||||
## Secret redaction expectations
|
||||
|
||||
API keys and configured secret values are redacted from:
|
||||
- reports (`--report-json` and run-dir `report.json`);
|
||||
- diagnostics artifacts (including effective config and LLM interaction artifacts);
|
||||
- surfaced adapter/runtime errors;
|
||||
- test fixtures and regression outputs.
|
||||
|
||||
Parent-process logs should still avoid printing raw environment variables.
|
||||
|
||||
## Parent-process pipe guidance
|
||||
|
||||
To avoid deadlocks in orchestrators:
|
||||
- always read both stdout and stderr concurrently when invoking as a subprocess;
|
||||
- prefer file outputs (`--output`, `--report-json`) for machine workflows;
|
||||
- treat stderr as human-readable diagnostics, not structured data;
|
||||
- parse structured results from output/report files.
|
||||
|
||||
For Go callers, prefer `exec.CommandContext` with explicit timeout/cancellation and buffered/streamed readers for both pipes.
|
||||
119
docs/integrations/openai-compatible-llm.md
Normal file
119
docs/integrations/openai-compatible-llm.md
Normal file
@@ -0,0 +1,119 @@
|
||||
# OpenAI-Compatible LLM Integration
|
||||
|
||||
## Scope
|
||||
|
||||
This document defines the external LLM endpoint contract Audita currently uses.
|
||||
|
||||
It covers:
|
||||
- endpoint and auth expectations;
|
||||
- structured request and response shape;
|
||||
- retry and timeout behavior;
|
||||
- diagnostics and secret redaction.
|
||||
|
||||
For user-facing CLI flags and config keys, see [`docs/cli.md`](../cli.md) and [`docs/config.md`](../config.md).
|
||||
|
||||
## Endpoint Contract
|
||||
|
||||
Audita sends HTTPS `POST` requests to:
|
||||
|
||||
- `<base_url>/chat/completions`
|
||||
|
||||
`base_url` comes from primary or validation LLM config and is required.
|
||||
|
||||
## Authentication Contract
|
||||
|
||||
When an API key is configured, Audita sends:
|
||||
|
||||
- `Authorization: Bearer <api_key>`
|
||||
|
||||
When no API key is configured, the `Authorization` header is omitted.
|
||||
|
||||
## Request Shape
|
||||
|
||||
Audita sends a chat-completions payload with:
|
||||
- `model`;
|
||||
- `messages` (role/content pairs);
|
||||
- `response_format` using JSON Schema strict mode.
|
||||
|
||||
Representative shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "example-model",
|
||||
"messages": [
|
||||
{"role": "system", "content": "..."},
|
||||
{"role": "user", "content": "..."}
|
||||
],
|
||||
"response_format": {
|
||||
"type": "json_schema",
|
||||
"json_schema": {
|
||||
"name": "correction_set",
|
||||
"strict": true,
|
||||
"schema": {"type": "object"}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Behavioral requirements enforced by Audita:
|
||||
- `model` must resolve to a non-empty value;
|
||||
- each message must have non-empty `role` and `content`;
|
||||
- `response_format.type` is always `json_schema`;
|
||||
- `response_format.json_schema.name` and `schema` must be present;
|
||||
- request schema JSON must be valid JSON.
|
||||
|
||||
## Response Handling Contract
|
||||
|
||||
Audita expects a successful JSON response with at least one choice and assistant content that can be interpreted as JSON.
|
||||
|
||||
Supported assistant content forms:
|
||||
- string containing JSON;
|
||||
- raw JSON value.
|
||||
|
||||
Audita then decodes the JSON against the expected structured output type.
|
||||
|
||||
Current structured schema identities used by Audita runtime:
|
||||
- `correction_set`
|
||||
- `validator_decision_set`
|
||||
|
||||
## Retries and Timeouts
|
||||
|
||||
Retry behavior:
|
||||
- default max retries is `3` when unset;
|
||||
- retries apply to retryable transport/decode/server-side errors;
|
||||
- HTTP `429` and `5xx` responses are retryable;
|
||||
- retry stops immediately when context is canceled or deadline expires.
|
||||
|
||||
Timeout behavior:
|
||||
- request timeout is derived from configured LLM timeout settings;
|
||||
- timeout/cancellation propagate through HTTP requests and return nonzero process failures.
|
||||
|
||||
## Error Behavior
|
||||
|
||||
Non-2xx responses fail the request.
|
||||
|
||||
Error message extraction behavior:
|
||||
- if provider JSON includes `error.message`, Audita surfaces that message;
|
||||
- else if provider JSON includes top-level `message`, Audita surfaces that;
|
||||
- otherwise Audita surfaces status code plus response body text.
|
||||
|
||||
Malformed or incompatible structured responses fail safely and are surfaced as runtime errors or validator/proposal warnings depending on call site.
|
||||
|
||||
## Secret Redaction
|
||||
|
||||
Configured LLM secrets are redacted from:
|
||||
- surfaced adapter/runtime errors;
|
||||
- LLM diagnostics request/response/error artifacts;
|
||||
- effective config/report artifacts that include LLM configuration material.
|
||||
|
||||
Redaction marker:
|
||||
- `[REDACTED]`
|
||||
|
||||
## Compatibility Boundaries
|
||||
|
||||
This integration documentation applies only to the implemented OpenAI-compatible chat completions flow.
|
||||
|
||||
Not part of current behavior:
|
||||
- provider SDK integration;
|
||||
- non-OpenAI-compatible API contracts;
|
||||
- server-side model routing features beyond explicitly configured model/base URL.
|
||||
99
docs/integrations/subprocess.md
Normal file
99
docs/integrations/subprocess.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# Subprocess Integration
|
||||
|
||||
## Scope
|
||||
|
||||
This document describes how a parent process should invoke Audita as a subprocess.
|
||||
|
||||
It covers:
|
||||
- invocation shape;
|
||||
- stdout/stderr behavior;
|
||||
- output/report file behavior;
|
||||
- diagnostics and exit behavior.
|
||||
|
||||
For full CLI and config references, see [`docs/cli.md`](../cli.md) and [`docs/config.md`](../config.md).
|
||||
|
||||
## Recommended Invocation
|
||||
|
||||
Use explicit output and report paths for machine workflows:
|
||||
|
||||
```sh
|
||||
audita process <transcript.json> \
|
||||
--glossary <glossary.yaml> \
|
||||
--output <output-transcript.json> \
|
||||
--report-json <report.json>
|
||||
```
|
||||
|
||||
Optional commonly used flags:
|
||||
- `--config <path>`
|
||||
- `--output-schema <bare-segments|audita-v1>`
|
||||
- `--work-dir <dir>`
|
||||
- `--work-dir-retention <always|auto|never>`
|
||||
- `--transcript-description <text>`
|
||||
|
||||
## Stdout Contract
|
||||
|
||||
On success:
|
||||
- with `--output`: stdout is expected to be empty;
|
||||
- without `--output`: stdout contains transcript JSON only.
|
||||
|
||||
`--report-json` output is never written to stdout.
|
||||
|
||||
## Stderr Contract
|
||||
|
||||
Stderr is human-readable status/error output.
|
||||
|
||||
On failures:
|
||||
- stderr includes a concise top-level error;
|
||||
- when diagnostics are initialized, stderr includes diagnostics directory path.
|
||||
|
||||
Do not treat stderr as a machine-stable JSON channel.
|
||||
|
||||
## Output and Report File Contract
|
||||
|
||||
Transcript output:
|
||||
- `--output` writes corrected transcript JSON to the provided path;
|
||||
- output write failures return nonzero.
|
||||
|
||||
Report output:
|
||||
- `--report-json` writes machine-readable process report JSON to the provided path;
|
||||
- run diagnostics also attempt to write their own `report.json`;
|
||||
- report write failures return nonzero;
|
||||
- on failure paths, report writing is best-effort and does not mask the primary run error.
|
||||
|
||||
## Diagnostics Contract
|
||||
|
||||
When run-directory initialization succeeds, per-run diagnostics artifacts are written under the configured work directory.
|
||||
|
||||
Typical artifacts include:
|
||||
- `source-transcript.json`
|
||||
- `source-transcript-parsed.json`
|
||||
- `normalized-transcript.json`
|
||||
- `normalization-summary.json`
|
||||
- `chunking-summary.json`
|
||||
- `invocation.json`
|
||||
- `effective-config.json`
|
||||
- `utilization-diagnostics.json`
|
||||
- `correction-ledger.json`
|
||||
- `report.json`
|
||||
- `error.log` (failure)
|
||||
|
||||
Retention behavior is controlled by `--work-dir-retention` / config.
|
||||
|
||||
## Exit Behavior
|
||||
|
||||
Exit codes:
|
||||
- `0`: success;
|
||||
- `1`: runtime processing/output/report failure;
|
||||
- `2`: CLI usage or configuration input error.
|
||||
|
||||
Treat any nonzero as subprocess failure.
|
||||
|
||||
## Parent-Process Guidance
|
||||
|
||||
For reliable orchestration:
|
||||
- read stdout and stderr concurrently to avoid pipe blocking;
|
||||
- prefer `--output` and `--report-json` for machine parsing;
|
||||
- use timeout/cancellation in the parent process;
|
||||
- inspect diagnostics path and `report.json`/`error.log` on failure.
|
||||
|
||||
For input file contracts, see [`docs/integrations/transcript-glossary-files.md`](transcript-glossary-files.md).
|
||||
98
docs/integrations/transcript-glossary-files.md
Normal file
98
docs/integrations/transcript-glossary-files.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# Transcript and Glossary File Integration
|
||||
|
||||
## Scope
|
||||
|
||||
This document defines the input file contracts for:
|
||||
- transcript JSON;
|
||||
- glossary YAML.
|
||||
|
||||
These files are loaded and validated before processing begins.
|
||||
|
||||
## Transcript JSON Contract
|
||||
|
||||
Audita accepts either top-level shape:
|
||||
- JSON array of segments; or
|
||||
- JSON object with a `segments` array.
|
||||
|
||||
Segment fields:
|
||||
- `id` (optional integer in source form);
|
||||
- `speaker` (required non-empty string);
|
||||
- `start` (required finite non-negative number);
|
||||
- `end` (required finite non-negative number, `>= start`);
|
||||
- `text` (required non-empty string);
|
||||
- `categories` (optional string array; entries must be non-empty).
|
||||
|
||||
Additional rules:
|
||||
- transcript must contain at least one segment;
|
||||
- duplicate segment IDs are rejected when IDs are present.
|
||||
|
||||
Example (`examples/tiny-transcript.json`):
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 1,
|
||||
"speaker": "A",
|
||||
"start": 0.0,
|
||||
"end": 1.2,
|
||||
"text": "hello world"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
## Glossary YAML Contract
|
||||
|
||||
Audita expects top-level `glossary` list entries.
|
||||
|
||||
Entry fields:
|
||||
- `name` (required non-empty string);
|
||||
- `category` (required non-empty string);
|
||||
- `summary` (required non-empty string);
|
||||
- `aliases` (optional list of strings; entries must be non-empty);
|
||||
- `plural` (optional string).
|
||||
|
||||
Additional rules:
|
||||
- glossary must contain at least one entry.
|
||||
|
||||
Example (`examples/tiny-glossary.yaml`):
|
||||
|
||||
```yaml
|
||||
glossary:
|
||||
- name: Audita
|
||||
aliases:
|
||||
- audita
|
||||
category: product
|
||||
summary: The Audita transcript correction CLI.
|
||||
```
|
||||
|
||||
## Validation Failure Behavior
|
||||
|
||||
Representative transcript validation failures:
|
||||
- invalid JSON;
|
||||
- unsupported top-level shape;
|
||||
- empty `speaker` or `text`;
|
||||
- invalid times (`NaN`, `Inf`, negative, or `end < start`);
|
||||
- duplicate IDs;
|
||||
- empty transcript array.
|
||||
|
||||
Representative glossary validation failures:
|
||||
- invalid YAML;
|
||||
- empty or missing glossary entries;
|
||||
- missing required entry fields;
|
||||
- empty alias values.
|
||||
|
||||
These failures surface as schema errors and the process exits nonzero.
|
||||
|
||||
## CLI Usage
|
||||
|
||||
Minimal invocation:
|
||||
|
||||
```sh
|
||||
audita process ./transcript.json --glossary ./glossary.yaml --output ./corrected.json
|
||||
```
|
||||
|
||||
See also:
|
||||
- [`docs/cli.md`](../cli.md)
|
||||
- [`docs/config.md`](../config.md)
|
||||
- [`examples/tiny-transcript.json`](../../examples/tiny-transcript.json)
|
||||
- [`examples/tiny-glossary.yaml`](../../examples/tiny-glossary.yaml)
|
||||
79
docs/internal/diagnostics-reporting.md
Normal file
79
docs/internal/diagnostics-reporting.md
Normal file
@@ -0,0 +1,79 @@
|
||||
# Audita Diagnostics and Reporting
|
||||
|
||||
## Scope
|
||||
|
||||
This document describes diagnostics artifacts, process report mapping, and correction ledger generation.
|
||||
|
||||
## Run Directory Ownership
|
||||
|
||||
`internal/core/diagnostics` owns run-directory creation, artifact writes, and retention decisions.
|
||||
|
||||
Stable artifact names include:
|
||||
- `source-transcript.json`
|
||||
- `source-transcript-parsed.json`
|
||||
- `normalized-transcript.json`
|
||||
- `normalization-summary.json`
|
||||
- `chunking-summary.json`
|
||||
- `utilization-diagnostics.json`
|
||||
- `correction-ledger.json`
|
||||
- `invocation.json`
|
||||
- `effective-config.json`
|
||||
- `report.json`
|
||||
- `error.log` (failure)
|
||||
|
||||
## Process Report Mapping
|
||||
|
||||
`internal/framework/processreport` maps runner/CLI execution facts into `reporting.ProcessReport`.
|
||||
|
||||
Report metadata fields include:
|
||||
- `report_schema_name` (`audita-process-report`)
|
||||
- `report_schema_version` (`v1`)
|
||||
- `output_schema`
|
||||
- `config_version` (when file config exists)
|
||||
|
||||
The report includes:
|
||||
- top-level status/error phase/error message;
|
||||
- normalization/chunking summaries;
|
||||
- diagnostics metadata paths;
|
||||
- per-module results and module summary.
|
||||
|
||||
## Correction Ledger
|
||||
|
||||
`internal/framework/processreport/BuildCorrectionLedger` flattens run results into `correction-ledger.json` entries.
|
||||
|
||||
Dispositions:
|
||||
- `applied`
|
||||
- `skipped`
|
||||
- `rejected`
|
||||
- `failed`
|
||||
|
||||
Validator decisions are split into deterministic and LLM-backed groups using validator metadata classification.
|
||||
|
||||
## Report Write Paths
|
||||
|
||||
- run directory always attempts to write `report.json` when possible;
|
||||
- optional `--report-json` writes an external report file;
|
||||
- on failure paths, report writing is best-effort and does not mask primary run errors.
|
||||
|
||||
## Retention Interaction
|
||||
|
||||
Current retention behavior:
|
||||
- failed runs are retained;
|
||||
- `always` keeps successful runs;
|
||||
- `auto` removes only clean successful runs;
|
||||
- `never` currently retains successful runs in current implementation.
|
||||
|
||||
## Redaction
|
||||
|
||||
Redacted data expectations:
|
||||
- effective config artifact uses config redaction;
|
||||
- diagnostics payloads and surfaced errors use LLM secret redaction;
|
||||
- reports should not include raw API key values.
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/core/diagnostics/*_test.go`
|
||||
- `internal/framework/processreport/*_test.go`
|
||||
- `internal/core/reporting/report_test.go`
|
||||
- `internal/cli/run_test.go`
|
||||
- `cmd/audita/main_integration_test.go`
|
||||
67
docs/internal/llm-runtime.md
Normal file
67
docs/internal/llm-runtime.md
Normal file
@@ -0,0 +1,67 @@
|
||||
# Audita LLM Runtime
|
||||
|
||||
## Scope
|
||||
|
||||
This document describes the structured LLM runtime and scheduler behavior.
|
||||
|
||||
## Client Boundary
|
||||
|
||||
All runtime LLM calls go through `contracts.StructuredLLMClient`.
|
||||
|
||||
Primary adapter:
|
||||
- `internal/framework/llm/OpenAICompatibleClient`
|
||||
|
||||
## Request/Response Behavior
|
||||
|
||||
The OpenAI-compatible adapter sends chat completions requests with:
|
||||
- model;
|
||||
- messages;
|
||||
- `response_format.type = json_schema`;
|
||||
- strict schema envelope (`name`, `schema`, `strict=true`).
|
||||
|
||||
The response is decoded into the requested structured output target.
|
||||
|
||||
## Response Schema Registry
|
||||
|
||||
Structured response schemas are registered in `internal/framework/responseschema`:
|
||||
- `correction_set`
|
||||
- `validator_decision_set`
|
||||
|
||||
Each schema includes stable diagnostics metadata (`id`, `version`, `name`, `sha256`).
|
||||
|
||||
## Retries and Error Handling
|
||||
|
||||
Adapter retries apply to retryable conditions (for example transport/decoding/retryable status classes) up to configured `max_retries`.
|
||||
|
||||
Errors are sanitized to redact configured API-key values before surfacing.
|
||||
|
||||
Malformed structured output detection is shared through `internal/framework/structuredoutput` and is used by:
|
||||
- proposal generation;
|
||||
- LLM-backed validators.
|
||||
|
||||
## Scheduling and Concurrency
|
||||
|
||||
`internal/framework/llm/Scheduler` provides FIFO, context-aware permit gating.
|
||||
|
||||
Runner composes scheduler limits across:
|
||||
- total LLM concurrency;
|
||||
- proposal LLM concurrency;
|
||||
- validation LLM concurrency.
|
||||
|
||||
Scheduler release is guarded to avoid permit leaks on cancellation/error.
|
||||
|
||||
## Diagnostics and Redaction
|
||||
|
||||
`internal/framework/llm/DiagnosticsWriter` writes request/response/error artifacts.
|
||||
|
||||
Configured secrets are derived from `llm.ConfiguredSecrets(cfg)` and redacted from:
|
||||
- diagnostics payloads;
|
||||
- surfaced runtime/adapter errors.
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/framework/llm/openai_compatible_client_test.go`
|
||||
- `internal/framework/llm/scheduler_test.go`
|
||||
- `internal/framework/llm/diagnostics_test.go`
|
||||
- `internal/framework/responseschema/registry_test.go`
|
||||
- `internal/framework/structuredoutput/malformed_test.go`
|
||||
58
docs/internal/modules.md
Normal file
58
docs/internal/modules.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Audita Modules
|
||||
|
||||
## Scope
|
||||
|
||||
This document covers module contracts and built-in module packages.
|
||||
|
||||
## Module Contract
|
||||
|
||||
Modules implement `contracts.TranscriptModule`:
|
||||
- `Key()`
|
||||
- `ReplacementPolicy()`
|
||||
- `Validators()`
|
||||
- `Propose(ctx, req)`
|
||||
|
||||
Runner resolves configured module specs to module instances through `internal/framework/modules`.
|
||||
|
||||
## Built-In Modules
|
||||
|
||||
Current module packages:
|
||||
- `internal/modules/glossary`
|
||||
- `internal/modules/homophones`
|
||||
- `internal/modules/spoken_word`
|
||||
- `internal/modules/grammar`
|
||||
|
||||
Current replacement policies:
|
||||
- `glossary`: `replace_all`
|
||||
- `homophones`: `require_unique`
|
||||
- `spoken_word`: `require_unique`
|
||||
- `grammar`: `require_unique`
|
||||
|
||||
## Proposal Generation Ownership
|
||||
|
||||
Shared proposal-generation plumbing is centralized in:
|
||||
- `internal/framework/proposal_generation`
|
||||
|
||||
Module packages own:
|
||||
- prompt selection (`internal/prompts` prompt IDs);
|
||||
- module-specific prompt payload construction.
|
||||
|
||||
Shared prompt helpers live in `internal/framework/promptcontext`.
|
||||
|
||||
## Validator Chain Ownership
|
||||
|
||||
Built-in chains are resolved in `internal/validators` per module key.
|
||||
Module packages call the built-in chain resolver at construction.
|
||||
|
||||
## Failure and Warning Behavior
|
||||
|
||||
- module setup failures surface as `runner_setup` or module setup errors;
|
||||
- module runtime failures surface as `runner_execution` with partial module results preserved;
|
||||
- malformed structured proposal payloads are downgraded to warnings and section-level proposal rejection.
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/modules/*/module_test.go`
|
||||
- `internal/framework/modules/registry_test.go`
|
||||
- `internal/framework/proposal_generation/*_test.go`
|
||||
- `internal/cli/run_test.go` (pipeline/report integration)
|
||||
46
docs/internal/output-schemas.md
Normal file
46
docs/internal/output-schemas.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# Audita Output Schemas
|
||||
|
||||
## Scope
|
||||
|
||||
This document describes the implemented transcript output schema registry.
|
||||
|
||||
## Registry Ownership
|
||||
|
||||
Output schema registry is owned by `internal/core/outputschema`.
|
||||
|
||||
Supported schema keys:
|
||||
- `bare-segments`
|
||||
- `audita-v1`
|
||||
|
||||
## Schemas
|
||||
|
||||
`bare-segments`:
|
||||
- top-level JSON array of transcript segments.
|
||||
|
||||
`audita-v1`:
|
||||
- top-level JSON object with:
|
||||
- `schema: "audita-v1"`
|
||||
- `version: "v1"`
|
||||
- `segments: [...]`
|
||||
|
||||
Segment fields include `id`, `speaker`, `start`, `end`, `text`, and optional `categories`.
|
||||
|
||||
## Validation and Resolution
|
||||
|
||||
Config validation and runtime resolution both reject unsupported schema keys.
|
||||
|
||||
Unknown schema keys fail with `unsupported output schema` before output emission.
|
||||
|
||||
## Output Emission
|
||||
|
||||
The selected schema is used by `audita process` when writing:
|
||||
- output file (`--output`) or
|
||||
- stdout (when no `--output`).
|
||||
|
||||
Report metadata records selected `output_schema`.
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/core/outputschema/registry_test.go`
|
||||
- `internal/core/config/config_test.go`
|
||||
- `internal/cli/run_test.go`
|
||||
92
docs/internal/overview.md
Normal file
92
docs/internal/overview.md
Normal file
@@ -0,0 +1,92 @@
|
||||
# Audita Internal Overview
|
||||
|
||||
## Scope
|
||||
|
||||
This document is the internal architecture entry point for developers and coding agents.
|
||||
|
||||
It summarizes:
|
||||
- package boundaries;
|
||||
- the main `process` execution path;
|
||||
- where to add new code safely.
|
||||
|
||||
## Package Map
|
||||
|
||||
CLI and command orchestration:
|
||||
- `internal/cli`
|
||||
|
||||
Core deterministic components:
|
||||
- `internal/core/config`
|
||||
- `internal/core/schema`
|
||||
- `internal/core/normalization`
|
||||
- `internal/core/chunking`
|
||||
- `internal/core/outputschema`
|
||||
- `internal/core/diagnostics`
|
||||
- `internal/core/reporting`
|
||||
- `internal/core/modulecatalog`
|
||||
|
||||
Framework orchestration and contracts:
|
||||
- `internal/framework/contracts`
|
||||
- `internal/framework/modules`
|
||||
- `internal/framework/proposal_generation`
|
||||
- `internal/framework/proposals`
|
||||
- `internal/framework/runner`
|
||||
- `internal/framework/validators`
|
||||
- `internal/framework/llm`
|
||||
- `internal/framework/responseschema`
|
||||
- `internal/framework/structuredoutput`
|
||||
- `internal/framework/processreport`
|
||||
- `internal/framework/promptcontext`
|
||||
- `internal/framework/stagename`
|
||||
|
||||
Domain implementations:
|
||||
- `internal/modules/*`
|
||||
- `internal/validators/*`
|
||||
- `internal/prompts`
|
||||
|
||||
## Main Execution Path (`audita process`)
|
||||
|
||||
High-level flow:
|
||||
1. CLI loads effective config and validates CLI requirements.
|
||||
2. Run directory is created and invocation/effective config artifacts are written.
|
||||
3. Transcript/glossary files are loaded and parsed.
|
||||
4. Transcript is normalized and chunked.
|
||||
5. `runner.Run` executes configured module instances.
|
||||
6. Proposals are validated, applied deterministically, and serialized in selected output schema.
|
||||
7. Process report, utilization diagnostics, correction ledger, and retention decisions are finalized.
|
||||
|
||||
## Boundary Summary
|
||||
|
||||
- `internal/core/*` owns deterministic, reusable logic and persistence-independent rules.
|
||||
- `internal/framework/*` owns orchestration contracts and reusable runtime plumbing.
|
||||
- `internal/modules/*` owns module-specific proposal behavior and prompt usage.
|
||||
- `internal/validators/*` owns validator composition and built-in chain assembly.
|
||||
- `internal/prompts` owns embedded prompt assets and metadata registry.
|
||||
|
||||
## Where To Add New Code
|
||||
|
||||
Add config fields:
|
||||
- `internal/core/config`
|
||||
|
||||
Add module behavior:
|
||||
- one package under `internal/modules/<module_key>`
|
||||
- registration/wiring through `internal/framework/modules` and config module list
|
||||
|
||||
Add validators:
|
||||
- implementation under `internal/validators/<validator_key>`
|
||||
- registry/chain wiring in `internal/validators`
|
||||
|
||||
Add runtime orchestration behavior:
|
||||
- `internal/framework/*` (runner/proposal/validator/LLM plumbing)
|
||||
|
||||
Add CLI surface:
|
||||
- `internal/cli`
|
||||
|
||||
## Related Internal Docs
|
||||
|
||||
- [`docs/internal/pipeline.md`](pipeline.md)
|
||||
- [`docs/internal/modules.md`](modules.md)
|
||||
- [`docs/internal/validators.md`](validators.md)
|
||||
- [`docs/internal/llm-runtime.md`](llm-runtime.md)
|
||||
- [`docs/internal/diagnostics-reporting.md`](diagnostics-reporting.md)
|
||||
- [`docs/internal/prompts.md`](prompts.md)
|
||||
- [`docs/internal/output-schemas.md`](output-schemas.md)
|
||||
79
docs/internal/pipeline.md
Normal file
79
docs/internal/pipeline.md
Normal file
@@ -0,0 +1,79 @@
|
||||
# Audita Internal Pipeline
|
||||
|
||||
## Scope
|
||||
|
||||
This document describes the implemented `audita process` pipeline.
|
||||
|
||||
## Inputs
|
||||
|
||||
Pipeline inputs are:
|
||||
- effective config (`internal/core/config`);
|
||||
- transcript JSON (`internal/core/schema`);
|
||||
- glossary YAML (`internal/core/schema`).
|
||||
|
||||
## Pipeline Phases
|
||||
|
||||
1. Input loading and schema validation
|
||||
- transcript and glossary files are read and parsed.
|
||||
- schema failures stop the run with `transcript_schema` or `glossary_schema`.
|
||||
|
||||
2. Normalization
|
||||
- canonical transcript segments are normalized by configured gap/duration/token settings.
|
||||
- normalization summary artifacts are written.
|
||||
|
||||
3. Chunking
|
||||
- normalized transcript is chunked with configured max/min tokens and target sections.
|
||||
|
||||
4. Module proposal generation
|
||||
- runner executes configured module instances in sequence.
|
||||
- each module proposes corrections per section.
|
||||
- per-section proposal generation can run concurrently.
|
||||
|
||||
5. Validator filtering
|
||||
- validators run on candidate proposals before apply.
|
||||
- deterministic validators run before LLM-backed validators.
|
||||
- LLM validator inputs are batched by max prompt token limit.
|
||||
|
||||
6. Deterministic apply
|
||||
- approved proposals are applied via replacement policy.
|
||||
- applied/skipped/rejected outcomes are recorded.
|
||||
|
||||
7. Output and reporting
|
||||
- final transcript is serialized with selected output schema.
|
||||
- report, utilization diagnostics, and correction ledger are written.
|
||||
- retention policy is applied to run directory.
|
||||
|
||||
## Runner Outputs
|
||||
|
||||
`runner.Run` returns:
|
||||
- final transcript;
|
||||
- per-module results;
|
||||
- utilization diagnostics.
|
||||
|
||||
CLI/reporting then map this into process report and diagnostics artifacts.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Representative failure phases include:
|
||||
- `run_dir_creation`
|
||||
- `transcript_read`, `glossary_read`
|
||||
- `transcript_schema`, `glossary_schema`
|
||||
- `chunking`
|
||||
- `runner_setup`, `runner_execution`
|
||||
- `output_schema`, `serialization`, `output_write`, `stdout_write`
|
||||
|
||||
When diagnostics are available, failure stderr includes diagnostics path.
|
||||
|
||||
## Invariants
|
||||
|
||||
- module execution order follows configured module sequence;
|
||||
- proposal/validator nondeterminism is isolated before deterministic apply;
|
||||
- proposal indices are assigned deterministically by section order;
|
||||
- output/report artifacts are generated from run results, not speculative state.
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/framework/runner/runner_test.go`
|
||||
- `internal/framework/proposal_generation/*_test.go`
|
||||
- `internal/cli/run_test.go`
|
||||
- `cmd/audita/main_integration_test.go`
|
||||
62
docs/internal/prompts.md
Normal file
62
docs/internal/prompts.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# Audita Prompt Registry
|
||||
|
||||
## Scope
|
||||
|
||||
This document describes embedded prompt assets, prompt metadata, and rendering behavior.
|
||||
|
||||
## Registry Ownership
|
||||
|
||||
Prompt registry lives in `internal/prompts` and embeds assets under `internal/prompts/assets/**`.
|
||||
|
||||
Registered prompt IDs:
|
||||
- `modules.glossary.proposal`
|
||||
- `modules.homophones.proposal`
|
||||
- `modules.spoken_word.proposal`
|
||||
- `modules.grammar.proposal`
|
||||
- `validators.spoken_form_plausibility`
|
||||
- `validators.meaning_reversal_review`
|
||||
- `validators.editorial_review`
|
||||
- `validators.grammar_review`
|
||||
- `validators.spoken_word_review`
|
||||
|
||||
## Metadata Model
|
||||
|
||||
Each prompt has metadata:
|
||||
- `prompt_id`
|
||||
- `prompt_version`
|
||||
- `prompt_source`
|
||||
- `embedded_path`
|
||||
- `sha256`
|
||||
|
||||
Current source/version values:
|
||||
- `prompt_source = builtin`
|
||||
- `prompt_version = v1`
|
||||
|
||||
## Rendering
|
||||
|
||||
`prompts.RenderUserSystem(promptID, data)` renders system/user templates.
|
||||
|
||||
Template behavior:
|
||||
- uses Go `text/template`;
|
||||
- `missingkey=error` is enabled;
|
||||
- output is trimmed.
|
||||
|
||||
A shared hardening fragment is embedded once and referenced by prompt templates.
|
||||
|
||||
## Prompt Context Inputs
|
||||
|
||||
Shared prompt payload helpers:
|
||||
- transcript section JSON (`internal/framework/promptcontext/MarshalTranscriptSectionJSON`)
|
||||
- transcript description block (`TranscriptDescriptionBlock`)
|
||||
|
||||
Modules and LLM validators provide typed data maps to render prompt assets.
|
||||
|
||||
## Diagnostics Integration
|
||||
|
||||
Prompt metadata is attached to proposal/validator diagnostics request metadata using `Metadata.DiagnosticsMap()`.
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/prompts/registry_test.go`
|
||||
- `internal/framework/promptcontext/*_test.go`
|
||||
- module and validator prompt builder tests
|
||||
72
docs/internal/validators.md
Normal file
72
docs/internal/validators.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# Audita Validators
|
||||
|
||||
## Scope
|
||||
|
||||
This document describes validator composition, execution order, and decision handling.
|
||||
|
||||
## Ownership
|
||||
|
||||
Built-in validator keys and chains:
|
||||
- `internal/validators`
|
||||
|
||||
Shared validator runtime mechanics:
|
||||
- `internal/framework/validators`
|
||||
|
||||
Execution-class metadata:
|
||||
- `internal/validators/metadata`
|
||||
|
||||
## Built-In Validator Keys
|
||||
|
||||
Deterministic:
|
||||
- `proposal_shape`
|
||||
- `confidence_threshold`
|
||||
- `original_text_presence`
|
||||
- `non_empty_corrected_text`
|
||||
- `no_effect`
|
||||
- `protected_terms`
|
||||
|
||||
LLM-backed:
|
||||
- `spoken_form_plausibility`
|
||||
- `meaning_reversal_review`
|
||||
- `editorial_review`
|
||||
|
||||
## Built-In Chains
|
||||
|
||||
Module chains are defined in `internal/validators/chains.go`.
|
||||
Glossary, homophones, spoken_word, and grammar each resolve a fixed ordered chain.
|
||||
|
||||
## Runtime Execution
|
||||
|
||||
For each module section:
|
||||
1. run deterministic validators;
|
||||
2. run LLM-backed validators;
|
||||
3. record decisions and warnings;
|
||||
4. carry only approved proposals forward.
|
||||
|
||||
Decision cardinality is enforced: each candidate proposal must receive exactly one decision per validator.
|
||||
|
||||
## LLM Validator Batching
|
||||
|
||||
LLM validators:
|
||||
- build canonical validation request payloads;
|
||||
- batch by `validation_max_prompt_tokens`;
|
||||
- call structured LLM client using response schema registry.
|
||||
|
||||
Oversized single proposals are rejected with `validator_input_too_large`.
|
||||
Malformed LLM validator responses are downgraded to warnings and rejected batch decisions.
|
||||
|
||||
## Decision and Rejection Reporting
|
||||
|
||||
Runner records:
|
||||
- `validator_decisions`
|
||||
- `validator_rejected`
|
||||
- warning records (including malformed response warnings)
|
||||
|
||||
Correction ledger classifies deterministic vs LLM validator decisions using canonical metadata classes.
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/validators/*_test.go`
|
||||
- `internal/framework/validators/*_test.go`
|
||||
- `internal/framework/processreport/correction_ledger_test.go`
|
||||
- `internal/cli/run_test.go`
|
||||
113
docs/operations.md
Normal file
113
docs/operations.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# Audita Operations
|
||||
|
||||
## Scope
|
||||
|
||||
This document covers operational behavior for `audita process` as currently implemented:
|
||||
- run lifecycle;
|
||||
- output and report files;
|
||||
- diagnostics artifacts;
|
||||
- run-directory retention behavior;
|
||||
- failure inspection and recovery.
|
||||
|
||||
For command syntax, see [`docs/cli.md`](cli.md).
|
||||
|
||||
## Process Run Lifecycle
|
||||
|
||||
A `process` run performs these high-level steps:
|
||||
1. load effective config (defaults + optional file + env + CLI);
|
||||
2. create a per-run diagnostics directory;
|
||||
3. load transcript JSON and glossary YAML;
|
||||
4. parse/validate input schemas;
|
||||
5. normalize transcript and compute chunking;
|
||||
6. run configured modules/validators;
|
||||
7. serialize output schema and write transcript output;
|
||||
8. build and write process report;
|
||||
9. apply run-directory retention.
|
||||
|
||||
If a failure happens after diagnostics initialization, the run writes failure details and returns nonzero.
|
||||
|
||||
## Output Files
|
||||
|
||||
Transcript output:
|
||||
- when `--output <path>` is set, corrected transcript JSON is written to that file;
|
||||
- when `--output` is omitted, corrected transcript JSON is written to stdout.
|
||||
|
||||
Report output:
|
||||
- when `--report-json <path>` is set, Audita writes a process report JSON file;
|
||||
- the run directory also writes its own `report.json` artifact.
|
||||
|
||||
On success with `--output`, stdout is expected to be empty.
|
||||
|
||||
## Diagnostics Directory
|
||||
|
||||
By default, runs use `work_dir` from effective config (default `/tmp/audita`).
|
||||
Each run directory is created under the work dir using a generated ID like `run-<unix-nanos>`.
|
||||
|
||||
Top-level diagnostics artifacts:
|
||||
- `source-transcript.json`
|
||||
- `source-transcript-parsed.json`
|
||||
- `normalized-transcript.json`
|
||||
- `normalization-summary.json`
|
||||
- `chunking-summary.json`
|
||||
- `utilization-diagnostics.json`
|
||||
- `correction-ledger.json`
|
||||
- `invocation.json`
|
||||
- `effective-config.json` (redacted)
|
||||
- `report.json`
|
||||
- `error.log` (failure runs)
|
||||
|
||||
Report diagnostics metadata includes resolved paths to these artifacts.
|
||||
|
||||
## Correction Ledger and Utilization Diagnostics
|
||||
|
||||
`correction-ledger.json` records correction dispositions:
|
||||
- `applied`
|
||||
- `skipped`
|
||||
- `rejected`
|
||||
- `failed`
|
||||
|
||||
`utilization-diagnostics.json` records effective concurrency and execution timing summaries for run/module/validator activity.
|
||||
|
||||
## Retention Behavior
|
||||
|
||||
Retention is controlled by `work_dir_retention` (`auto|always|never`).
|
||||
|
||||
Current behavior:
|
||||
- failed runs are always retained;
|
||||
- `always`: successful runs are retained;
|
||||
- `auto`: successful runs are retained only when skipped/rejected corrections occurred; clean successful runs are removed;
|
||||
- `never`: successful runs are currently retained (same net retention outcome as `always` in current implementation).
|
||||
|
||||
Even when a successful run directory is removed under `auto`, an explicit `--report-json` file is still preserved at its target path.
|
||||
|
||||
## Failure Inspection
|
||||
|
||||
For failed runs:
|
||||
1. read stderr for the top-level failure and diagnostics path;
|
||||
2. open `error.log` in the reported run directory;
|
||||
3. inspect run `report.json` (`status`, `error_phase`, `error_message`);
|
||||
4. inspect related artifacts referenced by report diagnostics metadata.
|
||||
|
||||
Typical `error_phase` values include:
|
||||
- `transcript_read`
|
||||
- `glossary_read`
|
||||
- `transcript_schema`
|
||||
- `glossary_schema`
|
||||
- `chunking`
|
||||
- `runner_setup`
|
||||
- `runner_execution`
|
||||
- `output_schema`
|
||||
- `serialization`
|
||||
- `output_write`
|
||||
- `stdout_write`
|
||||
|
||||
## Recovery Guidance
|
||||
|
||||
Safe recovery pattern:
|
||||
1. correct the immediate input/config/output-path problem;
|
||||
2. rerun with `--work-dir-retention always` during debugging;
|
||||
3. once stable, restore your normal retention mode.
|
||||
|
||||
Not implemented:
|
||||
- resume/checkpoint APIs
|
||||
- remote diagnostics/report storage
|
||||
207
docs/policy/architecture.md
Normal file
207
docs/policy/architecture.md
Normal file
@@ -0,0 +1,207 @@
|
||||
# Architecture Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines Audita's development architecture and invariants for maintainers and LLM coding agents. It describes how the project is intended to be changed safely, based on behavior implemented in this repository today.
|
||||
|
||||
User-facing behavior belongs in the README and focused runtime docs. Future or proposed work belongs only under `docs/roadmap/`.
|
||||
|
||||
## Project Shape
|
||||
|
||||
Audita is a single-process Go CLI for transcript polishing. The executable entrypoint is `cmd/audita`; command handling lives in `internal/cli`.
|
||||
|
||||
The implemented `audita process` flow is:
|
||||
|
||||
1. load effective config;
|
||||
2. read and validate transcript JSON and glossary YAML;
|
||||
3. normalize transcript segments;
|
||||
4. chunk the working transcript into sections;
|
||||
5. resolve configured module instances;
|
||||
6. run correction modules and validator chains;
|
||||
7. apply approved proposals deterministically;
|
||||
8. write transcript output, reports, and diagnostics artifacts.
|
||||
|
||||
The current built-in modules are `glossary`, `homophones`, `spoken_word`, and `grammar`. The default configured module sequence repeats `glossary`.
|
||||
|
||||
For external behavior and compatibility details, prefer links to existing behavior docs:
|
||||
|
||||
- [CLI reference](../cli.md)
|
||||
- [Configuration](../config.md)
|
||||
- [Operations](../operations.md)
|
||||
- [Troubleshooting](../troubleshooting.md)
|
||||
- [Integration docs](../integrations/subprocess.md)
|
||||
- [Internal docs](../internal/overview.md)
|
||||
|
||||
## Core Design Principles
|
||||
|
||||
- **Hexagonal architecture:** keep domain behavior behind narrow internal contracts. CLI, filesystem, config loading, diagnostics writing, and LLM transport are adapters around the core processing flow.
|
||||
- **Composable modules and validators:** correction stages and validators should remain small, explicit, and independently testable.
|
||||
- **Deterministic orchestration around LLM calls:** LLM responses are nondeterministic inputs. Proposal indexing, validator ordering, proposal application, reports, and output serialization must remain deterministic.
|
||||
- **Bounded and observable concurrency:** use the implemented schedulers and configured concurrency limits for LLM call sites. Preserve utilization diagnostics when changing scheduling or orchestration.
|
||||
- **Conservative correction behavior:** validate proposed corrections before application; apply accepted proposals through deterministic apply-time safety checks.
|
||||
- **Standard-library-first:** prefer the Go standard library. Narrow third-party dependencies are acceptable when they materially improve maintainability, such as `gopkg.in/yaml.v3` for YAML parsing.
|
||||
- **Current-behavior documentation:** non-roadmap docs must describe implemented behavior only.
|
||||
|
||||
## Architectural Boundaries
|
||||
|
||||
`internal/core` owns domain data handling and stable runtime contracts that do not require CLI or provider transport knowledge:
|
||||
|
||||
- config defaults, loading, validation, redaction, and catalogs;
|
||||
- transcript and glossary schemas;
|
||||
- normalization and chunking;
|
||||
- output-schema encoding;
|
||||
- diagnostics artifact naming and run-directory helpers;
|
||||
- public process report shapes.
|
||||
|
||||
`internal/framework` owns orchestration contracts and reusable runtime mechanics:
|
||||
|
||||
- module and validator interfaces;
|
||||
- proposal generation, proposal application, and prompt context;
|
||||
- runner orchestration;
|
||||
- LLM scheduler, OpenAI-compatible adapter, redaction helpers, and diagnostics writers;
|
||||
- structured response schema registry;
|
||||
- process report and correction-ledger assembly.
|
||||
|
||||
`internal/modules/*` owns module-specific correction stages. `internal/validators/*` owns built-in validator implementations, registry, chains, and execution-class metadata. `internal/prompts` owns embedded prompt assets and prompt metadata.
|
||||
|
||||
`internal/cli` owns command parsing, exit codes, stdout/stderr behavior, config command behavior, filesystem input/output wiring, and top-level process orchestration. CLI concerns should not move into modules, validators, or schema logic.
|
||||
|
||||
Tests should stay close to the behavior they protect. Shared test helpers are acceptable when they remove clear duplication without hiding module-specific behavior.
|
||||
|
||||
## Modules and Validators
|
||||
|
||||
Modules implement `contracts.TranscriptModule`. A module must provide:
|
||||
|
||||
- a stable key;
|
||||
- a replacement policy;
|
||||
- a validator chain;
|
||||
- proposal generation from explicit request inputs.
|
||||
|
||||
Module packages should stay separate. Do not collapse module-specific prompts, scope, or validation choices into a broad generic stage abstraction.
|
||||
|
||||
Validators implement the shared validator contract and return one decision per candidate proposal. Deterministic validators and LLM-backed validators are both composable chain elements. Validator identity and execution class metadata are stable enough to affect ordering, diagnostics, reports, and correction-ledger classification.
|
||||
|
||||
Future module or validator changes should preserve:
|
||||
|
||||
- explicit inputs and outputs;
|
||||
- no hidden global state;
|
||||
- explicit config dependencies;
|
||||
- deterministic proposal index handling;
|
||||
- validation before final mutation;
|
||||
- stable reason codes and validator keys where already exposed.
|
||||
|
||||
## LLM Integration and Concurrency
|
||||
|
||||
LLM calls are external effects behind narrow contracts. Production structured completions use `contracts.StructuredLLMClient`; the implemented provider adapter is OpenAI-compatible HTTP code in `internal/framework/llm`.
|
||||
|
||||
Structured response schemas are registered in `internal/framework/responseschema`. Provider-side schema enforcement is not a substitute for local validation: Audita still validates proposal structure, validator decision cardinality, and apply-time safety.
|
||||
|
||||
Concurrency is bounded by configured scheduler limits:
|
||||
|
||||
- total LLM concurrency;
|
||||
- proposal LLM concurrency;
|
||||
- validation LLM concurrency.
|
||||
|
||||
The scheduler is context-aware and releases permits on success, failure, and cancellation. Runner code may collect section-level work concurrently, but transcript mutation is applied later in deterministic proposal-index order.
|
||||
|
||||
Diagnostics for LLM interactions should be useful for debugging without leaking configured secrets. Use the existing redaction helpers and `llm.ConfiguredSecrets`.
|
||||
|
||||
## State, Inputs, and Outputs
|
||||
|
||||
Audita does not implement resume, checkpoint, manifest, or remote storage behavior. Runtime state is in memory plus per-run diagnostics artifacts written under the configured work directory.
|
||||
|
||||
Transcript input accepts the implemented JSON forms documented in the public contract. Parsed source transcripts are normalized into Audita's internal transcript shape before chunking and module execution.
|
||||
|
||||
Proposals and validator decisions are intermediate runtime data. Approved proposals are applied through `internal/framework/proposals`, which clones transcript state, orders by proposal index, and records applied or skipped changes.
|
||||
|
||||
Transcript output is encoded through `internal/core/outputschema`. Reports and correction ledgers are machine-readable artifacts derived from runner outputs; their public shape should not be changed casually.
|
||||
|
||||
## Configuration and CLI Boundaries
|
||||
|
||||
Config behavior is owned by `internal/core/config`; command usage and process wiring are owned by `internal/cli`.
|
||||
|
||||
`audita process` uses implemented precedence: defaults, config file, environment, then CLI flags. `config validate` validates defaults plus a file config and intentionally does not apply environment overrides. `config print-effective` applies defaults, file config, and environment overrides, then prints redacted JSON.
|
||||
|
||||
Do not duplicate full CLI or config reference material here. Use [Configuration](../config.md), [CLI reference](../cli.md), [Operations](../operations.md), and integration docs under [`docs/integrations/`](../integrations/subprocess.md) for current external behavior.
|
||||
|
||||
When adding config fields or CLI flags, update:
|
||||
|
||||
- config defaults, file/env/CLI application, and validation;
|
||||
- CLI flag extraction if applicable;
|
||||
- redaction when secrets are involved;
|
||||
- tests for precedence and source-specific behavior;
|
||||
- user-facing docs if external behavior changes.
|
||||
|
||||
## Errors, Logging, and Diagnostics
|
||||
|
||||
Errors should be phase-specific enough for CLI users and subprocess callers. The CLI writes human-readable errors to stderr and preserves transcript JSON-only stdout behavior on successful stdout output.
|
||||
|
||||
Run diagnostics are best-effort after run-directory creation. Failed runs are retained. Successful run retention follows the implemented work-dir retention policy.
|
||||
|
||||
Diagnostics and reports must not leak configured LLM secrets. Config redaction and LLM payload/error redaction are separate responsibilities and should remain separate.
|
||||
|
||||
Process reports, diagnostics metadata, utilization diagnostics, and correction ledgers are part of the public contract. Prefer additive, compatible changes.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Use targeted package tests for touched behavior and `go test ./...` for substantial changes.
|
||||
|
||||
When changing modules, inspect or add:
|
||||
|
||||
- package-local module tests under `internal/modules/*`;
|
||||
- prompt rendering or proposal-generation tests when prompt inputs change;
|
||||
- parity or release fixtures when public output behavior changes.
|
||||
|
||||
When changing validators, inspect or add:
|
||||
|
||||
- validator package tests;
|
||||
- registry and chain tests under `internal/validators`;
|
||||
- framework validator tests for batching, malformed output, diagnostics, and cardinality.
|
||||
|
||||
When changing LLM integration or concurrency, inspect or add:
|
||||
|
||||
- `internal/framework/llm` scheduler/client/redaction tests;
|
||||
- `internal/framework/runner` orchestration and utilization tests;
|
||||
- structured-output malformed classification tests.
|
||||
|
||||
When changing config, CLI, schema, output, reports, or diagnostics, inspect or add:
|
||||
|
||||
- `internal/core/config` tests;
|
||||
- CLI tests under `internal/cli`;
|
||||
- schema and output-schema tests under `internal/core`;
|
||||
- report, diagnostics, parity, and release-fixture tests.
|
||||
|
||||
## Dependency Policy
|
||||
|
||||
Audita should remain dependency-light. Prefer standard-library solutions for CLI parsing, HTTP, JSON, filesystem, synchronization, and tests.
|
||||
|
||||
Third-party dependencies should be narrow, justified, and preferably de facto standard for their purpose. YAML parsing is the current direct dependency exception.
|
||||
|
||||
Do not add broad frameworks for CLI, dependency injection, workflow orchestration, logging, or plugin systems without a concrete implemented need and focused tests.
|
||||
|
||||
## Documentation Expectations
|
||||
|
||||
Follow [Documentation Policy](./documentation.md). Architecture policy must stay concise and aligned with implemented behavior.
|
||||
|
||||
Do not use architecture docs as changelogs. Do not describe planned modules, adapters, modes, persistence, or configuration unless they are implemented. Put future work under `docs/roadmap/`.
|
||||
|
||||
## Architectural Invariants
|
||||
|
||||
- Keep LLM transport behind `StructuredLLMClient` and framework adapter boundaries.
|
||||
- Keep correction modules narrowly scoped and package-separated.
|
||||
- Keep validators modular, composable, and identified by stable keys.
|
||||
- Keep CLI/config/filesystem concerns out of module and validator domain logic.
|
||||
- Preserve deterministic transcript mutation and output handling around nondeterministic LLM calls.
|
||||
- Keep LLM concurrency bounded, configurable, and observable where implemented.
|
||||
- Keep run diagnostics and reports redacted and machine-readable.
|
||||
- Keep public CLI, config, output-schema, diagnostics, report, prompt, module, and validator contracts stable unless a change is explicit and tested.
|
||||
- Prefer small shared helpers over broad rewrites.
|
||||
- Avoid new dependencies unless they are narrow and clearly justified.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No plugin framework is implemented.
|
||||
- No generic workflow engine is implemented.
|
||||
- No resume, checkpoint, manifest, or remote storage system is implemented.
|
||||
- No multi-process service mode is implemented.
|
||||
- No provider SDK abstraction beyond the current structured LLM client contract and OpenAI-compatible HTTP adapter is implemented.
|
||||
124
docs/policy/development.md
Normal file
124
docs/policy/development.md
Normal file
@@ -0,0 +1,124 @@
|
||||
# Audita Development Workflow
|
||||
|
||||
## Scope
|
||||
|
||||
This is the canonical contributor workflow for Audita maintainers and coding agents.
|
||||
|
||||
It defines:
|
||||
- repository layout and boundaries;
|
||||
- setup and test commands;
|
||||
- expectations for code changes;
|
||||
- how to add config, CLI, modules, validators, docs, and examples.
|
||||
|
||||
## Setup
|
||||
|
||||
Prerequisites:
|
||||
- Go `1.24` or newer.
|
||||
|
||||
Common commands:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go build ./cmd/audita
|
||||
```
|
||||
|
||||
## Repository Layout
|
||||
|
||||
Top-level areas:
|
||||
- `cmd/audita`: executable entrypoint.
|
||||
- `internal/cli`: command parsing and process/config command orchestration.
|
||||
- `internal/core`: deterministic config/schema/normalization/chunking/output/diagnostics/reporting logic.
|
||||
- `internal/framework`: runner orchestration, contracts, proposal generation/application, validators runtime, LLM runtime, response schemas.
|
||||
- `internal/modules/*`: module-specific correction behavior.
|
||||
- `internal/validators/*`: validator implementations, chains, and metadata.
|
||||
- `internal/prompts`: embedded prompts and prompt metadata.
|
||||
- `docs/`: canonical documentation.
|
||||
- `examples/`: maintained copyable inputs/configs.
|
||||
|
||||
## Change Workflow
|
||||
|
||||
1. Confirm scope and behavior contract before editing.
|
||||
2. Make focused changes in the appropriate ownership area.
|
||||
3. Add or update tests for changed behavior.
|
||||
4. Run targeted package tests for touched areas.
|
||||
5. Run `go test ./...` for substantial changes.
|
||||
6. Update docs/examples when external behavior changes.
|
||||
|
||||
## How To Add or Change Configuration
|
||||
|
||||
1. Add fields/defaults/validation under `internal/core/config`.
|
||||
2. Apply source precedence correctly (defaults, file, env, CLI for `process`).
|
||||
3. Ensure `config validate` remains file-only and `config print-effective` remains redacted.
|
||||
4. Update tests in `internal/core/config` and related CLI tests.
|
||||
5. Update [`docs/config.md`](../config.md) and relevant examples under `examples/`.
|
||||
|
||||
## How To Add or Change CLI Behavior
|
||||
|
||||
1. Implement parsing/wiring in `internal/cli`.
|
||||
2. Keep stdout/stderr and exit behavior compatible unless intentional and documented.
|
||||
3. Update CLI tests under `internal/cli` and integration tests under `cmd/audita`.
|
||||
4. Update [`docs/cli.md`](../cli.md) and related integration docs.
|
||||
|
||||
## How To Add or Change Modules
|
||||
|
||||
1. Add or update one module package under `internal/modules/<module_key>`.
|
||||
2. Keep module-specific prompt ownership in the module + `internal/prompts`.
|
||||
3. Wire module registration/catalog resolution through framework/core module catalog code.
|
||||
4. Verify replacement policy and validator chain selection.
|
||||
5. Add/update module tests and proposal-generation tests.
|
||||
6. Update internal docs when behavior/contracts change.
|
||||
|
||||
## How To Add or Change Validators
|
||||
|
||||
1. Implement validator behavior in `internal/validators` and shared runtime pieces in `internal/framework/validators` only when needed.
|
||||
2. Preserve stable validator keys and decision semantics where already exposed.
|
||||
3. Keep deterministic vs LLM-backed execution-class behavior explicit.
|
||||
4. Add/update validator, chain, batching, and malformed-output tests.
|
||||
5. Update validator documentation when external or developer-facing behavior changes.
|
||||
|
||||
## Documentation and Examples Expectations
|
||||
|
||||
- Keep one canonical home per topic (see [`docs/policy/documentation.md`](documentation.md)).
|
||||
- Do not document future/unimplemented behavior outside `docs/roadmap/`.
|
||||
- Keep command examples and config/examples in sync with current code.
|
||||
- Keep examples secret-free and copyable.
|
||||
|
||||
## Practical Validation Checklist
|
||||
|
||||
Use this checklist for meaningful runtime-impacting changes:
|
||||
|
||||
1. Run core tests:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
```
|
||||
|
||||
2. Verify config commands and examples:
|
||||
|
||||
```sh
|
||||
go run ./cmd/audita config validate --config examples/minimal-config.yml
|
||||
go run ./cmd/audita config validate --config examples/production-config.yml
|
||||
go run ./cmd/audita config print-effective --config examples/minimal-config.yml
|
||||
```
|
||||
|
||||
3. Re-check subprocess/runtime contract when touching CLI/process/report paths:
|
||||
- `--output` success keeps stdout empty;
|
||||
- no `--output` success writes transcript JSON to stdout;
|
||||
- `--report-json` writes file output and is not written to stdout;
|
||||
- failures return nonzero and include diagnostics path when available.
|
||||
|
||||
4. Re-check diagnostics/report/redaction when touching LLM, reporting, or diagnostics code:
|
||||
- report schema metadata fields remain present;
|
||||
- diagnostics artifact paths remain valid;
|
||||
- configured secret values remain redacted in reports/diagnostics/errors.
|
||||
|
||||
5. Re-check output schema behavior when touching serialization/schema code:
|
||||
- default `bare-segments` behavior remains correct unless intentionally changed;
|
||||
- `audita-v1` behavior remains correct unless intentionally changed;
|
||||
- unsupported schemas fail validation/resolve paths clearly.
|
||||
|
||||
## Commit Discipline
|
||||
|
||||
- Keep commits scoped and reviewable.
|
||||
- Avoid mixing unrelated refactors with behavior changes.
|
||||
- Use concise plain-English commit messages.
|
||||
356
docs/policy/documentation.md
Normal file
356
docs/policy/documentation.md
Normal file
@@ -0,0 +1,356 @@
|
||||
# Go Project Documentation Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
Project documentation must help four audiences:
|
||||
|
||||
1. users who need to run the application;
|
||||
2. administrators/operators who need to configure and operate it;
|
||||
3. developers who need to understand and change it safely;
|
||||
4. LLM coding agents that need clear scope, boundaries, and invariants.
|
||||
|
||||
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
|
||||
|
||||
## Core Rules
|
||||
|
||||
### 1. Keep docs concise
|
||||
|
||||
Each document should cover a defined scope and only the essentials for that scope.
|
||||
|
||||
Avoid:
|
||||
- long background explanations;
|
||||
- repeated reference material;
|
||||
- implementation detail in user-facing docs;
|
||||
- aspirational language outside roadmap docs;
|
||||
- verbose examples where one minimal example is clearer.
|
||||
|
||||
### 2. Document only implemented behavior outside roadmap files
|
||||
|
||||
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
|
||||
|
||||
- `docs/roadmap/`
|
||||
|
||||
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
|
||||
|
||||
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
|
||||
|
||||
### 3. Use canonical homes
|
||||
|
||||
Each type of information should have one canonical location.
|
||||
|
||||
Canonical homes:
|
||||
|
||||
- project purpose and quickstart: `README.md`
|
||||
- development principles: `docs/policy/architecture.md`
|
||||
- configuration reference: `docs/config.md`
|
||||
- CLI reference: `docs/cli.md`
|
||||
- operations and recovery: `docs/operations.md`
|
||||
- troubleshooting: `docs/troubleshooting.md`
|
||||
- implemented internals: `docs/internal/`
|
||||
- future work: `docs/roadmap/`
|
||||
- contributor workflow: `docs/policy/development.md`
|
||||
- copyable examples: `examples/`
|
||||
|
||||
Other files should summarize briefly and link to the canonical source.
|
||||
|
||||
### 4. Keep examples real
|
||||
|
||||
Examples should be valid, maintained, and free of secrets.
|
||||
|
||||
Where practical:
|
||||
- example configs should load successfully;
|
||||
- example commands should match real CLI syntax;
|
||||
- important examples should be covered by tests.
|
||||
|
||||
## Documentation Profiles
|
||||
|
||||
All projects require:
|
||||
|
||||
- `README.md`
|
||||
- `docs/policy/architecture.md`
|
||||
|
||||
Additional docs depend on the project.
|
||||
|
||||
### Small library
|
||||
|
||||
Recommended:
|
||||
- `docs/policy/development.md`, if contributor conventions are non-obvious
|
||||
|
||||
### Simple CLI
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/policy/development.md`
|
||||
|
||||
### Config-driven CLI
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`
|
||||
- `docs/config.md`
|
||||
|
||||
Recommended:
|
||||
- `examples/`
|
||||
- `docs/policy/development.md`
|
||||
|
||||
### Stateful or operator-facing application
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- `examples/`
|
||||
- `docs/policy/development.md`
|
||||
|
||||
### Modular, staged, service-oriented, or orchestration application
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/`
|
||||
- `docs/policy/development.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- validated examples under `examples/`
|
||||
|
||||
## Required Documents
|
||||
|
||||
### README.md
|
||||
|
||||
**Audience:** users, administrators, operators
|
||||
|
||||
The README is the outward-facing project orientation page.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. concise description;
|
||||
2. elevator pitch;
|
||||
3. shortest useful command or usage example;
|
||||
4. links to targeted docs.
|
||||
|
||||
The README should be short. It is not a manual.
|
||||
|
||||
The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
|
||||
|
||||
### docs/policy/architecture.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
`docs/policy/architecture.md` is required for every project.
|
||||
|
||||
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
|
||||
|
||||
It should include:
|
||||
|
||||
- project shape;
|
||||
- core design principles;
|
||||
- package and boundary philosophy;
|
||||
- state/persistence philosophy, if applicable;
|
||||
- external integration philosophy, if applicable;
|
||||
- error-handling and logging principles;
|
||||
- testing expectations;
|
||||
- documentation expectations;
|
||||
- architectural invariants;
|
||||
- explicit non-goals, if useful.
|
||||
|
||||
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
|
||||
|
||||
### docs/policy/development.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for projects maintained by humans and LLM coding agents.
|
||||
|
||||
It should include:
|
||||
|
||||
- repository layout;
|
||||
- build/test commands;
|
||||
- coding conventions;
|
||||
- dependency policy;
|
||||
- how to add config fields;
|
||||
- how to add CLI flags;
|
||||
- how to add stages/modules/adapters, if applicable;
|
||||
- how to update examples;
|
||||
- documentation update expectations.
|
||||
|
||||
### docs/config.md
|
||||
|
||||
**Audience:** administrators, operators, advanced users
|
||||
|
||||
Required for applications with configuration files.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. config file locations and discovery precedence;
|
||||
2. minimal working config;
|
||||
3. production-oriented config;
|
||||
4. full configuration reference;
|
||||
5. secrets handling, if applicable;
|
||||
6. links to maintained examples.
|
||||
|
||||
The full configuration reference should be canonical.
|
||||
|
||||
### docs/cli.md
|
||||
|
||||
**Audience:** users, administrators, operators
|
||||
|
||||
Required for CLI applications.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. shortest useful command;
|
||||
2. command overview;
|
||||
3. complete flag reference;
|
||||
4. common workflows;
|
||||
5. diagnostic or recovery commands, if applicable.
|
||||
|
||||
Explain when commands are useful, not just their syntax.
|
||||
|
||||
### docs/operations.md
|
||||
|
||||
**Audience:** administrators, operators
|
||||
|
||||
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
|
||||
|
||||
It should cover:
|
||||
|
||||
- normal workflow;
|
||||
- filesystem layout;
|
||||
- remote storage layout, if applicable;
|
||||
- logs and manifests;
|
||||
- resume/retry behavior;
|
||||
- cleanup behavior;
|
||||
- archive/backup behavior;
|
||||
- safe recovery procedures;
|
||||
- operational caveats.
|
||||
|
||||
### docs/troubleshooting.md
|
||||
|
||||
**Audience:** administrators, operators
|
||||
|
||||
Recommended once recurring failure modes exist.
|
||||
|
||||
Each entry should include:
|
||||
|
||||
- symptom;
|
||||
- likely cause;
|
||||
- diagnostic command or inspection step;
|
||||
- safe fix;
|
||||
- relevant links.
|
||||
|
||||
### docs/internal/
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for modular, staged, service-oriented, or orchestration projects.
|
||||
|
||||
This directory describes implemented internal components. It is not the roadmap.
|
||||
|
||||
Use one file per major component where useful.
|
||||
|
||||
Each component doc should include:
|
||||
|
||||
1. purpose;
|
||||
2. inputs and outputs;
|
||||
3. boundaries;
|
||||
4. config fields used;
|
||||
5. external adapters used;
|
||||
6. state or manifest behavior, if applicable;
|
||||
7. skip/resume behavior, if applicable;
|
||||
8. failure behavior;
|
||||
9. tests to inspect before changing;
|
||||
10. architectural invariants.
|
||||
|
||||
### docs/roadmap/
|
||||
|
||||
**Audience:** maintainers, developers, LLM coding agents
|
||||
|
||||
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
|
||||
|
||||
Roadmap docs should clearly distinguish:
|
||||
|
||||
- proposed work;
|
||||
- accepted plans;
|
||||
- deferred ideas;
|
||||
- rejected ideas;
|
||||
- implementation prompts or task breakdowns, if useful.
|
||||
|
||||
Roadmap docs should not be confused with current behavior.
|
||||
|
||||
### docs/integrations/
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
|
||||
|
||||
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
|
||||
|
||||
Use one file per integration where useful.
|
||||
|
||||
## Examples Directory
|
||||
|
||||
Projects with non-trivial configuration or workflows should include `examples/`.
|
||||
|
||||
Useful examples include:
|
||||
|
||||
- minimal working config;
|
||||
- production-oriented config;
|
||||
- full annotated config;
|
||||
- local development config;
|
||||
- remote/object-storage config;
|
||||
- minimal session/input file.
|
||||
|
||||
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
|
||||
|
||||
## Security and Privacy
|
||||
|
||||
Docs and examples must not include:
|
||||
|
||||
- real API keys;
|
||||
- tokens;
|
||||
- passwords;
|
||||
- private keys;
|
||||
- private environment dumps;
|
||||
- sensitive user data;
|
||||
- raw private transcripts;
|
||||
- private infrastructure details unless intentionally public.
|
||||
|
||||
Document secret-handling mechanisms, not actual secret values.
|
||||
|
||||
## Maintenance Rules
|
||||
|
||||
When docs change, verify the affected behavior.
|
||||
|
||||
Where practical:
|
||||
|
||||
- load example config files in tests;
|
||||
- test CLI examples or command parser behavior;
|
||||
- validate documented flags against real flags;
|
||||
- remove stale references;
|
||||
- update links after renames;
|
||||
- keep roadmap content out of non-roadmap docs.
|
||||
|
||||
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
|
||||
|
||||
Documentation is complete only when it matches the current code.
|
||||
|
||||
## Documentation Change Checklist
|
||||
|
||||
Before merging documentation changes, verify:
|
||||
|
||||
- README is concise and orientation-focused.
|
||||
- `docs/policy/architecture.md` describes development principles.
|
||||
- Future work appears only under `docs/roadmap/`.
|
||||
- User-facing docs avoid unnecessary internals.
|
||||
- Developer-facing docs preserve boundaries and invariants.
|
||||
- Config examples match the schema.
|
||||
- CLI examples match real commands and flags.
|
||||
- Defaults appear in the canonical config reference.
|
||||
- No secrets or private data are included.
|
||||
- Links are accurate.
|
||||
@@ -1,128 +0,0 @@
|
||||
# Audita Release Checklist
|
||||
|
||||
Use this checklist before cutting a pre-1.0 or 1.0 release candidate.
|
||||
|
||||
## Core test pass
|
||||
|
||||
- Run:
|
||||
- `go test ./...`
|
||||
- Confirm tests pass without live LLM credentials and without Python dependencies.
|
||||
|
||||
## Config validation and precedence
|
||||
|
||||
- Validate a representative config:
|
||||
- `audita config validate --config <path>`
|
||||
- Inspect redacted effective config:
|
||||
- `audita config print-effective --config <path>`
|
||||
- Confirm precedence behavior:
|
||||
- defaults -> file config -> environment -> CLI.
|
||||
- Confirm default config search order:
|
||||
- `/usr/local/etc/audita/config.yml` first, then `/etc/audita/config.yml`.
|
||||
- Confirm missing both default-path config files is non-fatal when `--config`/`AUDITA_CONFIG` are unset.
|
||||
|
||||
## Output schema checks
|
||||
|
||||
- Verify default output schema remains `bare-segments`.
|
||||
- Verify `--output-schema audita-v1` emits object payload with `schema` and `version`.
|
||||
- Verify unknown schema (for example `seriatim-intermediate`) fails clearly.
|
||||
|
||||
## Subprocess contract checks
|
||||
|
||||
- With `--output`, verify stdout is empty on success.
|
||||
- Without `--output`, verify stdout contains transcript JSON only.
|
||||
- Verify `--report-json` writes file output and does not write report JSON to stdout.
|
||||
- Verify failure stderr remains human-readable and includes diagnostics path when available.
|
||||
- Verify nonzero exit on failures.
|
||||
|
||||
## Structured LLM checks
|
||||
|
||||
- Verify runtime uses the Audita-owned OpenAI-compatible adapter.
|
||||
- Verify structured response schemas are attached via `response_format.type=json_schema`.
|
||||
- Verify diagnostics metadata includes structured schema `id/version/name/sha256`.
|
||||
- Verify provider output is still locally decoded/validated before use.
|
||||
- Verify malformed module-stage structured payloads degrade to warnings/rejections instead of failing the run.
|
||||
|
||||
## Report and diagnostics schema checks
|
||||
|
||||
- Verify report metadata fields:
|
||||
- `report_schema_name`
|
||||
- `report_schema_version`
|
||||
- `output_schema`
|
||||
- `config_version` when file config is used.
|
||||
- Verify diagnostics artifact references exist in reports:
|
||||
- transcript/normalization/chunking/invocation/effective-config artifacts
|
||||
- utilization diagnostics artifact
|
||||
- correction ledger artifact
|
||||
- error log on failures.
|
||||
|
||||
## Redaction checks
|
||||
|
||||
- Verify secrets are redacted from:
|
||||
- `effective-config.json`
|
||||
- run-dir and `--report-json` reports
|
||||
- LLM request/response/error diagnostics payloads.
|
||||
- Verify no API keys/bearer tokens leak into fixtures or outputs.
|
||||
|
||||
## Prompt and validator metadata checks
|
||||
|
||||
- Verify prompt metadata appears in LLM request metadata diagnostics:
|
||||
- `prompt_id`, `prompt_version`, `prompt_source`, `embedded_path`, `sha256`.
|
||||
- Verify stable validator keys appear in report decisions/rejections.
|
||||
- Verify module warning records appear in reports for malformed proposal-generation payloads and malformed validator batches.
|
||||
- Verify built-in validator chains resolve and execute for default and explicit module runs.
|
||||
|
||||
## Utilization diagnostics checks
|
||||
|
||||
- Verify `utilization-diagnostics.json` exists on successful runs.
|
||||
- Verify partial utilization artifact behavior on controlled failure paths.
|
||||
- Verify utilization fields are structurally present and nonnegative:
|
||||
- effective concurrency
|
||||
- run timing
|
||||
- module timing summaries
|
||||
- per-validator timing summaries.
|
||||
|
||||
## Correction ledger checks
|
||||
|
||||
- Verify `correction-ledger.json` exists on successful runs.
|
||||
- Verify report references ledger artifact path.
|
||||
- Verify ledger dispositions include applied/rejected and skipped/failed where exercised.
|
||||
- Verify validator rejection and proposal-application skip remain distinct.
|
||||
|
||||
## Pipeline behavior checks
|
||||
|
||||
- Verify default full pipeline run remains:
|
||||
- `glossary`, `homophones`, `glossary`, `spoken_word`, `grammar`
|
||||
- with deterministic repeated instance naming (`glossary_1`, `glossary_2`).
|
||||
- Verify explicit module runs (`--modules`) still work.
|
||||
|
||||
## Failure and cancellation checks
|
||||
|
||||
- Verify controlled failure paths retain diagnostics and produce best-effort failure reports.
|
||||
- Verify malformed proposal-generation payloads keep exit code `0`, keep stderr empty on success, and record warnings in reports/diagnostics.
|
||||
- Verify malformed validator payloads reject only the affected batch and do not fail the module.
|
||||
- Verify timeout/cancellation paths exit nonzero, do not hang, and retain failure diagnostics when initialized.
|
||||
|
||||
## Release fixture/idempotence checks
|
||||
|
||||
- Run release fixtures (`internal/cli/testdata/release`) through `go test ./...`.
|
||||
- Confirm fixture checks cover:
|
||||
- must-apply and must-not-apply expectations
|
||||
- protected-term survival
|
||||
- report and diagnostics contracts
|
||||
- output-schema checks
|
||||
- prompt/schema metadata diagnostics
|
||||
- utilization/ledger artifacts
|
||||
- idempotence-oriented second pass no-op behavior with deterministic fake responses.
|
||||
|
||||
## Deferred-feature guardrail
|
||||
|
||||
- Confirm release docs do not claim support for deferred items:
|
||||
- filesystem prompt overrides
|
||||
- user-configurable validator chains
|
||||
- arbitrary user-supplied output schemas
|
||||
- resume/start-at/stop-after execution
|
||||
- diff/check/propose-only modes
|
||||
- generated transcript descriptions enabled by default
|
||||
- interactive review UI
|
||||
- UI/server wrapper
|
||||
- provider benchmarking harness.
|
||||
@@ -1,791 +0,0 @@
|
||||
# Pre-1.0 Code Quality and Deduplication Audit
|
||||
|
||||
## 1. Executive summary
|
||||
|
||||
Audita is in good shape for a limited pre-1.0 cleanup pass. The repository is small, package boundaries are mostly explicit, and the core public contract is already documented around `audita process`, config loading, output schemas, diagnostics, reports, embedded prompts, modules, and validators. The highest-value improvements are targeted centralization, not a rewrite.
|
||||
|
||||
Top three refactoring targets before 1.0:
|
||||
|
||||
1. Centralize module proposal plumbing and prompt payload construction across the four production modules.
|
||||
2. Centralize effective config loading plus schema/module catalog validation so `process`, `config print-effective`, and `config validate` cannot drift.
|
||||
3. Centralize diagnostics artifact names, stage names, and validator classification metadata used by reports and the correction ledger.
|
||||
|
||||
No major architectural risk is apparent. The main pre-1.0 risk is public-behavior drift from repeated policy strings, catalog values, artifact paths, and nearly identical command/module scaffolding.
|
||||
|
||||
This report was written to `docs/roadmap/audit.md`. `docs/roadmap/` already exists in the repository, although its previous `publish.md` file is currently deleted in the worktree by an unrelated change.
|
||||
|
||||
## 2. Repository map reviewed
|
||||
|
||||
Reviewed documentation:
|
||||
|
||||
- `README.md`
|
||||
- `docs/configuration.md`
|
||||
- `docs/architecture/architecture.md`
|
||||
- `docs/architecture/public-contract.md`
|
||||
- `docs/architecture/diagnostics.md`
|
||||
- `docs/architecture/output-schemas.md`
|
||||
- `docs/architecture/prompts.md`
|
||||
- `docs/architecture/validators.md`
|
||||
- `docs/architecture/structured-llm.md`
|
||||
- `docs/integration/subprocess-operations.md`
|
||||
- `docs/release-checklist.md`
|
||||
|
||||
Reviewed implementation areas:
|
||||
|
||||
- `cmd/audita`
|
||||
- `internal/cli`
|
||||
- `internal/core/config`
|
||||
- `internal/core/schema`
|
||||
- `internal/core/io`
|
||||
- `internal/core/normalization`
|
||||
- `internal/core/chunking`
|
||||
- `internal/core/diagnostics`
|
||||
- `internal/core/outputschema`
|
||||
- `internal/core/reporting`
|
||||
- `internal/framework/contracts`
|
||||
- `internal/framework/modules`
|
||||
- `internal/framework/proposal_generation`
|
||||
- `internal/framework/proposals`
|
||||
- `internal/framework/runner`
|
||||
- `internal/framework/validators`
|
||||
- `internal/framework/llm`
|
||||
- `internal/framework/responseschema`
|
||||
- `internal/framework/promptcontext`
|
||||
- `internal/framework/warnings`
|
||||
- `internal/modules/glossary`
|
||||
- `internal/modules/homophones`
|
||||
- `internal/modules/spoken_word`
|
||||
- `internal/modules/grammar`
|
||||
- `internal/prompts`
|
||||
- `internal/validators`
|
||||
- package tests and CLI parity/release fixtures under `internal/cli/testdata`
|
||||
|
||||
Major execution paths reviewed:
|
||||
|
||||
- `audita process <transcript.json> --glossary <glossary.yaml>`
|
||||
- `audita config validate --config <path>`
|
||||
- `audita config print-effective [--config <path>]`
|
||||
- default module sequence resolution and repeated glossary instance naming
|
||||
- proposal generation, validator execution, proposal application, report writing, diagnostics writing, and retention
|
||||
|
||||
Important absent or not-applicable areas:
|
||||
|
||||
- No `pkg/` directory exists.
|
||||
- No `examples/` directory exists.
|
||||
- No `docs/internal/` directory exists.
|
||||
- No `internal/app`, `internal/stage`, `internal/storage`, `internal/artifacts`, or `internal/manifest` packages exist. Their closest equivalents are `internal/cli`, `internal/framework/runner`, `internal/core/diagnostics`, and `internal/core/reporting`.
|
||||
|
||||
## 3. High-confidence deduplication opportunities
|
||||
|
||||
### 3.1 Module proposal plumbing is duplicated across all production modules
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/modules/glossary/module.go`
|
||||
- `internal/modules/homophones/module.go`
|
||||
- `internal/modules/spoken_word/module.go`
|
||||
- `internal/modules/grammar/module.go`
|
||||
- `internal/modules/*/prompt.go`
|
||||
- `internal/framework/proposal_generation`
|
||||
- `internal/framework/promptcontext`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- Each module has the same `Module` struct shape, `Validators` copy behavior, `Propose` flow, section transcript extraction, transcript description extraction, `proposal_generation.GenerateCandidates` request construction, prompt metadata map construction, and stage-name formatting.
|
||||
- Each module also has a near-identical prompt payload builder with local `promptSegment` and `promptTranscriptSection` types, glossary JSON marshaling, transcript section JSON marshaling, transcript description block rendering, and two-message return shape.
|
||||
- `collectSectionProposals` already passes a section transcript to each module, but each module then filters that transcript again by section metadata.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- A diagnostics or prompt-context bug fix would need to be repeated in four modules.
|
||||
- Prompt metadata fields and stage names are diagnostics-visible and could drift by module.
|
||||
- The double section filtering is currently harmless, but it obscures the runner/module contract.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Add a small shared helper for module proposal execution, likely in `internal/framework/proposal_generation` or a narrow `internal/modules/modulekit` package.
|
||||
- Keep domain-specific prompt IDs and prompt text local to each module.
|
||||
- Move transcript section prompt payload construction into a shared prompt-context helper, for example `promptcontext.MarshalTranscriptSection`.
|
||||
- Provide one helper for prompt metadata maps instead of manually expanding `prompt_id`, `prompt_version`, `prompt_source`, `embedded_path`, and `sha256` in every module.
|
||||
- Preserve current module `Key`, replacement policy, and validator chain ownership.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Keep one golden or table-driven prompt payload test per module for domain-specific wording.
|
||||
- Add shared tests for transcript section JSON shape, empty transcript handling, categories copy behavior, and prompt metadata fields.
|
||||
- Add a parity test that all four module `Propose` methods still write diagnostics under the same module instance directory and produce the same correction mapping.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Low to medium. The behavior is highly duplicated, but prompt and diagnostics behavior is sensitive. Refactor behind existing module tests and CLI parity fixtures.
|
||||
|
||||
### 3.2 Effective config loading is repeated between commands
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/cli/run.go`
|
||||
- `internal/core/config`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- `runProcess` and `runConfigPrintEffective` both resolve config path, start from defaults, optionally load/apply file config, then apply environment overrides.
|
||||
- `runConfigValidate` separately loads a file, applies it to defaults, and validates it.
|
||||
- Path source metadata is computed in `internal/cli`, not `internal/core/config`, even though the precedence contract is documented as config behavior.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- Config precedence is part of the public contract. If a future setting is added, three command paths may need coordinated updates.
|
||||
- `config print-effective` is the user-visible diagnostic for effective config. It should use the same loader as `process`, except for intentionally omitted CLI overrides.
|
||||
- The current code is understandable, but the behavior is repeated in a way that makes drift likely as config grows.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Add a narrow effective-config loader in `internal/core/config`, returning `Config`, source path, source type, and version metadata.
|
||||
- Keep command-specific CLI flag parsing in `internal/cli`.
|
||||
- Model the intentional differences explicitly:
|
||||
- `process`: defaults + file + env + CLI overrides
|
||||
- `config print-effective`: defaults + file + env
|
||||
- `config validate`: file schema + default-backed config validation, no env
|
||||
- Move `resolveConfigPath` or an equivalent path resolver into `internal/core/config`.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- One table-driven config loader test covering explicit `--config`, `AUDITA_CONFIG`, default search paths, missing explicit paths, and missing default paths.
|
||||
- CLI tests asserting `process` and `print-effective` share file+env behavior.
|
||||
- A regression test that `config validate` remains file-only and does not read environment overrides.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Low. Behavior is already explicit and well tested; the refactor can be done by moving code without changing precedence.
|
||||
|
||||
### 3.3 Module catalog validation is split across config, contracts, and module factory
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/core/config/validation.go`
|
||||
- `internal/framework/contracts/contracts.go`
|
||||
- `internal/framework/modules/registry.go`
|
||||
- `internal/validators/chains.go`
|
||||
- `internal/framework/validators/models.go`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- Module keys appear in multiple places:
|
||||
- config default CSV: `glossary,homophones,glossary,spoken_word,grammar`
|
||||
- module factory constants and known-key map
|
||||
- built-in validator chains
|
||||
- confidence threshold lookup
|
||||
- individual module `Key()` methods
|
||||
- `Config.Validate` checks only that module names are non-empty. An unsupported configured module can pass `audita config validate` and fail later in `process` runner setup.
|
||||
- `contracts.ResolveModuleRunSpecs` only assigns instance names; it does not validate production module support.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- `audita config validate` is documented as a CI/preflight command. Letting unsupported modules pass weakens that preflight.
|
||||
- Module key drift could affect thresholds, validator chains, reports, and unsupported-module errors.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Introduce a small canonical module catalog or key package that can be imported by config validation, module factory construction, validator chain resolution, and threshold lookup without creating a cycle.
|
||||
- Keep module construction in `internal/framework/modules`; the catalog should expose keys and validation only.
|
||||
- Make `Config.Validate` reject unknown built-in module keys through that catalog.
|
||||
- Keep repeated module instances valid.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- `internal/core/config` test: unknown `pipeline.modules` fails validation.
|
||||
- `internal/cli` test: `audita config validate --config` rejects an unsupported module before runtime.
|
||||
- Existing `internal/framework/modules` unknown-module tests should continue to pass.
|
||||
- Validator chain tests should assert every catalog module has a built-in chain.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Medium. This tightens validation behavior. It is desirable before 1.0, but if unknown modules were intentionally allowed for future extension, document that explicitly instead.
|
||||
|
||||
### 3.4 Output schema support is hardcoded in config validation and registry
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/core/config/validation.go`
|
||||
- `internal/core/outputschema/registry.go`
|
||||
- `docs/architecture/output-schemas.md`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- `Config.Validate` hardcodes `bare-segments` and `audita-v1`.
|
||||
- `outputschema.Resolve` owns the actual output schema registry and returns the runtime error for unsupported schema names.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- Adding or deferring a schema requires updating multiple places.
|
||||
- Public behavior could drift: a schema might validate in config but fail at output time, or vice versa.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Make `internal/core/outputschema` expose `IsSupported`, `SupportedKeys`, or a validation function.
|
||||
- Have config validation call that helper or consume shared constants.
|
||||
- Keep actual encoding logic in `outputschema`; config should not know encoder details.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Config validation test for every output schema returned by the registry.
|
||||
- Output schema registry test that unsupported `seriatim-intermediate` still fails clearly until implemented.
|
||||
- CLI test that unsupported `--output-schema` fails before output write.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Low. This is a straightforward catalog centralization.
|
||||
|
||||
### 3.5 Diagnostics artifact names and report metadata paths are repeated
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/core/diagnostics/run_dir.go`
|
||||
- `internal/cli/run.go`
|
||||
- `internal/core/reporting/report.go`
|
||||
- docs under `docs/architecture` and `docs/integration`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- Artifact filenames such as `source-transcript.json`, `source-transcript-parsed.json`, `normalized-transcript.json`, `normalization-summary.json`, `chunking-summary.json`, `utilization-diagnostics.json`, `correction-ledger.json`, `invocation.json`, `effective-config.json`, `report.json`, and `error.log` are repeated between run-directory writers and `buildProcessReport`.
|
||||
- `runProcess` writes `utilization-diagnostics.json` and `correction-ledger.json` by raw string on both success and failure paths.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- These names are part of the documented diagnostics contract.
|
||||
- A filename change would need to be made in multiple places, and report metadata could point at files that are no longer written.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Define diagnostics artifact name constants in `internal/core/diagnostics`.
|
||||
- Add a helper that returns `reporting.DiagnosticsMetadata` for a run directory and status.
|
||||
- Add named methods for utilization diagnostics and correction ledger writes, or at least constants used by `WriteJSONArtifact`.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Unit test that `diagnostics.MetadataForRunDirectory` matches files written by `RunDirectory`.
|
||||
- CLI success/failure tests should continue to assert report metadata paths and actual file existence.
|
||||
- Add a test for failure report metadata including `error.log`.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Low. This is mostly string centralization, with high public-contract value.
|
||||
|
||||
### 3.6 Validator execution class is duplicated and partially hardcoded
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/validators/registry.go`
|
||||
- `internal/validators/metadata/metadata.go`
|
||||
- `internal/validators/*/validator.go`
|
||||
- `internal/framework/runner/runner.go`
|
||||
- `internal/cli/review_artifacts.go`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- Validator constructors wrap validators with execution class metadata.
|
||||
- `BuiltInValidatorDefinition` also has an `LLMBacked` field.
|
||||
- Runner uses `metadata.ClassOf` to order deterministic validators before LLM-backed validators.
|
||||
- Correction ledger classification uses a local hardcoded map of LLM-backed validator names.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- Adding a new LLM-backed validator could be ordered correctly by runner metadata but appear in the wrong correction-ledger section.
|
||||
- Validator class is domain metadata, not report-building policy. It should have one source of truth.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Make validator classification resolvable by validator instance or stable key from a single metadata source.
|
||||
- Remove the unused or redundant `LLMBacked` field, or make it the canonical source used by constructors, runner ordering, and ledger formatting.
|
||||
- Replace the local ledger map with `metadata.ClassOf` when possible, or a registry lookup by stable key.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Correction ledger test that LLM-backed decisions are classified from validator metadata, not a local string map.
|
||||
- Registry test that every registered LLM-backed validator reports the same class through every public metadata path.
|
||||
- Runner ordering test should remain in place.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Low to medium. The implementation is small, but correction-ledger shape is diagnostics-visible.
|
||||
|
||||
### 3.7 Malformed structured-output classification is duplicated
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/framework/proposal_generation/generate.go`
|
||||
- `internal/framework/validators/llm_validators.go`
|
||||
- `internal/framework/llm/openai_compatible_client.go`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- Proposal generation and LLM validators both classify malformed structured-output errors by scanning error message substrings.
|
||||
- The marker lists are currently the same, but they are maintained independently.
|
||||
- The actual errors originate in the LLM adapter.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- Proposal-generation malformed payloads become warnings with zero proposals, while validator malformed payloads reject affected batches with warnings. If classifiers drift, similar adapter failures could be downgraded in one workflow and hard-fail in another.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Prefer a typed error or exported classifier from `internal/framework/llm`.
|
||||
- If typed errors are too invasive, create one shared classifier function in a lower framework package used by both proposal generation and validators.
|
||||
- Preserve the different handling semantics at each call site.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Shared classifier table for all adapter malformed-output errors.
|
||||
- Proposal-generation test and validator test should assert the same representative malformed adapter errors are downgraded.
|
||||
- Adapter tests should assert typed/classified errors wrap useful context and still redact secrets.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Medium. Error typing can accidentally affect retry and wrapping behavior; do this with focused tests.
|
||||
|
||||
## 4. Medium-confidence opportunities
|
||||
|
||||
### 4.1 CLI flag registration and override extraction are large and repetitive
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/cli/run.go`
|
||||
- `internal/core/config/flags.go`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- Each process flag has a field in `processFlags`, a registration entry in `newProcessFlagSet`, a case in `fs.Visit`, and an assignment in `config.ApplyCLIOverrides`.
|
||||
- File config and environment config also set many of the same effective config fields.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- Adding a new config option requires multiple edits. Missing one edit could create a flag that displays but does not override, or a config field with no CLI override.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Avoid a generic reflection-heavy flag system before 1.0.
|
||||
- Consider a small metadata table only for simple scalar flags, or a focused helper that maps visited flags to `CLIOverrides`.
|
||||
- Keep nontrivial semantics, such as legacy concurrency alias precedence, explicit in code.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- CLI override parity test for every stable flag that mutates config.
|
||||
- A test that default flag values reflect file+env effective config before CLI overrides.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Medium. A broad flag abstraction would be riskier than the current duplication. Do only a small helper if it clearly reduces missed updates.
|
||||
|
||||
### 4.2 Config source application repeats field-level assignments
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/core/config/file_config.go`
|
||||
- `internal/core/config/env.go`
|
||||
- `internal/core/config/flags.go`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- The same effective fields are assigned from file config, env vars, and CLI overrides.
|
||||
- Some semantics differ intentionally: file config supports `api_key_env`, env supports `OPENROUTER_API_KEY` fallback, CLI uses direct values.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- Field additions are easy to miss in one source.
|
||||
- Error messages and trimming behavior can drift.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Do not force all config sources through one generic mapper.
|
||||
- Add small setter helpers for repeated config subdomains such as LLM target, concurrency, thresholds, normalization, and diagnostics.
|
||||
- Keep source-specific parsing and error labels local.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Cross-source table proving file, env, and CLI all reach the same effective fields where they are meant to.
|
||||
- Tests for intentional differences: API key env resolution, `OPENROUTER_API_KEY` fallback, CLI direct API key, and transcript description trimming.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Medium. Useful, but only after the effective loader and catalog cleanup.
|
||||
|
||||
### 4.3 Prompt metadata and response schema metadata map construction repeats
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/modules/*/module.go`
|
||||
- `internal/framework/proposal_generation/generate.go`
|
||||
- `internal/framework/validators/llm_validators.go`
|
||||
- `internal/prompts`
|
||||
- `internal/framework/responseschema`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- Prompt metadata maps are manually expanded in module proposal generation and validator diagnostics.
|
||||
- Response schema metadata maps are built independently in proposal generation and validator diagnostics.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- Metadata fields are diagnostics-visible and useful for reproducibility.
|
||||
- Adding a metadata field requires updating multiple call sites.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Add `Metadata.Map()` or a typed diagnostics metadata struct in `internal/prompts`.
|
||||
- Add `responseschema.Metadata()` or a method returning a stable diagnostics shape.
|
||||
- Prefer typed structs over `map[string]any` where possible.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Prompt metadata rendering test should assert all registered prompts expose stable metadata.
|
||||
- Proposal and validator diagnostics tests should assert the shared metadata helper is used.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Low.
|
||||
|
||||
### 4.4 Secret redaction logic is split across config, LLM diagnostics, and adapter errors
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/core/config/redaction.go`
|
||||
- `internal/framework/llm/diagnostics.go`
|
||||
- `internal/framework/llm/client_common.go`
|
||||
- `internal/framework/proposal_generation/generate.go`
|
||||
- `internal/framework/runner/runner.go`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- Config redaction replaces non-empty API keys with `[REDACTED]`.
|
||||
- LLM diagnostics replace configured secret values and `Bearer <secret>`.
|
||||
- Adapter error sanitization separately replaces secrets and bearer values.
|
||||
- Proposal and validator paths separately assemble secret lists.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- Secret redaction is a public guarantee.
|
||||
- New secret-bearing config fields could be missed in one path.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Add a small redaction helper package or keep it in `internal/framework/llm` only if it remains LLM-specific.
|
||||
- Centralize `[]string` secret extraction from `config.Config`.
|
||||
- Keep config structural redaction separate from byte/string payload redaction, but share the redaction token and value replacement behavior.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- One test that a proposal-generation error, validator diagnostic artifact, effective config artifact, and surfaced provider error all redact the same configured secrets.
|
||||
- Existing subprocess no-secret-leak test should remain as an end-to-end guard.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Medium. The current coverage appears strong; change carefully.
|
||||
|
||||
### 4.5 Test fakes and fixture helpers are duplicated across packages
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/modules/*/module_test.go`
|
||||
- `internal/framework/proposal_generation/generate_test.go`
|
||||
- `internal/framework/validators/llm_validators_test.go`
|
||||
- `internal/cli/run_test.go`
|
||||
- `cmd/audita/main_integration_test.go`
|
||||
- `internal/cli/release_fixtures_test.go`
|
||||
- `internal/cli/parity_test.go`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- Several packages define fake structured LLM clients, fixture path helpers, read/write helpers, diagnostics glob assertions, and run-directory helpers.
|
||||
- The four module test files have particularly similar fake clients and proposal-diagnostics assertions.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- Refactors in LLM or diagnostics behavior require updating many tests.
|
||||
- Some duplicated tests are valuable because they preserve per-module public behavior; the issue is helper duplication, not coverage volume.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Add package-local helper files where duplication is within a package.
|
||||
- For cross-package fakes, prefer a small internal test support package only if it does not create import cycles or hide test intent.
|
||||
- Keep module-specific assertions local.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- This is test infrastructure cleanup. Existing tests should remain semantically equivalent.
|
||||
- Add helper tests only if helpers contain nontrivial behavior, such as fake response sequencing.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Low.
|
||||
|
||||
### 4.6 Stage-name construction is inconsistent enough to centralize, but not enough to redesign
|
||||
|
||||
Affected files/packages:
|
||||
|
||||
- `internal/modules/*/module.go`
|
||||
- `internal/framework/proposal_generation/generate.go`
|
||||
- `internal/framework/validators/llm_validators.go`
|
||||
- `internal/framework/runner/observability.go`
|
||||
|
||||
Duplicated or near-duplicated behavior:
|
||||
|
||||
- Modules pass stage names like `<module_instance>:proposal:section-0001`.
|
||||
- `proposal_generation` has a default builder using `<module_instance>:proposal-generation:section-0001`, but production modules bypass it.
|
||||
- Validators build `<module_instance>:<validator>:batch-0001`.
|
||||
- Utilization extracts module instance by splitting stage names on `:`.
|
||||
|
||||
Why it matters:
|
||||
|
||||
- Stage names affect diagnostics filenames and observability grouping.
|
||||
- Current behavior works, but the naming grammar is implicit.
|
||||
|
||||
Recommended refactor:
|
||||
|
||||
- Add narrow helpers for proposal and validator stage names.
|
||||
- Preserve current production stage names unless there is a deliberate pre-1.0 diagnostics compatibility decision.
|
||||
- Keep filename sanitization in `internal/framework/llm`.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Unit tests for stage-name helper output.
|
||||
- Utilization test that module instance extraction still works for proposal and validator stage names.
|
||||
|
||||
Risk level:
|
||||
|
||||
- Medium. Renaming stages can change diagnostics filenames, so avoid unnecessary churn.
|
||||
|
||||
## 5. Boundary and responsibility concerns
|
||||
|
||||
### CLI owns too much report and diagnostics metadata assembly
|
||||
|
||||
`internal/cli/run.go` is doing orchestration, command parsing, config loading, output routing, report assembly, diagnostics metadata path assembly, and correction-ledger construction. This is acceptable for a small CLI, but two pieces are drifting beyond command responsibility:
|
||||
|
||||
- diagnostics artifact path metadata belongs closer to `internal/core/diagnostics`;
|
||||
- report assembly and correction-ledger mapping belong closer to `internal/core/reporting` or a narrow reporting adapter package.
|
||||
|
||||
Recommended home:
|
||||
|
||||
- `internal/core/diagnostics`: artifact constants and diagnostics metadata path construction.
|
||||
- `internal/core/reporting`: pure mapping from runner/config/diagnostics state into report payloads.
|
||||
- `internal/cli`: command parsing, invocation wiring, exit codes, stdout/stderr behavior.
|
||||
|
||||
### Config validation lacks catalog ownership
|
||||
|
||||
`internal/core/config` currently validates only generic module list shape and hardcodes output schema keys. Because modules and output schemas are public contract values, config validation should use a catalog owned by the relevant domain.
|
||||
|
||||
Recommended home:
|
||||
|
||||
- output schema validation: `internal/core/outputschema`;
|
||||
- module key validation: a small catalog package or lower-level constants package importable by config, module factory, validator chains, and threshold lookup.
|
||||
|
||||
### Runner owns adapter shims between contracts and validator framework
|
||||
|
||||
`internal/framework/runner` contains `validationLLMClientAdapter` and `llmDiagnosticsWriterAdapter`. This is not a serious problem today because runner wires proposal and validation workflows. If these adapters grow, move them to `internal/framework/validators` or a small integration package so runner remains focused on orchestration.
|
||||
|
||||
### LLM malformed-output policy is spread across callers
|
||||
|
||||
The LLM adapter emits the errors, while proposal generation and validators classify them by message text. The policy decision is caller-specific, but the classification should live with the LLM/framework error type.
|
||||
|
||||
## 6. Path, key, and naming construction review
|
||||
|
||||
Centralized enough:
|
||||
|
||||
- LLM diagnostics artifact suffixes and stage sanitization are centralized in `internal/framework/llm/diagnostics.go`.
|
||||
- Output file writing is routed through `internal/core/io.WriteFile`.
|
||||
- Run directories are created in `internal/core/diagnostics.NewRunDirectory`.
|
||||
|
||||
Needs cleanup:
|
||||
|
||||
- Core diagnostics artifact names are repeated between `RunDirectory` writer methods and `buildProcessReport`.
|
||||
- `utilization-diagnostics.json` and `correction-ledger.json` are raw strings in both success and failure paths.
|
||||
- Proposal and validator diagnostics subdirectory construction repeats `filepath.Join(diagnosticsDir, moduleInstance)`.
|
||||
- Proposal and validator stage names are manually formatted in multiple packages.
|
||||
- Module keys are repeated across config defaults, module factory, validator chains, confidence threshold lookup, and module implementations.
|
||||
- Output schema names are repeated between config validation and `outputschema`.
|
||||
|
||||
Recommendation:
|
||||
|
||||
- Start with artifact constants and metadata helpers because that is the lowest-risk path/key cleanup.
|
||||
- Then centralize stage-name helpers without changing current production naming.
|
||||
- Defer any broader "path manager" abstraction.
|
||||
|
||||
## 7. Resolution and catalog review
|
||||
|
||||
Modules:
|
||||
|
||||
- Runtime module construction has a production registry in `internal/framework/modules`.
|
||||
- Instance naming for repeated modules is centralized in `contracts.ResolveModuleRunSpecs`.
|
||||
- Unknown module failure exists in the factory, but config validation does not catch unknown modules.
|
||||
- Built-in validator chain resolution separately maps module key to validator keys.
|
||||
|
||||
Output schemas:
|
||||
|
||||
- Encoding is centralized in `internal/core/outputschema`.
|
||||
- Validation is duplicated in config.
|
||||
|
||||
Prompts:
|
||||
|
||||
- Prompt asset lookup and metadata are centralized in `internal/prompts`.
|
||||
- Prompt metadata map construction is repeated at call sites.
|
||||
- Prompt source selection is intentionally built-in only and should remain that way for 1.0.
|
||||
|
||||
Validators:
|
||||
|
||||
- Validator construction is package-owned under `internal/validators`.
|
||||
- Chains are centralized in `internal/validators/chains.go`.
|
||||
- Execution class metadata exists, but reporting/correction-ledger classification does not fully use it.
|
||||
|
||||
Schemas:
|
||||
|
||||
- Transcript and glossary parsing/validation are centralized in `internal/core/schema`.
|
||||
- Structured LLM response schemas are centralized in `internal/framework/responseschema`.
|
||||
- Output schema registry and response schema registry are appropriately separate.
|
||||
|
||||
Recommendation:
|
||||
|
||||
- Introduce only small catalog helpers for module keys, output schema keys, prompt metadata maps, response schema metadata maps, and validator execution class.
|
||||
- Avoid user-configurable modules, validators, prompts, or schemas before 1.0 unless already planned elsewhere.
|
||||
|
||||
## 8. Config and command-loading review
|
||||
|
||||
Consistent behavior:
|
||||
|
||||
- The documented precedence for `process` is implemented: defaults, file config, environment, CLI.
|
||||
- `config print-effective` intentionally omits CLI process flags and uses defaults, file config, and environment.
|
||||
- `config validate` intentionally requires `--config` and does not require transcript/glossary inputs.
|
||||
- Missing explicit config paths are hard failures; missing default paths are non-fatal.
|
||||
- Environment parsing and CLI parsing both preserve legacy total-concurrency alias behavior.
|
||||
|
||||
Likely accidental or high-risk differences:
|
||||
|
||||
- Unsupported module names pass `Config.Validate` and `audita config validate`.
|
||||
- Output schema support is duplicated instead of delegated to the output schema registry.
|
||||
- Config path resolution lives in CLI even though it is part of config behavior.
|
||||
|
||||
Intentional differences:
|
||||
|
||||
- File config resolves `api_key_env`; env and CLI set direct API key values.
|
||||
- `OPENROUTER_API_KEY` is an environment fallback only for the primary LLM.
|
||||
- `transcript-description` has CLI/config support but no `AUDITA_*` environment variable, matching documentation.
|
||||
|
||||
Recommendation:
|
||||
|
||||
- Build a shared effective config context helper and keep source-specific parsing semantics explicit.
|
||||
- Tighten catalog validation before 1.0 if unknown modules are not meant to be accepted.
|
||||
|
||||
## 9. State, manifest, or progress handling review
|
||||
|
||||
Audita does not currently have a manifest/checkpoint/resume model. State is per-run diagnostics and report artifacts.
|
||||
|
||||
Consistent behavior:
|
||||
|
||||
- `process` creates one diagnostics run directory when diagnostics initialization succeeds.
|
||||
- Failures after run-dir creation write `error.log`, best-effort report artifacts, and retain diagnostics.
|
||||
- Success writes optional `--report-json`, run-dir `report.json`, utilization diagnostics, and correction ledger.
|
||||
- Retention is centralized in `diagnostics.ShouldRetainRunDirectory`.
|
||||
- There is no resume/retry/force behavior to preserve.
|
||||
|
||||
Drift risks:
|
||||
|
||||
- Success and failure paths both write utilization and correction-ledger artifacts with duplicated raw filenames.
|
||||
- Report diagnostics metadata is assembled independently from the run-directory writer methods.
|
||||
- Retention mode `never` currently still retains successful run directories in `ShouldRetainRunDirectory`, which may be intentional per tests or a naming/documentation mismatch. Do not change it in a dedup pass without first confirming semantics.
|
||||
|
||||
Recommendation:
|
||||
|
||||
- Centralize artifact names and report metadata path construction.
|
||||
- Keep retention behavior unchanged unless a separate bug review confirms the intended meaning of `never`.
|
||||
|
||||
## 10. Refactors to avoid before 1.0
|
||||
|
||||
- Do not introduce a generic workflow engine. The current sequential runner is clear and explicit.
|
||||
- Do not add a plugin architecture for modules, validators, prompts, or schemas before 1.0.
|
||||
- Do not redesign the CLI or replace `flag` with a larger framework only for deduplication.
|
||||
- Do not collapse all config source parsing into a reflection-based mapper; source semantics differ intentionally.
|
||||
- Do not merge module packages into one generic module type. Keep domain-specific prompt assets, keys, validator chains, and replacement policies visible.
|
||||
- Do not rewrite diagnostics or reporting schemas broadly. Centralize names and mapping helpers first.
|
||||
- Do not change diagnostics stage names casually; they affect artifact filenames and debugging workflows.
|
||||
- Do not consolidate deterministic and LLM validator behavior just because both return decisions. Their failure and batching semantics differ.
|
||||
- Do not generalize transcript/glossary schema parsing into a broad schema framework.
|
||||
- Do not reduce duplicated tests where the duplication protects distinct public command/module behavior.
|
||||
|
||||
## 11. Recommended implementation sequence
|
||||
|
||||
1. Centralize diagnostics artifact constants and diagnostics metadata path construction.
|
||||
2. Centralize output schema validation through `internal/core/outputschema`.
|
||||
3. Introduce a small module key catalog and use it in config validation, module factory, validator chains, and threshold lookup.
|
||||
4. Add an effective config loading context helper for defaults + file + env, then update `process` and `config print-effective`.
|
||||
5. Extract shared module proposal plumbing and prompt transcript-section payload construction.
|
||||
6. Centralize prompt metadata and response schema metadata map construction.
|
||||
7. Centralize validator execution-class lookup and update correction-ledger classification.
|
||||
8. Centralize malformed structured-output classification through a typed/shared LLM error helper.
|
||||
9. Add or consolidate focused test helpers for module LLM fakes, diagnostics assertions, and fixture paths.
|
||||
10. Do a final dead-code and legacy sweep for redundant helper fields such as unused validator definition metadata.
|
||||
|
||||
Each item can be a separate commit with package-level tests and at least one CLI regression where public behavior is involved.
|
||||
|
||||
## 12. Test strategy
|
||||
|
||||
Tests to add before refactoring:
|
||||
|
||||
- `internal/core/config`: unknown module key fails validation, if unsupported modules are not intended to be accepted.
|
||||
- `internal/core/config`: every output schema registry key validates through config.
|
||||
- `internal/core/diagnostics`: report metadata paths match run-directory artifact names.
|
||||
- `internal/validators`: validator class by key/instance is consistent for all registered validators.
|
||||
- `internal/framework/llm`: shared malformed structured-output classifier covers all current adapter malformed errors.
|
||||
|
||||
Tests to add during refactoring:
|
||||
|
||||
- `internal/framework/promptcontext`: transcript section prompt payload preserves IDs, speaker, timestamps, text, and categories.
|
||||
- `internal/framework/proposal_generation`: shared module proposal helper preserves current stage name, diagnostics dir, schema metadata, and malformed-output warning behavior.
|
||||
- `internal/cli`: `process` and `config print-effective` share defaults+file+env behavior.
|
||||
- `internal/cli`: `config validate` remains file-only and does not read env overrides.
|
||||
- `internal/cli`: correction ledger classifies deterministic and LLM validator decisions through canonical metadata.
|
||||
|
||||
Existing tests to run after each cleanup:
|
||||
|
||||
- `go test ./internal/core/config ./internal/core/outputschema`
|
||||
- `go test ./internal/core/diagnostics ./internal/core/reporting`
|
||||
- `go test ./internal/framework/proposal_generation ./internal/framework/validators ./internal/framework/runner`
|
||||
- `go test ./internal/validators/...`
|
||||
- `go test ./internal/modules/...`
|
||||
- `go test ./internal/cli ./cmd/audita`
|
||||
- Run `go test ./...` before merging a multi-package cleanup.
|
||||
|
||||
Validation note:
|
||||
|
||||
- During this report-only pass, no full test suite was run. A lightweight `go list ./...` completed package listing but emitted a sandbox warning while trying to write the Go module stat cache outside the repository.
|
||||
|
||||
## 13. Appendix: findings not worth acting on
|
||||
|
||||
### Separate module packages
|
||||
|
||||
The four production module packages contain visible repetition, but keeping separate packages is useful. The module domains, prompt assets, validator chains, and tests are distinct enough that a single generic module package would hide important behavior.
|
||||
|
||||
Do not refactor now beyond shared proposal/prompt plumbing.
|
||||
|
||||
### Report type duplication between runner and reporting
|
||||
|
||||
`runner.ModuleResult` and `reporting.ModuleReport` look similar. Keeping separate runtime and public report shapes is reasonable because runner owns execution state and reporting owns serialized public schema.
|
||||
|
||||
Only centralize mapping helpers; do not merge the types.
|
||||
|
||||
### Transcript and glossary parsing stay separate
|
||||
|
||||
Transcript JSON and glossary YAML parsing have different formats, validation rules, and error messages. There is no useful shared parser abstraction to extract.
|
||||
|
||||
### Response schema registry and output schema registry stay separate
|
||||
|
||||
Structured LLM response schemas and transcript output schemas are both "schemas", but they serve different users and have different lifecycles. Do not combine their registries.
|
||||
|
||||
### `flag` package usage
|
||||
|
||||
The CLI command surface is small. Replacing `flag` with a larger CLI framework would not pay for itself before 1.0.
|
||||
|
||||
### Local test duplication that protects public behavior
|
||||
|
||||
Some test duplication in CLI, subprocess, parity, and release fixtures is intentional. These tests exercise different public surfaces and should remain explicit even if helpers are shared.
|
||||
|
||||
### Filesystem state as diagnostics state
|
||||
|
||||
Audita has no resume/checkpoint semantics. Treating diagnostics artifacts as filesystem outputs is currently acceptable. A manifest system would be speculative before there is a resume or audit workflow that needs it.
|
||||
@@ -1,329 +0,0 @@
|
||||
# Pre-1.0 Deduplication Implementation Plan
|
||||
|
||||
This plan turns `docs/roadmap/audit.md` into staged, prompt-sized cleanup work for an LLM coding agent. Each stage should be implemented in order and kept small enough to review as an independent commit.
|
||||
|
||||
## Operating rules
|
||||
|
||||
- Read `docs/roadmap/audit.md` before starting any stage.
|
||||
- Preserve public CLI, report, diagnostics, config precedence, prompt metadata, and output-schema behavior unless a stage explicitly calls out an intended behavior change.
|
||||
- Keep the four production module packages separate: `glossary`, `homophones`, `spoken_word`, and `grammar`.
|
||||
- Do not introduce plugin systems, generic workflow engines, broad CLI framework rewrites, reflection-heavy config mappers, or merged module packages.
|
||||
- Prefer narrow helpers, catalogs, constants, and pure mapping functions over broad abstractions.
|
||||
- Run the targeted tests listed in each stage before moving to the next stage.
|
||||
- Run `go test ./...` before declaring the full sequence complete.
|
||||
- Ignore unrelated worktree changes, including the existing deletion of `docs/roadmap/publish.md`, unless the user explicitly asks to handle them.
|
||||
- Do not reduce parity, release-fixture, subprocess, or module-specific behavior coverage while consolidating helpers.
|
||||
|
||||
## Stages
|
||||
|
||||
### Stage 1: Diagnostics artifact constants and metadata paths
|
||||
|
||||
Goal:
|
||||
|
||||
- Centralize diagnostics artifact names and report diagnostics metadata path construction without changing any filenames or report fields.
|
||||
|
||||
Key edits:
|
||||
|
||||
- Define constants in `internal/core/diagnostics` for:
|
||||
- `source-transcript.json`
|
||||
- `source-transcript-parsed.json`
|
||||
- `normalized-transcript.json`
|
||||
- `normalization-summary.json`
|
||||
- `chunking-summary.json`
|
||||
- `utilization-diagnostics.json`
|
||||
- `correction-ledger.json`
|
||||
- `invocation.json`
|
||||
- `effective-config.json`
|
||||
- `report.json`
|
||||
- `error.log`
|
||||
- Add a diagnostics helper that builds `reporting.DiagnosticsMetadata` from a run directory path and failure/success status.
|
||||
- Update `RunDirectory` methods to use the constants.
|
||||
- Update CLI report assembly and utilization/correction-ledger writes to use the constants/helper instead of raw strings.
|
||||
|
||||
Behavior changes:
|
||||
|
||||
- None. All artifact names, report JSON keys, and path values must remain byte-for-byte compatible except for normal timestamp/order differences in existing outputs.
|
||||
|
||||
Tests:
|
||||
|
||||
- Add or update `internal/core/diagnostics` tests proving metadata helper paths match the artifact constants.
|
||||
- Run `go test ./internal/core/diagnostics ./internal/core/reporting ./internal/cli`.
|
||||
- Run any existing CLI report/diagnostics tests touched by this stage.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- No raw core diagnostics artifact filename strings remain in CLI report metadata assembly.
|
||||
- Existing success and failure reports still point to files that are actually written.
|
||||
- Retention behavior is unchanged.
|
||||
|
||||
### Stage 2: Output schema validation and module catalog
|
||||
|
||||
Goal:
|
||||
|
||||
- Move public key validation to small canonical catalogs so config validation, runtime resolution, and factory behavior cannot drift.
|
||||
|
||||
Key edits:
|
||||
|
||||
- Add `SupportedKeys`, `IsSupported`, or an equivalent validation helper to `internal/core/outputschema`.
|
||||
- Update `config.Validate` to use `internal/core/outputschema` for output schema validation.
|
||||
- Add a small canonical module key catalog that is importable by:
|
||||
- `internal/core/config`
|
||||
- `internal/framework/modules`
|
||||
- `internal/validators`
|
||||
- `internal/framework/validators`
|
||||
- Use the module catalog for default module key constants, known-key checks, validator chain keys, and confidence-threshold lookup.
|
||||
- Keep module construction in `internal/framework/modules`; the catalog must not construct modules.
|
||||
|
||||
Behavior changes:
|
||||
|
||||
- Intended behavior change: unsupported configured module keys should fail during config validation, including `audita config validate`.
|
||||
- Repeated supported module keys remain valid.
|
||||
- Output schema behavior remains unchanged for `bare-segments`, `audita-v1`, and unsupported names.
|
||||
|
||||
Tests:
|
||||
|
||||
- Add `internal/core/config` tests for unsupported module keys and repeated supported module keys.
|
||||
- Add config validation tests that every supported output schema validates.
|
||||
- Add or update output schema registry tests for supported and unsupported schemas.
|
||||
- Update module registry and validator chain tests to use the shared catalog where appropriate.
|
||||
- Run `go test ./internal/core/config ./internal/core/outputschema ./internal/framework/modules ./internal/framework/validators ./internal/validators/... ./internal/cli`.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Unknown modules fail before runner setup in config validation paths.
|
||||
- No duplicated hardcoded output schema support list remains in config validation.
|
||||
- No import cycle is introduced.
|
||||
|
||||
### Stage 3: Effective config loading context
|
||||
|
||||
Goal:
|
||||
|
||||
- Centralize config path resolution and defaults+file+env loading while keeping command-specific CLI overrides explicit.
|
||||
|
||||
Key edits:
|
||||
|
||||
- Move config path resolution from `internal/cli` into `internal/core/config` or add an equivalent exported helper there.
|
||||
- Add an effective config loader that returns:
|
||||
- effective `config.Config`
|
||||
- config path
|
||||
- config source (`flag`, `env`, `default`, or empty)
|
||||
- config version pointer when a file was loaded
|
||||
- Use the shared loader in `audita process` before applying CLI overrides.
|
||||
- Use the shared loader in `audita config print-effective`.
|
||||
- Keep `audita config validate` as file-only: load file, apply to defaults, validate, and do not apply environment overrides.
|
||||
|
||||
Behavior changes:
|
||||
|
||||
- None. Preserve existing precedence:
|
||||
- `process`: defaults, file config, environment, CLI flags
|
||||
- `config print-effective`: defaults, file config, environment
|
||||
- `config validate`: file config applied to defaults only
|
||||
- Preserve explicit config path failure behavior and missing default path non-fatal behavior.
|
||||
|
||||
Tests:
|
||||
|
||||
- Add table-driven config loader tests for:
|
||||
- explicit `--config`
|
||||
- `AUDITA_CONFIG`
|
||||
- default search paths
|
||||
- missing explicit path
|
||||
- missing env path
|
||||
- missing default paths
|
||||
- Add or update CLI tests proving `process` and `config print-effective` share file+env behavior.
|
||||
- Add or update CLI tests proving `config validate` ignores environment overrides.
|
||||
- Run `go test ./internal/core/config ./internal/cli ./cmd/audita`.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Config precedence is unchanged.
|
||||
- Config source/path/version metadata in invocation and reports is unchanged.
|
||||
- Config command stdout/stderr and exit-code behavior is unchanged except for the intended unknown-module validation from Stage 2.
|
||||
|
||||
### Stage 4: Prompt/schema metadata and stage-name helpers
|
||||
|
||||
Goal:
|
||||
|
||||
- Centralize diagnostics-visible metadata and stage-name construction without changing production diagnostics names.
|
||||
|
||||
Key edits:
|
||||
|
||||
- Add a helper or method in `internal/prompts` that returns the stable prompt metadata diagnostics shape currently expanded by call sites.
|
||||
- Add a helper or method in `internal/framework/responseschema` that returns the stable response schema metadata diagnostics shape currently expanded by call sites.
|
||||
- Add shared proposal and validator stage-name helpers in the lowest package that avoids import cycles.
|
||||
- Use the helpers in proposal generation, LLM validators, and production modules.
|
||||
|
||||
Behavior changes:
|
||||
|
||||
- None. Preserve current production stage names:
|
||||
- module proposal stages keep their existing `proposal` naming form;
|
||||
- validator batch stages keep their existing validator/batch naming form.
|
||||
- Preserve all prompt metadata and response schema metadata field names and values.
|
||||
|
||||
Tests:
|
||||
|
||||
- Add prompt metadata helper tests covering every registered prompt.
|
||||
- Add response schema metadata helper tests covering every registered response schema.
|
||||
- Add stage-name helper tests for no-section, section, and validator batch cases.
|
||||
- Run `go test ./internal/prompts ./internal/framework/responseschema ./internal/framework/proposal_generation ./internal/framework/validators ./internal/modules/...`.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- No manual prompt metadata map expansion remains in production module proposal plumbing.
|
||||
- No duplicated response schema metadata map construction remains in proposal generation and LLM validators.
|
||||
- Existing diagnostics fixture/path assertions still pass.
|
||||
|
||||
### Stage 5: Shared module proposal and prompt payload plumbing
|
||||
|
||||
Goal:
|
||||
|
||||
- Remove duplicated proposal execution and transcript-section prompt payload construction while preserving module-specific domain behavior.
|
||||
|
||||
Key edits:
|
||||
|
||||
- Add a narrow shared proposal execution helper, preferably in `internal/framework/proposal_generation` unless import cycles require a small module helper package.
|
||||
- The helper should own:
|
||||
- transcript description extraction from config;
|
||||
- `GenerateCandidates` request construction;
|
||||
- prompt metadata attachment;
|
||||
- stage-name selection;
|
||||
- conversion from generated corrections/warnings to `contracts.ProposalResult`.
|
||||
- Add shared transcript-section prompt payload construction in `internal/framework/promptcontext`.
|
||||
- Update each production module to provide only:
|
||||
- module key;
|
||||
- replacement policy;
|
||||
- validator chain;
|
||||
- prompt ID;
|
||||
- domain-specific `BuildProposalMessages` call or message builder.
|
||||
- Remove each module's redundant section transcript filtering if the runner already passes section-limited transcripts.
|
||||
|
||||
Behavior changes:
|
||||
|
||||
- None. Preserve module keys, replacement policies, validator chains, prompt IDs, diagnostics directories, proposal indexes, warning behavior, and correction mapping.
|
||||
|
||||
Tests:
|
||||
|
||||
- Add promptcontext tests for transcript section payload shape, empty transcript handling, section index, and category copying.
|
||||
- Keep one module-specific prompt test per production module for domain wording and constraints.
|
||||
- Add or update module proposal tests proving diagnostics are still written under the same module instance directory.
|
||||
- Run `go test ./internal/framework/promptcontext ./internal/framework/proposal_generation ./internal/modules/... ./internal/cli`.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Four production modules share proposal execution plumbing.
|
||||
- Module packages remain separate and readable.
|
||||
- CLI parity and release fixture behavior is unchanged.
|
||||
|
||||
### Stage 6: Validator classification and malformed LLM output policy
|
||||
|
||||
Goal:
|
||||
|
||||
- Use one source of truth for validator execution class and one shared classifier for malformed structured-output errors.
|
||||
|
||||
Key edits:
|
||||
|
||||
- Make validator execution class resolvable by stable validator key and by validator instance.
|
||||
- Replace the correction-ledger hardcoded LLM-backed validator map with the canonical metadata source.
|
||||
- Remove redundant validator metadata fields only after all call sites use the canonical source.
|
||||
- Add a shared malformed structured-output classifier in `internal/framework/llm` or another low-level framework package.
|
||||
- Update proposal generation and LLM validators to use the shared classifier while preserving their different handling outcomes.
|
||||
|
||||
Behavior changes:
|
||||
|
||||
- None. Proposal-generation malformed payloads still downgrade to warnings with zero proposals for affected sections.
|
||||
- Validator malformed payloads still reject affected batches with warnings.
|
||||
- Correction-ledger deterministic vs LLM validator sections should be unchanged for current validators.
|
||||
|
||||
Tests:
|
||||
|
||||
- Add validator metadata tests proving every registered validator has the expected execution class by key and instance.
|
||||
- Add correction-ledger tests proving deterministic and LLM-backed decisions are classified through canonical metadata.
|
||||
- Add shared malformed-output classifier tests covering current adapter malformed-output messages.
|
||||
- Update proposal-generation and validator tests to assert representative malformed adapter errors are still downgraded.
|
||||
- Run `go test ./internal/validators/... ./internal/framework/validators ./internal/framework/proposal_generation ./internal/framework/llm ./internal/cli`.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- No local hardcoded LLM-backed validator map remains in correction-ledger construction.
|
||||
- Proposal-generation and validator malformed-output classifier lists cannot drift.
|
||||
- Existing runner validator ordering is unchanged.
|
||||
|
||||
### Stage 7: Redaction and adapter workflow cleanup
|
||||
|
||||
Goal:
|
||||
|
||||
- Reduce duplicated secret extraction/redaction setup while preserving all no-secret-leak guarantees.
|
||||
|
||||
Key edits:
|
||||
|
||||
- Add a shared helper that extracts all configured LLM secret values from `config.Config`.
|
||||
- Use the helper in proposal-generation diagnostics and validator diagnostics setup.
|
||||
- Keep config structural redaction (`Config.Redacted`) separate from byte/string payload redaction.
|
||||
- Keep adapter error redaction behavior compatible with current surfaced errors.
|
||||
- Move runner adapter shims only if Stage 6 or this stage makes them materially larger; otherwise leave them in runner.
|
||||
|
||||
Behavior changes:
|
||||
|
||||
- None. Redaction token and no-secret-leak behavior remain unchanged.
|
||||
|
||||
Tests:
|
||||
|
||||
- Add or update tests proving proposal diagnostics, validator diagnostics, effective config artifacts, and surfaced adapter errors redact the same configured secrets.
|
||||
- Keep existing subprocess no-secret-leak tests.
|
||||
- Run `go test ./internal/core/config ./internal/framework/llm ./internal/framework/proposal_generation ./internal/framework/validators ./internal/cli ./cmd/audita`.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Secret-list assembly is no longer duplicated between proposal and validator paths.
|
||||
- No plaintext configured API key appears in diagnostics, reports, stdout, or stderr in existing redaction tests.
|
||||
- No unrelated adapter behavior changes.
|
||||
|
||||
### Stage 8: Test helper cleanup and dead-code sweep
|
||||
|
||||
Goal:
|
||||
|
||||
- Consolidate test-only duplication and remove dead/redundant code left by prior stages.
|
||||
|
||||
Key edits:
|
||||
|
||||
- Consolidate package-local fake LLM clients, fixture readers, diagnostics glob helpers, and run-directory helpers where duplication is clear.
|
||||
- Use cross-package test support only if it does not obscure test intent or introduce awkward imports.
|
||||
- Remove redundant metadata fields, constants, or helper functions made obsolete by earlier stages.
|
||||
- Keep module-specific prompt and behavior assertions local to each module package.
|
||||
|
||||
Behavior changes:
|
||||
|
||||
- None.
|
||||
|
||||
Tests:
|
||||
|
||||
- Run all package tests touched by helper cleanup.
|
||||
- Run `go test ./internal/modules/... ./internal/framework/... ./internal/cli ./cmd/audita`.
|
||||
- Run `go test ./...` before completing the full sequence.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Test helpers are simpler without reducing coverage.
|
||||
- No parity or release fixture assertions are removed unless replaced by equivalent or stronger assertions.
|
||||
- No production behavior changes.
|
||||
|
||||
## Final verification
|
||||
|
||||
Before declaring the staged cleanup complete:
|
||||
|
||||
- Run:
|
||||
- `go test ./internal/core/config ./internal/core/outputschema`
|
||||
- `go test ./internal/core/diagnostics ./internal/core/reporting`
|
||||
- `go test ./internal/framework/proposal_generation ./internal/framework/validators ./internal/framework/runner`
|
||||
- `go test ./internal/validators/...`
|
||||
- `go test ./internal/modules/...`
|
||||
- `go test ./internal/cli ./cmd/audita`
|
||||
- `go test ./...`
|
||||
- Inspect `git diff` for accidental public CLI, config, report, diagnostics, prompt metadata, stage-name, or output-schema changes.
|
||||
- Update docs only when behavior intentionally changes, especially the intended Stage 2 unknown-module validation change.
|
||||
- Keep commits stage-sized and mention behavior-preservation tests in each commit message or PR description.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- Unknown configured module keys should become config-validation failures before 1.0.
|
||||
- Diagnostics filenames and stage names are public enough to preserve unless a stage explicitly says otherwise.
|
||||
- Each stage should be implemented and reviewed separately.
|
||||
153
docs/troubleshooting.md
Normal file
153
docs/troubleshooting.md
Normal file
@@ -0,0 +1,153 @@
|
||||
# Audita Troubleshooting
|
||||
|
||||
## Scope
|
||||
|
||||
This guide lists recurring implemented failure modes for `audita process` and `audita config`.
|
||||
|
||||
For each entry: symptom, likely cause, inspect, and fix.
|
||||
|
||||
## Config Validation Fails
|
||||
|
||||
Symptom:
|
||||
- `audita config validate --config <path>` exits nonzero.
|
||||
|
||||
Likely causes:
|
||||
- missing `version`;
|
||||
- unsupported config version;
|
||||
- unknown YAML field;
|
||||
- unsupported module key or output schema;
|
||||
- invalid numeric/range/concurrency/retention values.
|
||||
|
||||
Inspect:
|
||||
1. rerun `audita config validate --config <path>` and read stderr.
|
||||
2. if needed, inspect effective config with `audita config print-effective --config <path>`.
|
||||
|
||||
Fix:
|
||||
- set `version: 1`;
|
||||
- remove unknown fields;
|
||||
- use supported module keys and output schemas (`bare-segments`, `audita-v1`);
|
||||
- correct invalid values to satisfy validation constraints.
|
||||
|
||||
## Config File Resolution Errors
|
||||
|
||||
Symptom:
|
||||
- `audita process` fails before processing with config-related errors like `config file not found`.
|
||||
|
||||
Likely causes:
|
||||
- `--config` points to a missing path;
|
||||
- `AUDITA_CONFIG` points to a missing path;
|
||||
- unreadable config path.
|
||||
|
||||
Inspect:
|
||||
1. confirm `--config` or `AUDITA_CONFIG` path exists;
|
||||
2. run `audita config validate --config <path>` directly.
|
||||
|
||||
Fix:
|
||||
- correct the path or unset invalid `AUDITA_CONFIG`;
|
||||
- fix permissions for the config file.
|
||||
|
||||
## Transcript or Glossary Schema Errors
|
||||
|
||||
Symptom:
|
||||
- stderr includes `transcript_schema` or `glossary_schema` and run exits nonzero.
|
||||
|
||||
Likely causes:
|
||||
- transcript is not valid JSON or has invalid segment fields;
|
||||
- glossary is not valid YAML or has missing required glossary entry fields.
|
||||
|
||||
Inspect:
|
||||
1. check stderr for parser/validation details;
|
||||
2. if diagnostics were created, inspect `error.log` and run `report.json` (`error_phase`);
|
||||
3. inspect `source-transcript.json` and `source-transcript-parsed.json` in the run directory.
|
||||
|
||||
Fix:
|
||||
- correct transcript JSON shape/content;
|
||||
- correct glossary YAML shape/content and required entry fields;
|
||||
- rerun validation with known-good tiny examples for comparison:
|
||||
- `examples/tiny-transcript.json`
|
||||
- `examples/tiny-glossary.yaml`
|
||||
|
||||
## LLM Runtime/Backend Failures
|
||||
|
||||
Symptom:
|
||||
- stderr includes `runner_execution` (or backend timeout/error details) and nonzero exit.
|
||||
|
||||
Likely causes:
|
||||
- unreachable/failed LLM endpoint;
|
||||
- timeout/cancellation;
|
||||
- runtime module execution failure.
|
||||
|
||||
Inspect:
|
||||
1. inspect stderr for backend message details;
|
||||
2. inspect run `report.json` (`error_phase`, `module_results`);
|
||||
3. inspect diagnostics payloads and `error.log`.
|
||||
|
||||
Fix:
|
||||
- verify model/base URL/API key settings;
|
||||
- increase timeout if needed;
|
||||
- rerun with `--work-dir-retention always` while debugging.
|
||||
|
||||
## Output File Write Failure
|
||||
|
||||
Symptom:
|
||||
- stderr includes `failed to write output file` and run exits nonzero.
|
||||
|
||||
Likely causes:
|
||||
- output path directory missing;
|
||||
- insufficient filesystem permissions;
|
||||
- invalid output target path.
|
||||
|
||||
Inspect:
|
||||
1. check `--output` target directory exists and is writable;
|
||||
2. inspect run diagnostics `error.log` and report `error_phase`.
|
||||
|
||||
Fix:
|
||||
- write to a valid writable path;
|
||||
- create missing directories;
|
||||
- adjust permissions.
|
||||
|
||||
## Report File Write Failure
|
||||
|
||||
Symptom:
|
||||
- stderr includes `failed to write report JSON file` and run exits nonzero.
|
||||
|
||||
Likely causes:
|
||||
- invalid or unwritable `--report-json` target path.
|
||||
|
||||
Inspect:
|
||||
1. verify parent directory exists and is writable;
|
||||
2. inspect diagnostics `error.log` for `report_write` context.
|
||||
|
||||
Fix:
|
||||
- choose a writable report path;
|
||||
- create missing directories;
|
||||
- rerun.
|
||||
|
||||
## Unsupported Output Schema
|
||||
|
||||
Symptom:
|
||||
- stderr includes `unsupported output schema` and run exits nonzero.
|
||||
|
||||
Likely causes:
|
||||
- unsupported `--output-schema` value;
|
||||
- unsupported `output.schema` in config.
|
||||
|
||||
Inspect:
|
||||
1. check CLI/config schema key;
|
||||
2. run `audita config validate --config <path>` when config is involved.
|
||||
|
||||
Fix:
|
||||
- use `bare-segments` or `audita-v1`.
|
||||
|
||||
## Diagnostics Directory Lookup
|
||||
|
||||
Symptom:
|
||||
- run fails and you need artifacts for debugging.
|
||||
|
||||
Inspect:
|
||||
1. read stderr for `audita process: diagnostics: <run-dir>`;
|
||||
2. open `<run-dir>/report.json` and `<run-dir>/error.log`;
|
||||
3. use diagnostics paths embedded in report metadata for artifact lookup.
|
||||
|
||||
Fix:
|
||||
- rerun with `--work-dir-retention always` to preserve run directories during investigation.
|
||||
6
examples/minimal-config.yml
Normal file
6
examples/minimal-config.yml
Normal file
@@ -0,0 +1,6 @@
|
||||
version: 1
|
||||
output:
|
||||
schema: bare-segments
|
||||
llm:
|
||||
proposal:
|
||||
api_key_env: AUDITA_LLM_API_KEY
|
||||
41
examples/production-config.yml
Normal file
41
examples/production-config.yml
Normal file
@@ -0,0 +1,41 @@
|
||||
version: 1
|
||||
pipeline:
|
||||
modules: [glossary, homophones, glossary, spoken_word, grammar]
|
||||
output:
|
||||
schema: audita-v1
|
||||
llm:
|
||||
proposal:
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
model: openrouter/google/gemma-4-31b-it
|
||||
api_key_env: AUDITA_LLM_API_KEY
|
||||
timeout: 120s
|
||||
max_retries: 3
|
||||
validation:
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
model: openrouter/google/gemma-4-31b-it
|
||||
api_key_env: AUDITA_VALIDATION_LLM_API_KEY
|
||||
timeout: 120
|
||||
max_retries: 3
|
||||
concurrency:
|
||||
total_llm: 2
|
||||
proposal_llm: 2
|
||||
validation_llm: 1
|
||||
chunking:
|
||||
target_sections: 8
|
||||
max_section_tokens: 8192
|
||||
min_section_tokens: 2048
|
||||
normalization:
|
||||
max_segment_gap: 4s
|
||||
ellipsis_gap: 3.5s
|
||||
max_segment_duration: 60s
|
||||
max_segment_tokens: 2048
|
||||
thresholds:
|
||||
glossary: 0.8
|
||||
homophones: 0.8
|
||||
spoken_word: 0.8
|
||||
grammar: 0.8
|
||||
context:
|
||||
description: "General context for domain vocabulary and speaker names."
|
||||
diagnostics:
|
||||
work_dir: /tmp/audita
|
||||
retention: auto
|
||||
6
examples/tiny-glossary.yaml
Normal file
6
examples/tiny-glossary.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
glossary:
|
||||
- name: Audita
|
||||
aliases:
|
||||
- audita
|
||||
category: product
|
||||
summary: The Audita transcript correction CLI.
|
||||
9
examples/tiny-transcript.json
Normal file
9
examples/tiny-transcript.json
Normal file
@@ -0,0 +1,9 @@
|
||||
[
|
||||
{
|
||||
"id": 1,
|
||||
"speaker": "A",
|
||||
"start": 0.0,
|
||||
"end": 1.2,
|
||||
"text": "hello world"
|
||||
}
|
||||
]
|
||||
121
internal/cli/process_flags.go
Normal file
121
internal/cli/process_flags.go
Normal file
@@ -0,0 +1,121 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"flag"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||
)
|
||||
|
||||
type processOverrideBinding func(*config.CLIOverrides, processFlags)
|
||||
|
||||
var processOverrideBindings = map[string]processOverrideBinding{
|
||||
"modules": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ModulesCSV = flags.modules
|
||||
},
|
||||
"output-schema": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.OutputSchema = flags.outputSchema
|
||||
},
|
||||
"llm-api-key": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.PrimaryLLMAPIKey = flags.llmAPIKey
|
||||
},
|
||||
"validation-llm-api-key": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationLLMAPIKey = flags.validationLLMAPIKey
|
||||
},
|
||||
"model": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.PrimaryModel = flags.model
|
||||
},
|
||||
"validation-model": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationModel = flags.validationModel
|
||||
},
|
||||
"base-url": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.PrimaryBaseURL = flags.baseURL
|
||||
},
|
||||
"validation-base-url": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationBaseURL = flags.validationBaseURL
|
||||
},
|
||||
"llm-timeout-seconds": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.PrimaryLLMTimeoutSeconds = flags.llmTimeoutSeconds
|
||||
},
|
||||
"total-llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.TotalLLMConcurrency = flags.totalLLMConcurrency
|
||||
},
|
||||
"proposal-llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ProposalLLMConcurrency = flags.proposalLLMConcurrency
|
||||
},
|
||||
"llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.PrimaryLLMConcurrency = flags.llmConcurrency
|
||||
},
|
||||
"validation-llm-timeout-seconds": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationLLMTimeoutSeconds = flags.validationLLMTimeoutSeconds
|
||||
},
|
||||
"max-retries": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.MaxRetries = flags.maxRetries
|
||||
},
|
||||
"validation-max-retries": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationMaxRetries = flags.validationMaxRetries
|
||||
},
|
||||
"validation-llm-concurrency": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationLLMConcurrency = flags.validationLLMConcurrency
|
||||
},
|
||||
"validation-max-prompt-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.ValidationMaxPromptTokens = flags.validationMaxPromptTokens
|
||||
},
|
||||
"max-section-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.MaxSectionTokens = flags.maxSectionTokens
|
||||
},
|
||||
"min-section-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.MinSectionTokens = flags.minSectionTokens
|
||||
},
|
||||
"target-sections": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.TargetSections = flags.targetSections
|
||||
},
|
||||
"glossary-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.GlossaryConfidenceThreshold = flags.glossaryConfidenceThreshold
|
||||
},
|
||||
"grammar-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.GrammarConfidenceThreshold = flags.grammarConfidenceThreshold
|
||||
},
|
||||
"homophones-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.HomophonesConfidenceThreshold = flags.homophonesConfidenceThreshold
|
||||
},
|
||||
"spoken-word-confidence-threshold": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.SpokenWordConfidenceThreshold = flags.spokenWordConfidenceThreshold
|
||||
},
|
||||
"normalize-max-segment-gap": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.NormalizeMaxSegmentGap = flags.normalizeMaxSegmentGap
|
||||
},
|
||||
"normalize-ellipsis-gap": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.NormalizeEllipsisGap = flags.normalizeEllipsisGap
|
||||
},
|
||||
"normalize-max-segment-duration": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.NormalizeMaxSegmentDuration = flags.normalizeMaxSegmentDuration
|
||||
},
|
||||
"normalize-max-segment-tokens": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.NormalizeMaxSegmentTokens = flags.normalizeMaxSegmentTokens
|
||||
},
|
||||
"transcript-description": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.TranscriptDescription = flags.transcriptDescription
|
||||
},
|
||||
"work-dir": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.WorkDir = flags.workDir
|
||||
},
|
||||
"work-dir-retention": func(overrides *config.CLIOverrides, flags processFlags) {
|
||||
overrides.WorkDirRetention = flags.workDirRetention
|
||||
},
|
||||
}
|
||||
|
||||
func processCLIOverrides(fs *flag.FlagSet, flags processFlags) (config.CLIOverrides, bool) {
|
||||
overrides := config.CLIOverrides{}
|
||||
explicitModules := false
|
||||
fs.Visit(func(f *flag.Flag) {
|
||||
if f.Name == "modules" {
|
||||
explicitModules = true
|
||||
}
|
||||
binding, ok := processOverrideBindings[f.Name]
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
binding(&overrides, flags)
|
||||
})
|
||||
return overrides, explicitModules
|
||||
}
|
||||
433
internal/cli/process_flags_test.go
Normal file
433
internal/cli/process_flags_test.go
Normal file
@@ -0,0 +1,433 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"io"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||
)
|
||||
|
||||
func TestProcessCLIOverridesMapsEveryConfigMutatingFlag(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
flagName string
|
||||
value string
|
||||
wantExplicitModules bool
|
||||
assertOverrideFields func(t *testing.T, overrides config.CLIOverrides)
|
||||
}{
|
||||
{
|
||||
name: "modules",
|
||||
flagName: "modules",
|
||||
value: "grammar,glossary",
|
||||
wantExplicitModules: true,
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "ModulesCSV", overrides.ModulesCSV, "grammar,glossary")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "output schema",
|
||||
flagName: "output-schema",
|
||||
value: "audita-v1",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "OutputSchema", overrides.OutputSchema, "audita-v1")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "primary api key",
|
||||
flagName: "llm-api-key",
|
||||
value: "primary-key",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "PrimaryLLMAPIKey", overrides.PrimaryLLMAPIKey, "primary-key")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation api key",
|
||||
flagName: "validation-llm-api-key",
|
||||
value: "validation-key",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "ValidationLLMAPIKey", overrides.ValidationLLMAPIKey, "validation-key")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "primary model",
|
||||
flagName: "model",
|
||||
value: "primary-model",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "PrimaryModel", overrides.PrimaryModel, "primary-model")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation model",
|
||||
flagName: "validation-model",
|
||||
value: "validation-model",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "ValidationModel", overrides.ValidationModel, "validation-model")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "primary base url",
|
||||
flagName: "base-url",
|
||||
value: "https://primary.example.test",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "PrimaryBaseURL", overrides.PrimaryBaseURL, "https://primary.example.test")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation base url",
|
||||
flagName: "validation-base-url",
|
||||
value: "https://validation.example.test",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "ValidationBaseURL", overrides.ValidationBaseURL, "https://validation.example.test")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "primary timeout",
|
||||
flagName: "llm-timeout-seconds",
|
||||
value: "101",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "PrimaryLLMTimeoutSeconds", overrides.PrimaryLLMTimeoutSeconds, 101)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "total concurrency",
|
||||
flagName: "total-llm-concurrency",
|
||||
value: "5",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "TotalLLMConcurrency", overrides.TotalLLMConcurrency, 5)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "proposal concurrency",
|
||||
flagName: "proposal-llm-concurrency",
|
||||
value: "3",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "ProposalLLMConcurrency", overrides.ProposalLLMConcurrency, 3)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "legacy concurrency alias",
|
||||
flagName: "llm-concurrency",
|
||||
value: "4",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "PrimaryLLMConcurrency", overrides.PrimaryLLMConcurrency, 4)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation timeout",
|
||||
flagName: "validation-llm-timeout-seconds",
|
||||
value: "202",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "ValidationLLMTimeoutSeconds", overrides.ValidationLLMTimeoutSeconds, 202)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "max retries",
|
||||
flagName: "max-retries",
|
||||
value: "6",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "MaxRetries", overrides.MaxRetries, 6)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation max retries",
|
||||
flagName: "validation-max-retries",
|
||||
value: "7",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "ValidationMaxRetries", overrides.ValidationMaxRetries, 7)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation concurrency",
|
||||
flagName: "validation-llm-concurrency",
|
||||
value: "8",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "ValidationLLMConcurrency", overrides.ValidationLLMConcurrency, 8)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "validation max prompt tokens",
|
||||
flagName: "validation-max-prompt-tokens",
|
||||
value: "4096",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "ValidationMaxPromptTokens", overrides.ValidationMaxPromptTokens, 4096)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "max section tokens",
|
||||
flagName: "max-section-tokens",
|
||||
value: "9000",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "MaxSectionTokens", overrides.MaxSectionTokens, 9000)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "min section tokens",
|
||||
flagName: "min-section-tokens",
|
||||
value: "1000",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "MinSectionTokens", overrides.MinSectionTokens, 1000)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "target sections",
|
||||
flagName: "target-sections",
|
||||
value: "12",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "TargetSections", overrides.TargetSections, 12)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "glossary threshold",
|
||||
flagName: "glossary-confidence-threshold",
|
||||
value: "0.91",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "GlossaryConfidenceThreshold", overrides.GlossaryConfidenceThreshold, 0.91)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "grammar threshold",
|
||||
flagName: "grammar-confidence-threshold",
|
||||
value: "0.92",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "GrammarConfidenceThreshold", overrides.GrammarConfidenceThreshold, 0.92)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "homophones threshold",
|
||||
flagName: "homophones-confidence-threshold",
|
||||
value: "0.93",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "HomophonesConfidenceThreshold", overrides.HomophonesConfidenceThreshold, 0.93)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "spoken word threshold",
|
||||
flagName: "spoken-word-confidence-threshold",
|
||||
value: "0.94",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "SpokenWordConfidenceThreshold", overrides.SpokenWordConfidenceThreshold, 0.94)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "normalize max segment gap",
|
||||
flagName: "normalize-max-segment-gap",
|
||||
value: "1.2",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "NormalizeMaxSegmentGap", overrides.NormalizeMaxSegmentGap, 1.2)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "normalize ellipsis gap",
|
||||
flagName: "normalize-ellipsis-gap",
|
||||
value: "2.3",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "NormalizeEllipsisGap", overrides.NormalizeEllipsisGap, 2.3)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "normalize max segment duration",
|
||||
flagName: "normalize-max-segment-duration",
|
||||
value: "45.6",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertFloatOverride(t, "NormalizeMaxSegmentDuration", overrides.NormalizeMaxSegmentDuration, 45.6)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "normalize max segment tokens",
|
||||
flagName: "normalize-max-segment-tokens",
|
||||
value: "321",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertIntOverride(t, "NormalizeMaxSegmentTokens", overrides.NormalizeMaxSegmentTokens, 321)
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "transcript description",
|
||||
flagName: "transcript-description",
|
||||
value: "podcast episode",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "TranscriptDescription", overrides.TranscriptDescription, "podcast episode")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "work dir",
|
||||
flagName: "work-dir",
|
||||
value: "/tmp/custom-audita",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "WorkDir", overrides.WorkDir, "/tmp/custom-audita")
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "work dir retention",
|
||||
flagName: "work-dir-retention",
|
||||
value: "always",
|
||||
assertOverrideFields: func(t *testing.T, overrides config.CLIOverrides) {
|
||||
assertStringOverride(t, "WorkDirRetention", overrides.WorkDirRetention, "always")
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
fs, flags := newProcessFlagSet(config.Default(), io.Discard)
|
||||
if err := fs.Parse([]string{"--" + tc.flagName, tc.value}); err != nil {
|
||||
t.Fatalf("parse flag: %v", err)
|
||||
}
|
||||
|
||||
overrides, explicitModules := processCLIOverrides(fs, flags)
|
||||
if explicitModules != tc.wantExplicitModules {
|
||||
t.Fatalf("explicitModules=%v, want %v", explicitModules, tc.wantExplicitModules)
|
||||
}
|
||||
tc.assertOverrideFields(t, overrides)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessCLIOverridesIgnoresNonConfigFlags(t *testing.T) {
|
||||
fs, flags := newProcessFlagSet(config.Default(), io.Discard)
|
||||
if err := fs.Parse([]string{
|
||||
"--config", "/tmp/config.yml",
|
||||
"--glossary", "/tmp/glossary.yml",
|
||||
"--output", "/tmp/output.json",
|
||||
"--report-json", "/tmp/report.json",
|
||||
}); err != nil {
|
||||
t.Fatalf("parse flags: %v", err)
|
||||
}
|
||||
|
||||
overrides, explicitModules := processCLIOverrides(fs, flags)
|
||||
if explicitModules {
|
||||
t.Fatal("non-config flags should not mark modules explicit")
|
||||
}
|
||||
assertNoCLIOverrides(t, overrides)
|
||||
}
|
||||
|
||||
func TestNewProcessFlagSetDefaultsReflectEffectiveConfig(t *testing.T) {
|
||||
cfg := config.Default()
|
||||
cfg.Modules = []string{"grammar", "glossary"}
|
||||
cfg.OutputSchema = "audita-v1"
|
||||
cfg.PrimaryLLM.APIKey = "primary-key"
|
||||
cfg.ValidationLLM.APIKey = "validation-key"
|
||||
cfg.PrimaryLLM.Model = "primary-model"
|
||||
cfg.ValidationLLM.Model = "validation-model"
|
||||
cfg.PrimaryLLM.BaseURL = "https://primary.example.test"
|
||||
cfg.ValidationLLM.BaseURL = "https://validation.example.test"
|
||||
cfg.PrimaryLLM.TimeoutSeconds = 101
|
||||
cfg.TotalLLMConcurrency = 5
|
||||
cfg.ProposalLLMConcurrency = 3
|
||||
cfg.PrimaryLLM.MaxRetries = 6
|
||||
cfg.ValidationMaxPromptTokens = 4096
|
||||
cfg.MaxSectionTokens = 9000
|
||||
cfg.MinSectionTokens = 1000
|
||||
cfg.Thresholds.Glossary = 0.91
|
||||
cfg.Thresholds.Grammar = 0.92
|
||||
cfg.Thresholds.Homophones = 0.93
|
||||
cfg.Thresholds.SpokenWord = 0.94
|
||||
cfg.Normalization.MaxSegmentGap = 1.2
|
||||
cfg.Normalization.EllipsisGap = 2.3
|
||||
cfg.Normalization.MaxSegmentDuration = 45.6
|
||||
cfg.Normalization.MaxSegmentTokens = 321
|
||||
cfg.TranscriptDescription = "podcast episode"
|
||||
cfg.WorkDir = "/tmp/custom-audita"
|
||||
cfg.WorkDirRetention = config.WorkDirRetentionAlways
|
||||
|
||||
validationTimeout := 202
|
||||
validationRetries := 7
|
||||
validationConcurrency := 8
|
||||
targetSections := 12
|
||||
cfg.ValidationLLM.TimeoutSeconds = &validationTimeout
|
||||
cfg.ValidationLLM.MaxRetries = &validationRetries
|
||||
cfg.ValidationLLMConcurrency = &validationConcurrency
|
||||
cfg.TargetSections = &targetSections
|
||||
|
||||
_, flags := newProcessFlagSet(cfg, io.Discard)
|
||||
|
||||
assertStringOverride(t, "modules default", flags.modules, "grammar,glossary")
|
||||
assertStringOverride(t, "output schema default", flags.outputSchema, "audita-v1")
|
||||
assertStringOverride(t, "primary api key default", flags.llmAPIKey, "primary-key")
|
||||
assertStringOverride(t, "validation api key default", flags.validationLLMAPIKey, "validation-key")
|
||||
assertStringOverride(t, "primary model default", flags.model, "primary-model")
|
||||
assertStringOverride(t, "validation model default", flags.validationModel, "validation-model")
|
||||
assertStringOverride(t, "primary base url default", flags.baseURL, "https://primary.example.test")
|
||||
assertStringOverride(t, "validation base url default", flags.validationBaseURL, "https://validation.example.test")
|
||||
assertIntOverride(t, "primary timeout default", flags.llmTimeoutSeconds, 101)
|
||||
assertIntOverride(t, "total concurrency default", flags.totalLLMConcurrency, 5)
|
||||
assertIntOverride(t, "proposal concurrency default", flags.proposalLLMConcurrency, 3)
|
||||
assertIntOverride(t, "legacy concurrency alias default", flags.llmConcurrency, 5)
|
||||
assertIntOverride(t, "validation timeout default", flags.validationLLMTimeoutSeconds, validationTimeout)
|
||||
assertIntOverride(t, "max retries default", flags.maxRetries, 6)
|
||||
assertIntOverride(t, "validation max retries default", flags.validationMaxRetries, validationRetries)
|
||||
assertIntOverride(t, "validation concurrency default", flags.validationLLMConcurrency, validationConcurrency)
|
||||
assertIntOverride(t, "validation max prompt tokens default", flags.validationMaxPromptTokens, 4096)
|
||||
assertIntOverride(t, "max section tokens default", flags.maxSectionTokens, 9000)
|
||||
assertIntOverride(t, "min section tokens default", flags.minSectionTokens, 1000)
|
||||
assertIntOverride(t, "target sections default", flags.targetSections, targetSections)
|
||||
assertFloatOverride(t, "glossary threshold default", flags.glossaryConfidenceThreshold, 0.91)
|
||||
assertFloatOverride(t, "grammar threshold default", flags.grammarConfidenceThreshold, 0.92)
|
||||
assertFloatOverride(t, "homophones threshold default", flags.homophonesConfidenceThreshold, 0.93)
|
||||
assertFloatOverride(t, "spoken word threshold default", flags.spokenWordConfidenceThreshold, 0.94)
|
||||
assertFloatOverride(t, "normalize max segment gap default", flags.normalizeMaxSegmentGap, 1.2)
|
||||
assertFloatOverride(t, "normalize ellipsis gap default", flags.normalizeEllipsisGap, 2.3)
|
||||
assertFloatOverride(t, "normalize max segment duration default", flags.normalizeMaxSegmentDuration, 45.6)
|
||||
assertIntOverride(t, "normalize max segment tokens default", flags.normalizeMaxSegmentTokens, 321)
|
||||
assertStringOverride(t, "transcript description default", flags.transcriptDescription, "podcast episode")
|
||||
assertStringOverride(t, "work dir default", flags.workDir, "/tmp/custom-audita")
|
||||
assertStringOverride(t, "work dir retention default", flags.workDirRetention, "always")
|
||||
}
|
||||
|
||||
func TestNewProcessFlagSetUsesFallbackDefaultsForUnsetOptionalConfig(t *testing.T) {
|
||||
cfg := config.Default()
|
||||
|
||||
_, flags := newProcessFlagSet(cfg, io.Discard)
|
||||
|
||||
assertIntOverride(t, "validation timeout fallback", flags.validationLLMTimeoutSeconds, cfg.PrimaryLLM.TimeoutSeconds)
|
||||
assertIntOverride(t, "validation retries fallback", flags.validationMaxRetries, cfg.PrimaryLLM.MaxRetries)
|
||||
assertIntOverride(t, "validation concurrency fallback", flags.validationLLMConcurrency, cfg.TotalLLMConcurrency)
|
||||
assertIntOverride(t, "target sections fallback", flags.targetSections, 0)
|
||||
}
|
||||
|
||||
func assertStringOverride(t *testing.T, name string, got *string, want string) {
|
||||
t.Helper()
|
||||
if got == nil || *got != want {
|
||||
t.Fatalf("%s=%v, want %q", name, pointerValue(got), want)
|
||||
}
|
||||
}
|
||||
|
||||
func assertIntOverride(t *testing.T, name string, got *int, want int) {
|
||||
t.Helper()
|
||||
if got == nil || *got != want {
|
||||
t.Fatalf("%s=%v, want %d", name, pointerValue(got), want)
|
||||
}
|
||||
}
|
||||
|
||||
func assertFloatOverride(t *testing.T, name string, got *float64, want float64) {
|
||||
t.Helper()
|
||||
if got == nil || *got != want {
|
||||
t.Fatalf("%s=%v, want %v", name, pointerValue(got), want)
|
||||
}
|
||||
}
|
||||
|
||||
func assertNoCLIOverrides(t *testing.T, overrides config.CLIOverrides) {
|
||||
t.Helper()
|
||||
value := reflect.ValueOf(overrides)
|
||||
typ := value.Type()
|
||||
for i := 0; i < value.NumField(); i++ {
|
||||
field := value.Field(i)
|
||||
if field.Kind() != reflect.Ptr {
|
||||
t.Fatalf("unexpected non-pointer CLIOverrides field %s", typ.Field(i).Name)
|
||||
}
|
||||
if !field.IsNil() {
|
||||
t.Fatalf("expected no CLI overrides, field %s was set", typ.Field(i).Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func pointerValue[T any](ptr *T) any {
|
||||
if ptr == nil {
|
||||
return "<nil>"
|
||||
}
|
||||
if stringer, ok := any(*ptr).(interface{ String() string }); ok {
|
||||
return strings.TrimSpace(stringer.String())
|
||||
}
|
||||
return *ptr
|
||||
}
|
||||
@@ -1,46 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/runner"
|
||||
)
|
||||
|
||||
func TestBuildCorrectionLedgerClassifiesValidatorDecisionsFromCanonicalMetadata(t *testing.T) {
|
||||
output := &runner.RunOutput{
|
||||
ModuleResults: []runner.ModuleResult{
|
||||
{
|
||||
ModuleKey: "glossary",
|
||||
ModuleInstance: "glossary",
|
||||
ReplacementPolicy: proposals.ReplacementPolicyReplaceAll,
|
||||
ValidatorDecisions: []runner.ValidatorDecisionRecord{
|
||||
{ValidatorName: "proposal_shape", ProposalIndex: 3, Approved: true, ReasonCode: "approved"},
|
||||
{ValidatorName: "spoken_form_plausibility", ProposalIndex: 3, Approved: true, ReasonCode: "approved"},
|
||||
},
|
||||
AppliedChanges: []proposals.AppliedChange{
|
||||
{
|
||||
ProposalIndex: 3,
|
||||
ModuleKey: "glossary",
|
||||
ModuleInstance: "glossary",
|
||||
TargetSegmentID: 1,
|
||||
OriginalText: "gestures",
|
||||
CorrectedText: "Jesters",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
ledger := buildCorrectionLedger("/tmp/audita-run-id", output)
|
||||
if len(ledger) != 1 {
|
||||
t.Fatalf("expected one ledger entry, got %d", len(ledger))
|
||||
}
|
||||
entry := ledger[0]
|
||||
if len(entry.DeterministicValidatorResults) != 1 || entry.DeterministicValidatorResults[0].ValidatorKey != "proposal_shape" {
|
||||
t.Fatalf("unexpected deterministic decision split: %+v", entry.DeterministicValidatorResults)
|
||||
}
|
||||
if len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].ValidatorKey != "spoken_form_plausibility" {
|
||||
t.Fatalf("unexpected llm-backed decision split: %+v", entry.LLMValidatorResults)
|
||||
}
|
||||
}
|
||||
@@ -23,10 +23,10 @@ import (
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/modules"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/processreport"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/runner"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||
stagewarnings "gitea.maximumdirect.net/eric/audita/internal/framework/warnings"
|
||||
)
|
||||
|
||||
type noOpStructuredLLMClient struct{}
|
||||
@@ -420,75 +420,7 @@ func runProcess(args []string, stdout, stderr io.Writer) int {
|
||||
return 2
|
||||
}
|
||||
|
||||
overrides := config.CLIOverrides{}
|
||||
explicitModules := false
|
||||
fs.Visit(func(f *flag.Flag) {
|
||||
switch f.Name {
|
||||
case "modules":
|
||||
explicitModules = true
|
||||
overrides.ModulesCSV = pFlags.modules
|
||||
case "output-schema":
|
||||
overrides.OutputSchema = pFlags.outputSchema
|
||||
case "llm-api-key":
|
||||
overrides.PrimaryLLMAPIKey = pFlags.llmAPIKey
|
||||
case "validation-llm-api-key":
|
||||
overrides.ValidationLLMAPIKey = pFlags.validationLLMAPIKey
|
||||
case "model":
|
||||
overrides.PrimaryModel = pFlags.model
|
||||
case "validation-model":
|
||||
overrides.ValidationModel = pFlags.validationModel
|
||||
case "base-url":
|
||||
overrides.PrimaryBaseURL = pFlags.baseURL
|
||||
case "validation-base-url":
|
||||
overrides.ValidationBaseURL = pFlags.validationBaseURL
|
||||
case "llm-timeout-seconds":
|
||||
overrides.PrimaryLLMTimeoutSeconds = pFlags.llmTimeoutSeconds
|
||||
case "total-llm-concurrency":
|
||||
overrides.TotalLLMConcurrency = pFlags.totalLLMConcurrency
|
||||
case "proposal-llm-concurrency":
|
||||
overrides.ProposalLLMConcurrency = pFlags.proposalLLMConcurrency
|
||||
case "llm-concurrency":
|
||||
overrides.PrimaryLLMConcurrency = pFlags.llmConcurrency
|
||||
case "validation-llm-timeout-seconds":
|
||||
overrides.ValidationLLMTimeoutSeconds = pFlags.validationLLMTimeoutSeconds
|
||||
case "max-retries":
|
||||
overrides.MaxRetries = pFlags.maxRetries
|
||||
case "validation-max-retries":
|
||||
overrides.ValidationMaxRetries = pFlags.validationMaxRetries
|
||||
case "validation-llm-concurrency":
|
||||
overrides.ValidationLLMConcurrency = pFlags.validationLLMConcurrency
|
||||
case "validation-max-prompt-tokens":
|
||||
overrides.ValidationMaxPromptTokens = pFlags.validationMaxPromptTokens
|
||||
case "max-section-tokens":
|
||||
overrides.MaxSectionTokens = pFlags.maxSectionTokens
|
||||
case "min-section-tokens":
|
||||
overrides.MinSectionTokens = pFlags.minSectionTokens
|
||||
case "target-sections":
|
||||
overrides.TargetSections = pFlags.targetSections
|
||||
case "glossary-confidence-threshold":
|
||||
overrides.GlossaryConfidenceThreshold = pFlags.glossaryConfidenceThreshold
|
||||
case "grammar-confidence-threshold":
|
||||
overrides.GrammarConfidenceThreshold = pFlags.grammarConfidenceThreshold
|
||||
case "homophones-confidence-threshold":
|
||||
overrides.HomophonesConfidenceThreshold = pFlags.homophonesConfidenceThreshold
|
||||
case "spoken-word-confidence-threshold":
|
||||
overrides.SpokenWordConfidenceThreshold = pFlags.spokenWordConfidenceThreshold
|
||||
case "normalize-max-segment-gap":
|
||||
overrides.NormalizeMaxSegmentGap = pFlags.normalizeMaxSegmentGap
|
||||
case "normalize-ellipsis-gap":
|
||||
overrides.NormalizeEllipsisGap = pFlags.normalizeEllipsisGap
|
||||
case "normalize-max-segment-duration":
|
||||
overrides.NormalizeMaxSegmentDuration = pFlags.normalizeMaxSegmentDuration
|
||||
case "normalize-max-segment-tokens":
|
||||
overrides.NormalizeMaxSegmentTokens = pFlags.normalizeMaxSegmentTokens
|
||||
case "transcript-description":
|
||||
overrides.TranscriptDescription = pFlags.transcriptDescription
|
||||
case "work-dir":
|
||||
overrides.WorkDir = pFlags.workDir
|
||||
case "work-dir-retention":
|
||||
overrides.WorkDirRetention = pFlags.workDirRetention
|
||||
}
|
||||
})
|
||||
overrides, explicitModules := processCLIOverrides(fs, pFlags)
|
||||
|
||||
if err := cfg.ApplyCLIOverrides(overrides); err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: invalid CLI configuration: %v\n", err)
|
||||
@@ -531,10 +463,13 @@ func runProcess(args []string, stdout, stderr io.Writer) int {
|
||||
if runOutput.Utilization != nil {
|
||||
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactUtilizationSummary, runOutput.Utilization)
|
||||
}
|
||||
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactCorrectionLedger, buildCorrectionLedger(runDir.Path(), runOutput))
|
||||
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactCorrectionLedger, processreport.BuildCorrectionLedger(processreport.CorrectionLedgerInput{
|
||||
RunDirectoryPath: runDir.Path(),
|
||||
RunOutput: runOutput,
|
||||
}))
|
||||
}
|
||||
errorPhase, errorMessage := extractErrorPhase(runErr)
|
||||
report := buildProcessReport("failed", inv, runDir, startedAt, completedAt, errorMessage, errorPhase, nil, nil, runOutput)
|
||||
report := processreport.Build(processReportInput("failed", inv, runDir, startedAt, completedAt, errorMessage, errorPhase, nil, nil, runOutput))
|
||||
|
||||
if strings.TrimSpace(inv.ReportJSONPath) != "" {
|
||||
if err := reporting.WriteProcessReport(inv.ReportJSONPath, report); err != nil {
|
||||
@@ -560,10 +495,13 @@ func runProcess(args []string, stdout, stderr io.Writer) int {
|
||||
if runOutput.Utilization != nil {
|
||||
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactUtilizationSummary, runOutput.Utilization)
|
||||
}
|
||||
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactCorrectionLedger, buildCorrectionLedger(runDir.Path(), runOutput))
|
||||
_ = runDir.WriteJSONArtifact(diagnostics.ArtifactCorrectionLedger, processreport.BuildCorrectionLedger(processreport.CorrectionLedgerInput{
|
||||
RunDirectoryPath: runDir.Path(),
|
||||
RunOutput: runOutput,
|
||||
}))
|
||||
}
|
||||
|
||||
report := buildProcessReport("success", inv, runDir, startedAt, completedAt, "", "", normSummary, chunkSummary, runOutput)
|
||||
report := processreport.Build(processReportInput("success", inv, runDir, startedAt, completedAt, "", "", normSummary, chunkSummary, runOutput))
|
||||
|
||||
if strings.TrimSpace(inv.ReportJSONPath) != "" {
|
||||
if err := reporting.WriteProcessReport(inv.ReportJSONPath, report); err != nil {
|
||||
@@ -578,21 +516,11 @@ func runProcess(args []string, stdout, stderr io.Writer) int {
|
||||
}
|
||||
}
|
||||
|
||||
hasSkippedCorrections := false
|
||||
if runOutput != nil {
|
||||
for _, mr := range runOutput.ModuleResults {
|
||||
if len(mr.SkippedChanges) > 0 || len(mr.ValidatorRejected) > 0 {
|
||||
hasSkippedCorrections = true
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if runDir != nil {
|
||||
_ = runDir.WriteReport(report)
|
||||
if err := runDir.ApplyRetention(diagnostics.RetentionDecisionInput{
|
||||
RunSucceeded: true,
|
||||
HasSkippedCorrections: hasSkippedCorrections,
|
||||
HasSkippedCorrections: processreport.HasSkippedCorrections(runOutput),
|
||||
}); err != nil {
|
||||
fmt.Fprintf(stderr, "audita process: failed to apply work-dir retention: %v\n", err)
|
||||
return 1
|
||||
@@ -710,130 +638,28 @@ func extractErrorPhase(err error) (phase string, message string) {
|
||||
return "", msg
|
||||
}
|
||||
|
||||
func buildProcessReport(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) reporting.ProcessReport {
|
||||
report := reporting.ProcessReport{
|
||||
ReportMetadata: reporting.ReportMetadata{
|
||||
ReportSchemaName: reporting.DefaultProcessReportSchemaName,
|
||||
ReportSchemaVersion: reporting.DefaultProcessReportSchemaVersion,
|
||||
OutputSchema: inv.Config.OutputSchema,
|
||||
ConfigVersion: inv.ConfigVersion,
|
||||
},
|
||||
Phase: "default_pipeline",
|
||||
Status: status,
|
||||
Operation: "process",
|
||||
TranscriptPath: inv.TranscriptPath,
|
||||
GlossaryPath: inv.GlossaryPath,
|
||||
OutputPath: inv.OutputPath,
|
||||
Modules: append([]string(nil), inv.Config.Modules...),
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: &completedAt,
|
||||
ErrorPhase: errorPhase,
|
||||
}
|
||||
func processReportInput(status string, inv processInvocation, runDir *diagnostics.RunDirectory, startedAt, completedAt time.Time, errorMessage string, errorPhase string, normalizationSummary *normalization.NormalizationSummary, chunkingSummary *chunking.Summary, runOutput *runner.RunOutput) processreport.BuildInput {
|
||||
runDirectoryPath := ""
|
||||
if runDir != nil {
|
||||
runSucceeded := status == "success"
|
||||
metadata := diagnostics.BuildDiagnosticsMetadata(runDir.Path(), runSucceeded)
|
||||
report.Diagnostics = &metadata
|
||||
runDirectoryPath = runDir.Path()
|
||||
}
|
||||
if errorMessage != "" {
|
||||
report.ErrorMessage = errorMessage
|
||||
return processreport.BuildInput{
|
||||
Status: status,
|
||||
TranscriptPath: inv.TranscriptPath,
|
||||
GlossaryPath: inv.GlossaryPath,
|
||||
OutputPath: inv.OutputPath,
|
||||
Modules: inv.Config.Modules,
|
||||
OutputSchema: inv.Config.OutputSchema,
|
||||
ConfigVersion: inv.ConfigVersion,
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: completedAt,
|
||||
ErrorMessage: errorMessage,
|
||||
ErrorPhase: errorPhase,
|
||||
RunDirectoryPath: runDirectoryPath,
|
||||
NormalizationSummary: normalizationSummary,
|
||||
ChunkingSummary: chunkingSummary,
|
||||
RunOutput: runOutput,
|
||||
}
|
||||
if normalizationSummary != nil {
|
||||
report.InputSegmentCount = &normalizationSummary.InputSegmentCount
|
||||
report.NormalizedSegmentCount = &normalizationSummary.OutputSegmentCount
|
||||
report.NormalizationMerges = &normalizationSummary.MergesPerformed
|
||||
report.NormalizationIDReassignments = &normalizationSummary.IDsReassigned
|
||||
report.NormalizationSkipped.DifferentSpeakers = &normalizationSummary.SkippedMerges.DifferentSpeakers
|
||||
report.NormalizationSkipped.GapTooLarge = &normalizationSummary.SkippedMerges.GapTooLarge
|
||||
report.NormalizationSkipped.DurationExceeded = &normalizationSummary.SkippedMerges.DurationExceeded
|
||||
report.NormalizationSkipped.TokenLimitExceeded = &normalizationSummary.SkippedMerges.TokenLimitExceeded
|
||||
}
|
||||
if chunkingSummary != nil {
|
||||
report.Chunking = &reporting.ChunkingSummary{
|
||||
ChunkCount: chunkingSummary.ChunkCount,
|
||||
MinEstimatedTokens: chunkingSummary.MinEstimatedTokens,
|
||||
MaxEstimatedTokens: chunkingSummary.MaxEstimatedTokens,
|
||||
TotalEstimatedTokens: chunkingSummary.TotalEstimatedTokens,
|
||||
TargetSections: chunkingSummary.TargetSections,
|
||||
MaxSectionTokens: chunkingSummary.MaxSectionTokens,
|
||||
MinSectionTokens: chunkingSummary.MinSectionTokens,
|
||||
}
|
||||
}
|
||||
report.ModulesSummary, report.ModuleResults = buildModuleReporting(runOutput)
|
||||
return report
|
||||
}
|
||||
|
||||
func buildModuleReporting(runOutput *runner.RunOutput) (*reporting.ModulesSummary, []reporting.ModuleReport) {
|
||||
if runOutput == nil || len(runOutput.ModuleResults) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
moduleReports := make([]reporting.ModuleReport, 0, len(runOutput.ModuleResults))
|
||||
summary := &reporting.ModulesSummary{ModuleCount: len(runOutput.ModuleResults)}
|
||||
for _, r := range runOutput.ModuleResults {
|
||||
startedAt := r.StartedAt
|
||||
completedAt := r.CompletedAt
|
||||
moduleReports = append(moduleReports, reporting.ModuleReport{
|
||||
ModuleKey: r.ModuleKey,
|
||||
ModuleInstance: r.ModuleInstance,
|
||||
ReplacementPolicy: string(r.ReplacementPolicy),
|
||||
Status: r.Status,
|
||||
ProposalCount: r.ProposalCount,
|
||||
Warnings: append([]stagewarnings.StageWarning(nil), r.Warnings...),
|
||||
ValidatorDecisions: mapValidatorDecisions(r.ValidatorDecisions),
|
||||
ValidatorRejected: mapValidatorRejected(r.ValidatorRejected),
|
||||
AppliedChanges: r.AppliedChanges,
|
||||
SkippedChanges: r.SkippedChanges,
|
||||
ErrorMessage: r.ErrorMessage,
|
||||
StartedAt: &startedAt,
|
||||
CompletedAt: &completedAt,
|
||||
})
|
||||
summary.TotalAppliedChanges += len(r.AppliedChanges)
|
||||
summary.TotalSkippedChanges += len(r.SkippedChanges) + len(r.ValidatorRejected)
|
||||
if r.Status == runner.ModuleStatusFailed && summary.FailedModuleInstance == "" {
|
||||
summary.FailedModuleInstance = r.ModuleInstance
|
||||
}
|
||||
}
|
||||
|
||||
return summary, moduleReports
|
||||
}
|
||||
|
||||
func mapValidatorDecisions(in []runner.ValidatorDecisionRecord) []reporting.ValidatorDecisionReport {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]reporting.ValidatorDecisionReport, len(in))
|
||||
for i, d := range in {
|
||||
out[i] = reporting.ValidatorDecisionReport{
|
||||
ValidatorName: d.ValidatorName,
|
||||
ProposalIndex: d.ProposalIndex,
|
||||
Approved: d.Approved,
|
||||
ReasonCode: d.ReasonCode,
|
||||
Message: d.Message,
|
||||
DiagnosticArtifactPath: d.DiagnosticArtifactPath,
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func mapValidatorRejected(in []runner.ValidatorRejectedChange) []reporting.ValidatorRejectedReport {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]reporting.ValidatorRejectedReport, len(in))
|
||||
for i, d := range in {
|
||||
out[i] = reporting.ValidatorRejectedReport{
|
||||
ValidatorName: d.ValidatorName,
|
||||
ProposalIndex: d.ProposalIndex,
|
||||
ModuleKey: d.ModuleKey,
|
||||
ModuleInstance: d.ModuleInstance,
|
||||
TargetSegmentID: d.TargetSegmentID,
|
||||
OriginalText: d.OriginalText,
|
||||
CorrectedText: d.CorrectedText,
|
||||
ReasonCode: d.ReasonCode,
|
||||
Message: d.Message,
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type processFlags struct {
|
||||
|
||||
171
internal/core/config/apply_helpers.go
Normal file
171
internal/core/config/apply_helpers.go
Normal file
@@ -0,0 +1,171 @@
|
||||
package config
|
||||
|
||||
import "strings"
|
||||
|
||||
type llmTargetPatch struct {
|
||||
apiKey *string
|
||||
model *string
|
||||
baseURL *string
|
||||
timeoutSeconds *int
|
||||
maxRetries *int
|
||||
}
|
||||
|
||||
type concurrencyPatch struct {
|
||||
totalLLM *int
|
||||
legacyTotalLLM *int
|
||||
proposalLLM *int
|
||||
validationLLM *int
|
||||
inheritProposal bool
|
||||
allowLegacyAlias bool
|
||||
}
|
||||
|
||||
type chunkingPatch struct {
|
||||
targetSections *int
|
||||
maxSectionTokens *int
|
||||
minSectionTokens *int
|
||||
}
|
||||
|
||||
type thresholdsPatch struct {
|
||||
glossary *float64
|
||||
grammar *float64
|
||||
homophones *float64
|
||||
spokenWord *float64
|
||||
}
|
||||
|
||||
type normalizationPatch struct {
|
||||
maxSegmentGap *float64
|
||||
ellipsisGap *float64
|
||||
maxSegmentDuration *float64
|
||||
maxSegmentTokens *int
|
||||
}
|
||||
|
||||
type contextPatch struct {
|
||||
transcriptDescription *string
|
||||
}
|
||||
|
||||
type diagnosticsPatch struct {
|
||||
workDir *string
|
||||
workDirRetention *string
|
||||
}
|
||||
|
||||
func (c *Config) applyPrimaryLLMTargetPatch(patch llmTargetPatch) {
|
||||
if patch.apiKey != nil {
|
||||
c.PrimaryLLM.APIKey = *patch.apiKey
|
||||
}
|
||||
if patch.model != nil {
|
||||
c.PrimaryLLM.Model = *patch.model
|
||||
}
|
||||
if patch.baseURL != nil {
|
||||
c.PrimaryLLM.BaseURL = *patch.baseURL
|
||||
}
|
||||
if patch.timeoutSeconds != nil {
|
||||
c.PrimaryLLM.TimeoutSeconds = *patch.timeoutSeconds
|
||||
}
|
||||
if patch.maxRetries != nil {
|
||||
c.PrimaryLLM.MaxRetries = *patch.maxRetries
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Config) applyValidationLLMTargetPatch(patch llmTargetPatch) {
|
||||
if patch.apiKey != nil {
|
||||
c.ValidationLLM.APIKey = *patch.apiKey
|
||||
}
|
||||
if patch.model != nil {
|
||||
c.ValidationLLM.Model = *patch.model
|
||||
}
|
||||
if patch.baseURL != nil {
|
||||
c.ValidationLLM.BaseURL = *patch.baseURL
|
||||
}
|
||||
if patch.timeoutSeconds != nil {
|
||||
value := *patch.timeoutSeconds
|
||||
c.ValidationLLM.TimeoutSeconds = &value
|
||||
}
|
||||
if patch.maxRetries != nil {
|
||||
value := *patch.maxRetries
|
||||
c.ValidationLLM.MaxRetries = &value
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Config) applyConcurrencyPatch(patch concurrencyPatch) {
|
||||
totalSet := false
|
||||
if patch.totalLLM != nil {
|
||||
c.TotalLLMConcurrency = *patch.totalLLM
|
||||
totalSet = true
|
||||
}
|
||||
if patch.allowLegacyAlias && patch.legacyTotalLLM != nil && !totalSet {
|
||||
c.TotalLLMConcurrency = *patch.legacyTotalLLM
|
||||
totalSet = true
|
||||
}
|
||||
|
||||
proposalSet := false
|
||||
if patch.proposalLLM != nil {
|
||||
c.ProposalLLMConcurrency = *patch.proposalLLM
|
||||
proposalSet = true
|
||||
}
|
||||
if patch.inheritProposal && totalSet && !proposalSet {
|
||||
c.ProposalLLMConcurrency = c.TotalLLMConcurrency
|
||||
}
|
||||
|
||||
if patch.validationLLM != nil {
|
||||
value := *patch.validationLLM
|
||||
c.ValidationLLMConcurrency = &value
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Config) applyChunkingPatch(patch chunkingPatch) {
|
||||
if patch.targetSections != nil {
|
||||
value := *patch.targetSections
|
||||
c.TargetSections = &value
|
||||
}
|
||||
if patch.maxSectionTokens != nil {
|
||||
c.MaxSectionTokens = *patch.maxSectionTokens
|
||||
}
|
||||
if patch.minSectionTokens != nil {
|
||||
c.MinSectionTokens = *patch.minSectionTokens
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Config) applyThresholdsPatch(patch thresholdsPatch) {
|
||||
if patch.glossary != nil {
|
||||
c.Thresholds.Glossary = *patch.glossary
|
||||
}
|
||||
if patch.grammar != nil {
|
||||
c.Thresholds.Grammar = *patch.grammar
|
||||
}
|
||||
if patch.homophones != nil {
|
||||
c.Thresholds.Homophones = *patch.homophones
|
||||
}
|
||||
if patch.spokenWord != nil {
|
||||
c.Thresholds.SpokenWord = *patch.spokenWord
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Config) applyNormalizationPatch(patch normalizationPatch) {
|
||||
if patch.maxSegmentGap != nil {
|
||||
c.Normalization.MaxSegmentGap = *patch.maxSegmentGap
|
||||
}
|
||||
if patch.ellipsisGap != nil {
|
||||
c.Normalization.EllipsisGap = *patch.ellipsisGap
|
||||
}
|
||||
if patch.maxSegmentDuration != nil {
|
||||
c.Normalization.MaxSegmentDuration = *patch.maxSegmentDuration
|
||||
}
|
||||
if patch.maxSegmentTokens != nil {
|
||||
c.Normalization.MaxSegmentTokens = *patch.maxSegmentTokens
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Config) applyContextPatch(patch contextPatch) {
|
||||
if patch.transcriptDescription != nil {
|
||||
c.TranscriptDescription = strings.TrimSpace(*patch.transcriptDescription)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Config) applyDiagnosticsPatch(patch diagnosticsPatch) {
|
||||
if patch.workDir != nil {
|
||||
c.WorkDir = *patch.workDir
|
||||
}
|
||||
if patch.workDirRetention != nil {
|
||||
c.WorkDirRetention = WorkDirRetention(*patch.workDirRetention)
|
||||
}
|
||||
}
|
||||
@@ -239,6 +239,186 @@ func TestApplyCLIOverridesTrimsTranscriptDescription(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigSourcesApplySharedEffectiveFieldsConsistently(t *testing.T) {
|
||||
fileCfg := mustParseFileConfigYAML(t, `
|
||||
version: 1
|
||||
output:
|
||||
schema: " audita-v1 "
|
||||
llm:
|
||||
proposal:
|
||||
base_url: https://proposal.example.test/v1
|
||||
model: provider/proposal
|
||||
timeout: 101
|
||||
max_retries: 5
|
||||
validation:
|
||||
base_url: https://validation.example.test/v1
|
||||
model: provider/validation
|
||||
timeout: 202
|
||||
max_retries: 6
|
||||
chunking:
|
||||
target_sections: 7
|
||||
max_section_tokens: 9000
|
||||
min_section_tokens: 1000
|
||||
thresholds:
|
||||
glossary: 0.91
|
||||
grammar: 0.92
|
||||
homophones: 0.93
|
||||
spoken_word: 0.94
|
||||
normalization:
|
||||
max_segment_gap: 1.2
|
||||
ellipsis_gap: 2.3
|
||||
max_segment_duration: 45.6
|
||||
max_segment_tokens: 321
|
||||
context:
|
||||
description: " shared context "
|
||||
diagnostics:
|
||||
work_dir: /tmp/audita-shared
|
||||
retention: always
|
||||
`)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
apply func(*Config) error
|
||||
}{
|
||||
{
|
||||
name: "file",
|
||||
apply: func(cfg *Config) error {
|
||||
return cfg.applyFileConfigWithLookup(fileCfg, mapLookup(map[string]string{}))
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "env",
|
||||
apply: func(cfg *Config) error {
|
||||
return cfg.applyEnvOverrides(mapLookup(map[string]string{
|
||||
"AUDITA_MODEL": "provider/proposal",
|
||||
"AUDITA_BASE_URL": "https://proposal.example.test/v1",
|
||||
"AUDITA_LLM_TIMEOUT_SECONDS": "101",
|
||||
"AUDITA_MAX_RETRIES": "5",
|
||||
"AUDITA_VALIDATION_MODEL": "provider/validation",
|
||||
"AUDITA_VALIDATION_BASE_URL": "https://validation.example.test/v1",
|
||||
"AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS": "202",
|
||||
"AUDITA_VALIDATION_MAX_RETRIES": "6",
|
||||
"AUDITA_TARGET_SECTIONS": "7",
|
||||
"AUDITA_MAX_SECTION_TOKENS": "9000",
|
||||
"AUDITA_MIN_SECTION_TOKENS": "1000",
|
||||
"AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD": "0.91",
|
||||
"AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD": "0.92",
|
||||
"AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD": "0.93",
|
||||
"AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD": "0.94",
|
||||
"AUDITA_NORMALIZE_MAX_SEGMENT_GAP": "1.2",
|
||||
"AUDITA_NORMALIZE_ELLIPSIS_GAP": "2.3",
|
||||
"AUDITA_NORMALIZE_MAX_SEGMENT_DURATION": "45.6",
|
||||
"AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS": "321",
|
||||
"AUDITA_WORK_DIR": "/tmp/audita-shared",
|
||||
"AUDITA_WORK_DIR_RETENTION": "always",
|
||||
}))
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "cli",
|
||||
apply: func(cfg *Config) error {
|
||||
outputSchema := " audita-v1 "
|
||||
proposalModel := "provider/proposal"
|
||||
proposalBaseURL := "https://proposal.example.test/v1"
|
||||
proposalTimeout := 101
|
||||
proposalMaxRetries := 5
|
||||
validationModel := "provider/validation"
|
||||
validationBaseURL := "https://validation.example.test/v1"
|
||||
validationTimeout := 202
|
||||
validationMaxRetries := 6
|
||||
targetSections := 7
|
||||
maxSectionTokens := 9000
|
||||
minSectionTokens := 1000
|
||||
glossaryThreshold := 0.91
|
||||
grammarThreshold := 0.92
|
||||
homophonesThreshold := 0.93
|
||||
spokenWordThreshold := 0.94
|
||||
normalizeMaxSegmentGap := 1.2
|
||||
normalizeEllipsisGap := 2.3
|
||||
normalizeMaxSegmentDuration := 45.6
|
||||
normalizeMaxSegmentTokens := 321
|
||||
description := " shared context "
|
||||
workDir := "/tmp/audita-shared"
|
||||
workDirRetention := "always"
|
||||
return cfg.ApplyCLIOverrides(CLIOverrides{
|
||||
OutputSchema: &outputSchema,
|
||||
PrimaryModel: &proposalModel,
|
||||
PrimaryBaseURL: &proposalBaseURL,
|
||||
PrimaryLLMTimeoutSeconds: &proposalTimeout,
|
||||
MaxRetries: &proposalMaxRetries,
|
||||
ValidationModel: &validationModel,
|
||||
ValidationBaseURL: &validationBaseURL,
|
||||
ValidationLLMTimeoutSeconds: &validationTimeout,
|
||||
ValidationMaxRetries: &validationMaxRetries,
|
||||
TargetSections: &targetSections,
|
||||
MaxSectionTokens: &maxSectionTokens,
|
||||
MinSectionTokens: &minSectionTokens,
|
||||
GlossaryConfidenceThreshold: &glossaryThreshold,
|
||||
GrammarConfidenceThreshold: &grammarThreshold,
|
||||
HomophonesConfidenceThreshold: &homophonesThreshold,
|
||||
SpokenWordConfidenceThreshold: &spokenWordThreshold,
|
||||
NormalizeMaxSegmentGap: &normalizeMaxSegmentGap,
|
||||
NormalizeEllipsisGap: &normalizeEllipsisGap,
|
||||
NormalizeMaxSegmentDuration: &normalizeMaxSegmentDuration,
|
||||
NormalizeMaxSegmentTokens: &normalizeMaxSegmentTokens,
|
||||
TranscriptDescription: &description,
|
||||
WorkDir: &workDir,
|
||||
WorkDirRetention: &workDirRetention,
|
||||
})
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
cfg := Default()
|
||||
if err := tc.apply(&cfg); err != nil {
|
||||
t.Fatalf("apply config source: %v", err)
|
||||
}
|
||||
assertSharedEffectiveFields(t, cfg, sharedEffectiveFieldOptions{
|
||||
wantOutputSchemaOverride: tc.name != "env",
|
||||
wantTranscriptDescriptionPatch: tc.name != "env",
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCLIAPIKeyOverrideIsDirectValue(t *testing.T) {
|
||||
cfg := Default()
|
||||
apiKey := "NOT_AN_ENV_VAR_NAME"
|
||||
validationAPIKey := "also direct"
|
||||
|
||||
if err := cfg.ApplyCLIOverrides(CLIOverrides{PrimaryLLMAPIKey: &apiKey, ValidationLLMAPIKey: &validationAPIKey}); err != nil {
|
||||
t.Fatalf("ApplyCLIOverrides failed: %v", err)
|
||||
}
|
||||
if cfg.PrimaryLLM.APIKey != apiKey {
|
||||
t.Fatalf("expected direct primary api key, got %q", cfg.PrimaryLLM.APIKey)
|
||||
}
|
||||
if cfg.ValidationLLM.APIKey != validationAPIKey {
|
||||
t.Fatalf("expected direct validation api key, got %q", cfg.ValidationLLM.APIKey)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyFileConfigTotalConcurrencyDoesNotChangeProposalWhenProposalUnset(t *testing.T) {
|
||||
fileCfg := mustParseFileConfigYAML(t, `
|
||||
version: 1
|
||||
concurrency:
|
||||
total_llm: 4
|
||||
`)
|
||||
cfg := Default()
|
||||
|
||||
if err := cfg.applyFileConfigWithLookup(fileCfg, mapLookup(map[string]string{})); err != nil {
|
||||
t.Fatalf("applyFileConfigWithLookup failed: %v", err)
|
||||
}
|
||||
|
||||
if cfg.TotalLLMConcurrency != 4 {
|
||||
t.Fatalf("expected file total concurrency 4, got %d", cfg.TotalLLMConcurrency)
|
||||
}
|
||||
if cfg.ProposalLLMConcurrency != DefaultLLMConcurrency {
|
||||
t.Fatalf("expected file config to preserve proposal concurrency when unset, got %d", cfg.ProposalLLMConcurrency)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidationRejectsOverlyLongTranscriptDescription(t *testing.T) {
|
||||
cfg := Default()
|
||||
cfg.TranscriptDescription = strings.Repeat("a", DefaultTranscriptDescriptionMaxChars+1)
|
||||
@@ -456,3 +636,70 @@ func mapLookup(values map[string]string) func(string) (string, bool) {
|
||||
return value, ok
|
||||
}
|
||||
}
|
||||
|
||||
func mustParseFileConfigYAML(t *testing.T, raw string) FileConfig {
|
||||
t.Helper()
|
||||
fileCfg, err := ParseFileConfigYAML([]byte(raw))
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFileConfigYAML failed: %v", err)
|
||||
}
|
||||
return fileCfg
|
||||
}
|
||||
|
||||
type sharedEffectiveFieldOptions struct {
|
||||
wantOutputSchemaOverride bool
|
||||
wantTranscriptDescriptionPatch bool
|
||||
}
|
||||
|
||||
func assertSharedEffectiveFields(t *testing.T, cfg Config, opts sharedEffectiveFieldOptions) {
|
||||
t.Helper()
|
||||
wantOutputSchema := DefaultOutputSchema
|
||||
if opts.wantOutputSchemaOverride {
|
||||
wantOutputSchema = "audita-v1"
|
||||
}
|
||||
if cfg.OutputSchema != wantOutputSchema {
|
||||
t.Fatalf("unexpected output schema: %q", cfg.OutputSchema)
|
||||
}
|
||||
if cfg.PrimaryLLM.Model != "provider/proposal" ||
|
||||
cfg.PrimaryLLM.BaseURL != "https://proposal.example.test/v1" ||
|
||||
cfg.PrimaryLLM.TimeoutSeconds != 101 ||
|
||||
cfg.PrimaryLLM.MaxRetries != 5 {
|
||||
t.Fatalf("unexpected primary llm config: %+v", cfg.PrimaryLLM)
|
||||
}
|
||||
if cfg.ValidationLLM.Model != "provider/validation" ||
|
||||
cfg.ValidationLLM.BaseURL != "https://validation.example.test/v1" ||
|
||||
cfg.ValidationLLM.TimeoutSeconds == nil ||
|
||||
*cfg.ValidationLLM.TimeoutSeconds != 202 ||
|
||||
cfg.ValidationLLM.MaxRetries == nil ||
|
||||
*cfg.ValidationLLM.MaxRetries != 6 {
|
||||
t.Fatalf("unexpected validation llm config: %+v", cfg.ValidationLLM)
|
||||
}
|
||||
if cfg.TargetSections == nil || *cfg.TargetSections != 7 ||
|
||||
cfg.MaxSectionTokens != 9000 ||
|
||||
cfg.MinSectionTokens != 1000 {
|
||||
t.Fatalf("unexpected chunking config: target=%v max=%d min=%d", cfg.TargetSections, cfg.MaxSectionTokens, cfg.MinSectionTokens)
|
||||
}
|
||||
if cfg.Thresholds.Glossary != 0.91 ||
|
||||
cfg.Thresholds.Grammar != 0.92 ||
|
||||
cfg.Thresholds.Homophones != 0.93 ||
|
||||
cfg.Thresholds.SpokenWord != 0.94 {
|
||||
t.Fatalf("unexpected thresholds: %+v", cfg.Thresholds)
|
||||
}
|
||||
if cfg.Normalization.MaxSegmentGap != 1.2 ||
|
||||
cfg.Normalization.EllipsisGap != 2.3 ||
|
||||
cfg.Normalization.MaxSegmentDuration != 45.6 ||
|
||||
cfg.Normalization.MaxSegmentTokens != 321 {
|
||||
t.Fatalf("unexpected normalization: %+v", cfg.Normalization)
|
||||
}
|
||||
wantDescription := ""
|
||||
if opts.wantTranscriptDescriptionPatch {
|
||||
wantDescription = "shared context"
|
||||
}
|
||||
if cfg.TranscriptDescription != wantDescription {
|
||||
t.Fatalf("unexpected transcript description: %q", cfg.TranscriptDescription)
|
||||
}
|
||||
if cfg.WorkDir != "/tmp/audita-shared" ||
|
||||
cfg.WorkDirRetention != WorkDirRetentionAlways {
|
||||
t.Fatalf("unexpected diagnostics config: work_dir=%q retention=%q", cfg.WorkDir, cfg.WorkDirRetention)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,8 +7,8 @@ import (
|
||||
)
|
||||
|
||||
const (
|
||||
DefaultConfigPath = "/etc/audita/config.yml"
|
||||
DefaultConfigPathUsrLocal = "/usr/local/etc/audita/config.yml"
|
||||
DefaultConfigPath = "/etc/audita/config.yml"
|
||||
DefaultConfigPathUsrLocal = "/usr/local/etc/audita/config.yml"
|
||||
)
|
||||
|
||||
var DefaultConfigSearchPaths = []string{
|
||||
@@ -50,100 +50,93 @@ func (c *Config) applyEnvOverrides(lookup func(string) (string, bool)) error {
|
||||
cfg.Modules = modules
|
||||
}
|
||||
|
||||
primaryLLM := llmTargetPatch{}
|
||||
if raw, ok := lookup("AUDITA_LLM_API_KEY"); ok {
|
||||
cfg.PrimaryLLM.APIKey = raw
|
||||
primaryLLM.apiKey = &raw
|
||||
} else if raw, ok := lookup("OPENROUTER_API_KEY"); ok {
|
||||
cfg.PrimaryLLM.APIKey = raw
|
||||
primaryLLM.apiKey = &raw
|
||||
}
|
||||
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_LLM_API_KEY"); ok {
|
||||
cfg.ValidationLLM.APIKey = raw
|
||||
}
|
||||
|
||||
if raw, ok := lookup("AUDITA_MODEL"); ok {
|
||||
cfg.PrimaryLLM.Model = raw
|
||||
primaryLLM.model = &raw
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_MODEL"); ok {
|
||||
cfg.ValidationLLM.Model = raw
|
||||
}
|
||||
|
||||
if raw, ok := lookup("AUDITA_BASE_URL"); ok {
|
||||
cfg.PrimaryLLM.BaseURL = raw
|
||||
primaryLLM.baseURL = &raw
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_BASE_URL"); ok {
|
||||
cfg.ValidationLLM.BaseURL = raw
|
||||
}
|
||||
|
||||
if raw, ok := lookup("AUDITA_LLM_TIMEOUT_SECONDS"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_LLM_TIMEOUT_SECONDS: %w", err)
|
||||
}
|
||||
cfg.PrimaryLLM.TimeoutSeconds = value
|
||||
primaryLLM.timeoutSeconds = &value
|
||||
}
|
||||
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS: %w", err)
|
||||
}
|
||||
cfg.ValidationLLM.TimeoutSeconds = &value
|
||||
}
|
||||
|
||||
if raw, ok := lookup("AUDITA_MAX_RETRIES"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_MAX_RETRIES: %w", err)
|
||||
}
|
||||
cfg.PrimaryLLM.MaxRetries = value
|
||||
primaryLLM.maxRetries = &value
|
||||
}
|
||||
cfg.applyPrimaryLLMTargetPatch(primaryLLM)
|
||||
|
||||
validationLLM := llmTargetPatch{}
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_LLM_API_KEY"); ok {
|
||||
validationLLM.apiKey = &raw
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_MODEL"); ok {
|
||||
validationLLM.model = &raw
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_BASE_URL"); ok {
|
||||
validationLLM.baseURL = &raw
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS: %w", err)
|
||||
}
|
||||
validationLLM.timeoutSeconds = &value
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_MAX_RETRIES"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_VALIDATION_MAX_RETRIES: %w", err)
|
||||
}
|
||||
validationLLM.maxRetries = &value
|
||||
}
|
||||
cfg.applyValidationLLMTargetPatch(validationLLM)
|
||||
|
||||
concurrency := concurrencyPatch{
|
||||
inheritProposal: true,
|
||||
allowLegacyAlias: true,
|
||||
}
|
||||
totalConcurrencySet := false
|
||||
if raw, ok := lookup("AUDITA_TOTAL_LLM_CONCURRENCY"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_TOTAL_LLM_CONCURRENCY: %w", err)
|
||||
}
|
||||
cfg.TotalLLMConcurrency = value
|
||||
totalConcurrencySet = true
|
||||
concurrency.totalLLM = &value
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_LLM_CONCURRENCY"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_LLM_CONCURRENCY: %w", err)
|
||||
}
|
||||
if !totalConcurrencySet {
|
||||
cfg.TotalLLMConcurrency = value
|
||||
totalConcurrencySet = true
|
||||
}
|
||||
concurrency.legacyTotalLLM = &value
|
||||
}
|
||||
|
||||
proposalConcurrencySet := false
|
||||
if raw, ok := lookup("AUDITA_PROPOSAL_LLM_CONCURRENCY"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_PROPOSAL_LLM_CONCURRENCY: %w", err)
|
||||
}
|
||||
cfg.ProposalLLMConcurrency = value
|
||||
proposalConcurrencySet = true
|
||||
}
|
||||
if totalConcurrencySet && !proposalConcurrencySet {
|
||||
cfg.ProposalLLMConcurrency = cfg.TotalLLMConcurrency
|
||||
}
|
||||
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_MAX_RETRIES"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_VALIDATION_MAX_RETRIES: %w", err)
|
||||
}
|
||||
cfg.ValidationLLM.MaxRetries = &value
|
||||
concurrency.proposalLLM = &value
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_LLM_CONCURRENCY"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_VALIDATION_LLM_CONCURRENCY: %w", err)
|
||||
}
|
||||
cfg.ValidationLLMConcurrency = &value
|
||||
concurrency.validationLLM = &value
|
||||
}
|
||||
cfg.applyConcurrencyPatch(concurrency)
|
||||
|
||||
if raw, ok := lookup("AUDITA_VALIDATION_MAX_PROMPT_TOKENS"); ok {
|
||||
value, err := parseInt(raw)
|
||||
@@ -153,12 +146,13 @@ func (c *Config) applyEnvOverrides(lookup func(string) (string, bool)) error {
|
||||
cfg.ValidationMaxPromptTokens = value
|
||||
}
|
||||
|
||||
chunking := chunkingPatch{}
|
||||
if raw, ok := lookup("AUDITA_MAX_SECTION_TOKENS"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_MAX_SECTION_TOKENS: %w", err)
|
||||
}
|
||||
cfg.MaxSectionTokens = value
|
||||
chunking.maxSectionTokens = &value
|
||||
}
|
||||
|
||||
if raw, ok := lookup("AUDITA_MIN_SECTION_TOKENS"); ok {
|
||||
@@ -166,7 +160,7 @@ func (c *Config) applyEnvOverrides(lookup func(string) (string, bool)) error {
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_MIN_SECTION_TOKENS: %w", err)
|
||||
}
|
||||
cfg.MinSectionTokens = value
|
||||
chunking.minSectionTokens = &value
|
||||
}
|
||||
|
||||
if raw, ok := lookup("AUDITA_TARGET_SECTIONS"); ok {
|
||||
@@ -174,73 +168,80 @@ func (c *Config) applyEnvOverrides(lookup func(string) (string, bool)) error {
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_TARGET_SECTIONS: %w", err)
|
||||
}
|
||||
cfg.TargetSections = &value
|
||||
chunking.targetSections = &value
|
||||
}
|
||||
cfg.applyChunkingPatch(chunking)
|
||||
|
||||
thresholds := thresholdsPatch{}
|
||||
if raw, ok := lookup("AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD"); ok {
|
||||
value, err := parseFloat(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD: %w", err)
|
||||
}
|
||||
cfg.Thresholds.Glossary = value
|
||||
thresholds.glossary = &value
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD"); ok {
|
||||
value, err := parseFloat(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD: %w", err)
|
||||
}
|
||||
cfg.Thresholds.Grammar = value
|
||||
thresholds.grammar = &value
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD"); ok {
|
||||
value, err := parseFloat(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD: %w", err)
|
||||
}
|
||||
cfg.Thresholds.Homophones = value
|
||||
thresholds.homophones = &value
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD"); ok {
|
||||
value, err := parseFloat(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD: %w", err)
|
||||
}
|
||||
cfg.Thresholds.SpokenWord = value
|
||||
thresholds.spokenWord = &value
|
||||
}
|
||||
cfg.applyThresholdsPatch(thresholds)
|
||||
|
||||
normalization := normalizationPatch{}
|
||||
if raw, ok := lookup("AUDITA_NORMALIZE_MAX_SEGMENT_GAP"); ok {
|
||||
value, err := parseFloat(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_NORMALIZE_MAX_SEGMENT_GAP: %w", err)
|
||||
}
|
||||
cfg.Normalization.MaxSegmentGap = value
|
||||
normalization.maxSegmentGap = &value
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_NORMALIZE_ELLIPSIS_GAP"); ok {
|
||||
value, err := parseFloat(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_NORMALIZE_ELLIPSIS_GAP: %w", err)
|
||||
}
|
||||
cfg.Normalization.EllipsisGap = value
|
||||
normalization.ellipsisGap = &value
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_NORMALIZE_MAX_SEGMENT_DURATION"); ok {
|
||||
value, err := parseFloat(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_NORMALIZE_MAX_SEGMENT_DURATION: %w", err)
|
||||
}
|
||||
cfg.Normalization.MaxSegmentDuration = value
|
||||
normalization.maxSegmentDuration = &value
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS"); ok {
|
||||
value, err := parseInt(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS: %w", err)
|
||||
}
|
||||
cfg.Normalization.MaxSegmentTokens = value
|
||||
normalization.maxSegmentTokens = &value
|
||||
}
|
||||
cfg.applyNormalizationPatch(normalization)
|
||||
|
||||
diagnostics := diagnosticsPatch{}
|
||||
if raw, ok := lookup("AUDITA_WORK_DIR"); ok {
|
||||
cfg.WorkDir = raw
|
||||
diagnostics.workDir = &raw
|
||||
}
|
||||
if raw, ok := lookup("AUDITA_WORK_DIR_RETENTION"); ok {
|
||||
cfg.WorkDirRetention = WorkDirRetention(raw)
|
||||
diagnostics.workDirRetention = &raw
|
||||
}
|
||||
cfg.applyDiagnosticsPatch(diagnostics)
|
||||
|
||||
cfg.syncLegacyConcurrencyAliases()
|
||||
|
||||
|
||||
@@ -205,118 +205,98 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
|
||||
if fileCfg.LLM != nil {
|
||||
if fileCfg.LLM.Proposal != nil {
|
||||
if fileCfg.LLM.Proposal.BaseURL != nil {
|
||||
c.PrimaryLLM.BaseURL = *fileCfg.LLM.Proposal.BaseURL
|
||||
}
|
||||
if fileCfg.LLM.Proposal.Model != nil {
|
||||
c.PrimaryLLM.Model = *fileCfg.LLM.Proposal.Model
|
||||
patch := llmTargetPatch{
|
||||
model: fileCfg.LLM.Proposal.Model,
|
||||
baseURL: fileCfg.LLM.Proposal.BaseURL,
|
||||
maxRetries: fileCfg.LLM.Proposal.MaxRetries,
|
||||
}
|
||||
if fileCfg.LLM.Proposal.Timeout != nil {
|
||||
c.PrimaryLLM.TimeoutSeconds = fileCfg.LLM.Proposal.Timeout.Seconds()
|
||||
}
|
||||
if fileCfg.LLM.Proposal.MaxRetries != nil {
|
||||
c.PrimaryLLM.MaxRetries = *fileCfg.LLM.Proposal.MaxRetries
|
||||
timeoutSeconds := fileCfg.LLM.Proposal.Timeout.Seconds()
|
||||
patch.timeoutSeconds = &timeoutSeconds
|
||||
}
|
||||
if fileCfg.LLM.Proposal.APIKeyEnv != nil {
|
||||
apiKey, err := resolveAPIKeyEnv(*fileCfg.LLM.Proposal.APIKeyEnv, lookup)
|
||||
if err != nil {
|
||||
return fmt.Errorf("llm.proposal.api_key_env: %w", err)
|
||||
}
|
||||
c.PrimaryLLM.APIKey = apiKey
|
||||
patch.apiKey = &apiKey
|
||||
}
|
||||
c.applyPrimaryLLMTargetPatch(patch)
|
||||
}
|
||||
if fileCfg.LLM.Validation != nil {
|
||||
if fileCfg.LLM.Validation.BaseURL != nil {
|
||||
c.ValidationLLM.BaseURL = *fileCfg.LLM.Validation.BaseURL
|
||||
}
|
||||
if fileCfg.LLM.Validation.Model != nil {
|
||||
c.ValidationLLM.Model = *fileCfg.LLM.Validation.Model
|
||||
patch := llmTargetPatch{
|
||||
model: fileCfg.LLM.Validation.Model,
|
||||
baseURL: fileCfg.LLM.Validation.BaseURL,
|
||||
maxRetries: fileCfg.LLM.Validation.MaxRetries,
|
||||
}
|
||||
if fileCfg.LLM.Validation.Timeout != nil {
|
||||
v := fileCfg.LLM.Validation.Timeout.Seconds()
|
||||
c.ValidationLLM.TimeoutSeconds = &v
|
||||
}
|
||||
if fileCfg.LLM.Validation.MaxRetries != nil {
|
||||
v := *fileCfg.LLM.Validation.MaxRetries
|
||||
c.ValidationLLM.MaxRetries = &v
|
||||
timeoutSeconds := fileCfg.LLM.Validation.Timeout.Seconds()
|
||||
patch.timeoutSeconds = &timeoutSeconds
|
||||
}
|
||||
if fileCfg.LLM.Validation.APIKeyEnv != nil {
|
||||
apiKey, err := resolveAPIKeyEnv(*fileCfg.LLM.Validation.APIKeyEnv, lookup)
|
||||
if err != nil {
|
||||
return fmt.Errorf("llm.validation.api_key_env: %w", err)
|
||||
}
|
||||
c.ValidationLLM.APIKey = apiKey
|
||||
patch.apiKey = &apiKey
|
||||
}
|
||||
c.applyValidationLLMTargetPatch(patch)
|
||||
}
|
||||
}
|
||||
|
||||
if fileCfg.Concurrency != nil {
|
||||
if fileCfg.Concurrency.TotalLLM != nil {
|
||||
c.TotalLLMConcurrency = *fileCfg.Concurrency.TotalLLM
|
||||
}
|
||||
if fileCfg.Concurrency.ProposalLLM != nil {
|
||||
c.ProposalLLMConcurrency = *fileCfg.Concurrency.ProposalLLM
|
||||
}
|
||||
if fileCfg.Concurrency.ValidationLLM != nil {
|
||||
v := *fileCfg.Concurrency.ValidationLLM
|
||||
c.ValidationLLMConcurrency = &v
|
||||
}
|
||||
c.applyConcurrencyPatch(concurrencyPatch{
|
||||
totalLLM: fileCfg.Concurrency.TotalLLM,
|
||||
proposalLLM: fileCfg.Concurrency.ProposalLLM,
|
||||
validationLLM: fileCfg.Concurrency.ValidationLLM,
|
||||
})
|
||||
}
|
||||
|
||||
if fileCfg.Chunking != nil {
|
||||
if fileCfg.Chunking.TargetSections != nil {
|
||||
v := *fileCfg.Chunking.TargetSections
|
||||
c.TargetSections = &v
|
||||
}
|
||||
if fileCfg.Chunking.MaxSectionTokens != nil {
|
||||
c.MaxSectionTokens = *fileCfg.Chunking.MaxSectionTokens
|
||||
}
|
||||
if fileCfg.Chunking.MinSectionTokens != nil {
|
||||
c.MinSectionTokens = *fileCfg.Chunking.MinSectionTokens
|
||||
}
|
||||
c.applyChunkingPatch(chunkingPatch{
|
||||
targetSections: fileCfg.Chunking.TargetSections,
|
||||
maxSectionTokens: fileCfg.Chunking.MaxSectionTokens,
|
||||
minSectionTokens: fileCfg.Chunking.MinSectionTokens,
|
||||
})
|
||||
}
|
||||
|
||||
if fileCfg.Normalization != nil {
|
||||
patch := normalizationPatch{
|
||||
maxSegmentTokens: fileCfg.Normalization.MaxSegmentTokens,
|
||||
}
|
||||
if fileCfg.Normalization.MaxSegmentGap != nil {
|
||||
c.Normalization.MaxSegmentGap = fileCfg.Normalization.MaxSegmentGap.Seconds()
|
||||
maxSegmentGap := fileCfg.Normalization.MaxSegmentGap.Seconds()
|
||||
patch.maxSegmentGap = &maxSegmentGap
|
||||
}
|
||||
if fileCfg.Normalization.EllipsisGap != nil {
|
||||
c.Normalization.EllipsisGap = fileCfg.Normalization.EllipsisGap.Seconds()
|
||||
ellipsisGap := fileCfg.Normalization.EllipsisGap.Seconds()
|
||||
patch.ellipsisGap = &ellipsisGap
|
||||
}
|
||||
if fileCfg.Normalization.MaxSegmentDuration != nil {
|
||||
c.Normalization.MaxSegmentDuration = fileCfg.Normalization.MaxSegmentDuration.Seconds()
|
||||
}
|
||||
if fileCfg.Normalization.MaxSegmentTokens != nil {
|
||||
c.Normalization.MaxSegmentTokens = *fileCfg.Normalization.MaxSegmentTokens
|
||||
maxSegmentDuration := fileCfg.Normalization.MaxSegmentDuration.Seconds()
|
||||
patch.maxSegmentDuration = &maxSegmentDuration
|
||||
}
|
||||
c.applyNormalizationPatch(patch)
|
||||
}
|
||||
|
||||
if fileCfg.Thresholds != nil {
|
||||
if fileCfg.Thresholds.Glossary != nil {
|
||||
c.Thresholds.Glossary = *fileCfg.Thresholds.Glossary
|
||||
}
|
||||
if fileCfg.Thresholds.Homophones != nil {
|
||||
c.Thresholds.Homophones = *fileCfg.Thresholds.Homophones
|
||||
}
|
||||
if fileCfg.Thresholds.SpokenWord != nil {
|
||||
c.Thresholds.SpokenWord = *fileCfg.Thresholds.SpokenWord
|
||||
}
|
||||
if fileCfg.Thresholds.Grammar != nil {
|
||||
c.Thresholds.Grammar = *fileCfg.Thresholds.Grammar
|
||||
}
|
||||
c.applyThresholdsPatch(thresholdsPatch{
|
||||
glossary: fileCfg.Thresholds.Glossary,
|
||||
grammar: fileCfg.Thresholds.Grammar,
|
||||
homophones: fileCfg.Thresholds.Homophones,
|
||||
spokenWord: fileCfg.Thresholds.SpokenWord,
|
||||
})
|
||||
}
|
||||
|
||||
if fileCfg.Context != nil && fileCfg.Context.Description != nil {
|
||||
c.TranscriptDescription = strings.TrimSpace(*fileCfg.Context.Description)
|
||||
c.applyContextPatch(contextPatch{transcriptDescription: fileCfg.Context.Description})
|
||||
}
|
||||
|
||||
if fileCfg.Diagnostics != nil {
|
||||
if fileCfg.Diagnostics.WorkDir != nil {
|
||||
c.WorkDir = *fileCfg.Diagnostics.WorkDir
|
||||
}
|
||||
if fileCfg.Diagnostics.Retention != nil {
|
||||
c.WorkDirRetention = WorkDirRetention(*fileCfg.Diagnostics.Retention)
|
||||
}
|
||||
c.applyDiagnosticsPatch(diagnosticsPatch{
|
||||
workDir: fileCfg.Diagnostics.WorkDir,
|
||||
workDirRetention: fileCfg.Diagnostics.Retention,
|
||||
})
|
||||
}
|
||||
|
||||
c.syncLegacyConcurrencyAliases()
|
||||
|
||||
@@ -51,107 +51,54 @@ func (c *Config) ApplyCLIOverrides(overrides CLIOverrides) error {
|
||||
c.OutputSchema = strings.TrimSpace(*overrides.OutputSchema)
|
||||
}
|
||||
|
||||
if overrides.PrimaryLLMAPIKey != nil {
|
||||
c.PrimaryLLM.APIKey = *overrides.PrimaryLLMAPIKey
|
||||
}
|
||||
if overrides.ValidationLLMAPIKey != nil {
|
||||
c.ValidationLLM.APIKey = *overrides.ValidationLLMAPIKey
|
||||
}
|
||||
if overrides.PrimaryModel != nil {
|
||||
c.PrimaryLLM.Model = *overrides.PrimaryModel
|
||||
}
|
||||
if overrides.ValidationModel != nil {
|
||||
c.ValidationLLM.Model = *overrides.ValidationModel
|
||||
}
|
||||
if overrides.PrimaryBaseURL != nil {
|
||||
c.PrimaryLLM.BaseURL = *overrides.PrimaryBaseURL
|
||||
}
|
||||
if overrides.ValidationBaseURL != nil {
|
||||
c.ValidationLLM.BaseURL = *overrides.ValidationBaseURL
|
||||
}
|
||||
if overrides.PrimaryLLMTimeoutSeconds != nil {
|
||||
c.PrimaryLLM.TimeoutSeconds = *overrides.PrimaryLLMTimeoutSeconds
|
||||
}
|
||||
totalConcurrencySet := false
|
||||
if overrides.TotalLLMConcurrency != nil {
|
||||
c.TotalLLMConcurrency = *overrides.TotalLLMConcurrency
|
||||
totalConcurrencySet = true
|
||||
}
|
||||
// Backward-compatible alias: --llm-concurrency maps to total concurrency
|
||||
// only when --total-llm-concurrency is not set in the same CLI invocation.
|
||||
if overrides.PrimaryLLMConcurrency != nil && !totalConcurrencySet {
|
||||
c.TotalLLMConcurrency = *overrides.PrimaryLLMConcurrency
|
||||
totalConcurrencySet = true
|
||||
}
|
||||
proposalConcurrencySet := false
|
||||
if overrides.ProposalLLMConcurrency != nil {
|
||||
c.ProposalLLMConcurrency = *overrides.ProposalLLMConcurrency
|
||||
proposalConcurrencySet = true
|
||||
}
|
||||
if totalConcurrencySet && !proposalConcurrencySet {
|
||||
c.ProposalLLMConcurrency = c.TotalLLMConcurrency
|
||||
}
|
||||
if overrides.ValidationLLMTimeoutSeconds != nil {
|
||||
value := *overrides.ValidationLLMTimeoutSeconds
|
||||
c.ValidationLLM.TimeoutSeconds = &value
|
||||
}
|
||||
if overrides.MaxRetries != nil {
|
||||
c.PrimaryLLM.MaxRetries = *overrides.MaxRetries
|
||||
}
|
||||
if overrides.ValidationMaxRetries != nil {
|
||||
value := *overrides.ValidationMaxRetries
|
||||
c.ValidationLLM.MaxRetries = &value
|
||||
}
|
||||
if overrides.ValidationLLMConcurrency != nil {
|
||||
value := *overrides.ValidationLLMConcurrency
|
||||
c.ValidationLLMConcurrency = &value
|
||||
}
|
||||
c.applyPrimaryLLMTargetPatch(llmTargetPatch{
|
||||
apiKey: overrides.PrimaryLLMAPIKey,
|
||||
model: overrides.PrimaryModel,
|
||||
baseURL: overrides.PrimaryBaseURL,
|
||||
timeoutSeconds: overrides.PrimaryLLMTimeoutSeconds,
|
||||
maxRetries: overrides.MaxRetries,
|
||||
})
|
||||
c.applyValidationLLMTargetPatch(llmTargetPatch{
|
||||
apiKey: overrides.ValidationLLMAPIKey,
|
||||
model: overrides.ValidationModel,
|
||||
baseURL: overrides.ValidationBaseURL,
|
||||
timeoutSeconds: overrides.ValidationLLMTimeoutSeconds,
|
||||
maxRetries: overrides.ValidationMaxRetries,
|
||||
})
|
||||
c.applyConcurrencyPatch(concurrencyPatch{
|
||||
totalLLM: overrides.TotalLLMConcurrency,
|
||||
legacyTotalLLM: overrides.PrimaryLLMConcurrency,
|
||||
proposalLLM: overrides.ProposalLLMConcurrency,
|
||||
validationLLM: overrides.ValidationLLMConcurrency,
|
||||
inheritProposal: true,
|
||||
allowLegacyAlias: true,
|
||||
})
|
||||
|
||||
if overrides.ValidationMaxPromptTokens != nil {
|
||||
c.ValidationMaxPromptTokens = *overrides.ValidationMaxPromptTokens
|
||||
}
|
||||
if overrides.MaxSectionTokens != nil {
|
||||
c.MaxSectionTokens = *overrides.MaxSectionTokens
|
||||
}
|
||||
if overrides.MinSectionTokens != nil {
|
||||
c.MinSectionTokens = *overrides.MinSectionTokens
|
||||
}
|
||||
if overrides.TargetSections != nil {
|
||||
value := *overrides.TargetSections
|
||||
c.TargetSections = &value
|
||||
}
|
||||
if overrides.GlossaryConfidenceThreshold != nil {
|
||||
c.Thresholds.Glossary = *overrides.GlossaryConfidenceThreshold
|
||||
}
|
||||
if overrides.GrammarConfidenceThreshold != nil {
|
||||
c.Thresholds.Grammar = *overrides.GrammarConfidenceThreshold
|
||||
}
|
||||
if overrides.HomophonesConfidenceThreshold != nil {
|
||||
c.Thresholds.Homophones = *overrides.HomophonesConfidenceThreshold
|
||||
}
|
||||
if overrides.SpokenWordConfidenceThreshold != nil {
|
||||
c.Thresholds.SpokenWord = *overrides.SpokenWordConfidenceThreshold
|
||||
}
|
||||
if overrides.NormalizeMaxSegmentGap != nil {
|
||||
c.Normalization.MaxSegmentGap = *overrides.NormalizeMaxSegmentGap
|
||||
}
|
||||
if overrides.NormalizeEllipsisGap != nil {
|
||||
c.Normalization.EllipsisGap = *overrides.NormalizeEllipsisGap
|
||||
}
|
||||
if overrides.NormalizeMaxSegmentDuration != nil {
|
||||
c.Normalization.MaxSegmentDuration = *overrides.NormalizeMaxSegmentDuration
|
||||
}
|
||||
if overrides.NormalizeMaxSegmentTokens != nil {
|
||||
c.Normalization.MaxSegmentTokens = *overrides.NormalizeMaxSegmentTokens
|
||||
}
|
||||
if overrides.TranscriptDescription != nil {
|
||||
c.TranscriptDescription = strings.TrimSpace(*overrides.TranscriptDescription)
|
||||
}
|
||||
if overrides.WorkDir != nil {
|
||||
c.WorkDir = *overrides.WorkDir
|
||||
}
|
||||
if overrides.WorkDirRetention != nil {
|
||||
c.WorkDirRetention = WorkDirRetention(*overrides.WorkDirRetention)
|
||||
}
|
||||
c.applyChunkingPatch(chunkingPatch{
|
||||
targetSections: overrides.TargetSections,
|
||||
maxSectionTokens: overrides.MaxSectionTokens,
|
||||
minSectionTokens: overrides.MinSectionTokens,
|
||||
})
|
||||
c.applyThresholdsPatch(thresholdsPatch{
|
||||
glossary: overrides.GlossaryConfidenceThreshold,
|
||||
grammar: overrides.GrammarConfidenceThreshold,
|
||||
homophones: overrides.HomophonesConfidenceThreshold,
|
||||
spokenWord: overrides.SpokenWordConfidenceThreshold,
|
||||
})
|
||||
c.applyNormalizationPatch(normalizationPatch{
|
||||
maxSegmentGap: overrides.NormalizeMaxSegmentGap,
|
||||
ellipsisGap: overrides.NormalizeEllipsisGap,
|
||||
maxSegmentDuration: overrides.NormalizeMaxSegmentDuration,
|
||||
maxSegmentTokens: overrides.NormalizeMaxSegmentTokens,
|
||||
})
|
||||
c.applyContextPatch(contextPatch{transcriptDescription: overrides.TranscriptDescription})
|
||||
c.applyDiagnosticsPatch(diagnosticsPatch{
|
||||
workDir: overrides.WorkDir,
|
||||
workDirRetention: overrides.WorkDirRetention,
|
||||
})
|
||||
|
||||
c.syncLegacyConcurrencyAliases()
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package cli
|
||||
package processreport
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
@@ -15,7 +15,12 @@ const (
|
||||
correctionDispositionFailed = "failed"
|
||||
)
|
||||
|
||||
type correctionLedgerEntry struct {
|
||||
type CorrectionLedgerInput struct {
|
||||
RunDirectoryPath string
|
||||
RunOutput *runner.RunOutput
|
||||
}
|
||||
|
||||
type CorrectionLedgerEntry struct {
|
||||
RunID string `json:"run_id,omitempty"`
|
||||
ModuleKey string `json:"module_key"`
|
||||
ModuleInstance string `json:"module_instance"`
|
||||
@@ -28,27 +33,28 @@ type correctionLedgerEntry struct {
|
||||
Disposition string `json:"disposition"`
|
||||
DispositionReasonCode string `json:"disposition_reason_code,omitempty"`
|
||||
DispositionMessage string `json:"disposition_message,omitempty"`
|
||||
DeterministicValidatorResults []ledgerValidatorDecisionRecord `json:"deterministic_validator_decisions,omitempty"`
|
||||
LLMValidatorResults []ledgerValidatorDecisionRecord `json:"llm_validator_decisions,omitempty"`
|
||||
DeterministicValidatorResults []LedgerValidatorDecisionRecord `json:"deterministic_validator_decisions,omitempty"`
|
||||
LLMValidatorResults []LedgerValidatorDecisionRecord `json:"llm_validator_decisions,omitempty"`
|
||||
}
|
||||
|
||||
type ledgerValidatorDecisionRecord struct {
|
||||
type LedgerValidatorDecisionRecord struct {
|
||||
ValidatorKey string `json:"validator_key"`
|
||||
Approved bool `json:"approved"`
|
||||
ReasonCode string `json:"reason_code"`
|
||||
Message string `json:"message,omitempty"`
|
||||
}
|
||||
|
||||
func buildCorrectionLedger(runDirPath string, runOutput *runner.RunOutput) []correctionLedgerEntry {
|
||||
func BuildCorrectionLedger(input CorrectionLedgerInput) []CorrectionLedgerEntry {
|
||||
runOutput := input.RunOutput
|
||||
if runOutput == nil || len(runOutput.ModuleResults) == 0 {
|
||||
return nil
|
||||
}
|
||||
runID := ""
|
||||
if runDirPath != "" {
|
||||
runID = filepath.Base(runDirPath)
|
||||
if input.RunDirectoryPath != "" {
|
||||
runID = filepath.Base(input.RunDirectoryPath)
|
||||
}
|
||||
|
||||
entries := make([]correctionLedgerEntry, 0)
|
||||
entries := make([]CorrectionLedgerEntry, 0)
|
||||
for _, module := range runOutput.ModuleResults {
|
||||
decisionsByProposal := make(map[int][]runner.ValidatorDecisionRecord)
|
||||
for _, decision := range module.ValidatorDecisions {
|
||||
@@ -56,7 +62,7 @@ func buildCorrectionLedger(runDirPath string, runOutput *runner.RunOutput) []cor
|
||||
}
|
||||
|
||||
for _, change := range module.AppliedChanges {
|
||||
entries = append(entries, correctionLedgerEntry{
|
||||
entries = append(entries, CorrectionLedgerEntry{
|
||||
RunID: runID,
|
||||
ModuleKey: module.ModuleKey,
|
||||
ModuleInstance: module.ModuleInstance,
|
||||
@@ -72,7 +78,7 @@ func buildCorrectionLedger(runDirPath string, runOutput *runner.RunOutput) []cor
|
||||
})
|
||||
}
|
||||
for _, change := range module.SkippedChanges {
|
||||
entries = append(entries, correctionLedgerEntry{
|
||||
entries = append(entries, CorrectionLedgerEntry{
|
||||
RunID: runID,
|
||||
ModuleKey: module.ModuleKey,
|
||||
ModuleInstance: module.ModuleInstance,
|
||||
@@ -89,7 +95,7 @@ func buildCorrectionLedger(runDirPath string, runOutput *runner.RunOutput) []cor
|
||||
})
|
||||
}
|
||||
for _, rejection := range module.ValidatorRejected {
|
||||
entries = append(entries, correctionLedgerEntry{
|
||||
entries = append(entries, CorrectionLedgerEntry{
|
||||
RunID: runID,
|
||||
ModuleKey: module.ModuleKey,
|
||||
ModuleInstance: module.ModuleInstance,
|
||||
@@ -106,7 +112,7 @@ func buildCorrectionLedger(runDirPath string, runOutput *runner.RunOutput) []cor
|
||||
})
|
||||
}
|
||||
if module.Status == runner.ModuleStatusFailed {
|
||||
entries = append(entries, correctionLedgerEntry{
|
||||
entries = append(entries, CorrectionLedgerEntry{
|
||||
RunID: runID,
|
||||
ModuleKey: module.ModuleKey,
|
||||
ModuleInstance: module.ModuleInstance,
|
||||
@@ -130,17 +136,29 @@ func buildCorrectionLedger(runDirPath string, runOutput *runner.RunOutput) []cor
|
||||
return entries
|
||||
}
|
||||
|
||||
func filterLedgerDecisions(in []runner.ValidatorDecisionRecord, wantLLM bool) []ledgerValidatorDecisionRecord {
|
||||
func HasSkippedCorrections(runOutput *runner.RunOutput) bool {
|
||||
if runOutput == nil {
|
||||
return false
|
||||
}
|
||||
for _, mr := range runOutput.ModuleResults {
|
||||
if len(mr.SkippedChanges) > 0 || len(mr.ValidatorRejected) > 0 {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func filterLedgerDecisions(in []runner.ValidatorDecisionRecord, wantLLM bool) []LedgerValidatorDecisionRecord {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]ledgerValidatorDecisionRecord, 0, len(in))
|
||||
out := make([]LedgerValidatorDecisionRecord, 0, len(in))
|
||||
for _, decision := range in {
|
||||
isLLMBacked := validatormetadata.ClassForKey(decision.ValidatorName) == validatormetadata.ExecutionClassLLMBacked
|
||||
if isLLMBacked != wantLLM {
|
||||
continue
|
||||
}
|
||||
out = append(out, ledgerValidatorDecisionRecord{
|
||||
out = append(out, LedgerValidatorDecisionRecord{
|
||||
ValidatorKey: decision.ValidatorName,
|
||||
Approved: decision.Approved,
|
||||
ReasonCode: decision.ReasonCode,
|
||||
128
internal/framework/processreport/correction_ledger_test.go
Normal file
128
internal/framework/processreport/correction_ledger_test.go
Normal file
@@ -0,0 +1,128 @@
|
||||
package processreport
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/runner"
|
||||
)
|
||||
|
||||
func TestBuildCorrectionLedgerClassifiesValidatorDecisionsFromCanonicalMetadata(t *testing.T) {
|
||||
output := &runner.RunOutput{
|
||||
ModuleResults: []runner.ModuleResult{
|
||||
{
|
||||
ModuleKey: "glossary",
|
||||
ModuleInstance: "glossary",
|
||||
ReplacementPolicy: proposals.ReplacementPolicyReplaceAll,
|
||||
ValidatorDecisions: []runner.ValidatorDecisionRecord{
|
||||
{ValidatorName: "proposal_shape", ProposalIndex: 3, Approved: true, ReasonCode: "approved"},
|
||||
{ValidatorName: "spoken_form_plausibility", ProposalIndex: 3, Approved: true, ReasonCode: "approved"},
|
||||
},
|
||||
AppliedChanges: []proposals.AppliedChange{
|
||||
{
|
||||
ProposalIndex: 3,
|
||||
ModuleKey: "glossary",
|
||||
ModuleInstance: "glossary",
|
||||
TargetSegmentID: 1,
|
||||
OriginalText: "gestures",
|
||||
CorrectedText: "Jesters",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
ledger := BuildCorrectionLedger(CorrectionLedgerInput{
|
||||
RunDirectoryPath: "/tmp/audita-run-id",
|
||||
RunOutput: output,
|
||||
})
|
||||
if len(ledger) != 1 {
|
||||
t.Fatalf("expected one ledger entry, got %d", len(ledger))
|
||||
}
|
||||
entry := ledger[0]
|
||||
if entry.RunID != "audita-run-id" || entry.Disposition != "applied" || entry.AppliedCorrectedText != "Jesters" {
|
||||
t.Fatalf("unexpected applied ledger entry: %+v", entry)
|
||||
}
|
||||
if len(entry.DeterministicValidatorResults) != 1 || entry.DeterministicValidatorResults[0].ValidatorKey != "proposal_shape" {
|
||||
t.Fatalf("unexpected deterministic decision split: %+v", entry.DeterministicValidatorResults)
|
||||
}
|
||||
if len(entry.LLMValidatorResults) != 1 || entry.LLMValidatorResults[0].ValidatorKey != "spoken_form_plausibility" {
|
||||
t.Fatalf("unexpected llm-backed decision split: %+v", entry.LLMValidatorResults)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildCorrectionLedgerPreservesDispositionPolicy(t *testing.T) {
|
||||
output := &runner.RunOutput{
|
||||
ModuleResults: []runner.ModuleResult{
|
||||
{
|
||||
ModuleKey: "grammar",
|
||||
ModuleInstance: "grammar",
|
||||
ReplacementPolicy: proposals.ReplacementPolicyRequireUnique,
|
||||
Status: runner.ModuleStatusSuccess,
|
||||
SkippedChanges: []proposals.SkippedChange{
|
||||
{
|
||||
ProposalIndex: 2,
|
||||
TargetSegmentID: 7,
|
||||
OriginalText: "old",
|
||||
CorrectedText: "new",
|
||||
SkipReason: proposals.SkipReasonAmbiguousOriginal,
|
||||
Message: "ambiguous",
|
||||
},
|
||||
},
|
||||
ValidatorRejected: []runner.ValidatorRejectedChange{
|
||||
{
|
||||
ProposalIndex: 3,
|
||||
TargetSegmentID: 8,
|
||||
OriginalText: "before",
|
||||
CorrectedText: "after",
|
||||
ReasonCode: "protected_term",
|
||||
Message: "blocked",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
ModuleKey: "capitalization",
|
||||
ModuleInstance: "capitalization",
|
||||
Status: runner.ModuleStatusFailed,
|
||||
ErrorMessage: "failed",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
ledger := BuildCorrectionLedger(CorrectionLedgerInput{RunOutput: output})
|
||||
if len(ledger) != 3 {
|
||||
t.Fatalf("expected skipped, rejected, and failed entries, got %+v", ledger)
|
||||
}
|
||||
|
||||
byDisposition := make(map[string]CorrectionLedgerEntry)
|
||||
for _, entry := range ledger {
|
||||
byDisposition[entry.Disposition] = entry
|
||||
}
|
||||
if byDisposition["skipped"].DispositionReasonCode != string(proposals.SkipReasonAmbiguousOriginal) ||
|
||||
byDisposition["skipped"].DispositionMessage != "ambiguous" {
|
||||
t.Fatalf("unexpected skipped ledger entry: %+v", byDisposition["skipped"])
|
||||
}
|
||||
if byDisposition["rejected"].DispositionReasonCode != "protected_term" ||
|
||||
byDisposition["rejected"].ProposedCorrectedText != "after" {
|
||||
t.Fatalf("unexpected rejected ledger entry: %+v", byDisposition["rejected"])
|
||||
}
|
||||
if byDisposition["failed"].DispositionReasonCode != "module_failed" ||
|
||||
byDisposition["failed"].DispositionMessage != "failed" {
|
||||
t.Fatalf("unexpected failed ledger entry: %+v", byDisposition["failed"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestHasSkippedCorrectionsIncludesApplicationSkipsAndValidatorRejections(t *testing.T) {
|
||||
if HasSkippedCorrections(nil) {
|
||||
t.Fatal("nil output should not have skipped corrections")
|
||||
}
|
||||
if HasSkippedCorrections(&runner.RunOutput{ModuleResults: []runner.ModuleResult{{AppliedChanges: []proposals.AppliedChange{{ProposalIndex: 1}}}}}) {
|
||||
t.Fatal("applied-only output should not have skipped corrections")
|
||||
}
|
||||
if !HasSkippedCorrections(&runner.RunOutput{ModuleResults: []runner.ModuleResult{{SkippedChanges: []proposals.SkippedChange{{ProposalIndex: 1}}}}}) {
|
||||
t.Fatal("application skips should count as skipped corrections")
|
||||
}
|
||||
if !HasSkippedCorrections(&runner.RunOutput{ModuleResults: []runner.ModuleResult{{ValidatorRejected: []runner.ValidatorRejectedChange{{ProposalIndex: 1}}}}}) {
|
||||
t.Fatal("validator rejections should count as skipped corrections")
|
||||
}
|
||||
}
|
||||
158
internal/framework/processreport/report.go
Normal file
158
internal/framework/processreport/report.go
Normal file
@@ -0,0 +1,158 @@
|
||||
package processreport
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/chunking"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/diagnostics"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/normalization"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/reporting"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/runner"
|
||||
stagewarnings "gitea.maximumdirect.net/eric/audita/internal/framework/warnings"
|
||||
)
|
||||
|
||||
// BuildInput contains already-computed process execution facts for report assembly.
|
||||
type BuildInput struct {
|
||||
Status string
|
||||
TranscriptPath string
|
||||
GlossaryPath string
|
||||
OutputPath string
|
||||
Modules []string
|
||||
OutputSchema string
|
||||
ConfigVersion *int
|
||||
StartedAt time.Time
|
||||
CompletedAt time.Time
|
||||
ErrorMessage string
|
||||
ErrorPhase string
|
||||
RunDirectoryPath string
|
||||
NormalizationSummary *normalization.NormalizationSummary
|
||||
ChunkingSummary *chunking.Summary
|
||||
RunOutput *runner.RunOutput
|
||||
}
|
||||
|
||||
// Build creates the public process report without owning command parsing or config loading.
|
||||
func Build(input BuildInput) reporting.ProcessReport {
|
||||
report := reporting.ProcessReport{
|
||||
ReportMetadata: reporting.ReportMetadata{
|
||||
ReportSchemaName: reporting.DefaultProcessReportSchemaName,
|
||||
ReportSchemaVersion: reporting.DefaultProcessReportSchemaVersion,
|
||||
OutputSchema: input.OutputSchema,
|
||||
ConfigVersion: input.ConfigVersion,
|
||||
},
|
||||
Phase: "default_pipeline",
|
||||
Status: input.Status,
|
||||
Operation: "process",
|
||||
TranscriptPath: input.TranscriptPath,
|
||||
GlossaryPath: input.GlossaryPath,
|
||||
OutputPath: input.OutputPath,
|
||||
Modules: append([]string(nil), input.Modules...),
|
||||
StartedAt: input.StartedAt,
|
||||
CompletedAt: &input.CompletedAt,
|
||||
ErrorPhase: input.ErrorPhase,
|
||||
}
|
||||
if input.RunDirectoryPath != "" {
|
||||
runSucceeded := input.Status == "success"
|
||||
metadata := diagnostics.BuildDiagnosticsMetadata(input.RunDirectoryPath, runSucceeded)
|
||||
report.Diagnostics = &metadata
|
||||
}
|
||||
if input.ErrorMessage != "" {
|
||||
report.ErrorMessage = input.ErrorMessage
|
||||
}
|
||||
if input.NormalizationSummary != nil {
|
||||
report.InputSegmentCount = &input.NormalizationSummary.InputSegmentCount
|
||||
report.NormalizedSegmentCount = &input.NormalizationSummary.OutputSegmentCount
|
||||
report.NormalizationMerges = &input.NormalizationSummary.MergesPerformed
|
||||
report.NormalizationIDReassignments = &input.NormalizationSummary.IDsReassigned
|
||||
report.NormalizationSkipped.DifferentSpeakers = &input.NormalizationSummary.SkippedMerges.DifferentSpeakers
|
||||
report.NormalizationSkipped.GapTooLarge = &input.NormalizationSummary.SkippedMerges.GapTooLarge
|
||||
report.NormalizationSkipped.DurationExceeded = &input.NormalizationSummary.SkippedMerges.DurationExceeded
|
||||
report.NormalizationSkipped.TokenLimitExceeded = &input.NormalizationSummary.SkippedMerges.TokenLimitExceeded
|
||||
}
|
||||
if input.ChunkingSummary != nil {
|
||||
report.Chunking = &reporting.ChunkingSummary{
|
||||
ChunkCount: input.ChunkingSummary.ChunkCount,
|
||||
MinEstimatedTokens: input.ChunkingSummary.MinEstimatedTokens,
|
||||
MaxEstimatedTokens: input.ChunkingSummary.MaxEstimatedTokens,
|
||||
TotalEstimatedTokens: input.ChunkingSummary.TotalEstimatedTokens,
|
||||
TargetSections: input.ChunkingSummary.TargetSections,
|
||||
MaxSectionTokens: input.ChunkingSummary.MaxSectionTokens,
|
||||
MinSectionTokens: input.ChunkingSummary.MinSectionTokens,
|
||||
}
|
||||
}
|
||||
report.ModulesSummary, report.ModuleResults = buildModuleReporting(input.RunOutput)
|
||||
return report
|
||||
}
|
||||
|
||||
func buildModuleReporting(runOutput *runner.RunOutput) (*reporting.ModulesSummary, []reporting.ModuleReport) {
|
||||
if runOutput == nil || len(runOutput.ModuleResults) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
moduleReports := make([]reporting.ModuleReport, 0, len(runOutput.ModuleResults))
|
||||
summary := &reporting.ModulesSummary{ModuleCount: len(runOutput.ModuleResults)}
|
||||
for _, r := range runOutput.ModuleResults {
|
||||
startedAt := r.StartedAt
|
||||
completedAt := r.CompletedAt
|
||||
moduleReports = append(moduleReports, reporting.ModuleReport{
|
||||
ModuleKey: r.ModuleKey,
|
||||
ModuleInstance: r.ModuleInstance,
|
||||
ReplacementPolicy: string(r.ReplacementPolicy),
|
||||
Status: r.Status,
|
||||
ProposalCount: r.ProposalCount,
|
||||
Warnings: append([]stagewarnings.StageWarning(nil), r.Warnings...),
|
||||
ValidatorDecisions: mapValidatorDecisions(r.ValidatorDecisions),
|
||||
ValidatorRejected: mapValidatorRejected(r.ValidatorRejected),
|
||||
AppliedChanges: r.AppliedChanges,
|
||||
SkippedChanges: r.SkippedChanges,
|
||||
ErrorMessage: r.ErrorMessage,
|
||||
StartedAt: &startedAt,
|
||||
CompletedAt: &completedAt,
|
||||
})
|
||||
summary.TotalAppliedChanges += len(r.AppliedChanges)
|
||||
summary.TotalSkippedChanges += len(r.SkippedChanges) + len(r.ValidatorRejected)
|
||||
if r.Status == runner.ModuleStatusFailed && summary.FailedModuleInstance == "" {
|
||||
summary.FailedModuleInstance = r.ModuleInstance
|
||||
}
|
||||
}
|
||||
|
||||
return summary, moduleReports
|
||||
}
|
||||
|
||||
func mapValidatorDecisions(in []runner.ValidatorDecisionRecord) []reporting.ValidatorDecisionReport {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]reporting.ValidatorDecisionReport, len(in))
|
||||
for i, d := range in {
|
||||
out[i] = reporting.ValidatorDecisionReport{
|
||||
ValidatorName: d.ValidatorName,
|
||||
ProposalIndex: d.ProposalIndex,
|
||||
Approved: d.Approved,
|
||||
ReasonCode: d.ReasonCode,
|
||||
Message: d.Message,
|
||||
DiagnosticArtifactPath: d.DiagnosticArtifactPath,
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func mapValidatorRejected(in []runner.ValidatorRejectedChange) []reporting.ValidatorRejectedReport {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]reporting.ValidatorRejectedReport, len(in))
|
||||
for i, d := range in {
|
||||
out[i] = reporting.ValidatorRejectedReport{
|
||||
ValidatorName: d.ValidatorName,
|
||||
ProposalIndex: d.ProposalIndex,
|
||||
ModuleKey: d.ModuleKey,
|
||||
ModuleInstance: d.ModuleInstance,
|
||||
TargetSegmentID: d.TargetSegmentID,
|
||||
OriginalText: d.OriginalText,
|
||||
CorrectedText: d.CorrectedText,
|
||||
ReasonCode: d.ReasonCode,
|
||||
Message: d.Message,
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
180
internal/framework/processreport/report_test.go
Normal file
180
internal/framework/processreport/report_test.go
Normal file
@@ -0,0 +1,180 @@
|
||||
package processreport
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/chunking"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/diagnostics"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/normalization"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/reporting"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/runner"
|
||||
)
|
||||
|
||||
func TestBuildSuccessReportMapsExecutionFacts(t *testing.T) {
|
||||
startedAt := time.Date(2026, 5, 23, 10, 0, 0, 0, time.UTC)
|
||||
completedAt := startedAt.Add(time.Second)
|
||||
configVersion := 4
|
||||
targetSections := 2
|
||||
inputSegments := 5
|
||||
outputSegments := 4
|
||||
merges := 1
|
||||
reassigned := 2
|
||||
differentSpeakers := 3
|
||||
|
||||
report := Build(BuildInput{
|
||||
Status: "success",
|
||||
TranscriptPath: "transcript.json",
|
||||
GlossaryPath: "glossary.yaml",
|
||||
OutputPath: "out.json",
|
||||
Modules: []string{"grammar"},
|
||||
OutputSchema: "default",
|
||||
ConfigVersion: &configVersion,
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: completedAt,
|
||||
RunDirectoryPath: filepath.Join(
|
||||
"tmp",
|
||||
"audita-run",
|
||||
),
|
||||
NormalizationSummary: &normalization.NormalizationSummary{
|
||||
InputSegmentCount: inputSegments,
|
||||
OutputSegmentCount: outputSegments,
|
||||
MergesPerformed: merges,
|
||||
IDsReassigned: reassigned,
|
||||
SkippedMerges: struct {
|
||||
DifferentSpeakers int `json:"different_speakers"`
|
||||
GapTooLarge int `json:"gap_too_large"`
|
||||
DurationExceeded int `json:"duration_exceeded"`
|
||||
TokenLimitExceeded int `json:"token_limit_exceeded"`
|
||||
}{
|
||||
DifferentSpeakers: differentSpeakers,
|
||||
},
|
||||
},
|
||||
ChunkingSummary: &chunking.Summary{
|
||||
ChunkCount: 3,
|
||||
MinEstimatedTokens: 10,
|
||||
MaxEstimatedTokens: 20,
|
||||
TotalEstimatedTokens: 45,
|
||||
TargetSections: &targetSections,
|
||||
MaxSectionTokens: 200,
|
||||
MinSectionTokens: 50,
|
||||
},
|
||||
RunOutput: &runner.RunOutput{
|
||||
ModuleResults: []runner.ModuleResult{
|
||||
{
|
||||
ModuleKey: "grammar",
|
||||
ModuleInstance: "grammar",
|
||||
ReplacementPolicy: proposals.ReplacementPolicyRequireUnique,
|
||||
Status: runner.ModuleStatusSuccess,
|
||||
ProposalCount: 2,
|
||||
ValidatorDecisions: []runner.ValidatorDecisionRecord{
|
||||
{
|
||||
ValidatorName: "proposal_shape",
|
||||
ProposalIndex: 1,
|
||||
Approved: true,
|
||||
ReasonCode: "approved",
|
||||
Message: "ok",
|
||||
DiagnosticArtifactPath: "diagnostics/validator.json",
|
||||
},
|
||||
},
|
||||
ValidatorRejected: []runner.ValidatorRejectedChange{
|
||||
{
|
||||
ValidatorName: "protected_term",
|
||||
ProposalIndex: 2,
|
||||
ModuleKey: "grammar",
|
||||
ModuleInstance: "grammar",
|
||||
TargetSegmentID: 7,
|
||||
OriginalText: "old",
|
||||
CorrectedText: "new",
|
||||
ReasonCode: "protected_term",
|
||||
Message: "blocked",
|
||||
},
|
||||
},
|
||||
AppliedChanges: []proposals.AppliedChange{
|
||||
{ProposalIndex: 1, TargetSegmentID: 7, OriginalText: "old", CorrectedText: "new"},
|
||||
},
|
||||
SkippedChanges: []proposals.SkippedChange{
|
||||
{ProposalIndex: 3, TargetSegmentID: 8, SkipReason: proposals.SkipReasonMissingSegment},
|
||||
},
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: completedAt,
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
if report.ReportMetadata.ReportSchemaName != reporting.DefaultProcessReportSchemaName ||
|
||||
report.ReportMetadata.ReportSchemaVersion != reporting.DefaultProcessReportSchemaVersion ||
|
||||
report.ReportMetadata.OutputSchema != "default" ||
|
||||
report.ReportMetadata.ConfigVersion == nil ||
|
||||
*report.ReportMetadata.ConfigVersion != configVersion {
|
||||
t.Fatalf("unexpected report metadata: %+v", report.ReportMetadata)
|
||||
}
|
||||
if report.Phase != "default_pipeline" || report.Operation != "process" || report.Status != "success" {
|
||||
t.Fatalf("unexpected process identity fields: phase=%q operation=%q status=%q", report.Phase, report.Operation, report.Status)
|
||||
}
|
||||
if report.Diagnostics == nil || report.Diagnostics.CorrectionLedgerPath != filepath.Join("tmp", "audita-run", diagnostics.ArtifactCorrectionLedger) {
|
||||
t.Fatalf("unexpected diagnostics metadata: %+v", report.Diagnostics)
|
||||
}
|
||||
if report.InputSegmentCount == nil || *report.InputSegmentCount != inputSegments ||
|
||||
report.NormalizationSkipped.DifferentSpeakers == nil ||
|
||||
*report.NormalizationSkipped.DifferentSpeakers != differentSpeakers {
|
||||
t.Fatalf("unexpected normalization summary: %+v", report)
|
||||
}
|
||||
if report.Chunking == nil || report.Chunking.ChunkCount != 3 || report.Chunking.TargetSections == nil || *report.Chunking.TargetSections != targetSections {
|
||||
t.Fatalf("unexpected chunking summary: %+v", report.Chunking)
|
||||
}
|
||||
if report.ModulesSummary == nil ||
|
||||
report.ModulesSummary.ModuleCount != 1 ||
|
||||
report.ModulesSummary.TotalAppliedChanges != 1 ||
|
||||
report.ModulesSummary.TotalSkippedChanges != 2 {
|
||||
t.Fatalf("unexpected modules summary: %+v", report.ModulesSummary)
|
||||
}
|
||||
if len(report.ModuleResults) != 1 ||
|
||||
len(report.ModuleResults[0].ValidatorDecisions) != 1 ||
|
||||
len(report.ModuleResults[0].ValidatorRejected) != 1 {
|
||||
t.Fatalf("unexpected module reports: %+v", report.ModuleResults)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildFailedReportPreservesErrorAndFailureSummary(t *testing.T) {
|
||||
startedAt := time.Date(2026, 5, 23, 10, 0, 0, 0, time.UTC)
|
||||
completedAt := startedAt.Add(time.Second)
|
||||
|
||||
report := Build(BuildInput{
|
||||
Status: "failed",
|
||||
TranscriptPath: "transcript.json",
|
||||
GlossaryPath: "glossary.yaml",
|
||||
Modules: []string{"grammar"},
|
||||
OutputSchema: "default",
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: completedAt,
|
||||
ErrorPhase: "module",
|
||||
ErrorMessage: "module failed",
|
||||
RunDirectoryPath: "run-dir",
|
||||
RunOutput: &runner.RunOutput{
|
||||
ModuleResults: []runner.ModuleResult{
|
||||
{
|
||||
ModuleKey: "grammar",
|
||||
ModuleInstance: "grammar",
|
||||
Status: runner.ModuleStatusFailed,
|
||||
ErrorMessage: "module failed",
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: completedAt,
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
if report.Status != "failed" || report.ErrorPhase != "module" || report.ErrorMessage != "module failed" {
|
||||
t.Fatalf("unexpected failure fields: %+v", report)
|
||||
}
|
||||
if report.Diagnostics == nil || report.Diagnostics.ErrorLogPath == "" {
|
||||
t.Fatalf("expected failure diagnostics metadata, got %+v", report.Diagnostics)
|
||||
}
|
||||
if report.ModulesSummary == nil || report.ModulesSummary.FailedModuleInstance != "grammar" {
|
||||
t.Fatalf("unexpected failed module summary: %+v", report.ModulesSummary)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"corrections"
|
||||
],
|
||||
"properties": {
|
||||
"corrections": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"original_text",
|
||||
"corrected_text",
|
||||
"confidence"
|
||||
],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
},
|
||||
"original_text": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"corrected_text": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"confidence": {
|
||||
"type": "number",
|
||||
"minimum": 0,
|
||||
"maximum": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"validations"
|
||||
],
|
||||
"properties": {
|
||||
"validations": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"correction_index",
|
||||
"approved",
|
||||
"confidence",
|
||||
"reason"
|
||||
],
|
||||
"properties": {
|
||||
"correction_index": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"approved": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"confidence": {
|
||||
"type": "number",
|
||||
"minimum": 0,
|
||||
"maximum": 1
|
||||
},
|
||||
"reason": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package responseschema
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"embed"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
@@ -20,6 +21,9 @@ const (
|
||||
schemaVersionV1 = "v1"
|
||||
)
|
||||
|
||||
//go:embed assets/*.json
|
||||
var schemaAssets embed.FS
|
||||
|
||||
// Schema describes one registered structured response schema.
|
||||
type Schema struct {
|
||||
ID string `json:"id"`
|
||||
@@ -39,17 +43,17 @@ func (s Schema) DiagnosticsMap() map[string]any {
|
||||
}
|
||||
|
||||
var registry = map[Key]Schema{
|
||||
CorrectionSetKey: mustBuildSchema(
|
||||
CorrectionSetKey: mustBuildSchemaFromAsset(
|
||||
correctionSetSchemaID,
|
||||
schemaVersionV1,
|
||||
"audita_correction_set_v1",
|
||||
[]byte(`{"type":"object","additionalProperties":false,"required":["corrections"],"properties":{"corrections":{"type":"array","items":{"type":"object","additionalProperties":false,"required":["id","original_text","corrected_text","confidence"],"properties":{"id":{"type":"integer","minimum":1},"original_text":{"type":"string","minLength":1},"corrected_text":{"type":"string","minLength":1},"confidence":{"type":"number","minimum":0,"maximum":1}}}}}}`),
|
||||
"assets/correction_set.v1.json",
|
||||
),
|
||||
ValidatorDecisionSetKey: mustBuildSchema(
|
||||
ValidatorDecisionSetKey: mustBuildSchemaFromAsset(
|
||||
validatorDecisionSchemaID,
|
||||
schemaVersionV1,
|
||||
"audita_validator_decision_set_v1",
|
||||
[]byte(`{"type":"object","additionalProperties":false,"required":["validations"],"properties":{"validations":{"type":"array","items":{"type":"object","additionalProperties":false,"required":["correction_index","approved","confidence","reason"],"properties":{"correction_index":{"type":"integer","minimum":0},"approved":{"type":"boolean"},"confidence":{"type":"number","minimum":0,"maximum":1},"reason":{"type":"string"}}}}}}`),
|
||||
"assets/validator_decision_set.v1.json",
|
||||
),
|
||||
}
|
||||
|
||||
@@ -93,6 +97,14 @@ func cloneSchema(in Schema) Schema {
|
||||
return out
|
||||
}
|
||||
|
||||
func mustBuildSchemaFromAsset(id string, version string, name string, assetPath string) Schema {
|
||||
rawSchema, err := schemaAssets.ReadFile(assetPath)
|
||||
if err != nil {
|
||||
panic(fmt.Sprintf("read response schema asset %q: %v", assetPath, err))
|
||||
}
|
||||
return mustBuildSchema(id, version, name, rawSchema)
|
||||
}
|
||||
|
||||
func mustBuildSchema(id string, version string, name string, rawSchema []byte) Schema {
|
||||
id = strings.TrimSpace(id)
|
||||
version = strings.TrimSpace(version)
|
||||
|
||||
@@ -1,14 +1,16 @@
|
||||
package responseschema
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const (
|
||||
expectedCorrectionSetSHA256 = "05f8ff3fa04f68115c0cb1859d2656f51aa5c0bae8ff2470b2d4f6f531953195"
|
||||
expectedValidatorDecisionSetSHA256 = "b73f4790b98fbb955f0aec5496dd8ce9a8fe14aa2f35c700b4b4e5634f106fd5"
|
||||
expectedCorrectionSetSHA256 = "b86a2dde38d7f440d26470fa8830167512bb0aa1b35e5fee5547057be583c388"
|
||||
expectedValidatorDecisionSetSHA256 = "2fe90d450e2595b57885aba91c8cc5cedf783dd36758ab430309e3eace401f54"
|
||||
)
|
||||
|
||||
func TestLookupKnownSchemas(t *testing.T) {
|
||||
@@ -47,6 +49,20 @@ func TestLookupUnknownSchema(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegisteredSchemasLoadReadableEmbeddedJSON(t *testing.T) {
|
||||
for _, schema := range Registered() {
|
||||
if len(schema.JSONSchema) == 0 {
|
||||
t.Fatalf("expected non-empty JSON schema for %q", schema.ID)
|
||||
}
|
||||
if !json.Valid(schema.JSONSchema) {
|
||||
t.Fatalf("expected valid JSON schema for %q", schema.ID)
|
||||
}
|
||||
if !bytes.Contains(schema.JSONSchema, []byte("\n ")) {
|
||||
t.Fatalf("expected readable formatted JSON schema for %q", schema.ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSchemaHashesMatchRegisteredJSON(t *testing.T) {
|
||||
expectedByKey := map[Key]string{
|
||||
CorrectionSetKey: expectedCorrectionSetSHA256,
|
||||
|
||||
Reference in New Issue
Block a user