Compare commits
8 Commits
76651333b1
...
4e4801dc98
| Author | SHA1 | Date | |
|---|---|---|---|
| 4e4801dc98 | |||
| 900ad74958 | |||
| 28d5201a69 | |||
| e5944f9875 | |||
| 6344fc91ba | |||
| 7f3a91cc9e | |||
| 72fb021453 | |||
| 40e8b54d3b |
305
README.md
305
README.md
@@ -1,305 +1,46 @@
|
|||||||
# Audita
|
# Audita
|
||||||
|
|
||||||
Audita is a transcript polishing CLI.
|
Audita is a CLI that polishes transcript JSON using glossary-aware and LLM-backed correction modules.
|
||||||
|
|
||||||
`audita process` validates transcript/glossary input, normalizes and chunks transcript segments, runs the default correction pipeline, and emits corrected transcript output plus machine-readable diagnostics and reports.
|
## Quickstart
|
||||||
|
|
||||||
## What Audita Does
|
Build:
|
||||||
|
|
||||||
Default module sequence:
|
|
||||||
- `glossary`
|
|
||||||
- `homophones`
|
|
||||||
- `glossary`
|
|
||||||
- `spoken_word`
|
|
||||||
- `grammar`
|
|
||||||
|
|
||||||
Pipeline behavior includes:
|
|
||||||
- glossary-backed domain/acoustic corrections
|
|
||||||
- conservative homophone and mistranscription corrections
|
|
||||||
- conservative spoken-word dysfluency cleanup with semantic guardrails
|
|
||||||
- grammar/punctuation/capitalization/formatting cleanup
|
|
||||||
- validator-chain enforcement before application
|
|
||||||
- malformed module-stage LLM payloads degrade to warnings/rejections instead of failing the run
|
|
||||||
- run reports and diagnostics artifacts with secret redaction
|
|
||||||
|
|
||||||
## Build and Install
|
|
||||||
|
|
||||||
Build a local binary:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go build -o ./bin/audita ./cmd/audita
|
go build -o ./bin/audita ./cmd/audita
|
||||||
```
|
```
|
||||||
|
|
||||||
Install into your Go bin directory:
|
Run the shortest useful command:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go install ./cmd/audita
|
audita process ./transcript.json --glossary ./glossary.yaml --output ./corrected.json
|
||||||
```
|
```
|
||||||
|
|
||||||
CLI help:
|
Notes:
|
||||||
|
- the transcript JSON path is required as a positional argument;
|
||||||
```sh
|
- `--glossary` is required;
|
||||||
audita --help
|
- without `--output`, corrected transcript JSON is written to stdout.
|
||||||
audita process --help
|
|
||||||
audita config --help
|
|
||||||
```
|
|
||||||
|
|
||||||
## Test
|
|
||||||
|
|
||||||
Run all tests:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./...
|
|
||||||
```
|
|
||||||
|
|
||||||
## Basic Usage
|
|
||||||
|
|
||||||
Required inputs:
|
|
||||||
- transcript JSON path (positional argument)
|
|
||||||
- `--glossary <glossary.yaml>`
|
|
||||||
|
|
||||||
Recommended run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
audita process transcript.json \
|
|
||||||
--glossary glossary.yaml \
|
|
||||||
--output corrected.json \
|
|
||||||
--report-json report.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Select an explicit output schema (default is `bare-segments`):
|
|
||||||
|
|
||||||
```sh
|
|
||||||
audita process transcript.json \
|
|
||||||
--glossary glossary.yaml \
|
|
||||||
--output-schema audita-v1 \
|
|
||||||
--output corrected.json \
|
|
||||||
--report-json report.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Recommended config-based run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
audita process transcript.json \
|
|
||||||
--glossary glossary.yaml \
|
|
||||||
--config audita.yml \
|
|
||||||
--output corrected.json \
|
|
||||||
--report-json report.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Explicit module override:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
audita process transcript.json \
|
|
||||||
--glossary glossary.yaml \
|
|
||||||
--modules glossary,homophones,grammar \
|
|
||||||
--output corrected.json \
|
|
||||||
--report-json report.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Optional transcript background context:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
audita process transcript.json \
|
|
||||||
--glossary glossary.yaml \
|
|
||||||
--transcript-description "Brief context that may help resolve ambiguous terms." \
|
|
||||||
--output corrected.json
|
|
||||||
```
|
|
||||||
|
|
||||||
The transcript description is background context only and does not override transcript content.
|
|
||||||
|
|
||||||
Write transcript JSON to stdout (no `--output`):
|
|
||||||
|
|
||||||
```sh
|
|
||||||
audita process transcript.json --glossary glossary.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
Control diagnostics location/retention:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
audita process transcript.json \
|
|
||||||
--glossary glossary.yaml \
|
|
||||||
--work-dir /tmp/audita \
|
|
||||||
--work-dir-retention auto \
|
|
||||||
--output corrected.json \
|
|
||||||
--report-json report.json
|
|
||||||
```
|
|
||||||
|
|
||||||
## Stdout/Stderr Contract
|
|
||||||
|
|
||||||
- With `--output`, stdout is expected to be empty on success.
|
|
||||||
- Without `--output`, stdout contains transcript JSON only on success.
|
|
||||||
- `--report-json` writes a file and is never printed to stdout.
|
|
||||||
- stderr is human-readable diagnostics/errors.
|
|
||||||
- successful runs remain quiet on stderr even when module warnings are recorded in report/diagnostics artifacts.
|
|
||||||
|
|
||||||
For subprocess orchestration guidance, see [`docs/subprocess-operations.md`](docs/subprocess-operations.md).
|
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
Precedence:
|
Audita loads defaults, optional file config, environment overrides, then CLI overrides.
|
||||||
1. defaults
|
|
||||||
2. config file (`--config`, `AUDITA_CONFIG`, or default search paths when present: `/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`)
|
|
||||||
3. environment (`AUDITA_*`)
|
|
||||||
4. CLI flags
|
|
||||||
|
|
||||||
Config commands:
|
Use these commands to validate and inspect config:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
audita config validate --config audita.yml
|
audita config validate --config ./audita.yml
|
||||||
audita config print-effective --config audita.yml
|
audita config print-effective --config ./audita.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
For full config-file schema and examples, see [`docs/configuration.md`](docs/configuration.md).
|
|
||||||
For output-schema details, see [`docs/architecture/output-schemas.md`](docs/architecture/output-schemas.md).
|
|
||||||
For built-in validator keys and chain definitions, see [`docs/architecture/validators.md`](docs/architecture/validators.md).
|
|
||||||
For embedded prompt assets and prompt metadata behavior, see [`docs/architecture/prompts.md`](docs/architecture/prompts.md).
|
|
||||||
For CLI/process compatibility guarantees, see [`docs/architecture/public-contract.md`](docs/architecture/public-contract.md).
|
|
||||||
|
|
||||||
### Modules
|
|
||||||
|
|
||||||
- `AUDITA_MODULES` (CSV)
|
|
||||||
- CLI: `--modules`
|
|
||||||
|
|
||||||
### Transcript Description
|
|
||||||
|
|
||||||
CLI:
|
|
||||||
- `--transcript-description`
|
|
||||||
|
|
||||||
Behavior:
|
|
||||||
- optional background context for proposal and LLM-validator prompts;
|
|
||||||
- trimmed and length-limited by CLI validation;
|
|
||||||
- does not override transcript content;
|
|
||||||
- no `AUDITA_*` environment variable is currently defined for this setting.
|
|
||||||
|
|
||||||
### Primary LLM
|
|
||||||
|
|
||||||
Environment:
|
|
||||||
- `AUDITA_LLM_API_KEY` (or `OPENROUTER_API_KEY` fallback)
|
|
||||||
- `AUDITA_MODEL`
|
|
||||||
- `AUDITA_BASE_URL`
|
|
||||||
- `AUDITA_LLM_TIMEOUT_SECONDS`
|
|
||||||
- `AUDITA_MAX_RETRIES`
|
|
||||||
|
|
||||||
CLI:
|
|
||||||
- `--llm-api-key`
|
|
||||||
- `--model`
|
|
||||||
- `--base-url`
|
|
||||||
- `--llm-timeout-seconds`
|
|
||||||
- `--max-retries`
|
|
||||||
|
|
||||||
### Validation LLM
|
|
||||||
|
|
||||||
Environment:
|
|
||||||
- `AUDITA_VALIDATION_LLM_API_KEY`
|
|
||||||
- `AUDITA_VALIDATION_MODEL`
|
|
||||||
- `AUDITA_VALIDATION_BASE_URL`
|
|
||||||
- `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS`
|
|
||||||
- `AUDITA_VALIDATION_MAX_RETRIES`
|
|
||||||
- `AUDITA_VALIDATION_LLM_CONCURRENCY`
|
|
||||||
- `AUDITA_VALIDATION_MAX_PROMPT_TOKENS`
|
|
||||||
|
|
||||||
CLI:
|
|
||||||
- `--validation-llm-api-key`
|
|
||||||
- `--validation-model`
|
|
||||||
- `--validation-base-url`
|
|
||||||
- `--validation-llm-timeout-seconds`
|
|
||||||
- `--validation-max-retries`
|
|
||||||
- `--validation-llm-concurrency`
|
|
||||||
- `--validation-max-prompt-tokens`
|
|
||||||
|
|
||||||
### LLM Concurrency
|
|
||||||
|
|
||||||
Environment:
|
|
||||||
- `AUDITA_TOTAL_LLM_CONCURRENCY`
|
|
||||||
- `AUDITA_PROPOSAL_LLM_CONCURRENCY`
|
|
||||||
- `AUDITA_VALIDATION_LLM_CONCURRENCY`
|
|
||||||
- `AUDITA_LLM_CONCURRENCY` (legacy alias for `AUDITA_TOTAL_LLM_CONCURRENCY`)
|
|
||||||
|
|
||||||
CLI:
|
|
||||||
- `--total-llm-concurrency`
|
|
||||||
- `--proposal-llm-concurrency`
|
|
||||||
- `--validation-llm-concurrency`
|
|
||||||
- `--llm-concurrency` (legacy alias for `--total-llm-concurrency`)
|
|
||||||
|
|
||||||
Behavior:
|
|
||||||
- all proposal and validation LLM calls are bounded by total LLM concurrency
|
|
||||||
- proposal LLM calls are additionally bounded by proposal LLM concurrency
|
|
||||||
- when validation concurrency is unset, it inherits total LLM concurrency
|
|
||||||
- when explicitly set, proposal and validation concurrency must each be `<= total-llm-concurrency`
|
|
||||||
- canonical total settings win when both canonical and legacy alias settings are provided at the same precedence layer
|
|
||||||
|
|
||||||
### Confidence Thresholds
|
|
||||||
|
|
||||||
Environment:
|
|
||||||
- `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD`
|
|
||||||
- `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD`
|
|
||||||
- `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD`
|
|
||||||
- `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD`
|
|
||||||
|
|
||||||
CLI:
|
|
||||||
- `--glossary-confidence-threshold`
|
|
||||||
- `--homophones-confidence-threshold`
|
|
||||||
- `--spoken-word-confidence-threshold`
|
|
||||||
- `--grammar-confidence-threshold`
|
|
||||||
|
|
||||||
### Normalization and Chunking
|
|
||||||
|
|
||||||
Environment:
|
|
||||||
- `AUDITA_NORMALIZE_MAX_SEGMENT_GAP`
|
|
||||||
- `AUDITA_NORMALIZE_ELLIPSIS_GAP`
|
|
||||||
- `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION`
|
|
||||||
- `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS`
|
|
||||||
- `AUDITA_MAX_SECTION_TOKENS`
|
|
||||||
- `AUDITA_MIN_SECTION_TOKENS`
|
|
||||||
- `AUDITA_TARGET_SECTIONS`
|
|
||||||
|
|
||||||
CLI:
|
|
||||||
- `--normalize-max-segment-gap`
|
|
||||||
- `--normalize-ellipsis-gap`
|
|
||||||
- `--normalize-max-segment-duration`
|
|
||||||
- `--normalize-max-segment-tokens`
|
|
||||||
- `--max-section-tokens`
|
|
||||||
- `--min-section-tokens`
|
|
||||||
- `--target-sections`
|
|
||||||
|
|
||||||
### Work Directory
|
|
||||||
|
|
||||||
Environment:
|
|
||||||
- `AUDITA_WORK_DIR`
|
|
||||||
- `AUDITA_WORK_DIR_RETENTION` (`auto`, `always`, `never`)
|
|
||||||
|
|
||||||
CLI:
|
|
||||||
- `--work-dir`
|
|
||||||
- `--work-dir-retention`
|
|
||||||
|
|
||||||
Retention behavior:
|
|
||||||
- `always`: keep all run directories
|
|
||||||
- `never`: keep successful run directories
|
|
||||||
- `auto`: keep failed runs and successful runs with skipped/rejected corrections
|
|
||||||
|
|
||||||
## Reports and Diagnostics
|
|
||||||
|
|
||||||
Per-run diagnostics include:
|
|
||||||
- source transcript artifacts
|
|
||||||
- normalized transcript artifact
|
|
||||||
- normalization summary
|
|
||||||
- chunking summary
|
|
||||||
- utilization diagnostics summary
|
|
||||||
- correction ledger
|
|
||||||
- invocation metadata
|
|
||||||
- redacted effective config
|
|
||||||
- module/validator prompt-response diagnostics
|
|
||||||
- `report.json`
|
|
||||||
- `error.log` on failure
|
|
||||||
|
|
||||||
Optional external report output:
|
|
||||||
- `--report-json <path>`
|
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
- Architecture: [`docs/architecture.md`](docs/architecture.md)
|
- CLI reference: [`docs/cli.md`](docs/cli.md)
|
||||||
- Diagnostics: [`docs/diagnostics.md`](docs/diagnostics.md)
|
- Configuration reference: [`docs/config.md`](docs/config.md)
|
||||||
- Structured LLM adapter: [`docs/structured-llm.md`](docs/structured-llm.md)
|
- Operations guide: [`docs/operations.md`](docs/operations.md)
|
||||||
- Subprocess operations: [`docs/subprocess-operations.md`](docs/subprocess-operations.md)
|
- Troubleshooting: [`docs/troubleshooting.md`](docs/troubleshooting.md)
|
||||||
- Release checklist: [`docs/release-checklist.md`](docs/release-checklist.md)
|
- Subprocess integration: [`docs/integrations/subprocess.md`](docs/integrations/subprocess.md)
|
||||||
|
- OpenAI-compatible LLM integration: [`docs/integrations/openai-compatible-llm.md`](docs/integrations/openai-compatible-llm.md)
|
||||||
|
- Transcript and glossary file integration: [`docs/integrations/transcript-glossary-files.md`](docs/integrations/transcript-glossary-files.md)
|
||||||
|
- Development workflow: [`docs/policy/development.md`](docs/policy/development.md)
|
||||||
|
- Architecture policy: [`docs/policy/architecture.md`](docs/policy/architecture.md)
|
||||||
|
- Documentation policy: [`docs/policy/documentation.md`](docs/policy/documentation.md)
|
||||||
|
|||||||
@@ -1,14 +0,0 @@
|
|||||||
# Audita Architecture Index
|
|
||||||
|
|
||||||
This file is the entrypoint for architecture documentation.
|
|
||||||
|
|
||||||
Core architecture overview:
|
|
||||||
- [Architecture Overview](./architecture/architecture.md)
|
|
||||||
|
|
||||||
Focused architecture contracts:
|
|
||||||
- [Public Contract](./architecture/public-contract.md)
|
|
||||||
- [Diagnostics](./architecture/diagnostics.md)
|
|
||||||
- [Structured LLM](./architecture/structured-llm.md)
|
|
||||||
- [Validators](./architecture/validators.md)
|
|
||||||
- [Prompts](./architecture/prompts.md)
|
|
||||||
- [Output Schemas](./architecture/output-schemas.md)
|
|
||||||
@@ -1,153 +0,0 @@
|
|||||||
# Audita Architecture
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
This document describes the production architecture implemented in this repository today.
|
|
||||||
|
|
||||||
Audita is a single-process Go CLI that:
|
|
||||||
- loads effective runtime configuration;
|
|
||||||
- reads transcript and glossary inputs;
|
|
||||||
- normalizes and sections transcripts;
|
|
||||||
- runs a built-in module pipeline with validator chains;
|
|
||||||
- writes transcript output and run diagnostics.
|
|
||||||
|
|
||||||
## Runtime entrypoints
|
|
||||||
Primary CLI commands:
|
|
||||||
- `audita process <transcript.json> --glossary <glossary.yaml> [flags]`
|
|
||||||
- `audita config validate --config <config.yml>`
|
|
||||||
- `audita config print-effective [--config <config.yml>]`
|
|
||||||
|
|
||||||
Command ownership lives in `internal/cli/run.go`.
|
|
||||||
|
|
||||||
## Configuration model
|
|
||||||
`internal/core/config` owns defaults, file parsing, environment overrides, CLI overrides, and validation.
|
|
||||||
|
|
||||||
Effective-config loading for `process` and `config print-effective` is centralized in:
|
|
||||||
- `ResolveConfigPath`
|
|
||||||
- `LoadEffectiveConfig`
|
|
||||||
|
|
||||||
Effective precedence for `audita process`:
|
|
||||||
1. defaults
|
|
||||||
2. config file
|
|
||||||
3. environment overrides
|
|
||||||
4. CLI overrides
|
|
||||||
|
|
||||||
`audita config validate` is intentionally file-only validation:
|
|
||||||
- load versioned file;
|
|
||||||
- apply onto defaults;
|
|
||||||
- validate;
|
|
||||||
- do not apply environment overrides.
|
|
||||||
|
|
||||||
Supported module and output-schema keys are validated through shared catalogs:
|
|
||||||
- module keys: `internal/core/modulecatalog`
|
|
||||||
- output schemas: `internal/core/outputschema`
|
|
||||||
|
|
||||||
## Pipeline and module orchestration
|
|
||||||
The built-in module sequence is configured in runtime config and executed by `internal/framework/runner` through resolved module specs.
|
|
||||||
|
|
||||||
Current default sequence:
|
|
||||||
- `glossary`
|
|
||||||
- `homophones`
|
|
||||||
- `glossary`
|
|
||||||
- `spoken_word`
|
|
||||||
- `grammar`
|
|
||||||
|
|
||||||
Execution behavior:
|
|
||||||
- modules execute serially over the working transcript;
|
|
||||||
- section proposal work can run concurrently within a module;
|
|
||||||
- validator execution happens on generated proposals before application;
|
|
||||||
- approved proposals are applied once per module in deterministic proposal-index order.
|
|
||||||
|
|
||||||
Production modules remain separate packages:
|
|
||||||
- `internal/modules/glossary`
|
|
||||||
- `internal/modules/homophones`
|
|
||||||
- `internal/modules/spoken_word`
|
|
||||||
- `internal/modules/grammar`
|
|
||||||
|
|
||||||
## Proposal generation and prompt context
|
|
||||||
Shared proposal plumbing is centralized in `internal/framework/proposal_generation`.
|
|
||||||
|
|
||||||
Module packages provide:
|
|
||||||
- module identity and replacement policy;
|
|
||||||
- module-specific prompt message building;
|
|
||||||
- built-in validator chain selection.
|
|
||||||
|
|
||||||
Shared prompt payload helpers are in `internal/framework/promptcontext`.
|
|
||||||
|
|
||||||
## Validator architecture
|
|
||||||
Built-in validator construction and chain composition live in `internal/validators`.
|
|
||||||
|
|
||||||
Shared validator runtime mechanics live in `internal/framework/validators`.
|
|
||||||
|
|
||||||
Execution class metadata (deterministic vs LLM-backed) is centralized in `internal/validators/metadata` and used for ordering and reporting classification.
|
|
||||||
|
|
||||||
## Structured LLM boundary
|
|
||||||
All production LLM calls go through the internal contract:
|
|
||||||
- `contracts.StructuredLLMClient`
|
|
||||||
- `CompleteStructured(ctx, req, out)`
|
|
||||||
|
|
||||||
The OpenAI-compatible HTTP adapter is implemented in `internal/framework/llm`.
|
|
||||||
|
|
||||||
Structured response schemas are registered in `internal/framework/responseschema` and attached to requests via `response_format` metadata.
|
|
||||||
|
|
||||||
Malformed structured-output detection is centralized in `internal/framework/structuredoutput` and reused by proposal generation and validator execution so downgrade behavior stays consistent.
|
|
||||||
|
|
||||||
## Stage naming and diagnostics metadata
|
|
||||||
Diagnostics stage naming is centralized in `internal/framework/stagename`:
|
|
||||||
- module proposal stage names;
|
|
||||||
- proposal-generation stage names;
|
|
||||||
- validator batch stage names.
|
|
||||||
|
|
||||||
Prompt metadata and response-schema metadata each expose canonical diagnostics maps via:
|
|
||||||
- `prompts.Metadata.DiagnosticsMap()`
|
|
||||||
- `responseschema.Schema.DiagnosticsMap()`
|
|
||||||
|
|
||||||
## Diagnostics and reporting
|
|
||||||
Run-directory artifacts are owned by `internal/core/diagnostics`.
|
|
||||||
|
|
||||||
Stable artifact names are centralized constants (for example transcript artifacts, `invocation.json`, `effective-config.json`, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, `error.log`).
|
|
||||||
|
|
||||||
Report diagnostics path metadata is constructed through `BuildDiagnosticsMetadata`, which keeps run-directory artifact references consistent between success and failure reports.
|
|
||||||
|
|
||||||
## Secret redaction
|
|
||||||
Redaction responsibilities are split by concern:
|
|
||||||
- structural config redaction: `config.Config.Redacted()`
|
|
||||||
- byte/string payload redaction for diagnostics and surfaced errors: framework redaction utilities.
|
|
||||||
|
|
||||||
Configured LLM secret extraction is centralized in `llm.ConfiguredSecrets(cfg)` and reused across proposal and validator diagnostics paths.
|
|
||||||
|
|
||||||
## Output contracts
|
|
||||||
Transcript output schema selection is owned by `internal/core/outputschema`.
|
|
||||||
|
|
||||||
Supported schemas:
|
|
||||||
- `bare-segments`
|
|
||||||
- `audita-v1`
|
|
||||||
|
|
||||||
Unknown schema keys fail validation and runtime resolution.
|
|
||||||
|
|
||||||
## Key package map
|
|
||||||
Core packages:
|
|
||||||
- `internal/core/config`
|
|
||||||
- `internal/core/schema`
|
|
||||||
- `internal/core/normalization`
|
|
||||||
- `internal/core/chunking`
|
|
||||||
- `internal/core/diagnostics`
|
|
||||||
- `internal/core/reporting`
|
|
||||||
- `internal/core/modulecatalog`
|
|
||||||
- `internal/core/outputschema`
|
|
||||||
|
|
||||||
Framework packages:
|
|
||||||
- `internal/framework/contracts`
|
|
||||||
- `internal/framework/proposals`
|
|
||||||
- `internal/framework/proposal_generation`
|
|
||||||
- `internal/framework/promptcontext`
|
|
||||||
- `internal/framework/runner`
|
|
||||||
- `internal/framework/validators`
|
|
||||||
- `internal/framework/llm`
|
|
||||||
- `internal/framework/responseschema`
|
|
||||||
- `internal/framework/stagename`
|
|
||||||
- `internal/framework/structuredoutput`
|
|
||||||
|
|
||||||
Domain packages:
|
|
||||||
- `internal/modules/*`
|
|
||||||
- `internal/validators/*`
|
|
||||||
- `internal/prompts`
|
|
||||||
@@ -1,104 +0,0 @@
|
|||||||
# Audita Diagnostics
|
|
||||||
|
|
||||||
This document describes the run-directory diagnostics artifacts produced by `audita process`.
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Diagnostics provide machine-readable run context and execution artifacts for:
|
|
||||||
- failure debugging;
|
|
||||||
- validator/correction review;
|
|
||||||
- post-run performance analysis.
|
|
||||||
|
|
||||||
Diagnostics are written under the configured work directory (`--work-dir`) when run-directory initialization succeeds.
|
|
||||||
|
|
||||||
## Core artifacts
|
|
||||||
|
|
||||||
Typical artifacts in each run directory:
|
|
||||||
- `source-transcript.json`
|
|
||||||
- `source-transcript-parsed.json`
|
|
||||||
- `normalized-transcript.json`
|
|
||||||
- `normalization-summary.json`
|
|
||||||
- `chunking-summary.json`
|
|
||||||
- `invocation.json`
|
|
||||||
- `effective-config.json` (redacted)
|
|
||||||
- module/validator LLM interaction artifacts
|
|
||||||
- `report.json`
|
|
||||||
- `error.log` on failure
|
|
||||||
|
|
||||||
## Utilization diagnostics artifact
|
|
||||||
|
|
||||||
Artifact:
|
|
||||||
- `utilization-diagnostics.json`
|
|
||||||
|
|
||||||
High-level fields:
|
|
||||||
- `effective_concurrency`:
|
|
||||||
- total/proposal/validation LLM concurrency limits in effect.
|
|
||||||
- `run_timing`:
|
|
||||||
- run wall time;
|
|
||||||
- scheduler queue wait time;
|
|
||||||
- LLM execution time;
|
|
||||||
- deterministic validator time;
|
|
||||||
- max/average in-flight LLM calls.
|
|
||||||
- `llm_calls`:
|
|
||||||
- total proposal and validation LLM call counts.
|
|
||||||
- `modules`:
|
|
||||||
- module-level timing summaries.
|
|
||||||
- `validators`:
|
|
||||||
- per-validator timing summaries keyed by stable validator key.
|
|
||||||
|
|
||||||
## Correction ledger artifact
|
|
||||||
|
|
||||||
Artifact:
|
|
||||||
- `correction-ledger.json`
|
|
||||||
|
|
||||||
Ledger records are flattened review entries derived from module results and include:
|
|
||||||
- module/proposal identity (`module_key`, `module_instance`, `proposal_index`, `segment_id`);
|
|
||||||
- correction text fields and replacement policy when available;
|
|
||||||
- disposition:
|
|
||||||
- `applied`
|
|
||||||
- `rejected`
|
|
||||||
- `skipped`
|
|
||||||
- `failed`
|
|
||||||
- stable reason codes/messages;
|
|
||||||
- deterministic and LLM validator decision snapshots using stable validator keys.
|
|
||||||
|
|
||||||
Validator rejection and proposal-application skip are distinct dispositions.
|
|
||||||
Module warnings are reported in module results and diagnostics metadata, but do not create standalone correction-ledger rows.
|
|
||||||
|
|
||||||
## Report references
|
|
||||||
|
|
||||||
`report.json` and optional `--report-json` output include diagnostics metadata paths for:
|
|
||||||
- utilization diagnostics artifact;
|
|
||||||
- correction ledger artifact;
|
|
||||||
- existing transcript/normalization/chunking/invocation/effective-config artifacts.
|
|
||||||
|
|
||||||
Module report entries also include warning records for malformed proposal-generation payloads and malformed validator batches.
|
|
||||||
|
|
||||||
## Retention behavior
|
|
||||||
|
|
||||||
Run-directory retention follows configured policy:
|
|
||||||
- `always`: keep all run directories;
|
|
||||||
- `never`: keep successful run directories;
|
|
||||||
- `auto`: keep failed runs and successful runs with skipped/rejected corrections.
|
|
||||||
|
|
||||||
## Redaction guarantees
|
|
||||||
|
|
||||||
API keys and other configured secrets are redacted from:
|
|
||||||
- `effective-config.json`;
|
|
||||||
- LLM interaction diagnostics artifacts;
|
|
||||||
- reports and surfaced errors.
|
|
||||||
|
|
||||||
## Debugging guide
|
|
||||||
|
|
||||||
When debugging:
|
|
||||||
- slow runs:
|
|
||||||
- inspect `utilization-diagnostics.json` (`run_timing`, `modules`, `validators`, in-flight metrics).
|
|
||||||
- validator rejections:
|
|
||||||
- inspect `correction-ledger.json` rejected entries and matching validator decisions;
|
|
||||||
- inspect validator response diagnostics payloads.
|
|
||||||
- module warnings:
|
|
||||||
- inspect module `warnings` entries in `report.json` or `--report-json`;
|
|
||||||
- follow any diagnostic artifact path on the warning to the recorded error/response payload.
|
|
||||||
- application skips:
|
|
||||||
- inspect `correction-ledger.json` skipped entries and skip reason codes;
|
|
||||||
- compare with validator decisions to distinguish validation rejection vs apply-time skip.
|
|
||||||
@@ -1,88 +0,0 @@
|
|||||||
# Audita Output Schemas
|
|
||||||
|
|
||||||
This document describes the built-in transcript output schema registry used by `audita process`.
|
|
||||||
|
|
||||||
## Supported schema names
|
|
||||||
|
|
||||||
### `bare-segments`
|
|
||||||
|
|
||||||
Status:
|
|
||||||
- implemented
|
|
||||||
- default output schema
|
|
||||||
|
|
||||||
Shape:
|
|
||||||
- top-level JSON array of transcript segments
|
|
||||||
|
|
||||||
Segment fields:
|
|
||||||
- `id`
|
|
||||||
- `speaker`
|
|
||||||
- `start`
|
|
||||||
- `end`
|
|
||||||
- `text`
|
|
||||||
- optional `categories`
|
|
||||||
|
|
||||||
Compatibility:
|
|
||||||
- this preserves the long-standing output shape used by existing consumers.
|
|
||||||
|
|
||||||
### `audita-v1`
|
|
||||||
|
|
||||||
Status:
|
|
||||||
- implemented
|
|
||||||
|
|
||||||
Shape:
|
|
||||||
- top-level JSON object:
|
|
||||||
- `schema`: `"audita-v1"`
|
|
||||||
- `version`: `"v1"`
|
|
||||||
- `segments`: transcript segment array
|
|
||||||
|
|
||||||
Segment fields inside `segments` match `bare-segments` segment fields.
|
|
||||||
|
|
||||||
Compatibility:
|
|
||||||
- this is the Audita-native object format with explicit schema/version metadata.
|
|
||||||
|
|
||||||
### `seriatim-intermediate`
|
|
||||||
|
|
||||||
Status:
|
|
||||||
- deferred / not implemented
|
|
||||||
|
|
||||||
Current behavior:
|
|
||||||
- selecting `seriatim-intermediate` fails clearly as an unsupported output schema.
|
|
||||||
|
|
||||||
Reason:
|
|
||||||
- a concrete, repository-backed contract for this schema has not been finalized yet.
|
|
||||||
|
|
||||||
## Selection
|
|
||||||
|
|
||||||
Choose output schema with CLI:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
audita process <transcript.json> --glossary <glossary.yaml> --output-schema audita-v1
|
|
||||||
```
|
|
||||||
|
|
||||||
Or in file config:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
version: 1
|
|
||||||
output:
|
|
||||||
schema: audita-v1
|
|
||||||
```
|
|
||||||
|
|
||||||
Precedence remains:
|
|
||||||
1. defaults
|
|
||||||
2. file config
|
|
||||||
3. environment overrides
|
|
||||||
4. CLI overrides
|
|
||||||
|
|
||||||
`--output-schema` overrides `output.schema` when both are supplied.
|
|
||||||
|
|
||||||
## Output routing behavior
|
|
||||||
|
|
||||||
- With `--output`, transcript JSON is written to file using the selected schema and stdout stays empty on success.
|
|
||||||
- Without `--output`, stdout contains transcript JSON only, using the selected schema.
|
|
||||||
- `--report-json` writes report JSON to file and does not write report payloads to stdout.
|
|
||||||
|
|
||||||
## Backward-compatibility expectations
|
|
||||||
|
|
||||||
- default schema stays `bare-segments` for compatibility unless explicitly changed in a future breaking release;
|
|
||||||
- supported schema names are treated as stable public contract values;
|
|
||||||
- unsupported schema names fail before output write.
|
|
||||||
@@ -1,118 +0,0 @@
|
|||||||
# Audita Prompts
|
|
||||||
|
|
||||||
This document describes Audita's built-in embedded prompt assets and prompt registry behavior.
|
|
||||||
|
|
||||||
## Why embedded prompt assets
|
|
||||||
|
|
||||||
Audita embeds production prompt text into the binary so runtime behavior is:
|
|
||||||
- deterministic;
|
|
||||||
- auditable;
|
|
||||||
- dependency-light;
|
|
||||||
- not dependent on external prompt files at execution time.
|
|
||||||
|
|
||||||
Prompt text is authored as Markdown assets and rendered by Go code using typed template data.
|
|
||||||
|
|
||||||
## Built-in prompt registry
|
|
||||||
|
|
||||||
The prompt registry lives in `internal/prompts` and is responsible for:
|
|
||||||
- loading embedded prompt assets;
|
|
||||||
- registering stable prompt IDs and versions;
|
|
||||||
- recording prompt source metadata;
|
|
||||||
- computing deterministic SHA-256 source hashes;
|
|
||||||
- rendering system/user prompts with strict missing-key failures.
|
|
||||||
|
|
||||||
Current prompt source behavior:
|
|
||||||
- built-in embedded prompts only (`prompt_source = builtin`).
|
|
||||||
- filesystem prompt overrides are not supported.
|
|
||||||
|
|
||||||
## Built-in prompt IDs
|
|
||||||
|
|
||||||
Module proposal prompts:
|
|
||||||
- `modules.glossary.proposal`
|
|
||||||
- `modules.homophones.proposal`
|
|
||||||
- `modules.spoken_word.proposal`
|
|
||||||
- `modules.grammar.proposal`
|
|
||||||
|
|
||||||
LLM-backed validator prompts:
|
|
||||||
- `validators.spoken_form_plausibility`
|
|
||||||
- `validators.meaning_reversal_review`
|
|
||||||
- `validators.editorial_review`
|
|
||||||
- `validators.grammar_review`
|
|
||||||
- `validators.spoken_word_review`
|
|
||||||
|
|
||||||
## Prompt version semantics
|
|
||||||
|
|
||||||
Current built-in prompt version value is `v1`.
|
|
||||||
|
|
||||||
Version is a stable metadata identifier for diagnostics and debugging. It is not a dynamic prompt-selection mechanism.
|
|
||||||
|
|
||||||
## Prompt hash semantics
|
|
||||||
|
|
||||||
Each registered prompt includes a deterministic SHA-256 hash of embedded source text.
|
|
||||||
|
|
||||||
Hash purpose:
|
|
||||||
- identify exact prompt source used in a run;
|
|
||||||
- support diagnostics reproducibility and change auditing.
|
|
||||||
|
|
||||||
Current hash scope:
|
|
||||||
- source prompt text (system + user assets for a registered prompt), not a runtime secret-bearing payload.
|
|
||||||
|
|
||||||
## Template rendering behavior
|
|
||||||
|
|
||||||
Prompt rendering uses Go `text/template` with typed template data from module/validator builders.
|
|
||||||
|
|
||||||
Missing-key behavior:
|
|
||||||
- rendering uses missing-key errors;
|
|
||||||
- missing/renamed template fields fail quickly instead of silently producing incomplete prompts.
|
|
||||||
|
|
||||||
Go code still owns:
|
|
||||||
- structured request/response models;
|
|
||||||
- response schema selection;
|
|
||||||
- transcript/glossary/payload formatting;
|
|
||||||
- module and validator selection;
|
|
||||||
- diagnostics wiring.
|
|
||||||
|
|
||||||
## Shared prompt hardening policy
|
|
||||||
|
|
||||||
A shared hardening fragment is embedded once and included in every module proposal prompt and every LLM-validator prompt.
|
|
||||||
|
|
||||||
Hardening policy includes:
|
|
||||||
- transcript text is untrusted data;
|
|
||||||
- glossary entries and transcript descriptions are reference data, not instructions;
|
|
||||||
- instructions found inside transcript text must not be obeyed;
|
|
||||||
- model must perform only the requested correction/validation task;
|
|
||||||
- no invention of facts, names, events, motivations, speaker intent, or corrections;
|
|
||||||
- transcript remains the source of truth.
|
|
||||||
|
|
||||||
## Transcript description behavior
|
|
||||||
|
|
||||||
Transcript description remains background-only prompt context:
|
|
||||||
- it may help interpret ambiguous terms;
|
|
||||||
- it is explicitly non-authoritative and must not override transcript content;
|
|
||||||
- empty descriptions do not render awkward blank context sections.
|
|
||||||
|
|
||||||
Generated transcript descriptions are not implemented in this workstream.
|
|
||||||
|
|
||||||
## Diagnostics and report metadata boundaries
|
|
||||||
|
|
||||||
Current metadata flow:
|
|
||||||
- proposal-generation diagnostics request metadata includes prompt metadata;
|
|
||||||
- LLM-validator diagnostics request metadata includes prompt metadata.
|
|
||||||
|
|
||||||
Prompt metadata fields used in diagnostics:
|
|
||||||
- `prompt_id`
|
|
||||||
- `prompt_version`
|
|
||||||
- `prompt_source`
|
|
||||||
- `embedded_path`
|
|
||||||
- `sha256`
|
|
||||||
|
|
||||||
Current boundary:
|
|
||||||
- detailed prompt metadata is diagnostics-first;
|
|
||||||
- broad report-level prompt registries/ledgers are deferred.
|
|
||||||
|
|
||||||
## 1.0 boundary
|
|
||||||
|
|
||||||
Not implemented for 1.0 in this workstream:
|
|
||||||
- filesystem prompt overrides;
|
|
||||||
- user-configurable prompt selection;
|
|
||||||
- external prompt directories.
|
|
||||||
@@ -1,121 +0,0 @@
|
|||||||
# Audita Public Contract
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
This document defines stability expectations for Audita's external runtime interfaces.
|
|
||||||
|
|
||||||
Covered interfaces:
|
|
||||||
- CLI commands and major flags;
|
|
||||||
- versioned config behavior and precedence;
|
|
||||||
- transcript/glossary input forms;
|
|
||||||
- output schema selection;
|
|
||||||
- report schema metadata;
|
|
||||||
- diagnostics artifact path metadata;
|
|
||||||
- stdout/stderr and exit-code behavior;
|
|
||||||
- redaction guarantees.
|
|
||||||
|
|
||||||
## CLI contract
|
|
||||||
Stable commands:
|
|
||||||
- `audita process`
|
|
||||||
- `audita config validate`
|
|
||||||
- `audita config print-effective`
|
|
||||||
|
|
||||||
Stable high-value `process` flags:
|
|
||||||
- `--config`
|
|
||||||
- `--glossary`
|
|
||||||
- `--output`
|
|
||||||
- `--report-json`
|
|
||||||
- `--modules`
|
|
||||||
- `--output-schema`
|
|
||||||
|
|
||||||
## Config contract
|
|
||||||
Supported config format:
|
|
||||||
- YAML;
|
|
||||||
- `version: 1`;
|
|
||||||
- strict unknown-field rejection.
|
|
||||||
|
|
||||||
Path resolution for `process` and `config print-effective`:
|
|
||||||
1. `--config`
|
|
||||||
2. `AUDITA_CONFIG`
|
|
||||||
3. `/usr/local/etc/audita/config.yml`
|
|
||||||
4. `/etc/audita/config.yml`
|
|
||||||
|
|
||||||
Missing explicit path is an error. Missing default paths is non-fatal.
|
|
||||||
|
|
||||||
Precedence for `process`:
|
|
||||||
1. defaults
|
|
||||||
2. file config
|
|
||||||
3. environment overrides
|
|
||||||
4. CLI overrides
|
|
||||||
|
|
||||||
`config validate` remains file-only validation (defaults + file config; no env overrides).
|
|
||||||
|
|
||||||
Module and output-schema keys are validated against built-in catalogs. Unknown keys fail validation.
|
|
||||||
|
|
||||||
## Input contract
|
|
||||||
Supported transcript JSON top-level forms:
|
|
||||||
- array of segments
|
|
||||||
- object with `segments` array
|
|
||||||
|
|
||||||
Supported glossary YAML form:
|
|
||||||
- top-level `glossary` list with required entry fields validated by schema parsing.
|
|
||||||
|
|
||||||
## Output schema contract
|
|
||||||
Supported transcript output schemas:
|
|
||||||
- `bare-segments` (default)
|
|
||||||
- `audita-v1`
|
|
||||||
|
|
||||||
Unknown schema keys fail before output write.
|
|
||||||
|
|
||||||
## Report metadata contract
|
|
||||||
Process reports include stable report metadata fields:
|
|
||||||
- `report_schema_name`
|
|
||||||
- `report_schema_version`
|
|
||||||
- `output_schema`
|
|
||||||
- `config_version` (when file config is loaded)
|
|
||||||
|
|
||||||
Current values:
|
|
||||||
- `report_schema_name = audita-process-report`
|
|
||||||
- `report_schema_version = v1`
|
|
||||||
|
|
||||||
`--report-json` output and run-directory `report.json` use the same report schema metadata.
|
|
||||||
|
|
||||||
Validator decision/rejection records use stable validator keys via `validator_name`.
|
|
||||||
|
|
||||||
## Diagnostics metadata contract
|
|
||||||
When run-directory initialization succeeds, diagnostics metadata paths reference stable artifacts, including:
|
|
||||||
- transcript and normalization artifacts;
|
|
||||||
- chunking summary;
|
|
||||||
- invocation metadata;
|
|
||||||
- redacted effective config;
|
|
||||||
- utilization diagnostics;
|
|
||||||
- correction ledger;
|
|
||||||
- `error.log` on failures.
|
|
||||||
|
|
||||||
LLM interaction diagnostics include stable prompt and structured-schema identifiers where applicable.
|
|
||||||
|
|
||||||
## Stdout/stderr and exit codes
|
|
||||||
Success:
|
|
||||||
- with `--output`, stdout is empty;
|
|
||||||
- without `--output`, stdout contains transcript JSON only;
|
|
||||||
- report JSON is not written to stdout.
|
|
||||||
|
|
||||||
Failures:
|
|
||||||
- nonzero exit;
|
|
||||||
- human-readable stderr summary;
|
|
||||||
- diagnostics directory path on stderr when available.
|
|
||||||
|
|
||||||
Exit codes:
|
|
||||||
- `0` success
|
|
||||||
- nonzero failure
|
|
||||||
|
|
||||||
## Redaction contract
|
|
||||||
Configured secrets are redacted from:
|
|
||||||
- effective config outputs;
|
|
||||||
- diagnostics artifacts;
|
|
||||||
- report artifacts;
|
|
||||||
- surfaced adapter/runtime errors.
|
|
||||||
|
|
||||||
## Compatibility policy
|
|
||||||
Stable command behavior, schema names, report metadata keys, diagnostics-path field semantics, and validator key identities are treated as public contract.
|
|
||||||
|
|
||||||
Additive fields are acceptable when existing fields and behavior remain compatible.
|
|
||||||
@@ -1,72 +0,0 @@
|
|||||||
# Structured LLM Architecture
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
This document describes Audita's structured LLM runtime boundary and adapter behavior.
|
|
||||||
|
|
||||||
## Runtime boundary
|
|
||||||
Production LLM integration depends on the internal contract only:
|
|
||||||
- `contracts.StructuredLLMClient`
|
|
||||||
- `CompleteStructured(ctx, req, out)`
|
|
||||||
|
|
||||||
Provider SDK types do not leak past this boundary.
|
|
||||||
|
|
||||||
## Adapter ownership
|
|
||||||
`internal/framework/llm` owns the OpenAI-compatible HTTP adapter and shared LLM runtime utilities.
|
|
||||||
|
|
||||||
Key responsibilities:
|
|
||||||
- request assembly;
|
|
||||||
- timeout/cancellation propagation;
|
|
||||||
- bounded retry behavior;
|
|
||||||
- scheduler integration;
|
|
||||||
- provider response decoding;
|
|
||||||
- error redaction.
|
|
||||||
|
|
||||||
## Structured schema registry
|
|
||||||
Structured response schemas are registered in `internal/framework/responseschema` and include stable metadata:
|
|
||||||
- `id`
|
|
||||||
- `version`
|
|
||||||
- `name`
|
|
||||||
- `json_schema`
|
|
||||||
- `sha256`
|
|
||||||
|
|
||||||
Current schema keys:
|
|
||||||
- `correction_set`
|
|
||||||
- `validator_decision_set`
|
|
||||||
|
|
||||||
Schema metadata is attached to diagnostics through `Schema.DiagnosticsMap()`.
|
|
||||||
|
|
||||||
## Request shape assumptions
|
|
||||||
Audita targets OpenAI-compatible chat-completions endpoints and sends structured requests with:
|
|
||||||
- model;
|
|
||||||
- chat messages;
|
|
||||||
- `response_format.type = json_schema`;
|
|
||||||
- schema name and JSON schema payload.
|
|
||||||
|
|
||||||
## Local validation remains mandatory
|
|
||||||
Provider schema enforcement is treated as transport-level guardrails.
|
|
||||||
|
|
||||||
Audita still validates output locally before applying behavior changes:
|
|
||||||
- proposal decoding and proposal invariants;
|
|
||||||
- validator decision decoding and cardinality checks;
|
|
||||||
- deterministic validation and apply-time rules.
|
|
||||||
|
|
||||||
## Shared malformed-output policy
|
|
||||||
Malformed structured-output classification is centralized in `internal/framework/structuredoutput`.
|
|
||||||
|
|
||||||
Proposal generation and validator execution both use this shared classifier so downgrade behavior cannot drift between the two paths.
|
|
||||||
|
|
||||||
## Secrets and redaction
|
|
||||||
Secret extraction for LLM redaction is centralized in `llm.ConfiguredSecrets(cfg)` and reused by proposal and validator diagnostics writers.
|
|
||||||
|
|
||||||
Secrets are redacted from:
|
|
||||||
- diagnostics artifacts;
|
|
||||||
- report artifacts;
|
|
||||||
- surfaced adapter/runtime errors.
|
|
||||||
|
|
||||||
## Concurrency and scheduling
|
|
||||||
LLM execution is constrained by composed scheduler limits:
|
|
||||||
- total LLM concurrency;
|
|
||||||
- proposal LLM concurrency;
|
|
||||||
- validation LLM concurrency.
|
|
||||||
|
|
||||||
The scheduler is FIFO and context-aware so permits are released on success, failure, and cancellation.
|
|
||||||
@@ -1,96 +0,0 @@
|
|||||||
# Audita Validators
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
This document defines the built-in validator system used by production module runs.
|
|
||||||
|
|
||||||
## Ownership boundaries
|
|
||||||
Built-in validator keys, constructors, and module chains are owned by `internal/validators`.
|
|
||||||
|
|
||||||
Shared runtime execution mechanics are owned by `internal/framework/validators`, including:
|
|
||||||
- validator request/result models;
|
|
||||||
- deterministic proposal checks;
|
|
||||||
- LLM validator batching and execution;
|
|
||||||
- decision-cardinality enforcement;
|
|
||||||
- diagnostics integration.
|
|
||||||
|
|
||||||
Execution class metadata is owned by `internal/validators/metadata`.
|
|
||||||
|
|
||||||
## Stable validator keys
|
|
||||||
Deterministic:
|
|
||||||
- `proposal_shape`
|
|
||||||
- `confidence_threshold`
|
|
||||||
- `original_text_presence`
|
|
||||||
- `non_empty_corrected_text`
|
|
||||||
- `no_effect`
|
|
||||||
- `protected_terms`
|
|
||||||
|
|
||||||
LLM-backed:
|
|
||||||
- `spoken_form_plausibility`
|
|
||||||
- `meaning_reversal_review`
|
|
||||||
- `editorial_review`
|
|
||||||
|
|
||||||
## Built-in module chains
|
|
||||||
`glossary`:
|
|
||||||
- `proposal_shape`
|
|
||||||
- `no_effect`
|
|
||||||
- `original_text_presence`
|
|
||||||
- `confidence_threshold`
|
|
||||||
- `protected_terms`
|
|
||||||
- `non_empty_corrected_text`
|
|
||||||
- `spoken_form_plausibility`
|
|
||||||
- `meaning_reversal_review`
|
|
||||||
|
|
||||||
`homophones`:
|
|
||||||
- `proposal_shape`
|
|
||||||
- `no_effect`
|
|
||||||
- `original_text_presence`
|
|
||||||
- `confidence_threshold`
|
|
||||||
- `protected_terms`
|
|
||||||
- `non_empty_corrected_text`
|
|
||||||
- `spoken_form_plausibility`
|
|
||||||
- `meaning_reversal_review`
|
|
||||||
|
|
||||||
`spoken_word`:
|
|
||||||
- `proposal_shape`
|
|
||||||
- `no_effect`
|
|
||||||
- `original_text_presence`
|
|
||||||
- `confidence_threshold`
|
|
||||||
- `protected_terms`
|
|
||||||
- `non_empty_corrected_text`
|
|
||||||
- `editorial_review`
|
|
||||||
- `meaning_reversal_review`
|
|
||||||
|
|
||||||
`grammar`:
|
|
||||||
- `proposal_shape`
|
|
||||||
- `no_effect`
|
|
||||||
- `original_text_presence`
|
|
||||||
- `confidence_threshold`
|
|
||||||
- `protected_terms`
|
|
||||||
- `non_empty_corrected_text`
|
|
||||||
- `editorial_review`
|
|
||||||
- `meaning_reversal_review`
|
|
||||||
|
|
||||||
## Ordering and execution semantics
|
|
||||||
Validator ordering is based on canonical metadata:
|
|
||||||
- deterministic validators run before LLM-backed validators.
|
|
||||||
|
|
||||||
Within each module stage:
|
|
||||||
- proposals are generated per section;
|
|
||||||
- validator chains execute on those proposals;
|
|
||||||
- approved proposals are applied once after section work settles.
|
|
||||||
|
|
||||||
## Malformed payload behavior
|
|
||||||
Malformed structured-output from proposal generation and LLM validator calls is downgraded, not treated as a process-fatal transport error.
|
|
||||||
|
|
||||||
Current outcomes:
|
|
||||||
- malformed proposal-generation payloads produce section/module warnings and zero proposals for the affected section;
|
|
||||||
- malformed validator decision payloads reject the affected validator batch with warnings;
|
|
||||||
- deterministic validator behavior and runner order remain unchanged.
|
|
||||||
|
|
||||||
## Reporting identity
|
|
||||||
Reports and diagnostics use stable validator keys as identifiers.
|
|
||||||
|
|
||||||
Correction-ledger deterministic-vs-LLM classification is derived from canonical validator metadata, not package-local hardcoded maps.
|
|
||||||
|
|
||||||
## Prompt assets
|
|
||||||
LLM validator prompt assets and prompt metadata are documented in [Prompts](./prompts.md).
|
|
||||||
186
docs/cli.md
Normal file
186
docs/cli.md
Normal file
@@ -0,0 +1,186 @@
|
|||||||
|
# Audita CLI Reference
|
||||||
|
|
||||||
|
## Shortest Useful Command
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita process <transcript.json> --glossary <glossary.yaml> --output <corrected.json>
|
||||||
|
```
|
||||||
|
|
||||||
|
This command validates input files, runs the configured correction pipeline, and writes corrected transcript JSON.
|
||||||
|
|
||||||
|
## Command Overview
|
||||||
|
|
||||||
|
- `audita process`: process one transcript JSON file.
|
||||||
|
- `audita config validate`: validate a versioned YAML config file.
|
||||||
|
- `audita config print-effective`: print redacted effective config JSON.
|
||||||
|
|
||||||
|
General help:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita --help
|
||||||
|
audita process --help
|
||||||
|
audita config --help
|
||||||
|
```
|
||||||
|
|
||||||
|
## `process`
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita process <transcript.json> [flags]
|
||||||
|
```
|
||||||
|
|
||||||
|
Input requirements:
|
||||||
|
- exactly one transcript JSON positional argument is required;
|
||||||
|
- `--glossary <path>` is required.
|
||||||
|
|
||||||
|
Config path selection for `process`:
|
||||||
|
1. `--config <path>`
|
||||||
|
2. `AUDITA_CONFIG`
|
||||||
|
3. `/usr/local/etc/audita/config.yml` (if present)
|
||||||
|
4. `/etc/audita/config.yml` (if present)
|
||||||
|
|
||||||
|
For precedence and full config schema, see [`docs/config.md`](config.md).
|
||||||
|
|
||||||
|
### `process` Flag Reference
|
||||||
|
|
||||||
|
Core I/O flags:
|
||||||
|
- `--config <path>`: path to versioned YAML config file.
|
||||||
|
- `--glossary <path>`: glossary YAML input path (required).
|
||||||
|
- `--output <path>`: corrected transcript JSON output file path.
|
||||||
|
- `--report-json <path>`: machine-readable report JSON output path.
|
||||||
|
- `--output-schema <key>`: output schema key (`bare-segments` or `audita-v1`).
|
||||||
|
- `--modules <csv>`: comma-separated module sequence override.
|
||||||
|
|
||||||
|
Primary LLM flags:
|
||||||
|
- `--llm-api-key <value>`: primary LLM API key.
|
||||||
|
- `--model <name>`: primary LLM model name.
|
||||||
|
- `--base-url <url>`: primary OpenAI-compatible base URL.
|
||||||
|
- `--llm-timeout-seconds <int>`: primary timeout in seconds.
|
||||||
|
- `--max-retries <int>`: primary structured-output retries.
|
||||||
|
|
||||||
|
Validation LLM flags:
|
||||||
|
- `--validation-llm-api-key <value>`: validation LLM API key.
|
||||||
|
- `--validation-model <name>`: validation LLM model name.
|
||||||
|
- `--validation-base-url <url>`: validation OpenAI-compatible base URL.
|
||||||
|
- `--validation-llm-timeout-seconds <int>`: validation timeout in seconds.
|
||||||
|
- `--validation-max-retries <int>`: validation structured-output retries.
|
||||||
|
- `--validation-max-prompt-tokens <int>`: validation max prompt tokens.
|
||||||
|
|
||||||
|
Concurrency flags:
|
||||||
|
- `--total-llm-concurrency <int>`: total concurrent proposal+validation LLM calls.
|
||||||
|
- `--proposal-llm-concurrency <int>`: concurrent proposal-generation LLM calls.
|
||||||
|
- `--validation-llm-concurrency <int>`: concurrent validation LLM calls.
|
||||||
|
- `--llm-concurrency <int>`: alias for `--total-llm-concurrency`.
|
||||||
|
|
||||||
|
Chunking and normalization flags:
|
||||||
|
- `--target-sections <int>`: target number of transcript sections.
|
||||||
|
- `--max-section-tokens <int>`: maximum section tokens.
|
||||||
|
- `--min-section-tokens <int>`: minimum section tokens.
|
||||||
|
- `--normalize-max-segment-gap <float>`: maximum same-speaker merge gap.
|
||||||
|
- `--normalize-ellipsis-gap <float>`: gap threshold for ellipsis insertion.
|
||||||
|
- `--normalize-max-segment-duration <float>`: maximum merged segment duration.
|
||||||
|
- `--normalize-max-segment-tokens <int>`: maximum merged segment token estimate.
|
||||||
|
|
||||||
|
Threshold flags:
|
||||||
|
- `--glossary-confidence-threshold <float>`
|
||||||
|
- `--homophones-confidence-threshold <float>`
|
||||||
|
- `--spoken-word-confidence-threshold <float>`
|
||||||
|
- `--grammar-confidence-threshold <float>`
|
||||||
|
|
||||||
|
Context and diagnostics flags:
|
||||||
|
- `--transcript-description <text>`: background context for prompts; does not override transcript content.
|
||||||
|
- `--work-dir <path>`: per-run diagnostics work directory.
|
||||||
|
- `--work-dir-retention <auto|always|never>`: run-directory retention policy.
|
||||||
|
|
||||||
|
### `process` Output and Exit Behavior
|
||||||
|
|
||||||
|
- With `--output`: stdout is expected to be empty on success.
|
||||||
|
- Without `--output`: stdout contains transcript JSON only on success.
|
||||||
|
- `--report-json` writes a file and is never printed to stdout.
|
||||||
|
- Stderr is human-readable diagnostics/errors.
|
||||||
|
- On failures after diagnostics initialization, stderr includes the diagnostics directory path.
|
||||||
|
|
||||||
|
Exit behavior:
|
||||||
|
- `0`: success.
|
||||||
|
- `1`: runtime failure during processing/reporting/output paths.
|
||||||
|
- `2`: CLI usage or configuration input error.
|
||||||
|
|
||||||
|
Integration references:
|
||||||
|
- subprocess contract: [`docs/integrations/subprocess.md`](integrations/subprocess.md)
|
||||||
|
- transcript/glossary file contract: [`docs/integrations/transcript-glossary-files.md`](integrations/transcript-glossary-files.md)
|
||||||
|
|
||||||
|
### `process` Examples
|
||||||
|
|
||||||
|
Write corrected transcript to a file:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita process transcript.json \
|
||||||
|
--glossary glossary.yaml \
|
||||||
|
--output corrected.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Emit transcript JSON to stdout:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita process transcript.json --glossary glossary.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Use explicit config and write report JSON:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita process transcript.json \
|
||||||
|
--glossary glossary.yaml \
|
||||||
|
--config audita.yml \
|
||||||
|
--output corrected.json \
|
||||||
|
--report-json report.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Override the module sequence:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita process transcript.json \
|
||||||
|
--glossary glossary.yaml \
|
||||||
|
--modules glossary,homophones,grammar \
|
||||||
|
--output corrected.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## `config validate`
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita config validate --config <path>
|
||||||
|
```
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
- validates defaults merged with file config;
|
||||||
|
- does not apply environment overrides;
|
||||||
|
- prints `config is valid` on success.
|
||||||
|
|
||||||
|
Errors:
|
||||||
|
- `--config` is required;
|
||||||
|
- positional arguments are rejected;
|
||||||
|
- validation failures are printed to stderr.
|
||||||
|
|
||||||
|
## `config print-effective`
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita config print-effective [--config <path>]
|
||||||
|
```
|
||||||
|
|
||||||
|
Config path selection:
|
||||||
|
1. `--config <path>` when provided
|
||||||
|
2. `AUDITA_CONFIG`
|
||||||
|
3. `/usr/local/etc/audita/config.yml` (if present)
|
||||||
|
4. `/etc/audita/config.yml` (if present)
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
- merges defaults, optional config file, and environment overrides;
|
||||||
|
- prints redacted JSON to stdout.
|
||||||
|
|
||||||
|
Errors:
|
||||||
|
- positional arguments are rejected;
|
||||||
|
- resolution or parse failures are printed to stderr.
|
||||||
238
docs/config.md
Normal file
238
docs/config.md
Normal file
@@ -0,0 +1,238 @@
|
|||||||
|
# Audita Configuration
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This is the canonical configuration reference for Audita.
|
||||||
|
|
||||||
|
It documents:
|
||||||
|
- config path resolution;
|
||||||
|
- effective precedence across defaults, file config, environment, and CLI;
|
||||||
|
- supported `version: 1` YAML schema;
|
||||||
|
- environment overrides;
|
||||||
|
- CLI override relationship;
|
||||||
|
- validation and secrets behavior.
|
||||||
|
|
||||||
|
For CLI command syntax, see [`docs/cli.md`](cli.md).
|
||||||
|
For OpenAI-compatible endpoint behavior, see [`docs/integrations/openai-compatible-llm.md`](integrations/openai-compatible-llm.md).
|
||||||
|
For transcript/glossary input file contracts, see [`docs/integrations/transcript-glossary-files.md`](integrations/transcript-glossary-files.md).
|
||||||
|
|
||||||
|
## Loading Model
|
||||||
|
|
||||||
|
Path resolution for `audita process` and `audita config print-effective`:
|
||||||
|
1. `--config <path>`
|
||||||
|
2. `AUDITA_CONFIG`
|
||||||
|
3. `/usr/local/etc/audita/config.yml` (if present)
|
||||||
|
4. `/etc/audita/config.yml` (if present)
|
||||||
|
|
||||||
|
Missing explicit path behavior:
|
||||||
|
- missing `--config` target is an error;
|
||||||
|
- missing `AUDITA_CONFIG` target is an error.
|
||||||
|
|
||||||
|
Missing default-path files are non-fatal.
|
||||||
|
|
||||||
|
## Effective Precedence
|
||||||
|
|
||||||
|
`audita process`:
|
||||||
|
1. defaults
|
||||||
|
2. file config
|
||||||
|
3. environment overrides
|
||||||
|
4. CLI overrides
|
||||||
|
|
||||||
|
`audita config print-effective`:
|
||||||
|
1. defaults
|
||||||
|
2. file config
|
||||||
|
3. environment overrides
|
||||||
|
|
||||||
|
`audita config validate`:
|
||||||
|
1. defaults
|
||||||
|
2. file config
|
||||||
|
|
||||||
|
`config validate` is intentionally file-only (no environment overrides).
|
||||||
|
|
||||||
|
## Defaults
|
||||||
|
|
||||||
|
Current defaults:
|
||||||
|
- modules: `glossary,homophones,glossary,spoken_word,grammar`
|
||||||
|
- output schema: `bare-segments`
|
||||||
|
- primary model: `openrouter/google/gemma-4-31b-it`
|
||||||
|
- primary base URL: `https://openrouter.ai/api/v1`
|
||||||
|
- primary timeout: `600` seconds
|
||||||
|
- max retries: `3`
|
||||||
|
- total/proposal LLM concurrency: `1`
|
||||||
|
- validation max prompt tokens: `2048`
|
||||||
|
- max section tokens: `8192`
|
||||||
|
- min section tokens: `2048`
|
||||||
|
- confidence thresholds: `0.8`
|
||||||
|
- normalization max segment gap: `4.0`
|
||||||
|
- normalization ellipsis gap: `3.5`
|
||||||
|
- normalization max segment duration: `60.0`
|
||||||
|
- normalization max segment tokens: `2048`
|
||||||
|
- transcript description: empty
|
||||||
|
- work dir: `/tmp/audita`
|
||||||
|
- work dir retention: `auto`
|
||||||
|
|
||||||
|
## YAML Schema (`version: 1`)
|
||||||
|
|
||||||
|
Supported file version:
|
||||||
|
- `version: 1` (required)
|
||||||
|
|
||||||
|
Unknown YAML fields are rejected.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
version: 1
|
||||||
|
|
||||||
|
pipeline:
|
||||||
|
modules: [glossary, homophones, glossary, spoken_word, grammar]
|
||||||
|
|
||||||
|
output:
|
||||||
|
schema: bare-segments
|
||||||
|
|
||||||
|
llm:
|
||||||
|
proposal:
|
||||||
|
base_url: https://openrouter.ai/api/v1
|
||||||
|
model: openrouter/google/gemma-4-31b-it
|
||||||
|
api_key_env: AUDITA_LLM_API_KEY
|
||||||
|
timeout: 600s
|
||||||
|
max_retries: 3
|
||||||
|
validation:
|
||||||
|
base_url: https://openrouter.ai/api/v1
|
||||||
|
model: openrouter/google/gemma-4-31b-it
|
||||||
|
api_key_env: AUDITA_VALIDATION_LLM_API_KEY
|
||||||
|
timeout: 600
|
||||||
|
max_retries: 3
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
total_llm: 1
|
||||||
|
proposal_llm: 1
|
||||||
|
validation_llm: 1
|
||||||
|
|
||||||
|
chunking:
|
||||||
|
target_sections: 8
|
||||||
|
max_section_tokens: 8192
|
||||||
|
min_section_tokens: 2048
|
||||||
|
|
||||||
|
normalization:
|
||||||
|
max_segment_gap: 4s
|
||||||
|
ellipsis_gap: 3.5s
|
||||||
|
max_segment_duration: 60s
|
||||||
|
max_segment_tokens: 2048
|
||||||
|
|
||||||
|
thresholds:
|
||||||
|
glossary: 0.8
|
||||||
|
homophones: 0.8
|
||||||
|
spoken_word: 0.8
|
||||||
|
grammar: 0.8
|
||||||
|
|
||||||
|
context:
|
||||||
|
description: optional background context
|
||||||
|
|
||||||
|
diagnostics:
|
||||||
|
work_dir: /tmp/audita
|
||||||
|
retention: auto
|
||||||
|
```
|
||||||
|
|
||||||
|
Duration-parsing behavior:
|
||||||
|
- `llm.*.timeout`: integer seconds or duration string; duration strings must resolve to whole seconds.
|
||||||
|
- `normalization.*` duration-like fields: numeric seconds or duration string.
|
||||||
|
|
||||||
|
## Environment Overrides
|
||||||
|
|
||||||
|
Modules:
|
||||||
|
- `AUDITA_MODULES`
|
||||||
|
|
||||||
|
Config path:
|
||||||
|
- `AUDITA_CONFIG`
|
||||||
|
|
||||||
|
Primary LLM:
|
||||||
|
- `AUDITA_LLM_API_KEY` (falls back to `OPENROUTER_API_KEY` when unset)
|
||||||
|
- `AUDITA_MODEL`
|
||||||
|
- `AUDITA_BASE_URL`
|
||||||
|
- `AUDITA_LLM_TIMEOUT_SECONDS`
|
||||||
|
- `AUDITA_MAX_RETRIES`
|
||||||
|
|
||||||
|
Validation LLM:
|
||||||
|
- `AUDITA_VALIDATION_LLM_API_KEY`
|
||||||
|
- `AUDITA_VALIDATION_MODEL`
|
||||||
|
- `AUDITA_VALIDATION_BASE_URL`
|
||||||
|
- `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS`
|
||||||
|
- `AUDITA_VALIDATION_MAX_RETRIES`
|
||||||
|
- `AUDITA_VALIDATION_MAX_PROMPT_TOKENS`
|
||||||
|
|
||||||
|
Concurrency:
|
||||||
|
- `AUDITA_TOTAL_LLM_CONCURRENCY`
|
||||||
|
- `AUDITA_PROPOSAL_LLM_CONCURRENCY`
|
||||||
|
- `AUDITA_VALIDATION_LLM_CONCURRENCY`
|
||||||
|
- `AUDITA_LLM_CONCURRENCY` (legacy alias for total)
|
||||||
|
|
||||||
|
Chunking:
|
||||||
|
- `AUDITA_MAX_SECTION_TOKENS`
|
||||||
|
- `AUDITA_MIN_SECTION_TOKENS`
|
||||||
|
- `AUDITA_TARGET_SECTIONS`
|
||||||
|
|
||||||
|
Thresholds:
|
||||||
|
- `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD`
|
||||||
|
- `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD`
|
||||||
|
- `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD`
|
||||||
|
- `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD`
|
||||||
|
|
||||||
|
Normalization:
|
||||||
|
- `AUDITA_NORMALIZE_MAX_SEGMENT_GAP`
|
||||||
|
- `AUDITA_NORMALIZE_ELLIPSIS_GAP`
|
||||||
|
- `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION`
|
||||||
|
- `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS`
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
- `AUDITA_WORK_DIR`
|
||||||
|
- `AUDITA_WORK_DIR_RETENTION` (`auto`, `always`, `never`)
|
||||||
|
|
||||||
|
Transcript description:
|
||||||
|
- no `AUDITA_*` environment variable is currently defined.
|
||||||
|
|
||||||
|
## CLI Override Relationship
|
||||||
|
|
||||||
|
CLI flags override file and environment values for `audita process`.
|
||||||
|
|
||||||
|
The CLI supports canonical total concurrency (`--total-llm-concurrency`) and legacy alias (`--llm-concurrency`):
|
||||||
|
- when both are provided at the same precedence layer, canonical total wins;
|
||||||
|
- if proposal concurrency is not explicitly set and total is set via environment or CLI, proposal concurrency inherits that total;
|
||||||
|
- validation concurrency inherits total only when validation concurrency is unset.
|
||||||
|
|
||||||
|
For full flag syntax, see [`docs/cli.md`](cli.md).
|
||||||
|
|
||||||
|
## Validation Rules
|
||||||
|
|
||||||
|
Validation includes:
|
||||||
|
- supported module keys only;
|
||||||
|
- supported output schema keys only (`bare-segments`, `audita-v1`);
|
||||||
|
- positive timeout/concurrency/token constraints;
|
||||||
|
- `proposal_llm <= total_llm` and `validation_llm <= total_llm` when validation is set;
|
||||||
|
- confidence thresholds in `[0.0, 1.0]`;
|
||||||
|
- transcript description length `<= 500` characters;
|
||||||
|
- non-empty work dir;
|
||||||
|
- work-dir retention in `auto|always|never`.
|
||||||
|
|
||||||
|
## Secrets
|
||||||
|
|
||||||
|
Recommended secret handling:
|
||||||
|
- use `llm.proposal.api_key_env` and `llm.validation.api_key_env` in file config;
|
||||||
|
- use `AUDITA_*_API_KEY` environment overrides or CLI key flags when needed.
|
||||||
|
|
||||||
|
`api_key_env` fields contain environment variable names, not secret values.
|
||||||
|
|
||||||
|
Redaction behavior:
|
||||||
|
- `audita config print-effective` redacts resolved API keys.
|
||||||
|
- diagnostics and report paths redact configured secret values.
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
- Minimal config: [`examples/minimal-config.yml`](../examples/minimal-config.yml)
|
||||||
|
- Production-style config: [`examples/production-config.yml`](../examples/production-config.yml)
|
||||||
|
- Tiny transcript input: [`examples/tiny-transcript.json`](../examples/tiny-transcript.json)
|
||||||
|
- Tiny glossary input: [`examples/tiny-glossary.yaml`](../examples/tiny-glossary.yaml)
|
||||||
|
|
||||||
|
Validate the config examples:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita config validate --config examples/minimal-config.yml
|
||||||
|
audita config validate --config examples/production-config.yml
|
||||||
|
```
|
||||||
@@ -1,149 +0,0 @@
|
|||||||
# Audita Configuration
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
This document defines the supported versioned YAML configuration model and runtime precedence behavior.
|
|
||||||
|
|
||||||
## Supported file version
|
|
||||||
Current supported config file version:
|
|
||||||
- `version: 1`
|
|
||||||
|
|
||||||
Validation rules:
|
|
||||||
- missing `version` fails;
|
|
||||||
- unsupported version fails;
|
|
||||||
- unknown YAML fields fail (strict decoding).
|
|
||||||
|
|
||||||
## Config path resolution
|
|
||||||
For `audita process` and `audita config print-effective`, path resolution order is:
|
|
||||||
1. `--config <path>`
|
|
||||||
2. `AUDITA_CONFIG`
|
|
||||||
3. `/usr/local/etc/audita/config.yml` (if present)
|
|
||||||
4. `/etc/audita/config.yml` (if present)
|
|
||||||
|
|
||||||
Missing-path behavior:
|
|
||||||
- missing `--config` path is an error;
|
|
||||||
- missing `AUDITA_CONFIG` path is an error;
|
|
||||||
- missing both default paths is non-fatal.
|
|
||||||
|
|
||||||
## Effective precedence
|
|
||||||
`audita process` effective precedence:
|
|
||||||
1. defaults
|
|
||||||
2. file config
|
|
||||||
3. environment overrides
|
|
||||||
4. CLI overrides
|
|
||||||
|
|
||||||
`audita config print-effective` uses:
|
|
||||||
1. defaults
|
|
||||||
2. file config
|
|
||||||
3. environment overrides
|
|
||||||
|
|
||||||
`audita config validate` intentionally uses file-only validation:
|
|
||||||
1. defaults
|
|
||||||
2. file config
|
|
||||||
|
|
||||||
Environment overrides are not applied in `config validate`.
|
|
||||||
|
|
||||||
## Supported top-level YAML fields
|
|
||||||
```yaml
|
|
||||||
version: 1
|
|
||||||
|
|
||||||
pipeline:
|
|
||||||
modules: [glossary, homophones, glossary, spoken_word, grammar]
|
|
||||||
|
|
||||||
output:
|
|
||||||
schema: bare-segments
|
|
||||||
|
|
||||||
llm:
|
|
||||||
proposal:
|
|
||||||
base_url: https://openrouter.ai/api/v1
|
|
||||||
model: openrouter/google/gemma-4-31b-it
|
|
||||||
api_key_env: AUDITA_LLM_API_KEY
|
|
||||||
timeout: 120s
|
|
||||||
max_retries: 3
|
|
||||||
validation:
|
|
||||||
base_url: https://openrouter.ai/api/v1
|
|
||||||
model: openrouter/google/gemma-4-31b-it
|
|
||||||
api_key_env: AUDITA_VALIDATION_LLM_API_KEY
|
|
||||||
timeout: 120s
|
|
||||||
max_retries: 3
|
|
||||||
|
|
||||||
concurrency:
|
|
||||||
total_llm: 2
|
|
||||||
proposal_llm: 2
|
|
||||||
validation_llm: 1
|
|
||||||
|
|
||||||
chunking:
|
|
||||||
target_sections: 8
|
|
||||||
max_section_tokens: 8192
|
|
||||||
min_section_tokens: 2048
|
|
||||||
|
|
||||||
normalization:
|
|
||||||
max_segment_gap: 4s
|
|
||||||
ellipsis_gap: 3.5s
|
|
||||||
max_segment_duration: 60s
|
|
||||||
max_segment_tokens: 2048
|
|
||||||
|
|
||||||
thresholds:
|
|
||||||
glossary: 0.8
|
|
||||||
homophones: 0.8
|
|
||||||
spoken_word: 0.8
|
|
||||||
grammar: 0.8
|
|
||||||
|
|
||||||
context:
|
|
||||||
description: "optional transcript background context"
|
|
||||||
|
|
||||||
diagnostics:
|
|
||||||
work_dir: /tmp/audita
|
|
||||||
retention: auto
|
|
||||||
```
|
|
||||||
|
|
||||||
## Module and output-schema validation
|
|
||||||
`pipeline.modules` keys are validated against the built-in supported module catalog.
|
|
||||||
|
|
||||||
Supported module keys:
|
|
||||||
- `glossary`
|
|
||||||
- `homophones`
|
|
||||||
- `spoken_word`
|
|
||||||
- `grammar`
|
|
||||||
|
|
||||||
Repeated supported module keys are allowed.
|
|
||||||
|
|
||||||
`output.schema` is validated against the built-in output schema catalog.
|
|
||||||
|
|
||||||
Supported output schema keys:
|
|
||||||
- `bare-segments`
|
|
||||||
- `audita-v1`
|
|
||||||
|
|
||||||
Unknown module keys and unknown output schema keys fail validation.
|
|
||||||
|
|
||||||
## Duration field parsing
|
|
||||||
Duration-like fields support:
|
|
||||||
- numeric seconds (for example `120`, `3.5`)
|
|
||||||
- duration strings (for example `120s`, `2m`)
|
|
||||||
|
|
||||||
LLM timeout duration strings must resolve to whole seconds.
|
|
||||||
|
|
||||||
## Secret handling
|
|
||||||
Use `api_key_env` fields for secrets:
|
|
||||||
- `llm.proposal.api_key_env`
|
|
||||||
- `llm.validation.api_key_env`
|
|
||||||
|
|
||||||
These fields store environment variable names, not secret values.
|
|
||||||
|
|
||||||
Resolved secret values are redacted from:
|
|
||||||
- `audita config print-effective` output;
|
|
||||||
- diagnostics `effective-config.json`;
|
|
||||||
- report and diagnostics payloads.
|
|
||||||
|
|
||||||
## Commands
|
|
||||||
Validate a file config:
|
|
||||||
```sh
|
|
||||||
audita config validate --config ./audita.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
Print redacted effective config:
|
|
||||||
```sh
|
|
||||||
audita config print-effective --config ./audita.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
## Compatibility notes
|
|
||||||
Legacy compatibility flags and environment aliases remain available where implemented, but the stable configuration surface is the versioned YAML model described above.
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
# Audita Development Workflow
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
This document defines the canonical contributor workflow and engineering conventions for this repository.
|
|
||||||
|
|
||||||
## Workflow
|
|
||||||
1. Start from a clean understanding of scope and constraints.
|
|
||||||
2. Make focused changes that preserve existing public behavior unless behavior change is explicitly intended.
|
|
||||||
3. Run targeted tests for touched packages.
|
|
||||||
4. Run `go test ./...` before finalizing substantial changes.
|
|
||||||
5. Update affected documentation so it describes current behavior only.
|
|
||||||
|
|
||||||
## Engineering conventions
|
|
||||||
- Keep module packages separate: `glossary`, `homophones`, `spoken_word`, `grammar`.
|
|
||||||
- Prefer narrow shared helpers and catalogs over broad abstractions.
|
|
||||||
- Preserve diagnostics artifact naming and report field contracts unless intentionally changed.
|
|
||||||
- Preserve CLI/config precedence semantics unless intentionally changed.
|
|
||||||
- Treat stable validator keys, prompt identifiers, and output-schema keys as contract surfaces.
|
|
||||||
|
|
||||||
## Configuration and runtime expectations
|
|
||||||
- `audita process` precedence is defaults -> file -> env -> CLI.
|
|
||||||
- `audita config validate` validates file config merged onto defaults only.
|
|
||||||
- `audita config print-effective` includes environment overrides and prints redacted JSON.
|
|
||||||
|
|
||||||
## Testing expectations
|
|
||||||
- Add tests for new behavior and for bug fixes.
|
|
||||||
- Keep deterministic fixtures stable.
|
|
||||||
- Do not reduce existing parity, release-fixture, subprocess, or module-specific coverage without equivalent replacement.
|
|
||||||
|
|
||||||
## Commit discipline
|
|
||||||
- Keep commits scoped and reviewable.
|
|
||||||
- Avoid mixing unrelated refactors with behavior changes.
|
|
||||||
- Use clear plain-English commit messages.
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# Documentation Policy
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
This policy defines how project documentation should be authored and maintained.
|
|
||||||
|
|
||||||
## Core rules
|
|
||||||
- Document the current behavior of the codebase.
|
|
||||||
- Remove stale behavior descriptions promptly when code changes.
|
|
||||||
- Do not describe development history in architecture or behavior docs unless a document is explicitly historical.
|
|
||||||
- Do not use architecture or behavior docs as changelogs.
|
|
||||||
- Prefer rewriting stale sections from scratch when substantial behavior or ownership changes occur.
|
|
||||||
|
|
||||||
## Consistency requirements
|
|
||||||
- Keep command examples aligned with current CLI surfaces.
|
|
||||||
- Keep configuration examples aligned with supported fields and precedence.
|
|
||||||
- Keep architecture package ownership descriptions aligned with current code layout.
|
|
||||||
- Keep stable contract identifiers accurate (module keys, validator keys, output-schema keys, report metadata fields).
|
|
||||||
|
|
||||||
## Cross-document expectations
|
|
||||||
- `docs/architecture/*` documents runtime behavior and package ownership.
|
|
||||||
- `docs/configuration.md` documents config schema and precedence.
|
|
||||||
- `docs/development.md` documents contributor workflow and engineering conventions.
|
|
||||||
|
|
||||||
## Review expectations for documentation changes
|
|
||||||
- Verify referenced files and links exist.
|
|
||||||
- Verify examples match current behavior.
|
|
||||||
- Prefer concise, direct language and avoid speculative future claims.
|
|
||||||
@@ -1,96 +0,0 @@
|
|||||||
# Audita Subprocess Operations
|
|
||||||
|
|
||||||
This document describes how parent processes should invoke `audita process` safely in production orchestration.
|
|
||||||
|
|
||||||
## Recommended command form
|
|
||||||
|
|
||||||
Use explicit file outputs for orchestrated runs:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
audita process <transcript.json> \
|
|
||||||
--transcript-description "Brief context that may help resolve ambiguous terms." \
|
|
||||||
--glossary <glossary.yaml> \
|
|
||||||
--output <output-transcript.json> \
|
|
||||||
--report-json <report.json>
|
|
||||||
```
|
|
||||||
|
|
||||||
Additional flags that may be situationally appropriate:
|
|
||||||
- `--config <path>` to select an explicit versioned config file.
|
|
||||||
- `--output-schema <bare-segments|audita-v1>` to select transcript output shape.
|
|
||||||
- `--work-dir <dir>` to control diagnostics location.
|
|
||||||
- `--work-dir-retention <always|auto|never>` to control retained run directories.
|
|
||||||
- `--total-llm-concurrency`, `--proposal-llm-concurrency`, and `--validation-llm-concurrency` when orchestration needs to set explicit LLM throughput controls.
|
|
||||||
- `--modules ...` only when intentionally overriding the default sequence.
|
|
||||||
|
|
||||||
For config-driven orchestration, validate config files in CI/preflight:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
audita config validate --config <path>
|
|
||||||
```
|
|
||||||
|
|
||||||
## Stdout behavior
|
|
||||||
|
|
||||||
- With `--output`: stdout is expected to be empty on success.
|
|
||||||
- Without `--output`: stdout contains transcript JSON only on success.
|
|
||||||
- Report JSON is never written to stdout.
|
|
||||||
|
|
||||||
## Stderr behavior
|
|
||||||
|
|
||||||
- Success path should be quiet or minimal human-readable logs.
|
|
||||||
- Failure path writes concise human-readable errors.
|
|
||||||
- When a diagnostics run directory exists, failure stderr includes its path.
|
|
||||||
- Prompt/response diagnostic payloads are not streamed to stderr.
|
|
||||||
|
|
||||||
## Output file behavior
|
|
||||||
|
|
||||||
- `--output` writes transcript JSON in the selected output schema to the provided path.
|
|
||||||
- Output write failures return nonzero and surface actionable errors.
|
|
||||||
- The command does not silently ignore output write errors.
|
|
||||||
|
|
||||||
## Report JSON behavior
|
|
||||||
|
|
||||||
- `--report-json` writes a machine-readable process report to the requested path.
|
|
||||||
- Run-directory `report.json` is written independently under diagnostics.
|
|
||||||
- Best-effort failure reports are emitted when possible without masking the primary failure.
|
|
||||||
- Report write failures return nonzero with clear stderr messaging.
|
|
||||||
- Report diagnostics metadata references run-directory artifacts including utilization diagnostics and correction ledger paths when available.
|
|
||||||
|
|
||||||
## Diagnostics directory behavior
|
|
||||||
|
|
||||||
- Each run creates (when possible) a per-run diagnostics directory.
|
|
||||||
- Typical artifacts include transcript, normalization, chunking, invocation, effective config, LLM diagnostics, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, and `error.log` on failure.
|
|
||||||
- Failed runs retain diagnostics.
|
|
||||||
- Under `auto` retention, successful runs with skipped/rejected corrections are retained; clean successful runs may be removed.
|
|
||||||
|
|
||||||
## Exit codes
|
|
||||||
|
|
||||||
- `0`: success.
|
|
||||||
- Nonzero: failure (input/schema/config/module/LLM/runtime/output/report/diagnostics errors).
|
|
||||||
|
|
||||||
Treat any nonzero as a failed subprocess invocation.
|
|
||||||
|
|
||||||
## Timeout and cancellation
|
|
||||||
|
|
||||||
- Runtime operations propagate context cancellation and request timeouts through LLM/scheduler paths.
|
|
||||||
- On cancellation or timeout, the process exits nonzero and should not hang.
|
|
||||||
- If diagnostics were initialized before failure, failure artifacts remain available for debugging.
|
|
||||||
|
|
||||||
## Secret redaction expectations
|
|
||||||
|
|
||||||
API keys and configured secret values are redacted from:
|
|
||||||
- reports (`--report-json` and run-dir `report.json`);
|
|
||||||
- diagnostics artifacts (including effective config and LLM interaction artifacts);
|
|
||||||
- surfaced adapter/runtime errors;
|
|
||||||
- test fixtures and regression outputs.
|
|
||||||
|
|
||||||
Parent-process logs should still avoid printing raw environment variables.
|
|
||||||
|
|
||||||
## Parent-process pipe guidance
|
|
||||||
|
|
||||||
To avoid deadlocks in orchestrators:
|
|
||||||
- always read both stdout and stderr concurrently when invoking as a subprocess;
|
|
||||||
- prefer file outputs (`--output`, `--report-json`) for machine workflows;
|
|
||||||
- treat stderr as human-readable diagnostics, not structured data;
|
|
||||||
- parse structured results from output/report files.
|
|
||||||
|
|
||||||
For Go callers, prefer `exec.CommandContext` with explicit timeout/cancellation and buffered/streamed readers for both pipes.
|
|
||||||
119
docs/integrations/openai-compatible-llm.md
Normal file
119
docs/integrations/openai-compatible-llm.md
Normal file
@@ -0,0 +1,119 @@
|
|||||||
|
# OpenAI-Compatible LLM Integration
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document defines the external LLM endpoint contract Audita currently uses.
|
||||||
|
|
||||||
|
It covers:
|
||||||
|
- endpoint and auth expectations;
|
||||||
|
- structured request and response shape;
|
||||||
|
- retry and timeout behavior;
|
||||||
|
- diagnostics and secret redaction.
|
||||||
|
|
||||||
|
For user-facing CLI flags and config keys, see [`docs/cli.md`](../cli.md) and [`docs/config.md`](../config.md).
|
||||||
|
|
||||||
|
## Endpoint Contract
|
||||||
|
|
||||||
|
Audita sends HTTPS `POST` requests to:
|
||||||
|
|
||||||
|
- `<base_url>/chat/completions`
|
||||||
|
|
||||||
|
`base_url` comes from primary or validation LLM config and is required.
|
||||||
|
|
||||||
|
## Authentication Contract
|
||||||
|
|
||||||
|
When an API key is configured, Audita sends:
|
||||||
|
|
||||||
|
- `Authorization: Bearer <api_key>`
|
||||||
|
|
||||||
|
When no API key is configured, the `Authorization` header is omitted.
|
||||||
|
|
||||||
|
## Request Shape
|
||||||
|
|
||||||
|
Audita sends a chat-completions payload with:
|
||||||
|
- `model`;
|
||||||
|
- `messages` (role/content pairs);
|
||||||
|
- `response_format` using JSON Schema strict mode.
|
||||||
|
|
||||||
|
Representative shape:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"model": "example-model",
|
||||||
|
"messages": [
|
||||||
|
{"role": "system", "content": "..."},
|
||||||
|
{"role": "user", "content": "..."}
|
||||||
|
],
|
||||||
|
"response_format": {
|
||||||
|
"type": "json_schema",
|
||||||
|
"json_schema": {
|
||||||
|
"name": "correction_set",
|
||||||
|
"strict": true,
|
||||||
|
"schema": {"type": "object"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Behavioral requirements enforced by Audita:
|
||||||
|
- `model` must resolve to a non-empty value;
|
||||||
|
- each message must have non-empty `role` and `content`;
|
||||||
|
- `response_format.type` is always `json_schema`;
|
||||||
|
- `response_format.json_schema.name` and `schema` must be present;
|
||||||
|
- request schema JSON must be valid JSON.
|
||||||
|
|
||||||
|
## Response Handling Contract
|
||||||
|
|
||||||
|
Audita expects a successful JSON response with at least one choice and assistant content that can be interpreted as JSON.
|
||||||
|
|
||||||
|
Supported assistant content forms:
|
||||||
|
- string containing JSON;
|
||||||
|
- raw JSON value.
|
||||||
|
|
||||||
|
Audita then decodes the JSON against the expected structured output type.
|
||||||
|
|
||||||
|
Current structured schema identities used by Audita runtime:
|
||||||
|
- `correction_set`
|
||||||
|
- `validator_decision_set`
|
||||||
|
|
||||||
|
## Retries and Timeouts
|
||||||
|
|
||||||
|
Retry behavior:
|
||||||
|
- default max retries is `3` when unset;
|
||||||
|
- retries apply to retryable transport/decode/server-side errors;
|
||||||
|
- HTTP `429` and `5xx` responses are retryable;
|
||||||
|
- retry stops immediately when context is canceled or deadline expires.
|
||||||
|
|
||||||
|
Timeout behavior:
|
||||||
|
- request timeout is derived from configured LLM timeout settings;
|
||||||
|
- timeout/cancellation propagate through HTTP requests and return nonzero process failures.
|
||||||
|
|
||||||
|
## Error Behavior
|
||||||
|
|
||||||
|
Non-2xx responses fail the request.
|
||||||
|
|
||||||
|
Error message extraction behavior:
|
||||||
|
- if provider JSON includes `error.message`, Audita surfaces that message;
|
||||||
|
- else if provider JSON includes top-level `message`, Audita surfaces that;
|
||||||
|
- otherwise Audita surfaces status code plus response body text.
|
||||||
|
|
||||||
|
Malformed or incompatible structured responses fail safely and are surfaced as runtime errors or validator/proposal warnings depending on call site.
|
||||||
|
|
||||||
|
## Secret Redaction
|
||||||
|
|
||||||
|
Configured LLM secrets are redacted from:
|
||||||
|
- surfaced adapter/runtime errors;
|
||||||
|
- LLM diagnostics request/response/error artifacts;
|
||||||
|
- effective config/report artifacts that include LLM configuration material.
|
||||||
|
|
||||||
|
Redaction marker:
|
||||||
|
- `[REDACTED]`
|
||||||
|
|
||||||
|
## Compatibility Boundaries
|
||||||
|
|
||||||
|
This integration documentation applies only to the implemented OpenAI-compatible chat completions flow.
|
||||||
|
|
||||||
|
Not part of current behavior:
|
||||||
|
- provider SDK integration;
|
||||||
|
- non-OpenAI-compatible API contracts;
|
||||||
|
- server-side model routing features beyond explicitly configured model/base URL.
|
||||||
99
docs/integrations/subprocess.md
Normal file
99
docs/integrations/subprocess.md
Normal file
@@ -0,0 +1,99 @@
|
|||||||
|
# Subprocess Integration
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document describes how a parent process should invoke Audita as a subprocess.
|
||||||
|
|
||||||
|
It covers:
|
||||||
|
- invocation shape;
|
||||||
|
- stdout/stderr behavior;
|
||||||
|
- output/report file behavior;
|
||||||
|
- diagnostics and exit behavior.
|
||||||
|
|
||||||
|
For full CLI and config references, see [`docs/cli.md`](../cli.md) and [`docs/config.md`](../config.md).
|
||||||
|
|
||||||
|
## Recommended Invocation
|
||||||
|
|
||||||
|
Use explicit output and report paths for machine workflows:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita process <transcript.json> \
|
||||||
|
--glossary <glossary.yaml> \
|
||||||
|
--output <output-transcript.json> \
|
||||||
|
--report-json <report.json>
|
||||||
|
```
|
||||||
|
|
||||||
|
Optional commonly used flags:
|
||||||
|
- `--config <path>`
|
||||||
|
- `--output-schema <bare-segments|audita-v1>`
|
||||||
|
- `--work-dir <dir>`
|
||||||
|
- `--work-dir-retention <always|auto|never>`
|
||||||
|
- `--transcript-description <text>`
|
||||||
|
|
||||||
|
## Stdout Contract
|
||||||
|
|
||||||
|
On success:
|
||||||
|
- with `--output`: stdout is expected to be empty;
|
||||||
|
- without `--output`: stdout contains transcript JSON only.
|
||||||
|
|
||||||
|
`--report-json` output is never written to stdout.
|
||||||
|
|
||||||
|
## Stderr Contract
|
||||||
|
|
||||||
|
Stderr is human-readable status/error output.
|
||||||
|
|
||||||
|
On failures:
|
||||||
|
- stderr includes a concise top-level error;
|
||||||
|
- when diagnostics are initialized, stderr includes diagnostics directory path.
|
||||||
|
|
||||||
|
Do not treat stderr as a machine-stable JSON channel.
|
||||||
|
|
||||||
|
## Output and Report File Contract
|
||||||
|
|
||||||
|
Transcript output:
|
||||||
|
- `--output` writes corrected transcript JSON to the provided path;
|
||||||
|
- output write failures return nonzero.
|
||||||
|
|
||||||
|
Report output:
|
||||||
|
- `--report-json` writes machine-readable process report JSON to the provided path;
|
||||||
|
- run diagnostics also attempt to write their own `report.json`;
|
||||||
|
- report write failures return nonzero;
|
||||||
|
- on failure paths, report writing is best-effort and does not mask the primary run error.
|
||||||
|
|
||||||
|
## Diagnostics Contract
|
||||||
|
|
||||||
|
When run-directory initialization succeeds, per-run diagnostics artifacts are written under the configured work directory.
|
||||||
|
|
||||||
|
Typical artifacts include:
|
||||||
|
- `source-transcript.json`
|
||||||
|
- `source-transcript-parsed.json`
|
||||||
|
- `normalized-transcript.json`
|
||||||
|
- `normalization-summary.json`
|
||||||
|
- `chunking-summary.json`
|
||||||
|
- `invocation.json`
|
||||||
|
- `effective-config.json`
|
||||||
|
- `utilization-diagnostics.json`
|
||||||
|
- `correction-ledger.json`
|
||||||
|
- `report.json`
|
||||||
|
- `error.log` (failure)
|
||||||
|
|
||||||
|
Retention behavior is controlled by `--work-dir-retention` / config.
|
||||||
|
|
||||||
|
## Exit Behavior
|
||||||
|
|
||||||
|
Exit codes:
|
||||||
|
- `0`: success;
|
||||||
|
- `1`: runtime processing/output/report failure;
|
||||||
|
- `2`: CLI usage or configuration input error.
|
||||||
|
|
||||||
|
Treat any nonzero as subprocess failure.
|
||||||
|
|
||||||
|
## Parent-Process Guidance
|
||||||
|
|
||||||
|
For reliable orchestration:
|
||||||
|
- read stdout and stderr concurrently to avoid pipe blocking;
|
||||||
|
- prefer `--output` and `--report-json` for machine parsing;
|
||||||
|
- use timeout/cancellation in the parent process;
|
||||||
|
- inspect diagnostics path and `report.json`/`error.log` on failure.
|
||||||
|
|
||||||
|
For input file contracts, see [`docs/integrations/transcript-glossary-files.md`](transcript-glossary-files.md).
|
||||||
98
docs/integrations/transcript-glossary-files.md
Normal file
98
docs/integrations/transcript-glossary-files.md
Normal file
@@ -0,0 +1,98 @@
|
|||||||
|
# Transcript and Glossary File Integration
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document defines the input file contracts for:
|
||||||
|
- transcript JSON;
|
||||||
|
- glossary YAML.
|
||||||
|
|
||||||
|
These files are loaded and validated before processing begins.
|
||||||
|
|
||||||
|
## Transcript JSON Contract
|
||||||
|
|
||||||
|
Audita accepts either top-level shape:
|
||||||
|
- JSON array of segments; or
|
||||||
|
- JSON object with a `segments` array.
|
||||||
|
|
||||||
|
Segment fields:
|
||||||
|
- `id` (optional integer in source form);
|
||||||
|
- `speaker` (required non-empty string);
|
||||||
|
- `start` (required finite non-negative number);
|
||||||
|
- `end` (required finite non-negative number, `>= start`);
|
||||||
|
- `text` (required non-empty string);
|
||||||
|
- `categories` (optional string array; entries must be non-empty).
|
||||||
|
|
||||||
|
Additional rules:
|
||||||
|
- transcript must contain at least one segment;
|
||||||
|
- duplicate segment IDs are rejected when IDs are present.
|
||||||
|
|
||||||
|
Example (`examples/tiny-transcript.json`):
|
||||||
|
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"speaker": "A",
|
||||||
|
"start": 0.0,
|
||||||
|
"end": 1.2,
|
||||||
|
"text": "hello world"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Glossary YAML Contract
|
||||||
|
|
||||||
|
Audita expects top-level `glossary` list entries.
|
||||||
|
|
||||||
|
Entry fields:
|
||||||
|
- `name` (required non-empty string);
|
||||||
|
- `category` (required non-empty string);
|
||||||
|
- `summary` (required non-empty string);
|
||||||
|
- `aliases` (optional list of strings; entries must be non-empty);
|
||||||
|
- `plural` (optional string).
|
||||||
|
|
||||||
|
Additional rules:
|
||||||
|
- glossary must contain at least one entry.
|
||||||
|
|
||||||
|
Example (`examples/tiny-glossary.yaml`):
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
glossary:
|
||||||
|
- name: Audita
|
||||||
|
aliases:
|
||||||
|
- audita
|
||||||
|
category: product
|
||||||
|
summary: The Audita transcript correction CLI.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation Failure Behavior
|
||||||
|
|
||||||
|
Representative transcript validation failures:
|
||||||
|
- invalid JSON;
|
||||||
|
- unsupported top-level shape;
|
||||||
|
- empty `speaker` or `text`;
|
||||||
|
- invalid times (`NaN`, `Inf`, negative, or `end < start`);
|
||||||
|
- duplicate IDs;
|
||||||
|
- empty transcript array.
|
||||||
|
|
||||||
|
Representative glossary validation failures:
|
||||||
|
- invalid YAML;
|
||||||
|
- empty or missing glossary entries;
|
||||||
|
- missing required entry fields;
|
||||||
|
- empty alias values.
|
||||||
|
|
||||||
|
These failures surface as schema errors and the process exits nonzero.
|
||||||
|
|
||||||
|
## CLI Usage
|
||||||
|
|
||||||
|
Minimal invocation:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
audita process ./transcript.json --glossary ./glossary.yaml --output ./corrected.json
|
||||||
|
```
|
||||||
|
|
||||||
|
See also:
|
||||||
|
- [`docs/cli.md`](../cli.md)
|
||||||
|
- [`docs/config.md`](../config.md)
|
||||||
|
- [`examples/tiny-transcript.json`](../../examples/tiny-transcript.json)
|
||||||
|
- [`examples/tiny-glossary.yaml`](../../examples/tiny-glossary.yaml)
|
||||||
79
docs/internal/diagnostics-reporting.md
Normal file
79
docs/internal/diagnostics-reporting.md
Normal file
@@ -0,0 +1,79 @@
|
|||||||
|
# Audita Diagnostics and Reporting
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document describes diagnostics artifacts, process report mapping, and correction ledger generation.
|
||||||
|
|
||||||
|
## Run Directory Ownership
|
||||||
|
|
||||||
|
`internal/core/diagnostics` owns run-directory creation, artifact writes, and retention decisions.
|
||||||
|
|
||||||
|
Stable artifact names include:
|
||||||
|
- `source-transcript.json`
|
||||||
|
- `source-transcript-parsed.json`
|
||||||
|
- `normalized-transcript.json`
|
||||||
|
- `normalization-summary.json`
|
||||||
|
- `chunking-summary.json`
|
||||||
|
- `utilization-diagnostics.json`
|
||||||
|
- `correction-ledger.json`
|
||||||
|
- `invocation.json`
|
||||||
|
- `effective-config.json`
|
||||||
|
- `report.json`
|
||||||
|
- `error.log` (failure)
|
||||||
|
|
||||||
|
## Process Report Mapping
|
||||||
|
|
||||||
|
`internal/framework/processreport` maps runner/CLI execution facts into `reporting.ProcessReport`.
|
||||||
|
|
||||||
|
Report metadata fields include:
|
||||||
|
- `report_schema_name` (`audita-process-report`)
|
||||||
|
- `report_schema_version` (`v1`)
|
||||||
|
- `output_schema`
|
||||||
|
- `config_version` (when file config exists)
|
||||||
|
|
||||||
|
The report includes:
|
||||||
|
- top-level status/error phase/error message;
|
||||||
|
- normalization/chunking summaries;
|
||||||
|
- diagnostics metadata paths;
|
||||||
|
- per-module results and module summary.
|
||||||
|
|
||||||
|
## Correction Ledger
|
||||||
|
|
||||||
|
`internal/framework/processreport/BuildCorrectionLedger` flattens run results into `correction-ledger.json` entries.
|
||||||
|
|
||||||
|
Dispositions:
|
||||||
|
- `applied`
|
||||||
|
- `skipped`
|
||||||
|
- `rejected`
|
||||||
|
- `failed`
|
||||||
|
|
||||||
|
Validator decisions are split into deterministic and LLM-backed groups using validator metadata classification.
|
||||||
|
|
||||||
|
## Report Write Paths
|
||||||
|
|
||||||
|
- run directory always attempts to write `report.json` when possible;
|
||||||
|
- optional `--report-json` writes an external report file;
|
||||||
|
- on failure paths, report writing is best-effort and does not mask primary run errors.
|
||||||
|
|
||||||
|
## Retention Interaction
|
||||||
|
|
||||||
|
Current retention behavior:
|
||||||
|
- failed runs are retained;
|
||||||
|
- `always` keeps successful runs;
|
||||||
|
- `auto` removes only clean successful runs;
|
||||||
|
- `never` currently retains successful runs in current implementation.
|
||||||
|
|
||||||
|
## Redaction
|
||||||
|
|
||||||
|
Redacted data expectations:
|
||||||
|
- effective config artifact uses config redaction;
|
||||||
|
- diagnostics payloads and surfaced errors use LLM secret redaction;
|
||||||
|
- reports should not include raw API key values.
|
||||||
|
|
||||||
|
## Key Tests
|
||||||
|
|
||||||
|
- `internal/core/diagnostics/*_test.go`
|
||||||
|
- `internal/framework/processreport/*_test.go`
|
||||||
|
- `internal/core/reporting/report_test.go`
|
||||||
|
- `internal/cli/run_test.go`
|
||||||
|
- `cmd/audita/main_integration_test.go`
|
||||||
67
docs/internal/llm-runtime.md
Normal file
67
docs/internal/llm-runtime.md
Normal file
@@ -0,0 +1,67 @@
|
|||||||
|
# Audita LLM Runtime
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document describes the structured LLM runtime and scheduler behavior.
|
||||||
|
|
||||||
|
## Client Boundary
|
||||||
|
|
||||||
|
All runtime LLM calls go through `contracts.StructuredLLMClient`.
|
||||||
|
|
||||||
|
Primary adapter:
|
||||||
|
- `internal/framework/llm/OpenAICompatibleClient`
|
||||||
|
|
||||||
|
## Request/Response Behavior
|
||||||
|
|
||||||
|
The OpenAI-compatible adapter sends chat completions requests with:
|
||||||
|
- model;
|
||||||
|
- messages;
|
||||||
|
- `response_format.type = json_schema`;
|
||||||
|
- strict schema envelope (`name`, `schema`, `strict=true`).
|
||||||
|
|
||||||
|
The response is decoded into the requested structured output target.
|
||||||
|
|
||||||
|
## Response Schema Registry
|
||||||
|
|
||||||
|
Structured response schemas are registered in `internal/framework/responseschema`:
|
||||||
|
- `correction_set`
|
||||||
|
- `validator_decision_set`
|
||||||
|
|
||||||
|
Each schema includes stable diagnostics metadata (`id`, `version`, `name`, `sha256`).
|
||||||
|
|
||||||
|
## Retries and Error Handling
|
||||||
|
|
||||||
|
Adapter retries apply to retryable conditions (for example transport/decoding/retryable status classes) up to configured `max_retries`.
|
||||||
|
|
||||||
|
Errors are sanitized to redact configured API-key values before surfacing.
|
||||||
|
|
||||||
|
Malformed structured output detection is shared through `internal/framework/structuredoutput` and is used by:
|
||||||
|
- proposal generation;
|
||||||
|
- LLM-backed validators.
|
||||||
|
|
||||||
|
## Scheduling and Concurrency
|
||||||
|
|
||||||
|
`internal/framework/llm/Scheduler` provides FIFO, context-aware permit gating.
|
||||||
|
|
||||||
|
Runner composes scheduler limits across:
|
||||||
|
- total LLM concurrency;
|
||||||
|
- proposal LLM concurrency;
|
||||||
|
- validation LLM concurrency.
|
||||||
|
|
||||||
|
Scheduler release is guarded to avoid permit leaks on cancellation/error.
|
||||||
|
|
||||||
|
## Diagnostics and Redaction
|
||||||
|
|
||||||
|
`internal/framework/llm/DiagnosticsWriter` writes request/response/error artifacts.
|
||||||
|
|
||||||
|
Configured secrets are derived from `llm.ConfiguredSecrets(cfg)` and redacted from:
|
||||||
|
- diagnostics payloads;
|
||||||
|
- surfaced runtime/adapter errors.
|
||||||
|
|
||||||
|
## Key Tests
|
||||||
|
|
||||||
|
- `internal/framework/llm/openai_compatible_client_test.go`
|
||||||
|
- `internal/framework/llm/scheduler_test.go`
|
||||||
|
- `internal/framework/llm/diagnostics_test.go`
|
||||||
|
- `internal/framework/responseschema/registry_test.go`
|
||||||
|
- `internal/framework/structuredoutput/malformed_test.go`
|
||||||
58
docs/internal/modules.md
Normal file
58
docs/internal/modules.md
Normal file
@@ -0,0 +1,58 @@
|
|||||||
|
# Audita Modules
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document covers module contracts and built-in module packages.
|
||||||
|
|
||||||
|
## Module Contract
|
||||||
|
|
||||||
|
Modules implement `contracts.TranscriptModule`:
|
||||||
|
- `Key()`
|
||||||
|
- `ReplacementPolicy()`
|
||||||
|
- `Validators()`
|
||||||
|
- `Propose(ctx, req)`
|
||||||
|
|
||||||
|
Runner resolves configured module specs to module instances through `internal/framework/modules`.
|
||||||
|
|
||||||
|
## Built-In Modules
|
||||||
|
|
||||||
|
Current module packages:
|
||||||
|
- `internal/modules/glossary`
|
||||||
|
- `internal/modules/homophones`
|
||||||
|
- `internal/modules/spoken_word`
|
||||||
|
- `internal/modules/grammar`
|
||||||
|
|
||||||
|
Current replacement policies:
|
||||||
|
- `glossary`: `replace_all`
|
||||||
|
- `homophones`: `require_unique`
|
||||||
|
- `spoken_word`: `require_unique`
|
||||||
|
- `grammar`: `require_unique`
|
||||||
|
|
||||||
|
## Proposal Generation Ownership
|
||||||
|
|
||||||
|
Shared proposal-generation plumbing is centralized in:
|
||||||
|
- `internal/framework/proposal_generation`
|
||||||
|
|
||||||
|
Module packages own:
|
||||||
|
- prompt selection (`internal/prompts` prompt IDs);
|
||||||
|
- module-specific prompt payload construction.
|
||||||
|
|
||||||
|
Shared prompt helpers live in `internal/framework/promptcontext`.
|
||||||
|
|
||||||
|
## Validator Chain Ownership
|
||||||
|
|
||||||
|
Built-in chains are resolved in `internal/validators` per module key.
|
||||||
|
Module packages call the built-in chain resolver at construction.
|
||||||
|
|
||||||
|
## Failure and Warning Behavior
|
||||||
|
|
||||||
|
- module setup failures surface as `runner_setup` or module setup errors;
|
||||||
|
- module runtime failures surface as `runner_execution` with partial module results preserved;
|
||||||
|
- malformed structured proposal payloads are downgraded to warnings and section-level proposal rejection.
|
||||||
|
|
||||||
|
## Key Tests
|
||||||
|
|
||||||
|
- `internal/modules/*/module_test.go`
|
||||||
|
- `internal/framework/modules/registry_test.go`
|
||||||
|
- `internal/framework/proposal_generation/*_test.go`
|
||||||
|
- `internal/cli/run_test.go` (pipeline/report integration)
|
||||||
46
docs/internal/output-schemas.md
Normal file
46
docs/internal/output-schemas.md
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
# Audita Output Schemas
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document describes the implemented transcript output schema registry.
|
||||||
|
|
||||||
|
## Registry Ownership
|
||||||
|
|
||||||
|
Output schema registry is owned by `internal/core/outputschema`.
|
||||||
|
|
||||||
|
Supported schema keys:
|
||||||
|
- `bare-segments`
|
||||||
|
- `audita-v1`
|
||||||
|
|
||||||
|
## Schemas
|
||||||
|
|
||||||
|
`bare-segments`:
|
||||||
|
- top-level JSON array of transcript segments.
|
||||||
|
|
||||||
|
`audita-v1`:
|
||||||
|
- top-level JSON object with:
|
||||||
|
- `schema: "audita-v1"`
|
||||||
|
- `version: "v1"`
|
||||||
|
- `segments: [...]`
|
||||||
|
|
||||||
|
Segment fields include `id`, `speaker`, `start`, `end`, `text`, and optional `categories`.
|
||||||
|
|
||||||
|
## Validation and Resolution
|
||||||
|
|
||||||
|
Config validation and runtime resolution both reject unsupported schema keys.
|
||||||
|
|
||||||
|
Unknown schema keys fail with `unsupported output schema` before output emission.
|
||||||
|
|
||||||
|
## Output Emission
|
||||||
|
|
||||||
|
The selected schema is used by `audita process` when writing:
|
||||||
|
- output file (`--output`) or
|
||||||
|
- stdout (when no `--output`).
|
||||||
|
|
||||||
|
Report metadata records selected `output_schema`.
|
||||||
|
|
||||||
|
## Key Tests
|
||||||
|
|
||||||
|
- `internal/core/outputschema/registry_test.go`
|
||||||
|
- `internal/core/config/config_test.go`
|
||||||
|
- `internal/cli/run_test.go`
|
||||||
92
docs/internal/overview.md
Normal file
92
docs/internal/overview.md
Normal file
@@ -0,0 +1,92 @@
|
|||||||
|
# Audita Internal Overview
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document is the internal architecture entry point for developers and coding agents.
|
||||||
|
|
||||||
|
It summarizes:
|
||||||
|
- package boundaries;
|
||||||
|
- the main `process` execution path;
|
||||||
|
- where to add new code safely.
|
||||||
|
|
||||||
|
## Package Map
|
||||||
|
|
||||||
|
CLI and command orchestration:
|
||||||
|
- `internal/cli`
|
||||||
|
|
||||||
|
Core deterministic components:
|
||||||
|
- `internal/core/config`
|
||||||
|
- `internal/core/schema`
|
||||||
|
- `internal/core/normalization`
|
||||||
|
- `internal/core/chunking`
|
||||||
|
- `internal/core/outputschema`
|
||||||
|
- `internal/core/diagnostics`
|
||||||
|
- `internal/core/reporting`
|
||||||
|
- `internal/core/modulecatalog`
|
||||||
|
|
||||||
|
Framework orchestration and contracts:
|
||||||
|
- `internal/framework/contracts`
|
||||||
|
- `internal/framework/modules`
|
||||||
|
- `internal/framework/proposal_generation`
|
||||||
|
- `internal/framework/proposals`
|
||||||
|
- `internal/framework/runner`
|
||||||
|
- `internal/framework/validators`
|
||||||
|
- `internal/framework/llm`
|
||||||
|
- `internal/framework/responseschema`
|
||||||
|
- `internal/framework/structuredoutput`
|
||||||
|
- `internal/framework/processreport`
|
||||||
|
- `internal/framework/promptcontext`
|
||||||
|
- `internal/framework/stagename`
|
||||||
|
|
||||||
|
Domain implementations:
|
||||||
|
- `internal/modules/*`
|
||||||
|
- `internal/validators/*`
|
||||||
|
- `internal/prompts`
|
||||||
|
|
||||||
|
## Main Execution Path (`audita process`)
|
||||||
|
|
||||||
|
High-level flow:
|
||||||
|
1. CLI loads effective config and validates CLI requirements.
|
||||||
|
2. Run directory is created and invocation/effective config artifacts are written.
|
||||||
|
3. Transcript/glossary files are loaded and parsed.
|
||||||
|
4. Transcript is normalized and chunked.
|
||||||
|
5. `runner.Run` executes configured module instances.
|
||||||
|
6. Proposals are validated, applied deterministically, and serialized in selected output schema.
|
||||||
|
7. Process report, utilization diagnostics, correction ledger, and retention decisions are finalized.
|
||||||
|
|
||||||
|
## Boundary Summary
|
||||||
|
|
||||||
|
- `internal/core/*` owns deterministic, reusable logic and persistence-independent rules.
|
||||||
|
- `internal/framework/*` owns orchestration contracts and reusable runtime plumbing.
|
||||||
|
- `internal/modules/*` owns module-specific proposal behavior and prompt usage.
|
||||||
|
- `internal/validators/*` owns validator composition and built-in chain assembly.
|
||||||
|
- `internal/prompts` owns embedded prompt assets and metadata registry.
|
||||||
|
|
||||||
|
## Where To Add New Code
|
||||||
|
|
||||||
|
Add config fields:
|
||||||
|
- `internal/core/config`
|
||||||
|
|
||||||
|
Add module behavior:
|
||||||
|
- one package under `internal/modules/<module_key>`
|
||||||
|
- registration/wiring through `internal/framework/modules` and config module list
|
||||||
|
|
||||||
|
Add validators:
|
||||||
|
- implementation under `internal/validators/<validator_key>`
|
||||||
|
- registry/chain wiring in `internal/validators`
|
||||||
|
|
||||||
|
Add runtime orchestration behavior:
|
||||||
|
- `internal/framework/*` (runner/proposal/validator/LLM plumbing)
|
||||||
|
|
||||||
|
Add CLI surface:
|
||||||
|
- `internal/cli`
|
||||||
|
|
||||||
|
## Related Internal Docs
|
||||||
|
|
||||||
|
- [`docs/internal/pipeline.md`](pipeline.md)
|
||||||
|
- [`docs/internal/modules.md`](modules.md)
|
||||||
|
- [`docs/internal/validators.md`](validators.md)
|
||||||
|
- [`docs/internal/llm-runtime.md`](llm-runtime.md)
|
||||||
|
- [`docs/internal/diagnostics-reporting.md`](diagnostics-reporting.md)
|
||||||
|
- [`docs/internal/prompts.md`](prompts.md)
|
||||||
|
- [`docs/internal/output-schemas.md`](output-schemas.md)
|
||||||
79
docs/internal/pipeline.md
Normal file
79
docs/internal/pipeline.md
Normal file
@@ -0,0 +1,79 @@
|
|||||||
|
# Audita Internal Pipeline
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document describes the implemented `audita process` pipeline.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
Pipeline inputs are:
|
||||||
|
- effective config (`internal/core/config`);
|
||||||
|
- transcript JSON (`internal/core/schema`);
|
||||||
|
- glossary YAML (`internal/core/schema`).
|
||||||
|
|
||||||
|
## Pipeline Phases
|
||||||
|
|
||||||
|
1. Input loading and schema validation
|
||||||
|
- transcript and glossary files are read and parsed.
|
||||||
|
- schema failures stop the run with `transcript_schema` or `glossary_schema`.
|
||||||
|
|
||||||
|
2. Normalization
|
||||||
|
- canonical transcript segments are normalized by configured gap/duration/token settings.
|
||||||
|
- normalization summary artifacts are written.
|
||||||
|
|
||||||
|
3. Chunking
|
||||||
|
- normalized transcript is chunked with configured max/min tokens and target sections.
|
||||||
|
|
||||||
|
4. Module proposal generation
|
||||||
|
- runner executes configured module instances in sequence.
|
||||||
|
- each module proposes corrections per section.
|
||||||
|
- per-section proposal generation can run concurrently.
|
||||||
|
|
||||||
|
5. Validator filtering
|
||||||
|
- validators run on candidate proposals before apply.
|
||||||
|
- deterministic validators run before LLM-backed validators.
|
||||||
|
- LLM validator inputs are batched by max prompt token limit.
|
||||||
|
|
||||||
|
6. Deterministic apply
|
||||||
|
- approved proposals are applied via replacement policy.
|
||||||
|
- applied/skipped/rejected outcomes are recorded.
|
||||||
|
|
||||||
|
7. Output and reporting
|
||||||
|
- final transcript is serialized with selected output schema.
|
||||||
|
- report, utilization diagnostics, and correction ledger are written.
|
||||||
|
- retention policy is applied to run directory.
|
||||||
|
|
||||||
|
## Runner Outputs
|
||||||
|
|
||||||
|
`runner.Run` returns:
|
||||||
|
- final transcript;
|
||||||
|
- per-module results;
|
||||||
|
- utilization diagnostics.
|
||||||
|
|
||||||
|
CLI/reporting then map this into process report and diagnostics artifacts.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Representative failure phases include:
|
||||||
|
- `run_dir_creation`
|
||||||
|
- `transcript_read`, `glossary_read`
|
||||||
|
- `transcript_schema`, `glossary_schema`
|
||||||
|
- `chunking`
|
||||||
|
- `runner_setup`, `runner_execution`
|
||||||
|
- `output_schema`, `serialization`, `output_write`, `stdout_write`
|
||||||
|
|
||||||
|
When diagnostics are available, failure stderr includes diagnostics path.
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- module execution order follows configured module sequence;
|
||||||
|
- proposal/validator nondeterminism is isolated before deterministic apply;
|
||||||
|
- proposal indices are assigned deterministically by section order;
|
||||||
|
- output/report artifacts are generated from run results, not speculative state.
|
||||||
|
|
||||||
|
## Key Tests
|
||||||
|
|
||||||
|
- `internal/framework/runner/runner_test.go`
|
||||||
|
- `internal/framework/proposal_generation/*_test.go`
|
||||||
|
- `internal/cli/run_test.go`
|
||||||
|
- `cmd/audita/main_integration_test.go`
|
||||||
62
docs/internal/prompts.md
Normal file
62
docs/internal/prompts.md
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
# Audita Prompt Registry
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document describes embedded prompt assets, prompt metadata, and rendering behavior.
|
||||||
|
|
||||||
|
## Registry Ownership
|
||||||
|
|
||||||
|
Prompt registry lives in `internal/prompts` and embeds assets under `internal/prompts/assets/**`.
|
||||||
|
|
||||||
|
Registered prompt IDs:
|
||||||
|
- `modules.glossary.proposal`
|
||||||
|
- `modules.homophones.proposal`
|
||||||
|
- `modules.spoken_word.proposal`
|
||||||
|
- `modules.grammar.proposal`
|
||||||
|
- `validators.spoken_form_plausibility`
|
||||||
|
- `validators.meaning_reversal_review`
|
||||||
|
- `validators.editorial_review`
|
||||||
|
- `validators.grammar_review`
|
||||||
|
- `validators.spoken_word_review`
|
||||||
|
|
||||||
|
## Metadata Model
|
||||||
|
|
||||||
|
Each prompt has metadata:
|
||||||
|
- `prompt_id`
|
||||||
|
- `prompt_version`
|
||||||
|
- `prompt_source`
|
||||||
|
- `embedded_path`
|
||||||
|
- `sha256`
|
||||||
|
|
||||||
|
Current source/version values:
|
||||||
|
- `prompt_source = builtin`
|
||||||
|
- `prompt_version = v1`
|
||||||
|
|
||||||
|
## Rendering
|
||||||
|
|
||||||
|
`prompts.RenderUserSystem(promptID, data)` renders system/user templates.
|
||||||
|
|
||||||
|
Template behavior:
|
||||||
|
- uses Go `text/template`;
|
||||||
|
- `missingkey=error` is enabled;
|
||||||
|
- output is trimmed.
|
||||||
|
|
||||||
|
A shared hardening fragment is embedded once and referenced by prompt templates.
|
||||||
|
|
||||||
|
## Prompt Context Inputs
|
||||||
|
|
||||||
|
Shared prompt payload helpers:
|
||||||
|
- transcript section JSON (`internal/framework/promptcontext/MarshalTranscriptSectionJSON`)
|
||||||
|
- transcript description block (`TranscriptDescriptionBlock`)
|
||||||
|
|
||||||
|
Modules and LLM validators provide typed data maps to render prompt assets.
|
||||||
|
|
||||||
|
## Diagnostics Integration
|
||||||
|
|
||||||
|
Prompt metadata is attached to proposal/validator diagnostics request metadata using `Metadata.DiagnosticsMap()`.
|
||||||
|
|
||||||
|
## Key Tests
|
||||||
|
|
||||||
|
- `internal/prompts/registry_test.go`
|
||||||
|
- `internal/framework/promptcontext/*_test.go`
|
||||||
|
- module and validator prompt builder tests
|
||||||
72
docs/internal/validators.md
Normal file
72
docs/internal/validators.md
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
# Audita Validators
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document describes validator composition, execution order, and decision handling.
|
||||||
|
|
||||||
|
## Ownership
|
||||||
|
|
||||||
|
Built-in validator keys and chains:
|
||||||
|
- `internal/validators`
|
||||||
|
|
||||||
|
Shared validator runtime mechanics:
|
||||||
|
- `internal/framework/validators`
|
||||||
|
|
||||||
|
Execution-class metadata:
|
||||||
|
- `internal/validators/metadata`
|
||||||
|
|
||||||
|
## Built-In Validator Keys
|
||||||
|
|
||||||
|
Deterministic:
|
||||||
|
- `proposal_shape`
|
||||||
|
- `confidence_threshold`
|
||||||
|
- `original_text_presence`
|
||||||
|
- `non_empty_corrected_text`
|
||||||
|
- `no_effect`
|
||||||
|
- `protected_terms`
|
||||||
|
|
||||||
|
LLM-backed:
|
||||||
|
- `spoken_form_plausibility`
|
||||||
|
- `meaning_reversal_review`
|
||||||
|
- `editorial_review`
|
||||||
|
|
||||||
|
## Built-In Chains
|
||||||
|
|
||||||
|
Module chains are defined in `internal/validators/chains.go`.
|
||||||
|
Glossary, homophones, spoken_word, and grammar each resolve a fixed ordered chain.
|
||||||
|
|
||||||
|
## Runtime Execution
|
||||||
|
|
||||||
|
For each module section:
|
||||||
|
1. run deterministic validators;
|
||||||
|
2. run LLM-backed validators;
|
||||||
|
3. record decisions and warnings;
|
||||||
|
4. carry only approved proposals forward.
|
||||||
|
|
||||||
|
Decision cardinality is enforced: each candidate proposal must receive exactly one decision per validator.
|
||||||
|
|
||||||
|
## LLM Validator Batching
|
||||||
|
|
||||||
|
LLM validators:
|
||||||
|
- build canonical validation request payloads;
|
||||||
|
- batch by `validation_max_prompt_tokens`;
|
||||||
|
- call structured LLM client using response schema registry.
|
||||||
|
|
||||||
|
Oversized single proposals are rejected with `validator_input_too_large`.
|
||||||
|
Malformed LLM validator responses are downgraded to warnings and rejected batch decisions.
|
||||||
|
|
||||||
|
## Decision and Rejection Reporting
|
||||||
|
|
||||||
|
Runner records:
|
||||||
|
- `validator_decisions`
|
||||||
|
- `validator_rejected`
|
||||||
|
- warning records (including malformed response warnings)
|
||||||
|
|
||||||
|
Correction ledger classifies deterministic vs LLM validator decisions using canonical metadata classes.
|
||||||
|
|
||||||
|
## Key Tests
|
||||||
|
|
||||||
|
- `internal/validators/*_test.go`
|
||||||
|
- `internal/framework/validators/*_test.go`
|
||||||
|
- `internal/framework/processreport/correction_ledger_test.go`
|
||||||
|
- `internal/cli/run_test.go`
|
||||||
113
docs/operations.md
Normal file
113
docs/operations.md
Normal file
@@ -0,0 +1,113 @@
|
|||||||
|
# Audita Operations
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This document covers operational behavior for `audita process` as currently implemented:
|
||||||
|
- run lifecycle;
|
||||||
|
- output and report files;
|
||||||
|
- diagnostics artifacts;
|
||||||
|
- run-directory retention behavior;
|
||||||
|
- failure inspection and recovery.
|
||||||
|
|
||||||
|
For command syntax, see [`docs/cli.md`](cli.md).
|
||||||
|
|
||||||
|
## Process Run Lifecycle
|
||||||
|
|
||||||
|
A `process` run performs these high-level steps:
|
||||||
|
1. load effective config (defaults + optional file + env + CLI);
|
||||||
|
2. create a per-run diagnostics directory;
|
||||||
|
3. load transcript JSON and glossary YAML;
|
||||||
|
4. parse/validate input schemas;
|
||||||
|
5. normalize transcript and compute chunking;
|
||||||
|
6. run configured modules/validators;
|
||||||
|
7. serialize output schema and write transcript output;
|
||||||
|
8. build and write process report;
|
||||||
|
9. apply run-directory retention.
|
||||||
|
|
||||||
|
If a failure happens after diagnostics initialization, the run writes failure details and returns nonzero.
|
||||||
|
|
||||||
|
## Output Files
|
||||||
|
|
||||||
|
Transcript output:
|
||||||
|
- when `--output <path>` is set, corrected transcript JSON is written to that file;
|
||||||
|
- when `--output` is omitted, corrected transcript JSON is written to stdout.
|
||||||
|
|
||||||
|
Report output:
|
||||||
|
- when `--report-json <path>` is set, Audita writes a process report JSON file;
|
||||||
|
- the run directory also writes its own `report.json` artifact.
|
||||||
|
|
||||||
|
On success with `--output`, stdout is expected to be empty.
|
||||||
|
|
||||||
|
## Diagnostics Directory
|
||||||
|
|
||||||
|
By default, runs use `work_dir` from effective config (default `/tmp/audita`).
|
||||||
|
Each run directory is created under the work dir using a generated ID like `run-<unix-nanos>`.
|
||||||
|
|
||||||
|
Top-level diagnostics artifacts:
|
||||||
|
- `source-transcript.json`
|
||||||
|
- `source-transcript-parsed.json`
|
||||||
|
- `normalized-transcript.json`
|
||||||
|
- `normalization-summary.json`
|
||||||
|
- `chunking-summary.json`
|
||||||
|
- `utilization-diagnostics.json`
|
||||||
|
- `correction-ledger.json`
|
||||||
|
- `invocation.json`
|
||||||
|
- `effective-config.json` (redacted)
|
||||||
|
- `report.json`
|
||||||
|
- `error.log` (failure runs)
|
||||||
|
|
||||||
|
Report diagnostics metadata includes resolved paths to these artifacts.
|
||||||
|
|
||||||
|
## Correction Ledger and Utilization Diagnostics
|
||||||
|
|
||||||
|
`correction-ledger.json` records correction dispositions:
|
||||||
|
- `applied`
|
||||||
|
- `skipped`
|
||||||
|
- `rejected`
|
||||||
|
- `failed`
|
||||||
|
|
||||||
|
`utilization-diagnostics.json` records effective concurrency and execution timing summaries for run/module/validator activity.
|
||||||
|
|
||||||
|
## Retention Behavior
|
||||||
|
|
||||||
|
Retention is controlled by `work_dir_retention` (`auto|always|never`).
|
||||||
|
|
||||||
|
Current behavior:
|
||||||
|
- failed runs are always retained;
|
||||||
|
- `always`: successful runs are retained;
|
||||||
|
- `auto`: successful runs are retained only when skipped/rejected corrections occurred; clean successful runs are removed;
|
||||||
|
- `never`: successful runs are currently retained (same net retention outcome as `always` in current implementation).
|
||||||
|
|
||||||
|
Even when a successful run directory is removed under `auto`, an explicit `--report-json` file is still preserved at its target path.
|
||||||
|
|
||||||
|
## Failure Inspection
|
||||||
|
|
||||||
|
For failed runs:
|
||||||
|
1. read stderr for the top-level failure and diagnostics path;
|
||||||
|
2. open `error.log` in the reported run directory;
|
||||||
|
3. inspect run `report.json` (`status`, `error_phase`, `error_message`);
|
||||||
|
4. inspect related artifacts referenced by report diagnostics metadata.
|
||||||
|
|
||||||
|
Typical `error_phase` values include:
|
||||||
|
- `transcript_read`
|
||||||
|
- `glossary_read`
|
||||||
|
- `transcript_schema`
|
||||||
|
- `glossary_schema`
|
||||||
|
- `chunking`
|
||||||
|
- `runner_setup`
|
||||||
|
- `runner_execution`
|
||||||
|
- `output_schema`
|
||||||
|
- `serialization`
|
||||||
|
- `output_write`
|
||||||
|
- `stdout_write`
|
||||||
|
|
||||||
|
## Recovery Guidance
|
||||||
|
|
||||||
|
Safe recovery pattern:
|
||||||
|
1. correct the immediate input/config/output-path problem;
|
||||||
|
2. rerun with `--work-dir-retention always` during debugging;
|
||||||
|
3. once stable, restore your normal retention mode.
|
||||||
|
|
||||||
|
Not implemented:
|
||||||
|
- resume/checkpoint APIs
|
||||||
|
- remote diagnostics/report storage
|
||||||
@@ -25,14 +25,12 @@ The current built-in modules are `glossary`, `homophones`, `spoken_word`, and `g
|
|||||||
|
|
||||||
For external behavior and compatibility details, prefer links to existing behavior docs:
|
For external behavior and compatibility details, prefer links to existing behavior docs:
|
||||||
|
|
||||||
- [Architecture overview](../architecture/architecture.md)
|
- [CLI reference](../cli.md)
|
||||||
- [Public contract](../architecture/public-contract.md)
|
- [Configuration](../config.md)
|
||||||
- [Diagnostics](../architecture/diagnostics.md)
|
- [Operations](../operations.md)
|
||||||
- [Structured LLM](../architecture/structured-llm.md)
|
- [Troubleshooting](../troubleshooting.md)
|
||||||
- [Validators](../architecture/validators.md)
|
- [Integration docs](../integrations/subprocess.md)
|
||||||
- [Prompts](../architecture/prompts.md)
|
- [Internal docs](../internal/overview.md)
|
||||||
- [Output schemas](../architecture/output-schemas.md)
|
|
||||||
- [Configuration](../configuration.md)
|
|
||||||
|
|
||||||
## Core Design Principles
|
## Core Design Principles
|
||||||
|
|
||||||
@@ -124,7 +122,7 @@ Config behavior is owned by `internal/core/config`; command usage and process wi
|
|||||||
|
|
||||||
`audita process` uses implemented precedence: defaults, config file, environment, then CLI flags. `config validate` validates defaults plus a file config and intentionally does not apply environment overrides. `config print-effective` applies defaults, file config, and environment overrides, then prints redacted JSON.
|
`audita process` uses implemented precedence: defaults, config file, environment, then CLI flags. `config validate` validates defaults plus a file config and intentionally does not apply environment overrides. `config print-effective` applies defaults, file config, and environment overrides, then prints redacted JSON.
|
||||||
|
|
||||||
Do not duplicate full CLI or config reference material here. Use [Configuration](../configuration.md), the README, and [Public contract](../architecture/public-contract.md) for current external behavior.
|
Do not duplicate full CLI or config reference material here. Use [Configuration](../config.md), [CLI reference](../cli.md), [Operations](../operations.md), and integration docs under [`docs/integrations/`](../integrations/subprocess.md) for current external behavior.
|
||||||
|
|
||||||
When adding config fields or CLI flags, update:
|
When adding config fields or CLI flags, update:
|
||||||
|
|
||||||
|
|||||||
124
docs/policy/development.md
Normal file
124
docs/policy/development.md
Normal file
@@ -0,0 +1,124 @@
|
|||||||
|
# Audita Development Workflow
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This is the canonical contributor workflow for Audita maintainers and coding agents.
|
||||||
|
|
||||||
|
It defines:
|
||||||
|
- repository layout and boundaries;
|
||||||
|
- setup and test commands;
|
||||||
|
- expectations for code changes;
|
||||||
|
- how to add config, CLI, modules, validators, docs, and examples.
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
Prerequisites:
|
||||||
|
- Go `1.24` or newer.
|
||||||
|
|
||||||
|
Common commands:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go test ./...
|
||||||
|
go build ./cmd/audita
|
||||||
|
```
|
||||||
|
|
||||||
|
## Repository Layout
|
||||||
|
|
||||||
|
Top-level areas:
|
||||||
|
- `cmd/audita`: executable entrypoint.
|
||||||
|
- `internal/cli`: command parsing and process/config command orchestration.
|
||||||
|
- `internal/core`: deterministic config/schema/normalization/chunking/output/diagnostics/reporting logic.
|
||||||
|
- `internal/framework`: runner orchestration, contracts, proposal generation/application, validators runtime, LLM runtime, response schemas.
|
||||||
|
- `internal/modules/*`: module-specific correction behavior.
|
||||||
|
- `internal/validators/*`: validator implementations, chains, and metadata.
|
||||||
|
- `internal/prompts`: embedded prompts and prompt metadata.
|
||||||
|
- `docs/`: canonical documentation.
|
||||||
|
- `examples/`: maintained copyable inputs/configs.
|
||||||
|
|
||||||
|
## Change Workflow
|
||||||
|
|
||||||
|
1. Confirm scope and behavior contract before editing.
|
||||||
|
2. Make focused changes in the appropriate ownership area.
|
||||||
|
3. Add or update tests for changed behavior.
|
||||||
|
4. Run targeted package tests for touched areas.
|
||||||
|
5. Run `go test ./...` for substantial changes.
|
||||||
|
6. Update docs/examples when external behavior changes.
|
||||||
|
|
||||||
|
## How To Add or Change Configuration
|
||||||
|
|
||||||
|
1. Add fields/defaults/validation under `internal/core/config`.
|
||||||
|
2. Apply source precedence correctly (defaults, file, env, CLI for `process`).
|
||||||
|
3. Ensure `config validate` remains file-only and `config print-effective` remains redacted.
|
||||||
|
4. Update tests in `internal/core/config` and related CLI tests.
|
||||||
|
5. Update [`docs/config.md`](../config.md) and relevant examples under `examples/`.
|
||||||
|
|
||||||
|
## How To Add or Change CLI Behavior
|
||||||
|
|
||||||
|
1. Implement parsing/wiring in `internal/cli`.
|
||||||
|
2. Keep stdout/stderr and exit behavior compatible unless intentional and documented.
|
||||||
|
3. Update CLI tests under `internal/cli` and integration tests under `cmd/audita`.
|
||||||
|
4. Update [`docs/cli.md`](../cli.md) and related integration docs.
|
||||||
|
|
||||||
|
## How To Add or Change Modules
|
||||||
|
|
||||||
|
1. Add or update one module package under `internal/modules/<module_key>`.
|
||||||
|
2. Keep module-specific prompt ownership in the module + `internal/prompts`.
|
||||||
|
3. Wire module registration/catalog resolution through framework/core module catalog code.
|
||||||
|
4. Verify replacement policy and validator chain selection.
|
||||||
|
5. Add/update module tests and proposal-generation tests.
|
||||||
|
6. Update internal docs when behavior/contracts change.
|
||||||
|
|
||||||
|
## How To Add or Change Validators
|
||||||
|
|
||||||
|
1. Implement validator behavior in `internal/validators` and shared runtime pieces in `internal/framework/validators` only when needed.
|
||||||
|
2. Preserve stable validator keys and decision semantics where already exposed.
|
||||||
|
3. Keep deterministic vs LLM-backed execution-class behavior explicit.
|
||||||
|
4. Add/update validator, chain, batching, and malformed-output tests.
|
||||||
|
5. Update validator documentation when external or developer-facing behavior changes.
|
||||||
|
|
||||||
|
## Documentation and Examples Expectations
|
||||||
|
|
||||||
|
- Keep one canonical home per topic (see [`docs/policy/documentation.md`](documentation.md)).
|
||||||
|
- Do not document future/unimplemented behavior outside `docs/roadmap/`.
|
||||||
|
- Keep command examples and config/examples in sync with current code.
|
||||||
|
- Keep examples secret-free and copyable.
|
||||||
|
|
||||||
|
## Practical Validation Checklist
|
||||||
|
|
||||||
|
Use this checklist for meaningful runtime-impacting changes:
|
||||||
|
|
||||||
|
1. Run core tests:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go test ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Verify config commands and examples:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go run ./cmd/audita config validate --config examples/minimal-config.yml
|
||||||
|
go run ./cmd/audita config validate --config examples/production-config.yml
|
||||||
|
go run ./cmd/audita config print-effective --config examples/minimal-config.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Re-check subprocess/runtime contract when touching CLI/process/report paths:
|
||||||
|
- `--output` success keeps stdout empty;
|
||||||
|
- no `--output` success writes transcript JSON to stdout;
|
||||||
|
- `--report-json` writes file output and is not written to stdout;
|
||||||
|
- failures return nonzero and include diagnostics path when available.
|
||||||
|
|
||||||
|
4. Re-check diagnostics/report/redaction when touching LLM, reporting, or diagnostics code:
|
||||||
|
- report schema metadata fields remain present;
|
||||||
|
- diagnostics artifact paths remain valid;
|
||||||
|
- configured secret values remain redacted in reports/diagnostics/errors.
|
||||||
|
|
||||||
|
5. Re-check output schema behavior when touching serialization/schema code:
|
||||||
|
- default `bare-segments` behavior remains correct unless intentionally changed;
|
||||||
|
- `audita-v1` behavior remains correct unless intentionally changed;
|
||||||
|
- unsupported schemas fail validation/resolve paths clearly.
|
||||||
|
|
||||||
|
## Commit Discipline
|
||||||
|
|
||||||
|
- Keep commits scoped and reviewable.
|
||||||
|
- Avoid mixing unrelated refactors with behavior changes.
|
||||||
|
- Use concise plain-English commit messages.
|
||||||
@@ -1,128 +0,0 @@
|
|||||||
# Audita Release Checklist
|
|
||||||
|
|
||||||
Use this checklist before cutting a pre-1.0 or 1.0 release candidate.
|
|
||||||
|
|
||||||
## Core test pass
|
|
||||||
|
|
||||||
- Run:
|
|
||||||
- `go test ./...`
|
|
||||||
- Confirm tests pass without live LLM credentials and without Python dependencies.
|
|
||||||
|
|
||||||
## Config validation and precedence
|
|
||||||
|
|
||||||
- Validate a representative config:
|
|
||||||
- `audita config validate --config <path>`
|
|
||||||
- Inspect redacted effective config:
|
|
||||||
- `audita config print-effective --config <path>`
|
|
||||||
- Confirm precedence behavior:
|
|
||||||
- defaults -> file config -> environment -> CLI.
|
|
||||||
- Confirm default config search order:
|
|
||||||
- `/usr/local/etc/audita/config.yml` first, then `/etc/audita/config.yml`.
|
|
||||||
- Confirm missing both default-path config files is non-fatal when `--config`/`AUDITA_CONFIG` are unset.
|
|
||||||
|
|
||||||
## Output schema checks
|
|
||||||
|
|
||||||
- Verify default output schema remains `bare-segments`.
|
|
||||||
- Verify `--output-schema audita-v1` emits object payload with `schema` and `version`.
|
|
||||||
- Verify unknown schema (for example `seriatim-intermediate`) fails clearly.
|
|
||||||
|
|
||||||
## Subprocess contract checks
|
|
||||||
|
|
||||||
- With `--output`, verify stdout is empty on success.
|
|
||||||
- Without `--output`, verify stdout contains transcript JSON only.
|
|
||||||
- Verify `--report-json` writes file output and does not write report JSON to stdout.
|
|
||||||
- Verify failure stderr remains human-readable and includes diagnostics path when available.
|
|
||||||
- Verify nonzero exit on failures.
|
|
||||||
|
|
||||||
## Structured LLM checks
|
|
||||||
|
|
||||||
- Verify runtime uses the Audita-owned OpenAI-compatible adapter.
|
|
||||||
- Verify structured response schemas are attached via `response_format.type=json_schema`.
|
|
||||||
- Verify diagnostics metadata includes structured schema `id/version/name/sha256`.
|
|
||||||
- Verify provider output is still locally decoded/validated before use.
|
|
||||||
- Verify malformed module-stage structured payloads degrade to warnings/rejections instead of failing the run.
|
|
||||||
|
|
||||||
## Report and diagnostics schema checks
|
|
||||||
|
|
||||||
- Verify report metadata fields:
|
|
||||||
- `report_schema_name`
|
|
||||||
- `report_schema_version`
|
|
||||||
- `output_schema`
|
|
||||||
- `config_version` when file config is used.
|
|
||||||
- Verify diagnostics artifact references exist in reports:
|
|
||||||
- transcript/normalization/chunking/invocation/effective-config artifacts
|
|
||||||
- utilization diagnostics artifact
|
|
||||||
- correction ledger artifact
|
|
||||||
- error log on failures.
|
|
||||||
|
|
||||||
## Redaction checks
|
|
||||||
|
|
||||||
- Verify secrets are redacted from:
|
|
||||||
- `effective-config.json`
|
|
||||||
- run-dir and `--report-json` reports
|
|
||||||
- LLM request/response/error diagnostics payloads.
|
|
||||||
- Verify no API keys/bearer tokens leak into fixtures or outputs.
|
|
||||||
|
|
||||||
## Prompt and validator metadata checks
|
|
||||||
|
|
||||||
- Verify prompt metadata appears in LLM request metadata diagnostics:
|
|
||||||
- `prompt_id`, `prompt_version`, `prompt_source`, `embedded_path`, `sha256`.
|
|
||||||
- Verify stable validator keys appear in report decisions/rejections.
|
|
||||||
- Verify module warning records appear in reports for malformed proposal-generation payloads and malformed validator batches.
|
|
||||||
- Verify built-in validator chains resolve and execute for default and explicit module runs.
|
|
||||||
|
|
||||||
## Utilization diagnostics checks
|
|
||||||
|
|
||||||
- Verify `utilization-diagnostics.json` exists on successful runs.
|
|
||||||
- Verify partial utilization artifact behavior on controlled failure paths.
|
|
||||||
- Verify utilization fields are structurally present and nonnegative:
|
|
||||||
- effective concurrency
|
|
||||||
- run timing
|
|
||||||
- module timing summaries
|
|
||||||
- per-validator timing summaries.
|
|
||||||
|
|
||||||
## Correction ledger checks
|
|
||||||
|
|
||||||
- Verify `correction-ledger.json` exists on successful runs.
|
|
||||||
- Verify report references ledger artifact path.
|
|
||||||
- Verify ledger dispositions include applied/rejected and skipped/failed where exercised.
|
|
||||||
- Verify validator rejection and proposal-application skip remain distinct.
|
|
||||||
|
|
||||||
## Pipeline behavior checks
|
|
||||||
|
|
||||||
- Verify default full pipeline run remains:
|
|
||||||
- `glossary`, `homophones`, `glossary`, `spoken_word`, `grammar`
|
|
||||||
- with deterministic repeated instance naming (`glossary_1`, `glossary_2`).
|
|
||||||
- Verify explicit module runs (`--modules`) still work.
|
|
||||||
|
|
||||||
## Failure and cancellation checks
|
|
||||||
|
|
||||||
- Verify controlled failure paths retain diagnostics and produce best-effort failure reports.
|
|
||||||
- Verify malformed proposal-generation payloads keep exit code `0`, keep stderr empty on success, and record warnings in reports/diagnostics.
|
|
||||||
- Verify malformed validator payloads reject only the affected batch and do not fail the module.
|
|
||||||
- Verify timeout/cancellation paths exit nonzero, do not hang, and retain failure diagnostics when initialized.
|
|
||||||
|
|
||||||
## Release fixture/idempotence checks
|
|
||||||
|
|
||||||
- Run release fixtures (`internal/cli/testdata/release`) through `go test ./...`.
|
|
||||||
- Confirm fixture checks cover:
|
|
||||||
- must-apply and must-not-apply expectations
|
|
||||||
- protected-term survival
|
|
||||||
- report and diagnostics contracts
|
|
||||||
- output-schema checks
|
|
||||||
- prompt/schema metadata diagnostics
|
|
||||||
- utilization/ledger artifacts
|
|
||||||
- idempotence-oriented second pass no-op behavior with deterministic fake responses.
|
|
||||||
|
|
||||||
## Deferred-feature guardrail
|
|
||||||
|
|
||||||
- Confirm release docs do not claim support for deferred items:
|
|
||||||
- filesystem prompt overrides
|
|
||||||
- user-configurable validator chains
|
|
||||||
- arbitrary user-supplied output schemas
|
|
||||||
- resume/start-at/stop-after execution
|
|
||||||
- diff/check/propose-only modes
|
|
||||||
- generated transcript descriptions enabled by default
|
|
||||||
- interactive review UI
|
|
||||||
- UI/server wrapper
|
|
||||||
- provider benchmarking harness.
|
|
||||||
153
docs/troubleshooting.md
Normal file
153
docs/troubleshooting.md
Normal file
@@ -0,0 +1,153 @@
|
|||||||
|
# Audita Troubleshooting
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This guide lists recurring implemented failure modes for `audita process` and `audita config`.
|
||||||
|
|
||||||
|
For each entry: symptom, likely cause, inspect, and fix.
|
||||||
|
|
||||||
|
## Config Validation Fails
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
- `audita config validate --config <path>` exits nonzero.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
- missing `version`;
|
||||||
|
- unsupported config version;
|
||||||
|
- unknown YAML field;
|
||||||
|
- unsupported module key or output schema;
|
||||||
|
- invalid numeric/range/concurrency/retention values.
|
||||||
|
|
||||||
|
Inspect:
|
||||||
|
1. rerun `audita config validate --config <path>` and read stderr.
|
||||||
|
2. if needed, inspect effective config with `audita config print-effective --config <path>`.
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
- set `version: 1`;
|
||||||
|
- remove unknown fields;
|
||||||
|
- use supported module keys and output schemas (`bare-segments`, `audita-v1`);
|
||||||
|
- correct invalid values to satisfy validation constraints.
|
||||||
|
|
||||||
|
## Config File Resolution Errors
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
- `audita process` fails before processing with config-related errors like `config file not found`.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
- `--config` points to a missing path;
|
||||||
|
- `AUDITA_CONFIG` points to a missing path;
|
||||||
|
- unreadable config path.
|
||||||
|
|
||||||
|
Inspect:
|
||||||
|
1. confirm `--config` or `AUDITA_CONFIG` path exists;
|
||||||
|
2. run `audita config validate --config <path>` directly.
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
- correct the path or unset invalid `AUDITA_CONFIG`;
|
||||||
|
- fix permissions for the config file.
|
||||||
|
|
||||||
|
## Transcript or Glossary Schema Errors
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
- stderr includes `transcript_schema` or `glossary_schema` and run exits nonzero.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
- transcript is not valid JSON or has invalid segment fields;
|
||||||
|
- glossary is not valid YAML or has missing required glossary entry fields.
|
||||||
|
|
||||||
|
Inspect:
|
||||||
|
1. check stderr for parser/validation details;
|
||||||
|
2. if diagnostics were created, inspect `error.log` and run `report.json` (`error_phase`);
|
||||||
|
3. inspect `source-transcript.json` and `source-transcript-parsed.json` in the run directory.
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
- correct transcript JSON shape/content;
|
||||||
|
- correct glossary YAML shape/content and required entry fields;
|
||||||
|
- rerun validation with known-good tiny examples for comparison:
|
||||||
|
- `examples/tiny-transcript.json`
|
||||||
|
- `examples/tiny-glossary.yaml`
|
||||||
|
|
||||||
|
## LLM Runtime/Backend Failures
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
- stderr includes `runner_execution` (or backend timeout/error details) and nonzero exit.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
- unreachable/failed LLM endpoint;
|
||||||
|
- timeout/cancellation;
|
||||||
|
- runtime module execution failure.
|
||||||
|
|
||||||
|
Inspect:
|
||||||
|
1. inspect stderr for backend message details;
|
||||||
|
2. inspect run `report.json` (`error_phase`, `module_results`);
|
||||||
|
3. inspect diagnostics payloads and `error.log`.
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
- verify model/base URL/API key settings;
|
||||||
|
- increase timeout if needed;
|
||||||
|
- rerun with `--work-dir-retention always` while debugging.
|
||||||
|
|
||||||
|
## Output File Write Failure
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
- stderr includes `failed to write output file` and run exits nonzero.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
- output path directory missing;
|
||||||
|
- insufficient filesystem permissions;
|
||||||
|
- invalid output target path.
|
||||||
|
|
||||||
|
Inspect:
|
||||||
|
1. check `--output` target directory exists and is writable;
|
||||||
|
2. inspect run diagnostics `error.log` and report `error_phase`.
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
- write to a valid writable path;
|
||||||
|
- create missing directories;
|
||||||
|
- adjust permissions.
|
||||||
|
|
||||||
|
## Report File Write Failure
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
- stderr includes `failed to write report JSON file` and run exits nonzero.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
- invalid or unwritable `--report-json` target path.
|
||||||
|
|
||||||
|
Inspect:
|
||||||
|
1. verify parent directory exists and is writable;
|
||||||
|
2. inspect diagnostics `error.log` for `report_write` context.
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
- choose a writable report path;
|
||||||
|
- create missing directories;
|
||||||
|
- rerun.
|
||||||
|
|
||||||
|
## Unsupported Output Schema
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
- stderr includes `unsupported output schema` and run exits nonzero.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
- unsupported `--output-schema` value;
|
||||||
|
- unsupported `output.schema` in config.
|
||||||
|
|
||||||
|
Inspect:
|
||||||
|
1. check CLI/config schema key;
|
||||||
|
2. run `audita config validate --config <path>` when config is involved.
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
- use `bare-segments` or `audita-v1`.
|
||||||
|
|
||||||
|
## Diagnostics Directory Lookup
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
- run fails and you need artifacts for debugging.
|
||||||
|
|
||||||
|
Inspect:
|
||||||
|
1. read stderr for `audita process: diagnostics: <run-dir>`;
|
||||||
|
2. open `<run-dir>/report.json` and `<run-dir>/error.log`;
|
||||||
|
3. use diagnostics paths embedded in report metadata for artifact lookup.
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
- rerun with `--work-dir-retention always` to preserve run directories during investigation.
|
||||||
6
examples/minimal-config.yml
Normal file
6
examples/minimal-config.yml
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
version: 1
|
||||||
|
output:
|
||||||
|
schema: bare-segments
|
||||||
|
llm:
|
||||||
|
proposal:
|
||||||
|
api_key_env: AUDITA_LLM_API_KEY
|
||||||
41
examples/production-config.yml
Normal file
41
examples/production-config.yml
Normal file
@@ -0,0 +1,41 @@
|
|||||||
|
version: 1
|
||||||
|
pipeline:
|
||||||
|
modules: [glossary, homophones, glossary, spoken_word, grammar]
|
||||||
|
output:
|
||||||
|
schema: audita-v1
|
||||||
|
llm:
|
||||||
|
proposal:
|
||||||
|
base_url: https://openrouter.ai/api/v1
|
||||||
|
model: openrouter/google/gemma-4-31b-it
|
||||||
|
api_key_env: AUDITA_LLM_API_KEY
|
||||||
|
timeout: 120s
|
||||||
|
max_retries: 3
|
||||||
|
validation:
|
||||||
|
base_url: https://openrouter.ai/api/v1
|
||||||
|
model: openrouter/google/gemma-4-31b-it
|
||||||
|
api_key_env: AUDITA_VALIDATION_LLM_API_KEY
|
||||||
|
timeout: 120
|
||||||
|
max_retries: 3
|
||||||
|
concurrency:
|
||||||
|
total_llm: 2
|
||||||
|
proposal_llm: 2
|
||||||
|
validation_llm: 1
|
||||||
|
chunking:
|
||||||
|
target_sections: 8
|
||||||
|
max_section_tokens: 8192
|
||||||
|
min_section_tokens: 2048
|
||||||
|
normalization:
|
||||||
|
max_segment_gap: 4s
|
||||||
|
ellipsis_gap: 3.5s
|
||||||
|
max_segment_duration: 60s
|
||||||
|
max_segment_tokens: 2048
|
||||||
|
thresholds:
|
||||||
|
glossary: 0.8
|
||||||
|
homophones: 0.8
|
||||||
|
spoken_word: 0.8
|
||||||
|
grammar: 0.8
|
||||||
|
context:
|
||||||
|
description: "General context for domain vocabulary and speaker names."
|
||||||
|
diagnostics:
|
||||||
|
work_dir: /tmp/audita
|
||||||
|
retention: auto
|
||||||
6
examples/tiny-glossary.yaml
Normal file
6
examples/tiny-glossary.yaml
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
glossary:
|
||||||
|
- name: Audita
|
||||||
|
aliases:
|
||||||
|
- audita
|
||||||
|
category: product
|
||||||
|
summary: The Audita transcript correction CLI.
|
||||||
9
examples/tiny-transcript.json
Normal file
9
examples/tiny-transcript.json
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"speaker": "A",
|
||||||
|
"start": 0.0,
|
||||||
|
"end": 1.2,
|
||||||
|
"text": "hello world"
|
||||||
|
}
|
||||||
|
]
|
||||||
Reference in New Issue
Block a user