From 7995c416759c144c657e78bf5c50c5783c1d9953 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 16 May 2026 08:02:18 -0500 Subject: [PATCH] Update the audita integration documentation reference --- docs/integrations/audita.md | 179 +++++++++++++----------------------- 1 file changed, 64 insertions(+), 115 deletions(-) diff --git a/docs/integrations/audita.md b/docs/integrations/audita.md index e13d571..d94aeef 100644 --- a/docs/integrations/audita.md +++ b/docs/integrations/audita.md @@ -1,147 +1,96 @@ -# Audita +# Audita Subprocess Operations -Audita is a framework-first transcript correction application. The public `audita` package provides: +This document describes how parent processes should invoke `audita process` safely in production orchestration. -- deterministic transcript normalization -- token-batched module orchestration -- concrete `glossary`, `homophones`, `spoken_word`, and `grammar` modules built on reusable proposal / validator contracts -- structured run reporting and work-dir diagnostics +## Recommended command form -The previous working implementation has been preserved as `audita_prototype` inside this repository. Its full regression suite lives under `tests/audita_prototype`. - -## Development - -This project is set up for `uv`. +Use explicit file outputs for orchestrated runs: ```sh -uv sync --extra dev -uv run pytest +audita process \ + --transcript-description "Brief context that may help resolve ambiguous terms." \ + --glossary \ + --output \ + --report-json ``` -## Usage +Additional flags that may be situationally appropriate: +- `--config ` to select an explicit versioned config file. +- `--output-schema ` to select transcript output shape. +- `--work-dir ` to control diagnostics location. +- `--work-dir-retention ` to control retained run directories. +- `--total-llm-concurrency`, `--proposal-llm-concurrency`, and `--validation-llm-concurrency` when orchestration needs to set explicit LLM throughput controls. +- `--modules ...` only when intentionally overriding the default sequence. -Process a transcript with the current framework implementation: +For config-driven orchestration, validate config files in CI/preflight: ```sh -uv run audita process transcript.json --glossary glossary.yaml --output corrected.json +audita config validate --config ``` -The framework currently runs this default module sequence: +## Stdout behavior -1. `glossary` -2. `homophones` -3. `glossary` -4. `spoken_word` -5. `grammar` +- With `--output`: stdout is expected to be empty on success. +- Without `--output`: stdout contains transcript JSON only on success. +- Report JSON is never written to stdout. -Resolved run instance names are auto-numbered for repeats, so the default report pipeline is: +## Stderr behavior -1. `glossary_1` -2. `homophones` -3. `glossary_2` -4. `spoken_word` -5. `grammar` +- Success path should be quiet or minimal human-readable logs. +- Failure path writes concise human-readable errors. +- When a diagnostics run directory exists, failure stderr includes its path. +- Prompt/response diagnostic payloads are not streamed to stderr. -The default module sequence is fully implemented today: +## Output file behavior -- `glossary` proposes glossary-supported acoustic corrections -- `homophones` proposes conservative homophone and mistranscription corrections -- `spoken_word` proposes conservative dysfluency cleanup -- `grammar` proposes punctuation, capitalization, and spacing cleanup only +- `--output` writes transcript JSON in the selected output schema to the provided path. +- Output write failures return nonzero and surface actionable errors. +- The command does not silently ignore output write errors. -To run a custom module sequence, pass `--modules`: +## Report JSON behavior -```sh -uv run audita process transcript.json --glossary glossary.yaml --modules grammar --output corrected.json -``` +- `--report-json` writes a machine-readable process report to the requested path. +- Run-directory `report.json` is written independently under diagnostics. +- Best-effort failure reports are emitted when possible without masking the primary failure. +- Report write failures return nonzero with clear stderr messaging. +- Report diagnostics metadata references run-directory artifacts including utilization diagnostics and correction ledger paths when available. -To also write a structured JSON report: +## Diagnostics directory behavior -```sh -uv run audita process transcript.json --glossary glossary.yaml --output corrected.json --report-json report.json -``` +- Each run creates (when possible) a per-run diagnostics directory. +- Typical artifacts include transcript, normalization, chunking, invocation, effective config, LLM diagnostics, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, and `error.log` on failure. +- Failed runs retain diagnostics. +- Under `auto` retention, successful runs with skipped/rejected corrections are retained; clean successful runs may be removed. -From a checked-out repository, you can also use the root launcher: +## Exit codes -```sh -./audita process transcript.json --glossary glossary.yaml --output corrected.json -``` +- `0`: success. +- Nonzero: failure (input/schema/config/module/LLM/runtime/output/report/diagnostics errors). -For a system-wide command, install the source tree under `/usr/local/src/audita`, sync dependencies there, and symlink the root launcher into your `PATH`: +Treat any nonzero as a failed subprocess invocation. -```sh -cd /usr/local/src/audita -uv sync --extra dev -ln -s /usr/local/src/audita/audita /usr/local/bin/audita -audita process transcript.json --glossary glossary.yaml --output corrected.json -``` +## Timeout and cancellation -Without `--output`, Audita writes the corrected transcript JSON to stdout and progress logs to stderr. -`--report-json` writes a separate machine-readable run report and never mixes report data into stdout. +- Runtime operations propagate context cancellation and request timeouts through LLM/scheduler paths. +- On cancellation or timeout, the process exits nonzero and should not hang. +- If diagnostics were initialized before failure, failure artifacts remain available for debugging. -Useful configuration can be supplied by CLI flag or environment variable. CLI flags take precedence over environment variables. Normal runs now require LLM API credentials, because the `glossary`, `homophones`, `spoken_word`, and `grammar` modules make real LLM calls. `AUDITA_LLM_API_KEY` and `--llm-api-key` are the preferred provider-neutral credential surfaces, while `OPENROUTER_API_KEY` remains supported as a backward-compatible fallback. +## Secret redaction expectations -| Environment variable | CLI flag | Default | Purpose | -| --- | --- | --- | --- | -| `AUDITA_MODULES` | `--modules` | `glossary,homophones,glossary,spoken_word,grammar` | Comma-separated logical module keys to run; CLI overrides the environment value | -| `AUDITA_LLM_API_KEY` | `--llm-api-key` | unset | Preferred provider-neutral LLM API credential; CLI overrides both environment-key variants | -| `AUDITA_VALIDATION_LLM_API_KEY` | `--validation-llm-api-key` | unset | Validation-phase LLM API credential; defaults to the primary LLM API key | -| `AUDITA_MODEL` | `--model` | `openrouter/google/gemma-4-31b-it` | LLM model name sent to the configured OpenAI-compatible endpoint | -| `AUDITA_VALIDATION_MODEL` | `--validation-model` | unset | Validation-phase LLM model; defaults to `AUDITA_MODEL` | -| `AUDITA_BASE_URL` | `--base-url` | `https://openrouter.ai/api/v1` | OpenAI-compatible API base URL | -| `AUDITA_VALIDATION_BASE_URL` | `--validation-base-url` | unset | Validation-phase OpenAI-compatible API base URL; defaults to `AUDITA_BASE_URL` | -| `AUDITA_LLM_TIMEOUT_SECONDS` | `--llm-timeout-seconds` | `600` | Per-request timeout in seconds for LLM calls to the configured OpenAI-compatible endpoint | -| `AUDITA_VALIDATION_LLM_TIMEOUT_SECONDS` | `--validation-llm-timeout-seconds` | unset | Validation-phase per-request timeout in seconds; defaults to `AUDITA_LLM_TIMEOUT_SECONDS` | -| `AUDITA_VALIDATION_MAX_PROMPT_TOKENS` | `--validation-max-prompt-tokens` | `2048` | Maximum estimated tokens per validation-phase LLM prompt batch | -| `AUDITA_TARGET_SECTIONS` | `--target-sections` | unset | Exact number of contiguous proposal-stage transcript sections; errors if min/max token bounds cannot be satisfied | -| `AUDITA_MAX_RETRIES` | `--max-retries` | `3` | Maximum Instructor retries for structured responses | -| `AUDITA_VALIDATION_MAX_RETRIES` | `--validation-max-retries` | unset | Validation-phase structured-output retries; defaults to `AUDITA_MAX_RETRIES` | -| `AUDITA_VALIDATION_LLM_CONCURRENCY` | `--validation-llm-concurrency` | unset | Validation-phase LLM concurrency; defaults to `AUDITA_LLM_CONCURRENCY` | -| `AUDITA_MAX_SECTION_TOKENS` | `--max-section-tokens` | `8192` | Maximum estimated tokens per proposal-stage transcript section | -| `AUDITA_MIN_SECTION_TOKENS` | `--min-section-tokens` | `2048` | Minimum estimated tokens per proposal-stage transcript section when balancing for concurrency | -| `AUDITA_GLOSSARY_CONFIDENCE_THRESHOLD` | `--glossary-confidence-threshold` | `0.8` | Minimum confidence required for glossary proposals to survive validation | -| `AUDITA_GRAMMAR_CONFIDENCE_THRESHOLD` | `--grammar-confidence-threshold` | `0.8` | Minimum confidence required for grammar proposals to survive validation | -| `AUDITA_HOMOPHONES_CONFIDENCE_THRESHOLD` | `--homophones-confidence-threshold` | `0.8` | Minimum confidence required for homophone proposals to survive validation | -| `AUDITA_SPOKEN_WORD_CONFIDENCE_THRESHOLD` | `--spoken-word-confidence-threshold` | `0.8` | Minimum confidence required for spoken-word proposals to survive validation | -| `AUDITA_NORMALIZE_MAX_SEGMENT_GAP` | `--normalize-max-segment-gap` | `4.0` | Same-speaker gaps eligible for deterministic merging | -| `AUDITA_NORMALIZE_ELLIPSIS_GAP` | `--normalize-ellipsis-gap` | `3.5` | Same-speaker gaps above this value are joined with ` ... ` | -| `AUDITA_NORMALIZE_MAX_SEGMENT_DURATION` | `--normalize-max-segment-duration` | `60.0` | Maximum merged segment duration | -| `AUDITA_NORMALIZE_MAX_SEGMENT_TOKENS` | `--normalize-max-segment-tokens` | `2048` | Maximum merged segment prompt payload size | -| `AUDITA_WORK_DIR` | `--work-dir` | `/tmp/audita` | Per-run scratch diagnostics directory | -| `AUDITA_WORK_DIR_RETENTION` | `--work-dir-retention` | `auto` | Whether to retain the per-run work directory: `auto`, `always`, or `never` | +API keys and configured secret values are redacted from: +- reports (`--report-json` and run-dir `report.json`); +- diagnostics artifacts (including effective config and LLM interaction artifacts); +- surfaced adapter/runtime errors; +- test fixtures and regression outputs. -Set `AUDITA_MODULES=grammar` to run only the grammar module by default, or override it per command with `--modules`. +Parent-process logs should still avoid printing raw environment variables. -Validation-phase LLM settings inherit from the primary `AUDITA_*` LLM settings by default. Set any of the `AUDITA_VALIDATION_*` values only when you want LLM-backed validators to use a different model, endpoint, credential, timeout, retry budget, or concurrency level. +## Parent-process pipe guidance -OpenRouter remains the default out of the box: +To avoid deadlocks in orchestrators: +- always read both stdout and stderr concurrently when invoking as a subprocess; +- prefer file outputs (`--output`, `--report-json`) for machine workflows; +- treat stderr as human-readable diagnostics, not structured data; +- parse structured results from output/report files. -```sh -export AUDITA_LLM_API_KEY=your-openrouter-key -audita process transcript.json --glossary glossary.yaml --output corrected.json -``` - -You can point Audita at any OpenAI-compatible endpoint by changing `AUDITA_BASE_URL` and, if needed, `AUDITA_MODEL`. For example, a local vLLM server: - -```sh -export AUDITA_LLM_API_KEY=local-dev-key -export AUDITA_BASE_URL=http://localhost:8000/v1 -export AUDITA_MODEL=meta-llama/Llama-3.1-8B-Instruct -audita process transcript.json --glossary glossary.yaml --output corrected.json -``` - -Or the actual OpenAI API: - -```sh -export AUDITA_LLM_API_KEY=your-openai-key -export AUDITA_BASE_URL=https://api.openai.com/v1 -export AUDITA_MODEL=gpt-4.1-mini -audita process transcript.json --glossary glossary.yaml --output corrected.json -``` - -`AUDITA_WORK_DIR` stores per-run diagnostics while processing. Under the default `AUDITA_WORK_DIR_RETENTION=auto`, clean successful runs are removed, while failed runs and successful runs with final skipped corrections are preserved. Use `always` to keep every run directory and `never` to remove successful run directories even when skips remain. -Failed runs always preserve the run directory and include an authoritative `report.json` alongside normalization and prompt/response diagnostics. - -## Prototype Archive - -The archived prototype remains importable as `audita_prototype` and is still covered by its original regression suite. This is intentional: the new `audita` package is a framework-oriented rewrite, not a thin wrapper around the old code. +For Go callers, prefer `exec.CommandContext` with explicit timeout/cancellation and buffered/streamed readers for both pipes.