diff --git a/README.md b/README.md index 99c3f1c..3e584dc 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,8 @@ audita config print-effective --config ./audita.yml - CLI reference: [`docs/cli.md`](docs/cli.md) - Configuration reference: [`docs/config.md`](docs/config.md) - Subprocess integration: [`docs/integrations/subprocess.md`](docs/integrations/subprocess.md) +- OpenAI-compatible LLM integration: [`docs/integrations/openai-compatible-llm.md`](docs/integrations/openai-compatible-llm.md) +- Transcript and glossary file integration: [`docs/integrations/transcript-glossary-files.md`](docs/integrations/transcript-glossary-files.md) - Development workflow: [`docs/policy/development.md`](docs/policy/development.md) - Architecture policy: [`docs/policy/architecture.md`](docs/policy/architecture.md) - Documentation policy: [`docs/policy/documentation.md`](docs/policy/documentation.md) diff --git a/docs/cli.md b/docs/cli.md index 8fa49c1..264ab2b 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -106,6 +106,10 @@ Exit behavior: - `1`: runtime failure during processing/reporting/output paths. - `2`: CLI usage or configuration input error. +Integration references: +- subprocess contract: [`docs/integrations/subprocess.md`](integrations/subprocess.md) +- transcript/glossary file contract: [`docs/integrations/transcript-glossary-files.md`](integrations/transcript-glossary-files.md) + ### `process` Examples Write corrected transcript to a file: diff --git a/docs/config.md b/docs/config.md index 4202935..ed76e97 100644 --- a/docs/config.md +++ b/docs/config.md @@ -13,6 +13,8 @@ It documents: - validation and secrets behavior. For CLI command syntax, see [`docs/cli.md`](cli.md). +For OpenAI-compatible endpoint behavior, see [`docs/integrations/openai-compatible-llm.md`](integrations/openai-compatible-llm.md). +For transcript/glossary input file contracts, see [`docs/integrations/transcript-glossary-files.md`](integrations/transcript-glossary-files.md). ## Loading Model diff --git a/docs/integration/subprocess-operations.md b/docs/integration/subprocess-operations.md deleted file mode 100644 index d7bd17f..0000000 --- a/docs/integration/subprocess-operations.md +++ /dev/null @@ -1,5 +0,0 @@ -# Moved: Subprocess Integration - -The canonical subprocess integration guide now lives at [`docs/integrations/subprocess.md`](../integrations/subprocess.md). - -This file is retained temporarily as a migration shim for older links. diff --git a/docs/integrations/openai-compatible-llm.md b/docs/integrations/openai-compatible-llm.md new file mode 100644 index 0000000..856fb02 --- /dev/null +++ b/docs/integrations/openai-compatible-llm.md @@ -0,0 +1,119 @@ +# OpenAI-Compatible LLM Integration + +## Scope + +This document defines the external LLM endpoint contract Audita currently uses. + +It covers: +- endpoint and auth expectations; +- structured request and response shape; +- retry and timeout behavior; +- diagnostics and secret redaction. + +For user-facing CLI flags and config keys, see [`docs/cli.md`](../cli.md) and [`docs/config.md`](../config.md). + +## Endpoint Contract + +Audita sends HTTPS `POST` requests to: + +- `/chat/completions` + +`base_url` comes from primary or validation LLM config and is required. + +## Authentication Contract + +When an API key is configured, Audita sends: + +- `Authorization: Bearer ` + +When no API key is configured, the `Authorization` header is omitted. + +## Request Shape + +Audita sends a chat-completions payload with: +- `model`; +- `messages` (role/content pairs); +- `response_format` using JSON Schema strict mode. + +Representative shape: + +```json +{ + "model": "example-model", + "messages": [ + {"role": "system", "content": "..."}, + {"role": "user", "content": "..."} + ], + "response_format": { + "type": "json_schema", + "json_schema": { + "name": "correction_set", + "strict": true, + "schema": {"type": "object"} + } + } +} +``` + +Behavioral requirements enforced by Audita: +- `model` must resolve to a non-empty value; +- each message must have non-empty `role` and `content`; +- `response_format.type` is always `json_schema`; +- `response_format.json_schema.name` and `schema` must be present; +- request schema JSON must be valid JSON. + +## Response Handling Contract + +Audita expects a successful JSON response with at least one choice and assistant content that can be interpreted as JSON. + +Supported assistant content forms: +- string containing JSON; +- raw JSON value. + +Audita then decodes the JSON against the expected structured output type. + +Current structured schema identities used by Audita runtime: +- `correction_set` +- `validator_decision_set` + +## Retries and Timeouts + +Retry behavior: +- default max retries is `3` when unset; +- retries apply to retryable transport/decode/server-side errors; +- HTTP `429` and `5xx` responses are retryable; +- retry stops immediately when context is canceled or deadline expires. + +Timeout behavior: +- request timeout is derived from configured LLM timeout settings; +- timeout/cancellation propagate through HTTP requests and return nonzero process failures. + +## Error Behavior + +Non-2xx responses fail the request. + +Error message extraction behavior: +- if provider JSON includes `error.message`, Audita surfaces that message; +- else if provider JSON includes top-level `message`, Audita surfaces that; +- otherwise Audita surfaces status code plus response body text. + +Malformed or incompatible structured responses fail safely and are surfaced as runtime errors or validator/proposal warnings depending on call site. + +## Secret Redaction + +Configured LLM secrets are redacted from: +- surfaced adapter/runtime errors; +- LLM diagnostics request/response/error artifacts; +- effective config/report artifacts that include LLM configuration material. + +Redaction marker: +- `[REDACTED]` + +## Compatibility Boundaries + +This integration documentation applies only to the implemented OpenAI-compatible chat completions flow. + +Not part of current behavior: +- provider SDK integration; +- non-OpenAI-compatible API contracts; +- server-side model routing features beyond explicitly configured model/base URL. diff --git a/docs/integrations/subprocess.md b/docs/integrations/subprocess.md index d94aeef..33f0c41 100644 --- a/docs/integrations/subprocess.md +++ b/docs/integrations/subprocess.md @@ -1,96 +1,99 @@ -# Audita Subprocess Operations +# Subprocess Integration -This document describes how parent processes should invoke `audita process` safely in production orchestration. +## Scope -## Recommended command form +This document describes how a parent process should invoke Audita as a subprocess. -Use explicit file outputs for orchestrated runs: +It covers: +- invocation shape; +- stdout/stderr behavior; +- output/report file behavior; +- diagnostics and exit behavior. + +For full CLI and config references, see [`docs/cli.md`](../cli.md) and [`docs/config.md`](../config.md). + +## Recommended Invocation + +Use explicit output and report paths for machine workflows: ```sh audita process \ - --transcript-description "Brief context that may help resolve ambiguous terms." \ --glossary \ --output \ --report-json ``` -Additional flags that may be situationally appropriate: -- `--config ` to select an explicit versioned config file. -- `--output-schema ` to select transcript output shape. -- `--work-dir ` to control diagnostics location. -- `--work-dir-retention ` to control retained run directories. -- `--total-llm-concurrency`, `--proposal-llm-concurrency`, and `--validation-llm-concurrency` when orchestration needs to set explicit LLM throughput controls. -- `--modules ...` only when intentionally overriding the default sequence. +Optional commonly used flags: +- `--config ` +- `--output-schema ` +- `--work-dir ` +- `--work-dir-retention ` +- `--transcript-description ` -For config-driven orchestration, validate config files in CI/preflight: +## Stdout Contract -```sh -audita config validate --config -``` +On success: +- with `--output`: stdout is expected to be empty; +- without `--output`: stdout contains transcript JSON only. -## Stdout behavior +`--report-json` output is never written to stdout. -- With `--output`: stdout is expected to be empty on success. -- Without `--output`: stdout contains transcript JSON only on success. -- Report JSON is never written to stdout. +## Stderr Contract -## Stderr behavior +Stderr is human-readable status/error output. -- Success path should be quiet or minimal human-readable logs. -- Failure path writes concise human-readable errors. -- When a diagnostics run directory exists, failure stderr includes its path. -- Prompt/response diagnostic payloads are not streamed to stderr. +On failures: +- stderr includes a concise top-level error; +- when diagnostics are initialized, stderr includes diagnostics directory path. -## Output file behavior +Do not treat stderr as a machine-stable JSON channel. -- `--output` writes transcript JSON in the selected output schema to the provided path. -- Output write failures return nonzero and surface actionable errors. -- The command does not silently ignore output write errors. +## Output and Report File Contract -## Report JSON behavior +Transcript output: +- `--output` writes corrected transcript JSON to the provided path; +- output write failures return nonzero. -- `--report-json` writes a machine-readable process report to the requested path. -- Run-directory `report.json` is written independently under diagnostics. -- Best-effort failure reports are emitted when possible without masking the primary failure. -- Report write failures return nonzero with clear stderr messaging. -- Report diagnostics metadata references run-directory artifacts including utilization diagnostics and correction ledger paths when available. +Report output: +- `--report-json` writes machine-readable process report JSON to the provided path; +- run diagnostics also attempt to write their own `report.json`; +- report write failures return nonzero; +- on failure paths, report writing is best-effort and does not mask the primary run error. -## Diagnostics directory behavior +## Diagnostics Contract -- Each run creates (when possible) a per-run diagnostics directory. -- Typical artifacts include transcript, normalization, chunking, invocation, effective config, LLM diagnostics, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, and `error.log` on failure. -- Failed runs retain diagnostics. -- Under `auto` retention, successful runs with skipped/rejected corrections are retained; clean successful runs may be removed. +When run-directory initialization succeeds, per-run diagnostics artifacts are written under the configured work directory. -## Exit codes +Typical artifacts include: +- `source-transcript.json` +- `source-transcript-parsed.json` +- `normalized-transcript.json` +- `normalization-summary.json` +- `chunking-summary.json` +- `invocation.json` +- `effective-config.json` +- `utilization-diagnostics.json` +- `correction-ledger.json` +- `report.json` +- `error.log` (failure) -- `0`: success. -- Nonzero: failure (input/schema/config/module/LLM/runtime/output/report/diagnostics errors). +Retention behavior is controlled by `--work-dir-retention` / config. -Treat any nonzero as a failed subprocess invocation. +## Exit Behavior -## Timeout and cancellation +Exit codes: +- `0`: success; +- `1`: runtime processing/output/report failure; +- `2`: CLI usage or configuration input error. -- Runtime operations propagate context cancellation and request timeouts through LLM/scheduler paths. -- On cancellation or timeout, the process exits nonzero and should not hang. -- If diagnostics were initialized before failure, failure artifacts remain available for debugging. +Treat any nonzero as subprocess failure. -## Secret redaction expectations +## Parent-Process Guidance -API keys and configured secret values are redacted from: -- reports (`--report-json` and run-dir `report.json`); -- diagnostics artifacts (including effective config and LLM interaction artifacts); -- surfaced adapter/runtime errors; -- test fixtures and regression outputs. +For reliable orchestration: +- read stdout and stderr concurrently to avoid pipe blocking; +- prefer `--output` and `--report-json` for machine parsing; +- use timeout/cancellation in the parent process; +- inspect diagnostics path and `report.json`/`error.log` on failure. -Parent-process logs should still avoid printing raw environment variables. - -## Parent-process pipe guidance - -To avoid deadlocks in orchestrators: -- always read both stdout and stderr concurrently when invoking as a subprocess; -- prefer file outputs (`--output`, `--report-json`) for machine workflows; -- treat stderr as human-readable diagnostics, not structured data; -- parse structured results from output/report files. - -For Go callers, prefer `exec.CommandContext` with explicit timeout/cancellation and buffered/streamed readers for both pipes. +For input file contracts, see [`docs/integrations/transcript-glossary-files.md`](transcript-glossary-files.md). diff --git a/docs/integrations/transcript-glossary-files.md b/docs/integrations/transcript-glossary-files.md new file mode 100644 index 0000000..9fb9f9d --- /dev/null +++ b/docs/integrations/transcript-glossary-files.md @@ -0,0 +1,98 @@ +# Transcript and Glossary File Integration + +## Scope + +This document defines the input file contracts for: +- transcript JSON; +- glossary YAML. + +These files are loaded and validated before processing begins. + +## Transcript JSON Contract + +Audita accepts either top-level shape: +- JSON array of segments; or +- JSON object with a `segments` array. + +Segment fields: +- `id` (optional integer in source form); +- `speaker` (required non-empty string); +- `start` (required finite non-negative number); +- `end` (required finite non-negative number, `>= start`); +- `text` (required non-empty string); +- `categories` (optional string array; entries must be non-empty). + +Additional rules: +- transcript must contain at least one segment; +- duplicate segment IDs are rejected when IDs are present. + +Example (`examples/tiny-transcript.json`): + +```json +[ + { + "id": 1, + "speaker": "A", + "start": 0.0, + "end": 1.2, + "text": "hello world" + } +] +``` + +## Glossary YAML Contract + +Audita expects top-level `glossary` list entries. + +Entry fields: +- `name` (required non-empty string); +- `category` (required non-empty string); +- `summary` (required non-empty string); +- `aliases` (optional list of strings; entries must be non-empty); +- `plural` (optional string). + +Additional rules: +- glossary must contain at least one entry. + +Example (`examples/tiny-glossary.yaml`): + +```yaml +glossary: + - name: Audita + aliases: + - audita + category: product + summary: The Audita transcript correction CLI. +``` + +## Validation Failure Behavior + +Representative transcript validation failures: +- invalid JSON; +- unsupported top-level shape; +- empty `speaker` or `text`; +- invalid times (`NaN`, `Inf`, negative, or `end < start`); +- duplicate IDs; +- empty transcript array. + +Representative glossary validation failures: +- invalid YAML; +- empty or missing glossary entries; +- missing required entry fields; +- empty alias values. + +These failures surface as schema errors and the process exits nonzero. + +## CLI Usage + +Minimal invocation: + +```sh +audita process ./transcript.json --glossary ./glossary.yaml --output ./corrected.json +``` + +See also: +- [`docs/cli.md`](../cli.md) +- [`docs/config.md`](../config.md) +- [`examples/tiny-transcript.json`](../../examples/tiny-transcript.json) +- [`examples/tiny-glossary.yaml`](../../examples/tiny-glossary.yaml)