eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Refresh documentation and policies and simplify releases

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-23 15:03:20 +00:00
parent 588ca36dfc
commit e8b8fbd509
36 changed files with 2143 additions and 2261 deletions
Download Patch File Download Diff File Expand all files Collapse all files

141
docs/integrations/openai-compatible-llm.md
View File

@@ -1,119 +1,76 @@
# OpenAI-Compatible LLM Integration
## Scope
This document owns the external protocol used by Audita's structured LLM adapter.
Settings, inheritance, defaults, and credential supply belong to
[configuration](../config.md#llm-settings); implementation and scheduling belong to
[LLM runtime internals](../internal/llm-runtime.md).
This document defines the external LLM endpoint contract Audita currently uses.
## Endpoint and Authentication
It covers:
- endpoint and auth expectations;
- structured request and response shape;
- retry and timeout behavior;
- diagnostics and secret redaction.
Audita sends an HTTP `POST` to the configured base URL with trailing slashes
removed and `/chat/completions` appended. Supply an HTTP or HTTPS base URL that
makes this the intended endpoint; the adapter does not require HTTPS in code.
Requests have `Content-Type: application/json`. A non-empty configured API key
adds `Authorization: Bearer <api_key>`; otherwise that header is omitted.
For user-facing CLI flags and config keys, see [`docs/cli.md`](../cli.md) and [`docs/config.md`](../config.md).
## Endpoint Contract
Audita sends HTTPS `POST` requests to:
- `<base_url>/chat/completions`
`base_url` comes from primary or validation LLM config and is required.
## Authentication Contract
When an API key is configured, Audita sends:
- `Authorization: Bearer <api_key>`
When no API key is configured, the `Authorization` header is omitted.
The endpoint must support the configured model and structured JSON-schema
responses. Local configuration validation does not probe that support.
## Request Shape
Audita sends a chat-completions payload with:
- `model`;
- `messages` (role/content pairs);
- `response_format` using JSON Schema strict mode.
Representative shape:
The payload contains `model`, an array of `messages` with string `role`/`content`,
and `response_format`. The structured-output portion has this illustrative shape:
```json
{
"model": "example-model",
"messages": [
{"role": "system", "content": "..."},
{"role": "user", "content": "..."}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "correction_set",
"strict": true,
"schema": {"type": "object"}
}
"type": "json_schema",
"json_schema": {
"name": "correction_set",
"strict": true,
"schema": {"type": "object"}
}
}
```
Behavioral requirements enforced by Audita:
- `model` must resolve to a non-empty value;
- each message must have non-empty `role` and `content`;
- `response_format.type` is always `json_schema`;
- `response_format.json_schema.name` and `schema` must be present;
- request schema JSON must be valid JSON.
The actual schema is supplied for the call; the fragment above is not a complete
request schema. Runtime calls use `correction_set` and `validator_decision_set`
identities. The adapter requires a non-empty resolved model, non-empty message
roles/content, a schema name, and valid schema JSON. It sends strict mode to the
provider; local typed decoding and proposal/decision checks still apply afterward.
## Response Handling Contract
Audita expects a successful JSON response with at least one choice and assistant content that can be interpreted as JSON.
A successful response must be a JSON envelope with at least one `choices` item.
Audita reads the first choice's `message.content`, accepting a non-empty string
containing JSON or a non-null raw JSON value that can decode into the expected
output type. Empty content, missing choices, and incompatible payloads fail.
The adapter also reads optional `model` and token-usage metadata.
Supported assistant content forms:
- string containing JSON;
- raw JSON value.
Audita then decodes the JSON against the expected structured output type.
Current structured schema identities used by Audita runtime:
- `correction_set`
- `validator_decision_set`
There is no streaming protocol or provider SDK dependency in this integration.
Audita uses this chat-completions contract, not other provider API shapes.
## Retries and Timeouts
Retry behavior:
- default max retries is `3` when unset;
- retries apply to retryable transport/decode/server-side errors;
- HTTP `429` and `5xx` responses are retryable;
- retry stops immediately when context is canceled or deadline expires.
The effective adapter retry count is additional attempts after the initial request;
see [configuration](../config.md#llm-settings) for the current zero-value behavior.
Transport/read errors, malformed response envelopes/content, typed decode failures,
HTTP `429`, and server-error responses are retryable. Other non-success HTTP
responses and a response missing choices are not retried by the adapter.
Cancellation or an expired context stops retrying.
Timeout behavior:
- request timeout is derived from configured LLM timeout settings;
- timeout/cancellation propagate through HTTP requests and return nonzero process failures.
The configured HTTP timeout applies per request; it is not a whole-process time
budget. A parent process can impose its own overall timeout. See
[configuration](../config.md#llm-settings) for timeout and retry values and
[subprocess guidance](subprocess.md) for parent responsibilities.
## Error Behavior
Non-2xx responses fail the request.
For non-success responses, error detail comes from provider `error.message`, then
a top-level `message`, otherwise the HTTP status and response body. Configured
keys are redacted from surfaced errors and LLM diagnostic payloads.
Error message extraction behavior:
- if provider JSON includes `error.message`, Audita surfaces that message;
- else if provider JSON includes top-level `message`, Audita surfaces that;
- otherwise Audita surfaces status code plus response body text.
Malformed or incompatible structured responses fail safely and are surfaced as runtime errors or validator/proposal warnings depending on call site.
## Secret Redaction
Configured LLM secrets are redacted from:
- surfaced adapter/runtime errors;
- LLM diagnostics request/response/error artifacts;
- effective config/report artifacts that include LLM configuration material.
Redaction marker:
- `[REDACTED]`
## Compatibility Boundaries
This integration documentation applies only to the implemented OpenAI-compatible chat completions flow.
Not part of current behavior:
- provider SDK integration;
- non-OpenAI-compatible API contracts;
- server-side model routing features beyond explicitly configured model/base URL.
Malformed structured responses can become section/batch warnings and rejected
proposals rather than fatal run errors, depending on the call site. Other adapter
failures can terminate processing with partial results. Consult the
[report contract](output-files.md#module-outcomes) for those outcomes and
[operations](../operations.md#artifact-handling) for handling diagnostic contents.

186
docs/integrations/output-files.md Normal file
View File

@@ -0,0 +1,186 @@
# Output Files and Report Formats
This document owns externally consumed corrected-transcript and report formats,
and describes the correction ledger and utilization diagnostics. It documents
current serialized shapes without making every debugging artifact a versioned
compatibility promise. Physical filenames, placement, retention, and permissions
belong to [operations](../operations.md); invocation and output channels belong to
the [CLI](../cli.md#process-output-and-exit-behavior).
## Corrected Transcript
The [output setting](../config.md#pipeline-and-output) selects the envelope:
| Encoder | JSON shape |
| --- | --- |
| `bare-segments` | Top-level array of segment objects |
| `audita-v1` | Object with `schema: "audita-v1"`, `version: "v1"`, and `segments` containing the segment array |
Both emit these segment fields:
| Field | Type | Meaning |
| --- | --- | --- |
| `id` | Integer | Identifier in the normalized transcript |
| `speaker` | String | Speaker label |
| `start`, `end` | Numbers | Start/end times in seconds |
| `text` | String | Text after normalization and accepted corrections |
| `categories` | Array of strings, optional | Segment categories; omitted when empty |
Input IDs and segment boundaries need not survive normalization unchanged;
normalization can merge segments and reassign IDs. The output describes the
resulting transcript, not a lossless copy of the source JSON. Input acceptance
rules belong to the [input-file contract](transcript-glossary-files.md).
## Process Report
The report is a JSON object. `report_metadata` identifies its schema separately
from the transcript output envelope:
| Metadata field | Type / current value |
| --- | --- |
| `report_schema_name` | String: `audita-process-report` |
| `report_schema_version` | String: `v1` |
| `output_schema` | String identifying the selected transcript encoder |
| `config_version` | Optional integer, present when file configuration was loaded |
The remaining top-level fields are:
| Field | Type and meaning |
| --- | --- |
| `phase` | String; currently `default_pipeline` |
| `status` | String: `success` or `failed` |
| `operation` | String; currently `process` |
| `transcript_path`, `glossary_path` | Input path strings |
| `output_path` | Optional transcript destination string |
| `modules` | Array of configured module keys, including repeats |
| `started_at`, `completed_at` | RFC 3339 timestamps, possibly fractional; completion is an optional field populated by completed process runs |
| `error_phase`, `error_message` | Optional failure classification and human-readable detail |
| `input_segment_count`, `normalized_segment_count` | Optional integer normalization counts |
| `normalization_merges`, `normalization_id_reassignments` | Optional integer transformation counts |
| `normalization_skipped` | Object of optional integer counters: `different_speakers`, `gap_too_large`, `duration_exceeded`, `token_limit_exceeded`; can be empty |
| `chunking` | Optional section summary described below |
| `diagnostics` | Optional artifact-path metadata described below |
| `modules_summary` | Optional aggregate module outcome |
| `module_results` | Optional array of module outcomes |
Failures can preserve partial module results. Missing summaries indicate unavailable
information, not necessarily zero work. Early configuration failures occur before
report construction. Report success describes processing results; a later file-write
or cleanup error can still cause a nonzero CLI exit. Use the exit result and report
together as explained in [subprocess guidance](subprocess.md).
Error phases identify where processing failed, for example `transcript_read`,
`glossary_read`, `transcript_schema`, `glossary_schema`, `chunking`, `runner_setup`,
`runner_execution`, `output_schema`, `serialization`, `output_write`, and
`stdout_write`. This is not an exhaustive enumeration or a promise that every
late failure is represented in a successfully written report. Error messages are
human-readable, not structured fields to parse for control flow.
### Chunking and Diagnostics Metadata
`chunking` contains integer fields `chunk_count`, `min_estimated_chunk_tokens`,
`max_estimated_chunk_tokens`, `total_estimated_transcript_tokens`,
`max_section_tokens`, and `min_section_tokens`, plus optional integer
`target_sections`. Token counts are estimates.
`diagnostics` contains optional path strings:
- `directory_path`
- `source_transcript_path`, `parsed_source_transcript_path`, `normalized_transcript_path`
- `normalization_summary_path`, `chunking_summary_path`
- `utilization_summary_path`, `correction_ledger_path`
- `invocation_metadata_path`, `redacted_effective_config_path`, `error_log_path`
The [operations guide](../operations.md#diagnostics-directory) defines the physical
artifacts and path-lifetime caveats.
### Module Outcomes
`modules_summary` contains integer `module_count`, `total_applied_changes`, and
`total_skipped_changes`, plus optional string `failed_module_instance`.
`total_skipped_changes` counts both apply-time skipped changes and validator
rejections. Their detailed records are stored in separate arrays.
Each `module_results` object contains:
| Field | Type and meaning |
| --- | --- |
| `module_key`, `module_instance` | Strings; the instance distinguishes repeated occurrences of a configured module |
| `replacement_policy` | Optional string describing the module's replacement policy |
| `status` | String describing module outcome (`success` or `failed`) |
| `proposal_count` | Integer |
| `started_at`, `completed_at` | Optional timestamps |
| `error_message` | Optional failure detail |
| `warnings` | Optional array of warning records |
| `validator_decisions` | Optional array of decision records |
| `validator_rejected` | Optional array of rejected-correction records |
| `applied_changes`, `skipped_changes` | Optional arrays of apply outcomes |
A validator decision has `validator_name` (string), `proposal_index` (integer),
`approved` (boolean), `reason_code` and `message` (strings), and optional
`diagnostic_artifact_path` (string).
Rejected corrections include `validator_name`, `proposal_index`, `module_key`,
`module_instance`, `target_segment_id`, `original_text`, `corrected_text`,
`reason_code`, and `message`. Applied changes include the same correction identity
and text fields without validator/reason/message fields, plus integer
`replacement_count`. Skipped changes include correction identity/text fields,
`skip_reason`, and `message`.
A warning has string `scope`, `reason_code`, and `message`; optional integer
`section_index` and `batch_index`; and optional string `validator_name` and
`diagnostic_artifact_path`. Warnings, skipped changes, and rejected corrections
can occur without a failed run. Proposal indexes associate outcomes with their
proposal within a module instance; they are not transcript segment IDs.
## Correction Ledger
The ledger is an array of correction entries; it can be `null` when no module
results are available and empty when modules produced no entries. Unlike the
process report, it has no separate schema-version envelope.
Each entry has required `module_key`, `module_instance`, integer `proposal_index`,
and `disposition`. The current dispositions are:
| Disposition | Meaning |
| --- | --- |
| `applied` | The proposed change was applied |
| `skipped` | The proposal reached application but was skipped |
| `rejected` | A validator rejected the proposal |
| `failed` | Module-level failure record, not necessarily an individual proposal |
Optional fields are `run_id`, integer `segment_id`, `original_text`,
`proposed_corrected_text`, `applied_corrected_text`, `replacement_policy`,
`disposition_reason_code`, and `disposition_message`. Optional validator arrays
are `deterministic_validator_decisions` and `llm_validator_decisions`; each record
contains string `validator_key`, boolean `approved`, string `reason_code`, and
optional string `message`.
Module failure entries can have the zero-value proposal index and no correction
text or segment. Do not interpret them as evidence that proposal zero failed.
Empty optional values are omitted. The ledger records disposition, not a replay
or resume protocol.
## Utilization Diagnostics
This unversioned JSON object describes observations from the runtime, not service
limits or performance guarantees:
- `effective_concurrency`: integer `total_llm`, `proposal_llm`, `validation_llm`.
- `run_timing`: millisecond integers `run_wall_time_ms`, `scheduler_queue_wait_ms`,
`llm_execution_time_ms`, `deterministic_validation_time_ms`; integer
`max_in_flight_llm_calls` and `average_in_flight_llm_calls`.
- `llm_calls`: integer `total_proposal_calls` and `total_validation_calls`.
- Optional `modules`: records with `module_key`, `module_instance`, integer
`proposal_llm_calls` and `validation_llm_calls`, and millisecond integer
`scheduler_queue_wait_ms`, `llm_execution_time_ms`,
`deterministic_validation_time_ms`, and `module_wall_time_ms`.
- Optional `validators`: records with `module_key`, `module_instance`,
`validator_key`, boolean `llm_backed`, integer `calls`, and millisecond integer
`elapsed_ms`.
Aggregated execution/wait durations can overlap in concurrent work; they need
not sum to wall time. The observed average concurrency is serialized as an integer.
Other source snapshots, prompt payloads, and diagnostic files are debugging
material; their placement is documented in operations without defining a separate
public schema for every file.

119
docs/integrations/subprocess.md
View File

@@ -1,99 +1,40 @@
# Subprocess Integration
## Scope
This guide explains how a parent process can invoke Audita and consume its results.
The [CLI reference](../cli.md) owns command syntax, streams, and exit semantics;
[configuration](../config.md) owns discovery and precedence.
This document describes how a parent process should invoke Audita as a subprocess.
## Invocation
It covers:
- invocation shape;
- stdout/stderr behavior;
- output/report file behavior;
- diagnostics and exit behavior.
Use an argument array with the transcript path and required glossary argument,
following the [process examples](../cli.md#process-examples). For machine workflows,
select explicit transcript and report destinations rather than depending on the
working directory or run-directory retention. Supplying a configuration path also
makes file selection predictable; inherited environment overrides still apply.
For full CLI and config references, see [`docs/cli.md`](../cli.md) and [`docs/config.md`](../config.md).
Resolve the executable deliberately and set the child's working directory so
relative input/output paths and reported artifact paths have a known base.
Read stdout and stderr concurrently to avoid pipe blocking, and impose timeout
or cancellation in the parent process when needed.
## Recommended Invocation
## Result Handling
Use explicit output and report paths for machine workflows:
Interpret the exit result according to the
[CLI output and exit contract](../cli.md#process-output-and-exit-behavior).
Consume transcript/report data through the
[output-file contract](output-files.md), and treat stderr as human-readable detail.
The report supplements the exit result: it may be absent after early failures,
and output may have been written before a later reporting or cleanup failure.
```sh
audita process <transcript.json> \
--glossary <glossary.yaml> \
--output <output-transcript.json> \
--report-json <report.json>
```
If processing fails, preserve available evidence and follow
[operations recovery](../operations.md#failure-inspection-and-recovery).
For reports retained outside the run directory, account for the
[artifact lifecycle](../operations.md#retention-behavior) before dereferencing
reported paths. Neither the existence of an output file nor a diagnostics path
alone establishes successful completion.
Optional commonly used flags:
- `--config <path>`
- `--output-schema <bare-segments|audita-v1>`
- `--work-dir <dir>`
- `--work-dir-retention <always|auto|never>`
- `--transcript-description <text>`
## Related Contracts
## Stdout Contract
On success:
- with `--output`: stdout is expected to be empty;
- without `--output`: stdout contains transcript JSON only.
`--report-json` output is never written to stdout.
## Stderr Contract
Stderr is human-readable status/error output.
On failures:
- stderr includes a concise top-level error;
- when diagnostics are initialized, stderr includes diagnostics directory path.
Do not treat stderr as a machine-stable JSON channel.
## Output and Report File Contract
Transcript output:
- `--output` writes corrected transcript JSON to the provided path;
- output write failures return nonzero.
Report output:
- `--report-json` writes machine-readable process report JSON to the provided path;
- run diagnostics also attempt to write their own `report.json`;
- report write failures return nonzero;
- on failure paths, report writing is best-effort and does not mask the primary run error.
## Diagnostics Contract
When run-directory initialization succeeds, per-run diagnostics artifacts are written under the configured work directory.
Typical artifacts include:
- `source-transcript.json`
- `source-transcript-parsed.json`
- `normalized-transcript.json`
- `normalization-summary.json`
- `chunking-summary.json`
- `invocation.json`
- `effective-config.json`
- `utilization-diagnostics.json`
- `correction-ledger.json`
- `report.json`
- `error.log` (failure)
Retention behavior is controlled by `--work-dir-retention` / config.
## Exit Behavior
Exit codes:
- `0`: success;
- `1`: runtime processing/output/report failure;
- `2`: CLI usage or configuration input error.
Treat any nonzero as subprocess failure.
## Parent-Process Guidance
For reliable orchestration:
- read stdout and stderr concurrently to avoid pipe blocking;
- prefer `--output` and `--report-json` for machine parsing;
- use timeout/cancellation in the parent process;
- inspect diagnostics path and `report.json`/`error.log` on failure.
For input file contracts, see [`docs/integrations/transcript-glossary-files.md`](transcript-glossary-files.md).
- [Transcript and glossary inputs](transcript-glossary-files.md)
- [LLM endpoint compatibility](openai-compatible-llm.md)
- [Configuration inspection commands](../cli.md#configuration-commands)

122
docs/integrations/transcript-glossary-files.md
View File

@@ -1,98 +1,50 @@
# Transcript and Glossary File Integration
# Transcript and Glossary Input Files
## Scope
This document defines the input file contracts for:
- transcript JSON;
- glossary YAML.
These files are loaded and validated before processing begins.
This document owns accepted input formats. Use the [CLI reference](../cli.md) for
invocation and error exit semantics, [operations](../operations.md) for artifact
handling, and [output files](output-files.md) for corrected-transcript serialization.
## Transcript JSON Contract
Audita accepts either top-level shape:
- JSON array of segments; or
- JSON object with a `segments` array.
Accepted top-level shapes are an array of segments or an object containing a
`segments` array. The array must contain at least one segment. Unknown object
fields are ignored by the current parser; an input envelope does not require a
schema/version marker.
Segment fields:
- `id` (optional integer in source form);
- `speaker` (required non-empty string);
- `start` (required finite non-negative number);
- `end` (required finite non-negative number, `>= start`);
- `text` (required non-empty string);
- `categories` (optional string array; entries must be non-empty).
| Segment field | Type and validation |
| --- | --- |
| `id` | Optional integer (or null). Supplied IDs must be unique; missing IDs are accepted. |
| `speaker` | Non-empty string |
| `start`, `end` | Numeric seconds, finite and non-negative; end must be at least start |
| `text` | Non-empty string |
| `categories` | Optional array of non-empty strings |
Additional rules:
- transcript must contain at least one segment;
- duplicate segment IDs are rejected when IDs are present.
Supply explicit start and end times for meaningful timing. The current decoder
uses zero for missing or null timestamps rather than enforcing their presence.
String validation checks empty values, not whitespace trimming. Normalization can
merge segments and reassign IDs, so source IDs need not match output IDs.
Example (`examples/tiny-transcript.json`):
```json
[
{
"id": 1,
"speaker": "A",
"start": 0.0,
"end": 1.2,
"text": "hello world"
}
]
```
The [tiny transcript](../../examples/tiny-transcript.json) is a complete copyable
input. Invalid JSON, unsupported top-level shapes, empty transcripts, invalid field
types, duplicate supplied IDs, and the validation failures above are rejected.
## Glossary YAML Contract
Audita expects top-level `glossary` list entries.
The top-level object contains a `glossary` list with at least one entry.
Entry fields:
- `name` (required non-empty string);
- `category` (required non-empty string);
- `summary` (required non-empty string);
- `aliases` (optional list of strings; entries must be non-empty);
- `plural` (optional string).
| Entry field | Type and validation |
| --- | --- |
| `name` | Non-empty string |
| `category` | Non-empty string |
| `summary` | Non-empty string |
| `aliases` | Optional list of non-empty strings |
| `plural` | Optional string |
Additional rules:
- glossary must contain at least one entry.
The parser checks empty strings without trimming them. Unknown YAML fields are
ignored by the glossary parser, unlike the stricter
[configuration-file schema](../config.md#file-schema).
Example (`examples/tiny-glossary.yaml`):
```yaml
glossary:
- name: Audita
aliases:
- audita
category: product
summary: The Audita transcript correction CLI.
```
## Validation Failure Behavior
Representative transcript validation failures:
- invalid JSON;
- unsupported top-level shape;
- empty `speaker` or `text`;
- invalid times (`NaN`, `Inf`, negative, or `end < start`);
- duplicate IDs;
- empty transcript array.
Representative glossary validation failures:
- invalid YAML;
- empty or missing glossary entries;
- missing required entry fields;
- empty alias values.
These failures surface as schema errors and the process exits nonzero.
## CLI Usage
Minimal invocation:
```sh
audita process ./transcript.json --glossary ./glossary.yaml --output ./corrected.json
```
See also:
- [`docs/cli.md`](../cli.md)
- [`docs/config.md`](../config.md)
- [`examples/tiny-transcript.json`](../../examples/tiny-transcript.json)
- [`examples/tiny-glossary.yaml`](../../examples/tiny-glossary.yaml)
The [tiny glossary](../../examples/tiny-glossary.yaml) is a complete copyable input.
Malformed YAML, an empty/missing glossary, invalid shapes, missing required text,
and empty aliases are rejected. These formats describe input acceptance, not
additional vocabulary restrictions.