eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Document resilient reports and remove obsolete warning material

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-24 17:55:37 +00:00
parent 2f036b0356
commit a770c77a23
26 changed files with 346 additions and 536 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docs/integrations/openai-compatible-llm.md
View File

@@ -22,10 +22,13 @@ cancellation or an earlier deadline also applies. See [configuration](../config.
for defaults and [PromptKit's consumer contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/consumers/pkg-promptkit.md)
for the pinned protocol and retry details.
A failed JSON Schema result after repair is classified as malformed generated
output; proposal and validator callers can turn it into a warning. Provider,
transport, malformed HTTP envelope, retry, admission, and deadline failures stay
runtime failures. Audita decodes into domain types only after PromptKit validates
the JSON. Errors exposed to users carry safe messages while retaining typed
causes for internal classification. Diagnostic request/response content is
sensitive and follows [operations guidance](../operations.md#artifact-handling).
A model operation that exhausts structural repair, a live-parent local operation
timeout, capacity rejection, transport failure, malformed provider response,
or provider HTTP failure is recoverable for that operation. Proposal generation
then contributes no candidates from that response; a failed validation batch
marks its candidates unavailable while other batches and later modules can
continue. Caller cancellation, broken assets or configuration, and unknown
internal failures remain fatal and cancel peers. Audita decodes into domain
types only after PromptKit validates the JSON. Ordinary reports use safe reason
codes and counts rather than provider messages. Full request/response content
is written only with `--debug`; see [operations](../operations.md#diagnostics-directory).

253
docs/integrations/output-files.md
View File

@@ -1,189 +1,90 @@
# Output Files and Report Formats
This document owns externally consumed corrected-transcript and report formats,
and describes the correction ledger and utilization diagnostics. It documents
current serialized shapes without making every debugging artifact a versioned
compatibility promise. Physical filenames, placement, retention, and permissions
belong to [operations](../operations.md); invocation and output channels belong to
the [CLI](../cli.md#streams-and-exit-status).
# Output Files and Reports
## Corrected Transcript
The [output setting](../config.md#settings) selects the envelope:
The corrected transcript uses the selected `output.schema`: `bare-segments`
or `audita-v1`. Its schema and text rules are unchanged by resilience
reporting. See [configuration](../config.md#settings) for selection and
[operations](../operations.md#required-output-files) for placement and writes.
| Encoder | JSON shape |
With reporting enabled, Audita produces three ordinary files:
- `errors.yml`: affected work units and run-level failures;
- `recoveries.yml`: successful logical model operations with repair, service
retry, or defective-proposal filtering;
- `report.md`: a concise overview and complete per-chunk matrices.
The YAML documents have Audita-owned schemas `audita-errors` and
`audita-recoveries`, each with integer `version: 1`. Both contain, in order,
`schema`, `version`, `run_id`, `status`, `started_at`, `completed_at`, `summary`,
and an always-present `errors` or `recoveries` sequence. Status is
`in_progress`, `completed`, or `failed`. Times are UTC RFC3339 strings;
`completed_at` is null until finalization. A recoverable model error can appear
with `status: completed`.
## Summary Counts
Both documents carry the same summary, in this order:
| Count | Meaning |
| --- | --- |
| `bare-segments` | Top-level array of segment objects |
| `audita-v1` | Object with `schema: "audita-v1"`, `version: "v1"`, and `segments` containing the segment array |
| `work_units_total` | Configured module instances × initial chunks. |
| `work_units_started`, `work_units_completed`, `work_units_interrupted`, `work_units_not_started` | Explicit processing states. |
| `work_units_with_errors` | Distinct units with recoverable operation failures. |
| `run_errors` | Fatal processing or output failures outside that unit count. |
| `corrections_applied` | Approved proposals applied, not replacement occurrences. |
| `llm_operations` | Structured-client invocations, including pre-dispatch failures. |
| `proposal_responses` | Structurally usable proposal responses. |
| `recoveries` | Successful operations with one or more recovery reasons. |
| `proposal_responses_with_filtering` | Proposal responses that filtered at least one defective proposal. |
Both emit these segment fields:
Unknown counts before processing are `null`; completed empty counts are `0`.
Retries and repair generations are details of one operation, not additional
operations. Provider token usage, execution timing, and attempt-level evidence
remain in opt-in debug files.
| Field | Type | Meaning |
| --- | --- | --- |
| `id` | Integer | Identifier in the normalized transcript |
| `speaker` | String | Speaker label |
| `start`, `end` | Numbers | Start/end times in seconds |
| `text` | String | Text after normalization and accepted corrections |
| `categories` | Array of strings, optional | Segment categories; omitted when empty |
## Error and Recovery Records
Input IDs and segment boundaries need not survive normalization unchanged;
normalization can merge segments and reassign IDs. The output describes the
resulting transcript, not a lossless copy of the source JSON. Input acceptance
rules belong to the [input-file contract](transcript-glossary-files.md).
An `errors` record with `scope: work_unit` identifies zero-based `stage_index`
and `chunk_index`, module key/instance, initial segment range, applied-correction
count, and a `failures` sequence. Each failure identifies the operation
(`proposal`, `validation`, or `validation_input`), optional validator and
zero-based batch index, safe reason/message, sorted proposal indices when
known, and execution counts when available. Multiple failed batches in one
work unit form one error record. `scope: run` instead records phase, safe reason
and message, optional origin, and output path where relevant. It can describe
a fatal error or secondary output-write failure without inventing a failed
model work unit. Raw provider errors and transcript text are absent.
## Process Report
Each recovery identifies its work unit and successful operation, with sorted
`reasons`, a safe message, available execution counts, and sorted
`filtered_proposals` details. One recovery can contain `output_repaired`,
`service_recovered`, and `proposals_filtered` together. Filtering covers
defective response proposals; low confidence, protected-term rejection,
ordinary editorial rejection, and an apply-time conflict caused by an earlier
accepted correction are not filtering recoveries. Failed operations are errors,
not recoveries. The initial chunk identity remains stable even if earlier
modules change its text.
The report is a JSON object. `report_metadata` identifies its schema separately
from the transcript output envelope:
## Markdown Overview
| Metadata field | Type / current value |
| --- | --- |
| `report_schema_name` | String: `audita-process-report` |
| `report_schema_version` | String: `v2` |
| `output_schema` | String identifying the selected transcript encoder |
| `config_version` | Optional integer, present when file configuration was loaded |
`report.md` includes run ID, status, corrections, work coverage, error and
recovery totals, input/output paths, and full work-unit error and recovery
matrices. Rows show every initial chunk with 1-based chunk number and initial
segment range; columns follow configured module-instance order, including
repeats. Completed error cells are `0` or `1`; recovery cells are operation
counts. `—` means not started and `n*` means interrupted with n facts recorded.
An empty stage or chunk set says “No work units.” Short reason breakdowns count
records containing each reason; a recovery can have several reasons, so those
counts need not sum to the recovery total. The report refers to both YAML
files. It does not include transcript excerpts or correction lists.
The remaining top-level fields are:
## Debug-Only Files
| Field | Type and meaning |
| --- | --- |
| `phase` | String; currently `default_pipeline` |
| `status` | String: `success` or `failed` |
| `operation` | String; currently `process` |
| `transcript_path`, `glossary_path` | Input path strings |
| `output_path` | Optional transcript destination string |
| `modules` | Array of configured module keys, including repeats |
| `started_at`, `completed_at` | RFC 3339 timestamps, possibly fractional; completion is an optional field populated by completed process runs |
| `error_phase`, `error_message` | Optional failure classification and human-readable detail |
| `input_segment_count`, `normalized_segment_count` | Optional integer normalization counts |
| `normalization_merges`, `normalization_id_reassignments` | Optional integer transformation counts |
| `normalization_skipped` | Object of optional integer counters: `different_speakers`, `gap_too_large`, `duration_exceeded`, `token_limit_exceeded`; can be empty |
| `chunking` | Optional section summary described below |
| `diagnostics` | Optional artifact-path metadata described below |
| `llm_summary` | Optional logical LLM execution summary, present when observations were initialized |
| `modules_summary` | Optional aggregate module outcome |
| `module_results` | Optional array of module outcomes |
Failures can preserve partial module results. Missing summaries indicate unavailable
information, not necessarily zero work. Early configuration failures occur before
report construction. Report success describes processing results; a later file-write
or cleanup error can still cause a nonzero CLI exit. Use the exit result and report
together as explained in [subprocess guidance](subprocess.md).
Error phases identify where processing failed, for example `transcript_read`,
`glossary_read`, `transcript_schema`, `glossary_schema`, `chunking`, `runner_setup`,
`runner_execution`, `output_schema`, `serialization`, `output_write`, and
`stdout_write`. This is not an exhaustive enumeration or a promise that every
late failure is represented in a successfully written report. Error messages are
human-readable, not structured fields to parse for control flow.
### Chunking and Diagnostics Metadata
`chunking` contains integer fields `chunk_count`, `min_estimated_chunk_tokens`,
`max_estimated_chunk_tokens`, `total_estimated_transcript_tokens`,
`max_section_tokens`, and `min_section_tokens`, plus optional integer
`target_sections`. Token counts are estimates.
`diagnostics` contains optional path strings:
- `directory_path`
- `source_transcript_path`, `parsed_source_transcript_path`, `normalized_transcript_path`
- `normalization_summary_path`, `chunking_summary_path`
- `utilization_summary_path`, `llm_execution_path`, `selected_profiles_path`, `correction_ledger_path`
- `invocation_metadata_path`, `redacted_effective_config_path`, `error_log_path`
The [operations guide](../operations.md#diagnostics-directory) defines the physical
artifacts and path-lifetime caveats.
### Module Outcomes
`modules_summary` contains integer `module_count`, `total_applied_changes`, and
`total_skipped_changes`, plus optional string `failed_module_instance`.
`total_skipped_changes` counts both apply-time skipped changes and validator
rejections. Their detailed records are stored in separate arrays.
Each `module_results` object contains:
| Field | Type and meaning |
| --- | --- |
| `module_key`, `module_instance` | Strings; the instance distinguishes repeated occurrences of a configured module |
| `replacement_policy` | Optional string describing the module's replacement policy |
| `status` | String describing module outcome (`success` or `failed`) |
| `proposal_count` | Integer |
| `started_at`, `completed_at` | Optional timestamps |
| `error_message` | Optional failure detail |
| `warnings` | Optional array of warning records |
| `validator_decisions` | Optional array of decision records |
| `validator_rejected` | Optional array of rejected-correction records |
| `applied_changes`, `skipped_changes` | Optional arrays of apply outcomes |
A validator decision has `validator_name` (string), `proposal_index` (integer),
`approved` (boolean), `reason_code` and `message` (strings), and optional
`diagnostic_artifact_path` (string).
Rejected corrections include `validator_name`, `proposal_index`, `module_key`,
`module_instance`, `target_segment_id`, `original_text`, `corrected_text`,
`reason_code`, and `message`. Applied changes include the same correction identity
and text fields without validator/reason/message fields, plus integer
`replacement_count`. Skipped changes include correction identity/text fields,
`skip_reason`, and `message`.
A warning has string `scope`, `reason_code`, and `message`; optional integer
`section_index` and `batch_index`; and optional string `validator_name` and
`diagnostic_artifact_path`. Warnings, skipped changes, and rejected corrections
can occur without a failed run. Proposal indexes associate outcomes with their
proposal within a module instance; they are not transcript segment IDs.
## Correction Ledger
The ledger is an array of correction entries; it can be `null` when no module
results are available and empty when modules produced no entries. Unlike the
process report, it has no separate schema-version envelope.
Each entry has required `module_key`, `module_instance`, integer `proposal_index`,
and `disposition`. The current dispositions are:
| Disposition | Meaning |
| --- | --- |
| `applied` | The proposed change was applied |
| `skipped` | The proposal reached application but was skipped |
| `rejected` | A validator rejected the proposal |
| `failed` | Module-level failure record, not necessarily an individual proposal |
Optional fields are `run_id`, integer `segment_id`, `original_text`,
`proposed_corrected_text`, `applied_corrected_text`, `replacement_policy`,
`disposition_reason_code`, and `disposition_message`. Optional validator arrays
are `deterministic_validator_decisions` and `llm_validator_decisions`; each record
contains string `validator_key`, boolean `approved`, string `reason_code`, and
optional string `message`.
Module failure entries can have the zero-value proposal index and no correction
text or segment. Do not interpret them as evidence that proposal zero failed.
Empty optional values are omitted. The ledger records disposition, not a replay
or resume protocol.
## Utilization Diagnostics
`llm_summary` contains integer `logical_calls`, `provider_attempts`,
`transient_retries`, and `structural_repairs`. Its `usage` contains `prompt`,
`completion`, `total`, `cached`, and `cache_write`, each with integer `value`
and boolean `known` and `complete`. A call counts once even if preparation fails.
If no call dispatches, usage is known-zero and complete. Missing `llm_summary`
means observations were unavailable.
The unversioned utilization JSON contains `backends`, keyed by backend ID. Each
entry has configured `concurrency` and `queue_capacity`, the same execution
summary fields, and integer `capacity_wait_ms`, `generation_ms`,
`retry_delay_ms`, and `logical_wall_ms`. It also has `run_wall_time_ms` and
optional `modules` and `validators` arrays. Module records contain the module
identity, logical proposal/validation call counts, and `module_wall_time_ms`.
Validator records contain module/validator identity, `llm_backed`, `calls`, and
`elapsed_ms`. Timings can overlap; they do not decompose total wall time.
`llm-execution.json` has `calls` and `events` arrays joined by `call_id`. Calls
include logical attribution, final classification, summary, elapsed milliseconds,
and available prompt/profile/model metadata. Events contain safe execution
metadata, including attempt and terminal records, with millisecond durations.
Preparation failures have a call and no fabricated events. This artifact has no
provider messages, endpoints, credentials, or transcript content. Other request
and response artifacts can contain redacted prompt and transcript content.
With `--debug`, `correction-ledger.json` contains proposed candidates and
their application, rejection, or interruption details. `llm-execution.json`
holds logical calls and PromptKit execution events; `utilization-diagnostics.json`
groups backend capacity and timing observations while preserving known and
complete usage semantics. Interaction request/response artifacts, selected
profiles, configuration provenance, and input snapshots are also debug-only.
These files are diagnostic evidence, not an alternate ordinary report schema.

23
docs/integrations/subprocess.md
View File

@@ -8,8 +8,10 @@ The [CLI reference](../cli.md) owns command syntax, streams, and exit semantics;
Use an argument array with the transcript path and required glossary argument,
following the [process examples](../cli.md#commands-and-help). For machine workflows,
select explicit transcript and report destinations rather than depending on the
working directory or run-directory retention. Supplying a configuration path also
select an explicit transcript file destination and a configuration path. Reports
then go beside the transcript. For stdout output, select `--report-dir` or use
the invocation working directory. Debug is independent and off by default.
Supplying a configuration path also
makes file selection predictable; only explicitly referenced credential variables
are read from the environment.
@@ -24,15 +26,20 @@ Interpret the exit result according to the
[CLI output and exit contract](../cli.md#streams-and-exit-status).
Consume transcript/report data through the
[output-file contract](output-files.md), and treat stderr as human-readable detail.
The report supplements the exit result: it may be absent after early failures,
and output may have been written before a later reporting or cleanup failure.
The three reports may be absent after early configuration or destination
failures. After initialization, failed reports can describe invalid input or
partial processing. Exit `0` can include recoverable model errors and applied
corrections; inspect `errors.yml` and `recoveries.yml` for those outcomes. A
transcript may have been written before a later required report failure, so
the file alone does not prove completion.
If processing fails, preserve available evidence and follow
[operations recovery](../operations.md#failure-inspection-and-recovery).
For reports retained outside the run directory, account for the
[artifact lifecycle](../operations.md#retention-behavior) before dereferencing
reported paths. Neither the existence of an output file nor a diagnostics path
alone establishes successful completion.
Ordinary reports are independent of optional retained debug evidence. Parent
cancellation reaches the child through its context; Audita joins workers and
attempts failed-status report finalization before exiting. Uncatchable
termination can leave `in_progress` reports. Neither the existence of an output
file nor a debug directory alone establishes successful completion.
## Related Contracts