105 lines
3.4 KiB
Markdown
105 lines
3.4 KiB
Markdown
# Audita Diagnostics
|
|
|
|
This document describes the run-directory diagnostics artifacts produced by `audita process`.
|
|
|
|
## Purpose
|
|
|
|
Diagnostics provide machine-readable run context and execution artifacts for:
|
|
- failure debugging;
|
|
- validator/correction review;
|
|
- post-run performance analysis.
|
|
|
|
Diagnostics are written under the configured work directory (`--work-dir`) when run-directory initialization succeeds.
|
|
|
|
## Core artifacts
|
|
|
|
Typical artifacts in each run directory:
|
|
- `source-transcript.json`
|
|
- `source-transcript-parsed.json`
|
|
- `normalized-transcript.json`
|
|
- `normalization-summary.json`
|
|
- `chunking-summary.json`
|
|
- `invocation.json`
|
|
- `effective-config.json` (redacted)
|
|
- module/validator LLM interaction artifacts
|
|
- `report.json`
|
|
- `error.log` on failure
|
|
|
|
## Utilization diagnostics artifact
|
|
|
|
Artifact:
|
|
- `utilization-diagnostics.json`
|
|
|
|
High-level fields:
|
|
- `effective_concurrency`:
|
|
- total/proposal/validation LLM concurrency limits in effect.
|
|
- `run_timing`:
|
|
- run wall time;
|
|
- scheduler queue wait time;
|
|
- LLM execution time;
|
|
- deterministic validator time;
|
|
- max/average in-flight LLM calls.
|
|
- `llm_calls`:
|
|
- total proposal and validation LLM call counts.
|
|
- `modules`:
|
|
- module-level timing summaries.
|
|
- `validators`:
|
|
- per-validator timing summaries keyed by stable validator key.
|
|
|
|
## Correction ledger artifact
|
|
|
|
Artifact:
|
|
- `correction-ledger.json`
|
|
|
|
Ledger records are flattened review entries derived from module results and include:
|
|
- module/proposal identity (`module_key`, `module_instance`, `proposal_index`, `segment_id`);
|
|
- correction text fields and replacement policy when available;
|
|
- disposition:
|
|
- `applied`
|
|
- `rejected`
|
|
- `skipped`
|
|
- `failed`
|
|
- stable reason codes/messages;
|
|
- deterministic and LLM validator decision snapshots using stable validator keys.
|
|
|
|
Validator rejection and proposal-application skip are distinct dispositions.
|
|
Module warnings are reported in module results and diagnostics metadata, but do not create standalone correction-ledger rows.
|
|
|
|
## Report references
|
|
|
|
`report.json` and optional `--report-json` output include diagnostics metadata paths for:
|
|
- utilization diagnostics artifact;
|
|
- correction ledger artifact;
|
|
- existing transcript/normalization/chunking/invocation/effective-config artifacts.
|
|
|
|
Module report entries also include warning records for malformed proposal-generation payloads and malformed validator batches.
|
|
|
|
## Retention behavior
|
|
|
|
Run-directory retention follows configured policy:
|
|
- `always`: keep all run directories;
|
|
- `never`: keep successful run directories;
|
|
- `auto`: keep failed runs and successful runs with skipped/rejected corrections.
|
|
|
|
## Redaction guarantees
|
|
|
|
API keys and other configured secrets are redacted from:
|
|
- `effective-config.json`;
|
|
- LLM interaction diagnostics artifacts;
|
|
- reports and surfaced errors.
|
|
|
|
## Debugging guide
|
|
|
|
When debugging:
|
|
- slow runs:
|
|
- inspect `utilization-diagnostics.json` (`run_timing`, `modules`, `validators`, in-flight metrics).
|
|
- validator rejections:
|
|
- inspect `correction-ledger.json` rejected entries and matching validator decisions;
|
|
- inspect validator response diagnostics payloads.
|
|
- module warnings:
|
|
- inspect module `warnings` entries in `report.json` or `--report-json`;
|
|
- follow any diagnostic artifact path on the warning to the recorded error/response payload.
|
|
- application skips:
|
|
- inspect `correction-ledger.json` skipped entries and skip reason codes;
|
|
- compare with validator decisions to distinguish validation rejection vs apply-time skip.
|