Add operations and troubleshooting documentation
This commit is contained in:
113
docs/operations.md
Normal file
113
docs/operations.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# Audita Operations
|
||||
|
||||
## Scope
|
||||
|
||||
This document covers operational behavior for `audita process` as currently implemented:
|
||||
- run lifecycle;
|
||||
- output and report files;
|
||||
- diagnostics artifacts;
|
||||
- run-directory retention behavior;
|
||||
- failure inspection and recovery.
|
||||
|
||||
For command syntax, see [`docs/cli.md`](cli.md).
|
||||
|
||||
## Process Run Lifecycle
|
||||
|
||||
A `process` run performs these high-level steps:
|
||||
1. load effective config (defaults + optional file + env + CLI);
|
||||
2. create a per-run diagnostics directory;
|
||||
3. load transcript JSON and glossary YAML;
|
||||
4. parse/validate input schemas;
|
||||
5. normalize transcript and compute chunking;
|
||||
6. run configured modules/validators;
|
||||
7. serialize output schema and write transcript output;
|
||||
8. build and write process report;
|
||||
9. apply run-directory retention.
|
||||
|
||||
If a failure happens after diagnostics initialization, the run writes failure details and returns nonzero.
|
||||
|
||||
## Output Files
|
||||
|
||||
Transcript output:
|
||||
- when `--output <path>` is set, corrected transcript JSON is written to that file;
|
||||
- when `--output` is omitted, corrected transcript JSON is written to stdout.
|
||||
|
||||
Report output:
|
||||
- when `--report-json <path>` is set, Audita writes a process report JSON file;
|
||||
- the run directory also writes its own `report.json` artifact.
|
||||
|
||||
On success with `--output`, stdout is expected to be empty.
|
||||
|
||||
## Diagnostics Directory
|
||||
|
||||
By default, runs use `work_dir` from effective config (default `/tmp/audita`).
|
||||
Each run directory is created under the work dir using a generated ID like `run-<unix-nanos>`.
|
||||
|
||||
Top-level diagnostics artifacts:
|
||||
- `source-transcript.json`
|
||||
- `source-transcript-parsed.json`
|
||||
- `normalized-transcript.json`
|
||||
- `normalization-summary.json`
|
||||
- `chunking-summary.json`
|
||||
- `utilization-diagnostics.json`
|
||||
- `correction-ledger.json`
|
||||
- `invocation.json`
|
||||
- `effective-config.json` (redacted)
|
||||
- `report.json`
|
||||
- `error.log` (failure runs)
|
||||
|
||||
Report diagnostics metadata includes resolved paths to these artifacts.
|
||||
|
||||
## Correction Ledger and Utilization Diagnostics
|
||||
|
||||
`correction-ledger.json` records correction dispositions:
|
||||
- `applied`
|
||||
- `skipped`
|
||||
- `rejected`
|
||||
- `failed`
|
||||
|
||||
`utilization-diagnostics.json` records effective concurrency and execution timing summaries for run/module/validator activity.
|
||||
|
||||
## Retention Behavior
|
||||
|
||||
Retention is controlled by `work_dir_retention` (`auto|always|never`).
|
||||
|
||||
Current behavior:
|
||||
- failed runs are always retained;
|
||||
- `always`: successful runs are retained;
|
||||
- `auto`: successful runs are retained only when skipped/rejected corrections occurred; clean successful runs are removed;
|
||||
- `never`: successful runs are currently retained (same net retention outcome as `always` in current implementation).
|
||||
|
||||
Even when a successful run directory is removed under `auto`, an explicit `--report-json` file is still preserved at its target path.
|
||||
|
||||
## Failure Inspection
|
||||
|
||||
For failed runs:
|
||||
1. read stderr for the top-level failure and diagnostics path;
|
||||
2. open `error.log` in the reported run directory;
|
||||
3. inspect run `report.json` (`status`, `error_phase`, `error_message`);
|
||||
4. inspect related artifacts referenced by report diagnostics metadata.
|
||||
|
||||
Typical `error_phase` values include:
|
||||
- `transcript_read`
|
||||
- `glossary_read`
|
||||
- `transcript_schema`
|
||||
- `glossary_schema`
|
||||
- `chunking`
|
||||
- `runner_setup`
|
||||
- `runner_execution`
|
||||
- `output_schema`
|
||||
- `serialization`
|
||||
- `output_write`
|
||||
- `stdout_write`
|
||||
|
||||
## Recovery Guidance
|
||||
|
||||
Safe recovery pattern:
|
||||
1. correct the immediate input/config/output-path problem;
|
||||
2. rerun with `--work-dir-retention always` during debugging;
|
||||
3. once stable, restore your normal retention mode.
|
||||
|
||||
Not implemented:
|
||||
- resume/checkpoint APIs
|
||||
- remote diagnostics/report storage
|
||||
153
docs/troubleshooting.md
Normal file
153
docs/troubleshooting.md
Normal file
@@ -0,0 +1,153 @@
|
||||
# Audita Troubleshooting
|
||||
|
||||
## Scope
|
||||
|
||||
This guide lists recurring implemented failure modes for `audita process` and `audita config`.
|
||||
|
||||
For each entry: symptom, likely cause, inspect, and fix.
|
||||
|
||||
## Config Validation Fails
|
||||
|
||||
Symptom:
|
||||
- `audita config validate --config <path>` exits nonzero.
|
||||
|
||||
Likely causes:
|
||||
- missing `version`;
|
||||
- unsupported config version;
|
||||
- unknown YAML field;
|
||||
- unsupported module key or output schema;
|
||||
- invalid numeric/range/concurrency/retention values.
|
||||
|
||||
Inspect:
|
||||
1. rerun `audita config validate --config <path>` and read stderr.
|
||||
2. if needed, inspect effective config with `audita config print-effective --config <path>`.
|
||||
|
||||
Fix:
|
||||
- set `version: 1`;
|
||||
- remove unknown fields;
|
||||
- use supported module keys and output schemas (`bare-segments`, `audita-v1`);
|
||||
- correct invalid values to satisfy validation constraints.
|
||||
|
||||
## Config File Resolution Errors
|
||||
|
||||
Symptom:
|
||||
- `audita process` fails before processing with config-related errors like `config file not found`.
|
||||
|
||||
Likely causes:
|
||||
- `--config` points to a missing path;
|
||||
- `AUDITA_CONFIG` points to a missing path;
|
||||
- unreadable config path.
|
||||
|
||||
Inspect:
|
||||
1. confirm `--config` or `AUDITA_CONFIG` path exists;
|
||||
2. run `audita config validate --config <path>` directly.
|
||||
|
||||
Fix:
|
||||
- correct the path or unset invalid `AUDITA_CONFIG`;
|
||||
- fix permissions for the config file.
|
||||
|
||||
## Transcript or Glossary Schema Errors
|
||||
|
||||
Symptom:
|
||||
- stderr includes `transcript_schema` or `glossary_schema` and run exits nonzero.
|
||||
|
||||
Likely causes:
|
||||
- transcript is not valid JSON or has invalid segment fields;
|
||||
- glossary is not valid YAML or has missing required glossary entry fields.
|
||||
|
||||
Inspect:
|
||||
1. check stderr for parser/validation details;
|
||||
2. if diagnostics were created, inspect `error.log` and run `report.json` (`error_phase`);
|
||||
3. inspect `source-transcript.json` and `source-transcript-parsed.json` in the run directory.
|
||||
|
||||
Fix:
|
||||
- correct transcript JSON shape/content;
|
||||
- correct glossary YAML shape/content and required entry fields;
|
||||
- rerun validation with known-good tiny examples for comparison:
|
||||
- `examples/tiny-transcript.json`
|
||||
- `examples/tiny-glossary.yaml`
|
||||
|
||||
## LLM Runtime/Backend Failures
|
||||
|
||||
Symptom:
|
||||
- stderr includes `runner_execution` (or backend timeout/error details) and nonzero exit.
|
||||
|
||||
Likely causes:
|
||||
- unreachable/failed LLM endpoint;
|
||||
- timeout/cancellation;
|
||||
- runtime module execution failure.
|
||||
|
||||
Inspect:
|
||||
1. inspect stderr for backend message details;
|
||||
2. inspect run `report.json` (`error_phase`, `module_results`);
|
||||
3. inspect diagnostics payloads and `error.log`.
|
||||
|
||||
Fix:
|
||||
- verify model/base URL/API key settings;
|
||||
- increase timeout if needed;
|
||||
- rerun with `--work-dir-retention always` while debugging.
|
||||
|
||||
## Output File Write Failure
|
||||
|
||||
Symptom:
|
||||
- stderr includes `failed to write output file` and run exits nonzero.
|
||||
|
||||
Likely causes:
|
||||
- output path directory missing;
|
||||
- insufficient filesystem permissions;
|
||||
- invalid output target path.
|
||||
|
||||
Inspect:
|
||||
1. check `--output` target directory exists and is writable;
|
||||
2. inspect run diagnostics `error.log` and report `error_phase`.
|
||||
|
||||
Fix:
|
||||
- write to a valid writable path;
|
||||
- create missing directories;
|
||||
- adjust permissions.
|
||||
|
||||
## Report File Write Failure
|
||||
|
||||
Symptom:
|
||||
- stderr includes `failed to write report JSON file` and run exits nonzero.
|
||||
|
||||
Likely causes:
|
||||
- invalid or unwritable `--report-json` target path.
|
||||
|
||||
Inspect:
|
||||
1. verify parent directory exists and is writable;
|
||||
2. inspect diagnostics `error.log` for `report_write` context.
|
||||
|
||||
Fix:
|
||||
- choose a writable report path;
|
||||
- create missing directories;
|
||||
- rerun.
|
||||
|
||||
## Unsupported Output Schema
|
||||
|
||||
Symptom:
|
||||
- stderr includes `unsupported output schema` and run exits nonzero.
|
||||
|
||||
Likely causes:
|
||||
- unsupported `--output-schema` value;
|
||||
- unsupported `output.schema` in config.
|
||||
|
||||
Inspect:
|
||||
1. check CLI/config schema key;
|
||||
2. run `audita config validate --config <path>` when config is involved.
|
||||
|
||||
Fix:
|
||||
- use `bare-segments` or `audita-v1`.
|
||||
|
||||
## Diagnostics Directory Lookup
|
||||
|
||||
Symptom:
|
||||
- run fails and you need artifacts for debugging.
|
||||
|
||||
Inspect:
|
||||
1. read stderr for `audita process: diagnostics: <run-dir>`;
|
||||
2. open `<run-dir>/report.json` and `<run-dir>/error.log`;
|
||||
3. use diagnostics paths embedded in report metadata for artifact lookup.
|
||||
|
||||
Fix:
|
||||
- rerun with `--work-dir-retention always` to preserve run directories during investigation.
|
||||
Reference in New Issue
Block a user