audita/docs/roadmap/resilience.md

# Resilient Processing and Focused Run Reports

**Last updated:** 2026-09-24

**Status:** Implemented. This document records the selected behavior and user
intent. Concrete design decisions and completion evidence are in
[historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/a83a518/docs/roadmap/implementation.md).

## Purpose and User Intent

Audita should reliably produce a useful transcript even when its LLM services
cannot improve it. Exhausted retries or unusable model responses must not turn
an otherwise processable transcript into a failed production run.

> Failure of every LLM operation across every stage and chunk still produces a
> successful output containing the deterministically normalized transcript, with
> no LLM corrections applied. Partial success preserves as many fully validated,
> applicable corrections as possible.

Deterministic normalization remains intentional and may change segment layout,
identifiers, and text before LLM work begins. Output uses the selected transcript
encoder; byte-for-byte preservation of the original input and mapping corrections
back to pre-normalization segmentation are not requirements.

The overriding goal is to minimize how much useful processing is lost because of
one problematic transcript segment. A stage/chunk work unit is an attribution
boundary, not an all-or-nothing transaction.

Existing reports contain useful information, but there are no compatibility
obligations for reporting or debug output. Reuse existing machinery only when it
makes the implementation simpler and clearer; preservation is not itself a goal.
Internal structures and measurements must serve Audita's processing workflow.
The reporting package adapts those outcomes to structured and human-facing
presentations, rather than reshaping processing around a reporting convenience.

## Historical Baseline and Implemented Change

The [pipeline](../internal/pipeline.md), [LLM integration](../integrations/openai-compatible-llm.md),
[report assembly](../internal/diagnostics-reporting.md), and
[artifact lifecycle](../operations.md) describe current behavior.

Before this work, exhausted structural repairs could become warnings and skipped
proposals or rejected validation batches. Other model runtime failures canceled
peer work, stopped subsequent modules, and prevented final transcript output.
Successful repairs appeared in execution observations rather than a dedicated
recovery classification. Detailed artifacts were written during ordinary runs
and could later be removed by retention. The working transcript was re-chunked
between modules.

This feature changed those behaviors. It builds on the completed
[library adoption](library-adoption.md) and [review remediation](remediation.md).
Fatal cancellation remains necessary; unsuccessful model work must be separated
from failures that actually require cancellation of the run.

## Fixed Chunks and Ordered Stages

Normalize first, then partition the transcript once. Preserve chunk boundaries,
segment membership, segment identity, and ordering throughout LLM processing.
Text evolves as accepted corrections are applied. Recombine chunks in original
order after the LLM stages finish.

A stage is a configured module instance. Repeated instances of the same module
have distinct identities. Proposal generation and each validator are operations
within a stage; validation may use multiple batches within a chunk. A work unit
is one stage instance applied to one chunk.

Preserve the overall stage sequence: finish a stage across its chunks before
advancing to the next stage. Concurrent chunk work shares PromptKit's backend
capacity. Failure in one unit does not prevent other chunks or later stages
from executing.

Retain operator control over chunk size and the current defaults. Production
experience indicates that chunk size affects model latency nonlinearly and that
useful settings are well below backend context limits. Tuning is an empirical
operator decision. Do not add automatic headroom calculations, cross-profile
sizing, adaptive repartitioning, or automatic tuning in this feature. A later
operation that cannot process its input follows the same safe error handling;
it does not silently alter chunk boundaries. Validation batching within a chunk
remains supported.

Independent whole-pipeline traversal by chunks and domain-aware response repair
are explicitly deferred in [future work](future.md). Read-only neighboring
context, cross-chunk edits, and transcript-wide restructuring are outside this
unit, without becoming permanent prohibitions on future capabilities.

## Partial Success and Failure Isolation

- Keep bounded transport retries, structural repairs, operation deadlines, and
  shared backend capacity. Completion comes from safe fallback, not unlimited
  attempts or an outer retry loop around PromptKit.
- Apply a correction only after it passes every required validator and the
  application checks. Failed validation must never be treated as approval.
- If a validation batch fails, exclude its affected candidates while allowing
  other candidates to continue through the remaining validators. Preserve fully
  validated, applicable corrections from successful batches in the same unit.
- Track candidates by stable proposal identity. Batch membership can change
  between validators; equal batch numbers do not establish candidate identity.
- A failed proposal-generation operation leaves its chunk unchanged for that
  stage because there are no reliable proposals to salvage. Failures later in
  validation need not discard unrelated successful candidates.
- Preserve earlier stages' corrections and other chunks' successful work.
  Record partial success explicitly; a unit can apply corrections and have errors.
- Continue despite model-operation failures, including exhausted repairs/retries,
  unusable model decisions, provider/transport failures, local operation deadlines,
  and admission failures. An operation need not exhaust its attempt budget to fail.
- A valid empty proposal list or valid assessments rejecting all corrections is
  successful processing. Correction count alone does not determine failure.
- Keep safe filtering of defective proposals. Do not retry ordinary editorial
  rejections. For now, an unusable validator decision set excludes the affected
  candidates and records an error after existing permitted recovery. New bounded
  feedback/regeneration for domain-invalid responses belongs to future work.
- Retain fatal treatment for invalid input/configuration, internal defects,
  explicit run cancellation, and inability to produce/write the transcript.
  Caller cancellation is distinct from a local model-operation timeout.
- Completed runs with model errors return success. Required report-write failures
  are an additional fatal output condition when reporting is enabled, as below.

PromptKit owns transport, transient retries, structural repair, and admission.
Audita owns domain acceptance, partial outcomes, fallback, and classification.
ConfigKit owns configuration resolution. Debug and reporting choices must not
change correction decisions, retry budgets, or continuation.

## Errors, Recoveries, and Debug

Classify by eventual outcome and impact. These are three reporting categories;
debug is evidence, not a failure severity. Whether execution continues is a
separate property.

| Category | Meaning | Representative cases |
| --- | --- | --- |
| Errors | Intended work could not complete reliably, or a fatal run-level operation failed. Affected candidates are excluded; other corrections can succeed. | Exhausted repair; unavailable provider; unusable validation decisions; operation deadline. |
| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient service failure recovered; defective proposals safely discarded. |
| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompts/responses, individual attempts, execution traces. |

Normal editorial disagreement belongs in debug. A validator correctly rejecting
a meaning-changing edit is expected behavior. A proposal referring to nonexistent
original text is defective; safely filtering it is a recovery. An unresolved
missing/duplicate decision-index response is an error, not a recovery.

Record an operation's eventual outcome once. Several failed attempts followed by
success produce a consolidated recovery. Attempts ending in failure contribute
to error details, not additional recoveries. Earlier operations that recovered
remain recorded if a later operation introduces an error in the same unit.

Separate output-quality recovery, defective-proposal filtering, and service
recovery with structured reasons. Network instability must not look like a prompt
defect. Do not introduce an arbitrary warning threshold or a fourth category.

### Counting and Processing Volume

- Count error-bearing work units once per stage/chunk, consolidating underlying
  operation failures as details. A partially successful unit still counts once.
- Count recoveries once per recovered operation. Aggregate defective-proposal
  filtering per proposal response, retaining affected-proposal counts as detail.
  Attempts and individual discarded proposals are not independent incidents.
- Keep work-unit coverage separate from operation/recovery counts. Show processing
  volume with explicitly labeled denominators; do not divide recovered-operation
  counts by work-unit counts and call that a recovery rate.
- Report fatal run-level errors separately from the stage/chunk grid when they
  cannot be attributed to a work unit. Do not invent a chunk to accommodate them.
- Derive YAML records, Markdown summaries, and CLI counts from actual processing
  outcomes with deterministic ordering, rather than independently calculating them.

## Production Outputs, Placement, and Lifecycle

Reporting is enabled by default. The ordinary bundle is:

- `errors.yml`: structured error-bearing units, their underlying failures, and
  applicable run-level failures.
- `recoveries.yml`: structured recovered operations and pertinent counts.
- `report.md`: a concise human-facing summary of outcome, corrections, processing
  volume, errors, recoveries, and their locations in the pipeline.

Write all three documents, including empty event lists for uneventful completed
runs. Use Audita-owned schemas with safe explanations and stable reason codes;
retain run, stage-instance, chunk, operation, and validator/batch identities where
applicable. The implementation plan specifies the versioned schemas; neither old
report formats nor raw library structures define the new output contract.

Place reports beside the output transcript. Retain transcript-to-stdout support
with an explicit report-directory selection, defaulting to the invocation working
directory. Preserve the existing overwrite convention: a new run can replace files
at its selected destinations. Do not add a mandatory fresh-directory restriction
or compatibility exports.

Before model work, create initial versions of all three reports carrying a common
run identifier and explicit `in_progress` status (rendered as prose in Markdown).
Fail immediately if initialization fails. Also check the transcript destination;
creating neighboring reports does not prove that an existing transcript file is
writable. Preflight must not prematurely destroy an existing transcript.

Finalize all required outputs and check every final write. Initial write success
cannot guarantee later success if space or permissions change. Failure to write
the transcript or finalize a required report returns nonzero and must not erase
an already produced transcript. Where possible, finalize available reports with
an explicit failed status on fatal termination. Abrupt termination may leave
`in_progress`; elaborate crash-recovery machinery is not required.

Provide an option to disable reporting entirely. It suppresses all three files
(`errors.yml`, `recoveries.yml`, and `report.md`), including initialization and
final writes, while retaining the brief CLI summary and independently selectable
debug output. Disabled reporting must not create or require a writable report
destination, and must not remove reports from earlier runs.
Transcript output failures remain fatal regardless of reporting selection.

Ordinary reports survive independently of debug cleanup. Reporting configuration
and finalization must not depend on retaining a debug run directory.

## Debug and Internal Boundaries

Production issue records are compact. Full responses, transcript excerpts,
correction lists, and complete validation decisions are opt-in debug evidence.
Use a single ConfigKit-backed debug boolean with a `--debug` CLI override. Debug
artifacts may use JSON where useful. Preserve configured-secret redaction in every
presentation.

Debug-off prevents detailed artifact persistence; writing and later deleting
those files is insufficient. In-memory domain state needed to process candidates
correctly is not debug output. Debug and ordinary reporting are independent
selections and do not alter processing behavior.

There is no obligation to retain `--report-json`, old artifact layouts, parallel
exports, or old retention settings. Reuse code only where useful to the new
workflow. Enabled debug output is retained on success or failure; disabled debug
creates no debug directory. This supersedes the retention question in the
[runtime follow-up](runtime-behavior.md). The legacy `never` behavior is not the
meaning of the new reporting-disable option. Optional debug-write failures produce
a safe stderr diagnostic without changing correction decisions or the run exit
status; ordinary report and transcript-write failures retain their fatal semantics.

## Deterministic Human-Facing Presentation

Use the CLI style:

> Completed: 18 corrections applied; 3 work units with errors; 7 recoveries.

Do not say that all error-bearing units were skipped: some apply valid corrections.
Include report locations when reporting is enabled and the files exist. Send the
summary to stderr so stdout remains exclusively available for transcript output.
Disabled reporting does not suppress the summary or fatal-error messages.

Keep `report.md` concise and high-level. Use separate error and recovery matrices
with stages as columns and chunks as rows. Include every chunk in fixed order;
retain configured stage order and distinct labels for repeated module instances.
No automatic width detection, thresholds, adaptive omission, or table splitting.
A wide or long table is acceptable.

The error matrix counts error-bearing work units: each attempted cell is 0 or 1.
Recovery cells count recovered operations and can exceed 1. An error cell may
coexist with recoveries and successfully applied corrections in the same unit.
Distinguish completed work with zero events from unattempted work; do not portray
canceled/unavailable processing as clean completion. Include totals, processing
volume, and a short reason breakdown. Detailed inspection belongs in YAML or debug.

## Target Acceptance Criteria

- Total LLM failure still returns success and the encoded normalized transcript,
  provided required output writes succeed.
- Mixed-success runs preserve independently validated, applicable corrections
  within the same stage/chunk as a failed batch. No candidate bypasses a required
  validator; later validators handle survivors by stable proposal identity.
- Subsequent stages and unaffected chunks execute after recoverable failures.
  Fatal errors and explicit cancellation stop and join active work appropriately.
- Chunk boundaries and identity remain fixed, with operator-selected sizing and
  unchanged defaults; final recombination preserves ordering.
- Valid no-change/editorial-rejection outcomes are successful. Repaired/filtered
  output and unresolved failures receive distinct classifications and counts.
- YAML, Markdown, and CLI agree on partial outcomes and totals; error matrices
  count affected work units, recovery matrices count recovered operations.
- With reporting enabled, all three reports are initialized before model work,
  finalized and retained, including empty lists. Initial or final required-output
  write failures produce nonzero exits without destroying completed transcript output.
- Reporting can be disabled without requiring its destination. Debug can be enabled
  independently; neither selection changes processing outcomes. Debug-off writes
  no detailed debug artifacts, and stdout transcript output remains supported.
- Reports use the fixed deterministic layout without adaptive presentation logic.
- Meaningful controlled-provider tests cover exhausted recovery, failed validation
  batches with successful survivors, total failure, fatal cancellation, output
  failures, and reporting/debug selections. Preserve domain safeguards, capacity
  limits, and redaction.
- Canonical CLI, configuration, operations, integration, internal, and architecture
  documentation describes the implemented final behavior.

## Implementation Reference

The product decisions are settled. The [historical implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/a83a518/docs/roadmap/implementation.md) defines
the selected settings and precedence, YAML schemas, counts and ordering, interrupted
work presentation, destination checks, and output finalization. It also owns the
ordered implementation stages and validation work. This document remains the
reference for the desired behavior and its acceptance criteria.