audita/docs/roadmap/resilience.md

Resilient Processing and Focused Run Reports

Last updated: 2026-09-24

Status: Implemented. This document records the selected behavior and user intent. Concrete design decisions and completion evidence are in historical implementation plan.

Purpose and User Intent

Audita should reliably produce a useful transcript even when its LLM services cannot improve it. Exhausted retries or unusable model responses must not turn an otherwise processable transcript into a failed production run.

Failure of every LLM operation across every stage and chunk still produces a successful output containing the deterministically normalized transcript, with no LLM corrections applied. Partial success preserves as many fully validated, applicable corrections as possible.

Deterministic normalization remains intentional and may change segment layout, identifiers, and text before LLM work begins. Output uses the selected transcript encoder; byte-for-byte preservation of the original input and mapping corrections back to pre-normalization segmentation are not requirements.

The overriding goal is to minimize how much useful processing is lost because of one problematic transcript segment. A stage/chunk work unit is an attribution boundary, not an all-or-nothing transaction.

Existing reports contain useful information, but there are no compatibility obligations for reporting or debug output. Reuse existing machinery only when it makes the implementation simpler and clearer; preservation is not itself a goal. Internal structures and measurements must serve Audita's processing workflow. The reporting package adapts those outcomes to structured and human-facing presentations, rather than reshaping processing around a reporting convenience.

Historical Baseline and Implemented Change

The pipeline, LLM integration, report assembly, and artifact lifecycle describe current behavior.

Before this work, exhausted structural repairs could become warnings and skipped proposals or rejected validation batches. Other model runtime failures canceled peer work, stopped subsequent modules, and prevented final transcript output. Successful repairs appeared in execution observations rather than a dedicated recovery classification. Detailed artifacts were written during ordinary runs and could later be removed by retention. The working transcript was re-chunked between modules.

This feature changed those behaviors. It builds on the completed library adoption and review remediation. Fatal cancellation remains necessary; unsuccessful model work must be separated from failures that actually require cancellation of the run.

Fixed Chunks and Ordered Stages

Normalize first, then partition the transcript once. Preserve chunk boundaries, segment membership, segment identity, and ordering throughout LLM processing. Text evolves as accepted corrections are applied. Recombine chunks in original order after the LLM stages finish.

A stage is a configured module instance. Repeated instances of the same module have distinct identities. Proposal generation and each validator are operations within a stage; validation may use multiple batches within a chunk. A work unit is one stage instance applied to one chunk.

Preserve the overall stage sequence: finish a stage across its chunks before advancing to the next stage. Concurrent chunk work shares PromptKit's backend capacity. Failure in one unit does not prevent other chunks or later stages from executing.

Retain operator control over chunk size and the current defaults. Production experience indicates that chunk size affects model latency nonlinearly and that useful settings are well below backend context limits. Tuning is an empirical operator decision. Do not add automatic headroom calculations, cross-profile sizing, adaptive repartitioning, or automatic tuning in this feature. A later operation that cannot process its input follows the same safe error handling; it does not silently alter chunk boundaries. Validation batching within a chunk remains supported.

Independent whole-pipeline traversal by chunks and domain-aware response repair are explicitly deferred in future work. Read-only neighboring context, cross-chunk edits, and transcript-wide restructuring are outside this unit, without becoming permanent prohibitions on future capabilities.

Partial Success and Failure Isolation

• Keep bounded transport retries, structural repairs, operation deadlines, and shared backend capacity. Completion comes from safe fallback, not unlimited attempts or an outer retry loop around PromptKit.
• Apply a correction only after it passes every required validator and the application checks. Failed validation must never be treated as approval.
• If a validation batch fails, exclude its affected candidates while allowing other candidates to continue through the remaining validators. Preserve fully validated, applicable corrections from successful batches in the same unit.
• Track candidates by stable proposal identity. Batch membership can change between validators; equal batch numbers do not establish candidate identity.
• A failed proposal-generation operation leaves its chunk unchanged for that stage because there are no reliable proposals to salvage. Failures later in validation need not discard unrelated successful candidates.
• Preserve earlier stages' corrections and other chunks' successful work. Record partial success explicitly; a unit can apply corrections and have errors.
• Continue despite model-operation failures, including exhausted repairs/retries, unusable model decisions, provider/transport failures, local operation deadlines, and admission failures. An operation need not exhaust its attempt budget to fail.
• A valid empty proposal list or valid assessments rejecting all corrections is successful processing. Correction count alone does not determine failure.
• Keep safe filtering of defective proposals. Do not retry ordinary editorial rejections. For now, an unusable validator decision set excludes the affected candidates and records an error after existing permitted recovery. New bounded feedback/regeneration for domain-invalid responses belongs to future work.
• Retain fatal treatment for invalid input/configuration, internal defects, explicit run cancellation, and inability to produce/write the transcript. Caller cancellation is distinct from a local model-operation timeout.
• Completed runs with model errors return success. Required report-write failures are an additional fatal output condition when reporting is enabled, as below.

PromptKit owns transport, transient retries, structural repair, and admission. Audita owns domain acceptance, partial outcomes, fallback, and classification. ConfigKit owns configuration resolution. Debug and reporting choices must not change correction decisions, retry budgets, or continuation.

Errors, Recoveries, and Debug

Classify by eventual outcome and impact. These are three reporting categories; debug is evidence, not a failure severity. Whether execution continues is a separate property.

Category	Meaning	Representative cases
Errors	Intended work could not complete reliably, or a fatal run-level operation failed. Affected candidates are excluded; other corrections can succeed.	Exhausted repair; unavailable provider; unusable validation decisions; operation deadline.
Recoveries	A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied.	Invalid output repaired; transient service failure recovered; defective proposals safely discarded.
Debug	Detailed evidence of processing and ordinary decisions.	Proposals, applied corrections, validator decisions, prompts/responses, individual attempts, execution traces.

Normal editorial disagreement belongs in debug. A validator correctly rejecting a meaning-changing edit is expected behavior. A proposal referring to nonexistent original text is defective; safely filtering it is a recovery. An unresolved missing/duplicate decision-index response is an error, not a recovery.

Record an operation's eventual outcome once. Several failed attempts followed by success produce a consolidated recovery. Attempts ending in failure contribute to error details, not additional recoveries. Earlier operations that recovered remain recorded if a later operation introduces an error in the same unit.

Separate output-quality recovery, defective-proposal filtering, and service recovery with structured reasons. Network instability must not look like a prompt defect. Do not introduce an arbitrary warning threshold or a fourth category.

Counting and Processing Volume

• Count error-bearing work units once per stage/chunk, consolidating underlying operation failures as details. A partially successful unit still counts once.
• Count recoveries once per recovered operation. Aggregate defective-proposal filtering per proposal response, retaining affected-proposal counts as detail. Attempts and individual discarded proposals are not independent incidents.
• Keep work-unit coverage separate from operation/recovery counts. Show processing volume with explicitly labeled denominators; do not divide recovered-operation counts by work-unit counts and call that a recovery rate.
• Report fatal run-level errors separately from the stage/chunk grid when they cannot be attributed to a work unit. Do not invent a chunk to accommodate them.
• Derive YAML records, Markdown summaries, and CLI counts from actual processing outcomes with deterministic ordering, rather than independently calculating them.

Production Outputs, Placement, and Lifecycle

Reporting is enabled by default. The ordinary bundle is:

• errors.yml: structured error-bearing units, their underlying failures, and applicable run-level failures.
• recoveries.yml: structured recovered operations and pertinent counts.
• report.md: a concise human-facing summary of outcome, corrections, processing volume, errors, recoveries, and their locations in the pipeline.

Write all three documents, including empty event lists for uneventful completed runs. Use Audita-owned schemas with safe explanations and stable reason codes; retain run, stage-instance, chunk, operation, and validator/batch identities where applicable. The implementation plan specifies the versioned schemas; neither old report formats nor raw library structures define the new output contract.

Place reports beside the output transcript. Retain transcript-to-stdout support with an explicit report-directory selection, defaulting to the invocation working directory. Preserve the existing overwrite convention: a new run can replace files at its selected destinations. Do not add a mandatory fresh-directory restriction or compatibility exports.

Before model work, create initial versions of all three reports carrying a common run identifier and explicit in_progress status (rendered as prose in Markdown). Fail immediately if initialization fails. Also check the transcript destination; creating neighboring reports does not prove that an existing transcript file is writable. Preflight must not prematurely destroy an existing transcript.

Finalize all required outputs and check every final write. Initial write success cannot guarantee later success if space or permissions change. Failure to write the transcript or finalize a required report returns nonzero and must not erase an already produced transcript. Where possible, finalize available reports with an explicit failed status on fatal termination. Abrupt termination may leave in_progress; elaborate crash-recovery machinery is not required.

Provide an option to disable reporting entirely. It suppresses all three files (errors.yml, recoveries.yml, and report.md), including initialization and final writes, while retaining the brief CLI summary and independently selectable debug output. Disabled reporting must not create or require a writable report destination, and must not remove reports from earlier runs. Transcript output failures remain fatal regardless of reporting selection.

Ordinary reports survive independently of debug cleanup. Reporting configuration and finalization must not depend on retaining a debug run directory.

Debug and Internal Boundaries

Production issue records are compact. Full responses, transcript excerpts, correction lists, and complete validation decisions are opt-in debug evidence. Use a single ConfigKit-backed debug boolean with a --debug CLI override. Debug artifacts may use JSON where useful. Preserve configured-secret redaction in every presentation.

Debug-off prevents detailed artifact persistence; writing and later deleting those files is insufficient. In-memory domain state needed to process candidates correctly is not debug output. Debug and ordinary reporting are independent selections and do not alter processing behavior.

There is no obligation to retain --report-json, old artifact layouts, parallel exports, or old retention settings. Reuse code only where useful to the new workflow. Enabled debug output is retained on success or failure; disabled debug creates no debug directory. This supersedes the retention question in the runtime follow-up. The legacy never behavior is not the meaning of the new reporting-disable option. Optional debug-write failures produce a safe stderr diagnostic without changing correction decisions or the run exit status; ordinary report and transcript-write failures retain their fatal semantics.

Deterministic Human-Facing Presentation

Use the CLI style:

Completed: 18 corrections applied; 3 work units with errors; 7 recoveries.

Do not say that all error-bearing units were skipped: some apply valid corrections. Include report locations when reporting is enabled and the files exist. Send the summary to stderr so stdout remains exclusively available for transcript output. Disabled reporting does not suppress the summary or fatal-error messages.

Keep report.md concise and high-level. Use separate error and recovery matrices with stages as columns and chunks as rows. Include every chunk in fixed order; retain configured stage order and distinct labels for repeated module instances. No automatic width detection, thresholds, adaptive omission, or table splitting. A wide or long table is acceptable.

The error matrix counts error-bearing work units: each attempted cell is 0 or 1. Recovery cells count recovered operations and can exceed 1. An error cell may coexist with recoveries and successfully applied corrections in the same unit. Distinguish completed work with zero events from unattempted work; do not portray canceled/unavailable processing as clean completion. Include totals, processing volume, and a short reason breakdown. Detailed inspection belongs in YAML or debug.

Target Acceptance Criteria

• Total LLM failure still returns success and the encoded normalized transcript, provided required output writes succeed.
• Mixed-success runs preserve independently validated, applicable corrections within the same stage/chunk as a failed batch. No candidate bypasses a required validator; later validators handle survivors by stable proposal identity.
• Subsequent stages and unaffected chunks execute after recoverable failures. Fatal errors and explicit cancellation stop and join active work appropriately.
• Chunk boundaries and identity remain fixed, with operator-selected sizing and unchanged defaults; final recombination preserves ordering.
• Valid no-change/editorial-rejection outcomes are successful. Repaired/filtered output and unresolved failures receive distinct classifications and counts.
• YAML, Markdown, and CLI agree on partial outcomes and totals; error matrices count affected work units, recovery matrices count recovered operations.
• With reporting enabled, all three reports are initialized before model work, finalized and retained, including empty lists. Initial or final required-output write failures produce nonzero exits without destroying completed transcript output.
• Reporting can be disabled without requiring its destination. Debug can be enabled independently; neither selection changes processing outcomes. Debug-off writes no detailed debug artifacts, and stdout transcript output remains supported.
• Reports use the fixed deterministic layout without adaptive presentation logic.
• Meaningful controlled-provider tests cover exhausted recovery, failed validation batches with successful survivors, total failure, fatal cancellation, output failures, and reporting/debug selections. Preserve domain safeguards, capacity limits, and redaction.
• Canonical CLI, configuration, operations, integration, internal, and architecture documentation describes the implemented final behavior.

Implementation Reference

The product decisions are settled. The historical implementation plan defines the selected settings and precedence, YAML schemas, counts and ordering, interrupted work presentation, destination checks, and output finalization. It also owns the ordered implementation stages and validation work. This document remains the reference for the desired behavior and its acceptance criteria.