audita/docs/roadmap/resilience.md

# Resilient Processing and Focused Run Reports

**Last updated:** 2026-09-24

**Status:** Feature direction agreed; not implemented. This document records the
intended end state and remaining product decisions. It is not a staged
implementation plan. Recommendations under Open Questions are not adopted policy.

## Purpose and User Intent

Audita should reliably produce a useful transcript even when its LLM services
cannot improve it. Exhausted retries or unusable model responses must not turn
an otherwise processable transcript into a failed production run.

The production guarantee is:

> Failure of every LLM operation across every stage and chunk still produces a
> successful output containing the deterministically normalized transcript, with
> no LLM corrections applied. Partial LLM success preserves corrections that
> completed successfully, subject to the selected work-unit commit boundary.

This does not promise the original input byte-for-byte. Deterministic
normalization remains intentional and may change segment layout, identifiers,
and text before LLM work begins. Output still uses the selected transcript
encoder. There is no requirement to map corrections back to the original,
pre-normalization segmentation.

Existing reports contain useful information. Preserve meaningful identities,
reason codes, correction/decision evidence, and execution measurements where
appropriate; reorganize their classification and audience rather than replace
everything with an unrelated reporting system.

## Current Baseline and Intended Change

The [pipeline](../internal/pipeline.md), [LLM integration](../integrations/openai-compatible-llm.md),
[report assembly](../internal/diagnostics-reporting.md), and
[artifact lifecycle](../operations.md) describe current behavior.

Currently, exhausted structural repairs can become warnings and skipped proposals
or rejected validation batches. Other model runtime failures cancel peer work,
stop subsequent modules, and prevent final transcript output. Successful repairs
appear in execution observations rather than a dedicated recovery classification.
Detailed artifacts are written during ordinary runs and may subsequently be
removed by retention. The working transcript is re-chunked between modules.

This feature deliberately changes those behaviors. It builds on the completed
[library adoption](library-adoption.md) and [review remediation](remediation.md);
it does not undo their separation of library and application responsibilities.
The existing fatal-cancellation mechanism remains useful, but model work failures
must be separated from failures that actually require cancellation of the run.

## Fixed Chunks and Ordered Stages

Normalize first, then partition the transcript once. Maintain the same chunk
boundaries, segment membership, segment identity, and ordering throughout LLM
processing. Text inside a chunk evolves as accepted corrections are applied.
Recombine the chunks in their original order after the LLM stages finish.

For this feature, a stage is a configured module instance. Repeated instances of
the same module have distinct identities. Proposal generation and each validator
are operations within a stage; validation may use multiple batches within a chunk.
A work unit is one stage instance applied to one chunk.

Preserve the overall stage sequence: finish a stage across its chunks before
advancing to the next stage. Concurrent chunk work within a stage continues to
share PromptKit's backend capacity. A skipped work unit does not prevent other
chunks or later stages from executing.

Fixed chunking makes failure isolation and context stable, as well as making
cross-stage reporting meaningful. It trades away adaptive repartitioning after
edits and task-specific chunk sizes. Initial sizing should allow reasonable room
for prompt overhead and text growth, without claiming that every later request
will fit. An oversized later model request is an explicit skipped-work outcome;
do not silently change chunk boundaries. Exact sizing policy remains open.

Whole-pipeline traversal by independently advancing chunks is explicitly deferred
to a future roadmap. Fixed chunking does not authorize changing the stage barrier.
Read-only neighboring context, cross-chunk edits, and transcript-wide restructuring
are also outside this unit; fixed chunks are not a permanent product prohibition
on such future capabilities.

## Resilient Execution Contract

- Keep bounded transport retries, structural repairs, operation deadlines, and
  shared backend capacity. Completion comes from safe fallback, not unlimited
  attempts or an Audita retry loop around PromptKit's existing recovery.
- Generate proposals, enforce deterministic/domain checks, and complete required
  LLM validation before applying a correction. Failure must never bypass validation
  or cause uncertain corrections to be accepted.
- Unsuccessful model work preserves the affected work unit's incoming text and
  records an error. Preserve corrections committed by earlier stages and other
  successful work units. The handling of partially successful batches within a
  failing work unit is an explicit open question below.
- Continue despite model-operation failures, including exhausted repairs/retries,
  unusable model decisions, provider/transport failures, deadlines, and admission
  failures. An operation can fail without consuming its entire attempt budget;
  budget exhaustion is not the sole definition of an error.
- A valid empty proposal list or valid assessments rejecting every correction is
  a successful outcome. Correction count does not determine success or failure.
- Retain fatal treatment for invalid input/configuration, internal defects,
  explicit run cancellation, and inability to produce/write the transcript.
  Distinguish caller cancellation from a local model-operation timeout.
- A completed run with skipped LLM work returns success and makes that degradation
  visible in its summary and error records. Fatal run failures remain nonzero.

PromptKit continues to own model transport, transient retries, structural repair,
and admission. Audita owns work-unit outcomes, domain acceptance, fallback, and
report classification. ConfigKit continues to own configuration resolution.
Debugging must not change correction decisions, retry budgets, or continuation.

## Errors, Recoveries, and Debug

Classify by eventual outcome and impact, not exception type or retry count alone.
These are three reporting categories; debug is evidence, not a failure severity.
Whether execution can continue is separate from the category.

| Category | Meaning | Representative cases |
| --- | --- | --- |
| Errors | Intended work could not complete reliably and was skipped, or a fatal run-level operation failed. | Exhausted output repair; unavailable provider; unusable validation decisions; operation deadline. |
| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient provider failure recovered; defective proposals safely discarded. |
| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompt/response content, individual attempts, detailed execution traces. |

Normal editorial disagreement belongs in debug: a validator correctly rejecting
a meaning-changing edit is expected behavior. A proposal referring to nonexistent
original text is defective; safely filtering it is a recovery. A schema-valid
validation response with missing or duplicate decision indices is an operation
failure when it cannot be recovered. Do not label unresolved output as recovered.

Record the eventual outcome of an operation once. Several unsuccessful attempts
followed by success yield a consolidated recovery. Attempts ending in failure
contribute to the error, not additional recoveries. Earlier operations that did
recover remain recorded if a later operation fails the work unit.

Separate output-quality recovery, defective-proposal filtering, and service
recovery using structured reasons. Network instability should not be mistaken
for a prompt deficiency. Preserve attempt counts and affected-proposal counts
without counting each attempt or rejected proposal as an independent incident.

Recovery frequency needs processing-volume context. Seven recoveries need not
warrant attention; seventy may or may not, depending on the amount and kind of
work. Do not introduce an arbitrary warning threshold or restore “warnings” as a
fourth category. Final counting/denominator details remain open below.

## Production Outputs and Debug Boundary

The selected ordinary report set is:

- `errors.yml`: structured records of unsuccessful work and applicable run failures.
- `recoveries.yml`: structured records of successfully handled problems.
- `report.md`: a human-facing summary of outcome, processing volume, applied
  corrections, errors, recoveries, and their locations in the pipeline.

Use Audita-owned structured records as the common source for YAML, Markdown, and
CLI summaries. Reuse useful existing report concepts rather than expose raw
library structures or maintain independently calculated totals. Exact schemas
are not finalized. They should identify the run, stage instance, stable chunk,
operation, validator/batch where relevant, reason, outcome, and pertinent counts.
Keep safe human-readable explanations alongside machine-readable reason codes.

Production issue records should be compact. Full responses, transcript excerpts,
proposed/applied correction lists, and complete validation decisions belong in
opt-in debug artifacts. Debug output may continue using JSON where suitable.
Preserve configured-secret redaction across every presentation and artifact.

Provide an explicit debug opt-in, with `--debug` as the proposed CLI spelling.
A configuration counterpart is desirable but its exact interface remains open.
Debug-off must prevent detailed artifact persistence in the first place; writing
those files and deleting them later does not satisfy the target. Temporary
in-memory processing data needed for correct execution is not debug output.

Ordinary reports must survive independently of debug retention. The current
`--report-json`, run-local report, ledger, execution/utilization artifacts, and
retention settings need a deliberate disposition. Do not silently retain all
existing verbose output as additional production defaults. Coordinate the
retention decision with the existing [runtime follow-up](runtime-behavior.md).

## Human-Facing Presentation

Provide a concise completion message showing corrections applied, work units
skipped due to errors, and recoveries. For example, the intended style is:

> Completed: 18 corrections applied; 3 work units skipped due to errors; 7 recoveries.

Include report locations when available. Send the summary to stderr so stdout
remains available exclusively for transcript output. Successful completion must
not imply that every stage/chunk succeeded.

`report.md` should help the reader locate concentrations of failures and recoveries
without opening debug data. Stable chunks support the proposed presentation of
separate error and recovery matrices: chunks as rows, configured stage instances
as columns, and counts in each cell. Preserve execution order and distinguish
repeated stages. Include totals and a short breakdown by reason.

Distinguish completed work with zero events from work that was not attempted.
Do not present canceled or unavailable work as cleanly completed. Exact table
counting, omission of uneventful rows, and large-report layout remain open below.

## Target Acceptance Criteria

- A controlled run in which every LLM operation fails still returns success and
  the normally encoded normalized transcript, with no LLM corrections applied.
- Mixed-success runs preserve committed corrections, skip unsuccessful work
  safely, and attempt subsequent stages and unaffected chunks.
- Chunks retain their identity and boundaries across all stages; validation
  batching does not repartition them. Final recombination preserves ordering.
- Recoverable model failures do not cancel unrelated work. Fatal failures and
  explicit cancellation still stop and join active work appropriately.
- Valid no-change and ordinary editorial-rejection outcomes remain successful.
  Repaired/filtered output and unrecovered failures receive distinct classifications.
- YAML records, Markdown tables, and CLI totals agree, with deterministic ordering
  and no duplicate incident counts from retries or repairs.
- Debug-off produces the selected ordinary output set without detailed debugging
  artifacts; debug-on adds evidence without changing processing outcomes.
- Tests exercise real runner/runtime/reporting paths with controlled providers,
  including exhausted recovery, partial success, and total LLM failure. Existing
  domain safeguards, capacity limits, redaction, and output contracts remain
  covered where they are retained.
- On implementation, update the canonical CLI, configuration, operations,
  integration, internal, and architecture documents to describe the new behavior.
  This roadmap alone does not change their current contracts.

## Open Questions

### Work-unit atomicity when only some validation batches succeed

**Recommendation:** Commit a stage/chunk unit only after all required operations
complete reliably. If a required batch fails, discard that unit's pending edits,
record one work-unit error with the underlying operation failures, and preserve
its incoming text. Normal valid rejections are not operation failures. This gives
predictable fallback and straightforward error counting.

**Alternative:** Preserve fully validated corrections from successful batches in
an otherwise failing unit. This retains more useful work but requires explicit
partial-success outcomes and clearer counts than “work unit skipped.”

### Domain-invalid responses and recovery opportunities

**Recommendation:** Continue safely filtering independently defective proposals;
do not retry ordinary editorial rejections. Treat an unusable validator decision
set as an error when existing permitted recovery cannot resolve it. Decide
explicitly whether dynamic decision-index/cardinality checks should become part
of the repair contract; this roadmap does not silently add semantic retry loops.

**Alternative:** Extend recovery to regenerate domain-invalid decision sets with
bounded feedback. This may improve completion quality, but its ownership,
feedback contract, and cost budget need a separate concrete design.

### Initial chunk sizing and later prompt growth

**Recommendation:** Retain operator control over chunk size and choose reasonable
headroom for known stage prompts. Keep validation batching within fixed chunks;
report a later operation that cannot fit as skipped work. Establish the concrete
sizing rule during detailed design rather than invent a universal safety margin.

**Alternative:** Size initial chunks against conservative estimates for all
selected stage/profile combinations. This reduces later size failures but may
increase call volume and still cannot guarantee fit after arbitrary text growth.

### Exact event counts and recovery-rate denominators

**Recommendation:** Count errors once per failed stage/chunk unit, linking multiple
underlying failures as details. Count recoveries once per recovered operation;
aggregate defective-proposal filtering per proposal response with an affected
count. Report work-unit coverage separately from logical-call recovery counts,
with explicit denominators. Earlier recoveries may coexist with a later error in
the same unit.

**Alternative:** Count affected work units for both matrices, placing operation
counts only in details. This makes the two tables directly comparable but hides
multiple distinct recoveries within a single unit unless the reader drills down.

### Output placement, empty files, and report-write failures

**Recommendation:** Produce all three ordinary documents for every completed run,
using empty event lists where appropriate. Select a durable report directory
independent of debug cleanup, including when transcript output goes to stdout.
Specify its CLI/default path and overwrite policy before implementation. Attempt
partial reports on fatal failures when possible. Report-writing failures should
be explicit and should not erase an already produced transcript; whether they
make the invocation nonzero must be settled as part of the output contract.

**Alternative:** Write event files only when nonempty and identify their absence
explicitly in `report.md`. This reduces empty files but requires consumers to
understand missing-file semantics. Best-effort ancillary reporting could preserve
a zero exit status after transcript success, at the cost of weaker evidence that
the promised output bundle is complete.

### Existing report interfaces and debug configuration

**Recommendation:** Keep useful internal structures and measurements, adapting the
ordinary schema to its audience and moving verbose evidence behind debug.
Finalize a versioned YAML envelope, identifiers/index display, and the disposition
of `--report-json` and retention settings in detailed design. Use one debug boolean
with a CLI override and optional ConfigKit-backed setting; avoid a matrix of
independent artifact switches.

**Alternative:** Retain explicit machine-oriented JSON export alongside the new
ordinary outputs for consumers that need it. This preserves an existing interface
but adds another maintained presentation. Compatibility/export requirements have
not been selected for this feature; do not infer them from prior migration policy.

### Large Markdown reports

**Recommendation:** Use separate matrices with totals and reason summaries; omit
uneventful chunk rows when that improves readability, explicitly reporting the
omitted count. Show zero separately from unattempted work. Keep full incident
details in YAML and identify chunks by stable labels and segment ranges.

**Alternative:** Always include every chunk row, splitting wide tables into stage
groups. This gives an exhaustive visual grid but can make long transcripts harder
to review. Exact layout and any row-selection threshold remain undecided.