audita/docs/roadmap/resilience.md

Resilient Processing and Focused Run Reports

Last updated: 2026-09-24

Status: Feature direction agreed; not implemented. This document records the intended end state and remaining product decisions. It is not a staged implementation plan. Recommendations under Open Questions are not adopted policy.

Purpose and User Intent

Audita should reliably produce a useful transcript even when its LLM services cannot improve it. Exhausted retries or unusable model responses must not turn an otherwise processable transcript into a failed production run.

The production guarantee is:

Failure of every LLM operation across every stage and chunk still produces a successful output containing the deterministically normalized transcript, with no LLM corrections applied. Partial LLM success preserves corrections that completed successfully, subject to the selected work-unit commit boundary.

This does not promise the original input byte-for-byte. Deterministic normalization remains intentional and may change segment layout, identifiers, and text before LLM work begins. Output still uses the selected transcript encoder. There is no requirement to map corrections back to the original, pre-normalization segmentation.

Existing reports contain useful information. Preserve meaningful identities, reason codes, correction/decision evidence, and execution measurements where appropriate; reorganize their classification and audience rather than replace everything with an unrelated reporting system.

Current Baseline and Intended Change

The pipeline, LLM integration, report assembly, and artifact lifecycle describe current behavior.

Currently, exhausted structural repairs can become warnings and skipped proposals or rejected validation batches. Other model runtime failures cancel peer work, stop subsequent modules, and prevent final transcript output. Successful repairs appear in execution observations rather than a dedicated recovery classification. Detailed artifacts are written during ordinary runs and may subsequently be removed by retention. The working transcript is re-chunked between modules.

This feature deliberately changes those behaviors. It builds on the completed library adoption and review remediation; it does not undo their separation of library and application responsibilities. The existing fatal-cancellation mechanism remains useful, but model work failures must be separated from failures that actually require cancellation of the run.

Fixed Chunks and Ordered Stages

Normalize first, then partition the transcript once. Maintain the same chunk boundaries, segment membership, segment identity, and ordering throughout LLM processing. Text inside a chunk evolves as accepted corrections are applied. Recombine the chunks in their original order after the LLM stages finish.

For this feature, a stage is a configured module instance. Repeated instances of the same module have distinct identities. Proposal generation and each validator are operations within a stage; validation may use multiple batches within a chunk. A work unit is one stage instance applied to one chunk.

Preserve the overall stage sequence: finish a stage across its chunks before advancing to the next stage. Concurrent chunk work within a stage continues to share PromptKit's backend capacity. A skipped work unit does not prevent other chunks or later stages from executing.

Fixed chunking makes failure isolation and context stable, as well as making cross-stage reporting meaningful. It trades away adaptive repartitioning after edits and task-specific chunk sizes. Initial sizing should allow reasonable room for prompt overhead and text growth, without claiming that every later request will fit. An oversized later model request is an explicit skipped-work outcome; do not silently change chunk boundaries. Exact sizing policy remains open.

Whole-pipeline traversal by independently advancing chunks is explicitly deferred to a future roadmap. Fixed chunking does not authorize changing the stage barrier. Read-only neighboring context, cross-chunk edits, and transcript-wide restructuring are also outside this unit; fixed chunks are not a permanent product prohibition on such future capabilities.

Resilient Execution Contract

• Keep bounded transport retries, structural repairs, operation deadlines, and shared backend capacity. Completion comes from safe fallback, not unlimited attempts or an Audita retry loop around PromptKit's existing recovery.
• Generate proposals, enforce deterministic/domain checks, and complete required LLM validation before applying a correction. Failure must never bypass validation or cause uncertain corrections to be accepted.
• Unsuccessful model work preserves the affected work unit's incoming text and records an error. Preserve corrections committed by earlier stages and other successful work units. The handling of partially successful batches within a failing work unit is an explicit open question below.
• Continue despite model-operation failures, including exhausted repairs/retries, unusable model decisions, provider/transport failures, deadlines, and admission failures. An operation can fail without consuming its entire attempt budget; budget exhaustion is not the sole definition of an error.
• A valid empty proposal list or valid assessments rejecting every correction is a successful outcome. Correction count does not determine success or failure.
• Retain fatal treatment for invalid input/configuration, internal defects, explicit run cancellation, and inability to produce/write the transcript. Distinguish caller cancellation from a local model-operation timeout.
• A completed run with skipped LLM work returns success and makes that degradation visible in its summary and error records. Fatal run failures remain nonzero.

PromptKit continues to own model transport, transient retries, structural repair, and admission. Audita owns work-unit outcomes, domain acceptance, fallback, and report classification. ConfigKit continues to own configuration resolution. Debugging must not change correction decisions, retry budgets, or continuation.

Errors, Recoveries, and Debug

Classify by eventual outcome and impact, not exception type or retry count alone. These are three reporting categories; debug is evidence, not a failure severity. Whether execution can continue is separate from the category.

Category	Meaning	Representative cases
Errors	Intended work could not complete reliably and was skipped, or a fatal run-level operation failed.	Exhausted output repair; unavailable provider; unusable validation decisions; operation deadline.
Recoveries	A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied.	Invalid output repaired; transient provider failure recovered; defective proposals safely discarded.
Debug	Detailed evidence of processing and ordinary decisions.	Proposals, applied corrections, validator decisions, prompt/response content, individual attempts, detailed execution traces.

Normal editorial disagreement belongs in debug: a validator correctly rejecting a meaning-changing edit is expected behavior. A proposal referring to nonexistent original text is defective; safely filtering it is a recovery. A schema-valid validation response with missing or duplicate decision indices is an operation failure when it cannot be recovered. Do not label unresolved output as recovered.

Record the eventual outcome of an operation once. Several unsuccessful attempts followed by success yield a consolidated recovery. Attempts ending in failure contribute to the error, not additional recoveries. Earlier operations that did recover remain recorded if a later operation fails the work unit.

Separate output-quality recovery, defective-proposal filtering, and service recovery using structured reasons. Network instability should not be mistaken for a prompt deficiency. Preserve attempt counts and affected-proposal counts without counting each attempt or rejected proposal as an independent incident.

Recovery frequency needs processing-volume context. Seven recoveries need not warrant attention; seventy may or may not, depending on the amount and kind of work. Do not introduce an arbitrary warning threshold or restore “warnings” as a fourth category. Final counting/denominator details remain open below.

Production Outputs and Debug Boundary

The selected ordinary report set is:

• errors.yml: structured records of unsuccessful work and applicable run failures.
• recoveries.yml: structured records of successfully handled problems.
• report.md: a human-facing summary of outcome, processing volume, applied corrections, errors, recoveries, and their locations in the pipeline.

Use Audita-owned structured records as the common source for YAML, Markdown, and CLI summaries. Reuse useful existing report concepts rather than expose raw library structures or maintain independently calculated totals. Exact schemas are not finalized. They should identify the run, stage instance, stable chunk, operation, validator/batch where relevant, reason, outcome, and pertinent counts. Keep safe human-readable explanations alongside machine-readable reason codes.

Production issue records should be compact. Full responses, transcript excerpts, proposed/applied correction lists, and complete validation decisions belong in opt-in debug artifacts. Debug output may continue using JSON where suitable. Preserve configured-secret redaction across every presentation and artifact.

Provide an explicit debug opt-in, with --debug as the proposed CLI spelling. A configuration counterpart is desirable but its exact interface remains open. Debug-off must prevent detailed artifact persistence in the first place; writing those files and deleting them later does not satisfy the target. Temporary in-memory processing data needed for correct execution is not debug output.

Ordinary reports must survive independently of debug retention. The current --report-json, run-local report, ledger, execution/utilization artifacts, and retention settings need a deliberate disposition. Do not silently retain all existing verbose output as additional production defaults. Coordinate the retention decision with the existing runtime follow-up.

Human-Facing Presentation

Provide a concise completion message showing corrections applied, work units skipped due to errors, and recoveries. For example, the intended style is:

Completed: 18 corrections applied; 3 work units skipped due to errors; 7 recoveries.

Include report locations when available. Send the summary to stderr so stdout remains available exclusively for transcript output. Successful completion must not imply that every stage/chunk succeeded.

report.md should help the reader locate concentrations of failures and recoveries without opening debug data. Stable chunks support the proposed presentation of separate error and recovery matrices: chunks as rows, configured stage instances as columns, and counts in each cell. Preserve execution order and distinguish repeated stages. Include totals and a short breakdown by reason.

Distinguish completed work with zero events from work that was not attempted. Do not present canceled or unavailable work as cleanly completed. Exact table counting, omission of uneventful rows, and large-report layout remain open below.

Target Acceptance Criteria

• A controlled run in which every LLM operation fails still returns success and the normally encoded normalized transcript, with no LLM corrections applied.
• Mixed-success runs preserve committed corrections, skip unsuccessful work safely, and attempt subsequent stages and unaffected chunks.
• Chunks retain their identity and boundaries across all stages; validation batching does not repartition them. Final recombination preserves ordering.
• Recoverable model failures do not cancel unrelated work. Fatal failures and explicit cancellation still stop and join active work appropriately.
• Valid no-change and ordinary editorial-rejection outcomes remain successful. Repaired/filtered output and unrecovered failures receive distinct classifications.
• YAML records, Markdown tables, and CLI totals agree, with deterministic ordering and no duplicate incident counts from retries or repairs.
• Debug-off produces the selected ordinary output set without detailed debugging artifacts; debug-on adds evidence without changing processing outcomes.
• Tests exercise real runner/runtime/reporting paths with controlled providers, including exhausted recovery, partial success, and total LLM failure. Existing domain safeguards, capacity limits, redaction, and output contracts remain covered where they are retained.
• On implementation, update the canonical CLI, configuration, operations, integration, internal, and architecture documents to describe the new behavior. This roadmap alone does not change their current contracts.

Open Questions

Work-unit atomicity when only some validation batches succeed

Recommendation: Commit a stage/chunk unit only after all required operations complete reliably. If a required batch fails, discard that unit's pending edits, record one work-unit error with the underlying operation failures, and preserve its incoming text. Normal valid rejections are not operation failures. This gives predictable fallback and straightforward error counting.

Alternative: Preserve fully validated corrections from successful batches in an otherwise failing unit. This retains more useful work but requires explicit partial-success outcomes and clearer counts than “work unit skipped.”

Domain-invalid responses and recovery opportunities

Recommendation: Continue safely filtering independently defective proposals; do not retry ordinary editorial rejections. Treat an unusable validator decision set as an error when existing permitted recovery cannot resolve it. Decide explicitly whether dynamic decision-index/cardinality checks should become part of the repair contract; this roadmap does not silently add semantic retry loops.

Alternative: Extend recovery to regenerate domain-invalid decision sets with bounded feedback. This may improve completion quality, but its ownership, feedback contract, and cost budget need a separate concrete design.

Initial chunk sizing and later prompt growth

Recommendation: Retain operator control over chunk size and choose reasonable headroom for known stage prompts. Keep validation batching within fixed chunks; report a later operation that cannot fit as skipped work. Establish the concrete sizing rule during detailed design rather than invent a universal safety margin.

Alternative: Size initial chunks against conservative estimates for all selected stage/profile combinations. This reduces later size failures but may increase call volume and still cannot guarantee fit after arbitrary text growth.

Exact event counts and recovery-rate denominators

Recommendation: Count errors once per failed stage/chunk unit, linking multiple underlying failures as details. Count recoveries once per recovered operation; aggregate defective-proposal filtering per proposal response with an affected count. Report work-unit coverage separately from logical-call recovery counts, with explicit denominators. Earlier recoveries may coexist with a later error in the same unit.

Alternative: Count affected work units for both matrices, placing operation counts only in details. This makes the two tables directly comparable but hides multiple distinct recoveries within a single unit unless the reader drills down.

Output placement, empty files, and report-write failures

Recommendation: Produce all three ordinary documents for every completed run, using empty event lists where appropriate. Select a durable report directory independent of debug cleanup, including when transcript output goes to stdout. Specify its CLI/default path and overwrite policy before implementation. Attempt partial reports on fatal failures when possible. Report-writing failures should be explicit and should not erase an already produced transcript; whether they make the invocation nonzero must be settled as part of the output contract.

Alternative: Write event files only when nonempty and identify their absence explicitly in report.md. This reduces empty files but requires consumers to understand missing-file semantics. Best-effort ancillary reporting could preserve a zero exit status after transcript success, at the cost of weaker evidence that the promised output bundle is complete.

Existing report interfaces and debug configuration

Recommendation: Keep useful internal structures and measurements, adapting the ordinary schema to its audience and moving verbose evidence behind debug. Finalize a versioned YAML envelope, identifiers/index display, and the disposition of --report-json and retention settings in detailed design. Use one debug boolean with a CLI override and optional ConfigKit-backed setting; avoid a matrix of independent artifact switches.

Alternative: Retain explicit machine-oriented JSON export alongside the new ordinary outputs for consumers that need it. This preserves an existing interface but adds another maintained presentation. Compatibility/export requirements have not been selected for this feature; do not infer them from prior migration policy.

Large Markdown reports

Recommendation: Use separate matrices with totals and reason summaries; omit uneventful chunk rows when that improves readability, explicitly reporting the omitted count. Show zero separately from unattempted work. Keep full incident details in YAML and identify chunks by stable labels and segment ranges.

Alternative: Always include every chunk row, splitting wide tables into stage groups. This gives an exhaustive visual grid but can make long transcripts harder to review. Exact layout and any row-selection threshold remain undecided.