diff --git a/docs/roadmap/resilience.md b/docs/roadmap/resilience.md new file mode 100644 index 0000000..d45bfba --- /dev/null +++ b/docs/roadmap/resilience.md @@ -0,0 +1,311 @@ +# Resilient Processing and Focused Run Reports + +**Last updated:** 2026-09-24 + +**Status:** Feature direction agreed; not implemented. This document records the +intended end state and remaining product decisions. It is not a staged +implementation plan. Recommendations under Open Questions are not adopted policy. + +## Purpose and User Intent + +Audita should reliably produce a useful transcript even when its LLM services +cannot improve it. Exhausted retries or unusable model responses must not turn +an otherwise processable transcript into a failed production run. + +The production guarantee is: + +> Failure of every LLM operation across every stage and chunk still produces a +> successful output containing the deterministically normalized transcript, with +> no LLM corrections applied. Partial LLM success preserves corrections that +> completed successfully, subject to the selected work-unit commit boundary. + +This does not promise the original input byte-for-byte. Deterministic +normalization remains intentional and may change segment layout, identifiers, +and text before LLM work begins. Output still uses the selected transcript +encoder. There is no requirement to map corrections back to the original, +pre-normalization segmentation. + +Existing reports contain useful information. Preserve meaningful identities, +reason codes, correction/decision evidence, and execution measurements where +appropriate; reorganize their classification and audience rather than replace +everything with an unrelated reporting system. + +## Current Baseline and Intended Change + +The [pipeline](../internal/pipeline.md), [LLM integration](../integrations/openai-compatible-llm.md), +[report assembly](../internal/diagnostics-reporting.md), and +[artifact lifecycle](../operations.md) describe current behavior. + +Currently, exhausted structural repairs can become warnings and skipped proposals +or rejected validation batches. Other model runtime failures cancel peer work, +stop subsequent modules, and prevent final transcript output. Successful repairs +appear in execution observations rather than a dedicated recovery classification. +Detailed artifacts are written during ordinary runs and may subsequently be +removed by retention. The working transcript is re-chunked between modules. + +This feature deliberately changes those behaviors. It builds on the completed +[library adoption](library-adoption.md) and [review remediation](remediation.md); +it does not undo their separation of library and application responsibilities. +The existing fatal-cancellation mechanism remains useful, but model work failures +must be separated from failures that actually require cancellation of the run. + +## Fixed Chunks and Ordered Stages + +Normalize first, then partition the transcript once. Maintain the same chunk +boundaries, segment membership, segment identity, and ordering throughout LLM +processing. Text inside a chunk evolves as accepted corrections are applied. +Recombine the chunks in their original order after the LLM stages finish. + +For this feature, a stage is a configured module instance. Repeated instances of +the same module have distinct identities. Proposal generation and each validator +are operations within a stage; validation may use multiple batches within a chunk. +A work unit is one stage instance applied to one chunk. + +Preserve the overall stage sequence: finish a stage across its chunks before +advancing to the next stage. Concurrent chunk work within a stage continues to +share PromptKit's backend capacity. A skipped work unit does not prevent other +chunks or later stages from executing. + +Fixed chunking makes failure isolation and context stable, as well as making +cross-stage reporting meaningful. It trades away adaptive repartitioning after +edits and task-specific chunk sizes. Initial sizing should allow reasonable room +for prompt overhead and text growth, without claiming that every later request +will fit. An oversized later model request is an explicit skipped-work outcome; +do not silently change chunk boundaries. Exact sizing policy remains open. + +Whole-pipeline traversal by independently advancing chunks is explicitly deferred +to a future roadmap. Fixed chunking does not authorize changing the stage barrier. +Read-only neighboring context, cross-chunk edits, and transcript-wide restructuring +are also outside this unit; fixed chunks are not a permanent product prohibition +on such future capabilities. + +## Resilient Execution Contract + +- Keep bounded transport retries, structural repairs, operation deadlines, and + shared backend capacity. Completion comes from safe fallback, not unlimited + attempts or an Audita retry loop around PromptKit's existing recovery. +- Generate proposals, enforce deterministic/domain checks, and complete required + LLM validation before applying a correction. Failure must never bypass validation + or cause uncertain corrections to be accepted. +- Unsuccessful model work preserves the affected work unit's incoming text and + records an error. Preserve corrections committed by earlier stages and other + successful work units. The handling of partially successful batches within a + failing work unit is an explicit open question below. +- Continue despite model-operation failures, including exhausted repairs/retries, + unusable model decisions, provider/transport failures, deadlines, and admission + failures. An operation can fail without consuming its entire attempt budget; + budget exhaustion is not the sole definition of an error. +- A valid empty proposal list or valid assessments rejecting every correction is + a successful outcome. Correction count does not determine success or failure. +- Retain fatal treatment for invalid input/configuration, internal defects, + explicit run cancellation, and inability to produce/write the transcript. + Distinguish caller cancellation from a local model-operation timeout. +- A completed run with skipped LLM work returns success and makes that degradation + visible in its summary and error records. Fatal run failures remain nonzero. + +PromptKit continues to own model transport, transient retries, structural repair, +and admission. Audita owns work-unit outcomes, domain acceptance, fallback, and +report classification. ConfigKit continues to own configuration resolution. +Debugging must not change correction decisions, retry budgets, or continuation. + +## Errors, Recoveries, and Debug + +Classify by eventual outcome and impact, not exception type or retry count alone. +These are three reporting categories; debug is evidence, not a failure severity. +Whether execution can continue is separate from the category. + +| Category | Meaning | Representative cases | +| --- | --- | --- | +| Errors | Intended work could not complete reliably and was skipped, or a fatal run-level operation failed. | Exhausted output repair; unavailable provider; unusable validation decisions; operation deadline. | +| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient provider failure recovered; defective proposals safely discarded. | +| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompt/response content, individual attempts, detailed execution traces. | + +Normal editorial disagreement belongs in debug: a validator correctly rejecting +a meaning-changing edit is expected behavior. A proposal referring to nonexistent +original text is defective; safely filtering it is a recovery. A schema-valid +validation response with missing or duplicate decision indices is an operation +failure when it cannot be recovered. Do not label unresolved output as recovered. + +Record the eventual outcome of an operation once. Several unsuccessful attempts +followed by success yield a consolidated recovery. Attempts ending in failure +contribute to the error, not additional recoveries. Earlier operations that did +recover remain recorded if a later operation fails the work unit. + +Separate output-quality recovery, defective-proposal filtering, and service +recovery using structured reasons. Network instability should not be mistaken +for a prompt deficiency. Preserve attempt counts and affected-proposal counts +without counting each attempt or rejected proposal as an independent incident. + +Recovery frequency needs processing-volume context. Seven recoveries need not +warrant attention; seventy may or may not, depending on the amount and kind of +work. Do not introduce an arbitrary warning threshold or restore “warnings” as a +fourth category. Final counting/denominator details remain open below. + +## Production Outputs and Debug Boundary + +The selected ordinary report set is: + +- `errors.yml`: structured records of unsuccessful work and applicable run failures. +- `recoveries.yml`: structured records of successfully handled problems. +- `report.md`: a human-facing summary of outcome, processing volume, applied + corrections, errors, recoveries, and their locations in the pipeline. + +Use Audita-owned structured records as the common source for YAML, Markdown, and +CLI summaries. Reuse useful existing report concepts rather than expose raw +library structures or maintain independently calculated totals. Exact schemas +are not finalized. They should identify the run, stage instance, stable chunk, +operation, validator/batch where relevant, reason, outcome, and pertinent counts. +Keep safe human-readable explanations alongside machine-readable reason codes. + +Production issue records should be compact. Full responses, transcript excerpts, +proposed/applied correction lists, and complete validation decisions belong in +opt-in debug artifacts. Debug output may continue using JSON where suitable. +Preserve configured-secret redaction across every presentation and artifact. + +Provide an explicit debug opt-in, with `--debug` as the proposed CLI spelling. +A configuration counterpart is desirable but its exact interface remains open. +Debug-off must prevent detailed artifact persistence in the first place; writing +those files and deleting them later does not satisfy the target. Temporary +in-memory processing data needed for correct execution is not debug output. + +Ordinary reports must survive independently of debug retention. The current +`--report-json`, run-local report, ledger, execution/utilization artifacts, and +retention settings need a deliberate disposition. Do not silently retain all +existing verbose output as additional production defaults. Coordinate the +retention decision with the existing [runtime follow-up](runtime-behavior.md). + +## Human-Facing Presentation + +Provide a concise completion message showing corrections applied, work units +skipped due to errors, and recoveries. For example, the intended style is: + +> Completed: 18 corrections applied; 3 work units skipped due to errors; 7 recoveries. + +Include report locations when available. Send the summary to stderr so stdout +remains available exclusively for transcript output. Successful completion must +not imply that every stage/chunk succeeded. + +`report.md` should help the reader locate concentrations of failures and recoveries +without opening debug data. Stable chunks support the proposed presentation of +separate error and recovery matrices: chunks as rows, configured stage instances +as columns, and counts in each cell. Preserve execution order and distinguish +repeated stages. Include totals and a short breakdown by reason. + +Distinguish completed work with zero events from work that was not attempted. +Do not present canceled or unavailable work as cleanly completed. Exact table +counting, omission of uneventful rows, and large-report layout remain open below. + +## Target Acceptance Criteria + +- A controlled run in which every LLM operation fails still returns success and + the normally encoded normalized transcript, with no LLM corrections applied. +- Mixed-success runs preserve committed corrections, skip unsuccessful work + safely, and attempt subsequent stages and unaffected chunks. +- Chunks retain their identity and boundaries across all stages; validation + batching does not repartition them. Final recombination preserves ordering. +- Recoverable model failures do not cancel unrelated work. Fatal failures and + explicit cancellation still stop and join active work appropriately. +- Valid no-change and ordinary editorial-rejection outcomes remain successful. + Repaired/filtered output and unrecovered failures receive distinct classifications. +- YAML records, Markdown tables, and CLI totals agree, with deterministic ordering + and no duplicate incident counts from retries or repairs. +- Debug-off produces the selected ordinary output set without detailed debugging + artifacts; debug-on adds evidence without changing processing outcomes. +- Tests exercise real runner/runtime/reporting paths with controlled providers, + including exhausted recovery, partial success, and total LLM failure. Existing + domain safeguards, capacity limits, redaction, and output contracts remain + covered where they are retained. +- On implementation, update the canonical CLI, configuration, operations, + integration, internal, and architecture documents to describe the new behavior. + This roadmap alone does not change their current contracts. + +## Open Questions + +### Work-unit atomicity when only some validation batches succeed + +**Recommendation:** Commit a stage/chunk unit only after all required operations +complete reliably. If a required batch fails, discard that unit's pending edits, +record one work-unit error with the underlying operation failures, and preserve +its incoming text. Normal valid rejections are not operation failures. This gives +predictable fallback and straightforward error counting. + +**Alternative:** Preserve fully validated corrections from successful batches in +an otherwise failing unit. This retains more useful work but requires explicit +partial-success outcomes and clearer counts than “work unit skipped.” + +### Domain-invalid responses and recovery opportunities + +**Recommendation:** Continue safely filtering independently defective proposals; +do not retry ordinary editorial rejections. Treat an unusable validator decision +set as an error when existing permitted recovery cannot resolve it. Decide +explicitly whether dynamic decision-index/cardinality checks should become part +of the repair contract; this roadmap does not silently add semantic retry loops. + +**Alternative:** Extend recovery to regenerate domain-invalid decision sets with +bounded feedback. This may improve completion quality, but its ownership, +feedback contract, and cost budget need a separate concrete design. + +### Initial chunk sizing and later prompt growth + +**Recommendation:** Retain operator control over chunk size and choose reasonable +headroom for known stage prompts. Keep validation batching within fixed chunks; +report a later operation that cannot fit as skipped work. Establish the concrete +sizing rule during detailed design rather than invent a universal safety margin. + +**Alternative:** Size initial chunks against conservative estimates for all +selected stage/profile combinations. This reduces later size failures but may +increase call volume and still cannot guarantee fit after arbitrary text growth. + +### Exact event counts and recovery-rate denominators + +**Recommendation:** Count errors once per failed stage/chunk unit, linking multiple +underlying failures as details. Count recoveries once per recovered operation; +aggregate defective-proposal filtering per proposal response with an affected +count. Report work-unit coverage separately from logical-call recovery counts, +with explicit denominators. Earlier recoveries may coexist with a later error in +the same unit. + +**Alternative:** Count affected work units for both matrices, placing operation +counts only in details. This makes the two tables directly comparable but hides +multiple distinct recoveries within a single unit unless the reader drills down. + +### Output placement, empty files, and report-write failures + +**Recommendation:** Produce all three ordinary documents for every completed run, +using empty event lists where appropriate. Select a durable report directory +independent of debug cleanup, including when transcript output goes to stdout. +Specify its CLI/default path and overwrite policy before implementation. Attempt +partial reports on fatal failures when possible. Report-writing failures should +be explicit and should not erase an already produced transcript; whether they +make the invocation nonzero must be settled as part of the output contract. + +**Alternative:** Write event files only when nonempty and identify their absence +explicitly in `report.md`. This reduces empty files but requires consumers to +understand missing-file semantics. Best-effort ancillary reporting could preserve +a zero exit status after transcript success, at the cost of weaker evidence that +the promised output bundle is complete. + +### Existing report interfaces and debug configuration + +**Recommendation:** Keep useful internal structures and measurements, adapting the +ordinary schema to its audience and moving verbose evidence behind debug. +Finalize a versioned YAML envelope, identifiers/index display, and the disposition +of `--report-json` and retention settings in detailed design. Use one debug boolean +with a CLI override and optional ConfigKit-backed setting; avoid a matrix of +independent artifact switches. + +**Alternative:** Retain explicit machine-oriented JSON export alongside the new +ordinary outputs for consumers that need it. This preserves an existing interface +but adds another maintained presentation. Compatibility/export requirements have +not been selected for this feature; do not infer them from prior migration policy. + +### Large Markdown reports + +**Recommendation:** Use separate matrices with totals and reason summaries; omit +uneventful chunk rows when that improves readability, explicitly reporting the +omitted count. Show zero separately from unattempted work. Keep full incident +details in YAML and identify chunks by stable labels and segment ranges. + +**Alternative:** Always include every chunk row, splitting wide tables into stage +groups. This gives an exhaustive visual grid but can make long transcripts harder +to review. Exact layout and any row-selection threshold remain undecided.