Record the resilience and reporting roadmap

docs/roadmap/resilience.md

@@ -0,0 +1,311 @@
+# Resilient Processing and Focused Run Reports
+
+**Last updated:** 2026-09-24
+
+**Status:** Feature direction agreed; not implemented. This document records the
+intended end state and remaining product decisions. It is not a staged
+implementation plan. Recommendations under Open Questions are not adopted policy.
+
+## Purpose and User Intent
+
+Audita should reliably produce a useful transcript even when its LLM services
+cannot improve it. Exhausted retries or unusable model responses must not turn
+an otherwise processable transcript into a failed production run.
+
+The production guarantee is:
+
+> Failure of every LLM operation across every stage and chunk still produces a
+> successful output containing the deterministically normalized transcript, with
+> no LLM corrections applied. Partial LLM success preserves corrections that
+> completed successfully, subject to the selected work-unit commit boundary.
+
+This does not promise the original input byte-for-byte. Deterministic
+normalization remains intentional and may change segment layout, identifiers,
+and text before LLM work begins. Output still uses the selected transcript
+encoder. There is no requirement to map corrections back to the original,
+pre-normalization segmentation.
+
+Existing reports contain useful information. Preserve meaningful identities,
+reason codes, correction/decision evidence, and execution measurements where
+appropriate; reorganize their classification and audience rather than replace
+everything with an unrelated reporting system.
+
+## Current Baseline and Intended Change
+
+The [pipeline](../internal/pipeline.md), [LLM integration](../integrations/openai-compatible-llm.md),
+[report assembly](../internal/diagnostics-reporting.md), and
+[artifact lifecycle](../operations.md) describe current behavior.
+
+Currently, exhausted structural repairs can become warnings and skipped proposals
+or rejected validation batches. Other model runtime failures cancel peer work,
+stop subsequent modules, and prevent final transcript output. Successful repairs
+appear in execution observations rather than a dedicated recovery classification.
+Detailed artifacts are written during ordinary runs and may subsequently be
+removed by retention. The working transcript is re-chunked between modules.
+
+This feature deliberately changes those behaviors. It builds on the completed
+[library adoption](library-adoption.md) and [review remediation](remediation.md);
+it does not undo their separation of library and application responsibilities.
+The existing fatal-cancellation mechanism remains useful, but model work failures
+must be separated from failures that actually require cancellation of the run.
+
+## Fixed Chunks and Ordered Stages
+
+Normalize first, then partition the transcript once. Maintain the same chunk
+boundaries, segment membership, segment identity, and ordering throughout LLM
+processing. Text inside a chunk evolves as accepted corrections are applied.
+Recombine the chunks in their original order after the LLM stages finish.
+
+For this feature, a stage is a configured module instance. Repeated instances of
+the same module have distinct identities. Proposal generation and each validator
+are operations within a stage; validation may use multiple batches within a chunk.
+A work unit is one stage instance applied to one chunk.
+
+Preserve the overall stage sequence: finish a stage across its chunks before
+advancing to the next stage. Concurrent chunk work within a stage continues to
+share PromptKit's backend capacity. A skipped work unit does not prevent other
+chunks or later stages from executing.
+
+Fixed chunking makes failure isolation and context stable, as well as making
+cross-stage reporting meaningful. It trades away adaptive repartitioning after
+edits and task-specific chunk sizes. Initial sizing should allow reasonable room
+for prompt overhead and text growth, without claiming that every later request
+will fit. An oversized later model request is an explicit skipped-work outcome;
+do not silently change chunk boundaries. Exact sizing policy remains open.
+
+Whole-pipeline traversal by independently advancing chunks is explicitly deferred
+to a future roadmap. Fixed chunking does not authorize changing the stage barrier.
+Read-only neighboring context, cross-chunk edits, and transcript-wide restructuring
+are also outside this unit; fixed chunks are not a permanent product prohibition
+on such future capabilities.
+
+## Resilient Execution Contract
+
+- Keep bounded transport retries, structural repairs, operation deadlines, and
+  shared backend capacity. Completion comes from safe fallback, not unlimited
+  attempts or an Audita retry loop around PromptKit's existing recovery.
+- Generate proposals, enforce deterministic/domain checks, and complete required
+  LLM validation before applying a correction. Failure must never bypass validation
+  or cause uncertain corrections to be accepted.
+- Unsuccessful model work preserves the affected work unit's incoming text and
+  records an error. Preserve corrections committed by earlier stages and other
+  successful work units. The handling of partially successful batches within a
+  failing work unit is an explicit open question below.
+- Continue despite model-operation failures, including exhausted repairs/retries,
+  unusable model decisions, provider/transport failures, deadlines, and admission
+  failures. An operation can fail without consuming its entire attempt budget;
+  budget exhaustion is not the sole definition of an error.
+- A valid empty proposal list or valid assessments rejecting every correction is
+  a successful outcome. Correction count does not determine success or failure.
+- Retain fatal treatment for invalid input/configuration, internal defects,
+  explicit run cancellation, and inability to produce/write the transcript.
+  Distinguish caller cancellation from a local model-operation timeout.
+- A completed run with skipped LLM work returns success and makes that degradation
+  visible in its summary and error records. Fatal run failures remain nonzero.
+
+PromptKit continues to own model transport, transient retries, structural repair,
+and admission. Audita owns work-unit outcomes, domain acceptance, fallback, and
+report classification. ConfigKit continues to own configuration resolution.
+Debugging must not change correction decisions, retry budgets, or continuation.
+
+## Errors, Recoveries, and Debug
+
+Classify by eventual outcome and impact, not exception type or retry count alone.
+These are three reporting categories; debug is evidence, not a failure severity.
+Whether execution can continue is separate from the category.
+
+| Category | Meaning | Representative cases |
+| --- | --- | --- |
+| Errors | Intended work could not complete reliably and was skipped, or a fatal run-level operation failed. | Exhausted output repair; unavailable provider; unusable validation decisions; operation deadline. |
+| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient provider failure recovered; defective proposals safely discarded. |
+| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompt/response content, individual attempts, detailed execution traces. |
+
+Normal editorial disagreement belongs in debug: a validator correctly rejecting
+a meaning-changing edit is expected behavior. A proposal referring to nonexistent
+original text is defective; safely filtering it is a recovery. A schema-valid
+validation response with missing or duplicate decision indices is an operation
+failure when it cannot be recovered. Do not label unresolved output as recovered.
+
+Record the eventual outcome of an operation once. Several unsuccessful attempts
+followed by success yield a consolidated recovery. Attempts ending in failure
+contribute to the error, not additional recoveries. Earlier operations that did
+recover remain recorded if a later operation fails the work unit.
+
+Separate output-quality recovery, defective-proposal filtering, and service
+recovery using structured reasons. Network instability should not be mistaken
+for a prompt deficiency. Preserve attempt counts and affected-proposal counts
+without counting each attempt or rejected proposal as an independent incident.
+
+Recovery frequency needs processing-volume context. Seven recoveries need not
+warrant attention; seventy may or may not, depending on the amount and kind of
+work. Do not introduce an arbitrary warning threshold or restore “warnings” as a
+fourth category. Final counting/denominator details remain open below.
+
+## Production Outputs and Debug Boundary
+
+The selected ordinary report set is:
+
+- `errors.yml`: structured records of unsuccessful work and applicable run failures.
+- `recoveries.yml`: structured records of successfully handled problems.
+- `report.md`: a human-facing summary of outcome, processing volume, applied
+  corrections, errors, recoveries, and their locations in the pipeline.
+
+Use Audita-owned structured records as the common source for YAML, Markdown, and
+CLI summaries. Reuse useful existing report concepts rather than expose raw
+library structures or maintain independently calculated totals. Exact schemas
+are not finalized. They should identify the run, stage instance, stable chunk,
+operation, validator/batch where relevant, reason, outcome, and pertinent counts.
+Keep safe human-readable explanations alongside machine-readable reason codes.
+
+Production issue records should be compact. Full responses, transcript excerpts,
+proposed/applied correction lists, and complete validation decisions belong in
+opt-in debug artifacts. Debug output may continue using JSON where suitable.
+Preserve configured-secret redaction across every presentation and artifact.
+
+Provide an explicit debug opt-in, with `--debug` as the proposed CLI spelling.
+A configuration counterpart is desirable but its exact interface remains open.
+Debug-off must prevent detailed artifact persistence in the first place; writing
+those files and deleting them later does not satisfy the target. Temporary
+in-memory processing data needed for correct execution is not debug output.
+
+Ordinary reports must survive independently of debug retention. The current
+`--report-json`, run-local report, ledger, execution/utilization artifacts, and
+retention settings need a deliberate disposition. Do not silently retain all
+existing verbose output as additional production defaults. Coordinate the
+retention decision with the existing [runtime follow-up](runtime-behavior.md).
+
+## Human-Facing Presentation
+
+Provide a concise completion message showing corrections applied, work units
+skipped due to errors, and recoveries. For example, the intended style is:
+
+> Completed: 18 corrections applied; 3 work units skipped due to errors; 7 recoveries.
+
+Include report locations when available. Send the summary to stderr so stdout
+remains available exclusively for transcript output. Successful completion must
+not imply that every stage/chunk succeeded.
+
+`report.md` should help the reader locate concentrations of failures and recoveries
+without opening debug data. Stable chunks support the proposed presentation of
+separate error and recovery matrices: chunks as rows, configured stage instances
+as columns, and counts in each cell. Preserve execution order and distinguish
+repeated stages. Include totals and a short breakdown by reason.
+
+Distinguish completed work with zero events from work that was not attempted.
+Do not present canceled or unavailable work as cleanly completed. Exact table
+counting, omission of uneventful rows, and large-report layout remain open below.
+
+## Target Acceptance Criteria
+
+- A controlled run in which every LLM operation fails still returns success and
+  the normally encoded normalized transcript, with no LLM corrections applied.
+- Mixed-success runs preserve committed corrections, skip unsuccessful work
+  safely, and attempt subsequent stages and unaffected chunks.
+- Chunks retain their identity and boundaries across all stages; validation
+  batching does not repartition them. Final recombination preserves ordering.
+- Recoverable model failures do not cancel unrelated work. Fatal failures and
+  explicit cancellation still stop and join active work appropriately.
+- Valid no-change and ordinary editorial-rejection outcomes remain successful.
+  Repaired/filtered output and unrecovered failures receive distinct classifications.
+- YAML records, Markdown tables, and CLI totals agree, with deterministic ordering
+  and no duplicate incident counts from retries or repairs.
+- Debug-off produces the selected ordinary output set without detailed debugging
+  artifacts; debug-on adds evidence without changing processing outcomes.
+- Tests exercise real runner/runtime/reporting paths with controlled providers,
+  including exhausted recovery, partial success, and total LLM failure. Existing
+  domain safeguards, capacity limits, redaction, and output contracts remain
+  covered where they are retained.
+- On implementation, update the canonical CLI, configuration, operations,
+  integration, internal, and architecture documents to describe the new behavior.
+  This roadmap alone does not change their current contracts.
+
+## Open Questions
+
+### Work-unit atomicity when only some validation batches succeed
+
+**Recommendation:** Commit a stage/chunk unit only after all required operations
+complete reliably. If a required batch fails, discard that unit's pending edits,
+record one work-unit error with the underlying operation failures, and preserve
+its incoming text. Normal valid rejections are not operation failures. This gives
+predictable fallback and straightforward error counting.
+
+**Alternative:** Preserve fully validated corrections from successful batches in
+an otherwise failing unit. This retains more useful work but requires explicit
+partial-success outcomes and clearer counts than “work unit skipped.”
+
+### Domain-invalid responses and recovery opportunities
+
+**Recommendation:** Continue safely filtering independently defective proposals;
+do not retry ordinary editorial rejections. Treat an unusable validator decision
+set as an error when existing permitted recovery cannot resolve it. Decide
+explicitly whether dynamic decision-index/cardinality checks should become part
+of the repair contract; this roadmap does not silently add semantic retry loops.
+
+**Alternative:** Extend recovery to regenerate domain-invalid decision sets with
+bounded feedback. This may improve completion quality, but its ownership,
+feedback contract, and cost budget need a separate concrete design.
+
+### Initial chunk sizing and later prompt growth
+
+**Recommendation:** Retain operator control over chunk size and choose reasonable
+headroom for known stage prompts. Keep validation batching within fixed chunks;
+report a later operation that cannot fit as skipped work. Establish the concrete
+sizing rule during detailed design rather than invent a universal safety margin.
+
+**Alternative:** Size initial chunks against conservative estimates for all
+selected stage/profile combinations. This reduces later size failures but may
+increase call volume and still cannot guarantee fit after arbitrary text growth.
+
+### Exact event counts and recovery-rate denominators
+
+**Recommendation:** Count errors once per failed stage/chunk unit, linking multiple
+underlying failures as details. Count recoveries once per recovered operation;
+aggregate defective-proposal filtering per proposal response with an affected
+count. Report work-unit coverage separately from logical-call recovery counts,
+with explicit denominators. Earlier recoveries may coexist with a later error in
+the same unit.
+
+**Alternative:** Count affected work units for both matrices, placing operation
+counts only in details. This makes the two tables directly comparable but hides
+multiple distinct recoveries within a single unit unless the reader drills down.
+
+### Output placement, empty files, and report-write failures
+
+**Recommendation:** Produce all three ordinary documents for every completed run,
+using empty event lists where appropriate. Select a durable report directory
+independent of debug cleanup, including when transcript output goes to stdout.
+Specify its CLI/default path and overwrite policy before implementation. Attempt
+partial reports on fatal failures when possible. Report-writing failures should
+be explicit and should not erase an already produced transcript; whether they
+make the invocation nonzero must be settled as part of the output contract.
+
+**Alternative:** Write event files only when nonempty and identify their absence
+explicitly in `report.md`. This reduces empty files but requires consumers to
+understand missing-file semantics. Best-effort ancillary reporting could preserve
+a zero exit status after transcript success, at the cost of weaker evidence that
+the promised output bundle is complete.
+
+### Existing report interfaces and debug configuration
+
+**Recommendation:** Keep useful internal structures and measurements, adapting the
+ordinary schema to its audience and moving verbose evidence behind debug.
+Finalize a versioned YAML envelope, identifiers/index display, and the disposition
+of `--report-json` and retention settings in detailed design. Use one debug boolean
+with a CLI override and optional ConfigKit-backed setting; avoid a matrix of
+independent artifact switches.
+
+**Alternative:** Retain explicit machine-oriented JSON export alongside the new
+ordinary outputs for consumers that need it. This preserves an existing interface
+but adds another maintained presentation. Compatibility/export requirements have
+not been selected for this feature; do not infer them from prior migration policy.
+
+### Large Markdown reports
+
+**Recommendation:** Use separate matrices with totals and reason summaries; omit
+uneventful chunk rows when that improves readability, explicitly reporting the
+omitted count. Show zero separately from unattempted work. Keep full incident
+details in YAML and identify chunks by stable labels and segment ranges.
+
+**Alternative:** Always include every chunk row, splitting wide tables into stage
+groups. This gives an exhaustive visual grid but can make long transcripts harder
+to review. Exact layout and any row-selection threshold remain undecided.