diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md new file mode 100644 index 0000000..7c782f0 --- /dev/null +++ b/docs/roadmap/future.md @@ -0,0 +1,51 @@ +# Deferred Feature Work + +**Last updated:** 2026-09-24 + +**Status:** Intended future work; not implemented or scheduled. These entries +preserve scope deferred from the [resilience roadmap](resilience.md). They are +not prerequisites for completing that feature or staged implementation plans. + +## Domain-Aware Response Recovery + +Add bounded feedback and regeneration for model responses that satisfy the +structural schema but violate Audita's domain response contract, such as missing, +duplicate, or unexpected validator decision indices. The goal is to recover more +usable work before excluding affected candidates, reducing the impact of an +isolated malformed decision set. + +This capability is desired but explicitly deferred from the current resilience +work. That work retains existing PromptKit structural repair and safe filtering; +unresolved domain-invalid decision sets produce errors and exclude their affected +candidates while other candidates can succeed. + +Before implementation, define the Audita/PromptKit ownership boundary, which +checks are eligible for feedback, how feedback identifies defects without exposing +secrets, and how regeneration fits bounded budgets and operation deadlines. Reuse +shared library capabilities where appropriate; do not introduce an uncoordinated +outer retry loop that multiplies existing retry/repair budgets. + +Ordinary editorial rejection is a valid decision, not a reason to regenerate until +an edit is approved. Recovery must preserve required validation, candidate identity, +partial-success safety, and truthful error/recovery reporting. Validate with +controlled responses that distinguish recoverable protocol defects from valid +negative decisions. Exact APIs, budgets, and sequencing remain undecided. + +## Independent Whole-Pipeline Traversal by Chunk + +Explore allowing each fixed chunk to advance through its ordered stages without +waiting for other chunks to finish the current stage. This could reduce waiting +behind a slow chunk and improve utilization while preserving stage order within +each chunk and deterministic final recombination. + +The resilience feature deliberately retains stage barriers: finish a stage across +chunks before advancing to the next stage. Fixed boundaries alone do not authorize +independent traversal. + +A future design must establish that stages do not depend on other chunks' evolving +state, preserve stable identities and partial-success rules, and continue to share +PromptKit's per-backend capacity across all roles and stages. Reporting must remain +deterministic despite different completion orders, and fatal cancellation must +still stop and join active work. Assess whether the scheduling change materially +improves real workloads before introducing additional orchestration machinery. +No new scheduler, global limiter, or execution mode is selected by this entry. diff --git a/docs/roadmap/resilience.md b/docs/roadmap/resilience.md index d45bfba..08102ce 100644 --- a/docs/roadmap/resilience.md +++ b/docs/roadmap/resilience.md @@ -2,9 +2,9 @@ **Last updated:** 2026-09-24 -**Status:** Feature direction agreed; not implemented. This document records the -intended end state and remaining product decisions. It is not a staged -implementation plan. Recommendations under Open Questions are not adopted policy. +**Status:** Feature direction agreed; not implemented. This document defines the +target behavior and user intent, not a staged implementation plan. Remaining +interface details are listed under Open Questions. ## Purpose and User Intent @@ -12,23 +12,26 @@ Audita should reliably produce a useful transcript even when its LLM services cannot improve it. Exhausted retries or unusable model responses must not turn an otherwise processable transcript into a failed production run. -The production guarantee is: - > Failure of every LLM operation across every stage and chunk still produces a > successful output containing the deterministically normalized transcript, with -> no LLM corrections applied. Partial LLM success preserves corrections that -> completed successfully, subject to the selected work-unit commit boundary. +> no LLM corrections applied. Partial success preserves as many fully validated, +> applicable corrections as possible. -This does not promise the original input byte-for-byte. Deterministic -normalization remains intentional and may change segment layout, identifiers, -and text before LLM work begins. Output still uses the selected transcript -encoder. There is no requirement to map corrections back to the original, -pre-normalization segmentation. +Deterministic normalization remains intentional and may change segment layout, +identifiers, and text before LLM work begins. Output uses the selected transcript +encoder; byte-for-byte preservation of the original input and mapping corrections +back to pre-normalization segmentation are not requirements. -Existing reports contain useful information. Preserve meaningful identities, -reason codes, correction/decision evidence, and execution measurements where -appropriate; reorganize their classification and audience rather than replace -everything with an unrelated reporting system. +The overriding goal is to minimize how much useful processing is lost because of +one problematic transcript segment. A stage/chunk work unit is an attribution +boundary, not an all-or-nothing transaction. + +Existing reports contain useful information, but there are no compatibility +obligations for reporting or debug output. Reuse existing machinery only when it +makes the implementation simpler and clearer; preservation is not itself a goal. +Internal structures and measurements must serve Audita's processing workflow. +The reporting package adapts those outcomes to structured and human-facing +presentations, rather than reshaping processing around a reporting convenience. ## Current Baseline and Intended Change @@ -44,268 +47,248 @@ Detailed artifacts are written during ordinary runs and may subsequently be removed by retention. The working transcript is re-chunked between modules. This feature deliberately changes those behaviors. It builds on the completed -[library adoption](library-adoption.md) and [review remediation](remediation.md); -it does not undo their separation of library and application responsibilities. -The existing fatal-cancellation mechanism remains useful, but model work failures -must be separated from failures that actually require cancellation of the run. +[library adoption](library-adoption.md) and [review remediation](remediation.md). +Fatal cancellation remains necessary; unsuccessful model work must be separated +from failures that actually require cancellation of the run. ## Fixed Chunks and Ordered Stages -Normalize first, then partition the transcript once. Maintain the same chunk -boundaries, segment membership, segment identity, and ordering throughout LLM -processing. Text inside a chunk evolves as accepted corrections are applied. -Recombine the chunks in their original order after the LLM stages finish. +Normalize first, then partition the transcript once. Preserve chunk boundaries, +segment membership, segment identity, and ordering throughout LLM processing. +Text evolves as accepted corrections are applied. Recombine chunks in original +order after the LLM stages finish. -For this feature, a stage is a configured module instance. Repeated instances of -the same module have distinct identities. Proposal generation and each validator -are operations within a stage; validation may use multiple batches within a chunk. -A work unit is one stage instance applied to one chunk. +A stage is a configured module instance. Repeated instances of the same module +have distinct identities. Proposal generation and each validator are operations +within a stage; validation may use multiple batches within a chunk. A work unit +is one stage instance applied to one chunk. Preserve the overall stage sequence: finish a stage across its chunks before -advancing to the next stage. Concurrent chunk work within a stage continues to -share PromptKit's backend capacity. A skipped work unit does not prevent other -chunks or later stages from executing. +advancing to the next stage. Concurrent chunk work shares PromptKit's backend +capacity. Failure in one unit does not prevent other chunks or later stages +from executing. -Fixed chunking makes failure isolation and context stable, as well as making -cross-stage reporting meaningful. It trades away adaptive repartitioning after -edits and task-specific chunk sizes. Initial sizing should allow reasonable room -for prompt overhead and text growth, without claiming that every later request -will fit. An oversized later model request is an explicit skipped-work outcome; -do not silently change chunk boundaries. Exact sizing policy remains open. +Retain operator control over chunk size and the current defaults. Production +experience indicates that chunk size affects model latency nonlinearly and that +useful settings are well below backend context limits. Tuning is an empirical +operator decision. Do not add automatic headroom calculations, cross-profile +sizing, adaptive repartitioning, or automatic tuning in this feature. A later +operation that cannot process its input follows the same safe error handling; +it does not silently alter chunk boundaries. Validation batching within a chunk +remains supported. -Whole-pipeline traversal by independently advancing chunks is explicitly deferred -to a future roadmap. Fixed chunking does not authorize changing the stage barrier. -Read-only neighboring context, cross-chunk edits, and transcript-wide restructuring -are also outside this unit; fixed chunks are not a permanent product prohibition -on such future capabilities. +Independent whole-pipeline traversal by chunks and domain-aware response repair +are explicitly deferred in [future work](future.md). Read-only neighboring +context, cross-chunk edits, and transcript-wide restructuring are outside this +unit, without becoming permanent prohibitions on future capabilities. -## Resilient Execution Contract +## Partial Success and Failure Isolation - Keep bounded transport retries, structural repairs, operation deadlines, and shared backend capacity. Completion comes from safe fallback, not unlimited - attempts or an Audita retry loop around PromptKit's existing recovery. -- Generate proposals, enforce deterministic/domain checks, and complete required - LLM validation before applying a correction. Failure must never bypass validation - or cause uncertain corrections to be accepted. -- Unsuccessful model work preserves the affected work unit's incoming text and - records an error. Preserve corrections committed by earlier stages and other - successful work units. The handling of partially successful batches within a - failing work unit is an explicit open question below. + attempts or an outer retry loop around PromptKit. +- Apply a correction only after it passes every required validator and the + application checks. Failed validation must never be treated as approval. +- If a validation batch fails, exclude its affected candidates while allowing + other candidates to continue through the remaining validators. Preserve fully + validated, applicable corrections from successful batches in the same unit. +- Track candidates by stable proposal identity. Batch membership can change + between validators; equal batch numbers do not establish candidate identity. +- A failed proposal-generation operation leaves its chunk unchanged for that + stage because there are no reliable proposals to salvage. Failures later in + validation need not discard unrelated successful candidates. +- Preserve earlier stages' corrections and other chunks' successful work. + Record partial success explicitly; a unit can apply corrections and have errors. - Continue despite model-operation failures, including exhausted repairs/retries, - unusable model decisions, provider/transport failures, deadlines, and admission - failures. An operation can fail without consuming its entire attempt budget; - budget exhaustion is not the sole definition of an error. -- A valid empty proposal list or valid assessments rejecting every correction is - a successful outcome. Correction count does not determine success or failure. + unusable model decisions, provider/transport failures, local operation deadlines, + and admission failures. An operation need not exhaust its attempt budget to fail. +- A valid empty proposal list or valid assessments rejecting all corrections is + successful processing. Correction count alone does not determine failure. +- Keep safe filtering of defective proposals. Do not retry ordinary editorial + rejections. For now, an unusable validator decision set excludes the affected + candidates and records an error after existing permitted recovery. New bounded + feedback/regeneration for domain-invalid responses belongs to future work. - Retain fatal treatment for invalid input/configuration, internal defects, explicit run cancellation, and inability to produce/write the transcript. - Distinguish caller cancellation from a local model-operation timeout. -- A completed run with skipped LLM work returns success and makes that degradation - visible in its summary and error records. Fatal run failures remain nonzero. + Caller cancellation is distinct from a local model-operation timeout. +- Completed runs with model errors return success. Required report-write failures + are an additional fatal output condition when reporting is enabled, as below. -PromptKit continues to own model transport, transient retries, structural repair, -and admission. Audita owns work-unit outcomes, domain acceptance, fallback, and -report classification. ConfigKit continues to own configuration resolution. -Debugging must not change correction decisions, retry budgets, or continuation. +PromptKit owns transport, transient retries, structural repair, and admission. +Audita owns domain acceptance, partial outcomes, fallback, and classification. +ConfigKit owns configuration resolution. Debug and reporting choices must not +change correction decisions, retry budgets, or continuation. ## Errors, Recoveries, and Debug -Classify by eventual outcome and impact, not exception type or retry count alone. -These are three reporting categories; debug is evidence, not a failure severity. -Whether execution can continue is separate from the category. +Classify by eventual outcome and impact. These are three reporting categories; +debug is evidence, not a failure severity. Whether execution continues is a +separate property. | Category | Meaning | Representative cases | | --- | --- | --- | -| Errors | Intended work could not complete reliably and was skipped, or a fatal run-level operation failed. | Exhausted output repair; unavailable provider; unusable validation decisions; operation deadline. | -| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient provider failure recovered; defective proposals safely discarded. | -| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompt/response content, individual attempts, detailed execution traces. | +| Errors | Intended work could not complete reliably, or a fatal run-level operation failed. Affected candidates are excluded; other corrections can succeed. | Exhausted repair; unavailable provider; unusable validation decisions; operation deadline. | +| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient service failure recovered; defective proposals safely discarded. | +| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompts/responses, individual attempts, execution traces. | -Normal editorial disagreement belongs in debug: a validator correctly rejecting +Normal editorial disagreement belongs in debug. A validator correctly rejecting a meaning-changing edit is expected behavior. A proposal referring to nonexistent -original text is defective; safely filtering it is a recovery. A schema-valid -validation response with missing or duplicate decision indices is an operation -failure when it cannot be recovered. Do not label unresolved output as recovered. +original text is defective; safely filtering it is a recovery. An unresolved +missing/duplicate decision-index response is an error, not a recovery. -Record the eventual outcome of an operation once. Several unsuccessful attempts -followed by success yield a consolidated recovery. Attempts ending in failure -contribute to the error, not additional recoveries. Earlier operations that did -recover remain recorded if a later operation fails the work unit. +Record an operation's eventual outcome once. Several failed attempts followed by +success produce a consolidated recovery. Attempts ending in failure contribute +to error details, not additional recoveries. Earlier operations that recovered +remain recorded if a later operation introduces an error in the same unit. Separate output-quality recovery, defective-proposal filtering, and service -recovery using structured reasons. Network instability should not be mistaken -for a prompt deficiency. Preserve attempt counts and affected-proposal counts -without counting each attempt or rejected proposal as an independent incident. +recovery with structured reasons. Network instability must not look like a prompt +defect. Do not introduce an arbitrary warning threshold or a fourth category. -Recovery frequency needs processing-volume context. Seven recoveries need not -warrant attention; seventy may or may not, depending on the amount and kind of -work. Do not introduce an arbitrary warning threshold or restore “warnings” as a -fourth category. Final counting/denominator details remain open below. +### Counting and Processing Volume -## Production Outputs and Debug Boundary +- Count error-bearing work units once per stage/chunk, consolidating underlying + operation failures as details. A partially successful unit still counts once. +- Count recoveries once per recovered operation. Aggregate defective-proposal + filtering per proposal response, retaining affected-proposal counts as detail. + Attempts and individual discarded proposals are not independent incidents. +- Keep work-unit coverage separate from operation/recovery counts. Show processing + volume with explicitly labeled denominators; do not divide recovered-operation + counts by work-unit counts and call that a recovery rate. +- Report fatal run-level errors separately from the stage/chunk grid when they + cannot be attributed to a work unit. Do not invent a chunk to accommodate them. +- Derive YAML records, Markdown summaries, and CLI counts from actual processing + outcomes with deterministic ordering, rather than independently calculating them. -The selected ordinary report set is: +## Production Outputs, Placement, and Lifecycle -- `errors.yml`: structured records of unsuccessful work and applicable run failures. -- `recoveries.yml`: structured records of successfully handled problems. -- `report.md`: a human-facing summary of outcome, processing volume, applied - corrections, errors, recoveries, and their locations in the pipeline. +Reporting is enabled by default. The ordinary bundle is: -Use Audita-owned structured records as the common source for YAML, Markdown, and -CLI summaries. Reuse useful existing report concepts rather than expose raw -library structures or maintain independently calculated totals. Exact schemas -are not finalized. They should identify the run, stage instance, stable chunk, -operation, validator/batch where relevant, reason, outcome, and pertinent counts. -Keep safe human-readable explanations alongside machine-readable reason codes. +- `errors.yml`: structured error-bearing units, their underlying failures, and + applicable run-level failures. +- `recoveries.yml`: structured recovered operations and pertinent counts. +- `report.md`: a concise human-facing summary of outcome, corrections, processing + volume, errors, recoveries, and their locations in the pipeline. -Production issue records should be compact. Full responses, transcript excerpts, -proposed/applied correction lists, and complete validation decisions belong in -opt-in debug artifacts. Debug output may continue using JSON where suitable. -Preserve configured-secret redaction across every presentation and artifact. +Write all three documents, including empty event lists for uneventful completed +runs. Use Audita-owned schemas with safe explanations and stable reason codes; +retain run, stage-instance, chunk, operation, and validator/batch identities where +applicable. Exact schemas are detailed-design work, not a reason to preserve the +old report formats or expose raw library structures. -Provide an explicit debug opt-in, with `--debug` as the proposed CLI spelling. -A configuration counterpart is desirable but its exact interface remains open. -Debug-off must prevent detailed artifact persistence in the first place; writing -those files and deleting them later does not satisfy the target. Temporary -in-memory processing data needed for correct execution is not debug output. +Place reports beside the output transcript. Retain transcript-to-stdout support +with an explicit report-directory selection; its default is a remaining interface +detail. Preserve the existing overwrite convention: a new run can replace files +at its selected destinations. Do not add a mandatory fresh-directory restriction +or compatibility exports. -Ordinary reports must survive independently of debug retention. The current -`--report-json`, run-local report, ledger, execution/utilization artifacts, and -retention settings need a deliberate disposition. Do not silently retain all -existing verbose output as additional production defaults. Coordinate the -retention decision with the existing [runtime follow-up](runtime-behavior.md). +Before model work, create initial versions of all three reports carrying a common +run identifier and explicit `in_progress` status (rendered as prose in Markdown). +Fail immediately if initialization fails. Also check the transcript destination; +creating neighboring reports does not prove that an existing transcript file is +writable. Preflight must not prematurely destroy an existing transcript. -## Human-Facing Presentation +Finalize all required outputs and check every final write. Initial write success +cannot guarantee later success if space or permissions change. Failure to write +the transcript or finalize a required report returns nonzero and must not erase +an already produced transcript. Where possible, finalize available reports with +an explicit failed status on fatal termination. Abrupt termination may leave +`in_progress`; elaborate crash-recovery machinery is not required. -Provide a concise completion message showing corrections applied, work units -skipped due to errors, and recoveries. For example, the intended style is: +Provide an option to disable reporting entirely. It suppresses all three files +(`errors.yml`, `recoveries.yml`, and `report.md`), including initialization and +final writes, while retaining the brief CLI summary and independently selectable +debug output. Disabled reporting must not create or require a writable report +destination, and must not remove reports from earlier runs. +Transcript output failures remain fatal regardless of reporting selection. -> Completed: 18 corrections applied; 3 work units skipped due to errors; 7 recoveries. +Ordinary reports survive independently of debug cleanup. Reporting configuration +and finalization must not depend on retaining a debug run directory. -Include report locations when available. Send the summary to stderr so stdout -remains available exclusively for transcript output. Successful completion must -not imply that every stage/chunk succeeded. +## Debug and Internal Boundaries -`report.md` should help the reader locate concentrations of failures and recoveries -without opening debug data. Stable chunks support the proposed presentation of -separate error and recovery matrices: chunks as rows, configured stage instances -as columns, and counts in each cell. Preserve execution order and distinguish -repeated stages. Include totals and a short breakdown by reason. +Production issue records are compact. Full responses, transcript excerpts, +correction lists, and complete validation decisions are opt-in debug evidence. +Use a single debug boolean, with `--debug` as the proposed CLI spelling and a +ConfigKit-backed setting/override design to be finalized. Debug artifacts may use +JSON where useful. Preserve configured-secret redaction in every presentation. -Distinguish completed work with zero events from work that was not attempted. -Do not present canceled or unavailable work as cleanly completed. Exact table -counting, omission of uneventful rows, and large-report layout remain open below. +Debug-off prevents detailed artifact persistence; writing and later deleting +those files is insufficient. In-memory domain state needed to process candidates +correctly is not debug output. Debug and ordinary reporting are independent +selections and do not alter processing behavior. + +There is no obligation to retain `--report-json`, old artifact layouts, parallel +exports, or old retention settings. Reuse code only where useful to the new +workflow. Resolve the disposition of debug retention alongside the existing +[runtime follow-up](runtime-behavior.md); the legacy `never` behavior must not +quietly become the meaning of the new reporting-disable option. + +## Deterministic Human-Facing Presentation + +Use the CLI style: + +> Completed: 18 corrections applied; 3 work units with errors; 7 recoveries. + +Do not say that all error-bearing units were skipped: some apply valid corrections. +Include report locations when reporting is enabled and the files exist. Send the +summary to stderr so stdout remains exclusively available for transcript output. +Disabled reporting does not suppress the summary or fatal-error messages. + +Keep `report.md` concise and high-level. Use separate error and recovery matrices +with stages as columns and chunks as rows. Include every chunk in fixed order; +retain configured stage order and distinct labels for repeated module instances. +No automatic width detection, thresholds, adaptive omission, or table splitting. +A wide or long table is acceptable. + +The error matrix counts error-bearing work units: each attempted cell is 0 or 1. +Recovery cells count recovered operations and can exceed 1. An error cell may +coexist with recoveries and successfully applied corrections in the same unit. +Distinguish completed work with zero events from unattempted work; do not portray +canceled/unavailable processing as clean completion. Include totals, processing +volume, and a short reason breakdown. Detailed inspection belongs in YAML or debug. ## Target Acceptance Criteria -- A controlled run in which every LLM operation fails still returns success and - the normally encoded normalized transcript, with no LLM corrections applied. -- Mixed-success runs preserve committed corrections, skip unsuccessful work - safely, and attempt subsequent stages and unaffected chunks. -- Chunks retain their identity and boundaries across all stages; validation - batching does not repartition them. Final recombination preserves ordering. -- Recoverable model failures do not cancel unrelated work. Fatal failures and - explicit cancellation still stop and join active work appropriately. -- Valid no-change and ordinary editorial-rejection outcomes remain successful. - Repaired/filtered output and unrecovered failures receive distinct classifications. -- YAML records, Markdown tables, and CLI totals agree, with deterministic ordering - and no duplicate incident counts from retries or repairs. -- Debug-off produces the selected ordinary output set without detailed debugging - artifacts; debug-on adds evidence without changing processing outcomes. -- Tests exercise real runner/runtime/reporting paths with controlled providers, - including exhausted recovery, partial success, and total LLM failure. Existing - domain safeguards, capacity limits, redaction, and output contracts remain - covered where they are retained. -- On implementation, update the canonical CLI, configuration, operations, - integration, internal, and architecture documents to describe the new behavior. - This roadmap alone does not change their current contracts. +- Total LLM failure still returns success and the encoded normalized transcript, + provided required output writes succeed. +- Mixed-success runs preserve independently validated, applicable corrections + within the same stage/chunk as a failed batch. No candidate bypasses a required + validator; later validators handle survivors by stable proposal identity. +- Subsequent stages and unaffected chunks execute after recoverable failures. + Fatal errors and explicit cancellation stop and join active work appropriately. +- Chunk boundaries and identity remain fixed, with operator-selected sizing and + unchanged defaults; final recombination preserves ordering. +- Valid no-change/editorial-rejection outcomes are successful. Repaired/filtered + output and unresolved failures receive distinct classifications and counts. +- YAML, Markdown, and CLI agree on partial outcomes and totals; error matrices + count affected work units, recovery matrices count recovered operations. +- With reporting enabled, all three reports are initialized before model work, + finalized and retained, including empty lists. Initial or final required-output + write failures produce nonzero exits without destroying completed transcript output. +- Reporting can be disabled without requiring its destination. Debug can be enabled + independently; neither selection changes processing outcomes. Debug-off writes + no detailed debug artifacts, and stdout transcript output remains supported. +- Reports use the fixed deterministic layout without adaptive presentation logic. +- Meaningful controlled-provider tests cover exhausted recovery, failed validation + batches with successful survivors, total failure, fatal cancellation, output + failures, and reporting/debug selections. Preserve domain safeguards, capacity + limits, and redaction. +- On implementation, update canonical CLI, configuration, operations, integration, + internal, and architecture documentation. This roadmap does not change their + current implemented contracts. ## Open Questions -### Work-unit atomicity when only some validation batches succeed +The product choices above replace the earlier alternatives and user comments. +Remaining details belong in the eventual decision-complete implementation plan: -**Recommendation:** Commit a stage/chunk unit only after all required operations -complete reliably. If a required batch fails, discard that unit's pending edits, -record one work-unit error with the underlying operation failures, and preserve -its incoming text. Normal valid rejections are not operation failures. This gives -predictable fallback and straightforward error counting. - -**Alternative:** Preserve fully validated corrections from successful batches in -an otherwise failing unit. This retains more useful work but requires explicit -partial-success outcomes and clearer counts than “work unit skipped.” - -### Domain-invalid responses and recovery opportunities - -**Recommendation:** Continue safely filtering independently defective proposals; -do not retry ordinary editorial rejections. Treat an unusable validator decision -set as an error when existing permitted recovery cannot resolve it. Decide -explicitly whether dynamic decision-index/cardinality checks should become part -of the repair contract; this roadmap does not silently add semantic retry loops. - -**Alternative:** Extend recovery to regenerate domain-invalid decision sets with -bounded feedback. This may improve completion quality, but its ownership, -feedback contract, and cost budget need a separate concrete design. - -### Initial chunk sizing and later prompt growth - -**Recommendation:** Retain operator control over chunk size and choose reasonable -headroom for known stage prompts. Keep validation batching within fixed chunks; -report a later operation that cannot fit as skipped work. Establish the concrete -sizing rule during detailed design rather than invent a universal safety margin. - -**Alternative:** Size initial chunks against conservative estimates for all -selected stage/profile combinations. This reduces later size failures but may -increase call volume and still cannot guarantee fit after arbitrary text growth. - -### Exact event counts and recovery-rate denominators - -**Recommendation:** Count errors once per failed stage/chunk unit, linking multiple -underlying failures as details. Count recoveries once per recovered operation; -aggregate defective-proposal filtering per proposal response with an affected -count. Report work-unit coverage separately from logical-call recovery counts, -with explicit denominators. Earlier recoveries may coexist with a later error in -the same unit. - -**Alternative:** Count affected work units for both matrices, placing operation -counts only in details. This makes the two tables directly comparable but hides -multiple distinct recoveries within a single unit unless the reader drills down. - -### Output placement, empty files, and report-write failures - -**Recommendation:** Produce all three ordinary documents for every completed run, -using empty event lists where appropriate. Select a durable report directory -independent of debug cleanup, including when transcript output goes to stdout. -Specify its CLI/default path and overwrite policy before implementation. Attempt -partial reports on fatal failures when possible. Report-writing failures should -be explicit and should not erase an already produced transcript; whether they -make the invocation nonzero must be settled as part of the output contract. - -**Alternative:** Write event files only when nonempty and identify their absence -explicitly in `report.md`. This reduces empty files but requires consumers to -understand missing-file semantics. Best-effort ancillary reporting could preserve -a zero exit status after transcript success, at the cost of weaker evidence that -the promised output bundle is complete. - -### Existing report interfaces and debug configuration - -**Recommendation:** Keep useful internal structures and measurements, adapting the -ordinary schema to its audience and moving verbose evidence behind debug. -Finalize a versioned YAML envelope, identifiers/index display, and the disposition -of `--report-json` and retention settings in detailed design. Use one debug boolean -with a CLI override and optional ConfigKit-backed setting; avoid a matrix of -independent artifact switches. - -**Alternative:** Retain explicit machine-oriented JSON export alongside the new -ordinary outputs for consumers that need it. This preserves an existing interface -but adds another maintained presentation. Compatibility/export requirements have -not been selected for this feature; do not infer them from prior migration policy. - -### Large Markdown reports - -**Recommendation:** Use separate matrices with totals and reason summaries; omit -uneventful chunk rows when that improves readability, explicitly reporting the -omitted count. Show zero separately from unattempted work. Keep full incident -details in YAML and identify chunks by stable labels and segment ranges. - -**Alternative:** Always include every chunk row, splitting wide tables into stage -groups. This gives an exhaustive visual grid but can make long transcripts harder -to review. Exact layout and any row-selection threshold remain undecided. +- Select exact CLI/configuration names and precedence for reporting enablement, + debug, and the report directory for stdout runs. Reports for file-output runs + belong beside the transcript; the stdout report-directory default is not yet set. +- Finalize YAML versions/fields, machine identifiers versus human chunk labels, + operation-volume denominators, deterministic record order, and failed/not-attempted + presentation without changing the agreed counting or fixed table layout. +- Specify how destinations are checked without destructive transcript truncation, + finalization order, path-collision handling, and debug placement/retention. Keep + ordinary filesystem behavior proportionate; do not add a transaction framework.