Finalize resilience decisions and record deferred work

This commit is contained in:
2026-09-24 16:34:35 +00:00
parent aa2f486845
commit 053b714ce6
2 changed files with 267 additions and 233 deletions

51
docs/roadmap/future.md Normal file
View File

@@ -0,0 +1,51 @@
# Deferred Feature Work
**Last updated:** 2026-09-24
**Status:** Intended future work; not implemented or scheduled. These entries
preserve scope deferred from the [resilience roadmap](resilience.md). They are
not prerequisites for completing that feature or staged implementation plans.
## Domain-Aware Response Recovery
Add bounded feedback and regeneration for model responses that satisfy the
structural schema but violate Audita's domain response contract, such as missing,
duplicate, or unexpected validator decision indices. The goal is to recover more
usable work before excluding affected candidates, reducing the impact of an
isolated malformed decision set.
This capability is desired but explicitly deferred from the current resilience
work. That work retains existing PromptKit structural repair and safe filtering;
unresolved domain-invalid decision sets produce errors and exclude their affected
candidates while other candidates can succeed.
Before implementation, define the Audita/PromptKit ownership boundary, which
checks are eligible for feedback, how feedback identifies defects without exposing
secrets, and how regeneration fits bounded budgets and operation deadlines. Reuse
shared library capabilities where appropriate; do not introduce an uncoordinated
outer retry loop that multiplies existing retry/repair budgets.
Ordinary editorial rejection is a valid decision, not a reason to regenerate until
an edit is approved. Recovery must preserve required validation, candidate identity,
partial-success safety, and truthful error/recovery reporting. Validate with
controlled responses that distinguish recoverable protocol defects from valid
negative decisions. Exact APIs, budgets, and sequencing remain undecided.
## Independent Whole-Pipeline Traversal by Chunk
Explore allowing each fixed chunk to advance through its ordered stages without
waiting for other chunks to finish the current stage. This could reduce waiting
behind a slow chunk and improve utilization while preserving stage order within
each chunk and deterministic final recombination.
The resilience feature deliberately retains stage barriers: finish a stage across
chunks before advancing to the next stage. Fixed boundaries alone do not authorize
independent traversal.
A future design must establish that stages do not depend on other chunks' evolving
state, preserve stable identities and partial-success rules, and continue to share
PromptKit's per-backend capacity across all roles and stages. Reporting must remain
deterministic despite different completion orders, and fatal cancellation must
still stop and join active work. Assess whether the scheduling change materially
improves real workloads before introducing additional orchestration machinery.
No new scheduler, global limiter, or execution mode is selected by this entry.

View File

@@ -2,9 +2,9 @@
**Last updated:** 2026-09-24
**Status:** Feature direction agreed; not implemented. This document records the
intended end state and remaining product decisions. It is not a staged
implementation plan. Recommendations under Open Questions are not adopted policy.
**Status:** Feature direction agreed; not implemented. This document defines the
target behavior and user intent, not a staged implementation plan. Remaining
interface details are listed under Open Questions.
## Purpose and User Intent
@@ -12,23 +12,26 @@ Audita should reliably produce a useful transcript even when its LLM services
cannot improve it. Exhausted retries or unusable model responses must not turn
an otherwise processable transcript into a failed production run.
The production guarantee is:
> Failure of every LLM operation across every stage and chunk still produces a
> successful output containing the deterministically normalized transcript, with
> no LLM corrections applied. Partial LLM success preserves corrections that
> completed successfully, subject to the selected work-unit commit boundary.
> no LLM corrections applied. Partial success preserves as many fully validated,
> applicable corrections as possible.
This does not promise the original input byte-for-byte. Deterministic
normalization remains intentional and may change segment layout, identifiers,
and text before LLM work begins. Output still uses the selected transcript
encoder. There is no requirement to map corrections back to the original,
pre-normalization segmentation.
Deterministic normalization remains intentional and may change segment layout,
identifiers, and text before LLM work begins. Output uses the selected transcript
encoder; byte-for-byte preservation of the original input and mapping corrections
back to pre-normalization segmentation are not requirements.
Existing reports contain useful information. Preserve meaningful identities,
reason codes, correction/decision evidence, and execution measurements where
appropriate; reorganize their classification and audience rather than replace
everything with an unrelated reporting system.
The overriding goal is to minimize how much useful processing is lost because of
one problematic transcript segment. A stage/chunk work unit is an attribution
boundary, not an all-or-nothing transaction.
Existing reports contain useful information, but there are no compatibility
obligations for reporting or debug output. Reuse existing machinery only when it
makes the implementation simpler and clearer; preservation is not itself a goal.
Internal structures and measurements must serve Audita's processing workflow.
The reporting package adapts those outcomes to structured and human-facing
presentations, rather than reshaping processing around a reporting convenience.
## Current Baseline and Intended Change
@@ -44,268 +47,248 @@ Detailed artifacts are written during ordinary runs and may subsequently be
removed by retention. The working transcript is re-chunked between modules.
This feature deliberately changes those behaviors. It builds on the completed
[library adoption](library-adoption.md) and [review remediation](remediation.md);
it does not undo their separation of library and application responsibilities.
The existing fatal-cancellation mechanism remains useful, but model work failures
must be separated from failures that actually require cancellation of the run.
[library adoption](library-adoption.md) and [review remediation](remediation.md).
Fatal cancellation remains necessary; unsuccessful model work must be separated
from failures that actually require cancellation of the run.
## Fixed Chunks and Ordered Stages
Normalize first, then partition the transcript once. Maintain the same chunk
boundaries, segment membership, segment identity, and ordering throughout LLM
processing. Text inside a chunk evolves as accepted corrections are applied.
Recombine the chunks in their original order after the LLM stages finish.
Normalize first, then partition the transcript once. Preserve chunk boundaries,
segment membership, segment identity, and ordering throughout LLM processing.
Text evolves as accepted corrections are applied. Recombine chunks in original
order after the LLM stages finish.
For this feature, a stage is a configured module instance. Repeated instances of
the same module have distinct identities. Proposal generation and each validator
are operations within a stage; validation may use multiple batches within a chunk.
A work unit is one stage instance applied to one chunk.
A stage is a configured module instance. Repeated instances of the same module
have distinct identities. Proposal generation and each validator are operations
within a stage; validation may use multiple batches within a chunk. A work unit
is one stage instance applied to one chunk.
Preserve the overall stage sequence: finish a stage across its chunks before
advancing to the next stage. Concurrent chunk work within a stage continues to
share PromptKit's backend capacity. A skipped work unit does not prevent other
chunks or later stages from executing.
advancing to the next stage. Concurrent chunk work shares PromptKit's backend
capacity. Failure in one unit does not prevent other chunks or later stages
from executing.
Fixed chunking makes failure isolation and context stable, as well as making
cross-stage reporting meaningful. It trades away adaptive repartitioning after
edits and task-specific chunk sizes. Initial sizing should allow reasonable room
for prompt overhead and text growth, without claiming that every later request
will fit. An oversized later model request is an explicit skipped-work outcome;
do not silently change chunk boundaries. Exact sizing policy remains open.
Retain operator control over chunk size and the current defaults. Production
experience indicates that chunk size affects model latency nonlinearly and that
useful settings are well below backend context limits. Tuning is an empirical
operator decision. Do not add automatic headroom calculations, cross-profile
sizing, adaptive repartitioning, or automatic tuning in this feature. A later
operation that cannot process its input follows the same safe error handling;
it does not silently alter chunk boundaries. Validation batching within a chunk
remains supported.
Whole-pipeline traversal by independently advancing chunks is explicitly deferred
to a future roadmap. Fixed chunking does not authorize changing the stage barrier.
Read-only neighboring context, cross-chunk edits, and transcript-wide restructuring
are also outside this unit; fixed chunks are not a permanent product prohibition
on such future capabilities.
Independent whole-pipeline traversal by chunks and domain-aware response repair
are explicitly deferred in [future work](future.md). Read-only neighboring
context, cross-chunk edits, and transcript-wide restructuring are outside this
unit, without becoming permanent prohibitions on future capabilities.
## Resilient Execution Contract
## Partial Success and Failure Isolation
- Keep bounded transport retries, structural repairs, operation deadlines, and
shared backend capacity. Completion comes from safe fallback, not unlimited
attempts or an Audita retry loop around PromptKit's existing recovery.
- Generate proposals, enforce deterministic/domain checks, and complete required
LLM validation before applying a correction. Failure must never bypass validation
or cause uncertain corrections to be accepted.
- Unsuccessful model work preserves the affected work unit's incoming text and
records an error. Preserve corrections committed by earlier stages and other
successful work units. The handling of partially successful batches within a
failing work unit is an explicit open question below.
attempts or an outer retry loop around PromptKit.
- Apply a correction only after it passes every required validator and the
application checks. Failed validation must never be treated as approval.
- If a validation batch fails, exclude its affected candidates while allowing
other candidates to continue through the remaining validators. Preserve fully
validated, applicable corrections from successful batches in the same unit.
- Track candidates by stable proposal identity. Batch membership can change
between validators; equal batch numbers do not establish candidate identity.
- A failed proposal-generation operation leaves its chunk unchanged for that
stage because there are no reliable proposals to salvage. Failures later in
validation need not discard unrelated successful candidates.
- Preserve earlier stages' corrections and other chunks' successful work.
Record partial success explicitly; a unit can apply corrections and have errors.
- Continue despite model-operation failures, including exhausted repairs/retries,
unusable model decisions, provider/transport failures, deadlines, and admission
failures. An operation can fail without consuming its entire attempt budget;
budget exhaustion is not the sole definition of an error.
- A valid empty proposal list or valid assessments rejecting every correction is
a successful outcome. Correction count does not determine success or failure.
unusable model decisions, provider/transport failures, local operation deadlines,
and admission failures. An operation need not exhaust its attempt budget to fail.
- A valid empty proposal list or valid assessments rejecting all corrections is
successful processing. Correction count alone does not determine failure.
- Keep safe filtering of defective proposals. Do not retry ordinary editorial
rejections. For now, an unusable validator decision set excludes the affected
candidates and records an error after existing permitted recovery. New bounded
feedback/regeneration for domain-invalid responses belongs to future work.
- Retain fatal treatment for invalid input/configuration, internal defects,
explicit run cancellation, and inability to produce/write the transcript.
Distinguish caller cancellation from a local model-operation timeout.
- A completed run with skipped LLM work returns success and makes that degradation
visible in its summary and error records. Fatal run failures remain nonzero.
Caller cancellation is distinct from a local model-operation timeout.
- Completed runs with model errors return success. Required report-write failures
are an additional fatal output condition when reporting is enabled, as below.
PromptKit continues to own model transport, transient retries, structural repair,
and admission. Audita owns work-unit outcomes, domain acceptance, fallback, and
report classification. ConfigKit continues to own configuration resolution.
Debugging must not change correction decisions, retry budgets, or continuation.
PromptKit owns transport, transient retries, structural repair, and admission.
Audita owns domain acceptance, partial outcomes, fallback, and classification.
ConfigKit owns configuration resolution. Debug and reporting choices must not
change correction decisions, retry budgets, or continuation.
## Errors, Recoveries, and Debug
Classify by eventual outcome and impact, not exception type or retry count alone.
These are three reporting categories; debug is evidence, not a failure severity.
Whether execution can continue is separate from the category.
Classify by eventual outcome and impact. These are three reporting categories;
debug is evidence, not a failure severity. Whether execution continues is a
separate property.
| Category | Meaning | Representative cases |
| --- | --- | --- |
| Errors | Intended work could not complete reliably and was skipped, or a fatal run-level operation failed. | Exhausted output repair; unavailable provider; unusable validation decisions; operation deadline. |
| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient provider failure recovered; defective proposals safely discarded. |
| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompt/response content, individual attempts, detailed execution traces. |
| Errors | Intended work could not complete reliably, or a fatal run-level operation failed. Affected candidates are excluded; other corrections can succeed. | Exhausted repair; unavailable provider; unusable validation decisions; operation deadline. |
| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient service failure recovered; defective proposals safely discarded. |
| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompts/responses, individual attempts, execution traces. |
Normal editorial disagreement belongs in debug: a validator correctly rejecting
Normal editorial disagreement belongs in debug. A validator correctly rejecting
a meaning-changing edit is expected behavior. A proposal referring to nonexistent
original text is defective; safely filtering it is a recovery. A schema-valid
validation response with missing or duplicate decision indices is an operation
failure when it cannot be recovered. Do not label unresolved output as recovered.
original text is defective; safely filtering it is a recovery. An unresolved
missing/duplicate decision-index response is an error, not a recovery.
Record the eventual outcome of an operation once. Several unsuccessful attempts
followed by success yield a consolidated recovery. Attempts ending in failure
contribute to the error, not additional recoveries. Earlier operations that did
recover remain recorded if a later operation fails the work unit.
Record an operation's eventual outcome once. Several failed attempts followed by
success produce a consolidated recovery. Attempts ending in failure contribute
to error details, not additional recoveries. Earlier operations that recovered
remain recorded if a later operation introduces an error in the same unit.
Separate output-quality recovery, defective-proposal filtering, and service
recovery using structured reasons. Network instability should not be mistaken
for a prompt deficiency. Preserve attempt counts and affected-proposal counts
without counting each attempt or rejected proposal as an independent incident.
recovery with structured reasons. Network instability must not look like a prompt
defect. Do not introduce an arbitrary warning threshold or a fourth category.
Recovery frequency needs processing-volume context. Seven recoveries need not
warrant attention; seventy may or may not, depending on the amount and kind of
work. Do not introduce an arbitrary warning threshold or restore “warnings” as a
fourth category. Final counting/denominator details remain open below.
### Counting and Processing Volume
## Production Outputs and Debug Boundary
- Count error-bearing work units once per stage/chunk, consolidating underlying
operation failures as details. A partially successful unit still counts once.
- Count recoveries once per recovered operation. Aggregate defective-proposal
filtering per proposal response, retaining affected-proposal counts as detail.
Attempts and individual discarded proposals are not independent incidents.
- Keep work-unit coverage separate from operation/recovery counts. Show processing
volume with explicitly labeled denominators; do not divide recovered-operation
counts by work-unit counts and call that a recovery rate.
- Report fatal run-level errors separately from the stage/chunk grid when they
cannot be attributed to a work unit. Do not invent a chunk to accommodate them.
- Derive YAML records, Markdown summaries, and CLI counts from actual processing
outcomes with deterministic ordering, rather than independently calculating them.
The selected ordinary report set is:
## Production Outputs, Placement, and Lifecycle
- `errors.yml`: structured records of unsuccessful work and applicable run failures.
- `recoveries.yml`: structured records of successfully handled problems.
- `report.md`: a human-facing summary of outcome, processing volume, applied
corrections, errors, recoveries, and their locations in the pipeline.
Reporting is enabled by default. The ordinary bundle is:
Use Audita-owned structured records as the common source for YAML, Markdown, and
CLI summaries. Reuse useful existing report concepts rather than expose raw
library structures or maintain independently calculated totals. Exact schemas
are not finalized. They should identify the run, stage instance, stable chunk,
operation, validator/batch where relevant, reason, outcome, and pertinent counts.
Keep safe human-readable explanations alongside machine-readable reason codes.
- `errors.yml`: structured error-bearing units, their underlying failures, and
applicable run-level failures.
- `recoveries.yml`: structured recovered operations and pertinent counts.
- `report.md`: a concise human-facing summary of outcome, corrections, processing
volume, errors, recoveries, and their locations in the pipeline.
Production issue records should be compact. Full responses, transcript excerpts,
proposed/applied correction lists, and complete validation decisions belong in
opt-in debug artifacts. Debug output may continue using JSON where suitable.
Preserve configured-secret redaction across every presentation and artifact.
Write all three documents, including empty event lists for uneventful completed
runs. Use Audita-owned schemas with safe explanations and stable reason codes;
retain run, stage-instance, chunk, operation, and validator/batch identities where
applicable. Exact schemas are detailed-design work, not a reason to preserve the
old report formats or expose raw library structures.
Provide an explicit debug opt-in, with `--debug` as the proposed CLI spelling.
A configuration counterpart is desirable but its exact interface remains open.
Debug-off must prevent detailed artifact persistence in the first place; writing
those files and deleting them later does not satisfy the target. Temporary
in-memory processing data needed for correct execution is not debug output.
Place reports beside the output transcript. Retain transcript-to-stdout support
with an explicit report-directory selection; its default is a remaining interface
detail. Preserve the existing overwrite convention: a new run can replace files
at its selected destinations. Do not add a mandatory fresh-directory restriction
or compatibility exports.
Ordinary reports must survive independently of debug retention. The current
`--report-json`, run-local report, ledger, execution/utilization artifacts, and
retention settings need a deliberate disposition. Do not silently retain all
existing verbose output as additional production defaults. Coordinate the
retention decision with the existing [runtime follow-up](runtime-behavior.md).
Before model work, create initial versions of all three reports carrying a common
run identifier and explicit `in_progress` status (rendered as prose in Markdown).
Fail immediately if initialization fails. Also check the transcript destination;
creating neighboring reports does not prove that an existing transcript file is
writable. Preflight must not prematurely destroy an existing transcript.
## Human-Facing Presentation
Finalize all required outputs and check every final write. Initial write success
cannot guarantee later success if space or permissions change. Failure to write
the transcript or finalize a required report returns nonzero and must not erase
an already produced transcript. Where possible, finalize available reports with
an explicit failed status on fatal termination. Abrupt termination may leave
`in_progress`; elaborate crash-recovery machinery is not required.
Provide a concise completion message showing corrections applied, work units
skipped due to errors, and recoveries. For example, the intended style is:
Provide an option to disable reporting entirely. It suppresses all three files
(`errors.yml`, `recoveries.yml`, and `report.md`), including initialization and
final writes, while retaining the brief CLI summary and independently selectable
debug output. Disabled reporting must not create or require a writable report
destination, and must not remove reports from earlier runs.
Transcript output failures remain fatal regardless of reporting selection.
> Completed: 18 corrections applied; 3 work units skipped due to errors; 7 recoveries.
Ordinary reports survive independently of debug cleanup. Reporting configuration
and finalization must not depend on retaining a debug run directory.
Include report locations when available. Send the summary to stderr so stdout
remains available exclusively for transcript output. Successful completion must
not imply that every stage/chunk succeeded.
## Debug and Internal Boundaries
`report.md` should help the reader locate concentrations of failures and recoveries
without opening debug data. Stable chunks support the proposed presentation of
separate error and recovery matrices: chunks as rows, configured stage instances
as columns, and counts in each cell. Preserve execution order and distinguish
repeated stages. Include totals and a short breakdown by reason.
Production issue records are compact. Full responses, transcript excerpts,
correction lists, and complete validation decisions are opt-in debug evidence.
Use a single debug boolean, with `--debug` as the proposed CLI spelling and a
ConfigKit-backed setting/override design to be finalized. Debug artifacts may use
JSON where useful. Preserve configured-secret redaction in every presentation.
Distinguish completed work with zero events from work that was not attempted.
Do not present canceled or unavailable work as cleanly completed. Exact table
counting, omission of uneventful rows, and large-report layout remain open below.
Debug-off prevents detailed artifact persistence; writing and later deleting
those files is insufficient. In-memory domain state needed to process candidates
correctly is not debug output. Debug and ordinary reporting are independent
selections and do not alter processing behavior.
There is no obligation to retain `--report-json`, old artifact layouts, parallel
exports, or old retention settings. Reuse code only where useful to the new
workflow. Resolve the disposition of debug retention alongside the existing
[runtime follow-up](runtime-behavior.md); the legacy `never` behavior must not
quietly become the meaning of the new reporting-disable option.
## Deterministic Human-Facing Presentation
Use the CLI style:
> Completed: 18 corrections applied; 3 work units with errors; 7 recoveries.
Do not say that all error-bearing units were skipped: some apply valid corrections.
Include report locations when reporting is enabled and the files exist. Send the
summary to stderr so stdout remains exclusively available for transcript output.
Disabled reporting does not suppress the summary or fatal-error messages.
Keep `report.md` concise and high-level. Use separate error and recovery matrices
with stages as columns and chunks as rows. Include every chunk in fixed order;
retain configured stage order and distinct labels for repeated module instances.
No automatic width detection, thresholds, adaptive omission, or table splitting.
A wide or long table is acceptable.
The error matrix counts error-bearing work units: each attempted cell is 0 or 1.
Recovery cells count recovered operations and can exceed 1. An error cell may
coexist with recoveries and successfully applied corrections in the same unit.
Distinguish completed work with zero events from unattempted work; do not portray
canceled/unavailable processing as clean completion. Include totals, processing
volume, and a short reason breakdown. Detailed inspection belongs in YAML or debug.
## Target Acceptance Criteria
- A controlled run in which every LLM operation fails still returns success and
the normally encoded normalized transcript, with no LLM corrections applied.
- Mixed-success runs preserve committed corrections, skip unsuccessful work
safely, and attempt subsequent stages and unaffected chunks.
- Chunks retain their identity and boundaries across all stages; validation
batching does not repartition them. Final recombination preserves ordering.
- Recoverable model failures do not cancel unrelated work. Fatal failures and
explicit cancellation still stop and join active work appropriately.
- Valid no-change and ordinary editorial-rejection outcomes remain successful.
Repaired/filtered output and unrecovered failures receive distinct classifications.
- YAML records, Markdown tables, and CLI totals agree, with deterministic ordering
and no duplicate incident counts from retries or repairs.
- Debug-off produces the selected ordinary output set without detailed debugging
artifacts; debug-on adds evidence without changing processing outcomes.
- Tests exercise real runner/runtime/reporting paths with controlled providers,
including exhausted recovery, partial success, and total LLM failure. Existing
domain safeguards, capacity limits, redaction, and output contracts remain
covered where they are retained.
- On implementation, update the canonical CLI, configuration, operations,
integration, internal, and architecture documents to describe the new behavior.
This roadmap alone does not change their current contracts.
- Total LLM failure still returns success and the encoded normalized transcript,
provided required output writes succeed.
- Mixed-success runs preserve independently validated, applicable corrections
within the same stage/chunk as a failed batch. No candidate bypasses a required
validator; later validators handle survivors by stable proposal identity.
- Subsequent stages and unaffected chunks execute after recoverable failures.
Fatal errors and explicit cancellation stop and join active work appropriately.
- Chunk boundaries and identity remain fixed, with operator-selected sizing and
unchanged defaults; final recombination preserves ordering.
- Valid no-change/editorial-rejection outcomes are successful. Repaired/filtered
output and unresolved failures receive distinct classifications and counts.
- YAML, Markdown, and CLI agree on partial outcomes and totals; error matrices
count affected work units, recovery matrices count recovered operations.
- With reporting enabled, all three reports are initialized before model work,
finalized and retained, including empty lists. Initial or final required-output
write failures produce nonzero exits without destroying completed transcript output.
- Reporting can be disabled without requiring its destination. Debug can be enabled
independently; neither selection changes processing outcomes. Debug-off writes
no detailed debug artifacts, and stdout transcript output remains supported.
- Reports use the fixed deterministic layout without adaptive presentation logic.
- Meaningful controlled-provider tests cover exhausted recovery, failed validation
batches with successful survivors, total failure, fatal cancellation, output
failures, and reporting/debug selections. Preserve domain safeguards, capacity
limits, and redaction.
- On implementation, update canonical CLI, configuration, operations, integration,
internal, and architecture documentation. This roadmap does not change their
current implemented contracts.
## Open Questions
### Work-unit atomicity when only some validation batches succeed
The product choices above replace the earlier alternatives and user comments.
Remaining details belong in the eventual decision-complete implementation plan:
**Recommendation:** Commit a stage/chunk unit only after all required operations
complete reliably. If a required batch fails, discard that unit's pending edits,
record one work-unit error with the underlying operation failures, and preserve
its incoming text. Normal valid rejections are not operation failures. This gives
predictable fallback and straightforward error counting.
**Alternative:** Preserve fully validated corrections from successful batches in
an otherwise failing unit. This retains more useful work but requires explicit
partial-success outcomes and clearer counts than “work unit skipped.”
### Domain-invalid responses and recovery opportunities
**Recommendation:** Continue safely filtering independently defective proposals;
do not retry ordinary editorial rejections. Treat an unusable validator decision
set as an error when existing permitted recovery cannot resolve it. Decide
explicitly whether dynamic decision-index/cardinality checks should become part
of the repair contract; this roadmap does not silently add semantic retry loops.
**Alternative:** Extend recovery to regenerate domain-invalid decision sets with
bounded feedback. This may improve completion quality, but its ownership,
feedback contract, and cost budget need a separate concrete design.
### Initial chunk sizing and later prompt growth
**Recommendation:** Retain operator control over chunk size and choose reasonable
headroom for known stage prompts. Keep validation batching within fixed chunks;
report a later operation that cannot fit as skipped work. Establish the concrete
sizing rule during detailed design rather than invent a universal safety margin.
**Alternative:** Size initial chunks against conservative estimates for all
selected stage/profile combinations. This reduces later size failures but may
increase call volume and still cannot guarantee fit after arbitrary text growth.
### Exact event counts and recovery-rate denominators
**Recommendation:** Count errors once per failed stage/chunk unit, linking multiple
underlying failures as details. Count recoveries once per recovered operation;
aggregate defective-proposal filtering per proposal response with an affected
count. Report work-unit coverage separately from logical-call recovery counts,
with explicit denominators. Earlier recoveries may coexist with a later error in
the same unit.
**Alternative:** Count affected work units for both matrices, placing operation
counts only in details. This makes the two tables directly comparable but hides
multiple distinct recoveries within a single unit unless the reader drills down.
### Output placement, empty files, and report-write failures
**Recommendation:** Produce all three ordinary documents for every completed run,
using empty event lists where appropriate. Select a durable report directory
independent of debug cleanup, including when transcript output goes to stdout.
Specify its CLI/default path and overwrite policy before implementation. Attempt
partial reports on fatal failures when possible. Report-writing failures should
be explicit and should not erase an already produced transcript; whether they
make the invocation nonzero must be settled as part of the output contract.
**Alternative:** Write event files only when nonempty and identify their absence
explicitly in `report.md`. This reduces empty files but requires consumers to
understand missing-file semantics. Best-effort ancillary reporting could preserve
a zero exit status after transcript success, at the cost of weaker evidence that
the promised output bundle is complete.
### Existing report interfaces and debug configuration
**Recommendation:** Keep useful internal structures and measurements, adapting the
ordinary schema to its audience and moving verbose evidence behind debug.
Finalize a versioned YAML envelope, identifiers/index display, and the disposition
of `--report-json` and retention settings in detailed design. Use one debug boolean
with a CLI override and optional ConfigKit-backed setting; avoid a matrix of
independent artifact switches.
**Alternative:** Retain explicit machine-oriented JSON export alongside the new
ordinary outputs for consumers that need it. This preserves an existing interface
but adds another maintained presentation. Compatibility/export requirements have
not been selected for this feature; do not infer them from prior migration policy.
### Large Markdown reports
**Recommendation:** Use separate matrices with totals and reason summaries; omit
uneventful chunk rows when that improves readability, explicitly reporting the
omitted count. Show zero separately from unattempted work. Keep full incident
details in YAML and identify chunks by stable labels and segment ranges.
**Alternative:** Always include every chunk row, splitting wide tables into stage
groups. This gives an exhaustive visual grid but can make long transcripts harder
to review. Exact layout and any row-selection threshold remain undecided.
- Select exact CLI/configuration names and precedence for reporting enablement,
debug, and the report directory for stdout runs. Reports for file-output runs
belong beside the transcript; the stdout report-directory default is not yet set.
- Finalize YAML versions/fields, machine identifiers versus human chunk labels,
operation-volume denominators, deterministic record order, and failed/not-attempted
presentation without changing the agreed counting or fixed table layout.
- Specify how destinations are checked without destructive transcript truncation,
finalization order, path-collision handling, and debug placement/retention. Keep
ordinary filesystem behavior proportionate; do not add a transaction framework.