Finalize resilience decisions and record deferred work
This commit is contained in:
51
docs/roadmap/future.md
Normal file
51
docs/roadmap/future.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# Deferred Feature Work
|
||||
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Intended future work; not implemented or scheduled. These entries
|
||||
preserve scope deferred from the [resilience roadmap](resilience.md). They are
|
||||
not prerequisites for completing that feature or staged implementation plans.
|
||||
|
||||
## Domain-Aware Response Recovery
|
||||
|
||||
Add bounded feedback and regeneration for model responses that satisfy the
|
||||
structural schema but violate Audita's domain response contract, such as missing,
|
||||
duplicate, or unexpected validator decision indices. The goal is to recover more
|
||||
usable work before excluding affected candidates, reducing the impact of an
|
||||
isolated malformed decision set.
|
||||
|
||||
This capability is desired but explicitly deferred from the current resilience
|
||||
work. That work retains existing PromptKit structural repair and safe filtering;
|
||||
unresolved domain-invalid decision sets produce errors and exclude their affected
|
||||
candidates while other candidates can succeed.
|
||||
|
||||
Before implementation, define the Audita/PromptKit ownership boundary, which
|
||||
checks are eligible for feedback, how feedback identifies defects without exposing
|
||||
secrets, and how regeneration fits bounded budgets and operation deadlines. Reuse
|
||||
shared library capabilities where appropriate; do not introduce an uncoordinated
|
||||
outer retry loop that multiplies existing retry/repair budgets.
|
||||
|
||||
Ordinary editorial rejection is a valid decision, not a reason to regenerate until
|
||||
an edit is approved. Recovery must preserve required validation, candidate identity,
|
||||
partial-success safety, and truthful error/recovery reporting. Validate with
|
||||
controlled responses that distinguish recoverable protocol defects from valid
|
||||
negative decisions. Exact APIs, budgets, and sequencing remain undecided.
|
||||
|
||||
## Independent Whole-Pipeline Traversal by Chunk
|
||||
|
||||
Explore allowing each fixed chunk to advance through its ordered stages without
|
||||
waiting for other chunks to finish the current stage. This could reduce waiting
|
||||
behind a slow chunk and improve utilization while preserving stage order within
|
||||
each chunk and deterministic final recombination.
|
||||
|
||||
The resilience feature deliberately retains stage barriers: finish a stage across
|
||||
chunks before advancing to the next stage. Fixed boundaries alone do not authorize
|
||||
independent traversal.
|
||||
|
||||
A future design must establish that stages do not depend on other chunks' evolving
|
||||
state, preserve stable identities and partial-success rules, and continue to share
|
||||
PromptKit's per-backend capacity across all roles and stages. Reporting must remain
|
||||
deterministic despite different completion orders, and fatal cancellation must
|
||||
still stop and join active work. Assess whether the scheduling change materially
|
||||
improves real workloads before introducing additional orchestration machinery.
|
||||
No new scheduler, global limiter, or execution mode is selected by this entry.
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Feature direction agreed; not implemented. This document records the
|
||||
intended end state and remaining product decisions. It is not a staged
|
||||
implementation plan. Recommendations under Open Questions are not adopted policy.
|
||||
**Status:** Feature direction agreed; not implemented. This document defines the
|
||||
target behavior and user intent, not a staged implementation plan. Remaining
|
||||
interface details are listed under Open Questions.
|
||||
|
||||
## Purpose and User Intent
|
||||
|
||||
@@ -12,23 +12,26 @@ Audita should reliably produce a useful transcript even when its LLM services
|
||||
cannot improve it. Exhausted retries or unusable model responses must not turn
|
||||
an otherwise processable transcript into a failed production run.
|
||||
|
||||
The production guarantee is:
|
||||
|
||||
> Failure of every LLM operation across every stage and chunk still produces a
|
||||
> successful output containing the deterministically normalized transcript, with
|
||||
> no LLM corrections applied. Partial LLM success preserves corrections that
|
||||
> completed successfully, subject to the selected work-unit commit boundary.
|
||||
> no LLM corrections applied. Partial success preserves as many fully validated,
|
||||
> applicable corrections as possible.
|
||||
|
||||
This does not promise the original input byte-for-byte. Deterministic
|
||||
normalization remains intentional and may change segment layout, identifiers,
|
||||
and text before LLM work begins. Output still uses the selected transcript
|
||||
encoder. There is no requirement to map corrections back to the original,
|
||||
pre-normalization segmentation.
|
||||
Deterministic normalization remains intentional and may change segment layout,
|
||||
identifiers, and text before LLM work begins. Output uses the selected transcript
|
||||
encoder; byte-for-byte preservation of the original input and mapping corrections
|
||||
back to pre-normalization segmentation are not requirements.
|
||||
|
||||
Existing reports contain useful information. Preserve meaningful identities,
|
||||
reason codes, correction/decision evidence, and execution measurements where
|
||||
appropriate; reorganize their classification and audience rather than replace
|
||||
everything with an unrelated reporting system.
|
||||
The overriding goal is to minimize how much useful processing is lost because of
|
||||
one problematic transcript segment. A stage/chunk work unit is an attribution
|
||||
boundary, not an all-or-nothing transaction.
|
||||
|
||||
Existing reports contain useful information, but there are no compatibility
|
||||
obligations for reporting or debug output. Reuse existing machinery only when it
|
||||
makes the implementation simpler and clearer; preservation is not itself a goal.
|
||||
Internal structures and measurements must serve Audita's processing workflow.
|
||||
The reporting package adapts those outcomes to structured and human-facing
|
||||
presentations, rather than reshaping processing around a reporting convenience.
|
||||
|
||||
## Current Baseline and Intended Change
|
||||
|
||||
@@ -44,268 +47,248 @@ Detailed artifacts are written during ordinary runs and may subsequently be
|
||||
removed by retention. The working transcript is re-chunked between modules.
|
||||
|
||||
This feature deliberately changes those behaviors. It builds on the completed
|
||||
[library adoption](library-adoption.md) and [review remediation](remediation.md);
|
||||
it does not undo their separation of library and application responsibilities.
|
||||
The existing fatal-cancellation mechanism remains useful, but model work failures
|
||||
must be separated from failures that actually require cancellation of the run.
|
||||
[library adoption](library-adoption.md) and [review remediation](remediation.md).
|
||||
Fatal cancellation remains necessary; unsuccessful model work must be separated
|
||||
from failures that actually require cancellation of the run.
|
||||
|
||||
## Fixed Chunks and Ordered Stages
|
||||
|
||||
Normalize first, then partition the transcript once. Maintain the same chunk
|
||||
boundaries, segment membership, segment identity, and ordering throughout LLM
|
||||
processing. Text inside a chunk evolves as accepted corrections are applied.
|
||||
Recombine the chunks in their original order after the LLM stages finish.
|
||||
Normalize first, then partition the transcript once. Preserve chunk boundaries,
|
||||
segment membership, segment identity, and ordering throughout LLM processing.
|
||||
Text evolves as accepted corrections are applied. Recombine chunks in original
|
||||
order after the LLM stages finish.
|
||||
|
||||
For this feature, a stage is a configured module instance. Repeated instances of
|
||||
the same module have distinct identities. Proposal generation and each validator
|
||||
are operations within a stage; validation may use multiple batches within a chunk.
|
||||
A work unit is one stage instance applied to one chunk.
|
||||
A stage is a configured module instance. Repeated instances of the same module
|
||||
have distinct identities. Proposal generation and each validator are operations
|
||||
within a stage; validation may use multiple batches within a chunk. A work unit
|
||||
is one stage instance applied to one chunk.
|
||||
|
||||
Preserve the overall stage sequence: finish a stage across its chunks before
|
||||
advancing to the next stage. Concurrent chunk work within a stage continues to
|
||||
share PromptKit's backend capacity. A skipped work unit does not prevent other
|
||||
chunks or later stages from executing.
|
||||
advancing to the next stage. Concurrent chunk work shares PromptKit's backend
|
||||
capacity. Failure in one unit does not prevent other chunks or later stages
|
||||
from executing.
|
||||
|
||||
Fixed chunking makes failure isolation and context stable, as well as making
|
||||
cross-stage reporting meaningful. It trades away adaptive repartitioning after
|
||||
edits and task-specific chunk sizes. Initial sizing should allow reasonable room
|
||||
for prompt overhead and text growth, without claiming that every later request
|
||||
will fit. An oversized later model request is an explicit skipped-work outcome;
|
||||
do not silently change chunk boundaries. Exact sizing policy remains open.
|
||||
Retain operator control over chunk size and the current defaults. Production
|
||||
experience indicates that chunk size affects model latency nonlinearly and that
|
||||
useful settings are well below backend context limits. Tuning is an empirical
|
||||
operator decision. Do not add automatic headroom calculations, cross-profile
|
||||
sizing, adaptive repartitioning, or automatic tuning in this feature. A later
|
||||
operation that cannot process its input follows the same safe error handling;
|
||||
it does not silently alter chunk boundaries. Validation batching within a chunk
|
||||
remains supported.
|
||||
|
||||
Whole-pipeline traversal by independently advancing chunks is explicitly deferred
|
||||
to a future roadmap. Fixed chunking does not authorize changing the stage barrier.
|
||||
Read-only neighboring context, cross-chunk edits, and transcript-wide restructuring
|
||||
are also outside this unit; fixed chunks are not a permanent product prohibition
|
||||
on such future capabilities.
|
||||
Independent whole-pipeline traversal by chunks and domain-aware response repair
|
||||
are explicitly deferred in [future work](future.md). Read-only neighboring
|
||||
context, cross-chunk edits, and transcript-wide restructuring are outside this
|
||||
unit, without becoming permanent prohibitions on future capabilities.
|
||||
|
||||
## Resilient Execution Contract
|
||||
## Partial Success and Failure Isolation
|
||||
|
||||
- Keep bounded transport retries, structural repairs, operation deadlines, and
|
||||
shared backend capacity. Completion comes from safe fallback, not unlimited
|
||||
attempts or an Audita retry loop around PromptKit's existing recovery.
|
||||
- Generate proposals, enforce deterministic/domain checks, and complete required
|
||||
LLM validation before applying a correction. Failure must never bypass validation
|
||||
or cause uncertain corrections to be accepted.
|
||||
- Unsuccessful model work preserves the affected work unit's incoming text and
|
||||
records an error. Preserve corrections committed by earlier stages and other
|
||||
successful work units. The handling of partially successful batches within a
|
||||
failing work unit is an explicit open question below.
|
||||
attempts or an outer retry loop around PromptKit.
|
||||
- Apply a correction only after it passes every required validator and the
|
||||
application checks. Failed validation must never be treated as approval.
|
||||
- If a validation batch fails, exclude its affected candidates while allowing
|
||||
other candidates to continue through the remaining validators. Preserve fully
|
||||
validated, applicable corrections from successful batches in the same unit.
|
||||
- Track candidates by stable proposal identity. Batch membership can change
|
||||
between validators; equal batch numbers do not establish candidate identity.
|
||||
- A failed proposal-generation operation leaves its chunk unchanged for that
|
||||
stage because there are no reliable proposals to salvage. Failures later in
|
||||
validation need not discard unrelated successful candidates.
|
||||
- Preserve earlier stages' corrections and other chunks' successful work.
|
||||
Record partial success explicitly; a unit can apply corrections and have errors.
|
||||
- Continue despite model-operation failures, including exhausted repairs/retries,
|
||||
unusable model decisions, provider/transport failures, deadlines, and admission
|
||||
failures. An operation can fail without consuming its entire attempt budget;
|
||||
budget exhaustion is not the sole definition of an error.
|
||||
- A valid empty proposal list or valid assessments rejecting every correction is
|
||||
a successful outcome. Correction count does not determine success or failure.
|
||||
unusable model decisions, provider/transport failures, local operation deadlines,
|
||||
and admission failures. An operation need not exhaust its attempt budget to fail.
|
||||
- A valid empty proposal list or valid assessments rejecting all corrections is
|
||||
successful processing. Correction count alone does not determine failure.
|
||||
- Keep safe filtering of defective proposals. Do not retry ordinary editorial
|
||||
rejections. For now, an unusable validator decision set excludes the affected
|
||||
candidates and records an error after existing permitted recovery. New bounded
|
||||
feedback/regeneration for domain-invalid responses belongs to future work.
|
||||
- Retain fatal treatment for invalid input/configuration, internal defects,
|
||||
explicit run cancellation, and inability to produce/write the transcript.
|
||||
Distinguish caller cancellation from a local model-operation timeout.
|
||||
- A completed run with skipped LLM work returns success and makes that degradation
|
||||
visible in its summary and error records. Fatal run failures remain nonzero.
|
||||
Caller cancellation is distinct from a local model-operation timeout.
|
||||
- Completed runs with model errors return success. Required report-write failures
|
||||
are an additional fatal output condition when reporting is enabled, as below.
|
||||
|
||||
PromptKit continues to own model transport, transient retries, structural repair,
|
||||
and admission. Audita owns work-unit outcomes, domain acceptance, fallback, and
|
||||
report classification. ConfigKit continues to own configuration resolution.
|
||||
Debugging must not change correction decisions, retry budgets, or continuation.
|
||||
PromptKit owns transport, transient retries, structural repair, and admission.
|
||||
Audita owns domain acceptance, partial outcomes, fallback, and classification.
|
||||
ConfigKit owns configuration resolution. Debug and reporting choices must not
|
||||
change correction decisions, retry budgets, or continuation.
|
||||
|
||||
## Errors, Recoveries, and Debug
|
||||
|
||||
Classify by eventual outcome and impact, not exception type or retry count alone.
|
||||
These are three reporting categories; debug is evidence, not a failure severity.
|
||||
Whether execution can continue is separate from the category.
|
||||
Classify by eventual outcome and impact. These are three reporting categories;
|
||||
debug is evidence, not a failure severity. Whether execution continues is a
|
||||
separate property.
|
||||
|
||||
| Category | Meaning | Representative cases |
|
||||
| --- | --- | --- |
|
||||
| Errors | Intended work could not complete reliably and was skipped, or a fatal run-level operation failed. | Exhausted output repair; unavailable provider; unusable validation decisions; operation deadline. |
|
||||
| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient provider failure recovered; defective proposals safely discarded. |
|
||||
| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompt/response content, individual attempts, detailed execution traces. |
|
||||
| Errors | Intended work could not complete reliably, or a fatal run-level operation failed. Affected candidates are excluded; other corrections can succeed. | Exhausted repair; unavailable provider; unusable validation decisions; operation deadline. |
|
||||
| Recoveries | A problem was handled successfully through retries, repair, or safe filtering. No operator intervention is implied. | Invalid output repaired; transient service failure recovered; defective proposals safely discarded. |
|
||||
| Debug | Detailed evidence of processing and ordinary decisions. | Proposals, applied corrections, validator decisions, prompts/responses, individual attempts, execution traces. |
|
||||
|
||||
Normal editorial disagreement belongs in debug: a validator correctly rejecting
|
||||
Normal editorial disagreement belongs in debug. A validator correctly rejecting
|
||||
a meaning-changing edit is expected behavior. A proposal referring to nonexistent
|
||||
original text is defective; safely filtering it is a recovery. A schema-valid
|
||||
validation response with missing or duplicate decision indices is an operation
|
||||
failure when it cannot be recovered. Do not label unresolved output as recovered.
|
||||
original text is defective; safely filtering it is a recovery. An unresolved
|
||||
missing/duplicate decision-index response is an error, not a recovery.
|
||||
|
||||
Record the eventual outcome of an operation once. Several unsuccessful attempts
|
||||
followed by success yield a consolidated recovery. Attempts ending in failure
|
||||
contribute to the error, not additional recoveries. Earlier operations that did
|
||||
recover remain recorded if a later operation fails the work unit.
|
||||
Record an operation's eventual outcome once. Several failed attempts followed by
|
||||
success produce a consolidated recovery. Attempts ending in failure contribute
|
||||
to error details, not additional recoveries. Earlier operations that recovered
|
||||
remain recorded if a later operation introduces an error in the same unit.
|
||||
|
||||
Separate output-quality recovery, defective-proposal filtering, and service
|
||||
recovery using structured reasons. Network instability should not be mistaken
|
||||
for a prompt deficiency. Preserve attempt counts and affected-proposal counts
|
||||
without counting each attempt or rejected proposal as an independent incident.
|
||||
recovery with structured reasons. Network instability must not look like a prompt
|
||||
defect. Do not introduce an arbitrary warning threshold or a fourth category.
|
||||
|
||||
Recovery frequency needs processing-volume context. Seven recoveries need not
|
||||
warrant attention; seventy may or may not, depending on the amount and kind of
|
||||
work. Do not introduce an arbitrary warning threshold or restore “warnings” as a
|
||||
fourth category. Final counting/denominator details remain open below.
|
||||
### Counting and Processing Volume
|
||||
|
||||
## Production Outputs and Debug Boundary
|
||||
- Count error-bearing work units once per stage/chunk, consolidating underlying
|
||||
operation failures as details. A partially successful unit still counts once.
|
||||
- Count recoveries once per recovered operation. Aggregate defective-proposal
|
||||
filtering per proposal response, retaining affected-proposal counts as detail.
|
||||
Attempts and individual discarded proposals are not independent incidents.
|
||||
- Keep work-unit coverage separate from operation/recovery counts. Show processing
|
||||
volume with explicitly labeled denominators; do not divide recovered-operation
|
||||
counts by work-unit counts and call that a recovery rate.
|
||||
- Report fatal run-level errors separately from the stage/chunk grid when they
|
||||
cannot be attributed to a work unit. Do not invent a chunk to accommodate them.
|
||||
- Derive YAML records, Markdown summaries, and CLI counts from actual processing
|
||||
outcomes with deterministic ordering, rather than independently calculating them.
|
||||
|
||||
The selected ordinary report set is:
|
||||
## Production Outputs, Placement, and Lifecycle
|
||||
|
||||
- `errors.yml`: structured records of unsuccessful work and applicable run failures.
|
||||
- `recoveries.yml`: structured records of successfully handled problems.
|
||||
- `report.md`: a human-facing summary of outcome, processing volume, applied
|
||||
corrections, errors, recoveries, and their locations in the pipeline.
|
||||
Reporting is enabled by default. The ordinary bundle is:
|
||||
|
||||
Use Audita-owned structured records as the common source for YAML, Markdown, and
|
||||
CLI summaries. Reuse useful existing report concepts rather than expose raw
|
||||
library structures or maintain independently calculated totals. Exact schemas
|
||||
are not finalized. They should identify the run, stage instance, stable chunk,
|
||||
operation, validator/batch where relevant, reason, outcome, and pertinent counts.
|
||||
Keep safe human-readable explanations alongside machine-readable reason codes.
|
||||
- `errors.yml`: structured error-bearing units, their underlying failures, and
|
||||
applicable run-level failures.
|
||||
- `recoveries.yml`: structured recovered operations and pertinent counts.
|
||||
- `report.md`: a concise human-facing summary of outcome, corrections, processing
|
||||
volume, errors, recoveries, and their locations in the pipeline.
|
||||
|
||||
Production issue records should be compact. Full responses, transcript excerpts,
|
||||
proposed/applied correction lists, and complete validation decisions belong in
|
||||
opt-in debug artifacts. Debug output may continue using JSON where suitable.
|
||||
Preserve configured-secret redaction across every presentation and artifact.
|
||||
Write all three documents, including empty event lists for uneventful completed
|
||||
runs. Use Audita-owned schemas with safe explanations and stable reason codes;
|
||||
retain run, stage-instance, chunk, operation, and validator/batch identities where
|
||||
applicable. Exact schemas are detailed-design work, not a reason to preserve the
|
||||
old report formats or expose raw library structures.
|
||||
|
||||
Provide an explicit debug opt-in, with `--debug` as the proposed CLI spelling.
|
||||
A configuration counterpart is desirable but its exact interface remains open.
|
||||
Debug-off must prevent detailed artifact persistence in the first place; writing
|
||||
those files and deleting them later does not satisfy the target. Temporary
|
||||
in-memory processing data needed for correct execution is not debug output.
|
||||
Place reports beside the output transcript. Retain transcript-to-stdout support
|
||||
with an explicit report-directory selection; its default is a remaining interface
|
||||
detail. Preserve the existing overwrite convention: a new run can replace files
|
||||
at its selected destinations. Do not add a mandatory fresh-directory restriction
|
||||
or compatibility exports.
|
||||
|
||||
Ordinary reports must survive independently of debug retention. The current
|
||||
`--report-json`, run-local report, ledger, execution/utilization artifacts, and
|
||||
retention settings need a deliberate disposition. Do not silently retain all
|
||||
existing verbose output as additional production defaults. Coordinate the
|
||||
retention decision with the existing [runtime follow-up](runtime-behavior.md).
|
||||
Before model work, create initial versions of all three reports carrying a common
|
||||
run identifier and explicit `in_progress` status (rendered as prose in Markdown).
|
||||
Fail immediately if initialization fails. Also check the transcript destination;
|
||||
creating neighboring reports does not prove that an existing transcript file is
|
||||
writable. Preflight must not prematurely destroy an existing transcript.
|
||||
|
||||
## Human-Facing Presentation
|
||||
Finalize all required outputs and check every final write. Initial write success
|
||||
cannot guarantee later success if space or permissions change. Failure to write
|
||||
the transcript or finalize a required report returns nonzero and must not erase
|
||||
an already produced transcript. Where possible, finalize available reports with
|
||||
an explicit failed status on fatal termination. Abrupt termination may leave
|
||||
`in_progress`; elaborate crash-recovery machinery is not required.
|
||||
|
||||
Provide a concise completion message showing corrections applied, work units
|
||||
skipped due to errors, and recoveries. For example, the intended style is:
|
||||
Provide an option to disable reporting entirely. It suppresses all three files
|
||||
(`errors.yml`, `recoveries.yml`, and `report.md`), including initialization and
|
||||
final writes, while retaining the brief CLI summary and independently selectable
|
||||
debug output. Disabled reporting must not create or require a writable report
|
||||
destination, and must not remove reports from earlier runs.
|
||||
Transcript output failures remain fatal regardless of reporting selection.
|
||||
|
||||
> Completed: 18 corrections applied; 3 work units skipped due to errors; 7 recoveries.
|
||||
Ordinary reports survive independently of debug cleanup. Reporting configuration
|
||||
and finalization must not depend on retaining a debug run directory.
|
||||
|
||||
Include report locations when available. Send the summary to stderr so stdout
|
||||
remains available exclusively for transcript output. Successful completion must
|
||||
not imply that every stage/chunk succeeded.
|
||||
## Debug and Internal Boundaries
|
||||
|
||||
`report.md` should help the reader locate concentrations of failures and recoveries
|
||||
without opening debug data. Stable chunks support the proposed presentation of
|
||||
separate error and recovery matrices: chunks as rows, configured stage instances
|
||||
as columns, and counts in each cell. Preserve execution order and distinguish
|
||||
repeated stages. Include totals and a short breakdown by reason.
|
||||
Production issue records are compact. Full responses, transcript excerpts,
|
||||
correction lists, and complete validation decisions are opt-in debug evidence.
|
||||
Use a single debug boolean, with `--debug` as the proposed CLI spelling and a
|
||||
ConfigKit-backed setting/override design to be finalized. Debug artifacts may use
|
||||
JSON where useful. Preserve configured-secret redaction in every presentation.
|
||||
|
||||
Distinguish completed work with zero events from work that was not attempted.
|
||||
Do not present canceled or unavailable work as cleanly completed. Exact table
|
||||
counting, omission of uneventful rows, and large-report layout remain open below.
|
||||
Debug-off prevents detailed artifact persistence; writing and later deleting
|
||||
those files is insufficient. In-memory domain state needed to process candidates
|
||||
correctly is not debug output. Debug and ordinary reporting are independent
|
||||
selections and do not alter processing behavior.
|
||||
|
||||
There is no obligation to retain `--report-json`, old artifact layouts, parallel
|
||||
exports, or old retention settings. Reuse code only where useful to the new
|
||||
workflow. Resolve the disposition of debug retention alongside the existing
|
||||
[runtime follow-up](runtime-behavior.md); the legacy `never` behavior must not
|
||||
quietly become the meaning of the new reporting-disable option.
|
||||
|
||||
## Deterministic Human-Facing Presentation
|
||||
|
||||
Use the CLI style:
|
||||
|
||||
> Completed: 18 corrections applied; 3 work units with errors; 7 recoveries.
|
||||
|
||||
Do not say that all error-bearing units were skipped: some apply valid corrections.
|
||||
Include report locations when reporting is enabled and the files exist. Send the
|
||||
summary to stderr so stdout remains exclusively available for transcript output.
|
||||
Disabled reporting does not suppress the summary or fatal-error messages.
|
||||
|
||||
Keep `report.md` concise and high-level. Use separate error and recovery matrices
|
||||
with stages as columns and chunks as rows. Include every chunk in fixed order;
|
||||
retain configured stage order and distinct labels for repeated module instances.
|
||||
No automatic width detection, thresholds, adaptive omission, or table splitting.
|
||||
A wide or long table is acceptable.
|
||||
|
||||
The error matrix counts error-bearing work units: each attempted cell is 0 or 1.
|
||||
Recovery cells count recovered operations and can exceed 1. An error cell may
|
||||
coexist with recoveries and successfully applied corrections in the same unit.
|
||||
Distinguish completed work with zero events from unattempted work; do not portray
|
||||
canceled/unavailable processing as clean completion. Include totals, processing
|
||||
volume, and a short reason breakdown. Detailed inspection belongs in YAML or debug.
|
||||
|
||||
## Target Acceptance Criteria
|
||||
|
||||
- A controlled run in which every LLM operation fails still returns success and
|
||||
the normally encoded normalized transcript, with no LLM corrections applied.
|
||||
- Mixed-success runs preserve committed corrections, skip unsuccessful work
|
||||
safely, and attempt subsequent stages and unaffected chunks.
|
||||
- Chunks retain their identity and boundaries across all stages; validation
|
||||
batching does not repartition them. Final recombination preserves ordering.
|
||||
- Recoverable model failures do not cancel unrelated work. Fatal failures and
|
||||
explicit cancellation still stop and join active work appropriately.
|
||||
- Valid no-change and ordinary editorial-rejection outcomes remain successful.
|
||||
Repaired/filtered output and unrecovered failures receive distinct classifications.
|
||||
- YAML records, Markdown tables, and CLI totals agree, with deterministic ordering
|
||||
and no duplicate incident counts from retries or repairs.
|
||||
- Debug-off produces the selected ordinary output set without detailed debugging
|
||||
artifacts; debug-on adds evidence without changing processing outcomes.
|
||||
- Tests exercise real runner/runtime/reporting paths with controlled providers,
|
||||
including exhausted recovery, partial success, and total LLM failure. Existing
|
||||
domain safeguards, capacity limits, redaction, and output contracts remain
|
||||
covered where they are retained.
|
||||
- On implementation, update the canonical CLI, configuration, operations,
|
||||
integration, internal, and architecture documents to describe the new behavior.
|
||||
This roadmap alone does not change their current contracts.
|
||||
- Total LLM failure still returns success and the encoded normalized transcript,
|
||||
provided required output writes succeed.
|
||||
- Mixed-success runs preserve independently validated, applicable corrections
|
||||
within the same stage/chunk as a failed batch. No candidate bypasses a required
|
||||
validator; later validators handle survivors by stable proposal identity.
|
||||
- Subsequent stages and unaffected chunks execute after recoverable failures.
|
||||
Fatal errors and explicit cancellation stop and join active work appropriately.
|
||||
- Chunk boundaries and identity remain fixed, with operator-selected sizing and
|
||||
unchanged defaults; final recombination preserves ordering.
|
||||
- Valid no-change/editorial-rejection outcomes are successful. Repaired/filtered
|
||||
output and unresolved failures receive distinct classifications and counts.
|
||||
- YAML, Markdown, and CLI agree on partial outcomes and totals; error matrices
|
||||
count affected work units, recovery matrices count recovered operations.
|
||||
- With reporting enabled, all three reports are initialized before model work,
|
||||
finalized and retained, including empty lists. Initial or final required-output
|
||||
write failures produce nonzero exits without destroying completed transcript output.
|
||||
- Reporting can be disabled without requiring its destination. Debug can be enabled
|
||||
independently; neither selection changes processing outcomes. Debug-off writes
|
||||
no detailed debug artifacts, and stdout transcript output remains supported.
|
||||
- Reports use the fixed deterministic layout without adaptive presentation logic.
|
||||
- Meaningful controlled-provider tests cover exhausted recovery, failed validation
|
||||
batches with successful survivors, total failure, fatal cancellation, output
|
||||
failures, and reporting/debug selections. Preserve domain safeguards, capacity
|
||||
limits, and redaction.
|
||||
- On implementation, update canonical CLI, configuration, operations, integration,
|
||||
internal, and architecture documentation. This roadmap does not change their
|
||||
current implemented contracts.
|
||||
|
||||
## Open Questions
|
||||
|
||||
### Work-unit atomicity when only some validation batches succeed
|
||||
The product choices above replace the earlier alternatives and user comments.
|
||||
Remaining details belong in the eventual decision-complete implementation plan:
|
||||
|
||||
**Recommendation:** Commit a stage/chunk unit only after all required operations
|
||||
complete reliably. If a required batch fails, discard that unit's pending edits,
|
||||
record one work-unit error with the underlying operation failures, and preserve
|
||||
its incoming text. Normal valid rejections are not operation failures. This gives
|
||||
predictable fallback and straightforward error counting.
|
||||
|
||||
**Alternative:** Preserve fully validated corrections from successful batches in
|
||||
an otherwise failing unit. This retains more useful work but requires explicit
|
||||
partial-success outcomes and clearer counts than “work unit skipped.”
|
||||
|
||||
### Domain-invalid responses and recovery opportunities
|
||||
|
||||
**Recommendation:** Continue safely filtering independently defective proposals;
|
||||
do not retry ordinary editorial rejections. Treat an unusable validator decision
|
||||
set as an error when existing permitted recovery cannot resolve it. Decide
|
||||
explicitly whether dynamic decision-index/cardinality checks should become part
|
||||
of the repair contract; this roadmap does not silently add semantic retry loops.
|
||||
|
||||
**Alternative:** Extend recovery to regenerate domain-invalid decision sets with
|
||||
bounded feedback. This may improve completion quality, but its ownership,
|
||||
feedback contract, and cost budget need a separate concrete design.
|
||||
|
||||
### Initial chunk sizing and later prompt growth
|
||||
|
||||
**Recommendation:** Retain operator control over chunk size and choose reasonable
|
||||
headroom for known stage prompts. Keep validation batching within fixed chunks;
|
||||
report a later operation that cannot fit as skipped work. Establish the concrete
|
||||
sizing rule during detailed design rather than invent a universal safety margin.
|
||||
|
||||
**Alternative:** Size initial chunks against conservative estimates for all
|
||||
selected stage/profile combinations. This reduces later size failures but may
|
||||
increase call volume and still cannot guarantee fit after arbitrary text growth.
|
||||
|
||||
### Exact event counts and recovery-rate denominators
|
||||
|
||||
**Recommendation:** Count errors once per failed stage/chunk unit, linking multiple
|
||||
underlying failures as details. Count recoveries once per recovered operation;
|
||||
aggregate defective-proposal filtering per proposal response with an affected
|
||||
count. Report work-unit coverage separately from logical-call recovery counts,
|
||||
with explicit denominators. Earlier recoveries may coexist with a later error in
|
||||
the same unit.
|
||||
|
||||
**Alternative:** Count affected work units for both matrices, placing operation
|
||||
counts only in details. This makes the two tables directly comparable but hides
|
||||
multiple distinct recoveries within a single unit unless the reader drills down.
|
||||
|
||||
### Output placement, empty files, and report-write failures
|
||||
|
||||
**Recommendation:** Produce all three ordinary documents for every completed run,
|
||||
using empty event lists where appropriate. Select a durable report directory
|
||||
independent of debug cleanup, including when transcript output goes to stdout.
|
||||
Specify its CLI/default path and overwrite policy before implementation. Attempt
|
||||
partial reports on fatal failures when possible. Report-writing failures should
|
||||
be explicit and should not erase an already produced transcript; whether they
|
||||
make the invocation nonzero must be settled as part of the output contract.
|
||||
|
||||
**Alternative:** Write event files only when nonempty and identify their absence
|
||||
explicitly in `report.md`. This reduces empty files but requires consumers to
|
||||
understand missing-file semantics. Best-effort ancillary reporting could preserve
|
||||
a zero exit status after transcript success, at the cost of weaker evidence that
|
||||
the promised output bundle is complete.
|
||||
|
||||
### Existing report interfaces and debug configuration
|
||||
|
||||
**Recommendation:** Keep useful internal structures and measurements, adapting the
|
||||
ordinary schema to its audience and moving verbose evidence behind debug.
|
||||
Finalize a versioned YAML envelope, identifiers/index display, and the disposition
|
||||
of `--report-json` and retention settings in detailed design. Use one debug boolean
|
||||
with a CLI override and optional ConfigKit-backed setting; avoid a matrix of
|
||||
independent artifact switches.
|
||||
|
||||
**Alternative:** Retain explicit machine-oriented JSON export alongside the new
|
||||
ordinary outputs for consumers that need it. This preserves an existing interface
|
||||
but adds another maintained presentation. Compatibility/export requirements have
|
||||
not been selected for this feature; do not infer them from prior migration policy.
|
||||
|
||||
### Large Markdown reports
|
||||
|
||||
**Recommendation:** Use separate matrices with totals and reason summaries; omit
|
||||
uneventful chunk rows when that improves readability, explicitly reporting the
|
||||
omitted count. Show zero separately from unattempted work. Keep full incident
|
||||
details in YAML and identify chunks by stable labels and segment ranges.
|
||||
|
||||
**Alternative:** Always include every chunk row, splitting wide tables into stage
|
||||
groups. This gives an exhaustive visual grid but can make long transcripts harder
|
||||
to review. Exact layout and any row-selection threshold remain undecided.
|
||||
- Select exact CLI/configuration names and precedence for reporting enablement,
|
||||
debug, and the report directory for stdout runs. Reports for file-output runs
|
||||
belong beside the transcript; the stdout report-directory default is not yet set.
|
||||
- Finalize YAML versions/fields, machine identifiers versus human chunk labels,
|
||||
operation-volume denominators, deterministic record order, and failed/not-attempted
|
||||
presentation without changing the agreed counting or fixed table layout.
|
||||
- Specify how destinations are checked without destructive transcript truncation,
|
||||
finalization order, path-collision handling, and debug placement/retention. Keep
|
||||
ordinary filesystem behavior proportionate; do not add a transaction framework.
|
||||
|
||||
Reference in New Issue
Block a user