Record reliability sprint review findings

This commit is contained in:
2026-11-17 01:51:41 +00:00
parent e8c070805e
commit 087e388a03

150
docs/roadmap/remediation.md Normal file
View File

@@ -0,0 +1,150 @@
# Reliability and Maintainability Review Remediation
**Reviewed revision:** `e8c0708`
**Status:** Five open findings from the review of all 20
[implementation stages](implementation.md) against the
[target roadmap](reliability-and-maintainability.md).
## Review Result
The sprint established the target architecture. ConfigKit and native
CommandKit integrations handle configuration and inspection, and preflight
hands checked processing selections to the runner. Audita decodes and
normalizes transcripts directly and partitions them once. A bounded section
window allows proposal, validation, and repair to overlap within a stage.
The runner assigns final proposal indices in section order and commits edits
at the stage boundary. PromptKit owns model execution, backend admission,
retries, and repair history. Required publication uses checked destinations.
Ordinary reports use typed processing facts, while optional debug capture
retains interactions and decision history. CommandKit renders live progress.
The offline CLI build and full Go suite passed, as did targeted race checks
for the runner, LLM adapter, validators, report projection, diagnostics,
glossary, and CLI. The ordered repair artifact test also passed 150 repetitions
under the race detector. A built CLI returned status 2 with source diagnostics
for malformed PromptKit and application configuration across the delegated
commands and processing cases in the plan. Focused review probes reproduced
R1, R3, and R4. R2 and R5 follow from the versioned output contract and current
documentation.
## Findings
### R1 — Include Later Delivery Failures in Writable Reports (P2)
**Affected plan:** Stage 4, item 3, and Stage 17, item 1.
**Evidence:** [Report publication](../../internal/core/diagnostics/output_session.go#L311)
renders a failed bundle using the failures accumulated at that point. Later
writes append to the returned error list, but the remaining reports and the
best-effort pass reuse the same bytes. The
[report projection](../../internal/framework/processreport/failures.go#L41)
can include secondary file failures only if publication supplies them.
**Reproduced behavior:** Start finalization with a processing failure. Make
`errors.yml` and `recoveries.yml` unwritable and let `report.md` succeed.
The returned error contains the processing failure and both delivery failures.
The renderer receives only the processing failure, so the writable report
omits the two already-observed failures. The same stale-bundle problem can
follow a report error that first switches a successful run to failed status.
**Required change:** Use the freshest observed failures at the report
publication boundary and make a bounded best-effort update of already-written
reports when their failure evidence becomes stale. Retain existing evidence
if rendering the updated failed bundle fails. Keep transcript-first,
nontransactional publication.
**Acceptance:** Extend the output-session regression to inspect the facts
rendered into a surviving report when an earlier failure is followed by a
second write failure. Check failure attribution and counts as well as status.
### R2 — Preserve the Version 1 Error Record Discriminator (P2)
**Affected plan:** Stage 17, item 1.
**Evidence:** [Report projection](../../internal/framework/processreport/report.go#L108)
adds a top-level `scope: interrupted` record to the
[version 1 error document](../integrations/output-files.md#error-and-recovery-records).
The baseline contract described `work_unit` and `run` records. An exhaustive
reader of those variants encounters a third variant under the same version.
The target asks to retain peer attribution and existing counts and schema.
Audita's [release policy](../policy/release.md#versioning-and-compatibility)
also preserves the documented output contract for minor and patch releases.
**Required change:** Keep the existing top-level variants. Carry available
peer interruption details as optional evidence on an existing record, for
example an optional sequence on the initiating run failure. Keep the primary
origin and peer origins separately identifiable and preserve the work-unit and
run-error counts. Treat a new top-level variant as an explicit contract change
if that representation is preferred.
**Acceptance:** Exercise a fatal operation and interrupted peer in one run.
Check that a consumer of the existing record variants can still read the
report, and that the peer's available phase, reason, and origin remain present.
### R3 — Use the Same Missing-Reason Fallback in Run Reports (P3)
**Affected plan:** Stage 12, item 2, and Stage 17, item 1.
**Evidence:** [Operation projection](../../internal/framework/structuredoutput/failure.go#L42)
uses `internal_failure` when a typed model failure has no reason and retains
its original class and cause. [Run-failure projection](../../internal/framework/processreport/failures.go#L16)
recognizes a typed failure only when its reason is nonempty. It can then derive
a different reason from a wrapped cause or the processing phase.
**Reproduced behavior:** A typed fatal failure with an empty reason and a
wrapped `context.DeadlineExceeded` is recorded on the operation as
`fatal` / `internal_failure` and serialized in the run report as
`canceled`. The cause is available, but it does not make this a parent
cancellation.
**Required change:** Share the typed reason projection and its empty-reason
fallback. Derive a reason from the raw cause only when a typed failure is absent.
**Acceptance:** Verify matching fallbacks for the operation and the run record
and retain the phase, origin, and original cause. Keep genuine parent
cancellation distinct.
### R4 — Preserve Effective Transcript Members During Typed Decoding (P3)
**Affected plan:** Stage 5, item 1.
**Evidence:** The baseline selected the final value of an exact duplicate JSON
member when it decoded the segment into a generic object, before applying
field types. [Typed transcript decoding](../../internal/core/schema/transcript.go#L39)
now applies the field type to every occurrence. An earlier type error survives
even when the final value is valid. The refactor was meant to retain accepted
inputs while improving numeric fidelity.
**Reproduced behavior:** The baseline at `d226bdd` accepts
`[{"speaker":17,"speaker":"A","text":"hello","start":0,"end":1}]` and reads
speaker `A`. The sprint revision rejects the same input because of the
superseded numeric value.
**Required change:** Preserve the effective-member behavior for existing
inputs without reviving whole-transcript generic decoding or float conversion
for IDs. Keep the compatibility handling focused on this changed boundary.
**Acceptance:** Cover an earlier superseded type mismatch and a type mismatch
in the effective member. Keep the existing exact-ID and unknown-field tests
for both input envelopes.
### R5 — Align Roadmap Status and Runtime Guidance (P3)
**Affected plan:** Stages 15, 19, and 20.
**Evidence:** The [feature roadmap](reliability-and-maintainability.md) still
describes the implementation as pending despite the completed sprint.
The [pipeline guide](../internal/pipeline.md) says work-unit
evidence preserves each peer's interruption or error. The runner preserves
available outcomes and projects detailed work units according to output
selection. The [CLI guide](../cli.md#inspection-and-outcomes) contains a
literal `\n` between “selected” and “profiles.”
**Required change:** Mark the roadmap with the landed sprint and this review,
while retaining its historical baseline context. Describe interruption detail
in terms of outcomes actually available and selected for retention. Fix the
CLI sentence and keep public guidance focused on the operator-visible behavior.
**Acceptance:** The roadmap and implementation status agree, the pipeline guide
reflects the evidence modes, and the CLI paragraph renders normally.