Plan shared stage checkpoint mechanics
This commit is contained in:
@@ -1,416 +1,272 @@
|
|||||||
# Implementation Plan: Speaker Checkpoints and Replayable Uploads
|
# Implementation Plan: Shared Stage Checkpoint Mechanics
|
||||||
|
|
||||||
Status: Implementation complete; Stages 1-7 complete.
|
Status: Not started.
|
||||||
|
|
||||||
## Scope and Instructions
|
## Scope and Authority
|
||||||
|
|
||||||
|
Implement the agreed follow-up to v2.2.0: consolidate the thin checkpoint
|
||||||
|
mechanism shared by transcribe and analyze without combining their progress
|
||||||
|
models or changing operator behavior. There is no separate consolidation
|
||||||
|
roadmap in this checkout. This plan uses the accompanying discussion as its
|
||||||
|
scope: persistence is already centralized; candidate isolation and
|
||||||
|
save-before-promotion are the duplication to remove. The completed
|
||||||
|
[transcription roadmap](transcribe-checkpoints-and-replay.md) provides background,
|
||||||
|
not authorization to reimplement transcription or HTTP replay. The preceding
|
||||||
|
completed implementation plan remains in Git history.
|
||||||
|
|
||||||
Implement [the feature roadmap](transcribe-checkpoints-and-replay.md). This plan
|
|
||||||
replaces the completed Notarius-profile plan; that work remains in Git history.
|
|
||||||
Complete the numbered stages in order, one bounded implementation prompt per
|
Complete the numbered stages in order, one bounded implementation prompt per
|
||||||
stage for gpt-5.6-terra. Record completion and actual verification results here.
|
stage for gpt-5.6-terra. Record completion and actual verification results here.
|
||||||
Intermediate stages are not a finished feature and must not be released alone.
|
This planning pass changes documentation only; the steps below are for the
|
||||||
|
subsequent implementation sprint.
|
||||||
|
|
||||||
Before working, read `AGENTS.md`, `docs/development.md`, all three policies in
|
Before each stage, read `AGENTS.md`, `docs/development.md`, and all three
|
||||||
`docs/policy/`, the roadmap, and the stage-specific preparation below. Use the
|
documents in `docs/policy/`: architecture, documentation, and testing. Use
|
||||||
maintained Go 1.26 toolchain. Preserve operator choices, speaker filenames,
|
maintained Go 1.26. Prefer graph tools for code discovery; inspect current
|
||||||
bounded concurrency, streaming, and existing credentials/settings. Do not add
|
files if their index is stale. Read the stage-specific preparation below,
|
||||||
new commands, dependencies, service requirements, arbitrary speaker limits,
|
rather than unrelated historical roadmaps.
|
||||||
strict transcript schemas, or a generic checkpoint framework. No release or
|
|
||||||
upstream service work is part of this plan.
|
|
||||||
|
|
||||||
## Settled Design Decisions
|
## Target and Scope Boundaries
|
||||||
|
|
||||||
### Progress Representation and Recovery
|
The existing `internal/manifest.Store` and `LocalStore` remain the session
|
||||||
|
manifest persistence interface and implementation. Encoding, schemas,
|
||||||
|
validation, normalization, and atomic file replacement remain unchanged.
|
||||||
|
There is no new store, journal, file format, dependency, command, configuration,
|
||||||
|
or migration.
|
||||||
|
|
||||||
Add optional `transcribe_state_version` (version 1) and `transcribe_speakers`
|
One private helper in `internal/stage` owns candidate preparation, calling the
|
||||||
to `manifest.StageRecord`. The latter is a map keyed by the existing derived
|
existing store, and promoting the selected stage record only after successful
|
||||||
speaker ID, containing completed results only. Absence means that work must be
|
save. The existing typed analyze/transcribe checkpoint methods become thin
|
||||||
done; a separate pending/running/failed state machine is unnecessary. Aggregate
|
callers. This is execution plumbing: keep it beside the callers rather than
|
||||||
stage status/error remains authoritative for invocation failure. A versioned
|
moving orchestration into the storage package or exporting a reusable API.
|
||||||
empty collection is meaningful: no earlier completions are reusable.
|
|
||||||
|
|
||||||
Use a typed `TranscribeSpeakerRecord` with fingerprint version/digest,
|
Keep these responsibilities stage-owned:
|
||||||
session-relative audio path, audio SHA-256 and size, one `ArtifactRecord` output
|
|
||||||
of kind `transcript_raw`, output size, and producer run ID. The output carries
|
|
||||||
its session-relative canonical path and SHA-256. No new transcript source IDs,
|
|
||||||
synthetic schema contracts, timestamps, or historical attempt collections are
|
|
||||||
needed. Existing aggregate diagnostics can retain attempt/status/duration data
|
|
||||||
for work performed in the current invocation; do not fabricate that information
|
|
||||||
for reused outputs.
|
|
||||||
|
|
||||||
Keep the session-manifest schema identifier unchanged for this optional additive
|
- Transcribe's completed-only speaker map, evidence checks, bounded concurrency,
|
||||||
state; validate supported per-stage versions and retain strict decoding. New
|
mutex, checkpoint-failure latch, cancellation, pre-dispatch force reset,
|
||||||
Narratio reads old manifests. Old binaries may reject the added fields under
|
output installation, and aggregate repair.
|
||||||
their existing strict decoding; document this downgrade limitation.
|
- Analyze's dispositions, selection, dependencies, catalog updates, working and
|
||||||
|
committed collections, failure reconciliation, invalidation, and outputs.
|
||||||
|
- Each stage's decision whether/when to checkpoint, typed validation, contextual
|
||||||
|
errors, and post-save working-state updates.
|
||||||
|
- Runner aggregate transitions, session locking, terminal failure saves, and
|
||||||
|
fixed dependent-stage invalidation.
|
||||||
|
|
||||||
For pre-feature manifests with no speaker evidence, regenerate transcription
|
Do not introduce a common subtask model, scheduler, transaction manager, global
|
||||||
once when transcribe is next selected, even if the aggregate status succeeded.
|
mutex, automatic retry, additional checkpoint, or batching policy. Do not
|
||||||
Old records do not bind individual transcript results to the audio bytes they
|
migrate publish/restore, planning clones, cleanup, or ordinary runner saves:
|
||||||
used. Do not guess that association from current files, old `per_file` metadata,
|
their operations are not the demonstrated duplication. Leave HTTP replay and
|
||||||
or abandoned run directories. This is the conservative upgrade path, not a new
|
all adapters unchanged.
|
||||||
general migration system. No other stage or unselected range is forcibly run.
|
|
||||||
|
|
||||||
### Reuse and Persistence
|
## Selected Design
|
||||||
|
|
||||||
Compute a versioned per-speaker fingerprint from speaker ID, session-relative
|
### Shared Save-and-Promote Operation
|
||||||
audio identity, actual audio SHA-256/size, and the existing transcribe semantic
|
|
||||||
configuration fingerprint (URL and language). Do not include invocation ID,
|
|
||||||
absolute workspace root, concurrency, retry settings, timeout, or credentials.
|
|
||||||
Stream input hashing with cancellation; do not trust stored input checksums
|
|
||||||
alone. Recheck input bytes before accepting a new result to catch changes
|
|
||||||
observed during transcription; fail that job without checkpointing mismatched
|
|
||||||
evidence. This is not a hostile-filesystem snapshot or concurrent-editor
|
|
||||||
transaction guarantee and requires no extra audio copies or file locks.
|
|
||||||
|
|
||||||
Validate reusable output identity, canonical location, JSON, checksum, and size.
|
Add `internal/stage/checkpoint.go` with a private helper shaped as:
|
||||||
Use existing confinement/file primitives. Missing or corrupt regular outputs
|
|
||||||
become work; unsafe paths/non-regular conflicts follow existing safety errors,
|
|
||||||
not recursive deletion. Missing required audio is an input error, not permission
|
|
||||||
to reuse an unverifiable transcript. Removed inputs are removed from the current
|
|
||||||
collection and aggregate outputs without deleting arbitrary files.
|
|
||||||
|
|
||||||
One stage-owned mutex serializes candidate collection construction, manifest
|
```go
|
||||||
save, and installation of committed state. Save a detached candidate and only
|
func checkpointStageCollection(
|
||||||
then update live authority, following analyze's pattern. Workers must not mutate
|
ctx context.Context,
|
||||||
the live manifest outside that serialized section. Keep expensive HTTP work
|
store manifest.Store,
|
||||||
outside the mutex. Stop dispatch/acceptance on checkpoint failure, join workers
|
path string,
|
||||||
before return, and preserve the last successfully committed collection.
|
live *manifest.Manifest,
|
||||||
|
stageName string,
|
||||||
|
update func(*manifest.StageRecord) error,
|
||||||
|
) error
|
||||||
|
```
|
||||||
|
|
||||||
Before launching requests, durably checkpoint the reconciled collection. For
|
Names are implementation choices, not public contracts. Use a function and
|
||||||
force, that collection is empty: this prevents later ordinary retries from
|
small copying helpers, not a factory/options/interface framework. The two
|
||||||
resurrecting pre-force outputs. If this initial save fails, do not start requests
|
existing typed checkpoint methods are the only production callers. They pass
|
||||||
or overwrite outputs. Force again clears the current collection again; ordinary
|
the already-resolved manifest path; the helper does not discover paths or
|
||||||
retry retains replacements already checkpointed after the previous force.
|
inspect configuration.
|
||||||
|
|
||||||
Save each validated result at its canonical raw-transcript location before
|
The helper performs this sequence:
|
||||||
checkpointing it. Run-local files remain staging/diagnostics, not reuse
|
|
||||||
dependencies. Bytes left by an unsuccessful checkpoint have no completion
|
|
||||||
authority. A worker may checkpoint a completed result using the parent stage
|
|
||||||
context even if a sibling canceled the work context; parent cancellation or a
|
|
||||||
checkpoint failure may prevent that save. Guarantee retention of committed
|
|
||||||
work, not all responses that happened to finish during cancellation.
|
|
||||||
|
|
||||||
### Multipart Replay Mechanism
|
1. Check required live/store inputs without mutation. Copy a checkpoint
|
||||||
|
candidate and select a copied target record. If absent, provide an empty
|
||||||
|
candidate record to the caller, without inferring its name, status, or time.
|
||||||
|
2. Apply the supplied update to that candidate record only. On update error,
|
||||||
|
return without saving or promoting. The typed caller validates and fills
|
||||||
|
its version, collection, and stage-owned initialization/timestamps.
|
||||||
|
3. Invoke `Store.Save` once with the supplied context. Preserve error identity
|
||||||
|
through `%w` and useful path context. On error, do not promote, retry,
|
||||||
|
reload, or synthesize aggregate failure.
|
||||||
|
4. After success, install the saved target record and candidate manifest
|
||||||
|
`UpdatedAt` into live state. Preserve an existing target record's pointer
|
||||||
|
identity by assigning through it. Allocate missing live maps/records only
|
||||||
|
after success. Do not replace the entire live manifest or overwrite other
|
||||||
|
stage records. Copy mutable checkpoint collections on promotion so saved
|
||||||
|
candidates and caller working collections cannot mutate live progress.
|
||||||
|
|
||||||
Replace the pipe/producer upload mechanism with a small pull-based multipart
|
Callers serialize the whole operation. Transcribe's current mutex does so;
|
||||||
body backed by the audio reader. Use `mime/multipart.Writer` to encode a bounded
|
analyze executes artifacts sequentially. Do not add a lock to the helper or
|
||||||
metadata prefix and suffix around the file payload: capture the prefix after
|
promise support for concurrent independent manifest writers. Atomic replacement
|
||||||
`CreateFormFile`, then encode the language field and closing boundary as the
|
visibility does not become power-loss durability or a transaction covering
|
||||||
suffix. Generate this framing once per application attempt. Each body combines
|
transcript files.
|
||||||
fresh readers for prefix, freshly opened audio, and suffix using `io.MultiReader`.
|
|
||||||
No full-audio buffer or temporary multipart spool is needed.
|
|
||||||
|
|
||||||
The initial body and `Request.GetBody` use the same factory/framing. Each owns
|
### Bounded Candidate Isolation
|
||||||
an independent audio handle. Give bodies idempotent Close and context-triggered
|
|
||||||
closure (for example `context.AfterFunc`), without holding a mutex across a
|
|
||||||
blocking Read. Read/Close coordination must release a blocked source and retain
|
|
||||||
genuine source-read/close errors while not treating deliberate early closure as
|
|
||||||
source failure. The attempt owns cleanup of all created bodies, including
|
|
||||||
replays; reject new body creation after its context expires.
|
|
||||||
|
|
||||||
Remove the old producer `Wait` dependency. Track body-local source errors with
|
Factor `cloneManifestForTranscribeCheckpoint` into a shared checkpoint-scoped
|
||||||
synchronization so a swallowed upload read error cannot become a successful
|
copy helper in the new file. Use ordinary typed copies and existing
|
||||||
transcript, and a deliberately closed original body cannot poison a successful
|
`manifest.CloneAnalyzeArtifactCollection` and
|
||||||
replay. Keep the outer retry policy and per-attempt context. Do not add replay
|
`manifest.CloneTranscribeSpeakerCollection`. Do not introduce reflection
|
||||||
counters to the user retry budget or depend on Go's exact internal retry count.
|
cloning, JSON round trips, or a public general-purpose clone API.
|
||||||
Preserve ordinary HTTP redirect policy; body replay can also enable Go's
|
|
||||||
body-preserving redirects. Do not introduce a new redirect restriction.
|
|
||||||
|
|
||||||
## Stage 1: Add Durable Speaker Completion State
|
Copy the manifest value, stages map, stage records, and mutable collections
|
||||||
|
touched by these updates or `LocalStore.Save`. In particular, Save normalizes
|
||||||
|
analyze collections throughout the candidate, including on a transcribe
|
||||||
|
checkpoint. Clone those collections for every stage so normalization cannot
|
||||||
|
leak into live state on failure. Preserve the existing bounded copies of
|
||||||
|
input/artifact and output/log/config slices. Retain nil entries for existing
|
||||||
|
validation to reject, rather than silently repairing invalid manifests.
|
||||||
|
|
||||||
Status: Complete.
|
This is not arbitrary deep isolation of metadata. Neither the two update
|
||||||
|
callbacks nor the current Store mutates `map[string]any` metadata, so it may
|
||||||
|
remain shared read-only. Document that boundary and inspect every current
|
||||||
|
normalization write. Use existing typed clones for nested checkpoint evidence;
|
||||||
|
do not restrict legal metadata values or alter their Go/JSON representations
|
||||||
|
to simplify copying.
|
||||||
|
|
||||||
Completion marker (2026-09-21): Added optional versioned transcribe-owned
|
### Differences to Preserve
|
||||||
per-speaker completion records, strict load/save validation, isolated collection
|
|
||||||
cloning, legacy aggregate-only compatibility, and lifecycle-preservation
|
|
||||||
coverage. Verified with `go test ./internal/manifest` and `git diff --check`.
|
|
||||||
|
|
||||||
Preparation: `internal/manifest/manifest.go`, `store.go`, `analyze_state.go`,
|
- Analyze still requires an existing stage record. Its callback changes only
|
||||||
their tests, and `docs/internal/manifest.md`.
|
analyze version/collection, not transcribe-style initialization or timestamp
|
||||||
|
policy. It refreshes working and committed collections only after success.
|
||||||
|
- Transcribe retains missing-record initialization for direct stage calls:
|
||||||
|
its typed caller supplies name, running status, and initial timestamps when
|
||||||
|
needed. Its callback retains the existing stage `UpdatedAt` update. Copying
|
||||||
|
an existing record must not clear aggregate fields or invoke lifecycle
|
||||||
|
transitions.
|
||||||
|
- Existing transcribe live-record initialization is incomplete on the unusual
|
||||||
|
missing-record path; promoting the actual successfully saved candidate makes
|
||||||
|
its identity agree with disk. This does not add a new missing-record
|
||||||
|
rejection or change ordinary runner status transitions.
|
||||||
|
- Analyze rollback/failure handling and transcribe failure latching remain in
|
||||||
|
their execution owners. The shared operation never invents failed items.
|
||||||
|
- Preserve starting-checkpoint elision, force/reset semantics, empty versioned
|
||||||
|
collections, evidence and producer identity, legacy transcription handling,
|
||||||
|
supported zero-sized audio, and aggregate repair without HTTP calls.
|
||||||
|
- Do not freeze exact error wording, wall-clock values, copy choreography, or
|
||||||
|
iteration order into new contracts. No new schema, limits, filename rules,
|
||||||
|
statuses, or compatibility guarantees are authorized.
|
||||||
|
|
||||||
1. Implement the additive fields, record type, version constants, validation,
|
## Stage 1: Add the Shared Checkpoint Operation
|
||||||
and deep-copy helper described above in a focused transcribe state file.
|
|
||||||
2. Validate ownership by the transcribe stage, supported versions, required
|
|
||||||
completed-record evidence, output kind, safe relative paths, SHA-256 values,
|
|
||||||
sizes, and producer identity consistency. Use existing speaker IDs as-is;
|
|
||||||
do not apply analysis-artifact key regexes or copy unrelated analysis limits.
|
|
||||||
3. Integrate validation with both load and save. Preserve speaker state through
|
|
||||||
running/failed/stale transitions and aggregate result clearing. Generic stage
|
|
||||||
invalidation does not discard checkpoints; reconciliation checks their data.
|
|
||||||
4. Test round-trip populated and versioned-empty collections, old manifests
|
|
||||||
without fields, malformed evidence, deep-copy isolation, and lifecycle
|
|
||||||
preservation. Avoid imposing JSON formatting or map order as contracts.
|
|
||||||
|
|
||||||
Verify: `go test ./internal/manifest`.
|
Status: Pending.
|
||||||
|
|
||||||
Done when durable storage can represent partial completion independently of
|
Preparation: `internal/manifest/store.go`, `manifest.go`, `analyze_state.go`,
|
||||||
aggregate status, without activating new stage behavior yet.
|
`transcribe_state.go`; `internal/stage/analyze_execution_state.go`,
|
||||||
|
`transcribe_execution_state.go`; `docs/internal/manifest.md`.
|
||||||
|
|
||||||
## Stage 2: Implement Read-Only Speaker Reconciliation
|
1. Add the private helper and bounded candidate copying described above.
|
||||||
|
Leave existing production checkpoint paths unchanged until their migration
|
||||||
|
stages. Do not duplicate manifest record-validation rules.
|
||||||
|
2. Add focused `internal/stage/checkpoint_test.go` coverage with real
|
||||||
|
`manifest.LocalStore` and temporary directories for successful save/reload.
|
||||||
|
Use a small Store wrapper only to observe/fail the persistence boundary.
|
||||||
|
3. Prove live progress remains old during Save, matches saved progress after
|
||||||
|
success, and does not change on update, validation, or save failure. Include
|
||||||
|
both typed collections and a transcribe checkpoint alongside analyze
|
||||||
|
records subject to save-time normalization. A failure test must exercise
|
||||||
|
candidate timestamp/normalization mutation before returning an error, not
|
||||||
|
merely fail before any work.
|
||||||
|
4. Prove unrelated stage/aggregate data survives, existing target references
|
||||||
|
observe updates, missing-target creation is invisible on failure, and later
|
||||||
|
caller collection edits do not mutate committed records. Preserve context
|
||||||
|
and error identity. Test the supported copy/write boundary, not arbitrary
|
||||||
|
malicious Store mutation.
|
||||||
|
|
||||||
Status: Complete.
|
Verify: `go test ./internal/manifest ./internal/stage` and
|
||||||
|
|
||||||
Completion marker (2026-09-21): Extracted shared deterministic transcribe job
|
|
||||||
construction and added read-only, cancellation-aware per-speaker input and
|
|
||||||
output reconciliation. Covered unchanged reuse, semantic and operational
|
|
||||||
settings, changed/added/removed inputs, legacy and forced work, missing/corrupt
|
|
||||||
outputs, unsafe evidence, and missing audio without connecting runtime skip or
|
|
||||||
checkpoint behavior. Verified with
|
|
||||||
`go test ./internal/stage ./internal/manifest` and `git diff --check`.
|
|
||||||
|
|
||||||
Preparation: `internal/stage/transcribe.go`, `semantic_contracts_initial.go`,
|
|
||||||
`analyze_reconciliation.go`, `extract_resume.go`, their tests, and the artifact
|
|
||||||
path/integrity helpers they use.
|
|
||||||
|
|
||||||
1. Extract the existing audio discovery/job construction into a shared,
|
|
||||||
transcribe-local reconciliation path. Preserve current input selection and
|
|
||||||
speaker identity behavior. Build sorted jobs and classify completed results
|
|
||||||
against the current input set and evidence defined above.
|
|
||||||
2. Add pure fingerprint construction and streamed input/output inspection.
|
|
||||||
Resolve manifest input paths with existing session-relative/absolute reading
|
|
||||||
helpers; do not change input precedence or assume the current working directory
|
|
||||||
is the session root. Newly written checkpoint paths are always relative.
|
|
||||||
3. Return current records and work-to-do without writing files or manifest state.
|
|
||||||
Force selects all work and an empty starting collection; old unversioned
|
|
||||||
transcribe state supplies no reusable records. Do not connect aggregate skip
|
|
||||||
validation until execution can produce the new evidence in Stage 3.
|
|
||||||
4. Test unchanged reuse; one changed audio among unchanged speakers; URL/language
|
|
||||||
changes; operational-setting exclusions; added/removed speakers; missing and
|
|
||||||
corrupt regular outputs; unsafe record paths; missing audio; and force.
|
|
||||||
Prefer compact tables and real temporary files over mocked hashing.
|
|
||||||
|
|
||||||
Verify: `go test ./internal/stage ./internal/manifest`.
|
|
||||||
|
|
||||||
Done when one read-only decision path describes exactly which speakers need
|
|
||||||
work, without altering existing runtime behavior prematurely.
|
|
||||||
|
|
||||||
## Stage 3: Execute and Checkpoint Speakers Incrementally
|
|
||||||
|
|
||||||
Status: Complete.
|
|
||||||
|
|
||||||
Completion marker (2026-09-21): Connected reconciliation to bounded execution,
|
|
||||||
durably checkpointed the starting and per-speaker collections through detached
|
|
||||||
manifest candidates, preserved committed work across failures and cancellation,
|
|
||||||
rebuilt deterministic aggregate outputs from reusable evidence, and added
|
|
||||||
speaker-aware resume validation. Updated production-style fixtures and covered
|
|
||||||
serialized concurrent saves, force/retry replacement, input rechecks, save
|
|
||||||
failures, partial recovery, and aggregate reconstruction. Verified with
|
|
||||||
`go test ./internal/stage ./internal/app`, `go test -race ./internal/stage`, and
|
|
||||||
`git diff --check`.
|
|
||||||
|
|
||||||
Preparation: `internal/stage/transcribe.go`, `analyze_execution_state.go`,
|
|
||||||
`run_local.go`,
|
|
||||||
`internal/app/runner.go`, and transcribe/checkpoint test fixtures.
|
|
||||||
|
|
||||||
1. Wire reconciliation into Run, require the existing ManifestStore collaborator,
|
|
||||||
and checkpoint the starting collection before launching HTTP work. Implement
|
|
||||||
detached candidate saves and serialized live-state updates. Audit Store.Save's
|
|
||||||
normalization side effects so a failed candidate save cannot mutate shared
|
|
||||||
authority; use detached records/collections wherever it may write.
|
|
||||||
2. Retain bounded errgroup execution and fail-fast cancellation. Check the work
|
|
||||||
context before beginning each queued request. After success, validate adapter
|
|
||||||
output path/JSON, recheck audio evidence, materialize canonical bytes, compute
|
|
||||||
output evidence, and checkpoint the new record. Check a checkpoint-failure
|
|
||||||
latch under the serialization lock before accepting further completions.
|
|
||||||
3. Preserve committed successes on all error/cancellation exits, including
|
|
||||||
starting-save, canonical-write, per-speaker-save, and aggregate-save failures.
|
|
||||||
Return meaningful speaker/error context and let the runner retain its existing
|
|
||||||
terminal failure behavior. Do not report partial aggregate outputs as success.
|
|
||||||
4. Assemble final outputs in deterministic speaker order from the reconciled
|
|
||||||
completed records, including reused speakers, with original producer IDs and
|
|
||||||
checksums. If all records are current but aggregate status is failed/running,
|
|
||||||
finish without WhisperX calls and rebuild the aggregate success.
|
|
||||||
5. Implement `transcribeStage.ValidateResume` using the same read-only
|
|
||||||
reconciliation, so a succeeded stage cannot skip changed audio or invalid
|
|
||||||
transcripts. Retain the existing URL/language semantic check and runner
|
|
||||||
invalidation behavior; no whole-config digest invalidation.
|
|
||||||
6. Adapt existing tests that require no canonical partial files: the new contract
|
|
||||||
permits checkpointed files but forbids aggregate success. Update fixtures to
|
|
||||||
provide real manifest stores and records rather than a production fallback
|
|
||||||
that silently disables checkpointing.
|
|
||||||
7. Test A/B completion followed by C failure and D unattempted (concurrency 1),
|
|
||||||
reload and retry only C/D; concurrently finishing speakers without lost saves;
|
|
||||||
failure and cancellation after a checkpoint; save failure retaining earlier
|
|
||||||
authority; and force failure followed by ordinary retry without old-result reuse.
|
|
||||||
|
|
||||||
Verify: `go test ./internal/stage ./internal/app` and
|
|
||||||
`go test -race ./internal/stage`.
|
`go test -race ./internal/stage`.
|
||||||
|
|
||||||
Done when incremental progress and aggregate success work together safely under
|
Done when the shared integrity boundary is tested without changing existing
|
||||||
ordinary, concurrent, canceled, and forced execution.
|
stage behavior. Temporary copying overlap is removed in Stage 3.
|
||||||
|
|
||||||
## Stage 4: Verify Runner, Merge, and Durable Lifecycle Integration
|
## Stage 2: Migrate Analyze
|
||||||
|
|
||||||
Status: Complete.
|
Status: Pending.
|
||||||
|
|
||||||
Completion marker (2026-09-22): Added real-persistence application coverage for
|
Preparation: `internal/stage/analyze_execution_state.go`,
|
||||||
cross-invocation retry, interrupted running state, failed aggregate saves,
|
`analyze_checkpoint_test.go`, `analyze_failure_test.go`,
|
||||||
force/reset persistence, downstream invalidation, and one-time legacy upgrade.
|
`internal/app/analyze_checkpoint_recovery_test.go`, and
|
||||||
Versioned merge input discovery now requires aggregate success and an exact
|
`docs/internal/stage-analyze.md`.
|
||||||
match between current speaker evidence and declared aggregate outputs, while
|
|
||||||
legacy/manual directory fallback remains available. Restore coverage preserves
|
|
||||||
session-relative speaker paths, leaves unpublished raw transcripts absent, and
|
|
||||||
canonical transcripts remain reusable after run-directory cleanup. Verified
|
|
||||||
with `go test ./internal/app ./internal/stage ./internal/manifest`,
|
|
||||||
`go test -race ./internal/app`, and `git diff --check`.
|
|
||||||
|
|
||||||
Preparation: `internal/app/runner.go`, `runner_runtime.go`,
|
1. Replace manual candidate copying, saving, and live promotion in analyze
|
||||||
`analyze_checkpoint_recovery_test.go`, `runner_terminal_failure_test.go`,
|
with the shared helper. Retain typed validation, required-record checking,
|
||||||
`restore_manifest_paths.go`, `internal/stage/merge.go`, and existing publish,
|
the typed update callback, and committed/working collection refresh.
|
||||||
restore, and cleanup tests routed from `docs/internal/overview.md`.
|
2. Preserve starting-checkpoint elision, selected-artifact behavior, catalog
|
||||||
|
sequencing, dependent invalidation on changed output, and rollback after
|
||||||
|
checkpoint failure. Do not change the runner's terminal failure save or
|
||||||
|
joined errors.
|
||||||
|
3. Run existing tests before adding coverage. Retain tests proving earlier
|
||||||
|
artifacts survive failure/cancellation, unsaved output bytes gain no
|
||||||
|
authority, selected/forced work preserves appropriate unrelated artifacts,
|
||||||
|
and interrupted runs recover across store reloads. Add only missing
|
||||||
|
integration coverage; do not duplicate Stage 1's isolation matrix.
|
||||||
|
|
||||||
1. Add application-level tests with real manifest persistence and fake WhisperX:
|
Verify: `go test ./internal/stage ./internal/app ./internal/manifest` and
|
||||||
failed stage across separate invocations; all speakers committed before a
|
`go test -race ./internal/stage ./internal/app`.
|
||||||
failed aggregate save; interrupted running state; force/reset persistence;
|
|
||||||
and unchanged upstream/succeeded downstream invalidation behavior. Inject
|
|
||||||
errors at durable boundaries, not every private method call.
|
|
||||||
2. Close `discoverRawTranscripts`' directory-scan fallback for versioned
|
|
||||||
transcribe state: incomplete aggregate state is not usable, and completed
|
|
||||||
state consumes only declared current aggregate outputs. Keep legacy/manual
|
|
||||||
fallback behavior for manifests without the new state rather than banning
|
|
||||||
previously allowed standalone workflows. Test that partial files and removed
|
|
||||||
speaker files cannot leak into a versioned merge, including direct stage calls.
|
|
||||||
3. Audit manifest copies/projections, remote manifest publication, restored path
|
|
||||||
handling, and cleanup. Preserve relative speaker paths on restore and verify
|
|
||||||
they resolve under the destination session, not the producer machine. Reuse
|
|
||||||
existing confinement validation, not a second path-policy layer.
|
|
||||||
4. Add focused round-trip coverage through existing publish/restore fixtures for
|
|
||||||
speaker records. Restored outputs absent from the published selection are
|
|
||||||
non-reusable locally; do not silently expand remote publication to every raw
|
|
||||||
speaker or upload audio as a new default. Removing run directories must not
|
|
||||||
remove canonical checkpointed transcripts or make reuse require run files.
|
|
||||||
5. Verify old succeeded manifests are readable and rerun once when transcribe is
|
|
||||||
selected, then reuse normally. Keep old manifests usable for other commands
|
|
||||||
and bounded ranges. Do not automatically import old unrecorded JSON files.
|
|
||||||
|
|
||||||
Verify: `go test ./internal/app ./internal/stage ./internal/manifest` and
|
Done when analyze delegates persistence choreography while progress, artifact
|
||||||
`go test -race ./internal/app`.
|
selection, and failure behavior remain unchanged.
|
||||||
|
|
||||||
Done when the new authority survives runner and storage lifecycles and cannot
|
## Stage 3: Migrate Transcribe and Remove Duplicate Machinery
|
||||||
expose incomplete transcripts downstream.
|
|
||||||
|
|
||||||
## Stage 5: Implement Replayable Streaming Multipart Bodies
|
Status: Pending.
|
||||||
|
|
||||||
Status: Complete.
|
Preparation: `internal/stage/transcribe_execution_state.go`, `transcribe.go`,
|
||||||
|
`transcribe_resume.go`, `transcribe_checkpoint_test.go`,
|
||||||
|
`transcribe_reconciliation_test.go`,
|
||||||
|
`internal/app/transcribe_checkpoint_recovery_test.go`, and
|
||||||
|
`docs/internal/stage-transcribe.md`.
|
||||||
|
|
||||||
Completion marker (2026-09-22): Replaced the producer-goroutine/pipe upload
|
1. Delegate `checkpointCollectionLocked` to the shared helper, retaining
|
||||||
with bounded standard-library multipart framing and independently opened,
|
typed initialization/update and successful working-collection refresh.
|
||||||
pull-based streaming bodies shared by the initial request and `GetBody`.
|
Keep the existing mutex across candidate creation, save, and promotion.
|
||||||
Bodies are tracked, idempotently closed, canceled with their attempt context,
|
Do not move HTTP/hashing work into that critical section or add a lock.
|
||||||
and retain genuine read/close failures without treating an intentionally
|
2. Preserve failure-latch checks and worker joining, the parent versus worker
|
||||||
abandoned original as a replay failure. Added focused coverage for identical
|
context distinction, force's durable empty starting collection, and
|
||||||
framing, partial-read restart, simultaneous handles, source failures, and
|
ordinary retry's reuse of committed forced replacements.
|
||||||
blocked-reader release. Verified with `go test ./internal/adapters/whisperx`,
|
3. Remove `cloneManifestForTranscribeCheckpoint`. Its current use by a
|
||||||
`go test -race ./internal/adapters/whisperx` (including ten repeated runs), and
|
read-only resume test is not a reason to retain duplicate production code.
|
||||||
`git diff --check`.
|
Use an independent test snapshot or the shared bounded copy where sufficient,
|
||||||
|
without exporting a helper for tests.
|
||||||
|
4. Retain existing real-store and concurrency coverage: no lost completions;
|
||||||
|
pre-dispatch save failure prevents requests/canonical writes; later failure
|
||||||
|
retains prior authority; interrupted and aggregate-save recovery; failed
|
||||||
|
force followed by retry; run-directory independence; zero-sized audio;
|
||||||
|
removed/mismatched aggregate repair without additional WhisperX requests
|
||||||
|
or removed-speaker merge input. Do not weaken these regression assertions.
|
||||||
|
|
||||||
Preparation: `internal/adapters/whisperx/http.go`, `http_test.go`,
|
Verify: `go test ./internal/stage ./internal/app ./internal/manifest` and
|
||||||
`docs/integrations/whisperx.md`, and the installed Go net/http Request/GetBody
|
`go test -race ./internal/stage ./internal/app`.
|
||||||
contract. Keep this change inside the WhisperX adapter.
|
|
||||||
|
|
||||||
1. Implement the pull-based framing/body factory described above, using standard
|
Done when both stages use one save/promote implementation and one bounded
|
||||||
multipart encoding rather than hand-escaped MIME headers. Add no new payload
|
copy implementation, with their different models and synchronization intact.
|
||||||
cap or upload-size setting. Preserve file-before-language wire semantics and
|
|
||||||
the existing form names.
|
|
||||||
2. Install the initial body and GetBody with the same per-attempt boundary.
|
|
||||||
Track and close all opened bodies even if a custom transport returns early;
|
|
||||||
make cleanup/context callbacks safe against overlapping reads and closes.
|
|
||||||
A GetBody failure must be returned, not converted into an empty upload.
|
|
||||||
3. Replace pipe-specific Wait/error paths in doTranscribeAttempt with source-error
|
|
||||||
handling appropriate to pull bodies. Genuine source errors remain visible;
|
|
||||||
intentional closure during transport retry is not itself a source error.
|
|
||||||
Keep response bounds, JSON validation, output installation, retry
|
|
||||||
classification/delay, cancellation, and application attempt reporting intact.
|
|
||||||
4. Remove obsolete producer-only helpers rather than retaining two upload paths.
|
|
||||||
Adapt tests tied to producer goroutines to assert streaming and resource
|
|
||||||
release, which are the actual contracts.
|
|
||||||
5. Add focused body tests: original/replayed bytes parse identically; replay
|
|
||||||
after partial consumption starts from the beginning; simultaneous bodies use
|
|
||||||
independent handles; source failures propagate; Close/cancel releases a
|
|
||||||
blocked reader without deadlock. Use synthetic audio and gated readers.
|
|
||||||
|
|
||||||
Verify: `go test ./internal/adapters/whisperx` and
|
## Stage 4: Document, Review Scope, and Validate
|
||||||
`go test -race ./internal/adapters/whisperx`.
|
|
||||||
|
|
||||||
Done when replay is available without whole-file buffering or producer-lifetime
|
Status: Pending.
|
||||||
ambiguity, and all existing HTTP behavior tests still pass.
|
|
||||||
|
|
||||||
## Stage 6: Verify Replay Through the HTTP Adapter Boundary
|
Preparation: `docs/internal/manifest.md`, `docs/internal/stage-transcribe.md`,
|
||||||
|
`docs/internal/stage-analyze.md`, documentation/testing policies, and the
|
||||||
|
implementation diff against this plan's starting commit.
|
||||||
|
|
||||||
Status: Complete.
|
1. Update manifest internals to identify the shared checkpoint owner and copy/
|
||||||
|
save/promote boundary. Stage guides retain stage-specific policy and link
|
||||||
Completion marker (2026-09-22): Added a deterministic transport regression
|
to that explanation instead of duplicating shared implementation details.
|
||||||
that abandons a partially consumed original request, obtains and parses a fresh
|
Do not rewrite immutable release notes or announce new operator behavior.
|
||||||
multipart replay from `GetBody`, and installs the returned transcript within one
|
2. Assess tests collectively. Consolidate genuinely duplicated persistence
|
||||||
application attempt. Covered replay open/read failures, early responses,
|
mechanics coverage, but retain stage/application recovery tests proving
|
||||||
parent cancellation during replay, and disposal of every opened source, while
|
correct integration. Avoid test-count/LOC quotas. Record a brief summary
|
||||||
proving an abandoned original does not poison a valid replay. Added a real HTTP
|
here of duplicated mechanisms removed and thin wrappers retained.
|
||||||
retry-exhaustion integration that durably retains an earlier speaker and resumes
|
3. Confirm no schema, validation policy, scheduling, force, concurrency, output
|
||||||
only the uncommitted speakers. This controlled boundary verifies Narratio's
|
selection, stage order, adapter call, or configuration change slipped into
|
||||||
replay contract, not recovery from every real proxy protocol failure. Verified
|
the refactor. Additional restrictions require a separate decision, not
|
||||||
with `go test ./internal/adapters/whisperx ./internal/app ./internal/stage`,
|
tests that retrospectively declare new policy.
|
||||||
`go test -race ./internal/adapters/whisperx ./internal/app ./internal/stage`,
|
4. Run all gates below and record actual results. Mark this plan complete only
|
||||||
ten repeated race-enabled adapter runs, and `git diff --check`.
|
when both migrations and verification pass. Do not release, tag, push, delete
|
||||||
|
other roadmaps, or expand into a repository-wide cloning refactor.
|
||||||
Preparation: Existing injectable HTTPClient/RoundTripper seam and local-server
|
|
||||||
tests in `internal/adapters/whisperx/http_test.go`; app WhisperX wiring tests.
|
|
||||||
|
|
||||||
1. Add a deterministic transport-boundary regression: consume part of the
|
|
||||||
original request, close it as a transport would, call GetBody, consume and
|
|
||||||
parse the replay using the request Content-Type, then return valid JSON.
|
|
||||||
Verify exact audio/language, installed output, and one application attempt.
|
|
||||||
This must fail before replay support; merely checking non-nil GetBody is
|
|
||||||
insufficient. Do not assert Go's internal retry count or require a flaky
|
|
||||||
handcrafted HTTP/2 peer in the default suite.
|
|
||||||
2. Cover replay-open/read failure, early response, parent cancellation during
|
|
||||||
replay, and disposal of all body handles. Reuse Stage 5 body coverage rather
|
|
||||||
than repeat every low-level case. Prove an intentionally abandoned original
|
|
||||||
body does not turn a valid replay response into failure.
|
|
||||||
3. Retain local HTTP tests for application retries, non-retryable responses,
|
|
||||||
response limits, invalid JSON, and bounded streaming; exercise that an
|
|
||||||
exhausted HTTP failure still leaves earlier speaker checkpoints reusable
|
|
||||||
using the existing stage/app seams.
|
|
||||||
4. Record the verification boundary honestly: controlled transport replay proves
|
|
||||||
Narratio's body contract, not recovery from every real proxy PROTOCOL_ERROR.
|
|
||||||
|
|
||||||
Verify: `go test ./internal/adapters/whisperx ./internal/app ./internal/stage`
|
|
||||||
and race tests for changed packages.
|
|
||||||
|
|
||||||
Done when the replay defect is protected at the adapter boundary and the
|
|
||||||
application retry/checkpoint behavior remains intact.
|
|
||||||
|
|
||||||
## Stage 7: Document and Validate the Completed Feature
|
|
||||||
|
|
||||||
Status: Complete.
|
|
||||||
|
|
||||||
Completion marker (2026-09-22): Updated the canonical transcribe, merge,
|
|
||||||
manifest, WhisperX, operations, and troubleshooting documentation for durable
|
|
||||||
speaker reconciliation, serialized checkpoints, aggregate gating, durable
|
|
||||||
force reset, legacy upgrade and downgrade limits, replayable streaming bodies,
|
|
||||||
resource ownership, and the absence of an exactly-once remote guarantee.
|
|
||||||
Reviewed the completed implementation against the feature roadmap and
|
|
||||||
implementation-discipline guidance. Verified with `go test ./...`,
|
|
||||||
`go test -race ./...`, `go vet ./...`, `go build ./...`,
|
|
||||||
`go test ./internal/doccheck`,
|
|
||||||
`go test ./internal/config -run '^TestExamplesLoadAndValidate$'`, and
|
|
||||||
`git diff --check`.
|
|
||||||
|
|
||||||
1. Update canonical transcribe and manifest internals for the collection,
|
|
||||||
reconciliation, durable-reset force behavior, serialized saves, and aggregate
|
|
||||||
gate. Remove obsolete all-speakers-before-canonical-write descriptions.
|
|
||||||
2. Update the WhisperX contract for replayable bounded streaming, application
|
|
||||||
attempts versus transport replay, resource ownership, and the lack of an
|
|
||||||
exactly-once server guarantee. Do not claim the remote protocol issue is fixed.
|
|
||||||
3. Update operations/troubleshooting with ordinary resume versus force, durable
|
|
||||||
partial success, old-manifest one-time retranscription on selecting transcribe,
|
|
||||||
downgrade limitations, and private-model-change guidance. No new CLI flags
|
|
||||||
or promises of automatic historical-file recovery. Keep detailed wire and
|
|
||||||
storage contracts in their canonical owners.
|
|
||||||
4. Review changes against the feature roadmap and the implementation-discipline
|
|
||||||
guidance. Do not elevate selected storage encoding, internal ordering, test
|
|
||||||
synchronization, or exact error wording into new product guarantees.
|
|
||||||
5. Run the full repository gates below; document results and update completion
|
|
||||||
statuses only after they pass. Leave release preparation for a separate task.
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go test ./...
|
go test ./...
|
||||||
@@ -422,10 +278,12 @@ go test ./internal/config -run '^TestExamplesLoadAndValidate$'
|
|||||||
git diff --check
|
git diff --check
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Done when duplication is removed, recovery remains covered, documentation
|
||||||
|
matches the implementation, and all gates pass.
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
None requiring user input. The agreed roadmap supplies the product decisions;
|
None requiring a product decision. A private stage-level helper is selected
|
||||||
this plan selects a completed-only manifest collection, conservative old-record
|
because persistence already has an owner and the duplication is execution
|
||||||
handling, and pull-based replay as implementation choices. No stronger remote
|
plumbing. Model unification, library extraction, generalized transactions, and
|
||||||
reliability guarantee, new operator restriction, or generic orchestration
|
broader cloning cleanup remain out of scope.
|
||||||
surface is proposed.
|
|
||||||
|
|||||||
@@ -15,9 +15,9 @@ peer `PROTOCOL_ERROR` and an inability to replay the request body without
|
|||||||
`Request.GetBody`.
|
`Request.GetBody`.
|
||||||
|
|
||||||
These are related reliability improvements, not a promise to eliminate all
|
These are related reliability improvements, not a promise to eliminate all
|
||||||
service failures. This document owns the target state and scope; the
|
service failures. This document records the target state and scope. The
|
||||||
[implementation plan](implementation.md) owns ordered changes and concrete
|
completed implementation sequence is retained in Git history; current behavior
|
||||||
storage and replay decisions.
|
is documented in the canonical guides linked below.
|
||||||
|
|
||||||
## Pre-implementation Limitations
|
## Pre-implementation Limitations
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user