narratio/docs/roadmap/implementation.md

Implementation Plan: Speaker Checkpoints and Replayable Uploads

Status: Ready; all stages pending.

Scope and Instructions

Implement the feature roadmap. This plan replaces the completed Notarius-profile plan; that work remains in Git history. Complete the numbered stages in order, one bounded implementation prompt per stage for gpt-5.6-terra. Record completion and actual verification results here. Intermediate stages are not a finished feature and must not be released alone.

Before working, read AGENTS.md, docs/development.md, all three policies in docs/policy/, the roadmap, and the stage-specific preparation below. Use the maintained Go 1.26 toolchain. Preserve operator choices, speaker filenames, bounded concurrency, streaming, and existing credentials/settings. Do not add new commands, dependencies, service requirements, arbitrary speaker limits, strict transcript schemas, or a generic checkpoint framework. No release or upstream service work is part of this plan.

Settled Design Decisions

Progress Representation and Recovery

Add optional transcribe_state_version (version 1) and transcribe_speakers to manifest.StageRecord. The latter is a map keyed by the existing derived speaker ID, containing completed results only. Absence means that work must be done; a separate pending/running/failed state machine is unnecessary. Aggregate stage status/error remains authoritative for invocation failure. A versioned empty collection is meaningful: no earlier completions are reusable.

Use a typed TranscribeSpeakerRecord with fingerprint version/digest, session-relative audio path, audio SHA-256 and size, one ArtifactRecord output of kind transcript_raw, output size, and producer run ID. The output carries its session-relative canonical path and SHA-256. No new transcript source IDs, synthetic schema contracts, timestamps, or historical attempt collections are needed. Existing aggregate diagnostics can retain attempt/status/duration data for work performed in the current invocation; do not fabricate that information for reused outputs.

Keep the session-manifest schema identifier unchanged for this optional additive state; validate supported per-stage versions and retain strict decoding. New Narratio reads old manifests. Old binaries may reject the added fields under their existing strict decoding; document this downgrade limitation.

For pre-feature manifests with no speaker evidence, regenerate transcription once when transcribe is next selected, even if the aggregate status succeeded. Old records do not bind individual transcript results to the audio bytes they used. Do not guess that association from current files, old per_file metadata, or abandoned run directories. This is the conservative upgrade path, not a new general migration system. No other stage or unselected range is forcibly run.

Reuse and Persistence

Compute a versioned per-speaker fingerprint from speaker ID, session-relative audio identity, actual audio SHA-256/size, and the existing transcribe semantic configuration fingerprint (URL and language). Do not include invocation ID, absolute workspace root, concurrency, retry settings, timeout, or credentials. Stream input hashing with cancellation; do not trust stored input checksums alone. Recheck input bytes before accepting a new result to catch changes observed during transcription; fail that job without checkpointing mismatched evidence. This is not a hostile-filesystem snapshot or concurrent-editor transaction guarantee and requires no extra audio copies or file locks.

Validate reusable output identity, canonical location, JSON, checksum, and size. Use existing confinement/file primitives. Missing or corrupt regular outputs become work; unsafe paths/non-regular conflicts follow existing safety errors, not recursive deletion. Missing required audio is an input error, not permission to reuse an unverifiable transcript. Removed inputs are removed from the current collection and aggregate outputs without deleting arbitrary files.

One stage-owned mutex serializes candidate collection construction, manifest save, and installation of committed state. Save a detached candidate and only then update live authority, following analyze's pattern. Workers must not mutate the live manifest outside that serialized section. Keep expensive HTTP work outside the mutex. Stop dispatch/acceptance on checkpoint failure, join workers before return, and preserve the last successfully committed collection.

Before launching requests, durably checkpoint the reconciled collection. For force, that collection is empty: this prevents later ordinary retries from resurrecting pre-force outputs. If this initial save fails, do not start requests or overwrite outputs. Force again clears the current collection again; ordinary retry retains replacements already checkpointed after the previous force.

Save each validated result at its canonical raw-transcript location before checkpointing it. Run-local files remain staging/diagnostics, not reuse dependencies. Bytes left by an unsuccessful checkpoint have no completion authority. A worker may checkpoint a completed result using the parent stage context even if a sibling canceled the work context; parent cancellation or a checkpoint failure may prevent that save. Guarantee retention of committed work, not all responses that happened to finish during cancellation.

Multipart Replay Mechanism

Replace the pipe/producer upload mechanism with a small pull-based multipart body backed by the audio reader. Use mime/multipart.Writer to encode a bounded metadata prefix and suffix around the file payload: capture the prefix after CreateFormFile, then encode the language field and closing boundary as the suffix. Generate this framing once per application attempt. Each body combines fresh readers for prefix, freshly opened audio, and suffix using io.MultiReader. No full-audio buffer or temporary multipart spool is needed.

The initial body and Request.GetBody use the same factory/framing. Each owns an independent audio handle. Give bodies idempotent Close and context-triggered closure (for example context.AfterFunc), without holding a mutex across a blocking Read. Read/Close coordination must release a blocked source and retain genuine source-read/close errors while not treating deliberate early closure as source failure. The attempt owns cleanup of all created bodies, including replays; reject new body creation after its context expires.

Remove the old producer Wait dependency. Track body-local source errors with synchronization so a swallowed upload read error cannot become a successful transcript, and a deliberately closed original body cannot poison a successful replay. Keep the outer retry policy and per-attempt context. Do not add replay counters to the user retry budget or depend on Go's exact internal retry count. Preserve ordinary HTTP redirect policy; body replay can also enable Go's body-preserving redirects. Do not introduce a new redirect restriction.

Stage 1: Add Durable Speaker Completion State

Status: Pending.

Preparation: internal/manifest/manifest.go, store.go, analyze_state.go, their tests, and docs/internal/manifest.md.

1. Implement the additive fields, record type, version constants, validation, and deep-copy helper described above in a focused transcribe state file.
2. Validate ownership by the transcribe stage, supported versions, required completed-record evidence, output kind, safe relative paths, SHA-256 values, sizes, and producer identity consistency. Use existing speaker IDs as-is; do not apply analysis-artifact key regexes or copy unrelated analysis limits.
3. Integrate validation with both load and save. Preserve speaker state through running/failed/stale transitions and aggregate result clearing. Generic stage invalidation does not discard checkpoints; reconciliation checks their data.
4. Test round-trip populated and versioned-empty collections, old manifests without fields, malformed evidence, deep-copy isolation, and lifecycle preservation. Avoid imposing JSON formatting or map order as contracts.

Verify: go test ./internal/manifest.

Done when durable storage can represent partial completion independently of aggregate status, without activating new stage behavior yet.

Stage 2: Implement Read-Only Speaker Reconciliation

Status: Pending.

Preparation: internal/stage/transcribe.go, semantic_contracts_initial.go, analyze_reconciliation.go, extract_resume.go, their tests, and the artifact path/integrity helpers they use.

1. Extract the existing audio discovery/job construction into a shared, transcribe-local reconciliation path. Preserve current input selection and speaker identity behavior. Build sorted jobs and classify completed results against the current input set and evidence defined above.
2. Add pure fingerprint construction and streamed input/output inspection. Resolve manifest input paths with existing session-relative/absolute reading helpers; do not change input precedence or assume the current working directory is the session root. Newly written checkpoint paths are always relative.
3. Return current records and work-to-do without writing files or manifest state. Force selects all work and an empty starting collection; old unversioned transcribe state supplies no reusable records. Do not connect aggregate skip validation until execution can produce the new evidence in Stage 3.
4. Test unchanged reuse; one changed audio among unchanged speakers; URL/language changes; operational-setting exclusions; added/removed speakers; missing and corrupt regular outputs; unsafe record paths; missing audio; and force. Prefer compact tables and real temporary files over mocked hashing.

Verify: go test ./internal/stage ./internal/manifest.

Done when one read-only decision path describes exactly which speakers need work, without altering existing runtime behavior prematurely.

Stage 3: Execute and Checkpoint Speakers Incrementally

Status: Pending.

Preparation: internal/stage/transcribe.go, analyze_execution_state.go, run_local.go, internal/app/runner.go, and transcribe/checkpoint test fixtures.

1. Wire reconciliation into Run, require the existing ManifestStore collaborator, and checkpoint the starting collection before launching HTTP work. Implement detached candidate saves and serialized live-state updates. Audit Store.Save's normalization side effects so a failed candidate save cannot mutate shared authority; use detached records/collections wherever it may write.
2. Retain bounded errgroup execution and fail-fast cancellation. Check the work context before beginning each queued request. After success, validate adapter output path/JSON, recheck audio evidence, materialize canonical bytes, compute output evidence, and checkpoint the new record. Check a checkpoint-failure latch under the serialization lock before accepting further completions.
3. Preserve committed successes on all error/cancellation exits, including starting-save, canonical-write, per-speaker-save, and aggregate-save failures. Return meaningful speaker/error context and let the runner retain its existing terminal failure behavior. Do not report partial aggregate outputs as success.
4. Assemble final outputs in deterministic speaker order from the reconciled completed records, including reused speakers, with original producer IDs and checksums. If all records are current but aggregate status is failed/running, finish without WhisperX calls and rebuild the aggregate success.
5. Implement transcribeStage.ValidateResume using the same read-only reconciliation, so a succeeded stage cannot skip changed audio or invalid transcripts. Retain the existing URL/language semantic check and runner invalidation behavior; no whole-config digest invalidation.
6. Adapt existing tests that require no canonical partial files: the new contract permits checkpointed files but forbids aggregate success. Update fixtures to provide real manifest stores and records rather than a production fallback that silently disables checkpointing.
7. Test A/B completion followed by C failure and D unattempted (concurrency 1), reload and retry only C/D; concurrently finishing speakers without lost saves; failure and cancellation after a checkpoint; save failure retaining earlier authority; and force failure followed by ordinary retry without old-result reuse.

Verify: go test ./internal/stage ./internal/app and go test -race ./internal/stage.

Done when incremental progress and aggregate success work together safely under ordinary, concurrent, canceled, and forced execution.

Stage 4: Verify Runner, Merge, and Durable Lifecycle Integration

Status: Pending.

Preparation: internal/app/runner.go, runner_runtime.go, analyze_checkpoint_recovery_test.go, runner_terminal_failure_test.go, restore_manifest_paths.go, internal/stage/merge.go, and existing publish, restore, and cleanup tests routed from docs/internal/overview.md.

1. Add application-level tests with real manifest persistence and fake WhisperX: failed stage across separate invocations; all speakers committed before a failed aggregate save; interrupted running state; force/reset persistence; and unchanged upstream/succeeded downstream invalidation behavior. Inject errors at durable boundaries, not every private method call.
2. Close discoverRawTranscripts' directory-scan fallback for versioned transcribe state: incomplete aggregate state is not usable, and completed state consumes only declared current aggregate outputs. Keep legacy/manual fallback behavior for manifests without the new state rather than banning previously allowed standalone workflows. Test that partial files and removed speaker files cannot leak into a versioned merge, including direct stage calls.
3. Audit manifest copies/projections, remote manifest publication, restored path handling, and cleanup. Preserve relative speaker paths on restore and verify they resolve under the destination session, not the producer machine. Reuse existing confinement validation, not a second path-policy layer.
4. Add focused round-trip coverage through existing publish/restore fixtures for speaker records. Restored outputs absent from the published selection are non-reusable locally; do not silently expand remote publication to every raw speaker or upload audio as a new default. Removing run directories must not remove canonical checkpointed transcripts or make reuse require run files.
5. Verify old succeeded manifests are readable and rerun once when transcribe is selected, then reuse normally. Keep old manifests usable for other commands and bounded ranges. Do not automatically import old unrecorded JSON files.

Verify: go test ./internal/app ./internal/stage ./internal/manifest and go test -race ./internal/app.

Done when the new authority survives runner and storage lifecycles and cannot expose incomplete transcripts downstream.

Stage 5: Implement Replayable Streaming Multipart Bodies

Status: Pending.

Preparation: internal/adapters/whisperx/http.go, http_test.go, docs/integrations/whisperx.md, and the installed Go net/http Request/GetBody contract. Keep this change inside the WhisperX adapter.

1. Implement the pull-based framing/body factory described above, using standard multipart encoding rather than hand-escaped MIME headers. Add no new payload cap or upload-size setting. Preserve file-before-language wire semantics and the existing form names.
2. Install the initial body and GetBody with the same per-attempt boundary. Track and close all opened bodies even if a custom transport returns early; make cleanup/context callbacks safe against overlapping reads and closes. A GetBody failure must be returned, not converted into an empty upload.
3. Replace pipe-specific Wait/error paths in doTranscribeAttempt with source-error handling appropriate to pull bodies. Genuine source errors remain visible; intentional closure during transport retry is not itself a source error. Keep response bounds, JSON validation, output installation, retry classification/delay, cancellation, and application attempt reporting intact.
4. Remove obsolete producer-only helpers rather than retaining two upload paths. Adapt tests tied to producer goroutines to assert streaming and resource release, which are the actual contracts.
5. Add focused body tests: original/replayed bytes parse identically; replay after partial consumption starts from the beginning; simultaneous bodies use independent handles; source failures propagate; Close/cancel releases a blocked reader without deadlock. Use synthetic audio and gated readers.

Verify: go test ./internal/adapters/whisperx and go test -race ./internal/adapters/whisperx.

Done when replay is available without whole-file buffering or producer-lifetime ambiguity, and all existing HTTP behavior tests still pass.

Stage 6: Verify Replay Through the HTTP Adapter Boundary

Status: Pending.

Preparation: Existing injectable HTTPClient/RoundTripper seam and local-server tests in internal/adapters/whisperx/http_test.go; app WhisperX wiring tests.

1. Add a deterministic transport-boundary regression: consume part of the original request, close it as a transport would, call GetBody, consume and parse the replay using the request Content-Type, then return valid JSON. Verify exact audio/language, installed output, and one application attempt. This must fail before replay support; merely checking non-nil GetBody is insufficient. Do not assert Go's internal retry count or require a flaky handcrafted HTTP/2 peer in the default suite.
2. Cover replay-open/read failure, early response, parent cancellation during replay, and disposal of all body handles. Reuse Stage 5 body coverage rather than repeat every low-level case. Prove an intentionally abandoned original body does not turn a valid replay response into failure.
3. Retain local HTTP tests for application retries, non-retryable responses, response limits, invalid JSON, and bounded streaming; exercise that an exhausted HTTP failure still leaves earlier speaker checkpoints reusable using the existing stage/app seams.
4. Record the verification boundary honestly: controlled transport replay proves Narratio's body contract, not recovery from every real proxy PROTOCOL_ERROR.

Verify: go test ./internal/adapters/whisperx ./internal/app ./internal/stage and race tests for changed packages.

Done when the replay defect is protected at the adapter boundary and the application retry/checkpoint behavior remains intact.

Stage 7: Document and Validate the Completed Feature

Status: Pending.

1. Update canonical transcribe and manifest internals for the collection, reconciliation, durable-reset force behavior, serialized saves, and aggregate gate. Remove obsolete all-speakers-before-canonical-write descriptions.
2. Update the WhisperX contract for replayable bounded streaming, application attempts versus transport replay, resource ownership, and the lack of an exactly-once server guarantee. Do not claim the remote protocol issue is fixed.
3. Update operations/troubleshooting with ordinary resume versus force, durable partial success, old-manifest one-time retranscription on selecting transcribe, downgrade limitations, and private-model-change guidance. No new CLI flags or promises of automatic historical-file recovery. Keep detailed wire and storage contracts in their canonical owners.
4. Review changes against the feature roadmap and the implementation-discipline guidance. Do not elevate selected storage encoding, internal ordering, test synchronization, or exact error wording into new product guarantees.
5. Run the full repository gates below; document results and update completion statuses only after they pass. Leave release preparation for a separate task.

go test ./...
go test -race ./...
go vet ./...
go build ./...
go test ./internal/doccheck
go test ./internal/config -run '^TestExamplesLoadAndValidate$'
git diff --check

Open Questions

None requiring user input. The agreed roadmap supplies the product decisions; this plan selects a completed-only manifest collection, conservative old-record handling, and pull-based replay as implementation choices. No stronger remote reliability guarantee, new operator restriction, or generic orchestration surface is proposed.