narratio/docs/roadmap/transcribe-checkpoints-and-replay.md

Resumable Speaker Transcription and Replayable Uploads

Status: Implemented.

Goal and User Intent

Preserve completed speaker transcriptions when another speaker fails, so an ordinary run or resume does not repeat expensive successful work. Separately, allow Go's HTTP transport to replay WhisperX uploads when it encounters a recoverable connection or stream failure.

The motivating incident exhausted four transcription attempts for one speaker after three other speakers had completed. The final error reported an HTTP/2 peer PROTOCOL_ERROR and an inability to replay the request body without Request.GetBody.

These are related reliability improvements, not a promise to eliminate all service failures. This document records the target state and scope. The completed implementation sequence is retained in Git history; current behavior is documented in the canonical guides linked below.

Pre-implementation Limitations

The transcribe stage writes successful responses into the invocation's run directory. It installs canonical transcripts and returns its per-file metadata only after every speaker succeeds. A failure therefore leaves no durable per-speaker progress that ordinary resume can use, even though some completed JSON files may remain on disk.

The WhisperX adapter already retries requests at the application level, reopening audio for each attempt. Its multipart request body is streamed through a pipe and has no replay callback. Transport-level recovery within an attempt cannot reconstruct a consumed body. Providing that capability addresses the replay limitation, not the underlying peer protocol error; persistent failures can still require service or proxy investigation.

Targeted End State

Durable Per-Speaker Completion

Each successfully transcribed speaker file is validated, saved, and recorded durably before that work is treated as reusable. The session manifest remains the progress authority. Follow the existing analyze-stage checkpoint pattern without turning speakers into pipeline stages or introducing a general task engine.

Ordinary execution reconciles the current prepared audio set with recorded speaker results. It reuses valid completed work and schedules failed, unattempted, missing, or obsolete results. A result is not reusable merely because a matching filename exists. Evidence binds it to the speaker/audio identity, audio content, Narratio-visible recognition settings, and the saved transcript's integrity.

The relevant recognition settings remain the WhisperX service URL and language. Retry count/delay, concurrency, timeout, credentials, and diagnostic locations remain operational settings rather than reasons to discard completed results. Changes to private service models or configuration remain an operator reason to force transcription; Narratio does not inspect the service's private state.

Changes to individual audio inputs require replacement of affected results, not automatic retranscription of unrelated unchanged speakers. New speakers need transcription; removed speakers do not remain in the current aggregate output set. Records must describe the inputs actually used, not grant reuse to old bytes under a changed identity.

Aggregate Completion, Failure, and Force

The overall stage succeeds only when every currently required speaker has a validated, durably recorded result. Incomplete per-speaker progress does not make the aggregate stage successful or authorize downstream consumption of an incomplete collection. Existing dependent-stage invalidation remains in effect.

Retain bounded concurrency and the current fail-fast behavior: a speaker failure cancels outstanding work. Results already checkpointed remain reusable after failure, cancellation, or process interruption. A later ordinary run or resume retries the remaining work; it does not need a new CLI command.

Checkpoint writes must be serialized so concurrently finishing speakers cannot overwrite each other's progress. A failed persistence operation must not advertise an unrecorded result as complete. Previously committed progress remains the recovery authority, and checkpoint errors must be surfaced rather than reported as successful transcription.

--force on transcribe regenerates all currently required speakers instead of reusing their prior completions. Successful replacements are checkpointed too, so an ordinary retry after a failed forced invocation can retain the newly completed work. Old results for speakers not successfully regenerated must not satisfy that unfinished forced replacement on the next ordinary retry.

This deliberately relaxes the current all-speakers-before-canonical-installation implementation: individual durable results may exist while the stage remains failed or incomplete. The all-speakers-before-stage-success boundary remains.

Replayable WhisperX Requests

Provide Request.GetBody so the HTTP transport can obtain a fresh body for the same request. Replay reopens the audio and reproduces the multipart boundary, filename, fields, and framing used by the original request. Independent bodies must not share a consumed reader or mutable read position.

Preserve bounded-memory streaming rather than buffering entire audio files in memory. Correctly close every original and replay body and its audio resources; preserve source-read errors and cancellation, including partial uploads and early server responses. Successful response handling must not accidentally wait on, or report errors solely from, an abandoned body instead of the replay that produced the response.

Keep the existing application-level retry count, delay, classification, and per-attempt timeout semantics. Transport replays occur within an application attempt and its context; they are not additional Narratio-configured attempts. Do not add another custom retry loop, silently increase retries, or disable HTTP/2 globally as a substitute for replay support.

The adapter still validates and installs only successful JSON responses. Request replay does not provide exactly-once server execution: a lost response can cause repeated transcription work, as can existing application retries. No new service-side idempotency protocol is required by this feature.

Required Change Areas

• Manifest and persistence: Add transcribe-owned per-speaker progress and evidence, with safe checkpoint updates and preservation through runner failure handling. Review manifest validation, copying, serialization, and existing publish/restore handling wherever the new records pass through those paths.
• Stage execution and reuse: Reconcile speaker records before scheduling, checkpoint validated completions, rebuild aggregate outputs from the current set, and account for partial progress in ordinary and forced execution. Review successful-stage reuse as well as failed-stage retries so stale audio or missing/corrupt transcript evidence is not bypassed by an aggregate skip.
• Artifact lifecycle: Use existing path, installation, integrity, and manifest mechanisms. Ensure checkpointed results do not depend on an ephemeral run directory surviving cleanup. Do not introduce a second authoritative ledger or infer completion by scanning abandoned runs.
• HTTP adapter: Make multipart bodies reproducible and correctly manage every replay's lifetime, without changing the WhisperX HTTP schema or requiring an upstream service change.
• Tests: Cover per-speaker recovery after a later failure, interruption and checkpoint failure; changed/missing inputs or outputs; force followed by an ordinary retry; concurrent successful completions; and aggregate downstream gating. Extend existing fixtures rather than duplicating analyze's complete test suite. Exercise actual body reconstruction after partial consumption, multipart equivalence, cancellation/resource cleanup, and existing bounded streaming and retry behavior—not just the presence of a callback. Default tests remain offline and use local/fake HTTP boundaries, never paid services.
• Documentation: Update the canonical stage guide, manifest guide, WhisperX contract, operations guide, and troubleshooting guide when implemented. Explain partial completion, ordinary retry versus force, and the distinction between request replay limitations and persistent peer protocol failures.

Compatibility and Scope Boundaries

Existing speaker identities, raw transcript formats, and the fixed pipeline order remain unchanged. Existing manifests remain readable, but must not be treated as containing per-speaker evidence they never recorded. Recovery relies on verified recorded evidence, not guessed associations between old transcripts and current audio. No general legacy-migration framework is part of the feature.

Automatic adoption of unrecorded outputs from previously failed runs is outside this feature. Such files may be recoverable after verifying their inputs, but their presence alone does not prove completion. No recovery/import command, speaker-selection CLI, historical invocation ledger, new service configuration discovery, or broader orchestration refactor is included.

The feature does not diagnose the remote service/proxy's underlying HTTP/2 problem, guarantee recovery from persistent protocol errors, or change the policy for continuing unrelated work after failure.

Completion Criteria

For speakers A, B, C, and D: if A and B are checkpointed, C fails, and D has not completed, ordinary resume retains A and B, retries C, and completes D. No downstream stage receives an incomplete successful transcription collection. Force replaces all four, while an ordinary retry can reuse checkpoints from that forced replacement attempt.

The HTTP transport can reconstruct the upload after consuming the original body, without whole-file memory buffering, lost source errors, or stranded upload resources. A persistent peer error still fails normally after the configured application attempts, without discarding checkpointed speakers.