Plan resumable transcription and replayable uploads

docs/roadmap/transcribe-checkpoints-and-replay.md

@@ -0,0 +1,183 @@
+# Resumable Speaker Transcription and Replayable Uploads
+
+Status: Proposed; not implemented.
+
+## Goal and User Intent
+
+Preserve completed speaker transcriptions when another speaker fails, so an
+ordinary run or resume does not repeat expensive successful work. Separately,
+allow Go's HTTP transport to replay WhisperX uploads when it encounters a
+recoverable connection or stream failure.
+
+The motivating incident exhausted four transcription attempts for one speaker
+after three other speakers had completed. The final error reported an HTTP/2
+peer `PROTOCOL_ERROR` and an inability to replay the request body without
+`Request.GetBody`.
+
+These are related reliability improvements, not a promise to eliminate all
+service failures. This document owns the target state and scope; the
+[implementation plan](implementation.md) owns ordered changes and concrete
+storage and replay decisions.
+
+## Current Limitations
+
+The [transcribe stage](../internal/stage-transcribe.md) writes successful
+responses into the invocation's run directory. It installs canonical transcripts
+and returns its per-file metadata only after every speaker succeeds. A failure
+therefore leaves no durable per-speaker progress that ordinary resume can use,
+even though some completed JSON files may remain on disk.
+
+The [WhisperX adapter](../integrations/whisperx.md) already retries requests at
+the application level, reopening audio for each attempt. Its multipart request
+body is streamed through a pipe and has no replay callback. Transport-level
+recovery within an attempt cannot reconstruct a consumed body. Providing that
+capability addresses the replay limitation, not the underlying peer protocol
+error; persistent failures can still require service or proxy investigation.
+
+## Targeted End State
+
+### Durable Per-Speaker Completion
+
+Each successfully transcribed speaker file is validated, saved, and recorded
+durably before that work is treated as reusable. The session manifest remains
+the progress authority. Follow the existing analyze-stage checkpoint pattern
+without turning speakers into pipeline stages or introducing a general task
+engine.
+
+Ordinary execution reconciles the current prepared audio set with recorded
+speaker results. It reuses valid completed work and schedules failed,
+unattempted, missing, or obsolete results. A result is not reusable merely
+because a matching filename exists. Evidence binds it to the speaker/audio
+identity, audio content, Narratio-visible recognition settings, and the saved
+transcript's integrity.
+
+The relevant recognition settings remain the WhisperX service URL and language.
+Retry count/delay, concurrency, timeout, credentials, and diagnostic locations
+remain operational settings rather than reasons to discard completed results.
+Changes to private service models or configuration remain an operator reason
+to force transcription; Narratio does not inspect the service's private state.
+
+Changes to individual audio inputs require replacement of affected results,
+not automatic retranscription of unrelated unchanged speakers. New speakers
+need transcription; removed speakers do not remain in the current aggregate
+output set. Records must describe the inputs actually used, not grant reuse to
+old bytes under a changed identity.
+
+### Aggregate Completion, Failure, and Force
+
+The overall stage succeeds only when every currently required speaker has a
+validated, durably recorded result. Incomplete per-speaker progress does not
+make the aggregate stage successful or authorize downstream consumption of an
+incomplete collection. Existing dependent-stage invalidation remains in effect.
+
+Retain bounded concurrency and the current fail-fast behavior: a speaker
+failure cancels outstanding work. Results already checkpointed remain reusable
+after failure, cancellation, or process interruption. A later ordinary run or
+resume retries the remaining work; it does not need a new CLI command.
+
+Checkpoint writes must be serialized so concurrently finishing speakers cannot
+overwrite each other's progress. A failed persistence operation must not
+advertise an unrecorded result as complete. Previously committed progress remains
+the recovery authority, and checkpoint errors must be surfaced rather than
+reported as successful transcription.
+
+`--force` on transcribe regenerates all currently required speakers instead of
+reusing their prior completions. Successful replacements are checkpointed too,
+so an ordinary retry after a failed forced invocation can retain the newly
+completed work. Old results for speakers not successfully regenerated must not
+satisfy that unfinished forced replacement on the next ordinary retry.
+
+This deliberately relaxes the current all-speakers-before-canonical-installation
+implementation: individual durable results may exist while the stage remains
+failed or incomplete. The all-speakers-before-stage-success boundary remains.
+
+### Replayable WhisperX Requests
+
+Provide `Request.GetBody` so the HTTP transport can obtain a fresh body for the
+same request. Replay reopens the audio and reproduces the multipart boundary,
+filename, fields, and framing used by the original request. Independent bodies
+must not share a consumed reader or mutable read position.
+
+Preserve bounded-memory streaming rather than buffering entire audio files in
+memory. Correctly close every original and replay body and its audio resources;
+preserve source-read errors and cancellation, including partial uploads and
+early server responses. Successful response handling must not accidentally wait
+on, or report errors solely from, an abandoned body instead of the replay that
+produced the response.
+
+Keep the existing application-level retry count, delay, classification, and
+per-attempt timeout semantics. Transport replays occur within an application
+attempt and its context; they are not additional Narratio-configured attempts.
+Do not add another custom retry loop, silently increase retries, or disable
+HTTP/2 globally as a substitute for replay support.
+
+The adapter still validates and installs only successful JSON responses.
+Request replay does not provide exactly-once server execution: a lost response
+can cause repeated transcription work, as can existing application retries.
+No new service-side idempotency protocol is required by this feature.
+
+## Required Change Areas
+
+- **Manifest and persistence:** Add transcribe-owned per-speaker progress and
+  evidence, with safe checkpoint updates and preservation through runner failure
+  handling. Review manifest validation, copying, serialization, and existing
+  publish/restore handling wherever the new records pass through those paths.
+- **Stage execution and reuse:** Reconcile speaker records before scheduling,
+  checkpoint validated completions, rebuild aggregate outputs from the current
+  set, and account for partial progress in ordinary and forced execution.
+  Review successful-stage reuse as well as failed-stage retries so stale audio
+  or missing/corrupt transcript evidence is not bypassed by an aggregate skip.
+- **Artifact lifecycle:** Use existing path, installation, integrity, and manifest
+  mechanisms. Ensure checkpointed results do not depend on an ephemeral run
+  directory surviving cleanup. Do not introduce a second authoritative ledger
+  or infer completion by scanning abandoned runs.
+- **HTTP adapter:** Make multipart bodies reproducible and correctly manage
+  every replay's lifetime, without changing the WhisperX HTTP schema or requiring
+  an upstream service change.
+- **Tests:** Cover per-speaker recovery after a later failure, interruption and
+  checkpoint failure; changed/missing inputs or outputs; force followed by an
+  ordinary retry; concurrent successful completions; and aggregate downstream
+  gating. Extend existing fixtures rather than duplicating analyze's complete
+  test suite. Exercise actual body reconstruction after partial consumption,
+  multipart equivalence, cancellation/resource cleanup, and existing bounded
+  streaming and retry behavior—not just the presence of a callback. Default
+  tests remain offline and use local/fake HTTP boundaries, never paid services.
+- **Documentation:** Update the canonical
+  [stage guide](../internal/stage-transcribe.md),
+  [manifest guide](../internal/manifest.md),
+  [WhisperX contract](../integrations/whisperx.md),
+  [operations guide](../operations.md), and
+  [troubleshooting guide](../troubleshooting.md) when implemented. Explain
+  partial completion, ordinary retry versus force, and the distinction between
+  request replay limitations and persistent peer protocol failures.
+
+## Compatibility and Scope Boundaries
+
+Existing speaker identities, raw transcript formats, and the fixed pipeline
+order remain unchanged. Existing manifests remain readable, but must not be
+treated as containing per-speaker evidence they never recorded. Recovery relies
+on verified recorded evidence, not guessed associations between old transcripts
+and current audio. No general legacy-migration framework is part of the feature.
+
+Automatic adoption of unrecorded outputs from previously failed runs is outside
+this feature. Such files may be recoverable after verifying their inputs, but
+their presence alone does not prove completion. No recovery/import command,
+speaker-selection CLI, historical invocation ledger, new service configuration
+discovery, or broader orchestration refactor is included.
+
+The feature does not diagnose the remote service/proxy's underlying HTTP/2
+problem, guarantee recovery from persistent protocol errors, or change the
+policy for continuing unrelated work after failure.
+
+## Completion Criteria
+
+For speakers A, B, C, and D: if A and B are checkpointed, C fails, and D has not
+completed, ordinary resume retains A and B, retries C, and completes D. No
+downstream stage receives an incomplete successful transcription collection.
+Force replaces all four, while an ordinary retry can reuse checkpoints from
+that forced replacement attempt.
+
+The HTTP transport can reconstruct the upload after consuming the original
+body, without whole-file memory buffering, lost source errors, or stranded
+upload resources. A persistent peer error still fails normally after the
+configured application attempts, without discarding checkpointed speakers.