diff --git a/docs/integrations/whisperx.md b/docs/integrations/whisperx.md index 3850b3a..de41b88 100644 --- a/docs/integrations/whisperx.md +++ b/docs/integrations/whisperx.md @@ -17,11 +17,20 @@ Narratio sends an HTTP `POST` to the configured transcription URL using The server must return a `2xx` response whose body is valid JSON. Narratio does not currently require a more specific response schema at this boundary. -The transcription URL must be an absolute `http` or `https` URL. The audio body -is streamed through a fresh multipart writer for every attempt, so its memory -use is bounded by the transport buffer rather than by the complete audio file. +The transcription URL must be an absolute `http` or `https` URL. Each +application attempt creates one bounded multipart framing and streams the audio +from an open file between that framing; it does not buffer the complete audio +or spool a multipart copy. The file field precedes the language field. WhisperX response acquisition is capped at 10 MiB. +The initial request body and any transport-requested replay use the same +multipart boundary and framing. Every body opens an independent audio handle +and replay starts at the beginning. Narratio closes all bodies created for the +attempt on response, failure, timeout, or cancellation, including when a +transport abandons an earlier body. Genuine source read/close failures remain +request failures; deliberate early closure of an abandoned original does not +poison a valid replay. + ## Request And Result Contract Each adapter request identifies a speaker, a readable audio file, and the @@ -43,6 +52,11 @@ transcript output. - Narratio waits the configured retry delay between attempts and aborts that wait when the parent context is canceled. +Transport replay occurs inside one application attempt and does not consume an +entry from the configured retry budget or increment the reported attempt +count. The standard HTTP redirect policy remains in effect; body replay can +also support body-preserving redirects. + ## Validation And Failure Semantics Client construction rejects a missing or non-HTTP(S) absolute transcription URL, @@ -55,12 +69,19 @@ failure to install the output causes the transcription to fail. Errors include attempt context, and the result retains attempts, final status when available, and elapsed duration for diagnostics. +Replay makes a fresh request body available to the HTTP transport; it does not +provide an exactly-once server guarantee and does not prove recovery from every +proxy or protocol-level failure. A server may have received work before the +client learns that a request failed. WhisperX operations should therefore be +safe to retry for the same input when practical. + ## Determinism And Concurrency Each audio request has stable multipart field names, and successful bytes are installed atomically. The transcribe stage may process speaker files in -parallel, bounded by the configured concurrency. It records results in stable -speaker order after all work completes; any speaker failure fails the stage. +parallel, bounded by the configured concurrency. Each validated speaker is +checkpointed independently; any speaker failure still fails the aggregate +stage, while committed earlier speakers remain reusable on an ordinary retry. ## Related Canonical Docs diff --git a/docs/internal/manifest.md b/docs/internal/manifest.md index d463cbd..c7ded58 100644 --- a/docs/internal/manifest.md +++ b/docs/internal/manifest.md @@ -50,6 +50,43 @@ The model admits these stage states: - `stale` - `interrupted` +### Transcribe-owned speaker state + +A `transcribe` stage record may carry `transcribe_state_version: 1` and a +`transcribe_speakers` map keyed by the existing derived speaker ID. The version +is the authority marker: a versioned empty map means that no earlier speaker +completion is reusable. Absence of both fields is accepted as legacy +aggregate-only state so current Narratio can read older manifests. + +Each map entry represents one completed speaker only. It binds a versioned +fingerprint to the session-relative audio path, audio SHA-256 and size, one +canonical `transcript_raw` output with its path, SHA-256 and size, and the +producer run ID. Paths must be canonical and session-relative, checksums must +be lowercase SHA-256 values, and the record and output producer identities +must agree. Other stages cannot carry this transcribe-owned state. + +Transcribe reconciles this collection against the current prepared audio set +and observable WhisperX semantics. Missing or corrupt regular outputs become +work; unsafe or non-regular evidence is rejected. It saves the reconciled +collection before HTTP dispatch and saves a detached candidate after each +validated completion. Only after a save succeeds is the candidate installed as +live authority. Canonical bytes produced before an unsuccessful save are not a +completion record. + +Aggregate transcribe status still describes the whole invocation. A failed, +running, stale, or interrupted aggregate may therefore retain reusable speaker +records, while aggregate outputs are declared only after the current +collection is complete. Ordinary retry reconciles and reuses those records. +Force first checkpoints an empty current collection, preventing later retries +from reviving pre-force results. + +Older aggregate-only transcribe success is conservatively regenerated once +when transcribe is selected because it does not bind individual outputs to the +audio bytes used. The session-manifest schema identifier remains +`narratio.session-manifest.v1`; this additive state is accepted by current +Narratio, but older strict-decoding binaries may reject manifests after these +fields have been written. + ### Analyze-owned artifact state Every `analyze` stage record carries `analyze_state_version: 1` and an @@ -156,10 +193,10 @@ establish a current immutable snapshot. Manifest replacement provides atomic visibility, not a crash- or power-loss durability guarantee. Callers reload after a returned save error before retrying. -An analyze checkpoint error leaves the prior manifest as per-artifact authority. -The runner then attempts to persist the bounded failed/stale retry collection; -if that terminal save also fails, its error is joined with the originating -failure and the next invocation must reconcile from the last readable manifest. +A transcribe or analyze checkpoint error leaves the prior manifest collection +as authority. The runner then attempts to persist the aggregate failure; if +that terminal save also fails, its error is joined with the originating failure +and the next invocation must reconcile from the last readable manifest. ## Execution Semantics @@ -193,11 +230,11 @@ only fields returned by the new result. Marking a record stale does not clear those details because resume validation and diagnosis may still require them before execution begins. -Aggregate lifecycle clearing deliberately preserves the analyze-owned -per-artifact collection. This lets later reconciliation replace only evaluated -entries without erasing unrelated current results. Other stages retain their -existing aggregate-only lifecycle behavior and are forbidden from carrying the -analyze-specific fields. +Aggregate lifecycle clearing deliberately preserves both the transcribe-owned +speaker collection and the analyze-owned per-artifact collection. This lets +later reconciliation replace only non-current entries without erasing durable +completions. Other stages retain aggregate-only lifecycle behavior and are +forbidden from carrying either collection. A stage may explicitly return a skipped disposition and stable reason. The runner persists that outcome in the session manifest, clears older outputs, @@ -228,7 +265,7 @@ evidence and focused durable validators: | Work | Reuse authority | Focused owners | | --- | --- | --- | | prepare | aggregate semantic fingerprint | [prepare](stage-prepare.md) | -| transcribe | aggregate semantic fingerprint | [transcribe](stage-transcribe.md), [WhisperX](../integrations/whisperx.md) | +| transcribe speakers | aggregate semantic fingerprint plus per-speaker input/output reconciliation | [transcribe](stage-transcribe.md), [WhisperX](../integrations/whisperx.md) | | merge | aggregate semantic fingerprint | [merge](stage-merge.md), [Seriatim](../integrations/seriatim.md) | | polish | aggregate semantic fingerprint | [polish](stage-polish.md), [Audita](../integrations/audita.md) | | normalize | aggregate semantic fingerprint | [normalize](stage-normalize.md), [Seriatim](../integrations/seriatim.md) | @@ -310,6 +347,7 @@ where the last session transition can require operator interpretation. - Remote commit model and readers: `internal/artifacts/remote_commit.go`, `internal/artifacts/current_state_commit.go`, - Persistence and validation: `internal/manifest/store.go` +- Transcribe speaker state: `internal/manifest/transcribe_state.go` - Package tests: `internal/manifest/*_test.go` - Assembled execution behavior: `internal/app/runner_test.go`, `internal/app/run_stage_test.go` diff --git a/docs/internal/stage-merge.md b/docs/internal/stage-merge.md index 395dafa..2917ee9 100644 --- a/docs/internal/stage-merge.md +++ b/docs/internal/stage-merge.md @@ -17,7 +17,11 @@ Normalize raw transcript inputs and merge into base transcript via Seriatim. ## Key Behavior -- discovers and validates raw transcript inputs. +- discovers and validates raw transcript inputs. For versioned transcribe + state, discovery requires aggregate transcribe success and consumes only the + aggregate outputs that exactly match the current per-speaker collection. + Partial checkpoints and orphaned files are not merge inputs. Legacy/manual + workflows without versioned speaker state retain directory discovery. - normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output. - merges normalized inputs (`seriatim.Run`) into base transcript. - validates merged transcript and optional report JSON. @@ -26,6 +30,8 @@ Normalize raw transcript inputs and merge into base transcript via Seriatim. ## Invariants - merge always consumes normalized forms of raw inputs. +- a versioned transcribe collection is usable only through its complete, + successful aggregate output declaration. - base transcript must validate before stage success. - report output is config-gated. diff --git a/docs/internal/stage-transcribe.md b/docs/internal/stage-transcribe.md index b26f8be..6d69938 100644 --- a/docs/internal/stage-transcribe.md +++ b/docs/internal/stage-transcribe.md @@ -14,12 +14,19 @@ Generate raw per-speaker transcripts from prepared audio using WhisperX. ## Key Behavior -- discovers prepared audio from manifest inputs or canonical audio directory. -- derives the transcript identity from the prepared `.flac` filename. -- dispatches one WhisperX request per prepared file through a bounded errgroup. -- validates each output as JSON. -- writes run-local outputs then materializes canonical transcript outputs only - after every planned request succeeds. +- discovers prepared audio from manifest inputs or the canonical audio + directory and derives each speaker identity from its `.flac` filename. +- hashes the current audio and reconciles it with the versioned per-speaker + completion collection before starting HTTP work. +- reuses only records whose audio identity, observable WhisperX semantics, + canonical output location, JSON, checksum, and size remain current. +- durably checkpoints the reconciled collection before dispatch, then sends + only missing or obsolete speakers through the bounded worker group. +- validates each successful run-local response, rechecks the audio bytes, + materializes the canonical raw transcript, and checkpoints that speaker + before accepting another completion. +- rebuilds aggregate stage outputs in stable speaker order from the complete + current collection, including records produced by earlier invocations. ## Invariants @@ -28,21 +35,44 @@ Generate raw per-speaker transcripts from prepared audio using WhisperX. - output path returned by adapter must match requested output path. - an empty adapter result path means the requested path; adapters cannot select an alternate destination. -- each successful output is validated before stage success, and cancellation or - incomplete dispatch cannot be reported as a successful result. +- candidate collection construction, manifest save, and installation into live + state are serialized. Expensive hashing and HTTP work remain outside that + critical section. +- a canonical file has no completion authority until its manifest checkpoint + succeeds. A save failure preserves the last successfully committed + collection for ordinary retry. +- each successful output is validated before its checkpoint, and cancellation, + incomplete dispatch, or a failed aggregate save cannot be reported as + aggregate stage success. +- removed audio inputs are omitted from the current collection and aggregate + output list; reconciliation does not delete arbitrary transcript files. +- merge cannot consume a partial versioned collection: it requires transcribe + aggregate success and an exact match between current speaker records and + declared aggregate outputs. ## Resume Evidence -Transcribe records a versioned semantic-configuration fingerprint containing -the Narratio-visible WhisperX service URL and recognition language. Changes to -either rerun transcription and stale its fixed descendants while leaving -prepare reusable. Retry count/delay, concurrency, timeout, credentials, and -diagnostic locations are operational and do not change this evidence. +Each completed speaker record binds the speaker ID, session-relative audio +path, streamed audio checksum and size, the Narratio-visible WhisperX service +URL and recognition language, and the canonical transcript path, checksum, +size, and producer run. Changed audio bytes select that speaker for replacement; +a changed URL or language selects every speaker. Either case stales fixed +descendants through the normal runner decision. Retry count/delay, concurrency, +timeout, credentials, and diagnostic locations are operational and do not +change this evidence. + +An ordinary retry reuses every valid speaker checkpoint even when aggregate +transcribe status is failed, running, or stale. `--force` first durably replaces +the collection with an empty versioned collection, so a later ordinary retry +cannot resurrect pre-force results; speakers checkpointed by that forced +attempt remain reusable. WhisperX models or private service configuration not exposed by Narratio's adapter contract cannot be fingerprinted; use `--force` after changing them. -An existing successful transcribe record without evidence reruns once when -selected. +An older aggregate-only transcribe record has no trustworthy audio-to-output +binding. When transcribe is next selected, every current speaker is therefore +generated once and recorded in the versioned collection; later invocations can +reuse it normally. ## Related Contracts And Tests @@ -51,6 +81,9 @@ selected. - [Configuration](../config.md#pipeline) owns concurrency and other operator-selected values. - Implementation and tests: `internal/stage/transcribe.go`, + `internal/stage/transcribe_reconciliation.go`, + `internal/stage/transcribe_execution_state.go`, + `internal/stage/transcribe_resume.go`, `internal/stage/transcribe_test.go`, - `internal/stage/semantic_contracts_initial.go`, and - `internal/stage/semantic_contracts_initial_test.go` + `internal/stage/transcribe_reconciliation_test.go`, and + `internal/stage/transcribe_checkpoint_test.go` diff --git a/docs/operations.md b/docs/operations.md index a5c3fec..d67fac4 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -116,8 +116,8 @@ Execution rules: perpetually rerun dependent work. Every aggregate stage except analyze currently provides semantic-configuration -evidence; analyze retains its more precise per-artifact fingerprints and -validator. +evidence; transcribe additionally reconciles per-speaker completion evidence, +while analyze retains its more precise per-artifact fingerprints and validator. Prepare additionally validates current stable/local/S3 source identity and the checksums of its durable prepared copies before reuse. Changed bytes, audio membership, S3 object identity, or missing/tampered copies rerun prepare and @@ -187,6 +187,54 @@ shared session lifecycle. In particular, render does not require Notarius or Scriptorium, extract does not require Scriptorium, and analyze does not require the transcription, Seriatim, Audita, or Notarius adapters. +### Transcription continuation and replacement + +Transcribe checkpoints each validated speaker independently. If a later +speaker exhausts HTTP retries, the process is canceled, or the aggregate stage +save fails, the aggregate stage is not successful and downstream merge cannot +consume the partial set. The last successfully saved speaker records and their +canonical `transcripts/raw/.json` files remain durable, however. + +After correcting the underlying problem, retry normally: + +```bash +narratio run-stage transcribe 2026-04-04 +``` + +Narratio hashes the current prepared audio, validates recorded transcript JSON, +checksum, size, and location, and calls WhisperX only for missing or obsolete +speakers. A changed audio file replaces only its speaker; a changed WhisperX +URL or language makes every current speaker non-reusable. Retry, timeout, and +concurrency tuning do not invalidate completed speakers. + +Use force only when every current transcript must be replaced, including after +a WhisperX model or private server-side configuration changes behind the same +URL and language: + +```bash +narratio run-stage transcribe 2026-04-04 --force +``` + +Force durably clears the reusable collection before sending requests. If that +forced attempt fails, a later ordinary retry reuses only replacement speakers +checkpointed after the reset; it does not revive pre-force results. Do not edit +speaker checksums or copy older JSON into the canonical directory to influence +reuse. + +The first current Narratio invocation that selects transcribe for an older +aggregate-only manifest retranscribes every current speaker once because the +old record cannot prove which audio bytes produced each file. Subsequent runs +reuse the new versioned evidence normally. The overall session-manifest schema +identifier is unchanged, but older strict-decoding Narratio binaries may reject +a manifest after current Narratio writes the additive speaker fields; retain a +backup or complete the upgrade before relying on downgrade. + +Restoring a manifest does not implicitly publish or restore every raw speaker +file. If a recorded canonical transcript is absent locally, ordinary +reconciliation schedules that speaker again. Removing run directories does not +remove canonical checkpointed transcripts and does not by itself invalidate +them. + ## Artifact Development Workflows Choose the narrowest workflow that matches the intended replacement: diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 7b7a2e3..94525c6 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,6 +1,6 @@ # Implementation Plan: Speaker Checkpoints and Replayable Uploads -Status: Implementation in progress; Stages 1-6 complete. +Status: Implementation complete; Stages 1-7 complete. ## Scope and Instructions @@ -381,7 +381,19 @@ application retry/checkpoint behavior remains intact. ## Stage 7: Document and Validate the Completed Feature -Status: Pending. +Status: Complete. + +Completion marker (2026-09-22): Updated the canonical transcribe, merge, +manifest, WhisperX, operations, and troubleshooting documentation for durable +speaker reconciliation, serialized checkpoints, aggregate gating, durable +force reset, legacy upgrade and downgrade limits, replayable streaming bodies, +resource ownership, and the absence of an exactly-once remote guarantee. +Reviewed the completed implementation against the feature roadmap and +implementation-discipline guidance. Verified with `go test ./...`, +`go test -race ./...`, `go vet ./...`, `go build ./...`, +`go test ./internal/doccheck`, +`go test ./internal/config -run '^TestExamplesLoadAndValidate$'`, and +`git diff --check`. 1. Update canonical transcribe and manifest internals for the collection, reconciliation, durable-reset force behavior, serialized saves, and aggregate diff --git a/docs/roadmap/transcribe-checkpoints-and-replay.md b/docs/roadmap/transcribe-checkpoints-and-replay.md index 7f3ceb7..97600fd 100644 --- a/docs/roadmap/transcribe-checkpoints-and-replay.md +++ b/docs/roadmap/transcribe-checkpoints-and-replay.md @@ -1,6 +1,6 @@ # Resumable Speaker Transcription and Replayable Uploads -Status: Proposed; not implemented. +Status: Implemented. ## Goal and User Intent @@ -19,7 +19,7 @@ service failures. This document owns the target state and scope; the [implementation plan](implementation.md) owns ordered changes and concrete storage and replay decisions. -## Current Limitations +## Pre-implementation Limitations The [transcribe stage](../internal/stage-transcribe.md) writes successful responses into the invocation's run directory. It installs canonical transcripts diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index c5c4d47..0c6d74b 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -213,6 +213,100 @@ input validation. Relevant reference: [Operations: Stage Execution and Continuation Behavior](./operations.md#stage-execution-and-continuation-behavior). +## Transcription retries speakers that previously finished + +Symptom: + +- an ordinary transcribe retry calls WhisperX for a speaker that appeared to + finish earlier; or +- an older successful session retranscribes all speakers once after upgrade. + +Likely causes: + +- the earlier response or canonical file was not followed by a successful + speaker manifest checkpoint; +- the prepared audio bytes/path, WhisperX URL, or language changed; +- the canonical transcript is missing, unsafe, non-regular, invalid JSON, or + no longer matches its recorded size/checksum; +- the previous invocation used `--force`, which durably cleared pre-force + speaker authority; or +- the manifest has only legacy aggregate transcribe state and cannot bind old + outputs to their source audio. + +Safe fix: + +- inspect the reported speaker and filesystem error, repair the prepared input + or canonical-output path when needed, then retry transcribe normally; +- allow a legacy aggregate-only record to regenerate once; and +- do not edit the manifest or copy an old JSON file into place as a substitute + for a successful checkpoint. + +```bash +narratio run-stage transcribe 2026-04-04 +``` + +Valid earlier speaker checkpoints remain reusable even while aggregate +transcribe status is failed or interrupted. Merge remains gated until +transcribe reaches aggregate success with the complete current set. + +Relevant reference: [Operations: Transcription continuation and replacement](./operations.md#transcription-continuation-and-replacement). + +## WhisperX model changed but transcription was reused + +Symptom: + +- WhisperX model weights, prompts, or private server configuration changed at + the same configured URL, but Narratio still considers speaker transcripts + current. + +Likely cause: + +- Narratio can fingerprint its configured URL and language, not private state + loaded behind the service boundary. + +Safe fix: + +- explicitly force transcribe to establish a durable reset, then use ordinary + retry if the replacement attempt is interrupted. + +```bash +narratio run-stage transcribe 2026-04-04 --force +``` + +Transport replay and application retries do not provide an exactly-once remote +execution guarantee. Investigate server-side failures or duplicate work using +WhisperX diagnostics; do not infer completion from a server log without the +local speaker checkpoint. + +Relevant references: [WhisperX integration](./integrations/whisperx.md) and +[Transcribe internals](./internal/stage-transcribe.md). + +## Transcription manifest checkpoint failed + +Symptom: + +- transcribe reports that a speaker completed but the session manifest could + not be saved. + +Likely causes: + +- the workspace is unwritable or out of space; +- the manifest path or parent directory is unsafe or the wrong type; or +- a transient filesystem error interrupted atomic replacement. + +Safe fix: + +- correct the reported filesystem or permission problem and retry transcribe + normally. Narratio reconciles from the last readable collection and reuses + only successfully checkpointed speakers. + +A canonical raw transcript left by the failed checkpoint is incidental and +does not become reusable authority. Preserve the manifest and diagnostics if +the failure repeats; do not advertise the file by editing manifest evidence. + +Relevant references: [Manifest persistence](./internal/manifest.md#persistence-semantics) +and [Transcribe internals](./internal/stage-transcribe.md). + ## Notarius credential reference unavailable Symptom: @@ -580,18 +674,22 @@ Symptom: or unsupported; or - restore rejects a committed remote session manifest for the same reason. -Likely cause: +Likely causes: - the manifest predates the supported canonical state contract. Narratio rejects it at the schema boundary rather than interpreting aggregate output - lists or incidental files as current authority. + lists or incidental files as current authority; or +- an older strict-decoding Narratio binary is reading a current manifest that + contains additive transcribe speaker state it does not recognize. Safe fix: - for unsupported local state, move the old manifest aside and regenerate, or restore a verified current committed snapshot when one is authoritative; - for unsupported committed remote state, regenerate healthy local state as - needed and republish the session. + needed and republish the session; and +- for a downgrade rejection, return to the current binary or restore a verified + pre-upgrade backup. Do not delete only the speaker fields by hand. ```bash narratio run 2026-04-04 @@ -603,7 +701,7 @@ old files into canonical paths merely to influence selection; availability is manifest-owned. Relevant references: [Operations: Stage Execution and Continuation Behavior](./operations.md#stage-execution-and-continuation-behavior) -and [Manifest Internals](./internal/manifest.md#analyze-owned-artifact-state). +and [Manifest Internals](./internal/manifest.md#transcribe-owned-speaker-state). ## Scriptorium private input changed without a rerun