diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index c7f865b..2d8f7ed 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,258 +1,346 @@ -# Implementation Plan: Notarius Configuration Profile Selection +# Implementation Plan: Speaker Checkpoints and Replayable Uploads -Status: Complete; all stages implemented and verified. +Status: Ready; all stages pending. -## Scope and Working Instructions +## Scope and Instructions -Implement the [feature roadmap](notarius-config-profile.md), whose authority is -the [consumer request](feature-request-notarius-config-profile.yml). Complete -the numbered stages in order. Each stage is a bounded task for a gpt-5.6-terra -coding agent. Do not treat completion of an intermediate stage as completion of -the feature. Record completed stages and actual verification results here. +Implement [the feature roadmap](transcribe-checkpoints-and-replay.md). This plan +replaces the completed Notarius-profile plan; that work remains in Git history. +Complete the numbered stages in order, one bounded implementation prompt per +stage for gpt-5.6-terra. Record completion and actual verification results here. +Intermediate stages are not a finished feature and must not be released alone. -Before each stage, read `docs/development.md`, all documents under -`docs/policy/`, the feature roadmap, and that stage's preparation list. Follow -AGENTS.md discovery guidance. Use existing collaborators and fixtures; do not -build a new configuration, cache, or profile-management subsystem. Tests must -remain offline and independent of installed Notarius binaries and credentials. -Do not change dependencies, implement Ansible changes, tag, or release as part -of this plan. +Before working, read `AGENTS.md`, `docs/development.md`, all three policies in +`docs/policy/`, the roadmap, and the stage-specific preparation below. Use the +maintained Go 1.26 toolchain. Preserve operator choices, speaker filenames, +bounded concurrency, streaming, and existing credentials/settings. Do not add +new commands, dependencies, service requirements, arbitrary speaker limits, +strict transcript schemas, or a generic checkpoint framework. No release or +upstream service work is part of this plan. -## Implementation Decisions +## Settled Design Decisions -These choices resolve implementation details without adding restrictions to -the feature request: +### Progress Representation and Recovery -- Represent the authored field as `ConfigProfile *string` with YAML/JSON name - `config_profile` and `omitempty`, in both typed configuration and its safe - YAML projection. Nil means omitted/null. A non-nil pointer preserves an - explicitly supplied empty string as well as `default` and named selections. -- Carry the same optional value in the Notarius adapter request. Nil emits no - flag; non-nil emits `--config-profile` followed by the exact supplied string - as a separate argument, including an empty string. Do not trim, lowercase, - substitute Narratio's profile, impose a name regex, or inspect profile files. - Notarius owns validation and selector interpretation. -- For reuse evidence only, use one small stage-local helper that maps nil, - `""`, and exact `"default"` to the empty identity; other values remain - unchanged. Both extraction fingerprints use it. This does not mutate the - authored field or change CLI forwarding. Do not attempt to recreate the - downstream parser's complete normalization rules. -- Add `ConfigProfile string` with `json:"config_profile,omitempty"` to both - existing fingerprint payloads, populated only where extraction is enabled. - Preserve existing field ordering and the current semantic fingerprint version - (1). With no non-base selector the serialized payload remains unchanged; - a named selection adds evidence that old records cannot already possess. - No dual-reader, manifest migration, new durable metadata field, or broad - fingerprint-version bump is needed. This is a minimal additive encoding, - not a permanent promise to freeze fingerprint bytes across future releases. -- Profile changes trigger the existing non-resumable/invalidation paths, not - force mode. Preserve configured resume/cache controls, force precedence, - credentials, output contracts, and stage-range behavior. Do not run extraction - outside the operator's selected range or add a separate testing workspace. -- Preserve the existing division of ownership: private downstream file changes - behind an unchanged selector require operator-directed force when needed. - No downstream configuration validation subprocess or runtime version gate - is added. +Add optional `transcribe_state_version` (version 1) and `transcribe_speakers` +to `manifest.StageRecord`. The latter is a map keyed by the existing derived +speaker ID, containing completed results only. Absence means that work must be +done; a separate pending/running/failed state machine is unnecessary. Aggregate +stage status/error remains authoritative for invocation failure. A versioned +empty collection is meaningful: no earlier completions are reusable. -## Stage 1: Add Configuration and Adapter Support +Use a typed `TranscribeSpeakerRecord` with fingerprint version/digest, +session-relative audio path, audio SHA-256 and size, one `ArtifactRecord` output +of kind `transcript_raw`, output size, and producer run ID. The output carries +its session-relative canonical path and SHA-256. No new transcript source IDs, +synthetic schema contracts, timestamps, or historical attempt collections are +needed. Existing aggregate diagnostics can retain attempt/status/duration data +for work performed in the current invocation; do not fabricate that information +for reused outputs. -Status: Complete. +Keep the session-manifest schema identifier unchanged for this optional additive +state; validate supported per-stage versions and retain strict decoding. New +Narratio reads old manifests. Old binaries may reject the added fields under +their existing strict decoding; document this downgrade limitation. -Completion marker (2026-09-21): Added composed optional configuration and safe -inspection support, adapter request/argument forwarding, copy auditing, and -focused configuration and subprocess coverage. Verified with -`go test ./internal/config ./internal/adapters/notarius`. +For pre-feature manifests with no speaker evidence, regenerate transcription +once when transcribe is next selected, even if the aggregate status succeeded. +Old records do not bind individual transcript results to the audio bytes they +used. Do not guess that association from current files, old `per_file` metadata, +or abandoned run directories. This is the conservative upgrade path, not a new +general migration system. No other stage or unselected range is forcibly run. -Preparation: +### Reuse and Persistence -- `docs/config.md` (composition, profiles, Notarius, credentials). -- `docs/integrations/notarius.md` (invocation and failure behavior). -- `internal/config/config.go`, `credential_projection.go`, - `pipeline_configkit.go`, `pipeline_normalize.go`, and `notarius_test.go`. -- `internal/adapters/notarius/runner.go`, `subprocess.go`, `fake.go`, and - `subprocess_test.go`. +Compute a versioned per-speaker fingerprint from speaker ID, session-relative +audio identity, actual audio SHA-256/size, and the existing transcribe semantic +configuration fingerprint (URL and language). Do not include invocation ID, +absolute workspace root, concurrency, retry settings, timeout, or credentials. +Stream input hashing with cancellation; do not trust stored input checksums +alone. Recheck input bytes before accepting a new result to catch changes +observed during transcription; fail that job without checkpointing mismatched +evidence. This is not a hostile-filesystem snapshot or concurrent-editor +transaction guarantee and requires no extra audio copies or file locks. -Work: +Validate reusable output identity, canonical location, JSON, checksum, and size. +Use existing confinement/file primitives. Missing or corrupt regular outputs +become work; unsafe paths/non-regular conflicts follow existing safety errors, +not recursive deletion. Missing required audio is an input error, not permission +to reuse an unverifiable transcript. Removed inputs are removed from the current +collection and aggregate outputs without deleting arbitrary files. -1. Add the optional field to `NotariusConfig` and the `safeNotarius` projection, - including its returned value. Let existing ConfigKit composition handle - overrides and null; do not add a default or resolve this name as a path. -2. Add the optional field to `notarius.RunRequest` and append the argument pair - when non-nil in `SubprocessRunner.Run`, before `--json`. Leave subprocess - environment construction and error propagation unchanged. -3. Audit any explicit Notarius configuration/request copies for the new field. - Update actual omissions without introducing a generic cloning framework. -4. Extend real configuration-load tests to cover a base selection, an overlay - replacing it with a differently named downstream selection, and an overlay - clearing it with null. Cover omitted, empty, and `default` values compactly. - Verify the safe effective-YAML output retains a supplied value, including - explicit empty, and round-trips without leaking credential values. Reuse - existing credential tests rather than duplicating their full matrix. -5. Extend adapter invocation tests for omitted and supplied selections, - including empty and `default`. Assert the argument/value boundary rather - than imposing new ordering requirements on unrelated arguments. Reuse the - existing nonzero-exit test with a selected profile to prove there is no - fallback or successful receipt processing after failure. +One stage-owned mutex serializes candidate collection construction, manifest +save, and installation of committed state. Save a detached candidate and only +then update live authority, following analyze's pattern. Workers must not mutate +the live manifest outside that serialized section. Keep expensive HTTP work +outside the mutex. Stop dispatch/acceptance on checkpoint failure, join workers +before return, and preserve the last successfully committed collection. -Verification: +Before launching requests, durably checkpoint the reconciled collection. For +force, that collection is empty: this prevents later ordinary retries from +resurrecting pre-force outputs. If this initial save fails, do not start requests +or overwrite outputs. Force again clears the current collection again; ordinary +retry retains replacements already checkpointed after the previous force. -```sh -go test ./internal/config ./internal/adapters/notarius -``` +Save each validated result at its canonical raw-transcript location before +checkpointing it. Run-local files remain staging/diagnostics, not reuse +dependencies. Bytes left by an unsuccessful checkpoint have no completion +authority. A worker may checkpoint a completed result using the parent stage +context even if a sibling canceled the work context; parent cancellation or a +checkpoint failure may prevent that save. Guarantee retention of committed +work, not all responses that happened to finish during cancellation. -Completion: Configuration composition and safe inspection preserve the field, -and the adapter forwards it correctly. The executing stage is connected in -Stage 2 together with reuse tracking, so it cannot start using untracked profile -selections as an intermediate implementation. +### Multipart Replay Mechanism -## Stage 2: Wire Extraction and Profile-Sensitive Reuse +Replace the pipe/producer upload mechanism with a small pull-based multipart +body backed by the audio reader. Use `mime/multipart.Writer` to encode a bounded +metadata prefix and suffix around the file payload: capture the prefix after +`CreateFormFile`, then encode the language field and closing boundary as the +suffix. Generate this framing once per application attempt. Each body combines +fresh readers for prefix, freshly opened audio, and suffix using `io.MultiReader`. +No full-audio buffer or temporary multipart spool is needed. -Status: Complete. +The initial body and `Request.GetBody` use the same factory/framing. Each owns +an independent audio handle. Give bodies idempotent Close and context-triggered +closure (for example `context.AfterFunc`), without holding a mutex across a +blocking Read. Read/Close coordination must release a blocked source and retain +genuine source-read/close errors while not treating deliberate early closure as +source failure. The attempt owns cleanup of all created bodies, including +replays; reject new body creation after its context expires. -Completion marker (2026-09-21): Wired the selected profile into extraction, -added a shared base-selector reuse identity to both existing fingerprints, and -covered request forwarding, force precedence, semantic sensitivity, prior -no-profile encoding, and focused resume validation. Verified with -`go test ./internal/stage ./internal/adapters/notarius ./internal/config`. +Remove the old producer `Wait` dependency. Track body-local source errors with +synchronization so a swallowed upload read error cannot become a successful +transcript, and a deliberately closed original body cannot poison a successful +replay. Keep the outer retry policy and per-attempt context. Do not add replay +counters to the user retry budget or depend on Go's exact internal retry count. +Preserve ordinary HTTP redirect policy; body replay can also enable Go's +body-preserving redirects. Do not introduce a new redirect restriction. -Preparation: +## Stage 1: Add Durable Speaker Completion State -- `docs/internal/stage-extract.md`. -- `internal/stage/extract.go`, `extract_resume.go`, - `semantic_contracts_delivery.go`, and their focused tests. -- In particular, `TestExtractStageResolvesManifestInputAndBuildsExactRequest`, - `TestExtractSemanticConfigSensitivityAndOperationalExclusions`, and - `TestExtractStageResumeValidationRequiresFreshEvidence`. +Status: Pending. -Work: +Preparation: `internal/manifest/manifest.go`, `store.go`, `analyze_state.go`, +their tests, and `docs/internal/manifest.md`. -1. Pass `cfg.ConfigProfile` to the adapter request when extraction executes. - Keep the field independent of credential forwarding and force overrides. -2. Implement the reuse-identity helper and additive fingerprint fields described - above. Update both `extractSemanticConfig` and `fingerprintDocument`, using - the same identity in semantic comparison and `extractionFingerprint`. - Keep disabled extraction insensitive to unused downstream settings. -3. Extend the existing stage request test to prove the authored selection - reaches the adapter. Cover ordinary and forced execution sufficiently to - show that profile forwarding does not alter existing cache-control precedence. -4. Extend semantic-sensitivity coverage: base to named, named to different named, - and named to base change evidence; unchanged selection does not. Verify - nil/empty/`default` share base evidence and disabled settings remain inert. -5. Exercise the focused resume validator with an actual valid recorded bundle: - it remains reusable for the same selection and becomes non-resumable for a - changed selection. This guards the second fingerprint path independently - of the application's semantic comparison. -6. Check the additive encoding against the prior no-profile payload shape once - in focused coverage or a recorded development check. Avoid whole-manifest - snapshots and a permanent historical compatibility harness; retain existing - input/output/reference sorting and all unrelated fingerprint fields. +1. Implement the additive fields, record type, version constants, validation, + and deep-copy helper described above in a focused transcribe state file. +2. Validate ownership by the transcribe stage, supported versions, required + completed-record evidence, output kind, safe relative paths, SHA-256 values, + sizes, and producer identity consistency. Use existing speaker IDs as-is; + do not apply analysis-artifact key regexes or copy unrelated analysis limits. +3. Integrate validation with both load and save. Preserve speaker state through + running/failed/stale transitions and aggregate result clearing. Generic stage + invalidation does not discard checkpoints; reconciliation checks their data. +4. Test round-trip populated and versioned-empty collections, old manifests + without fields, malformed evidence, deep-copy isolation, and lifecycle + preservation. Avoid imposing JSON formatting or map order as contracts. -Verification: +Verify: `go test ./internal/manifest`. -```sh -go test ./internal/stage ./internal/adapters/notarius ./internal/config -``` +Done when durable storage can represent partial completion independently of +aggregate status, without activating new stage behavior yet. -Completion: Selected profiles reach actual extraction, and both existing reuse -checks account for the selection without forcing downstream cache refresh or -changing the manifest schema. +## Stage 2: Implement Read-Only Speaker Reconciliation -## Stage 3: Verify Application Lifecycle Behavior +Status: Pending. -Status: Complete. +Preparation: `internal/stage/transcribe.go`, `semantic_contracts_initial.go`, +`analyze_reconciliation.go`, `extract_resume.go`, their tests, and the artifact +path/integrity helpers they use. -Completion marker (2026-09-21): Extended the existing application lifecycle -fixtures to cover same-selection reuse, named and base selection changes, -dependent invalidation/rerun, sibling preservation, ordinary downstream -failure, and real ConfigKit profile-to-execution composition without implicit -name forwarding or digest-wide invalidation. Verified with -`go test ./internal/app ./internal/stage`. +1. Extract the existing audio discovery/job construction into a shared, + transcribe-local reconciliation path. Preserve current input selection and + speaker identity behavior. Build sorted jobs and classify completed results + against the current input set and evidence defined above. +2. Add pure fingerprint construction and streamed input/output inspection. + Resolve manifest input paths with existing session-relative/absolute reading + helpers; do not change input precedence or assume the current working directory + is the session root. Newly written checkpoint paths are always relative. +3. Return current records and work-to-do without writing files or manifest state. + Force selects all work and an empty starting collection; old unversioned + transcribe state supplies no reusable records. Do not connect aggregate skip + validation until execution can produce the new evidence in Stage 3. +4. Test unchanged reuse; one changed audio among unchanged speakers; URL/language + changes; operational-setting exclusions; added/removed speakers; missing and + corrupt regular outputs; unsafe record paths; missing audio; and force. + Prefer compact tables and real temporary files over mocked hashing. -Preparation: +Verify: `go test ./internal/stage ./internal/manifest`. -- `internal/app/extract_lifecycle_test.go`, `runner_test.go`, and - `run_control_test.go`. -- Existing fixtures and tests for changed extraction outcomes, failure/retry, - and independent render/extract invalidation. +Done when one read-only decision path describes exactly which speakers need +work, without altering existing runtime behavior prematurely. -Work: +## Stage 3: Execute and Checkpoint Speakers Incrementally -1. Extend the existing lifecycle fixtures rather than building another harness. - Run a successful extraction with a configured profile, then verify ordinary - reuse for that same profile. Change only the downstream selection and verify - that extraction executes again with the new value, upstream transcripts and - render remain current, and succeeded dependents are invalidated/rerun through - the existing runner policy. Include returning from a named selection to base. -2. Cover the changed-profile failure boundary with the existing fake runner: - return an ordinary downstream error and verify extraction is failed, the - previous result is not advertised as current success, and dependents are not - silently reused. Reuse existing retry coverage where it already proves the - recovery behavior. Do not assert Notarius-specific diagnostic wording. -3. Ensure one configuration-to-execution path proves explicit composition: - Narratio's testing overlay can select a differently named Notarius overlay; - without the field, selecting a Narratio profile does not populate the - downstream request. Use real configuration loading and the existing injected - runner, not a production provider or sibling repository dependency. -4. Review existing operational-exclusion and force tests for continued validity. - Do not broaden invalidation to the full effective configuration digest or - Narratio's profile label, and do not automatically change cache options. +Status: Pending. -Verification: +Preparation: `internal/stage/transcribe.go`, `analyze_execution_state.go`, +`run_local.go`, +`internal/app/runner.go`, and transcribe/checkpoint test fixtures. -```sh -go test ./internal/app ./internal/stage -``` +1. Wire reconciliation into Run, require the existing ManifestStore collaborator, + and checkpoint the starting collection before launching HTTP work. Implement + detached candidate saves and serialized live-state updates. Audit Store.Save's + normalization side effects so a failed candidate save cannot mutate shared + authority; use detached records/collections wherever it may write. +2. Retain bounded errgroup execution and fail-fast cancellation. Check the work + context before beginning each queued request. After success, validate adapter + output path/JSON, recheck audio evidence, materialize canonical bytes, compute + output evidence, and checkpoint the new record. Check a checkpoint-failure + latch under the serialization lock before accepting further completions. +3. Preserve committed successes on all error/cancellation exits, including + starting-save, canonical-write, per-speaker-save, and aggregate-save failures. + Return meaningful speaker/error context and let the runner retain its existing + terminal failure behavior. Do not report partial aggregate outputs as success. +4. Assemble final outputs in deterministic speaker order from the reconciled + completed records, including reused speakers, with original producer IDs and + checksums. If all records are current but aggregate status is failed/running, + finish without WhisperX calls and rebuild the aggregate success. +5. Implement `transcribeStage.ValidateResume` using the same read-only + reconciliation, so a succeeded stage cannot skip changed audio or invalid + transcripts. Retain the existing URL/language semantic check and runner + invalidation behavior; no whole-config digest invalidation. +6. Adapt existing tests that require no canonical partial files: the new contract + permits checkpointed files but forbids aggregate success. Update fixtures to + provide real manifest stores and records rather than a production fallback + that silently disables checkpointing. +7. Test A/B completion followed by C failure and D unattempted (concurrency 1), + reload and retry only C/D; concurrently finishing speakers without lost saves; + failure and cancellation after a checkpoint; save failure retaining earlier + authority; and force failure followed by ordinary retry without old-result reuse. -Completion: Tests demonstrate the requested workflow, including selection -changes and failure, across the real Narratio configuration/runner boundaries. -Notarius's own overlay loader and cache internals remain outside these tests. +Verify: `go test ./internal/stage ./internal/app` and +`go test -race ./internal/stage`. -## Stage 4: Document the Feature and Validate the Completed Change +Done when incremental progress and aggregate success work together safely under +ordinary, concurrent, canceled, and forced execution. -Status: Complete. +## Stage 4: Verify Runner, Merge, and Durable Lifecycle Integration -Completion marker (2026-09-21): Updated the canonical configuration, -integration, extract-internals, operations, and maintained-example documentation; -marked the feature roadmap implemented; and reviewed the completed behavior -against the consumer request. Verified with Go 1.26.8 using `go test ./...`, -`go test -race ./...`, `go vet ./...`, `go build ./...`, -`go test ./internal/doccheck`, -`go test ./internal/config -run '^TestExamplesLoadAndValidate$'`, and -`git diff --check`. +Status: Pending. -Preparation: Canonical documentation owners listed in the feature roadmap, -`examples/README.md`, and the existing development-focused operations guidance. +Preparation: `internal/app/runner.go`, `runner_runtime.go`, +`analyze_checkpoint_recovery_test.go`, `runner_terminal_failure_test.go`, +`restore_manifest_paths.go`, `internal/stage/merge.go`, and existing publish, +restore, and cleanup tests routed from `docs/internal/overview.md`. -Work: +1. Add application-level tests with real manifest persistence and fake WhisperX: + failed stage across separate invocations; all speakers committed before a + failed aggregate save; interrupted running state; force/reset persistence; + and unchanged upstream/succeeded downstream invalidation behavior. Inject + errors at durable boundaries, not every private method call. +2. Close `discoverRawTranscripts`' directory-scan fallback for versioned + transcribe state: incomplete aggregate state is not usable, and completed + state consumes only declared current aggregate outputs. Keep legacy/manual + fallback behavior for manifests without the new state rather than banning + previously allowed standalone workflows. Test that partial files and removed + speaker files cannot leak into a versioned merge, including direct stage calls. +3. Audit manifest copies/projections, remote manifest publication, restored path + handling, and cleanup. Preserve relative speaker paths on restore and verify + they resolve under the destination session, not the producer machine. Reuse + existing confinement validation, not a second path-policy layer. +4. Add focused round-trip coverage through existing publish/restore fixtures for + speaker records. Restored outputs absent from the published selection are + non-reusable locally; do not silently expand remote publication to every raw + speaker or upload audio as a new default. Removing run directories must not + remove canonical checkpointed transcripts or make reuse require run files. +5. Verify old succeeded manifests are readable and rerun once when transcribe is + selected, then reuse normally. Keep old manifests usable for other commands + and bounded ranges. Do not automatically import old unrecorded JSON files. -1. Document the optional field in `docs/config.md`, including omitted/null, - explicit base selectors, and independence from Narratio's profile and - `notarius.llm_api_key`. Do not add profile-name restrictions. -2. Update the invocation in `docs/integrations/notarius.md` to include the - optional flag and identify Notarius v0.7.0 as a known supporting version, - not an unconditional version requirement. Preserve the distinction from - `--llm-profile` and ordinary downstream failure handling. -3. Update `docs/internal/stage-extract.md` to describe selection-sensitive reuse - and retain its distinction from private file content and operational settings. - Make clear this is an explicit downstream setting, not blanket invalidation - by Narratio's profile label or effective digest. -4. Add concise development workflow guidance in `docs/operations.md`: explicit - testing-overlay composition; different downstream names are allowed; - switching selection triggers extraction when that stage is in scope; edits - behind an unchanged selector may need force; testing is not a separate - workspace. Preserve the existing distinction between extract+analyze and - analyze-only regeneration. -5. Update the maintained annotated pipeline example to show the optional field - without requiring an externally installed testing overlay (use a commented - named-selection example). Do not make previously runnable examples depend - on new downstream files. Link to canonical owners instead of duplicating - full contracts or adding new top-level Narratio flags. -6. Review the final diff against every observable behavior and scope boundary - in the consumer request. Mark the implementation stages and roadmap status - accurately after verification; do not remove the original request or alter - historical release notes. +Verify: `go test ./internal/app ./internal/stage ./internal/manifest` and +`go test -race ./internal/app`. -Run the repository checks with the maintained Go 1.26 toolchain: +Done when the new authority survives runner and storage lifecycles and cannot +expose incomplete transcripts downstream. + +## Stage 5: Implement Replayable Streaming Multipart Bodies + +Status: Pending. + +Preparation: `internal/adapters/whisperx/http.go`, `http_test.go`, +`docs/integrations/whisperx.md`, and the installed Go net/http Request/GetBody +contract. Keep this change inside the WhisperX adapter. + +1. Implement the pull-based framing/body factory described above, using standard + multipart encoding rather than hand-escaped MIME headers. Add no new payload + cap or upload-size setting. Preserve file-before-language wire semantics and + the existing form names. +2. Install the initial body and GetBody with the same per-attempt boundary. + Track and close all opened bodies even if a custom transport returns early; + make cleanup/context callbacks safe against overlapping reads and closes. + A GetBody failure must be returned, not converted into an empty upload. +3. Replace pipe-specific Wait/error paths in doTranscribeAttempt with source-error + handling appropriate to pull bodies. Genuine source errors remain visible; + intentional closure during transport retry is not itself a source error. + Keep response bounds, JSON validation, output installation, retry + classification/delay, cancellation, and application attempt reporting intact. +4. Remove obsolete producer-only helpers rather than retaining two upload paths. + Adapt tests tied to producer goroutines to assert streaming and resource + release, which are the actual contracts. +5. Add focused body tests: original/replayed bytes parse identically; replay + after partial consumption starts from the beginning; simultaneous bodies use + independent handles; source failures propagate; Close/cancel releases a + blocked reader without deadlock. Use synthetic audio and gated readers. + +Verify: `go test ./internal/adapters/whisperx` and +`go test -race ./internal/adapters/whisperx`. + +Done when replay is available without whole-file buffering or producer-lifetime +ambiguity, and all existing HTTP behavior tests still pass. + +## Stage 6: Verify Replay Through the HTTP Adapter Boundary + +Status: Pending. + +Preparation: Existing injectable HTTPClient/RoundTripper seam and local-server +tests in `internal/adapters/whisperx/http_test.go`; app WhisperX wiring tests. + +1. Add a deterministic transport-boundary regression: consume part of the + original request, close it as a transport would, call GetBody, consume and + parse the replay using the request Content-Type, then return valid JSON. + Verify exact audio/language, installed output, and one application attempt. + This must fail before replay support; merely checking non-nil GetBody is + insufficient. Do not assert Go's internal retry count or require a flaky + handcrafted HTTP/2 peer in the default suite. +2. Cover replay-open/read failure, early response, parent cancellation during + replay, and disposal of all body handles. Reuse Stage 5 body coverage rather + than repeat every low-level case. Prove an intentionally abandoned original + body does not turn a valid replay response into failure. +3. Retain local HTTP tests for application retries, non-retryable responses, + response limits, invalid JSON, and bounded streaming; exercise that an + exhausted HTTP failure still leaves earlier speaker checkpoints reusable + using the existing stage/app seams. +4. Record the verification boundary honestly: controlled transport replay proves + Narratio's body contract, not recovery from every real proxy PROTOCOL_ERROR. + +Verify: `go test ./internal/adapters/whisperx ./internal/app ./internal/stage` +and race tests for changed packages. + +Done when the replay defect is protected at the adapter boundary and the +application retry/checkpoint behavior remains intact. + +## Stage 7: Document and Validate the Completed Feature + +Status: Pending. + +1. Update canonical transcribe and manifest internals for the collection, + reconciliation, durable-reset force behavior, serialized saves, and aggregate + gate. Remove obsolete all-speakers-before-canonical-write descriptions. +2. Update the WhisperX contract for replayable bounded streaming, application + attempts versus transport replay, resource ownership, and the lack of an + exactly-once server guarantee. Do not claim the remote protocol issue is fixed. +3. Update operations/troubleshooting with ordinary resume versus force, durable + partial success, old-manifest one-time retranscription on selecting transcribe, + downgrade limitations, and private-model-change guidance. No new CLI flags + or promises of automatic historical-file recovery. Keep detailed wire and + storage contracts in their canonical owners. +4. Review changes against the feature roadmap and the implementation-discipline + guidance. Do not elevate selected storage encoding, internal ordering, test + synchronization, or exact error wording into new product guarantees. +5. Run the full repository gates below; document results and update completion + statuses only after they pass. Leave release preparation for a separate task. ```sh go test ./... @@ -264,12 +352,10 @@ go test ./internal/config -run '^TestExamplesLoadAndValidate$' git diff --check ``` -Completion: The requested interface, failure and reuse behavior, examples, and -canonical documentation agree; all checks pass. No release is part of this -implementation sprint. - ## Open Questions -None. The consumer request resolves the product behavior. The choices above -settle implementation details without adding new operator restrictions, -downstream configuration ownership, or cache guarantees. +None requiring user input. The agreed roadmap supplies the product decisions; +this plan selects a completed-only manifest collection, conservative old-record +handling, and pull-based replay as implementation choices. No stronger remote +reliability guarantee, new operator restriction, or generic orchestration +surface is proposed. diff --git a/docs/roadmap/transcribe-checkpoints-and-replay.md b/docs/roadmap/transcribe-checkpoints-and-replay.md new file mode 100644 index 0000000..7f3ceb7 --- /dev/null +++ b/docs/roadmap/transcribe-checkpoints-and-replay.md @@ -0,0 +1,183 @@ +# Resumable Speaker Transcription and Replayable Uploads + +Status: Proposed; not implemented. + +## Goal and User Intent + +Preserve completed speaker transcriptions when another speaker fails, so an +ordinary run or resume does not repeat expensive successful work. Separately, +allow Go's HTTP transport to replay WhisperX uploads when it encounters a +recoverable connection or stream failure. + +The motivating incident exhausted four transcription attempts for one speaker +after three other speakers had completed. The final error reported an HTTP/2 +peer `PROTOCOL_ERROR` and an inability to replay the request body without +`Request.GetBody`. + +These are related reliability improvements, not a promise to eliminate all +service failures. This document owns the target state and scope; the +[implementation plan](implementation.md) owns ordered changes and concrete +storage and replay decisions. + +## Current Limitations + +The [transcribe stage](../internal/stage-transcribe.md) writes successful +responses into the invocation's run directory. It installs canonical transcripts +and returns its per-file metadata only after every speaker succeeds. A failure +therefore leaves no durable per-speaker progress that ordinary resume can use, +even though some completed JSON files may remain on disk. + +The [WhisperX adapter](../integrations/whisperx.md) already retries requests at +the application level, reopening audio for each attempt. Its multipart request +body is streamed through a pipe and has no replay callback. Transport-level +recovery within an attempt cannot reconstruct a consumed body. Providing that +capability addresses the replay limitation, not the underlying peer protocol +error; persistent failures can still require service or proxy investigation. + +## Targeted End State + +### Durable Per-Speaker Completion + +Each successfully transcribed speaker file is validated, saved, and recorded +durably before that work is treated as reusable. The session manifest remains +the progress authority. Follow the existing analyze-stage checkpoint pattern +without turning speakers into pipeline stages or introducing a general task +engine. + +Ordinary execution reconciles the current prepared audio set with recorded +speaker results. It reuses valid completed work and schedules failed, +unattempted, missing, or obsolete results. A result is not reusable merely +because a matching filename exists. Evidence binds it to the speaker/audio +identity, audio content, Narratio-visible recognition settings, and the saved +transcript's integrity. + +The relevant recognition settings remain the WhisperX service URL and language. +Retry count/delay, concurrency, timeout, credentials, and diagnostic locations +remain operational settings rather than reasons to discard completed results. +Changes to private service models or configuration remain an operator reason +to force transcription; Narratio does not inspect the service's private state. + +Changes to individual audio inputs require replacement of affected results, +not automatic retranscription of unrelated unchanged speakers. New speakers +need transcription; removed speakers do not remain in the current aggregate +output set. Records must describe the inputs actually used, not grant reuse to +old bytes under a changed identity. + +### Aggregate Completion, Failure, and Force + +The overall stage succeeds only when every currently required speaker has a +validated, durably recorded result. Incomplete per-speaker progress does not +make the aggregate stage successful or authorize downstream consumption of an +incomplete collection. Existing dependent-stage invalidation remains in effect. + +Retain bounded concurrency and the current fail-fast behavior: a speaker +failure cancels outstanding work. Results already checkpointed remain reusable +after failure, cancellation, or process interruption. A later ordinary run or +resume retries the remaining work; it does not need a new CLI command. + +Checkpoint writes must be serialized so concurrently finishing speakers cannot +overwrite each other's progress. A failed persistence operation must not +advertise an unrecorded result as complete. Previously committed progress remains +the recovery authority, and checkpoint errors must be surfaced rather than +reported as successful transcription. + +`--force` on transcribe regenerates all currently required speakers instead of +reusing their prior completions. Successful replacements are checkpointed too, +so an ordinary retry after a failed forced invocation can retain the newly +completed work. Old results for speakers not successfully regenerated must not +satisfy that unfinished forced replacement on the next ordinary retry. + +This deliberately relaxes the current all-speakers-before-canonical-installation +implementation: individual durable results may exist while the stage remains +failed or incomplete. The all-speakers-before-stage-success boundary remains. + +### Replayable WhisperX Requests + +Provide `Request.GetBody` so the HTTP transport can obtain a fresh body for the +same request. Replay reopens the audio and reproduces the multipart boundary, +filename, fields, and framing used by the original request. Independent bodies +must not share a consumed reader or mutable read position. + +Preserve bounded-memory streaming rather than buffering entire audio files in +memory. Correctly close every original and replay body and its audio resources; +preserve source-read errors and cancellation, including partial uploads and +early server responses. Successful response handling must not accidentally wait +on, or report errors solely from, an abandoned body instead of the replay that +produced the response. + +Keep the existing application-level retry count, delay, classification, and +per-attempt timeout semantics. Transport replays occur within an application +attempt and its context; they are not additional Narratio-configured attempts. +Do not add another custom retry loop, silently increase retries, or disable +HTTP/2 globally as a substitute for replay support. + +The adapter still validates and installs only successful JSON responses. +Request replay does not provide exactly-once server execution: a lost response +can cause repeated transcription work, as can existing application retries. +No new service-side idempotency protocol is required by this feature. + +## Required Change Areas + +- **Manifest and persistence:** Add transcribe-owned per-speaker progress and + evidence, with safe checkpoint updates and preservation through runner failure + handling. Review manifest validation, copying, serialization, and existing + publish/restore handling wherever the new records pass through those paths. +- **Stage execution and reuse:** Reconcile speaker records before scheduling, + checkpoint validated completions, rebuild aggregate outputs from the current + set, and account for partial progress in ordinary and forced execution. + Review successful-stage reuse as well as failed-stage retries so stale audio + or missing/corrupt transcript evidence is not bypassed by an aggregate skip. +- **Artifact lifecycle:** Use existing path, installation, integrity, and manifest + mechanisms. Ensure checkpointed results do not depend on an ephemeral run + directory surviving cleanup. Do not introduce a second authoritative ledger + or infer completion by scanning abandoned runs. +- **HTTP adapter:** Make multipart bodies reproducible and correctly manage + every replay's lifetime, without changing the WhisperX HTTP schema or requiring + an upstream service change. +- **Tests:** Cover per-speaker recovery after a later failure, interruption and + checkpoint failure; changed/missing inputs or outputs; force followed by an + ordinary retry; concurrent successful completions; and aggregate downstream + gating. Extend existing fixtures rather than duplicating analyze's complete + test suite. Exercise actual body reconstruction after partial consumption, + multipart equivalence, cancellation/resource cleanup, and existing bounded + streaming and retry behavior—not just the presence of a callback. Default + tests remain offline and use local/fake HTTP boundaries, never paid services. +- **Documentation:** Update the canonical + [stage guide](../internal/stage-transcribe.md), + [manifest guide](../internal/manifest.md), + [WhisperX contract](../integrations/whisperx.md), + [operations guide](../operations.md), and + [troubleshooting guide](../troubleshooting.md) when implemented. Explain + partial completion, ordinary retry versus force, and the distinction between + request replay limitations and persistent peer protocol failures. + +## Compatibility and Scope Boundaries + +Existing speaker identities, raw transcript formats, and the fixed pipeline +order remain unchanged. Existing manifests remain readable, but must not be +treated as containing per-speaker evidence they never recorded. Recovery relies +on verified recorded evidence, not guessed associations between old transcripts +and current audio. No general legacy-migration framework is part of the feature. + +Automatic adoption of unrecorded outputs from previously failed runs is outside +this feature. Such files may be recoverable after verifying their inputs, but +their presence alone does not prove completion. No recovery/import command, +speaker-selection CLI, historical invocation ledger, new service configuration +discovery, or broader orchestration refactor is included. + +The feature does not diagnose the remote service/proxy's underlying HTTP/2 +problem, guarantee recovery from persistent protocol errors, or change the +policy for continuing unrelated work after failure. + +## Completion Criteria + +For speakers A, B, C, and D: if A and B are checkpointed, C fails, and D has not +completed, ordinary resume retains A and B, retries C, and completes D. No +downstream stage receives an incomplete successful transcription collection. +Force replaces all four, while an ordinary retry can reuse checkpoints from +that forced replacement attempt. + +The HTTP transport can reconstruct the upload after consuming the original +body, without whole-file memory buffering, lost source errors, or stranded +upload resources. A persistent peer error still fails normally after the +configured application attempts, without discarding checkpointed speakers.