diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 94525c6..a75eba9 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,416 +1,272 @@ -# Implementation Plan: Speaker Checkpoints and Replayable Uploads +# Implementation Plan: Shared Stage Checkpoint Mechanics -Status: Implementation complete; Stages 1-7 complete. +Status: Not started. -## Scope and Instructions +## Scope and Authority + +Implement the agreed follow-up to v2.2.0: consolidate the thin checkpoint +mechanism shared by transcribe and analyze without combining their progress +models or changing operator behavior. There is no separate consolidation +roadmap in this checkout. This plan uses the accompanying discussion as its +scope: persistence is already centralized; candidate isolation and +save-before-promotion are the duplication to remove. The completed +[transcription roadmap](transcribe-checkpoints-and-replay.md) provides background, +not authorization to reimplement transcription or HTTP replay. The preceding +completed implementation plan remains in Git history. -Implement [the feature roadmap](transcribe-checkpoints-and-replay.md). This plan -replaces the completed Notarius-profile plan; that work remains in Git history. Complete the numbered stages in order, one bounded implementation prompt per stage for gpt-5.6-terra. Record completion and actual verification results here. -Intermediate stages are not a finished feature and must not be released alone. +This planning pass changes documentation only; the steps below are for the +subsequent implementation sprint. -Before working, read `AGENTS.md`, `docs/development.md`, all three policies in -`docs/policy/`, the roadmap, and the stage-specific preparation below. Use the -maintained Go 1.26 toolchain. Preserve operator choices, speaker filenames, -bounded concurrency, streaming, and existing credentials/settings. Do not add -new commands, dependencies, service requirements, arbitrary speaker limits, -strict transcript schemas, or a generic checkpoint framework. No release or -upstream service work is part of this plan. +Before each stage, read `AGENTS.md`, `docs/development.md`, and all three +documents in `docs/policy/`: architecture, documentation, and testing. Use +maintained Go 1.26. Prefer graph tools for code discovery; inspect current +files if their index is stale. Read the stage-specific preparation below, +rather than unrelated historical roadmaps. -## Settled Design Decisions +## Target and Scope Boundaries -### Progress Representation and Recovery +The existing `internal/manifest.Store` and `LocalStore` remain the session +manifest persistence interface and implementation. Encoding, schemas, +validation, normalization, and atomic file replacement remain unchanged. +There is no new store, journal, file format, dependency, command, configuration, +or migration. -Add optional `transcribe_state_version` (version 1) and `transcribe_speakers` -to `manifest.StageRecord`. The latter is a map keyed by the existing derived -speaker ID, containing completed results only. Absence means that work must be -done; a separate pending/running/failed state machine is unnecessary. Aggregate -stage status/error remains authoritative for invocation failure. A versioned -empty collection is meaningful: no earlier completions are reusable. +One private helper in `internal/stage` owns candidate preparation, calling the +existing store, and promoting the selected stage record only after successful +save. The existing typed analyze/transcribe checkpoint methods become thin +callers. This is execution plumbing: keep it beside the callers rather than +moving orchestration into the storage package or exporting a reusable API. -Use a typed `TranscribeSpeakerRecord` with fingerprint version/digest, -session-relative audio path, audio SHA-256 and size, one `ArtifactRecord` output -of kind `transcript_raw`, output size, and producer run ID. The output carries -its session-relative canonical path and SHA-256. No new transcript source IDs, -synthetic schema contracts, timestamps, or historical attempt collections are -needed. Existing aggregate diagnostics can retain attempt/status/duration data -for work performed in the current invocation; do not fabricate that information -for reused outputs. +Keep these responsibilities stage-owned: -Keep the session-manifest schema identifier unchanged for this optional additive -state; validate supported per-stage versions and retain strict decoding. New -Narratio reads old manifests. Old binaries may reject the added fields under -their existing strict decoding; document this downgrade limitation. +- Transcribe's completed-only speaker map, evidence checks, bounded concurrency, + mutex, checkpoint-failure latch, cancellation, pre-dispatch force reset, + output installation, and aggregate repair. +- Analyze's dispositions, selection, dependencies, catalog updates, working and + committed collections, failure reconciliation, invalidation, and outputs. +- Each stage's decision whether/when to checkpoint, typed validation, contextual + errors, and post-save working-state updates. +- Runner aggregate transitions, session locking, terminal failure saves, and + fixed dependent-stage invalidation. -For pre-feature manifests with no speaker evidence, regenerate transcription -once when transcribe is next selected, even if the aggregate status succeeded. -Old records do not bind individual transcript results to the audio bytes they -used. Do not guess that association from current files, old `per_file` metadata, -or abandoned run directories. This is the conservative upgrade path, not a new -general migration system. No other stage or unselected range is forcibly run. +Do not introduce a common subtask model, scheduler, transaction manager, global +mutex, automatic retry, additional checkpoint, or batching policy. Do not +migrate publish/restore, planning clones, cleanup, or ordinary runner saves: +their operations are not the demonstrated duplication. Leave HTTP replay and +all adapters unchanged. -### Reuse and Persistence +## Selected Design -Compute a versioned per-speaker fingerprint from speaker ID, session-relative -audio identity, actual audio SHA-256/size, and the existing transcribe semantic -configuration fingerprint (URL and language). Do not include invocation ID, -absolute workspace root, concurrency, retry settings, timeout, or credentials. -Stream input hashing with cancellation; do not trust stored input checksums -alone. Recheck input bytes before accepting a new result to catch changes -observed during transcription; fail that job without checkpointing mismatched -evidence. This is not a hostile-filesystem snapshot or concurrent-editor -transaction guarantee and requires no extra audio copies or file locks. +### Shared Save-and-Promote Operation -Validate reusable output identity, canonical location, JSON, checksum, and size. -Use existing confinement/file primitives. Missing or corrupt regular outputs -become work; unsafe paths/non-regular conflicts follow existing safety errors, -not recursive deletion. Missing required audio is an input error, not permission -to reuse an unverifiable transcript. Removed inputs are removed from the current -collection and aggregate outputs without deleting arbitrary files. +Add `internal/stage/checkpoint.go` with a private helper shaped as: -One stage-owned mutex serializes candidate collection construction, manifest -save, and installation of committed state. Save a detached candidate and only -then update live authority, following analyze's pattern. Workers must not mutate -the live manifest outside that serialized section. Keep expensive HTTP work -outside the mutex. Stop dispatch/acceptance on checkpoint failure, join workers -before return, and preserve the last successfully committed collection. +```go +func checkpointStageCollection( + ctx context.Context, + store manifest.Store, + path string, + live *manifest.Manifest, + stageName string, + update func(*manifest.StageRecord) error, +) error +``` -Before launching requests, durably checkpoint the reconciled collection. For -force, that collection is empty: this prevents later ordinary retries from -resurrecting pre-force outputs. If this initial save fails, do not start requests -or overwrite outputs. Force again clears the current collection again; ordinary -retry retains replacements already checkpointed after the previous force. +Names are implementation choices, not public contracts. Use a function and +small copying helpers, not a factory/options/interface framework. The two +existing typed checkpoint methods are the only production callers. They pass +the already-resolved manifest path; the helper does not discover paths or +inspect configuration. -Save each validated result at its canonical raw-transcript location before -checkpointing it. Run-local files remain staging/diagnostics, not reuse -dependencies. Bytes left by an unsuccessful checkpoint have no completion -authority. A worker may checkpoint a completed result using the parent stage -context even if a sibling canceled the work context; parent cancellation or a -checkpoint failure may prevent that save. Guarantee retention of committed -work, not all responses that happened to finish during cancellation. +The helper performs this sequence: -### Multipart Replay Mechanism +1. Check required live/store inputs without mutation. Copy a checkpoint + candidate and select a copied target record. If absent, provide an empty + candidate record to the caller, without inferring its name, status, or time. +2. Apply the supplied update to that candidate record only. On update error, + return without saving or promoting. The typed caller validates and fills + its version, collection, and stage-owned initialization/timestamps. +3. Invoke `Store.Save` once with the supplied context. Preserve error identity + through `%w` and useful path context. On error, do not promote, retry, + reload, or synthesize aggregate failure. +4. After success, install the saved target record and candidate manifest + `UpdatedAt` into live state. Preserve an existing target record's pointer + identity by assigning through it. Allocate missing live maps/records only + after success. Do not replace the entire live manifest or overwrite other + stage records. Copy mutable checkpoint collections on promotion so saved + candidates and caller working collections cannot mutate live progress. -Replace the pipe/producer upload mechanism with a small pull-based multipart -body backed by the audio reader. Use `mime/multipart.Writer` to encode a bounded -metadata prefix and suffix around the file payload: capture the prefix after -`CreateFormFile`, then encode the language field and closing boundary as the -suffix. Generate this framing once per application attempt. Each body combines -fresh readers for prefix, freshly opened audio, and suffix using `io.MultiReader`. -No full-audio buffer or temporary multipart spool is needed. +Callers serialize the whole operation. Transcribe's current mutex does so; +analyze executes artifacts sequentially. Do not add a lock to the helper or +promise support for concurrent independent manifest writers. Atomic replacement +visibility does not become power-loss durability or a transaction covering +transcript files. -The initial body and `Request.GetBody` use the same factory/framing. Each owns -an independent audio handle. Give bodies idempotent Close and context-triggered -closure (for example `context.AfterFunc`), without holding a mutex across a -blocking Read. Read/Close coordination must release a blocked source and retain -genuine source-read/close errors while not treating deliberate early closure as -source failure. The attempt owns cleanup of all created bodies, including -replays; reject new body creation after its context expires. +### Bounded Candidate Isolation -Remove the old producer `Wait` dependency. Track body-local source errors with -synchronization so a swallowed upload read error cannot become a successful -transcript, and a deliberately closed original body cannot poison a successful -replay. Keep the outer retry policy and per-attempt context. Do not add replay -counters to the user retry budget or depend on Go's exact internal retry count. -Preserve ordinary HTTP redirect policy; body replay can also enable Go's -body-preserving redirects. Do not introduce a new redirect restriction. +Factor `cloneManifestForTranscribeCheckpoint` into a shared checkpoint-scoped +copy helper in the new file. Use ordinary typed copies and existing +`manifest.CloneAnalyzeArtifactCollection` and +`manifest.CloneTranscribeSpeakerCollection`. Do not introduce reflection +cloning, JSON round trips, or a public general-purpose clone API. -## Stage 1: Add Durable Speaker Completion State +Copy the manifest value, stages map, stage records, and mutable collections +touched by these updates or `LocalStore.Save`. In particular, Save normalizes +analyze collections throughout the candidate, including on a transcribe +checkpoint. Clone those collections for every stage so normalization cannot +leak into live state on failure. Preserve the existing bounded copies of +input/artifact and output/log/config slices. Retain nil entries for existing +validation to reject, rather than silently repairing invalid manifests. -Status: Complete. +This is not arbitrary deep isolation of metadata. Neither the two update +callbacks nor the current Store mutates `map[string]any` metadata, so it may +remain shared read-only. Document that boundary and inspect every current +normalization write. Use existing typed clones for nested checkpoint evidence; +do not restrict legal metadata values or alter their Go/JSON representations +to simplify copying. -Completion marker (2026-09-21): Added optional versioned transcribe-owned -per-speaker completion records, strict load/save validation, isolated collection -cloning, legacy aggregate-only compatibility, and lifecycle-preservation -coverage. Verified with `go test ./internal/manifest` and `git diff --check`. +### Differences to Preserve -Preparation: `internal/manifest/manifest.go`, `store.go`, `analyze_state.go`, -their tests, and `docs/internal/manifest.md`. +- Analyze still requires an existing stage record. Its callback changes only + analyze version/collection, not transcribe-style initialization or timestamp + policy. It refreshes working and committed collections only after success. +- Transcribe retains missing-record initialization for direct stage calls: + its typed caller supplies name, running status, and initial timestamps when + needed. Its callback retains the existing stage `UpdatedAt` update. Copying + an existing record must not clear aggregate fields or invoke lifecycle + transitions. +- Existing transcribe live-record initialization is incomplete on the unusual + missing-record path; promoting the actual successfully saved candidate makes + its identity agree with disk. This does not add a new missing-record + rejection or change ordinary runner status transitions. +- Analyze rollback/failure handling and transcribe failure latching remain in + their execution owners. The shared operation never invents failed items. +- Preserve starting-checkpoint elision, force/reset semantics, empty versioned + collections, evidence and producer identity, legacy transcription handling, + supported zero-sized audio, and aggregate repair without HTTP calls. +- Do not freeze exact error wording, wall-clock values, copy choreography, or + iteration order into new contracts. No new schema, limits, filename rules, + statuses, or compatibility guarantees are authorized. -1. Implement the additive fields, record type, version constants, validation, - and deep-copy helper described above in a focused transcribe state file. -2. Validate ownership by the transcribe stage, supported versions, required - completed-record evidence, output kind, safe relative paths, SHA-256 values, - sizes, and producer identity consistency. Use existing speaker IDs as-is; - do not apply analysis-artifact key regexes or copy unrelated analysis limits. -3. Integrate validation with both load and save. Preserve speaker state through - running/failed/stale transitions and aggregate result clearing. Generic stage - invalidation does not discard checkpoints; reconciliation checks their data. -4. Test round-trip populated and versioned-empty collections, old manifests - without fields, malformed evidence, deep-copy isolation, and lifecycle - preservation. Avoid imposing JSON formatting or map order as contracts. +## Stage 1: Add the Shared Checkpoint Operation -Verify: `go test ./internal/manifest`. +Status: Pending. -Done when durable storage can represent partial completion independently of -aggregate status, without activating new stage behavior yet. +Preparation: `internal/manifest/store.go`, `manifest.go`, `analyze_state.go`, +`transcribe_state.go`; `internal/stage/analyze_execution_state.go`, +`transcribe_execution_state.go`; `docs/internal/manifest.md`. -## Stage 2: Implement Read-Only Speaker Reconciliation +1. Add the private helper and bounded candidate copying described above. + Leave existing production checkpoint paths unchanged until their migration + stages. Do not duplicate manifest record-validation rules. +2. Add focused `internal/stage/checkpoint_test.go` coverage with real + `manifest.LocalStore` and temporary directories for successful save/reload. + Use a small Store wrapper only to observe/fail the persistence boundary. +3. Prove live progress remains old during Save, matches saved progress after + success, and does not change on update, validation, or save failure. Include + both typed collections and a transcribe checkpoint alongside analyze + records subject to save-time normalization. A failure test must exercise + candidate timestamp/normalization mutation before returning an error, not + merely fail before any work. +4. Prove unrelated stage/aggregate data survives, existing target references + observe updates, missing-target creation is invisible on failure, and later + caller collection edits do not mutate committed records. Preserve context + and error identity. Test the supported copy/write boundary, not arbitrary + malicious Store mutation. -Status: Complete. - -Completion marker (2026-09-21): Extracted shared deterministic transcribe job -construction and added read-only, cancellation-aware per-speaker input and -output reconciliation. Covered unchanged reuse, semantic and operational -settings, changed/added/removed inputs, legacy and forced work, missing/corrupt -outputs, unsafe evidence, and missing audio without connecting runtime skip or -checkpoint behavior. Verified with -`go test ./internal/stage ./internal/manifest` and `git diff --check`. - -Preparation: `internal/stage/transcribe.go`, `semantic_contracts_initial.go`, -`analyze_reconciliation.go`, `extract_resume.go`, their tests, and the artifact -path/integrity helpers they use. - -1. Extract the existing audio discovery/job construction into a shared, - transcribe-local reconciliation path. Preserve current input selection and - speaker identity behavior. Build sorted jobs and classify completed results - against the current input set and evidence defined above. -2. Add pure fingerprint construction and streamed input/output inspection. - Resolve manifest input paths with existing session-relative/absolute reading - helpers; do not change input precedence or assume the current working directory - is the session root. Newly written checkpoint paths are always relative. -3. Return current records and work-to-do without writing files or manifest state. - Force selects all work and an empty starting collection; old unversioned - transcribe state supplies no reusable records. Do not connect aggregate skip - validation until execution can produce the new evidence in Stage 3. -4. Test unchanged reuse; one changed audio among unchanged speakers; URL/language - changes; operational-setting exclusions; added/removed speakers; missing and - corrupt regular outputs; unsafe record paths; missing audio; and force. - Prefer compact tables and real temporary files over mocked hashing. - -Verify: `go test ./internal/stage ./internal/manifest`. - -Done when one read-only decision path describes exactly which speakers need -work, without altering existing runtime behavior prematurely. - -## Stage 3: Execute and Checkpoint Speakers Incrementally - -Status: Complete. - -Completion marker (2026-09-21): Connected reconciliation to bounded execution, -durably checkpointed the starting and per-speaker collections through detached -manifest candidates, preserved committed work across failures and cancellation, -rebuilt deterministic aggregate outputs from reusable evidence, and added -speaker-aware resume validation. Updated production-style fixtures and covered -serialized concurrent saves, force/retry replacement, input rechecks, save -failures, partial recovery, and aggregate reconstruction. Verified with -`go test ./internal/stage ./internal/app`, `go test -race ./internal/stage`, and -`git diff --check`. - -Preparation: `internal/stage/transcribe.go`, `analyze_execution_state.go`, -`run_local.go`, -`internal/app/runner.go`, and transcribe/checkpoint test fixtures. - -1. Wire reconciliation into Run, require the existing ManifestStore collaborator, - and checkpoint the starting collection before launching HTTP work. Implement - detached candidate saves and serialized live-state updates. Audit Store.Save's - normalization side effects so a failed candidate save cannot mutate shared - authority; use detached records/collections wherever it may write. -2. Retain bounded errgroup execution and fail-fast cancellation. Check the work - context before beginning each queued request. After success, validate adapter - output path/JSON, recheck audio evidence, materialize canonical bytes, compute - output evidence, and checkpoint the new record. Check a checkpoint-failure - latch under the serialization lock before accepting further completions. -3. Preserve committed successes on all error/cancellation exits, including - starting-save, canonical-write, per-speaker-save, and aggregate-save failures. - Return meaningful speaker/error context and let the runner retain its existing - terminal failure behavior. Do not report partial aggregate outputs as success. -4. Assemble final outputs in deterministic speaker order from the reconciled - completed records, including reused speakers, with original producer IDs and - checksums. If all records are current but aggregate status is failed/running, - finish without WhisperX calls and rebuild the aggregate success. -5. Implement `transcribeStage.ValidateResume` using the same read-only - reconciliation, so a succeeded stage cannot skip changed audio or invalid - transcripts. Retain the existing URL/language semantic check and runner - invalidation behavior; no whole-config digest invalidation. -6. Adapt existing tests that require no canonical partial files: the new contract - permits checkpointed files but forbids aggregate success. Update fixtures to - provide real manifest stores and records rather than a production fallback - that silently disables checkpointing. -7. Test A/B completion followed by C failure and D unattempted (concurrency 1), - reload and retry only C/D; concurrently finishing speakers without lost saves; - failure and cancellation after a checkpoint; save failure retaining earlier - authority; and force failure followed by ordinary retry without old-result reuse. - -Verify: `go test ./internal/stage ./internal/app` and +Verify: `go test ./internal/manifest ./internal/stage` and `go test -race ./internal/stage`. -Done when incremental progress and aggregate success work together safely under -ordinary, concurrent, canceled, and forced execution. +Done when the shared integrity boundary is tested without changing existing +stage behavior. Temporary copying overlap is removed in Stage 3. -## Stage 4: Verify Runner, Merge, and Durable Lifecycle Integration +## Stage 2: Migrate Analyze -Status: Complete. +Status: Pending. -Completion marker (2026-09-22): Added real-persistence application coverage for -cross-invocation retry, interrupted running state, failed aggregate saves, -force/reset persistence, downstream invalidation, and one-time legacy upgrade. -Versioned merge input discovery now requires aggregate success and an exact -match between current speaker evidence and declared aggregate outputs, while -legacy/manual directory fallback remains available. Restore coverage preserves -session-relative speaker paths, leaves unpublished raw transcripts absent, and -canonical transcripts remain reusable after run-directory cleanup. Verified -with `go test ./internal/app ./internal/stage ./internal/manifest`, -`go test -race ./internal/app`, and `git diff --check`. +Preparation: `internal/stage/analyze_execution_state.go`, +`analyze_checkpoint_test.go`, `analyze_failure_test.go`, +`internal/app/analyze_checkpoint_recovery_test.go`, and +`docs/internal/stage-analyze.md`. -Preparation: `internal/app/runner.go`, `runner_runtime.go`, -`analyze_checkpoint_recovery_test.go`, `runner_terminal_failure_test.go`, -`restore_manifest_paths.go`, `internal/stage/merge.go`, and existing publish, -restore, and cleanup tests routed from `docs/internal/overview.md`. +1. Replace manual candidate copying, saving, and live promotion in analyze + with the shared helper. Retain typed validation, required-record checking, + the typed update callback, and committed/working collection refresh. +2. Preserve starting-checkpoint elision, selected-artifact behavior, catalog + sequencing, dependent invalidation on changed output, and rollback after + checkpoint failure. Do not change the runner's terminal failure save or + joined errors. +3. Run existing tests before adding coverage. Retain tests proving earlier + artifacts survive failure/cancellation, unsaved output bytes gain no + authority, selected/forced work preserves appropriate unrelated artifacts, + and interrupted runs recover across store reloads. Add only missing + integration coverage; do not duplicate Stage 1's isolation matrix. -1. Add application-level tests with real manifest persistence and fake WhisperX: - failed stage across separate invocations; all speakers committed before a - failed aggregate save; interrupted running state; force/reset persistence; - and unchanged upstream/succeeded downstream invalidation behavior. Inject - errors at durable boundaries, not every private method call. -2. Close `discoverRawTranscripts`' directory-scan fallback for versioned - transcribe state: incomplete aggregate state is not usable, and completed - state consumes only declared current aggregate outputs. Keep legacy/manual - fallback behavior for manifests without the new state rather than banning - previously allowed standalone workflows. Test that partial files and removed - speaker files cannot leak into a versioned merge, including direct stage calls. -3. Audit manifest copies/projections, remote manifest publication, restored path - handling, and cleanup. Preserve relative speaker paths on restore and verify - they resolve under the destination session, not the producer machine. Reuse - existing confinement validation, not a second path-policy layer. -4. Add focused round-trip coverage through existing publish/restore fixtures for - speaker records. Restored outputs absent from the published selection are - non-reusable locally; do not silently expand remote publication to every raw - speaker or upload audio as a new default. Removing run directories must not - remove canonical checkpointed transcripts or make reuse require run files. -5. Verify old succeeded manifests are readable and rerun once when transcribe is - selected, then reuse normally. Keep old manifests usable for other commands - and bounded ranges. Do not automatically import old unrecorded JSON files. +Verify: `go test ./internal/stage ./internal/app ./internal/manifest` and +`go test -race ./internal/stage ./internal/app`. -Verify: `go test ./internal/app ./internal/stage ./internal/manifest` and -`go test -race ./internal/app`. +Done when analyze delegates persistence choreography while progress, artifact +selection, and failure behavior remain unchanged. -Done when the new authority survives runner and storage lifecycles and cannot -expose incomplete transcripts downstream. +## Stage 3: Migrate Transcribe and Remove Duplicate Machinery -## Stage 5: Implement Replayable Streaming Multipart Bodies +Status: Pending. -Status: Complete. +Preparation: `internal/stage/transcribe_execution_state.go`, `transcribe.go`, +`transcribe_resume.go`, `transcribe_checkpoint_test.go`, +`transcribe_reconciliation_test.go`, +`internal/app/transcribe_checkpoint_recovery_test.go`, and +`docs/internal/stage-transcribe.md`. -Completion marker (2026-09-22): Replaced the producer-goroutine/pipe upload -with bounded standard-library multipart framing and independently opened, -pull-based streaming bodies shared by the initial request and `GetBody`. -Bodies are tracked, idempotently closed, canceled with their attempt context, -and retain genuine read/close failures without treating an intentionally -abandoned original as a replay failure. Added focused coverage for identical -framing, partial-read restart, simultaneous handles, source failures, and -blocked-reader release. Verified with `go test ./internal/adapters/whisperx`, -`go test -race ./internal/adapters/whisperx` (including ten repeated runs), and -`git diff --check`. +1. Delegate `checkpointCollectionLocked` to the shared helper, retaining + typed initialization/update and successful working-collection refresh. + Keep the existing mutex across candidate creation, save, and promotion. + Do not move HTTP/hashing work into that critical section or add a lock. +2. Preserve failure-latch checks and worker joining, the parent versus worker + context distinction, force's durable empty starting collection, and + ordinary retry's reuse of committed forced replacements. +3. Remove `cloneManifestForTranscribeCheckpoint`. Its current use by a + read-only resume test is not a reason to retain duplicate production code. + Use an independent test snapshot or the shared bounded copy where sufficient, + without exporting a helper for tests. +4. Retain existing real-store and concurrency coverage: no lost completions; + pre-dispatch save failure prevents requests/canonical writes; later failure + retains prior authority; interrupted and aggregate-save recovery; failed + force followed by retry; run-directory independence; zero-sized audio; + removed/mismatched aggregate repair without additional WhisperX requests + or removed-speaker merge input. Do not weaken these regression assertions. -Preparation: `internal/adapters/whisperx/http.go`, `http_test.go`, -`docs/integrations/whisperx.md`, and the installed Go net/http Request/GetBody -contract. Keep this change inside the WhisperX adapter. +Verify: `go test ./internal/stage ./internal/app ./internal/manifest` and +`go test -race ./internal/stage ./internal/app`. -1. Implement the pull-based framing/body factory described above, using standard - multipart encoding rather than hand-escaped MIME headers. Add no new payload - cap or upload-size setting. Preserve file-before-language wire semantics and - the existing form names. -2. Install the initial body and GetBody with the same per-attempt boundary. - Track and close all opened bodies even if a custom transport returns early; - make cleanup/context callbacks safe against overlapping reads and closes. - A GetBody failure must be returned, not converted into an empty upload. -3. Replace pipe-specific Wait/error paths in doTranscribeAttempt with source-error - handling appropriate to pull bodies. Genuine source errors remain visible; - intentional closure during transport retry is not itself a source error. - Keep response bounds, JSON validation, output installation, retry - classification/delay, cancellation, and application attempt reporting intact. -4. Remove obsolete producer-only helpers rather than retaining two upload paths. - Adapt tests tied to producer goroutines to assert streaming and resource - release, which are the actual contracts. -5. Add focused body tests: original/replayed bytes parse identically; replay - after partial consumption starts from the beginning; simultaneous bodies use - independent handles; source failures propagate; Close/cancel releases a - blocked reader without deadlock. Use synthetic audio and gated readers. +Done when both stages use one save/promote implementation and one bounded +copy implementation, with their different models and synchronization intact. -Verify: `go test ./internal/adapters/whisperx` and -`go test -race ./internal/adapters/whisperx`. +## Stage 4: Document, Review Scope, and Validate -Done when replay is available without whole-file buffering or producer-lifetime -ambiguity, and all existing HTTP behavior tests still pass. +Status: Pending. -## Stage 6: Verify Replay Through the HTTP Adapter Boundary +Preparation: `docs/internal/manifest.md`, `docs/internal/stage-transcribe.md`, +`docs/internal/stage-analyze.md`, documentation/testing policies, and the +implementation diff against this plan's starting commit. -Status: Complete. - -Completion marker (2026-09-22): Added a deterministic transport regression -that abandons a partially consumed original request, obtains and parses a fresh -multipart replay from `GetBody`, and installs the returned transcript within one -application attempt. Covered replay open/read failures, early responses, -parent cancellation during replay, and disposal of every opened source, while -proving an abandoned original does not poison a valid replay. Added a real HTTP -retry-exhaustion integration that durably retains an earlier speaker and resumes -only the uncommitted speakers. This controlled boundary verifies Narratio's -replay contract, not recovery from every real proxy protocol failure. Verified -with `go test ./internal/adapters/whisperx ./internal/app ./internal/stage`, -`go test -race ./internal/adapters/whisperx ./internal/app ./internal/stage`, -ten repeated race-enabled adapter runs, and `git diff --check`. - -Preparation: Existing injectable HTTPClient/RoundTripper seam and local-server -tests in `internal/adapters/whisperx/http_test.go`; app WhisperX wiring tests. - -1. Add a deterministic transport-boundary regression: consume part of the - original request, close it as a transport would, call GetBody, consume and - parse the replay using the request Content-Type, then return valid JSON. - Verify exact audio/language, installed output, and one application attempt. - This must fail before replay support; merely checking non-nil GetBody is - insufficient. Do not assert Go's internal retry count or require a flaky - handcrafted HTTP/2 peer in the default suite. -2. Cover replay-open/read failure, early response, parent cancellation during - replay, and disposal of all body handles. Reuse Stage 5 body coverage rather - than repeat every low-level case. Prove an intentionally abandoned original - body does not turn a valid replay response into failure. -3. Retain local HTTP tests for application retries, non-retryable responses, - response limits, invalid JSON, and bounded streaming; exercise that an - exhausted HTTP failure still leaves earlier speaker checkpoints reusable - using the existing stage/app seams. -4. Record the verification boundary honestly: controlled transport replay proves - Narratio's body contract, not recovery from every real proxy PROTOCOL_ERROR. - -Verify: `go test ./internal/adapters/whisperx ./internal/app ./internal/stage` -and race tests for changed packages. - -Done when the replay defect is protected at the adapter boundary and the -application retry/checkpoint behavior remains intact. - -## Stage 7: Document and Validate the Completed Feature - -Status: Complete. - -Completion marker (2026-09-22): Updated the canonical transcribe, merge, -manifest, WhisperX, operations, and troubleshooting documentation for durable -speaker reconciliation, serialized checkpoints, aggregate gating, durable -force reset, legacy upgrade and downgrade limits, replayable streaming bodies, -resource ownership, and the absence of an exactly-once remote guarantee. -Reviewed the completed implementation against the feature roadmap and -implementation-discipline guidance. Verified with `go test ./...`, -`go test -race ./...`, `go vet ./...`, `go build ./...`, -`go test ./internal/doccheck`, -`go test ./internal/config -run '^TestExamplesLoadAndValidate$'`, and -`git diff --check`. - -1. Update canonical transcribe and manifest internals for the collection, - reconciliation, durable-reset force behavior, serialized saves, and aggregate - gate. Remove obsolete all-speakers-before-canonical-write descriptions. -2. Update the WhisperX contract for replayable bounded streaming, application - attempts versus transport replay, resource ownership, and the lack of an - exactly-once server guarantee. Do not claim the remote protocol issue is fixed. -3. Update operations/troubleshooting with ordinary resume versus force, durable - partial success, old-manifest one-time retranscription on selecting transcribe, - downgrade limitations, and private-model-change guidance. No new CLI flags - or promises of automatic historical-file recovery. Keep detailed wire and - storage contracts in their canonical owners. -4. Review changes against the feature roadmap and the implementation-discipline - guidance. Do not elevate selected storage encoding, internal ordering, test - synchronization, or exact error wording into new product guarantees. -5. Run the full repository gates below; document results and update completion - statuses only after they pass. Leave release preparation for a separate task. +1. Update manifest internals to identify the shared checkpoint owner and copy/ + save/promote boundary. Stage guides retain stage-specific policy and link + to that explanation instead of duplicating shared implementation details. + Do not rewrite immutable release notes or announce new operator behavior. +2. Assess tests collectively. Consolidate genuinely duplicated persistence + mechanics coverage, but retain stage/application recovery tests proving + correct integration. Avoid test-count/LOC quotas. Record a brief summary + here of duplicated mechanisms removed and thin wrappers retained. +3. Confirm no schema, validation policy, scheduling, force, concurrency, output + selection, stage order, adapter call, or configuration change slipped into + the refactor. Additional restrictions require a separate decision, not + tests that retrospectively declare new policy. +4. Run all gates below and record actual results. Mark this plan complete only + when both migrations and verification pass. Do not release, tag, push, delete + other roadmaps, or expand into a repository-wide cloning refactor. ```sh go test ./... @@ -422,10 +278,12 @@ go test ./internal/config -run '^TestExamplesLoadAndValidate$' git diff --check ``` +Done when duplication is removed, recovery remains covered, documentation +matches the implementation, and all gates pass. + ## Open Questions -None requiring user input. The agreed roadmap supplies the product decisions; -this plan selects a completed-only manifest collection, conservative old-record -handling, and pull-based replay as implementation choices. No stronger remote -reliability guarantee, new operator restriction, or generic orchestration -surface is proposed. +None requiring a product decision. A private stage-level helper is selected +because persistence already has an owner and the duplication is execution +plumbing. Model unification, library extraction, generalized transactions, and +broader cloning cleanup remain out of scope. diff --git a/docs/roadmap/transcribe-checkpoints-and-replay.md b/docs/roadmap/transcribe-checkpoints-and-replay.md index 97600fd..98959d1 100644 --- a/docs/roadmap/transcribe-checkpoints-and-replay.md +++ b/docs/roadmap/transcribe-checkpoints-and-replay.md @@ -15,9 +15,9 @@ peer `PROTOCOL_ERROR` and an inability to replay the request body without `Request.GetBody`. These are related reliability improvements, not a promise to eliminate all -service failures. This document owns the target state and scope; the -[implementation plan](implementation.md) owns ordered changes and concrete -storage and replay decisions. +service failures. This document records the target state and scope. The +completed implementation sequence is retained in Git history; current behavior +is documented in the canonical guides linked below. ## Pre-implementation Limitations