eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 27 Wiki Activity

Plan resumable transcription and replayable uploads

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-21 15:50:48 +00:00
parent 3438fe82c9
commit 89ada1b269
2 changed files with 487 additions and 218 deletions
Download Patch File Download Diff File Expand all files Collapse all files

522
docs/roadmap/implementation.md
View File

@@ -1,258 +1,346 @@
# Implementation Plan: Notarius Configuration Profile Selection
# Implementation Plan: Speaker Checkpoints and Replayable Uploads
Status: Complete; all stages implemented and verified.
Status: Ready; all stages pending.
## Scope and Working Instructions
## Scope and Instructions
Implement the [feature roadmap](notarius-config-profile.md), whose authority is
the [consumer request](feature-request-notarius-config-profile.yml). Complete
the numbered stages in order. Each stage is a bounded task for a gpt-5.6-terra
coding agent. Do not treat completion of an intermediate stage as completion of
the feature. Record completed stages and actual verification results here.
Implement [the feature roadmap](transcribe-checkpoints-and-replay.md). This plan
replaces the completed Notarius-profile plan; that work remains in Git history.
Complete the numbered stages in order, one bounded implementation prompt per
stage for gpt-5.6-terra. Record completion and actual verification results here.
Intermediate stages are not a finished feature and must not be released alone.
Before each stage, read `docs/development.md`, all documents under
`docs/policy/`, the feature roadmap, and that stage's preparation list. Follow
AGENTS.md discovery guidance. Use existing collaborators and fixtures; do not
build a new configuration, cache, or profile-management subsystem. Tests must
remain offline and independent of installed Notarius binaries and credentials.
Do not change dependencies, implement Ansible changes, tag, or release as part
of this plan.
Before working, read `AGENTS.md`, `docs/development.md`, all three policies in
`docs/policy/`, the roadmap, and the stage-specific preparation below. Use the
maintained Go 1.26 toolchain. Preserve operator choices, speaker filenames,
bounded concurrency, streaming, and existing credentials/settings. Do not add
new commands, dependencies, service requirements, arbitrary speaker limits,
strict transcript schemas, or a generic checkpoint framework. No release or
upstream service work is part of this plan.
## Implementation Decisions
## Settled Design Decisions
These choices resolve implementation details without adding restrictions to
the feature request:
### Progress Representation and Recovery
- Represent the authored field as `ConfigProfile *string` with YAML/JSON name
`config_profile` and `omitempty`, in both typed configuration and its safe
YAML projection. Nil means omitted/null. A non-nil pointer preserves an
explicitly supplied empty string as well as `default` and named selections.
- Carry the same optional value in the Notarius adapter request. Nil emits no
flag; non-nil emits `--config-profile` followed by the exact supplied string
as a separate argument, including an empty string. Do not trim, lowercase,
substitute Narratio's profile, impose a name regex, or inspect profile files.
Notarius owns validation and selector interpretation.
- For reuse evidence only, use one small stage-local helper that maps nil,
`""`, and exact `"default"` to the empty identity; other values remain
unchanged. Both extraction fingerprints use it. This does not mutate the
authored field or change CLI forwarding. Do not attempt to recreate the
downstream parser's complete normalization rules.
- Add `ConfigProfile string` with `json:"config_profile,omitempty"` to both
existing fingerprint payloads, populated only where extraction is enabled.
Preserve existing field ordering and the current semantic fingerprint version
(1). With no non-base selector the serialized payload remains unchanged;
a named selection adds evidence that old records cannot already possess.
No dual-reader, manifest migration, new durable metadata field, or broad
fingerprint-version bump is needed. This is a minimal additive encoding,
not a permanent promise to freeze fingerprint bytes across future releases.
- Profile changes trigger the existing non-resumable/invalidation paths, not
force mode. Preserve configured resume/cache controls, force precedence,
credentials, output contracts, and stage-range behavior. Do not run extraction
outside the operator's selected range or add a separate testing workspace.
- Preserve the existing division of ownership: private downstream file changes
behind an unchanged selector require operator-directed force when needed.
No downstream configuration validation subprocess or runtime version gate
is added.
Add optional `transcribe_state_version` (version 1) and `transcribe_speakers`
to `manifest.StageRecord`. The latter is a map keyed by the existing derived
speaker ID, containing completed results only. Absence means that work must be
done; a separate pending/running/failed state machine is unnecessary. Aggregate
stage status/error remains authoritative for invocation failure. A versioned
empty collection is meaningful: no earlier completions are reusable.
## Stage 1: Add Configuration and Adapter Support
Use a typed `TranscribeSpeakerRecord` with fingerprint version/digest,
session-relative audio path, audio SHA-256 and size, one `ArtifactRecord` output
of kind `transcript_raw`, output size, and producer run ID. The output carries
its session-relative canonical path and SHA-256. No new transcript source IDs,
synthetic schema contracts, timestamps, or historical attempt collections are
needed. Existing aggregate diagnostics can retain attempt/status/duration data
for work performed in the current invocation; do not fabricate that information
for reused outputs.
Status: Complete.
Keep the session-manifest schema identifier unchanged for this optional additive
state; validate supported per-stage versions and retain strict decoding. New
Narratio reads old manifests. Old binaries may reject the added fields under
their existing strict decoding; document this downgrade limitation.
Completion marker (2026-09-21): Added composed optional configuration and safe
inspection support, adapter request/argument forwarding, copy auditing, and
focused configuration and subprocess coverage. Verified with
`go test ./internal/config ./internal/adapters/notarius`.
For pre-feature manifests with no speaker evidence, regenerate transcription
once when transcribe is next selected, even if the aggregate status succeeded.
Old records do not bind individual transcript results to the audio bytes they
used. Do not guess that association from current files, old `per_file` metadata,
or abandoned run directories. This is the conservative upgrade path, not a new
general migration system. No other stage or unselected range is forcibly run.
Preparation:
### Reuse and Persistence
- `docs/config.md` (composition, profiles, Notarius, credentials).
- `docs/integrations/notarius.md` (invocation and failure behavior).
- `internal/config/config.go`, `credential_projection.go`,
`pipeline_configkit.go`, `pipeline_normalize.go`, and `notarius_test.go`.
- `internal/adapters/notarius/runner.go`, `subprocess.go`, `fake.go`, and
`subprocess_test.go`.
Compute a versioned per-speaker fingerprint from speaker ID, session-relative
audio identity, actual audio SHA-256/size, and the existing transcribe semantic
configuration fingerprint (URL and language). Do not include invocation ID,
absolute workspace root, concurrency, retry settings, timeout, or credentials.
Stream input hashing with cancellation; do not trust stored input checksums
alone. Recheck input bytes before accepting a new result to catch changes
observed during transcription; fail that job without checkpointing mismatched
evidence. This is not a hostile-filesystem snapshot or concurrent-editor
transaction guarantee and requires no extra audio copies or file locks.
Work:
Validate reusable output identity, canonical location, JSON, checksum, and size.
Use existing confinement/file primitives. Missing or corrupt regular outputs
become work; unsafe paths/non-regular conflicts follow existing safety errors,
not recursive deletion. Missing required audio is an input error, not permission
to reuse an unverifiable transcript. Removed inputs are removed from the current
collection and aggregate outputs without deleting arbitrary files.
1. Add the optional field to `NotariusConfig` and the `safeNotarius` projection,
including its returned value. Let existing ConfigKit composition handle
overrides and null; do not add a default or resolve this name as a path.
2. Add the optional field to `notarius.RunRequest` and append the argument pair
when non-nil in `SubprocessRunner.Run`, before `--json`. Leave subprocess
environment construction and error propagation unchanged.
3. Audit any explicit Notarius configuration/request copies for the new field.
Update actual omissions without introducing a generic cloning framework.
4. Extend real configuration-load tests to cover a base selection, an overlay
replacing it with a differently named downstream selection, and an overlay
clearing it with null. Cover omitted, empty, and `default` values compactly.
Verify the safe effective-YAML output retains a supplied value, including
explicit empty, and round-trips without leaking credential values. Reuse
existing credential tests rather than duplicating their full matrix.
5. Extend adapter invocation tests for omitted and supplied selections,
including empty and `default`. Assert the argument/value boundary rather
than imposing new ordering requirements on unrelated arguments. Reuse the
existing nonzero-exit test with a selected profile to prove there is no
fallback or successful receipt processing after failure.
One stage-owned mutex serializes candidate collection construction, manifest
save, and installation of committed state. Save a detached candidate and only
then update live authority, following analyze's pattern. Workers must not mutate
the live manifest outside that serialized section. Keep expensive HTTP work
outside the mutex. Stop dispatch/acceptance on checkpoint failure, join workers
before return, and preserve the last successfully committed collection.
Verification:
Before launching requests, durably checkpoint the reconciled collection. For
force, that collection is empty: this prevents later ordinary retries from
resurrecting pre-force outputs. If this initial save fails, do not start requests
or overwrite outputs. Force again clears the current collection again; ordinary
retry retains replacements already checkpointed after the previous force.
```sh
go test ./internal/config ./internal/adapters/notarius
```
Save each validated result at its canonical raw-transcript location before
checkpointing it. Run-local files remain staging/diagnostics, not reuse
dependencies. Bytes left by an unsuccessful checkpoint have no completion
authority. A worker may checkpoint a completed result using the parent stage
context even if a sibling canceled the work context; parent cancellation or a
checkpoint failure may prevent that save. Guarantee retention of committed
work, not all responses that happened to finish during cancellation.
Completion: Configuration composition and safe inspection preserve the field,
and the adapter forwards it correctly. The executing stage is connected in
Stage 2 together with reuse tracking, so it cannot start using untracked profile
selections as an intermediate implementation.
### Multipart Replay Mechanism
## Stage 2: Wire Extraction and Profile-Sensitive Reuse
Replace the pipe/producer upload mechanism with a small pull-based multipart
body backed by the audio reader. Use `mime/multipart.Writer` to encode a bounded
metadata prefix and suffix around the file payload: capture the prefix after
`CreateFormFile`, then encode the language field and closing boundary as the
suffix. Generate this framing once per application attempt. Each body combines
fresh readers for prefix, freshly opened audio, and suffix using `io.MultiReader`.
No full-audio buffer or temporary multipart spool is needed.
Status: Complete.
The initial body and `Request.GetBody` use the same factory/framing. Each owns
an independent audio handle. Give bodies idempotent Close and context-triggered
closure (for example `context.AfterFunc`), without holding a mutex across a
blocking Read. Read/Close coordination must release a blocked source and retain
genuine source-read/close errors while not treating deliberate early closure as
source failure. The attempt owns cleanup of all created bodies, including
replays; reject new body creation after its context expires.
Completion marker (2026-09-21): Wired the selected profile into extraction,
added a shared base-selector reuse identity to both existing fingerprints, and
covered request forwarding, force precedence, semantic sensitivity, prior
no-profile encoding, and focused resume validation. Verified with
`go test ./internal/stage ./internal/adapters/notarius ./internal/config`.
Remove the old producer `Wait` dependency. Track body-local source errors with
synchronization so a swallowed upload read error cannot become a successful
transcript, and a deliberately closed original body cannot poison a successful
replay. Keep the outer retry policy and per-attempt context. Do not add replay
counters to the user retry budget or depend on Go's exact internal retry count.
Preserve ordinary HTTP redirect policy; body replay can also enable Go's
body-preserving redirects. Do not introduce a new redirect restriction.
Preparation:
## Stage 1: Add Durable Speaker Completion State
- `docs/internal/stage-extract.md`.
- `internal/stage/extract.go`, `extract_resume.go`,
`semantic_contracts_delivery.go`, and their focused tests.
- In particular, `TestExtractStageResolvesManifestInputAndBuildsExactRequest`,
`TestExtractSemanticConfigSensitivityAndOperationalExclusions`, and
`TestExtractStageResumeValidationRequiresFreshEvidence`.
Status: Pending.
Work:
Preparation: `internal/manifest/manifest.go`, `store.go`, `analyze_state.go`,
their tests, and `docs/internal/manifest.md`.
1. Pass `cfg.ConfigProfile` to the adapter request when extraction executes.
Keep the field independent of credential forwarding and force overrides.
2. Implement the reuse-identity helper and additive fingerprint fields described
above. Update both `extractSemanticConfig` and `fingerprintDocument`, using
the same identity in semantic comparison and `extractionFingerprint`.
Keep disabled extraction insensitive to unused downstream settings.
3. Extend the existing stage request test to prove the authored selection
reaches the adapter. Cover ordinary and forced execution sufficiently to
show that profile forwarding does not alter existing cache-control precedence.
4. Extend semantic-sensitivity coverage: base to named, named to different named,
and named to base change evidence; unchanged selection does not. Verify
nil/empty/`default` share base evidence and disabled settings remain inert.
5. Exercise the focused resume validator with an actual valid recorded bundle:
it remains reusable for the same selection and becomes non-resumable for a
changed selection. This guards the second fingerprint path independently
of the application's semantic comparison.
6. Check the additive encoding against the prior no-profile payload shape once
in focused coverage or a recorded development check. Avoid whole-manifest
snapshots and a permanent historical compatibility harness; retain existing
input/output/reference sorting and all unrelated fingerprint fields.
1. Implement the additive fields, record type, version constants, validation,
and deep-copy helper described above in a focused transcribe state file.
2. Validate ownership by the transcribe stage, supported versions, required
completed-record evidence, output kind, safe relative paths, SHA-256 values,
sizes, and producer identity consistency. Use existing speaker IDs as-is;
do not apply analysis-artifact key regexes or copy unrelated analysis limits.
3. Integrate validation with both load and save. Preserve speaker state through
running/failed/stale transitions and aggregate result clearing. Generic stage
invalidation does not discard checkpoints; reconciliation checks their data.
4. Test round-trip populated and versioned-empty collections, old manifests
without fields, malformed evidence, deep-copy isolation, and lifecycle
preservation. Avoid imposing JSON formatting or map order as contracts.
Verification:
Verify: `go test ./internal/manifest`.
```sh
go test ./internal/stage ./internal/adapters/notarius ./internal/config
```
Done when durable storage can represent partial completion independently of
aggregate status, without activating new stage behavior yet.
Completion: Selected profiles reach actual extraction, and both existing reuse
checks account for the selection without forcing downstream cache refresh or
changing the manifest schema.
## Stage 2: Implement Read-Only Speaker Reconciliation
## Stage 3: Verify Application Lifecycle Behavior
Status: Pending.
Status: Complete.
Preparation: `internal/stage/transcribe.go`, `semantic_contracts_initial.go`,
`analyze_reconciliation.go`, `extract_resume.go`, their tests, and the artifact
path/integrity helpers they use.
Completion marker (2026-09-21): Extended the existing application lifecycle
fixtures to cover same-selection reuse, named and base selection changes,
dependent invalidation/rerun, sibling preservation, ordinary downstream
failure, and real ConfigKit profile-to-execution composition without implicit
name forwarding or digest-wide invalidation. Verified with
`go test ./internal/app ./internal/stage`.
1. Extract the existing audio discovery/job construction into a shared,
transcribe-local reconciliation path. Preserve current input selection and
speaker identity behavior. Build sorted jobs and classify completed results
against the current input set and evidence defined above.
2. Add pure fingerprint construction and streamed input/output inspection.
Resolve manifest input paths with existing session-relative/absolute reading
helpers; do not change input precedence or assume the current working directory
is the session root. Newly written checkpoint paths are always relative.
3. Return current records and work-to-do without writing files or manifest state.
Force selects all work and an empty starting collection; old unversioned
transcribe state supplies no reusable records. Do not connect aggregate skip
validation until execution can produce the new evidence in Stage 3.
4. Test unchanged reuse; one changed audio among unchanged speakers; URL/language
changes; operational-setting exclusions; added/removed speakers; missing and
corrupt regular outputs; unsafe record paths; missing audio; and force.
Prefer compact tables and real temporary files over mocked hashing.
Preparation:
Verify: `go test ./internal/stage ./internal/manifest`.
- `internal/app/extract_lifecycle_test.go`, `runner_test.go`, and
`run_control_test.go`.
- Existing fixtures and tests for changed extraction outcomes, failure/retry,
and independent render/extract invalidation.
Done when one read-only decision path describes exactly which speakers need
work, without altering existing runtime behavior prematurely.
Work:
## Stage 3: Execute and Checkpoint Speakers Incrementally
1. Extend the existing lifecycle fixtures rather than building another harness.
Run a successful extraction with a configured profile, then verify ordinary
reuse for that same profile. Change only the downstream selection and verify
that extraction executes again with the new value, upstream transcripts and
render remain current, and succeeded dependents are invalidated/rerun through
the existing runner policy. Include returning from a named selection to base.
2. Cover the changed-profile failure boundary with the existing fake runner:
return an ordinary downstream error and verify extraction is failed, the
previous result is not advertised as current success, and dependents are not
silently reused. Reuse existing retry coverage where it already proves the
recovery behavior. Do not assert Notarius-specific diagnostic wording.
3. Ensure one configuration-to-execution path proves explicit composition:
Narratio's testing overlay can select a differently named Notarius overlay;
without the field, selecting a Narratio profile does not populate the
downstream request. Use real configuration loading and the existing injected
runner, not a production provider or sibling repository dependency.
4. Review existing operational-exclusion and force tests for continued validity.
Do not broaden invalidation to the full effective configuration digest or
Narratio's profile label, and do not automatically change cache options.
Status: Pending.
Verification:
Preparation: `internal/stage/transcribe.go`, `analyze_execution_state.go`,
`run_local.go`,
`internal/app/runner.go`, and transcribe/checkpoint test fixtures.
```sh
go test ./internal/app ./internal/stage
```
1. Wire reconciliation into Run, require the existing ManifestStore collaborator,
and checkpoint the starting collection before launching HTTP work. Implement
detached candidate saves and serialized live-state updates. Audit Store.Save's
normalization side effects so a failed candidate save cannot mutate shared
authority; use detached records/collections wherever it may write.
2. Retain bounded errgroup execution and fail-fast cancellation. Check the work
context before beginning each queued request. After success, validate adapter
output path/JSON, recheck audio evidence, materialize canonical bytes, compute
output evidence, and checkpoint the new record. Check a checkpoint-failure
latch under the serialization lock before accepting further completions.
3. Preserve committed successes on all error/cancellation exits, including
starting-save, canonical-write, per-speaker-save, and aggregate-save failures.
Return meaningful speaker/error context and let the runner retain its existing
terminal failure behavior. Do not report partial aggregate outputs as success.
4. Assemble final outputs in deterministic speaker order from the reconciled
completed records, including reused speakers, with original producer IDs and
checksums. If all records are current but aggregate status is failed/running,
finish without WhisperX calls and rebuild the aggregate success.
5. Implement `transcribeStage.ValidateResume` using the same read-only
reconciliation, so a succeeded stage cannot skip changed audio or invalid
transcripts. Retain the existing URL/language semantic check and runner
invalidation behavior; no whole-config digest invalidation.
6. Adapt existing tests that require no canonical partial files: the new contract
permits checkpointed files but forbids aggregate success. Update fixtures to
provide real manifest stores and records rather than a production fallback
that silently disables checkpointing.
7. Test A/B completion followed by C failure and D unattempted (concurrency 1),
reload and retry only C/D; concurrently finishing speakers without lost saves;
failure and cancellation after a checkpoint; save failure retaining earlier
authority; and force failure followed by ordinary retry without old-result reuse.
Completion: Tests demonstrate the requested workflow, including selection
changes and failure, across the real Narratio configuration/runner boundaries.
Notarius's own overlay loader and cache internals remain outside these tests.
Verify: `go test ./internal/stage ./internal/app` and
`go test -race ./internal/stage`.
## Stage 4: Document the Feature and Validate the Completed Change
Done when incremental progress and aggregate success work together safely under
ordinary, concurrent, canceled, and forced execution.
Status: Complete.
## Stage 4: Verify Runner, Merge, and Durable Lifecycle Integration
Completion marker (2026-09-21): Updated the canonical configuration,
integration, extract-internals, operations, and maintained-example documentation;
marked the feature roadmap implemented; and reviewed the completed behavior
against the consumer request. Verified with Go 1.26.8 using `go test ./...`,
`go test -race ./...`, `go vet ./...`, `go build ./...`,
`go test ./internal/doccheck`,
`go test ./internal/config -run '^TestExamplesLoadAndValidate$'`, and
`git diff --check`.
Status: Pending.
Preparation: Canonical documentation owners listed in the feature roadmap,
`examples/README.md`, and the existing development-focused operations guidance.
Preparation: `internal/app/runner.go`, `runner_runtime.go`,
`analyze_checkpoint_recovery_test.go`, `runner_terminal_failure_test.go`,
`restore_manifest_paths.go`, `internal/stage/merge.go`, and existing publish,
restore, and cleanup tests routed from `docs/internal/overview.md`.
Work:
1. Add application-level tests with real manifest persistence and fake WhisperX:
failed stage across separate invocations; all speakers committed before a
failed aggregate save; interrupted running state; force/reset persistence;
and unchanged upstream/succeeded downstream invalidation behavior. Inject
errors at durable boundaries, not every private method call.
2. Close `discoverRawTranscripts`' directory-scan fallback for versioned
transcribe state: incomplete aggregate state is not usable, and completed
state consumes only declared current aggregate outputs. Keep legacy/manual
fallback behavior for manifests without the new state rather than banning
previously allowed standalone workflows. Test that partial files and removed
speaker files cannot leak into a versioned merge, including direct stage calls.
3. Audit manifest copies/projections, remote manifest publication, restored path
handling, and cleanup. Preserve relative speaker paths on restore and verify
they resolve under the destination session, not the producer machine. Reuse
existing confinement validation, not a second path-policy layer.
4. Add focused round-trip coverage through existing publish/restore fixtures for
speaker records. Restored outputs absent from the published selection are
non-reusable locally; do not silently expand remote publication to every raw
speaker or upload audio as a new default. Removing run directories must not
remove canonical checkpointed transcripts or make reuse require run files.
5. Verify old succeeded manifests are readable and rerun once when transcribe is
selected, then reuse normally. Keep old manifests usable for other commands
and bounded ranges. Do not automatically import old unrecorded JSON files.
1. Document the optional field in `docs/config.md`, including omitted/null,
explicit base selectors, and independence from Narratio's profile and
`notarius.llm_api_key`. Do not add profile-name restrictions.
2. Update the invocation in `docs/integrations/notarius.md` to include the
optional flag and identify Notarius v0.7.0 as a known supporting version,
not an unconditional version requirement. Preserve the distinction from
`--llm-profile` and ordinary downstream failure handling.
3. Update `docs/internal/stage-extract.md` to describe selection-sensitive reuse
and retain its distinction from private file content and operational settings.
Make clear this is an explicit downstream setting, not blanket invalidation
by Narratio's profile label or effective digest.
4. Add concise development workflow guidance in `docs/operations.md`: explicit
testing-overlay composition; different downstream names are allowed;
switching selection triggers extraction when that stage is in scope; edits
behind an unchanged selector may need force; testing is not a separate
workspace. Preserve the existing distinction between extract+analyze and
analyze-only regeneration.
5. Update the maintained annotated pipeline example to show the optional field
without requiring an externally installed testing overlay (use a commented
named-selection example). Do not make previously runnable examples depend
on new downstream files. Link to canonical owners instead of duplicating
full contracts or adding new top-level Narratio flags.
6. Review the final diff against every observable behavior and scope boundary
in the consumer request. Mark the implementation stages and roadmap status
accurately after verification; do not remove the original request or alter
historical release notes.
Verify: `go test ./internal/app ./internal/stage ./internal/manifest` and
`go test -race ./internal/app`.
Run the repository checks with the maintained Go 1.26 toolchain:
Done when the new authority survives runner and storage lifecycles and cannot
expose incomplete transcripts downstream.
## Stage 5: Implement Replayable Streaming Multipart Bodies
Status: Pending.
Preparation: `internal/adapters/whisperx/http.go`, `http_test.go`,
`docs/integrations/whisperx.md`, and the installed Go net/http Request/GetBody
contract. Keep this change inside the WhisperX adapter.
1. Implement the pull-based framing/body factory described above, using standard
multipart encoding rather than hand-escaped MIME headers. Add no new payload
cap or upload-size setting. Preserve file-before-language wire semantics and
the existing form names.
2. Install the initial body and GetBody with the same per-attempt boundary.
Track and close all opened bodies even if a custom transport returns early;
make cleanup/context callbacks safe against overlapping reads and closes.
A GetBody failure must be returned, not converted into an empty upload.
3. Replace pipe-specific Wait/error paths in doTranscribeAttempt with source-error
handling appropriate to pull bodies. Genuine source errors remain visible;
intentional closure during transport retry is not itself a source error.
Keep response bounds, JSON validation, output installation, retry
classification/delay, cancellation, and application attempt reporting intact.
4. Remove obsolete producer-only helpers rather than retaining two upload paths.
Adapt tests tied to producer goroutines to assert streaming and resource
release, which are the actual contracts.
5. Add focused body tests: original/replayed bytes parse identically; replay
after partial consumption starts from the beginning; simultaneous bodies use
independent handles; source failures propagate; Close/cancel releases a
blocked reader without deadlock. Use synthetic audio and gated readers.
Verify: `go test ./internal/adapters/whisperx` and
`go test -race ./internal/adapters/whisperx`.
Done when replay is available without whole-file buffering or producer-lifetime
ambiguity, and all existing HTTP behavior tests still pass.
## Stage 6: Verify Replay Through the HTTP Adapter Boundary
Status: Pending.
Preparation: Existing injectable HTTPClient/RoundTripper seam and local-server
tests in `internal/adapters/whisperx/http_test.go`; app WhisperX wiring tests.
1. Add a deterministic transport-boundary regression: consume part of the
original request, close it as a transport would, call GetBody, consume and
parse the replay using the request Content-Type, then return valid JSON.
Verify exact audio/language, installed output, and one application attempt.
This must fail before replay support; merely checking non-nil GetBody is
insufficient. Do not assert Go's internal retry count or require a flaky
handcrafted HTTP/2 peer in the default suite.
2. Cover replay-open/read failure, early response, parent cancellation during
replay, and disposal of all body handles. Reuse Stage 5 body coverage rather
than repeat every low-level case. Prove an intentionally abandoned original
body does not turn a valid replay response into failure.
3. Retain local HTTP tests for application retries, non-retryable responses,
response limits, invalid JSON, and bounded streaming; exercise that an
exhausted HTTP failure still leaves earlier speaker checkpoints reusable
using the existing stage/app seams.
4. Record the verification boundary honestly: controlled transport replay proves
Narratio's body contract, not recovery from every real proxy PROTOCOL_ERROR.
Verify: `go test ./internal/adapters/whisperx ./internal/app ./internal/stage`
and race tests for changed packages.
Done when the replay defect is protected at the adapter boundary and the
application retry/checkpoint behavior remains intact.
## Stage 7: Document and Validate the Completed Feature
Status: Pending.
1. Update canonical transcribe and manifest internals for the collection,
reconciliation, durable-reset force behavior, serialized saves, and aggregate
gate. Remove obsolete all-speakers-before-canonical-write descriptions.
2. Update the WhisperX contract for replayable bounded streaming, application
attempts versus transport replay, resource ownership, and the lack of an
exactly-once server guarantee. Do not claim the remote protocol issue is fixed.
3. Update operations/troubleshooting with ordinary resume versus force, durable
partial success, old-manifest one-time retranscription on selecting transcribe,
downgrade limitations, and private-model-change guidance. No new CLI flags
or promises of automatic historical-file recovery. Keep detailed wire and
storage contracts in their canonical owners.
4. Review changes against the feature roadmap and the implementation-discipline
guidance. Do not elevate selected storage encoding, internal ordering, test
synchronization, or exact error wording into new product guarantees.
5. Run the full repository gates below; document results and update completion
statuses only after they pass. Leave release preparation for a separate task.
```sh
go test ./...
@@ -264,12 +352,10 @@ go test ./internal/config -run '^TestExamplesLoadAndValidate$'
git diff --check
```
Completion: The requested interface, failure and reuse behavior, examples, and
canonical documentation agree; all checks pass. No release is part of this
implementation sprint.
## Open Questions
None. The consumer request resolves the product behavior. The choices above
settle implementation details without adding new operator restrictions,
downstream configuration ownership, or cache guarantees.
None requiring user input. The agreed roadmap supplies the product decisions;
this plan selects a completed-only manifest collection, conservative old-record
handling, and pull-based replay as implementation choices. No stronger remote
reliability guarantee, new operator restriction, or generic orchestration
surface is proposed.

183
docs/roadmap/transcribe-checkpoints-and-replay.md Normal file
View File

@@ -0,0 +1,183 @@
# Resumable Speaker Transcription and Replayable Uploads
Status: Proposed; not implemented.
## Goal and User Intent
Preserve completed speaker transcriptions when another speaker fails, so an
ordinary run or resume does not repeat expensive successful work. Separately,
allow Go's HTTP transport to replay WhisperX uploads when it encounters a
recoverable connection or stream failure.
The motivating incident exhausted four transcription attempts for one speaker
after three other speakers had completed. The final error reported an HTTP/2
peer `PROTOCOL_ERROR` and an inability to replay the request body without
`Request.GetBody`.
These are related reliability improvements, not a promise to eliminate all
service failures. This document owns the target state and scope; the
[implementation plan](implementation.md) owns ordered changes and concrete
storage and replay decisions.
## Current Limitations
The [transcribe stage](../internal/stage-transcribe.md) writes successful
responses into the invocation's run directory. It installs canonical transcripts
and returns its per-file metadata only after every speaker succeeds. A failure
therefore leaves no durable per-speaker progress that ordinary resume can use,
even though some completed JSON files may remain on disk.
The [WhisperX adapter](../integrations/whisperx.md) already retries requests at
the application level, reopening audio for each attempt. Its multipart request
body is streamed through a pipe and has no replay callback. Transport-level
recovery within an attempt cannot reconstruct a consumed body. Providing that
capability addresses the replay limitation, not the underlying peer protocol
error; persistent failures can still require service or proxy investigation.
## Targeted End State
### Durable Per-Speaker Completion
Each successfully transcribed speaker file is validated, saved, and recorded
durably before that work is treated as reusable. The session manifest remains
the progress authority. Follow the existing analyze-stage checkpoint pattern
without turning speakers into pipeline stages or introducing a general task
engine.
Ordinary execution reconciles the current prepared audio set with recorded
speaker results. It reuses valid completed work and schedules failed,
unattempted, missing, or obsolete results. A result is not reusable merely
because a matching filename exists. Evidence binds it to the speaker/audio
identity, audio content, Narratio-visible recognition settings, and the saved
transcript's integrity.
The relevant recognition settings remain the WhisperX service URL and language.
Retry count/delay, concurrency, timeout, credentials, and diagnostic locations
remain operational settings rather than reasons to discard completed results.
Changes to private service models or configuration remain an operator reason
to force transcription; Narratio does not inspect the service's private state.
Changes to individual audio inputs require replacement of affected results,
not automatic retranscription of unrelated unchanged speakers. New speakers
need transcription; removed speakers do not remain in the current aggregate
output set. Records must describe the inputs actually used, not grant reuse to
old bytes under a changed identity.
### Aggregate Completion, Failure, and Force
The overall stage succeeds only when every currently required speaker has a
validated, durably recorded result. Incomplete per-speaker progress does not
make the aggregate stage successful or authorize downstream consumption of an
incomplete collection. Existing dependent-stage invalidation remains in effect.
Retain bounded concurrency and the current fail-fast behavior: a speaker
failure cancels outstanding work. Results already checkpointed remain reusable
after failure, cancellation, or process interruption. A later ordinary run or
resume retries the remaining work; it does not need a new CLI command.
Checkpoint writes must be serialized so concurrently finishing speakers cannot
overwrite each other's progress. A failed persistence operation must not
advertise an unrecorded result as complete. Previously committed progress remains
the recovery authority, and checkpoint errors must be surfaced rather than
reported as successful transcription.
`--force` on transcribe regenerates all currently required speakers instead of
reusing their prior completions. Successful replacements are checkpointed too,
so an ordinary retry after a failed forced invocation can retain the newly
completed work. Old results for speakers not successfully regenerated must not
satisfy that unfinished forced replacement on the next ordinary retry.
This deliberately relaxes the current all-speakers-before-canonical-installation
implementation: individual durable results may exist while the stage remains
failed or incomplete. The all-speakers-before-stage-success boundary remains.
### Replayable WhisperX Requests
Provide `Request.GetBody` so the HTTP transport can obtain a fresh body for the
same request. Replay reopens the audio and reproduces the multipart boundary,
filename, fields, and framing used by the original request. Independent bodies
must not share a consumed reader or mutable read position.
Preserve bounded-memory streaming rather than buffering entire audio files in
memory. Correctly close every original and replay body and its audio resources;
preserve source-read errors and cancellation, including partial uploads and
early server responses. Successful response handling must not accidentally wait
on, or report errors solely from, an abandoned body instead of the replay that
produced the response.
Keep the existing application-level retry count, delay, classification, and
per-attempt timeout semantics. Transport replays occur within an application
attempt and its context; they are not additional Narratio-configured attempts.
Do not add another custom retry loop, silently increase retries, or disable
HTTP/2 globally as a substitute for replay support.
The adapter still validates and installs only successful JSON responses.
Request replay does not provide exactly-once server execution: a lost response
can cause repeated transcription work, as can existing application retries.
No new service-side idempotency protocol is required by this feature.
## Required Change Areas
- **Manifest and persistence:** Add transcribe-owned per-speaker progress and
evidence, with safe checkpoint updates and preservation through runner failure
handling. Review manifest validation, copying, serialization, and existing
publish/restore handling wherever the new records pass through those paths.
- **Stage execution and reuse:** Reconcile speaker records before scheduling,
checkpoint validated completions, rebuild aggregate outputs from the current
set, and account for partial progress in ordinary and forced execution.
Review successful-stage reuse as well as failed-stage retries so stale audio
or missing/corrupt transcript evidence is not bypassed by an aggregate skip.
- **Artifact lifecycle:** Use existing path, installation, integrity, and manifest
mechanisms. Ensure checkpointed results do not depend on an ephemeral run
directory surviving cleanup. Do not introduce a second authoritative ledger
or infer completion by scanning abandoned runs.
- **HTTP adapter:** Make multipart bodies reproducible and correctly manage
every replay's lifetime, without changing the WhisperX HTTP schema or requiring
an upstream service change.
- **Tests:** Cover per-speaker recovery after a later failure, interruption and
checkpoint failure; changed/missing inputs or outputs; force followed by an
ordinary retry; concurrent successful completions; and aggregate downstream
gating. Extend existing fixtures rather than duplicating analyze's complete
test suite. Exercise actual body reconstruction after partial consumption,
multipart equivalence, cancellation/resource cleanup, and existing bounded
streaming and retry behavior—not just the presence of a callback. Default
tests remain offline and use local/fake HTTP boundaries, never paid services.
- **Documentation:** Update the canonical
[stage guide](../internal/stage-transcribe.md),
[manifest guide](../internal/manifest.md),
[WhisperX contract](../integrations/whisperx.md),
[operations guide](../operations.md), and
[troubleshooting guide](../troubleshooting.md) when implemented. Explain
partial completion, ordinary retry versus force, and the distinction between
request replay limitations and persistent peer protocol failures.
## Compatibility and Scope Boundaries
Existing speaker identities, raw transcript formats, and the fixed pipeline
order remain unchanged. Existing manifests remain readable, but must not be
treated as containing per-speaker evidence they never recorded. Recovery relies
on verified recorded evidence, not guessed associations between old transcripts
and current audio. No general legacy-migration framework is part of the feature.
Automatic adoption of unrecorded outputs from previously failed runs is outside
this feature. Such files may be recoverable after verifying their inputs, but
their presence alone does not prove completion. No recovery/import command,
speaker-selection CLI, historical invocation ledger, new service configuration
discovery, or broader orchestration refactor is included.
The feature does not diagnose the remote service/proxy's underlying HTTP/2
problem, guarantee recovery from persistent protocol errors, or change the
policy for continuing unrelated work after failure.
## Completion Criteria
For speakers A, B, C, and D: if A and B are checkpointed, C fails, and D has not
completed, ordinary resume retains A and B, retries C, and completes D. No
downstream stage receives an incomplete successful transcription collection.
Force replaces all four, while an ordinary retry can reuse checkpoints from
that forced replacement attempt.
The HTTP transport can reconstruct the upload after consuming the original
body, without whole-file memory buffering, lost source errors, or stranded
upload resources. A persistent peer error still fails normally after the
configured application attempts, without discarding checkpointed speakers.