Plan resumable transcription and replayable uploads
This commit is contained in:
@@ -1,258 +1,346 @@
|
||||
# Implementation Plan: Notarius Configuration Profile Selection
|
||||
# Implementation Plan: Speaker Checkpoints and Replayable Uploads
|
||||
|
||||
Status: Complete; all stages implemented and verified.
|
||||
Status: Ready; all stages pending.
|
||||
|
||||
## Scope and Working Instructions
|
||||
## Scope and Instructions
|
||||
|
||||
Implement the [feature roadmap](notarius-config-profile.md), whose authority is
|
||||
the [consumer request](feature-request-notarius-config-profile.yml). Complete
|
||||
the numbered stages in order. Each stage is a bounded task for a gpt-5.6-terra
|
||||
coding agent. Do not treat completion of an intermediate stage as completion of
|
||||
the feature. Record completed stages and actual verification results here.
|
||||
Implement [the feature roadmap](transcribe-checkpoints-and-replay.md). This plan
|
||||
replaces the completed Notarius-profile plan; that work remains in Git history.
|
||||
Complete the numbered stages in order, one bounded implementation prompt per
|
||||
stage for gpt-5.6-terra. Record completion and actual verification results here.
|
||||
Intermediate stages are not a finished feature and must not be released alone.
|
||||
|
||||
Before each stage, read `docs/development.md`, all documents under
|
||||
`docs/policy/`, the feature roadmap, and that stage's preparation list. Follow
|
||||
AGENTS.md discovery guidance. Use existing collaborators and fixtures; do not
|
||||
build a new configuration, cache, or profile-management subsystem. Tests must
|
||||
remain offline and independent of installed Notarius binaries and credentials.
|
||||
Do not change dependencies, implement Ansible changes, tag, or release as part
|
||||
of this plan.
|
||||
Before working, read `AGENTS.md`, `docs/development.md`, all three policies in
|
||||
`docs/policy/`, the roadmap, and the stage-specific preparation below. Use the
|
||||
maintained Go 1.26 toolchain. Preserve operator choices, speaker filenames,
|
||||
bounded concurrency, streaming, and existing credentials/settings. Do not add
|
||||
new commands, dependencies, service requirements, arbitrary speaker limits,
|
||||
strict transcript schemas, or a generic checkpoint framework. No release or
|
||||
upstream service work is part of this plan.
|
||||
|
||||
## Implementation Decisions
|
||||
## Settled Design Decisions
|
||||
|
||||
These choices resolve implementation details without adding restrictions to
|
||||
the feature request:
|
||||
### Progress Representation and Recovery
|
||||
|
||||
- Represent the authored field as `ConfigProfile *string` with YAML/JSON name
|
||||
`config_profile` and `omitempty`, in both typed configuration and its safe
|
||||
YAML projection. Nil means omitted/null. A non-nil pointer preserves an
|
||||
explicitly supplied empty string as well as `default` and named selections.
|
||||
- Carry the same optional value in the Notarius adapter request. Nil emits no
|
||||
flag; non-nil emits `--config-profile` followed by the exact supplied string
|
||||
as a separate argument, including an empty string. Do not trim, lowercase,
|
||||
substitute Narratio's profile, impose a name regex, or inspect profile files.
|
||||
Notarius owns validation and selector interpretation.
|
||||
- For reuse evidence only, use one small stage-local helper that maps nil,
|
||||
`""`, and exact `"default"` to the empty identity; other values remain
|
||||
unchanged. Both extraction fingerprints use it. This does not mutate the
|
||||
authored field or change CLI forwarding. Do not attempt to recreate the
|
||||
downstream parser's complete normalization rules.
|
||||
- Add `ConfigProfile string` with `json:"config_profile,omitempty"` to both
|
||||
existing fingerprint payloads, populated only where extraction is enabled.
|
||||
Preserve existing field ordering and the current semantic fingerprint version
|
||||
(1). With no non-base selector the serialized payload remains unchanged;
|
||||
a named selection adds evidence that old records cannot already possess.
|
||||
No dual-reader, manifest migration, new durable metadata field, or broad
|
||||
fingerprint-version bump is needed. This is a minimal additive encoding,
|
||||
not a permanent promise to freeze fingerprint bytes across future releases.
|
||||
- Profile changes trigger the existing non-resumable/invalidation paths, not
|
||||
force mode. Preserve configured resume/cache controls, force precedence,
|
||||
credentials, output contracts, and stage-range behavior. Do not run extraction
|
||||
outside the operator's selected range or add a separate testing workspace.
|
||||
- Preserve the existing division of ownership: private downstream file changes
|
||||
behind an unchanged selector require operator-directed force when needed.
|
||||
No downstream configuration validation subprocess or runtime version gate
|
||||
is added.
|
||||
Add optional `transcribe_state_version` (version 1) and `transcribe_speakers`
|
||||
to `manifest.StageRecord`. The latter is a map keyed by the existing derived
|
||||
speaker ID, containing completed results only. Absence means that work must be
|
||||
done; a separate pending/running/failed state machine is unnecessary. Aggregate
|
||||
stage status/error remains authoritative for invocation failure. A versioned
|
||||
empty collection is meaningful: no earlier completions are reusable.
|
||||
|
||||
## Stage 1: Add Configuration and Adapter Support
|
||||
Use a typed `TranscribeSpeakerRecord` with fingerprint version/digest,
|
||||
session-relative audio path, audio SHA-256 and size, one `ArtifactRecord` output
|
||||
of kind `transcript_raw`, output size, and producer run ID. The output carries
|
||||
its session-relative canonical path and SHA-256. No new transcript source IDs,
|
||||
synthetic schema contracts, timestamps, or historical attempt collections are
|
||||
needed. Existing aggregate diagnostics can retain attempt/status/duration data
|
||||
for work performed in the current invocation; do not fabricate that information
|
||||
for reused outputs.
|
||||
|
||||
Status: Complete.
|
||||
Keep the session-manifest schema identifier unchanged for this optional additive
|
||||
state; validate supported per-stage versions and retain strict decoding. New
|
||||
Narratio reads old manifests. Old binaries may reject the added fields under
|
||||
their existing strict decoding; document this downgrade limitation.
|
||||
|
||||
Completion marker (2026-09-21): Added composed optional configuration and safe
|
||||
inspection support, adapter request/argument forwarding, copy auditing, and
|
||||
focused configuration and subprocess coverage. Verified with
|
||||
`go test ./internal/config ./internal/adapters/notarius`.
|
||||
For pre-feature manifests with no speaker evidence, regenerate transcription
|
||||
once when transcribe is next selected, even if the aggregate status succeeded.
|
||||
Old records do not bind individual transcript results to the audio bytes they
|
||||
used. Do not guess that association from current files, old `per_file` metadata,
|
||||
or abandoned run directories. This is the conservative upgrade path, not a new
|
||||
general migration system. No other stage or unselected range is forcibly run.
|
||||
|
||||
Preparation:
|
||||
### Reuse and Persistence
|
||||
|
||||
- `docs/config.md` (composition, profiles, Notarius, credentials).
|
||||
- `docs/integrations/notarius.md` (invocation and failure behavior).
|
||||
- `internal/config/config.go`, `credential_projection.go`,
|
||||
`pipeline_configkit.go`, `pipeline_normalize.go`, and `notarius_test.go`.
|
||||
- `internal/adapters/notarius/runner.go`, `subprocess.go`, `fake.go`, and
|
||||
`subprocess_test.go`.
|
||||
Compute a versioned per-speaker fingerprint from speaker ID, session-relative
|
||||
audio identity, actual audio SHA-256/size, and the existing transcribe semantic
|
||||
configuration fingerprint (URL and language). Do not include invocation ID,
|
||||
absolute workspace root, concurrency, retry settings, timeout, or credentials.
|
||||
Stream input hashing with cancellation; do not trust stored input checksums
|
||||
alone. Recheck input bytes before accepting a new result to catch changes
|
||||
observed during transcription; fail that job without checkpointing mismatched
|
||||
evidence. This is not a hostile-filesystem snapshot or concurrent-editor
|
||||
transaction guarantee and requires no extra audio copies or file locks.
|
||||
|
||||
Work:
|
||||
Validate reusable output identity, canonical location, JSON, checksum, and size.
|
||||
Use existing confinement/file primitives. Missing or corrupt regular outputs
|
||||
become work; unsafe paths/non-regular conflicts follow existing safety errors,
|
||||
not recursive deletion. Missing required audio is an input error, not permission
|
||||
to reuse an unverifiable transcript. Removed inputs are removed from the current
|
||||
collection and aggregate outputs without deleting arbitrary files.
|
||||
|
||||
1. Add the optional field to `NotariusConfig` and the `safeNotarius` projection,
|
||||
including its returned value. Let existing ConfigKit composition handle
|
||||
overrides and null; do not add a default or resolve this name as a path.
|
||||
2. Add the optional field to `notarius.RunRequest` and append the argument pair
|
||||
when non-nil in `SubprocessRunner.Run`, before `--json`. Leave subprocess
|
||||
environment construction and error propagation unchanged.
|
||||
3. Audit any explicit Notarius configuration/request copies for the new field.
|
||||
Update actual omissions without introducing a generic cloning framework.
|
||||
4. Extend real configuration-load tests to cover a base selection, an overlay
|
||||
replacing it with a differently named downstream selection, and an overlay
|
||||
clearing it with null. Cover omitted, empty, and `default` values compactly.
|
||||
Verify the safe effective-YAML output retains a supplied value, including
|
||||
explicit empty, and round-trips without leaking credential values. Reuse
|
||||
existing credential tests rather than duplicating their full matrix.
|
||||
5. Extend adapter invocation tests for omitted and supplied selections,
|
||||
including empty and `default`. Assert the argument/value boundary rather
|
||||
than imposing new ordering requirements on unrelated arguments. Reuse the
|
||||
existing nonzero-exit test with a selected profile to prove there is no
|
||||
fallback or successful receipt processing after failure.
|
||||
One stage-owned mutex serializes candidate collection construction, manifest
|
||||
save, and installation of committed state. Save a detached candidate and only
|
||||
then update live authority, following analyze's pattern. Workers must not mutate
|
||||
the live manifest outside that serialized section. Keep expensive HTTP work
|
||||
outside the mutex. Stop dispatch/acceptance on checkpoint failure, join workers
|
||||
before return, and preserve the last successfully committed collection.
|
||||
|
||||
Verification:
|
||||
Before launching requests, durably checkpoint the reconciled collection. For
|
||||
force, that collection is empty: this prevents later ordinary retries from
|
||||
resurrecting pre-force outputs. If this initial save fails, do not start requests
|
||||
or overwrite outputs. Force again clears the current collection again; ordinary
|
||||
retry retains replacements already checkpointed after the previous force.
|
||||
|
||||
```sh
|
||||
go test ./internal/config ./internal/adapters/notarius
|
||||
```
|
||||
Save each validated result at its canonical raw-transcript location before
|
||||
checkpointing it. Run-local files remain staging/diagnostics, not reuse
|
||||
dependencies. Bytes left by an unsuccessful checkpoint have no completion
|
||||
authority. A worker may checkpoint a completed result using the parent stage
|
||||
context even if a sibling canceled the work context; parent cancellation or a
|
||||
checkpoint failure may prevent that save. Guarantee retention of committed
|
||||
work, not all responses that happened to finish during cancellation.
|
||||
|
||||
Completion: Configuration composition and safe inspection preserve the field,
|
||||
and the adapter forwards it correctly. The executing stage is connected in
|
||||
Stage 2 together with reuse tracking, so it cannot start using untracked profile
|
||||
selections as an intermediate implementation.
|
||||
### Multipart Replay Mechanism
|
||||
|
||||
## Stage 2: Wire Extraction and Profile-Sensitive Reuse
|
||||
Replace the pipe/producer upload mechanism with a small pull-based multipart
|
||||
body backed by the audio reader. Use `mime/multipart.Writer` to encode a bounded
|
||||
metadata prefix and suffix around the file payload: capture the prefix after
|
||||
`CreateFormFile`, then encode the language field and closing boundary as the
|
||||
suffix. Generate this framing once per application attempt. Each body combines
|
||||
fresh readers for prefix, freshly opened audio, and suffix using `io.MultiReader`.
|
||||
No full-audio buffer or temporary multipart spool is needed.
|
||||
|
||||
Status: Complete.
|
||||
The initial body and `Request.GetBody` use the same factory/framing. Each owns
|
||||
an independent audio handle. Give bodies idempotent Close and context-triggered
|
||||
closure (for example `context.AfterFunc`), without holding a mutex across a
|
||||
blocking Read. Read/Close coordination must release a blocked source and retain
|
||||
genuine source-read/close errors while not treating deliberate early closure as
|
||||
source failure. The attempt owns cleanup of all created bodies, including
|
||||
replays; reject new body creation after its context expires.
|
||||
|
||||
Completion marker (2026-09-21): Wired the selected profile into extraction,
|
||||
added a shared base-selector reuse identity to both existing fingerprints, and
|
||||
covered request forwarding, force precedence, semantic sensitivity, prior
|
||||
no-profile encoding, and focused resume validation. Verified with
|
||||
`go test ./internal/stage ./internal/adapters/notarius ./internal/config`.
|
||||
Remove the old producer `Wait` dependency. Track body-local source errors with
|
||||
synchronization so a swallowed upload read error cannot become a successful
|
||||
transcript, and a deliberately closed original body cannot poison a successful
|
||||
replay. Keep the outer retry policy and per-attempt context. Do not add replay
|
||||
counters to the user retry budget or depend on Go's exact internal retry count.
|
||||
Preserve ordinary HTTP redirect policy; body replay can also enable Go's
|
||||
body-preserving redirects. Do not introduce a new redirect restriction.
|
||||
|
||||
Preparation:
|
||||
## Stage 1: Add Durable Speaker Completion State
|
||||
|
||||
- `docs/internal/stage-extract.md`.
|
||||
- `internal/stage/extract.go`, `extract_resume.go`,
|
||||
`semantic_contracts_delivery.go`, and their focused tests.
|
||||
- In particular, `TestExtractStageResolvesManifestInputAndBuildsExactRequest`,
|
||||
`TestExtractSemanticConfigSensitivityAndOperationalExclusions`, and
|
||||
`TestExtractStageResumeValidationRequiresFreshEvidence`.
|
||||
Status: Pending.
|
||||
|
||||
Work:
|
||||
Preparation: `internal/manifest/manifest.go`, `store.go`, `analyze_state.go`,
|
||||
their tests, and `docs/internal/manifest.md`.
|
||||
|
||||
1. Pass `cfg.ConfigProfile` to the adapter request when extraction executes.
|
||||
Keep the field independent of credential forwarding and force overrides.
|
||||
2. Implement the reuse-identity helper and additive fingerprint fields described
|
||||
above. Update both `extractSemanticConfig` and `fingerprintDocument`, using
|
||||
the same identity in semantic comparison and `extractionFingerprint`.
|
||||
Keep disabled extraction insensitive to unused downstream settings.
|
||||
3. Extend the existing stage request test to prove the authored selection
|
||||
reaches the adapter. Cover ordinary and forced execution sufficiently to
|
||||
show that profile forwarding does not alter existing cache-control precedence.
|
||||
4. Extend semantic-sensitivity coverage: base to named, named to different named,
|
||||
and named to base change evidence; unchanged selection does not. Verify
|
||||
nil/empty/`default` share base evidence and disabled settings remain inert.
|
||||
5. Exercise the focused resume validator with an actual valid recorded bundle:
|
||||
it remains reusable for the same selection and becomes non-resumable for a
|
||||
changed selection. This guards the second fingerprint path independently
|
||||
of the application's semantic comparison.
|
||||
6. Check the additive encoding against the prior no-profile payload shape once
|
||||
in focused coverage or a recorded development check. Avoid whole-manifest
|
||||
snapshots and a permanent historical compatibility harness; retain existing
|
||||
input/output/reference sorting and all unrelated fingerprint fields.
|
||||
1. Implement the additive fields, record type, version constants, validation,
|
||||
and deep-copy helper described above in a focused transcribe state file.
|
||||
2. Validate ownership by the transcribe stage, supported versions, required
|
||||
completed-record evidence, output kind, safe relative paths, SHA-256 values,
|
||||
sizes, and producer identity consistency. Use existing speaker IDs as-is;
|
||||
do not apply analysis-artifact key regexes or copy unrelated analysis limits.
|
||||
3. Integrate validation with both load and save. Preserve speaker state through
|
||||
running/failed/stale transitions and aggregate result clearing. Generic stage
|
||||
invalidation does not discard checkpoints; reconciliation checks their data.
|
||||
4. Test round-trip populated and versioned-empty collections, old manifests
|
||||
without fields, malformed evidence, deep-copy isolation, and lifecycle
|
||||
preservation. Avoid imposing JSON formatting or map order as contracts.
|
||||
|
||||
Verification:
|
||||
Verify: `go test ./internal/manifest`.
|
||||
|
||||
```sh
|
||||
go test ./internal/stage ./internal/adapters/notarius ./internal/config
|
||||
```
|
||||
Done when durable storage can represent partial completion independently of
|
||||
aggregate status, without activating new stage behavior yet.
|
||||
|
||||
Completion: Selected profiles reach actual extraction, and both existing reuse
|
||||
checks account for the selection without forcing downstream cache refresh or
|
||||
changing the manifest schema.
|
||||
## Stage 2: Implement Read-Only Speaker Reconciliation
|
||||
|
||||
## Stage 3: Verify Application Lifecycle Behavior
|
||||
Status: Pending.
|
||||
|
||||
Status: Complete.
|
||||
Preparation: `internal/stage/transcribe.go`, `semantic_contracts_initial.go`,
|
||||
`analyze_reconciliation.go`, `extract_resume.go`, their tests, and the artifact
|
||||
path/integrity helpers they use.
|
||||
|
||||
Completion marker (2026-09-21): Extended the existing application lifecycle
|
||||
fixtures to cover same-selection reuse, named and base selection changes,
|
||||
dependent invalidation/rerun, sibling preservation, ordinary downstream
|
||||
failure, and real ConfigKit profile-to-execution composition without implicit
|
||||
name forwarding or digest-wide invalidation. Verified with
|
||||
`go test ./internal/app ./internal/stage`.
|
||||
1. Extract the existing audio discovery/job construction into a shared,
|
||||
transcribe-local reconciliation path. Preserve current input selection and
|
||||
speaker identity behavior. Build sorted jobs and classify completed results
|
||||
against the current input set and evidence defined above.
|
||||
2. Add pure fingerprint construction and streamed input/output inspection.
|
||||
Resolve manifest input paths with existing session-relative/absolute reading
|
||||
helpers; do not change input precedence or assume the current working directory
|
||||
is the session root. Newly written checkpoint paths are always relative.
|
||||
3. Return current records and work-to-do without writing files or manifest state.
|
||||
Force selects all work and an empty starting collection; old unversioned
|
||||
transcribe state supplies no reusable records. Do not connect aggregate skip
|
||||
validation until execution can produce the new evidence in Stage 3.
|
||||
4. Test unchanged reuse; one changed audio among unchanged speakers; URL/language
|
||||
changes; operational-setting exclusions; added/removed speakers; missing and
|
||||
corrupt regular outputs; unsafe record paths; missing audio; and force.
|
||||
Prefer compact tables and real temporary files over mocked hashing.
|
||||
|
||||
Preparation:
|
||||
Verify: `go test ./internal/stage ./internal/manifest`.
|
||||
|
||||
- `internal/app/extract_lifecycle_test.go`, `runner_test.go`, and
|
||||
`run_control_test.go`.
|
||||
- Existing fixtures and tests for changed extraction outcomes, failure/retry,
|
||||
and independent render/extract invalidation.
|
||||
Done when one read-only decision path describes exactly which speakers need
|
||||
work, without altering existing runtime behavior prematurely.
|
||||
|
||||
Work:
|
||||
## Stage 3: Execute and Checkpoint Speakers Incrementally
|
||||
|
||||
1. Extend the existing lifecycle fixtures rather than building another harness.
|
||||
Run a successful extraction with a configured profile, then verify ordinary
|
||||
reuse for that same profile. Change only the downstream selection and verify
|
||||
that extraction executes again with the new value, upstream transcripts and
|
||||
render remain current, and succeeded dependents are invalidated/rerun through
|
||||
the existing runner policy. Include returning from a named selection to base.
|
||||
2. Cover the changed-profile failure boundary with the existing fake runner:
|
||||
return an ordinary downstream error and verify extraction is failed, the
|
||||
previous result is not advertised as current success, and dependents are not
|
||||
silently reused. Reuse existing retry coverage where it already proves the
|
||||
recovery behavior. Do not assert Notarius-specific diagnostic wording.
|
||||
3. Ensure one configuration-to-execution path proves explicit composition:
|
||||
Narratio's testing overlay can select a differently named Notarius overlay;
|
||||
without the field, selecting a Narratio profile does not populate the
|
||||
downstream request. Use real configuration loading and the existing injected
|
||||
runner, not a production provider or sibling repository dependency.
|
||||
4. Review existing operational-exclusion and force tests for continued validity.
|
||||
Do not broaden invalidation to the full effective configuration digest or
|
||||
Narratio's profile label, and do not automatically change cache options.
|
||||
Status: Pending.
|
||||
|
||||
Verification:
|
||||
Preparation: `internal/stage/transcribe.go`, `analyze_execution_state.go`,
|
||||
`run_local.go`,
|
||||
`internal/app/runner.go`, and transcribe/checkpoint test fixtures.
|
||||
|
||||
```sh
|
||||
go test ./internal/app ./internal/stage
|
||||
```
|
||||
1. Wire reconciliation into Run, require the existing ManifestStore collaborator,
|
||||
and checkpoint the starting collection before launching HTTP work. Implement
|
||||
detached candidate saves and serialized live-state updates. Audit Store.Save's
|
||||
normalization side effects so a failed candidate save cannot mutate shared
|
||||
authority; use detached records/collections wherever it may write.
|
||||
2. Retain bounded errgroup execution and fail-fast cancellation. Check the work
|
||||
context before beginning each queued request. After success, validate adapter
|
||||
output path/JSON, recheck audio evidence, materialize canonical bytes, compute
|
||||
output evidence, and checkpoint the new record. Check a checkpoint-failure
|
||||
latch under the serialization lock before accepting further completions.
|
||||
3. Preserve committed successes on all error/cancellation exits, including
|
||||
starting-save, canonical-write, per-speaker-save, and aggregate-save failures.
|
||||
Return meaningful speaker/error context and let the runner retain its existing
|
||||
terminal failure behavior. Do not report partial aggregate outputs as success.
|
||||
4. Assemble final outputs in deterministic speaker order from the reconciled
|
||||
completed records, including reused speakers, with original producer IDs and
|
||||
checksums. If all records are current but aggregate status is failed/running,
|
||||
finish without WhisperX calls and rebuild the aggregate success.
|
||||
5. Implement `transcribeStage.ValidateResume` using the same read-only
|
||||
reconciliation, so a succeeded stage cannot skip changed audio or invalid
|
||||
transcripts. Retain the existing URL/language semantic check and runner
|
||||
invalidation behavior; no whole-config digest invalidation.
|
||||
6. Adapt existing tests that require no canonical partial files: the new contract
|
||||
permits checkpointed files but forbids aggregate success. Update fixtures to
|
||||
provide real manifest stores and records rather than a production fallback
|
||||
that silently disables checkpointing.
|
||||
7. Test A/B completion followed by C failure and D unattempted (concurrency 1),
|
||||
reload and retry only C/D; concurrently finishing speakers without lost saves;
|
||||
failure and cancellation after a checkpoint; save failure retaining earlier
|
||||
authority; and force failure followed by ordinary retry without old-result reuse.
|
||||
|
||||
Completion: Tests demonstrate the requested workflow, including selection
|
||||
changes and failure, across the real Narratio configuration/runner boundaries.
|
||||
Notarius's own overlay loader and cache internals remain outside these tests.
|
||||
Verify: `go test ./internal/stage ./internal/app` and
|
||||
`go test -race ./internal/stage`.
|
||||
|
||||
## Stage 4: Document the Feature and Validate the Completed Change
|
||||
Done when incremental progress and aggregate success work together safely under
|
||||
ordinary, concurrent, canceled, and forced execution.
|
||||
|
||||
Status: Complete.
|
||||
## Stage 4: Verify Runner, Merge, and Durable Lifecycle Integration
|
||||
|
||||
Completion marker (2026-09-21): Updated the canonical configuration,
|
||||
integration, extract-internals, operations, and maintained-example documentation;
|
||||
marked the feature roadmap implemented; and reviewed the completed behavior
|
||||
against the consumer request. Verified with Go 1.26.8 using `go test ./...`,
|
||||
`go test -race ./...`, `go vet ./...`, `go build ./...`,
|
||||
`go test ./internal/doccheck`,
|
||||
`go test ./internal/config -run '^TestExamplesLoadAndValidate$'`, and
|
||||
`git diff --check`.
|
||||
Status: Pending.
|
||||
|
||||
Preparation: Canonical documentation owners listed in the feature roadmap,
|
||||
`examples/README.md`, and the existing development-focused operations guidance.
|
||||
Preparation: `internal/app/runner.go`, `runner_runtime.go`,
|
||||
`analyze_checkpoint_recovery_test.go`, `runner_terminal_failure_test.go`,
|
||||
`restore_manifest_paths.go`, `internal/stage/merge.go`, and existing publish,
|
||||
restore, and cleanup tests routed from `docs/internal/overview.md`.
|
||||
|
||||
Work:
|
||||
1. Add application-level tests with real manifest persistence and fake WhisperX:
|
||||
failed stage across separate invocations; all speakers committed before a
|
||||
failed aggregate save; interrupted running state; force/reset persistence;
|
||||
and unchanged upstream/succeeded downstream invalidation behavior. Inject
|
||||
errors at durable boundaries, not every private method call.
|
||||
2. Close `discoverRawTranscripts`' directory-scan fallback for versioned
|
||||
transcribe state: incomplete aggregate state is not usable, and completed
|
||||
state consumes only declared current aggregate outputs. Keep legacy/manual
|
||||
fallback behavior for manifests without the new state rather than banning
|
||||
previously allowed standalone workflows. Test that partial files and removed
|
||||
speaker files cannot leak into a versioned merge, including direct stage calls.
|
||||
3. Audit manifest copies/projections, remote manifest publication, restored path
|
||||
handling, and cleanup. Preserve relative speaker paths on restore and verify
|
||||
they resolve under the destination session, not the producer machine. Reuse
|
||||
existing confinement validation, not a second path-policy layer.
|
||||
4. Add focused round-trip coverage through existing publish/restore fixtures for
|
||||
speaker records. Restored outputs absent from the published selection are
|
||||
non-reusable locally; do not silently expand remote publication to every raw
|
||||
speaker or upload audio as a new default. Removing run directories must not
|
||||
remove canonical checkpointed transcripts or make reuse require run files.
|
||||
5. Verify old succeeded manifests are readable and rerun once when transcribe is
|
||||
selected, then reuse normally. Keep old manifests usable for other commands
|
||||
and bounded ranges. Do not automatically import old unrecorded JSON files.
|
||||
|
||||
1. Document the optional field in `docs/config.md`, including omitted/null,
|
||||
explicit base selectors, and independence from Narratio's profile and
|
||||
`notarius.llm_api_key`. Do not add profile-name restrictions.
|
||||
2. Update the invocation in `docs/integrations/notarius.md` to include the
|
||||
optional flag and identify Notarius v0.7.0 as a known supporting version,
|
||||
not an unconditional version requirement. Preserve the distinction from
|
||||
`--llm-profile` and ordinary downstream failure handling.
|
||||
3. Update `docs/internal/stage-extract.md` to describe selection-sensitive reuse
|
||||
and retain its distinction from private file content and operational settings.
|
||||
Make clear this is an explicit downstream setting, not blanket invalidation
|
||||
by Narratio's profile label or effective digest.
|
||||
4. Add concise development workflow guidance in `docs/operations.md`: explicit
|
||||
testing-overlay composition; different downstream names are allowed;
|
||||
switching selection triggers extraction when that stage is in scope; edits
|
||||
behind an unchanged selector may need force; testing is not a separate
|
||||
workspace. Preserve the existing distinction between extract+analyze and
|
||||
analyze-only regeneration.
|
||||
5. Update the maintained annotated pipeline example to show the optional field
|
||||
without requiring an externally installed testing overlay (use a commented
|
||||
named-selection example). Do not make previously runnable examples depend
|
||||
on new downstream files. Link to canonical owners instead of duplicating
|
||||
full contracts or adding new top-level Narratio flags.
|
||||
6. Review the final diff against every observable behavior and scope boundary
|
||||
in the consumer request. Mark the implementation stages and roadmap status
|
||||
accurately after verification; do not remove the original request or alter
|
||||
historical release notes.
|
||||
Verify: `go test ./internal/app ./internal/stage ./internal/manifest` and
|
||||
`go test -race ./internal/app`.
|
||||
|
||||
Run the repository checks with the maintained Go 1.26 toolchain:
|
||||
Done when the new authority survives runner and storage lifecycles and cannot
|
||||
expose incomplete transcripts downstream.
|
||||
|
||||
## Stage 5: Implement Replayable Streaming Multipart Bodies
|
||||
|
||||
Status: Pending.
|
||||
|
||||
Preparation: `internal/adapters/whisperx/http.go`, `http_test.go`,
|
||||
`docs/integrations/whisperx.md`, and the installed Go net/http Request/GetBody
|
||||
contract. Keep this change inside the WhisperX adapter.
|
||||
|
||||
1. Implement the pull-based framing/body factory described above, using standard
|
||||
multipart encoding rather than hand-escaped MIME headers. Add no new payload
|
||||
cap or upload-size setting. Preserve file-before-language wire semantics and
|
||||
the existing form names.
|
||||
2. Install the initial body and GetBody with the same per-attempt boundary.
|
||||
Track and close all opened bodies even if a custom transport returns early;
|
||||
make cleanup/context callbacks safe against overlapping reads and closes.
|
||||
A GetBody failure must be returned, not converted into an empty upload.
|
||||
3. Replace pipe-specific Wait/error paths in doTranscribeAttempt with source-error
|
||||
handling appropriate to pull bodies. Genuine source errors remain visible;
|
||||
intentional closure during transport retry is not itself a source error.
|
||||
Keep response bounds, JSON validation, output installation, retry
|
||||
classification/delay, cancellation, and application attempt reporting intact.
|
||||
4. Remove obsolete producer-only helpers rather than retaining two upload paths.
|
||||
Adapt tests tied to producer goroutines to assert streaming and resource
|
||||
release, which are the actual contracts.
|
||||
5. Add focused body tests: original/replayed bytes parse identically; replay
|
||||
after partial consumption starts from the beginning; simultaneous bodies use
|
||||
independent handles; source failures propagate; Close/cancel releases a
|
||||
blocked reader without deadlock. Use synthetic audio and gated readers.
|
||||
|
||||
Verify: `go test ./internal/adapters/whisperx` and
|
||||
`go test -race ./internal/adapters/whisperx`.
|
||||
|
||||
Done when replay is available without whole-file buffering or producer-lifetime
|
||||
ambiguity, and all existing HTTP behavior tests still pass.
|
||||
|
||||
## Stage 6: Verify Replay Through the HTTP Adapter Boundary
|
||||
|
||||
Status: Pending.
|
||||
|
||||
Preparation: Existing injectable HTTPClient/RoundTripper seam and local-server
|
||||
tests in `internal/adapters/whisperx/http_test.go`; app WhisperX wiring tests.
|
||||
|
||||
1. Add a deterministic transport-boundary regression: consume part of the
|
||||
original request, close it as a transport would, call GetBody, consume and
|
||||
parse the replay using the request Content-Type, then return valid JSON.
|
||||
Verify exact audio/language, installed output, and one application attempt.
|
||||
This must fail before replay support; merely checking non-nil GetBody is
|
||||
insufficient. Do not assert Go's internal retry count or require a flaky
|
||||
handcrafted HTTP/2 peer in the default suite.
|
||||
2. Cover replay-open/read failure, early response, parent cancellation during
|
||||
replay, and disposal of all body handles. Reuse Stage 5 body coverage rather
|
||||
than repeat every low-level case. Prove an intentionally abandoned original
|
||||
body does not turn a valid replay response into failure.
|
||||
3. Retain local HTTP tests for application retries, non-retryable responses,
|
||||
response limits, invalid JSON, and bounded streaming; exercise that an
|
||||
exhausted HTTP failure still leaves earlier speaker checkpoints reusable
|
||||
using the existing stage/app seams.
|
||||
4. Record the verification boundary honestly: controlled transport replay proves
|
||||
Narratio's body contract, not recovery from every real proxy PROTOCOL_ERROR.
|
||||
|
||||
Verify: `go test ./internal/adapters/whisperx ./internal/app ./internal/stage`
|
||||
and race tests for changed packages.
|
||||
|
||||
Done when the replay defect is protected at the adapter boundary and the
|
||||
application retry/checkpoint behavior remains intact.
|
||||
|
||||
## Stage 7: Document and Validate the Completed Feature
|
||||
|
||||
Status: Pending.
|
||||
|
||||
1. Update canonical transcribe and manifest internals for the collection,
|
||||
reconciliation, durable-reset force behavior, serialized saves, and aggregate
|
||||
gate. Remove obsolete all-speakers-before-canonical-write descriptions.
|
||||
2. Update the WhisperX contract for replayable bounded streaming, application
|
||||
attempts versus transport replay, resource ownership, and the lack of an
|
||||
exactly-once server guarantee. Do not claim the remote protocol issue is fixed.
|
||||
3. Update operations/troubleshooting with ordinary resume versus force, durable
|
||||
partial success, old-manifest one-time retranscription on selecting transcribe,
|
||||
downgrade limitations, and private-model-change guidance. No new CLI flags
|
||||
or promises of automatic historical-file recovery. Keep detailed wire and
|
||||
storage contracts in their canonical owners.
|
||||
4. Review changes against the feature roadmap and the implementation-discipline
|
||||
guidance. Do not elevate selected storage encoding, internal ordering, test
|
||||
synchronization, or exact error wording into new product guarantees.
|
||||
5. Run the full repository gates below; document results and update completion
|
||||
statuses only after they pass. Leave release preparation for a separate task.
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
@@ -264,12 +352,10 @@ go test ./internal/config -run '^TestExamplesLoadAndValidate$'
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Completion: The requested interface, failure and reuse behavior, examples, and
|
||||
canonical documentation agree; all checks pass. No release is part of this
|
||||
implementation sprint.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The consumer request resolves the product behavior. The choices above
|
||||
settle implementation details without adding new operator restrictions,
|
||||
downstream configuration ownership, or cache guarantees.
|
||||
None requiring user input. The agreed roadmap supplies the product decisions;
|
||||
this plan selects a completed-only manifest collection, conservative old-record
|
||||
handling, and pull-based replay as implementation choices. No stronger remote
|
||||
reliability guarantee, new operator restriction, or generic orchestration
|
||||
surface is proposed.
|
||||
|
||||
Reference in New Issue
Block a user