From d1802f99f027df017875fc8d4b8791efde522934 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 3 May 2026 22:24:27 -0500 Subject: [PATCH] Document Seriatim merge integration --- README.md | 16 +++++++++------- architecture.md | 40 +++++++++++++++++++++++++--------------- examples/speakers.yml | 7 +++++-- 3 files changed, 39 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 6e2d483..11cf41b 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ `narratio` is a Go-based orchestration application for processing D&D session audio into transcripts and downstream artifacts. -This repository currently contains a **working scaffold** with strict config loading, local workdir/manifest handling, resumable stage control, a real WhisperX HTTP adapter, and a real `prepare` + `transcribe` path. +This repository currently contains a **working scaffold** with strict config loading, local workdir/manifest handling, resumable stage control, real WhisperX and Seriatim adapters, and real `prepare` + `transcribe` + `merge` stages. ## Expected Config Files @@ -26,7 +26,7 @@ Seriatim config contract in `pipeline.yml`: `speakers.yml` note: - use Seriatim’s documented `match:` format (not the legacy direct mapping style used by older scripts/scaffolds) -- TODO: add a concrete `speakers.yml` example once the Seriatim README/spec is vendored or linked in-repo. +- see [`examples/speakers.yml`](examples/speakers.yml) for a concrete `match:` example. Decoding is strict (`KnownFields(true)`), so unknown YAML fields fail fast. @@ -42,13 +42,12 @@ Implemented now: - strict config load + validation - local artifact/workdir creation and locking - manifest create/load/save and stage status tracking -- stage framework with real `prepare` and `transcribe` stages; placeholder downstream stages +- stage framework with real `prepare`, `transcribe`, and `merge` stages; placeholder downstream stages - resumable run control (`run`, `resume`, `run-stage`, `plan` with run/skip decisions) -- real WhisperX HTTP adapter plus fake/no-op adapters for test/scaffold usage +- real WhisperX HTTP adapter plus real Seriatim subprocess adapter (both with fake/no-op adapters for test/scaffold usage) Not implemented yet: -- real Seriatim execution - real Audita execution - real analyzer integration - real remote archive/storage backend @@ -68,9 +67,12 @@ go run ./cmd/narratio plan --config examples/pipeline.minimal.yml --session exam ## Run Pipeline (Current State) -The current `run` command executes `prepare` + real `transcribe` + placeholder downstream stages and records progress in `manifest.json`. +The current `run` command executes `prepare` + real `transcribe` + real `merge` + placeholder downstream stages and records progress in `manifest.json`. -Default CLI wiring builds and uses the real WhisperX HTTP adapter from `pipeline.whisperx` when `whisperx.transcribe_url` is configured. +Default CLI wiring builds and uses: + +- real WhisperX HTTP adapter from `pipeline.whisperx` +- real Seriatim subprocess adapter from `pipeline.seriatim` ```bash go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml diff --git a/architecture.md b/architecture.md index 25068b3..86cc9d5 100644 --- a/architecture.md +++ b/architecture.md @@ -4,7 +4,7 @@ `narratio` is a Go orchestrator for D&D session processing. It coordinates a stage-based pipeline from recorded audio through transcript and artifact generation, while preserving durable run state for skip/rerun/resume behavior. -The repository currently implements the orchestration scaffold, local state model, and stage framework, including a real WhisperX HTTP adapter and a real `transcribe` stage. It intentionally does **not** yet implement real Seriatim, Audita, analyzer, remote archive, or notification integrations. +The repository currently implements the orchestration scaffold, local state model, and stage framework, including real WhisperX and Seriatim adapters plus real `prepare`/`transcribe`/`merge` stages. It intentionally does **not** yet implement real Audita, analyzer, remote archive, or notification integrations. ## 2. Design Goals @@ -36,15 +36,16 @@ Implemented now: - Stage interface, canonical stage ordering, and runner main loop. - Real `prepare` stage (input resolution/materialization/provenance). - Real `transcribe` stage (prepared-audio discovery, WhisperX adapter execution, JSON transcript validation, provenance metadata). +- Real `merge` stage (raw-transcript discovery, Seriatim adapter execution, merged/report JSON validation, provenance metadata). - Real WhisperX HTTP adapter implementation (multipart POST + retries + timeout + atomic output writes). -- Placeholder downstream stages (`normalize`..`notify`) with adapter contract calls. +- Real Seriatim subprocess adapter implementation (deterministic CLI/env construction + output/report JSON validation). +- Placeholder downstream stages (`normalize`, `polish`, `analyze`, `archive`, `notify`) with adapter contract calls. - Adapter interfaces and fake/no-op implementations for all external boundaries. - Reusable subprocess helper and generated YAML/config writing helper. - Test coverage across config, manifest, artifacts, planning, runner control, and adapters. Still planned/future: -- Real Seriatim adapter. - Real Audita adapter. - Real analyzer integration. - Real remote archive/storage backend (S3/SFTP/etc). @@ -68,7 +69,8 @@ Execution status: - `prepare` is implemented with real local filesystem behavior. - `transcribe` is implemented and validates raw transcript JSON outputs. -- `normalize`/`merge`/`polish`/`analyze`/`archive`/`notify` remain placeholders. +- `merge` is implemented and validates merged transcript/report JSON outputs. +- `normalize`/`polish`/`analyze`/`archive`/`notify` remain placeholders. ## 6. CLI Commands @@ -249,7 +251,14 @@ Current `Declares` role: - Calls `whisperx.Client` with bounded parallelism from `pipeline.whisperx.concurrency`. - Validates each output file exists and is valid JSON before stage success. - Records output refs plus stage metadata (`audio_files_count`, language/concurrency/retry settings, output paths, per-file attempts/status/duration/path). -- `normalize`..`notify` (placeholder): +- `merge` (real): + - Discovers raw transcript inputs from `manifest.stages.transcribe.outputs` (`kind=transcript_raw`) or `work/.../transcripts/raw` fallback. + - Validates input transcript files as existing regular JSON files. + - Uses prepared `inputs/speakers.yml` and `inputs/autocorrect.yml` directly (no speaker-map format translation). + - Invokes `seriatim.Runner` using canonical merged/report/log/generated-config paths under the session workdir. + - Validates merged transcript output JSON and (when enabled) report JSON before stage success. + - Records merged/report output refs plus stage metadata (input paths/count, Seriatim settings, output paths, adapter duration/exit metadata). +- `normalize`/`polish`/`analyze`/`archive`/`notify` (placeholder): - Return placeholder metadata. - Optionally call adapters using expected request/result contract shapes. @@ -274,7 +283,7 @@ Adapter boundaries (`internal/adapters/*`): - `storage.Backend` - `notify.Sender` -All adapters currently have fake/no-op implementations for tests/scaffold execution. WhisperX additionally has a real HTTP adapter implementation under `internal/adapters/whisperx/http.go`. +All adapters currently have fake/no-op implementations for tests/scaffold execution. WhisperX has a real HTTP adapter (`internal/adapters/whisperx/http.go`) and Seriatim has a real subprocess adapter (`internal/adapters/seriatim/subprocess.go`). ### Subprocess helper @@ -309,11 +318,11 @@ Stale detection is intentionally TODO (`run_control.go`) pending checksum-based - Structured logger is initialized via `internal/logging` (`slog` text handler). - Runner emits concise stage lifecycle logs (skip/start/success/fail + manifest save points). -- Placeholder subprocess adapter requests include distinct stdout/stderr log paths to preserve future boundary design. +- Real WhisperX/Seriatim stages already use explicit output/log/config paths; placeholder downstream subprocess adapters keep that same boundary pattern. ### Long-running stage expectations -The architecture expects long-running stages (especially WhisperX and Audita). WhisperX already uses context timeout/cancellation and retry logic inside its HTTP adapter; other long-running integrations are still pending. +The architecture expects long-running stages (especially WhisperX, Seriatim, and Audita). WhisperX and Seriatim already use context timeout/cancellation in their adapters (with WhisperX retry logic); other long-running integrations are still pending. ## 13. Testing Strategy @@ -326,8 +335,10 @@ Current tests verify scaffold behavior without real external services: - Run/skip/force/resume/run-stage control behavior. - Prepare-stage input materialization/provenance/idempotency. - Real transcribe-stage audio discovery/concurrency/failure handling/output validation/provenance. +- Real merge-stage input discovery/validation, Seriatim adapter failure handling, merged/report output validation, and provenance recording. - Adapter fake behavior and error propagation. - WhisperX HTTP adapter behavior (request shape, retry policy, timeout/cancel, JSON validation, atomic writes). +- Seriatim subprocess adapter behavior (arg/env construction, timeout/failure handling, output/report JSON validation). - Subprocess helper behavior (success/failure/timeout/log capture). Tests intentionally avoid hardcoding arbitrary operational default values. @@ -337,13 +348,12 @@ Tests intentionally avoid hardcoding arbitrary operational default values. Recommended implementation sequence (one focused boundary at a time): 1. Implement real `normalize` transcript transformation/validation. -2. Implement real `merge` using Seriatim adapter + generated config + subprocess logs. -3. Implement real `polish` using Audita adapter + checkpoint/log handling. -4. Implement real `analyze` adapter integration and artifact validation. -5. Implement real `archive` remote backend behavior. -6. Implement real `notify` backend. -7. Add checksum-based stale detection and stale status transitions. -8. Add selective parallelism where architecturally safe (`transcribe` fan-out and/or downstream-safe boundaries). +2. Implement real `polish` using Audita adapter + checkpoint/log handling. +3. Implement real `analyze` adapter integration and artifact validation. +4. Implement real `archive` remote backend behavior. +5. Implement real `notify` backend. +6. Add checksum-based stale detection and stale status transitions. +7. Add selective parallelism where architecturally safe (`transcribe` fan-out and/or downstream-safe boundaries). Each step must preserve existing package boundaries and manifest-based control flow. diff --git a/examples/speakers.yml b/examples/speakers.yml index ed4acd7..7357b02 100644 --- a/examples/speakers.yml +++ b/examples/speakers.yml @@ -1,2 +1,5 @@ -sample-speaker: sample-speaker.flac - +match: + - speaker: "Eric Rakestraw" + match: + - "Eric_Rakestraw" + - "Eric"