Document WhisperX transcribe integration

.gitignore

@@ -25,6 +25,10 @@ go.work.sum
 # env file
 .env
 
+# Compiled binaries and test configuration
+narratio
+pipeline.yml
+
 # ---> VisualStudioCode
 .vscode/*
 !.vscode/settings.json

README.md

@@ -2,7 +2,7 @@
 
 `narratio` is a Go-based orchestration application for processing D&D session audio into transcripts and downstream artifacts.
 
-This repository currently contains a **working scaffold** with strict config loading, local workdir/manifest handling, resumable stage control, and placeholder adapters/stages.
+This repository currently contains a **working scaffold** with strict config loading, local workdir/manifest handling, resumable stage control, a real WhisperX HTTP adapter, and a real `prepare` + `transcribe` path.
 
 ## Expected Config Files
 
@@ -25,18 +25,17 @@ Example minimal files are available under `examples/`:
 
 ## Current Scaffold Status
 
-Implemented in scaffold form:
+Implemented now:
 
 - strict config load + validation
 - local artifact/workdir creation and locking
 - manifest create/load/save and stage status tracking
-- stage framework with real `prepare` stage and placeholder downstream stages
+- stage framework with real `prepare` and `transcribe` stages; placeholder downstream stages
 - resumable run control (`run`, `resume`, `run-stage`, `plan` with run/skip decisions)
-- fake/no-op adapters for external tool boundaries
+- real WhisperX HTTP adapter plus fake/no-op adapters for test/scaffold usage
 
 Not implemented yet:
 
-- real WhisperX HTTP integration
 - real Seriatim execution
 - real Audita execution
 - real analyzer integration
@@ -55,9 +54,11 @@ go test ./...
 go run ./cmd/narratio plan --config examples/pipeline.minimal.yml --session examples/session.minimal.yml
 ```
 
-## Run Placeholder Pipeline
+## Run Pipeline (Current State)
 
-The current `run` command executes available scaffold behavior (`prepare` + placeholder stages) and records progress in `manifest.json`.
+The current `run` command executes `prepare` + real `transcribe` + placeholder downstream stages and records progress in `manifest.json`.
+
+Note: default CLI wiring currently injects `whisperx.NoopClient` in the runner to avoid implicit network dependency; the real HTTP adapter is implemented under `internal/adapters/whisperx/http.go` and is used via explicit env wiring/tests.
 
 ```bash
 go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml

architecture.md

@@ -4,7 +4,7 @@
 
 `narratio` is a Go orchestrator for D&D session processing. It coordinates a stage-based pipeline from recorded audio through transcript and artifact generation, while preserving durable run state for skip/rerun/resume behavior.
 
-The repository currently implements the orchestration scaffold, local state model, and stage framework. It intentionally does **not** yet implement real external integrations (WhisperX, Seriatim, Audita, analyzer, remote archive, notifications).
+The repository currently implements the orchestration scaffold, local state model, and stage framework, including a real WhisperX HTTP adapter and a real `transcribe` stage. It intentionally does **not** yet implement real Seriatim, Audita, analyzer, remote archive, or notification integrations.
 
 ## 2. Design Goals
 
@@ -35,14 +35,15 @@ Implemented now:
 - Durable local JSON manifest store with atomic writes.
 - Stage interface, canonical stage ordering, and runner main loop.
 - Real `prepare` stage (input resolution/materialization/provenance).
-- Placeholder downstream stages (`transcribe`..`notify`) with adapter contract calls.
+- Real `transcribe` stage (prepared-audio discovery, WhisperX adapter execution, JSON transcript validation, provenance metadata).
+- Real WhisperX HTTP adapter implementation (multipart POST + retries + timeout + atomic output writes).
+- Placeholder downstream stages (`normalize`..`notify`) with adapter contract calls.
 - Adapter interfaces and fake/no-op implementations for all external boundaries.
 - Reusable subprocess helper and generated YAML/config writing helper.
 - Test coverage across config, manifest, artifacts, planning, runner control, and adapters.
 
 Still planned/future:
 
-- Real WhisperX HTTP adapter.
 - Real Seriatim adapter.
 - Real Audita adapter.
 - Real analyzer integration.
@@ -66,7 +67,8 @@ Canonical stage order is fixed in code:
 Execution status:
 
 - `prepare` is implemented with real local filesystem behavior.
-- All other stages are placeholders that return metadata and optionally exercise fake adapters.
+- `transcribe` is implemented and validates raw transcript JSON outputs.
+- `normalize`/`merge`/`polish`/`analyze`/`archive`/`notify` remain placeholders.
 
 ## 6. CLI Commands
 
@@ -104,6 +106,15 @@ Key behavior:
 - Combined resolved config type keeps source paths (`PipelinePath`, `SessionPath`) for provenance/errors.
 - WhisperX optional fields are defaulted during load for deterministic resolved config values.
 
+WhisperX config keys:
+
+- `pipeline.whisperx.transcribe_url` (required)
+- `pipeline.whisperx.language` (default: `en`)
+- `pipeline.whisperx.timeout` (default: `30m`)
+- `pipeline.whisperx.retries` (default: `3`)
+- `pipeline.whisperx.retry_delay` (default: `2s`)
+- `pipeline.whisperx.concurrency` (default: `2`)
+
 Validation currently enforces:
 
 - `pipeline.workspace.root` is required.
@@ -218,7 +229,13 @@ Current `Declares` role:
   - Resolves/copies `.flac` audio inputs from `audio_dir` or `audio_files`.
   - Computes checksums and records deterministic `manifest.inputs`.
   - Uses checksum-aware write/copy reuse for idempotency.
-- `transcribe`..`notify` (placeholder):
+- `transcribe` (real):
+  - Discovers prepared `.flac` audio inputs from `manifest.inputs` (`kind=audio`) or `work/.../audio` fallback.
+  - Derives per-speaker output files at `transcripts/raw/{audio_basename}.json`.
+  - Calls `whisperx.Client` with bounded parallelism from `pipeline.whisperx.concurrency`.
+  - Validates each output file exists and is valid JSON before stage success.
+  - Records output refs plus stage metadata (`audio_files_count`, language/concurrency/retry settings, output paths, per-file attempts/status/duration/path).
+- `normalize`..`notify` (placeholder):
   - Return placeholder metadata.
   - Optionally call adapters using expected request/result contract shapes.
 
@@ -243,7 +260,7 @@ Adapter boundaries (`internal/adapters/*`):
 - `storage.Backend`
 - `notify.Sender`
 
-All adapters currently have fake/no-op implementations for tests/scaffold execution. Real integration TODOs are explicitly marked in adapter contract files.
+All adapters currently have fake/no-op implementations for tests/scaffold execution. WhisperX additionally has a real HTTP adapter implementation under `internal/adapters/whisperx/http.go`.
 
 ### Subprocess helper
 
@@ -278,11 +295,11 @@ Stale detection is intentionally TODO (`run_control.go`) pending checksum-based
 
 - Structured logger is initialized via `internal/logging` (`slog` text handler).
 - Runner emits concise stage lifecycle logs (skip/start/success/fail + manifest save points).
-- Placeholder adapter requests include distinct stdout/stderr log paths to preserve future boundary design.
+- Placeholder subprocess adapter requests include distinct stdout/stderr log paths to preserve future boundary design.
 
 ### Long-running stage expectations
 
-The architecture expects long-running stages (especially Audita and WhisperX) and already provides context-based cancellation and subprocess scaffolding, but real long-running integrations are still pending.
+The architecture expects long-running stages (especially WhisperX and Audita). WhisperX already uses context timeout/cancellation and retry logic inside its HTTP adapter; other long-running integrations are still pending.
 
 ## 13. Testing Strategy
 
@@ -294,7 +311,9 @@ Current tests verify scaffold behavior without real external services:
 - Plan order and stage selection.
 - Run/skip/force/resume/run-stage control behavior.
 - Prepare-stage input materialization/provenance/idempotency.
+- Real transcribe-stage audio discovery/concurrency/failure handling/output validation/provenance.
 - Adapter fake behavior and error propagation.
+- WhisperX HTTP adapter behavior (request shape, retry policy, timeout/cancel, JSON validation, atomic writes).
 - Subprocess helper behavior (success/failure/timeout/log capture).
 
 Tests intentionally avoid hardcoding arbitrary operational default values.
@@ -303,14 +322,14 @@ Tests intentionally avoid hardcoding arbitrary operational default values.
 
 Recommended implementation sequence (one focused boundary at a time):
 
-1. Harden `transcribe` with real WhisperX adapter and transcript output validation.
+1. Implement real `normalize` transcript transformation/validation.
 2. Implement real `merge` using Seriatim adapter + generated config + subprocess logs.
 3. Implement real `polish` using Audita adapter + checkpoint/log handling.
 4. Implement real `analyze` adapter integration and artifact validation.
 5. Implement real `archive` remote backend behavior.
 6. Implement real `notify` backend.
 7. Add checksum-based stale detection and stale status transitions.
-8. Add selective parallelism where architecturally safe (`transcribe` fan-out).
+8. Add selective parallelism where architecturally safe (`transcribe` fan-out and/or downstream-safe boundaries).
 
 Each step must preserve existing package boundaries and manifest-based control flow.
 

examples/pipeline.minimal.yml

@@ -5,11 +5,11 @@ storage:
   backend: local
 
 whisperx:
-  transcribe_url: https://transcription.ai.rakestrawhome.com/transcribe
+  transcribe_url: "https://transcription.example.com/transcribe"
-  language: en
+  language: "en"
-  timeout: 30m
+  timeout: "30m"
   retries: 3
-  retry_delay: 2s
+  retry_delay: "2s"
   concurrency: 2
 
 seriatim:

internal/adapters/whisperx/client.go

@@ -6,8 +6,6 @@ import (
 	"time"
 )
 
-// TODO: implement a real WhisperX HTTP client adapter.
-
 // Client is the adapter boundary for WhisperX transcription jobs.
 type Client interface {
 	Transcribe(ctx context.Context, req TranscribeRequest) (TranscribeResult, error)