diff --git a/.gitignore b/.gitignore index 5c50ec6..6874f32 100644 --- a/.gitignore +++ b/.gitignore @@ -25,6 +25,10 @@ go.work.sum # env file .env +# Compiled binaries and test configuration +narratio +pipeline.yml + # ---> VisualStudioCode .vscode/* !.vscode/settings.json diff --git a/README.md b/README.md index 66a0f83..aaea0db 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ `narratio` is a Go-based orchestration application for processing D&D session audio into transcripts and downstream artifacts. -This repository currently contains a **working scaffold** with strict config loading, local workdir/manifest handling, resumable stage control, and placeholder adapters/stages. +This repository currently contains a **working scaffold** with strict config loading, local workdir/manifest handling, resumable stage control, a real WhisperX HTTP adapter, and a real `prepare` + `transcribe` path. ## Expected Config Files @@ -25,18 +25,17 @@ Example minimal files are available under `examples/`: ## Current Scaffold Status -Implemented in scaffold form: +Implemented now: - strict config load + validation - local artifact/workdir creation and locking - manifest create/load/save and stage status tracking -- stage framework with real `prepare` stage and placeholder downstream stages +- stage framework with real `prepare` and `transcribe` stages; placeholder downstream stages - resumable run control (`run`, `resume`, `run-stage`, `plan` with run/skip decisions) -- fake/no-op adapters for external tool boundaries +- real WhisperX HTTP adapter plus fake/no-op adapters for test/scaffold usage Not implemented yet: -- real WhisperX HTTP integration - real Seriatim execution - real Audita execution - real analyzer integration @@ -55,9 +54,11 @@ go test ./... go run ./cmd/narratio plan --config examples/pipeline.minimal.yml --session examples/session.minimal.yml ``` -## Run Placeholder Pipeline +## Run Pipeline (Current State) -The current `run` command executes available scaffold behavior (`prepare` + placeholder stages) and records progress in `manifest.json`. +The current `run` command executes `prepare` + real `transcribe` + placeholder downstream stages and records progress in `manifest.json`. + +Note: default CLI wiring currently injects `whisperx.NoopClient` in the runner to avoid implicit network dependency; the real HTTP adapter is implemented under `internal/adapters/whisperx/http.go` and is used via explicit env wiring/tests. ```bash go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml diff --git a/architecture.md b/architecture.md index 84c1964..7f4b76f 100644 --- a/architecture.md +++ b/architecture.md @@ -4,7 +4,7 @@ `narratio` is a Go orchestrator for D&D session processing. It coordinates a stage-based pipeline from recorded audio through transcript and artifact generation, while preserving durable run state for skip/rerun/resume behavior. -The repository currently implements the orchestration scaffold, local state model, and stage framework. It intentionally does **not** yet implement real external integrations (WhisperX, Seriatim, Audita, analyzer, remote archive, notifications). +The repository currently implements the orchestration scaffold, local state model, and stage framework, including a real WhisperX HTTP adapter and a real `transcribe` stage. It intentionally does **not** yet implement real Seriatim, Audita, analyzer, remote archive, or notification integrations. ## 2. Design Goals @@ -35,14 +35,15 @@ Implemented now: - Durable local JSON manifest store with atomic writes. - Stage interface, canonical stage ordering, and runner main loop. - Real `prepare` stage (input resolution/materialization/provenance). -- Placeholder downstream stages (`transcribe`..`notify`) with adapter contract calls. +- Real `transcribe` stage (prepared-audio discovery, WhisperX adapter execution, JSON transcript validation, provenance metadata). +- Real WhisperX HTTP adapter implementation (multipart POST + retries + timeout + atomic output writes). +- Placeholder downstream stages (`normalize`..`notify`) with adapter contract calls. - Adapter interfaces and fake/no-op implementations for all external boundaries. - Reusable subprocess helper and generated YAML/config writing helper. - Test coverage across config, manifest, artifacts, planning, runner control, and adapters. Still planned/future: -- Real WhisperX HTTP adapter. - Real Seriatim adapter. - Real Audita adapter. - Real analyzer integration. @@ -66,7 +67,8 @@ Canonical stage order is fixed in code: Execution status: - `prepare` is implemented with real local filesystem behavior. -- All other stages are placeholders that return metadata and optionally exercise fake adapters. +- `transcribe` is implemented and validates raw transcript JSON outputs. +- `normalize`/`merge`/`polish`/`analyze`/`archive`/`notify` remain placeholders. ## 6. CLI Commands @@ -104,6 +106,15 @@ Key behavior: - Combined resolved config type keeps source paths (`PipelinePath`, `SessionPath`) for provenance/errors. - WhisperX optional fields are defaulted during load for deterministic resolved config values. +WhisperX config keys: + +- `pipeline.whisperx.transcribe_url` (required) +- `pipeline.whisperx.language` (default: `en`) +- `pipeline.whisperx.timeout` (default: `30m`) +- `pipeline.whisperx.retries` (default: `3`) +- `pipeline.whisperx.retry_delay` (default: `2s`) +- `pipeline.whisperx.concurrency` (default: `2`) + Validation currently enforces: - `pipeline.workspace.root` is required. @@ -218,7 +229,13 @@ Current `Declares` role: - Resolves/copies `.flac` audio inputs from `audio_dir` or `audio_files`. - Computes checksums and records deterministic `manifest.inputs`. - Uses checksum-aware write/copy reuse for idempotency. -- `transcribe`..`notify` (placeholder): +- `transcribe` (real): + - Discovers prepared `.flac` audio inputs from `manifest.inputs` (`kind=audio`) or `work/.../audio` fallback. + - Derives per-speaker output files at `transcripts/raw/{audio_basename}.json`. + - Calls `whisperx.Client` with bounded parallelism from `pipeline.whisperx.concurrency`. + - Validates each output file exists and is valid JSON before stage success. + - Records output refs plus stage metadata (`audio_files_count`, language/concurrency/retry settings, output paths, per-file attempts/status/duration/path). +- `normalize`..`notify` (placeholder): - Return placeholder metadata. - Optionally call adapters using expected request/result contract shapes. @@ -243,7 +260,7 @@ Adapter boundaries (`internal/adapters/*`): - `storage.Backend` - `notify.Sender` -All adapters currently have fake/no-op implementations for tests/scaffold execution. Real integration TODOs are explicitly marked in adapter contract files. +All adapters currently have fake/no-op implementations for tests/scaffold execution. WhisperX additionally has a real HTTP adapter implementation under `internal/adapters/whisperx/http.go`. ### Subprocess helper @@ -278,11 +295,11 @@ Stale detection is intentionally TODO (`run_control.go`) pending checksum-based - Structured logger is initialized via `internal/logging` (`slog` text handler). - Runner emits concise stage lifecycle logs (skip/start/success/fail + manifest save points). -- Placeholder adapter requests include distinct stdout/stderr log paths to preserve future boundary design. +- Placeholder subprocess adapter requests include distinct stdout/stderr log paths to preserve future boundary design. ### Long-running stage expectations -The architecture expects long-running stages (especially Audita and WhisperX) and already provides context-based cancellation and subprocess scaffolding, but real long-running integrations are still pending. +The architecture expects long-running stages (especially WhisperX and Audita). WhisperX already uses context timeout/cancellation and retry logic inside its HTTP adapter; other long-running integrations are still pending. ## 13. Testing Strategy @@ -294,7 +311,9 @@ Current tests verify scaffold behavior without real external services: - Plan order and stage selection. - Run/skip/force/resume/run-stage control behavior. - Prepare-stage input materialization/provenance/idempotency. +- Real transcribe-stage audio discovery/concurrency/failure handling/output validation/provenance. - Adapter fake behavior and error propagation. +- WhisperX HTTP adapter behavior (request shape, retry policy, timeout/cancel, JSON validation, atomic writes). - Subprocess helper behavior (success/failure/timeout/log capture). Tests intentionally avoid hardcoding arbitrary operational default values. @@ -303,14 +322,14 @@ Tests intentionally avoid hardcoding arbitrary operational default values. Recommended implementation sequence (one focused boundary at a time): -1. Harden `transcribe` with real WhisperX adapter and transcript output validation. +1. Implement real `normalize` transcript transformation/validation. 2. Implement real `merge` using Seriatim adapter + generated config + subprocess logs. 3. Implement real `polish` using Audita adapter + checkpoint/log handling. 4. Implement real `analyze` adapter integration and artifact validation. 5. Implement real `archive` remote backend behavior. 6. Implement real `notify` backend. 7. Add checksum-based stale detection and stale status transitions. -8. Add selective parallelism where architecturally safe (`transcribe` fan-out). +8. Add selective parallelism where architecturally safe (`transcribe` fan-out and/or downstream-safe boundaries). Each step must preserve existing package boundaries and manifest-based control flow. diff --git a/examples/pipeline.minimal.yml b/examples/pipeline.minimal.yml index a992a44..8d1386f 100644 --- a/examples/pipeline.minimal.yml +++ b/examples/pipeline.minimal.yml @@ -5,11 +5,11 @@ storage: backend: local whisperx: - transcribe_url: https://transcription.ai.rakestrawhome.com/transcribe - language: en - timeout: 30m + transcribe_url: "https://transcription.example.com/transcribe" + language: "en" + timeout: "30m" retries: 3 - retry_delay: 2s + retry_delay: "2s" concurrency: 2 seriatim: diff --git a/internal/adapters/whisperx/client.go b/internal/adapters/whisperx/client.go index 9413f49..4fd5ed7 100644 --- a/internal/adapters/whisperx/client.go +++ b/internal/adapters/whisperx/client.go @@ -6,8 +6,6 @@ import ( "time" ) -// TODO: implement a real WhisperX HTTP client adapter. - // Client is the adapter boundary for WhisperX transcription jobs. type Client interface { Transcribe(ctx context.Context, req TranscribeRequest) (TranscribeResult, error)