narratio/architecture.md

# Narratio Architecture

## 1. Purpose

`narratio` is a Go orchestrator for D&D session processing. It coordinates a stage-based pipeline from recorded audio through transcript and artifact generation, while preserving durable run state for skip/rerun/resume behavior.

The repository currently implements the orchestration scaffold, local state model, and stage framework, including real WhisperX/Seriatim/Audita adapters plus real `prepare`/`transcribe`/`merge`/`polish` stages. It intentionally does **not** yet implement real analyzer, remote archive, or notification integrations.

## 2. Design Goals

- Modular boundaries between orchestration, stages, adapters, storage, config, and manifest.
- Config-driven behavior with strict decoding and conservative validation.
- Durable manifest-based control for resumability and idempotent reruns.
- Boring, explicit control flow (no workflow engine/DAG abstraction).
- Observable orchestration via structured logs plus per-tool log file paths.
- Safe extension points for incremental real-stage implementation.

## 3. Non-Goals

Current and near-term non-goals:

- Reimplementing WhisperX, Seriatim, Audita, or analyzer internals.
- Building a generic distributed workflow engine.
- Adding a DB dependency for v1.
- Embedding D&D prompt logic in orchestration core.
- Implementing checksum-based stale detection (planned, not implemented).

## 4. Current Implementation Status

Implemented now:

- CLI entrypoint and command dispatch (`run`, `plan`, `status`, `resume`, `run-stage`).
- Strict `pipeline.yml`/`session.yml` loading and validation.
- Local workdir and artifact-store abstraction with lock-file support.
- Durable local JSON manifest store with atomic writes.
- Stage interface, canonical stage ordering, and runner main loop.
- Real `prepare` stage (input resolution/materialization/provenance).
- Real `transcribe` stage (prepared-audio discovery, WhisperX adapter execution, JSON transcript validation, provenance metadata).
- Real `merge` stage (raw-transcript discovery, Seriatim adapter execution, merged/report JSON validation, provenance metadata).
- Real `polish` stage (merged-transcript discovery, Audita adapter execution, processed/report JSON validation, provenance metadata).
- Real WhisperX HTTP adapter implementation (multipart POST + retries + timeout + atomic output writes).
- Real Seriatim subprocess adapter implementation (deterministic CLI/env construction + output/report JSON validation).
- Real Audita subprocess adapter implementation (deterministic CLI/env construction + credential env handling + output/report JSON validation).
- Placeholder downstream stages (`normalize`, `analyze`, `archive`, `notify`) with adapter contract calls.
- Adapter interfaces and fake/no-op implementations for all external boundaries.
- Reusable subprocess helper and generated YAML/config writing helper.
- Test coverage across config, manifest, artifacts, planning, runner control, and adapters.

Still planned/future:

- Real analyzer integration.
- Real remote archive/storage backend (S3/SFTP/etc).
- Real notification backend.
- Stale detection based on input/config checksums.

## 5. High-Level Pipeline

Canonical stage order is fixed in code:

1. `prepare`
2. `transcribe`
3. `normalize`
4. `merge`
5. `polish`
6. `analyze`
7. `archive`
8. `notify`

Execution status:

- `prepare` is implemented with real local filesystem behavior.
- `transcribe` is implemented and validates raw transcript JSON outputs.
- `merge` is implemented and validates merged transcript/report JSON outputs.
- `polish` is implemented and validates processed transcript/report JSON outputs.
- `normalize`/`analyze`/`archive`/`notify` remain placeholders.

## 6. CLI Commands

`cmd/narratio/main.go` is intentionally thin and delegates to `internal/app`.

Current command behavior:

- `run --config <pipeline.yml> --session <session.yml> [--force]`
  - Loads + validates config, ensures workdir, acquires lock, loads/creates manifest, executes full stage plan.
  - Skips already-succeeded stages unless `--force`.
- `plan --config <pipeline.yml> --session <session.yml> [--force]`
  - Loads + validates config, ensures workdir, loads manifest if present, prints `run`/`skip` action per stage and totals.
- `resume --config <pipeline.yml> --session <session.yml> [--force]`
  - Resumes from first non-succeeded stage based on manifest (or full run when forced).
- `run-stage --config <pipeline.yml> --session <session.yml> [--force] <stage>`
  - Executes exactly one named stage; unknown stage is an error.
- `status --manifest <path>`
  - Loads manifest and prints session ID, updated timestamp, and stage statuses.

Invalid usage returns non-zero with usage/error text.

## 7. Configuration

Package: `internal/config`

Files:

- `pipeline.yml`
- `session.yml`

Key behavior:

- Strict YAML decoding via `yaml.Decoder.KnownFields(true)`.
- Unknown fields are rejected.
- Combined resolved config type keeps source paths (`PipelinePath`, `SessionPath`) for provenance/errors.
- WhisperX optional fields are defaulted during load for deterministic resolved config values.

WhisperX config keys:

- `pipeline.whisperx.transcribe_url` (required)
- `pipeline.whisperx.language` (default: `en`)
- `pipeline.whisperx.timeout` (default: `30m`)
- `pipeline.whisperx.retries` (default: `3`)
- `pipeline.whisperx.retry_delay` (default: `2s`)
- `pipeline.whisperx.concurrency` (default: `2`)

Seriatim config keys:

- `pipeline.seriatim.binary` (required)
- `pipeline.seriatim.timeout` (default: `10m`)
- `pipeline.seriatim.output_schema` (default: `seriatim-intermediate`; allowed: `seriatim-minimal|seriatim-intermediate|seriatim-full`)
- `pipeline.seriatim.coalesce_gap` (default: `3.0`, must be `>= 0`)
- `pipeline.seriatim.report` (default: `true`)
- optional tuning under `pipeline.seriatim.env.*` (must be `> 0` when provided)

Audita config keys:

- `pipeline.audita.binary` (required)
- `pipeline.audita.timeout` (default: `3h`)
- `pipeline.audita.llm_api_key_env` (required; default: `AUDITA_LLM_API_KEY`)
- `pipeline.audita.modules` (default sequence: `glossary,homophones,glossary,spoken_word,grammar,homophones,glossary`)
- `pipeline.audita.base_url` (default: `https://openrouter.ai/api/v1`)
- `pipeline.audita.model` (default: `openrouter/google/gemma-4-31b-it`)
- `pipeline.audita.llm_concurrency` (default: `1`, must be `> 0`)
- `pipeline.audita.validation_model` (default: empty string)
- `pipeline.audita.validation_llm_concurrency` (default: `1`, must be `> 0`)
- `pipeline.audita.report` (default: `true`)

Audita secret-handling policy:

- `llm_api_key_env` stores only the environment variable **name**.
- API key values are read from the process environment at runtime and are not stored in `pipeline.yml`, manifest metadata, generated configs, or logs.

Validation currently enforces:

- `pipeline.workspace.root` is required.
- `pipeline.whisperx.transcribe_url` is required and must be a valid URL.
- `pipeline.whisperx.timeout` and `pipeline.whisperx.retry_delay` must parse as Go durations.
- `pipeline.whisperx.retries` must be `>= 0`.
- `pipeline.whisperx.concurrency` must be `> 0`.
- `pipeline.seriatim.binary` is required.
- `pipeline.seriatim.timeout` must parse as Go duration.
- `pipeline.seriatim.output_schema` must be one of `seriatim-minimal|seriatim-intermediate|seriatim-full`.
- `pipeline.seriatim.coalesce_gap` must be `>= 0`.
- optional `pipeline.seriatim.env.*` values must be `> 0` when provided.
- `pipeline.audita.binary` is required.
- `pipeline.audita.timeout` must parse as Go duration.
- `pipeline.audita.llm_api_key_env` is required.
- `pipeline.audita.modules` must be non-empty and each entry must be one of `glossary|homophones|spoken_word|grammar`.
- `pipeline.audita.base_url` must be a valid URL when provided.
- `pipeline.audita.model` is required.
- `pipeline.audita.llm_concurrency` must be `> 0`.
- `pipeline.audita.validation_llm_concurrency` must be `> 0`.
- `session.session_id` is required.
- `session.inputs.speakers_file`, `autocorrect_file`, `glossary_file` are required.
- At least one audio source: `session.inputs.audio_dir` or non-empty `session.inputs.audio_files`.
- Configured timeout fields must parse as Go durations when present.

Validation scope is intentionally conservative (no deep business defaults, no full schema semantics, no remote connectivity checks).

## 8. Manifest and Stage Status

Package: `internal/manifest`

Model includes:

- `Manifest`
- `StageRecord`
- `ArtifactRecord`
- `InputRecord`
- `ErrorRecord`
- `StageStatus`

Supported stage statuses:

- `pending`
- `running`
- `succeeded`
- `failed`
- `skipped`
- `stale` (defined but not actively produced by runner yet)
- `interrupted` (defined for future use)

Store behavior:

- Local JSON store with create/load/save.
- Save is atomic (same-directory temp file + rename).
- `updated_at` refreshed on save.

Helpers implemented:

- `MarkStageRunning`
- `MarkStageSucceeded`
- `MarkStageFailed`
- `MarkStageSkipped`

Runner currently records `running/succeeded/failed` transitions and preserves existing succeeded stage metadata when skipping.

Manifest versioning note:

- Current manifest model includes optional `pipeline_version`, but does **not** include an explicit manifest schema/version field.
- Before manifests become long-term compatibility-sensitive across releases/storage backends, add explicit manifest schema/versioning and migration policy.

## 9. Workdir and Artifact Model

Package: `internal/artifacts`

Canonical local paths are resolved by `SessionPaths` under:

`{workspace.root}/work/{session_id}/`

Subpaths:

- `inputs/`
- `audio/`
- `transcripts/raw/`
- `transcripts/normalized/`
- `artifacts/`
- `config/`
- `logs/`
- `manifest.json`
- `.lock`

Implemented store capabilities:

- Ensure layout.
- Resolve session paths.
- Copy inputs.
- Existence checks.
- SHA-256 checksums.
- Atomic file writes.
- Session lock acquire/release (`.lock`, conflict-safe via `O_EXCL`).

`Ref` supports local path info plus optional future remote key/checksum metadata.

## 10. Stage Architecture

Package: `internal/stage`

Core contracts:

- `Stage` interface with `Name`, `Declares`, and `Run(ctx, env, manifest)`.
- `IODecl` for declared input/output artifact intent.
- `StageResult` for outputs, logs, generated configs, and metadata.

Current `Declares` role:

- `Stage.Declares` is currently contract metadata; the runner does not yet enforce declared inputs/outputs at execution time.
- Real-stage implementation work should strengthen enforcement by validating declared inputs before execution and validating declared outputs after execution.

### Current stage behaviors

- `prepare` (real):
  - Ensures layout.
  - Resolves session-relative input paths.
  - Materializes required input files into `inputs/`.
  - Writes `pipeline.resolved.yml`.
  - Resolves/copies `.flac` audio inputs from `audio_dir` or `audio_files`.
  - Computes checksums and records deterministic `manifest.inputs`.
  - Uses checksum-aware write/copy reuse for idempotency.
- `transcribe` (real):
  - Discovers prepared `.flac` audio inputs from `manifest.inputs` (`kind=audio`) or `work/.../audio` fallback.
  - Derives per-speaker output files at `transcripts/raw/{audio_basename}.json`.
  - Calls `whisperx.Client` with bounded parallelism from `pipeline.whisperx.concurrency`.
  - Validates each output file exists and is valid JSON before stage success.
  - Records output refs plus stage metadata (`audio_files_count`, language/concurrency/retry settings, output paths, per-file attempts/status/duration/path).
- `merge` (real):
  - Discovers raw transcript inputs from `manifest.stages.transcribe.outputs` (`kind=transcript_raw`) or `work/.../transcripts/raw` fallback.
  - Validates input transcript files as existing regular JSON files.
  - Uses prepared `inputs/speakers.yml` and `inputs/autocorrect.yml` directly (no speaker-map format translation).
  - Invokes `seriatim.Runner` using canonical merged/report/log/generated-config paths under the session workdir.
  - Validates merged transcript output JSON and (when enabled) report JSON before stage success.
  - Records merged/report output refs plus stage metadata (input paths/count, Seriatim settings, output paths, adapter duration/exit metadata).
- `polish` (real):
  - Discovers merged transcript input from `manifest.stages.merge.outputs` (`kind=transcript_merged`) or fallback `work/.../transcripts/merged.json`.
  - Validates merged transcript as JSON and requires prepared `inputs/glossary.yml`.
  - Invokes `audita.Runner` using canonical paths:
    - `transcripts/processed.json`
    - `artifacts/audita.report.json` (when enabled)
    - `artifacts/audita-work`
    - `logs/audita.stdout.log`
    - `logs/audita.stderr.log`
    - `config/audita.generated.yml`
  - Validates processed transcript output JSON with required top-level `segments` array and validates report JSON when enabled.
  - Records processed/report outputs plus non-secret provenance metadata (module/model/base-url/concurrency settings, paths, adapter duration/exit metadata, credential env-var name/presence).
- `normalize`/`analyze`/`archive`/`notify` (placeholder):
  - Return placeholder metadata.
  - Optionally call adapters using expected request/result contract shapes.

## 11. Application Environment and Adapters

`internal/app` uses `stage.Env` as the shared dependency container to avoid duplicate environment definitions.

Contained dependencies:

- Resolved config
- Artifact store
- Manifest store
- Logger
- Adapter interfaces

Adapter boundaries (`internal/adapters/*`):

- `whisperx.Client`
- `seriatim.Runner`
- `audita.Runner`
- `analyzer.Runner`
- `storage.Backend`
- `notify.Sender`

All adapters currently have fake/no-op implementations for tests/scaffold execution. WhisperX has a real HTTP adapter (`internal/adapters/whisperx/http.go`), Seriatim has a real subprocess adapter (`internal/adapters/seriatim/subprocess.go`), and Audita has a real subprocess adapter (`internal/adapters/audita/subprocess.go`).

### Subprocess helper

`internal/adapters/subprocess` provides reusable subprocess scaffolding:

- Context cancellation + optional timeout.
- Explicit executable/args, working dir, env overrides.
- Stdout/stderr log file handling.
- Exit code and timing capture.
- Actionable error wrapping.
- Atomic file/YAML writers for generated config/log scaffolding.

## 12. Run Control, Locking, Logging, and Long-Running Stages

### Run control

`internal/app` runner is sequential and manifest-driven:

- Loads/creates manifest.
- Computes per-stage action (`run` vs `skip`).
- Skip rule today: succeeded + not forced => skip.
- `--force` reruns previously succeeded stages.
- `resume` starts at first non-succeeded stage.

Stale detection is intentionally TODO (`run_control.go`) pending checksum-based invalidation logic.

### Locking

`run`/`resume`/`run-stage` execution paths acquire a session lock via artifact store and release it with `defer`.

### Logging

- Structured logger is initialized via `internal/logging` (`slog` text handler).
- Runner emits concise stage lifecycle logs (skip/start/success/fail + manifest save points).
- Real WhisperX/Seriatim/Audita stages already use explicit output/log/config paths; placeholder downstream subprocess adapters keep that same boundary pattern.

### Long-running stage expectations

The architecture expects long-running stages (especially WhisperX, Seriatim, and Audita). WhisperX/Seriatim/Audita already use context timeout/cancellation in their adapters (with WhisperX retry logic); other long-running integrations are still pending.

## 13. Testing Strategy

Current tests verify scaffold behavior without real external services:

- Strict config load/validate behavior and error quality.
- Manifest store round trips and transition semantics.
- Workdir layout, atomic writes, checksums, and lock behavior.
- Plan order and stage selection.
- Run/skip/force/resume/run-stage control behavior.
- Prepare-stage input materialization/provenance/idempotency.
- Real transcribe-stage audio discovery/concurrency/failure handling/output validation/provenance.
- Real merge-stage input discovery/validation, Seriatim adapter failure handling, merged/report output validation, and provenance recording.
- Real polish-stage merged-input discovery/validation, Audita adapter failure handling, processed/report output validation, and provenance recording.
- Adapter fake behavior and error propagation.
- WhisperX HTTP adapter behavior (request shape, retry policy, timeout/cancel, JSON validation, atomic writes).
- Seriatim subprocess adapter behavior (arg/env construction, timeout/failure handling, output/report JSON validation).
- Audita subprocess adapter behavior (arg/env construction, credential handling, timeout/failure handling, output/report JSON validation).
- Subprocess helper behavior (success/failure/timeout/log capture).

Tests intentionally avoid hardcoding arbitrary operational default values.

## 14. Future Work / Implementation Roadmap

Recommended implementation sequence (one focused boundary at a time):

1. Implement real `normalize` transcript transformation/validation.
2. Implement real `analyze` adapter integration and artifact validation.
3. Implement real `archive` remote backend behavior.
4. Implement real `notify` backend.
5. Add checksum-based stale detection and stale status transitions.
6. Add selective parallelism where architecturally safe (`transcribe` fan-out and/or downstream-safe boundaries).

Each step must preserve existing package boundaries and manifest-based control flow.

### Real Stage Implementation Checklist

For each stage moved from placeholder to real behavior:

- Implement and/or use the appropriate adapter behavior for that stage boundary.
- Generate and use config/log paths needed by the adapter/tool invocation.
- Validate expected stage inputs before execution.
- Write outputs atomically where practical (temp + promote/rename).
- Validate expected outputs after execution before marking success.
- Record outputs, logs, generated configs, and provenance metadata in manifest stage records.
- Preserve `skip`/`force`/`resume` semantics (do not regress run-control behavior).
- Add fake-adapter tests and at least one explicit failure-path test for stage behavior.

## 15. Architectural Invariants

The following invariants are mandatory for future changes:

1. Stage orchestration logic lives in `internal/app`; stage business logic lives in `internal/stage`.
2. External tool details remain behind adapter interfaces.
3. Config loading is strict and validation-first.
4. Manifest is the source of truth for stage status and resume/skip decisions.
5. Stage outputs/provenance are recorded durably and deterministically.
6. Local workdir is the primary execution area; remote storage is an adapter concern.
7. Main entrypoint stays thin (`cmd/narratio` delegates to `internal/app`).
8. Pipeline behavior remains explicit and maintainable; no generic DAG engine introduction.
9. Every stage must consume declared artifacts, produce declared artifacts, validate them, record provenance, and remain safely skippable/rerunnable.