diff --git a/architecture.md b/architecture.md index 33a1585..bee11f1 100644 --- a/architecture.md +++ b/architecture.md @@ -1,753 +1,337 @@ -# D&D Session Orchestrator Architecture +# Narratio Architecture -## Purpose - -This repository implements a Go-based orchestration application called `narratio` for processing recorded Dungeons & Dragons session audio into durable transcripts and downstream analysis artifacts. - -The orchestrator coordinates an existing pipeline consisting of: - -1. Per-speaker FLAC audio recordings from Mumble. -2. A self-hosted WhisperX HTTP transcription service. -3. `seriatim`, a deterministic transcript merger. -4. `audita`, an LLM-backed transcript polisher. -5. A future (not yet implemented, name subject to change) `dnd-session-analyzer`, which generates final D&D session artifacts such as session logs, event logs, meta-analysis, and table-read reports. -6. Long-term storage (which may be local filesystem, S3-compatible object storage, SFTP endpoint, or similar) for all inputs, intermediate outputs, logs, generated configs, and final artifacts. +## 1. Purpose -The orchestrator is not responsible for implementing transcription, transcript merging, transcript polishing, or artifact generation. Its job is to coordinate those components reliably, maintain durable state, validate stage boundaries, manage local and remote artifacts, and make the pipeline resumable. +`narratio` is a Go orchestrator for D&D session processing. It coordinates a stage-based pipeline from recorded audio through transcript and artifact generation, while preserving durable run state for skip/rerun/resume behavior. -## Design Goals +The repository currently implements the orchestration scaffold, local state model, and stage framework. It intentionally does **not** yet implement real external integrations (WhisperX, Seriatim, Audita, analyzer, remote archive, notifications). -The application should be: +## 2. Design Goals -- **Modular**: Each stage and external component should be isolated behind a narrow interface. -- **Composable**: Stages should consume and produce declared artifacts. -- **Resumable**: Long-running workflows should be restartable without repeating completed work. -- **Idempotent**: Re-running the same pipeline should not corrupt or duplicate outputs. -- **Observable**: The orchestrator should produce structured logs, captured subprocess logs, and a durable run manifest. -- **Config-driven**: Most user-adjustable behavior should live in configuration files, not hardcoded logic. -- **Strict at boundaries**: Inputs and outputs should be validated at every major stage boundary. -- **Boring and reliable**: Prefer explicit, maintainable code over clever workflow abstractions. +- Modular boundaries between orchestration, stages, adapters, storage, config, and manifest. +- Config-driven behavior with strict decoding and conservative validation. +- Durable manifest-based control for resumability and idempotent reruns. +- Boring, explicit control flow (no workflow engine/DAG abstraction). +- Observable orchestration via structured logs plus per-tool log file paths. +- Safe extension points for incremental real-stage implementation. -The orchestrator should be written in Go. - -## Non-Goals - -The orchestrator should not: - -- Reimplement WhisperX. -- Reimplement `seriatim`. -- Reimplement `audita`. -- Contain D&D-specific prompt logic directly in the core orchestration layer. -- Parse or modify LLM outputs except for high-level validation of expected files or schemas. -- Become a general-purpose distributed workflow engine. -- Require a database for v1. -- Hide important run state only in logs. - -## High-Level Pipeline - -The intended pipeline is: - -```text -prepare - ↓ -transcribe speaker tracks in parallel - ↓ -normalize speaker transcripts - ↓ -merge transcripts with seriatim - ↓ -polish transcript with audita - ↓ -generate D&D artifacts with dnd-session-analyzer - ↓ -archive outputs - ↓ -notify user -``` - -Some stages may be implemented in the initial scaffold only as interfaces or placeholders. - -The initial implementation should create the application framework and contracts without implementing the full real behavior of all stages. +## 3. Non-Goals -## Primary Concepts -### Configuration +Current and near-term non-goals: -Configuration is user-authored and describes how the pipeline should run. - -There should be two primary configuration files: - -pipeline.yml -session.yml - -pipeline.yml contains durable/default pipeline configuration: - -```text -workspace root -S3 bucket/prefix -WhisperX service settings -seriatim binary path and options -audita binary path and options -analyzer binary path and artifact settings -notification settings -concurrency and timeout defaults -``` - -session.yml contains per-session inputs and metadata: - -```text -session ID -campaign ID -session date/title -audio input directory or explicit audio files -speakers.yml -autocorrect.yml -glossary.yml -references to previous-session context -``` - -The application should strictly decode config files and reject unknown fields. Config validation should happen before any stage is run. - -Environment variables may be used for secrets and deployment-specific credentials, but ordinary pipeline behavior should live in config files. - -### Manifest +- Reimplementing WhisperX, Seriatim, Audita, or analyzer internals. +- Building a generic distributed workflow engine. +- Adding a DB dependency for v1. +- Embedding D&D prompt logic in orchestration core. +- Implementing checksum-based stale detection (planned, not implemented). -The manifest is machine-authored durable run state. +## 4. Current Implementation Status -Each session run should have a manifest.json in the local work directory and eventually mirrored to S3. +Implemented now: -The manifest should track: - -```text -session ID -pipeline version -created/updated timestamps -resolved input files -checksums for important inputs/configs -stage statuses -stage timestamps -stage outputs -captured logs -generated config files -component versions where available -errors -artifact locations -S3 object keys -``` - -Stage statuses should be explicit: - -```text -pending -running -succeeded -failed -skipped -stale -interrupted -``` - -The manifest is the source of truth for resume/skip decisions. - -### Stages - -A stage is a pipeline unit that consumes declared inputs and produces declared outputs. - -Examples: - -```text -prepare -transcribe -normalize -merge -polish -analyze -archive -notify -``` +- CLI entrypoint and command dispatch (`run`, `plan`, `status`, `resume`, `run-stage`). +- Strict `pipeline.yml`/`session.yml` loading and validation. +- Local workdir and artifact-store abstraction with lock-file support. +- Durable local JSON manifest store with atomic writes. +- Stage interface, canonical stage ordering, and runner main loop. +- Real `prepare` stage (input resolution/materialization/provenance). +- Placeholder downstream stages (`transcribe`..`notify`) with adapter contract calls. +- Adapter interfaces and fake/no-op implementations for all external boundaries. +- Reusable subprocess helper and generated YAML/config writing helper. +- Test coverage across config, manifest, artifacts, planning, runner control, and adapters. -Stages should be organized around the lifecycle of a session, not around executable names. - -For example, the merge stage may call seriatim, but the stage should be named for the pipeline operation, not the tool. - -Every stage should: - -- Determine required inputs. -- Check whether expected outputs already exist. -- Decide whether to run, skip, or fail. -- Execute through an adapter or local logic. -- Validate expected outputs. -- Write outputs to the local work directory. -- Record stage status and metadata in the manifest. -- Allow safe rerun with `--force`. - -### Adapters - -Adapters isolate interactions with external components. - -The orchestrator should have thin adapters for: - -- WhisperX HTTP API -- seriatim -- audita -- future dnd-session-analyzer -- storage layer (local filesystem, S3-compatible, SFTP, etc.) -- optional email/notification backend - -Adapters should hide details such as: - -- CLI argument construction -- generated config file format -- HTTP request/response details -- stdout/stderr capture -- process exit handling -- timeout and cancellation behavior - -The stage layer should call adapter methods and should not know implementation details of the underlying external tool. - -### Artifact Store - -The orchestrator should use a local work directory as the primary working area. - -Each stage writes outputs locally first. The orchestrator then validates and archives those outputs to S3. - -The storage layer should expose an abstraction for artifact references and common operations: - -write local artifact -read local artifact -calculate checksum -upload to S3 -record local path and S3 key -validate existence - -For v1, local filesystem + S3 is enough. A database is not required. - -#### Expected Local Work Directory Layout - -Each session should have an isolated local work directory. - -Example: - -```text -work/{session_id}/ - inputs/ - speakers.yml - autocorrect.yml - glossary.yml - session.yml - pipeline.resolved.yml - - audio/ - adam.flac - eric.flac - other-speaker.flac - - transcripts/ - raw/ - adam.json - eric.json - other-speaker.json - normalized/ - adam.json - eric.json - other-speaker.json - merged.json - processed.json - - artifacts/ - event-log.json - session-log.md - meta-analysis.md - table-read.md - - config/ - seriatim.generated.yml - audita.generated.yml - analyzer.event-log.generated.yml - analyzer.session-log.generated.yml - - logs/ - whisperx.adam.log - whisperx.eric.log - seriatim.stdout.log - seriatim.stderr.log - audita.stdout.log - audita.stderr.log - analyzer.session-log.stdout.log - analyzer.session-log.stderr.log - - manifest.json - .lock -``` - -The exact layout may evolve, but the distinction between inputs, audio, transcripts, artifacts, generated config, logs, and manifest should remain. - -#### Expected S3 Layout - -S3 should mirror the conceptual local layout. - -Example: - -```text -sessions/{session_id}/ - inputs/ - audio/ - transcripts/ - raw/ - normalized/ - merged.json - processed.json - artifacts/ - config/ - logs/ - manifest.json -``` - -The orchestrator should be able to upload outputs after each successful stage, not only at the end of the full workflow. - -## Suggested Go Package Layout - -Initial scaffold should use a package layout *similar* to: - -```text -cmd/narratio/ - main.go - -internal/app/ - app.go - plan.go - run.go - commands.go - -internal/config/ - config.go - load.go - validate.go - -internal/manifest/ - manifest.go - store.go - status.go - -internal/stage/ - stage.go - prepare.go - transcribe.go - normalize.go - merge.go - polish.go - analyze.go - archive.go - notify.go - -internal/adapters/ - whisperx/ - client.go - seriatim/ - runner.go - audita/ - runner.go - analyzer/ - runner.go - notify/ - sender.go - -internal/artifacts/ - store.go - local.go - s3.go - paths.go - checksum.go - -internal/contracts/ - transcript.go - session.go - artifact.go - -internal/logging/ - logging.go -``` - -The initial build step should create these packages and core interfaces, but should not fully implement all real external behavior. - -## CLI Shape - -The CLI should eventually support commands like: - -```bash -dnd-orchestrator run --config pipeline.yml --session session.yml -dnd-orchestrator plan --config pipeline.yml --session session.yml -dnd-orchestrator status --session-id 2026-05-03 -dnd-orchestrator resume --session-id 2026-05-03 -dnd-orchestrator run-stage polish --session-id 2026-05-03 --force -``` - -For the initial scaffold, it is acceptable to implement only partial command behavior, but the command structure should anticipate these operations. - -``` -run -``` - -Loads config, validates inputs, creates or loads a manifest, builds a plan, and executes the pipeline. - -``` -plan -``` - -Prints the stages that would run and which stages would be skipped. - -``` -status -``` - -Reads a manifest and prints stage status. - -``` -resume -``` -Continues a previous run from the latest valid manifest state. - -``` -run-stage -``` - -Runs one stage explicitly, with optional --force. - -## Core Stage Interface - -The application should define a narrow stage interface. - -Example shape: - -```go -type Stage interface { - Name() string - Run(ctx context.Context, env *Env, manifest *manifest.Manifest) (*StageResult, error) -} -``` - -The exact types may vary, but the idea should remain: -- a stage has a stable name -- a stage runs with context -- a stage receives shared dependencies through an environment object -- a stage may inspect/update manifest state through controlled methods -- a stage returns declared outputs and metadata - -Suggested supporting type: +Still planned/future: -```go -type StageResult struct { - Outputs []artifacts.Ref - Metadata map[string]any -} -``` +- Real WhisperX HTTP adapter. +- Real Seriatim adapter. +- Real Audita adapter. +- Real analyzer integration. +- Real remote archive/storage backend (S3/SFTP/etc). +- Real notification backend. +- Stale detection based on input/config checksums. -## Application Environment +## 5. High-Level Pipeline -The application should pass shared dependencies through an explicit environment object rather than globals. +Canonical stage order is fixed in code: -Example shape: +1. `prepare` +2. `transcribe` +3. `normalize` +4. `merge` +5. `polish` +6. `analyze` +7. `archive` +8. `notify` -```go -type Env struct { - Config *config.Config - ArtifactStore artifacts.Store - Logger *slog.Logger +Execution status: - WhisperX whisperx.Client - Seriatim seriatim.Runner - Audita audita.Runner - Analyzer analyzer.Runner - Notifier notify.Sender -} -``` +- `prepare` is implemented with real local filesystem behavior. +- All other stages are placeholders that return metadata and optionally exercise fake adapters. -Avoid package-level mutable state. +## 6. CLI Commands -## Main Loop +`cmd/narratio/main.go` is intentionally thin and delegates to `internal/app`. -The main application loop should be a simple state machine. +Current command behavior: -Conceptual behavior: +- `run --config --session [--force]` + - Loads + validates config, ensures workdir, acquires lock, loads/creates manifest, executes full stage plan. + - Skips already-succeeded stages unless `--force`. +- `plan --config --session [--force]` + - Loads + validates config, ensures workdir, loads manifest if present, prints `run`/`skip` action per stage and totals. +- `resume --config --session [--force]` + - Resumes from first non-succeeded stage based on manifest (or full run when forced). +- `run-stage --config --session [--force] ` + - Executes exactly one named stage; unknown stage is an error. +- `status --manifest ` + - Loads manifest and prints session ID, updated timestamp, and stage statuses. -```text -load configuration -validate configuration -create/load manifest -acquire session lock -resolve input files -build execution plan -for each stage: - decide whether stage should run - mark stage running - run stage - validate outputs - archive outputs as appropriate - mark stage succeeded or failed -release lock -``` +Invalid usage returns non-zero with usage/error text. -The implementation should prefer clear control flow over a generic workflow engine. +## 7. Configuration -## Long-Running Stage Handling +Package: `internal/config` -`audita` can run for more than an hour. `seriatim` is very fast (typically less than a second). Each `whisperx` file transcription job can take anywhere from 5 to 30 minutes. +Files: -The orchestrator should therefore treat long-running stages as normal and expected. +- `pipeline.yml` +- `session.yml` -Long-running subprocess stages should: -- receive a generous configurable timeout -- use context.Context for cancellation -- capture stdout and stderr continuously -- write logs to the work directory -- update manifest status before and after execution -- optionally update heartbeat timestamps while running -- preserve temporary work directories on failure -- avoid treating partial output as success +Key behavior: -A stage should write to a temporary output path first, then promote to the final output path only after validation. +- Strict YAML decoding via `yaml.Decoder.KnownFields(true)`. +- Unknown fields are rejected. +- Combined resolved config type keeps source paths (`PipelinePath`, `SessionPath`) for provenance/errors. -Example: +Validation currently enforces: -```text -processed.json.tmp -processed.json -``` -## Idempotency and Resume Behavior +- `pipeline.workspace.root` is required. +- `session.session_id` is required. +- `session.inputs.speakers_file`, `autocorrect_file`, `glossary_file` are required. +- At least one audio source: `session.inputs.audio_dir` or non-empty `session.inputs.audio_files`. +- Configured timeout fields must parse as Go durations when present. -Every stage should be designed around the question: +Validation scope is intentionally conservative (no deep business defaults, no full schema semantics, no remote connectivity checks). -```text -If this stage runs twice, what happens? -``` +## 8. Manifest and Stage Status -Preferred behavior: -- If expected output exists and validates, skip the stage. -- If expected output exists but input/config checksum changed, mark stale. -- If `--force` is supplied, rerun and replace output atomically. -- If a previous run failed, preserve logs and temporary files for inspection. -- If rerunning, do not delete unrelated successful outputs. +Package: `internal/manifest` -The manifest should allow the application to resume after interruption without rerunning completed expensive stages. +Model includes: -## Locking +- `Manifest` +- `StageRecord` +- `ArtifactRecord` +- `InputRecord` +- `ErrorRecord` +- `StageStatus` -The orchestrator should prevent two processes from operating on the same session work directory at the same time. +Supported stage statuses: -For v1, a lock file in the work directory is acceptable: +- `pending` +- `running` +- `succeeded` +- `failed` +- `skipped` +- `stale` (defined but not actively produced by runner yet) +- `interrupted` (defined for future use) -```text -work/{session_id}/.lock -``` +Store behavior: -The lock should be acquired before mutating manifest or stage outputs. +- Local JSON store with create/load/save. +- Save is atomic (same-directory temp file + rename). +- `updated_at` refreshed on save. -## Logging +Helpers implemented: -The orchestrator should use structured logs, preferably Go's log/slog. +- `MarkStageRunning` +- `MarkStageSucceeded` +- `MarkStageFailed` +- `MarkStageSkipped` -Subprocess logs should be captured separately: +Runner currently records `running/succeeded/failed` transitions and preserves existing succeeded stage metadata when skipping. -```text -orchestrator structured log -stdout log per external tool invocation -stderr log per external tool invocation -``` +Manifest versioning note: -Do not combine all component logs into one global text file. +- Current manifest model includes optional `pipeline_version`, but does **not** include an explicit manifest schema/version field. +- Before manifests become long-term compatibility-sensitive across releases/storage backends, add explicit manifest schema/versioning and migration policy. -## External Component Boundaries -### WhisperX +## 9. Workdir and Artifact Model -The `whisperx` adapter should handle: +Package: `internal/artifacts` -- HTTP submission of audio files -- polling or waiting for completion, depending on API shape -- timeout and retry policy -- returning raw transcript JSON -- per-speaker parallelism with bounded concurrency +Canonical local paths are resolved by `SessionPaths` under: -The orchestrator should not assume WhisperX internals beyond the adapter contract. +`{workspace.root}/work/{session_id}/` -### Seriatim +Subpaths: -`seriatim` is a deterministic transcript merger. +- `inputs/` +- `audio/` +- `transcripts/raw/` +- `transcripts/normalized/` +- `artifacts/` +- `config/` +- `logs/` +- `manifest.json` +- `.lock` -The `seriatim` adapter should handle: +Implemented store capabilities: -- generating Seriatim config -- invoking the binary -- passing input transcript paths -- passing output path -- capturing logs -- returning merged transcript path +- Ensure layout. +- Resolve session paths. +- Copy inputs. +- Existence checks. +- SHA-256 checksums. +- Atomic file writes. +- Session lock acquire/release (`.lock`, conflict-safe via `O_EXCL`). -The orchestrator should not implement `seriatim` merge logic. +`Ref` supports local path info plus optional future remote key/checksum metadata. -### Audita +## 10. Stage Architecture -`audita` is an LLM-backed transcript polisher. +Package: `internal/stage` -The `audita` adapter should handle: +Core contracts: -- generating `audita` config -- invoking the binary or service -- passing merged transcript path -- passing glossary/autocorrect paths -- capturing logs -- preserving Audita work/checkpoint files -- returning processed transcript path +- `Stage` interface with `Name`, `Declares`, and `Run(ctx, env, manifest)`. +- `IODecl` for declared input/output artifact intent. +- `StageResult` for outputs, logs, generated configs, and metadata. -The orchestrator should not implement LLM correction logic. +Current `Declares` role: -### D&D Session Analyzer +- `Stage.Declares` is currently contract metadata; the runner does not yet enforce declared inputs/outputs at execution time. +- Real-stage implementation work should strengthen enforcement by validating declared inputs before execution and validating declared outputs after execution. -The analyzer is a future component. +### Current stage behaviors -The initial orchestrator should include interfaces/placeholders for it, but should not need a complete implementation. +- `prepare` (real): + - Ensures layout. + - Resolves session-relative input paths. + - Materializes required input files into `inputs/`. + - Writes `pipeline.resolved.yml`. + - Resolves/copies `.flac` audio inputs from `audio_dir` or `audio_files`. + - Computes checksums and records deterministic `manifest.inputs`. + - Uses checksum-aware write/copy reuse for idempotency. +- `transcribe`..`notify` (placeholder): + - Return placeholder metadata. + - Optionally call adapters using expected request/result contract shapes. -The analyzer adapter should eventually handle: +## 11. Application Environment and Adapters -- artifact type selection -- generated analyzer config -- prompt template references -- processed transcript path -- prior session context references -- output artifact paths -- structured validation where appropriate +`internal/app` uses `stage.Env` as the shared dependency container to avoid duplicate environment definitions. -The orchestrator should not contain prompt-specific D&D logic. The analyzer should own artifact-specific prompt behavior. +Contained dependencies: -## Contracts and Schema Versioning +- Resolved config +- Artifact store +- Manifest store +- Logger +- Adapter interfaces -Core data artifacts should include schema versions where practical. +Adapter boundaries (`internal/adapters/*`): -Important contracts include: +- `whisperx.Client` +- `seriatim.Runner` +- `audita.Runner` +- `analyzer.Runner` +- `storage.Backend` +- `notify.Sender` -- SpeakerTranscript (`whisperx` output) -- CanonicalTranscript (`seriatim` output) -- ProcessedTranscript (`autida` output) -- ArtifactResult -- SessionManifest +All adapters currently have fake/no-op implementations for tests/scaffold execution. Real integration TODOs are explicitly marked in adapter contract files. -The orchestrator may not need to deeply understand every field in every artifact, but it should validate enough to know that expected outputs were produced and are parseable. +### Subprocess helper -Schema versions should appear in JSON artifacts where possible: +`internal/adapters/subprocess` provides reusable subprocess scaffolding: -```json -{ - "schema": "processed_transcript.v1", - "session_id": "2026-05-03", - "segments": [] -} -``` - -### Validation Boundaries - -Validate at each boundary: -```text -input config is valid -audio files exist -speakers.yml maps expected filenames -WhisperX output exists and is parseable -normalized speaker transcripts exist -Seriatim merged transcript exists and is parseable -Audita processed transcript exists and is parseable -analyzer artifacts exist in expected formats -``` - -For the initial scaffold, validation methods may be stubs or simple parse/existence checks, but the architecture should make validation a first-class concern. - -## Testing Strategy - -The orchestrator should be testable without running `whisperx`, `seriatim`, `audita`, or LLMs. - -Use fake adapters for tests: -- fake `whisperx` returns canned transcript JSON -- fake `seriatim` writes a known merged transcript -- fake `audita` writes a known processed transcript -- fake `analyzer` writes fixed artifacts -- fake artifact store writes to a temp directory - -Tests should cover: -- config loading and validation -- plan generation -- manifest creation/update -- stage skip behavior -- force rerun behavior -- failure handling -- resume behavior -- subprocess adapter construction where practical - -### Testing Non-Goals -Tests must **not** do any of the following: -- include hard-coded assertions of specific default configuration values (although a test may validate that a default value is defined and has the correct type, it may not require it to equal any particualar value) - -## Initial Implementation Scope - -The first implementation pass should create the application framework and interfaces, not the full working pipeline. - -Implement: -``` -Go module setup -CLI skeleton -config structs and loader -manifest structs and local manifest store -artifact reference/store interfaces -stage interface -app environment object -plan/main-loop skeleton -placeholder stages -adapter interfaces -fake/no-op adapter implementations where helpful -basic structured logging -basic tests for config/manifest/stage planning -``` - -Do not yet implement: -``` -real WhisperX HTTP API behavior -real Seriatim subprocess execution -real Audita subprocess execution -real S3 upload/download -real email notification -full transcript schema validation -full analyzer behavior -D&D prompt logic -``` - -The initial code should compile, include clear TODOs, and establish durable package boundaries. - -## Engineering Preferences - -Use idiomatic Go. - -Prefer: -``` -context.Context -log/slog -explicit interfaces -small packages -table-driven tests -strict config decoding -clear errors with context -local filesystem abstractions where useful -standard library where sufficient -``` - -Avoid: -``` -global mutable state -reflection-heavy frameworks unless justified -premature generic DAG engines -hidden side effects in constructors -mixing orchestration logic with adapter-specific implementation details -writing business logic directly in main.go -``` - -### Architectural Invariant - -The most important invariant of the application is: - -Every stage consumes declared artifacts, produces declared artifacts, validates them, records provenance, and can be safely skipped or rerun. - -All design and implementation choices should support that invariant. \ No newline at end of file +- Context cancellation + optional timeout. +- Explicit executable/args, working dir, env overrides. +- Stdout/stderr log file handling. +- Exit code and timing capture. +- Actionable error wrapping. +- Atomic file/YAML writers for generated config/log scaffolding. + +## 12. Run Control, Locking, Logging, and Long-Running Stages + +### Run control + +`internal/app` runner is sequential and manifest-driven: + +- Loads/creates manifest. +- Computes per-stage action (`run` vs `skip`). +- Skip rule today: succeeded + not forced => skip. +- `--force` reruns previously succeeded stages. +- `resume` starts at first non-succeeded stage. + +Stale detection is intentionally TODO (`run_control.go`) pending checksum-based invalidation logic. + +### Locking + +`run`/`resume`/`run-stage` execution paths acquire a session lock via artifact store and release it with `defer`. + +### Logging + +- Structured logger is initialized via `internal/logging` (`slog` text handler). +- Runner emits concise stage lifecycle logs (skip/start/success/fail + manifest save points). +- Placeholder adapter requests include distinct stdout/stderr log paths to preserve future boundary design. + +### Long-running stage expectations + +The architecture expects long-running stages (especially Audita and WhisperX) and already provides context-based cancellation and subprocess scaffolding, but real long-running integrations are still pending. + +## 13. Testing Strategy + +Current tests verify scaffold behavior without real external services: + +- Strict config load/validate behavior and error quality. +- Manifest store round trips and transition semantics. +- Workdir layout, atomic writes, checksums, and lock behavior. +- Plan order and stage selection. +- Run/skip/force/resume/run-stage control behavior. +- Prepare-stage input materialization/provenance/idempotency. +- Adapter fake behavior and error propagation. +- Subprocess helper behavior (success/failure/timeout/log capture). + +Tests intentionally avoid hardcoding arbitrary operational default values. + +## 14. Future Work / Implementation Roadmap + +Recommended implementation sequence (one focused boundary at a time): + +1. Harden `transcribe` with real WhisperX adapter and transcript output validation. +2. Implement real `merge` using Seriatim adapter + generated config + subprocess logs. +3. Implement real `polish` using Audita adapter + checkpoint/log handling. +4. Implement real `analyze` adapter integration and artifact validation. +5. Implement real `archive` remote backend behavior. +6. Implement real `notify` backend. +7. Add checksum-based stale detection and stale status transitions. +8. Add selective parallelism where architecturally safe (`transcribe` fan-out). + +Each step must preserve existing package boundaries and manifest-based control flow. + +### Real Stage Implementation Checklist + +For each stage moved from placeholder to real behavior: + +- Implement and/or use the appropriate adapter behavior for that stage boundary. +- Generate and use config/log paths needed by the adapter/tool invocation. +- Validate expected stage inputs before execution. +- Write outputs atomically where practical (temp + promote/rename). +- Validate expected outputs after execution before marking success. +- Record outputs, logs, generated configs, and provenance metadata in manifest stage records. +- Preserve `skip`/`force`/`resume` semantics (do not regress run-control behavior). +- Add fake-adapter tests and at least one explicit failure-path test for stage behavior. + +## 15. Architectural Invariants + +The following invariants are mandatory for future changes: + +1. Stage orchestration logic lives in `internal/app`; stage business logic lives in `internal/stage`. +2. External tool details remain behind adapter interfaces. +3. Config loading is strict and validation-first. +4. Manifest is the source of truth for stage status and resume/skip decisions. +5. Stage outputs/provenance are recorded durably and deterministically. +6. Local workdir is the primary execution area; remote storage is an adapter concern. +7. Main entrypoint stays thin (`cmd/narratio` delegates to `internal/app`). +8. Pipeline behavior remains explicit and maintainable; no generic DAG engine introduction. +9. Every stage must consume declared artifacts, produce declared artifacts, validate them, record provenance, and remain safely skippable/rerunnable.