narratio/architecture.md

Narratio Architecture

1. Purpose

narratio is a Go orchestrator for D&D session processing. It coordinates a stage-based pipeline from recorded audio through transcript and artifact generation, while preserving durable run state for skip/rerun/resume behavior.

The repository currently implements the orchestration scaffold, local state model, and stage framework, including a real WhisperX HTTP adapter and a real transcribe stage. It intentionally does not yet implement real Seriatim, Audita, analyzer, remote archive, or notification integrations.

2. Design Goals

• Modular boundaries between orchestration, stages, adapters, storage, config, and manifest.
• Config-driven behavior with strict decoding and conservative validation.
• Durable manifest-based control for resumability and idempotent reruns.
• Boring, explicit control flow (no workflow engine/DAG abstraction).
• Observable orchestration via structured logs plus per-tool log file paths.
• Safe extension points for incremental real-stage implementation.

3. Non-Goals

Current and near-term non-goals:

• Reimplementing WhisperX, Seriatim, Audita, or analyzer internals.
• Building a generic distributed workflow engine.
• Adding a DB dependency for v1.
• Embedding D&D prompt logic in orchestration core.
• Implementing checksum-based stale detection (planned, not implemented).

4. Current Implementation Status

Implemented now:

• CLI entrypoint and command dispatch (run, plan, status, resume, run-stage).
• Strict pipeline.yml/session.yml loading and validation.
• Local workdir and artifact-store abstraction with lock-file support.
• Durable local JSON manifest store with atomic writes.
• Stage interface, canonical stage ordering, and runner main loop.
• Real prepare stage (input resolution/materialization/provenance).
• Real transcribe stage (prepared-audio discovery, WhisperX adapter execution, JSON transcript validation, provenance metadata).
• Real WhisperX HTTP adapter implementation (multipart POST + retries + timeout + atomic output writes).
• Placeholder downstream stages (normalize..notify) with adapter contract calls.
• Adapter interfaces and fake/no-op implementations for all external boundaries.
• Reusable subprocess helper and generated YAML/config writing helper.
• Test coverage across config, manifest, artifacts, planning, runner control, and adapters.

Still planned/future:

• Real Seriatim adapter.
• Real Audita adapter.
• Real analyzer integration.
• Real remote archive/storage backend (S3/SFTP/etc).
• Real notification backend.
• Stale detection based on input/config checksums.

5. High-Level Pipeline

Canonical stage order is fixed in code:

1. prepare
2. transcribe
3. normalize
4. merge
5. polish
6. analyze
7. archive
8. notify

Execution status:

• prepare is implemented with real local filesystem behavior.
• transcribe is implemented and validates raw transcript JSON outputs.
• normalize/merge/polish/analyze/archive/notify remain placeholders.

6. CLI Commands

cmd/narratio/main.go is intentionally thin and delegates to internal/app.

Current command behavior:

• run --config <pipeline.yml> --session <session.yml> [--force]
  • Loads + validates config, ensures workdir, acquires lock, loads/creates manifest, executes full stage plan.
  • Skips already-succeeded stages unless --force.
• plan --config <pipeline.yml> --session <session.yml> [--force]
  • Loads + validates config, ensures workdir, loads manifest if present, prints run/skip action per stage and totals.
• resume --config <pipeline.yml> --session <session.yml> [--force]
  • Resumes from first non-succeeded stage based on manifest (or full run when forced).
• run-stage --config <pipeline.yml> --session <session.yml> [--force] <stage>
  • Executes exactly one named stage; unknown stage is an error.
• status --manifest <path>
  • Loads manifest and prints session ID, updated timestamp, and stage statuses.

Invalid usage returns non-zero with usage/error text.

7. Configuration

Package: internal/config

Files:

• pipeline.yml
• session.yml

Key behavior:

• Strict YAML decoding via yaml.Decoder.KnownFields(true).
• Unknown fields are rejected.
• Combined resolved config type keeps source paths (PipelinePath, SessionPath) for provenance/errors.
• WhisperX optional fields are defaulted during load for deterministic resolved config values.

WhisperX config keys:

• pipeline.whisperx.transcribe_url (required)
• pipeline.whisperx.language (default: en)
• pipeline.whisperx.timeout (default: 30m)
• pipeline.whisperx.retries (default: 3)
• pipeline.whisperx.retry_delay (default: 2s)
• pipeline.whisperx.concurrency (default: 2)

Seriatim config keys:

• pipeline.seriatim.binary (required)
• pipeline.seriatim.timeout (default: 10m)
• pipeline.seriatim.output_schema (default: seriatim-intermediate; allowed: seriatim-minimal|seriatim-intermediate|seriatim-full)
• pipeline.seriatim.coalesce_gap (default: 3.0, must be >= 0)
• pipeline.seriatim.report (default: true)
• optional tuning under pipeline.seriatim.env.* (must be > 0 when provided)

Validation currently enforces:

• pipeline.workspace.root is required.
• pipeline.whisperx.transcribe_url is required and must be a valid URL.
• pipeline.whisperx.timeout and pipeline.whisperx.retry_delay must parse as Go durations.
• pipeline.whisperx.retries must be >= 0.
• pipeline.whisperx.concurrency must be > 0.
• pipeline.seriatim.binary is required.
• pipeline.seriatim.timeout must parse as Go duration.
• pipeline.seriatim.output_schema must be one of seriatim-minimal|seriatim-intermediate|seriatim-full.
• pipeline.seriatim.coalesce_gap must be >= 0.
• optional pipeline.seriatim.env.* values must be > 0 when provided.
• session.session_id is required.
• session.inputs.speakers_file, autocorrect_file, glossary_file are required.
• At least one audio source: session.inputs.audio_dir or non-empty session.inputs.audio_files.
• Configured timeout fields must parse as Go durations when present.

Validation scope is intentionally conservative (no deep business defaults, no full schema semantics, no remote connectivity checks).

8. Manifest and Stage Status

Package: internal/manifest

Model includes:

• Manifest
• StageRecord
• ArtifactRecord
• InputRecord
• ErrorRecord
• StageStatus

Supported stage statuses:

• pending
• running
• succeeded
• failed
• skipped
• stale (defined but not actively produced by runner yet)
• interrupted (defined for future use)

Store behavior:

• Local JSON store with create/load/save.
• Save is atomic (same-directory temp file + rename).
• updated_at refreshed on save.

Helpers implemented:

• MarkStageRunning
• MarkStageSucceeded
• MarkStageFailed
• MarkStageSkipped

Runner currently records running/succeeded/failed transitions and preserves existing succeeded stage metadata when skipping.

Manifest versioning note:

• Current manifest model includes optional pipeline_version, but does not include an explicit manifest schema/version field.
• Before manifests become long-term compatibility-sensitive across releases/storage backends, add explicit manifest schema/versioning and migration policy.

9. Workdir and Artifact Model

Package: internal/artifacts

Canonical local paths are resolved by SessionPaths under:

{workspace.root}/work/{session_id}/

Subpaths:

• inputs/
• audio/
• transcripts/raw/
• transcripts/normalized/
• artifacts/
• config/
• logs/
• manifest.json
• .lock

Implemented store capabilities:

• Ensure layout.
• Resolve session paths.
• Copy inputs.
• Existence checks.
• SHA-256 checksums.
• Atomic file writes.
• Session lock acquire/release (.lock, conflict-safe via O_EXCL).

Ref supports local path info plus optional future remote key/checksum metadata.

10. Stage Architecture

Package: internal/stage

Core contracts:

• Stage interface with Name, Declares, and Run(ctx, env, manifest).
• IODecl for declared input/output artifact intent.
• StageResult for outputs, logs, generated configs, and metadata.

Current Declares role:

• Stage.Declares is currently contract metadata; the runner does not yet enforce declared inputs/outputs at execution time.
• Real-stage implementation work should strengthen enforcement by validating declared inputs before execution and validating declared outputs after execution.

Current stage behaviors

• prepare (real):
  • Ensures layout.
  • Resolves session-relative input paths.
  • Materializes required input files into inputs/.
  • Writes pipeline.resolved.yml.
  • Resolves/copies .flac audio inputs from audio_dir or audio_files.
  • Computes checksums and records deterministic manifest.inputs.
  • Uses checksum-aware write/copy reuse for idempotency.
• transcribe (real):
  • Discovers prepared .flac audio inputs from manifest.inputs (kind=audio) or work/.../audio fallback.
  • Derives per-speaker output files at transcripts/raw/{audio_basename}.json.
  • Calls whisperx.Client with bounded parallelism from pipeline.whisperx.concurrency.
  • Validates each output file exists and is valid JSON before stage success.
  • Records output refs plus stage metadata (audio_files_count, language/concurrency/retry settings, output paths, per-file attempts/status/duration/path).
• normalize..notify (placeholder):
  • Return placeholder metadata.
  • Optionally call adapters using expected request/result contract shapes.

11. Application Environment and Adapters

internal/app uses stage.Env as the shared dependency container to avoid duplicate environment definitions.

Contained dependencies:

• Resolved config
• Artifact store
• Manifest store
• Logger
• Adapter interfaces

Adapter boundaries (internal/adapters/*):

• whisperx.Client
• seriatim.Runner
• audita.Runner
• analyzer.Runner
• storage.Backend
• notify.Sender

All adapters currently have fake/no-op implementations for tests/scaffold execution. WhisperX additionally has a real HTTP adapter implementation under internal/adapters/whisperx/http.go.

Subprocess helper

internal/adapters/subprocess provides reusable subprocess scaffolding:

• Context cancellation + optional timeout.
• Explicit executable/args, working dir, env overrides.
• Stdout/stderr log file handling.
• Exit code and timing capture.
• Actionable error wrapping.
• Atomic file/YAML writers for generated config/log scaffolding.

12. Run Control, Locking, Logging, and Long-Running Stages

Run control

internal/app runner is sequential and manifest-driven:

• Loads/creates manifest.
• Computes per-stage action (run vs skip).
• Skip rule today: succeeded + not forced => skip.
• --force reruns previously succeeded stages.
• resume starts at first non-succeeded stage.

Stale detection is intentionally TODO (run_control.go) pending checksum-based invalidation logic.

Locking

run/resume/run-stage execution paths acquire a session lock via artifact store and release it with defer.

Logging

• Structured logger is initialized via internal/logging (slog text handler).
• Runner emits concise stage lifecycle logs (skip/start/success/fail + manifest save points).
• Placeholder subprocess adapter requests include distinct stdout/stderr log paths to preserve future boundary design.

Long-running stage expectations

The architecture expects long-running stages (especially WhisperX and Audita). WhisperX already uses context timeout/cancellation and retry logic inside its HTTP adapter; other long-running integrations are still pending.

13. Testing Strategy

Current tests verify scaffold behavior without real external services:

• Strict config load/validate behavior and error quality.
• Manifest store round trips and transition semantics.
• Workdir layout, atomic writes, checksums, and lock behavior.
• Plan order and stage selection.
• Run/skip/force/resume/run-stage control behavior.
• Prepare-stage input materialization/provenance/idempotency.
• Real transcribe-stage audio discovery/concurrency/failure handling/output validation/provenance.
• Adapter fake behavior and error propagation.
• WhisperX HTTP adapter behavior (request shape, retry policy, timeout/cancel, JSON validation, atomic writes).
• Subprocess helper behavior (success/failure/timeout/log capture).

Tests intentionally avoid hardcoding arbitrary operational default values.

14. Future Work / Implementation Roadmap

Recommended implementation sequence (one focused boundary at a time):

1. Implement real normalize transcript transformation/validation.
2. Implement real merge using Seriatim adapter + generated config + subprocess logs.
3. Implement real polish using Audita adapter + checkpoint/log handling.
4. Implement real analyze adapter integration and artifact validation.
5. Implement real archive remote backend behavior.
6. Implement real notify backend.
7. Add checksum-based stale detection and stale status transitions.
8. Add selective parallelism where architecturally safe (transcribe fan-out and/or downstream-safe boundaries).

Each step must preserve existing package boundaries and manifest-based control flow.

Real Stage Implementation Checklist

For each stage moved from placeholder to real behavior:

• Implement and/or use the appropriate adapter behavior for that stage boundary.
• Generate and use config/log paths needed by the adapter/tool invocation.
• Validate expected stage inputs before execution.
• Write outputs atomically where practical (temp + promote/rename).
• Validate expected outputs after execution before marking success.
• Record outputs, logs, generated configs, and provenance metadata in manifest stage records.
• Preserve skip/force/resume semantics (do not regress run-control behavior).
• Add fake-adapter tests and at least one explicit failure-path test for stage behavior.

15. Architectural Invariants

The following invariants are mandatory for future changes:

1. Stage orchestration logic lives in internal/app; stage business logic lives in internal/stage.
2. External tool details remain behind adapter interfaces.
3. Config loading is strict and validation-first.
4. Manifest is the source of truth for stage status and resume/skip decisions.
5. Stage outputs/provenance are recorded durably and deterministically.
6. Local workdir is the primary execution area; remote storage is an adapter concern.
7. Main entrypoint stays thin (cmd/narratio delegates to internal/app).
8. Pipeline behavior remains explicit and maintainable; no generic DAG engine introduction.
9. Every stage must consume declared artifacts, produce declared artifacts, validate them, record provenance, and remain safely skippable/rerunnable.