narratio/architecture.md

D&D Session Orchestrator Architecture

Purpose

This repository implements a Go-based orchestration application called narratio for processing recorded Dungeons & Dragons session audio into durable transcripts and downstream analysis artifacts.

The orchestrator coordinates an existing pipeline consisting of:

1. Per-speaker FLAC audio recordings from Mumble.
2. A self-hosted WhisperX HTTP transcription service.
3. seriatim, a deterministic transcript merger.
4. audita, an LLM-backed transcript polisher.
5. A future (not yet implemented, name subject to change) dnd-session-analyzer, which generates final D&D session artifacts such as session logs, event logs, meta-analysis, and table-read reports.
6. Long-term storage (which may be local filesystem, S3-compatible object storage, SFTP endpoint, or similar) for all inputs, intermediate outputs, logs, generated configs, and final artifacts.

The orchestrator is not responsible for implementing transcription, transcript merging, transcript polishing, or artifact generation. Its job is to coordinate those components reliably, maintain durable state, validate stage boundaries, manage local and remote artifacts, and make the pipeline resumable.

Design Goals

The application should be:

• Modular: Each stage and external component should be isolated behind a narrow interface.
• Composable: Stages should consume and produce declared artifacts.
• Resumable: Long-running workflows should be restartable without repeating completed work.
• Idempotent: Re-running the same pipeline should not corrupt or duplicate outputs.
• Observable: The orchestrator should produce structured logs, captured subprocess logs, and a durable run manifest.
• Config-driven: Most user-adjustable behavior should live in configuration files, not hardcoded logic.
• Strict at boundaries: Inputs and outputs should be validated at every major stage boundary.
• Boring and reliable: Prefer explicit, maintainable code over clever workflow abstractions.

The orchestrator should be written in Go.

Non-Goals

The orchestrator should not:

• Reimplement WhisperX.
• Reimplement seriatim.
• Reimplement audita.
• Contain D&D-specific prompt logic directly in the core orchestration layer.
• Parse or modify LLM outputs except for high-level validation of expected files or schemas.
• Become a general-purpose distributed workflow engine.
• Require a database for v1.
• Hide important run state only in logs.

High-Level Pipeline

The intended pipeline is:

prepare
  ↓
transcribe speaker tracks in parallel
  ↓
normalize speaker transcripts
  ↓
merge transcripts with seriatim
  ↓
polish transcript with audita
  ↓
generate D&D artifacts with dnd-session-analyzer
  ↓
archive outputs
  ↓
notify user

Some stages may be implemented in the initial scaffold only as interfaces or placeholders.

The initial implementation should create the application framework and contracts without implementing the full real behavior of all stages.

Primary Concepts

Configuration

Configuration is user-authored and describes how the pipeline should run.

There should be two primary configuration files:

pipeline.yml session.yml

pipeline.yml contains durable/default pipeline configuration:

workspace root
S3 bucket/prefix
WhisperX service settings
seriatim binary path and options
audita binary path and options
analyzer binary path and artifact settings
notification settings
concurrency and timeout defaults

session.yml contains per-session inputs and metadata:

session ID
campaign ID
session date/title
audio input directory or explicit audio files
speakers.yml
autocorrect.yml
glossary.yml
references to previous-session context

The application should strictly decode config files and reject unknown fields. Config validation should happen before any stage is run.

Environment variables may be used for secrets and deployment-specific credentials, but ordinary pipeline behavior should live in config files.

Manifest

The manifest is machine-authored durable run state.

Each session run should have a manifest.json in the local work directory and eventually mirrored to S3.

The manifest should track:

session ID
pipeline version
created/updated timestamps
resolved input files
checksums for important inputs/configs
stage statuses
stage timestamps
stage outputs
captured logs
generated config files
component versions where available
errors
artifact locations
S3 object keys

Stage statuses should be explicit:

pending
running
succeeded
failed
skipped
stale
interrupted

The manifest is the source of truth for resume/skip decisions.

Stages

A stage is a pipeline unit that consumes declared inputs and produces declared outputs.

Examples:

prepare
transcribe
normalize
merge
polish
analyze
archive
notify

Stages should be organized around the lifecycle of a session, not around executable names.

For example, the merge stage may call seriatim, but the stage should be named for the pipeline operation, not the tool.

Every stage should:

• Determine required inputs.
• Check whether expected outputs already exist.
• Decide whether to run, skip, or fail.
• Execute through an adapter or local logic.
• Validate expected outputs.
• Write outputs to the local work directory.
• Record stage status and metadata in the manifest.
• Allow safe rerun with --force.

Adapters

Adapters isolate interactions with external components.

The orchestrator should have thin adapters for:

• WhisperX HTTP API
• seriatim
• audita
• future dnd-session-analyzer
• storage layer (local filesystem, S3-compatible, SFTP, etc.)
• optional email/notification backend

Adapters should hide details such as:

• CLI argument construction
• generated config file format
• HTTP request/response details
• stdout/stderr capture
• process exit handling
• timeout and cancellation behavior

The stage layer should call adapter methods and should not know implementation details of the underlying external tool.

Artifact Store

The orchestrator should use a local work directory as the primary working area.

Each stage writes outputs locally first. The orchestrator then validates and archives those outputs to S3.

The storage layer should expose an abstraction for artifact references and common operations:

write local artifact read local artifact calculate checksum upload to S3 record local path and S3 key validate existence

For v1, local filesystem + S3 is enough. A database is not required.

Expected Local Work Directory Layout

Each session should have an isolated local work directory.

Example:

work/{session_id}/
  inputs/
    speakers.yml
    autocorrect.yml
    glossary.yml
    session.yml
    pipeline.resolved.yml

  audio/
    adam.flac
    eric.flac
    other-speaker.flac

  transcripts/
    raw/
      adam.json
      eric.json
      other-speaker.json
    normalized/
      adam.json
      eric.json
      other-speaker.json
    merged.json
    processed.json

  artifacts/
    event-log.json
    session-log.md
    meta-analysis.md
    table-read.md

  config/
    seriatim.generated.yml
    audita.generated.yml
    analyzer.event-log.generated.yml
    analyzer.session-log.generated.yml

  logs/
    whisperx.adam.log
    whisperx.eric.log
    seriatim.stdout.log
    seriatim.stderr.log
    audita.stdout.log
    audita.stderr.log
    analyzer.session-log.stdout.log
    analyzer.session-log.stderr.log

  manifest.json
  .lock

The exact layout may evolve, but the distinction between inputs, audio, transcripts, artifacts, generated config, logs, and manifest should remain.

Expected S3 Layout

S3 should mirror the conceptual local layout.

Example:

sessions/{session_id}/
  inputs/
  audio/
  transcripts/
    raw/
    normalized/
    merged.json
    processed.json
  artifacts/
  config/
  logs/
  manifest.json

The orchestrator should be able to upload outputs after each successful stage, not only at the end of the full workflow.

Suggested Go Package Layout

Initial scaffold should use a package layout similar to:

cmd/narratio/
  main.go

internal/app/
  app.go
  plan.go
  run.go
  commands.go

internal/config/
  config.go
  load.go
  validate.go

internal/manifest/
  manifest.go
  store.go
  status.go

internal/stage/
  stage.go
  prepare.go
  transcribe.go
  normalize.go
  merge.go
  polish.go
  analyze.go
  archive.go
  notify.go

internal/adapters/
  whisperx/
    client.go
  seriatim/
    runner.go
  audita/
    runner.go
  analyzer/
    runner.go
  notify/
    sender.go

internal/artifacts/
  store.go
  local.go
  s3.go
  paths.go
  checksum.go

internal/contracts/
  transcript.go
  session.go
  artifact.go

internal/logging/
  logging.go

The initial build step should create these packages and core interfaces, but should not fully implement all real external behavior.

CLI Shape

The CLI should eventually support commands like:

dnd-orchestrator run --config pipeline.yml --session session.yml
dnd-orchestrator plan --config pipeline.yml --session session.yml
dnd-orchestrator status --session-id 2026-05-03
dnd-orchestrator resume --session-id 2026-05-03
dnd-orchestrator run-stage polish --session-id 2026-05-03 --force

For the initial scaffold, it is acceptable to implement only partial command behavior, but the command structure should anticipate these operations.

run

Loads config, validates inputs, creates or loads a manifest, builds a plan, and executes the pipeline.

plan

Prints the stages that would run and which stages would be skipped.

status

Reads a manifest and prints stage status.

resume

Continues a previous run from the latest valid manifest state.

run-stage

Runs one stage explicitly, with optional --force.

Core Stage Interface

The application should define a narrow stage interface.

Example shape:

type Stage interface {
    Name() string
    Run(ctx context.Context, env *Env, manifest *manifest.Manifest) (*StageResult, error)
}

The exact types may vary, but the idea should remain:

• a stage has a stable name
• a stage runs with context
• a stage receives shared dependencies through an environment object
• a stage may inspect/update manifest state through controlled methods
• a stage returns declared outputs and metadata

Suggested supporting type:

type StageResult struct {
    Outputs  []artifacts.Ref
    Metadata map[string]any
}

Application Environment

The application should pass shared dependencies through an explicit environment object rather than globals.

Example shape:

type Env struct {
    Config        *config.Config
    ArtifactStore artifacts.Store
    Logger        *slog.Logger

    WhisperX      whisperx.Client
    Seriatim      seriatim.Runner
    Audita        audita.Runner
    Analyzer      analyzer.Runner
    Notifier      notify.Sender
}

Avoid package-level mutable state.

Main Loop

The main application loop should be a simple state machine.

Conceptual behavior:

load configuration
validate configuration
create/load manifest
acquire session lock
resolve input files
build execution plan
for each stage:
  decide whether stage should run
  mark stage running
  run stage
  validate outputs
  archive outputs as appropriate
  mark stage succeeded or failed
release lock

The implementation should prefer clear control flow over a generic workflow engine.

Long-Running Stage Handling

audita can run for more than an hour. seriatim is very fast (typically less than a second). Each whisperx file transcription job can take anywhere from 5 to 30 minutes.

The orchestrator should therefore treat long-running stages as normal and expected.

Long-running subprocess stages should:

• receive a generous configurable timeout
• use context.Context for cancellation
• capture stdout and stderr continuously
• write logs to the work directory
• update manifest status before and after execution
• optionally update heartbeat timestamps while running
• preserve temporary work directories on failure
• avoid treating partial output as success

A stage should write to a temporary output path first, then promote to the final output path only after validation.

Example:

processed.json.tmp
processed.json

Idempotency and Resume Behavior

Every stage should be designed around the question:

If this stage runs twice, what happens?

Preferred behavior:

• If expected output exists and validates, skip the stage.
• If expected output exists but input/config checksum changed, mark stale.
• If --force is supplied, rerun and replace output atomically.
• If a previous run failed, preserve logs and temporary files for inspection.
• If rerunning, do not delete unrelated successful outputs.

The manifest should allow the application to resume after interruption without rerunning completed expensive stages.

Locking

The orchestrator should prevent two processes from operating on the same session work directory at the same time.

For v1, a lock file in the work directory is acceptable:

work/{session_id}/.lock

The lock should be acquired before mutating manifest or stage outputs.

Logging

The orchestrator should use structured logs, preferably Go's log/slog.

Subprocess logs should be captured separately:

orchestrator structured log
stdout log per external tool invocation
stderr log per external tool invocation

Do not combine all component logs into one global text file.

External Component Boundaries

WhisperX

The whisperx adapter should handle:

• HTTP submission of audio files
• polling or waiting for completion, depending on API shape
• timeout and retry policy
• returning raw transcript JSON
• per-speaker parallelism with bounded concurrency

The orchestrator should not assume WhisperX internals beyond the adapter contract.

Seriatim

seriatim is a deterministic transcript merger.

The seriatim adapter should handle:

• generating Seriatim config
• invoking the binary
• passing input transcript paths
• passing output path
• capturing logs
• returning merged transcript path

The orchestrator should not implement seriatim merge logic.

Audita

audita is an LLM-backed transcript polisher.

The audita adapter should handle:

• generating audita config
• invoking the binary or service
• passing merged transcript path
• passing glossary/autocorrect paths
• capturing logs
• preserving Audita work/checkpoint files
• returning processed transcript path

The orchestrator should not implement LLM correction logic.

D&D Session Analyzer

The analyzer is a future component.

The initial orchestrator should include interfaces/placeholders for it, but should not need a complete implementation.

The analyzer adapter should eventually handle:

• artifact type selection
• generated analyzer config
• prompt template references
• processed transcript path
• prior session context references
• output artifact paths
• structured validation where appropriate

The orchestrator should not contain prompt-specific D&D logic. The analyzer should own artifact-specific prompt behavior.

Contracts and Schema Versioning

Core data artifacts should include schema versions where practical.

Important contracts include:

• SpeakerTranscript (whisperx output)
• CanonicalTranscript (seriatim output)
• ProcessedTranscript (autida output)
• ArtifactResult
• SessionManifest

The orchestrator may not need to deeply understand every field in every artifact, but it should validate enough to know that expected outputs were produced and are parseable.

Schema versions should appear in JSON artifacts where possible:

{
  "schema": "processed_transcript.v1",
  "session_id": "2026-05-03",
  "segments": []
}

Validation Boundaries

Validate at each boundary:

input config is valid
audio files exist
speakers.yml maps expected filenames
WhisperX output exists and is parseable
normalized speaker transcripts exist
Seriatim merged transcript exists and is parseable
Audita processed transcript exists and is parseable
analyzer artifacts exist in expected formats

For the initial scaffold, validation methods may be stubs or simple parse/existence checks, but the architecture should make validation a first-class concern.

Testing Strategy

The orchestrator should be testable without running whisperx, seriatim, audita, or LLMs.

Use fake adapters for tests:

• fake whisperx returns canned transcript JSON
• fake seriatim writes a known merged transcript
• fake audita writes a known processed transcript
• fake analyzer writes fixed artifacts
• fake artifact store writes to a temp directory

Tests should cover:

• config loading and validation
• plan generation
• manifest creation/update
• stage skip behavior
• force rerun behavior
• failure handling
• resume behavior
• subprocess adapter construction where practical

Testing Non-Goals

Tests must not do any of the following:

• include hard-coded assertions of specific default configuration values (although a test may validate that a default value is defined and has the correct type, it may not require it to equal any particualar value)

Initial Implementation Scope

The first implementation pass should create the application framework and interfaces, not the full working pipeline.

Implement:

Go module setup
CLI skeleton
config structs and loader
manifest structs and local manifest store
artifact reference/store interfaces
stage interface
app environment object
plan/main-loop skeleton
placeholder stages
adapter interfaces
fake/no-op adapter implementations where helpful
basic structured logging
basic tests for config/manifest/stage planning

Do not yet implement:

real WhisperX HTTP API behavior
real Seriatim subprocess execution
real Audita subprocess execution
real S3 upload/download
real email notification
full transcript schema validation
full analyzer behavior
D&D prompt logic

The initial code should compile, include clear TODOs, and establish durable package boundaries.

Engineering Preferences

Use idiomatic Go.

Prefer:

context.Context
log/slog
explicit interfaces
small packages
table-driven tests
strict config decoding
clear errors with context
local filesystem abstractions where useful
standard library where sufficient

Avoid:

global mutable state
reflection-heavy frameworks unless justified
premature generic DAG engines
hidden side effects in constructors
mixing orchestration logic with adapter-specific implementation details
writing business logic directly in main.go

Architectural Invariant

The most important invariant of the application is:

Every stage consumes declared artifacts, produces declared artifacts, validates them, records provenance, and can be safely skipped or rerun.

All design and implementation choices should support that invariant.