narratio/README.md

narratio

narratio is a Go orchestration application for processing D&D session audio into transcripts and generated artifacts.

Current Implementation

Implemented now:

• strict config loading/validation (pipeline.yml and session.yml)
• local workspace/session layout, locking, and manifest persistence
• resumable stage control (run, plan, resume, run-stage, status)
• real prepare, transcribe, merge, polish, normalize, trim, and analyze stages
• real WhisperX, Seriatim, and Audita adapters
• real Scriptorium subprocess adapter
• optional Scriptorium render diagnostics (render_debug)

Not implemented yet:

• notify stage behavior
• additional analyze artifacts beyond session_recap
• generic DAG orchestration

Config Files

Narratio expects two YAML files:

• pipeline.yml: pipeline/workspace settings
• session.yml: per-session settings

Pipeline config lookup for CLI commands:

• if --config <path> is provided, Narratio uses that path
• if --config is omitted, Narratio searches in this order:
  • /usr/local/etc/narratio/pipeline.yml
  • /etc/narratio/pipeline.yml

Session config lookup for CLI commands:

• if --session <path> is provided, Narratio uses that path
• if --session is omitted, Narratio searches in this order:
  • ./session.yml
  • /usr/local/etc/narratio/session.yml
  • /etc/narratio/session.yml

Session template support:

• Narratio renders session.yml templates before strict YAML decode.
• --session-id <value> provides the session_id template variable.
• Supported placeholder forms:
  • {{session_id}}
  • {{ session_id }}
• unresolved template placeholders fail with a clear error.
• strict YAML validation still runs after rendering.
• concrete session.yml files without templates remain fully supported.

Optional secrets-from-files config:

• pipeline.secrets.env_dir may point to a directory of secret files
• each top-level file with an env-var-style name is loaded as an environment variable:
  • file name = env var name
  • file contents = env var value (trailing newline/CRLF trimmed)
• process environment wins: existing env vars are not overwritten
• if configured, Narratio fails fast when env_dir is missing/unreadable
• relative env_dir values resolve from Narratio’s current working directory

YAML decoding is strict (KnownFields(true)), so unknown fields fail fast.

Storage And Archive Foundations

Narratio now includes configuration and path-model foundations for archive support, plus implemented prepare-stage S3 audio input.

Implemented foundations:

• pipeline.storage.s3 config shape (bucket, root_prefix, region, endpoint, force_path_style)
• pipeline.spool config shape (root, delete_audio_after_archive)
• pipeline.archive config shape (enabled, upload_run, promote_artifacts)
• promotion-rule validation (from/to required, relative-only paths, traversal rejected)
• session.campaign requirement for campaign-aware path construction
• optional session.inputs.audio_s3.prefix modeling and prepare-stage S3 audio download
• run ID generation and S3/local path helper foundations
• manifest run/path identity fields

Current defaults:

• pipeline.storage.s3.root_prefix: dnd
• pipeline.workspace.cleanup_after_archive: false
• pipeline.spool.root: /var/spool/narratio
• pipeline.spool.delete_audio_after_archive: false
• pipeline.archive.enabled: true
• pipeline.archive.upload_run: true
• default pipeline.archive.promote_artifacts:
  • transcripts/trimmed.json -> transcripts/trimmed.json (required: true)
  • artifacts/session_recap.md -> artifacts/session_recap.md (required: true)

Current boundaries:

• local development audio (audio_dir / audio_files) still works
• audio_dir/audio_files and audio_s3 are mutually exclusive
• real S3-compatible backend now exists in the storage adapter package
• storage backend tests use fake storage and do not require live S3
• archive uploads successful run records under runs/{run_id}/
• archive does not upload local audio by default
• archive uploads promoted outputs to session-level keys using archive.promote_artifacts
• archive uploads current/manifest.json
• archive uploads current/run_id.txt last as the effective commit marker
• required missing promotions fail archive
• optional missing promotions are skipped and recorded
• cleanup remains conservative and opt-in:
  • pipeline.spool.delete_audio_after_archive: true removes only the run-scoped spool audio directory after successful archive commit
  • pipeline.workspace.cleanup_after_archive: true removes only the run-scoped local workdir after successful archive commit
  • cleanup executes only after all selected stages for the command invocation succeed
  • cleanup does not run for failed, incomplete, skipped, or unarchived runs
  • local development audio_dir/audio_files source inputs are never deleted by spool cleanup

S3 input details and current boundaries are documented in docs/s3-audio-input.md.

Remote Storage Backend

Narratio includes an object-store backend layer for future prepare/archive work:

• List(ctx, prefix)
• Download(ctx, key, localPath)
• Upload(ctx, localPath, key, opts)
• Exists(ctx, key)

Implemented backends:

• fake storage backend for deterministic tests
• S3-compatible backend built from pipeline.storage.s3

Key invariant:

• callers pass full bucket-relative object keys
• storage backends do not prepend root_prefix and do not infer session/campaign paths

Current boundary:

• prepare uses List + Download through the backend when session.inputs.audio_s3 is configured
• archive uses Upload through the backend for successful run-record uploads
• archive also uses Upload for promotion writes and current pointers
• no failed or incomplete runs are uploaded
• local audio is not re-uploaded by default

Archive run-upload details and boundaries are documented in docs/archive-storage.md.

Canonical Stage Order

1. prepare
2. transcribe
3. merge
4. polish
5. normalize
6. trim
7. analyze
8. archive
9. notify

Transcript Tiers

• transcripts/merged.json: canonical deterministic merged transcript from Seriatim merge
• transcripts/processed.json: full raw Audita-polished transcript output
• transcripts/normalized.json: Seriatim-normalized transcript from the normalize stage
• transcripts/trimmed.json: gameplay-only normalized polished transcript from trim stage

Seriatim Configuration

pipeline.seriatim configures the Seriatim subprocess adapter used by merge, normalize, and trim.

Minimal behavior:

• pipeline.seriatim may be omitted entirely.
• when omitted, Narratio defaults to:
  • binary: seriatim
  • timeout: 10m
  • output_schema: seriatim-intermediate
  • coalesce_gap: 3.0
  • report: true

Optional overrides in pipeline.seriatim continue to work, including explicit binary paths and advanced env tuning values.

Audita Configuration

pipeline.audita configures the real Audita subprocess adapter used by polish.

Minimal behavior:

• pipeline.audita may be omitted entirely.
• when omitted, Narratio defaults to:
  • binary: audita
  • timeout: 3h
  • report: true

Optional:

• llm_api_key_env (when set, Narratio requires that env var and passes it to Audita as AUDITA_LLM_API_KEY)
• modules override list (when empty/omitted, Narratio does not pass --modules)
• base_url (when omitted, Narratio does not pass --base-url; Audita runtime defaults/config may apply)
• model (when omitted, Narratio does not pass --model; Audita runtime defaults/config may apply)
• transcript_description
• config_path
• output_schema (bare-segments or audita-v1)
• work_dir_retention (always, auto, or never)
• total_llm_concurrency (> 0 when provided)
• proposal_llm_concurrency (> 0 when provided)
• validation_model
• validation_llm_concurrency (> 0 when provided)
• report (defaults to true)

Narratio passes only configured optional Audita flags. Omitted optional values are left to Audita runtime defaults/config.

Normalize Configuration

pipeline.normalize is optional. When omitted, Narratio defaults to:

• output_path: transcripts/normalized.json
• output_schema: seriatim-intermediate
• report: true

Allowed normalize.output_schema values:

• seriatim-minimal
• seriatim-intermediate
• seriatim-full

normalize.output_path is treated as session-workdir-relative when not absolute.

Normalize stage behavior summary:

• normalize runs after polish and before trim
• normalize resolves transcripts/processed.json
• normalize runs Seriatim normalize to produce transcripts/normalized.json
• normalize diagnostics are written to:
  • artifacts/seriatim.normalize.report.json (when enabled)
  • logs/seriatim.normalize.stdout.log
  • logs/seriatim.normalize.stderr.log
  • config/seriatim.normalize.generated.yml

Trim Configuration

pipeline.trim is optional. If omitted, no trim config is loaded. If trim.enabled is omitted, it defaults to false.

When trim.enabled: true:

• trim.output_path is required
• trim.bounds.prompt_id is required
• trim.bounds.transcript_input_name is required
• trim.bounds.output_path is required
• trim.bounds.timeout must be a valid Go duration when provided
• trim.bounds.render_debug: true requires trim.bounds.render_output_path
• trim.bounds.profile_id may be empty to use the prompt default profile

Trim paths are treated as session-workdir-relative when not absolute.

Example trim config:

trim:
  enabled: true
  output_path: "transcripts/trimmed.json"
  bounds:
    prompt_id: "dnd_session.bounds"
    profile_id: ""
    transcript_input_name: "transcript"
    output_path: "artifacts/session_bounds.json"
    timeout: "10m"
    render_debug: false
    render_output_path: "artifacts/session_bounds.render.json"
  seriatim:
    report: false

Trim behavior summary:

• trim discovers and validates transcripts/normalized.json
• trim uses Scriptorium bounds (dnd_session.bounds by example config) to produce artifacts/session_bounds.json
• bounds IDs are validated against the same normalized transcript ID space that Seriatim trim will consume
• trim converts bounds to Seriatim keep selector (for example 10-868) and runs Seriatim trim
• if trim is disabled, Narratio copies normalized transcript to trimmed transcript and records trim_action=copy_disabled

Trim outputs and diagnostics:

• artifacts/session_bounds.json
• transcripts/trimmed.json
• logs/scriptorium.bounds.stdout.log
• logs/scriptorium.bounds.stderr.log
• config/scriptorium.bounds.generated.yml
• logs/seriatim.trim.stdout.log
• logs/seriatim.trim.stderr.log
• config/seriatim.trim.generated.yml
• optional bounds render-debug outputs:
  • artifacts/session_bounds.render.json
  • logs/scriptorium.bounds.render.stdout.log
  • logs/scriptorium.bounds.render.stderr.log
  • config/scriptorium.bounds.render.generated.yml

Render-debug files are diagnostics and are not treated as canonical stage output artifact refs.

Scriptorium Configuration

pipeline.scriptorium is optional. When present, Narratio validates and uses it for analyze-stage artifact generation.

Key points:

• scriptorium.binary defaults to scriptorium when section is present
• scriptorium.config_path is optional
• scriptorium.timeout defaults to 10m when omitted
• scriptorium.render_debug enables render diagnostics globally
• artifacts are configured under scriptorium.artifacts (map shape supports multiple artifacts)
• enabled artifacts require prompt_id and output_path
• artifact render_debug may override global render setting
• vars currently support boolean and string values

Example session_recap artifact definition:

scriptorium:
  binary: "scriptorium"
  config_path: "/etc/scriptorium/config.yml"
  timeout: "10m"
  render_debug: false

  artifacts:
    session_recap:
      enabled: true
      prompt_id: "dnd.session_recap"
      profile_id: "local-quality" # optional
      output_path: "artifacts/session_recap.md"
      timeout: "10m"
      # render_debug: true # optional per-artifact override

      inputs:
        transcript:
          source: "trimmed_transcript"
          required: true

        previous_recap:
          source: "previous_session_artifact"
          artifact: "session_recap"
          path: "" # optional; set when available
          required: false

      vars:
        session_id: true
        session_date: true
        campaign_name: true
        previous_session_id: true
        output_kind: "session_recap"

Prompt IDs and profile IDs are configuration values. They are not hardcoded in analyze-stage logic.

Do not put secrets in pipeline.yml. If API-key behavior is configured, use env var names only.

If pipeline.secrets.env_dir is configured, keep only references and secret files there; secret values are still not written to manifests, generated configs, or Narratio-managed logs.

Scriptorium Runtime Behavior

Narratio integrates with Scriptorium through the public CLI subprocess contract:

• generation: scriptorium run
• diagnostics/testing: scriptorium render --format json when render_debug is enabled

For the initial implementation, only session_recap generation is supported.

Analyze-stage session recap behavior:

• available transcript input sources for configured artifacts: processed_transcript, normalized_transcript, trimmed_transcript
• session recap should use gameplay-only transcript input (source: trimmed_transcript)
• Narratio resolves trimmed_transcript from trim manifest output (transcript_trimmed) or fallback transcripts/trimmed.json
• Narratio resolves normalized_transcript from normalize manifest output (transcript_normalized) or fallback transcripts/normalized.json
• missing trimmed transcript fails clearly and advises running trim stage first
• normalized_transcript is the preferred full-transcript source for future table/meta-analysis artifacts
• processed_transcript remains supported for advanced/debug use cases
• optionally includes previous_recap when configured and resolvable
• omits optional previous recap when unavailable
• fails if required inputs are missing
• validates output file exists and is non-empty

Expected session output paths:

• artifacts/session_recap.md
• logs/scriptorium.session_recap.stdout.log
• logs/scriptorium.session_recap.stderr.log
• config/scriptorium.session_recap.generated.yml
• artifacts/session_recap.render.json when render diagnostics are enabled

Examples

Starter files:

• examples/pipeline.minimal.yml
• examples/pipeline.audita-overrides.yml
• examples/session.minimal.yml
• examples/session.template.yml
• examples/speakers.yml

Commands

Run tests:

go test ./...

Plan a run:

go run ./cmd/narratio plan --session examples/session.minimal.yml

Use --config <path> to override default pipeline lookup when needed.

Run with a discoverable session template:

go run ./cmd/narratio run --session-id 2026-04-04

Run full pipeline:

go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml

Run analyze only:

go run ./cmd/narratio run-stage --config examples/pipeline.minimal.yml --session examples/session.minimal.yml analyze

Resume with a template session ID:

go run ./cmd/narratio resume --config examples/pipeline.minimal.yml --session examples/session.template.yml --session-id 2026-04-04

Operational Note

Checksum-based stale detection is not implemented yet.

If prepared inputs or prompt/runtime config change, rerun the appropriate upstream stages before relying on downstream artifacts.

Examples:

• glossary/autocorrect/speaker-context changes: rerun at least merge, polish, normalize, trim, and analyze
• trim bounds prompt/profile/config changes: rerun at least normalize, trim, and analyze
• session recap prompt/profile/input-source changes: rerun analyze

Roadmap

Near-term roadmap:

• extend analyze to additional configured artifacts
• support workflows where later artifacts consume earlier generated artifacts
• keep orchestration explicit without a generic DAG engine
• implement archive and notify backends