narratio/README.md

narratio

narratio is a Go orchestration application for processing D&D session audio into transcripts and generated artifacts.

Current Implementation

Implemented now:

• strict config loading/validation (pipeline.yml and session.yml)
• local workspace/session layout, locking, and manifest persistence
• resumable stage control (run, plan, resume, run-stage, status)
• real prepare, transcribe, merge, polish, normalize, trim, and analyze stages
• real WhisperX, Seriatim, and Audita adapters
• real Scriptorium subprocess adapter
• optional Scriptorium render diagnostics (render_debug)

Not implemented yet:

• notify stage behavior
• additional analyze artifacts beyond session_recap
• generic DAG orchestration

Config Files

Narratio expects two YAML files:

• pipeline.yml: pipeline/workspace settings
• session.yml: per-session settings

Pipeline config lookup for CLI commands:

• if --config <path> is provided, Narratio uses that path
• if --config is omitted, Narratio searches in this order:
  • /usr/local/etc/narratio/pipeline.yml
  • /etc/narratio/pipeline.yml

Optional secrets-from-files config:

• pipeline.secrets.env_dir may point to a directory of secret files
• each top-level file with an env-var-style name is loaded as an environment variable:
  • file name = env var name
  • file contents = env var value (trailing newline/CRLF trimmed)
• process environment wins: existing env vars are not overwritten
• if configured, Narratio fails fast when env_dir is missing/unreadable
• relative env_dir values resolve from Narratio’s current working directory

YAML decoding is strict (KnownFields(true)), so unknown fields fail fast.

Storage And Archive Foundations

Narratio now includes configuration and path-model foundations for archive support, plus implemented prepare-stage S3 audio input.

Implemented foundations:

• pipeline.storage.s3 config shape (bucket, root_prefix, region, endpoint, force_path_style)
• pipeline.spool config shape (root, delete_audio_after_archive)
• pipeline.archive config shape (enabled, upload_run, promote_artifacts)
• promotion-rule validation (from/to required, relative-only paths, traversal rejected)
• session.campaign requirement for campaign-aware path construction
• optional session.inputs.audio_s3.prefix modeling and prepare-stage S3 audio download
• run ID generation and S3/local path helper foundations
• manifest run/path identity fields

Current defaults:

• pipeline.storage.s3.root_prefix: dnd
• pipeline.spool.root: /var/spool/narratio
• pipeline.spool.delete_audio_after_archive: false
• pipeline.archive.enabled: true
• pipeline.archive.upload_run: true
• default pipeline.archive.promote_artifacts:
  • transcripts/trimmed.json -> transcripts/trimmed.json (required: true)
  • artifacts/session_recap.md -> artifacts/session_recap.md (required: true)

Current boundaries:

• local development audio (audio_dir / audio_files) still works
• audio_dir/audio_files and audio_s3 are mutually exclusive
• real S3-compatible backend now exists in the storage adapter package
• storage backend tests use fake storage and do not require live S3
• archive uploads successful run records under runs/{run_id}/
• archive does not upload local audio by default
• archive does not yet perform promotion writes
• no current/manifest.json or current/run_id.txt uploads yet

S3 input details and current boundaries are documented in docs/s3-audio-input.md.

Remote Storage Backend

Narratio includes an object-store backend layer for future prepare/archive work:

• List(ctx, prefix)
• Download(ctx, key, localPath)
• Upload(ctx, localPath, key, opts)
• Exists(ctx, key)

Implemented backends:

• fake storage backend for deterministic tests
• S3-compatible backend built from pipeline.storage.s3

Key invariant:

• callers pass full bucket-relative object keys
• storage backends do not prepend root_prefix and do not infer session/campaign paths

Current boundary:

• prepare uses List + Download through the backend when session.inputs.audio_s3 is configured
• archive uses Upload through the backend for successful run-record uploads
• promotion uploads and current-pointer writes are still not implemented

Archive run-upload details and boundaries are documented in docs/archive-storage.md.

Canonical Stage Order

1. prepare
2. transcribe
3. merge
4. polish
5. normalize
6. trim
7. analyze
8. archive
9. notify

Transcript Tiers

• transcripts/merged.json: canonical deterministic merged transcript from Seriatim merge
• transcripts/processed.json: full raw Audita-polished transcript output
• transcripts/normalized.json: Seriatim-normalized transcript from the normalize stage
• transcripts/trimmed.json: gameplay-only normalized polished transcript from trim stage

Audita Configuration

pipeline.audita configures the real Audita subprocess adapter used by polish.

Required:

• binary
• timeout
• base_url
• model

Optional:

• llm_api_key_env (when set, Narratio requires that env var and passes it to Audita as AUDITA_LLM_API_KEY)
• modules override list (when empty/omitted, Narratio does not pass --modules)
• transcript_description
• config_path
• output_schema (bare-segments or audita-v1)
• work_dir_retention (always, auto, or never)
• total_llm_concurrency (> 0 when provided)
• proposal_llm_concurrency (> 0 when provided)
• validation_model
• validation_llm_concurrency (> 0 when provided)
• report (defaults to true)

Narratio passes only configured optional Audita flags. Omitted optional values are left to Audita runtime defaults.

Normalize Configuration

pipeline.normalize is optional. When omitted, Narratio defaults to:

• output_path: transcripts/normalized.json
• output_schema: seriatim-intermediate
• report: true

Allowed normalize.output_schema values:

• seriatim-minimal
• seriatim-intermediate
• seriatim-full

normalize.output_path is treated as session-workdir-relative when not absolute.

Normalize stage behavior summary:

• normalize runs after polish and before trim
• normalize resolves transcripts/processed.json
• normalize runs Seriatim normalize to produce transcripts/normalized.json
• normalize diagnostics are written to:
  • artifacts/seriatim.normalize.report.json (when enabled)
  • logs/seriatim.normalize.stdout.log
  • logs/seriatim.normalize.stderr.log
  • config/seriatim.normalize.generated.yml

Trim Configuration

pipeline.trim is optional. If omitted, no trim config is loaded. If trim.enabled is omitted, it defaults to false.

When trim.enabled: true:

• trim.output_path is required
• trim.bounds.prompt_id is required
• trim.bounds.transcript_input_name is required
• trim.bounds.output_path is required
• trim.bounds.timeout must be a valid Go duration when provided
• trim.bounds.render_debug: true requires trim.bounds.render_output_path
• trim.bounds.profile_id may be empty to use the prompt default profile

Trim paths are treated as session-workdir-relative when not absolute.

Example trim config:

trim:
  enabled: true
  output_path: "transcripts/trimmed.json"
  bounds:
    prompt_id: "dnd_session.bounds"
    profile_id: ""
    transcript_input_name: "transcript"
    output_path: "artifacts/session_bounds.json"
    timeout: "10m"
    render_debug: false
    render_output_path: "artifacts/session_bounds.render.json"
  seriatim:
    report: false

Trim behavior summary:

• trim discovers and validates transcripts/normalized.json
• trim uses Scriptorium bounds (dnd_session.bounds by example config) to produce artifacts/session_bounds.json
• bounds IDs are validated against the same normalized transcript ID space that Seriatim trim will consume
• trim converts bounds to Seriatim keep selector (for example 10-868) and runs Seriatim trim
• if trim is disabled, Narratio copies normalized transcript to trimmed transcript and records trim_action=copy_disabled

Trim outputs and diagnostics:

• artifacts/session_bounds.json
• transcripts/trimmed.json
• logs/scriptorium.bounds.stdout.log
• logs/scriptorium.bounds.stderr.log
• config/scriptorium.bounds.generated.yml
• logs/seriatim.trim.stdout.log
• logs/seriatim.trim.stderr.log
• config/seriatim.trim.generated.yml
• optional bounds render-debug outputs:
  • artifacts/session_bounds.render.json
  • logs/scriptorium.bounds.render.stdout.log
  • logs/scriptorium.bounds.render.stderr.log
  • config/scriptorium.bounds.render.generated.yml

Render-debug files are diagnostics and are not treated as canonical stage output artifact refs.

Scriptorium Configuration

pipeline.scriptorium is optional. When present, Narratio validates and uses it for analyze-stage artifact generation.

Key points:

• scriptorium.binary is required when section is present
• scriptorium.config_path is optional
• scriptorium.timeout defaults to 10m when omitted
• scriptorium.render_debug enables render diagnostics globally
• artifacts are configured under scriptorium.artifacts (map shape supports multiple artifacts)
• enabled artifacts require prompt_id and output_path
• artifact render_debug may override global render setting
• vars currently support boolean and string values

Example session_recap artifact definition:

scriptorium:
  binary: "scriptorium"
  config_path: "/etc/scriptorium/config.yml"
  timeout: "10m"
  render_debug: false

  artifacts:
    session_recap:
      enabled: true
      prompt_id: "dnd.session_recap"
      profile_id: "local-quality" # optional
      output_path: "artifacts/session_recap.md"
      timeout: "10m"
      # render_debug: true # optional per-artifact override

      inputs:
        transcript:
          source: "trimmed_transcript"
          required: true

        previous_recap:
          source: "previous_session_artifact"
          artifact: "session_recap"
          path: "" # optional; set when available
          required: false

      vars:
        session_id: true
        session_date: true
        campaign_name: true
        previous_session_id: true
        output_kind: "session_recap"

Prompt IDs and profile IDs are configuration values. They are not hardcoded in analyze-stage logic.

Do not put secrets in pipeline.yml. If API-key behavior is configured, use env var names only.

If pipeline.secrets.env_dir is configured, keep only references and secret files there; secret values are still not written to manifests, generated configs, or Narratio-managed logs.

Scriptorium Runtime Behavior

Narratio integrates with Scriptorium through the public CLI subprocess contract:

• generation: scriptorium run
• diagnostics/testing: scriptorium render --format json when render_debug is enabled

For the initial implementation, only session_recap generation is supported.

Analyze-stage session recap behavior:

• available transcript input sources for configured artifacts: processed_transcript, normalized_transcript, trimmed_transcript
• session recap should use gameplay-only transcript input (source: trimmed_transcript)
• Narratio resolves trimmed_transcript from trim manifest output (transcript_trimmed) or fallback transcripts/trimmed.json
• Narratio resolves normalized_transcript from normalize manifest output (transcript_normalized) or fallback transcripts/normalized.json
• missing trimmed transcript fails clearly and advises running trim stage first
• normalized_transcript is the preferred full-transcript source for future table/meta-analysis artifacts
• processed_transcript remains supported for advanced/debug use cases
• optionally includes previous_recap when configured and resolvable
• omits optional previous recap when unavailable
• fails if required inputs are missing
• validates output file exists and is non-empty

Expected session output paths:

• artifacts/session_recap.md
• logs/scriptorium.session_recap.stdout.log
• logs/scriptorium.session_recap.stderr.log
• config/scriptorium.session_recap.generated.yml
• artifacts/session_recap.render.json when render diagnostics are enabled

Examples

Starter files:

• examples/pipeline.minimal.yml
• examples/session.minimal.yml
• examples/speakers.yml

Commands

Run tests:

go test ./...

Plan a run:

go run ./cmd/narratio plan --session examples/session.minimal.yml

Use --config <path> to override default pipeline lookup when needed.

Run full pipeline:

go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml

Run analyze only:

go run ./cmd/narratio run-stage --config examples/pipeline.minimal.yml --session examples/session.minimal.yml analyze

Operational Note

Checksum-based stale detection is not implemented yet.

If prepared inputs or prompt/runtime config change, rerun the appropriate upstream stages before relying on downstream artifacts.

Examples:

• glossary/autocorrect/speaker-context changes: rerun at least merge, polish, normalize, trim, and analyze
• trim bounds prompt/profile/config changes: rerun at least normalize, trim, and analyze
• session recap prompt/profile/input-source changes: rerun analyze

Roadmap

Near-term roadmap:

• extend analyze to additional configured artifacts
• support workflows where later artifacts consume earlier generated artifacts
• keep orchestration explicit without a generic DAG engine
• implement archive and notify backends