narratio/architecture.md

Narratio Architecture

1. Purpose

narratio is a Go orchestrator for D&D session processing. It runs a stage-based local pipeline from audio input through transcript processing and artifact generation, with manifest-based skip/force/resume behavior.

Narratio integrates with Scriptorium through the public CLI (scriptorium run and scriptorium render) via synchronous subprocess execution.

2. Current Status

Implemented:

• strict pipeline.yml + session.yml loading with strict YAML field checking (KnownFields(true))
• local workspace/session layout, lock file handling, artifact path helpers, checksums, and atomic writes
• manifest store and stage status transitions for resumable runs
• real prepare, transcribe, merge, and polish stages
• real WhisperX HTTP adapter
• real Seriatim subprocess adapter
• real Audita subprocess adapter
• real Scriptorium subprocess adapter
• real normalize stage producing transcripts/normalized.json
• real trim stage producing transcripts/trimmed.json
• real analyze stage for initial session_recap generation
• optional Scriptorium render diagnostics (render_debug) before production run
• storage/archive configuration and validation foundations for:
  • pipeline.storage.s3
  • pipeline.spool
  • pipeline.archive promotion rules
  • session.inputs.audio_s3
• run identity and path-model foundations:
  • run ID generation (YYYYMMDDTHHMMSSZ-xxxxxxxx)
  • S3 session/run/current key builders
  • campaign/session/run local work/spool path helpers
  • manifest run/path identity fields (campaign, run_id, local and S3 prefixes)
• remote storage backend layer:
  • narrow object-store interface (List, Download, Upload, Exists)
  • fake storage backend for deterministic tests (no network dependency)
  • S3-compatible backend using AWS SDK v2
  • config-based object-store construction helper

Still placeholder/future:

• archive stage behavior
• prepare-stage S3 audio download behavior
• archive-stage S3 upload/promotion behavior
• notify stage behavior
• additional Scriptorium artifact types beyond session_recap
• artifact-to-artifact workflows beyond the initial single-artifact implementation
• generic stale detection based on input/config checksums

3. Pipeline and Stage Boundaries

Canonical stage order:

1. prepare
2. transcribe
3. merge
4. polish
5. normalize
6. trim
7. analyze
8. archive
9. notify

Boundary rules:

• orchestration logic lives in internal/app
• stage business logic lives in internal/stage
• external-tool CLI construction lives in adapter packages
• Scriptorium CLI details stay in internal/adapters/scriptorium

4. Scriptorium Integration Model

Integration mode:

• public CLI subprocesses only (no Scriptorium internal Go packages, no HTTP API)
• production generation uses scriptorium run
• diagnostics/testing render uses scriptorium render --format json

Run invocation shape used by adapter:

scriptorium run --prompt <prompt_id> --input name=path --out <output_path>

Optional flags passed when configured:

• --config <path>
• --profile <profile_id>
• repeated --var name=value
• repeated --input name=path
• --timeout <duration>
• --api-key-env <ENV_NAME> when configured

Render invocation shape used by adapter:

scriptorium render --prompt <prompt_id> --input name=path --format json --out <render_output_path>

Adapter behavior:

• always passes --out
• captures stdout/stderr separately
• writes generated invocation metadata YAML (redacted, no secrets)
• treats exit code 0 as success
• treats exit code 1 as failure
• treats exit code 2 as failure with validation_failed=true and preserves output metadata when available
• validates successful output files exist and are non-empty
• does not treat non-empty stderr as failure by itself

5. Configuration Contract

CLI pipeline config path resolution:

• when --config <path> is provided, that path is used
• when --config is omitted, Narratio searches defaults in order:
  • /usr/local/etc/narratio/pipeline.yml
  • /etc/narratio/pipeline.yml

Optional pipeline secrets directory:

• pipeline.secrets.env_dir enables loading environment variables from local files before command execution
• file name = env var name; file contents = env var value (trailing newline/CRLF trimmed)
• only env-var-style file names are considered; other entries are ignored
• existing process environment values are preserved (not overwritten)
• if configured, unreadable/missing env_dir fails command execution early
• relative env_dir values are resolved from current working directory

Storage and archive foundations:

• pipeline.storage.s3 is available for modeling S3 coordinates:
  • bucket
  • root_prefix (default dnd)
  • region
  • endpoint
  • force_path_style (default false)
• pipeline.spool.root defaults to /var/spool/narratio
• pipeline.spool.delete_audio_after_archive defaults to false (cleanup behavior not implemented yet)
• pipeline.archive is optional and defaults to:
  • enabled: true
  • upload_run: true
  • default promote_artifacts:
    • transcripts/trimmed.json
    • artifacts/session_recap.md
• archive promotion rules enforce safe relative paths:
  • from and to are required
  • absolute paths are rejected
  • traversal segments such as .. are rejected

Session input foundations:

• session.campaign is required
• local audio remains supported through session.inputs.audio_dir or session.inputs.audio_files
• optional S3 audio input shape is modeled as session.inputs.audio_s3.prefix
• audio_dir/audio_files and audio_s3 are mutually exclusive
• S3 input execution (object listing/downloading) is not implemented yet

Cross-config validation scope:

• pipeline.storage.s3.bucket is required only when an S3-dependent feature is explicitly configured (for current foundations, that includes session.inputs.audio_s3, and archive upload intent when using storage.backend: s3)
• no AWS credentials are stored in Narratio config; credential resolution remains an external runtime concern

Remote object-store backend scope:

• remote storage APIs are isolated to internal/adapters/storage
• AWS SDK types remain contained within the S3 backend implementation package
• S3 key/session path semantics remain outside the backend, with this invariant:
  • callers pass full bucket-relative object keys
  • backend methods do not prepend root_prefix or infer campaign/session/run paths
• the backend layer is available for future prepare/archive usage, but no stage currently invokes List/Download/Upload/Exists

pipeline.scriptorium is optional. Existing pipelines without Scriptorium continue to work.

pipeline.trim is optional. Existing pipelines without trim config continue to work.

pipeline.normalize is optional. Existing pipelines without normalize config continue to work.

pipeline.audita drives the real Audita subprocess adapter for the polish stage.

Audita required fields:

• binary
• timeout
• base_url
• model

Audita optional fields:

• llm_api_key_env (enforced only when configured)
• modules override list (when omitted/empty, Narratio does not pass --modules)
• transcript_description
• config_path
• output_schema (bare-segments or audita-v1)
• work_dir_retention (always, auto, never)
• total_llm_concurrency (> 0 when provided)
• proposal_llm_concurrency (> 0 when provided)
• validation_model
• validation_llm_concurrency (> 0 when provided)
• report (default true)

Narratio passes only configured optional Audita flags; omitted optional values defer to Audita defaults.

When pipeline.normalize is omitted, defaults are applied:

• output_path: transcripts/normalized.json
• output_schema: seriatim-intermediate
• report: true

When pipeline.normalize is present:

• output_path must be non-empty
• output_schema must be one of seriatim-minimal, seriatim-intermediate, or seriatim-full
• relative output_path values are session-workdir-relative paths
• Seriatim binary settings still come from pipeline.seriatim

When pipeline.trim is present:

• enabled is optional and defaults to false when omitted
• relative output_path, bounds.output_path, and bounds.render_output_path values are session-workdir-relative paths
• do not store secrets in trim config values

When pipeline.trim.enabled: true:

• output_path is required and non-empty
• bounds.prompt_id is required and non-empty
• bounds.transcript_input_name is required and non-empty
• bounds.output_path is required and non-empty
• bounds.timeout must parse as a Go duration when provided
• bounds.render_debug: true requires non-empty bounds.render_output_path
• bounds.profile_id may be empty to use the prompt default profile
• prompt IDs are config values, not hardcoded stage logic

When pipeline.scriptorium is present:

• binary is required and non-empty
• config_path is optional; when provided it must be non-empty
• timeout is optional; when provided it must parse as a Go duration
• default timeout is 10m
• unknown YAML fields fail strict decode

Artifacts are configured as a map under pipeline.scriptorium.artifacts so multiple artifacts are possible in the config shape.

For each artifact definition:

• enabled: true requires non-empty prompt_id
• enabled: true requires non-empty output_path
• timeout must parse as Go duration when present
• optional per-artifact render_debug may override global scriptorium.render_debug
• inputs are named and each input requires non-empty source
• inputs may be optional (required: false)
• vars values currently support string and bool

Prompt IDs and profile IDs are configuration values, not hardcoded stage logic.

Trim config shape:

trim:
  enabled: true
  output_path: "transcripts/trimmed.json"
  bounds:
    prompt_id: "dnd_session.bounds"
    profile_id: ""
    transcript_input_name: "transcript"
    output_path: "artifacts/session_bounds.json"
    timeout: "10m"
    render_debug: false
    render_output_path: "artifacts/session_bounds.render.json"
  seriatim:
    report: false

6. Transcript Tiers

Narratio currently produces and uses four transcript tiers:

• transcripts/merged.json: canonical deterministic merged transcript from Seriatim merge
• transcripts/processed.json: full raw Audita-polished transcript output (includes pre/post-game content)
• transcripts/normalized.json: normalized transcript generated by Seriatim normalize
• transcripts/trimmed.json: gameplay-only normalized polished transcript from trim stage

Trim reads transcripts/normalized.json, validates bounds IDs against that same transcript ID space, and writes transcripts/trimmed.json.

7. Normalize Stage (Current Implementation)

Normalize stage behavior:

• stage order position: after polish and before trim
• discovers processed transcript from manifest polish outputs (transcript_processed) when present, else work/<session_id>/transcripts/processed.json
• validates processed transcript JSON shape (segments array required)
• runs Seriatim normalize to produce normalized transcript
• validates normalized transcript JSON shape (segments array required)
• validates normalize report JSON when enabled

Expected normalize outputs and diagnostics:

• transcripts/normalized.json
• artifacts/seriatim.normalize.report.json (when normalize report is enabled)
• logs/seriatim.normalize.stdout.log
• logs/seriatim.normalize.stderr.log
• config/seriatim.normalize.generated.yml

8. Trim Stage (Current Implementation)

Trim stage behavior:

• stage order position: after normalize and before analyze
• discovers normalized transcript from manifest normalize outputs (transcript_normalized) when present, else work/<session_id>/transcripts/normalized.json
• validates normalized transcript JSON shape (segments array required)
• when trim.enabled: false (or trim config omitted), deterministically copies normalized transcript to transcripts/trimmed.json and records trim_action=copy_disabled
• when trim.enabled: true:
  • runs Scriptorium bounds prompt using configured trim.bounds.prompt_id
  • writes bounds output to configured path (typically artifacts/session_bounds.json)
  • parses and validates bounds output against the same normalized transcript being trimmed
  • converts bounds range to Seriatim keep selector (for example 10-868)
  • runs Seriatim trim to produce transcripts/trimmed.json
  • supports no-trim bounds actions (none/copy) by copying normalized transcript unchanged
• validates trimmed transcript JSON shape (segments array required)

Expected trim outputs and diagnostics:

• artifacts/session_bounds.json
• transcripts/trimmed.json
• logs/scriptorium.bounds.stdout.log
• logs/scriptorium.bounds.stderr.log
• config/scriptorium.bounds.generated.yml
• logs/seriatim.trim.stdout.log
• logs/seriatim.trim.stderr.log
• config/seriatim.trim.generated.yml
• optional bounds render-debug outputs when enabled:
  • artifacts/session_bounds.render.json
  • logs/scriptorium.bounds.render.stdout.log
  • logs/scriptorium.bounds.render.stderr.log
  • config/scriptorium.bounds.render.generated.yml

Render-debug files are diagnostics. They are recorded in stage metadata/log/config refs and are not treated as canonical stage output artifact refs.

9. Analyze Stage (Current Implementation)

The current real analyze implementation supports only scriptorium.artifacts.session_recap.

Behavior:

• if pipeline.scriptorium is missing, analyze returns a skipped result with metadata
• if no Scriptorium artifacts are enabled, analyze returns a skipped result with metadata
• if enabled artifacts exist but session_recap is not enabled, analyze fails clearly
• available transcript input sources for configured artifacts: processed_transcript, normalized_transcript, trimmed_transcript
• session_recap should use trimmed_transcript input (transcripts/trimmed.json) for in-universe recap generation
• trimmed_transcript input is resolved from manifest (trim output kind transcript_trimmed) when available, otherwise fallback path work/<session_id>/transcripts/trimmed.json
• normalized_transcript input is resolved from manifest (normalize output kind transcript_normalized) when available, otherwise fallback path work/<session_id>/transcripts/normalized.json
• processed_transcript input is resolved from manifest (polish output kind transcript_processed) when available, otherwise fallback path work/<session_id>/transcripts/processed.json
• normalized_transcript is the preferred full-transcript source for future table/meta-analysis artifacts
• processed_transcript remains available for advanced/debug use cases
• transcript inputs are validated as JSON with top-level segments array
• configured inputs are resolved by source
• optional previous_recap is omitted when unavailable
• required previous_recap fails before invocation when unavailable
• vars are built from config + session metadata
• render_debug controls pre-run scriptorium render diagnostics
• render failure stops stage before production run
• render output is validated as JSON
• production call uses Scriptorium adapter RunArtifact
• successful run output must exist and be non-empty
• missing trimmed_transcript input for configured trimmed_transcript source fails clearly with guidance to run trim stage first
• manifest records output refs, logs, generated config paths, and non-secret provenance metadata

10. Session Recap Paths

Current expected paths for session_recap:

• artifact output: artifacts/session_recap.md
• run stdout log: logs/scriptorium.session_recap.stdout.log
• run stderr log: logs/scriptorium.session_recap.stderr.log
• run generated invocation/config: config/scriptorium.session_recap.generated.yml
• render output (when enabled): artifacts/session_recap.render.json
• render stdout log: logs/scriptorium.session_recap.render.stdout.log
• render stderr log: logs/scriptorium.session_recap.render.stderr.log
• render generated invocation/config: config/scriptorium.session_recap.render.generated.yml

11. Security and Privacy

• do not store secrets in pipeline YAML, generated invocation YAML, logs, or manifest metadata
• if API-key integration is configured, pass env var names only (never raw key values)
• with pipeline.secrets.env_dir, secret file values are loaded into process env only and are not persisted in manifest metadata or generated configs
• avoid logging transcript content or rendered prompt content by default
• treat generated artifacts and logs as potentially sensitive session material

12. Operational Caveat (Pre-Stale-Detection)

Checksum-based stale detection is not implemented yet.

If prepared inputs or prompt/runtime configuration change (for example glossary files, prompt IDs, profile IDs, or relevant pipeline settings), rerun the appropriate prior stages to refresh downstream artifacts.

Examples:

• glossary or autocorrect changes usually require rerunning at least merge, polish, normalize, trim, and analyze
• trim prompt/profile changes require rerunning at least normalize, trim, and analyze
• session recap prompt/profile/input-source changes require rerunning analyze

13. Roadmap

Planned next steps:

• extend analyze beyond session_recap to additional configured artifacts
• support artifact inputs that consume prior generated artifacts
• keep this composable without adding a generic DAG engine in the near term
• implement real archive backend behavior
• implement real notify backend behavior
• add checksum-based stale detection and stale transitions

Architectural invariants remain:

• strict config decoding/validation
• manifest-driven run control
• clear stage/adapter separation
• configuration-driven prompt/profile/input/vars/output mapping
• Scriptorium integration through public CLI subprocess contract