narratio/architecture.md

Narratio Architecture

1. Purpose

narratio is a Go orchestrator for D&D session processing. It runs a stage-based local pipeline from audio input through transcript processing and artifact generation, with manifest-based skip/force/resume behavior.

Narratio integrates with Scriptorium through the public CLI (scriptorium run and scriptorium render) via synchronous subprocess execution.

2. Current Status

Implemented:

• strict pipeline.yml + session.yml loading with strict YAML field checking (KnownFields(true))
• local workspace/session layout, lock file handling, artifact path helpers, checksums, and atomic writes
• manifest store and stage status transitions for resumable runs
• real prepare, transcribe, merge, and polish stages
• real WhisperX HTTP adapter
• real Seriatim subprocess adapter
• real Audita subprocess adapter
• real Scriptorium subprocess adapter
• real analyze stage for initial session_recap generation
• optional Scriptorium render diagnostics (render_debug) before production run

Still placeholder/future:

• trim stage behavior
• archive stage behavior
• notify stage behavior
• additional Scriptorium artifact types beyond session_recap
• artifact-to-artifact workflows beyond the initial single-artifact implementation
• generic stale detection based on input/config checksums

3. Pipeline and Stage Boundaries

Canonical stage order:

1. prepare
2. transcribe
3. merge
4. polish
5. trim
6. analyze
7. archive
8. notify

Boundary rules:

• orchestration logic lives in internal/app
• stage business logic lives in internal/stage
• external-tool CLI construction lives in adapter packages
• Scriptorium CLI details stay in internal/adapters/scriptorium

4. Scriptorium Integration Model

Integration mode:

• public CLI subprocesses only (no Scriptorium internal Go packages, no HTTP API)
• production generation uses scriptorium run
• diagnostics/testing render uses scriptorium render --format json

Run invocation shape used by adapter:

scriptorium run --prompt <prompt_id> --input name=path --out <output_path>

Optional flags passed when configured:

• --config <path>
• --profile <profile_id>
• repeated --var name=value
• repeated --input name=path
• --timeout <duration>
• --api-key-env <ENV_NAME> when configured

Render invocation shape used by adapter:

scriptorium render --prompt <prompt_id> --input name=path --format json --out <render_output_path>

Adapter behavior:

• always passes --out
• captures stdout/stderr separately
• writes generated invocation metadata YAML (redacted, no secrets)
• treats exit code 0 as success
• treats exit code 1 as failure
• treats exit code 2 as failure with validation_failed=true and preserves output metadata when available
• validates successful output files exist and are non-empty
• does not treat non-empty stderr as failure by itself

5. Configuration Contract

pipeline.scriptorium is optional. Existing pipelines without Scriptorium continue to work.

pipeline.trim is optional. Existing pipelines without trim config continue to work.

When pipeline.trim is present:

• enabled is optional and defaults to false when omitted
• relative output_path, bounds.output_path, and bounds.render_output_path values are session-workdir-relative paths
• do not store secrets in trim config values

When pipeline.trim.enabled: true:

• output_path is required and non-empty
• bounds.prompt_id is required and non-empty
• bounds.transcript_input_name is required and non-empty
• bounds.output_path is required and non-empty
• bounds.timeout must parse as a Go duration when provided
• bounds.render_debug: true requires non-empty bounds.render_output_path
• bounds.profile_id may be empty to use the prompt default profile
• prompt IDs are config values, not hardcoded stage logic

When pipeline.scriptorium is present:

• binary is required and non-empty
• config_path is optional; when provided it must be non-empty
• timeout is optional; when provided it must parse as a Go duration
• default timeout is 10m
• unknown YAML fields fail strict decode

Artifacts are configured as a map under pipeline.scriptorium.artifacts so multiple artifacts are possible in the config shape.

For each artifact definition:

• enabled: true requires non-empty prompt_id
• enabled: true requires non-empty output_path
• timeout must parse as Go duration when present
• optional per-artifact render_debug may override global scriptorium.render_debug
• inputs are named and each input requires non-empty source
• inputs may be optional (required: false)
• vars values currently support string and bool

Prompt IDs and profile IDs are configuration values, not hardcoded stage logic.

Trim scaffold config shape:

trim:
  enabled: true
  output_path: "transcripts/trimmed.json"
  bounds:
    prompt_id: "dnd_session.bounds"
    profile_id: ""
    transcript_input_name: "transcript"
    output_path: "artifacts/session_bounds.json"
    timeout: "10m"
    render_debug: false
    render_output_path: "artifacts/session_bounds.render.json"
  seriatim:
    report: false

6. Analyze Stage (Current Implementation)

The current real analyze implementation supports only scriptorium.artifacts.session_recap.

Behavior:

• if pipeline.scriptorium is missing, analyze returns a skipped result with metadata
• if no Scriptorium artifacts are enabled, analyze returns a skipped result with metadata
• if enabled artifacts exist but session_recap is not enabled, analyze fails clearly
• processed transcript input is resolved from manifest (polish output kind transcript_processed) when available, otherwise fallback path work/<session_id>/transcripts/processed.json
• processed transcript is validated as JSON with top-level segments array
• configured inputs are resolved by source
• optional previous_recap is omitted when unavailable
• required previous_recap fails before invocation when unavailable
• vars are built from config + session metadata
• render_debug controls pre-run scriptorium render diagnostics
• render failure stops stage before production run
• render output is validated as JSON
• production call uses Scriptorium adapter RunArtifact
• successful run output must exist and be non-empty
• manifest records output refs, logs, generated config paths, and non-secret provenance metadata

7. Session Recap Paths

Current expected paths for session_recap:

• artifact output: artifacts/session_recap.md
• run stdout log: logs/scriptorium.session_recap.stdout.log
• run stderr log: logs/scriptorium.session_recap.stderr.log
• run generated invocation/config: config/scriptorium.session_recap.generated.yml
• render output (when enabled): artifacts/session_recap.render.json
• render stdout log: logs/scriptorium.session_recap.render.stdout.log
• render stderr log: logs/scriptorium.session_recap.render.stderr.log
• render generated invocation/config: config/scriptorium.session_recap.render.generated.yml

8. Security and Privacy

• do not store secrets in pipeline YAML, generated invocation YAML, logs, or manifest metadata
• if API-key integration is configured, pass env var names only (never raw key values)
• avoid logging transcript content or rendered prompt content by default
• treat generated artifacts and logs as potentially sensitive session material

9. Roadmap

Planned next steps:

• extend analyze beyond session_recap to additional configured artifacts
• support artifact inputs that consume prior generated artifacts
• keep this composable without adding a generic DAG engine in the near term
• implement real archive backend behavior
• implement real notify backend behavior
• add checksum-based stale detection and stale transitions

Architectural invariants remain:

• strict config decoding/validation
• manifest-driven run control
• clear stage/adapter separation
• configuration-driven prompt/profile/input/vars/output mapping
• Scriptorium integration through public CLI subprocess contract