narratio/architecture.md

# Narratio Architecture

## 1. Purpose

`narratio` is a Go orchestrator for D&D session processing. It runs a stage-based local pipeline from audio input through transcript processing and artifact generation, with manifest-based skip/force/resume behavior.

Narratio integrates with Scriptorium through the **public CLI** (`scriptorium run` and `scriptorium render`) via synchronous subprocess execution.

## 2. Current Status

Implemented:

- strict `pipeline.yml` + `session.yml` loading with strict YAML field checking (`KnownFields(true)`)
- local workspace/session layout, lock file handling, artifact path helpers, checksums, and atomic writes
- manifest store and stage status transitions for resumable runs
- real `prepare`, `transcribe`, `merge`, and `polish` stages
- real WhisperX HTTP adapter
- real Seriatim subprocess adapter
- real Audita subprocess adapter
- real Scriptorium subprocess adapter
- real `normalize` stage producing `transcripts/normalized.json`
- real `trim` stage producing `transcripts/trimmed.json`
- real `analyze` stage for initial `session_recap` generation
- optional Scriptorium render diagnostics (`render_debug`) before production run
- storage/archive configuration and validation foundations for:
  - `pipeline.storage.s3`
  - `pipeline.spool`
  - `pipeline.archive` promotion rules
  - `session.inputs.audio_s3`
- run identity and path-model foundations:
  - run ID generation (`YYYYMMDDTHHMMSSZ-xxxxxxxx`)
  - S3 session/run/current key builders
  - campaign/session/run local work/spool path helpers
  - manifest run/path identity fields (`campaign`, `run_id`, local and S3 prefixes)
- remote storage backend layer:
  - narrow object-store interface (`List`, `Download`, `Upload`, `Exists`)
  - fake storage backend for deterministic tests (no network dependency)
  - S3-compatible backend using AWS SDK v2
  - config-based object-store construction helper

Still placeholder/future:

- archive-stage promotion behavior
- `notify` stage behavior
- additional Scriptorium artifact types beyond `session_recap`
- artifact-to-artifact workflows beyond the initial single-artifact implementation
- generic stale detection based on input/config checksums

## 3. Pipeline and Stage Boundaries

Canonical stage order:

1. `prepare`
2. `transcribe`
3. `merge`
4. `polish`
5. `normalize`
6. `trim`
7. `analyze`
8. `archive`
9. `notify`

Boundary rules:

- orchestration logic lives in `internal/app`
- stage business logic lives in `internal/stage`
- external-tool CLI construction lives in adapter packages
- Scriptorium CLI details stay in `internal/adapters/scriptorium`

## 4. Scriptorium Integration Model

Integration mode:

- public CLI subprocesses only (no Scriptorium internal Go packages, no HTTP API)
- production generation uses `scriptorium run`
- diagnostics/testing render uses `scriptorium render --format json`

Run invocation shape used by adapter:

```bash
scriptorium run --prompt <prompt_id> --input name=path --out <output_path>
```

Optional flags passed when configured:

- `--config <path>`
- `--profile <profile_id>`
- repeated `--var name=value`
- repeated `--input name=path`
- `--timeout <duration>`
- `--api-key-env <ENV_NAME>` when configured

Render invocation shape used by adapter:

```bash
scriptorium render --prompt <prompt_id> --input name=path --format json --out <render_output_path>
```

Adapter behavior:

- always passes `--out`
- captures stdout/stderr separately
- writes generated invocation metadata YAML (redacted, no secrets)
- treats exit code `0` as success
- treats exit code `1` as failure
- treats exit code `2` as failure with `validation_failed=true` and preserves output metadata when available
- validates successful output files exist and are non-empty
- does not treat non-empty stderr as failure by itself

## 5. Configuration Contract

CLI pipeline config path resolution:

- when `--config <path>` is provided, that path is used
- when `--config` is omitted, Narratio searches defaults in order:
  - `/usr/local/etc/narratio/pipeline.yml`
  - `/etc/narratio/pipeline.yml`

Optional pipeline secrets directory:

- `pipeline.secrets.env_dir` enables loading environment variables from local files before command execution
- file name = env var name; file contents = env var value (trailing newline/CRLF trimmed)
- only env-var-style file names are considered; other entries are ignored
- existing process environment values are preserved (not overwritten)
- if configured, unreadable/missing `env_dir` fails command execution early
- relative `env_dir` values are resolved from current working directory

Storage and archive foundations:

- `pipeline.storage.s3` is available for modeling S3 coordinates:
  - `bucket`
  - `root_prefix` (default `dnd`)
  - `region`
  - `endpoint`
  - `force_path_style` (default `false`)
- `pipeline.spool.root` defaults to `/var/spool/narratio`
- `pipeline.spool.delete_audio_after_archive` defaults to `false` (cleanup behavior not implemented yet)
- `pipeline.archive` is optional and defaults to:
  - `enabled: true`
  - `upload_run: true`
  - default `promote_artifacts`:
    - `transcripts/trimmed.json`
    - `artifacts/session_recap.md`
- archive promotion rules enforce safe relative paths:
  - `from` and `to` are required
  - absolute paths are rejected
  - traversal segments such as `..` are rejected

Session input foundations:

- `session.campaign` is required
- local audio remains supported through `session.inputs.audio_dir` or `session.inputs.audio_files`
- optional S3 audio input shape is `session.inputs.audio_s3.prefix`
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
- when `audio_s3` is configured, `prepare` lists and downloads `.flac` objects through the object-store backend

Cross-config validation scope:

- `pipeline.storage.s3.bucket` is required only when an S3-dependent feature is explicitly configured (for current foundations, that includes `session.inputs.audio_s3`, and archive upload intent when using `storage.backend: s3`)
- no AWS credentials are stored in Narratio config; credential resolution remains an external runtime concern

Remote object-store backend scope:

- remote storage APIs are isolated to `internal/adapters/storage`
- AWS SDK types remain contained within the S3 backend implementation package
- S3 key/session path semantics remain outside the backend, with this invariant:
  - callers pass full bucket-relative object keys
  - backend methods do not prepend `root_prefix` or infer campaign/session/run paths
- `prepare` now uses object-store `List` and `Download` for S3 audio input
- `archive` now uses object-store `Upload` for successful run-record upload under the run prefix
- archive promotion behavior using top-level writes and current pointers remains future work

Prepare S3 audio behavior (implemented):

- compute session prefix as `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
- resolve `session.inputs.audio_s3.prefix` under that session prefix
- list objects under the computed audio prefix and filter `.flac` keys
- fail clearly when no `.flac` objects are found
- download selected objects to spool audio path:
  - `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
- materialize audio files into workdir audio path:
  - `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
- record S3 provenance in manifest input records (bucket/key/metadata/local paths/checksum)
- no AWS SDK types are used in stage code; storage implementation details stay in storage adapter packages

Archive run upload behavior (implemented):

- `archive` verifies prerequisite stage success before upload:
  - `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`
- only successful/completed runs are uploaded
- uploaded run record destination is:
  - `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
- uploaded existing local paths include:
  - `inputs/`, `transcripts/`, `artifacts/`, optional `reports/`, `config/`, `logs/`, and `manifest.json`
- local `audio/` is intentionally excluded from upload by default
- file upload order is deterministic (sorted relative paths)
- `archive.enabled: false` and `archive.upload_run: false` skip upload cleanly
- stage metadata records non-secret upload context:
  - `s3_bucket`, `s3_run_prefix`, `files_uploaded`, `uploaded_paths`, `audio_upload_skipped`
- archive does not yet:
  - write top-level promoted outputs
  - write `current/manifest.json`
  - write `current/run_id.txt`

`pipeline.scriptorium` is optional. Existing pipelines without Scriptorium continue to work.

`pipeline.trim` is optional. Existing pipelines without trim config continue to work.

`pipeline.normalize` is optional. Existing pipelines without normalize config continue to work.

`pipeline.audita` drives the real Audita subprocess adapter for the `polish` stage.

Audita required fields:

- `binary`
- `timeout`
- `base_url`
- `model`

Audita optional fields:

- `llm_api_key_env` (enforced only when configured)
- `modules` override list (when omitted/empty, Narratio does not pass `--modules`)
- `transcript_description`
- `config_path`
- `output_schema` (`bare-segments` or `audita-v1`)
- `work_dir_retention` (`always`, `auto`, `never`)
- `total_llm_concurrency` (> 0 when provided)
- `proposal_llm_concurrency` (> 0 when provided)
- `validation_model`
- `validation_llm_concurrency` (> 0 when provided)
- `report` (default `true`)

Narratio passes only configured optional Audita flags; omitted optional values defer to Audita defaults.

When `pipeline.normalize` is omitted, defaults are applied:

- `output_path: transcripts/normalized.json`
- `output_schema: seriatim-intermediate`
- `report: true`

When `pipeline.normalize` is present:

- `output_path` must be non-empty
- `output_schema` must be one of `seriatim-minimal`, `seriatim-intermediate`, or `seriatim-full`
- relative `output_path` values are session-workdir-relative paths
- Seriatim binary settings still come from `pipeline.seriatim`

When `pipeline.trim` is present:

- `enabled` is optional and defaults to `false` when omitted
- relative `output_path`, `bounds.output_path`, and `bounds.render_output_path` values are session-workdir-relative paths
- do not store secrets in trim config values

When `pipeline.trim.enabled: true`:

- `output_path` is required and non-empty
- `bounds.prompt_id` is required and non-empty
- `bounds.transcript_input_name` is required and non-empty
- `bounds.output_path` is required and non-empty
- `bounds.timeout` must parse as a Go duration when provided
- `bounds.render_debug: true` requires non-empty `bounds.render_output_path`
- `bounds.profile_id` may be empty to use the prompt default profile
- prompt IDs are config values, not hardcoded stage logic

When `pipeline.scriptorium` is present:

- `binary` is required and non-empty
- `config_path` is optional; when provided it must be non-empty
- `timeout` is optional; when provided it must parse as a Go duration
- default `timeout` is `10m`
- unknown YAML fields fail strict decode

Artifacts are configured as a map under `pipeline.scriptorium.artifacts` so multiple artifacts are possible in the config shape.

For each artifact definition:

- `enabled: true` requires non-empty `prompt_id`
- `enabled: true` requires non-empty `output_path`
- `timeout` must parse as Go duration when present
- optional per-artifact `render_debug` may override global `scriptorium.render_debug`
- `inputs` are named and each input requires non-empty `source`
- inputs may be optional (`required: false`)
- `vars` values currently support `string` and `bool`

Prompt IDs and profile IDs are configuration values, not hardcoded stage logic.

Trim config shape:

```yaml
trim:
  enabled: true
  output_path: "transcripts/trimmed.json"
  bounds:
    prompt_id: "dnd_session.bounds"
    profile_id: ""
    transcript_input_name: "transcript"
    output_path: "artifacts/session_bounds.json"
    timeout: "10m"
    render_debug: false
    render_output_path: "artifacts/session_bounds.render.json"
  seriatim:
    report: false
```

## 6. Transcript Tiers

Narratio currently produces and uses four transcript tiers:

- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
- `transcripts/processed.json`: full raw Audita-polished transcript output (includes pre/post-game content)
- `transcripts/normalized.json`: normalized transcript generated by Seriatim normalize
- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage

Trim reads `transcripts/normalized.json`, validates bounds IDs against that same transcript ID space, and writes `transcripts/trimmed.json`.

## 7. Normalize Stage (Current Implementation)

Normalize stage behavior:

- stage order position: after `polish` and before `trim`
- discovers processed transcript from manifest polish outputs (`transcript_processed`) when present, else `work/<session_id>/transcripts/processed.json`
- validates processed transcript JSON shape (`segments` array required)
- runs Seriatim `normalize` to produce normalized transcript
- validates normalized transcript JSON shape (`segments` array required)
- validates normalize report JSON when enabled

Expected normalize outputs and diagnostics:

- `transcripts/normalized.json`
- `artifacts/seriatim.normalize.report.json` (when normalize report is enabled)
- `logs/seriatim.normalize.stdout.log`
- `logs/seriatim.normalize.stderr.log`
- `config/seriatim.normalize.generated.yml`

## 8. Trim Stage (Current Implementation)

Trim stage behavior:

- stage order position: after `normalize` and before `analyze`
- discovers normalized transcript from manifest normalize outputs (`transcript_normalized`) when present, else `work/<session_id>/transcripts/normalized.json`
- validates normalized transcript JSON shape (`segments` array required)
- when `trim.enabled: false` (or trim config omitted), deterministically copies normalized transcript to `transcripts/trimmed.json` and records `trim_action=copy_disabled`
- when `trim.enabled: true`:
  - runs Scriptorium bounds prompt using configured `trim.bounds.prompt_id`
  - writes bounds output to configured path (typically `artifacts/session_bounds.json`)
  - parses and validates bounds output against the same normalized transcript being trimmed
  - converts bounds range to Seriatim keep selector (for example `10-868`)
  - runs Seriatim `trim` to produce `transcripts/trimmed.json`
  - supports no-trim bounds actions (`none`/`copy`) by copying normalized transcript unchanged
- validates trimmed transcript JSON shape (`segments` array required)

Expected trim outputs and diagnostics:

- `artifacts/session_bounds.json`
- `transcripts/trimmed.json`
- `logs/scriptorium.bounds.stdout.log`
- `logs/scriptorium.bounds.stderr.log`
- `config/scriptorium.bounds.generated.yml`
- `logs/seriatim.trim.stdout.log`
- `logs/seriatim.trim.stderr.log`
- `config/seriatim.trim.generated.yml`
- optional bounds render-debug outputs when enabled:
  - `artifacts/session_bounds.render.json`
  - `logs/scriptorium.bounds.render.stdout.log`
  - `logs/scriptorium.bounds.render.stderr.log`
  - `config/scriptorium.bounds.render.generated.yml`

Render-debug files are diagnostics. They are recorded in stage metadata/log/config refs and are not treated as canonical stage output artifact refs.

## 9. Analyze Stage (Current Implementation)

The current real analyze implementation supports only `scriptorium.artifacts.session_recap`.

Behavior:

- if `pipeline.scriptorium` is missing, analyze returns a skipped result with metadata
- if no Scriptorium artifacts are enabled, analyze returns a skipped result with metadata
- if enabled artifacts exist but `session_recap` is not enabled, analyze fails clearly
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
- `session_recap` should use `trimmed_transcript` input (`transcripts/trimmed.json`) for in-universe recap generation
- `trimmed_transcript` input is resolved from manifest (`trim` output kind `transcript_trimmed`) when available, otherwise fallback path `work/<session_id>/transcripts/trimmed.json`
- `normalized_transcript` input is resolved from manifest (`normalize` output kind `transcript_normalized`) when available, otherwise fallback path `work/<session_id>/transcripts/normalized.json`
- `processed_transcript` input is resolved from manifest (`polish` output kind `transcript_processed`) when available, otherwise fallback path `work/<session_id>/transcripts/processed.json`
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
- `processed_transcript` remains available for advanced/debug use cases
- transcript inputs are validated as JSON with top-level `segments` array
- configured inputs are resolved by source
- optional `previous_recap` is omitted when unavailable
- required `previous_recap` fails before invocation when unavailable
- vars are built from config + session metadata
- `render_debug` controls pre-run `scriptorium render` diagnostics
- render failure stops stage before production run
- render output is validated as JSON
- production call uses Scriptorium adapter `RunArtifact`
- successful run output must exist and be non-empty
- missing `trimmed_transcript` input for configured `trimmed_transcript` source fails clearly with guidance to run trim stage first
- manifest records output refs, logs, generated config paths, and non-secret provenance metadata

## 10. Session Recap Paths

Current expected paths for `session_recap`:

- artifact output: `artifacts/session_recap.md`
- run stdout log: `logs/scriptorium.session_recap.stdout.log`
- run stderr log: `logs/scriptorium.session_recap.stderr.log`
- run generated invocation/config: `config/scriptorium.session_recap.generated.yml`
- render output (when enabled): `artifacts/session_recap.render.json`
- render stdout log: `logs/scriptorium.session_recap.render.stdout.log`
- render stderr log: `logs/scriptorium.session_recap.render.stderr.log`
- render generated invocation/config: `config/scriptorium.session_recap.render.generated.yml`

## 11. Security and Privacy

- do not store secrets in pipeline YAML, generated invocation YAML, logs, or manifest metadata
- if API-key integration is configured, pass env var names only (never raw key values)
- with `pipeline.secrets.env_dir`, secret file values are loaded into process env only and are not persisted in manifest metadata or generated configs
- avoid logging transcript content or rendered prompt content by default
- treat generated artifacts and logs as potentially sensitive session material

## 12. Operational Caveat (Pre-Stale-Detection)

Checksum-based stale detection is not implemented yet.

If prepared inputs or prompt/runtime configuration change (for example glossary files, prompt IDs, profile IDs, or relevant pipeline settings), rerun the appropriate prior stages to refresh downstream artifacts.

Examples:

- glossary or autocorrect changes usually require rerunning at least `merge`, `polish`, `normalize`, `trim`, and `analyze`
- trim prompt/profile changes require rerunning at least `normalize`, `trim`, and `analyze`
- session recap prompt/profile/input-source changes require rerunning `analyze`

## 13. Roadmap

Planned next steps:

- extend analyze beyond `session_recap` to additional configured artifacts
- support artifact inputs that consume prior generated artifacts
- keep this composable without adding a generic DAG engine in the near term
- implement real `archive` backend behavior
- implement real `notify` backend behavior
- add checksum-based stale detection and stale transitions

Architectural invariants remain:

- strict config decoding/validation
- manifest-driven run control
- clear stage/adapter separation
- configuration-driven prompt/profile/input/vars/output mapping
- Scriptorium integration through public CLI subprocess contract