eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Rewrite docs for the publish stage contract and current behavior

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-23 04:51:16 +00:00
parent 79737edf79
commit 71395bb076
22 changed files with 496 additions and 1174 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -2,7 +2,7 @@
Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts.
It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, archive publishing, and resumable run state in one operator workflow.
It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, publish-stage uploads, and resumable run state in one operator workflow.
```bash
narratio run 2026-04-04

18
docs/architecture.md
View File

@@ -19,7 +19,7 @@ It coordinates specialized downstream systems rather than reimplementing their d
- Audita handles transcript correction and polishing.
- Scriptorium handles prompt execution and generated artifacts.
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and archive semantics.
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and publish semantics.
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
@@ -80,7 +80,7 @@ It should record:
- input and output refs;
- logs and generated config refs;
- checksums or provenance where useful;
- non-secret adapter and archive metadata.
- non-secret adapter and publish metadata.
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
@@ -117,19 +117,19 @@ Narratio should not become a secondary configuration system for downstream tools
Local and remote paths are part of Narratio’s application contract.
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and archive paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and publish/current paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
## Archive Invariants
## Publish Invariants
Archive behavior must preserve a clear commit boundary.
Publish behavior must preserve a clear commit boundary.
A remote run is current only after the archive stage has successfully uploaded the run record, required promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`.
A remote run is current only after the publish stage has successfully uploaded the run record, required published outputs, `current/manifest.json`, and finally `current/run_id.txt`.
`current/run_id.txt` is the final remote commit marker and must be written last.
Failed, incomplete, skipped, or uncommitted archive attempts must not be presented as current remote state. Local cleanup is permitted only after successful archive commit and only when explicitly configured.
Failed, incomplete, skipped, or uncommitted publish attempts must not be presented as current remote state. Local cleanup is permitted only after successful publish commit and only when explicitly configured.
## Security and Privacy
@@ -139,7 +139,7 @@ Rules:
- Do not store raw secrets in pipeline or session YAML.
- Use environment variable names or secret-file references for secret handling.
- Do not write raw secret values to manifests, logs, generated configs, or archive metadata.
- Do not write raw secret values to manifests, logs, generated configs, or publish metadata.
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
@@ -177,7 +177,7 @@ Tests should cover:
- stage success, failure, skip, and resume behavior;
- adapter command construction;
- fake storage behavior;
- archive commit ordering;
- publish commit ordering;
- example config validity where practical.
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.

285
docs/cli.md
View File

@@ -6,91 +6,64 @@
narratio run 2026-04-04
```
This command uses default system discovery for `pipeline.yml`, the pipeline default campaign ID, and local `session.yml`. If local session discovery misses and S3 storage is configured, the positional session ID loads remote `session.yml` from the canonical session prefix.
Default pipeline and session discovery checks system config locations only. Pass `--config`, `--campaign-file`, and `--session` to use files from the current working directory. Pass `--campaign <id>` to select a campaign from `pipeline.campaigns.root`.
Ordinary local and remote `session.yml` files must be concrete YAML. Templates belong to `narratio session init`, which renders a configured campaign template before writing the concrete file.
This runs the full pipeline for the given session ID using default config discovery and campaign selection.
## Command Overview
Top-level commands:
- `run <session_id>`: execute pipeline stages and persist manifest state.
- `run <session_id>`: execute the pipeline.
- `resume <session_id>`: continue from first non-succeeded stage.
- `run-stage <stage> <session_id>`: execute exactly one stage.
- `resume <session_id>`: continue from first non-succeeded stage unless forced.
- `analyze <session_id>`: force-rerun the analyze stage.
- `publish <session_id>`: force-rerun the archive stage.
- `analyze <session_id>`: force-rerun analyze stage.
- `publish <session_id>`: force-rerun publish stage.
- `clean <session_id>|--all`: remove local workspace/spool state.
- `session <subcommand>`: session-scoped helper commands.
Session subcommands:
- `session init <session_id>`: create local or remote `session.yml`.
- `session validate <session_id>`: run read-only preflight checks.
- `session status <session_id>`: inspect local/remote session state.
- `session plan <session_id>`: validate config, prepare workspace layout, and print stage run/skip decisions.
- `session restore <session_id>`: restore durable local state from committed remote archive state.
- `session artifacts <session_id>`: list effective artifact source IDs.
- `session locks <session_id>`: list archive promotion locks.
- `session locks add <session_id> <source>`: add or update a remote lock.
- `session locks remove <session_id> <source>`: remove a remote lock.
Unknown commands print usage and exit non-zero.
For config semantics, see [docs/config.md](./config.md). For operator lifecycle and recovery, see [docs/operations.md](./operations.md).
- `session init <session_id>`
- `session plan <session_id>`
- `session validate <session_id>`
- `session status <session_id>`
- `session restore <session_id>`
- `session artifacts <session_id>`
- `session locks <session_id>`
- `session locks add <session_id> <source>`
- `session locks remove <session_id> <source>`
## Common Flags
Most session-aware commands accept:
- `--config <path>`: optional explicit `pipeline.yml` path.
- `--campaign <id>`: optional campaign ID selector.
- `--campaign-file <path>`: optional explicit `campaign.yml` path.
- `--session <path>`: optional explicit concrete `session.yml` path.
- `--previous-session-id <value>`: expected previous session identifier.
- `--config <pipeline.yml>`
- `--campaign <id>`
- `--campaign-file <campaign.yml>`
- `--session <session.yml>`
- `--previous-session-id <id>`
The positional `<session_id>` is required even when `--session` is provided. It is used as the expected session identity and as the remote session lookup value when local session discovery misses.
`--campaign` and `--campaign-file` are mutually exclusive.
## Command Reference
### `run`
```bash
narratio run <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>]
narratio run <session_id> [--force] [--artifacts <name[,name...]>] [...common flags]
```
Purpose:
- Execute configured stages in canonical order.
Success output:
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
Common failure cases:
- missing system default config/session paths when flags are omitted.
- missing selected campaign under `pipeline.campaigns.root`.
- missing local session plus missing/unavailable remote `session.yml`.
- templated `session.yml`; run `narratio session init` to generate concrete YAML.
- concrete session identity mismatch.
- unknown configured artifact key in `--artifacts`.
Runs stages in canonical order and writes manifest state.
### `resume`
```bash
narratio resume <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>]
narratio resume <session_id> [--force] [--artifacts <name[,name...]>] [...common flags]
```
Purpose:
- Continue from session-manifest stage status.
Success output:
- `narratio resume: session <session_id> has no remaining stages`
- or `narratio resume: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
Starts at the first non-succeeded stage from the session manifest.
### `run-stage`
```bash
narratio run-stage <stage> <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>]
narratio run-stage <stage> <session_id> [--force] [--artifacts <name[,name...]>] [...common flags]
```
Valid stage names:
@@ -102,175 +75,130 @@ Valid stage names:
- `normalize`
- `trim`
- `analyze`
- `archive`
- `publish`
- `notify`
Success output:
- `narratio run-stage: stage=<name> executed=<n> skipped=<n> force=<true|false>; manifest=<path>`
`--artifacts` is accepted only for `analyze` and `archive`.
`--artifacts` is accepted only for `analyze` and `publish`.
### `analyze`
```bash
narratio analyze <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--artifacts <name[,name...]>]
narratio analyze <session_id> [--artifacts <name[,name...]>] [...common flags]
```
Purpose:
- Force-rerun the analyze stage.
- Shorter equivalent for `narratio run-stage analyze <session_id> --force`.
`analyze` is force-by-design and does not accept `--force`.
Equivalent to `narratio run-stage analyze <session_id> --force`.
### `publish`
```bash
narratio publish <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--artifacts <name[,name...]>]
narratio publish <session_id> [--artifacts <name[,name...]>] [...common flags]
```
Purpose:
- Force-rerun the archive stage.
- Shorter equivalent for `narratio run-stage archive <session_id> --force`.
`publish` is force-by-design and does not accept `--force` or a stage positional argument.
Equivalent to `narratio run-stage publish <session_id> --force`.
### `clean`
```bash
narratio clean <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--dry-run] [--clear-cache]
narratio clean --all [--config <pipeline.yml>] [--dry-run] [--clear-cache]
narratio clean <session_id> [--dry-run] [--clear-cache] [...common flags]
narratio clean --all [--dry-run] [--clear-cache] [--config <pipeline.yml>]
```
Session cleanup deletes:
- `{workspace.root}/work/{campaign}/{session_id}`
- `{spool.root}/{campaign}/{session_id}`
All-session cleanup deletes:
- `{workspace.root}/work`
- the contents of `{spool.root}`, while preserving the spool root directory itself.
Cache behavior:
- cache is preserved by default.
- `--clear-cache` in session mode removes cached S3 audio files for the resolved session.
- `--all --clear-cache` removes the configured Narratio S3 audio cache namespace for the configured bucket/root prefix.
- `--clear-cache` does not delete arbitrary files under `pipeline.cache.root`.
- session mode deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`.
- `--all` deletes all session work and spool children.
- cache is preserved unless `--clear-cache` is passed.
### `session plan`
```bash
narratio session plan <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force]
narratio session plan <session_id> [--force] [...common flags]
```
Purpose:
- Validate config, load secrets if configured, prepare workdir, and print stage run/skip decisions.
Success output includes:
- `narratio session plan: workdir prepared at <path>`
- one line per stage (`<stage>: run|skip`)
- `totals: run=<n> skip=<n>`
### `session status`
```bash
narratio session status <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>]
```
Output includes:
- session ID, campaign, workspace, and session config source.
- local manifest state when present.
- remote current archive state when storage is configured.
- catalog-based promoted output availability for expected transcript and artifact sources.
- effective archive locks and conservative next actions.
Validates config and session inputs, prepares workdir layout, and prints stage run/skip decisions.
### `session validate`
```bash
narratio session validate <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>]
narratio session validate <session_id> [...common flags]
```
Checks include:
- effective config and session source.
- stable input files.
- local or remote audio availability.
- previous-session requirements.
- archive promotions and effective locks.
Read-only preflight checks for config, inputs, audio availability, previous-session requirements, publish outputs, and effective locks.
Warnings do not fail the command. Any `ERROR` finding exits non-zero.
### `session status`
```bash
narratio session status <session_id> [...common flags]
```
Shows local manifest state, remote current state (when storage is configured), published-output availability, and effective locks.
### `session init`
```bash
narratio session init <session_id> --output ./session.yml
narratio session init <session_id> --remote
narratio session init <session_id> --config <pipeline.yml> --campaign icewind --remote
narratio session init <session_id> --config <pipeline.yml> --campaign-file ./campaign.yml --remote
narratio session init <session_id> --remote --force
```
Additional flags:
Flags:
- `--previous-session-id <value>`
- `--date <value>`
- `--title <value>`
- `--audio-s3-prefix <prefix>`: defaults to `audio/` when neither audio flag is provided.
- `--audio-dir <path>`: local audio directory; mutually exclusive with `--audio-s3-prefix`.
- `--force`: overwrite existing local or remote target.
Behavior:
- exactly one of `--output` or `--remote` is required.
- `--config`, `--campaign`, and `--campaign-file` are optional overrides; omitted campaign selection uses `pipeline.campaigns.default_campaign_id`.
- `--campaign <id>` selects a campaign under `pipeline.campaigns.root`.
- `--campaign-file <path>` loads an explicit campaign file.
- if `campaign.yml` sets `session_template_file`, the template path is resolved relative to `campaign.yml` and rendered from init flags.
- if no session template is configured, a minimal concrete session file is generated directly.
- template variables must be supplied by matching flags, and supplied template-related flags must be used by the template.
- remote writes target `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
- existing local or remote targets fail unless `--force` is passed.
- remote writes use existence checks, not compare-and-swap.
- `--output <path>` or `--remote` (exactly one is required)
- `--previous-session-id <id>`
- `--date <date>`
- `--title <title>`
- `--audio-dir <path>`
- `--audio-s3-prefix <prefix>`
- `--force`
- common config/campaign flags
### `session restore`
```bash
narratio session restore <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--dry-run] [--force] [--include-audio]
narratio session restore <session_id> [--dry-run] [--force] [--include-audio] [...common flags]
```
Purpose:
- Restore durable session state from the committed remote archive current state.
- Default restore installs `manifest.json`, `transcripts/**`, and `artifacts/**` from the current session archive.
- When configured previous-session inputs require it, restore reconstructs `previous/**` from the previous session's committed current archive.
- `audio/**` is restored only with `--include-audio`.
Restores durable local session files from committed remote current state.
Dry-run output may include planned previous-cache downloads. Existing differing files under `previous/**` follow the normal restore conflict policy and require `--force` to overwrite.
Default restore scope:
When `--include-audio` is set, S3 audio files are restored through the shared audio cache. Cache hits avoid re-downloading large audio objects.
- `manifest.json`
- `transcripts/**`
- `artifacts/**`
- `previous/**` when required by configured previous-session artifact inputs
`audio/**` is restored only when `--include-audio` is set.
### `session artifacts`
```bash
narratio session artifacts <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--remote]
narratio session artifacts <session_id> [--remote] [...common flags]
```
Purpose:
- List built-in, configured, previous-session, promoted, and locked artifact sources.
`--remote` checks promoted top-level object availability through the storage adapter. Remote markers appear only in the `Promoted` section, which reports each configured archive promotion destination and includes `dest=<path>` when that destination differs from the source's canonical path.
Lists built-in sources, configured artifact sources, previous-session sources, publish output rules, and lock status. With `--remote`, includes remote published-state markers.
### `session locks`
```bash
narratio session locks <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>]
narratio session locks add <session_id> <source> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--reason <text>] [--force]
narratio session locks remove <session_id> <source> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>]
narratio session locks <session_id> [...common flags]
narratio session locks add <session_id> <source> [--reason <text>] [--force] [...common flags]
narratio session locks remove <session_id> <source> [...common flags]
```
Behavior:
- list mode prints effective locks from static `pipeline.archive.locks` and remote `{session_prefix}/locks.yml`.
- `locks add` writes only the remote lock store and fails if the source is already locked by pipeline config.
- `locks remove` removes only remote locks and cannot remove static pipeline locks.
- `locks add --force` is required to update an existing remote lock reason.
- list mode prints effective locks from static `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`.
- add/remove mutate only the remote lock store.
- static pipeline locks cannot be removed by lock commands.
## `--artifacts` Rules
- accepted on `run`, `resume`, `run-stage`, `analyze`, and `publish`.
- on `run-stage`, only valid for `analyze` and `publish`.
- filters configured analyze artifact execution.
- filters configured `pipeline.publish.outputs` entries for `narratio.artifact.<key>` sources.
- does not suppress built-in transcript/bounds publish outputs.
- does not imply `--force` for `run`, `resume`, or `run-stage`.
## Common Workflows
Default-discovery run:
Run full pipeline:
```bash
narratio run 2026-04-04
@@ -282,60 +210,21 @@ Run only selected analyze artifacts:
narratio run 2026-04-04 --artifacts session_recap,player_handout
```
Resume with selected analyze artifacts:
```bash
narratio resume 2026-04-04 --artifacts player_handout
```
Force-rerun analyze with selected artifacts:
Force analyze only:
```bash
narratio analyze 2026-04-04 --artifacts player_handout
```
Force-rerun archive publishing:
Force publish only:
```bash
narratio publish 2026-04-04
```
Preview restore actions without writes:
Restore preview then apply:
```bash
narratio session restore 2026-04-04 --dry-run
```
Restore and then force analyze:
```bash
narratio session restore 2026-04-04
narratio analyze 2026-04-04
```
Rehydrate canonical previous-session inputs after artifact-input changes:
```bash
narratio run-stage prepare 2026-04-04 --force
```
Reset local state before testing restore:
```bash
narratio clean 2026-04-04 --dry-run
narratio clean 2026-04-04
narratio session restore 2026-04-04 --include-audio
```
Clean all local sessions while keeping cached S3 audio:
```bash
narratio clean --all
```
## `--artifacts` and `--force`
- `--artifacts` filters which configured artifacts are executable when analyze runs and which configured artifact promotions archive publishes.
- `--artifacts` does not imply `--force`.
- if analyze is already `succeeded` and `--force` is not set, runner-level skip still applies.
- `--artifacts` does not suppress built-in transcript or bounds promotions.

448
docs/config.md
View File

@@ -1,14 +1,13 @@
# Configuration
## 1. Overview
## Overview
Narratio loads three YAML files:
- `pipeline.yml`: pipeline-level runtime configuration.
- `pipeline.yml`: pipeline-level runtime settings.
- `campaign.yml`: stable campaign identity and campaign-level input defaults.
- `session.yml`: per-session metadata and input selection, loaded locally or from the configured S3 backend.
- `session.yml`: per-session metadata and input selection.
These commands load and validate all three files before running:
Commands that load and validate all three files include:
- `narratio run`
- `narratio resume`
@@ -23,94 +22,36 @@ These commands load and validate all three files before running:
- `narratio session locks`
- `narratio clean <session_id>`
Behavior:
Validation behavior:
- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail.
- ordinary local and remote `session.yml` files must be concrete YAML; template placeholders are rejected.
- strict YAML decode is enabled (`KnownFields(true)`); unknown fields fail.
- loaded `session.yml` files must be concrete YAML (no `{{ ... }}` placeholders).
- defaults are applied for optional pipeline fields.
- campaign identity is selected by ID from the pipeline campaign registry unless `--campaign-file` is used.
- campaign-level stable input paths fill missing session input paths.
- session-level stable input paths override campaign-level input paths.
- campaign config may point `session init` to a session template.
- validation enforces required fields, value formats, and cross-field constraints.
- campaign/session identity mismatches fail load.
## 2. Config file discovery
## File Discovery
Pipeline discovery order when `--config` is omitted:
These commands use the same config discovery behavior:
1. `/usr/local/etc/narratio/pipeline.yml`
2. `/etc/narratio/pipeline.yml`
- `narratio run`
- `narratio resume`
- `narratio run-stage`
- `narratio analyze`
- `narratio publish`
- `narratio session plan`
- `narratio session status`
- `narratio session validate`
- `narratio session restore`
- `narratio session artifacts`
- `narratio session locks`
- `narratio clean <session_id>`
Session discovery order when `--session` is omitted:
Pipeline config lookup:
1. `/usr/local/etc/narratio/session.yml`
2. `/etc/narratio/session.yml`
- if `--config <path>` is provided, that path is used.
- if omitted, Narratio searches in order:
1. `/usr/local/etc/narratio/pipeline.yml`
2. `/etc/narratio/pipeline.yml`
- first existing file wins.
Campaign discovery when `--campaign-file` is omitted:
Campaign config lookup:
- if `--campaign <id>` is set: `{pipeline.campaigns.root}/{id}/campaign.yml`
- otherwise: `{pipeline.campaigns.root}/{pipeline.campaigns.default_campaign_id}/campaign.yml`
- pipeline config is loaded first.
- if `--campaign-file <path>` is provided, that path is used.
- otherwise, if `--campaign <id>` is provided, Narratio loads:
- `{pipeline.campaigns.root}/{id}/campaign.yml`
- otherwise, Narratio uses `pipeline.campaigns.default_campaign_id` and loads:
- `{pipeline.campaigns.root}/{default_campaign_id}/campaign.yml`
- `--campaign` and `--campaign-file` are mutually exclusive.
- campaign IDs must be single path segments, not paths.
Remote `session.yml` fallback:
Session config lookup:
- if `--session <path>` is provided, that path is used.
- if `--session` is omitted, Narratio searches locally in order:
1. `/usr/local/etc/narratio/session.yml`
2. `/etc/narratio/session.yml`
- first existing local file wins.
- if no local session file is found, a positional `<session_id>` is present, storage is configured, and campaign identity is resolved, Narratio loads remote `session.yml` from:
- if local session discovery fails and storage is configured, Narratio can load:
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
- local discovery always runs before remote fallback.
- local files in the current working directory are used only when passed explicitly, for example `--config ./pipeline.yml --campaign-file ./campaign.yml --session ./session.yml`.
## 3. Session templating
Template behavior for local and remote `session.yml` loaded by downstream commands:
- downstream commands do not render templates.
- local and remote `session.yml` must be concrete.
- any `{{ ... }}` placeholder in loaded `session.yml` fails with guidance to run `narratio session init`.
- if concrete `session_id` mismatches the positional `<session_id>`, load fails.
- if concrete `previous_session_id` mismatches `--previous-session-id`, load fails.
Template behavior for `narratio session init`:
- `campaign.yml` may set `session_template_file`.
- relative template paths resolve relative to `campaign.yml`.
- supported init template variables:
- `{{ session_id }}`
- `{{ previous_session_id }}`
- `{{ date }}`
- `{{ title }}`
- `{{ audio_s3_prefix }}`
- `{{ audio_dir }}`
- each template variable must be supplied by the matching `session init` flag.
- template-related flags such as `--date`, `--title`, `--audio-s3-prefix`, `--audio-dir`, and `--previous-session-id` fail if the configured template does not use them.
- rendered output is strict-decoded and validated before it is written locally or remotely.
- if `session_template_file` is omitted, `session init` generates the minimal concrete session YAML directly.
## 4. Minimal config set
### `pipeline.yml`
## Minimal Working Config
`pipeline.yml`
```yaml
campaigns:
@@ -120,30 +61,17 @@ whisperx:
transcribe_url: "https://transcription.example.com/transcribe"
```
Why this is sufficient:
- `whisperx.transcribe_url` is required.
- `campaigns.default_campaign_id` selects the default campaign when `--campaign` is omitted.
- `workspace.root` defaults to `/var/lib/narratio`.
- optional sections (`seriatim`, `audita`, `archive`, `scriptorium`, `trim`, `normalize`, etc.) receive defaults or stay inactive.
### `campaign.yml`
`campaign.yml`
```yaml
campaign_id: sample-campaign
session_template_file: ./session.template.yml
inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
```
Why this is sufficient:
- `campaign_id` supplies the stable campaign identity.
- stable input files are required and resolve relative to `campaign.yml` when copied during `prepare`.
### `session.yml`
`session.yml`
```yaml
session_id: 2026-05-03
@@ -151,67 +79,14 @@ inputs:
audio_dir: ./audio
```
Why this is sufficient:
- `session_id` is required.
- `campaign` can be omitted because it is supplied by `campaign.yml`.
- stable input paths can be omitted because `campaign.yml` supplies defaults.
- local `audio_dir` resolves relative to `session.yml`.
Minimal local-file usage:
```bash
narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign sample-campaign --session ./session.yml
narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign-file ./campaign.yml --session ./session.yml
```
Previous-session-enabled variant:
## Publish Config
Top-level publish settings live at `pipeline.publish`.
```yaml
session_id: 2026-05-03
previous_session_id: 2026-04-26
inputs:
audio_dir: ./audio
```
```bash
narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign sample-campaign --session ./session.yml --previous-session-id 2026-04-26
```
## 5. Production-oriented config set
### `pipeline.yml`
```yaml
workspace:
root: /var/lib/narratio/workspace
cleanup_after_archive: true
storage:
backend: s3
s3:
bucket: my-dnd-archive
root_prefix: dnd
region: us-east-1
access_key_id_env: OBJECT_STORAGE_KEY_ID
secret_access_key_env: OBJECT_STORAGE_KEY
campaigns:
root: /srv/narratio/campaigns
default_campaign_id: forsaken
spool:
root: /var/spool/narratio
delete_audio_after_archive: true
cache:
root: /var/cache/narratio
s3_audio: true
archive:
publish:
enabled: true
upload_run: true
promote_artifacts:
outputs:
- source: narratio.transcript.final_trimmed
dest: transcripts/final.trimmed.json
required: true
@@ -221,80 +96,31 @@ archive:
locks:
- source: narratio.artifact.session_recap
reason: Final recap was manually edited.
whisperx:
transcribe_url: "https://transcription.example.com/transcribe"
scriptorium:
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
inputs:
transcript:
source: narratio.transcript.final_trimmed
required: true
previous_recap:
source: narratio.previous_session.artifact.session_recap
required: false
```
### `campaign.yml`
Rules:
```yaml
campaign_id: forsaken
inputs:
speakers_file: /srv/narratio/campaigns/forsaken/speakers.yml
autocorrect_file: /srv/narratio/campaigns/forsaken/autocorrect.yml
glossary_file: /srv/narratio/campaigns/forsaken/glossary.yml
```
- `outputs[].source` is required.
- `outputs[].dest` is optional; when omitted, Narratio derives destination from the source.
- `outputs[].required` defaults to `true`.
- static `publish.locks` and remote `{session_prefix}/locks.yml` are merged; static locks win on duplicates.
- locks prevent overwrite of top-level published destinations.
### Local `session.yml`
Supported publish source families:
```yaml
session_id: 2026-05-03
previous_session_id: 2026-04-26
date: 2026-05-03
title: The Black Cabin
inputs:
audio_s3:
prefix: audio/
```
- built-ins: `narratio.transcript.base`, `narratio.transcript.polished`, `narratio.transcript.final`, `narratio.transcript.final_trimmed`, `narratio.bounds.session`
- configured artifacts: `narratio.artifact.<artifact_key>`
### S3-first session config
For S3-first operation, upload the same `session.yml` content to:
```text
{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml
```
Then run with explicit or discovered pipeline/campaign config and no `--session`:
```bash
narratio run 2026-05-03 --config /usr/local/etc/narratio/pipeline.yml --campaign forsaken --previous-session-id 2026-04-26
```
Operational notes:
- archive promotion is explicit and source-based via `archive.promote_artifacts`.
- `source` is required; `dest` is optional and derived when omitted.
- `archive.locks` skips top-level promotion overwrites for static locked sources while preserving run-local uploads.
- operator-created mutable locks are stored at `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/locks.yml` and are merged with static locks.
- Narratio does not auto-promote all generated analyze artifacts.
- `restore` reads the same config/campaign/session inputs and restore scope is bounded by committed archive current state.
- `clean` removes workspace/spool state by default and preserves `pipeline.cache.root` unless `--clear-cache` is passed.
## 6. Full pipeline reference
## Full Reference
### Pipeline
| Path | Type | Required | Default |
| --- | --- | --- | --- |
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
| `pipeline.workspace.cleanup_after_archive` | bool | No | `false` |
| `pipeline.workspace.cleanup_after_publish` | bool | No | `false` |
| `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` |
| `pipeline.campaigns.default_campaign_id` | string | No | empty |
| `pipeline.secrets.env_dir` | string | Conditional | none |
| `pipeline.secrets.env_dir` | string | No | empty |
| `pipeline.storage.backend` | string | No | empty |
| `pipeline.storage.s3.bucket` | string | Conditional | empty |
| `pipeline.storage.s3.root_prefix` | string | No | `dnd` |
@@ -304,18 +130,18 @@ Operational notes:
| `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` |
| `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` |
| `pipeline.spool.root` | string | No | `/var/spool/narratio` |
| `pipeline.spool.delete_audio_after_archive` | bool | No | `false` |
| `pipeline.spool.delete_audio_after_publish` | bool | No | `false` |
| `pipeline.cache.root` | string | No | `/var/cache/narratio` |
| `pipeline.cache.s3_audio` | bool | No | `true` |
| `pipeline.archive.enabled` | bool | No | `true` |
| `pipeline.archive.upload_run` | bool | No | `true` |
| `pipeline.archive.promote_artifacts[]` | list | No | final-trimmed transcript rule |
| `pipeline.archive.promote_artifacts[].source` | string | Yes (per rule) | none |
| `pipeline.archive.promote_artifacts[].dest` | string | No | derived from source |
| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` |
| `pipeline.archive.locks[]` | list | No | empty |
| `pipeline.archive.locks[].source` | string | Yes (per lock) | none |
| `pipeline.archive.locks[].reason` | string | No | empty |
| `pipeline.publish.enabled` | bool | No | `true` |
| `pipeline.publish.upload_run` | bool | No | `true` |
| `pipeline.publish.outputs[]` | list | No | one final-trimmed output rule |
| `pipeline.publish.outputs[].source` | string | Yes (per rule) | none |
| `pipeline.publish.outputs[].dest` | string | No | derived from source |
| `pipeline.publish.outputs[].required` | bool | No | `true` |
| `pipeline.publish.locks[]` | list | No | empty |
| `pipeline.publish.locks[].source` | string | Yes (per lock) | none |
| `pipeline.publish.locks[].reason` | string | No | empty |
| `pipeline.whisperx.transcribe_url` | string | Yes | none |
| `pipeline.whisperx.language` | string | No | `en` |
| `pipeline.whisperx.timeout` | duration string | No | `30m` |
@@ -364,162 +190,38 @@ Operational notes:
| `pipeline.scriptorium.timeout` | duration string | No | `10m` |
| `pipeline.scriptorium.render_debug` | bool | No | `false` |
| `pipeline.scriptorium.artifacts` | map | No | empty |
| `pipeline.scriptorium.artifacts.<name>.enabled` | bool | No | `false` |
| `pipeline.scriptorium.artifacts.<name>.depends_on[]` | list[string] | No | empty |
| `pipeline.scriptorium.artifacts.<name>.render_debug` | bool | No | unset |
| `pipeline.scriptorium.artifacts.<name>.prompt_id` | string | Conditional | none |
| `pipeline.scriptorium.artifacts.<name>.profile_id` | string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.output_path` | string | Conditional | none |
| `pipeline.scriptorium.artifacts.<name>.timeout` | duration string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` | string | Conditional | none |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.artifact` | string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.path` | string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required` | bool | No | `false` |
| `pipeline.scriptorium.artifacts.<name>.vars.<key>` | map value | No | empty |
| `pipeline.notification.backend` | string | No | empty |
| `pipeline.notification.recipient` | string | No | empty |
| `pipeline.notification.timeout` | duration string | No | empty |
| `pipeline.notification.timeout` | duration string | No | `30s` |
Scriptorium artifact-key and dependency rules:
### Campaign
| Path | Type | Required |
| --- | --- | --- |
| `campaign_id` | string | Yes |
| `session_template_file` | string | No |
| `inputs.speakers_file` | string | Yes |
| `inputs.autocorrect_file` | string | Yes |
| `inputs.glossary_file` | string | Yes |
- artifact keys must match `^[a-z][a-z0-9_]*$`.
- enabled artifacts require `prompt_id` and `output_path`.
- `output_path` must be relative, traversal-safe, and under `artifacts/`.
- configured artifact input sources use `narratio.artifact.<name>`.
- if input source references `narratio.artifact.<name>`, artifact `<name>` must exist and must be listed in `depends_on`.
- every `depends_on` entry must be a configured artifact key.
- self-dependency is rejected.
- enabled dependency cycles are rejected.
- any artifact referenced by `depends_on` or `narratio.artifact.<name>` source must define `output_path` (even if not enabled).
Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
- `narratio.previous_session.artifact.<configured_artifact_key>`
- `narratio.transcript.base`
- `narratio.transcript.polished`
- `narratio.transcript.final`
- `narratio.transcript.final_trimmed`
- `narratio.bounds.session`
- `narratio.artifact.<configured_artifact_key>`
`pipeline.archive.promote_artifacts[].source` values:
- `narratio.transcript.base`
- `narratio.transcript.polished`
- `narratio.transcript.final`
- `narratio.transcript.final_trimmed`
- `narratio.bounds.session`
- `narratio.artifact.<configured_artifact_key>`
`pipeline.archive.locks[].source` accepts the same source values as `pipeline.archive.promote_artifacts[].source`.
Archive promotion destination rules:
- `dest` must be a clean relative path (not absolute, no traversal).
- duplicate `dest` values are rejected.
- if `dest` is omitted:
- built-in sources derive their canonical destination path;
- configured sources derive from `pipeline.scriptorium.artifacts.<name>.output_path`;
- derivation failure is a config validation error.
Archive lock rules:
- locks are source-based and do not accept `dest`.
- duplicate lock sources are rejected.
- static `pipeline.archive.locks` win over remote mutable locks for the same source.
- locked promotions are recorded as intentional skips in archive metadata.
- locked required promotions do not fail archive by default.
- ordinary `--force` reruns do not override locks.
Remote mutable lock store:
- path: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/locks.yml`.
- strict YAML shape: top-level `locks`, each with `source` and optional `reason`.
- `narratio session locks add` and `narratio session locks remove` mutate only the remote lock store.
- writes use existence checks plus `--force` for updates; they are not compare-and-swap atomic.
Restore-related implications:
- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs).
- restore scope considers committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, optional `audio/**`).
- S3 audio downloads use `pipeline.spool.root` for active downloads and `pipeline.cache.root` for reusable cached audio when `pipeline.cache.s3_audio` is true.
- `pipeline.cache.root` is durable local cache state. It is not workspace state and is preserved by default by `narratio clean`.
## 7. Full campaign reference
| Path | Type | Required | Default |
| --- | --- | --- | --- |
| `campaign.campaign_id` | string | Yes | none |
| `campaign.session_template_file` | string | No | none |
| `campaign.inputs.speakers_file` | string | Yes | none |
| `campaign.inputs.autocorrect_file` | string | Yes | none |
| `campaign.inputs.glossary_file` | string | Yes | none |
Campaign input paths and `campaign.session_template_file` may be absolute or relative. Relative paths resolve from the directory containing `campaign.yml`.
## 8. Full session reference
| Path | Type | Required | Default |
| --- | --- | --- | --- |
| `session.session_id` | string | Yes | none |
| `session.previous_session_id` | string | No | empty |
| `session.campaign` | string | No | `campaign.campaign_id` |
| `session.date` | string | No | empty |
| `session.title` | string | No | empty |
| `session.inputs.audio_dir` | string | Conditional | empty |
| `session.inputs.audio_files[]` | list[string] | Conditional | empty |
| `session.inputs.audio_s3.prefix` | string | Conditional | none |
| `session.inputs.speakers_file` | string | No | `campaign.inputs.speakers_file` |
| `session.inputs.autocorrect_file` | string | No | `campaign.inputs.autocorrect_file` |
| `session.inputs.glossary_file` | string | No | `campaign.inputs.glossary_file` |
Session input paths may be absolute or relative. Relative audio paths and session-level stable input overrides resolve from the directory containing `session.yml`. If both `campaign.yml` and `session.yml` specify campaign identity, the values must match.
Audio-source rule:
- configure exactly one mode:
- `audio_dir`, or
- `audio_files` (at least one), or
- `audio_s3.prefix`
- `audio_s3` cannot be combined with local audio fields.
Previous-session rule:
- if `session.previous_session_id` is set, it must not equal `session.session_id`.
- canonical previous-session sources (`narratio.previous_session.artifact.<name>`) are hydrated during `prepare` from archive current state when required by enabled configured artifacts.
## 9. Secrets
Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`.
Behavior:
- `env_dir` may be absolute or relative.
- relative `env_dir` resolves from current working directory.
- files with valid env-var names (`[A-Za-z_][A-Za-z0-9_]*`) are loaded.
- values are loaded from file contents with trailing newline trimming.
- existing process env vars are preserved.
- invalid names and subdirectories are skipped.
- missing/unreadable `env_dir` fails command execution.
Guidance:
- do not put secret values directly in YAML.
- configure env var names in config and provide values via env/secrets files.
## 10. Examples
Maintained examples:
### Session
| Path | Type | Required |
| --- | --- | --- |
| `session_id` | string | Yes |
| `previous_session_id` | string | No |
| `campaign` | string | No |
| `date` | string | No |
| `title` | string | No |
| `inputs.speakers_file` | string | No |
| `inputs.autocorrect_file` | string | No |
| `inputs.glossary_file` | string | No |
| `inputs.audio_dir` | string | Conditional |
| `inputs.audio_files[]` | list[string] | Conditional |
| `inputs.audio_s3.prefix` | string | Conditional |
## Maintained Examples
- `examples/pipeline.minimal.yml`
- `examples/pipeline.production.yml`
- `examples/pipeline.full.annotated.yml`
- `examples/campaigns/sample-campaign/campaign.yml`
- `examples/campaigns/sample-campaign/speakers.yml`
- `examples/campaigns/sample-campaign/autocorrect.yml`
- `examples/campaigns/sample-campaign/glossary.yml`
- `examples/campaigns/sample-campaign/session.template.yml`
- `examples/session.local-audio.yml`
- `examples/session.s3-audio.yml`
These examples are validated by `internal/config` tests.

2
docs/development.md
View File

@@ -75,7 +75,7 @@ Remote-storage commands must obtain object storage through the app-level command
1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
2. Keep external transport/subprocess details in `internal/adapters`.
3. Preserve manifest and promotion semantics expected by runner and archive logic.
3. Preserve manifest and publish-output semantics expected by runner and publish logic.
4. Add/update stage and adapter tests.
5. Update internal component contracts in `docs/internal/`.

2
docs/integrations/seriatim.md
View File

@@ -20,7 +20,7 @@ Owns:
- JSON output validation
Does not own:
- Transcript input selection/promotion logic (stage-owned)
- Transcript input selection/materialization logic (stage-owned)
- Bounds computation (scriptorium/trim-stage-owned)
## Config Fields Used

4
docs/internal/README.md
View File

@@ -11,7 +11,7 @@ Implementation-accurate contracts for workspace/state, manifests, stages, artifa
- `storage.md`: remote storage backend contracts and object-store invariants.
- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics.
- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior.
- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants.
- `workspace.md`: local state model, manifests, run-local layout, materialization, and cleanup invariants.
- `command-restore.md`: restore command discovery/planning/execution/reporting contract.
- `stage-prepare.md`: input materialization and provenance capture.
- `stage-transcribe.md`: WhisperX transcript generation.
@@ -20,7 +20,7 @@ Implementation-accurate contracts for workspace/state, manifests, stages, artifa
- `stage-normalize.md`: post-polish normalization.
- `stage-trim.md`: bounds-driven transcript trimming.
- `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts.
- `stage-archive.md`: archive upload and current-pointer publish contract.
- `stage-publish.md`: publish upload and current-pointer commit contract.
## External Integration Notes
- `../integrations/README.md`: canonical location for external integration contracts (`audita.md`, `seriatim.md`, `scriptorium.md`).

2
docs/internal/adapters.md
View File

@@ -28,7 +28,7 @@ Default wiring and adapter calls consume:
- `pipeline.seriatim.*`
- `pipeline.audita.*`
- `pipeline.scriptorium.*`
- `pipeline.storage.*` and `pipeline.archive.*` (object-store construction/gating)
- `pipeline.storage.*` and `pipeline.publish.*` (object-store construction/gating)
- `pipeline.notification.*` (sender boundary exists; placeholder behavior today)
## External adapters used

8
docs/internal/artifacts.md
View File

@@ -15,8 +15,8 @@ Inputs:
Outputs:
- resolved artifact path + provenance (`ResolvedSessionArtifact`);
- runtime catalog entries for built-ins and configured artifacts;
- requirement sets for canonical previous-session inputs.
- canonical S3 session, run, current, session config, session locks, audio, and promoted artifact keys.
- requirement sets for canonical previous-session inputs;
- canonical S3 session, run, current, session config, session locks, audio, and published output keys.
## Boundaries
Owns:
@@ -28,7 +28,7 @@ Owns:
Does not own:
- prepare-stage remote hydration;
- stage success/skip transitions;
- archive upload orchestration.
- publish upload orchestration.
## Built-in IDs
| Artifact ID | Canonical file | Producer stage | Output kind |
@@ -71,7 +71,7 @@ Previous-session canonical provenance values include:
- Built-ins resolve via manifest producer outputs first, then canonical fallback paths.
- Configured `narratio.artifact.<name>` sources resolve through catalog availability.
- Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths.
- Archive-relative configured artifact paths under `artifacts/` are cached without a redundant nested `artifacts/` segment.
- Publish-relative configured artifact paths under `artifacts/` are cached without a redundant nested `artifacts/` segment.
- Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`.
## Previous-session requirement scanning

10
docs/internal/command-restore.md
View File

@@ -27,14 +27,14 @@ Owns:
Does not own:
- Stage execution orchestration (`run`, `resume`, `run-stage`).
- Archive publish behavior (owned by archive stage).
- Publish-stage behavior.
- Storage transport implementation details (owned by storage adapters).
## Config fields used
- Config/session discovery and templating fields consumed by all commands.
- `pipeline.workspace.root` (local restore target root).
- `pipeline.storage.*` (remote backend + archive identity derivation).
- `pipeline.storage.s3.*` identity components used by archive prefix helpers.
- `pipeline.storage.*` (remote backend + publish identity derivation).
- `pipeline.storage.s3.*` identity components used by session-prefix helpers.
- `pipeline.spool.root` for active audio downloads.
- `pipeline.cache.root` and `pipeline.cache.s3_audio` for reusable S3 audio cache.
- `session.session_id`
@@ -80,7 +80,7 @@ Restore path scope:
- Dry-run is read-only and returns plan output only.
## Failure behavior
- Fails when storage backend is unavailable or archive identity cannot be resolved.
- Fails when storage backend is unavailable or publish identity cannot be resolved.
- Fails when remote current pointer/manifest is missing or invalid.
- Fails when remote manifest identity mismatches requested campaign/session.
- Fails on local conflicts unless `--force` is set.
@@ -96,7 +96,7 @@ Restore path scope:
- `internal/artifacts/archive_identity_test.go`
## Architectural invariants
- Restore relies on centralized archive identity/key helpers (`internal/artifacts`) rather than ad hoc key building.
- Restore relies on centralized path/key helpers (`internal/artifacts`) rather than ad hoc key building.
- `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files.
- Local path mapping is traversal-safe and constrained to session root.
- Restore scope is deterministic and path-classified:

2
docs/internal/manifest.md
View File

@@ -30,7 +30,7 @@ Manifest identity fields are populated by app/stage orchestration from:
- `session.session_id`
- `session.campaign`
- `pipeline.workspace.root`
- `pipeline.storage.s3.*` (when archive/S3 identity is set)
- `pipeline.storage.s3.*` (when publish/S3 identity is set)
## External adapters used
- No external service adapters.

10
docs/internal/stage-analyze.md
View File

@@ -1,7 +1,7 @@
# Stage: analyze
## Purpose
Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths.
Execute selected configured Scriptorium artifacts in deterministic dependency order and materialize successful outputs to canonical session artifact paths.
## Inputs and outputs
Inputs:
@@ -15,7 +15,7 @@ Source types used by analyze:
- canonical previous-session artifacts: `narratio.previous_session.artifact.<artifact_key>`.
Outputs:
- promoted configured artifact files at each configured `output_path`;
- materialized configured artifact files at each configured `output_path`;
- stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info).
## Boundaries
@@ -24,12 +24,12 @@ Owns:
- selected-artifact planning and dependency ordering;
- per-input resolution and required/optional handling;
- Scriptorium render/run invocation;
- run-local output generation and canonical promotion.
- run-local output generation and canonical materialization.
Does not own:
- prepare-time previous-session hydration;
- object-store access for previous-session sources;
- archive promotion policy.
- publish output rule behavior.
## Config fields used
- `session.session_id`
@@ -76,5 +76,5 @@ Does not own:
## Architectural invariants
- Canonical previous-session behavior is local-cache only during analyze.
- Generated outputs are validated and promoted before stage success is recorded.
- Generated outputs are validated and materialized before stage success is recorded.
- Resolver/catalog decisions stay deterministic and validation-gated.

6
docs/internal/stage-merge.md
View File

@@ -19,7 +19,7 @@ Owns:
- Per-input normalize calls to Seriatim
- Final merge call to Seriatim
- Run-local log/config/report path wiring
- Promotion of base/report outputs to canonical paths
- Materialization of base/report outputs to canonical paths
Does not own:
- Transcript polishing or downstream artifact generation
@@ -43,7 +43,7 @@ Does not own:
## State and Manifest Behavior
- Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory.
- Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled.
- Promotes canonical base transcript and optional report.
- Materializes canonical base transcript and optional report.
- Records normalized-input provenance and adapter metadata in stage metadata.
## Skip and Resume Behavior
@@ -59,5 +59,5 @@ Does not own:
## Architectural Invariants
- Merge consumes normalized forms of each raw transcript.
- Base transcript must validate before promotion.
- Base transcript must validate before materialization.
- Report output is optional and gated by config.

14
docs/internal/stage-prepare.md
View File

@@ -17,7 +17,7 @@ Inputs:
- local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
- S3: `session.inputs.audio_s3.prefix`;
- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning);
- remote previous-session current archive state when previous hydration is required.
- remote previous-session current publish state when previous hydration is required.
Outputs:
- `inputs/campaign.yml`;
@@ -41,7 +41,7 @@ Owns:
Does not own:
- transcript or artifact generation;
- analyze-stage source resolution;
- archive commit behavior.
- publish commit behavior.
## Config fields used
- `session.session_id`
@@ -83,10 +83,10 @@ Does not own:
- `narratio.previous_session.artifact.<artifact_key>`
- If one or more canonical previous-session requirements exist:
- clears managed `previous/` state;
- hydrates required/optional previous artifacts from the configured previous session’s committed archive current state;
- hydrates required/optional previous artifacts from the configured previous session’s committed publish current state;
- writes `previous/manifest.json` and hydrated `previous/artifacts/**`;
- stores archive-relative artifact paths such as `artifacts/session_recap.md` as `previous/artifacts/session_recap.md`, not `previous/artifacts/artifacts/session_recap.md`;
- records hydrated previous inputs in `manifest.Inputs` with source `previous_session_archive.current`.
- stores publish-relative artifact paths such as `artifacts/session_recap.md` as `previous/artifacts/session_recap.md`, not `previous/artifacts/artifacts/session_recap.md`;
- records hydrated previous inputs in `manifest.Inputs` with source `previous_session_publish.current`.
- If no canonical previous-session requirements exist, prepare does not manage `previous/`.
- `manifest.Inputs` is sorted deterministically by `(kind, path)`.
- S3 audio `manifest.Inputs` retain S3 provenance and include `cache_path`; `spool_path` is present only when the current prepare invocation downloaded the file.
@@ -95,7 +95,7 @@ Does not own:
- `previous_session_id` unset:
- if any referenced previous artifact is required: fail;
- if all referenced previous artifacts are optional: continue and omit them.
- Previous session archive current pointer or manifest missing:
- Previous session publish current pointer or manifest missing:
- if any referenced previous artifact is required: fail;
- if all referenced previous artifacts are optional: continue and omit missing ones.
- Missing required previous artifact object: fail.
@@ -120,5 +120,5 @@ Does not own:
## Architectural invariants
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
- Storage keys are computed by callers using archive/path helpers; storage adapter receives explicit keys.
- Storage keys are computed by callers using path helpers; storage adapter receives explicit keys.
- `prepare` is the only stage that hydrates canonical previous-session cache state.

62
docs/internal/stage-archive.md → docs/internal/stage-publish.md
View File

@@ -1,4 +1,4 @@
# Stage: archive
# Stage: publish
## Purpose
Publish durable run/session state to object storage, then atomically advance remote current state.
@@ -7,37 +7,37 @@ Publish durable run/session state to object storage, then atomically advance rem
Inputs:
- session manifest and prerequisite stage records
- run root contents under `runs/{run_id}/`
- promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`)
- effective source-based promotion locks from static config and remote session lock store
- publish output rules with artifact `source` IDs and publish `dest` paths (`pipeline.publish.outputs`)
- effective source-based publish locks from static config and remote session lock store
- session-level `previous/**` cache files when present
Outputs:
- uploaded run files under `{session_prefix}/runs/{run_id}/...`
- uploaded promoted artifacts under `{session_prefix}/...`
- uploaded published outputs under `{session_prefix}/...`
- uploaded session previous-cache files under `{session_prefix}/previous/...` when present
- `{session_prefix}/current/manifest.json`
- `{session_prefix}/current/run_id.txt` written last
## Boundaries
Owns:
- Archive enable/disable gate behavior
- Prerequisite stage success enforcement
- Run file collection and upload (excluding `audio/`)
- Promotion rule resolution and upload
- Promotion lock enforcement
- Session previous-cache file collection/upload
- Commit pointer publish order
- publish enable/disable gate behavior
- prerequisite stage success enforcement
- run file collection and upload (excluding `audio/`)
- publish output rule resolution and upload
- publish lock enforcement
- session previous-cache file collection/upload
- commit pointer publish order
Does not own:
- Stage execution before archive
- Post-archive local cleanup policy execution (handled by app cleanup logic)
- stage execution before publish
- post-publish local cleanup policy execution (handled by app cleanup logic)
## Config Fields Used
- `pipeline.archive.enabled`
- `pipeline.archive.upload_run`
- `pipeline.archive.promote_artifacts`
- `pipeline.archive.locks`
- `{session_prefix}/locks.yml` loaded by app orchestration before archive execution
- `pipeline.publish.enabled`
- `pipeline.publish.upload_run`
- `pipeline.publish.outputs`
- `pipeline.publish.locks`
- `{session_prefix}/locks.yml` loaded by app orchestration before publish execution
- `pipeline.storage.s3.bucket`
- `pipeline.storage.s3.root_prefix`
- `pipeline.workspace.root`
@@ -51,26 +51,28 @@ Does not own:
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
- Resolves bucket/prefix from manifest identity first, then config fallback.
- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
- Skips top-level promotion uploads for effective locked sources; run-local uploads still publish.
- When selected configured artifact keys are supplied, skips promotion rules for unselected `narratio.artifact.<key>` sources; built-in transcript and bounds promotions still publish.
- Effective locks are the union of `pipeline.archive.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources.
- Skips top-level published output uploads for effective locked sources; run-local materialized outputs remain unchanged.
- When selected configured artifact keys are supplied, skips publish rules for unselected `narratio.artifact.<key>` sources; built-in transcript and bounds outputs still publish.
- Effective locks are the union of `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources.
- Writes metadata including:
- upload counts/paths
- `previous_files_uploaded` and `previous_uploaded_paths`
- `skipped_unselected_promotions`
- `locked_promotion_count` and `locked_promotions`
- `published_files_uploaded` and `published_paths`
- `skipped_optional_outputs`
- `skipped_unselected_outputs`
- `locked_output_count` and `locked_outputs`
- `current_manifest_key`
- `current_run_id_key`
- `current_pointer_written`
- On skipped archive path, returns metadata with `skipped=true` and pointer not written.
- On skipped publish path, returns metadata with `skipped=true` and pointer not written.
## Skip and Resume Behavior
- Stage may self-skip (metadata skip) when archive disabled or run upload disabled.
- Stage may self-skip (metadata skip) when publish disabled or run upload disabled.
- Runner-level skip also applies for previously succeeded stage unless forced.
## Failure Behavior
- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required promotion source, upload failures, or pointer write failures.
- Locked required promotions are intentional skips and do not fail archive.
- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required output source, upload failures, or pointer write failures.
- Locked required outputs are intentional skips and do not fail publish.
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
## Tests to Inspect Before Changing
@@ -79,8 +81,8 @@ Does not own:
## Architectural Invariants
- Run upload excludes `audio/` subtree.
- Session `previous/**` is archiveable durable input/provenance state, not run-local output.
- Ordinary `--force` does not override archive locks.
- Malformed or unreadable remote lock store fails archive-capable execution before promotion.
- Session `previous/**` is publishable durable input/provenance state, not run-local output.
- Ordinary `--force` does not override publish locks.
- Malformed or unreadable remote lock store fails publish-capable execution before output uploads.
- `current/manifest.json` uploads before `current/run_id.txt`.
- `current/run_id.txt` is the remote publish commit marker.

8
docs/internal/stage-transcribe.md
View File

@@ -15,7 +15,7 @@ Owns:
- Discovering prepared audio inputs
- Deriving speaker ids from audio basenames
- Parallel WhisperX invocation with bounded concurrency
- Validating produced JSON and promoting run-local outputs
- Validating produced JSON and materializing run-local outputs
Does not own:
- Transcript merge/polish/normalize/trim/analyze
@@ -36,8 +36,8 @@ Does not own:
## State and Manifest Behavior
- Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled.
- Validates each generated transcript JSON before promotion.
- Promotes canonical outputs to `transcripts/raw/*.json`.
- Validates each generated transcript JSON before materialization.
- Materializes canonical outputs to `transcripts/raw/*.json`.
- Records per-file metadata (attempts/status/duration/output path) in stage metadata.
## Skip and Resume Behavior
@@ -54,5 +54,5 @@ Does not own:
## Architectural Invariants
- Speaker identity is derived from `.flac` basename and must be unique.
- Every successful speaker output must be valid JSON before promotion.
- Every successful speaker output must be valid JSON before materialization.
- Canonical raw transcript set is the only supported merge input surface.

2
docs/internal/stage-trim.md
View File

@@ -52,7 +52,7 @@ Does not own:
## State and Manifest Behavior
- Reads final transcript from normalize manifest outputs when available; falls back to canonical path.
- Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled.
- Promotes canonical final-trimmed transcript; promotes session bounds when trim enabled.
- Materializes canonical final-trimmed transcript and session bounds when trim is enabled.
- Records bounds diagnostics, trim action, keep selector, and adapter metadata.
## Skip and Resume Behavior

8
docs/internal/storage.md
View File

@@ -37,12 +37,12 @@ Does not own:
## External adapters used
Storage package contracts:
- `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`.
- `Backend` (archive request boundary): currently implemented with `NoopBackend` only.
- `Backend` (legacy compatibility boundary): currently implemented with `NoopBackend` only.
Implementations:
- `S3Backend`: AWS SDK-backed `ObjectStore` implementation.
- `FakeBackend`: deterministic test `ObjectStore` and archive backend.
- `NoopBackend`: deterministic no-op archive backend for compatibility wiring.
- `FakeBackend`: deterministic test `ObjectStore` and compatibility backend.
- `NoopBackend`: deterministic no-op compatibility backend for wiring/tests.
## State and manifest behavior
- Storage implementations are stateless with respect to manifest/session lifecycle.
@@ -67,7 +67,7 @@ Implementations:
- `internal/adapters/storage/s3_backend_test.go`
- `internal/adapters/storage/fake_test.go`
- `internal/adapters/storage/keys_test.go`
- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `archive`)
- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `publish`)
## Architectural invariants
- Callers pass full bucket-relative keys.

18
docs/internal/workspace.md
View File

@@ -1,7 +1,7 @@
# Workspace internals
## Purpose
Define the local durable and run-local workspace model used by stages, manifests, resume, and archive.
Define the local durable and run-local workspace model used by stages, manifests, resume, and publish.
## Inputs and Outputs
Inputs:
@@ -24,14 +24,14 @@ Owns:
Does not own:
- Stage business logic
- Remote archive semantics (documented in `stage-archive.md`)
- Remote publish semantics (documented in `stage-publish.md`)
- CLI argument parsing
## Config Fields Used
- `pipeline.workspace.root`
- `pipeline.workspace.cleanup_after_archive`
- `pipeline.workspace.cleanup_after_publish`
- `pipeline.spool.root`
- `pipeline.spool.delete_audio_after_archive`
- `pipeline.spool.delete_audio_after_publish`
- `pipeline.cache.root`
- `pipeline.cache.s3_audio`
- `session.campaign`
@@ -43,10 +43,10 @@ None directly in this subsystem. Stages may use object storage adapters and then
## State and Manifest Behavior
- Session state is persisted in the session manifest (`manifest.Manifest`).
- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`.
- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and promoted to canonical session paths after stage success.
- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and then materialized to canonical session paths after stage success.
- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs.
- For S3 audio sessions, `prepare` records work/cache paths, S3 provenance, and spool path when the invocation downloaded the object.
- `previous/**` is reconstructed from configured previous-session requirements; restore uses the previous session's committed current archive rather than treating current-session archived `previous/**` as authoritative.
- `previous/**` is reconstructed from configured previous-session requirements; restore uses the previous session's committed current publish state rather than treating current-session stored `previous/**` as authoritative.
- Durable cache state under `pipeline.cache.root` is not workspace state and is preserved by default by `narratio clean`.
- `narratio clean <id>` removes the session work root and session spool root.
- `narratio clean --all` removes all local session work under `workspace.root/work` and spool children under `spool.root`.
@@ -60,7 +60,7 @@ None directly in this subsystem. Stages may use object storage adapters and then
## Failure Behavior
- Failures preserve manifests and run-local files for inspection.
- Lock conflicts fail fast via `ErrLockConflict`.
- Cleanup can fail post-archive; failure is recorded in archive stage metadata and returned by the run.
- Cleanup can fail post-publish; failure is recorded in publish stage metadata and returned by the run.
## Tests to Inspect Before Changing
- `internal/artifacts/local_test.go`
@@ -72,7 +72,7 @@ None directly in this subsystem. Stages may use object storage adapters and then
## Architectural Invariants
- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`.
- Run roots are always nested: `runs/{run_id}` under the session root.
- Run-local output promotion must end in canonical session paths.
- Run-local output materialization must end in canonical session paths.
- `previous/**` is session-durable state and must not be treated as run-local output scratch state.
- Automatic post-archive cleanup only targets run-scoped directories and must never delete configured root directories.
- Automatic post-publish cleanup only targets run-scoped directories and must never delete configured root directories.
- Manual `clean` may delete session-scoped directories or the `workspace.root/work` directory, but it must preserve configured root directories and reject unsafe targets.

401
docs/operations.md
View File

@@ -1,285 +1,170 @@
# Operations
This guide describes the implemented operator lifecycle for Narratio.
This guide covers the implemented operator lifecycle for Narratio.
For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md).
For field-level settings, see [docs/config.md](./config.md). For syntax/flags, see [docs/cli.md](./cli.md).
## Normal workflow (S3-first path)
## Normal Workflow
1. Create or upload `session.yml`, or pass a local `session.yml` explicitly.
2. Upload session `.flac` files to object storage under the configured session audio prefix.
3. Run Narratio:
1. Ensure `pipeline.yml`, `campaign.yml`, and `session.yml` are available.
2. Ensure session audio is available (local `audio_dir`/`audio_files` or S3 prefix).
3. Run:
```bash
narratio run 2026-04-04
```
4. Read success output:
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
- use `narratio session status <session_id>` for inspection.
Notes:
- default pipeline/session discovery checks system config locations; campaign selection uses `pipeline.campaigns.default_campaign_id` unless `--campaign <id>` or `--campaign-file <path>` is passed.
- when local `session.yml` discovery misses, positional `<session_id>` loads remote `session.yml` from `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
Initialize a remote session skeleton:
```bash
narratio session init 2026-04-04 --remote
```
Remote init uses normal default config discovery and writes `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`. If `campaign.yml` sets `session_template_file`, init renders that template from the supplied flags and writes concrete YAML. Pass `--config`, `--campaign <id>`, or `--campaign-file <path>` when testing non-system config files. It fails if the object already exists unless `--force` is passed.
Validate before running:
```bash
narratio session validate 2026-04-04
```
## Restore workflow
Use restore when local durable session state is missing or stale and archive current state is authoritative.
Dry-run (no local writes):
```bash
narratio session restore 2026-04-04 --dry-run
```
Execution:
```bash
narratio session restore 2026-04-04
```
Post-restore analyze rerun pattern:
```bash
narratio analyze 2026-04-04
```
Restore source-of-truth:
- remote commit marker: `current/run_id.txt`
- remote current manifest: `current/manifest.json`
- configured previous-session requirements are reconstructed from the previous session's remote `current/` state, not from archived `previous/**` objects in the current session.
Restore default scope:
- includes `manifest.json`, `transcripts/**`, and `artifacts/**` from the current session archive.
- includes `previous/**` only when configured previous-session artifact inputs require it; restore hydrates those files the same way `prepare` would.
- includes `audio/**` only with `--include-audio`
- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source)
Reset local state before restore testing:
```bash
narratio clean 2026-04-04 --dry-run
narratio clean 2026-04-04
narratio session restore 2026-04-04 --include-audio
```
`clean` removes the local session work directory and session spool directory. It preserves the durable S3 audio cache by default, so repeated restore or forced prepare tests do not re-download large audio files.
## Local filesystem layout and state artifacts
Session root:
- `{workspace.root}/work/{campaign}/{session_id}/`
Primary state:
- `manifest.json`: session-level stage state.
- `runs/{run_id}/manifest.json`: invocation-level state.
- `.lock`: session lock while a modifying command is active.
- `inputs/campaign.yml`, `inputs/session.yml`, and `inputs/pipeline.resolved.yml`: materialized config inputs for the run.
Canonical session directories:
- `inputs/`
- `audio/`
- `transcripts/`
- `artifacts/`
- `previous/`
- `reports/`
- `logs/`
- `config/`
- `current/`
- `runs/`
Run-local stage directories:
- `runs/{run_id}/{stage}/` with stage-local `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`.
Behavior:
- directory creation is idempotent.
- stage outputs are generally generated run-local first, then promoted to canonical paths on success.
- restore installs downloaded files to canonical session paths and does not recreate historical run sandboxes.
## Analyze artifact execution lifecycle
Analyze executes configured artifacts from `pipeline.scriptorium.artifacts`.
Execution model:
- executable set = enabled artifacts, filtered by `--artifacts` when provided.
- artifact-to-artifact dependencies are declared via `depends_on`.
- selected artifacts run in deterministic dependency order.
- after each successful artifact run, output is promoted to configured canonical `output_path`.
Configured artifact source reuse:
- a non-executable configured artifact can satisfy inputs if its configured output file already exists and is valid.
- reused configured artifact provenance is `filesystem.disabled_artifact_output`.
`--artifacts` behavior:
- accepted on `run`, `resume`, `run-stage analyze`, `run-stage archive`, `analyze`, and `publish`.
- filters analyze execution and configured artifact promotions.
- built-in transcript and bounds promotions are not filtered.
- does not imply force on `run`, `resume`, or `run-stage`; `narratio analyze` is force-by-design.
- `publish` is force-by-design and accepts `--artifacts` for configured artifact promotions.
Canonical previous-session input behavior:
- canonical sources use `narratio.previous_session.artifact.<artifact_key>`.
- these inputs are hydrated by `prepare` and by `restore`; `analyze` expects the local previous cache to already exist.
- if analyze fails due to missing canonical previous cache, rerun:
- `narratio run-stage prepare <id> --force`
- or `narratio session restore <id>` when remote archive current state is authoritative.
## Remote archive layout and publish contract
Preferred manual publish command:
```bash
narratio publish <id>
```
`publish` is equivalent to `narratio run-stage archive <id> --force`; use `run-stage` when you need the general single-stage command form.
When archive is enabled and run upload is enabled, archive publishes under:
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
- run prefix: `{session_prefix}/runs/{run_id}/`
Archive uploads:
- run record files from run root (excluding `audio/`).
- promoted files from explicit `archive.promote_artifacts` rules.
- mutable session locks from helper commands live at `{session_prefix}/locks.yml`.
Publish order:
1. upload `current/manifest.json`
2. upload `current/run_id.txt` last
`current/run_id.txt` is the remote commit marker.
Archive promotion is explicit and source-based:
- Narratio does not auto-promote all generated analyze artifacts.
- each rule resolves `source` through the artifact resolver/catalog model, then uploads to `dest`.
- missing required promotion sources fail archive stage.
- missing optional promotion sources are skipped.
- invalid resolved artifacts fail archive stage.
- `archive.locks` skips top-level promotion overwrites for locked sources while run-local uploads still publish.
- remote locks from `{session_prefix}/locks.yml` are merged with static `archive.locks`; static locks win on duplicate sources.
- locked required promotions are treated as intentional successful skips and are recorded in archive metadata.
Lock helper behavior:
- `narratio session locks <id>` lists effective static and remote locks.
- `narratio session locks add <id> <source> --reason <text>` writes a remote lock.
- `narratio session locks add <id> <source> --force --reason <text>` updates an existing remote lock reason.
- `narratio session locks remove <id> <source>` removes only a remote lock.
- `locks remove` cannot remove static pipeline locks.
- remote lock writes check whether the lock store exists, but are not compare-and-swap atomic.
## Resume, retry, restore, and safe rerun behavior
Default skip:
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set.
Resume:
- `resume` starts at first non-succeeded stage.
- `resume --force` runs full stage order.
Restore conflict policy:
- restore classifies local differences as conflicts.
- without `--force`, restore fails when conflicts exist.
- with `--force`, conflicting local files are overwritten by remote archive files.
Forced reruns:
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
- ordinary `--force` does not override archive locks.
Safe rerun pattern:
1. rerun the changed stage with `--force`.
2. run `resume` to rebuild downstream stages.
## Cleanup behavior
Automatic post-archive cleanup is considered only when archive stage executed and succeeded.
Automatic cleanup toggles:
- `pipeline.spool.delete_audio_after_archive=true` deletes run-scoped spool audio.
- `pipeline.workspace.cleanup_after_archive=true` deletes run-scoped local run directory.
Manual cleanup:
- `narratio clean <id>` deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`.
- `narratio clean --all` deletes all local session work under `{workspace.root}/work` and all spool children under `{spool.root}`.
- `--dry-run` prints targets without deleting.
- `--clear-cache` also removes matching S3 audio cache files. Without it, cache is preserved.
The S3 audio cache under `pipeline.cache.root` is durable input cache state, not workspace or spool state. Automatic cleanup and default manual cleanup do not delete it.
Cleanup eligibility gates:
- archive enabled
- archive run upload enabled
- run record upload completed
- current pointer write completed (`current/run_id.txt` written)
No cleanup for failed/incomplete/unarchived/archive-skipped runs.
## Failure and recovery playbooks
After run failure, Narratio keeps:
- session manifest
- run manifest
- run-local artifacts/logs/config/reports
Failed or incomplete runs remain local-only.
After restore failure:
- already-installed restore files remain in place.
- restore does not roll back prior successful installs.
- existing local manifest is preserved if restored manifest validation/install fails.
Recommended recovery:
1. inspect state:
4. Inspect status:
```bash
narratio session status 2026-04-04
```
This reports local manifest state, committed remote current state, expected remote transcript/artifact availability, and archive locks.
## Publish Workflow
2. for restore-specific checks, run:
Publish is the stage that commits remote current state.
```bash
narratio publish 2026-04-04
```
Equivalent command:
```bash
narratio run-stage publish 2026-04-04 --force
```
Publish uploads:
- run history files under `{session_prefix}/runs/{run_id}/` (excluding `audio/`)
- configured published outputs from `pipeline.publish.outputs`
- `previous/**` cache files when present
- `current/manifest.json`
- `current/run_id.txt` last
`current/run_id.txt` is the remote commit marker.
## Published Outputs and Locks
Published output behavior:
- outputs are source-based rules in `pipeline.publish.outputs`.
- required missing unlocked sources fail publish.
- optional missing unlocked sources are skipped.
- selected artifacts (`--artifacts`) only filter configured `narratio.artifact.<key>` output rules.
- built-in transcript and bounds output rules are not filtered by `--artifacts`.
Lock behavior:
- static locks: `pipeline.publish.locks`.
- mutable locks: `{session_prefix}/locks.yml`.
- effective lock set is static + mutable; static wins on duplicate sources.
- locked outputs are intentional skips and do not fail publish.
- lock commands mutate only remote mutable locks.
## Restore Workflow
Use restore when local durable session state is missing/stale and committed remote current state is authoritative.
Preview:
```bash
narratio session restore 2026-04-04 --dry-run
```
3. fix root cause (config/input/credentials/storage/service availability).
4. continue with `resume`, or targeted `run-stage <stage> <id> --force` followed by `resume`.
Apply:
## Restore report
```bash
narratio session restore 2026-04-04
```
Non-dry-run restore writes a durable report at:
- `reports/restore-latest.json`
Default restore scope:
Report content includes:
- identity (`campaign`, `session_id`, `run_id`)
- mode flags (`dry_run`, `force`, `include_audio`)
- plan counts and execution counts
- per-action status
- `manifest.json`
- `transcripts/**`
- `artifacts/**`
- `previous/**` when required by configured previous-session artifact inputs
Dry-run does not write restore report files.
Optional:
## Operational caveats
- add `--include-audio` to restore `audio/**`.
- `session status <session_id>` uses normal config/session loading, including remote session fallback.
- `session status <session_id>` includes the same promoted remote output availability view as `session artifacts <session_id> --remote` when storage is configured.
- local and S3 audio input modes are mutually exclusive.
- archive publish requires upstream stages through `analyze` to be `succeeded`.
- required configured artifact promotions for unselected `--artifacts` keys are skipped intentionally; selected required promotions still fail if their files are missing.
- restore requires configured remote object storage and committed remote current state.
Restore reads committed current state only (`current/run_id.txt`, `current/manifest.json`).
## Workspace and State Layout
Session root:
- `{workspace.root}/work/{campaign}/{session_id}/`
Durable session state:
- `manifest.json`
- `inputs/**`
- `audio/**`
- `transcripts/**`
- `artifacts/**`
- `previous/**`
- `reports/**`
- `logs/**`
- `config/**`
- `runs/**`
Run-local stage layout:
- `runs/{run_id}/{stage}/outputs|logs|reports|config|scratch`
Stages typically write run-local outputs first, then materialize canonical session outputs on success.
## Resume and Force Rules
- `run` and `run-stage` skip succeeded stages unless `--force` is set.
- `resume` starts at the first non-succeeded stage.
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
- `--force` does not bypass publish locks.
## Cleanup
Automatic post-publish cleanup is considered only when publish executes successfully and commits current state.
Config toggles:
- `pipeline.spool.delete_audio_after_publish=true`
- `pipeline.workspace.cleanup_after_publish=true`
Manual cleanup:
```bash
narratio clean 2026-04-04
narratio clean --all
```
Cache is preserved by default. Use `--clear-cache` to remove matching S3 audio cache entries.
## Failure and Recovery
After stage failure, Narratio keeps manifests and run-local files for inspection.
Standard recovery flow:
1. inspect status:
```bash
narratio session status 2026-04-04
```
2. if needed, inspect restore plan:
```bash
narratio session restore 2026-04-04 --dry-run
```
3. fix root cause.
4. continue with `resume`, or rerun a stage with `--force` then `resume`.
## Operational Caveats
- local and S3 audio modes are mutually exclusive.
- publish requires prerequisite stages through `analyze` to be `succeeded`.
- restore requires configured object storage and committed current state.
- `session status` and `session artifacts --remote` both report remote published-output availability when storage is configured.

356
docs/troubleshooting.md
View File

@@ -1,17 +1,17 @@
# Troubleshooting
## Purpose
Canonical operator troubleshooting guide for recurring implemented Narratio failures.
Canonical operator troubleshooting guide for recurring Narratio failures.
## Config file discovery failure
## Config discovery failure
Symptom:
- `run`, `resume`, `run-stage`, `session plan`, or `session restore` fails with config/session not found.
- command fails because `pipeline.yml`, `campaign.yml`, or `session.yml` was not found.
Likely Cause:
- `pipeline.yml` or `session.yml` is missing from system discovery paths.
- the selected campaign ID does not exist under `pipeline.campaigns.root`.
- a local working-directory config file was not passed explicitly.
Likely cause:
- missing files in discovery paths.
- missing/incorrect campaign selection.
- local file exists but was not passed explicitly.
Diagnostics:
@@ -20,160 +20,73 @@ ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
ls -l /usr/local/etc/narratio/session.yml /etc/narratio/session.yml
```
Safe Fix:
- pass explicit `--config`, `--campaign <id>`, `--campaign-file <path>`, and `--session` as appropriate.
- or place files in documented discovery paths and set `pipeline.campaigns.default_campaign_id`.
Links:
- [docs/config.md](./config.md)
- [docs/cli.md](./cli.md)
Safe fix:
- pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`.
## Templated session file rejected
Symptom:
- load fails with a message that `session.yml must be concrete`.
- load fails because `session.yml` must be concrete.
Likely Cause:
- a template authoring file such as `session.template.yml` was passed to `--session` or uploaded as remote `session.yml`.
- `session.yml` still contains `{{ ... }}` placeholders.
Likely cause:
- template placeholders (`{{ ... }}`) still present in loaded session config.
Diagnostics:
```bash
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session ./session.yml
narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
```
Safe Fix:
- generate concrete YAML with `narratio session init`.
- pass the generated concrete `session.yml` to downstream commands or upload it through `session init --remote`.
Safe fix:
- generate concrete session YAML via `narratio session init`.
Links:
- [docs/config.md](./config.md)
## Strict YAML decode or validation failure
## Strict decode or validation failure
Symptom:
- config load fails with unknown field or validation error.
- unknown field or invalid value error during config load.
Likely Cause:
- typo/stale field name.
- missing required fields or invalid constraints.
Likely cause:
- typo, stale field name, or invalid value.
Diagnostics:
```bash
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
```
Safe Fix:
- align fields/values to canonical config reference and examples.
Links:
- [docs/config.md](./config.md)
- [examples/](../examples/)
Safe fix:
- align config with [docs/config.md](./config.md) and maintained examples.
## `--artifacts` selection failure
Symptom:
- `run`/`resume`/`run-stage` fails with invalid or unknown artifact selection.
- command fails on unknown/invalid selected artifact key.
Likely Cause:
- `--artifacts` contains blank names or unknown artifact keys.
- `pipeline.scriptorium.artifacts` missing while using `--artifacts`.
Likely cause:
- artifact key not defined in `pipeline.scriptorium.artifacts`.
- empty token in `--artifacts` input.
Diagnostics:
Safe fix:
- use only configured artifact keys.
```bash
narratio run 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts player_handout
```
Safe Fix:
- use configured artifact keys only.
- ensure `pipeline.scriptorium.artifacts` is defined.
Links:
- [docs/cli.md](./cli.md)
- [docs/config.md](./config.md)
## `run-stage --artifacts` on unsupported stage
## `run-stage --artifacts` unsupported stage
Symptom:
- `run-stage` fails because `--artifacts` is only supported for `analyze` and `archive`.
- `run-stage` rejects `--artifacts` for the selected stage.
Likely Cause:
- `--artifacts` was used with a stage other than `analyze` or `archive`.
Likely cause:
- `--artifacts` used with a stage other than `analyze` or `publish`.
Diagnostics:
Safe fix:
- use `--artifacts` only with `run-stage analyze ...` or `run-stage publish ...`.
```bash
narratio run-stage polish 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts session_recap
```
Safe Fix:
- use `--artifacts` only with `run-stage analyze ...` or `run-stage archive ...`.
Links:
- [docs/cli.md](./cli.md)
## Configured artifact dependency/input validation failure
## Previous-session input unavailable
Symptom:
- config validation fails for `depends_on`, `narratio.artifact.<name>` source, or artifact output path.
- analyze fails on required previous-session artifact input.
Likely Cause:
- `narratio.artifact.<name>` source missing matching `depends_on` key.
- dependency references unknown artifact key.
- dependency self-reference or enabled dependency cycle.
- artifact output path missing/invalid/outside `artifacts/` root.
Diagnostics:
```bash
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
```
Safe Fix:
- ensure artifact-to-artifact inputs have explicit `depends_on` entries using artifact keys.
- ensure referenced artifacts exist and define valid `output_path` values.
- keep output paths relative and under `artifacts/`.
Links:
- [docs/config.md](./config.md)
- [docs/internal/stage-analyze.md](./internal/stage-analyze.md)
## Required configured artifact input unavailable at analyze time
Symptom:
- analyze fails because configured input source is unavailable.
Likely Cause:
- required upstream configured artifact was not selected/executed this run.
- non-executable dependency output file is missing or invalid on disk.
Diagnostics:
```bash
narratio session status 2026-04-04
narratio run-stage analyze 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts player_handout
```
Safe Fix:
- run analyze with needed artifacts selected.
- or ensure dependency output file exists at configured path and is valid.
Links:
- [docs/operations.md](./operations.md)
- [docs/config.md](./config.md)
## Manifest/status path failure
Symptom:
- `session status` fails because config/session state is missing, unreadable, or invalid.
Likely Cause:
- wrong session ID.
- wrong config/campaign/session file selected.
- manifest removed after cleanup.
Likely cause:
- `previous/**` cache not hydrated for this session.
Diagnostics:
@@ -181,137 +94,75 @@ Diagnostics:
narratio session status 2026-04-04
```
Safe Fix:
- use the same session ID and config files that will be used for `run`, `resume`, or `run-stage`.
Safe fix:
Links:
- [docs/cli.md](./cli.md)
- [docs/operations.md](./operations.md)
```bash
narratio run-stage prepare 2026-04-04 --force
```
Or rehydrate from remote current state:
```bash
narratio session restore 2026-04-04
```
## Session lock conflict (`.lock`)
Symptom:
- `run`, `resume`, `run-stage`, or `session restore` fails with lock conflict for session workdir.
- command fails with lock conflict.
Likely Cause:
- another Narratio process is running same session.
- stale lock from interrupted prior run.
Likely cause:
- another process is running for the same session.
- stale lock file from interrupted command.
Diagnostics:
```bash
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
cat {workspace.root}/work/{campaign}/{session_id}/.lock
ps aux | grep narratio
```
Safe Fix:
- wait for active process to finish.
- if no process is active, remove only stale session `.lock` file.
Safe fix:
- wait for active process; remove stale lock only if no process is active.
Links:
- [docs/operations.md](./operations.md)
- [docs/internal/workspace.md](./internal/workspace.md)
## Restore remote current pointer or manifest missing
## Restore current pointer/manifest missing
Symptom:
- `session restore` fails with remote current pointer or current manifest errors.
- restore fails reading remote current state.
Likely Cause:
- `current/run_id.txt` was never published.
- `current/manifest.json` is missing for the session prefix.
- archive commit did not complete.
Likely cause:
- publish commit did not complete.
- `current/run_id.txt` or `current/manifest.json` is missing.
Diagnostics:
```bash
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run
narratio session restore 2026-04-04 --dry-run
```
Safe Fix:
- verify archive stage succeeded for the target session.
- rerun/archive from a healthy source workspace so current pointers are published.
Links:
- [docs/operations.md](./operations.md)
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
## Restore manifest identity mismatch
Symptom:
- `session restore` fails because remote manifest session or campaign does not match requested values.
Likely Cause:
- wrong positional session ID or wrong session config selected.
- archive prefix points to a different campaign/session.
Diagnostics:
```bash
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run
```
Safe Fix:
- use the correct session config and positional session ID.
- verify campaign/session identity in local config before restore.
Links:
- [docs/config.md](./config.md)
- [docs/operations.md](./operations.md)
Safe fix:
- republish from a healthy local session state.
## Restore conflict without `--force`
Symptom:
- `session restore` fails with `restore conflict` and conflict counts.
- restore reports conflict and exits.
Likely Cause:
- local durable file differs from remote file for one or more planned restore paths.
Likely cause:
- local durable file differs from remote restore source.
Diagnostics:
Safe fix:
- inspect with `--dry-run`.
- rerun with `--force` only when remote should overwrite local.
```bash
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run
```
Safe Fix:
- review planned conflicts.
- rerun with `--force` only when remote state should overwrite local state.
Links:
- [docs/cli.md](./cli.md)
- [docs/operations.md](./operations.md)
## Restore report expectations
## Secrets or credentials failure
Symptom:
- operator expects restore report file but does not find one.
- startup fails loading secrets dir, or storage/tool auth fails at runtime.
Likely Cause:
- restore was executed in `--dry-run` mode.
- restore failed before report persistence path (for example lock acquisition failure).
Diagnostics:
```bash
ls -l {workspace.root}/work/{campaign}/{session_id}/reports/restore-latest.json
```
Safe Fix:
- run non-dry-run restore for durable report output.
- resolve lock or early preflight failures and retry.
Links:
- [docs/operations.md](./operations.md)
## Secrets env-dir or credential-env failure
Symptom:
- startup fails loading secrets directory, or stage fails due to missing credential env vars.
Likely Cause:
- invalid `pipeline.secrets.env_dir` path/permissions.
- required credential env var unset/empty.
Likely cause:
- invalid `pipeline.secrets.env_dir`.
- missing credential env vars.
Diagnostics:
@@ -320,60 +171,53 @@ ls -la /path/to/secrets_dir
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
```
Safe Fix:
- fix secrets directory and credential env vars.
- keep secret values out of YAML.
Safe fix:
- fix path/permissions/env vars; keep secret values out of YAML.
Links:
- [docs/config.md](./config.md)
## S3-audio prepare failure
## S3 audio prepare failure
Symptom:
- `prepare` fails in S3 mode (listing/downloading/no audio/backend error).
- prepare fails in S3 mode (list/download/no files/backend error).
Likely Cause:
- wrong `session.inputs.audio_s3.prefix`.
- no `.flac` files at resolved prefix.
- invalid/missing object-store credentials or backend config.
- mixed local+S3 audio input config.
Likely cause:
- bad `session.inputs.audio_s3.prefix`.
- no `.flac` objects at prefix.
- bad storage credentials/config.
- mixed local+S3 audio config.
Diagnostics:
```bash
narratio run-stage prepare 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
narratio run-stage prepare 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
```
Safe Fix:
- configure exactly one audio source mode.
- verify `.flac` files and storage access.
Safe fix:
- configure exactly one audio mode and verify storage access.
Links:
- [docs/config.md](./config.md)
- [docs/operations.md](./operations.md)
## Archive promotion/current-pointer failure
## Publish output or current-pointer failure
Symptom:
- archive fails on required promotion source missing or pointer write failure.
- publish fails on required output source missing, upload error, or commit-marker write failure.
Likely Cause:
- required promoted file absent (including analyze outputs not generated for this run).
- storage upload failed before `current/run_id.txt` commit marker write.
Likely cause:
- required source file not produced.
- storage upload failed before `current/run_id.txt` write.
Diagnostics:
```bash
narratio session status 2026-04-04
narratio run-stage archive 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
narratio run-stage publish 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
```
Safe Fix:
- rerun or resume upstream stages to generate required files.
- adjust promotion `source`/`dest` rules to match artifacts that must exist.
- retry after storage issue is resolved.
Safe fix:
- rerun upstream stages to regenerate required outputs.
- adjust `pipeline.publish.outputs` source/dest rules.
- retry after storage issue is fixed.
## Helpful Links
Links:
- [docs/operations.md](./operations.md)
- [docs/config.md](./config.md)
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
- [docs/cli.md](./cli.md)
- [docs/operations.md](./operations.md)
- [docs/internal/stage-publish.md](./internal/stage-publish.md)

2
examples/pipeline.full.annotated.yml
View File

@@ -43,7 +43,7 @@ publish:
# Optional booleans; defaults are true.
enabled: true
upload_run: true
# Optional promotion rules; sources use Narratio artifact source IDs.
# Optional publish output rules; sources use Narratio artifact source IDs.
outputs:
- source: narratio.transcript.final_trimmed
dest: transcripts/final.trimmed.json