eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Rewrite docs for the publish stage contract and current behavior

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-23 04:51:16 +00:00
parent 79737edf79
commit 71395bb076
22 changed files with 496 additions and 1174 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -2,7 +2,7 @@
Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts. Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts.
It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, archive publishing, and resumable run state in one operator workflow. It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, publish-stage uploads, and resumable run state in one operator workflow.
```bash ```bash
narratio run 2026-04-04 narratio run 2026-04-04

18
docs/architecture.md
View File

@@ -19,7 +19,7 @@ It coordinates specialized downstream systems rather than reimplementing their d
- Audita handles transcript correction and polishing. - Audita handles transcript correction and polishing.
- Scriptorium handles prompt execution and generated artifacts. - Scriptorium handles prompt execution and generated artifacts.
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and archive semantics. Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and publish semantics.
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine. Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
@@ -80,7 +80,7 @@ It should record:
- input and output refs; - input and output refs;
- logs and generated config refs; - logs and generated config refs;
- checksums or provenance where useful; - checksums or provenance where useful;
- non-secret adapter and archive metadata. - non-secret adapter and publish metadata.
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress. Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
@@ -117,19 +117,19 @@ Narratio should not become a secondary configuration system for downstream tools
Local and remote paths are part of Narratio’s application contract. Local and remote paths are part of Narratio’s application contract.
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and archive paths. Stages should avoid reconstructing canonical paths through scattered string concatenation. Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and publish/current paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics. Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
## Archive Invariants ## Publish Invariants
Archive behavior must preserve a clear commit boundary. Publish behavior must preserve a clear commit boundary.
A remote run is current only after the archive stage has successfully uploaded the run record, required promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`. A remote run is current only after the publish stage has successfully uploaded the run record, required published outputs, `current/manifest.json`, and finally `current/run_id.txt`.
`current/run_id.txt` is the final remote commit marker and must be written last. `current/run_id.txt` is the final remote commit marker and must be written last.
Failed, incomplete, skipped, or uncommitted archive attempts must not be presented as current remote state. Local cleanup is permitted only after successful archive commit and only when explicitly configured. Failed, incomplete, skipped, or uncommitted publish attempts must not be presented as current remote state. Local cleanup is permitted only after successful publish commit and only when explicitly configured.
## Security and Privacy ## Security and Privacy
@@ -139,7 +139,7 @@ Rules:
- Do not store raw secrets in pipeline or session YAML. - Do not store raw secrets in pipeline or session YAML.
- Use environment variable names or secret-file references for secret handling. - Use environment variable names or secret-file references for secret handling.
- Do not write raw secret values to manifests, logs, generated configs, or archive metadata. - Do not write raw secret values to manifests, logs, generated configs, or publish metadata.
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive. - Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason. - Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
@@ -177,7 +177,7 @@ Tests should cover:
- stage success, failure, skip, and resume behavior; - stage success, failure, skip, and resume behavior;
- adapter command construction; - adapter command construction;
- fake storage behavior; - fake storage behavior;
- archive commit ordering; - publish commit ordering;
- example config validity where practical. - example config validity where practical.
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs. Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.

285
docs/cli.md
View File

@@ -6,91 +6,64 @@
narratio run 2026-04-04 narratio run 2026-04-04
``` ```
This command uses default system discovery for `pipeline.yml`, the pipeline default campaign ID, and local `session.yml`. If local session discovery misses and S3 storage is configured, the positional session ID loads remote `session.yml` from the canonical session prefix. This runs the full pipeline for the given session ID using default config discovery and campaign selection.
Default pipeline and session discovery checks system config locations only. Pass `--config`, `--campaign-file`, and `--session` to use files from the current working directory. Pass `--campaign <id>` to select a campaign from `pipeline.campaigns.root`.
Ordinary local and remote `session.yml` files must be concrete YAML. Templates belong to `narratio session init`, which renders a configured campaign template before writing the concrete file.
## Command Overview ## Command Overview
Top-level commands: Top-level commands:
- `run <session_id>`: execute pipeline stages and persist manifest state. - `run <session_id>`: execute the pipeline.
- `resume <session_id>`: continue from first non-succeeded stage.
- `run-stage <stage> <session_id>`: execute exactly one stage. - `run-stage <stage> <session_id>`: execute exactly one stage.
- `resume <session_id>`: continue from first non-succeeded stage unless forced. - `analyze <session_id>`: force-rerun analyze stage.
- `analyze <session_id>`: force-rerun the analyze stage. - `publish <session_id>`: force-rerun publish stage.
- `publish <session_id>`: force-rerun the archive stage.
- `clean <session_id>|--all`: remove local workspace/spool state. - `clean <session_id>|--all`: remove local workspace/spool state.
- `session <subcommand>`: session-scoped helper commands. - `session <subcommand>`: session-scoped helper commands.
Session subcommands: Session subcommands:
- `session init <session_id>`: create local or remote `session.yml`. - `session init <session_id>`
- `session validate <session_id>`: run read-only preflight checks. - `session plan <session_id>`
- `session status <session_id>`: inspect local/remote session state. - `session validate <session_id>`
- `session plan <session_id>`: validate config, prepare workspace layout, and print stage run/skip decisions. - `session status <session_id>`
- `session restore <session_id>`: restore durable local state from committed remote archive state. - `session restore <session_id>`
- `session artifacts <session_id>`: list effective artifact source IDs. - `session artifacts <session_id>`
- `session locks <session_id>`: list archive promotion locks. - `session locks <session_id>`
- `session locks add <session_id> <source>`: add or update a remote lock. - `session locks add <session_id> <source>`
- `session locks remove <session_id> <source>`: remove a remote lock. - `session locks remove <session_id> <source>`
Unknown commands print usage and exit non-zero.
For config semantics, see [docs/config.md](./config.md). For operator lifecycle and recovery, see [docs/operations.md](./operations.md).
## Common Flags ## Common Flags
Most session-aware commands accept: Most session-aware commands accept:
- `--config <path>`: optional explicit `pipeline.yml` path. - `--config <pipeline.yml>`
- `--campaign <id>`: optional campaign ID selector. - `--campaign <id>`
- `--campaign-file <path>`: optional explicit `campaign.yml` path. - `--campaign-file <campaign.yml>`
- `--session <path>`: optional explicit concrete `session.yml` path. - `--session <session.yml>`
- `--previous-session-id <value>`: expected previous session identifier. - `--previous-session-id <id>`
The positional `<session_id>` is required even when `--session` is provided. It is used as the expected session identity and as the remote session lookup value when local session discovery misses. `--campaign` and `--campaign-file` are mutually exclusive.
## Command Reference ## Command Reference
### `run` ### `run`
```bash ```bash
narratio run <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>] narratio run <session_id> [--force] [--artifacts <name[,name...]>] [...common flags]
``` ```
Purpose: Runs stages in canonical order and writes manifest state.
- Execute configured stages in canonical order.
Success output:
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
Common failure cases:
- missing system default config/session paths when flags are omitted.
- missing selected campaign under `pipeline.campaigns.root`.
- missing local session plus missing/unavailable remote `session.yml`.
- templated `session.yml`; run `narratio session init` to generate concrete YAML.
- concrete session identity mismatch.
- unknown configured artifact key in `--artifacts`.
### `resume` ### `resume`
```bash ```bash
narratio resume <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>] narratio resume <session_id> [--force] [--artifacts <name[,name...]>] [...common flags]
``` ```
Purpose: Starts at the first non-succeeded stage from the session manifest.
- Continue from session-manifest stage status.
Success output:
- `narratio resume: session <session_id> has no remaining stages`
- or `narratio resume: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
### `run-stage` ### `run-stage`
```bash ```bash
narratio run-stage <stage> <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>] narratio run-stage <stage> <session_id> [--force] [--artifacts <name[,name...]>] [...common flags]
``` ```
Valid stage names: Valid stage names:
@@ -102,175 +75,130 @@ Valid stage names:
- `normalize` - `normalize`
- `trim` - `trim`
- `analyze` - `analyze`
- `archive` - `publish`
- `notify` - `notify`
Success output: `--artifacts` is accepted only for `analyze` and `publish`.
- `narratio run-stage: stage=<name> executed=<n> skipped=<n> force=<true|false>; manifest=<path>`
`--artifacts` is accepted only for `analyze` and `archive`.
### `analyze` ### `analyze`
```bash ```bash
narratio analyze <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--artifacts <name[,name...]>] narratio analyze <session_id> [--artifacts <name[,name...]>] [...common flags]
``` ```
Purpose: Equivalent to `narratio run-stage analyze <session_id> --force`.
- Force-rerun the analyze stage.
- Shorter equivalent for `narratio run-stage analyze <session_id> --force`.
`analyze` is force-by-design and does not accept `--force`.
### `publish` ### `publish`
```bash ```bash
narratio publish <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--artifacts <name[,name...]>] narratio publish <session_id> [--artifacts <name[,name...]>] [...common flags]
``` ```
Purpose: Equivalent to `narratio run-stage publish <session_id> --force`.
- Force-rerun the archive stage.
- Shorter equivalent for `narratio run-stage archive <session_id> --force`.
`publish` is force-by-design and does not accept `--force` or a stage positional argument.
### `clean` ### `clean`
```bash ```bash
narratio clean <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--dry-run] [--clear-cache] narratio clean <session_id> [--dry-run] [--clear-cache] [...common flags]
narratio clean --all [--config <pipeline.yml>] [--dry-run] [--clear-cache] narratio clean --all [--dry-run] [--clear-cache] [--config <pipeline.yml>]
``` ```
Session cleanup deletes: - session mode deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`.
- `{workspace.root}/work/{campaign}/{session_id}` - `--all` deletes all session work and spool children.
- `{spool.root}/{campaign}/{session_id}` - cache is preserved unless `--clear-cache` is passed.
All-session cleanup deletes:
- `{workspace.root}/work`
- the contents of `{spool.root}`, while preserving the spool root directory itself.
Cache behavior:
- cache is preserved by default.
- `--clear-cache` in session mode removes cached S3 audio files for the resolved session.
- `--all --clear-cache` removes the configured Narratio S3 audio cache namespace for the configured bucket/root prefix.
- `--clear-cache` does not delete arbitrary files under `pipeline.cache.root`.
### `session plan` ### `session plan`
```bash ```bash
narratio session plan <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force] narratio session plan <session_id> [--force] [...common flags]
``` ```
Purpose: Validates config and session inputs, prepares workdir layout, and prints stage run/skip decisions.
- Validate config, load secrets if configured, prepare workdir, and print stage run/skip decisions.
Success output includes:
- `narratio session plan: workdir prepared at <path>`
- one line per stage (`<stage>: run|skip`)
- `totals: run=<n> skip=<n>`
### `session status`
```bash
narratio session status <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>]
```
Output includes:
- session ID, campaign, workspace, and session config source.
- local manifest state when present.
- remote current archive state when storage is configured.
- catalog-based promoted output availability for expected transcript and artifact sources.
- effective archive locks and conservative next actions.
### `session validate` ### `session validate`
```bash ```bash
narratio session validate <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] narratio session validate <session_id> [...common flags]
``` ```
Checks include: Read-only preflight checks for config, inputs, audio availability, previous-session requirements, publish outputs, and effective locks.
- effective config and session source.
- stable input files.
- local or remote audio availability.
- previous-session requirements.
- archive promotions and effective locks.
Warnings do not fail the command. Any `ERROR` finding exits non-zero. ### `session status`
```bash
narratio session status <session_id> [...common flags]
```
Shows local manifest state, remote current state (when storage is configured), published-output availability, and effective locks.
### `session init` ### `session init`
```bash ```bash
narratio session init <session_id> --output ./session.yml narratio session init <session_id> --output ./session.yml
narratio session init <session_id> --remote narratio session init <session_id> --remote
narratio session init <session_id> --config <pipeline.yml> --campaign icewind --remote narratio session init <session_id> --remote --force
narratio session init <session_id> --config <pipeline.yml> --campaign-file ./campaign.yml --remote
``` ```
Additional flags: Flags:
- `--previous-session-id <value>` - `--output <path>` or `--remote` (exactly one is required)
- `--date <value>` - `--previous-session-id <id>`
- `--title <value>` - `--date <date>`
- `--audio-s3-prefix <prefix>`: defaults to `audio/` when neither audio flag is provided. - `--title <title>`
- `--audio-dir <path>`: local audio directory; mutually exclusive with `--audio-s3-prefix`. - `--audio-dir <path>`
- `--force`: overwrite existing local or remote target. - `--audio-s3-prefix <prefix>`
- `--force`
Behavior: - common config/campaign flags
- exactly one of `--output` or `--remote` is required.
- `--config`, `--campaign`, and `--campaign-file` are optional overrides; omitted campaign selection uses `pipeline.campaigns.default_campaign_id`.
- `--campaign <id>` selects a campaign under `pipeline.campaigns.root`.
- `--campaign-file <path>` loads an explicit campaign file.
- if `campaign.yml` sets `session_template_file`, the template path is resolved relative to `campaign.yml` and rendered from init flags.
- if no session template is configured, a minimal concrete session file is generated directly.
- template variables must be supplied by matching flags, and supplied template-related flags must be used by the template.
- remote writes target `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
- existing local or remote targets fail unless `--force` is passed.
- remote writes use existence checks, not compare-and-swap.
### `session restore` ### `session restore`
```bash ```bash
narratio session restore <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--dry-run] [--force] [--include-audio] narratio session restore <session_id> [--dry-run] [--force] [--include-audio] [...common flags]
``` ```
Purpose: Restores durable local session files from committed remote current state.
- Restore durable session state from the committed remote archive current state.
- Default restore installs `manifest.json`, `transcripts/**`, and `artifacts/**` from the current session archive.
- When configured previous-session inputs require it, restore reconstructs `previous/**` from the previous session's committed current archive.
- `audio/**` is restored only with `--include-audio`.
Dry-run output may include planned previous-cache downloads. Existing differing files under `previous/**` follow the normal restore conflict policy and require `--force` to overwrite. Default restore scope:
When `--include-audio` is set, S3 audio files are restored through the shared audio cache. Cache hits avoid re-downloading large audio objects. - `manifest.json`
- `transcripts/**`
- `artifacts/**`
- `previous/**` when required by configured previous-session artifact inputs
`audio/**` is restored only when `--include-audio` is set.
### `session artifacts` ### `session artifacts`
```bash ```bash
narratio session artifacts <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--remote] narratio session artifacts <session_id> [--remote] [...common flags]
``` ```
Purpose: Lists built-in sources, configured artifact sources, previous-session sources, publish output rules, and lock status. With `--remote`, includes remote published-state markers.
- List built-in, configured, previous-session, promoted, and locked artifact sources.
`--remote` checks promoted top-level object availability through the storage adapter. Remote markers appear only in the `Promoted` section, which reports each configured archive promotion destination and includes `dest=<path>` when that destination differs from the source's canonical path.
### `session locks` ### `session locks`
```bash ```bash
narratio session locks <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] narratio session locks <session_id> [...common flags]
narratio session locks add <session_id> <source> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--reason <text>] [--force] narratio session locks add <session_id> <source> [--reason <text>] [--force] [...common flags]
narratio session locks remove <session_id> <source> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] narratio session locks remove <session_id> <source> [...common flags]
``` ```
Behavior: - list mode prints effective locks from static `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`.
- list mode prints effective locks from static `pipeline.archive.locks` and remote `{session_prefix}/locks.yml`. - add/remove mutate only the remote lock store.
- `locks add` writes only the remote lock store and fails if the source is already locked by pipeline config. - static pipeline locks cannot be removed by lock commands.
- `locks remove` removes only remote locks and cannot remove static pipeline locks.
- `locks add --force` is required to update an existing remote lock reason. ## `--artifacts` Rules
- accepted on `run`, `resume`, `run-stage`, `analyze`, and `publish`.
- on `run-stage`, only valid for `analyze` and `publish`.
- filters configured analyze artifact execution.
- filters configured `pipeline.publish.outputs` entries for `narratio.artifact.<key>` sources.
- does not suppress built-in transcript/bounds publish outputs.
- does not imply `--force` for `run`, `resume`, or `run-stage`.
## Common Workflows ## Common Workflows
Default-discovery run: Run full pipeline:
```bash ```bash
narratio run 2026-04-04 narratio run 2026-04-04
@@ -282,60 +210,21 @@ Run only selected analyze artifacts:
narratio run 2026-04-04 --artifacts session_recap,player_handout narratio run 2026-04-04 --artifacts session_recap,player_handout
``` ```
Resume with selected analyze artifacts: Force analyze only:
```bash
narratio resume 2026-04-04 --artifacts player_handout
```
Force-rerun analyze with selected artifacts:
```bash ```bash
narratio analyze 2026-04-04 --artifacts player_handout narratio analyze 2026-04-04 --artifacts player_handout
``` ```
Force-rerun archive publishing: Force publish only:
```bash ```bash
narratio publish 2026-04-04 narratio publish 2026-04-04
``` ```
Preview restore actions without writes: Restore preview then apply:
```bash ```bash
narratio session restore 2026-04-04 --dry-run narratio session restore 2026-04-04 --dry-run
```
Restore and then force analyze:
```bash
narratio session restore 2026-04-04 narratio session restore 2026-04-04
narratio analyze 2026-04-04
``` ```
Rehydrate canonical previous-session inputs after artifact-input changes:
```bash
narratio run-stage prepare 2026-04-04 --force
```
Reset local state before testing restore:
```bash
narratio clean 2026-04-04 --dry-run
narratio clean 2026-04-04
narratio session restore 2026-04-04 --include-audio
```
Clean all local sessions while keeping cached S3 audio:
```bash
narratio clean --all
```
## `--artifacts` and `--force`
- `--artifacts` filters which configured artifacts are executable when analyze runs and which configured artifact promotions archive publishes.
- `--artifacts` does not imply `--force`.
- if analyze is already `succeeded` and `--force` is not set, runner-level skip still applies.
- `--artifacts` does not suppress built-in transcript or bounds promotions.

448
docs/config.md
View File

@@ -1,14 +1,13 @@
# Configuration # Configuration
## 1. Overview ## Overview
Narratio loads three YAML files: Narratio loads three YAML files:
- `pipeline.yml`: pipeline-level runtime configuration. - `pipeline.yml`: pipeline-level runtime settings.
- `campaign.yml`: stable campaign identity and campaign-level input defaults. - `campaign.yml`: stable campaign identity and campaign-level input defaults.
- `session.yml`: per-session metadata and input selection, loaded locally or from the configured S3 backend. - `session.yml`: per-session metadata and input selection.
These commands load and validate all three files before running: Commands that load and validate all three files include:
- `narratio run` - `narratio run`
- `narratio resume` - `narratio resume`
@@ -23,94 +22,36 @@ These commands load and validate all three files before running:
- `narratio session locks` - `narratio session locks`
- `narratio clean <session_id>` - `narratio clean <session_id>`
Behavior: Validation behavior:
- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail. - strict YAML decode is enabled (`KnownFields(true)`); unknown fields fail.
- ordinary local and remote `session.yml` files must be concrete YAML; template placeholders are rejected. - loaded `session.yml` files must be concrete YAML (no `{{ ... }}` placeholders).
- defaults are applied for optional pipeline fields. - defaults are applied for optional pipeline fields.
- campaign identity is selected by ID from the pipeline campaign registry unless `--campaign-file` is used. - campaign/session identity mismatches fail load.
- campaign-level stable input paths fill missing session input paths.
- session-level stable input paths override campaign-level input paths.
- campaign config may point `session init` to a session template.
- validation enforces required fields, value formats, and cross-field constraints.
## 2. Config file discovery ## File Discovery
Pipeline discovery order when `--config` is omitted:
These commands use the same config discovery behavior: 1. `/usr/local/etc/narratio/pipeline.yml`
2. `/etc/narratio/pipeline.yml`
- `narratio run` Session discovery order when `--session` is omitted:
- `narratio resume`
- `narratio run-stage`
- `narratio analyze`
- `narratio publish`
- `narratio session plan`
- `narratio session status`
- `narratio session validate`
- `narratio session restore`
- `narratio session artifacts`
- `narratio session locks`
- `narratio clean <session_id>`
Pipeline config lookup: 1. `/usr/local/etc/narratio/session.yml`
2. `/etc/narratio/session.yml`
- if `--config <path>` is provided, that path is used. Campaign discovery when `--campaign-file` is omitted:
- if omitted, Narratio searches in order:
1. `/usr/local/etc/narratio/pipeline.yml`
2. `/etc/narratio/pipeline.yml`
- first existing file wins.
Campaign config lookup: - if `--campaign <id>` is set: `{pipeline.campaigns.root}/{id}/campaign.yml`
- otherwise: `{pipeline.campaigns.root}/{pipeline.campaigns.default_campaign_id}/campaign.yml`
- pipeline config is loaded first. Remote `session.yml` fallback:
- if `--campaign-file <path>` is provided, that path is used.
- otherwise, if `--campaign <id>` is provided, Narratio loads:
- `{pipeline.campaigns.root}/{id}/campaign.yml`
- otherwise, Narratio uses `pipeline.campaigns.default_campaign_id` and loads:
- `{pipeline.campaigns.root}/{default_campaign_id}/campaign.yml`
- `--campaign` and `--campaign-file` are mutually exclusive.
- campaign IDs must be single path segments, not paths.
Session config lookup: - if local session discovery fails and storage is configured, Narratio can load:
- if `--session <path>` is provided, that path is used.
- if `--session` is omitted, Narratio searches locally in order:
1. `/usr/local/etc/narratio/session.yml`
2. `/etc/narratio/session.yml`
- first existing local file wins.
- if no local session file is found, a positional `<session_id>` is present, storage is configured, and campaign identity is resolved, Narratio loads remote `session.yml` from:
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml` - `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
- local discovery always runs before remote fallback.
- local files in the current working directory are used only when passed explicitly, for example `--config ./pipeline.yml --campaign-file ./campaign.yml --session ./session.yml`.
## 3. Session templating ## Minimal Working Config
`pipeline.yml`
Template behavior for local and remote `session.yml` loaded by downstream commands:
- downstream commands do not render templates.
- local and remote `session.yml` must be concrete.
- any `{{ ... }}` placeholder in loaded `session.yml` fails with guidance to run `narratio session init`.
- if concrete `session_id` mismatches the positional `<session_id>`, load fails.
- if concrete `previous_session_id` mismatches `--previous-session-id`, load fails.
Template behavior for `narratio session init`:
- `campaign.yml` may set `session_template_file`.
- relative template paths resolve relative to `campaign.yml`.
- supported init template variables:
- `{{ session_id }}`
- `{{ previous_session_id }}`
- `{{ date }}`
- `{{ title }}`
- `{{ audio_s3_prefix }}`
- `{{ audio_dir }}`
- each template variable must be supplied by the matching `session init` flag.
- template-related flags such as `--date`, `--title`, `--audio-s3-prefix`, `--audio-dir`, and `--previous-session-id` fail if the configured template does not use them.
- rendered output is strict-decoded and validated before it is written locally or remotely.
- if `session_template_file` is omitted, `session init` generates the minimal concrete session YAML directly.
## 4. Minimal config set
### `pipeline.yml`
```yaml ```yaml
campaigns: campaigns:
@@ -120,30 +61,17 @@ whisperx:
transcribe_url: "https://transcription.example.com/transcribe" transcribe_url: "https://transcription.example.com/transcribe"
``` ```
Why this is sufficient: `campaign.yml`
- `whisperx.transcribe_url` is required.
- `campaigns.default_campaign_id` selects the default campaign when `--campaign` is omitted.
- `workspace.root` defaults to `/var/lib/narratio`.
- optional sections (`seriatim`, `audita`, `archive`, `scriptorium`, `trim`, `normalize`, etc.) receive defaults or stay inactive.
### `campaign.yml`
```yaml ```yaml
campaign_id: sample-campaign campaign_id: sample-campaign
session_template_file: ./session.template.yml
inputs: inputs:
speakers_file: ./speakers.yml speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml glossary_file: ./glossary.yml
``` ```
Why this is sufficient: `session.yml`
- `campaign_id` supplies the stable campaign identity.
- stable input files are required and resolve relative to `campaign.yml` when copied during `prepare`.
### `session.yml`
```yaml ```yaml
session_id: 2026-05-03 session_id: 2026-05-03
@@ -151,67 +79,14 @@ inputs:
audio_dir: ./audio audio_dir: ./audio
``` ```
Why this is sufficient: ## Publish Config
Top-level publish settings live at `pipeline.publish`.
- `session_id` is required.
- `campaign` can be omitted because it is supplied by `campaign.yml`.
- stable input paths can be omitted because `campaign.yml` supplies defaults.
- local `audio_dir` resolves relative to `session.yml`.
Minimal local-file usage:
```bash
narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign sample-campaign --session ./session.yml
narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign-file ./campaign.yml --session ./session.yml
```
Previous-session-enabled variant:
```yaml ```yaml
session_id: 2026-05-03 publish:
previous_session_id: 2026-04-26
inputs:
audio_dir: ./audio
```
```bash
narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign sample-campaign --session ./session.yml --previous-session-id 2026-04-26
```
## 5. Production-oriented config set
### `pipeline.yml`
```yaml
workspace:
root: /var/lib/narratio/workspace
cleanup_after_archive: true
storage:
backend: s3
s3:
bucket: my-dnd-archive
root_prefix: dnd
region: us-east-1
access_key_id_env: OBJECT_STORAGE_KEY_ID
secret_access_key_env: OBJECT_STORAGE_KEY
campaigns:
root: /srv/narratio/campaigns
default_campaign_id: forsaken
spool:
root: /var/spool/narratio
delete_audio_after_archive: true
cache:
root: /var/cache/narratio
s3_audio: true
archive:
enabled: true enabled: true
upload_run: true upload_run: true
promote_artifacts: outputs:
- source: narratio.transcript.final_trimmed - source: narratio.transcript.final_trimmed
dest: transcripts/final.trimmed.json dest: transcripts/final.trimmed.json
required: true required: true
@@ -221,80 +96,31 @@ archive:
locks: locks:
- source: narratio.artifact.session_recap - source: narratio.artifact.session_recap
reason: Final recap was manually edited. reason: Final recap was manually edited.
whisperx:
transcribe_url: "https://transcription.example.com/transcribe"
scriptorium:
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
inputs:
transcript:
source: narratio.transcript.final_trimmed
required: true
previous_recap:
source: narratio.previous_session.artifact.session_recap
required: false
``` ```
### `campaign.yml` Rules:
```yaml - `outputs[].source` is required.
campaign_id: forsaken - `outputs[].dest` is optional; when omitted, Narratio derives destination from the source.
inputs: - `outputs[].required` defaults to `true`.
speakers_file: /srv/narratio/campaigns/forsaken/speakers.yml - static `publish.locks` and remote `{session_prefix}/locks.yml` are merged; static locks win on duplicates.
autocorrect_file: /srv/narratio/campaigns/forsaken/autocorrect.yml - locks prevent overwrite of top-level published destinations.
glossary_file: /srv/narratio/campaigns/forsaken/glossary.yml
```
### Local `session.yml` Supported publish source families:
```yaml - built-ins: `narratio.transcript.base`, `narratio.transcript.polished`, `narratio.transcript.final`, `narratio.transcript.final_trimmed`, `narratio.bounds.session`
session_id: 2026-05-03 - configured artifacts: `narratio.artifact.<artifact_key>`
previous_session_id: 2026-04-26
date: 2026-05-03
title: The Black Cabin
inputs:
audio_s3:
prefix: audio/
```
### S3-first session config ## Full Reference
For S3-first operation, upload the same `session.yml` content to:
```text
{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml
```
Then run with explicit or discovered pipeline/campaign config and no `--session`:
```bash
narratio run 2026-05-03 --config /usr/local/etc/narratio/pipeline.yml --campaign forsaken --previous-session-id 2026-04-26
```
Operational notes:
- archive promotion is explicit and source-based via `archive.promote_artifacts`.
- `source` is required; `dest` is optional and derived when omitted.
- `archive.locks` skips top-level promotion overwrites for static locked sources while preserving run-local uploads.
- operator-created mutable locks are stored at `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/locks.yml` and are merged with static locks.
- Narratio does not auto-promote all generated analyze artifacts.
- `restore` reads the same config/campaign/session inputs and restore scope is bounded by committed archive current state.
- `clean` removes workspace/spool state by default and preserves `pipeline.cache.root` unless `--clear-cache` is passed.
## 6. Full pipeline reference
### Pipeline
| Path | Type | Required | Default | | Path | Type | Required | Default |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` | | `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
| `pipeline.workspace.cleanup_after_archive` | bool | No | `false` | | `pipeline.workspace.cleanup_after_publish` | bool | No | `false` |
| `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` | | `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` |
| `pipeline.campaigns.default_campaign_id` | string | No | empty | | `pipeline.campaigns.default_campaign_id` | string | No | empty |
| `pipeline.secrets.env_dir` | string | Conditional | none | | `pipeline.secrets.env_dir` | string | No | empty |
| `pipeline.storage.backend` | string | No | empty | | `pipeline.storage.backend` | string | No | empty |
| `pipeline.storage.s3.bucket` | string | Conditional | empty | | `pipeline.storage.s3.bucket` | string | Conditional | empty |
| `pipeline.storage.s3.root_prefix` | string | No | `dnd` | | `pipeline.storage.s3.root_prefix` | string | No | `dnd` |
@@ -304,18 +130,18 @@ Operational notes:
| `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` | | `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` |
| `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` | | `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` |
| `pipeline.spool.root` | string | No | `/var/spool/narratio` | | `pipeline.spool.root` | string | No | `/var/spool/narratio` |
| `pipeline.spool.delete_audio_after_archive` | bool | No | `false` | | `pipeline.spool.delete_audio_after_publish` | bool | No | `false` |
| `pipeline.cache.root` | string | No | `/var/cache/narratio` | | `pipeline.cache.root` | string | No | `/var/cache/narratio` |
| `pipeline.cache.s3_audio` | bool | No | `true` | | `pipeline.cache.s3_audio` | bool | No | `true` |
| `pipeline.archive.enabled` | bool | No | `true` | | `pipeline.publish.enabled` | bool | No | `true` |
| `pipeline.archive.upload_run` | bool | No | `true` | | `pipeline.publish.upload_run` | bool | No | `true` |
| `pipeline.archive.promote_artifacts[]` | list | No | final-trimmed transcript rule | | `pipeline.publish.outputs[]` | list | No | one final-trimmed output rule |
| `pipeline.archive.promote_artifacts[].source` | string | Yes (per rule) | none | | `pipeline.publish.outputs[].source` | string | Yes (per rule) | none |
| `pipeline.archive.promote_artifacts[].dest` | string | No | derived from source | | `pipeline.publish.outputs[].dest` | string | No | derived from source |
| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` | | `pipeline.publish.outputs[].required` | bool | No | `true` |
| `pipeline.archive.locks[]` | list | No | empty | | `pipeline.publish.locks[]` | list | No | empty |
| `pipeline.archive.locks[].source` | string | Yes (per lock) | none | | `pipeline.publish.locks[].source` | string | Yes (per lock) | none |
| `pipeline.archive.locks[].reason` | string | No | empty | | `pipeline.publish.locks[].reason` | string | No | empty |
| `pipeline.whisperx.transcribe_url` | string | Yes | none | | `pipeline.whisperx.transcribe_url` | string | Yes | none |
| `pipeline.whisperx.language` | string | No | `en` | | `pipeline.whisperx.language` | string | No | `en` |
| `pipeline.whisperx.timeout` | duration string | No | `30m` | | `pipeline.whisperx.timeout` | duration string | No | `30m` |
@@ -364,162 +190,38 @@ Operational notes:
| `pipeline.scriptorium.timeout` | duration string | No | `10m` | | `pipeline.scriptorium.timeout` | duration string | No | `10m` |
| `pipeline.scriptorium.render_debug` | bool | No | `false` | | `pipeline.scriptorium.render_debug` | bool | No | `false` |
| `pipeline.scriptorium.artifacts` | map | No | empty | | `pipeline.scriptorium.artifacts` | map | No | empty |
| `pipeline.scriptorium.artifacts.<name>.enabled` | bool | No | `false` |
| `pipeline.scriptorium.artifacts.<name>.depends_on[]` | list[string] | No | empty |
| `pipeline.scriptorium.artifacts.<name>.render_debug` | bool | No | unset |
| `pipeline.scriptorium.artifacts.<name>.prompt_id` | string | Conditional | none |
| `pipeline.scriptorium.artifacts.<name>.profile_id` | string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.output_path` | string | Conditional | none |
| `pipeline.scriptorium.artifacts.<name>.timeout` | duration string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` | string | Conditional | none |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.artifact` | string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.path` | string | No | empty |
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required` | bool | No | `false` |
| `pipeline.scriptorium.artifacts.<name>.vars.<key>` | map value | No | empty |
| `pipeline.notification.backend` | string | No | empty | | `pipeline.notification.backend` | string | No | empty |
| `pipeline.notification.recipient` | string | No | empty | | `pipeline.notification.recipient` | string | No | empty |
| `pipeline.notification.timeout` | duration string | No | empty | | `pipeline.notification.timeout` | duration string | No | `30s` |
Scriptorium artifact-key and dependency rules: ### Campaign
| Path | Type | Required |
| --- | --- | --- |
| `campaign_id` | string | Yes |
| `session_template_file` | string | No |
| `inputs.speakers_file` | string | Yes |
| `inputs.autocorrect_file` | string | Yes |
| `inputs.glossary_file` | string | Yes |
- artifact keys must match `^[a-z][a-z0-9_]*$`. ### Session
- enabled artifacts require `prompt_id` and `output_path`. | Path | Type | Required |
- `output_path` must be relative, traversal-safe, and under `artifacts/`. | --- | --- | --- |
- configured artifact input sources use `narratio.artifact.<name>`. | `session_id` | string | Yes |
- if input source references `narratio.artifact.<name>`, artifact `<name>` must exist and must be listed in `depends_on`. | `previous_session_id` | string | No |
- every `depends_on` entry must be a configured artifact key. | `campaign` | string | No |
- self-dependency is rejected. | `date` | string | No |
- enabled dependency cycles are rejected. | `title` | string | No |
- any artifact referenced by `depends_on` or `narratio.artifact.<name>` source must define `output_path` (even if not enabled). | `inputs.speakers_file` | string | No |
| `inputs.autocorrect_file` | string | No |
Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values: | `inputs.glossary_file` | string | No |
| `inputs.audio_dir` | string | Conditional |
- `narratio.previous_session.artifact.<configured_artifact_key>` | `inputs.audio_files[]` | list[string] | Conditional |
- `narratio.transcript.base` | `inputs.audio_s3.prefix` | string | Conditional |
- `narratio.transcript.polished`
- `narratio.transcript.final`
- `narratio.transcript.final_trimmed`
- `narratio.bounds.session`
- `narratio.artifact.<configured_artifact_key>`
`pipeline.archive.promote_artifacts[].source` values:
- `narratio.transcript.base`
- `narratio.transcript.polished`
- `narratio.transcript.final`
- `narratio.transcript.final_trimmed`
- `narratio.bounds.session`
- `narratio.artifact.<configured_artifact_key>`
`pipeline.archive.locks[].source` accepts the same source values as `pipeline.archive.promote_artifacts[].source`.
Archive promotion destination rules:
- `dest` must be a clean relative path (not absolute, no traversal).
- duplicate `dest` values are rejected.
- if `dest` is omitted:
- built-in sources derive their canonical destination path;
- configured sources derive from `pipeline.scriptorium.artifacts.<name>.output_path`;
- derivation failure is a config validation error.
Archive lock rules:
- locks are source-based and do not accept `dest`.
- duplicate lock sources are rejected.
- static `pipeline.archive.locks` win over remote mutable locks for the same source.
- locked promotions are recorded as intentional skips in archive metadata.
- locked required promotions do not fail archive by default.
- ordinary `--force` reruns do not override locks.
Remote mutable lock store:
- path: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/locks.yml`.
- strict YAML shape: top-level `locks`, each with `source` and optional `reason`.
- `narratio session locks add` and `narratio session locks remove` mutate only the remote lock store.
- writes use existence checks plus `--force` for updates; they are not compare-and-swap atomic.
Restore-related implications:
- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs).
- restore scope considers committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, optional `audio/**`).
- S3 audio downloads use `pipeline.spool.root` for active downloads and `pipeline.cache.root` for reusable cached audio when `pipeline.cache.s3_audio` is true.
- `pipeline.cache.root` is durable local cache state. It is not workspace state and is preserved by default by `narratio clean`.
## 7. Full campaign reference
| Path | Type | Required | Default |
| --- | --- | --- | --- |
| `campaign.campaign_id` | string | Yes | none |
| `campaign.session_template_file` | string | No | none |
| `campaign.inputs.speakers_file` | string | Yes | none |
| `campaign.inputs.autocorrect_file` | string | Yes | none |
| `campaign.inputs.glossary_file` | string | Yes | none |
Campaign input paths and `campaign.session_template_file` may be absolute or relative. Relative paths resolve from the directory containing `campaign.yml`.
## 8. Full session reference
| Path | Type | Required | Default |
| --- | --- | --- | --- |
| `session.session_id` | string | Yes | none |
| `session.previous_session_id` | string | No | empty |
| `session.campaign` | string | No | `campaign.campaign_id` |
| `session.date` | string | No | empty |
| `session.title` | string | No | empty |
| `session.inputs.audio_dir` | string | Conditional | empty |
| `session.inputs.audio_files[]` | list[string] | Conditional | empty |
| `session.inputs.audio_s3.prefix` | string | Conditional | none |
| `session.inputs.speakers_file` | string | No | `campaign.inputs.speakers_file` |
| `session.inputs.autocorrect_file` | string | No | `campaign.inputs.autocorrect_file` |
| `session.inputs.glossary_file` | string | No | `campaign.inputs.glossary_file` |
Session input paths may be absolute or relative. Relative audio paths and session-level stable input overrides resolve from the directory containing `session.yml`. If both `campaign.yml` and `session.yml` specify campaign identity, the values must match.
Audio-source rule:
- configure exactly one mode:
- `audio_dir`, or
- `audio_files` (at least one), or
- `audio_s3.prefix`
- `audio_s3` cannot be combined with local audio fields.
Previous-session rule:
- if `session.previous_session_id` is set, it must not equal `session.session_id`.
- canonical previous-session sources (`narratio.previous_session.artifact.<name>`) are hydrated during `prepare` from archive current state when required by enabled configured artifacts.
## 9. Secrets
Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`.
Behavior:
- `env_dir` may be absolute or relative.
- relative `env_dir` resolves from current working directory.
- files with valid env-var names (`[A-Za-z_][A-Za-z0-9_]*`) are loaded.
- values are loaded from file contents with trailing newline trimming.
- existing process env vars are preserved.
- invalid names and subdirectories are skipped.
- missing/unreadable `env_dir` fails command execution.
Guidance:
- do not put secret values directly in YAML.
- configure env var names in config and provide values via env/secrets files.
## 10. Examples
Maintained examples:
## Maintained Examples
- `examples/pipeline.minimal.yml` - `examples/pipeline.minimal.yml`
- `examples/pipeline.production.yml` - `examples/pipeline.production.yml`
- `examples/pipeline.full.annotated.yml` - `examples/pipeline.full.annotated.yml`
- `examples/campaigns/sample-campaign/campaign.yml` - `examples/campaigns/sample-campaign/campaign.yml`
- `examples/campaigns/sample-campaign/speakers.yml`
- `examples/campaigns/sample-campaign/autocorrect.yml`
- `examples/campaigns/sample-campaign/glossary.yml`
- `examples/campaigns/sample-campaign/session.template.yml`
- `examples/session.local-audio.yml` - `examples/session.local-audio.yml`
- `examples/session.s3-audio.yml` - `examples/session.s3-audio.yml`
These examples are validated by `internal/config` tests.

2
docs/development.md
View File

@@ -75,7 +75,7 @@ Remote-storage commands must obtain object storage through the app-level command
1. Implement stage behavior in `internal/stage` with clear input/output boundaries. 1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
2. Keep external transport/subprocess details in `internal/adapters`. 2. Keep external transport/subprocess details in `internal/adapters`.
3. Preserve manifest and promotion semantics expected by runner and archive logic. 3. Preserve manifest and publish-output semantics expected by runner and publish logic.
4. Add/update stage and adapter tests. 4. Add/update stage and adapter tests.
5. Update internal component contracts in `docs/internal/`. 5. Update internal component contracts in `docs/internal/`.

2
docs/integrations/seriatim.md
View File

@@ -20,7 +20,7 @@ Owns:
- JSON output validation - JSON output validation
Does not own: Does not own:
- Transcript input selection/promotion logic (stage-owned) - Transcript input selection/materialization logic (stage-owned)
- Bounds computation (scriptorium/trim-stage-owned) - Bounds computation (scriptorium/trim-stage-owned)
## Config Fields Used ## Config Fields Used

4
docs/internal/README.md
View File

@@ -11,7 +11,7 @@ Implementation-accurate contracts for workspace/state, manifests, stages, artifa
- `storage.md`: remote storage backend contracts and object-store invariants. - `storage.md`: remote storage backend contracts and object-store invariants.
- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics. - `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics.
- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior. - `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior.
- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants. - `workspace.md`: local state model, manifests, run-local layout, materialization, and cleanup invariants.
- `command-restore.md`: restore command discovery/planning/execution/reporting contract. - `command-restore.md`: restore command discovery/planning/execution/reporting contract.
- `stage-prepare.md`: input materialization and provenance capture. - `stage-prepare.md`: input materialization and provenance capture.
- `stage-transcribe.md`: WhisperX transcript generation. - `stage-transcribe.md`: WhisperX transcript generation.
@@ -20,7 +20,7 @@ Implementation-accurate contracts for workspace/state, manifests, stages, artifa
- `stage-normalize.md`: post-polish normalization. - `stage-normalize.md`: post-polish normalization.
- `stage-trim.md`: bounds-driven transcript trimming. - `stage-trim.md`: bounds-driven transcript trimming.
- `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts. - `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts.
- `stage-archive.md`: archive upload and current-pointer publish contract. - `stage-publish.md`: publish upload and current-pointer commit contract.
## External Integration Notes ## External Integration Notes
- `../integrations/README.md`: canonical location for external integration contracts (`audita.md`, `seriatim.md`, `scriptorium.md`). - `../integrations/README.md`: canonical location for external integration contracts (`audita.md`, `seriatim.md`, `scriptorium.md`).

2
docs/internal/adapters.md
View File

@@ -28,7 +28,7 @@ Default wiring and adapter calls consume:
- `pipeline.seriatim.*` - `pipeline.seriatim.*`
- `pipeline.audita.*` - `pipeline.audita.*`
- `pipeline.scriptorium.*` - `pipeline.scriptorium.*`
- `pipeline.storage.*` and `pipeline.archive.*` (object-store construction/gating) - `pipeline.storage.*` and `pipeline.publish.*` (object-store construction/gating)
- `pipeline.notification.*` (sender boundary exists; placeholder behavior today) - `pipeline.notification.*` (sender boundary exists; placeholder behavior today)
## External adapters used ## External adapters used

8
docs/internal/artifacts.md
View File

@@ -15,8 +15,8 @@ Inputs:
Outputs: Outputs:
- resolved artifact path + provenance (`ResolvedSessionArtifact`); - resolved artifact path + provenance (`ResolvedSessionArtifact`);
- runtime catalog entries for built-ins and configured artifacts; - runtime catalog entries for built-ins and configured artifacts;
- requirement sets for canonical previous-session inputs. - requirement sets for canonical previous-session inputs;
- canonical S3 session, run, current, session config, session locks, audio, and promoted artifact keys. - canonical S3 session, run, current, session config, session locks, audio, and published output keys.
## Boundaries ## Boundaries
Owns: Owns:
@@ -28,7 +28,7 @@ Owns:
Does not own: Does not own:
- prepare-stage remote hydration; - prepare-stage remote hydration;
- stage success/skip transitions; - stage success/skip transitions;
- archive upload orchestration. - publish upload orchestration.
## Built-in IDs ## Built-in IDs
| Artifact ID | Canonical file | Producer stage | Output kind | | Artifact ID | Canonical file | Producer stage | Output kind |
@@ -71,7 +71,7 @@ Previous-session canonical provenance values include:
- Built-ins resolve via manifest producer outputs first, then canonical fallback paths. - Built-ins resolve via manifest producer outputs first, then canonical fallback paths.
- Configured `narratio.artifact.<name>` sources resolve through catalog availability. - Configured `narratio.artifact.<name>` sources resolve through catalog availability.
- Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths. - Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths.
- Archive-relative configured artifact paths under `artifacts/` are cached without a redundant nested `artifacts/` segment. - Publish-relative configured artifact paths under `artifacts/` are cached without a redundant nested `artifacts/` segment.
- Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`. - Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`.
## Previous-session requirement scanning ## Previous-session requirement scanning

10
docs/internal/command-restore.md
View File

@@ -27,14 +27,14 @@ Owns:
Does not own: Does not own:
- Stage execution orchestration (`run`, `resume`, `run-stage`). - Stage execution orchestration (`run`, `resume`, `run-stage`).
- Archive publish behavior (owned by archive stage). - Publish-stage behavior.
- Storage transport implementation details (owned by storage adapters). - Storage transport implementation details (owned by storage adapters).
## Config fields used ## Config fields used
- Config/session discovery and templating fields consumed by all commands. - Config/session discovery and templating fields consumed by all commands.
- `pipeline.workspace.root` (local restore target root). - `pipeline.workspace.root` (local restore target root).
- `pipeline.storage.*` (remote backend + archive identity derivation). - `pipeline.storage.*` (remote backend + publish identity derivation).
- `pipeline.storage.s3.*` identity components used by archive prefix helpers. - `pipeline.storage.s3.*` identity components used by session-prefix helpers.
- `pipeline.spool.root` for active audio downloads. - `pipeline.spool.root` for active audio downloads.
- `pipeline.cache.root` and `pipeline.cache.s3_audio` for reusable S3 audio cache. - `pipeline.cache.root` and `pipeline.cache.s3_audio` for reusable S3 audio cache.
- `session.session_id` - `session.session_id`
@@ -80,7 +80,7 @@ Restore path scope:
- Dry-run is read-only and returns plan output only. - Dry-run is read-only and returns plan output only.
## Failure behavior ## Failure behavior
- Fails when storage backend is unavailable or archive identity cannot be resolved. - Fails when storage backend is unavailable or publish identity cannot be resolved.
- Fails when remote current pointer/manifest is missing or invalid. - Fails when remote current pointer/manifest is missing or invalid.
- Fails when remote manifest identity mismatches requested campaign/session. - Fails when remote manifest identity mismatches requested campaign/session.
- Fails on local conflicts unless `--force` is set. - Fails on local conflicts unless `--force` is set.
@@ -96,7 +96,7 @@ Restore path scope:
- `internal/artifacts/archive_identity_test.go` - `internal/artifacts/archive_identity_test.go`
## Architectural invariants ## Architectural invariants
- Restore relies on centralized archive identity/key helpers (`internal/artifacts`) rather than ad hoc key building. - Restore relies on centralized path/key helpers (`internal/artifacts`) rather than ad hoc key building.
- `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files. - `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files.
- Local path mapping is traversal-safe and constrained to session root. - Local path mapping is traversal-safe and constrained to session root.
- Restore scope is deterministic and path-classified: - Restore scope is deterministic and path-classified:

2
docs/internal/manifest.md
View File

@@ -30,7 +30,7 @@ Manifest identity fields are populated by app/stage orchestration from:
- `session.session_id` - `session.session_id`
- `session.campaign` - `session.campaign`
- `pipeline.workspace.root` - `pipeline.workspace.root`
- `pipeline.storage.s3.*` (when archive/S3 identity is set) - `pipeline.storage.s3.*` (when publish/S3 identity is set)
## External adapters used ## External adapters used
- No external service adapters. - No external service adapters.

10
docs/internal/stage-analyze.md
View File

@@ -1,7 +1,7 @@
# Stage: analyze # Stage: analyze
## Purpose ## Purpose
Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths. Execute selected configured Scriptorium artifacts in deterministic dependency order and materialize successful outputs to canonical session artifact paths.
## Inputs and outputs ## Inputs and outputs
Inputs: Inputs:
@@ -15,7 +15,7 @@ Source types used by analyze:
- canonical previous-session artifacts: `narratio.previous_session.artifact.<artifact_key>`. - canonical previous-session artifacts: `narratio.previous_session.artifact.<artifact_key>`.
Outputs: Outputs:
- promoted configured artifact files at each configured `output_path`; - materialized configured artifact files at each configured `output_path`;
- stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info). - stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info).
## Boundaries ## Boundaries
@@ -24,12 +24,12 @@ Owns:
- selected-artifact planning and dependency ordering; - selected-artifact planning and dependency ordering;
- per-input resolution and required/optional handling; - per-input resolution and required/optional handling;
- Scriptorium render/run invocation; - Scriptorium render/run invocation;
- run-local output generation and canonical promotion. - run-local output generation and canonical materialization.
Does not own: Does not own:
- prepare-time previous-session hydration; - prepare-time previous-session hydration;
- object-store access for previous-session sources; - object-store access for previous-session sources;
- archive promotion policy. - publish output rule behavior.
## Config fields used ## Config fields used
- `session.session_id` - `session.session_id`
@@ -76,5 +76,5 @@ Does not own:
## Architectural invariants ## Architectural invariants
- Canonical previous-session behavior is local-cache only during analyze. - Canonical previous-session behavior is local-cache only during analyze.
- Generated outputs are validated and promoted before stage success is recorded. - Generated outputs are validated and materialized before stage success is recorded.
- Resolver/catalog decisions stay deterministic and validation-gated. - Resolver/catalog decisions stay deterministic and validation-gated.

6
docs/internal/stage-merge.md
View File

@@ -19,7 +19,7 @@ Owns:
- Per-input normalize calls to Seriatim - Per-input normalize calls to Seriatim
- Final merge call to Seriatim - Final merge call to Seriatim
- Run-local log/config/report path wiring - Run-local log/config/report path wiring
- Promotion of base/report outputs to canonical paths - Materialization of base/report outputs to canonical paths
Does not own: Does not own:
- Transcript polishing or downstream artifact generation - Transcript polishing or downstream artifact generation
@@ -43,7 +43,7 @@ Does not own:
## State and Manifest Behavior ## State and Manifest Behavior
- Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory. - Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory.
- Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled. - Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled.
- Promotes canonical base transcript and optional report. - Materializes canonical base transcript and optional report.
- Records normalized-input provenance and adapter metadata in stage metadata. - Records normalized-input provenance and adapter metadata in stage metadata.
## Skip and Resume Behavior ## Skip and Resume Behavior
@@ -59,5 +59,5 @@ Does not own:
## Architectural Invariants ## Architectural Invariants
- Merge consumes normalized forms of each raw transcript. - Merge consumes normalized forms of each raw transcript.
- Base transcript must validate before promotion. - Base transcript must validate before materialization.
- Report output is optional and gated by config. - Report output is optional and gated by config.

14
docs/internal/stage-prepare.md
View File

@@ -17,7 +17,7 @@ Inputs:
- local: `session.inputs.audio_dir` or `session.inputs.audio_files`; - local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
- S3: `session.inputs.audio_s3.prefix`; - S3: `session.inputs.audio_s3.prefix`;
- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning); - configured enabled Scriptorium artifact inputs (for previous-session requirement scanning);
- remote previous-session current archive state when previous hydration is required. - remote previous-session current publish state when previous hydration is required.
Outputs: Outputs:
- `inputs/campaign.yml`; - `inputs/campaign.yml`;
@@ -41,7 +41,7 @@ Owns:
Does not own: Does not own:
- transcript or artifact generation; - transcript or artifact generation;
- analyze-stage source resolution; - analyze-stage source resolution;
- archive commit behavior. - publish commit behavior.
## Config fields used ## Config fields used
- `session.session_id` - `session.session_id`
@@ -83,10 +83,10 @@ Does not own:
- `narratio.previous_session.artifact.<artifact_key>` - `narratio.previous_session.artifact.<artifact_key>`
- If one or more canonical previous-session requirements exist: - If one or more canonical previous-session requirements exist:
- clears managed `previous/` state; - clears managed `previous/` state;
- hydrates required/optional previous artifacts from the configured previous session’s committed archive current state; - hydrates required/optional previous artifacts from the configured previous session’s committed publish current state;
- writes `previous/manifest.json` and hydrated `previous/artifacts/**`; - writes `previous/manifest.json` and hydrated `previous/artifacts/**`;
- stores archive-relative artifact paths such as `artifacts/session_recap.md` as `previous/artifacts/session_recap.md`, not `previous/artifacts/artifacts/session_recap.md`; - stores publish-relative artifact paths such as `artifacts/session_recap.md` as `previous/artifacts/session_recap.md`, not `previous/artifacts/artifacts/session_recap.md`;
- records hydrated previous inputs in `manifest.Inputs` with source `previous_session_archive.current`. - records hydrated previous inputs in `manifest.Inputs` with source `previous_session_publish.current`.
- If no canonical previous-session requirements exist, prepare does not manage `previous/`. - If no canonical previous-session requirements exist, prepare does not manage `previous/`.
- `manifest.Inputs` is sorted deterministically by `(kind, path)`. - `manifest.Inputs` is sorted deterministically by `(kind, path)`.
- S3 audio `manifest.Inputs` retain S3 provenance and include `cache_path`; `spool_path` is present only when the current prepare invocation downloaded the file. - S3 audio `manifest.Inputs` retain S3 provenance and include `cache_path`; `spool_path` is present only when the current prepare invocation downloaded the file.
@@ -95,7 +95,7 @@ Does not own:
- `previous_session_id` unset: - `previous_session_id` unset:
- if any referenced previous artifact is required: fail; - if any referenced previous artifact is required: fail;
- if all referenced previous artifacts are optional: continue and omit them. - if all referenced previous artifacts are optional: continue and omit them.
- Previous session archive current pointer or manifest missing: - Previous session publish current pointer or manifest missing:
- if any referenced previous artifact is required: fail; - if any referenced previous artifact is required: fail;
- if all referenced previous artifacts are optional: continue and omit missing ones. - if all referenced previous artifacts are optional: continue and omit missing ones.
- Missing required previous artifact object: fail. - Missing required previous artifact object: fail.
@@ -120,5 +120,5 @@ Does not own:
## Architectural invariants ## Architectural invariants
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive. - `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
- Storage keys are computed by callers using archive/path helpers; storage adapter receives explicit keys. - Storage keys are computed by callers using path helpers; storage adapter receives explicit keys.
- `prepare` is the only stage that hydrates canonical previous-session cache state. - `prepare` is the only stage that hydrates canonical previous-session cache state.

62
docs/internal/stage-archive.md → docs/internal/stage-publish.md
View File

@@ -1,4 +1,4 @@
# Stage: archive # Stage: publish
## Purpose ## Purpose
Publish durable run/session state to object storage, then atomically advance remote current state. Publish durable run/session state to object storage, then atomically advance remote current state.
@@ -7,37 +7,37 @@ Publish durable run/session state to object storage, then atomically advance rem
Inputs: Inputs:
- session manifest and prerequisite stage records - session manifest and prerequisite stage records
- run root contents under `runs/{run_id}/` - run root contents under `runs/{run_id}/`
- promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`) - publish output rules with artifact `source` IDs and publish `dest` paths (`pipeline.publish.outputs`)
- effective source-based promotion locks from static config and remote session lock store - effective source-based publish locks from static config and remote session lock store
- session-level `previous/**` cache files when present - session-level `previous/**` cache files when present
Outputs: Outputs:
- uploaded run files under `{session_prefix}/runs/{run_id}/...` - uploaded run files under `{session_prefix}/runs/{run_id}/...`
- uploaded promoted artifacts under `{session_prefix}/...` - uploaded published outputs under `{session_prefix}/...`
- uploaded session previous-cache files under `{session_prefix}/previous/...` when present - uploaded session previous-cache files under `{session_prefix}/previous/...` when present
- `{session_prefix}/current/manifest.json` - `{session_prefix}/current/manifest.json`
- `{session_prefix}/current/run_id.txt` written last - `{session_prefix}/current/run_id.txt` written last
## Boundaries ## Boundaries
Owns: Owns:
- Archive enable/disable gate behavior - publish enable/disable gate behavior
- Prerequisite stage success enforcement - prerequisite stage success enforcement
- Run file collection and upload (excluding `audio/`) - run file collection and upload (excluding `audio/`)
- Promotion rule resolution and upload - publish output rule resolution and upload
- Promotion lock enforcement - publish lock enforcement
- Session previous-cache file collection/upload - session previous-cache file collection/upload
- Commit pointer publish order - commit pointer publish order
Does not own: Does not own:
- Stage execution before archive - stage execution before publish
- Post-archive local cleanup policy execution (handled by app cleanup logic) - post-publish local cleanup policy execution (handled by app cleanup logic)
## Config Fields Used ## Config Fields Used
- `pipeline.archive.enabled` - `pipeline.publish.enabled`
- `pipeline.archive.upload_run` - `pipeline.publish.upload_run`
- `pipeline.archive.promote_artifacts` - `pipeline.publish.outputs`
- `pipeline.archive.locks` - `pipeline.publish.locks`
- `{session_prefix}/locks.yml` loaded by app orchestration before archive execution - `{session_prefix}/locks.yml` loaded by app orchestration before publish execution
- `pipeline.storage.s3.bucket` - `pipeline.storage.s3.bucket`
- `pipeline.storage.s3.root_prefix` - `pipeline.storage.s3.root_prefix`
- `pipeline.workspace.root` - `pipeline.workspace.root`
@@ -51,26 +51,28 @@ Does not own:
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`. - Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
- Resolves bucket/prefix from manifest identity first, then config fallback. - Resolves bucket/prefix from manifest identity first, then config fallback.
- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists. - Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
- Skips top-level promotion uploads for effective locked sources; run-local uploads still publish. - Skips top-level published output uploads for effective locked sources; run-local materialized outputs remain unchanged.
- When selected configured artifact keys are supplied, skips promotion rules for unselected `narratio.artifact.<key>` sources; built-in transcript and bounds promotions still publish. - When selected configured artifact keys are supplied, skips publish rules for unselected `narratio.artifact.<key>` sources; built-in transcript and bounds outputs still publish.
- Effective locks are the union of `pipeline.archive.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources. - Effective locks are the union of `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources.
- Writes metadata including: - Writes metadata including:
- upload counts/paths - upload counts/paths
- `previous_files_uploaded` and `previous_uploaded_paths` - `previous_files_uploaded` and `previous_uploaded_paths`
- `skipped_unselected_promotions` - `published_files_uploaded` and `published_paths`
- `locked_promotion_count` and `locked_promotions` - `skipped_optional_outputs`
- `skipped_unselected_outputs`
- `locked_output_count` and `locked_outputs`
- `current_manifest_key` - `current_manifest_key`
- `current_run_id_key` - `current_run_id_key`
- `current_pointer_written` - `current_pointer_written`
- On skipped archive path, returns metadata with `skipped=true` and pointer not written. - On skipped publish path, returns metadata with `skipped=true` and pointer not written.
## Skip and Resume Behavior ## Skip and Resume Behavior
- Stage may self-skip (metadata skip) when archive disabled or run upload disabled. - Stage may self-skip (metadata skip) when publish disabled or run upload disabled.
- Runner-level skip also applies for previously succeeded stage unless forced. - Runner-level skip also applies for previously succeeded stage unless forced.
## Failure Behavior ## Failure Behavior
- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required promotion source, upload failures, or pointer write failures. - Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required output source, upload failures, or pointer write failures.
- Locked required promotions are intentional skips and do not fail archive. - Locked required outputs are intentional skips and do not fail publish.
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail. - Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
## Tests to Inspect Before Changing ## Tests to Inspect Before Changing
@@ -79,8 +81,8 @@ Does not own:
## Architectural Invariants ## Architectural Invariants
- Run upload excludes `audio/` subtree. - Run upload excludes `audio/` subtree.
- Session `previous/**` is archiveable durable input/provenance state, not run-local output. - Session `previous/**` is publishable durable input/provenance state, not run-local output.
- Ordinary `--force` does not override archive locks. - Ordinary `--force` does not override publish locks.
- Malformed or unreadable remote lock store fails archive-capable execution before promotion. - Malformed or unreadable remote lock store fails publish-capable execution before output uploads.
- `current/manifest.json` uploads before `current/run_id.txt`. - `current/manifest.json` uploads before `current/run_id.txt`.
- `current/run_id.txt` is the remote publish commit marker. - `current/run_id.txt` is the remote publish commit marker.

8
docs/internal/stage-transcribe.md
View File

@@ -15,7 +15,7 @@ Owns:
- Discovering prepared audio inputs - Discovering prepared audio inputs
- Deriving speaker ids from audio basenames - Deriving speaker ids from audio basenames
- Parallel WhisperX invocation with bounded concurrency - Parallel WhisperX invocation with bounded concurrency
- Validating produced JSON and promoting run-local outputs - Validating produced JSON and materializing run-local outputs
Does not own: Does not own:
- Transcript merge/polish/normalize/trim/analyze - Transcript merge/polish/normalize/trim/analyze
@@ -36,8 +36,8 @@ Does not own:
## State and Manifest Behavior ## State and Manifest Behavior
- Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled. - Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled.
- Validates each generated transcript JSON before promotion. - Validates each generated transcript JSON before materialization.
- Promotes canonical outputs to `transcripts/raw/*.json`. - Materializes canonical outputs to `transcripts/raw/*.json`.
- Records per-file metadata (attempts/status/duration/output path) in stage metadata. - Records per-file metadata (attempts/status/duration/output path) in stage metadata.
## Skip and Resume Behavior ## Skip and Resume Behavior
@@ -54,5 +54,5 @@ Does not own:
## Architectural Invariants ## Architectural Invariants
- Speaker identity is derived from `.flac` basename and must be unique. - Speaker identity is derived from `.flac` basename and must be unique.
- Every successful speaker output must be valid JSON before promotion. - Every successful speaker output must be valid JSON before materialization.
- Canonical raw transcript set is the only supported merge input surface. - Canonical raw transcript set is the only supported merge input surface.

2
docs/internal/stage-trim.md
View File

@@ -52,7 +52,7 @@ Does not own:
## State and Manifest Behavior ## State and Manifest Behavior
- Reads final transcript from normalize manifest outputs when available; falls back to canonical path. - Reads final transcript from normalize manifest outputs when available; falls back to canonical path.
- Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled. - Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled.
- Promotes canonical final-trimmed transcript; promotes session bounds when trim enabled. - Materializes canonical final-trimmed transcript and session bounds when trim is enabled.
- Records bounds diagnostics, trim action, keep selector, and adapter metadata. - Records bounds diagnostics, trim action, keep selector, and adapter metadata.
## Skip and Resume Behavior ## Skip and Resume Behavior

8
docs/internal/storage.md
View File

@@ -37,12 +37,12 @@ Does not own:
## External adapters used ## External adapters used
Storage package contracts: Storage package contracts:
- `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`. - `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`.
- `Backend` (archive request boundary): currently implemented with `NoopBackend` only. - `Backend` (legacy compatibility boundary): currently implemented with `NoopBackend` only.
Implementations: Implementations:
- `S3Backend`: AWS SDK-backed `ObjectStore` implementation. - `S3Backend`: AWS SDK-backed `ObjectStore` implementation.
- `FakeBackend`: deterministic test `ObjectStore` and archive backend. - `FakeBackend`: deterministic test `ObjectStore` and compatibility backend.
- `NoopBackend`: deterministic no-op archive backend for compatibility wiring. - `NoopBackend`: deterministic no-op compatibility backend for wiring/tests.
## State and manifest behavior ## State and manifest behavior
- Storage implementations are stateless with respect to manifest/session lifecycle. - Storage implementations are stateless with respect to manifest/session lifecycle.
@@ -67,7 +67,7 @@ Implementations:
- `internal/adapters/storage/s3_backend_test.go` - `internal/adapters/storage/s3_backend_test.go`
- `internal/adapters/storage/fake_test.go` - `internal/adapters/storage/fake_test.go`
- `internal/adapters/storage/keys_test.go` - `internal/adapters/storage/keys_test.go`
- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `archive`) - `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `publish`)
## Architectural invariants ## Architectural invariants
- Callers pass full bucket-relative keys. - Callers pass full bucket-relative keys.

18
docs/internal/workspace.md
View File

@@ -1,7 +1,7 @@
# Workspace internals # Workspace internals
## Purpose ## Purpose
Define the local durable and run-local workspace model used by stages, manifests, resume, and archive. Define the local durable and run-local workspace model used by stages, manifests, resume, and publish.
## Inputs and Outputs ## Inputs and Outputs
Inputs: Inputs:
@@ -24,14 +24,14 @@ Owns:
Does not own: Does not own:
- Stage business logic - Stage business logic
- Remote archive semantics (documented in `stage-archive.md`) - Remote publish semantics (documented in `stage-publish.md`)
- CLI argument parsing - CLI argument parsing
## Config Fields Used ## Config Fields Used
- `pipeline.workspace.root` - `pipeline.workspace.root`
- `pipeline.workspace.cleanup_after_archive` - `pipeline.workspace.cleanup_after_publish`
- `pipeline.spool.root` - `pipeline.spool.root`
- `pipeline.spool.delete_audio_after_archive` - `pipeline.spool.delete_audio_after_publish`
- `pipeline.cache.root` - `pipeline.cache.root`
- `pipeline.cache.s3_audio` - `pipeline.cache.s3_audio`
- `session.campaign` - `session.campaign`
@@ -43,10 +43,10 @@ None directly in this subsystem. Stages may use object storage adapters and then
## State and Manifest Behavior ## State and Manifest Behavior
- Session state is persisted in the session manifest (`manifest.Manifest`). - Session state is persisted in the session manifest (`manifest.Manifest`).
- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`. - Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`.
- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and promoted to canonical session paths after stage success. - During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and then materialized to canonical session paths after stage success.
- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs. - `manifest.Artifacts` entries record `ProducerRunID` for durable outputs.
- For S3 audio sessions, `prepare` records work/cache paths, S3 provenance, and spool path when the invocation downloaded the object. - For S3 audio sessions, `prepare` records work/cache paths, S3 provenance, and spool path when the invocation downloaded the object.
- `previous/**` is reconstructed from configured previous-session requirements; restore uses the previous session's committed current archive rather than treating current-session archived `previous/**` as authoritative. - `previous/**` is reconstructed from configured previous-session requirements; restore uses the previous session's committed current publish state rather than treating current-session stored `previous/**` as authoritative.
- Durable cache state under `pipeline.cache.root` is not workspace state and is preserved by default by `narratio clean`. - Durable cache state under `pipeline.cache.root` is not workspace state and is preserved by default by `narratio clean`.
- `narratio clean <id>` removes the session work root and session spool root. - `narratio clean <id>` removes the session work root and session spool root.
- `narratio clean --all` removes all local session work under `workspace.root/work` and spool children under `spool.root`. - `narratio clean --all` removes all local session work under `workspace.root/work` and spool children under `spool.root`.
@@ -60,7 +60,7 @@ None directly in this subsystem. Stages may use object storage adapters and then
## Failure Behavior ## Failure Behavior
- Failures preserve manifests and run-local files for inspection. - Failures preserve manifests and run-local files for inspection.
- Lock conflicts fail fast via `ErrLockConflict`. - Lock conflicts fail fast via `ErrLockConflict`.
- Cleanup can fail post-archive; failure is recorded in archive stage metadata and returned by the run. - Cleanup can fail post-publish; failure is recorded in publish stage metadata and returned by the run.
## Tests to Inspect Before Changing ## Tests to Inspect Before Changing
- `internal/artifacts/local_test.go` - `internal/artifacts/local_test.go`
@@ -72,7 +72,7 @@ None directly in this subsystem. Stages may use object storage adapters and then
## Architectural Invariants ## Architectural Invariants
- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`. - Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`.
- Run roots are always nested: `runs/{run_id}` under the session root. - Run roots are always nested: `runs/{run_id}` under the session root.
- Run-local output promotion must end in canonical session paths. - Run-local output materialization must end in canonical session paths.
- `previous/**` is session-durable state and must not be treated as run-local output scratch state. - `previous/**` is session-durable state and must not be treated as run-local output scratch state.
- Automatic post-archive cleanup only targets run-scoped directories and must never delete configured root directories. - Automatic post-publish cleanup only targets run-scoped directories and must never delete configured root directories.
- Manual `clean` may delete session-scoped directories or the `workspace.root/work` directory, but it must preserve configured root directories and reject unsafe targets. - Manual `clean` may delete session-scoped directories or the `workspace.root/work` directory, but it must preserve configured root directories and reject unsafe targets.

401
docs/operations.md
View File

@@ -1,285 +1,170 @@
# Operations # Operations
This guide describes the implemented operator lifecycle for Narratio. This guide covers the implemented operator lifecycle for Narratio.
For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md). For field-level settings, see [docs/config.md](./config.md). For syntax/flags, see [docs/cli.md](./cli.md).
## Normal workflow (S3-first path) ## Normal Workflow
1. Create or upload `session.yml`, or pass a local `session.yml` explicitly. 1. Ensure `pipeline.yml`, `campaign.yml`, and `session.yml` are available.
2. Upload session `.flac` files to object storage under the configured session audio prefix. 2. Ensure session audio is available (local `audio_dir`/`audio_files` or S3 prefix).
3. Run Narratio: 3. Run:
```bash ```bash
narratio run 2026-04-04 narratio run 2026-04-04
``` ```
4. Read success output: 4. Inspect status:
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
- use `narratio session status <session_id>` for inspection.
Notes:
- default pipeline/session discovery checks system config locations; campaign selection uses `pipeline.campaigns.default_campaign_id` unless `--campaign <id>` or `--campaign-file <path>` is passed.
- when local `session.yml` discovery misses, positional `<session_id>` loads remote `session.yml` from `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
Initialize a remote session skeleton:
```bash
narratio session init 2026-04-04 --remote
```
Remote init uses normal default config discovery and writes `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`. If `campaign.yml` sets `session_template_file`, init renders that template from the supplied flags and writes concrete YAML. Pass `--config`, `--campaign <id>`, or `--campaign-file <path>` when testing non-system config files. It fails if the object already exists unless `--force` is passed.
Validate before running:
```bash
narratio session validate 2026-04-04
```
## Restore workflow
Use restore when local durable session state is missing or stale and archive current state is authoritative.
Dry-run (no local writes):
```bash
narratio session restore 2026-04-04 --dry-run
```
Execution:
```bash
narratio session restore 2026-04-04
```
Post-restore analyze rerun pattern:
```bash
narratio analyze 2026-04-04
```
Restore source-of-truth:
- remote commit marker: `current/run_id.txt`
- remote current manifest: `current/manifest.json`
- configured previous-session requirements are reconstructed from the previous session's remote `current/` state, not from archived `previous/**` objects in the current session.
Restore default scope:
- includes `manifest.json`, `transcripts/**`, and `artifacts/**` from the current session archive.
- includes `previous/**` only when configured previous-session artifact inputs require it; restore hydrates those files the same way `prepare` would.
- includes `audio/**` only with `--include-audio`
- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source)
Reset local state before restore testing:
```bash
narratio clean 2026-04-04 --dry-run
narratio clean 2026-04-04
narratio session restore 2026-04-04 --include-audio
```
`clean` removes the local session work directory and session spool directory. It preserves the durable S3 audio cache by default, so repeated restore or forced prepare tests do not re-download large audio files.
## Local filesystem layout and state artifacts
Session root:
- `{workspace.root}/work/{campaign}/{session_id}/`
Primary state:
- `manifest.json`: session-level stage state.
- `runs/{run_id}/manifest.json`: invocation-level state.
- `.lock`: session lock while a modifying command is active.
- `inputs/campaign.yml`, `inputs/session.yml`, and `inputs/pipeline.resolved.yml`: materialized config inputs for the run.
Canonical session directories:
- `inputs/`
- `audio/`
- `transcripts/`
- `artifacts/`
- `previous/`
- `reports/`
- `logs/`
- `config/`
- `current/`
- `runs/`
Run-local stage directories:
- `runs/{run_id}/{stage}/` with stage-local `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`.
Behavior:
- directory creation is idempotent.
- stage outputs are generally generated run-local first, then promoted to canonical paths on success.
- restore installs downloaded files to canonical session paths and does not recreate historical run sandboxes.
## Analyze artifact execution lifecycle
Analyze executes configured artifacts from `pipeline.scriptorium.artifacts`.
Execution model:
- executable set = enabled artifacts, filtered by `--artifacts` when provided.
- artifact-to-artifact dependencies are declared via `depends_on`.
- selected artifacts run in deterministic dependency order.
- after each successful artifact run, output is promoted to configured canonical `output_path`.
Configured artifact source reuse:
- a non-executable configured artifact can satisfy inputs if its configured output file already exists and is valid.
- reused configured artifact provenance is `filesystem.disabled_artifact_output`.
`--artifacts` behavior:
- accepted on `run`, `resume`, `run-stage analyze`, `run-stage archive`, `analyze`, and `publish`.
- filters analyze execution and configured artifact promotions.
- built-in transcript and bounds promotions are not filtered.
- does not imply force on `run`, `resume`, or `run-stage`; `narratio analyze` is force-by-design.
- `publish` is force-by-design and accepts `--artifacts` for configured artifact promotions.
Canonical previous-session input behavior:
- canonical sources use `narratio.previous_session.artifact.<artifact_key>`.
- these inputs are hydrated by `prepare` and by `restore`; `analyze` expects the local previous cache to already exist.
- if analyze fails due to missing canonical previous cache, rerun:
- `narratio run-stage prepare <id> --force`
- or `narratio session restore <id>` when remote archive current state is authoritative.
## Remote archive layout and publish contract
Preferred manual publish command:
```bash
narratio publish <id>
```
`publish` is equivalent to `narratio run-stage archive <id> --force`; use `run-stage` when you need the general single-stage command form.
When archive is enabled and run upload is enabled, archive publishes under:
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
- run prefix: `{session_prefix}/runs/{run_id}/`
Archive uploads:
- run record files from run root (excluding `audio/`).
- promoted files from explicit `archive.promote_artifacts` rules.
- mutable session locks from helper commands live at `{session_prefix}/locks.yml`.
Publish order:
1. upload `current/manifest.json`
2. upload `current/run_id.txt` last
`current/run_id.txt` is the remote commit marker.
Archive promotion is explicit and source-based:
- Narratio does not auto-promote all generated analyze artifacts.
- each rule resolves `source` through the artifact resolver/catalog model, then uploads to `dest`.
- missing required promotion sources fail archive stage.
- missing optional promotion sources are skipped.
- invalid resolved artifacts fail archive stage.
- `archive.locks` skips top-level promotion overwrites for locked sources while run-local uploads still publish.
- remote locks from `{session_prefix}/locks.yml` are merged with static `archive.locks`; static locks win on duplicate sources.
- locked required promotions are treated as intentional successful skips and are recorded in archive metadata.
Lock helper behavior:
- `narratio session locks <id>` lists effective static and remote locks.
- `narratio session locks add <id> <source> --reason <text>` writes a remote lock.
- `narratio session locks add <id> <source> --force --reason <text>` updates an existing remote lock reason.
- `narratio session locks remove <id> <source>` removes only a remote lock.
- `locks remove` cannot remove static pipeline locks.
- remote lock writes check whether the lock store exists, but are not compare-and-swap atomic.
## Resume, retry, restore, and safe rerun behavior
Default skip:
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set.
Resume:
- `resume` starts at first non-succeeded stage.
- `resume --force` runs full stage order.
Restore conflict policy:
- restore classifies local differences as conflicts.
- without `--force`, restore fails when conflicts exist.
- with `--force`, conflicting local files are overwritten by remote archive files.
Forced reruns:
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
- ordinary `--force` does not override archive locks.
Safe rerun pattern:
1. rerun the changed stage with `--force`.
2. run `resume` to rebuild downstream stages.
## Cleanup behavior
Automatic post-archive cleanup is considered only when archive stage executed and succeeded.
Automatic cleanup toggles:
- `pipeline.spool.delete_audio_after_archive=true` deletes run-scoped spool audio.
- `pipeline.workspace.cleanup_after_archive=true` deletes run-scoped local run directory.
Manual cleanup:
- `narratio clean <id>` deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`.
- `narratio clean --all` deletes all local session work under `{workspace.root}/work` and all spool children under `{spool.root}`.
- `--dry-run` prints targets without deleting.
- `--clear-cache` also removes matching S3 audio cache files. Without it, cache is preserved.
The S3 audio cache under `pipeline.cache.root` is durable input cache state, not workspace or spool state. Automatic cleanup and default manual cleanup do not delete it.
Cleanup eligibility gates:
- archive enabled
- archive run upload enabled
- run record upload completed
- current pointer write completed (`current/run_id.txt` written)
No cleanup for failed/incomplete/unarchived/archive-skipped runs.
## Failure and recovery playbooks
After run failure, Narratio keeps:
- session manifest
- run manifest
- run-local artifacts/logs/config/reports
Failed or incomplete runs remain local-only.
After restore failure:
- already-installed restore files remain in place.
- restore does not roll back prior successful installs.
- existing local manifest is preserved if restored manifest validation/install fails.
Recommended recovery:
1. inspect state:
```bash ```bash
narratio session status 2026-04-04 narratio session status 2026-04-04
``` ```
This reports local manifest state, committed remote current state, expected remote transcript/artifact availability, and archive locks. ## Publish Workflow
2. for restore-specific checks, run: Publish is the stage that commits remote current state.
```bash
narratio publish 2026-04-04
```
Equivalent command:
```bash
narratio run-stage publish 2026-04-04 --force
```
Publish uploads:
- run history files under `{session_prefix}/runs/{run_id}/` (excluding `audio/`)
- configured published outputs from `pipeline.publish.outputs`
- `previous/**` cache files when present
- `current/manifest.json`
- `current/run_id.txt` last
`current/run_id.txt` is the remote commit marker.
## Published Outputs and Locks
Published output behavior:
- outputs are source-based rules in `pipeline.publish.outputs`.
- required missing unlocked sources fail publish.
- optional missing unlocked sources are skipped.
- selected artifacts (`--artifacts`) only filter configured `narratio.artifact.<key>` output rules.
- built-in transcript and bounds output rules are not filtered by `--artifacts`.
Lock behavior:
- static locks: `pipeline.publish.locks`.
- mutable locks: `{session_prefix}/locks.yml`.
- effective lock set is static + mutable; static wins on duplicate sources.
- locked outputs are intentional skips and do not fail publish.
- lock commands mutate only remote mutable locks.
## Restore Workflow
Use restore when local durable session state is missing/stale and committed remote current state is authoritative.
Preview:
```bash ```bash
narratio session restore 2026-04-04 --dry-run narratio session restore 2026-04-04 --dry-run
``` ```
3. fix root cause (config/input/credentials/storage/service availability). Apply:
4. continue with `resume`, or targeted `run-stage <stage> <id> --force` followed by `resume`.
## Restore report ```bash
narratio session restore 2026-04-04
```
Non-dry-run restore writes a durable report at: Default restore scope:
- `reports/restore-latest.json`
Report content includes: - `manifest.json`
- identity (`campaign`, `session_id`, `run_id`) - `transcripts/**`
- mode flags (`dry_run`, `force`, `include_audio`) - `artifacts/**`
- plan counts and execution counts - `previous/**` when required by configured previous-session artifact inputs
- per-action status
Dry-run does not write restore report files. Optional:
## Operational caveats - add `--include-audio` to restore `audio/**`.
- `session status <session_id>` uses normal config/session loading, including remote session fallback. Restore reads committed current state only (`current/run_id.txt`, `current/manifest.json`).
- `session status <session_id>` includes the same promoted remote output availability view as `session artifacts <session_id> --remote` when storage is configured.
- local and S3 audio input modes are mutually exclusive. ## Workspace and State Layout
- archive publish requires upstream stages through `analyze` to be `succeeded`.
- required configured artifact promotions for unselected `--artifacts` keys are skipped intentionally; selected required promotions still fail if their files are missing. Session root:
- restore requires configured remote object storage and committed remote current state.
- `{workspace.root}/work/{campaign}/{session_id}/`
Durable session state:
- `manifest.json`
- `inputs/**`
- `audio/**`
- `transcripts/**`
- `artifacts/**`
- `previous/**`
- `reports/**`
- `logs/**`
- `config/**`
- `runs/**`
Run-local stage layout:
- `runs/{run_id}/{stage}/outputs|logs|reports|config|scratch`
Stages typically write run-local outputs first, then materialize canonical session outputs on success.
## Resume and Force Rules
- `run` and `run-stage` skip succeeded stages unless `--force` is set.
- `resume` starts at the first non-succeeded stage.
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
- `--force` does not bypass publish locks.
## Cleanup
Automatic post-publish cleanup is considered only when publish executes successfully and commits current state.
Config toggles:
- `pipeline.spool.delete_audio_after_publish=true`
- `pipeline.workspace.cleanup_after_publish=true`
Manual cleanup:
```bash
narratio clean 2026-04-04
narratio clean --all
```
Cache is preserved by default. Use `--clear-cache` to remove matching S3 audio cache entries.
## Failure and Recovery
After stage failure, Narratio keeps manifests and run-local files for inspection.
Standard recovery flow:
1. inspect status:
```bash
narratio session status 2026-04-04
```
2. if needed, inspect restore plan:
```bash
narratio session restore 2026-04-04 --dry-run
```
3. fix root cause.
4. continue with `resume`, or rerun a stage with `--force` then `resume`.
## Operational Caveats
- local and S3 audio modes are mutually exclusive.
- publish requires prerequisite stages through `analyze` to be `succeeded`.
- restore requires configured object storage and committed current state.
- `session status` and `session artifacts --remote` both report remote published-output availability when storage is configured.

356
docs/troubleshooting.md
View File

@@ -1,17 +1,17 @@
# Troubleshooting # Troubleshooting
## Purpose ## Purpose
Canonical operator troubleshooting guide for recurring implemented Narratio failures. Canonical operator troubleshooting guide for recurring Narratio failures.
## Config file discovery failure ## Config discovery failure
Symptom: Symptom:
- `run`, `resume`, `run-stage`, `session plan`, or `session restore` fails with config/session not found. - command fails because `pipeline.yml`, `campaign.yml`, or `session.yml` was not found.
Likely Cause: Likely cause:
- `pipeline.yml` or `session.yml` is missing from system discovery paths. - missing files in discovery paths.
- the selected campaign ID does not exist under `pipeline.campaigns.root`. - missing/incorrect campaign selection.
- a local working-directory config file was not passed explicitly. - local file exists but was not passed explicitly.
Diagnostics: Diagnostics:
@@ -20,160 +20,73 @@ ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
ls -l /usr/local/etc/narratio/session.yml /etc/narratio/session.yml ls -l /usr/local/etc/narratio/session.yml /etc/narratio/session.yml
``` ```
Safe Fix: Safe fix:
- pass explicit `--config`, `--campaign <id>`, `--campaign-file <path>`, and `--session` as appropriate. - pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`.
- or place files in documented discovery paths and set `pipeline.campaigns.default_campaign_id`.
Links:
- [docs/config.md](./config.md)
- [docs/cli.md](./cli.md)
## Templated session file rejected ## Templated session file rejected
Symptom: Symptom:
- load fails with a message that `session.yml must be concrete`. - load fails because `session.yml` must be concrete.
Likely Cause: Likely cause:
- a template authoring file such as `session.template.yml` was passed to `--session` or uploaded as remote `session.yml`. - template placeholders (`{{ ... }}`) still present in loaded session config.
- `session.yml` still contains `{{ ... }}` placeholders.
Diagnostics: Diagnostics:
```bash ```bash
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session ./session.yml narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
``` ```
Safe Fix: Safe fix:
- generate concrete YAML with `narratio session init`. - generate concrete session YAML via `narratio session init`.
- pass the generated concrete `session.yml` to downstream commands or upload it through `session init --remote`.
Links: ## Strict decode or validation failure
- [docs/config.md](./config.md)
## Strict YAML decode or validation failure
Symptom: Symptom:
- config load fails with unknown field or validation error. - unknown field or invalid value error during config load.
Likely Cause: Likely cause:
- typo/stale field name. - typo, stale field name, or invalid value.
- missing required fields or invalid constraints.
Diagnostics: Diagnostics:
```bash ```bash
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
``` ```
Safe Fix: Safe fix:
- align fields/values to canonical config reference and examples. - align config with [docs/config.md](./config.md) and maintained examples.
Links:
- [docs/config.md](./config.md)
- [examples/](../examples/)
## `--artifacts` selection failure ## `--artifacts` selection failure
Symptom: Symptom:
- `run`/`resume`/`run-stage` fails with invalid or unknown artifact selection. - command fails on unknown/invalid selected artifact key.
Likely Cause: Likely cause:
- `--artifacts` contains blank names or unknown artifact keys. - artifact key not defined in `pipeline.scriptorium.artifacts`.
- `pipeline.scriptorium.artifacts` missing while using `--artifacts`. - empty token in `--artifacts` input.
Diagnostics: Safe fix:
- use only configured artifact keys.
```bash ## `run-stage --artifacts` unsupported stage
narratio run 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts player_handout
```
Safe Fix:
- use configured artifact keys only.
- ensure `pipeline.scriptorium.artifacts` is defined.
Links:
- [docs/cli.md](./cli.md)
- [docs/config.md](./config.md)
## `run-stage --artifacts` on unsupported stage
Symptom: Symptom:
- `run-stage` fails because `--artifacts` is only supported for `analyze` and `archive`. - `run-stage` rejects `--artifacts` for the selected stage.
Likely Cause: Likely cause:
- `--artifacts` was used with a stage other than `analyze` or `archive`. - `--artifacts` used with a stage other than `analyze` or `publish`.
Diagnostics: Safe fix:
- use `--artifacts` only with `run-stage analyze ...` or `run-stage publish ...`.
```bash ## Previous-session input unavailable
narratio run-stage polish 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts session_recap
```
Safe Fix:
- use `--artifacts` only with `run-stage analyze ...` or `run-stage archive ...`.
Links:
- [docs/cli.md](./cli.md)
## Configured artifact dependency/input validation failure
Symptom: Symptom:
- config validation fails for `depends_on`, `narratio.artifact.<name>` source, or artifact output path. - analyze fails on required previous-session artifact input.
Likely Cause: Likely cause:
- `narratio.artifact.<name>` source missing matching `depends_on` key. - `previous/**` cache not hydrated for this session.
- dependency references unknown artifact key.
- dependency self-reference or enabled dependency cycle.
- artifact output path missing/invalid/outside `artifacts/` root.
Diagnostics:
```bash
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
```
Safe Fix:
- ensure artifact-to-artifact inputs have explicit `depends_on` entries using artifact keys.
- ensure referenced artifacts exist and define valid `output_path` values.
- keep output paths relative and under `artifacts/`.
Links:
- [docs/config.md](./config.md)
- [docs/internal/stage-analyze.md](./internal/stage-analyze.md)
## Required configured artifact input unavailable at analyze time
Symptom:
- analyze fails because configured input source is unavailable.
Likely Cause:
- required upstream configured artifact was not selected/executed this run.
- non-executable dependency output file is missing or invalid on disk.
Diagnostics:
```bash
narratio session status 2026-04-04
narratio run-stage analyze 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts player_handout
```
Safe Fix:
- run analyze with needed artifacts selected.
- or ensure dependency output file exists at configured path and is valid.
Links:
- [docs/operations.md](./operations.md)
- [docs/config.md](./config.md)
## Manifest/status path failure
Symptom:
- `session status` fails because config/session state is missing, unreadable, or invalid.
Likely Cause:
- wrong session ID.
- wrong config/campaign/session file selected.
- manifest removed after cleanup.
Diagnostics: Diagnostics:
@@ -181,137 +94,75 @@ Diagnostics:
narratio session status 2026-04-04 narratio session status 2026-04-04
``` ```
Safe Fix: Safe fix:
- use the same session ID and config files that will be used for `run`, `resume`, or `run-stage`.
Links: ```bash
- [docs/cli.md](./cli.md) narratio run-stage prepare 2026-04-04 --force
- [docs/operations.md](./operations.md) ```
Or rehydrate from remote current state:
```bash
narratio session restore 2026-04-04
```
## Session lock conflict (`.lock`) ## Session lock conflict (`.lock`)
Symptom: Symptom:
- `run`, `resume`, `run-stage`, or `session restore` fails with lock conflict for session workdir. - command fails with lock conflict.
Likely Cause: Likely cause:
- another Narratio process is running same session. - another process is running for the same session.
- stale lock from interrupted prior run. - stale lock file from interrupted command.
Diagnostics: Diagnostics:
```bash ```bash
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
cat {workspace.root}/work/{campaign}/{session_id}/.lock
ps aux | grep narratio ps aux | grep narratio
``` ```
Safe Fix: Safe fix:
- wait for active process to finish. - wait for active process; remove stale lock only if no process is active.
- if no process is active, remove only stale session `.lock` file.
Links: ## Restore current pointer/manifest missing
- [docs/operations.md](./operations.md)
- [docs/internal/workspace.md](./internal/workspace.md)
## Restore remote current pointer or manifest missing
Symptom: Symptom:
- `session restore` fails with remote current pointer or current manifest errors. - restore fails reading remote current state.
Likely Cause: Likely cause:
- `current/run_id.txt` was never published. - publish commit did not complete.
- `current/manifest.json` is missing for the session prefix. - `current/run_id.txt` or `current/manifest.json` is missing.
- archive commit did not complete.
Diagnostics: Diagnostics:
```bash ```bash
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run narratio session restore 2026-04-04 --dry-run
``` ```
Safe Fix: Safe fix:
- verify archive stage succeeded for the target session. - republish from a healthy local session state.
- rerun/archive from a healthy source workspace so current pointers are published.
Links:
- [docs/operations.md](./operations.md)
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
## Restore manifest identity mismatch
Symptom:
- `session restore` fails because remote manifest session or campaign does not match requested values.
Likely Cause:
- wrong positional session ID or wrong session config selected.
- archive prefix points to a different campaign/session.
Diagnostics:
```bash
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run
```
Safe Fix:
- use the correct session config and positional session ID.
- verify campaign/session identity in local config before restore.
Links:
- [docs/config.md](./config.md)
- [docs/operations.md](./operations.md)
## Restore conflict without `--force` ## Restore conflict without `--force`
Symptom: Symptom:
- `session restore` fails with `restore conflict` and conflict counts. - restore reports conflict and exits.
Likely Cause: Likely cause:
- local durable file differs from remote file for one or more planned restore paths. - local durable file differs from remote restore source.
Diagnostics: Safe fix:
- inspect with `--dry-run`.
- rerun with `--force` only when remote should overwrite local.
```bash ## Secrets or credentials failure
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run
```
Safe Fix:
- review planned conflicts.
- rerun with `--force` only when remote state should overwrite local state.
Links:
- [docs/cli.md](./cli.md)
- [docs/operations.md](./operations.md)
## Restore report expectations
Symptom: Symptom:
- operator expects restore report file but does not find one. - startup fails loading secrets dir, or storage/tool auth fails at runtime.
Likely Cause: Likely cause:
- restore was executed in `--dry-run` mode. - invalid `pipeline.secrets.env_dir`.
- restore failed before report persistence path (for example lock acquisition failure). - missing credential env vars.
Diagnostics:
```bash
ls -l {workspace.root}/work/{campaign}/{session_id}/reports/restore-latest.json
```
Safe Fix:
- run non-dry-run restore for durable report output.
- resolve lock or early preflight failures and retry.
Links:
- [docs/operations.md](./operations.md)
## Secrets env-dir or credential-env failure
Symptom:
- startup fails loading secrets directory, or stage fails due to missing credential env vars.
Likely Cause:
- invalid `pipeline.secrets.env_dir` path/permissions.
- required credential env var unset/empty.
Diagnostics: Diagnostics:
@@ -320,60 +171,53 @@ ls -la /path/to/secrets_dir
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM' env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
``` ```
Safe Fix: Safe fix:
- fix secrets directory and credential env vars. - fix path/permissions/env vars; keep secret values out of YAML.
- keep secret values out of YAML.
Links: ## S3 audio prepare failure
- [docs/config.md](./config.md)
## S3-audio prepare failure
Symptom: Symptom:
- `prepare` fails in S3 mode (listing/downloading/no audio/backend error). - prepare fails in S3 mode (list/download/no files/backend error).
Likely Cause: Likely cause:
- wrong `session.inputs.audio_s3.prefix`. - bad `session.inputs.audio_s3.prefix`.
- no `.flac` files at resolved prefix. - no `.flac` objects at prefix.
- invalid/missing object-store credentials or backend config. - bad storage credentials/config.
- mixed local+S3 audio input config. - mixed local+S3 audio config.
Diagnostics: Diagnostics:
```bash ```bash
narratio run-stage prepare 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml narratio run-stage prepare 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
``` ```
Safe Fix: Safe fix:
- configure exactly one audio source mode. - configure exactly one audio mode and verify storage access.
- verify `.flac` files and storage access.
Links: ## Publish output or current-pointer failure
- [docs/config.md](./config.md)
- [docs/operations.md](./operations.md)
## Archive promotion/current-pointer failure
Symptom: Symptom:
- archive fails on required promotion source missing or pointer write failure. - publish fails on required output source missing, upload error, or commit-marker write failure.
Likely Cause: Likely cause:
- required promoted file absent (including analyze outputs not generated for this run). - required source file not produced.
- storage upload failed before `current/run_id.txt` commit marker write. - storage upload failed before `current/run_id.txt` write.
Diagnostics: Diagnostics:
```bash ```bash
narratio session status 2026-04-04 narratio session status 2026-04-04
narratio run-stage archive 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml narratio run-stage publish 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
``` ```
Safe Fix: Safe fix:
- rerun or resume upstream stages to generate required files. - rerun upstream stages to regenerate required outputs.
- adjust promotion `source`/`dest` rules to match artifacts that must exist. - adjust `pipeline.publish.outputs` source/dest rules.
- retry after storage issue is resolved. - retry after storage issue is fixed.
## Helpful Links
Links:
- [docs/operations.md](./operations.md)
- [docs/config.md](./config.md) - [docs/config.md](./config.md)
- [docs/internal/stage-archive.md](./internal/stage-archive.md) - [docs/cli.md](./cli.md)
- [docs/operations.md](./operations.md)
- [docs/internal/stage-publish.md](./internal/stage-publish.md)

2
examples/pipeline.full.annotated.yml
View File

@@ -43,7 +43,7 @@ publish:
# Optional booleans; defaults are true. # Optional booleans; defaults are true.
enabled: true enabled: true
upload_run: true upload_run: true
# Optional promotion rules; sources use Narratio artifact source IDs. # Optional publish output rules; sources use Narratio artifact source IDs.
outputs: outputs:
- source: narratio.transcript.final_trimmed - source: narratio.transcript.final_trimmed
dest: transcripts/final.trimmed.json dest: transcripts/final.trimmed.json