From d723384888d8ead5b6cf16a3d34b24d62b8c6ec2 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 23 May 2026 12:50:46 +0000 Subject: [PATCH] Rewrite user and operator documentation for current CLI and config behavior --- README.md | 14 +- docs/cli.md | 180 ++++++++++++++--------- docs/config.md | 212 +++++++++++++++------------ docs/operations.md | 229 +++++++++++++++++++---------- docs/troubleshooting.md | 310 +++++++++++++++++++++++----------------- 5 files changed, 577 insertions(+), 368 deletions(-) diff --git a/README.md b/README.md index 4bcc0d9..ca1d5c6 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,22 @@ # narratio -Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts. +Narratio is a stage-driven Go orchestrator for turning D&D session audio into polished transcripts and generated artifacts. -It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, publish-stage uploads, and resumable run state in one operator workflow. +It runs a deterministic workflow across `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`, and `publish`, with manifest-driven resume and restore support. ```bash narratio run 2026-04-04 ``` -This command requires discoverable `pipeline.yml` and `session.yml` files (or explicit `--config` and `--session` flags). +This requires resolvable `pipeline.yml`, `campaign.yml`, and concrete `session.yml` (or explicit config flags). ## Documentation -- [Configuration](docs/config.md) - [CLI Reference](docs/cli.md) -- [Operations and Recovery](docs/operations.md) +- [Configuration](docs/config.md) +- [Operations](docs/operations.md) - [Troubleshooting](docs/troubleshooting.md) +- [Internal Component Contracts](docs/internal/README.md) - [Development Guide](docs/policy/development.md) - [Architecture Principles](docs/policy/architecture.md) -- [Internal Component Contracts](docs/internal/README.md) -- [Config Examples](examples/) +- [Maintained Examples](examples/) diff --git a/docs/cli.md b/docs/cli.md index 0fd907d..3efa400 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -1,4 +1,4 @@ -# CLI +# CLI Reference ## Shortest Useful Command @@ -6,18 +6,19 @@ narratio run 2026-04-04 ``` -This runs the full pipeline for the given session ID using default config discovery and campaign selection. +This runs the canonical full pipeline for session `2026-04-04`. ## Command Overview + Top-level commands: -- `run `: execute the pipeline. +- `run `: run full stage order. - `resume `: continue from first non-succeeded stage. -- `run-stage `: execute exactly one stage. -- `analyze `: force-rerun analyze stage. -- `publish `: force-rerun publish stage. -- `clean |--all`: remove local workspace/spool state. -- `session `: session-scoped helper commands. +- `run-stage `: run one stage. +- `analyze `: force-run analyze. +- `publish `: force-run publish. +- `clean ` or `clean --all`: remove local work/spool state. +- `session `: session helper commands. Session subcommands: @@ -31,7 +32,8 @@ Session subcommands: - `session locks add ` - `session locks remove ` -## Common Flags +## Common Config Flags + Most session-aware commands accept: - `--config ` @@ -40,30 +42,41 @@ Most session-aware commands accept: - `--session ` - `--previous-session-id ` -`--campaign` and `--campaign-file` are mutually exclusive. +Rules: + +- `--campaign` and `--campaign-file` are mutually exclusive. +- `--session` is not used by `session init`. +- `clean --all` cannot be combined with campaign/session selectors. ## Command Reference ### `run` ```bash -narratio run [--force] [--artifacts ] [...common flags] +narratio run [--force] [--artifacts ] [...common config flags] ``` -Runs stages in canonical order and writes manifest state. +Behavior: + +- evaluates full stage order; +- skips already-succeeded stages unless `--force` is set; +- writes session and run manifests. ### `resume` ```bash -narratio resume [--force] [--artifacts ] [...common flags] +narratio resume [--force] [--artifacts ] [...common config flags] ``` -Starts at the first non-succeeded stage from the session manifest. +Behavior: + +- when not forced, starts at first non-succeeded stage in manifest order; +- with `--force`, reevaluates the selected stage list as runnable. ### `run-stage` ```bash -narratio run-stage [--force] [--artifacts ] [...common flags] +narratio run-stage [--force] [--artifacts ] [...common config flags] ``` Valid stage names: @@ -78,123 +91,161 @@ Valid stage names: - `publish` - `notify` -`--artifacts` is accepted only for `analyze` and `publish`. +Rules: + +- `--artifacts` is accepted only for `analyze` and `publish` stage targets. ### `analyze` ```bash -narratio analyze [--artifacts ] [...common flags] +narratio analyze [--artifacts ] [...common config flags] ``` -Equivalent to `narratio run-stage analyze --force`. +Equivalent to: + +```bash +narratio run-stage analyze --force [...common config flags] +``` ### `publish` ```bash -narratio publish [--artifacts ] [...common flags] +narratio publish [--artifacts ] [...common config flags] ``` -Equivalent to `narratio run-stage publish --force`. +Equivalent to: + +```bash +narratio run-stage publish --force [...common config flags] +``` ### `clean` ```bash -narratio clean [--dry-run] [--clear-cache] [...common flags] +narratio clean [--dry-run] [--clear-cache] [...common config flags] narratio clean --all [--dry-run] [--clear-cache] [--config ] ``` -- session mode deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`. -- `--all` deletes all session work and spool children. -- cache is preserved unless `--clear-cache` is passed. +Behavior: + +- session mode removes: + - `{workspace.root}/work/{campaign}/{session_id}` + - `{spool.root}/{campaign}/{session_id}` +- `--all` removes: + - `{workspace.root}/work/*` + - direct children under `{spool.root}` +- cache remains unless `--clear-cache` is provided. ### `session plan` ```bash -narratio session plan [--force] [...common flags] +narratio session plan [--force] [...common config flags] ``` -Validates config and session inputs, prepares workdir layout, and prints stage run/skip decisions. +Validates config, prepares local workdir layout, and prints run/skip decisions for each stage. ### `session validate` ```bash -narratio session validate [...common flags] +narratio session validate [...common config flags] ``` -Read-only preflight checks for config, inputs, audio availability, previous-session requirements, publish outputs, and effective locks. +Read-only preflight checks for config validity, required inputs, audio mode, previous-session requirements, publish outputs, and effective locks. ### `session status` ```bash -narratio session status [...common flags] +narratio session status [...common config flags] ``` -Shows local manifest state, remote current state (when storage is configured), published-output availability, and effective locks. +Prints local manifest state and, when storage is available, remote current-state and published-output status. ### `session init` ```bash -narratio session init --output ./session.yml -narratio session init --remote -narratio session init --remote --force +narratio session init --output ./session.yml [options] +narratio session init --remote [options] ``` -Flags: +Required target selection: -- `--output ` or `--remote` (exactly one is required) +- exactly one of: + - `--output ` + - `--remote` + +Options: + +- `--config ` +- `--campaign ` or `--campaign-file ` - `--previous-session-id ` -- `--date ` -- `--title ` +- `--date <YYYY-MM-DD>` +- `--title <text>` - `--audio-dir <path>` - `--audio-s3-prefix <prefix>` - `--force` -- common config/campaign flags + +Rules: + +- `--audio-dir` and `--audio-s3-prefix` are mutually exclusive. +- if campaign `session_template_file` is configured, `session init` renders it; +- generated session YAML must be concrete (no unresolved `{{ ... }}` placeholders). ### `session restore` ```bash -narratio session restore <session_id> [--dry-run] [--force] [--include-audio] [...common flags] +narratio session restore <session_id> [--dry-run] [--force] [--include-audio] [...common config flags] ``` -Restores durable local session files from committed remote current state. +Behavior: + +- discovers committed remote current state; +- plans local restores; +- writes `reports/restore-latest.json` on execution; +- blocks conflicting overwrites unless `--force` is set. Default restore scope: - `manifest.json` - `transcripts/**` - `artifacts/**` -- `previous/**` when required by configured previous-session artifact inputs +- `previous/**` when required by configured previous-session inputs -`audio/**` is restored only when `--include-audio` is set. +`audio/**` is included only with `--include-audio`. ### `session artifacts` ```bash -narratio session artifacts <session_id> [--remote] [...common flags] +narratio session artifacts <session_id> [--remote] [...common config flags] ``` -Lists built-in sources, configured artifact sources, previous-session sources, publish output rules, and lock status. With `--remote`, includes remote published-state markers. +Lists effective built-in and configured artifact sources, publish rules, lock state, and optional remote published-state availability. ### `session locks` ```bash -narratio session locks <session_id> [...common flags] -narratio session locks add <session_id> <source> [--reason <text>] [--force] [...common flags] -narratio session locks remove <session_id> <source> [...common flags] +narratio session locks <session_id> [...common config flags] +narratio session locks add <session_id> <source> [--reason <text>] [--force] [...common config flags] +narratio session locks remove <session_id> <source> [...common config flags] ``` -- list mode prints effective locks from static `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`. -- add/remove mutate only the remote lock store. -- static pipeline locks cannot be removed by lock commands. +Behavior: -## `--artifacts` Rules +- list mode merges static `pipeline.publish.locks` with remote `{session_prefix}/locks.yml`; +- add/remove mutate only remote locks; +- static locks from pipeline config cannot be removed by CLI commands. -- accepted on `run`, `resume`, `run-stage`, `analyze`, and `publish`. -- on `run-stage`, only valid for `analyze` and `publish`. -- filters configured analyze artifact execution. -- filters configured `pipeline.publish.outputs` entries for `narratio.artifact.<key>` sources. -- does not suppress built-in transcript/bounds publish outputs. -- does not imply `--force` for `run`, `resume`, or `run-stage`. +## `--artifacts` Selection Rules + +- accepted on `run`, `resume`, `run-stage`, `analyze`, and `publish`; +- names must exist in `pipeline.scriptorium.artifacts`; +- empty entries are invalid; +- repeated names are deduplicated. + +Effects: + +- filters analyze execution to selected configured artifacts; +- filters publish rules that source `narratio.artifact.<name>`; +- does not filter built-in transcript/bounds publish sources. ## Common Workflows @@ -204,16 +255,16 @@ Run full pipeline: narratio run 2026-04-04 ``` -Run only selected analyze artifacts: +Dry-run restore plan: ```bash -narratio run 2026-04-04 --artifacts session_recap,player_handout +narratio session restore 2026-04-04 --dry-run ``` -Force analyze only: +Generate a concrete session file from template/default structure: ```bash -narratio analyze 2026-04-04 --artifacts player_handout +narratio session init 2026-04-04 --output ./session.yml --date 2026-04-04 --title "Session 12" ``` Force publish only: @@ -221,10 +272,3 @@ Force publish only: ```bash narratio publish 2026-04-04 ``` - -Restore preview then apply: - -```bash -narratio session restore 2026-04-04 --dry-run -narratio session restore 2026-04-04 -``` diff --git a/docs/config.md b/docs/config.md index 4707a72..648dfaf 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1,56 +1,56 @@ -# Configuration +# Configuration Reference -## Overview -Narratio loads three YAML files: +## Purpose -- `pipeline.yml`: pipeline-level runtime settings. -- `campaign.yml`: stable campaign identity and campaign-level input defaults. -- `session.yml`: per-session metadata and input selection. +Narratio resolves three YAML documents: -Commands that load and validate all three files include: +- `pipeline.yml`: pipeline/runtime settings +- `campaign.yml`: campaign identity and stable input defaults +- `session.yml`: session identity, metadata, and audio source selection -- `narratio run` -- `narratio resume` -- `narratio run-stage` -- `narratio analyze` -- `narratio publish` -- `narratio session plan` -- `narratio session status` -- `narratio session validate` -- `narratio session restore` -- `narratio session artifacts` -- `narratio session locks` -- `narratio clean <session_id>` +## Discovery and Selection -Validation behavior: +### `pipeline.yml` -- strict YAML decode is enabled (`KnownFields(true)`); unknown fields fail. -- loaded `session.yml` files must be concrete YAML (no `{{ ... }}` placeholders). -- defaults are applied for optional pipeline fields. -- campaign/session identity mismatches fail load. - -## File Discovery -Pipeline discovery order when `--config` is omitted: +When `--config` is omitted, search order is: 1. `/usr/local/etc/narratio/pipeline.yml` 2. `/etc/narratio/pipeline.yml` -Session discovery order when `--session` is omitted: +### `campaign.yml` + +Selection rules: + +- if `--campaign-file` is set, use that path; +- else if `--campaign <id>` is set, use `{pipeline.campaigns.root}/{id}/campaign.yml`; +- else use `{pipeline.campaigns.root}/{pipeline.campaigns.default_campaign_id}/campaign.yml`. + +### `session.yml` + +When `--session` is omitted, local search order is: 1. `/usr/local/etc/narratio/session.yml` 2. `/etc/narratio/session.yml` -Campaign discovery when `--campaign-file` is omitted: +If local session discovery fails and a `session_id` is known, Narratio attempts remote session loading from: -- if `--campaign <id>` is set: `{pipeline.campaigns.root}/{id}/campaign.yml` -- otherwise: `{pipeline.campaigns.root}/{pipeline.campaigns.default_campaign_id}/campaign.yml` +- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml` -Remote `session.yml` fallback: +using configured object storage. -- if local session discovery fails and storage is configured, Narratio can load: - - `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml` +## Validation and Merge Rules + +- YAML decode is strict (`KnownFields(true)`): unknown fields fail load. +- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load. +- Pipeline defaults are applied before validation. +- Campaign and session identities must agree. +- Stable files (`speakers_file`, `autocorrect_file`, `glossary_file`) resolve from session overrides when provided, otherwise from campaign defaults. +- Exactly one audio mode must be configured in session input: + - local (`audio_dir` or `audio_files`), or + - S3 (`audio_s3.prefix`). + +## Minimal Working Configuration -## Minimal Working Config `pipeline.yml` ```yaml @@ -58,7 +58,7 @@ campaigns: root: /usr/local/share/narratio/campaigns default_campaign_id: sample-campaign whisperx: - transcribe_url: "https://transcription.example.com/transcribe" + transcribe_url: https://transcription.example.com/transcribe ``` `campaign.yml` @@ -71,7 +71,7 @@ inputs: glossary_file: ./glossary.yml ``` -`session.yml` +`session.yml` (local audio) ```yaml session_id: 2026-05-03 @@ -79,8 +79,16 @@ inputs: audio_dir: ./audio ``` -## Publish Config -Top-level publish settings live at `pipeline.publish`. +## Secrets Handling + +- Do not place raw secrets in YAML. +- Use env var names in config (for example `pipeline.audita.llm_api_key_env`). +- Optionally load env files from `pipeline.secrets.env_dir`. +- Commands that need storage/auth load filesystem secrets before constructing adapters. + +## Publish Configuration Summary + +Publish rules live under `pipeline.publish`. ```yaml publish: @@ -95,26 +103,21 @@ publish: required: true locks: - source: narratio.artifact.session_recap - reason: Final recap was manually edited. + reason: manual post-publish edits ``` Rules: - `outputs[].source` is required. -- `outputs[].dest` is optional; when omitted, Narratio derives destination from the source. +- `outputs[].dest` may be omitted when derivable from source. - `outputs[].required` defaults to `true`. -- static `publish.locks` and remote `{session_prefix}/locks.yml` are merged; static locks win on duplicates. -- locks prevent overwrite of top-level published destinations. +- static locks (`pipeline.publish.locks`) merge with remote locks (`{session_prefix}/locks.yml`), with static locks taking precedence on duplicates. -Supported publish source families: - -- built-ins: `narratio.transcript.base`, `narratio.transcript.polished`, `narratio.transcript.final`, `narratio.transcript.final_trimmed`, `narratio.bounds.session` -- configured artifacts: `narratio.artifact.<artifact_key>` - -## Full Reference +## Full Schema ### Pipeline -| Path | Type | Required | Default | + +| Field | Type | Required | Default / Rule | | --- | --- | --- | --- | | `pipeline.workspace.root` | string | No | `/var/lib/narratio` | | `pipeline.workspace.cleanup_after_publish` | bool | No | `false` | @@ -122,7 +125,7 @@ Supported publish source families: | `pipeline.campaigns.default_campaign_id` | string | No | empty | | `pipeline.secrets.env_dir` | string | No | empty | | `pipeline.storage.backend` | string | No | empty | -| `pipeline.storage.s3.bucket` | string | Conditional | empty | +| `pipeline.storage.s3.bucket` | string | Conditional | required for S3 session-audio and for publish upload when backend is `s3` | | `pipeline.storage.s3.root_prefix` | string | No | `dnd` | | `pipeline.storage.s3.region` | string | No | empty | | `pipeline.storage.s3.endpoint` | string | No | empty | @@ -135,21 +138,21 @@ Supported publish source families: | `pipeline.cache.s3_audio` | bool | No | `true` | | `pipeline.publish.enabled` | bool | No | `true` | | `pipeline.publish.upload_run` | bool | No | `true` | -| `pipeline.publish.outputs[]` | list | No | one final-trimmed output rule | -| `pipeline.publish.outputs[].source` | string | Yes (per rule) | none | -| `pipeline.publish.outputs[].dest` | string | No | derived from source | +| `pipeline.publish.outputs[]` | list | No | defaults to final trimmed transcript output | +| `pipeline.publish.outputs[].source` | string | Yes (per rule) | must reference built-in or configured artifact source | +| `pipeline.publish.outputs[].dest` | string | Conditional | derived if omitted and source supports derivation | | `pipeline.publish.outputs[].required` | bool | No | `true` | | `pipeline.publish.locks[]` | list | No | empty | -| `pipeline.publish.locks[].source` | string | Yes (per lock) | none | +| `pipeline.publish.locks[].source` | string | Yes (per lock) | must reference supported publish source | | `pipeline.publish.locks[].reason` | string | No | empty | -| `pipeline.whisperx.transcribe_url` | string | Yes | none | +| `pipeline.whisperx.transcribe_url` | string | Yes | valid URL | | `pipeline.whisperx.language` | string | No | `en` | -| `pipeline.whisperx.timeout` | duration string | No | `30m` | +| `pipeline.whisperx.timeout` | duration | No | `30m` | | `pipeline.whisperx.retries` | int | No | `3` | -| `pipeline.whisperx.retry_delay` | duration string | No | `2s` | +| `pipeline.whisperx.retry_delay` | duration | No | `2s` | | `pipeline.whisperx.concurrency` | int | No | `2` | | `pipeline.seriatim.binary` | string | No | `seriatim` | -| `pipeline.seriatim.timeout` | duration string | No | `10m` | +| `pipeline.seriatim.timeout` | duration | No | `10m` | | `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` | | `pipeline.seriatim.coalesce_gap` | float | No | `3.0` | | `pipeline.seriatim.report` | bool | No | `true` | @@ -158,7 +161,7 @@ Supported publish source families: | `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset | | `pipeline.seriatim.env.filler_max_duration` | float | No | unset | | `pipeline.audita.binary` | string | No | `audita` | -| `pipeline.audita.timeout` | duration string | No | `3h` | +| `pipeline.audita.timeout` | duration | No | `3h` | | `pipeline.audita.llm_api_key_env` | string | No | empty | | `pipeline.audita.modules[]` | list[string] | No | empty | | `pipeline.audita.base_url` | string | No | empty | @@ -176,52 +179,85 @@ Supported publish source families: | `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` | | `pipeline.normalize.report` | bool | No | `true` | | `pipeline.trim.enabled` | bool | No | `false` | -| `pipeline.trim.output_path` | string | Conditional | none | -| `pipeline.trim.bounds.prompt_id` | string | Conditional | none | +| `pipeline.trim.output_path` | string | Conditional | required when trim enabled | +| `pipeline.trim.bounds.prompt_id` | string | Conditional | required when trim enabled | | `pipeline.trim.bounds.profile_id` | string | No | empty | -| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | none | -| `pipeline.trim.bounds.output_path` | string | Conditional | none | -| `pipeline.trim.bounds.timeout` | duration string | No | `10m` | +| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | required when trim enabled | +| `pipeline.trim.bounds.output_path` | string | Conditional | required when trim enabled | +| `pipeline.trim.bounds.timeout` | duration | No | `10m` | | `pipeline.trim.bounds.render_debug` | bool | No | `false` | -| `pipeline.trim.bounds.render_output_path` | string | Conditional | none | +| `pipeline.trim.bounds.render_output_path` | string | Conditional | required when `render_debug` is true | | `pipeline.trim.seriatim.report` | bool | No | `false` | | `pipeline.scriptorium.binary` | string | No | `scriptorium` | | `pipeline.scriptorium.config_path` | string | No | empty | -| `pipeline.scriptorium.timeout` | duration string | No | `10m` | +| `pipeline.scriptorium.timeout` | duration | No | `10m` | | `pipeline.scriptorium.render_debug` | bool | No | `false` | | `pipeline.scriptorium.artifacts` | map | No | empty | | `pipeline.notification.backend` | string | No | empty | | `pipeline.notification.recipient` | string | No | empty | -| `pipeline.notification.timeout` | duration string | No | `30s` | +| `pipeline.notification.timeout` | duration | No | `30s` | + +### Scriptorium Artifact Entries + +For each `pipeline.scriptorium.artifacts.<name>`: + +| Field | Type | Required | Rule | +| --- | --- | --- | --- | +| `enabled` | bool | No | `false` if omitted | +| `depends_on[]` | list[string] | No | must reference configured artifact keys; no self-reference; enabled graph must be acyclic | +| `render_debug` | bool | No | per-artifact override | +| `prompt_id` | string | Conditional | required when artifact is enabled | +| `profile_id` | string | No | empty | +| `output_path` | string | Conditional | required when enabled; also required when referenced by publish/output/input rules | +| `timeout` | duration | No | artifact override | +| `inputs` | map | No | input key names must be non-empty | +| `vars` | map | No | values must be string or bool | + +For each artifact input `pipeline.scriptorium.artifacts.<name>.inputs.<input_name>`: + +| Field | Type | Required | Rule | +| --- | --- | --- | --- | +| `source` | string | Yes | built-in runtime source, `narratio.artifact.<name>`, or `narratio.previous_session.artifact.<name>` | +| `artifact` | string | No | optional passthrough adapter field | +| `path` | string | No | optional passthrough adapter field | +| `required` | bool | No | optional input requirement | ### Campaign -| Path | Type | Required | -| --- | --- | --- | -| `campaign_id` | string | Yes | -| `session_template_file` | string | No | -| `inputs.speakers_file` | string | Yes | -| `inputs.autocorrect_file` | string | Yes | -| `inputs.glossary_file` | string | Yes | + +| Field | Type | Required | Notes | +| --- | --- | --- | --- | +| `campaign_id` | string | Yes | canonical campaign identity | +| `session_template_file` | string | No | used by `session init` when set | +| `inputs.speakers_file` | string | Yes | stable input default | +| `inputs.autocorrect_file` | string | Yes | stable input default | +| `inputs.glossary_file` | string | Yes | stable input default | ### Session -| Path | Type | Required | -| --- | --- | --- | -| `session_id` | string | Yes | -| `previous_session_id` | string | No | -| `campaign` | string | No | -| `date` | string | No | -| `title` | string | No | -| `inputs.speakers_file` | string | No | -| `inputs.autocorrect_file` | string | No | -| `inputs.glossary_file` | string | No | -| `inputs.audio_dir` | string | Conditional | -| `inputs.audio_files[]` | list[string] | Conditional | -| `inputs.audio_s3.prefix` | string | Conditional | + +| Field | Type | Required in session file | Notes | +| --- | --- | --- | --- | +| `session_id` | string | Yes | must match CLI session target when provided | +| `previous_session_id` | string | No | must not equal `session_id` | +| `campaign` | string | No | filled from `campaign_id` during resolve if omitted | +| `date` | string | No | metadata | +| `title` | string | No | metadata | +| `inputs.speakers_file` | string | No | overrides campaign stable input | +| `inputs.autocorrect_file` | string | No | overrides campaign stable input | +| `inputs.glossary_file` | string | No | overrides campaign stable input | +| `inputs.audio_dir` | string | Conditional | local audio mode | +| `inputs.audio_files[]` | list[string] | Conditional | local audio mode | +| `inputs.audio_s3.prefix` | string | Conditional | S3 audio mode | + +Audio rules: + +- configure local mode (`audio_dir` or `audio_files`) or S3 mode (`audio_s3.prefix`), not both. ## Maintained Examples + - `examples/pipeline.minimal.yml` - `examples/pipeline.production.yml` - `examples/pipeline.full.annotated.yml` - `examples/campaigns/sample-campaign/campaign.yml` - `examples/session.local-audio.yml` - `examples/session.s3-audio.yml` +- `examples/session.template.yml` diff --git a/docs/operations.md b/docs/operations.md index 4e1ce07..454d5ac 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,72 +1,156 @@ -# Operations +# Operations Guide -This guide covers the implemented operator lifecycle for Narratio. +Operator workflow for running, recovering, and publishing Narratio sessions. -For field-level settings, see [docs/config.md](./config.md). For syntax/flags, see [docs/cli.md](./cli.md). +For command syntax, see [docs/cli.md](./cli.md). For field-level config, see [docs/config.md](./config.md). -## Normal Workflow +## Standard Session Workflow -1. Ensure `pipeline.yml`, `campaign.yml`, and `session.yml` are available. -2. Ensure session audio is available (local `audio_dir`/`audio_files` or S3 prefix). -3. Run: +1. Select pipeline/campaign/session config. +2. Validate session readiness: + +```bash +narratio session validate 2026-04-04 +``` + +3. (Optional) inspect stage decisions: + +```bash +narratio session plan 2026-04-04 +``` + +4. Run the pipeline: ```bash narratio run 2026-04-04 ``` -4. Inspect status: +5. Check state: ```bash narratio session status 2026-04-04 ``` +## Campaign and Session Selection + +Campaign selection priority: + +- `--campaign-file` +- `--campaign` +- `pipeline.campaigns.default_campaign_id` + +Session source priority: + +- `--session` +- local default search paths +- remote session object (S3) when local session file is not found and storage is configured + +## Session Initialization + +Use `session init` to generate a concrete session file for local or remote use. + +Local file: + +```bash +narratio session init 2026-04-04 --output ./session.yml --date 2026-04-04 --title "Session 12" +``` + +Remote session object: + +```bash +narratio session init 2026-04-04 --remote --force +``` + +If `campaign.yml` sets `session_template_file`, `session init` renders it. Template variables must resolve to concrete values. + +## Stage Execution and Resume Behavior + +Canonical stage order: + +1. `prepare` +2. `transcribe` +3. `merge` +4. `polish` +5. `normalize` +6. `trim` +7. `analyze` +8. `publish` +9. `notify` + +Execution rules: + +- succeeded stages are skipped unless `--force` is set; +- `resume` starts at first non-succeeded stage; +- force rerunning a succeeded upstream stage marks succeeded downstream stages as `stale`. + +Single-stage execution: + +```bash +narratio run-stage normalize 2026-04-04 --force +``` + +## Artifact Selection + +`--artifacts` can be used on `run`, `resume`, `run-stage`, `analyze`, and `publish`. + +Selection behavior: + +- validates names against `pipeline.scriptorium.artifacts`; +- filters analyze execution to selected configured artifacts; +- filters publish rules for `narratio.artifact.<name>` sources only; +- does not suppress built-in transcript or bounds publish sources. + ## Publish Workflow -Publish is the stage that commits remote current state. +Run publish only: ```bash narratio publish 2026-04-04 ``` -Equivalent command: +Equivalent: ```bash narratio run-stage publish 2026-04-04 --force ``` -Publish uploads: +Publish commit model: -- run history files under `{session_prefix}/runs/{run_id}/` (excluding `audio/`) -- configured published outputs from `pipeline.publish.outputs` -- `previous/**` cache files when present -- `current/manifest.json` -- `current/run_id.txt` last +- uploads run files under `{session_prefix}/runs/{run_id}/`; +- uploads configured published outputs; +- uploads `previous/**` cache files when present; +- writes `current/manifest.json`; +- writes `current/run_id.txt` last. -`current/run_id.txt` is the remote commit marker. +`current/run_id.txt` is the remote current-state commit marker. -## Published Outputs and Locks +## Publish Locks -Published output behavior: +Lock sources: -- outputs are source-based rules in `pipeline.publish.outputs`. -- required missing unlocked sources fail publish. -- optional missing unlocked sources are skipped. -- selected artifacts (`--artifacts`) only filter configured `narratio.artifact.<key>` output rules. -- built-in transcript and bounds output rules are not filtered by `--artifacts`. +- static locks in `pipeline.publish.locks` +- mutable remote locks in `{session_prefix}/locks.yml` -Lock behavior: +Effective lock rules: -- static locks: `pipeline.publish.locks`. -- mutable locks: `{session_prefix}/locks.yml`. -- effective lock set is static + mutable; static wins on duplicate sources. -- locked outputs are intentional skips and do not fail publish. -- lock commands mutate only remote mutable locks. +- static and remote locks are merged; +- static locks win on source collisions; +- locked outputs are intentional skips; +- lock add/remove commands mutate only remote lock state. + +Examples: + +```bash +narratio session locks 2026-04-04 +narratio session locks add 2026-04-04 narratio.artifact.session_recap --reason "manual edits" --force +narratio session locks remove 2026-04-04 narratio.artifact.session_recap +``` ## Restore Workflow -Use restore when local durable session state is missing/stale and committed remote current state is authoritative. +Use restore when local durable session state is missing or stale and remote committed current state is authoritative. -Preview: +Dry run: ```bash narratio session restore 2026-04-04 --dry-run @@ -83,21 +167,22 @@ Default restore scope: - `manifest.json` - `transcripts/**` - `artifacts/**` -- `previous/**` when required by configured previous-session artifact inputs +- `previous/**` when needed by configured previous-session artifact inputs Optional: -- add `--include-audio` to restore `audio/**`. +- `--include-audio` to include `audio/**` +- `--force` to overwrite local conflicts -Restore reads committed current state only (`current/run_id.txt`, `current/manifest.json`). +Restore writes an execution report at `reports/restore-latest.json`. -## Workspace and State Layout +## Local State Layout Session root: -- `{workspace.root}/work/{campaign}/{session_id}/` +- `{workspace.root}/work/{campaign}/{session_id}` -Durable session state: +Durable session paths: - `manifest.json` - `inputs/**` @@ -110,61 +195,55 @@ Durable session state: - `config/**` - `runs/**` -Run-local stage layout: +Run-local layout: -- `runs/{run_id}/{stage}/outputs|logs|reports|config|scratch` +- `runs/{run_id}/{stage}/outputs` +- `runs/{run_id}/{stage}/logs` +- `runs/{run_id}/{stage}/reports` +- `runs/{run_id}/{stage}/config` +- `runs/{run_id}/{stage}/scratch` -Stages typically write run-local outputs first, then materialize canonical session outputs on success. +Spool layout (runtime/transient): -## Resume and Force Rules +- `{spool.root}/{campaign}/{session_id}/{run_id}/...` +- restore audio spool under `{spool.root}/{campaign}/{session_id}/restore/audio` -- `run` and `run-stage` skip succeeded stages unless `--force` is set. -- `resume` starts at the first non-succeeded stage. -- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`. -- `--force` does not bypass publish locks. +Cache layout (durable S3 audio cache): + +- `{cache.root}/s3/{bucket}/...` ## Cleanup -Automatic post-publish cleanup is considered only when publish executes successfully and commits current state. - -Config toggles: - -- `pipeline.spool.delete_audio_after_publish=true` -- `pipeline.workspace.cleanup_after_publish=true` - -Manual cleanup: +Session-scoped cleanup: ```bash narratio clean 2026-04-04 +``` + +Global cleanup: + +```bash narratio clean --all ``` -Cache is preserved by default. Use `--clear-cache` to remove matching S3 audio cache entries. - -## Failure and Recovery - -After stage failure, Narratio keeps manifests and run-local files for inspection. - -Standard recovery flow: - -1. inspect status: +Dry-run and cache variants: ```bash -narratio session status 2026-04-04 +narratio clean 2026-04-04 --dry-run --clear-cache +narratio clean --all --dry-run --clear-cache ``` -2. if needed, inspect restore plan: +Rules: -```bash -narratio session restore 2026-04-04 --dry-run -``` - -3. fix root cause. -4. continue with `resume`, or rerun a stage with `--force` then `resume`. +- `clean` deletes work/spool session state; +- cache is preserved unless `--clear-cache` is set; +- automatic post-publish cleanup is gated by successful publish commit plus: + - `pipeline.spool.delete_audio_after_publish=true` + - `pipeline.workspace.cleanup_after_publish=true` ## Operational Caveats -- local and S3 audio modes are mutually exclusive. -- publish requires prerequisite stages through `analyze` to be `succeeded`. -- restore requires configured object storage and committed current state. -- `session status` and `session artifacts --remote` both report remote published-output availability when storage is configured. +- Local and S3 audio modes are mutually exclusive. +- Publish requires prerequisite stages through analyze to be succeeded. +- Restore requires configured object storage and committed remote current state. +- Storage-backed commands load filesystem secrets before object-store initialization. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index ae7e0e0..f16bc78 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,35 +1,58 @@ # Troubleshooting -## Purpose -Canonical operator troubleshooting guide for recurring Narratio failures. +Operational diagnosis guide for common Narratio failures. -## Config discovery failure +## Config file not found Symptom: -- command fails because `pipeline.yml`, `campaign.yml`, or `session.yml` was not found. -Likely cause: -- missing files in discovery paths. -- missing/incorrect campaign selection. -- local file exists but was not passed explicitly. +- command fails to resolve `pipeline.yml`, `campaign.yml`, or `session.yml`. + +Likely causes: + +- missing files in default search paths; +- wrong campaign selection; +- omitted explicit flags. Diagnostics: ```bash -ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml -ls -l /usr/local/etc/narratio/session.yml /etc/narratio/session.yml +narratio session plan 2026-04-04 ``` Safe fix: + - pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`. -## Templated session file rejected +## Session template placeholders rejected Symptom: -- load fails because `session.yml` must be concrete. + +- load error says session file must be concrete or contains `{{ ... }}` placeholders. Likely cause: -- template placeholders (`{{ ... }}`) still present in loaded session config. + +- using template content as runtime session config. + +Diagnostics: + +```bash +narratio session validate 2026-04-04 --session /path/session.yml +``` + +Safe fix: + +- generate concrete session YAML with `narratio session init`. + +## Strict decode or schema validation failure + +Symptom: + +- unknown field / invalid value error during config load. + +Likely cause: + +- stale field name, typo, invalid enum, or invalid duration/path format. Diagnostics: @@ -38,82 +61,79 @@ narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /pa ``` Safe fix: -- generate concrete session YAML via `narratio session init`. -## Strict decode or validation failure +- align config with [docs/config.md](./config.md) and maintained files under `examples/`. + +## Audio mode conflict Symptom: -- unknown field or invalid value error during config load. + +- validation fails on session audio configuration. Likely cause: -- typo, stale field name, or invalid value. - -Diagnostics: - -```bash -narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml -``` - -Safe fix: -- align config with [docs/config.md](./config.md) and maintained examples. - -## `--artifacts` selection failure - -Symptom: -- command fails on unknown/invalid selected artifact key. - -Likely cause: -- artifact key not defined in `pipeline.scriptorium.artifacts`. -- empty token in `--artifacts` input. - -Safe fix: -- use only configured artifact keys. - -## `run-stage --artifacts` unsupported stage - -Symptom: -- `run-stage` rejects `--artifacts` for the selected stage. - -Likely cause: -- `--artifacts` used with a stage other than `analyze` or `publish`. - -Safe fix: -- use `--artifacts` only with `run-stage analyze ...` or `run-stage publish ...`. - -## Previous-session input unavailable - -Symptom: -- analyze fails on required previous-session artifact input. - -Likely cause: -- `previous/**` cache not hydrated for this session. + +- configured both local and S3 session audio inputs. + +Safe fix: + +- use local mode (`audio_dir` or `audio_files`) or S3 mode (`audio_s3.prefix`), not both. + +## `--artifacts` selection error + +Symptom: + +- unknown artifact key or invalid `--artifacts` usage. + +Likely causes: + +- key not defined in `pipeline.scriptorium.artifacts`; +- empty list entry (for example trailing comma); +- `run-stage` used with non-`analyze`/`publish` target. + +Safe fix: + +- provide only configured keys and use `--artifacts` with supported commands/stages. + +## Previous-session artifact input missing + +Symptom: + +- prepare/analyze fails due to missing required previous-session artifact cache input. + +Likely causes: + +- missing `session.previous_session_id`; +- previous artifact not restored/published for source session. Diagnostics: ```bash +narratio session validate 2026-04-04 narratio session status 2026-04-04 ``` Safe fix: ```bash -narratio run-stage prepare 2026-04-04 --force +narratio session restore 2026-04-04 ``` -Or rehydrate from remote current state: +or rerun prepare after correcting session config: ```bash -narratio session restore 2026-04-04 +narratio run-stage prepare 2026-04-04 --force ``` ## Session lock conflict (`.lock`) Symptom: -- command fails with lock conflict. -Likely cause: -- another process is running for the same session. -- stale lock file from interrupted command. +- command fails acquiring session lock. + +Likely causes: + +- another process is running for the same session; +- stale lock left by interrupted process. Diagnostics: @@ -123,16 +143,19 @@ ps aux | grep narratio ``` Safe fix: -- wait for active process; remove stale lock only if no process is active. -## Restore current pointer/manifest missing +- wait for active process completion; +- remove stale lock only after confirming no live process owns it. + +## Restore conflict without `--force` Symptom: -- restore fails reading remote current state. + +- restore fails with conflict count. Likely cause: -- publish commit did not complete. -- `current/run_id.txt` or `current/manifest.json` is missing. + +- local durable files differ from remote restore sources. Diagnostics: @@ -141,83 +164,110 @@ narratio session restore 2026-04-04 --dry-run ``` Safe fix: -- republish from a healthy local session state. -## Restore conflict without `--force` +- review conflicts; +- rerun with `--force` only when remote state should overwrite local. + +## Restore current-state discovery failure Symptom: -- restore reports conflict and exits. -Likely cause: -- local durable file differs from remote restore source. +- restore cannot find current pointer or current manifest. -Safe fix: -- inspect with `--dry-run`. -- rerun with `--force` only when remote should overwrite local. +Likely causes: -## Secrets or credentials failure - -Symptom: -- startup fails loading secrets dir, or storage/tool auth fails at runtime. - -Likely cause: -- invalid `pipeline.secrets.env_dir`. -- missing credential env vars. - -Diagnostics: - -```bash -ls -la /path/to/secrets_dir -env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM' -``` - -Safe fix: -- fix path/permissions/env vars; keep secret values out of YAML. - -## S3 audio prepare failure - -Symptom: -- prepare fails in S3 mode (list/download/no files/backend error). - -Likely cause: -- bad `session.inputs.audio_s3.prefix`. -- no `.flac` objects at prefix. -- bad storage credentials/config. -- mixed local+S3 audio config. - -Diagnostics: - -```bash -narratio run-stage prepare 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml -``` - -Safe fix: -- configure exactly one audio mode and verify storage access. - -## Publish output or current-pointer failure - -Symptom: -- publish fails on required output source missing, upload error, or commit-marker write failure. - -Likely cause: -- required source file not produced. -- storage upload failed before `current/run_id.txt` write. +- no committed publish current state; +- storage credentials or connectivity failure. Diagnostics: ```bash narratio session status 2026-04-04 -narratio run-stage publish 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml +narratio session restore 2026-04-04 --dry-run ``` Safe fix: -- rerun upstream stages to regenerate required outputs. -- adjust `pipeline.publish.outputs` source/dest rules. -- retry after storage issue is fixed. -## Helpful Links +- resolve storage/auth issue; +- republish from healthy local state if current pointer is missing. + +## Publish output failure + +Symptom: + +- publish fails on missing required source, upload error, or commit write. + +Likely causes: + +- required source file not produced; +- lock/state expectations mismatch; +- remote storage failure. + +Diagnostics: + +```bash +narratio session artifacts 2026-04-04 --remote +narratio session status 2026-04-04 +narratio run-stage publish 2026-04-04 --force +``` + +Safe fix: + +- regenerate missing sources by rerunning prerequisite stages; +- correct publish source/destination rules; +- retry after storage failure is resolved. + +## Secrets or storage credential failure + +Symptom: + +- object-store command fails at initialization/auth. + +Likely causes: + +- invalid `pipeline.secrets.env_dir`; +- missing credential environment variables; +- invalid S3 endpoint/bucket settings. + +Diagnostics: + +```bash +ls -la /path/to/secrets_dir +env | grep -E 'OBJECT_STORAGE|AWS|AUDITA|SCRIPTORIUM' +``` + +Safe fix: + +- correct secret-file path and permissions; +- provide required env vars; +- keep secret values out of YAML. + +## S3 audio prepare failure + +Symptom: + +- prepare fails listing/downloading session S3 audio. + +Likely causes: + +- incorrect `session.inputs.audio_s3.prefix`; +- no matching `.flac` objects; +- storage connectivity or permissions failure. + +Diagnostics: + +```bash +narratio run-stage prepare 2026-04-04 --force +``` + +Safe fix: + +- verify prefix contents and storage access; +- keep session audio mode consistent. + +## References -- [docs/config.md](./config.md) - [docs/cli.md](./cli.md) +- [docs/config.md](./config.md) - [docs/operations.md](./operations.md) - [docs/internal/stage-publish.md](./internal/stage-publish.md)