From 71395bb07681ca34425ea382cef8bdd76c3dd76e Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 23 May 2026 04:51:16 +0000 Subject: [PATCH] Rewrite docs for the publish stage contract and current behavior --- README.md | 2 +- docs/architecture.md | 18 +- docs/cli.md | 285 ++++------- docs/config.md | 448 +++--------------- docs/development.md | 2 +- docs/integrations/seriatim.md | 2 +- docs/internal/README.md | 4 +- docs/internal/adapters.md | 2 +- docs/internal/artifacts.md | 8 +- docs/internal/command-restore.md | 10 +- docs/internal/manifest.md | 2 +- docs/internal/stage-analyze.md | 10 +- docs/internal/stage-merge.md | 6 +- docs/internal/stage-prepare.md | 14 +- .../{stage-archive.md => stage-publish.md} | 62 +-- docs/internal/stage-transcribe.md | 8 +- docs/internal/stage-trim.md | 2 +- docs/internal/storage.md | 8 +- docs/internal/workspace.md | 18 +- docs/operations.md | 401 ++++++---------- docs/troubleshooting.md | 356 ++++---------- examples/pipeline.full.annotated.yml | 2 +- 22 files changed, 496 insertions(+), 1174 deletions(-) rename docs/internal/{stage-archive.md => stage-publish.md} (54%) diff --git a/README.md b/README.md index 0cbf0df..7c9a327 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts. -It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, archive publishing, and resumable run state in one operator workflow. +It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, publish-stage uploads, and resumable run state in one operator workflow. ```bash narratio run 2026-04-04 diff --git a/docs/architecture.md b/docs/architecture.md index 3142fa2..923458e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -19,7 +19,7 @@ It coordinates specialized downstream systems rather than reimplementing their d - Audita handles transcript correction and polishing. - Scriptorium handles prompt execution and generated artifacts. -Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and archive semantics. +Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and publish semantics. Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine. @@ -80,7 +80,7 @@ It should record: - input and output refs; - logs and generated config refs; - checksums or provenance where useful; -- non-secret adapter and archive metadata. +- non-secret adapter and publish metadata. Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress. @@ -117,19 +117,19 @@ Narratio should not become a secondary configuration system for downstream tools Local and remote paths are part of Narratio’s application contract. -Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and archive paths. Stages should avoid reconstructing canonical paths through scattered string concatenation. +Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and publish/current paths. Stages should avoid reconstructing canonical paths through scattered string concatenation. Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics. -## Archive Invariants +## Publish Invariants -Archive behavior must preserve a clear commit boundary. +Publish behavior must preserve a clear commit boundary. -A remote run is current only after the archive stage has successfully uploaded the run record, required promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`. +A remote run is current only after the publish stage has successfully uploaded the run record, required published outputs, `current/manifest.json`, and finally `current/run_id.txt`. `current/run_id.txt` is the final remote commit marker and must be written last. -Failed, incomplete, skipped, or uncommitted archive attempts must not be presented as current remote state. Local cleanup is permitted only after successful archive commit and only when explicitly configured. +Failed, incomplete, skipped, or uncommitted publish attempts must not be presented as current remote state. Local cleanup is permitted only after successful publish commit and only when explicitly configured. ## Security and Privacy @@ -139,7 +139,7 @@ Rules: - Do not store raw secrets in pipeline or session YAML. - Use environment variable names or secret-file references for secret handling. -- Do not write raw secret values to manifests, logs, generated configs, or archive metadata. +- Do not write raw secret values to manifests, logs, generated configs, or publish metadata. - Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive. - Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason. @@ -177,7 +177,7 @@ Tests should cover: - stage success, failure, skip, and resume behavior; - adapter command construction; - fake storage behavior; -- archive commit ordering; +- publish commit ordering; - example config validity where practical. Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs. diff --git a/docs/cli.md b/docs/cli.md index 09fc742..0fd907d 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -6,91 +6,64 @@ narratio run 2026-04-04 ``` -This command uses default system discovery for `pipeline.yml`, the pipeline default campaign ID, and local `session.yml`. If local session discovery misses and S3 storage is configured, the positional session ID loads remote `session.yml` from the canonical session prefix. - -Default pipeline and session discovery checks system config locations only. Pass `--config`, `--campaign-file`, and `--session` to use files from the current working directory. Pass `--campaign ` to select a campaign from `pipeline.campaigns.root`. - -Ordinary local and remote `session.yml` files must be concrete YAML. Templates belong to `narratio session init`, which renders a configured campaign template before writing the concrete file. +This runs the full pipeline for the given session ID using default config discovery and campaign selection. ## Command Overview - Top-level commands: -- `run `: execute pipeline stages and persist manifest state. +- `run `: execute the pipeline. +- `resume `: continue from first non-succeeded stage. - `run-stage `: execute exactly one stage. -- `resume `: continue from first non-succeeded stage unless forced. -- `analyze `: force-rerun the analyze stage. -- `publish `: force-rerun the archive stage. +- `analyze `: force-rerun analyze stage. +- `publish `: force-rerun publish stage. - `clean |--all`: remove local workspace/spool state. - `session `: session-scoped helper commands. Session subcommands: -- `session init `: create local or remote `session.yml`. -- `session validate `: run read-only preflight checks. -- `session status `: inspect local/remote session state. -- `session plan `: validate config, prepare workspace layout, and print stage run/skip decisions. -- `session restore `: restore durable local state from committed remote archive state. -- `session artifacts `: list effective artifact source IDs. -- `session locks `: list archive promotion locks. -- `session locks add `: add or update a remote lock. -- `session locks remove `: remove a remote lock. - -Unknown commands print usage and exit non-zero. - -For config semantics, see [docs/config.md](./config.md). For operator lifecycle and recovery, see [docs/operations.md](./operations.md). +- `session init ` +- `session plan ` +- `session validate ` +- `session status ` +- `session restore ` +- `session artifacts ` +- `session locks ` +- `session locks add ` +- `session locks remove ` ## Common Flags - Most session-aware commands accept: -- `--config `: optional explicit `pipeline.yml` path. -- `--campaign `: optional campaign ID selector. -- `--campaign-file `: optional explicit `campaign.yml` path. -- `--session `: optional explicit concrete `session.yml` path. -- `--previous-session-id `: expected previous session identifier. +- `--config ` +- `--campaign ` +- `--campaign-file ` +- `--session ` +- `--previous-session-id ` -The positional `` is required even when `--session` is provided. It is used as the expected session identity and as the remote session lookup value when local session discovery misses. +`--campaign` and `--campaign-file` are mutually exclusive. ## Command Reference ### `run` ```bash -narratio run [--config ] [--campaign ] [--campaign-file ] [--session ] [--previous-session-id ] [--force] [--artifacts ] +narratio run [--force] [--artifacts ] [...common flags] ``` -Purpose: -- Execute configured stages in canonical order. - -Success output: -- `narratio run: session ; executed= skipped=; manifest=` - -Common failure cases: -- missing system default config/session paths when flags are omitted. -- missing selected campaign under `pipeline.campaigns.root`. -- missing local session plus missing/unavailable remote `session.yml`. -- templated `session.yml`; run `narratio session init` to generate concrete YAML. -- concrete session identity mismatch. -- unknown configured artifact key in `--artifacts`. +Runs stages in canonical order and writes manifest state. ### `resume` ```bash -narratio resume [--config ] [--campaign ] [--campaign-file ] [--session ] [--previous-session-id ] [--force] [--artifacts ] +narratio resume [--force] [--artifacts ] [...common flags] ``` -Purpose: -- Continue from session-manifest stage status. - -Success output: -- `narratio resume: session has no remaining stages` -- or `narratio resume: session ; executed= skipped=; manifest=` +Starts at the first non-succeeded stage from the session manifest. ### `run-stage` ```bash -narratio run-stage [--config ] [--campaign ] [--campaign-file ] [--session ] [--previous-session-id ] [--force] [--artifacts ] +narratio run-stage [--force] [--artifacts ] [...common flags] ``` Valid stage names: @@ -102,175 +75,130 @@ Valid stage names: - `normalize` - `trim` - `analyze` -- `archive` +- `publish` - `notify` -Success output: -- `narratio run-stage: stage= executed= skipped= force=; manifest=` - -`--artifacts` is accepted only for `analyze` and `archive`. +`--artifacts` is accepted only for `analyze` and `publish`. ### `analyze` ```bash -narratio analyze [--config ] [--campaign ] [--campaign-file ] [--session ] [--previous-session-id ] [--artifacts ] +narratio analyze [--artifacts ] [...common flags] ``` -Purpose: -- Force-rerun the analyze stage. -- Shorter equivalent for `narratio run-stage analyze --force`. - -`analyze` is force-by-design and does not accept `--force`. +Equivalent to `narratio run-stage analyze --force`. ### `publish` ```bash -narratio publish [--config ] [--campaign ] [--campaign-file ] [--session ] [--previous-session-id ] [--artifacts ] +narratio publish [--artifacts ] [...common flags] ``` -Purpose: -- Force-rerun the archive stage. -- Shorter equivalent for `narratio run-stage archive --force`. - -`publish` is force-by-design and does not accept `--force` or a stage positional argument. +Equivalent to `narratio run-stage publish --force`. ### `clean` ```bash -narratio clean [--config ] [--campaign ] [--campaign-file ] [--session ] [--previous-session-id ] [--dry-run] [--clear-cache] -narratio clean --all [--config ] [--dry-run] [--clear-cache] +narratio clean [--dry-run] [--clear-cache] [...common flags] +narratio clean --all [--dry-run] [--clear-cache] [--config ] ``` -Session cleanup deletes: -- `{workspace.root}/work/{campaign}/{session_id}` -- `{spool.root}/{campaign}/{session_id}` - -All-session cleanup deletes: -- `{workspace.root}/work` -- the contents of `{spool.root}`, while preserving the spool root directory itself. - -Cache behavior: -- cache is preserved by default. -- `--clear-cache` in session mode removes cached S3 audio files for the resolved session. -- `--all --clear-cache` removes the configured Narratio S3 audio cache namespace for the configured bucket/root prefix. -- `--clear-cache` does not delete arbitrary files under `pipeline.cache.root`. +- session mode deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`. +- `--all` deletes all session work and spool children. +- cache is preserved unless `--clear-cache` is passed. ### `session plan` ```bash -narratio session plan [--config ] [--campaign ] [--campaign-file ] [--session ] [--previous-session-id ] [--force] +narratio session plan [--force] [...common flags] ``` -Purpose: -- Validate config, load secrets if configured, prepare workdir, and print stage run/skip decisions. - -Success output includes: -- `narratio session plan: workdir prepared at ` -- one line per stage (`: run|skip`) -- `totals: run= skip=` - -### `session status` - -```bash -narratio session status [--config ] [--campaign ] [--campaign-file ] [--session ] [--previous-session-id ] -``` - -Output includes: -- session ID, campaign, workspace, and session config source. -- local manifest state when present. -- remote current archive state when storage is configured. -- catalog-based promoted output availability for expected transcript and artifact sources. -- effective archive locks and conservative next actions. +Validates config and session inputs, prepares workdir layout, and prints stage run/skip decisions. ### `session validate` ```bash -narratio session validate [--config ] [--campaign ] [--campaign-file ] [--session ] [--previous-session-id ] +narratio session validate [...common flags] ``` -Checks include: -- effective config and session source. -- stable input files. -- local or remote audio availability. -- previous-session requirements. -- archive promotions and effective locks. +Read-only preflight checks for config, inputs, audio availability, previous-session requirements, publish outputs, and effective locks. -Warnings do not fail the command. Any `ERROR` finding exits non-zero. +### `session status` + +```bash +narratio session status [...common flags] +``` + +Shows local manifest state, remote current state (when storage is configured), published-output availability, and effective locks. ### `session init` ```bash narratio session init --output ./session.yml narratio session init --remote -narratio session init --config --campaign icewind --remote -narratio session init --config --campaign-file ./campaign.yml --remote +narratio session init --remote --force ``` -Additional flags: +Flags: -- `--previous-session-id ` -- `--date ` -- `--title ` -- `--audio-s3-prefix `: defaults to `audio/` when neither audio flag is provided. -- `--audio-dir `: local audio directory; mutually exclusive with `--audio-s3-prefix`. -- `--force`: overwrite existing local or remote target. - -Behavior: -- exactly one of `--output` or `--remote` is required. -- `--config`, `--campaign`, and `--campaign-file` are optional overrides; omitted campaign selection uses `pipeline.campaigns.default_campaign_id`. -- `--campaign ` selects a campaign under `pipeline.campaigns.root`. -- `--campaign-file ` loads an explicit campaign file. -- if `campaign.yml` sets `session_template_file`, the template path is resolved relative to `campaign.yml` and rendered from init flags. -- if no session template is configured, a minimal concrete session file is generated directly. -- template variables must be supplied by matching flags, and supplied template-related flags must be used by the template. -- remote writes target `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`. -- existing local or remote targets fail unless `--force` is passed. -- remote writes use existence checks, not compare-and-swap. +- `--output ` or `--remote` (exactly one is required) +- `--previous-session-id ` +- `--date ` +- `--title ` +- `--audio-dir <path>` +- `--audio-s3-prefix <prefix>` +- `--force` +- common config/campaign flags ### `session restore` ```bash -narratio session restore <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--dry-run] [--force] [--include-audio] +narratio session restore <session_id> [--dry-run] [--force] [--include-audio] [...common flags] ``` -Purpose: -- Restore durable session state from the committed remote archive current state. -- Default restore installs `manifest.json`, `transcripts/**`, and `artifacts/**` from the current session archive. -- When configured previous-session inputs require it, restore reconstructs `previous/**` from the previous session's committed current archive. -- `audio/**` is restored only with `--include-audio`. +Restores durable local session files from committed remote current state. -Dry-run output may include planned previous-cache downloads. Existing differing files under `previous/**` follow the normal restore conflict policy and require `--force` to overwrite. +Default restore scope: -When `--include-audio` is set, S3 audio files are restored through the shared audio cache. Cache hits avoid re-downloading large audio objects. +- `manifest.json` +- `transcripts/**` +- `artifacts/**` +- `previous/**` when required by configured previous-session artifact inputs + +`audio/**` is restored only when `--include-audio` is set. ### `session artifacts` ```bash -narratio session artifacts <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--remote] +narratio session artifacts <session_id> [--remote] [...common flags] ``` -Purpose: -- List built-in, configured, previous-session, promoted, and locked artifact sources. - -`--remote` checks promoted top-level object availability through the storage adapter. Remote markers appear only in the `Promoted` section, which reports each configured archive promotion destination and includes `dest=<path>` when that destination differs from the source's canonical path. +Lists built-in sources, configured artifact sources, previous-session sources, publish output rules, and lock status. With `--remote`, includes remote published-state markers. ### `session locks` ```bash -narratio session locks <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] -narratio session locks add <session_id> <source> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--reason <text>] [--force] -narratio session locks remove <session_id> <source> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] +narratio session locks <session_id> [...common flags] +narratio session locks add <session_id> <source> [--reason <text>] [--force] [...common flags] +narratio session locks remove <session_id> <source> [...common flags] ``` -Behavior: -- list mode prints effective locks from static `pipeline.archive.locks` and remote `{session_prefix}/locks.yml`. -- `locks add` writes only the remote lock store and fails if the source is already locked by pipeline config. -- `locks remove` removes only remote locks and cannot remove static pipeline locks. -- `locks add --force` is required to update an existing remote lock reason. +- list mode prints effective locks from static `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`. +- add/remove mutate only the remote lock store. +- static pipeline locks cannot be removed by lock commands. + +## `--artifacts` Rules + +- accepted on `run`, `resume`, `run-stage`, `analyze`, and `publish`. +- on `run-stage`, only valid for `analyze` and `publish`. +- filters configured analyze artifact execution. +- filters configured `pipeline.publish.outputs` entries for `narratio.artifact.<key>` sources. +- does not suppress built-in transcript/bounds publish outputs. +- does not imply `--force` for `run`, `resume`, or `run-stage`. ## Common Workflows -Default-discovery run: +Run full pipeline: ```bash narratio run 2026-04-04 @@ -282,60 +210,21 @@ Run only selected analyze artifacts: narratio run 2026-04-04 --artifacts session_recap,player_handout ``` -Resume with selected analyze artifacts: - -```bash -narratio resume 2026-04-04 --artifacts player_handout -``` - -Force-rerun analyze with selected artifacts: +Force analyze only: ```bash narratio analyze 2026-04-04 --artifacts player_handout ``` -Force-rerun archive publishing: +Force publish only: ```bash narratio publish 2026-04-04 ``` -Preview restore actions without writes: +Restore preview then apply: ```bash narratio session restore 2026-04-04 --dry-run -``` - -Restore and then force analyze: - -```bash narratio session restore 2026-04-04 -narratio analyze 2026-04-04 ``` - -Rehydrate canonical previous-session inputs after artifact-input changes: - -```bash -narratio run-stage prepare 2026-04-04 --force -``` - -Reset local state before testing restore: - -```bash -narratio clean 2026-04-04 --dry-run -narratio clean 2026-04-04 -narratio session restore 2026-04-04 --include-audio -``` - -Clean all local sessions while keeping cached S3 audio: - -```bash -narratio clean --all -``` - -## `--artifacts` and `--force` - -- `--artifacts` filters which configured artifacts are executable when analyze runs and which configured artifact promotions archive publishes. -- `--artifacts` does not imply `--force`. -- if analyze is already `succeeded` and `--force` is not set, runner-level skip still applies. -- `--artifacts` does not suppress built-in transcript or bounds promotions. diff --git a/docs/config.md b/docs/config.md index f0ad026..4707a72 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1,14 +1,13 @@ # Configuration -## 1. Overview - +## Overview Narratio loads three YAML files: -- `pipeline.yml`: pipeline-level runtime configuration. +- `pipeline.yml`: pipeline-level runtime settings. - `campaign.yml`: stable campaign identity and campaign-level input defaults. -- `session.yml`: per-session metadata and input selection, loaded locally or from the configured S3 backend. +- `session.yml`: per-session metadata and input selection. -These commands load and validate all three files before running: +Commands that load and validate all three files include: - `narratio run` - `narratio resume` @@ -23,94 +22,36 @@ These commands load and validate all three files before running: - `narratio session locks` - `narratio clean <session_id>` -Behavior: +Validation behavior: -- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail. -- ordinary local and remote `session.yml` files must be concrete YAML; template placeholders are rejected. +- strict YAML decode is enabled (`KnownFields(true)`); unknown fields fail. +- loaded `session.yml` files must be concrete YAML (no `{{ ... }}` placeholders). - defaults are applied for optional pipeline fields. -- campaign identity is selected by ID from the pipeline campaign registry unless `--campaign-file` is used. -- campaign-level stable input paths fill missing session input paths. -- session-level stable input paths override campaign-level input paths. -- campaign config may point `session init` to a session template. -- validation enforces required fields, value formats, and cross-field constraints. +- campaign/session identity mismatches fail load. -## 2. Config file discovery +## File Discovery +Pipeline discovery order when `--config` is omitted: -These commands use the same config discovery behavior: +1. `/usr/local/etc/narratio/pipeline.yml` +2. `/etc/narratio/pipeline.yml` -- `narratio run` -- `narratio resume` -- `narratio run-stage` -- `narratio analyze` -- `narratio publish` -- `narratio session plan` -- `narratio session status` -- `narratio session validate` -- `narratio session restore` -- `narratio session artifacts` -- `narratio session locks` -- `narratio clean <session_id>` +Session discovery order when `--session` is omitted: -Pipeline config lookup: +1. `/usr/local/etc/narratio/session.yml` +2. `/etc/narratio/session.yml` -- if `--config <path>` is provided, that path is used. -- if omitted, Narratio searches in order: - 1. `/usr/local/etc/narratio/pipeline.yml` - 2. `/etc/narratio/pipeline.yml` -- first existing file wins. +Campaign discovery when `--campaign-file` is omitted: -Campaign config lookup: +- if `--campaign <id>` is set: `{pipeline.campaigns.root}/{id}/campaign.yml` +- otherwise: `{pipeline.campaigns.root}/{pipeline.campaigns.default_campaign_id}/campaign.yml` -- pipeline config is loaded first. -- if `--campaign-file <path>` is provided, that path is used. -- otherwise, if `--campaign <id>` is provided, Narratio loads: - - `{pipeline.campaigns.root}/{id}/campaign.yml` -- otherwise, Narratio uses `pipeline.campaigns.default_campaign_id` and loads: - - `{pipeline.campaigns.root}/{default_campaign_id}/campaign.yml` -- `--campaign` and `--campaign-file` are mutually exclusive. -- campaign IDs must be single path segments, not paths. +Remote `session.yml` fallback: -Session config lookup: - -- if `--session <path>` is provided, that path is used. -- if `--session` is omitted, Narratio searches locally in order: - 1. `/usr/local/etc/narratio/session.yml` - 2. `/etc/narratio/session.yml` -- first existing local file wins. -- if no local session file is found, a positional `<session_id>` is present, storage is configured, and campaign identity is resolved, Narratio loads remote `session.yml` from: +- if local session discovery fails and storage is configured, Narratio can load: - `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml` -- local discovery always runs before remote fallback. -- local files in the current working directory are used only when passed explicitly, for example `--config ./pipeline.yml --campaign-file ./campaign.yml --session ./session.yml`. -## 3. Session templating - -Template behavior for local and remote `session.yml` loaded by downstream commands: - -- downstream commands do not render templates. -- local and remote `session.yml` must be concrete. -- any `{{ ... }}` placeholder in loaded `session.yml` fails with guidance to run `narratio session init`. -- if concrete `session_id` mismatches the positional `<session_id>`, load fails. -- if concrete `previous_session_id` mismatches `--previous-session-id`, load fails. - -Template behavior for `narratio session init`: - -- `campaign.yml` may set `session_template_file`. -- relative template paths resolve relative to `campaign.yml`. -- supported init template variables: - - `{{ session_id }}` - - `{{ previous_session_id }}` - - `{{ date }}` - - `{{ title }}` - - `{{ audio_s3_prefix }}` - - `{{ audio_dir }}` -- each template variable must be supplied by the matching `session init` flag. -- template-related flags such as `--date`, `--title`, `--audio-s3-prefix`, `--audio-dir`, and `--previous-session-id` fail if the configured template does not use them. -- rendered output is strict-decoded and validated before it is written locally or remotely. -- if `session_template_file` is omitted, `session init` generates the minimal concrete session YAML directly. - -## 4. Minimal config set - -### `pipeline.yml` +## Minimal Working Config +`pipeline.yml` ```yaml campaigns: @@ -120,30 +61,17 @@ whisperx: transcribe_url: "https://transcription.example.com/transcribe" ``` -Why this is sufficient: - -- `whisperx.transcribe_url` is required. -- `campaigns.default_campaign_id` selects the default campaign when `--campaign` is omitted. -- `workspace.root` defaults to `/var/lib/narratio`. -- optional sections (`seriatim`, `audita`, `archive`, `scriptorium`, `trim`, `normalize`, etc.) receive defaults or stay inactive. - -### `campaign.yml` +`campaign.yml` ```yaml campaign_id: sample-campaign -session_template_file: ./session.template.yml inputs: speakers_file: ./speakers.yml autocorrect_file: ./autocorrect.yml glossary_file: ./glossary.yml ``` -Why this is sufficient: - -- `campaign_id` supplies the stable campaign identity. -- stable input files are required and resolve relative to `campaign.yml` when copied during `prepare`. - -### `session.yml` +`session.yml` ```yaml session_id: 2026-05-03 @@ -151,67 +79,14 @@ inputs: audio_dir: ./audio ``` -Why this is sufficient: - -- `session_id` is required. -- `campaign` can be omitted because it is supplied by `campaign.yml`. -- stable input paths can be omitted because `campaign.yml` supplies defaults. -- local `audio_dir` resolves relative to `session.yml`. - -Minimal local-file usage: - -```bash -narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign sample-campaign --session ./session.yml -narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign-file ./campaign.yml --session ./session.yml -``` - -Previous-session-enabled variant: +## Publish Config +Top-level publish settings live at `pipeline.publish`. ```yaml -session_id: 2026-05-03 -previous_session_id: 2026-04-26 -inputs: - audio_dir: ./audio -``` - -```bash -narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign sample-campaign --session ./session.yml --previous-session-id 2026-04-26 -``` - -## 5. Production-oriented config set - -### `pipeline.yml` - -```yaml -workspace: - root: /var/lib/narratio/workspace - cleanup_after_archive: true - -storage: - backend: s3 - s3: - bucket: my-dnd-archive - root_prefix: dnd - region: us-east-1 - access_key_id_env: OBJECT_STORAGE_KEY_ID - secret_access_key_env: OBJECT_STORAGE_KEY - -campaigns: - root: /srv/narratio/campaigns - default_campaign_id: forsaken - -spool: - root: /var/spool/narratio - delete_audio_after_archive: true - -cache: - root: /var/cache/narratio - s3_audio: true - -archive: +publish: enabled: true upload_run: true - promote_artifacts: + outputs: - source: narratio.transcript.final_trimmed dest: transcripts/final.trimmed.json required: true @@ -221,80 +96,31 @@ archive: locks: - source: narratio.artifact.session_recap reason: Final recap was manually edited. - -whisperx: - transcribe_url: "https://transcription.example.com/transcribe" - -scriptorium: - artifacts: - session_recap: - enabled: true - prompt_id: dnd.session_recap - output_path: artifacts/session_recap.md - inputs: - transcript: - source: narratio.transcript.final_trimmed - required: true - previous_recap: - source: narratio.previous_session.artifact.session_recap - required: false ``` -### `campaign.yml` +Rules: -```yaml -campaign_id: forsaken -inputs: - speakers_file: /srv/narratio/campaigns/forsaken/speakers.yml - autocorrect_file: /srv/narratio/campaigns/forsaken/autocorrect.yml - glossary_file: /srv/narratio/campaigns/forsaken/glossary.yml -``` +- `outputs[].source` is required. +- `outputs[].dest` is optional; when omitted, Narratio derives destination from the source. +- `outputs[].required` defaults to `true`. +- static `publish.locks` and remote `{session_prefix}/locks.yml` are merged; static locks win on duplicates. +- locks prevent overwrite of top-level published destinations. -### Local `session.yml` +Supported publish source families: -```yaml -session_id: 2026-05-03 -previous_session_id: 2026-04-26 -date: 2026-05-03 -title: The Black Cabin -inputs: - audio_s3: - prefix: audio/ -``` +- built-ins: `narratio.transcript.base`, `narratio.transcript.polished`, `narratio.transcript.final`, `narratio.transcript.final_trimmed`, `narratio.bounds.session` +- configured artifacts: `narratio.artifact.<artifact_key>` -### S3-first session config - -For S3-first operation, upload the same `session.yml` content to: - -```text -{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml -``` - -Then run with explicit or discovered pipeline/campaign config and no `--session`: - -```bash -narratio run 2026-05-03 --config /usr/local/etc/narratio/pipeline.yml --campaign forsaken --previous-session-id 2026-04-26 -``` - -Operational notes: - -- archive promotion is explicit and source-based via `archive.promote_artifacts`. -- `source` is required; `dest` is optional and derived when omitted. -- `archive.locks` skips top-level promotion overwrites for static locked sources while preserving run-local uploads. -- operator-created mutable locks are stored at `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/locks.yml` and are merged with static locks. -- Narratio does not auto-promote all generated analyze artifacts. -- `restore` reads the same config/campaign/session inputs and restore scope is bounded by committed archive current state. -- `clean` removes workspace/spool state by default and preserves `pipeline.cache.root` unless `--clear-cache` is passed. - -## 6. Full pipeline reference +## Full Reference +### Pipeline | Path | Type | Required | Default | | --- | --- | --- | --- | | `pipeline.workspace.root` | string | No | `/var/lib/narratio` | -| `pipeline.workspace.cleanup_after_archive` | bool | No | `false` | +| `pipeline.workspace.cleanup_after_publish` | bool | No | `false` | | `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` | | `pipeline.campaigns.default_campaign_id` | string | No | empty | -| `pipeline.secrets.env_dir` | string | Conditional | none | +| `pipeline.secrets.env_dir` | string | No | empty | | `pipeline.storage.backend` | string | No | empty | | `pipeline.storage.s3.bucket` | string | Conditional | empty | | `pipeline.storage.s3.root_prefix` | string | No | `dnd` | @@ -304,18 +130,18 @@ Operational notes: | `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` | | `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` | | `pipeline.spool.root` | string | No | `/var/spool/narratio` | -| `pipeline.spool.delete_audio_after_archive` | bool | No | `false` | +| `pipeline.spool.delete_audio_after_publish` | bool | No | `false` | | `pipeline.cache.root` | string | No | `/var/cache/narratio` | | `pipeline.cache.s3_audio` | bool | No | `true` | -| `pipeline.archive.enabled` | bool | No | `true` | -| `pipeline.archive.upload_run` | bool | No | `true` | -| `pipeline.archive.promote_artifacts[]` | list | No | final-trimmed transcript rule | -| `pipeline.archive.promote_artifacts[].source` | string | Yes (per rule) | none | -| `pipeline.archive.promote_artifacts[].dest` | string | No | derived from source | -| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` | -| `pipeline.archive.locks[]` | list | No | empty | -| `pipeline.archive.locks[].source` | string | Yes (per lock) | none | -| `pipeline.archive.locks[].reason` | string | No | empty | +| `pipeline.publish.enabled` | bool | No | `true` | +| `pipeline.publish.upload_run` | bool | No | `true` | +| `pipeline.publish.outputs[]` | list | No | one final-trimmed output rule | +| `pipeline.publish.outputs[].source` | string | Yes (per rule) | none | +| `pipeline.publish.outputs[].dest` | string | No | derived from source | +| `pipeline.publish.outputs[].required` | bool | No | `true` | +| `pipeline.publish.locks[]` | list | No | empty | +| `pipeline.publish.locks[].source` | string | Yes (per lock) | none | +| `pipeline.publish.locks[].reason` | string | No | empty | | `pipeline.whisperx.transcribe_url` | string | Yes | none | | `pipeline.whisperx.language` | string | No | `en` | | `pipeline.whisperx.timeout` | duration string | No | `30m` | @@ -364,162 +190,38 @@ Operational notes: | `pipeline.scriptorium.timeout` | duration string | No | `10m` | | `pipeline.scriptorium.render_debug` | bool | No | `false` | | `pipeline.scriptorium.artifacts` | map | No | empty | -| `pipeline.scriptorium.artifacts.<name>.enabled` | bool | No | `false` | -| `pipeline.scriptorium.artifacts.<name>.depends_on[]` | list[string] | No | empty | -| `pipeline.scriptorium.artifacts.<name>.render_debug` | bool | No | unset | -| `pipeline.scriptorium.artifacts.<name>.prompt_id` | string | Conditional | none | -| `pipeline.scriptorium.artifacts.<name>.profile_id` | string | No | empty | -| `pipeline.scriptorium.artifacts.<name>.output_path` | string | Conditional | none | -| `pipeline.scriptorium.artifacts.<name>.timeout` | duration string | No | empty | -| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` | string | Conditional | none | -| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.artifact` | string | No | empty | -| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.path` | string | No | empty | -| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required` | bool | No | `false` | -| `pipeline.scriptorium.artifacts.<name>.vars.<key>` | map value | No | empty | | `pipeline.notification.backend` | string | No | empty | | `pipeline.notification.recipient` | string | No | empty | -| `pipeline.notification.timeout` | duration string | No | empty | +| `pipeline.notification.timeout` | duration string | No | `30s` | -Scriptorium artifact-key and dependency rules: +### Campaign +| Path | Type | Required | +| --- | --- | --- | +| `campaign_id` | string | Yes | +| `session_template_file` | string | No | +| `inputs.speakers_file` | string | Yes | +| `inputs.autocorrect_file` | string | Yes | +| `inputs.glossary_file` | string | Yes | -- artifact keys must match `^[a-z][a-z0-9_]*$`. -- enabled artifacts require `prompt_id` and `output_path`. -- `output_path` must be relative, traversal-safe, and under `artifacts/`. -- configured artifact input sources use `narratio.artifact.<name>`. -- if input source references `narratio.artifact.<name>`, artifact `<name>` must exist and must be listed in `depends_on`. -- every `depends_on` entry must be a configured artifact key. -- self-dependency is rejected. -- enabled dependency cycles are rejected. -- any artifact referenced by `depends_on` or `narratio.artifact.<name>` source must define `output_path` (even if not enabled). - -Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values: - -- `narratio.previous_session.artifact.<configured_artifact_key>` -- `narratio.transcript.base` -- `narratio.transcript.polished` -- `narratio.transcript.final` -- `narratio.transcript.final_trimmed` -- `narratio.bounds.session` -- `narratio.artifact.<configured_artifact_key>` - -`pipeline.archive.promote_artifacts[].source` values: - -- `narratio.transcript.base` -- `narratio.transcript.polished` -- `narratio.transcript.final` -- `narratio.transcript.final_trimmed` -- `narratio.bounds.session` -- `narratio.artifact.<configured_artifact_key>` - -`pipeline.archive.locks[].source` accepts the same source values as `pipeline.archive.promote_artifacts[].source`. - -Archive promotion destination rules: - -- `dest` must be a clean relative path (not absolute, no traversal). -- duplicate `dest` values are rejected. -- if `dest` is omitted: - - built-in sources derive their canonical destination path; - - configured sources derive from `pipeline.scriptorium.artifacts.<name>.output_path`; - - derivation failure is a config validation error. - -Archive lock rules: - -- locks are source-based and do not accept `dest`. -- duplicate lock sources are rejected. -- static `pipeline.archive.locks` win over remote mutable locks for the same source. -- locked promotions are recorded as intentional skips in archive metadata. -- locked required promotions do not fail archive by default. -- ordinary `--force` reruns do not override locks. - -Remote mutable lock store: - -- path: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/locks.yml`. -- strict YAML shape: top-level `locks`, each with `source` and optional `reason`. -- `narratio session locks add` and `narratio session locks remove` mutate only the remote lock store. -- writes use existence checks plus `--force` for updates; they are not compare-and-swap atomic. - -Restore-related implications: - -- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs). -- restore scope considers committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, optional `audio/**`). -- S3 audio downloads use `pipeline.spool.root` for active downloads and `pipeline.cache.root` for reusable cached audio when `pipeline.cache.s3_audio` is true. -- `pipeline.cache.root` is durable local cache state. It is not workspace state and is preserved by default by `narratio clean`. - -## 7. Full campaign reference - -| Path | Type | Required | Default | -| --- | --- | --- | --- | -| `campaign.campaign_id` | string | Yes | none | -| `campaign.session_template_file` | string | No | none | -| `campaign.inputs.speakers_file` | string | Yes | none | -| `campaign.inputs.autocorrect_file` | string | Yes | none | -| `campaign.inputs.glossary_file` | string | Yes | none | - -Campaign input paths and `campaign.session_template_file` may be absolute or relative. Relative paths resolve from the directory containing `campaign.yml`. - -## 8. Full session reference - -| Path | Type | Required | Default | -| --- | --- | --- | --- | -| `session.session_id` | string | Yes | none | -| `session.previous_session_id` | string | No | empty | -| `session.campaign` | string | No | `campaign.campaign_id` | -| `session.date` | string | No | empty | -| `session.title` | string | No | empty | -| `session.inputs.audio_dir` | string | Conditional | empty | -| `session.inputs.audio_files[]` | list[string] | Conditional | empty | -| `session.inputs.audio_s3.prefix` | string | Conditional | none | -| `session.inputs.speakers_file` | string | No | `campaign.inputs.speakers_file` | -| `session.inputs.autocorrect_file` | string | No | `campaign.inputs.autocorrect_file` | -| `session.inputs.glossary_file` | string | No | `campaign.inputs.glossary_file` | - -Session input paths may be absolute or relative. Relative audio paths and session-level stable input overrides resolve from the directory containing `session.yml`. If both `campaign.yml` and `session.yml` specify campaign identity, the values must match. - -Audio-source rule: - -- configure exactly one mode: - - `audio_dir`, or - - `audio_files` (at least one), or - - `audio_s3.prefix` -- `audio_s3` cannot be combined with local audio fields. - -Previous-session rule: - -- if `session.previous_session_id` is set, it must not equal `session.session_id`. -- canonical previous-session sources (`narratio.previous_session.artifact.<name>`) are hydrated during `prepare` from archive current state when required by enabled configured artifacts. - -## 9. Secrets - -Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`. - -Behavior: - -- `env_dir` may be absolute or relative. -- relative `env_dir` resolves from current working directory. -- files with valid env-var names (`[A-Za-z_][A-Za-z0-9_]*`) are loaded. -- values are loaded from file contents with trailing newline trimming. -- existing process env vars are preserved. -- invalid names and subdirectories are skipped. -- missing/unreadable `env_dir` fails command execution. - -Guidance: - -- do not put secret values directly in YAML. -- configure env var names in config and provide values via env/secrets files. - -## 10. Examples - -Maintained examples: +### Session +| Path | Type | Required | +| --- | --- | --- | +| `session_id` | string | Yes | +| `previous_session_id` | string | No | +| `campaign` | string | No | +| `date` | string | No | +| `title` | string | No | +| `inputs.speakers_file` | string | No | +| `inputs.autocorrect_file` | string | No | +| `inputs.glossary_file` | string | No | +| `inputs.audio_dir` | string | Conditional | +| `inputs.audio_files[]` | list[string] | Conditional | +| `inputs.audio_s3.prefix` | string | Conditional | +## Maintained Examples - `examples/pipeline.minimal.yml` - `examples/pipeline.production.yml` - `examples/pipeline.full.annotated.yml` - `examples/campaigns/sample-campaign/campaign.yml` -- `examples/campaigns/sample-campaign/speakers.yml` -- `examples/campaigns/sample-campaign/autocorrect.yml` -- `examples/campaigns/sample-campaign/glossary.yml` -- `examples/campaigns/sample-campaign/session.template.yml` - `examples/session.local-audio.yml` - `examples/session.s3-audio.yml` - -These examples are validated by `internal/config` tests. diff --git a/docs/development.md b/docs/development.md index c06d8d0..e1c4991 100644 --- a/docs/development.md +++ b/docs/development.md @@ -75,7 +75,7 @@ Remote-storage commands must obtain object storage through the app-level command 1. Implement stage behavior in `internal/stage` with clear input/output boundaries. 2. Keep external transport/subprocess details in `internal/adapters`. -3. Preserve manifest and promotion semantics expected by runner and archive logic. +3. Preserve manifest and publish-output semantics expected by runner and publish logic. 4. Add/update stage and adapter tests. 5. Update internal component contracts in `docs/internal/`. diff --git a/docs/integrations/seriatim.md b/docs/integrations/seriatim.md index 16faddf..279b98a 100644 --- a/docs/integrations/seriatim.md +++ b/docs/integrations/seriatim.md @@ -20,7 +20,7 @@ Owns: - JSON output validation Does not own: -- Transcript input selection/promotion logic (stage-owned) +- Transcript input selection/materialization logic (stage-owned) - Bounds computation (scriptorium/trim-stage-owned) ## Config Fields Used diff --git a/docs/internal/README.md b/docs/internal/README.md index 965e77d..28bed9c 100644 --- a/docs/internal/README.md +++ b/docs/internal/README.md @@ -11,7 +11,7 @@ Implementation-accurate contracts for workspace/state, manifests, stages, artifa - `storage.md`: remote storage backend contracts and object-store invariants. - `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics. - `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior. -- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants. +- `workspace.md`: local state model, manifests, run-local layout, materialization, and cleanup invariants. - `command-restore.md`: restore command discovery/planning/execution/reporting contract. - `stage-prepare.md`: input materialization and provenance capture. - `stage-transcribe.md`: WhisperX transcript generation. @@ -20,7 +20,7 @@ Implementation-accurate contracts for workspace/state, manifests, stages, artifa - `stage-normalize.md`: post-polish normalization. - `stage-trim.md`: bounds-driven transcript trimming. - `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts. -- `stage-archive.md`: archive upload and current-pointer publish contract. +- `stage-publish.md`: publish upload and current-pointer commit contract. ## External Integration Notes - `../integrations/README.md`: canonical location for external integration contracts (`audita.md`, `seriatim.md`, `scriptorium.md`). diff --git a/docs/internal/adapters.md b/docs/internal/adapters.md index 4b85092..8355093 100644 --- a/docs/internal/adapters.md +++ b/docs/internal/adapters.md @@ -28,7 +28,7 @@ Default wiring and adapter calls consume: - `pipeline.seriatim.*` - `pipeline.audita.*` - `pipeline.scriptorium.*` -- `pipeline.storage.*` and `pipeline.archive.*` (object-store construction/gating) +- `pipeline.storage.*` and `pipeline.publish.*` (object-store construction/gating) - `pipeline.notification.*` (sender boundary exists; placeholder behavior today) ## External adapters used diff --git a/docs/internal/artifacts.md b/docs/internal/artifacts.md index 4dcc78d..d13dd75 100644 --- a/docs/internal/artifacts.md +++ b/docs/internal/artifacts.md @@ -15,8 +15,8 @@ Inputs: Outputs: - resolved artifact path + provenance (`ResolvedSessionArtifact`); - runtime catalog entries for built-ins and configured artifacts; -- requirement sets for canonical previous-session inputs. -- canonical S3 session, run, current, session config, session locks, audio, and promoted artifact keys. +- requirement sets for canonical previous-session inputs; +- canonical S3 session, run, current, session config, session locks, audio, and published output keys. ## Boundaries Owns: @@ -28,7 +28,7 @@ Owns: Does not own: - prepare-stage remote hydration; - stage success/skip transitions; -- archive upload orchestration. +- publish upload orchestration. ## Built-in IDs | Artifact ID | Canonical file | Producer stage | Output kind | @@ -71,7 +71,7 @@ Previous-session canonical provenance values include: - Built-ins resolve via manifest producer outputs first, then canonical fallback paths. - Configured `narratio.artifact.<name>` sources resolve through catalog availability. - Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths. -- Archive-relative configured artifact paths under `artifacts/` are cached without a redundant nested `artifacts/` segment. +- Publish-relative configured artifact paths under `artifacts/` are cached without a redundant nested `artifacts/` segment. - Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`. ## Previous-session requirement scanning diff --git a/docs/internal/command-restore.md b/docs/internal/command-restore.md index 2799da5..511a25a 100644 --- a/docs/internal/command-restore.md +++ b/docs/internal/command-restore.md @@ -27,14 +27,14 @@ Owns: Does not own: - Stage execution orchestration (`run`, `resume`, `run-stage`). -- Archive publish behavior (owned by archive stage). +- Publish-stage behavior. - Storage transport implementation details (owned by storage adapters). ## Config fields used - Config/session discovery and templating fields consumed by all commands. - `pipeline.workspace.root` (local restore target root). -- `pipeline.storage.*` (remote backend + archive identity derivation). -- `pipeline.storage.s3.*` identity components used by archive prefix helpers. +- `pipeline.storage.*` (remote backend + publish identity derivation). +- `pipeline.storage.s3.*` identity components used by session-prefix helpers. - `pipeline.spool.root` for active audio downloads. - `pipeline.cache.root` and `pipeline.cache.s3_audio` for reusable S3 audio cache. - `session.session_id` @@ -80,7 +80,7 @@ Restore path scope: - Dry-run is read-only and returns plan output only. ## Failure behavior -- Fails when storage backend is unavailable or archive identity cannot be resolved. +- Fails when storage backend is unavailable or publish identity cannot be resolved. - Fails when remote current pointer/manifest is missing or invalid. - Fails when remote manifest identity mismatches requested campaign/session. - Fails on local conflicts unless `--force` is set. @@ -96,7 +96,7 @@ Restore path scope: - `internal/artifacts/archive_identity_test.go` ## Architectural invariants -- Restore relies on centralized archive identity/key helpers (`internal/artifacts`) rather than ad hoc key building. +- Restore relies on centralized path/key helpers (`internal/artifacts`) rather than ad hoc key building. - `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files. - Local path mapping is traversal-safe and constrained to session root. - Restore scope is deterministic and path-classified: diff --git a/docs/internal/manifest.md b/docs/internal/manifest.md index 341bb2e..a8c3a69 100644 --- a/docs/internal/manifest.md +++ b/docs/internal/manifest.md @@ -30,7 +30,7 @@ Manifest identity fields are populated by app/stage orchestration from: - `session.session_id` - `session.campaign` - `pipeline.workspace.root` -- `pipeline.storage.s3.*` (when archive/S3 identity is set) +- `pipeline.storage.s3.*` (when publish/S3 identity is set) ## External adapters used - No external service adapters. diff --git a/docs/internal/stage-analyze.md b/docs/internal/stage-analyze.md index e4b15aa..a78b863 100644 --- a/docs/internal/stage-analyze.md +++ b/docs/internal/stage-analyze.md @@ -1,7 +1,7 @@ # Stage: analyze ## Purpose -Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths. +Execute selected configured Scriptorium artifacts in deterministic dependency order and materialize successful outputs to canonical session artifact paths. ## Inputs and outputs Inputs: @@ -15,7 +15,7 @@ Source types used by analyze: - canonical previous-session artifacts: `narratio.previous_session.artifact.<artifact_key>`. Outputs: -- promoted configured artifact files at each configured `output_path`; +- materialized configured artifact files at each configured `output_path`; - stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info). ## Boundaries @@ -24,12 +24,12 @@ Owns: - selected-artifact planning and dependency ordering; - per-input resolution and required/optional handling; - Scriptorium render/run invocation; -- run-local output generation and canonical promotion. +- run-local output generation and canonical materialization. Does not own: - prepare-time previous-session hydration; - object-store access for previous-session sources; -- archive promotion policy. +- publish output rule behavior. ## Config fields used - `session.session_id` @@ -76,5 +76,5 @@ Does not own: ## Architectural invariants - Canonical previous-session behavior is local-cache only during analyze. -- Generated outputs are validated and promoted before stage success is recorded. +- Generated outputs are validated and materialized before stage success is recorded. - Resolver/catalog decisions stay deterministic and validation-gated. diff --git a/docs/internal/stage-merge.md b/docs/internal/stage-merge.md index 1cac61b..21f92b6 100644 --- a/docs/internal/stage-merge.md +++ b/docs/internal/stage-merge.md @@ -19,7 +19,7 @@ Owns: - Per-input normalize calls to Seriatim - Final merge call to Seriatim - Run-local log/config/report path wiring -- Promotion of base/report outputs to canonical paths +- Materialization of base/report outputs to canonical paths Does not own: - Transcript polishing or downstream artifact generation @@ -43,7 +43,7 @@ Does not own: ## State and Manifest Behavior - Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory. - Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled. -- Promotes canonical base transcript and optional report. +- Materializes canonical base transcript and optional report. - Records normalized-input provenance and adapter metadata in stage metadata. ## Skip and Resume Behavior @@ -59,5 +59,5 @@ Does not own: ## Architectural Invariants - Merge consumes normalized forms of each raw transcript. -- Base transcript must validate before promotion. +- Base transcript must validate before materialization. - Report output is optional and gated by config. diff --git a/docs/internal/stage-prepare.md b/docs/internal/stage-prepare.md index 4ba1b1e..31b8eae 100644 --- a/docs/internal/stage-prepare.md +++ b/docs/internal/stage-prepare.md @@ -17,7 +17,7 @@ Inputs: - local: `session.inputs.audio_dir` or `session.inputs.audio_files`; - S3: `session.inputs.audio_s3.prefix`; - configured enabled Scriptorium artifact inputs (for previous-session requirement scanning); -- remote previous-session current archive state when previous hydration is required. +- remote previous-session current publish state when previous hydration is required. Outputs: - `inputs/campaign.yml`; @@ -41,7 +41,7 @@ Owns: Does not own: - transcript or artifact generation; - analyze-stage source resolution; -- archive commit behavior. +- publish commit behavior. ## Config fields used - `session.session_id` @@ -83,10 +83,10 @@ Does not own: - `narratio.previous_session.artifact.<artifact_key>` - If one or more canonical previous-session requirements exist: - clears managed `previous/` state; - - hydrates required/optional previous artifacts from the configured previous session’s committed archive current state; + - hydrates required/optional previous artifacts from the configured previous session’s committed publish current state; - writes `previous/manifest.json` and hydrated `previous/artifacts/**`; - - stores archive-relative artifact paths such as `artifacts/session_recap.md` as `previous/artifacts/session_recap.md`, not `previous/artifacts/artifacts/session_recap.md`; - - records hydrated previous inputs in `manifest.Inputs` with source `previous_session_archive.current`. + - stores publish-relative artifact paths such as `artifacts/session_recap.md` as `previous/artifacts/session_recap.md`, not `previous/artifacts/artifacts/session_recap.md`; + - records hydrated previous inputs in `manifest.Inputs` with source `previous_session_publish.current`. - If no canonical previous-session requirements exist, prepare does not manage `previous/`. - `manifest.Inputs` is sorted deterministically by `(kind, path)`. - S3 audio `manifest.Inputs` retain S3 provenance and include `cache_path`; `spool_path` is present only when the current prepare invocation downloaded the file. @@ -95,7 +95,7 @@ Does not own: - `previous_session_id` unset: - if any referenced previous artifact is required: fail; - if all referenced previous artifacts are optional: continue and omit them. -- Previous session archive current pointer or manifest missing: +- Previous session publish current pointer or manifest missing: - if any referenced previous artifact is required: fail; - if all referenced previous artifacts are optional: continue and omit missing ones. - Missing required previous artifact object: fail. @@ -120,5 +120,5 @@ Does not own: ## Architectural invariants - `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive. -- Storage keys are computed by callers using archive/path helpers; storage adapter receives explicit keys. +- Storage keys are computed by callers using path helpers; storage adapter receives explicit keys. - `prepare` is the only stage that hydrates canonical previous-session cache state. diff --git a/docs/internal/stage-archive.md b/docs/internal/stage-publish.md similarity index 54% rename from docs/internal/stage-archive.md rename to docs/internal/stage-publish.md index 923cd6e..8231473 100644 --- a/docs/internal/stage-archive.md +++ b/docs/internal/stage-publish.md @@ -1,4 +1,4 @@ -# Stage: archive +# Stage: publish ## Purpose Publish durable run/session state to object storage, then atomically advance remote current state. @@ -7,37 +7,37 @@ Publish durable run/session state to object storage, then atomically advance rem Inputs: - session manifest and prerequisite stage records - run root contents under `runs/{run_id}/` -- promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`) -- effective source-based promotion locks from static config and remote session lock store +- publish output rules with artifact `source` IDs and publish `dest` paths (`pipeline.publish.outputs`) +- effective source-based publish locks from static config and remote session lock store - session-level `previous/**` cache files when present Outputs: - uploaded run files under `{session_prefix}/runs/{run_id}/...` -- uploaded promoted artifacts under `{session_prefix}/...` +- uploaded published outputs under `{session_prefix}/...` - uploaded session previous-cache files under `{session_prefix}/previous/...` when present - `{session_prefix}/current/manifest.json` - `{session_prefix}/current/run_id.txt` written last ## Boundaries Owns: -- Archive enable/disable gate behavior -- Prerequisite stage success enforcement -- Run file collection and upload (excluding `audio/`) -- Promotion rule resolution and upload -- Promotion lock enforcement -- Session previous-cache file collection/upload -- Commit pointer publish order +- publish enable/disable gate behavior +- prerequisite stage success enforcement +- run file collection and upload (excluding `audio/`) +- publish output rule resolution and upload +- publish lock enforcement +- session previous-cache file collection/upload +- commit pointer publish order Does not own: -- Stage execution before archive -- Post-archive local cleanup policy execution (handled by app cleanup logic) +- stage execution before publish +- post-publish local cleanup policy execution (handled by app cleanup logic) ## Config Fields Used -- `pipeline.archive.enabled` -- `pipeline.archive.upload_run` -- `pipeline.archive.promote_artifacts` -- `pipeline.archive.locks` -- `{session_prefix}/locks.yml` loaded by app orchestration before archive execution +- `pipeline.publish.enabled` +- `pipeline.publish.upload_run` +- `pipeline.publish.outputs` +- `pipeline.publish.locks` +- `{session_prefix}/locks.yml` loaded by app orchestration before publish execution - `pipeline.storage.s3.bucket` - `pipeline.storage.s3.root_prefix` - `pipeline.workspace.root` @@ -51,26 +51,28 @@ Does not own: - Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`. - Resolves bucket/prefix from manifest identity first, then config fallback. - Uploads session `previous/**` files as durable session state when the local `previous/` directory exists. -- Skips top-level promotion uploads for effective locked sources; run-local uploads still publish. -- When selected configured artifact keys are supplied, skips promotion rules for unselected `narratio.artifact.<key>` sources; built-in transcript and bounds promotions still publish. -- Effective locks are the union of `pipeline.archive.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources. +- Skips top-level published output uploads for effective locked sources; run-local materialized outputs remain unchanged. +- When selected configured artifact keys are supplied, skips publish rules for unselected `narratio.artifact.<key>` sources; built-in transcript and bounds outputs still publish. +- Effective locks are the union of `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources. - Writes metadata including: - upload counts/paths - `previous_files_uploaded` and `previous_uploaded_paths` - - `skipped_unselected_promotions` - - `locked_promotion_count` and `locked_promotions` + - `published_files_uploaded` and `published_paths` + - `skipped_optional_outputs` + - `skipped_unselected_outputs` + - `locked_output_count` and `locked_outputs` - `current_manifest_key` - `current_run_id_key` - `current_pointer_written` -- On skipped archive path, returns metadata with `skipped=true` and pointer not written. +- On skipped publish path, returns metadata with `skipped=true` and pointer not written. ## Skip and Resume Behavior -- Stage may self-skip (metadata skip) when archive disabled or run upload disabled. +- Stage may self-skip (metadata skip) when publish disabled or run upload disabled. - Runner-level skip also applies for previously succeeded stage unless forced. ## Failure Behavior -- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required promotion source, upload failures, or pointer write failures. -- Locked required promotions are intentional skips and do not fail archive. +- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required output source, upload failures, or pointer write failures. +- Locked required outputs are intentional skips and do not fail publish. - Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail. ## Tests to Inspect Before Changing @@ -79,8 +81,8 @@ Does not own: ## Architectural Invariants - Run upload excludes `audio/` subtree. -- Session `previous/**` is archiveable durable input/provenance state, not run-local output. -- Ordinary `--force` does not override archive locks. -- Malformed or unreadable remote lock store fails archive-capable execution before promotion. +- Session `previous/**` is publishable durable input/provenance state, not run-local output. +- Ordinary `--force` does not override publish locks. +- Malformed or unreadable remote lock store fails publish-capable execution before output uploads. - `current/manifest.json` uploads before `current/run_id.txt`. - `current/run_id.txt` is the remote publish commit marker. diff --git a/docs/internal/stage-transcribe.md b/docs/internal/stage-transcribe.md index 907cdfa..8b7ce96 100644 --- a/docs/internal/stage-transcribe.md +++ b/docs/internal/stage-transcribe.md @@ -15,7 +15,7 @@ Owns: - Discovering prepared audio inputs - Deriving speaker ids from audio basenames - Parallel WhisperX invocation with bounded concurrency -- Validating produced JSON and promoting run-local outputs +- Validating produced JSON and materializing run-local outputs Does not own: - Transcript merge/polish/normalize/trim/analyze @@ -36,8 +36,8 @@ Does not own: ## State and Manifest Behavior - Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled. -- Validates each generated transcript JSON before promotion. -- Promotes canonical outputs to `transcripts/raw/*.json`. +- Validates each generated transcript JSON before materialization. +- Materializes canonical outputs to `transcripts/raw/*.json`. - Records per-file metadata (attempts/status/duration/output path) in stage metadata. ## Skip and Resume Behavior @@ -54,5 +54,5 @@ Does not own: ## Architectural Invariants - Speaker identity is derived from `.flac` basename and must be unique. -- Every successful speaker output must be valid JSON before promotion. +- Every successful speaker output must be valid JSON before materialization. - Canonical raw transcript set is the only supported merge input surface. diff --git a/docs/internal/stage-trim.md b/docs/internal/stage-trim.md index 292dfd8..d520e14 100644 --- a/docs/internal/stage-trim.md +++ b/docs/internal/stage-trim.md @@ -52,7 +52,7 @@ Does not own: ## State and Manifest Behavior - Reads final transcript from normalize manifest outputs when available; falls back to canonical path. - Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled. -- Promotes canonical final-trimmed transcript; promotes session bounds when trim enabled. +- Materializes canonical final-trimmed transcript and session bounds when trim is enabled. - Records bounds diagnostics, trim action, keep selector, and adapter metadata. ## Skip and Resume Behavior diff --git a/docs/internal/storage.md b/docs/internal/storage.md index 9e8e4e2..610f7a4 100644 --- a/docs/internal/storage.md +++ b/docs/internal/storage.md @@ -37,12 +37,12 @@ Does not own: ## External adapters used Storage package contracts: - `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`. -- `Backend` (archive request boundary): currently implemented with `NoopBackend` only. +- `Backend` (legacy compatibility boundary): currently implemented with `NoopBackend` only. Implementations: - `S3Backend`: AWS SDK-backed `ObjectStore` implementation. -- `FakeBackend`: deterministic test `ObjectStore` and archive backend. -- `NoopBackend`: deterministic no-op archive backend for compatibility wiring. +- `FakeBackend`: deterministic test `ObjectStore` and compatibility backend. +- `NoopBackend`: deterministic no-op compatibility backend for wiring/tests. ## State and manifest behavior - Storage implementations are stateless with respect to manifest/session lifecycle. @@ -67,7 +67,7 @@ Implementations: - `internal/adapters/storage/s3_backend_test.go` - `internal/adapters/storage/fake_test.go` - `internal/adapters/storage/keys_test.go` -- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `archive`) +- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `publish`) ## Architectural invariants - Callers pass full bucket-relative keys. diff --git a/docs/internal/workspace.md b/docs/internal/workspace.md index 00580e5..ca97482 100644 --- a/docs/internal/workspace.md +++ b/docs/internal/workspace.md @@ -1,7 +1,7 @@ # Workspace internals ## Purpose -Define the local durable and run-local workspace model used by stages, manifests, resume, and archive. +Define the local durable and run-local workspace model used by stages, manifests, resume, and publish. ## Inputs and Outputs Inputs: @@ -24,14 +24,14 @@ Owns: Does not own: - Stage business logic -- Remote archive semantics (documented in `stage-archive.md`) +- Remote publish semantics (documented in `stage-publish.md`) - CLI argument parsing ## Config Fields Used - `pipeline.workspace.root` -- `pipeline.workspace.cleanup_after_archive` +- `pipeline.workspace.cleanup_after_publish` - `pipeline.spool.root` -- `pipeline.spool.delete_audio_after_archive` +- `pipeline.spool.delete_audio_after_publish` - `pipeline.cache.root` - `pipeline.cache.s3_audio` - `session.campaign` @@ -43,10 +43,10 @@ None directly in this subsystem. Stages may use object storage adapters and then ## State and Manifest Behavior - Session state is persisted in the session manifest (`manifest.Manifest`). - Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`. -- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and promoted to canonical session paths after stage success. +- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and then materialized to canonical session paths after stage success. - `manifest.Artifacts` entries record `ProducerRunID` for durable outputs. - For S3 audio sessions, `prepare` records work/cache paths, S3 provenance, and spool path when the invocation downloaded the object. -- `previous/**` is reconstructed from configured previous-session requirements; restore uses the previous session's committed current archive rather than treating current-session archived `previous/**` as authoritative. +- `previous/**` is reconstructed from configured previous-session requirements; restore uses the previous session's committed current publish state rather than treating current-session stored `previous/**` as authoritative. - Durable cache state under `pipeline.cache.root` is not workspace state and is preserved by default by `narratio clean`. - `narratio clean <id>` removes the session work root and session spool root. - `narratio clean --all` removes all local session work under `workspace.root/work` and spool children under `spool.root`. @@ -60,7 +60,7 @@ None directly in this subsystem. Stages may use object storage adapters and then ## Failure Behavior - Failures preserve manifests and run-local files for inspection. - Lock conflicts fail fast via `ErrLockConflict`. -- Cleanup can fail post-archive; failure is recorded in archive stage metadata and returned by the run. +- Cleanup can fail post-publish; failure is recorded in publish stage metadata and returned by the run. ## Tests to Inspect Before Changing - `internal/artifacts/local_test.go` @@ -72,7 +72,7 @@ None directly in this subsystem. Stages may use object storage adapters and then ## Architectural Invariants - Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`. - Run roots are always nested: `runs/{run_id}` under the session root. -- Run-local output promotion must end in canonical session paths. +- Run-local output materialization must end in canonical session paths. - `previous/**` is session-durable state and must not be treated as run-local output scratch state. -- Automatic post-archive cleanup only targets run-scoped directories and must never delete configured root directories. +- Automatic post-publish cleanup only targets run-scoped directories and must never delete configured root directories. - Manual `clean` may delete session-scoped directories or the `workspace.root/work` directory, but it must preserve configured root directories and reject unsafe targets. diff --git a/docs/operations.md b/docs/operations.md index 2c4306d..4e1ce07 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,285 +1,170 @@ # Operations -This guide describes the implemented operator lifecycle for Narratio. +This guide covers the implemented operator lifecycle for Narratio. -For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md). +For field-level settings, see [docs/config.md](./config.md). For syntax/flags, see [docs/cli.md](./cli.md). -## Normal workflow (S3-first path) +## Normal Workflow -1. Create or upload `session.yml`, or pass a local `session.yml` explicitly. -2. Upload session `.flac` files to object storage under the configured session audio prefix. -3. Run Narratio: +1. Ensure `pipeline.yml`, `campaign.yml`, and `session.yml` are available. +2. Ensure session audio is available (local `audio_dir`/`audio_files` or S3 prefix). +3. Run: ```bash narratio run 2026-04-04 ``` -4. Read success output: -- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>` -- use `narratio session status <session_id>` for inspection. - -Notes: -- default pipeline/session discovery checks system config locations; campaign selection uses `pipeline.campaigns.default_campaign_id` unless `--campaign <id>` or `--campaign-file <path>` is passed. -- when local `session.yml` discovery misses, positional `<session_id>` loads remote `session.yml` from `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`. -- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access. - -Initialize a remote session skeleton: - -```bash -narratio session init 2026-04-04 --remote -``` - -Remote init uses normal default config discovery and writes `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`. If `campaign.yml` sets `session_template_file`, init renders that template from the supplied flags and writes concrete YAML. Pass `--config`, `--campaign <id>`, or `--campaign-file <path>` when testing non-system config files. It fails if the object already exists unless `--force` is passed. - -Validate before running: - -```bash -narratio session validate 2026-04-04 -``` - -## Restore workflow - -Use restore when local durable session state is missing or stale and archive current state is authoritative. - -Dry-run (no local writes): - -```bash -narratio session restore 2026-04-04 --dry-run -``` - -Execution: - -```bash -narratio session restore 2026-04-04 -``` - -Post-restore analyze rerun pattern: - -```bash -narratio analyze 2026-04-04 -``` - -Restore source-of-truth: -- remote commit marker: `current/run_id.txt` -- remote current manifest: `current/manifest.json` -- configured previous-session requirements are reconstructed from the previous session's remote `current/` state, not from archived `previous/**` objects in the current session. - -Restore default scope: -- includes `manifest.json`, `transcripts/**`, and `artifacts/**` from the current session archive. -- includes `previous/**` only when configured previous-session artifact inputs require it; restore hydrates those files the same way `prepare` would. -- includes `audio/**` only with `--include-audio` -- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source) - -Reset local state before restore testing: - -```bash -narratio clean 2026-04-04 --dry-run -narratio clean 2026-04-04 -narratio session restore 2026-04-04 --include-audio -``` - -`clean` removes the local session work directory and session spool directory. It preserves the durable S3 audio cache by default, so repeated restore or forced prepare tests do not re-download large audio files. - -## Local filesystem layout and state artifacts - -Session root: -- `{workspace.root}/work/{campaign}/{session_id}/` - -Primary state: -- `manifest.json`: session-level stage state. -- `runs/{run_id}/manifest.json`: invocation-level state. -- `.lock`: session lock while a modifying command is active. -- `inputs/campaign.yml`, `inputs/session.yml`, and `inputs/pipeline.resolved.yml`: materialized config inputs for the run. - -Canonical session directories: -- `inputs/` -- `audio/` -- `transcripts/` -- `artifacts/` -- `previous/` -- `reports/` -- `logs/` -- `config/` -- `current/` -- `runs/` - -Run-local stage directories: -- `runs/{run_id}/{stage}/` with stage-local `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`. - -Behavior: -- directory creation is idempotent. -- stage outputs are generally generated run-local first, then promoted to canonical paths on success. -- restore installs downloaded files to canonical session paths and does not recreate historical run sandboxes. - -## Analyze artifact execution lifecycle - -Analyze executes configured artifacts from `pipeline.scriptorium.artifacts`. - -Execution model: -- executable set = enabled artifacts, filtered by `--artifacts` when provided. -- artifact-to-artifact dependencies are declared via `depends_on`. -- selected artifacts run in deterministic dependency order. -- after each successful artifact run, output is promoted to configured canonical `output_path`. - -Configured artifact source reuse: -- a non-executable configured artifact can satisfy inputs if its configured output file already exists and is valid. -- reused configured artifact provenance is `filesystem.disabled_artifact_output`. - -`--artifacts` behavior: -- accepted on `run`, `resume`, `run-stage analyze`, `run-stage archive`, `analyze`, and `publish`. -- filters analyze execution and configured artifact promotions. -- built-in transcript and bounds promotions are not filtered. -- does not imply force on `run`, `resume`, or `run-stage`; `narratio analyze` is force-by-design. -- `publish` is force-by-design and accepts `--artifacts` for configured artifact promotions. - -Canonical previous-session input behavior: -- canonical sources use `narratio.previous_session.artifact.<artifact_key>`. -- these inputs are hydrated by `prepare` and by `restore`; `analyze` expects the local previous cache to already exist. -- if analyze fails due to missing canonical previous cache, rerun: - - `narratio run-stage prepare <id> --force` - - or `narratio session restore <id>` when remote archive current state is authoritative. - -## Remote archive layout and publish contract - -Preferred manual publish command: - -```bash -narratio publish <id> -``` - -`publish` is equivalent to `narratio run-stage archive <id> --force`; use `run-stage` when you need the general single-stage command form. - -When archive is enabled and run upload is enabled, archive publishes under: - -- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/` -- run prefix: `{session_prefix}/runs/{run_id}/` - -Archive uploads: -- run record files from run root (excluding `audio/`). -- promoted files from explicit `archive.promote_artifacts` rules. -- mutable session locks from helper commands live at `{session_prefix}/locks.yml`. - -Publish order: -1. upload `current/manifest.json` -2. upload `current/run_id.txt` last - -`current/run_id.txt` is the remote commit marker. - -Archive promotion is explicit and source-based: -- Narratio does not auto-promote all generated analyze artifacts. -- each rule resolves `source` through the artifact resolver/catalog model, then uploads to `dest`. -- missing required promotion sources fail archive stage. -- missing optional promotion sources are skipped. -- invalid resolved artifacts fail archive stage. -- `archive.locks` skips top-level promotion overwrites for locked sources while run-local uploads still publish. -- remote locks from `{session_prefix}/locks.yml` are merged with static `archive.locks`; static locks win on duplicate sources. -- locked required promotions are treated as intentional successful skips and are recorded in archive metadata. - -Lock helper behavior: -- `narratio session locks <id>` lists effective static and remote locks. -- `narratio session locks add <id> <source> --reason <text>` writes a remote lock. -- `narratio session locks add <id> <source> --force --reason <text>` updates an existing remote lock reason. -- `narratio session locks remove <id> <source>` removes only a remote lock. -- `locks remove` cannot remove static pipeline locks. -- remote lock writes check whether the lock store exists, but are not compare-and-swap atomic. - -## Resume, retry, restore, and safe rerun behavior - -Default skip: -- `run` and `run-stage` skip already-succeeded stages unless `--force` is set. - -Resume: -- `resume` starts at first non-succeeded stage. -- `resume --force` runs full stage order. - -Restore conflict policy: -- restore classifies local differences as conflicts. -- without `--force`, restore fails when conflicts exist. -- with `--force`, conflicting local files are overwritten by remote archive files. - -Forced reruns: -- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`. -- ordinary `--force` does not override archive locks. - -Safe rerun pattern: -1. rerun the changed stage with `--force`. -2. run `resume` to rebuild downstream stages. - -## Cleanup behavior - -Automatic post-archive cleanup is considered only when archive stage executed and succeeded. - -Automatic cleanup toggles: -- `pipeline.spool.delete_audio_after_archive=true` deletes run-scoped spool audio. -- `pipeline.workspace.cleanup_after_archive=true` deletes run-scoped local run directory. - -Manual cleanup: -- `narratio clean <id>` deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`. -- `narratio clean --all` deletes all local session work under `{workspace.root}/work` and all spool children under `{spool.root}`. -- `--dry-run` prints targets without deleting. -- `--clear-cache` also removes matching S3 audio cache files. Without it, cache is preserved. - -The S3 audio cache under `pipeline.cache.root` is durable input cache state, not workspace or spool state. Automatic cleanup and default manual cleanup do not delete it. - -Cleanup eligibility gates: -- archive enabled -- archive run upload enabled -- run record upload completed -- current pointer write completed (`current/run_id.txt` written) - -No cleanup for failed/incomplete/unarchived/archive-skipped runs. - -## Failure and recovery playbooks - -After run failure, Narratio keeps: -- session manifest -- run manifest -- run-local artifacts/logs/config/reports - -Failed or incomplete runs remain local-only. - -After restore failure: -- already-installed restore files remain in place. -- restore does not roll back prior successful installs. -- existing local manifest is preserved if restored manifest validation/install fails. - -Recommended recovery: - -1. inspect state: +4. Inspect status: ```bash narratio session status 2026-04-04 ``` -This reports local manifest state, committed remote current state, expected remote transcript/artifact availability, and archive locks. +## Publish Workflow -2. for restore-specific checks, run: +Publish is the stage that commits remote current state. + +```bash +narratio publish 2026-04-04 +``` + +Equivalent command: + +```bash +narratio run-stage publish 2026-04-04 --force +``` + +Publish uploads: + +- run history files under `{session_prefix}/runs/{run_id}/` (excluding `audio/`) +- configured published outputs from `pipeline.publish.outputs` +- `previous/**` cache files when present +- `current/manifest.json` +- `current/run_id.txt` last + +`current/run_id.txt` is the remote commit marker. + +## Published Outputs and Locks + +Published output behavior: + +- outputs are source-based rules in `pipeline.publish.outputs`. +- required missing unlocked sources fail publish. +- optional missing unlocked sources are skipped. +- selected artifacts (`--artifacts`) only filter configured `narratio.artifact.<key>` output rules. +- built-in transcript and bounds output rules are not filtered by `--artifacts`. + +Lock behavior: + +- static locks: `pipeline.publish.locks`. +- mutable locks: `{session_prefix}/locks.yml`. +- effective lock set is static + mutable; static wins on duplicate sources. +- locked outputs are intentional skips and do not fail publish. +- lock commands mutate only remote mutable locks. + +## Restore Workflow + +Use restore when local durable session state is missing/stale and committed remote current state is authoritative. + +Preview: ```bash narratio session restore 2026-04-04 --dry-run ``` -3. fix root cause (config/input/credentials/storage/service availability). -4. continue with `resume`, or targeted `run-stage <stage> <id> --force` followed by `resume`. +Apply: -## Restore report +```bash +narratio session restore 2026-04-04 +``` -Non-dry-run restore writes a durable report at: -- `reports/restore-latest.json` +Default restore scope: -Report content includes: -- identity (`campaign`, `session_id`, `run_id`) -- mode flags (`dry_run`, `force`, `include_audio`) -- plan counts and execution counts -- per-action status +- `manifest.json` +- `transcripts/**` +- `artifacts/**` +- `previous/**` when required by configured previous-session artifact inputs -Dry-run does not write restore report files. +Optional: -## Operational caveats +- add `--include-audio` to restore `audio/**`. -- `session status <session_id>` uses normal config/session loading, including remote session fallback. -- `session status <session_id>` includes the same promoted remote output availability view as `session artifacts <session_id> --remote` when storage is configured. -- local and S3 audio input modes are mutually exclusive. -- archive publish requires upstream stages through `analyze` to be `succeeded`. -- required configured artifact promotions for unselected `--artifacts` keys are skipped intentionally; selected required promotions still fail if their files are missing. -- restore requires configured remote object storage and committed remote current state. +Restore reads committed current state only (`current/run_id.txt`, `current/manifest.json`). + +## Workspace and State Layout + +Session root: + +- `{workspace.root}/work/{campaign}/{session_id}/` + +Durable session state: + +- `manifest.json` +- `inputs/**` +- `audio/**` +- `transcripts/**` +- `artifacts/**` +- `previous/**` +- `reports/**` +- `logs/**` +- `config/**` +- `runs/**` + +Run-local stage layout: + +- `runs/{run_id}/{stage}/outputs|logs|reports|config|scratch` + +Stages typically write run-local outputs first, then materialize canonical session outputs on success. + +## Resume and Force Rules + +- `run` and `run-stage` skip succeeded stages unless `--force` is set. +- `resume` starts at the first non-succeeded stage. +- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`. +- `--force` does not bypass publish locks. + +## Cleanup + +Automatic post-publish cleanup is considered only when publish executes successfully and commits current state. + +Config toggles: + +- `pipeline.spool.delete_audio_after_publish=true` +- `pipeline.workspace.cleanup_after_publish=true` + +Manual cleanup: + +```bash +narratio clean 2026-04-04 +narratio clean --all +``` + +Cache is preserved by default. Use `--clear-cache` to remove matching S3 audio cache entries. + +## Failure and Recovery + +After stage failure, Narratio keeps manifests and run-local files for inspection. + +Standard recovery flow: + +1. inspect status: + +```bash +narratio session status 2026-04-04 +``` + +2. if needed, inspect restore plan: + +```bash +narratio session restore 2026-04-04 --dry-run +``` + +3. fix root cause. +4. continue with `resume`, or rerun a stage with `--force` then `resume`. + +## Operational Caveats + +- local and S3 audio modes are mutually exclusive. +- publish requires prerequisite stages through `analyze` to be `succeeded`. +- restore requires configured object storage and committed current state. +- `session status` and `session artifacts --remote` both report remote published-output availability when storage is configured. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 4e4de1f..ae7e0e0 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,17 +1,17 @@ # Troubleshooting ## Purpose -Canonical operator troubleshooting guide for recurring implemented Narratio failures. +Canonical operator troubleshooting guide for recurring Narratio failures. -## Config file discovery failure +## Config discovery failure Symptom: -- `run`, `resume`, `run-stage`, `session plan`, or `session restore` fails with config/session not found. +- command fails because `pipeline.yml`, `campaign.yml`, or `session.yml` was not found. -Likely Cause: -- `pipeline.yml` or `session.yml` is missing from system discovery paths. -- the selected campaign ID does not exist under `pipeline.campaigns.root`. -- a local working-directory config file was not passed explicitly. +Likely cause: +- missing files in discovery paths. +- missing/incorrect campaign selection. +- local file exists but was not passed explicitly. Diagnostics: @@ -20,160 +20,73 @@ ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml ls -l /usr/local/etc/narratio/session.yml /etc/narratio/session.yml ``` -Safe Fix: -- pass explicit `--config`, `--campaign <id>`, `--campaign-file <path>`, and `--session` as appropriate. -- or place files in documented discovery paths and set `pipeline.campaigns.default_campaign_id`. - -Links: -- [docs/config.md](./config.md) -- [docs/cli.md](./cli.md) +Safe fix: +- pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`. ## Templated session file rejected Symptom: -- load fails with a message that `session.yml must be concrete`. +- load fails because `session.yml` must be concrete. -Likely Cause: -- a template authoring file such as `session.template.yml` was passed to `--session` or uploaded as remote `session.yml`. -- `session.yml` still contains `{{ ... }}` placeholders. +Likely cause: +- template placeholders (`{{ ... }}`) still present in loaded session config. Diagnostics: ```bash -narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session ./session.yml +narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml ``` -Safe Fix: -- generate concrete YAML with `narratio session init`. -- pass the generated concrete `session.yml` to downstream commands or upload it through `session init --remote`. +Safe fix: +- generate concrete session YAML via `narratio session init`. -Links: -- [docs/config.md](./config.md) - -## Strict YAML decode or validation failure +## Strict decode or validation failure Symptom: -- config load fails with unknown field or validation error. +- unknown field or invalid value error during config load. -Likely Cause: -- typo/stale field name. -- missing required fields or invalid constraints. +Likely cause: +- typo, stale field name, or invalid value. Diagnostics: ```bash -narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml +narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml ``` -Safe Fix: -- align fields/values to canonical config reference and examples. - -Links: -- [docs/config.md](./config.md) -- [examples/](../examples/) +Safe fix: +- align config with [docs/config.md](./config.md) and maintained examples. ## `--artifacts` selection failure Symptom: -- `run`/`resume`/`run-stage` fails with invalid or unknown artifact selection. +- command fails on unknown/invalid selected artifact key. -Likely Cause: -- `--artifacts` contains blank names or unknown artifact keys. -- `pipeline.scriptorium.artifacts` missing while using `--artifacts`. +Likely cause: +- artifact key not defined in `pipeline.scriptorium.artifacts`. +- empty token in `--artifacts` input. -Diagnostics: +Safe fix: +- use only configured artifact keys. -```bash -narratio run 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts player_handout -``` - -Safe Fix: -- use configured artifact keys only. -- ensure `pipeline.scriptorium.artifacts` is defined. - -Links: -- [docs/cli.md](./cli.md) -- [docs/config.md](./config.md) - -## `run-stage --artifacts` on unsupported stage +## `run-stage --artifacts` unsupported stage Symptom: -- `run-stage` fails because `--artifacts` is only supported for `analyze` and `archive`. +- `run-stage` rejects `--artifacts` for the selected stage. -Likely Cause: -- `--artifacts` was used with a stage other than `analyze` or `archive`. +Likely cause: +- `--artifacts` used with a stage other than `analyze` or `publish`. -Diagnostics: +Safe fix: +- use `--artifacts` only with `run-stage analyze ...` or `run-stage publish ...`. -```bash -narratio run-stage polish 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts session_recap -``` - -Safe Fix: -- use `--artifacts` only with `run-stage analyze ...` or `run-stage archive ...`. - -Links: -- [docs/cli.md](./cli.md) - -## Configured artifact dependency/input validation failure +## Previous-session input unavailable Symptom: -- config validation fails for `depends_on`, `narratio.artifact.<name>` source, or artifact output path. +- analyze fails on required previous-session artifact input. -Likely Cause: -- `narratio.artifact.<name>` source missing matching `depends_on` key. -- dependency references unknown artifact key. -- dependency self-reference or enabled dependency cycle. -- artifact output path missing/invalid/outside `artifacts/` root. - -Diagnostics: - -```bash -narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml -``` - -Safe Fix: -- ensure artifact-to-artifact inputs have explicit `depends_on` entries using artifact keys. -- ensure referenced artifacts exist and define valid `output_path` values. -- keep output paths relative and under `artifacts/`. - -Links: -- [docs/config.md](./config.md) -- [docs/internal/stage-analyze.md](./internal/stage-analyze.md) - -## Required configured artifact input unavailable at analyze time - -Symptom: -- analyze fails because configured input source is unavailable. - -Likely Cause: -- required upstream configured artifact was not selected/executed this run. -- non-executable dependency output file is missing or invalid on disk. - -Diagnostics: - -```bash -narratio session status 2026-04-04 -narratio run-stage analyze 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts player_handout -``` - -Safe Fix: -- run analyze with needed artifacts selected. -- or ensure dependency output file exists at configured path and is valid. - -Links: -- [docs/operations.md](./operations.md) -- [docs/config.md](./config.md) - -## Manifest/status path failure - -Symptom: -- `session status` fails because config/session state is missing, unreadable, or invalid. - -Likely Cause: -- wrong session ID. -- wrong config/campaign/session file selected. -- manifest removed after cleanup. +Likely cause: +- `previous/**` cache not hydrated for this session. Diagnostics: @@ -181,137 +94,75 @@ Diagnostics: narratio session status 2026-04-04 ``` -Safe Fix: -- use the same session ID and config files that will be used for `run`, `resume`, or `run-stage`. +Safe fix: -Links: -- [docs/cli.md](./cli.md) -- [docs/operations.md](./operations.md) +```bash +narratio run-stage prepare 2026-04-04 --force +``` + +Or rehydrate from remote current state: + +```bash +narratio session restore 2026-04-04 +``` ## Session lock conflict (`.lock`) Symptom: -- `run`, `resume`, `run-stage`, or `session restore` fails with lock conflict for session workdir. +- command fails with lock conflict. -Likely Cause: -- another Narratio process is running same session. -- stale lock from interrupted prior run. +Likely cause: +- another process is running for the same session. +- stale lock file from interrupted command. Diagnostics: ```bash ls -l {workspace.root}/work/{campaign}/{session_id}/.lock -cat {workspace.root}/work/{campaign}/{session_id}/.lock ps aux | grep narratio ``` -Safe Fix: -- wait for active process to finish. -- if no process is active, remove only stale session `.lock` file. +Safe fix: +- wait for active process; remove stale lock only if no process is active. -Links: -- [docs/operations.md](./operations.md) -- [docs/internal/workspace.md](./internal/workspace.md) - -## Restore remote current pointer or manifest missing +## Restore current pointer/manifest missing Symptom: -- `session restore` fails with remote current pointer or current manifest errors. +- restore fails reading remote current state. -Likely Cause: -- `current/run_id.txt` was never published. -- `current/manifest.json` is missing for the session prefix. -- archive commit did not complete. +Likely cause: +- publish commit did not complete. +- `current/run_id.txt` or `current/manifest.json` is missing. Diagnostics: ```bash -narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run +narratio session restore 2026-04-04 --dry-run ``` -Safe Fix: -- verify archive stage succeeded for the target session. -- rerun/archive from a healthy source workspace so current pointers are published. - -Links: -- [docs/operations.md](./operations.md) -- [docs/internal/stage-archive.md](./internal/stage-archive.md) - -## Restore manifest identity mismatch - -Symptom: -- `session restore` fails because remote manifest session or campaign does not match requested values. - -Likely Cause: -- wrong positional session ID or wrong session config selected. -- archive prefix points to a different campaign/session. - -Diagnostics: - -```bash -narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run -``` - -Safe Fix: -- use the correct session config and positional session ID. -- verify campaign/session identity in local config before restore. - -Links: -- [docs/config.md](./config.md) -- [docs/operations.md](./operations.md) +Safe fix: +- republish from a healthy local session state. ## Restore conflict without `--force` Symptom: -- `session restore` fails with `restore conflict` and conflict counts. +- restore reports conflict and exits. -Likely Cause: -- local durable file differs from remote file for one or more planned restore paths. +Likely cause: +- local durable file differs from remote restore source. -Diagnostics: +Safe fix: +- inspect with `--dry-run`. +- rerun with `--force` only when remote should overwrite local. -```bash -narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run -``` - -Safe Fix: -- review planned conflicts. -- rerun with `--force` only when remote state should overwrite local state. - -Links: -- [docs/cli.md](./cli.md) -- [docs/operations.md](./operations.md) - -## Restore report expectations +## Secrets or credentials failure Symptom: -- operator expects restore report file but does not find one. +- startup fails loading secrets dir, or storage/tool auth fails at runtime. -Likely Cause: -- restore was executed in `--dry-run` mode. -- restore failed before report persistence path (for example lock acquisition failure). - -Diagnostics: - -```bash -ls -l {workspace.root}/work/{campaign}/{session_id}/reports/restore-latest.json -``` - -Safe Fix: -- run non-dry-run restore for durable report output. -- resolve lock or early preflight failures and retry. - -Links: -- [docs/operations.md](./operations.md) - -## Secrets env-dir or credential-env failure - -Symptom: -- startup fails loading secrets directory, or stage fails due to missing credential env vars. - -Likely Cause: -- invalid `pipeline.secrets.env_dir` path/permissions. -- required credential env var unset/empty. +Likely cause: +- invalid `pipeline.secrets.env_dir`. +- missing credential env vars. Diagnostics: @@ -320,60 +171,53 @@ ls -la /path/to/secrets_dir env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM' ``` -Safe Fix: -- fix secrets directory and credential env vars. -- keep secret values out of YAML. +Safe fix: +- fix path/permissions/env vars; keep secret values out of YAML. -Links: -- [docs/config.md](./config.md) - -## S3-audio prepare failure +## S3 audio prepare failure Symptom: -- `prepare` fails in S3 mode (listing/downloading/no audio/backend error). +- prepare fails in S3 mode (list/download/no files/backend error). -Likely Cause: -- wrong `session.inputs.audio_s3.prefix`. -- no `.flac` files at resolved prefix. -- invalid/missing object-store credentials or backend config. -- mixed local+S3 audio input config. +Likely cause: +- bad `session.inputs.audio_s3.prefix`. +- no `.flac` objects at prefix. +- bad storage credentials/config. +- mixed local+S3 audio config. Diagnostics: ```bash -narratio run-stage prepare 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml +narratio run-stage prepare 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml ``` -Safe Fix: -- configure exactly one audio source mode. -- verify `.flac` files and storage access. +Safe fix: +- configure exactly one audio mode and verify storage access. -Links: -- [docs/config.md](./config.md) -- [docs/operations.md](./operations.md) - -## Archive promotion/current-pointer failure +## Publish output or current-pointer failure Symptom: -- archive fails on required promotion source missing or pointer write failure. +- publish fails on required output source missing, upload error, or commit-marker write failure. -Likely Cause: -- required promoted file absent (including analyze outputs not generated for this run). -- storage upload failed before `current/run_id.txt` commit marker write. +Likely cause: +- required source file not produced. +- storage upload failed before `current/run_id.txt` write. Diagnostics: ```bash narratio session status 2026-04-04 -narratio run-stage archive 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml +narratio run-stage publish 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml ``` -Safe Fix: -- rerun or resume upstream stages to generate required files. -- adjust promotion `source`/`dest` rules to match artifacts that must exist. -- retry after storage issue is resolved. +Safe fix: +- rerun upstream stages to regenerate required outputs. +- adjust `pipeline.publish.outputs` source/dest rules. +- retry after storage issue is fixed. + +## Helpful Links -Links: -- [docs/operations.md](./operations.md) - [docs/config.md](./config.md) -- [docs/internal/stage-archive.md](./internal/stage-archive.md) +- [docs/cli.md](./cli.md) +- [docs/operations.md](./operations.md) +- [docs/internal/stage-publish.md](./internal/stage-publish.md) diff --git a/examples/pipeline.full.annotated.yml b/examples/pipeline.full.annotated.yml index 0e82ed1..0aa8cb6 100644 --- a/examples/pipeline.full.annotated.yml +++ b/examples/pipeline.full.annotated.yml @@ -43,7 +43,7 @@ publish: # Optional booleans; defaults are true. enabled: true upload_run: true - # Optional promotion rules; sources use Narratio artifact source IDs. + # Optional publish output rules; sources use Narratio artifact source IDs. outputs: - source: narratio.transcript.final_trimmed dest: transcripts/final.trimmed.json