Rewrite docs for the publish stage contract and current behavior
This commit is contained in:
@@ -1,285 +1,170 @@
|
||||
# Operations
|
||||
|
||||
This guide describes the implemented operator lifecycle for Narratio.
|
||||
This guide covers the implemented operator lifecycle for Narratio.
|
||||
|
||||
For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md).
|
||||
For field-level settings, see [docs/config.md](./config.md). For syntax/flags, see [docs/cli.md](./cli.md).
|
||||
|
||||
## Normal workflow (S3-first path)
|
||||
## Normal Workflow
|
||||
|
||||
1. Create or upload `session.yml`, or pass a local `session.yml` explicitly.
|
||||
2. Upload session `.flac` files to object storage under the configured session audio prefix.
|
||||
3. Run Narratio:
|
||||
1. Ensure `pipeline.yml`, `campaign.yml`, and `session.yml` are available.
|
||||
2. Ensure session audio is available (local `audio_dir`/`audio_files` or S3 prefix).
|
||||
3. Run:
|
||||
|
||||
```bash
|
||||
narratio run 2026-04-04
|
||||
```
|
||||
|
||||
4. Read success output:
|
||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
- use `narratio session status <session_id>` for inspection.
|
||||
|
||||
Notes:
|
||||
- default pipeline/session discovery checks system config locations; campaign selection uses `pipeline.campaigns.default_campaign_id` unless `--campaign <id>` or `--campaign-file <path>` is passed.
|
||||
- when local `session.yml` discovery misses, positional `<session_id>` loads remote `session.yml` from `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
|
||||
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
|
||||
|
||||
Initialize a remote session skeleton:
|
||||
|
||||
```bash
|
||||
narratio session init 2026-04-04 --remote
|
||||
```
|
||||
|
||||
Remote init uses normal default config discovery and writes `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`. If `campaign.yml` sets `session_template_file`, init renders that template from the supplied flags and writes concrete YAML. Pass `--config`, `--campaign <id>`, or `--campaign-file <path>` when testing non-system config files. It fails if the object already exists unless `--force` is passed.
|
||||
|
||||
Validate before running:
|
||||
|
||||
```bash
|
||||
narratio session validate 2026-04-04
|
||||
```
|
||||
|
||||
## Restore workflow
|
||||
|
||||
Use restore when local durable session state is missing or stale and archive current state is authoritative.
|
||||
|
||||
Dry-run (no local writes):
|
||||
|
||||
```bash
|
||||
narratio session restore 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
Execution:
|
||||
|
||||
```bash
|
||||
narratio session restore 2026-04-04
|
||||
```
|
||||
|
||||
Post-restore analyze rerun pattern:
|
||||
|
||||
```bash
|
||||
narratio analyze 2026-04-04
|
||||
```
|
||||
|
||||
Restore source-of-truth:
|
||||
- remote commit marker: `current/run_id.txt`
|
||||
- remote current manifest: `current/manifest.json`
|
||||
- configured previous-session requirements are reconstructed from the previous session's remote `current/` state, not from archived `previous/**` objects in the current session.
|
||||
|
||||
Restore default scope:
|
||||
- includes `manifest.json`, `transcripts/**`, and `artifacts/**` from the current session archive.
|
||||
- includes `previous/**` only when configured previous-session artifact inputs require it; restore hydrates those files the same way `prepare` would.
|
||||
- includes `audio/**` only with `--include-audio`
|
||||
- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source)
|
||||
|
||||
Reset local state before restore testing:
|
||||
|
||||
```bash
|
||||
narratio clean 2026-04-04 --dry-run
|
||||
narratio clean 2026-04-04
|
||||
narratio session restore 2026-04-04 --include-audio
|
||||
```
|
||||
|
||||
`clean` removes the local session work directory and session spool directory. It preserves the durable S3 audio cache by default, so repeated restore or forced prepare tests do not re-download large audio files.
|
||||
|
||||
## Local filesystem layout and state artifacts
|
||||
|
||||
Session root:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/`
|
||||
|
||||
Primary state:
|
||||
- `manifest.json`: session-level stage state.
|
||||
- `runs/{run_id}/manifest.json`: invocation-level state.
|
||||
- `.lock`: session lock while a modifying command is active.
|
||||
- `inputs/campaign.yml`, `inputs/session.yml`, and `inputs/pipeline.resolved.yml`: materialized config inputs for the run.
|
||||
|
||||
Canonical session directories:
|
||||
- `inputs/`
|
||||
- `audio/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `previous/`
|
||||
- `reports/`
|
||||
- `logs/`
|
||||
- `config/`
|
||||
- `current/`
|
||||
- `runs/`
|
||||
|
||||
Run-local stage directories:
|
||||
- `runs/{run_id}/{stage}/` with stage-local `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`.
|
||||
|
||||
Behavior:
|
||||
- directory creation is idempotent.
|
||||
- stage outputs are generally generated run-local first, then promoted to canonical paths on success.
|
||||
- restore installs downloaded files to canonical session paths and does not recreate historical run sandboxes.
|
||||
|
||||
## Analyze artifact execution lifecycle
|
||||
|
||||
Analyze executes configured artifacts from `pipeline.scriptorium.artifacts`.
|
||||
|
||||
Execution model:
|
||||
- executable set = enabled artifacts, filtered by `--artifacts` when provided.
|
||||
- artifact-to-artifact dependencies are declared via `depends_on`.
|
||||
- selected artifacts run in deterministic dependency order.
|
||||
- after each successful artifact run, output is promoted to configured canonical `output_path`.
|
||||
|
||||
Configured artifact source reuse:
|
||||
- a non-executable configured artifact can satisfy inputs if its configured output file already exists and is valid.
|
||||
- reused configured artifact provenance is `filesystem.disabled_artifact_output`.
|
||||
|
||||
`--artifacts` behavior:
|
||||
- accepted on `run`, `resume`, `run-stage analyze`, `run-stage archive`, `analyze`, and `publish`.
|
||||
- filters analyze execution and configured artifact promotions.
|
||||
- built-in transcript and bounds promotions are not filtered.
|
||||
- does not imply force on `run`, `resume`, or `run-stage`; `narratio analyze` is force-by-design.
|
||||
- `publish` is force-by-design and accepts `--artifacts` for configured artifact promotions.
|
||||
|
||||
Canonical previous-session input behavior:
|
||||
- canonical sources use `narratio.previous_session.artifact.<artifact_key>`.
|
||||
- these inputs are hydrated by `prepare` and by `restore`; `analyze` expects the local previous cache to already exist.
|
||||
- if analyze fails due to missing canonical previous cache, rerun:
|
||||
- `narratio run-stage prepare <id> --force`
|
||||
- or `narratio session restore <id>` when remote archive current state is authoritative.
|
||||
|
||||
## Remote archive layout and publish contract
|
||||
|
||||
Preferred manual publish command:
|
||||
|
||||
```bash
|
||||
narratio publish <id>
|
||||
```
|
||||
|
||||
`publish` is equivalent to `narratio run-stage archive <id> --force`; use `run-stage` when you need the general single-stage command form.
|
||||
|
||||
When archive is enabled and run upload is enabled, archive publishes under:
|
||||
|
||||
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- run prefix: `{session_prefix}/runs/{run_id}/`
|
||||
|
||||
Archive uploads:
|
||||
- run record files from run root (excluding `audio/`).
|
||||
- promoted files from explicit `archive.promote_artifacts` rules.
|
||||
- mutable session locks from helper commands live at `{session_prefix}/locks.yml`.
|
||||
|
||||
Publish order:
|
||||
1. upload `current/manifest.json`
|
||||
2. upload `current/run_id.txt` last
|
||||
|
||||
`current/run_id.txt` is the remote commit marker.
|
||||
|
||||
Archive promotion is explicit and source-based:
|
||||
- Narratio does not auto-promote all generated analyze artifacts.
|
||||
- each rule resolves `source` through the artifact resolver/catalog model, then uploads to `dest`.
|
||||
- missing required promotion sources fail archive stage.
|
||||
- missing optional promotion sources are skipped.
|
||||
- invalid resolved artifacts fail archive stage.
|
||||
- `archive.locks` skips top-level promotion overwrites for locked sources while run-local uploads still publish.
|
||||
- remote locks from `{session_prefix}/locks.yml` are merged with static `archive.locks`; static locks win on duplicate sources.
|
||||
- locked required promotions are treated as intentional successful skips and are recorded in archive metadata.
|
||||
|
||||
Lock helper behavior:
|
||||
- `narratio session locks <id>` lists effective static and remote locks.
|
||||
- `narratio session locks add <id> <source> --reason <text>` writes a remote lock.
|
||||
- `narratio session locks add <id> <source> --force --reason <text>` updates an existing remote lock reason.
|
||||
- `narratio session locks remove <id> <source>` removes only a remote lock.
|
||||
- `locks remove` cannot remove static pipeline locks.
|
||||
- remote lock writes check whether the lock store exists, but are not compare-and-swap atomic.
|
||||
|
||||
## Resume, retry, restore, and safe rerun behavior
|
||||
|
||||
Default skip:
|
||||
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set.
|
||||
|
||||
Resume:
|
||||
- `resume` starts at first non-succeeded stage.
|
||||
- `resume --force` runs full stage order.
|
||||
|
||||
Restore conflict policy:
|
||||
- restore classifies local differences as conflicts.
|
||||
- without `--force`, restore fails when conflicts exist.
|
||||
- with `--force`, conflicting local files are overwritten by remote archive files.
|
||||
|
||||
Forced reruns:
|
||||
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
|
||||
- ordinary `--force` does not override archive locks.
|
||||
|
||||
Safe rerun pattern:
|
||||
1. rerun the changed stage with `--force`.
|
||||
2. run `resume` to rebuild downstream stages.
|
||||
|
||||
## Cleanup behavior
|
||||
|
||||
Automatic post-archive cleanup is considered only when archive stage executed and succeeded.
|
||||
|
||||
Automatic cleanup toggles:
|
||||
- `pipeline.spool.delete_audio_after_archive=true` deletes run-scoped spool audio.
|
||||
- `pipeline.workspace.cleanup_after_archive=true` deletes run-scoped local run directory.
|
||||
|
||||
Manual cleanup:
|
||||
- `narratio clean <id>` deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`.
|
||||
- `narratio clean --all` deletes all local session work under `{workspace.root}/work` and all spool children under `{spool.root}`.
|
||||
- `--dry-run` prints targets without deleting.
|
||||
- `--clear-cache` also removes matching S3 audio cache files. Without it, cache is preserved.
|
||||
|
||||
The S3 audio cache under `pipeline.cache.root` is durable input cache state, not workspace or spool state. Automatic cleanup and default manual cleanup do not delete it.
|
||||
|
||||
Cleanup eligibility gates:
|
||||
- archive enabled
|
||||
- archive run upload enabled
|
||||
- run record upload completed
|
||||
- current pointer write completed (`current/run_id.txt` written)
|
||||
|
||||
No cleanup for failed/incomplete/unarchived/archive-skipped runs.
|
||||
|
||||
## Failure and recovery playbooks
|
||||
|
||||
After run failure, Narratio keeps:
|
||||
- session manifest
|
||||
- run manifest
|
||||
- run-local artifacts/logs/config/reports
|
||||
|
||||
Failed or incomplete runs remain local-only.
|
||||
|
||||
After restore failure:
|
||||
- already-installed restore files remain in place.
|
||||
- restore does not roll back prior successful installs.
|
||||
- existing local manifest is preserved if restored manifest validation/install fails.
|
||||
|
||||
Recommended recovery:
|
||||
|
||||
1. inspect state:
|
||||
4. Inspect status:
|
||||
|
||||
```bash
|
||||
narratio session status 2026-04-04
|
||||
```
|
||||
|
||||
This reports local manifest state, committed remote current state, expected remote transcript/artifact availability, and archive locks.
|
||||
## Publish Workflow
|
||||
|
||||
2. for restore-specific checks, run:
|
||||
Publish is the stage that commits remote current state.
|
||||
|
||||
```bash
|
||||
narratio publish 2026-04-04
|
||||
```
|
||||
|
||||
Equivalent command:
|
||||
|
||||
```bash
|
||||
narratio run-stage publish 2026-04-04 --force
|
||||
```
|
||||
|
||||
Publish uploads:
|
||||
|
||||
- run history files under `{session_prefix}/runs/{run_id}/` (excluding `audio/`)
|
||||
- configured published outputs from `pipeline.publish.outputs`
|
||||
- `previous/**` cache files when present
|
||||
- `current/manifest.json`
|
||||
- `current/run_id.txt` last
|
||||
|
||||
`current/run_id.txt` is the remote commit marker.
|
||||
|
||||
## Published Outputs and Locks
|
||||
|
||||
Published output behavior:
|
||||
|
||||
- outputs are source-based rules in `pipeline.publish.outputs`.
|
||||
- required missing unlocked sources fail publish.
|
||||
- optional missing unlocked sources are skipped.
|
||||
- selected artifacts (`--artifacts`) only filter configured `narratio.artifact.<key>` output rules.
|
||||
- built-in transcript and bounds output rules are not filtered by `--artifacts`.
|
||||
|
||||
Lock behavior:
|
||||
|
||||
- static locks: `pipeline.publish.locks`.
|
||||
- mutable locks: `{session_prefix}/locks.yml`.
|
||||
- effective lock set is static + mutable; static wins on duplicate sources.
|
||||
- locked outputs are intentional skips and do not fail publish.
|
||||
- lock commands mutate only remote mutable locks.
|
||||
|
||||
## Restore Workflow
|
||||
|
||||
Use restore when local durable session state is missing/stale and committed remote current state is authoritative.
|
||||
|
||||
Preview:
|
||||
|
||||
```bash
|
||||
narratio session restore 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
3. fix root cause (config/input/credentials/storage/service availability).
|
||||
4. continue with `resume`, or targeted `run-stage <stage> <id> --force` followed by `resume`.
|
||||
Apply:
|
||||
|
||||
## Restore report
|
||||
```bash
|
||||
narratio session restore 2026-04-04
|
||||
```
|
||||
|
||||
Non-dry-run restore writes a durable report at:
|
||||
- `reports/restore-latest.json`
|
||||
Default restore scope:
|
||||
|
||||
Report content includes:
|
||||
- identity (`campaign`, `session_id`, `run_id`)
|
||||
- mode flags (`dry_run`, `force`, `include_audio`)
|
||||
- plan counts and execution counts
|
||||
- per-action status
|
||||
- `manifest.json`
|
||||
- `transcripts/**`
|
||||
- `artifacts/**`
|
||||
- `previous/**` when required by configured previous-session artifact inputs
|
||||
|
||||
Dry-run does not write restore report files.
|
||||
Optional:
|
||||
|
||||
## Operational caveats
|
||||
- add `--include-audio` to restore `audio/**`.
|
||||
|
||||
- `session status <session_id>` uses normal config/session loading, including remote session fallback.
|
||||
- `session status <session_id>` includes the same promoted remote output availability view as `session artifacts <session_id> --remote` when storage is configured.
|
||||
- local and S3 audio input modes are mutually exclusive.
|
||||
- archive publish requires upstream stages through `analyze` to be `succeeded`.
|
||||
- required configured artifact promotions for unselected `--artifacts` keys are skipped intentionally; selected required promotions still fail if their files are missing.
|
||||
- restore requires configured remote object storage and committed remote current state.
|
||||
Restore reads committed current state only (`current/run_id.txt`, `current/manifest.json`).
|
||||
|
||||
## Workspace and State Layout
|
||||
|
||||
Session root:
|
||||
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/`
|
||||
|
||||
Durable session state:
|
||||
|
||||
- `manifest.json`
|
||||
- `inputs/**`
|
||||
- `audio/**`
|
||||
- `transcripts/**`
|
||||
- `artifacts/**`
|
||||
- `previous/**`
|
||||
- `reports/**`
|
||||
- `logs/**`
|
||||
- `config/**`
|
||||
- `runs/**`
|
||||
|
||||
Run-local stage layout:
|
||||
|
||||
- `runs/{run_id}/{stage}/outputs|logs|reports|config|scratch`
|
||||
|
||||
Stages typically write run-local outputs first, then materialize canonical session outputs on success.
|
||||
|
||||
## Resume and Force Rules
|
||||
|
||||
- `run` and `run-stage` skip succeeded stages unless `--force` is set.
|
||||
- `resume` starts at the first non-succeeded stage.
|
||||
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
|
||||
- `--force` does not bypass publish locks.
|
||||
|
||||
## Cleanup
|
||||
|
||||
Automatic post-publish cleanup is considered only when publish executes successfully and commits current state.
|
||||
|
||||
Config toggles:
|
||||
|
||||
- `pipeline.spool.delete_audio_after_publish=true`
|
||||
- `pipeline.workspace.cleanup_after_publish=true`
|
||||
|
||||
Manual cleanup:
|
||||
|
||||
```bash
|
||||
narratio clean 2026-04-04
|
||||
narratio clean --all
|
||||
```
|
||||
|
||||
Cache is preserved by default. Use `--clear-cache` to remove matching S3 audio cache entries.
|
||||
|
||||
## Failure and Recovery
|
||||
|
||||
After stage failure, Narratio keeps manifests and run-local files for inspection.
|
||||
|
||||
Standard recovery flow:
|
||||
|
||||
1. inspect status:
|
||||
|
||||
```bash
|
||||
narratio session status 2026-04-04
|
||||
```
|
||||
|
||||
2. if needed, inspect restore plan:
|
||||
|
||||
```bash
|
||||
narratio session restore 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
3. fix root cause.
|
||||
4. continue with `resume`, or rerun a stage with `--force` then `resume`.
|
||||
|
||||
## Operational Caveats
|
||||
|
||||
- local and S3 audio modes are mutually exclusive.
|
||||
- publish requires prerequisite stages through `analyze` to be `succeeded`.
|
||||
- restore requires configured object storage and committed current state.
|
||||
- `session status` and `session artifacts --remote` both report remote published-output availability when storage is configured.
|
||||
|
||||
Reference in New Issue
Block a user