eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Update documentation for the new analyze stage and artifact registry

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-19 19:42:28 -05:00
parent ebb21b9201
commit 574b1cde6c
11 changed files with 464 additions and 344 deletions
Download Patch File Download Diff File Expand all files Collapse all files

171
docs/operations.md
View File

@@ -6,8 +6,7 @@ For field-level configuration, see [docs/config.md](./config.md). For full comma
## Normal workflow (S3-first path)
1. Upload session `.flac` files to the session audio prefix in object storage:
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/{audio_s3.prefix}`
1. Upload session `.flac` files to object storage under the session audio prefix.
2. Run Narratio:
```bash
@@ -15,28 +14,24 @@ narratio run --session-id 2026-04-04
```
3. Read success output:
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
- `manifest=<path>` is the local session manifest path to use with `status`.
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
- use `manifest=<path>` with `status` for inspection.
Notes:
- This command relies on discoverable `pipeline.yml` and `session.yml` unless `--config` and `--session` are passed explicitly.
- For S3 audio input, `session.inputs.audio_s3.prefix` must be configured and audio files must already exist remotely.
- default config/session discovery applies unless `--config` and `--session` are passed.
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
## Local filesystem layout and state artifacts
Session root:
- `{workspace.root}/work/{campaign}/{session_id}/`
Primary state:
- `manifest.json`: session-level manifest (authoritative local stage state).
- `runs/{run_id}/manifest.json`: run-level manifest for one invocation.
- `.lock`: session lock file while a run is active.
- `manifest.json`: session-level stage state.
- `runs/{run_id}/manifest.json`: invocation-level state.
- `.lock`: session lock while a run is active.
Canonical session directories:
- `inputs/`
- `audio/`
- `transcripts/`
@@ -48,123 +43,107 @@ Canonical session directories:
- `runs/`
Run-local stage directories:
- `runs/{run_id}/{stage}/` with stage-local `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`.
- `runs/{run_id}/{stage}/`
- Stage runtime files are written under deterministic run-local subdirectories such as:
- `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`
Behavior:
- directory creation is idempotent.
- stage outputs are generally generated run-local first, then promoted to canonical paths on success.
Behavior notes:
## Analyze artifact execution lifecycle
- Layout creation is idempotent.
- Durable outputs are promoted to canonical session paths after stage success.
- Run-local artifacts remain in `runs/{run_id}/...` unless configured post-archive cleanup removes that run scope.
Analyze executes configured artifacts from `pipeline.scriptorium.artifacts`.
Execution model:
- executable set = enabled artifacts, filtered by `--artifacts` when provided.
- artifact-to-artifact dependencies are declared via `depends_on`.
- selected artifacts run in deterministic dependency order.
- after each successful artifact run, output is promoted to configured canonical `output_path`.
Configured artifact source reuse:
- a non-executable configured artifact can satisfy inputs if its configured output file already exists and is valid.
- reused configured artifact provenance is `filesystem.disabled_artifact_output`.
`--artifacts` behavior:
- accepted on `run`, `resume`, and `run-stage analyze`.
- filters analyze execution only; does not force stage rerun.
## Remote archive layout and publish contract
When archive is enabled and run upload is enabled, archive publishes to object storage under:
When archive is enabled and run upload is enabled, archive publishes under:
- Session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
- Run prefix: `{session_prefix}/runs/{run_id}/`
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
- run prefix: `{session_prefix}/runs/{run_id}/`
Archive uploads:
- run record files from run root (excluding `audio/`).
- promoted files from explicit `archive.promote_artifacts` rules.
- Run record files from run root (including stage subtrees and run manifest), excluding local `audio/`.
- Promoted artifacts from `archive.promote_artifacts` to session-level keys.
Publish order:
1. upload `current/manifest.json`
2. upload `current/run_id.txt` last
Publish order (commit contract):
`current/run_id.txt` is the remote commit marker.
1. Upload `current/manifest.json`
2. Upload `current/run_id.txt` last
Meaning of `current/run_id.txt`:
- It is the effective remote commit marker for published session state.
- It is written only after required run uploads and required promotions succeed.
Archive promotion is explicit and path-based:
- Narratio does not auto-promote all generated analyze artifacts.
- missing required promotion sources fail archive stage.
- missing optional promotion sources are skipped.
## Resume, retry, and safe rerun behavior
Default skip behavior:
Default skip:
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set.
- `run` and `run-stage` skip stages already marked `succeeded` unless `--force` is set.
Resume:
- `resume` starts at first non-succeeded stage.
- `resume --force` runs full stage order.
Resume behavior:
Forced reruns:
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
- `resume` starts at the first non-`succeeded` stage in canonical stage order.
- If all stages are `succeeded`, `resume` prints that no stages remain.
- `resume --force` runs full stage order rather than starting at first non-succeeded.
Forced rerun behavior:
- Successful forced rerun of an upstream stage marks downstream previously `succeeded` stages as `stale`.
- `stale` stages are not treated as complete and are eligible to run in subsequent commands.
Targeted rerun with one stage:
```bash
narratio run-stage --force <stage>
```
Valid stage names:
- `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`, `archive`, `notify`
Safe operator pattern:
1. Force-rerun the stage that changed.
2. Run `resume` to rebuild downstream stages in order.
Safe rerun pattern:
1. rerun the changed stage with `--force`.
2. run `resume` to rebuild downstream stages.
## Cleanup behavior
Cleanup is considered only after run execution completes and only when archive stage both executed and succeeded.
Cleanup is considered only when archive stage executed and succeeded.
Configured cleanup toggles:
Cleanup toggles:
- `pipeline.spool.delete_audio_after_archive=true` deletes run-scoped spool audio.
- `pipeline.workspace.cleanup_after_archive=true` deletes run-scoped local run directory.
- `pipeline.spool.delete_audio_after_archive=true`
- deletes only run-scoped spool audio directory: `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
- `pipeline.workspace.cleanup_after_archive=true`
- deletes only run-scoped local run directory: `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/`
Cleanup eligibility gates:
- archive enabled
- archive run upload enabled
- run record upload completed
- current pointer write completed (`current/run_id.txt` written)
Eligibility gates for cleanup:
- archive is enabled
- archive run upload is enabled
- archive metadata indicates run record upload happened
- archive metadata indicates `current` pointer write completed (`current/run_id.txt` written)
Cleanup does not run for:
- failed runs
- incomplete runs
- unarchived runs
- archive-skipped runs (`archive.enabled=false` or `archive.upload_run=false`)
No cleanup for failed/incomplete/unarchived/archive-skipped runs.
## Failure and recovery playbooks
What remains after failure:
After failure, Narratio keeps:
- session manifest
- run manifest
- run-local artifacts/logs/config/reports
- Session manifest remains on disk.
- Run manifest remains under `runs/{run_id}/manifest.json`.
- Run-local stage artifacts/logs/config/reports remain under `runs/{run_id}/...`.
- Failed/incomplete runs remain local-only.
- Remote current pointer is not committed if archive prerequisite or pointer-write steps fail.
Failed or incomplete runs remain local-only.
Recommended recovery flow:
Recommended recovery:
1. Inspect current state:
1. inspect state:
```bash
narratio status --manifest <manifest-path-from-run-output>
narratio status --manifest <manifest-path>
```
2. Fix the root cause (config, input, credentials, adapter availability, etc.).
3. Continue with:
- `narratio resume --session-id <id>` for ordered continuation, or
- `narratio run-stage --force <stage>` for targeted correction, then `resume`.
2. fix root cause (config/input/credentials/service availability).
3. continue with `resume`, or targeted `run-stage --force` followed by `resume`.
## Operational caveats
- `status` requires an explicit manifest path; there is no direct session-id lookup command.
- S3 audio mode and local audio mode are mutually exclusive in session config.
- Archive verifies stage prerequisites (`prepare` through `analyze`) before publishing.
- By default, archive does not upload local `audio/` into run history.
- Unknown CLI commands fail and print usage.
- `status` requires explicit `--manifest`; there is no session-id lookup command.
- local and S3 audio input modes are mutually exclusive.
- archive publish requires upstream stages through `analyze` to be `succeeded`.
- required promotion rules can fail when selected analyze artifacts did not generate a required file path.