eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Documentation update for the restore subcommand

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-19 22:32:55 -05:00
parent 648001a8fe
commit 5bc8e8683f
7 changed files with 333 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
docs/cli.md
View File

@@ -6,7 +6,7 @@
narratio run --session-id 2026-04-04
```
This command uses default config discovery for `pipeline.yml` and `session.yml`; both files must be discoverable unless you pass explicit `--config` and `--session` paths.
This command uses default discovery for `pipeline.yml` and `session.yml`; both files must be discoverable unless you pass explicit `--config` and `--session` paths.
## Command Overview
@@ -17,6 +17,7 @@ Implemented commands:
- `resume`: continue from first non-succeeded stage unless forced.
- `status`: read and print stage statuses from an existing manifest.
- `run-stage`: execute exactly one stage.
- `restore`: restore durable local session state from the committed remote archive state.
Unknown commands print usage and exit non-zero.
@@ -68,6 +69,15 @@ Valid stage names:
- `archive`
- `notify`
### `restore`
- `--config <path>`
- `--session <path>`
- `--session-id <value>`
- `--dry-run`: plan restore actions without writing local files.
- `--force`: overwrite local conflicting files with remote archive files.
- `--include-audio`: include durable archived `audio/**` files in restore scope.
### `status`
- `--manifest <path>`: required manifest path.
@@ -178,6 +188,39 @@ Common failure cases:
- unknown stage name.
- using `--artifacts` with any non-`analyze` stage.
### `restore`
Purpose:
- Restore durable session state (`manifest.json`, `transcripts/**`, `artifacts/**`, and optional `audio/**`) from the committed remote archive current state.
Syntax:
```bash
narratio restore [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--dry-run] [--force] [--include-audio]
```
Success output (dry-run):
- `Restore plan for <campaign>/<session_id>`
- `Remote run: <run_id>`
- `Would download: <n>`
- `Would skip unchanged: <n>`
- `Conflicts: <n>`
Success output (non-dry-run):
- `Restored session archive for <campaign>/<session_id>`
- `Remote run: <run_id>`
- `Downloaded: <n>`
- `Skipped unchanged: <n>`
- `Conflicts: <n>`
Common failure cases:
- storage backend is not configured.
- remote `current/run_id.txt` missing/empty.
- remote `current/manifest.json` missing or invalid.
- remote manifest session/campaign mismatch.
- local conflicts without `--force`.
- session lock conflict.
## Common Workflows
Default-discovery run:
@@ -204,6 +247,19 @@ Run only analyze stage with selected artifacts:
narratio run-stage --session-id 2026-04-04 --artifacts player_handout analyze
```
Preview restore actions without writes:
```bash
narratio restore --session-id 2026-04-04 --dry-run
```
Restore and then force analyze:
```bash
narratio restore --session-id 2026-04-04
narratio run-stage --session-id 2026-04-04 --force analyze
```
## Diagnostic / Recovery Commands
Inspect stage status:
@@ -219,4 +275,4 @@ Get manifest path from previous output:
- `--artifacts` filters which configured artifacts are executable when analyze runs.
- `--artifacts` does not imply `--force`.
- If analyze is already `succeeded` and `--force` is not set, runner-level skip still applies.
- if analyze is already `succeeded` and `--force` is not set, runner-level skip still applies.

25
docs/config.md
View File

@@ -13,6 +13,7 @@ These commands load and validate both files before running:
- `narratio plan`
- `narratio resume`
- `narratio run-stage`
- `narratio restore`
Behavior:
@@ -23,28 +24,28 @@ Behavior:
## 2. Config file discovery
Pipeline config lookup for `run`, `plan`, `resume`, and `run-stage`:
Pipeline config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`:
- If `--config <path>` is provided, that path is used.
- If omitted, Narratio searches in order:
- if `--config <path>` is provided, that path is used.
- if omitted, Narratio searches in order:
1. `/usr/local/etc/narratio/pipeline.yml`
2. `/etc/narratio/pipeline.yml`
- First existing file wins.
- first existing file wins.
## 3. Session file discovery and templating
Session config lookup for `run`, `plan`, `resume`, and `run-stage`:
Session config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`:
- If `--session <path>` is provided, that path is used.
- If omitted, Narratio searches in order:
- if `--session <path>` is provided, that path is used.
- if omitted, Narratio searches in order:
1. `./session.yml`
2. `/usr/local/etc/narratio/session.yml`
3. `/etc/narratio/session.yml`
- First existing file wins.
- first existing file wins.
Template behavior:
- Supported placeholders:
- supported placeholders:
- `{{session_id}}`
- `{{ session_id }}`
- `--session-id <value>` supplies the placeholder value.
@@ -133,6 +134,7 @@ Operational notes:
- archive promotion is explicit and source-based via `archive.promote_artifacts`.
- `source` is required; `dest` is optional and derived when omitted.
- Narratio does not auto-promote all generated analyze artifacts.
- `restore` reads the same config/session inputs and restore scope is bounded by committed archive current state.
## 7. Full pipeline reference
@@ -267,6 +269,11 @@ Archive promotion destination rules:
- configured sources derive from `pipeline.scriptorium.artifacts.<name>.output_path`;
- derivation failure is a config validation error.
Restore-related implications:
- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs).
- restore scope considers only committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, optional `audio/**`).
## 8. Full session reference
| Path | Type | Required | Default |

3
docs/internal/README.md
View File

@@ -4,7 +4,7 @@
Developers and LLM coding agents changing Narratio internals.
## Scope
Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, and adapter boundaries.
Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, adapter boundaries, and restore command behavior.
## Component Docs
- `adapters.md`: external adapter map, runtime wiring, and boundary ownership.
@@ -12,6 +12,7 @@ Implementation-accurate contracts for workspace/state, manifests, stages, artifa
- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics.
- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior.
- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants.
- `command-restore.md`: restore command discovery/planning/execution/reporting contract.
- `stage-prepare.md`: input materialization and provenance capture.
- `stage-transcribe.md`: WhisperX transcript generation.
- `stage-merge.md`: Seriatim normalization + merge.

86
docs/internal/command-restore.md Normal file
View File

@@ -0,0 +1,86 @@
# Internal: Command Restore
## Purpose
Define the implemented `narratio restore` command contract: committed remote-state discovery, deterministic planning, safe file installation, conflict policy, and restore reporting.
## Inputs and outputs
Inputs:
- CLI flags: `--config`, `--session`, `--session-id`, `--dry-run`, `--force`, `--include-audio`.
- Resolved/validated `pipeline.yml` and `session.yml`.
- Configured remote object store.
- Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`).
Outputs:
- Dry-run summary to stdout (plan + counts).
- Non-dry-run completion summary to stdout.
- Local durable session files restored under canonical session root.
- Non-dry-run restore report at `reports/restore-latest.json`.
## Boundaries
Owns:
- Restore command flag parsing and command wiring.
- Remote current-state discovery and identity validation.
- Restore plan construction and conflict classification.
- Restore execution for planned downloads.
- Restore report model and persistence.
Does not own:
- Stage execution orchestration (`run`, `resume`, `run-stage`).
- Archive publish behavior (owned by archive stage).
- Storage transport implementation details (owned by storage adapters).
## Config fields used
- Config/session discovery and templating fields consumed by all commands.
- `pipeline.workspace.root` (local restore target root).
- `pipeline.storage.*` (remote backend + archive identity derivation).
- `pipeline.storage.s3.*` identity components used by archive prefix helpers.
- `session.session_id`
- `session.campaign`
## External adapters used
- `storage.ObjectStore` for `Exists`, `List`, `Download`.
- `artifacts.Store` (`LocalStore`) for layout and session lock management.
- `manifest.LocalStore` for manifest decode/validation and identity checks.
## State and manifest behavior
- Restore is not a pipeline run and does not create a run manifest.
- Restore uses committed remote current state only:
- `current/run_id.txt` must exist and be non-empty.
- `current/manifest.json` must decode and match requested session/campaign.
- Non-dry-run writes restore files to canonical session paths.
- Manifest install behavior:
- validated before replacement.
- installed last among download actions.
- existing local manifest is preserved if restored manifest validation/install fails.
- Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`.
## Skip and resume behavior
- Restore does not participate in stage skip/resume decisions.
- Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state.
- Dry-run is read-only and returns plan output only.
## Failure behavior
- Fails when storage backend is unavailable or archive identity cannot be resolved.
- Fails when remote current pointer/manifest is missing or invalid.
- Fails when remote manifest identity mismatches requested campaign/session.
- Fails on local conflicts unless `--force` is set.
- Fails fast on session lock acquisition conflict for non-dry-run execution.
- On execution failure, previously installed files remain; no rollback is performed.
## Tests to inspect before changing
- `internal/app/restore_test.go`
- `internal/app/restore_discovery_test.go`
- `internal/app/restore_plan_test.go`
- `internal/app/restore_execution_test.go`
- `internal/app/restore_workflow_test.go`
- `internal/artifacts/archive_identity_test.go`
## Architectural invariants
- Restore relies on centralized archive identity/key helpers (`internal/artifacts`) rather than ad hoc key building.
- `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files.
- Local path mapping is traversal-safe and constrained to session root.
- Restore scope is deterministic and path-classified:
- include `manifest.json`, `transcripts/**`, `artifacts/**`
- include `audio/**` only with `--include-audio`
- exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`
- Command remains standalone; no implicit `run --restore` behavior.

74
docs/operations.md
View File

@@ -6,7 +6,7 @@ For field-level configuration, see [docs/config.md](./config.md). For full comma
## Normal workflow (S3-first path)
1. Upload session `.flac` files to object storage under the session audio prefix.
1. Upload session `.flac` files to object storage under the configured session audio prefix.
2. Run Narratio:
```bash
@@ -21,6 +21,37 @@ Notes:
- default config/session discovery applies unless `--config` and `--session` are passed.
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
## Restore workflow
Use restore when local durable session state is missing or stale and archive current state is authoritative.
Dry-run (no local writes):
```bash
narratio restore --session-id 2026-04-04 --dry-run
```
Execution:
```bash
narratio restore --session-id 2026-04-04
```
Post-restore analyze rerun pattern:
```bash
narratio run-stage --session-id 2026-04-04 --force analyze
```
Restore source-of-truth:
- remote commit marker: `current/run_id.txt`
- remote current manifest: `current/manifest.json`
Restore default scope:
- includes `manifest.json`, `transcripts/**`, `artifacts/**`
- includes `audio/**` only with `--include-audio`
- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source)
## Local filesystem layout and state artifacts
Session root:
@@ -29,7 +60,7 @@ Session root:
Primary state:
- `manifest.json`: session-level stage state.
- `runs/{run_id}/manifest.json`: invocation-level state.
- `.lock`: session lock while a run is active.
- `.lock`: session lock while a modifying command is active.
Canonical session directories:
- `inputs/`
@@ -48,6 +79,7 @@ Run-local stage directories:
Behavior:
- directory creation is idempotent.
- stage outputs are generally generated run-local first, then promoted to canonical paths on success.
- restore installs downloaded files to canonical session paths and does not recreate historical run sandboxes.
## Analyze artifact execution lifecycle
@@ -91,7 +123,7 @@ Archive promotion is explicit and source-based:
- missing optional promotion sources are skipped.
- invalid resolved artifacts fail archive stage.
## Resume, retry, and safe rerun behavior
## Resume, retry, restore, and safe rerun behavior
Default skip:
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set.
@@ -100,6 +132,11 @@ Resume:
- `resume` starts at first non-succeeded stage.
- `resume --force` runs full stage order.
Restore conflict policy:
- restore classifies local differences as conflicts.
- without `--force`, restore fails when conflicts exist.
- with `--force`, conflicting local files are overwritten by remote archive files.
Forced reruns:
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
@@ -125,13 +162,18 @@ No cleanup for failed/incomplete/unarchived/archive-skipped runs.
## Failure and recovery playbooks
After failure, Narratio keeps:
After run failure, Narratio keeps:
- session manifest
- run manifest
- run-local artifacts/logs/config/reports
Failed or incomplete runs remain local-only.
After restore failure:
- already-installed restore files remain in place.
- restore does not roll back prior successful installs.
- existing local manifest is preserved if restored manifest validation/install fails.
Recommended recovery:
1. inspect state:
@@ -140,8 +182,27 @@ Recommended recovery:
narratio status --manifest <manifest-path>
```
2. fix root cause (config/input/credentials/service availability).
3. continue with `resume`, or targeted `run-stage --force` followed by `resume`.
2. for restore-specific checks, run:
```bash
narratio restore --session-id 2026-04-04 --dry-run
```
3. fix root cause (config/input/credentials/storage/service availability).
4. continue with `resume`, or targeted `run-stage --force` followed by `resume`.
## Restore report
Non-dry-run restore writes a durable report at:
- `reports/restore-latest.json`
Report content includes:
- identity (`campaign`, `session_id`, `run_id`)
- mode flags (`dry_run`, `force`, `include_audio`)
- plan counts and execution counts
- per-action status
Dry-run does not write restore report files.
## Operational caveats
@@ -149,3 +210,4 @@ narratio status --manifest <manifest-path>
- local and S3 audio input modes are mutually exclusive.
- archive publish requires upstream stages through `analyze` to be `succeeded`.
- required promotion rules can fail when selected analyze artifacts did not generate a required file path.
- restore requires configured remote object storage and committed remote current state.

18
docs/roadmap/restore.md
View File

@@ -2,7 +2,7 @@
## Status
Planned. This document is implementation guidance for 5.3-Codex.
Implemented through Step 8. This document remains as roadmap and design history for the restore feature, and as the home for future restore-related ideas (for example `run --restore`).
## Summary
@@ -546,7 +546,7 @@ Do not document future `narratio run --restore` behavior outside `docs/roadmap/`
## Implementation phases
### Phase 1: Audit existing archive and path helpers
### Phase 1: Audit existing archive and path helpers (completed)
Before coding behavior, inspect:
@@ -574,7 +574,7 @@ Deliverable:
- small code comments or internal helper selection;
- no large behavior change yet unless required by tests.
### Phase 2: Add CLI surface and command wiring
### Phase 2: Add CLI surface and command wiring (completed)
Add `narratio restore` command parsing.
@@ -599,7 +599,7 @@ Deliverable:
- command returns a clear “not yet implemented” or calls an empty planner if phased commits are desired;
- CLI tests pass.
### Phase 3: Implement remote current-state discovery
### Phase 3: Implement remote current-state discovery (completed)
Add restore code that:
@@ -615,7 +615,7 @@ Deliverable:
- fake-storage tests for current-state discovery;
- no local file writes beyond temporary files.
### Phase 4: Implement restore planning
### Phase 4: Implement restore planning (completed)
Build deterministic restore plans for default scope and `--include-audio`.
@@ -627,7 +627,7 @@ Deliverable:
- dry-run output works;
- no real file overwrite yet except temp comparisons as needed.
### Phase 5: Implement restore execution
### Phase 5: Implement restore execution (completed)
Execute the plan safely:
@@ -644,7 +644,7 @@ Deliverable:
- restore works end-to-end against fake storage;
- failures are clear and do not corrupt existing local manifest.
### Phase 6: Add restore report and operator summary
### Phase 6: Add restore report and operator summary (completed)
Add concise stdout summary and optional JSON restore report if consistent with project diagnostics.
@@ -654,7 +654,7 @@ Deliverable:
- durable diagnostic report if implemented;
- no content leakage.
### Phase 7: Workflow integration test
### Phase 7: Workflow integration test (completed)
Add a test for restoring a previous session and then forcing `analyze`.
@@ -664,7 +664,7 @@ Deliverable:
- no upstream stages rerun;
- no reliance on live subprocesses or S3.
### Phase 8: Documentation update
### Phase 8: Documentation update (completed)
Once implemented, update current-behavior docs and internal command docs.

97
docs/troubleshooting.md
View File

@@ -6,7 +6,7 @@ Canonical operator troubleshooting guide for recurring implemented Narratio fail
## Config file discovery failure
Symptom:
- `run`, `plan`, `resume`, or `run-stage` fails with config/session not found.
- `run`, `plan`, `resume`, `run-stage`, or `restore` fails with config/session not found.
Likely Cause:
- `pipeline.yml` or `session.yml` is missing from discovery paths.
@@ -192,7 +192,7 @@ Links:
## Session lock conflict (`.lock`)
Symptom:
- run fails with lock conflict for session workdir.
- `run`, `resume`, `run-stage`, or `restore` fails with lock conflict for session workdir.
Likely Cause:
- another Narratio process is running same session.
@@ -207,13 +207,104 @@ ps aux | grep narratio
```
Safe Fix:
- wait for active run to finish.
- wait for active process to finish.
- if no process is active, remove only stale session `.lock` file.
Links:
- [docs/operations.md](./operations.md)
- [docs/internal/workspace.md](./internal/workspace.md)
## Restore remote current pointer or manifest missing
Symptom:
- `restore` fails with remote current pointer or current manifest errors.
Likely Cause:
- `current/run_id.txt` was never published.
- `current/manifest.json` is missing for the session prefix.
- archive commit did not complete.
Diagnostics:
```bash
narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
```
Safe Fix:
- verify archive stage succeeded for the target session.
- rerun/archive from a healthy source workspace so current pointers are published.
Links:
- [docs/operations.md](./operations.md)
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
## Restore manifest identity mismatch
Symptom:
- `restore` fails because remote manifest session or campaign does not match requested values.
Likely Cause:
- wrong `--session-id` or wrong session config selected.
- archive prefix points to a different campaign/session.
Diagnostics:
```bash
narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
```
Safe Fix:
- use the correct session config and `--session-id`.
- verify campaign/session identity in local config before restore.
Links:
- [docs/config.md](./config.md)
- [docs/operations.md](./operations.md)
## Restore conflict without `--force`
Symptom:
- `restore` fails with `restore conflict` and conflict counts.
Likely Cause:
- local durable file differs from remote file for one or more planned restore paths.
Diagnostics:
```bash
narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
```
Safe Fix:
- review planned conflicts.
- rerun with `--force` only when remote state should overwrite local state.
Links:
- [docs/cli.md](./cli.md)
- [docs/operations.md](./operations.md)
## Restore report expectations
Symptom:
- operator expects restore report file but does not find one.
Likely Cause:
- restore was executed in `--dry-run` mode.
- restore failed before report persistence path (for example lock acquisition failure).
Diagnostics:
```bash
ls -l {workspace.root}/work/{campaign}/{session_id}/reports/restore-latest.json
```
Safe Fix:
- run non-dry-run restore for durable report output.
- resolve lock or early preflight failures and retry.
Links:
- [docs/operations.md](./operations.md)
## Secrets env-dir or credential-env failure
Symptom: