narratio/docs/roadmap/restore.md

Roadmap: narratio restore Subcommand

Status

Planned. This document is implementation guidance for 5.3-Codex.

Summary

Add a new narratio restore subcommand that hydrates a local session workspace from the current committed remote archive state.

The primary operator workflow is:

narratio restore --session-id 2026-04-04
narratio run-stage --force analyze

This should allow a new machine with no local workspace state to restore the durable session manifest, transcripts, and generated artifacts from S3, then generate new Scriptorium artifacts without re-running transcription, merge, normalize, polish, or trim.

This is intentionally a separate command. Do not fold this behavior into the prepare stage. The existing prepare stage should remain focused on materializing configured local/S3 inputs for a pipeline run.

Goals

• Add a first-class narratio restore command.
• Restore the current committed remote session state into the canonical local session workspace.
• Use the existing object storage adapter boundary.
• Preserve archive commit semantics: only restore from a remote state that has a valid current commit marker.
• Restore durable session-level outputs needed for downstream stages, especially analyze.
• Provide safe conflict behavior by default.
• Support --dry-run, --force, and --include-audio.
• Keep the implementation explicit, testable, and narrow.

Non-goals

• Do not make restore a pipeline stage.
• Do not change the prepare stage behavior as part of this work.
• Do not add implicit restore behavior to narratio run in this implementation.
• Do not restore historical run-local sandboxes by default.
• Do not implement a generic remote synchronization engine.
• Do not implement bidirectional sync.
• Do not delete local files merely because they are absent remotely.
• Do not merge remote and local manifests in the first implementation.
• Do not require live S3 for the ordinary unit test suite.

Future work explicitly out of scope

A future change may add:

narratio run --restore

That future flag should run narratio restore before starting the normal pipeline. Mention this as future work in roadmap/docs if useful, but do not implement it now.

Existing architecture to preserve

prepare remains input materialization

The prepare stage currently materializes required session inputs into canonical local workspace paths and records input provenance. It owns local copying/materialization of config and audio inputs, including S3 audio download when session.inputs.audio_s3.prefix is configured. It does not own transcript generation/processing or archive publish behavior.

restore should not be implemented by expanding prepare. It should be an app-level command that reuses shared helpers where appropriate.

Workspace model

The local durable session workspace is campaign-aware:

{workspace.root}/work/{campaign}/{session_id}/

It contains durable session paths such as:

manifest.json
inputs/
audio/
transcripts/
artifacts/
reports/
logs/
config/
current/
runs/

Run-local sandboxes live below:

runs/{run_id}/

Restore should target durable session-level paths, not old run-local stage sandboxes.

Storage boundary

The storage adapter owns object-store primitives only: List, Download, Upload, and Exists.

The storage adapter must not infer root prefixes, campaign names, session IDs, run IDs, or archive layout. Restore code must construct full bucket-relative keys before calling storage.

Archive commit boundary

A remote run is current only after the archive stage has uploaded the run record, promoted outputs, current/manifest.json, and finally current/run_id.txt.

current/run_id.txt is the final remote commit marker and must be written last.

Restore must not treat incomplete, skipped, failed, or uncommitted archive attempts as current remote state.

User-facing command

Add:

narratio restore [flags]

The command should use the same configuration/session discovery conventions as run, plan, resume, and run-stage where practical:

narratio restore --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-04-04

Required effective inputs:

• resolved pipeline config;
• resolved session config;
• session.campaign;
• session.session_id;
• configured remote storage backend.

Supported flags:

--config <path>       Existing pipeline config path behavior.
--session <path>      Existing session config path behavior.
--session-id <value>  Existing session template behavior.
--dry-run             Plan restore actions without writing local files.
--force               Overwrite conflicting local files with remote files.
--include-audio       Include archived session-level audio files.

Do not add --restore to run in this implementation.

Default restore scope

By default, restore:

1. Validates and reads the current remote commit marker.
2. Downloads the current remote manifest into the local session manifest path.
3. Downloads durable transcript files.
4. Downloads durable generated artifact files.

Default included remote/local durable paths:

manifest.json                 from remote current manifest
transcripts/**
artifacts/**

Default excluded paths:

audio/**                      unless --include-audio is passed
runs/**                       always excluded for this implementation
logs/**                       excluded for this implementation
reports/**                    excluded for this implementation unless needed for current manifest validation
config/**                     excluded for this implementation
inputs/**                     excluded for this implementation
current/**                    remote control metadata only; do not mirror blindly

If the existing archive implementation stores promoted files in a different remote layout, use the existing archive/path helpers and current archive semantics rather than inventing a parallel layout.

Remote state discovery

Implement restore around the current committed archive state.

Expected algorithm:

1. Resolve pipeline/session config.
2. Ensure storage is configured.
3. Ensure local workspace layout exists.
4. Acquire the session lock.
5. Build the remote session archive prefix using the same helpers/policy used by archive code.
6. Check for the remote current/run_id.txt commit marker.
7. Read the committed run ID.
8. Download current/manifest.json to a temporary file.
9. Validate that the manifest is parseable and belongs to the requested campaign/session.
10. Build a restore plan from the committed remote state.
11. Execute the restore plan unless --dry-run is set.
12. Emit a concise summary.

Important: current/run_id.txt is the commit marker. Do not restore from a remote session prefix merely because files exist under transcripts/ or artifacts/.

Restore planning

Create a planning layer before writing files.

A restore plan entry should include at least:

type RestoreAction struct {
    Kind        RestoreActionKind
    RemoteKey   string
    LocalPath   string
    Size        int64
    ETag        string
    ExistsLocal bool
    SameLocal   bool
    Conflict    bool
    Reason      string
}

Suggested action kinds:

download
skip_same
skip_missing_optional
conflict

The restore planner should be deterministic:

• sort remote objects by key;
• sort planned actions by local path or stable restore priority;
• write/report stable output for tests.

Conflict and overwrite policy

Default behavior should be safe.

For each planned file:

local absent:
  download

local present and same as remote:
  skip

local present and different:
  conflict; fail restore unless --force is set

--force:
  overwrite local conflicting files with remote versions

--dry-run:
  do not write any files; report what would happen

The first implementation may use size and checksum/hash comparison where available. If remote ETag cannot be treated as a content hash, compare by downloading to a temporary file and hashing locally before deciding whether a local file is the same. Prefer correctness over assuming provider-specific ETag semantics.

Do not delete local files that are not present remotely.

File writing and transactionality

Restore should avoid partial writes.

Implementation requirements:

• download each remote object to a temporary file under the session workspace or OS temp dir;
• validate downloaded content where possible before replacing local files;
• create parent directories as needed;
• atomically rename/copy into place only after successful download;
• do not overwrite local files unless --force is set;
• if a later file fails, preserve already-restored files but return a failure summary;
• never corrupt an existing local manifest on failed manifest download/parse.

Manifest restore is especially sensitive:

• download remote current/manifest.json to a temporary file;
• parse and validate it;
• if no local manifest exists, install it;
• if a local manifest exists and is equivalent, skip;
• if a local manifest exists and differs, fail unless --force is set;
• with --force, replace the local manifest with the remote manifest after validation;
• do not attempt a manifest merge in the initial implementation.

Manifest semantics

restore is not a pipeline run and should not mark stages as running/succeeded/failed.

The restored remote manifest becomes the local session manifest. That is what allows a subsequent command such as:

narratio run-stage --force analyze

to see existing upstream stage state and canonical durable outputs.

Do not create a new run manifest for restore.

It is acceptable to write a restore diagnostic report outside the manifest, for example:

reports/restore-latest.json

or a timestamped report, if that pattern fits the existing codebase. The report must not contain secrets.

Local workspace locking

restore should acquire the same session lock used by ordinary pipeline operations before modifying session workspace state.

If the lock is held, fail fast with the same lock-conflict behavior used elsewhere.

--dry-run may still acquire the lock for consistency, but it is acceptable to avoid the lock if the codebase already has a clear read-only command pattern. Prefer safety and simplicity.

Audio behavior

By default, do not restore audio.

If --include-audio is passed:

• restore archived durable session-level audio files only;
• do not use run-scoped spool paths;
• do not mutate or delete spool state;
• do not infer original session.inputs.audio_s3.prefix behavior;
• respect the same conflict/force/dry-run behavior used for transcripts/artifacts.

If the archive does not contain durable audio files, --include-audio should report that no archived audio was found rather than failing, unless the final implementation chooses to treat explicit audio restore as required. Prefer non-failure for absent archived audio unless tests or existing archive semantics suggest otherwise.

Remote object selection

Prefer using manifest/artifact metadata when it reliably identifies durable outputs.

Also support listing committed durable archive prefixes so restore can retrieve all top-level session artifacts that may not yet be fully represented in manifest metadata.

The implementation should inspect existing archive code before choosing the final object-selection method. Do not duplicate archive path construction.

Recommended selection priority:

1. Remote current manifest path.
2. Durable promoted transcript/artifact outputs recorded in the manifest or archive metadata, if available.
3. Objects under committed durable transcripts/ and artifacts/ archive prefixes.
4. Objects under durable audio/ only when --include-audio is passed.

Always exclude:

runs/**

for the first implementation.

Package and file organization

Expected areas to inspect and update:

cmd/narratio/
internal/app/
internal/adapters/storage/
internal/artifacts/
internal/manifest/
docs/
examples/

Suggested implementation shape:

internal/app/restore.go
internal/app/restore_test.go

internal/archive/restore/
  planner.go
  executor.go
  report.go
  keys.go
  *_test.go

The exact package name may vary. Use whatever best fits the existing repository, but keep these boundaries clear:

• internal/app owns CLI command handling, config/session loading, lock acquisition, and wiring.
• Restore planning/execution owns remote key discovery, conflict detection, downloads, and reporting.
• internal/adapters/storage remains a transport boundary only.
• Workspace/path helpers remain centralized; do not scatter string concatenation.

If the repository already has an internal/archive or archive-stage helper package, prefer extending that rather than creating a conflicting package layout.

CLI output

narratio restore should print a concise operator summary.

Example successful output:

Restored session archive for sample-campaign/2026-04-04
Remote run: 20260504T031500Z-a1b2c3
Downloaded: 4
Skipped unchanged: 2
Conflicts: 0

Example dry run:

Restore plan for sample-campaign/2026-04-04
Remote run: 20260504T031500Z-a1b2c3
Would download: transcripts/processed.json
Would download: transcripts/trimmed.json
Would skip unchanged: artifacts/session_recap.md

Example conflict:

restore conflict: local artifacts/session_recap.md differs from remote archive; rerun with --force to overwrite

Do not print transcript or artifact content.

Error behavior

Fail clearly when:

• storage backend is not configured;
• S3 bucket/config is missing or invalid;
• remote current commit marker is missing;
• remote current manifest is missing;
• remote manifest is invalid;
• remote manifest does not match requested campaign/session;
• local file differs from remote and --force is not set;
• a required remote object download fails;
• a local path would escape the session workspace;
• a remote key maps to an unsafe local path.

Skip or report non-fatal conditions when:

• optional audio restore finds no archived audio;
• an included prefix has no objects;
• a local file already matches the remote file.

Path safety

Every restored file must map to a safe path under the session root.

Validation rules:

• local restore paths must be relative to the session root;
• reject absolute paths;
• reject .. traversal;
• reject paths that escape through symlinks if the codebase has symlink-safe path checks;
• do not restore remote keys directly without mapping/classification;
• do not mirror arbitrary remote keys.

Testing plan

Add focused unit tests. Do not require live S3.

CLI tests

Add or update internal/app command tests for:

• narratio restore --help;
• restore accepts --config, --session, and --session-id;
• restore accepts --dry-run;
• restore accepts --force;
• restore accepts --include-audio;
• restore fails when storage is not configured;
• restore does not run pipeline stages.

Restore planner tests

Test:

• missing current/run_id.txt fails;
• missing current/manifest.json fails;
• invalid manifest fails;
• wrong campaign/session manifest fails;
• default scope includes manifest/transcripts/artifacts;
• default scope excludes audio/logs/reports/config/runs;
• --include-audio includes durable audio;
• run-local keys are excluded;
• keys are sorted deterministically;
• unsafe remote-to-local paths are rejected.

Conflict policy tests

Test:

• absent local file downloads;
• matching local file skips;
• differing local file conflicts by default;
• --force overwrites conflicts;
• --dry-run writes nothing;
• partial failure does not corrupt an existing local manifest.

Storage/fake tests

Use fake storage to simulate:

• object listing;
• object download;
• missing objects;
• download failures;
• metadata/ETag behavior.

Workspace/lock tests

Test:

• session layout is created before restore;
• session lock conflict fails;
• restored files land under the expected campaign/session workspace;
• no files are written outside the session root.

Follow-up command workflow test

Add at least one test that simulates:

narratio restore --session-id 2026-04-04
narratio run-stage --force analyze

The test does not need to run real Scriptorium. Use existing fake/stub behavior to verify that restored transcripts and manifest state are sufficient for analyze-stage input resolution.

Documentation updates when implemented

When the feature is implemented, update current-behavior docs:

docs/cli.md
docs/operations.md
docs/internal/storage.md or docs/internal/archive/restore.md

If the documentation set does not yet have an internal restore document, add one consistent with the existing internal-doc style:

docs/internal/command-restore.md

or:

docs/internal/archive-restore.md

Do not document future narratio run --restore behavior outside docs/roadmap/ until implemented.

Implementation phases

Phase 1: Audit existing archive and path helpers

Before coding behavior, inspect:

internal/app/
internal/stage/archive*
internal/adapters/storage/
internal/artifacts/
internal/manifest/
docs/internal/stage-archive.md, if present

Determine:

• exact remote archive key layout;
• how root prefix/campaign/session are modeled;
• how current commit marker keys are built;
• how current manifest is uploaded;
• where promoted outputs are uploaded;
• whether helper functions already exist for remote archive keys;
• whether local workspace path helpers can safely map restore destinations.

Deliverable:

• small code comments or internal helper selection;
• no large behavior change yet unless required by tests.

Phase 2: Add CLI surface and command wiring

Add narratio restore command parsing.

Wire flags:

--config
--session
--session-id
--dry-run
--force
--include-audio

Use the existing config/session load path where practical.

Deliverable:

• command exists;
• help output is sensible;
• command validates basic inputs;
• command returns a clear “not yet implemented” or calls an empty planner if phased commits are desired;
• CLI tests pass.

Phase 3: Implement remote current-state discovery

Add restore code that:

• creates an object store from resolved config;
• builds remote current marker key;
• reads current/run_id.txt;
• reads/downloads current/manifest.json;
• validates manifest identity;
• returns remote current-state metadata.

Deliverable:

• fake-storage tests for current-state discovery;
• no local file writes beyond temporary files.

Phase 4: Implement restore planning

Build deterministic restore plans for default scope and --include-audio.

Deliverable:

• plan lists manifest, transcript, artifact files;
• plan excludes run-local data;
• plan detects local same/conflict/missing states;
• dry-run output works;
• no real file overwrite yet except temp comparisons as needed.

Phase 5: Implement restore execution

Execute the plan safely:

• create directories;
• download to temporary files;
• validate content where practical;
• atomically install files;
• enforce default conflict failure;
• support --force;
• preserve existing manifest unless safe to replace.

Deliverable:

• restore works end-to-end against fake storage;
• failures are clear and do not corrupt existing local manifest.

Phase 6: Add restore report and operator summary

Add concise stdout summary and optional JSON restore report if consistent with project diagnostics.

Deliverable:

• user-friendly output;
• durable diagnostic report if implemented;
• no content leakage.

Phase 7: Workflow integration test

Add a test for restoring a previous session and then forcing analyze.

Deliverable:

• restored manifest/transcripts/artifacts are sufficient for analyze input resolution;
• no upstream stages rerun;
• no reliance on live subprocesses or S3.

Phase 8: Documentation update

Once implemented, update current-behavior docs and internal command docs.

Also leave future narratio run --restore in roadmap only.

Definition of done

The feature is complete when:

• narratio restore exists and is documented.
• It uses the same config/session discovery semantics as other commands where practical.
• It requires configured remote storage.
• It restores only from a committed current archive state.
• It restores the current manifest, transcripts, and artifacts by default.
• It restores audio only with --include-audio.
• It excludes run-local sandboxes.
• It fails on local/remote conflicts by default.
• --force overwrites conflicts.
• --dry-run writes nothing.
• It uses fake storage in tests.
• It does not change prepare behavior.
• It does not implement narratio run --restore.
• It avoids AWS SDK leakage outside the storage adapter.
• It uses centralized path/key helpers rather than scattered string concatenation.
• go test ./... passes.

Suggested test commands

Run focused tests first:

go test ./internal/app -run TestExecute -v
go test ./internal/adapters/storage -v
go test ./internal/artifacts -v
go test ./internal/manifest -v

Then run the full suite:

go test ./...

Suggested commit message

Add restore subcommand roadmap