narratio/docs/roadmap/restore.md

# Roadmap: `narratio restore` Subcommand

## Status

Implemented through Step 8. This document remains as roadmap and design history for the restore feature, and as the home for future restore-related ideas (for example `run --restore`).

## Summary

Add a new `narratio restore` subcommand that hydrates a local session workspace from the current committed remote archive state.

The primary operator workflow is:

```bash
narratio restore --session-id 2026-04-04
narratio run-stage --force analyze
```

This should allow a new machine with no local workspace state to restore the durable session manifest, transcripts, and generated artifacts from S3, then generate new Scriptorium artifacts without re-running transcription, merge, normalize, polish, or trim.

This is intentionally a separate command. Do not fold this behavior into the `prepare` stage. The existing `prepare` stage should remain focused on materializing configured local/S3 inputs for a pipeline run.

## Goals

- Add a first-class `narratio restore` command.
- Restore the current committed remote session state into the canonical local session workspace.
- Use the existing object storage adapter boundary.
- Preserve archive commit semantics: only restore from a remote state that has a valid current commit marker.
- Restore durable session-level outputs needed for downstream stages, especially `analyze`.
- Provide safe conflict behavior by default.
- Support `--dry-run`, `--force`, and `--include-audio`.
- Keep the implementation explicit, testable, and narrow.

## Non-goals

- Do not make `restore` a pipeline stage.
- Do not change the `prepare` stage behavior as part of this work.
- Do not add implicit restore behavior to `narratio run` in this implementation.
- Do not restore historical run-local sandboxes by default.
- Do not implement a generic remote synchronization engine.
- Do not implement bidirectional sync.
- Do not delete local files merely because they are absent remotely.
- Do not merge remote and local manifests in the first implementation.
- Do not require live S3 for the ordinary unit test suite.

## Future work explicitly out of scope

A future change may add:

```bash
narratio run --restore
```

That future flag should run `narratio restore` before starting the normal pipeline. Mention this as future work in roadmap/docs if useful, but do not implement it now.

## Existing architecture to preserve

### `prepare` remains input materialization

The `prepare` stage currently materializes required session inputs into canonical local workspace paths and records input provenance. It owns local copying/materialization of config and audio inputs, including S3 audio download when `session.inputs.audio_s3.prefix` is configured. It does not own transcript generation/processing or archive publish behavior.

`restore` should not be implemented by expanding `prepare`. It should be an app-level command that reuses shared helpers where appropriate.

### Workspace model

The local durable session workspace is campaign-aware:

```text
{workspace.root}/work/{campaign}/{session_id}/
```

It contains durable session paths such as:

```text
manifest.json
inputs/
audio/
transcripts/
artifacts/
reports/
logs/
config/
current/
runs/
```

Run-local sandboxes live below:

```text
runs/{run_id}/
```

Restore should target durable session-level paths, not old run-local stage sandboxes.

### Storage boundary

The storage adapter owns object-store primitives only: `List`, `Download`, `Upload`, and `Exists`.

The storage adapter must not infer root prefixes, campaign names, session IDs, run IDs, or archive layout. Restore code must construct full bucket-relative keys before calling storage.

### Archive commit boundary

A remote run is current only after the archive stage has uploaded the run record, promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`.

`current/run_id.txt` is the final remote commit marker and must be written last.

Restore must not treat incomplete, skipped, failed, or uncommitted archive attempts as current remote state.

## User-facing command

Add:

```bash
narratio restore [flags]
```

The command should use the same configuration/session discovery conventions as `run`, `plan`, `resume`, and `run-stage` where practical:

```bash
narratio restore --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-04-04
```

Required effective inputs:

- resolved pipeline config;
- resolved session config;
- `session.campaign`;
- `session.session_id`;
- configured remote storage backend.

Supported flags:

```text
--config <path>       Existing pipeline config path behavior.
--session <path>      Existing session config path behavior.
--session-id <value>  Existing session template behavior.
--dry-run             Plan restore actions without writing local files.
--force               Overwrite conflicting local files with remote files.
--include-audio       Include archived session-level audio files.
```

Do not add `--restore` to `run` in this implementation.

## Default restore scope

By default, restore:

1. Validates and reads the current remote commit marker.
2. Downloads the current remote manifest into the local session manifest path.
3. Downloads durable transcript files.
4. Downloads durable generated artifact files.

Default included remote/local durable paths:

```text
manifest.json                 from remote current manifest
transcripts/**
artifacts/**
```

Default excluded paths:

```text
audio/**                      unless --include-audio is passed
runs/**                       always excluded for this implementation
logs/**                       excluded for this implementation
reports/**                    excluded for this implementation unless needed for current manifest validation
config/**                     excluded for this implementation
inputs/**                     excluded for this implementation
current/**                    remote control metadata only; do not mirror blindly
```

If the existing archive implementation stores promoted files in a different remote layout, use the existing archive/path helpers and current archive semantics rather than inventing a parallel layout.

## Remote state discovery

Implement restore around the current committed archive state.

Expected algorithm:

1. Resolve pipeline/session config.
2. Ensure storage is configured.
3. Ensure local workspace layout exists.
4. Acquire the session lock.
5. Build the remote session archive prefix using the same helpers/policy used by archive code.
6. Check for the remote `current/run_id.txt` commit marker.
7. Read the committed run ID.
8. Download `current/manifest.json` to a temporary file.
9. Validate that the manifest is parseable and belongs to the requested campaign/session.
10. Build a restore plan from the committed remote state.
11. Execute the restore plan unless `--dry-run` is set.
12. Emit a concise summary.

Important: `current/run_id.txt` is the commit marker. Do not restore from a remote session prefix merely because files exist under `transcripts/` or `artifacts/`.

## Restore planning

Create a planning layer before writing files.

A restore plan entry should include at least:

```go
type RestoreAction struct {
    Kind        RestoreActionKind
    RemoteKey   string
    LocalPath   string
    Size        int64
    ETag        string
    ExistsLocal bool
    SameLocal   bool
    Conflict    bool
    Reason      string
}
```

Suggested action kinds:

```text
download
skip_same
skip_missing_optional
conflict
```

The restore planner should be deterministic:

- sort remote objects by key;
- sort planned actions by local path or stable restore priority;
- write/report stable output for tests.

## Conflict and overwrite policy

Default behavior should be safe.

For each planned file:

```text
local absent:
  download

local present and same as remote:
  skip

local present and different:
  conflict; fail restore unless --force is set

--force:
  overwrite local conflicting files with remote versions

--dry-run:
  do not write any files; report what would happen
```

The first implementation may use size and checksum/hash comparison where available. If remote ETag cannot be treated as a content hash, compare by downloading to a temporary file and hashing locally before deciding whether a local file is the same. Prefer correctness over assuming provider-specific ETag semantics.

Do not delete local files that are not present remotely.

## File writing and transactionality

Restore should avoid partial writes.

Implementation requirements:

- download each remote object to a temporary file under the session workspace or OS temp dir;
- validate downloaded content where possible before replacing local files;
- create parent directories as needed;
- atomically rename/copy into place only after successful download;
- do not overwrite local files unless `--force` is set;
- if a later file fails, preserve already-restored files but return a failure summary;
- never corrupt an existing local manifest on failed manifest download/parse.

Manifest restore is especially sensitive:

- download remote `current/manifest.json` to a temporary file;
- parse and validate it;
- if no local manifest exists, install it;
- if a local manifest exists and is equivalent, skip;
- if a local manifest exists and differs, fail unless `--force` is set;
- with `--force`, replace the local manifest with the remote manifest after validation;
- do not attempt a manifest merge in the initial implementation.

## Manifest semantics

`restore` is not a pipeline run and should not mark stages as running/succeeded/failed.

The restored remote manifest becomes the local session manifest. That is what allows a subsequent command such as:

```bash
narratio run-stage --force analyze
```

to see existing upstream stage state and canonical durable outputs.

Do not create a new run manifest for `restore`.

It is acceptable to write a restore diagnostic report outside the manifest, for example:

```text
reports/restore-latest.json
```

or a timestamped report, if that pattern fits the existing codebase. The report must not contain secrets.

## Local workspace locking

`restore` should acquire the same session lock used by ordinary pipeline operations before modifying session workspace state.

If the lock is held, fail fast with the same lock-conflict behavior used elsewhere.

`--dry-run` may still acquire the lock for consistency, but it is acceptable to avoid the lock if the codebase already has a clear read-only command pattern. Prefer safety and simplicity.

## Audio behavior

By default, do not restore audio.

If `--include-audio` is passed:

- restore archived durable session-level audio files only;
- do not use run-scoped spool paths;
- do not mutate or delete spool state;
- do not infer original `session.inputs.audio_s3.prefix` behavior;
- respect the same conflict/force/dry-run behavior used for transcripts/artifacts.

If the archive does not contain durable audio files, `--include-audio` should report that no archived audio was found rather than failing, unless the final implementation chooses to treat explicit audio restore as required. Prefer non-failure for absent archived audio unless tests or existing archive semantics suggest otherwise.

## Remote object selection

Prefer using manifest/artifact metadata when it reliably identifies durable outputs.

Also support listing committed durable archive prefixes so restore can retrieve all top-level session artifacts that may not yet be fully represented in manifest metadata.

The implementation should inspect existing archive code before choosing the final object-selection method. Do not duplicate archive path construction.

Recommended selection priority:

1. Remote current manifest path.
2. Durable promoted transcript/artifact outputs recorded in the manifest or archive metadata, if available.
3. Objects under committed durable `transcripts/` and `artifacts/` archive prefixes.
4. Objects under durable `audio/` only when `--include-audio` is passed.

Always exclude:

```text
runs/**
```

for the first implementation.

## Package and file organization

Expected areas to inspect and update:

```text
cmd/narratio/
internal/app/
internal/adapters/storage/
internal/artifacts/
internal/manifest/
docs/
examples/
```

Suggested implementation shape:

```text
internal/app/restore.go
internal/app/restore_test.go

internal/archive/restore/
  planner.go
  executor.go
  report.go
  keys.go
  *_test.go
```

The exact package name may vary. Use whatever best fits the existing repository, but keep these boundaries clear:

- `internal/app` owns CLI command handling, config/session loading, lock acquisition, and wiring.
- Restore planning/execution owns remote key discovery, conflict detection, downloads, and reporting.
- `internal/adapters/storage` remains a transport boundary only.
- Workspace/path helpers remain centralized; do not scatter string concatenation.

If the repository already has an `internal/archive` or archive-stage helper package, prefer extending that rather than creating a conflicting package layout.

## CLI output

`narratio restore` should print a concise operator summary.

Example successful output:

```text
Restored session archive for sample-campaign/2026-04-04
Remote run: 20260504T031500Z-a1b2c3
Downloaded: 4
Skipped unchanged: 2
Conflicts: 0
```

Example dry run:

```text
Restore plan for sample-campaign/2026-04-04
Remote run: 20260504T031500Z-a1b2c3
Would download: transcripts/processed.json
Would download: transcripts/trimmed.json
Would skip unchanged: artifacts/session_recap.md
```

Example conflict:

```text
restore conflict: local artifacts/session_recap.md differs from remote archive; rerun with --force to overwrite
```

Do not print transcript or artifact content.

## Error behavior

Fail clearly when:

- storage backend is not configured;
- S3 bucket/config is missing or invalid;
- remote current commit marker is missing;
- remote current manifest is missing;
- remote manifest is invalid;
- remote manifest does not match requested campaign/session;
- local file differs from remote and `--force` is not set;
- a required remote object download fails;
- a local path would escape the session workspace;
- a remote key maps to an unsafe local path.

Skip or report non-fatal conditions when:

- optional audio restore finds no archived audio;
- an included prefix has no objects;
- a local file already matches the remote file.

## Path safety

Every restored file must map to a safe path under the session root.

Validation rules:

- local restore paths must be relative to the session root;
- reject absolute paths;
- reject `..` traversal;
- reject paths that escape through symlinks if the codebase has symlink-safe path checks;
- do not restore remote keys directly without mapping/classification;
- do not mirror arbitrary remote keys.

## Testing plan

Add focused unit tests. Do not require live S3.

### CLI tests

Add or update `internal/app` command tests for:

- `narratio restore --help`;
- restore accepts `--config`, `--session`, and `--session-id`;
- restore accepts `--dry-run`;
- restore accepts `--force`;
- restore accepts `--include-audio`;
- restore fails when storage is not configured;
- restore does not run pipeline stages.

### Restore planner tests

Test:

- missing `current/run_id.txt` fails;
- missing `current/manifest.json` fails;
- invalid manifest fails;
- wrong campaign/session manifest fails;
- default scope includes manifest/transcripts/artifacts;
- default scope excludes audio/logs/reports/config/runs;
- `--include-audio` includes durable audio;
- run-local keys are excluded;
- keys are sorted deterministically;
- unsafe remote-to-local paths are rejected.

### Conflict policy tests

Test:

- absent local file downloads;
- matching local file skips;
- differing local file conflicts by default;
- `--force` overwrites conflicts;
- `--dry-run` writes nothing;
- partial failure does not corrupt an existing local manifest.

### Storage/fake tests

Use fake storage to simulate:

- object listing;
- object download;
- missing objects;
- download failures;
- metadata/ETag behavior.

### Workspace/lock tests

Test:

- session layout is created before restore;
- session lock conflict fails;
- restored files land under the expected campaign/session workspace;
- no files are written outside the session root.

### Follow-up command workflow test

Add at least one test that simulates:

```bash
narratio restore --session-id 2026-04-04
narratio run-stage --force analyze
```

The test does not need to run real Scriptorium. Use existing fake/stub behavior to verify that restored transcripts and manifest state are sufficient for analyze-stage input resolution.

## Documentation updates when implemented

When the feature is implemented, update current-behavior docs:

```text
docs/cli.md
docs/operations.md
docs/internal/storage.md or docs/internal/archive/restore.md
```

If the documentation set does not yet have an internal restore document, add one consistent with the existing internal-doc style:

```text
docs/internal/command-restore.md
```

or:

```text
docs/internal/archive-restore.md
```

Do not document future `narratio run --restore` behavior outside `docs/roadmap/` until implemented.

## Implementation phases

### Phase 1: Audit existing archive and path helpers (completed)

Before coding behavior, inspect:

```text
internal/app/
internal/stage/archive*
internal/adapters/storage/
internal/artifacts/
internal/manifest/
docs/internal/stage-archive.md, if present
```

Determine:

- exact remote archive key layout;
- how root prefix/campaign/session are modeled;
- how current commit marker keys are built;
- how current manifest is uploaded;
- where promoted outputs are uploaded;
- whether helper functions already exist for remote archive keys;
- whether local workspace path helpers can safely map restore destinations.

Deliverable:

- small code comments or internal helper selection;
- no large behavior change yet unless required by tests.

### Phase 2: Add CLI surface and command wiring (completed)

Add `narratio restore` command parsing.

Wire flags:

```text
--config
--session
--session-id
--dry-run
--force
--include-audio
```

Use the existing config/session load path where practical.

Deliverable:

- command exists;
- help output is sensible;
- command validates basic inputs;
- command returns a clear “not yet implemented” or calls an empty planner if phased commits are desired;
- CLI tests pass.

### Phase 3: Implement remote current-state discovery (completed)

Add restore code that:

- creates an object store from resolved config;
- builds remote current marker key;
- reads `current/run_id.txt`;
- reads/downloads `current/manifest.json`;
- validates manifest identity;
- returns remote current-state metadata.

Deliverable:

- fake-storage tests for current-state discovery;
- no local file writes beyond temporary files.

### Phase 4: Implement restore planning (completed)

Build deterministic restore plans for default scope and `--include-audio`.

Deliverable:

- plan lists manifest, transcript, artifact files;
- plan excludes run-local data;
- plan detects local same/conflict/missing states;
- dry-run output works;
- no real file overwrite yet except temp comparisons as needed.

### Phase 5: Implement restore execution (completed)

Execute the plan safely:

- create directories;
- download to temporary files;
- validate content where practical;
- atomically install files;
- enforce default conflict failure;
- support `--force`;
- preserve existing manifest unless safe to replace.

Deliverable:

- restore works end-to-end against fake storage;
- failures are clear and do not corrupt existing local manifest.

### Phase 6: Add restore report and operator summary (completed)

Add concise stdout summary and optional JSON restore report if consistent with project diagnostics.

Deliverable:

- user-friendly output;
- durable diagnostic report if implemented;
- no content leakage.

### Phase 7: Workflow integration test (completed)

Add a test for restoring a previous session and then forcing `analyze`.

Deliverable:

- restored manifest/transcripts/artifacts are sufficient for analyze input resolution;
- no upstream stages rerun;
- no reliance on live subprocesses or S3.

### Phase 8: Documentation update (completed)

Once implemented, update current-behavior docs and internal command docs.

Also leave future `narratio run --restore` in roadmap only.

## Definition of done

The feature is complete when:

- `narratio restore` exists and is documented.
- It uses the same config/session discovery semantics as other commands where practical.
- It requires configured remote storage.
- It restores only from a committed current archive state.
- It restores the current manifest, transcripts, and artifacts by default.
- It restores audio only with `--include-audio`.
- It excludes run-local sandboxes.
- It fails on local/remote conflicts by default.
- `--force` overwrites conflicts.
- `--dry-run` writes nothing.
- It uses fake storage in tests.
- It does not change `prepare` behavior.
- It does not implement `narratio run --restore`.
- It avoids AWS SDK leakage outside the storage adapter.
- It uses centralized path/key helpers rather than scattered string concatenation.
- `go test ./...` passes.

## Suggested test commands

Run focused tests first:

```bash
go test ./internal/app -run TestExecute -v
go test ./internal/adapters/storage -v
go test ./internal/artifacts -v
go test ./internal/manifest -v
```

Then run the full suite:

```bash
go test ./...
```

## Suggested commit message

```text
Add restore subcommand roadmap
```