diff --git a/docs/roadmap/previous.md b/docs/roadmap/previous.md new file mode 100644 index 0000000..45fe36f --- /dev/null +++ b/docs/roadmap/previous.md @@ -0,0 +1,629 @@ +# Roadmap: Previous-Session Artifacts + +## Status + +Planned. + +This roadmap describes the implementation strategy for first-class previous-session artifact support in Narratio. It belongs under `docs/roadmap/previous.md` until the feature is implemented. After implementation, current behavior should be documented in the appropriate user-facing and internal documentation files, and this roadmap should be removed or marked complete according to the documentation policy. + +## Summary + +Narratio should support using artifacts from a previous session as inputs to artifacts generated for the current session. + +The primary use case is session recap continuity: a current session recap should be able to consume the previous session recap. The design should support arbitrary previous-session artifacts from the start, not just `session_recap`. + +The canonical source syntax should be: + +```yaml +source: narratio.previous_session.artifact. +``` + +For example: + +```yaml +source: narratio.previous_session.artifact.session_recap +``` + +Previous-session artifacts are materialized during the `prepare` stage into a current-session top-level `previous/` directory. Downstream stages consume only the local `previous/` copies. The S3 backend is authoritative for previous-session state. + +## Design Decisions + +### 1. Add `previous_session_id` to `session.yml` + +Add an optional top-level session key: + +```yaml +session_id: "{{ session_id }}" +previous_session_id: "{{ previous_session_id }}" +campaign: sample-campaign +``` + +Rules: + +- `previous_session_id` is optional. +- If present, it identifies the previous session within the same campaign. +- It must not equal `session_id`. +- It should use the same validation rules as `session_id`. +- It may be supplied through session templating. +- Add a CLI/template value such as `--previous-session-id ` if required by the existing session templating implementation. +- If a template placeholder for `previous_session_id` is present and no value is supplied, loading should fail with a clear unresolved-template error. + +### 2. Use canonical previous-session artifact source IDs + +Support this source pattern in Scriptorium artifact input definitions: + +```text +narratio.previous_session.artifact. +``` + +Examples: + +```yaml +inputs: + previous_recap: + source: narratio.previous_session.artifact.session_recap + required: false + + previous_quest_log: + source: narratio.previous_session.artifact.quest_log + required: true +``` + +Rules: + +- `` must be a valid configured artifact key. +- Use the same artifact key validation rules as current-session runtime artifacts. +- Do not special-case `session_recap`. +- Do not limit implementation to a fixed list of previous artifacts. + +### 3. Add a top-level `previous/` workspace directory + +Extend the session workspace layout with: + +```text +previous/ + manifest.json + artifacts/ + +``` + +The `previous/` directory is current-session state. It is a prepared input cache, not a full mirror of the previous session workspace. + +Conceptually: + +```text +work/// + previous/ + manifest.json + artifacts/ + session_recap.md + quest_log.json +``` + +The current session should not read directly from the previous session's local workspace during ordinary operation. + +### 4. S3 is authoritative for previous-session state + +For this initial implementation, previous-session artifacts should be downloaded from the configured S3 backend. + +Do not compare local and remote copies. + +Do not prefer local previous-session workspace state. + +Do not implement a `--local` override in this roadmap. That can be considered later. + +The simplified stage behavior is: + +1. If the local manifest indicates `prepare` already succeeded and `--force` is not supplied, the runner skips `prepare`. No previous-session download occurs. +2. If `prepare` has not succeeded, `prepare` runs and downloads referenced previous-session artifacts from S3. +3. If `prepare` previously succeeded but `--force` is supplied, `prepare` runs again and overwrites local `previous/` state from S3. + +### 5. Materialize only referenced previous-session artifacts + +During `prepare`, scan the current resolved pipeline/session configuration for Scriptorium artifact inputs whose source matches: + +```text +narratio.previous_session.artifact. +``` + +Only those referenced previous-session artifacts need to be downloaded. + +Do not blindly download every previous-session artifact. + +If no previous-session artifact sources are referenced, `prepare` should not require `previous_session_id` and should not touch `previous/`. + +### 6. Preserve stage boundaries + +`prepare` owns previous-session artifact materialization because these files are inputs to later stages. + +`analyze` should not talk to S3. + +The Scriptorium adapter should not know about previous sessions. + +The storage adapter should not infer campaign, session, run, or root-prefix semantics. Callers should continue to provide explicit bucket-relative keys. + +### 7. Archive and restore `previous/` + +After implementation: + +- `archive` should include `previous/` as durable current-session prepared input state. +- `restore` should restore `previous/` along with the rest of the durable session state it already restores. +- `previous/` should not be treated as current-session generated artifacts. +- `previous/` entries should be recorded as inputs/provenance, not as outputs produced by the current session. + +## Target User Workflow + +A typical session config: + +```yaml +session_id: "{{ session_id }}" +previous_session_id: "{{ previous_session_id }}" +campaign: sample-campaign + +inputs: + audio_dir: ./audio + speakers_file: ./examples/speakers.yml + autocorrect_file: ./examples/autocorrect.yml + glossary_file: ./examples/glossary.yml +``` + +A typical artifact config: + +```yaml +scriptorium: + artifacts: + session_recap: + enabled: true + prompt_id: dnd_session.session_recap + output_path: artifacts/session_recap.md + inputs: + transcript: + source: narratio.transcript.trimmed + required: true + previous_recap: + source: narratio.previous_session.artifact.session_recap + required: false +``` + +Typical command: + +```bash +narratio run --session-id 2026-04-11 --previous-session-id 2026-04-04 +``` + +Expected behavior: + +1. `prepare` sees a referenced previous-session artifact: `session_recap`. +2. `prepare` downloads the previous session's `narratio.artifact.session_recap` from S3. +3. `prepare` writes it under the current session workspace, for example `previous/artifacts/session_recap.md`. +4. `prepare` records provenance in the current session manifest. +5. `analyze` resolves `narratio.previous_session.artifact.session_recap` from the local `previous/` directory. +6. Scriptorium receives the previous recap as a normal input file. + +## Implementation Plan + +### Phase 1: Session config and templating + +Update session configuration structs to include: + +```yaml +previous_session_id: "" +``` + +Implementation steps: + +1. Add `PreviousSessionID` or equivalent to the session config type. +2. Add validation: + - optional; + - same format constraints as `session_id`; + - must not equal `session_id`. +3. Extend session templating support to include: + - `{{previous_session_id}}` + - `{{ previous_session_id }}` +4. Add a CLI flag if required by current templating flow: + - `--previous-session-id ` +5. Ensure unresolved `previous_session_id` placeholders fail clearly. +6. Update config tests for: + - no previous session; + - valid previous session; + - previous session equal to current session; + - unresolved placeholder; + - CLI/template rendering. + +Do not add future workflow flags in this phase. + +### Phase 2: Workspace path helpers + +Add centralized path helpers for current-session previous-state paths. + +Suggested helpers: + +```text +SessionPreviousDir() +SessionPreviousManifestPath() +SessionPreviousArtifactsDir() +SessionPreviousArtifactPath(name or relative output path) +``` + +The exact names should match existing path-helper style. + +Rules: + +- Do not construct `previous/` paths through scattered string concatenation. +- Keep paths session-relative where possible. +- Ensure workspace layout creation includes `previous/` only when appropriate, or creates it idempotently with the rest of the layout if simpler. + +Tests: + +- path helper tests; +- workspace layout tests; +- ensure cleanup logic does not accidentally delete configured roots; +- ensure `previous/` is treated as session-durable state, not run-local state. + +### Phase 3: Previous-session source parsing + +Add parsing/recognition for: + +```text +narratio.previous_session.artifact. +``` + +Implementation steps: + +1. Add constants/helpers in the artifact/source parsing layer. +2. Validate artifact names using the same rules as current runtime artifact keys. +3. Add helpers such as: + - `IsPreviousSessionArtifactSource(source string) bool` + - `PreviousSessionArtifactName(source string) (string, bool)` +4. Ensure config validation accepts this source pattern. +5. Ensure invalid sources fail clearly. + +Tests: + +- valid previous-session artifact source; +- invalid/missing artifact name; +- invalid artifact key characters; +- ordinary current-session sources still validate; +- unknown sources still fail. + +### Phase 4: Scan configured artifacts for previous-session inputs + +Add a helper that inspects resolved Scriptorium artifact definitions and returns the set of referenced previous-session artifact names. + +Rules: + +- Scan enabled artifacts according to current artifact-enable semantics. +- Include all inputs whose source matches `narratio.previous_session.artifact.`. +- Deduplicate artifact names. +- Sort results deterministically. +- Preserve required/optional information per reference. +- If the same previous artifact is referenced both required and optional, treat it as required. + +Suggested output model: + +```go +type PreviousArtifactRequirement struct { + Name string + Required bool + Sources []string // optional diagnostics +} +``` + +Tests: + +- no artifacts; +- no previous inputs; +- one optional previous input; +- one required previous input; +- duplicate references; +- required plus optional reference to the same artifact; +- deterministic ordering. + +### Phase 5: Resolve previous-session archive keys + +Implement a narrow service/helper used by `prepare` to resolve previous-session artifact files from S3. + +Responsibilities: + +1. Locate the previous session's current remote state. +2. Download the previous session manifest, or the minimum remote metadata needed to resolve artifact source IDs. +3. Resolve `narratio.artifact.` inside the previous session's artifact catalog/manifest. +4. Download the resolved artifact into the current session's `previous/` directory. +5. Download/store the previous session manifest as `previous/manifest.json`. +6. Return provenance records for manifest input recording. + +Important archive invariant: + +- Remote current state must be based on the committed archive marker. +- Do not treat incomplete archive uploads as current state. +- Use the existing archive/current remote layout and commit-marker rules. +- `current/run_id.txt` is the final remote commit marker and should be respected when locating current remote state. + +Do not put prefix semantics into the storage adapter. Compute explicit bucket-relative keys in app/stage/archive helper code, then call the storage adapter. + +Required vs optional behavior: + +- Required previous artifact missing from S3/current manifest: fail `prepare`. +- Optional previous artifact missing from S3/current manifest: continue without materializing that input. +- Previous session missing entirely: + - fail if any referenced previous artifact is required; + - continue if all referenced previous artifacts are optional. +- If previous_session_id is unset: + - fail if any referenced previous artifact is required; + - continue and omit all previous-session inputs if all are optional. + +Validation behavior: + +- Downloaded artifacts should pass the same validation rules as current-session artifacts where practical. +- Generic Scriptorium artifacts should at least be non-empty. +- Invalid required artifact: fail. +- Invalid optional artifact: prefer fail if the object exists but is invalid, because invalid archived data is usually an operator problem rather than absence. + +Tests: + +- downloads previous manifest; +- downloads required previous artifact; +- skips missing optional previous artifact; +- fails missing required previous artifact; +- fails required previous artifact when previous_session_id is unset; +- optional previous artifact with no previous_session_id does not fail; +- respects current remote commit marker; +- does not use local previous-session workspace state; +- uses storage adapter with explicit keys. + +### Phase 6: Integrate with `prepare` + +Extend the `prepare` stage: + +1. Run existing input materialization as before. +2. Detect previous-session artifact requirements. +3. If requirements exist, hydrate `previous/` from S3 according to the rules above. +4. Record hydrated previous artifacts in `manifest.Inputs`. +5. Preserve existing prepare outputs and provenance behavior. + +Overwrite behavior: + +- If `prepare` runs, it owns `previous/`. +- Before hydrating, clear the managed `previous/` directory, or clear the managed previous artifact paths. +- Prefer clearing the whole `previous/` directory if no other feature writes there. +- Under `--force`, this naturally overwrites local `previous/` state. +- Do not compare local and remote copies. + +Skip behavior: + +- Do not add stage-local skip logic. +- Runner-level skip remains authoritative. +- If the manifest says `prepare` succeeded and `--force` is not supplied, `prepare` does not run and no S3 downloads occur. +- If users add a new previous-session input after `prepare` already succeeded, they must rerun prepare with `--force`. + +Error message requirement: + +If an analyze-stage input cannot be resolved because `previous/` is missing or stale, the error should tell the operator to run: + +```bash +narratio run-stage --force prepare +``` + +or the appropriate existing CLI command shape. + +Tests: + +- ordinary prepare without previous_session_id remains unchanged; +- prepare with optional previous artifact and no previous_session_id succeeds; +- prepare with required previous artifact and no previous_session_id fails; +- prepare with required previous artifact downloads to `previous/`; +- force prepare overwrites `previous/`; +- prepare records manifest inputs for previous artifacts; +- prepare skip behavior remains controlled by runner tests; +- no regression in S3 audio prepare behavior. + +### Phase 7: Artifact resolver support + +Update artifact resolution so Scriptorium inputs can resolve: + +```text +narratio.previous_session.artifact. +``` + +from the current session's `previous/` directory. + +Rules: + +- The resolver should not call S3. +- The resolver should not read the previous session's local workspace. +- The resolver should map the previous-session source ID to the local prepared copy under `previous/`. +- Resolution should use manifest input records when available. +- Fallback to the local `previous/` path may be allowed if consistent with existing resolver behavior, but manifest provenance should be preferred. +- Missing required input should fail with a clear prepare-oriented message. +- Optional missing input should be omitted. + +Suggested provenance: + +```text +previous_session.manifest.outputs +previous_session.archive.current +current_session.previous_cache +``` + +Use names that fit the existing manifest/resolver vocabulary. + +Tests: + +- resolves prepared previous artifact through manifest input record; +- resolves or fails appropriately when only filesystem copy exists, depending on chosen fallback policy; +- missing optional previous artifact is omitted; +- missing required previous artifact fails clearly; +- current-session `narratio.artifact.` behavior is unchanged. + +### Phase 8: Analyze-stage integration + +The analyze stage should require little or no special previous-session logic if the resolver is designed correctly. + +Confirm: + +- Scriptorium input resolution accepts previous-session source IDs. +- The Scriptorium adapter receives a normal local input path. +- Render-debug and run modes behave the same as for ordinary inputs. +- Stage metadata includes useful input provenance if current structures support it. + +Tests: + +- configured artifact receives previous recap input; +- optional previous recap omitted when not prepared; +- required previous recap fails when not prepared; +- render-debug path works with previous-session inputs; +- no S3 calls occur from analyze. + +### Phase 9: Archive `previous/` + +Update archive behavior so the current session's durable `previous/` directory is uploaded/preserved. + +Rules: + +- `previous/` is current-session input/provenance state. +- It is not a generated current-session artifact. +- Archive it with the current session's durable state, alongside other durable session files according to the current archive layout. +- Preserve existing archive commit ordering. +- Do not make `previous/` upload the final commit marker. +- Do not treat missing `previous/` as an error when no previous-session inputs were prepared. + +Tests: + +- archive includes `previous/manifest.json` and prepared previous artifacts when present; +- archive omits or tolerates absent `previous/` when unused; +- archive commit ordering remains valid; +- cleanup behavior does not delete durable `previous/` before archive commit. + +### Phase 10: Restore `previous/` + +Update `narratio restore` so it restores the current session's archived `previous/` directory. + +Rules: + +- Restore `previous/` as durable current-session state. +- Do not infer or restore the previous session workspace merely because `previous_session_id` exists. +- Do not redownload previous-session artifacts from the previous session archive during restore; restore the current session's archived `previous/` cache. +- Respect existing restore flags and overwrite behavior. + +Tests: + +- restore downloads `previous/` when present; +- restore succeeds when `previous/` is absent; +- restore with force overwrites local `previous/` according to existing restore semantics; +- restored current session can run analyze using `previous/` without needing previous session archive access. + +### Phase 11: Documentation updates after implementation + +After implementation, update current-behavior docs. Do not document implemented behavior only in this roadmap. + +Likely files: + +- `docs/config.md` +- `docs/cli.md` +- `docs/operations.md` +- `docs/internal/stage-prepare.md` +- `docs/internal/artifacts.md` +- `docs/internal/storage.md` only if storage contracts change +- `docs/internal/workspace.md` +- relevant maintained examples under `examples/` + +Docs should explain: + +- `session.previous_session_id`; +- `--previous-session-id` if added; +- `narratio.previous_session.artifact.`; +- `previous/` workspace directory; +- S3-authoritative previous-session behavior; +- required vs optional previous-session artifact handling; +- when to run `narratio run-stage --force prepare`. + +Do not document future `--local` support as current behavior. + +## Future Work + +These items are explicitly out of scope for the initial implementation. + +### `--local` previous-session source + +A future flag may allow `prepare` to copy previous-session artifacts from a local previous session workspace instead of S3. + +Possible future command shape: + +```bash +narratio run-stage --force prepare --local +``` + +or a more specific flag such as: + +```bash +narratio run-stage --force prepare --previous-source local +``` + +Do not implement this now. + +### `narratio run --restore` + +A future flag may run `narratio restore` before starting the regular pipeline: + +```bash +narratio run --restore --session-id 2026-04-11 +``` + +Do not implement this as part of previous-session artifact support unless it already exists and only requires documentation. + +### Multi-previous-session support + +A future design may support more than one previous/reference session. + +Do not implement this now. + +### Previous-session artifact version pinning + +A future design may pin previous-session artifact inputs to a specific previous run ID or checksum. + +Do not implement this now. + +## Test Plan Summary + +Run at least: + +```bash +go test ./internal/config -v +go test ./internal/artifacts -v +go test ./internal/stage -run Prepare -v +go test ./internal/stage -run Analyze -v +go test ./internal/app -run TestExecute -v +go test ./... +``` + +Add focused tests for: + +- session config and templating; +- previous-session source parsing; +- previous artifact requirement scanning; +- S3-backed previous artifact download; +- prepare skip/force semantics; +- manifest input provenance; +- resolver behavior; +- analyze integration; +- archive/restore `previous/` persistence; +- examples load/validate. + +## Acceptance Criteria + +The feature is complete when: + +1. `session.yml` supports optional `previous_session_id`. +2. Session templating supports `previous_session_id`. +3. Scriptorium artifact inputs accept `narratio.previous_session.artifact.`. +4. `prepare` scans enabled Scriptorium artifacts for previous-session artifact inputs. +5. `prepare` downloads referenced previous-session artifacts from S3 into `previous/`. +6. Required/optional behavior is correct. +7. `prepare` uses stage-level skip/force behavior and does not compare local and remote copies. +8. `analyze` resolves previous-session artifacts only from local prepared `previous/` state. +9. `archive` persists `previous/`. +10. `restore` restores `previous/`. +11. Tests cover config, prepare, resolver, analyze, archive, and restore behavior. +12. Current-behavior docs and maintained examples are updated after implementation. +13. No storage adapter implementation infers campaign/session/root-prefix semantics. +14. No previous-session S3 logic is added to `analyze` or the Scriptorium adapter. diff --git a/docs/roadmap/restore.md b/docs/roadmap/restore.md deleted file mode 100644 index 606d88a..0000000 --- a/docs/roadmap/restore.md +++ /dev/null @@ -1,715 +0,0 @@ -# Roadmap: `narratio restore` Subcommand - -## Status - -Implemented through Step 8. This document remains as roadmap and design history for the restore feature, and as the home for future restore-related ideas (for example `run --restore`). - -## Summary - -Add a new `narratio restore` subcommand that hydrates a local session workspace from the current committed remote archive state. - -The primary operator workflow is: - -```bash -narratio restore --session-id 2026-04-04 -narratio run-stage --force analyze -``` - -This should allow a new machine with no local workspace state to restore the durable session manifest, transcripts, and generated artifacts from S3, then generate new Scriptorium artifacts without re-running transcription, merge, normalize, polish, or trim. - -This is intentionally a separate command. Do not fold this behavior into the `prepare` stage. The existing `prepare` stage should remain focused on materializing configured local/S3 inputs for a pipeline run. - -## Goals - -- Add a first-class `narratio restore` command. -- Restore the current committed remote session state into the canonical local session workspace. -- Use the existing object storage adapter boundary. -- Preserve archive commit semantics: only restore from a remote state that has a valid current commit marker. -- Restore durable session-level outputs needed for downstream stages, especially `analyze`. -- Provide safe conflict behavior by default. -- Support `--dry-run`, `--force`, and `--include-audio`. -- Keep the implementation explicit, testable, and narrow. - -## Non-goals - -- Do not make `restore` a pipeline stage. -- Do not change the `prepare` stage behavior as part of this work. -- Do not add implicit restore behavior to `narratio run` in this implementation. -- Do not restore historical run-local sandboxes by default. -- Do not implement a generic remote synchronization engine. -- Do not implement bidirectional sync. -- Do not delete local files merely because they are absent remotely. -- Do not merge remote and local manifests in the first implementation. -- Do not require live S3 for the ordinary unit test suite. - -## Future work explicitly out of scope - -A future change may add: - -```bash -narratio run --restore -``` - -That future flag should run `narratio restore` before starting the normal pipeline. Mention this as future work in roadmap/docs if useful, but do not implement it now. - -## Existing architecture to preserve - -### `prepare` remains input materialization - -The `prepare` stage currently materializes required session inputs into canonical local workspace paths and records input provenance. It owns local copying/materialization of config and audio inputs, including S3 audio download when `session.inputs.audio_s3.prefix` is configured. It does not own transcript generation/processing or archive publish behavior. - -`restore` should not be implemented by expanding `prepare`. It should be an app-level command that reuses shared helpers where appropriate. - -### Workspace model - -The local durable session workspace is campaign-aware: - -```text -{workspace.root}/work/{campaign}/{session_id}/ -``` - -It contains durable session paths such as: - -```text -manifest.json -inputs/ -audio/ -transcripts/ -artifacts/ -reports/ -logs/ -config/ -current/ -runs/ -``` - -Run-local sandboxes live below: - -```text -runs/{run_id}/ -``` - -Restore should target durable session-level paths, not old run-local stage sandboxes. - -### Storage boundary - -The storage adapter owns object-store primitives only: `List`, `Download`, `Upload`, and `Exists`. - -The storage adapter must not infer root prefixes, campaign names, session IDs, run IDs, or archive layout. Restore code must construct full bucket-relative keys before calling storage. - -### Archive commit boundary - -A remote run is current only after the archive stage has uploaded the run record, promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`. - -`current/run_id.txt` is the final remote commit marker and must be written last. - -Restore must not treat incomplete, skipped, failed, or uncommitted archive attempts as current remote state. - -## User-facing command - -Add: - -```bash -narratio restore [flags] -``` - -The command should use the same configuration/session discovery conventions as `run`, `plan`, `resume`, and `run-stage` where practical: - -```bash -narratio restore --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-04-04 -``` - -Required effective inputs: - -- resolved pipeline config; -- resolved session config; -- `session.campaign`; -- `session.session_id`; -- configured remote storage backend. - -Supported flags: - -```text ---config Existing pipeline config path behavior. ---session Existing session config path behavior. ---session-id Existing session template behavior. ---dry-run Plan restore actions without writing local files. ---force Overwrite conflicting local files with remote files. ---include-audio Include archived session-level audio files. -``` - -Do not add `--restore` to `run` in this implementation. - -## Default restore scope - -By default, restore: - -1. Validates and reads the current remote commit marker. -2. Downloads the current remote manifest into the local session manifest path. -3. Downloads durable transcript files. -4. Downloads durable generated artifact files. - -Default included remote/local durable paths: - -```text -manifest.json from remote current manifest -transcripts/** -artifacts/** -``` - -Default excluded paths: - -```text -audio/** unless --include-audio is passed -runs/** always excluded for this implementation -logs/** excluded for this implementation -reports/** excluded for this implementation unless needed for current manifest validation -config/** excluded for this implementation -inputs/** excluded for this implementation -current/** remote control metadata only; do not mirror blindly -``` - -If the existing archive implementation stores promoted files in a different remote layout, use the existing archive/path helpers and current archive semantics rather than inventing a parallel layout. - -## Remote state discovery - -Implement restore around the current committed archive state. - -Expected algorithm: - -1. Resolve pipeline/session config. -2. Ensure storage is configured. -3. Ensure local workspace layout exists. -4. Acquire the session lock. -5. Build the remote session archive prefix using the same helpers/policy used by archive code. -6. Check for the remote `current/run_id.txt` commit marker. -7. Read the committed run ID. -8. Download `current/manifest.json` to a temporary file. -9. Validate that the manifest is parseable and belongs to the requested campaign/session. -10. Build a restore plan from the committed remote state. -11. Execute the restore plan unless `--dry-run` is set. -12. Emit a concise summary. - -Important: `current/run_id.txt` is the commit marker. Do not restore from a remote session prefix merely because files exist under `transcripts/` or `artifacts/`. - -## Restore planning - -Create a planning layer before writing files. - -A restore plan entry should include at least: - -```go -type RestoreAction struct { - Kind RestoreActionKind - RemoteKey string - LocalPath string - Size int64 - ETag string - ExistsLocal bool - SameLocal bool - Conflict bool - Reason string -} -``` - -Suggested action kinds: - -```text -download -skip_same -skip_missing_optional -conflict -``` - -The restore planner should be deterministic: - -- sort remote objects by key; -- sort planned actions by local path or stable restore priority; -- write/report stable output for tests. - -## Conflict and overwrite policy - -Default behavior should be safe. - -For each planned file: - -```text -local absent: - download - -local present and same as remote: - skip - -local present and different: - conflict; fail restore unless --force is set - ---force: - overwrite local conflicting files with remote versions - ---dry-run: - do not write any files; report what would happen -``` - -The first implementation may use size and checksum/hash comparison where available. If remote ETag cannot be treated as a content hash, compare by downloading to a temporary file and hashing locally before deciding whether a local file is the same. Prefer correctness over assuming provider-specific ETag semantics. - -Do not delete local files that are not present remotely. - -## File writing and transactionality - -Restore should avoid partial writes. - -Implementation requirements: - -- download each remote object to a temporary file under the session workspace or OS temp dir; -- validate downloaded content where possible before replacing local files; -- create parent directories as needed; -- atomically rename/copy into place only after successful download; -- do not overwrite local files unless `--force` is set; -- if a later file fails, preserve already-restored files but return a failure summary; -- never corrupt an existing local manifest on failed manifest download/parse. - -Manifest restore is especially sensitive: - -- download remote `current/manifest.json` to a temporary file; -- parse and validate it; -- if no local manifest exists, install it; -- if a local manifest exists and is equivalent, skip; -- if a local manifest exists and differs, fail unless `--force` is set; -- with `--force`, replace the local manifest with the remote manifest after validation; -- do not attempt a manifest merge in the initial implementation. - -## Manifest semantics - -`restore` is not a pipeline run and should not mark stages as running/succeeded/failed. - -The restored remote manifest becomes the local session manifest. That is what allows a subsequent command such as: - -```bash -narratio run-stage --force analyze -``` - -to see existing upstream stage state and canonical durable outputs. - -Do not create a new run manifest for `restore`. - -It is acceptable to write a restore diagnostic report outside the manifest, for example: - -```text -reports/restore-latest.json -``` - -or a timestamped report, if that pattern fits the existing codebase. The report must not contain secrets. - -## Local workspace locking - -`restore` should acquire the same session lock used by ordinary pipeline operations before modifying session workspace state. - -If the lock is held, fail fast with the same lock-conflict behavior used elsewhere. - -`--dry-run` may still acquire the lock for consistency, but it is acceptable to avoid the lock if the codebase already has a clear read-only command pattern. Prefer safety and simplicity. - -## Audio behavior - -By default, do not restore audio. - -If `--include-audio` is passed: - -- restore archived durable session-level audio files only; -- do not use run-scoped spool paths; -- do not mutate or delete spool state; -- do not infer original `session.inputs.audio_s3.prefix` behavior; -- respect the same conflict/force/dry-run behavior used for transcripts/artifacts. - -If the archive does not contain durable audio files, `--include-audio` should report that no archived audio was found rather than failing, unless the final implementation chooses to treat explicit audio restore as required. Prefer non-failure for absent archived audio unless tests or existing archive semantics suggest otherwise. - -## Remote object selection - -Prefer using manifest/artifact metadata when it reliably identifies durable outputs. - -Also support listing committed durable archive prefixes so restore can retrieve all top-level session artifacts that may not yet be fully represented in manifest metadata. - -The implementation should inspect existing archive code before choosing the final object-selection method. Do not duplicate archive path construction. - -Recommended selection priority: - -1. Remote current manifest path. -2. Durable promoted transcript/artifact outputs recorded in the manifest or archive metadata, if available. -3. Objects under committed durable `transcripts/` and `artifacts/` archive prefixes. -4. Objects under durable `audio/` only when `--include-audio` is passed. - -Always exclude: - -```text -runs/** -``` - -for the first implementation. - -## Package and file organization - -Expected areas to inspect and update: - -```text -cmd/narratio/ -internal/app/ -internal/adapters/storage/ -internal/artifacts/ -internal/manifest/ -docs/ -examples/ -``` - -Suggested implementation shape: - -```text -internal/app/restore.go -internal/app/restore_test.go - -internal/archive/restore/ - planner.go - executor.go - report.go - keys.go - *_test.go -``` - -The exact package name may vary. Use whatever best fits the existing repository, but keep these boundaries clear: - -- `internal/app` owns CLI command handling, config/session loading, lock acquisition, and wiring. -- Restore planning/execution owns remote key discovery, conflict detection, downloads, and reporting. -- `internal/adapters/storage` remains a transport boundary only. -- Workspace/path helpers remain centralized; do not scatter string concatenation. - -If the repository already has an `internal/archive` or archive-stage helper package, prefer extending that rather than creating a conflicting package layout. - -## CLI output - -`narratio restore` should print a concise operator summary. - -Example successful output: - -```text -Restored session archive for sample-campaign/2026-04-04 -Remote run: 20260504T031500Z-a1b2c3 -Downloaded: 4 -Skipped unchanged: 2 -Conflicts: 0 -``` - -Example dry run: - -```text -Restore plan for sample-campaign/2026-04-04 -Remote run: 20260504T031500Z-a1b2c3 -Would download: transcripts/processed.json -Would download: transcripts/trimmed.json -Would skip unchanged: artifacts/session_recap.md -``` - -Example conflict: - -```text -restore conflict: local artifacts/session_recap.md differs from remote archive; rerun with --force to overwrite -``` - -Do not print transcript or artifact content. - -## Error behavior - -Fail clearly when: - -- storage backend is not configured; -- S3 bucket/config is missing or invalid; -- remote current commit marker is missing; -- remote current manifest is missing; -- remote manifest is invalid; -- remote manifest does not match requested campaign/session; -- local file differs from remote and `--force` is not set; -- a required remote object download fails; -- a local path would escape the session workspace; -- a remote key maps to an unsafe local path. - -Skip or report non-fatal conditions when: - -- optional audio restore finds no archived audio; -- an included prefix has no objects; -- a local file already matches the remote file. - -## Path safety - -Every restored file must map to a safe path under the session root. - -Validation rules: - -- local restore paths must be relative to the session root; -- reject absolute paths; -- reject `..` traversal; -- reject paths that escape through symlinks if the codebase has symlink-safe path checks; -- do not restore remote keys directly without mapping/classification; -- do not mirror arbitrary remote keys. - -## Testing plan - -Add focused unit tests. Do not require live S3. - -### CLI tests - -Add or update `internal/app` command tests for: - -- `narratio restore --help`; -- restore accepts `--config`, `--session`, and `--session-id`; -- restore accepts `--dry-run`; -- restore accepts `--force`; -- restore accepts `--include-audio`; -- restore fails when storage is not configured; -- restore does not run pipeline stages. - -### Restore planner tests - -Test: - -- missing `current/run_id.txt` fails; -- missing `current/manifest.json` fails; -- invalid manifest fails; -- wrong campaign/session manifest fails; -- default scope includes manifest/transcripts/artifacts; -- default scope excludes audio/logs/reports/config/runs; -- `--include-audio` includes durable audio; -- run-local keys are excluded; -- keys are sorted deterministically; -- unsafe remote-to-local paths are rejected. - -### Conflict policy tests - -Test: - -- absent local file downloads; -- matching local file skips; -- differing local file conflicts by default; -- `--force` overwrites conflicts; -- `--dry-run` writes nothing; -- partial failure does not corrupt an existing local manifest. - -### Storage/fake tests - -Use fake storage to simulate: - -- object listing; -- object download; -- missing objects; -- download failures; -- metadata/ETag behavior. - -### Workspace/lock tests - -Test: - -- session layout is created before restore; -- session lock conflict fails; -- restored files land under the expected campaign/session workspace; -- no files are written outside the session root. - -### Follow-up command workflow test - -Add at least one test that simulates: - -```bash -narratio restore --session-id 2026-04-04 -narratio run-stage --force analyze -``` - -The test does not need to run real Scriptorium. Use existing fake/stub behavior to verify that restored transcripts and manifest state are sufficient for analyze-stage input resolution. - -## Documentation updates when implemented - -When the feature is implemented, update current-behavior docs: - -```text -docs/cli.md -docs/operations.md -docs/internal/storage.md or docs/internal/archive/restore.md -``` - -If the documentation set does not yet have an internal restore document, add one consistent with the existing internal-doc style: - -```text -docs/internal/command-restore.md -``` - -or: - -```text -docs/internal/archive-restore.md -``` - -Do not document future `narratio run --restore` behavior outside `docs/roadmap/` until implemented. - -## Implementation phases - -### Phase 1: Audit existing archive and path helpers (completed) - -Before coding behavior, inspect: - -```text -internal/app/ -internal/stage/archive* -internal/adapters/storage/ -internal/artifacts/ -internal/manifest/ -docs/internal/stage-archive.md, if present -``` - -Determine: - -- exact remote archive key layout; -- how root prefix/campaign/session are modeled; -- how current commit marker keys are built; -- how current manifest is uploaded; -- where promoted outputs are uploaded; -- whether helper functions already exist for remote archive keys; -- whether local workspace path helpers can safely map restore destinations. - -Deliverable: - -- small code comments or internal helper selection; -- no large behavior change yet unless required by tests. - -### Phase 2: Add CLI surface and command wiring (completed) - -Add `narratio restore` command parsing. - -Wire flags: - -```text ---config ---session ---session-id ---dry-run ---force ---include-audio -``` - -Use the existing config/session load path where practical. - -Deliverable: - -- command exists; -- help output is sensible; -- command validates basic inputs; -- command returns a clear “not yet implemented” or calls an empty planner if phased commits are desired; -- CLI tests pass. - -### Phase 3: Implement remote current-state discovery (completed) - -Add restore code that: - -- creates an object store from resolved config; -- builds remote current marker key; -- reads `current/run_id.txt`; -- reads/downloads `current/manifest.json`; -- validates manifest identity; -- returns remote current-state metadata. - -Deliverable: - -- fake-storage tests for current-state discovery; -- no local file writes beyond temporary files. - -### Phase 4: Implement restore planning (completed) - -Build deterministic restore plans for default scope and `--include-audio`. - -Deliverable: - -- plan lists manifest, transcript, artifact files; -- plan excludes run-local data; -- plan detects local same/conflict/missing states; -- dry-run output works; -- no real file overwrite yet except temp comparisons as needed. - -### Phase 5: Implement restore execution (completed) - -Execute the plan safely: - -- create directories; -- download to temporary files; -- validate content where practical; -- atomically install files; -- enforce default conflict failure; -- support `--force`; -- preserve existing manifest unless safe to replace. - -Deliverable: - -- restore works end-to-end against fake storage; -- failures are clear and do not corrupt existing local manifest. - -### Phase 6: Add restore report and operator summary (completed) - -Add concise stdout summary and optional JSON restore report if consistent with project diagnostics. - -Deliverable: - -- user-friendly output; -- durable diagnostic report if implemented; -- no content leakage. - -### Phase 7: Workflow integration test (completed) - -Add a test for restoring a previous session and then forcing `analyze`. - -Deliverable: - -- restored manifest/transcripts/artifacts are sufficient for analyze input resolution; -- no upstream stages rerun; -- no reliance on live subprocesses or S3. - -### Phase 8: Documentation update (completed) - -Once implemented, update current-behavior docs and internal command docs. - -Also leave future `narratio run --restore` in roadmap only. - -## Definition of done - -The feature is complete when: - -- `narratio restore` exists and is documented. -- It uses the same config/session discovery semantics as other commands where practical. -- It requires configured remote storage. -- It restores only from a committed current archive state. -- It restores the current manifest, transcripts, and artifacts by default. -- It restores audio only with `--include-audio`. -- It excludes run-local sandboxes. -- It fails on local/remote conflicts by default. -- `--force` overwrites conflicts. -- `--dry-run` writes nothing. -- It uses fake storage in tests. -- It does not change `prepare` behavior. -- It does not implement `narratio run --restore`. -- It avoids AWS SDK leakage outside the storage adapter. -- It uses centralized path/key helpers rather than scattered string concatenation. -- `go test ./...` passes. - -## Suggested test commands - -Run focused tests first: - -```bash -go test ./internal/app -run TestExecute -v -go test ./internal/adapters/storage -v -go test ./internal/artifacts -v -go test ./internal/manifest -v -``` - -Then run the full suite: - -```bash -go test ./... -``` - -## Suggested commit message - -```text -Add restore subcommand roadmap -```