Update documentation to include a roadmap for previous session artifact support
This commit is contained in:
629
docs/roadmap/previous.md
Normal file
629
docs/roadmap/previous.md
Normal file
@@ -0,0 +1,629 @@
|
|||||||
|
# Roadmap: Previous-Session Artifacts
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Planned.
|
||||||
|
|
||||||
|
This roadmap describes the implementation strategy for first-class previous-session artifact support in Narratio. It belongs under `docs/roadmap/previous.md` until the feature is implemented. After implementation, current behavior should be documented in the appropriate user-facing and internal documentation files, and this roadmap should be removed or marked complete according to the documentation policy.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Narratio should support using artifacts from a previous session as inputs to artifacts generated for the current session.
|
||||||
|
|
||||||
|
The primary use case is session recap continuity: a current session recap should be able to consume the previous session recap. The design should support arbitrary previous-session artifacts from the start, not just `session_recap`.
|
||||||
|
|
||||||
|
The canonical source syntax should be:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
source: narratio.previous_session.artifact.<artifact_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
source: narratio.previous_session.artifact.session_recap
|
||||||
|
```
|
||||||
|
|
||||||
|
Previous-session artifacts are materialized during the `prepare` stage into a current-session top-level `previous/` directory. Downstream stages consume only the local `previous/` copies. The S3 backend is authoritative for previous-session state.
|
||||||
|
|
||||||
|
## Design Decisions
|
||||||
|
|
||||||
|
### 1. Add `previous_session_id` to `session.yml`
|
||||||
|
|
||||||
|
Add an optional top-level session key:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
session_id: "{{ session_id }}"
|
||||||
|
previous_session_id: "{{ previous_session_id }}"
|
||||||
|
campaign: sample-campaign
|
||||||
|
```
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- `previous_session_id` is optional.
|
||||||
|
- If present, it identifies the previous session within the same campaign.
|
||||||
|
- It must not equal `session_id`.
|
||||||
|
- It should use the same validation rules as `session_id`.
|
||||||
|
- It may be supplied through session templating.
|
||||||
|
- Add a CLI/template value such as `--previous-session-id <id>` if required by the existing session templating implementation.
|
||||||
|
- If a template placeholder for `previous_session_id` is present and no value is supplied, loading should fail with a clear unresolved-template error.
|
||||||
|
|
||||||
|
### 2. Use canonical previous-session artifact source IDs
|
||||||
|
|
||||||
|
Support this source pattern in Scriptorium artifact input definitions:
|
||||||
|
|
||||||
|
```text
|
||||||
|
narratio.previous_session.artifact.<artifact_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
inputs:
|
||||||
|
previous_recap:
|
||||||
|
source: narratio.previous_session.artifact.session_recap
|
||||||
|
required: false
|
||||||
|
|
||||||
|
previous_quest_log:
|
||||||
|
source: narratio.previous_session.artifact.quest_log
|
||||||
|
required: true
|
||||||
|
```
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- `<artifact_name>` must be a valid configured artifact key.
|
||||||
|
- Use the same artifact key validation rules as current-session runtime artifacts.
|
||||||
|
- Do not special-case `session_recap`.
|
||||||
|
- Do not limit implementation to a fixed list of previous artifacts.
|
||||||
|
|
||||||
|
### 3. Add a top-level `previous/` workspace directory
|
||||||
|
|
||||||
|
Extend the session workspace layout with:
|
||||||
|
|
||||||
|
```text
|
||||||
|
previous/
|
||||||
|
manifest.json
|
||||||
|
artifacts/
|
||||||
|
<artifact outputs copied from the previous session>
|
||||||
|
```
|
||||||
|
|
||||||
|
The `previous/` directory is current-session state. It is a prepared input cache, not a full mirror of the previous session workspace.
|
||||||
|
|
||||||
|
Conceptually:
|
||||||
|
|
||||||
|
```text
|
||||||
|
work/<campaign>/<current_session_id>/
|
||||||
|
previous/
|
||||||
|
manifest.json
|
||||||
|
artifacts/
|
||||||
|
session_recap.md
|
||||||
|
quest_log.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The current session should not read directly from the previous session's local workspace during ordinary operation.
|
||||||
|
|
||||||
|
### 4. S3 is authoritative for previous-session state
|
||||||
|
|
||||||
|
For this initial implementation, previous-session artifacts should be downloaded from the configured S3 backend.
|
||||||
|
|
||||||
|
Do not compare local and remote copies.
|
||||||
|
|
||||||
|
Do not prefer local previous-session workspace state.
|
||||||
|
|
||||||
|
Do not implement a `--local` override in this roadmap. That can be considered later.
|
||||||
|
|
||||||
|
The simplified stage behavior is:
|
||||||
|
|
||||||
|
1. If the local manifest indicates `prepare` already succeeded and `--force` is not supplied, the runner skips `prepare`. No previous-session download occurs.
|
||||||
|
2. If `prepare` has not succeeded, `prepare` runs and downloads referenced previous-session artifacts from S3.
|
||||||
|
3. If `prepare` previously succeeded but `--force` is supplied, `prepare` runs again and overwrites local `previous/` state from S3.
|
||||||
|
|
||||||
|
### 5. Materialize only referenced previous-session artifacts
|
||||||
|
|
||||||
|
During `prepare`, scan the current resolved pipeline/session configuration for Scriptorium artifact inputs whose source matches:
|
||||||
|
|
||||||
|
```text
|
||||||
|
narratio.previous_session.artifact.<artifact_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
Only those referenced previous-session artifacts need to be downloaded.
|
||||||
|
|
||||||
|
Do not blindly download every previous-session artifact.
|
||||||
|
|
||||||
|
If no previous-session artifact sources are referenced, `prepare` should not require `previous_session_id` and should not touch `previous/`.
|
||||||
|
|
||||||
|
### 6. Preserve stage boundaries
|
||||||
|
|
||||||
|
`prepare` owns previous-session artifact materialization because these files are inputs to later stages.
|
||||||
|
|
||||||
|
`analyze` should not talk to S3.
|
||||||
|
|
||||||
|
The Scriptorium adapter should not know about previous sessions.
|
||||||
|
|
||||||
|
The storage adapter should not infer campaign, session, run, or root-prefix semantics. Callers should continue to provide explicit bucket-relative keys.
|
||||||
|
|
||||||
|
### 7. Archive and restore `previous/`
|
||||||
|
|
||||||
|
After implementation:
|
||||||
|
|
||||||
|
- `archive` should include `previous/` as durable current-session prepared input state.
|
||||||
|
- `restore` should restore `previous/` along with the rest of the durable session state it already restores.
|
||||||
|
- `previous/` should not be treated as current-session generated artifacts.
|
||||||
|
- `previous/` entries should be recorded as inputs/provenance, not as outputs produced by the current session.
|
||||||
|
|
||||||
|
## Target User Workflow
|
||||||
|
|
||||||
|
A typical session config:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
session_id: "{{ session_id }}"
|
||||||
|
previous_session_id: "{{ previous_session_id }}"
|
||||||
|
campaign: sample-campaign
|
||||||
|
|
||||||
|
inputs:
|
||||||
|
audio_dir: ./audio
|
||||||
|
speakers_file: ./examples/speakers.yml
|
||||||
|
autocorrect_file: ./examples/autocorrect.yml
|
||||||
|
glossary_file: ./examples/glossary.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
A typical artifact config:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
scriptorium:
|
||||||
|
artifacts:
|
||||||
|
session_recap:
|
||||||
|
enabled: true
|
||||||
|
prompt_id: dnd_session.session_recap
|
||||||
|
output_path: artifacts/session_recap.md
|
||||||
|
inputs:
|
||||||
|
transcript:
|
||||||
|
source: narratio.transcript.trimmed
|
||||||
|
required: true
|
||||||
|
previous_recap:
|
||||||
|
source: narratio.previous_session.artifact.session_recap
|
||||||
|
required: false
|
||||||
|
```
|
||||||
|
|
||||||
|
Typical command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run --session-id 2026-04-11 --previous-session-id 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected behavior:
|
||||||
|
|
||||||
|
1. `prepare` sees a referenced previous-session artifact: `session_recap`.
|
||||||
|
2. `prepare` downloads the previous session's `narratio.artifact.session_recap` from S3.
|
||||||
|
3. `prepare` writes it under the current session workspace, for example `previous/artifacts/session_recap.md`.
|
||||||
|
4. `prepare` records provenance in the current session manifest.
|
||||||
|
5. `analyze` resolves `narratio.previous_session.artifact.session_recap` from the local `previous/` directory.
|
||||||
|
6. Scriptorium receives the previous recap as a normal input file.
|
||||||
|
|
||||||
|
## Implementation Plan
|
||||||
|
|
||||||
|
### Phase 1: Session config and templating
|
||||||
|
|
||||||
|
Update session configuration structs to include:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
previous_session_id: ""
|
||||||
|
```
|
||||||
|
|
||||||
|
Implementation steps:
|
||||||
|
|
||||||
|
1. Add `PreviousSessionID` or equivalent to the session config type.
|
||||||
|
2. Add validation:
|
||||||
|
- optional;
|
||||||
|
- same format constraints as `session_id`;
|
||||||
|
- must not equal `session_id`.
|
||||||
|
3. Extend session templating support to include:
|
||||||
|
- `{{previous_session_id}}`
|
||||||
|
- `{{ previous_session_id }}`
|
||||||
|
4. Add a CLI flag if required by current templating flow:
|
||||||
|
- `--previous-session-id <id>`
|
||||||
|
5. Ensure unresolved `previous_session_id` placeholders fail clearly.
|
||||||
|
6. Update config tests for:
|
||||||
|
- no previous session;
|
||||||
|
- valid previous session;
|
||||||
|
- previous session equal to current session;
|
||||||
|
- unresolved placeholder;
|
||||||
|
- CLI/template rendering.
|
||||||
|
|
||||||
|
Do not add future workflow flags in this phase.
|
||||||
|
|
||||||
|
### Phase 2: Workspace path helpers
|
||||||
|
|
||||||
|
Add centralized path helpers for current-session previous-state paths.
|
||||||
|
|
||||||
|
Suggested helpers:
|
||||||
|
|
||||||
|
```text
|
||||||
|
SessionPreviousDir()
|
||||||
|
SessionPreviousManifestPath()
|
||||||
|
SessionPreviousArtifactsDir()
|
||||||
|
SessionPreviousArtifactPath(name or relative output path)
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact names should match existing path-helper style.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- Do not construct `previous/` paths through scattered string concatenation.
|
||||||
|
- Keep paths session-relative where possible.
|
||||||
|
- Ensure workspace layout creation includes `previous/` only when appropriate, or creates it idempotently with the rest of the layout if simpler.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- path helper tests;
|
||||||
|
- workspace layout tests;
|
||||||
|
- ensure cleanup logic does not accidentally delete configured roots;
|
||||||
|
- ensure `previous/` is treated as session-durable state, not run-local state.
|
||||||
|
|
||||||
|
### Phase 3: Previous-session source parsing
|
||||||
|
|
||||||
|
Add parsing/recognition for:
|
||||||
|
|
||||||
|
```text
|
||||||
|
narratio.previous_session.artifact.<artifact_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
Implementation steps:
|
||||||
|
|
||||||
|
1. Add constants/helpers in the artifact/source parsing layer.
|
||||||
|
2. Validate artifact names using the same rules as current runtime artifact keys.
|
||||||
|
3. Add helpers such as:
|
||||||
|
- `IsPreviousSessionArtifactSource(source string) bool`
|
||||||
|
- `PreviousSessionArtifactName(source string) (string, bool)`
|
||||||
|
4. Ensure config validation accepts this source pattern.
|
||||||
|
5. Ensure invalid sources fail clearly.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- valid previous-session artifact source;
|
||||||
|
- invalid/missing artifact name;
|
||||||
|
- invalid artifact key characters;
|
||||||
|
- ordinary current-session sources still validate;
|
||||||
|
- unknown sources still fail.
|
||||||
|
|
||||||
|
### Phase 4: Scan configured artifacts for previous-session inputs
|
||||||
|
|
||||||
|
Add a helper that inspects resolved Scriptorium artifact definitions and returns the set of referenced previous-session artifact names.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- Scan enabled artifacts according to current artifact-enable semantics.
|
||||||
|
- Include all inputs whose source matches `narratio.previous_session.artifact.<name>`.
|
||||||
|
- Deduplicate artifact names.
|
||||||
|
- Sort results deterministically.
|
||||||
|
- Preserve required/optional information per reference.
|
||||||
|
- If the same previous artifact is referenced both required and optional, treat it as required.
|
||||||
|
|
||||||
|
Suggested output model:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type PreviousArtifactRequirement struct {
|
||||||
|
Name string
|
||||||
|
Required bool
|
||||||
|
Sources []string // optional diagnostics
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- no artifacts;
|
||||||
|
- no previous inputs;
|
||||||
|
- one optional previous input;
|
||||||
|
- one required previous input;
|
||||||
|
- duplicate references;
|
||||||
|
- required plus optional reference to the same artifact;
|
||||||
|
- deterministic ordering.
|
||||||
|
|
||||||
|
### Phase 5: Resolve previous-session archive keys
|
||||||
|
|
||||||
|
Implement a narrow service/helper used by `prepare` to resolve previous-session artifact files from S3.
|
||||||
|
|
||||||
|
Responsibilities:
|
||||||
|
|
||||||
|
1. Locate the previous session's current remote state.
|
||||||
|
2. Download the previous session manifest, or the minimum remote metadata needed to resolve artifact source IDs.
|
||||||
|
3. Resolve `narratio.artifact.<name>` inside the previous session's artifact catalog/manifest.
|
||||||
|
4. Download the resolved artifact into the current session's `previous/` directory.
|
||||||
|
5. Download/store the previous session manifest as `previous/manifest.json`.
|
||||||
|
6. Return provenance records for manifest input recording.
|
||||||
|
|
||||||
|
Important archive invariant:
|
||||||
|
|
||||||
|
- Remote current state must be based on the committed archive marker.
|
||||||
|
- Do not treat incomplete archive uploads as current state.
|
||||||
|
- Use the existing archive/current remote layout and commit-marker rules.
|
||||||
|
- `current/run_id.txt` is the final remote commit marker and should be respected when locating current remote state.
|
||||||
|
|
||||||
|
Do not put prefix semantics into the storage adapter. Compute explicit bucket-relative keys in app/stage/archive helper code, then call the storage adapter.
|
||||||
|
|
||||||
|
Required vs optional behavior:
|
||||||
|
|
||||||
|
- Required previous artifact missing from S3/current manifest: fail `prepare`.
|
||||||
|
- Optional previous artifact missing from S3/current manifest: continue without materializing that input.
|
||||||
|
- Previous session missing entirely:
|
||||||
|
- fail if any referenced previous artifact is required;
|
||||||
|
- continue if all referenced previous artifacts are optional.
|
||||||
|
- If previous_session_id is unset:
|
||||||
|
- fail if any referenced previous artifact is required;
|
||||||
|
- continue and omit all previous-session inputs if all are optional.
|
||||||
|
|
||||||
|
Validation behavior:
|
||||||
|
|
||||||
|
- Downloaded artifacts should pass the same validation rules as current-session artifacts where practical.
|
||||||
|
- Generic Scriptorium artifacts should at least be non-empty.
|
||||||
|
- Invalid required artifact: fail.
|
||||||
|
- Invalid optional artifact: prefer fail if the object exists but is invalid, because invalid archived data is usually an operator problem rather than absence.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- downloads previous manifest;
|
||||||
|
- downloads required previous artifact;
|
||||||
|
- skips missing optional previous artifact;
|
||||||
|
- fails missing required previous artifact;
|
||||||
|
- fails required previous artifact when previous_session_id is unset;
|
||||||
|
- optional previous artifact with no previous_session_id does not fail;
|
||||||
|
- respects current remote commit marker;
|
||||||
|
- does not use local previous-session workspace state;
|
||||||
|
- uses storage adapter with explicit keys.
|
||||||
|
|
||||||
|
### Phase 6: Integrate with `prepare`
|
||||||
|
|
||||||
|
Extend the `prepare` stage:
|
||||||
|
|
||||||
|
1. Run existing input materialization as before.
|
||||||
|
2. Detect previous-session artifact requirements.
|
||||||
|
3. If requirements exist, hydrate `previous/` from S3 according to the rules above.
|
||||||
|
4. Record hydrated previous artifacts in `manifest.Inputs`.
|
||||||
|
5. Preserve existing prepare outputs and provenance behavior.
|
||||||
|
|
||||||
|
Overwrite behavior:
|
||||||
|
|
||||||
|
- If `prepare` runs, it owns `previous/`.
|
||||||
|
- Before hydrating, clear the managed `previous/` directory, or clear the managed previous artifact paths.
|
||||||
|
- Prefer clearing the whole `previous/` directory if no other feature writes there.
|
||||||
|
- Under `--force`, this naturally overwrites local `previous/` state.
|
||||||
|
- Do not compare local and remote copies.
|
||||||
|
|
||||||
|
Skip behavior:
|
||||||
|
|
||||||
|
- Do not add stage-local skip logic.
|
||||||
|
- Runner-level skip remains authoritative.
|
||||||
|
- If the manifest says `prepare` succeeded and `--force` is not supplied, `prepare` does not run and no S3 downloads occur.
|
||||||
|
- If users add a new previous-session input after `prepare` already succeeded, they must rerun prepare with `--force`.
|
||||||
|
|
||||||
|
Error message requirement:
|
||||||
|
|
||||||
|
If an analyze-stage input cannot be resolved because `previous/` is missing or stale, the error should tell the operator to run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage --force prepare
|
||||||
|
```
|
||||||
|
|
||||||
|
or the appropriate existing CLI command shape.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- ordinary prepare without previous_session_id remains unchanged;
|
||||||
|
- prepare with optional previous artifact and no previous_session_id succeeds;
|
||||||
|
- prepare with required previous artifact and no previous_session_id fails;
|
||||||
|
- prepare with required previous artifact downloads to `previous/`;
|
||||||
|
- force prepare overwrites `previous/`;
|
||||||
|
- prepare records manifest inputs for previous artifacts;
|
||||||
|
- prepare skip behavior remains controlled by runner tests;
|
||||||
|
- no regression in S3 audio prepare behavior.
|
||||||
|
|
||||||
|
### Phase 7: Artifact resolver support
|
||||||
|
|
||||||
|
Update artifact resolution so Scriptorium inputs can resolve:
|
||||||
|
|
||||||
|
```text
|
||||||
|
narratio.previous_session.artifact.<artifact_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
from the current session's `previous/` directory.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- The resolver should not call S3.
|
||||||
|
- The resolver should not read the previous session's local workspace.
|
||||||
|
- The resolver should map the previous-session source ID to the local prepared copy under `previous/`.
|
||||||
|
- Resolution should use manifest input records when available.
|
||||||
|
- Fallback to the local `previous/` path may be allowed if consistent with existing resolver behavior, but manifest provenance should be preferred.
|
||||||
|
- Missing required input should fail with a clear prepare-oriented message.
|
||||||
|
- Optional missing input should be omitted.
|
||||||
|
|
||||||
|
Suggested provenance:
|
||||||
|
|
||||||
|
```text
|
||||||
|
previous_session.manifest.outputs
|
||||||
|
previous_session.archive.current
|
||||||
|
current_session.previous_cache
|
||||||
|
```
|
||||||
|
|
||||||
|
Use names that fit the existing manifest/resolver vocabulary.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- resolves prepared previous artifact through manifest input record;
|
||||||
|
- resolves or fails appropriately when only filesystem copy exists, depending on chosen fallback policy;
|
||||||
|
- missing optional previous artifact is omitted;
|
||||||
|
- missing required previous artifact fails clearly;
|
||||||
|
- current-session `narratio.artifact.<name>` behavior is unchanged.
|
||||||
|
|
||||||
|
### Phase 8: Analyze-stage integration
|
||||||
|
|
||||||
|
The analyze stage should require little or no special previous-session logic if the resolver is designed correctly.
|
||||||
|
|
||||||
|
Confirm:
|
||||||
|
|
||||||
|
- Scriptorium input resolution accepts previous-session source IDs.
|
||||||
|
- The Scriptorium adapter receives a normal local input path.
|
||||||
|
- Render-debug and run modes behave the same as for ordinary inputs.
|
||||||
|
- Stage metadata includes useful input provenance if current structures support it.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- configured artifact receives previous recap input;
|
||||||
|
- optional previous recap omitted when not prepared;
|
||||||
|
- required previous recap fails when not prepared;
|
||||||
|
- render-debug path works with previous-session inputs;
|
||||||
|
- no S3 calls occur from analyze.
|
||||||
|
|
||||||
|
### Phase 9: Archive `previous/`
|
||||||
|
|
||||||
|
Update archive behavior so the current session's durable `previous/` directory is uploaded/preserved.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- `previous/` is current-session input/provenance state.
|
||||||
|
- It is not a generated current-session artifact.
|
||||||
|
- Archive it with the current session's durable state, alongside other durable session files according to the current archive layout.
|
||||||
|
- Preserve existing archive commit ordering.
|
||||||
|
- Do not make `previous/` upload the final commit marker.
|
||||||
|
- Do not treat missing `previous/` as an error when no previous-session inputs were prepared.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- archive includes `previous/manifest.json` and prepared previous artifacts when present;
|
||||||
|
- archive omits or tolerates absent `previous/` when unused;
|
||||||
|
- archive commit ordering remains valid;
|
||||||
|
- cleanup behavior does not delete durable `previous/` before archive commit.
|
||||||
|
|
||||||
|
### Phase 10: Restore `previous/`
|
||||||
|
|
||||||
|
Update `narratio restore` so it restores the current session's archived `previous/` directory.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- Restore `previous/` as durable current-session state.
|
||||||
|
- Do not infer or restore the previous session workspace merely because `previous_session_id` exists.
|
||||||
|
- Do not redownload previous-session artifacts from the previous session archive during restore; restore the current session's archived `previous/` cache.
|
||||||
|
- Respect existing restore flags and overwrite behavior.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- restore downloads `previous/` when present;
|
||||||
|
- restore succeeds when `previous/` is absent;
|
||||||
|
- restore with force overwrites local `previous/` according to existing restore semantics;
|
||||||
|
- restored current session can run analyze using `previous/` without needing previous session archive access.
|
||||||
|
|
||||||
|
### Phase 11: Documentation updates after implementation
|
||||||
|
|
||||||
|
After implementation, update current-behavior docs. Do not document implemented behavior only in this roadmap.
|
||||||
|
|
||||||
|
Likely files:
|
||||||
|
|
||||||
|
- `docs/config.md`
|
||||||
|
- `docs/cli.md`
|
||||||
|
- `docs/operations.md`
|
||||||
|
- `docs/internal/stage-prepare.md`
|
||||||
|
- `docs/internal/artifacts.md`
|
||||||
|
- `docs/internal/storage.md` only if storage contracts change
|
||||||
|
- `docs/internal/workspace.md`
|
||||||
|
- relevant maintained examples under `examples/`
|
||||||
|
|
||||||
|
Docs should explain:
|
||||||
|
|
||||||
|
- `session.previous_session_id`;
|
||||||
|
- `--previous-session-id` if added;
|
||||||
|
- `narratio.previous_session.artifact.<artifact_name>`;
|
||||||
|
- `previous/` workspace directory;
|
||||||
|
- S3-authoritative previous-session behavior;
|
||||||
|
- required vs optional previous-session artifact handling;
|
||||||
|
- when to run `narratio run-stage --force prepare`.
|
||||||
|
|
||||||
|
Do not document future `--local` support as current behavior.
|
||||||
|
|
||||||
|
## Future Work
|
||||||
|
|
||||||
|
These items are explicitly out of scope for the initial implementation.
|
||||||
|
|
||||||
|
### `--local` previous-session source
|
||||||
|
|
||||||
|
A future flag may allow `prepare` to copy previous-session artifacts from a local previous session workspace instead of S3.
|
||||||
|
|
||||||
|
Possible future command shape:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage --force prepare --local
|
||||||
|
```
|
||||||
|
|
||||||
|
or a more specific flag such as:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage --force prepare --previous-source local
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not implement this now.
|
||||||
|
|
||||||
|
### `narratio run --restore`
|
||||||
|
|
||||||
|
A future flag may run `narratio restore` before starting the regular pipeline:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run --restore --session-id 2026-04-11
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not implement this as part of previous-session artifact support unless it already exists and only requires documentation.
|
||||||
|
|
||||||
|
### Multi-previous-session support
|
||||||
|
|
||||||
|
A future design may support more than one previous/reference session.
|
||||||
|
|
||||||
|
Do not implement this now.
|
||||||
|
|
||||||
|
### Previous-session artifact version pinning
|
||||||
|
|
||||||
|
A future design may pin previous-session artifact inputs to a specific previous run ID or checksum.
|
||||||
|
|
||||||
|
Do not implement this now.
|
||||||
|
|
||||||
|
## Test Plan Summary
|
||||||
|
|
||||||
|
Run at least:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/config -v
|
||||||
|
go test ./internal/artifacts -v
|
||||||
|
go test ./internal/stage -run Prepare -v
|
||||||
|
go test ./internal/stage -run Analyze -v
|
||||||
|
go test ./internal/app -run TestExecute -v
|
||||||
|
go test ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
Add focused tests for:
|
||||||
|
|
||||||
|
- session config and templating;
|
||||||
|
- previous-session source parsing;
|
||||||
|
- previous artifact requirement scanning;
|
||||||
|
- S3-backed previous artifact download;
|
||||||
|
- prepare skip/force semantics;
|
||||||
|
- manifest input provenance;
|
||||||
|
- resolver behavior;
|
||||||
|
- analyze integration;
|
||||||
|
- archive/restore `previous/` persistence;
|
||||||
|
- examples load/validate.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
The feature is complete when:
|
||||||
|
|
||||||
|
1. `session.yml` supports optional `previous_session_id`.
|
||||||
|
2. Session templating supports `previous_session_id`.
|
||||||
|
3. Scriptorium artifact inputs accept `narratio.previous_session.artifact.<artifact_name>`.
|
||||||
|
4. `prepare` scans enabled Scriptorium artifacts for previous-session artifact inputs.
|
||||||
|
5. `prepare` downloads referenced previous-session artifacts from S3 into `previous/`.
|
||||||
|
6. Required/optional behavior is correct.
|
||||||
|
7. `prepare` uses stage-level skip/force behavior and does not compare local and remote copies.
|
||||||
|
8. `analyze` resolves previous-session artifacts only from local prepared `previous/` state.
|
||||||
|
9. `archive` persists `previous/`.
|
||||||
|
10. `restore` restores `previous/`.
|
||||||
|
11. Tests cover config, prepare, resolver, analyze, archive, and restore behavior.
|
||||||
|
12. Current-behavior docs and maintained examples are updated after implementation.
|
||||||
|
13. No storage adapter implementation infers campaign/session/root-prefix semantics.
|
||||||
|
14. No previous-session S3 logic is added to `analyze` or the Scriptorium adapter.
|
||||||
@@ -1,715 +0,0 @@
|
|||||||
# Roadmap: `narratio restore` Subcommand
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
Implemented through Step 8. This document remains as roadmap and design history for the restore feature, and as the home for future restore-related ideas (for example `run --restore`).
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Add a new `narratio restore` subcommand that hydrates a local session workspace from the current committed remote archive state.
|
|
||||||
|
|
||||||
The primary operator workflow is:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio restore --session-id 2026-04-04
|
|
||||||
narratio run-stage --force analyze
|
|
||||||
```
|
|
||||||
|
|
||||||
This should allow a new machine with no local workspace state to restore the durable session manifest, transcripts, and generated artifacts from S3, then generate new Scriptorium artifacts without re-running transcription, merge, normalize, polish, or trim.
|
|
||||||
|
|
||||||
This is intentionally a separate command. Do not fold this behavior into the `prepare` stage. The existing `prepare` stage should remain focused on materializing configured local/S3 inputs for a pipeline run.
|
|
||||||
|
|
||||||
## Goals
|
|
||||||
|
|
||||||
- Add a first-class `narratio restore` command.
|
|
||||||
- Restore the current committed remote session state into the canonical local session workspace.
|
|
||||||
- Use the existing object storage adapter boundary.
|
|
||||||
- Preserve archive commit semantics: only restore from a remote state that has a valid current commit marker.
|
|
||||||
- Restore durable session-level outputs needed for downstream stages, especially `analyze`.
|
|
||||||
- Provide safe conflict behavior by default.
|
|
||||||
- Support `--dry-run`, `--force`, and `--include-audio`.
|
|
||||||
- Keep the implementation explicit, testable, and narrow.
|
|
||||||
|
|
||||||
## Non-goals
|
|
||||||
|
|
||||||
- Do not make `restore` a pipeline stage.
|
|
||||||
- Do not change the `prepare` stage behavior as part of this work.
|
|
||||||
- Do not add implicit restore behavior to `narratio run` in this implementation.
|
|
||||||
- Do not restore historical run-local sandboxes by default.
|
|
||||||
- Do not implement a generic remote synchronization engine.
|
|
||||||
- Do not implement bidirectional sync.
|
|
||||||
- Do not delete local files merely because they are absent remotely.
|
|
||||||
- Do not merge remote and local manifests in the first implementation.
|
|
||||||
- Do not require live S3 for the ordinary unit test suite.
|
|
||||||
|
|
||||||
## Future work explicitly out of scope
|
|
||||||
|
|
||||||
A future change may add:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio run --restore
|
|
||||||
```
|
|
||||||
|
|
||||||
That future flag should run `narratio restore` before starting the normal pipeline. Mention this as future work in roadmap/docs if useful, but do not implement it now.
|
|
||||||
|
|
||||||
## Existing architecture to preserve
|
|
||||||
|
|
||||||
### `prepare` remains input materialization
|
|
||||||
|
|
||||||
The `prepare` stage currently materializes required session inputs into canonical local workspace paths and records input provenance. It owns local copying/materialization of config and audio inputs, including S3 audio download when `session.inputs.audio_s3.prefix` is configured. It does not own transcript generation/processing or archive publish behavior.
|
|
||||||
|
|
||||||
`restore` should not be implemented by expanding `prepare`. It should be an app-level command that reuses shared helpers where appropriate.
|
|
||||||
|
|
||||||
### Workspace model
|
|
||||||
|
|
||||||
The local durable session workspace is campaign-aware:
|
|
||||||
|
|
||||||
```text
|
|
||||||
{workspace.root}/work/{campaign}/{session_id}/
|
|
||||||
```
|
|
||||||
|
|
||||||
It contains durable session paths such as:
|
|
||||||
|
|
||||||
```text
|
|
||||||
manifest.json
|
|
||||||
inputs/
|
|
||||||
audio/
|
|
||||||
transcripts/
|
|
||||||
artifacts/
|
|
||||||
reports/
|
|
||||||
logs/
|
|
||||||
config/
|
|
||||||
current/
|
|
||||||
runs/
|
|
||||||
```
|
|
||||||
|
|
||||||
Run-local sandboxes live below:
|
|
||||||
|
|
||||||
```text
|
|
||||||
runs/{run_id}/
|
|
||||||
```
|
|
||||||
|
|
||||||
Restore should target durable session-level paths, not old run-local stage sandboxes.
|
|
||||||
|
|
||||||
### Storage boundary
|
|
||||||
|
|
||||||
The storage adapter owns object-store primitives only: `List`, `Download`, `Upload`, and `Exists`.
|
|
||||||
|
|
||||||
The storage adapter must not infer root prefixes, campaign names, session IDs, run IDs, or archive layout. Restore code must construct full bucket-relative keys before calling storage.
|
|
||||||
|
|
||||||
### Archive commit boundary
|
|
||||||
|
|
||||||
A remote run is current only after the archive stage has uploaded the run record, promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
|
||||||
|
|
||||||
`current/run_id.txt` is the final remote commit marker and must be written last.
|
|
||||||
|
|
||||||
Restore must not treat incomplete, skipped, failed, or uncommitted archive attempts as current remote state.
|
|
||||||
|
|
||||||
## User-facing command
|
|
||||||
|
|
||||||
Add:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio restore [flags]
|
|
||||||
```
|
|
||||||
|
|
||||||
The command should use the same configuration/session discovery conventions as `run`, `plan`, `resume`, and `run-stage` where practical:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio restore --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
Required effective inputs:
|
|
||||||
|
|
||||||
- resolved pipeline config;
|
|
||||||
- resolved session config;
|
|
||||||
- `session.campaign`;
|
|
||||||
- `session.session_id`;
|
|
||||||
- configured remote storage backend.
|
|
||||||
|
|
||||||
Supported flags:
|
|
||||||
|
|
||||||
```text
|
|
||||||
--config <path> Existing pipeline config path behavior.
|
|
||||||
--session <path> Existing session config path behavior.
|
|
||||||
--session-id <value> Existing session template behavior.
|
|
||||||
--dry-run Plan restore actions without writing local files.
|
|
||||||
--force Overwrite conflicting local files with remote files.
|
|
||||||
--include-audio Include archived session-level audio files.
|
|
||||||
```
|
|
||||||
|
|
||||||
Do not add `--restore` to `run` in this implementation.
|
|
||||||
|
|
||||||
## Default restore scope
|
|
||||||
|
|
||||||
By default, restore:
|
|
||||||
|
|
||||||
1. Validates and reads the current remote commit marker.
|
|
||||||
2. Downloads the current remote manifest into the local session manifest path.
|
|
||||||
3. Downloads durable transcript files.
|
|
||||||
4. Downloads durable generated artifact files.
|
|
||||||
|
|
||||||
Default included remote/local durable paths:
|
|
||||||
|
|
||||||
```text
|
|
||||||
manifest.json from remote current manifest
|
|
||||||
transcripts/**
|
|
||||||
artifacts/**
|
|
||||||
```
|
|
||||||
|
|
||||||
Default excluded paths:
|
|
||||||
|
|
||||||
```text
|
|
||||||
audio/** unless --include-audio is passed
|
|
||||||
runs/** always excluded for this implementation
|
|
||||||
logs/** excluded for this implementation
|
|
||||||
reports/** excluded for this implementation unless needed for current manifest validation
|
|
||||||
config/** excluded for this implementation
|
|
||||||
inputs/** excluded for this implementation
|
|
||||||
current/** remote control metadata only; do not mirror blindly
|
|
||||||
```
|
|
||||||
|
|
||||||
If the existing archive implementation stores promoted files in a different remote layout, use the existing archive/path helpers and current archive semantics rather than inventing a parallel layout.
|
|
||||||
|
|
||||||
## Remote state discovery
|
|
||||||
|
|
||||||
Implement restore around the current committed archive state.
|
|
||||||
|
|
||||||
Expected algorithm:
|
|
||||||
|
|
||||||
1. Resolve pipeline/session config.
|
|
||||||
2. Ensure storage is configured.
|
|
||||||
3. Ensure local workspace layout exists.
|
|
||||||
4. Acquire the session lock.
|
|
||||||
5. Build the remote session archive prefix using the same helpers/policy used by archive code.
|
|
||||||
6. Check for the remote `current/run_id.txt` commit marker.
|
|
||||||
7. Read the committed run ID.
|
|
||||||
8. Download `current/manifest.json` to a temporary file.
|
|
||||||
9. Validate that the manifest is parseable and belongs to the requested campaign/session.
|
|
||||||
10. Build a restore plan from the committed remote state.
|
|
||||||
11. Execute the restore plan unless `--dry-run` is set.
|
|
||||||
12. Emit a concise summary.
|
|
||||||
|
|
||||||
Important: `current/run_id.txt` is the commit marker. Do not restore from a remote session prefix merely because files exist under `transcripts/` or `artifacts/`.
|
|
||||||
|
|
||||||
## Restore planning
|
|
||||||
|
|
||||||
Create a planning layer before writing files.
|
|
||||||
|
|
||||||
A restore plan entry should include at least:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type RestoreAction struct {
|
|
||||||
Kind RestoreActionKind
|
|
||||||
RemoteKey string
|
|
||||||
LocalPath string
|
|
||||||
Size int64
|
|
||||||
ETag string
|
|
||||||
ExistsLocal bool
|
|
||||||
SameLocal bool
|
|
||||||
Conflict bool
|
|
||||||
Reason string
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Suggested action kinds:
|
|
||||||
|
|
||||||
```text
|
|
||||||
download
|
|
||||||
skip_same
|
|
||||||
skip_missing_optional
|
|
||||||
conflict
|
|
||||||
```
|
|
||||||
|
|
||||||
The restore planner should be deterministic:
|
|
||||||
|
|
||||||
- sort remote objects by key;
|
|
||||||
- sort planned actions by local path or stable restore priority;
|
|
||||||
- write/report stable output for tests.
|
|
||||||
|
|
||||||
## Conflict and overwrite policy
|
|
||||||
|
|
||||||
Default behavior should be safe.
|
|
||||||
|
|
||||||
For each planned file:
|
|
||||||
|
|
||||||
```text
|
|
||||||
local absent:
|
|
||||||
download
|
|
||||||
|
|
||||||
local present and same as remote:
|
|
||||||
skip
|
|
||||||
|
|
||||||
local present and different:
|
|
||||||
conflict; fail restore unless --force is set
|
|
||||||
|
|
||||||
--force:
|
|
||||||
overwrite local conflicting files with remote versions
|
|
||||||
|
|
||||||
--dry-run:
|
|
||||||
do not write any files; report what would happen
|
|
||||||
```
|
|
||||||
|
|
||||||
The first implementation may use size and checksum/hash comparison where available. If remote ETag cannot be treated as a content hash, compare by downloading to a temporary file and hashing locally before deciding whether a local file is the same. Prefer correctness over assuming provider-specific ETag semantics.
|
|
||||||
|
|
||||||
Do not delete local files that are not present remotely.
|
|
||||||
|
|
||||||
## File writing and transactionality
|
|
||||||
|
|
||||||
Restore should avoid partial writes.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- download each remote object to a temporary file under the session workspace or OS temp dir;
|
|
||||||
- validate downloaded content where possible before replacing local files;
|
|
||||||
- create parent directories as needed;
|
|
||||||
- atomically rename/copy into place only after successful download;
|
|
||||||
- do not overwrite local files unless `--force` is set;
|
|
||||||
- if a later file fails, preserve already-restored files but return a failure summary;
|
|
||||||
- never corrupt an existing local manifest on failed manifest download/parse.
|
|
||||||
|
|
||||||
Manifest restore is especially sensitive:
|
|
||||||
|
|
||||||
- download remote `current/manifest.json` to a temporary file;
|
|
||||||
- parse and validate it;
|
|
||||||
- if no local manifest exists, install it;
|
|
||||||
- if a local manifest exists and is equivalent, skip;
|
|
||||||
- if a local manifest exists and differs, fail unless `--force` is set;
|
|
||||||
- with `--force`, replace the local manifest with the remote manifest after validation;
|
|
||||||
- do not attempt a manifest merge in the initial implementation.
|
|
||||||
|
|
||||||
## Manifest semantics
|
|
||||||
|
|
||||||
`restore` is not a pipeline run and should not mark stages as running/succeeded/failed.
|
|
||||||
|
|
||||||
The restored remote manifest becomes the local session manifest. That is what allows a subsequent command such as:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio run-stage --force analyze
|
|
||||||
```
|
|
||||||
|
|
||||||
to see existing upstream stage state and canonical durable outputs.
|
|
||||||
|
|
||||||
Do not create a new run manifest for `restore`.
|
|
||||||
|
|
||||||
It is acceptable to write a restore diagnostic report outside the manifest, for example:
|
|
||||||
|
|
||||||
```text
|
|
||||||
reports/restore-latest.json
|
|
||||||
```
|
|
||||||
|
|
||||||
or a timestamped report, if that pattern fits the existing codebase. The report must not contain secrets.
|
|
||||||
|
|
||||||
## Local workspace locking
|
|
||||||
|
|
||||||
`restore` should acquire the same session lock used by ordinary pipeline operations before modifying session workspace state.
|
|
||||||
|
|
||||||
If the lock is held, fail fast with the same lock-conflict behavior used elsewhere.
|
|
||||||
|
|
||||||
`--dry-run` may still acquire the lock for consistency, but it is acceptable to avoid the lock if the codebase already has a clear read-only command pattern. Prefer safety and simplicity.
|
|
||||||
|
|
||||||
## Audio behavior
|
|
||||||
|
|
||||||
By default, do not restore audio.
|
|
||||||
|
|
||||||
If `--include-audio` is passed:
|
|
||||||
|
|
||||||
- restore archived durable session-level audio files only;
|
|
||||||
- do not use run-scoped spool paths;
|
|
||||||
- do not mutate or delete spool state;
|
|
||||||
- do not infer original `session.inputs.audio_s3.prefix` behavior;
|
|
||||||
- respect the same conflict/force/dry-run behavior used for transcripts/artifacts.
|
|
||||||
|
|
||||||
If the archive does not contain durable audio files, `--include-audio` should report that no archived audio was found rather than failing, unless the final implementation chooses to treat explicit audio restore as required. Prefer non-failure for absent archived audio unless tests or existing archive semantics suggest otherwise.
|
|
||||||
|
|
||||||
## Remote object selection
|
|
||||||
|
|
||||||
Prefer using manifest/artifact metadata when it reliably identifies durable outputs.
|
|
||||||
|
|
||||||
Also support listing committed durable archive prefixes so restore can retrieve all top-level session artifacts that may not yet be fully represented in manifest metadata.
|
|
||||||
|
|
||||||
The implementation should inspect existing archive code before choosing the final object-selection method. Do not duplicate archive path construction.
|
|
||||||
|
|
||||||
Recommended selection priority:
|
|
||||||
|
|
||||||
1. Remote current manifest path.
|
|
||||||
2. Durable promoted transcript/artifact outputs recorded in the manifest or archive metadata, if available.
|
|
||||||
3. Objects under committed durable `transcripts/` and `artifacts/` archive prefixes.
|
|
||||||
4. Objects under durable `audio/` only when `--include-audio` is passed.
|
|
||||||
|
|
||||||
Always exclude:
|
|
||||||
|
|
||||||
```text
|
|
||||||
runs/**
|
|
||||||
```
|
|
||||||
|
|
||||||
for the first implementation.
|
|
||||||
|
|
||||||
## Package and file organization
|
|
||||||
|
|
||||||
Expected areas to inspect and update:
|
|
||||||
|
|
||||||
```text
|
|
||||||
cmd/narratio/
|
|
||||||
internal/app/
|
|
||||||
internal/adapters/storage/
|
|
||||||
internal/artifacts/
|
|
||||||
internal/manifest/
|
|
||||||
docs/
|
|
||||||
examples/
|
|
||||||
```
|
|
||||||
|
|
||||||
Suggested implementation shape:
|
|
||||||
|
|
||||||
```text
|
|
||||||
internal/app/restore.go
|
|
||||||
internal/app/restore_test.go
|
|
||||||
|
|
||||||
internal/archive/restore/
|
|
||||||
planner.go
|
|
||||||
executor.go
|
|
||||||
report.go
|
|
||||||
keys.go
|
|
||||||
*_test.go
|
|
||||||
```
|
|
||||||
|
|
||||||
The exact package name may vary. Use whatever best fits the existing repository, but keep these boundaries clear:
|
|
||||||
|
|
||||||
- `internal/app` owns CLI command handling, config/session loading, lock acquisition, and wiring.
|
|
||||||
- Restore planning/execution owns remote key discovery, conflict detection, downloads, and reporting.
|
|
||||||
- `internal/adapters/storage` remains a transport boundary only.
|
|
||||||
- Workspace/path helpers remain centralized; do not scatter string concatenation.
|
|
||||||
|
|
||||||
If the repository already has an `internal/archive` or archive-stage helper package, prefer extending that rather than creating a conflicting package layout.
|
|
||||||
|
|
||||||
## CLI output
|
|
||||||
|
|
||||||
`narratio restore` should print a concise operator summary.
|
|
||||||
|
|
||||||
Example successful output:
|
|
||||||
|
|
||||||
```text
|
|
||||||
Restored session archive for sample-campaign/2026-04-04
|
|
||||||
Remote run: 20260504T031500Z-a1b2c3
|
|
||||||
Downloaded: 4
|
|
||||||
Skipped unchanged: 2
|
|
||||||
Conflicts: 0
|
|
||||||
```
|
|
||||||
|
|
||||||
Example dry run:
|
|
||||||
|
|
||||||
```text
|
|
||||||
Restore plan for sample-campaign/2026-04-04
|
|
||||||
Remote run: 20260504T031500Z-a1b2c3
|
|
||||||
Would download: transcripts/processed.json
|
|
||||||
Would download: transcripts/trimmed.json
|
|
||||||
Would skip unchanged: artifacts/session_recap.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Example conflict:
|
|
||||||
|
|
||||||
```text
|
|
||||||
restore conflict: local artifacts/session_recap.md differs from remote archive; rerun with --force to overwrite
|
|
||||||
```
|
|
||||||
|
|
||||||
Do not print transcript or artifact content.
|
|
||||||
|
|
||||||
## Error behavior
|
|
||||||
|
|
||||||
Fail clearly when:
|
|
||||||
|
|
||||||
- storage backend is not configured;
|
|
||||||
- S3 bucket/config is missing or invalid;
|
|
||||||
- remote current commit marker is missing;
|
|
||||||
- remote current manifest is missing;
|
|
||||||
- remote manifest is invalid;
|
|
||||||
- remote manifest does not match requested campaign/session;
|
|
||||||
- local file differs from remote and `--force` is not set;
|
|
||||||
- a required remote object download fails;
|
|
||||||
- a local path would escape the session workspace;
|
|
||||||
- a remote key maps to an unsafe local path.
|
|
||||||
|
|
||||||
Skip or report non-fatal conditions when:
|
|
||||||
|
|
||||||
- optional audio restore finds no archived audio;
|
|
||||||
- an included prefix has no objects;
|
|
||||||
- a local file already matches the remote file.
|
|
||||||
|
|
||||||
## Path safety
|
|
||||||
|
|
||||||
Every restored file must map to a safe path under the session root.
|
|
||||||
|
|
||||||
Validation rules:
|
|
||||||
|
|
||||||
- local restore paths must be relative to the session root;
|
|
||||||
- reject absolute paths;
|
|
||||||
- reject `..` traversal;
|
|
||||||
- reject paths that escape through symlinks if the codebase has symlink-safe path checks;
|
|
||||||
- do not restore remote keys directly without mapping/classification;
|
|
||||||
- do not mirror arbitrary remote keys.
|
|
||||||
|
|
||||||
## Testing plan
|
|
||||||
|
|
||||||
Add focused unit tests. Do not require live S3.
|
|
||||||
|
|
||||||
### CLI tests
|
|
||||||
|
|
||||||
Add or update `internal/app` command tests for:
|
|
||||||
|
|
||||||
- `narratio restore --help`;
|
|
||||||
- restore accepts `--config`, `--session`, and `--session-id`;
|
|
||||||
- restore accepts `--dry-run`;
|
|
||||||
- restore accepts `--force`;
|
|
||||||
- restore accepts `--include-audio`;
|
|
||||||
- restore fails when storage is not configured;
|
|
||||||
- restore does not run pipeline stages.
|
|
||||||
|
|
||||||
### Restore planner tests
|
|
||||||
|
|
||||||
Test:
|
|
||||||
|
|
||||||
- missing `current/run_id.txt` fails;
|
|
||||||
- missing `current/manifest.json` fails;
|
|
||||||
- invalid manifest fails;
|
|
||||||
- wrong campaign/session manifest fails;
|
|
||||||
- default scope includes manifest/transcripts/artifacts;
|
|
||||||
- default scope excludes audio/logs/reports/config/runs;
|
|
||||||
- `--include-audio` includes durable audio;
|
|
||||||
- run-local keys are excluded;
|
|
||||||
- keys are sorted deterministically;
|
|
||||||
- unsafe remote-to-local paths are rejected.
|
|
||||||
|
|
||||||
### Conflict policy tests
|
|
||||||
|
|
||||||
Test:
|
|
||||||
|
|
||||||
- absent local file downloads;
|
|
||||||
- matching local file skips;
|
|
||||||
- differing local file conflicts by default;
|
|
||||||
- `--force` overwrites conflicts;
|
|
||||||
- `--dry-run` writes nothing;
|
|
||||||
- partial failure does not corrupt an existing local manifest.
|
|
||||||
|
|
||||||
### Storage/fake tests
|
|
||||||
|
|
||||||
Use fake storage to simulate:
|
|
||||||
|
|
||||||
- object listing;
|
|
||||||
- object download;
|
|
||||||
- missing objects;
|
|
||||||
- download failures;
|
|
||||||
- metadata/ETag behavior.
|
|
||||||
|
|
||||||
### Workspace/lock tests
|
|
||||||
|
|
||||||
Test:
|
|
||||||
|
|
||||||
- session layout is created before restore;
|
|
||||||
- session lock conflict fails;
|
|
||||||
- restored files land under the expected campaign/session workspace;
|
|
||||||
- no files are written outside the session root.
|
|
||||||
|
|
||||||
### Follow-up command workflow test
|
|
||||||
|
|
||||||
Add at least one test that simulates:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio restore --session-id 2026-04-04
|
|
||||||
narratio run-stage --force analyze
|
|
||||||
```
|
|
||||||
|
|
||||||
The test does not need to run real Scriptorium. Use existing fake/stub behavior to verify that restored transcripts and manifest state are sufficient for analyze-stage input resolution.
|
|
||||||
|
|
||||||
## Documentation updates when implemented
|
|
||||||
|
|
||||||
When the feature is implemented, update current-behavior docs:
|
|
||||||
|
|
||||||
```text
|
|
||||||
docs/cli.md
|
|
||||||
docs/operations.md
|
|
||||||
docs/internal/storage.md or docs/internal/archive/restore.md
|
|
||||||
```
|
|
||||||
|
|
||||||
If the documentation set does not yet have an internal restore document, add one consistent with the existing internal-doc style:
|
|
||||||
|
|
||||||
```text
|
|
||||||
docs/internal/command-restore.md
|
|
||||||
```
|
|
||||||
|
|
||||||
or:
|
|
||||||
|
|
||||||
```text
|
|
||||||
docs/internal/archive-restore.md
|
|
||||||
```
|
|
||||||
|
|
||||||
Do not document future `narratio run --restore` behavior outside `docs/roadmap/` until implemented.
|
|
||||||
|
|
||||||
## Implementation phases
|
|
||||||
|
|
||||||
### Phase 1: Audit existing archive and path helpers (completed)
|
|
||||||
|
|
||||||
Before coding behavior, inspect:
|
|
||||||
|
|
||||||
```text
|
|
||||||
internal/app/
|
|
||||||
internal/stage/archive*
|
|
||||||
internal/adapters/storage/
|
|
||||||
internal/artifacts/
|
|
||||||
internal/manifest/
|
|
||||||
docs/internal/stage-archive.md, if present
|
|
||||||
```
|
|
||||||
|
|
||||||
Determine:
|
|
||||||
|
|
||||||
- exact remote archive key layout;
|
|
||||||
- how root prefix/campaign/session are modeled;
|
|
||||||
- how current commit marker keys are built;
|
|
||||||
- how current manifest is uploaded;
|
|
||||||
- where promoted outputs are uploaded;
|
|
||||||
- whether helper functions already exist for remote archive keys;
|
|
||||||
- whether local workspace path helpers can safely map restore destinations.
|
|
||||||
|
|
||||||
Deliverable:
|
|
||||||
|
|
||||||
- small code comments or internal helper selection;
|
|
||||||
- no large behavior change yet unless required by tests.
|
|
||||||
|
|
||||||
### Phase 2: Add CLI surface and command wiring (completed)
|
|
||||||
|
|
||||||
Add `narratio restore` command parsing.
|
|
||||||
|
|
||||||
Wire flags:
|
|
||||||
|
|
||||||
```text
|
|
||||||
--config
|
|
||||||
--session
|
|
||||||
--session-id
|
|
||||||
--dry-run
|
|
||||||
--force
|
|
||||||
--include-audio
|
|
||||||
```
|
|
||||||
|
|
||||||
Use the existing config/session load path where practical.
|
|
||||||
|
|
||||||
Deliverable:
|
|
||||||
|
|
||||||
- command exists;
|
|
||||||
- help output is sensible;
|
|
||||||
- command validates basic inputs;
|
|
||||||
- command returns a clear “not yet implemented” or calls an empty planner if phased commits are desired;
|
|
||||||
- CLI tests pass.
|
|
||||||
|
|
||||||
### Phase 3: Implement remote current-state discovery (completed)
|
|
||||||
|
|
||||||
Add restore code that:
|
|
||||||
|
|
||||||
- creates an object store from resolved config;
|
|
||||||
- builds remote current marker key;
|
|
||||||
- reads `current/run_id.txt`;
|
|
||||||
- reads/downloads `current/manifest.json`;
|
|
||||||
- validates manifest identity;
|
|
||||||
- returns remote current-state metadata.
|
|
||||||
|
|
||||||
Deliverable:
|
|
||||||
|
|
||||||
- fake-storage tests for current-state discovery;
|
|
||||||
- no local file writes beyond temporary files.
|
|
||||||
|
|
||||||
### Phase 4: Implement restore planning (completed)
|
|
||||||
|
|
||||||
Build deterministic restore plans for default scope and `--include-audio`.
|
|
||||||
|
|
||||||
Deliverable:
|
|
||||||
|
|
||||||
- plan lists manifest, transcript, artifact files;
|
|
||||||
- plan excludes run-local data;
|
|
||||||
- plan detects local same/conflict/missing states;
|
|
||||||
- dry-run output works;
|
|
||||||
- no real file overwrite yet except temp comparisons as needed.
|
|
||||||
|
|
||||||
### Phase 5: Implement restore execution (completed)
|
|
||||||
|
|
||||||
Execute the plan safely:
|
|
||||||
|
|
||||||
- create directories;
|
|
||||||
- download to temporary files;
|
|
||||||
- validate content where practical;
|
|
||||||
- atomically install files;
|
|
||||||
- enforce default conflict failure;
|
|
||||||
- support `--force`;
|
|
||||||
- preserve existing manifest unless safe to replace.
|
|
||||||
|
|
||||||
Deliverable:
|
|
||||||
|
|
||||||
- restore works end-to-end against fake storage;
|
|
||||||
- failures are clear and do not corrupt existing local manifest.
|
|
||||||
|
|
||||||
### Phase 6: Add restore report and operator summary (completed)
|
|
||||||
|
|
||||||
Add concise stdout summary and optional JSON restore report if consistent with project diagnostics.
|
|
||||||
|
|
||||||
Deliverable:
|
|
||||||
|
|
||||||
- user-friendly output;
|
|
||||||
- durable diagnostic report if implemented;
|
|
||||||
- no content leakage.
|
|
||||||
|
|
||||||
### Phase 7: Workflow integration test (completed)
|
|
||||||
|
|
||||||
Add a test for restoring a previous session and then forcing `analyze`.
|
|
||||||
|
|
||||||
Deliverable:
|
|
||||||
|
|
||||||
- restored manifest/transcripts/artifacts are sufficient for analyze input resolution;
|
|
||||||
- no upstream stages rerun;
|
|
||||||
- no reliance on live subprocesses or S3.
|
|
||||||
|
|
||||||
### Phase 8: Documentation update (completed)
|
|
||||||
|
|
||||||
Once implemented, update current-behavior docs and internal command docs.
|
|
||||||
|
|
||||||
Also leave future `narratio run --restore` in roadmap only.
|
|
||||||
|
|
||||||
## Definition of done
|
|
||||||
|
|
||||||
The feature is complete when:
|
|
||||||
|
|
||||||
- `narratio restore` exists and is documented.
|
|
||||||
- It uses the same config/session discovery semantics as other commands where practical.
|
|
||||||
- It requires configured remote storage.
|
|
||||||
- It restores only from a committed current archive state.
|
|
||||||
- It restores the current manifest, transcripts, and artifacts by default.
|
|
||||||
- It restores audio only with `--include-audio`.
|
|
||||||
- It excludes run-local sandboxes.
|
|
||||||
- It fails on local/remote conflicts by default.
|
|
||||||
- `--force` overwrites conflicts.
|
|
||||||
- `--dry-run` writes nothing.
|
|
||||||
- It uses fake storage in tests.
|
|
||||||
- It does not change `prepare` behavior.
|
|
||||||
- It does not implement `narratio run --restore`.
|
|
||||||
- It avoids AWS SDK leakage outside the storage adapter.
|
|
||||||
- It uses centralized path/key helpers rather than scattered string concatenation.
|
|
||||||
- `go test ./...` passes.
|
|
||||||
|
|
||||||
## Suggested test commands
|
|
||||||
|
|
||||||
Run focused tests first:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
go test ./internal/app -run TestExecute -v
|
|
||||||
go test ./internal/adapters/storage -v
|
|
||||||
go test ./internal/artifacts -v
|
|
||||||
go test ./internal/manifest -v
|
|
||||||
```
|
|
||||||
|
|
||||||
Then run the full suite:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
go test ./...
|
|
||||||
```
|
|
||||||
|
|
||||||
## Suggested commit message
|
|
||||||
|
|
||||||
```text
|
|
||||||
Add restore subcommand roadmap
|
|
||||||
```
|
|
||||||
Reference in New Issue
Block a user