diff --git a/docs/roadmap/previous.md b/docs/roadmap/previous.md deleted file mode 100644 index 9eaa728..0000000 --- a/docs/roadmap/previous.md +++ /dev/null @@ -1,629 +0,0 @@ -# Roadmap: Previous-Session Artifacts - -## Status - -Completed. - -This roadmap describes the implementation strategy for first-class previous-session artifact support in Narratio. It belongs under `docs/roadmap/previous.md` until the feature is implemented. After implementation, current behavior should be documented in the appropriate user-facing and internal documentation files, and this roadmap should be removed or marked complete according to the documentation policy. - -## Summary - -Narratio should support using artifacts from a previous session as inputs to artifacts generated for the current session. - -The primary use case is session recap continuity: a current session recap should be able to consume the previous session recap. The design should support arbitrary previous-session artifacts from the start, not just `session_recap`. - -The canonical source syntax should be: - -```yaml -source: narratio.previous_session.artifact. -``` - -For example: - -```yaml -source: narratio.previous_session.artifact.session_recap -``` - -Previous-session artifacts are materialized during the `prepare` stage into a current-session top-level `previous/` directory. Downstream stages consume only the local `previous/` copies. The S3 backend is authoritative for previous-session state. - -## Design Decisions - -### 1. Add `previous_session_id` to `session.yml` - -Add an optional top-level session key: - -```yaml -session_id: "{{ session_id }}" -previous_session_id: "{{ previous_session_id }}" -campaign: sample-campaign -``` - -Rules: - -- `previous_session_id` is optional. -- If present, it identifies the previous session within the same campaign. -- It must not equal `session_id`. -- It should use the same validation rules as `session_id`. -- It may be supplied through session templating. -- Add a CLI/template value such as `--previous-session-id ` if required by the existing session templating implementation. -- If a template placeholder for `previous_session_id` is present and no value is supplied, loading should fail with a clear unresolved-template error. - -### 2. Use canonical previous-session artifact source IDs - -Support this source pattern in Scriptorium artifact input definitions: - -```text -narratio.previous_session.artifact. -``` - -Examples: - -```yaml -inputs: - previous_recap: - source: narratio.previous_session.artifact.session_recap - required: false - - previous_quest_log: - source: narratio.previous_session.artifact.quest_log - required: true -``` - -Rules: - -- `` must be a valid configured artifact key. -- Use the same artifact key validation rules as current-session runtime artifacts. -- Do not special-case `session_recap`. -- Do not limit implementation to a fixed list of previous artifacts. - -### 3. Add a top-level `previous/` workspace directory - -Extend the session workspace layout with: - -```text -previous/ - manifest.json - artifacts/ - -``` - -The `previous/` directory is current-session state. It is a prepared input cache, not a full mirror of the previous session workspace. - -Conceptually: - -```text -work/// - previous/ - manifest.json - artifacts/ - session_recap.md - quest_log.json -``` - -The current session should not read directly from the previous session's local workspace during ordinary operation. - -### 4. S3 is authoritative for previous-session state - -For this initial implementation, previous-session artifacts should be downloaded from the configured S3 backend. - -Do not compare local and remote copies. - -Do not prefer local previous-session workspace state. - -Do not implement a `--local` override in this roadmap. That can be considered later. - -The simplified stage behavior is: - -1. If the local manifest indicates `prepare` already succeeded and `--force` is not supplied, the runner skips `prepare`. No previous-session download occurs. -2. If `prepare` has not succeeded, `prepare` runs and downloads referenced previous-session artifacts from S3. -3. If `prepare` previously succeeded but `--force` is supplied, `prepare` runs again and overwrites local `previous/` state from S3. - -### 5. Materialize only referenced previous-session artifacts - -During `prepare`, scan the current resolved pipeline/session configuration for Scriptorium artifact inputs whose source matches: - -```text -narratio.previous_session.artifact. -``` - -Only those referenced previous-session artifacts need to be downloaded. - -Do not blindly download every previous-session artifact. - -If no previous-session artifact sources are referenced, `prepare` should not require `previous_session_id` and should not touch `previous/`. - -### 6. Preserve stage boundaries - -`prepare` owns previous-session artifact materialization because these files are inputs to later stages. - -`analyze` should not talk to S3. - -The Scriptorium adapter should not know about previous sessions. - -The storage adapter should not infer campaign, session, run, or root-prefix semantics. Callers should continue to provide explicit bucket-relative keys. - -### 7. Archive and restore `previous/` - -After implementation: - -- `archive` should include `previous/` as durable current-session prepared input state. -- `restore` should restore `previous/` along with the rest of the durable session state it already restores. -- `previous/` should not be treated as current-session generated artifacts. -- `previous/` entries should be recorded as inputs/provenance, not as outputs produced by the current session. - -## Target User Workflow - -A typical session config: - -```yaml -session_id: "{{ session_id }}" -previous_session_id: "{{ previous_session_id }}" -campaign: sample-campaign - -inputs: - audio_dir: ./audio - speakers_file: ./examples/speakers.yml - autocorrect_file: ./examples/autocorrect.yml - glossary_file: ./examples/glossary.yml -``` - -A typical artifact config: - -```yaml -scriptorium: - artifacts: - session_recap: - enabled: true - prompt_id: dnd_session.session_recap - output_path: artifacts/session_recap.md - inputs: - transcript: - source: narratio.transcript.trimmed - required: true - previous_recap: - source: narratio.previous_session.artifact.session_recap - required: false -``` - -Typical command: - -```bash -narratio run --session-id 2026-04-11 --previous-session-id 2026-04-04 -``` - -Expected behavior: - -1. `prepare` sees a referenced previous-session artifact: `session_recap`. -2. `prepare` downloads the previous session's `narratio.artifact.session_recap` from S3. -3. `prepare` writes it under the current session workspace, for example `previous/artifacts/session_recap.md`. -4. `prepare` records provenance in the current session manifest. -5. `analyze` resolves `narratio.previous_session.artifact.session_recap` from the local `previous/` directory. -6. Scriptorium receives the previous recap as a normal input file. - -## Implementation Plan - -### Phase 1: Session config and templating - -Update session configuration structs to include: - -```yaml -previous_session_id: "" -``` - -Implementation steps: - -1. Add `PreviousSessionID` or equivalent to the session config type. -2. Add validation: - - optional; - - same format constraints as `session_id`; - - must not equal `session_id`. -3. Extend session templating support to include: - - `{{previous_session_id}}` - - `{{ previous_session_id }}` -4. Add a CLI flag if required by current templating flow: - - `--previous-session-id ` -5. Ensure unresolved `previous_session_id` placeholders fail clearly. -6. Update config tests for: - - no previous session; - - valid previous session; - - previous session equal to current session; - - unresolved placeholder; - - CLI/template rendering. - -Do not add future workflow flags in this phase. - -### Phase 2: Workspace path helpers - -Add centralized path helpers for current-session previous-state paths. - -Suggested helpers: - -```text -SessionPreviousDir() -SessionPreviousManifestPath() -SessionPreviousArtifactsDir() -SessionPreviousArtifactPath(name or relative output path) -``` - -The exact names should match existing path-helper style. - -Rules: - -- Do not construct `previous/` paths through scattered string concatenation. -- Keep paths session-relative where possible. -- Ensure workspace layout creation includes `previous/` only when appropriate, or creates it idempotently with the rest of the layout if simpler. - -Tests: - -- path helper tests; -- workspace layout tests; -- ensure cleanup logic does not accidentally delete configured roots; -- ensure `previous/` is treated as session-durable state, not run-local state. - -### Phase 3: Previous-session source parsing - -Add parsing/recognition for: - -```text -narratio.previous_session.artifact. -``` - -Implementation steps: - -1. Add constants/helpers in the artifact/source parsing layer. -2. Validate artifact names using the same rules as current runtime artifact keys. -3. Add helpers such as: - - `IsPreviousSessionArtifactSource(source string) bool` - - `PreviousSessionArtifactName(source string) (string, bool)` -4. Ensure config validation accepts this source pattern. -5. Ensure invalid sources fail clearly. - -Tests: - -- valid previous-session artifact source; -- invalid/missing artifact name; -- invalid artifact key characters; -- ordinary current-session sources still validate; -- unknown sources still fail. - -### Phase 4: Scan configured artifacts for previous-session inputs - -Add a helper that inspects resolved Scriptorium artifact definitions and returns the set of referenced previous-session artifact names. - -Rules: - -- Scan enabled artifacts according to current artifact-enable semantics. -- Include all inputs whose source matches `narratio.previous_session.artifact.`. -- Deduplicate artifact names. -- Sort results deterministically. -- Preserve required/optional information per reference. -- If the same previous artifact is referenced both required and optional, treat it as required. - -Suggested output model: - -```go -type PreviousArtifactRequirement struct { - Name string - Required bool - Sources []string // optional diagnostics -} -``` - -Tests: - -- no artifacts; -- no previous inputs; -- one optional previous input; -- one required previous input; -- duplicate references; -- required plus optional reference to the same artifact; -- deterministic ordering. - -### Phase 5: Resolve previous-session archive keys - -Implement a narrow service/helper used by `prepare` to resolve previous-session artifact files from S3. - -Responsibilities: - -1. Locate the previous session's current remote state. -2. Download the previous session manifest, or the minimum remote metadata needed to resolve artifact source IDs. -3. Resolve `narratio.artifact.` inside the previous session's artifact catalog/manifest. -4. Download the resolved artifact into the current session's `previous/` directory. -5. Download/store the previous session manifest as `previous/manifest.json`. -6. Return provenance records for manifest input recording. - -Important archive invariant: - -- Remote current state must be based on the committed archive marker. -- Do not treat incomplete archive uploads as current state. -- Use the existing archive/current remote layout and commit-marker rules. -- `current/run_id.txt` is the final remote commit marker and should be respected when locating current remote state. - -Do not put prefix semantics into the storage adapter. Compute explicit bucket-relative keys in app/stage/archive helper code, then call the storage adapter. - -Required vs optional behavior: - -- Required previous artifact missing from S3/current manifest: fail `prepare`. -- Optional previous artifact missing from S3/current manifest: continue without materializing that input. -- Previous session missing entirely: - - fail if any referenced previous artifact is required; - - continue if all referenced previous artifacts are optional. -- If previous_session_id is unset: - - fail if any referenced previous artifact is required; - - continue and omit all previous-session inputs if all are optional. - -Validation behavior: - -- Downloaded artifacts should pass the same validation rules as current-session artifacts where practical. -- Generic Scriptorium artifacts should at least be non-empty. -- Invalid required artifact: fail. -- Invalid optional artifact: prefer fail if the object exists but is invalid, because invalid archived data is usually an operator problem rather than absence. - -Tests: - -- downloads previous manifest; -- downloads required previous artifact; -- skips missing optional previous artifact; -- fails missing required previous artifact; -- fails required previous artifact when previous_session_id is unset; -- optional previous artifact with no previous_session_id does not fail; -- respects current remote commit marker; -- does not use local previous-session workspace state; -- uses storage adapter with explicit keys. - -### Phase 6: Integrate with `prepare` - -Extend the `prepare` stage: - -1. Run existing input materialization as before. -2. Detect previous-session artifact requirements. -3. If requirements exist, hydrate `previous/` from S3 according to the rules above. -4. Record hydrated previous artifacts in `manifest.Inputs`. -5. Preserve existing prepare outputs and provenance behavior. - -Overwrite behavior: - -- If `prepare` runs, it owns `previous/`. -- Before hydrating, clear the managed `previous/` directory, or clear the managed previous artifact paths. -- Prefer clearing the whole `previous/` directory if no other feature writes there. -- Under `--force`, this naturally overwrites local `previous/` state. -- Do not compare local and remote copies. - -Skip behavior: - -- Do not add stage-local skip logic. -- Runner-level skip remains authoritative. -- If the manifest says `prepare` succeeded and `--force` is not supplied, `prepare` does not run and no S3 downloads occur. -- If users add a new previous-session input after `prepare` already succeeded, they must rerun prepare with `--force`. - -Error message requirement: - -If an analyze-stage input cannot be resolved because `previous/` is missing or stale, the error should tell the operator to run: - -```bash -narratio run-stage --force prepare -``` - -or the appropriate existing CLI command shape. - -Tests: - -- ordinary prepare without previous_session_id remains unchanged; -- prepare with optional previous artifact and no previous_session_id succeeds; -- prepare with required previous artifact and no previous_session_id fails; -- prepare with required previous artifact downloads to `previous/`; -- force prepare overwrites `previous/`; -- prepare records manifest inputs for previous artifacts; -- prepare skip behavior remains controlled by runner tests; -- no regression in S3 audio prepare behavior. - -### Phase 7: Artifact resolver support - -Update artifact resolution so Scriptorium inputs can resolve: - -```text -narratio.previous_session.artifact. -``` - -from the current session's `previous/` directory. - -Rules: - -- The resolver should not call S3. -- The resolver should not read the previous session's local workspace. -- The resolver should map the previous-session source ID to the local prepared copy under `previous/`. -- Resolution should use manifest input records when available. -- Fallback to the local `previous/` path may be allowed if consistent with existing resolver behavior, but manifest provenance should be preferred. -- Missing required input should fail with a clear prepare-oriented message. -- Optional missing input should be omitted. - -Suggested provenance: - -```text -previous_session.manifest.outputs -previous_session.archive.current -current_session.previous_cache -``` - -Use names that fit the existing manifest/resolver vocabulary. - -Tests: - -- resolves prepared previous artifact through manifest input record; -- resolves or fails appropriately when only filesystem copy exists, depending on chosen fallback policy; -- missing optional previous artifact is omitted; -- missing required previous artifact fails clearly; -- current-session `narratio.artifact.` behavior is unchanged. - -### Phase 8: Analyze-stage integration - -The analyze stage should require little or no special previous-session logic if the resolver is designed correctly. - -Confirm: - -- Scriptorium input resolution accepts previous-session source IDs. -- The Scriptorium adapter receives a normal local input path. -- Render-debug and run modes behave the same as for ordinary inputs. -- Stage metadata includes useful input provenance if current structures support it. - -Tests: - -- configured artifact receives previous recap input; -- optional previous recap omitted when not prepared; -- required previous recap fails when not prepared; -- render-debug path works with previous-session inputs; -- no S3 calls occur from analyze. - -### Phase 9: Archive `previous/` - -Update archive behavior so the current session's durable `previous/` directory is uploaded/preserved. - -Rules: - -- `previous/` is current-session input/provenance state. -- It is not a generated current-session artifact. -- Archive it with the current session's durable state, alongside other durable session files according to the current archive layout. -- Preserve existing archive commit ordering. -- Do not make `previous/` upload the final commit marker. -- Do not treat missing `previous/` as an error when no previous-session inputs were prepared. - -Tests: - -- archive includes `previous/manifest.json` and prepared previous artifacts when present; -- archive omits or tolerates absent `previous/` when unused; -- archive commit ordering remains valid; -- cleanup behavior does not delete durable `previous/` before archive commit. - -### Phase 10: Restore `previous/` - -Update `narratio restore` so it restores the current session's archived `previous/` directory. - -Rules: - -- Restore `previous/` as durable current-session state. -- Do not infer or restore the previous session workspace merely because `previous_session_id` exists. -- Do not redownload previous-session artifacts from the previous session archive during restore; restore the current session's archived `previous/` cache. -- Respect existing restore flags and overwrite behavior. - -Tests: - -- restore downloads `previous/` when present; -- restore succeeds when `previous/` is absent; -- restore with force overwrites local `previous/` according to existing restore semantics; -- restored current session can run analyze using `previous/` without needing previous session archive access. - -### Phase 11: Documentation updates after implementation - -After implementation, update current-behavior docs. Do not document implemented behavior only in this roadmap. - -Likely files: - -- `docs/config.md` -- `docs/cli.md` -- `docs/operations.md` -- `docs/internal/stage-prepare.md` -- `docs/internal/artifacts.md` -- `docs/internal/storage.md` only if storage contracts change -- `docs/internal/workspace.md` -- relevant maintained examples under `examples/` - -Docs should explain: - -- `session.previous_session_id`; -- `--previous-session-id` if added; -- `narratio.previous_session.artifact.`; -- `previous/` workspace directory; -- S3-authoritative previous-session behavior; -- required vs optional previous-session artifact handling; -- when to run `narratio run-stage --force prepare`. - -Do not document future `--local` support as current behavior. - -## Future Work - -These items are explicitly out of scope for the initial implementation. - -### `--local` previous-session source - -A future flag may allow `prepare` to copy previous-session artifacts from a local previous session workspace instead of S3. - -Possible future command shape: - -```bash -narratio run-stage --force prepare --local -``` - -or a more specific flag such as: - -```bash -narratio run-stage --force prepare --previous-source local -``` - -Do not implement this now. - -### `narratio run --restore` - -A future flag may run `narratio restore` before starting the regular pipeline: - -```bash -narratio run --restore --session-id 2026-04-11 -``` - -Do not implement this as part of previous-session artifact support unless it already exists and only requires documentation. - -### Multi-previous-session support - -A future design may support more than one previous/reference session. - -Do not implement this now. - -### Previous-session artifact version pinning - -A future design may pin previous-session artifact inputs to a specific previous run ID or checksum. - -Do not implement this now. - -## Test Plan Summary - -Run at least: - -```bash -go test ./internal/config -v -go test ./internal/artifacts -v -go test ./internal/stage -run Prepare -v -go test ./internal/stage -run Analyze -v -go test ./internal/app -run TestExecute -v -go test ./... -``` - -Add focused tests for: - -- session config and templating; -- previous-session source parsing; -- previous artifact requirement scanning; -- S3-backed previous artifact download; -- prepare skip/force semantics; -- manifest input provenance; -- resolver behavior; -- analyze integration; -- archive/restore `previous/` persistence; -- examples load/validate. - -## Acceptance Criteria - -The feature is complete when: - -1. `session.yml` supports optional `previous_session_id`. -2. Session templating supports `previous_session_id`. -3. Scriptorium artifact inputs accept `narratio.previous_session.artifact.`. -4. `prepare` scans enabled Scriptorium artifacts for previous-session artifact inputs. -5. `prepare` downloads referenced previous-session artifacts from S3 into `previous/`. -6. Required/optional behavior is correct. -7. `prepare` uses stage-level skip/force behavior and does not compare local and remote copies. -8. `analyze` resolves previous-session artifacts only from local prepared `previous/` state. -9. `archive` persists `previous/`. -10. `restore` restores `previous/`. -11. Tests cover config, prepare, resolver, analyze, archive, and restore behavior. -12. Current-behavior docs and maintained examples are updated after implementation. -13. No storage adapter implementation infers campaign/session/root-prefix semantics. -14. No previous-session S3 logic is added to `analyze` or the Scriptorium adapter.