narratio/docs/roadmap/previous.md

Roadmap: Previous-Session Artifacts

Status

Completed.

This roadmap describes the implementation strategy for first-class previous-session artifact support in Narratio. It belongs under docs/roadmap/previous.md until the feature is implemented. After implementation, current behavior should be documented in the appropriate user-facing and internal documentation files, and this roadmap should be removed or marked complete according to the documentation policy.

Summary

Narratio should support using artifacts from a previous session as inputs to artifacts generated for the current session.

The primary use case is session recap continuity: a current session recap should be able to consume the previous session recap. The design should support arbitrary previous-session artifacts from the start, not just session_recap.

The canonical source syntax should be:

source: narratio.previous_session.artifact.<artifact_name>

For example:

source: narratio.previous_session.artifact.session_recap

Previous-session artifacts are materialized during the prepare stage into a current-session top-level previous/ directory. Downstream stages consume only the local previous/ copies. The S3 backend is authoritative for previous-session state.

Design Decisions

1. Add previous_session_id to session.yml

Add an optional top-level session key:

session_id: "{{ session_id }}"
previous_session_id: "{{ previous_session_id }}"
campaign: sample-campaign

Rules:

• previous_session_id is optional.
• If present, it identifies the previous session within the same campaign.
• It must not equal session_id.
• It should use the same validation rules as session_id.
• It may be supplied through session templating.
• Add a CLI/template value such as --previous-session-id <id> if required by the existing session templating implementation.
• If a template placeholder for previous_session_id is present and no value is supplied, loading should fail with a clear unresolved-template error.

2. Use canonical previous-session artifact source IDs

Support this source pattern in Scriptorium artifact input definitions:

narratio.previous_session.artifact.<artifact_name>

Examples:

inputs:
  previous_recap:
    source: narratio.previous_session.artifact.session_recap
    required: false

  previous_quest_log:
    source: narratio.previous_session.artifact.quest_log
    required: true

Rules:

• <artifact_name> must be a valid configured artifact key.
• Use the same artifact key validation rules as current-session runtime artifacts.
• Do not special-case session_recap.
• Do not limit implementation to a fixed list of previous artifacts.

3. Add a top-level previous/ workspace directory

Extend the session workspace layout with:

previous/
  manifest.json
  artifacts/
    <artifact outputs copied from the previous session>

The previous/ directory is current-session state. It is a prepared input cache, not a full mirror of the previous session workspace.

Conceptually:

work/<campaign>/<current_session_id>/
  previous/
    manifest.json
    artifacts/
      session_recap.md
      quest_log.json

The current session should not read directly from the previous session's local workspace during ordinary operation.

4. S3 is authoritative for previous-session state

For this initial implementation, previous-session artifacts should be downloaded from the configured S3 backend.

Do not compare local and remote copies.

Do not prefer local previous-session workspace state.

Do not implement a --local override in this roadmap. That can be considered later.

The simplified stage behavior is:

1. If the local manifest indicates prepare already succeeded and --force is not supplied, the runner skips prepare. No previous-session download occurs.
2. If prepare has not succeeded, prepare runs and downloads referenced previous-session artifacts from S3.
3. If prepare previously succeeded but --force is supplied, prepare runs again and overwrites local previous/ state from S3.

5. Materialize only referenced previous-session artifacts

During prepare, scan the current resolved pipeline/session configuration for Scriptorium artifact inputs whose source matches:

narratio.previous_session.artifact.<artifact_name>

Only those referenced previous-session artifacts need to be downloaded.

Do not blindly download every previous-session artifact.

If no previous-session artifact sources are referenced, prepare should not require previous_session_id and should not touch previous/.

6. Preserve stage boundaries

prepare owns previous-session artifact materialization because these files are inputs to later stages.

analyze should not talk to S3.

The Scriptorium adapter should not know about previous sessions.

The storage adapter should not infer campaign, session, run, or root-prefix semantics. Callers should continue to provide explicit bucket-relative keys.

7. Archive and restore previous/

After implementation:

• archive should include previous/ as durable current-session prepared input state.
• restore should restore previous/ along with the rest of the durable session state it already restores.
• previous/ should not be treated as current-session generated artifacts.
• previous/ entries should be recorded as inputs/provenance, not as outputs produced by the current session.

Target User Workflow

A typical session config:

session_id: "{{ session_id }}"
previous_session_id: "{{ previous_session_id }}"
campaign: sample-campaign

inputs:
  audio_dir: ./audio
  speakers_file: ./examples/speakers.yml
  autocorrect_file: ./examples/autocorrect.yml
  glossary_file: ./examples/glossary.yml

A typical artifact config:

scriptorium:
  artifacts:
    session_recap:
      enabled: true
      prompt_id: dnd_session.session_recap
      output_path: artifacts/session_recap.md
      inputs:
        transcript:
          source: narratio.transcript.trimmed
          required: true
        previous_recap:
          source: narratio.previous_session.artifact.session_recap
          required: false

Typical command:

narratio run --session-id 2026-04-11 --previous-session-id 2026-04-04

Expected behavior:

1. prepare sees a referenced previous-session artifact: session_recap.
2. prepare downloads the previous session's narratio.artifact.session_recap from S3.
3. prepare writes it under the current session workspace, for example previous/artifacts/session_recap.md.
4. prepare records provenance in the current session manifest.
5. analyze resolves narratio.previous_session.artifact.session_recap from the local previous/ directory.
6. Scriptorium receives the previous recap as a normal input file.

Implementation Plan

Phase 1: Session config and templating

Update session configuration structs to include:

previous_session_id: ""

Implementation steps:

1. Add PreviousSessionID or equivalent to the session config type.
2. Add validation:
   • optional;
   • same format constraints as session_id;
   • must not equal session_id.
3. Extend session templating support to include:
   • {{previous_session_id}}
   • {{ previous_session_id }}
4. Add a CLI flag if required by current templating flow:
   • --previous-session-id <id>
5. Ensure unresolved previous_session_id placeholders fail clearly.
6. Update config tests for:
   • no previous session;
   • valid previous session;
   • previous session equal to current session;
   • unresolved placeholder;
   • CLI/template rendering.

Do not add future workflow flags in this phase.

Phase 2: Workspace path helpers

Add centralized path helpers for current-session previous-state paths.

Suggested helpers:

SessionPreviousDir()
SessionPreviousManifestPath()
SessionPreviousArtifactsDir()
SessionPreviousArtifactPath(name or relative output path)

The exact names should match existing path-helper style.

Rules:

• Do not construct previous/ paths through scattered string concatenation.
• Keep paths session-relative where possible.
• Ensure workspace layout creation includes previous/ only when appropriate, or creates it idempotently with the rest of the layout if simpler.

Tests:

• path helper tests;
• workspace layout tests;
• ensure cleanup logic does not accidentally delete configured roots;
• ensure previous/ is treated as session-durable state, not run-local state.

Phase 3: Previous-session source parsing

Add parsing/recognition for:

narratio.previous_session.artifact.<artifact_name>

Implementation steps:

1. Add constants/helpers in the artifact/source parsing layer.
2. Validate artifact names using the same rules as current runtime artifact keys.
3. Add helpers such as:
   • IsPreviousSessionArtifactSource(source string) bool
   • PreviousSessionArtifactName(source string) (string, bool)
4. Ensure config validation accepts this source pattern.
5. Ensure invalid sources fail clearly.

Tests:

• valid previous-session artifact source;
• invalid/missing artifact name;
• invalid artifact key characters;
• ordinary current-session sources still validate;
• unknown sources still fail.

Phase 4: Scan configured artifacts for previous-session inputs

Add a helper that inspects resolved Scriptorium artifact definitions and returns the set of referenced previous-session artifact names.

Rules:

• Scan enabled artifacts according to current artifact-enable semantics.
• Include all inputs whose source matches narratio.previous_session.artifact.<name>.
• Deduplicate artifact names.
• Sort results deterministically.
• Preserve required/optional information per reference.
• If the same previous artifact is referenced both required and optional, treat it as required.

Suggested output model:

type PreviousArtifactRequirement struct {
    Name     string
    Required bool
    Sources  []string // optional diagnostics
}

Tests:

• no artifacts;
• no previous inputs;
• one optional previous input;
• one required previous input;
• duplicate references;
• required plus optional reference to the same artifact;
• deterministic ordering.

Phase 5: Resolve previous-session archive keys

Implement a narrow service/helper used by prepare to resolve previous-session artifact files from S3.

Responsibilities:

1. Locate the previous session's current remote state.
2. Download the previous session manifest, or the minimum remote metadata needed to resolve artifact source IDs.
3. Resolve narratio.artifact.<name> inside the previous session's artifact catalog/manifest.
4. Download the resolved artifact into the current session's previous/ directory.
5. Download/store the previous session manifest as previous/manifest.json.
6. Return provenance records for manifest input recording.

Important archive invariant:

• Remote current state must be based on the committed archive marker.
• Do not treat incomplete archive uploads as current state.
• Use the existing archive/current remote layout and commit-marker rules.
• current/run_id.txt is the final remote commit marker and should be respected when locating current remote state.

Do not put prefix semantics into the storage adapter. Compute explicit bucket-relative keys in app/stage/archive helper code, then call the storage adapter.

Required vs optional behavior:

• Required previous artifact missing from S3/current manifest: fail prepare.
• Optional previous artifact missing from S3/current manifest: continue without materializing that input.
• Previous session missing entirely:
  • fail if any referenced previous artifact is required;
  • continue if all referenced previous artifacts are optional.
• If previous_session_id is unset:
  • fail if any referenced previous artifact is required;
  • continue and omit all previous-session inputs if all are optional.

Validation behavior:

• Downloaded artifacts should pass the same validation rules as current-session artifacts where practical.
• Generic Scriptorium artifacts should at least be non-empty.
• Invalid required artifact: fail.
• Invalid optional artifact: prefer fail if the object exists but is invalid, because invalid archived data is usually an operator problem rather than absence.

Tests:

• downloads previous manifest;
• downloads required previous artifact;
• skips missing optional previous artifact;
• fails missing required previous artifact;
• fails required previous artifact when previous_session_id is unset;
• optional previous artifact with no previous_session_id does not fail;
• respects current remote commit marker;
• does not use local previous-session workspace state;
• uses storage adapter with explicit keys.

Phase 6: Integrate with prepare

Extend the prepare stage:

1. Run existing input materialization as before.
2. Detect previous-session artifact requirements.
3. If requirements exist, hydrate previous/ from S3 according to the rules above.
4. Record hydrated previous artifacts in manifest.Inputs.
5. Preserve existing prepare outputs and provenance behavior.

Overwrite behavior:

• If prepare runs, it owns previous/.
• Before hydrating, clear the managed previous/ directory, or clear the managed previous artifact paths.
• Prefer clearing the whole previous/ directory if no other feature writes there.
• Under --force, this naturally overwrites local previous/ state.
• Do not compare local and remote copies.

Skip behavior:

• Do not add stage-local skip logic.
• Runner-level skip remains authoritative.
• If the manifest says prepare succeeded and --force is not supplied, prepare does not run and no S3 downloads occur.
• If users add a new previous-session input after prepare already succeeded, they must rerun prepare with --force.

Error message requirement:

If an analyze-stage input cannot be resolved because previous/ is missing or stale, the error should tell the operator to run:

narratio run-stage --force prepare

or the appropriate existing CLI command shape.

Tests:

• ordinary prepare without previous_session_id remains unchanged;
• prepare with optional previous artifact and no previous_session_id succeeds;
• prepare with required previous artifact and no previous_session_id fails;
• prepare with required previous artifact downloads to previous/;
• force prepare overwrites previous/;
• prepare records manifest inputs for previous artifacts;
• prepare skip behavior remains controlled by runner tests;
• no regression in S3 audio prepare behavior.

Phase 7: Artifact resolver support

Update artifact resolution so Scriptorium inputs can resolve:

narratio.previous_session.artifact.<artifact_name>

from the current session's previous/ directory.

Rules:

• The resolver should not call S3.
• The resolver should not read the previous session's local workspace.
• The resolver should map the previous-session source ID to the local prepared copy under previous/.
• Resolution should use manifest input records when available.
• Fallback to the local previous/ path may be allowed if consistent with existing resolver behavior, but manifest provenance should be preferred.
• Missing required input should fail with a clear prepare-oriented message.
• Optional missing input should be omitted.

Suggested provenance:

previous_session.manifest.outputs
previous_session.archive.current
current_session.previous_cache

Use names that fit the existing manifest/resolver vocabulary.

Tests:

• resolves prepared previous artifact through manifest input record;
• resolves or fails appropriately when only filesystem copy exists, depending on chosen fallback policy;
• missing optional previous artifact is omitted;
• missing required previous artifact fails clearly;
• current-session narratio.artifact.<name> behavior is unchanged.

Phase 8: Analyze-stage integration

The analyze stage should require little or no special previous-session logic if the resolver is designed correctly.

Confirm:

• Scriptorium input resolution accepts previous-session source IDs.
• The Scriptorium adapter receives a normal local input path.
• Render-debug and run modes behave the same as for ordinary inputs.
• Stage metadata includes useful input provenance if current structures support it.

Tests:

• configured artifact receives previous recap input;
• optional previous recap omitted when not prepared;
• required previous recap fails when not prepared;
• render-debug path works with previous-session inputs;
• no S3 calls occur from analyze.

Phase 9: Archive previous/

Update archive behavior so the current session's durable previous/ directory is uploaded/preserved.

Rules:

• previous/ is current-session input/provenance state.
• It is not a generated current-session artifact.
• Archive it with the current session's durable state, alongside other durable session files according to the current archive layout.
• Preserve existing archive commit ordering.
• Do not make previous/ upload the final commit marker.
• Do not treat missing previous/ as an error when no previous-session inputs were prepared.

Tests:

• archive includes previous/manifest.json and prepared previous artifacts when present;
• archive omits or tolerates absent previous/ when unused;
• archive commit ordering remains valid;
• cleanup behavior does not delete durable previous/ before archive commit.

Phase 10: Restore previous/

Update narratio restore so it restores the current session's archived previous/ directory.

Rules:

• Restore previous/ as durable current-session state.
• Do not infer or restore the previous session workspace merely because previous_session_id exists.
• Do not redownload previous-session artifacts from the previous session archive during restore; restore the current session's archived previous/ cache.
• Respect existing restore flags and overwrite behavior.

Tests:

• restore downloads previous/ when present;
• restore succeeds when previous/ is absent;
• restore with force overwrites local previous/ according to existing restore semantics;
• restored current session can run analyze using previous/ without needing previous session archive access.

Phase 11: Documentation updates after implementation

After implementation, update current-behavior docs. Do not document implemented behavior only in this roadmap.

Likely files:

• docs/config.md
• docs/cli.md
• docs/operations.md
• docs/internal/stage-prepare.md
• docs/internal/artifacts.md
• docs/internal/storage.md only if storage contracts change
• docs/internal/workspace.md
• relevant maintained examples under examples/

Docs should explain:

• session.previous_session_id;
• --previous-session-id if added;
• narratio.previous_session.artifact.<artifact_name>;
• previous/ workspace directory;
• S3-authoritative previous-session behavior;
• required vs optional previous-session artifact handling;
• when to run narratio run-stage --force prepare.

Do not document future --local support as current behavior.

Future Work

These items are explicitly out of scope for the initial implementation.

--local previous-session source

A future flag may allow prepare to copy previous-session artifacts from a local previous session workspace instead of S3.

Possible future command shape:

narratio run-stage --force prepare --local

or a more specific flag such as:

narratio run-stage --force prepare --previous-source local

Do not implement this now.

narratio run --restore

A future flag may run narratio restore before starting the regular pipeline:

narratio run --restore --session-id 2026-04-11

Do not implement this as part of previous-session artifact support unless it already exists and only requires documentation.

Multi-previous-session support

A future design may support more than one previous/reference session.

Do not implement this now.

Previous-session artifact version pinning

A future design may pin previous-session artifact inputs to a specific previous run ID or checksum.

Do not implement this now.

Test Plan Summary

Run at least:

go test ./internal/config -v
go test ./internal/artifacts -v
go test ./internal/stage -run Prepare -v
go test ./internal/stage -run Analyze -v
go test ./internal/app -run TestExecute -v
go test ./...

Add focused tests for:

• session config and templating;
• previous-session source parsing;
• previous artifact requirement scanning;
• S3-backed previous artifact download;
• prepare skip/force semantics;
• manifest input provenance;
• resolver behavior;
• analyze integration;
• archive/restore previous/ persistence;
• examples load/validate.

Acceptance Criteria

The feature is complete when:

1. session.yml supports optional previous_session_id.
2. Session templating supports previous_session_id.
3. Scriptorium artifact inputs accept narratio.previous_session.artifact.<artifact_name>.
4. prepare scans enabled Scriptorium artifacts for previous-session artifact inputs.
5. prepare downloads referenced previous-session artifacts from S3 into previous/.
6. Required/optional behavior is correct.
7. prepare uses stage-level skip/force behavior and does not compare local and remote copies.
8. analyze resolves previous-session artifacts only from local prepared previous/ state.
9. archive persists previous/.
10. restore restores previous/.
11. Tests cover config, prepare, resolver, analyze, archive, and restore behavior.
12. Current-behavior docs and maintained examples are updated after implementation.
13. No storage adapter implementation infers campaign/session/root-prefix semantics.
14. No previous-session S3 logic is added to analyze or the Scriptorium adapter.