Finalize previous-session artifact documentation and restore-analyze continuity coverage

docs/internal/artifacts.md

@@ -1,41 +1,35 @@
 # Internal: Artifacts
 
 ## Purpose
-Define Narratio's artifact identity and resolution model for built-in transcript/bounds artifacts and runtime-configured analyze artifacts.
+Define Narratio artifact identity, catalog, and source-resolution behavior for:
+- built-in session artifacts;
+- configured analyze artifacts;
+- canonical previous-session artifact sources.
 
 ## Inputs and outputs
 Inputs:
-- artifact sources from config/runtime (`pipeline.scriptorium.artifacts.*.inputs.*.source`)
-- session paths and optional session manifest stage outputs
-- runtime artifact catalog state for configured artifact sources
+- configured input sources (`pipeline.scriptorium.artifacts.*.inputs.*.source`);
+- session paths and manifest inputs/outputs;
+- runtime catalog state.
 
 Outputs:
-- resolved local artifact path and provenance (`ResolvedSessionArtifact`)
-- runtime catalog entries for planned/executable/available artifacts
-- validation errors for unsupported, missing, or invalid artifact sources
+- resolved artifact path + provenance (`ResolvedSessionArtifact`);
+- runtime catalog entries for built-ins and configured artifacts;
+- requirement sets for canonical previous-session inputs.
 
 ## Boundaries
 Owns:
-- built-in artifact registry and content validation rules
-- runtime artifact catalog for configured artifact source IDs
-- source resolution behavior for built-in and configured artifact sources
+- built-in source registry and validation;
+- configured artifact catalog identity (`narratio.artifact.<name>`);
+- canonical previous-session source parsing and resolution;
+- previous-session requirement collection (`CollectPreviousArtifactRequirements`).
 
 Does not own:
-- artifact generation (stages produce files)
-- manifest transition policy
-- archive promotion behavior
-
-## Config fields used
-- `pipeline.scriptorium.artifacts.<name>.enabled`
-- `pipeline.scriptorium.artifacts.<name>.output_path`
-- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
-
-## External adapters used
-- none
-
-## State and manifest behavior
-Built-in registry entries:
+- prepare-stage remote hydration;
+- stage success/skip transitions;
+- archive upload orchestration.
 
+## Built-in IDs
 | Artifact ID | Canonical file | Producer stage | Output kind |
 | --- | --- | --- | --- |
 | `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` |
@@ -44,44 +38,58 @@ Built-in registry entries:
 | `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` |
 | `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` |
 
-Runtime catalog entries include built-ins and configured `narratio.artifact.<name>` sources.
+## Source families
+- built-in: `narratio.transcript.*`, `narratio.bounds.session`
+- configured artifact: `narratio.artifact.<artifact_key>`
+- canonical previous-session artifact: `narratio.previous_session.artifact.<artifact_key>`
 
-Catalog states:
-- `planned`: source is registered and known for this run
-- `executable`: configured artifact is selected for analyze execution
-- `available`: artifact has a usable file path (generated this run or reused from disk)
+## Runtime catalog model
+Catalog entries track:
+- `planned`: source is registered for this run;
+- `executable`: configured artifact is selected for analyze execution;
+- `available`: usable local file exists (generated this run or reused from disk).
 
-Resolution behavior:
-- built-in sources resolve via manifest producer outputs first, then canonical fallback path
-- configured `narratio.artifact.<name>` sources resolve through runtime catalog availability
-- configured source lookup requires catalog context
-
-Configured artifact provenance values:
+Configured artifact provenance values include:
 - `generated.current_analyze_run`
 - `filesystem.disabled_artifact_output`
 
-Content validation:
-- transcript built-ins: JSON with top-level `segments` array
-- bounds built-in: valid JSON
-- configured artifacts: non-empty text file
+Previous-session canonical provenance values include:
+- `manifest.inputs.previous_cache`
+- `current_session.previous_cache`
 
-## Skip and resume behavior
-- resolver and catalog have no direct skip/resume decisions
-- stage/runner skip-resume behavior consumes catalog/resolver results
+## Resolution behavior
+- Built-ins resolve via manifest producer outputs first, then canonical fallback paths.
+- Configured `narratio.artifact.<name>` sources resolve through catalog availability.
+- Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths.
+- Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`.
+
+## Previous-session requirement scanning
+`CollectPreviousArtifactRequirements`:
+- scans enabled configured artifacts only;
+- includes canonical previous-session sources only;
+- deduplicates by artifact key;
+- merges required/optional references (`required` wins);
+- records deterministic sorted source locations for diagnostics.
+
+## Validation behavior
+- transcript built-ins: JSON with top-level `segments` array;
+- bounds built-in: valid JSON;
+- configured and previous-session artifact files: non-empty text content.
 
 ## Failure behavior
-- unsupported source -> source validation error
-- known source unavailable -> `ErrSessionArtifactNotFound`
-- configured source without catalog -> resolution error
-- resolved file with invalid content -> validation error
+- unsupported source or malformed canonical previous source: validation/resolution error;
+- known source unavailable: `ErrSessionArtifactNotFound`;
+- configured/previous canonical source without catalog: error;
+- resolved invalid file content: validation error.
 
 ## Tests to inspect before changing
 - `internal/artifacts/artifact_resolver_test.go`
 - `internal/artifacts/catalog_test.go`
+- `internal/artifacts/previous_requirements_test.go`
+- `internal/stage/prepare_previous_test.go`
 - `internal/stage/analyze_test.go`
-- `internal/config/scriptorium_test.go`
 
 ## Architectural invariants
-- built-in IDs are static and registry-backed
-- configured artifact IDs are runtime-derived (`narratio.artifact.<name>`) and catalog-backed
-- built-in/source resolution remains deterministic and validation-gated
+- Built-in source IDs are static.
+- Configured and previous-session source IDs are artifact-key based and validation-gated.
+- Resolution behavior remains deterministic and manifest-aware.

docs/internal/command-restore.md

@@ -1,11 +1,11 @@
 # Internal: Command Restore
 
 ## Purpose
-Define the implemented `narratio restore` command contract: committed remote-state discovery, deterministic planning, safe file installation, conflict policy, and restore reporting.
+Define the implemented `narratio restore` contract: committed remote-state discovery, deterministic plan classification, safe file install semantics, and restore reporting.
 
 ## Inputs and outputs
 Inputs:
-- CLI flags: `--config`, `--session`, `--session-id`, `--dry-run`, `--force`, `--include-audio`.
+- CLI flags: `--config`, `--session`, `--session-id`, `--previous-session-id`, `--dry-run`, `--force`, `--include-audio`.
 - Resolved/validated `pipeline.yml` and `session.yml`.
 - Configured remote object store.
 - Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`).
@@ -54,6 +54,21 @@ Does not own:
   - existing local manifest is preserved if restored manifest validation/install fails.
 - Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`.
 
+Restore path scope:
+- includes:
+  - `manifest.json`
+  - `transcripts/**`
+  - `artifacts/**`
+  - `previous/**`
+  - `audio/**` only when `--include-audio` is set
+- excludes:
+  - `runs/**`
+  - `logs/**`
+  - `reports/**`
+  - `config/**`
+  - `inputs/**`
+  - remote `current/**` pointer files as local restore targets
+
 ## Skip and resume behavior
 - Restore does not participate in stage skip/resume decisions.
 - Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state.
@@ -80,7 +95,7 @@ Does not own:
 - `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files.
 - Local path mapping is traversal-safe and constrained to session root.
 - Restore scope is deterministic and path-classified:
-  - include `manifest.json`, `transcripts/**`, `artifacts/**`
+  - include `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`
   - include `audio/**` only with `--include-audio`
   - exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`
 - Command remains standalone; no implicit `run --restore` behavior.

docs/internal/stage-analyze.md

@@ -3,31 +3,36 @@
 ## Purpose
 Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths.
 
-## Inputs and Outputs
+## Inputs and outputs
 Inputs:
-- configured artifact definitions from `pipeline.scriptorium.artifacts`
-- selected artifact filter from runtime (`--artifacts`) when provided
-- resolved artifact input sources declared per artifact (`inputs.*.source`)
-- optional previous-session file inputs (`previous_session_artifact`)
+- configured artifact definitions from `pipeline.scriptorium.artifacts`;
+- selected artifact filter (`--artifacts`) when provided;
+- resolved artifact sources from resolver/catalog.
+
+Source types used by analyze:
+- built-ins: `narratio.transcript.*`, `narratio.bounds.session`;
+- configured artifacts: `narratio.artifact.<artifact_key>`;
+- canonical previous-session artifacts: `narratio.previous_session.artifact.<artifact_key>`;
+- legacy path-based previous-session source: `previous_session_artifact` (uses `inputs.*.path`).
 
 Outputs:
-- one promoted output file per executed configured artifact at that artifact's configured `output_path`
-- stage metadata containing generated artifact entries and reused disabled-artifact entries
+- promoted configured artifact files at each configured `output_path`;
+- stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info).
 
 ## Boundaries
 Owns:
-- runtime artifact catalog construction for analyze execution
-- selected-artifact planning and dependency ordering
-- per-artifact input resolution, var resolution, timeout/render-debug resolution
-- Scriptorium run/render invocation for each selected artifact
-- run-local output generation and canonical promotion
+- runtime artifact catalog construction;
+- selected-artifact planning and dependency ordering;
+- per-input resolution and required/optional handling;
+- Scriptorium render/run invocation;
+- run-local output generation and canonical promotion.
 
 Does not own:
-- transcript generation/processing stages
-- archive promotion policy
-- per-artifact resume semantics
+- prepare-time previous-session hydration;
+- object-store access for previous-session sources;
+- archive promotion policy.
 
-## Config Fields Used
+## Config fields used
 - `session.session_id`
 - `session.campaign`
 - `pipeline.workspace.root`
@@ -36,49 +41,41 @@ Does not own:
 - `pipeline.scriptorium.timeout`
 - `pipeline.scriptorium.render_debug`
 - `pipeline.scriptorium.artifacts.<name>.*`
-  - `enabled`
-  - `depends_on`
-  - `prompt_id`
-  - `profile_id`
-  - `timeout`
-  - `output_path`
-  - `render_debug`
-  - `inputs`
-  - `vars`
 
-## External Adapters Used
+## External adapters used
 - Scriptorium adapter:
-  - optional `RenderArtifact` (render debug)
-  - `RunArtifact` (artifact generation)
+  - optional `RenderArtifact` when render-debug is enabled;
+  - `RunArtifact` for artifact generation.
 
-## State and Manifest Behavior
-- If `pipeline.scriptorium` is absent, stage returns success metadata with `skipped=true`.
-- If no artifacts are configured, stage returns success metadata with `skipped=true`.
-- If zero artifacts are executable after `enabled` + `--artifacts` filtering, stage returns success metadata with `skipped=true`.
-- Builds runtime catalog with built-ins and configured artifacts.
-- Non-executable configured artifacts are marked available only when their configured output file exists and is valid on disk.
-- Executes selected configured artifacts in topological order with deterministic tie-breaking.
-- For each generated artifact, records metadata fields including `name`, `source_id`, `output_kind`, `path`, `prompt_id`, `profile_id`, and `provenance`.
-- Reused disabled artifacts are recorded separately in `reused_artifacts` with provenance `filesystem.disabled_artifact_output`.
+## State and manifest behavior
+- If Scriptorium config is absent, or no artifacts are executable after filtering, analyze returns success metadata with `skipped=true`.
+- Builds runtime catalog with built-ins and configured `narratio.artifact.<name>` entries.
+- Non-executable configured artifacts may still be marked available from existing canonical output files.
+- Resolves canonical previous-session sources from local prepared `previous/` cache:
+  - prefers manifest-backed previous input paths when present;
+  - may fall back to current-session `previous/` filesystem paths.
+- Analyze does not call object storage for canonical previous-session source resolution.
+- Required canonical previous-session input missing:
+  - fails with guidance to run `narratio run-stage --force prepare`.
+- Optional missing sources are omitted from adapter input paths.
 
-## Skip and Resume Behavior
+## Skip and resume behavior
 - Runner-level skip applies when analyze is already `succeeded` and `--force` is not set.
-- Analyze remains stage-scoped for resume/skip; there is no per-artifact resume state.
-- `--artifacts` filters which configured artifacts are executable when analyze runs; it does not imply `--force`.
+- Analyze is stage-scoped for resume; no per-artifact manifest resume state.
+- `--artifacts` filters executable artifacts but does not imply force rerun.
 
-## Failure Behavior
-- Fails on invalid dependency ordering, unavailable required configured inputs, invalid built-in input prerequisites, render/run adapter failures, validation-failed adapter results, or missing/empty outputs.
-- Required configured dependency missing from catalog availability fails clearly before invocation.
-- Optional missing inputs are omitted.
+## Failure behavior
+- Fails on dependency-order violations, missing required inputs, resolver validation failures, adapter errors, and missing/empty generated outputs.
+- Required unavailable configured artifact source (`narratio.artifact.<name>`) fails before invocation.
+- Required canonical previous-session source fails with prepare-rerun guidance.
 
-## Tests to Inspect Before Changing
+## Tests to inspect before changing
 - `internal/stage/analyze_test.go`
 - `internal/artifacts/catalog_test.go`
 - `internal/artifacts/artifact_resolver_test.go`
-- `internal/adapters/scriptorium/subprocess_test.go`
+- `internal/app/restore_workflow_test.go`
 
-## Architectural Invariants
-- Configured artifacts are identified by `narratio.artifact.<name>` source IDs.
-- Artifact-to-artifact references rely on explicit `depends_on` declarations validated in config.
-- Generated analyze outputs are treated uniformly as Scriptorium artifacts.
-- Successful outputs must exist and be non-empty before promotion.
+## Architectural invariants
+- Canonical previous-session behavior is local-cache only during analyze.
+- Generated outputs are validated and promoted before stage success is recorded.
+- Resolver/catalog decisions stay deterministic and validation-gated.

docs/internal/stage-archive.md

@@ -1,17 +1,19 @@
 # Stage: archive
 
 ## Purpose
-Publish run records and promoted session artifacts to object storage, then atomically advance the remote current pointer.
+Publish durable run/session state to object storage, then atomically advance remote current state.
 
 ## Inputs and Outputs
 Inputs:
 - session manifest and prerequisite stage records
 - run root contents under `runs/{run_id}/`
 - promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`)
+- session-level `previous/**` cache files when present
 
 Outputs:
 - uploaded run files under `{session_prefix}/runs/{run_id}/...`
 - uploaded promoted artifacts under `{session_prefix}/...`
+- uploaded session previous-cache files under `{session_prefix}/previous/...` when present
 - `{session_prefix}/current/manifest.json`
 - `{session_prefix}/current/run_id.txt` written last
 
@@ -21,6 +23,7 @@ Owns:
 - Prerequisite stage success enforcement
 - Run file collection and upload (excluding `audio/`)
 - Promotion rule resolution and upload
+- Session previous-cache file collection/upload
 - Commit pointer publish order
 
 Does not own:
@@ -43,8 +46,10 @@ Does not own:
 ## State and Manifest Behavior
 - Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
 - Resolves bucket/prefix from manifest identity first, then config fallback.
+- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
 - Writes metadata including:
   - upload counts/paths
+  - `previous_files_uploaded` and `previous_uploaded_paths`
   - `current_manifest_key`
   - `current_run_id_key`
   - `current_pointer_written`
@@ -64,5 +69,6 @@ Does not own:
 
 ## Architectural Invariants
 - Run upload excludes `audio/` subtree.
+- Session `previous/**` is archiveable durable input/provenance state, not run-local output.
 - `current/manifest.json` uploads before `current/run_id.txt`.
 - `current/run_id.txt` is the remote publish commit marker.

docs/internal/stage-prepare.md

@@ -1,40 +1,49 @@
 # Stage: prepare
 
 ## Purpose
-Materialize all required session inputs into canonical local workspace paths and record input provenance in the session manifest.
+Materialize canonical current-session input state and provenance before downstream stages run.
 
-## Inputs and Outputs
+Prepare owns:
+- local input file materialization (`inputs/**`);
+- audio input materialization (`audio/**`);
+- previous-session cache hydration (`previous/**`) for canonical previous-session artifact sources.
+
+## Inputs and outputs
 Inputs:
-- `session.yml` (resolved session config)
-- `pipeline.resolved.yml` (materialized from resolved pipeline config)
-- `speakers.yml`
-- `autocorrect.yml`
-- `glossary.yml`
+- resolved config/session (`pipeline.yml`, `session.yml`);
+- session-local input files (`speakers`, `autocorrect`, `glossary`);
 - audio source:
-  - local (`session.inputs.audio_dir` or `session.inputs.audio_files`), or
-  - S3 (`session.inputs.audio_s3.prefix`)
+  - local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
+  - S3: `session.inputs.audio_s3.prefix`;
+- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning);
+- remote previous-session current archive state when previous hydration is required.
 
 Outputs:
-- `inputs/session.yml`
-- `inputs/pipeline.resolved.yml`
-- `inputs/speakers.yml`
-- `inputs/autocorrect.yml`
-- `inputs/glossary.yml`
-- `audio/*.flac` in session workdir
-- `manifest.Inputs` records with checksums and source metadata
+- `inputs/session.yml`;
+- `inputs/pipeline.resolved.yml`;
+- `inputs/speakers.yml`;
+- `inputs/autocorrect.yml`;
+- `inputs/glossary.yml`;
+- `audio/*.flac` in canonical session `audio/`;
+- optional `previous/manifest.json`;
+- optional `previous/artifacts/**`;
+- deterministic `manifest.Inputs` records with checksums and provenance metadata.
 
 ## Boundaries
 Owns:
-- Input path resolution and validation
-- Local copy/materialization of configs and audio files
-- S3 audio download to run-scoped spool, then copy into work audio dir
+- input path resolution and materialization;
+- S3 audio list/download/copy flow;
+- previous-session artifact requirement collection from enabled configured artifacts;
+- previous cache lifecycle when requirements exist (clear and rehydrate managed `previous/` state).
 
 Does not own:
-- Transcript generation/processing
-- Archive publish behavior
+- transcript or artifact generation;
+- analyze-stage source resolution;
+- archive commit behavior.
 
-## Config Fields Used
+## Config fields used
 - `session.session_id`
+- `session.previous_session_id`
 - `session.campaign`
 - `session.inputs.speakers_file`
 - `session.inputs.autocorrect_file`
@@ -46,29 +55,56 @@ Does not own:
 - `pipeline.spool.root`
 - `pipeline.storage.s3.bucket`
 - `pipeline.storage.s3.root_prefix`
+- `pipeline.scriptorium.artifacts.<name>.enabled`
+- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
+- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required`
 
-## External Adapters Used
-- Object storage backend (`env.ObjectStore`) for S3 audio list/download when `audio_s3` is configured.
+## External adapters used
+- `storage.ObjectStore` for:
+  - S3 audio listing/downloads;
+  - previous-session current pointer/manifest/artifact object checks and downloads.
 
-## State and Manifest Behavior
+## State and manifest behavior
 - Ensures workspace layout exists.
-- Writes resolved config and input files to canonical `inputs/` paths.
-- Records all prepared inputs into `manifest.Inputs` (sorted deterministically by kind/path).
-- For S3 audio, records `S3Bucket`, `S3Key`, `S3Size`, `S3ETag`, and `SpoolPath` in each audio input record.
+- Materializes canonical input files and audio files.
+- Scans enabled configured artifact inputs for canonical sources:
+  - `narratio.previous_session.artifact.<artifact_key>`
+- If one or more canonical previous-session requirements exist:
+  - clears managed `previous/` state;
+  - hydrates required/optional previous artifacts from the configured previous session’s committed archive current state;
+  - writes `previous/manifest.json` and hydrated `previous/artifacts/**`;
+  - records hydrated previous inputs in `manifest.Inputs` with source `previous_session_archive.current`.
+- If no canonical previous-session requirements exist, prepare does not manage `previous/`.
+- `manifest.Inputs` is sorted deterministically by `(kind, path)`.
 
-## Skip and Resume Behavior
-- Runner-level skip applies when stage already `succeeded` and `--force` is not set.
-- Stage itself is deterministic/idempotent for unchanged inputs (`copyFileIfChanged`, `writeBytesIfChanged`).
+## Required and optional previous-session behavior
+- `previous_session_id` unset:
+  - if any referenced previous artifact is required: fail;
+  - if all referenced previous artifacts are optional: continue and omit them.
+- Previous session archive current pointer or manifest missing:
+  - if any referenced previous artifact is required: fail;
+  - if all referenced previous artifacts are optional: continue and omit missing ones.
+- Missing required previous artifact object: fail.
+- Missing optional previous artifact object: omit.
+- Downloaded previous artifacts must validate as non-empty files.
 
-## Failure Behavior
-- Fails on missing required files, invalid audio source combinations, no discoverable `.flac` files, duplicate audio basenames, missing object store for S3 mode, or S3 list/download failures.
+## Skip and resume behavior
+- Runner-level skip remains authoritative:
+  - if `prepare` already succeeded and run is not forced, `prepare` does not run and no hydration/download occurs.
+- If `prepare` runs (including with `--force`), it owns managed `previous/` state for canonical previous-session inputs.
 
-## Tests to Inspect Before Changing
+## Failure behavior
+- Fails on missing required input files, invalid audio-source combinations, empty/duplicate audio inputs, missing object store for S3 modes, and remote access/download/validation errors.
+- For required canonical previous-session inputs, analyze-time missing-input guidance is to rerun:
+  - `narratio run-stage --force prepare`
+
+## Tests to inspect before changing
 - `internal/stage/prepare_test.go`
-- `internal/app/session_cli_test.go`
-- `internal/config/load_validate_test.go`
+- `internal/stage/prepare_previous_test.go`
+- `internal/artifacts/previous_requirements_test.go`
+- `internal/app/runner_test.go`
 
-## Architectural Invariants
+## Architectural invariants
 - `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
-- Audio files must be `.flac`.
-- Canonical `inputs/*` and `audio/*` paths are the durable source for downstream stages.
+- Storage keys are computed by callers using archive/path helpers; storage adapter receives explicit keys.
+- `prepare` is the only stage that hydrates canonical previous-session cache state.