diff --git a/docs/cli.md b/docs/cli.md index 2d40241..8c0e5fc 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -196,7 +196,7 @@ Common failure cases: ### `restore` Purpose: -- Restore durable session state (`manifest.json`, `transcripts/**`, `artifacts/**`, and optional `audio/**`) from the committed remote archive current state. +- Restore durable session state (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, and optional `audio/**`) from the committed remote archive current state. Syntax: @@ -265,6 +265,12 @@ narratio restore --session-id 2026-04-04 narratio run-stage --session-id 2026-04-04 --force analyze ``` +Rehydrate canonical previous-session inputs after artifact-input changes: + +```bash +narratio run-stage --session-id 2026-04-04 --force prepare +``` + ## Diagnostic / Recovery Commands Inspect stage status: diff --git a/docs/config.md b/docs/config.md index 5de4611..046d700 100644 --- a/docs/config.md +++ b/docs/config.md @@ -73,7 +73,6 @@ Why this is sufficient: ```yaml session_id: "{{ session_id }}" -previous_session_id: "{{ previous_session_id }}" campaign: sample-campaign inputs: audio_dir: ./audio @@ -88,6 +87,23 @@ Usage: narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03 ``` +Previous-session-enabled variant: + +```yaml +session_id: "{{ session_id }}" +previous_session_id: "{{ previous_session_id }}" +campaign: sample-campaign +inputs: + audio_dir: ./audio + speakers_file: ./examples/speakers.yml + autocorrect_file: ./examples/autocorrect.yml + glossary_file: ./examples/glossary.yml +``` + +```bash +narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03 --previous-session-id 2026-04-26 +``` + ## 6. Production-oriented config ```yaml @@ -132,6 +148,9 @@ scriptorium: transcript: source: narratio.transcript.trimmed required: true + previous_recap: + source: narratio.previous_session.artifact.session_recap + required: false ``` Operational notes: @@ -248,7 +267,6 @@ Scriptorium artifact-key and dependency rules: Allowed `pipeline.scriptorium.artifacts..inputs..source` values: -- `previous_session_artifact` - `narratio.previous_session.artifact.` - `narratio.transcript.merged` - `narratio.transcript.polished` @@ -256,6 +274,7 @@ Allowed `pipeline.scriptorium.artifacts..inputs..source` values: - `narratio.transcript.trimmed` - `narratio.bounds.session` - `narratio.artifact.` +- `previous_session_artifact` (legacy path-based source; uses `inputs..path`) `pipeline.archive.promote_artifacts[].source` values: @@ -278,7 +297,7 @@ Archive promotion destination rules: Restore-related implications: - restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs). -- restore scope considers only committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, optional `audio/**`). +- restore scope considers committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, optional `audio/**`). ## 8. Full session reference @@ -307,6 +326,7 @@ Audio-source rule: Previous-session rule: - if `session.previous_session_id` is set, it must not equal `session.session_id`. +- canonical previous-session sources (`narratio.previous_session.artifact.`) are hydrated during `prepare` from archive current state when required by enabled configured artifacts. ## 9. Secrets diff --git a/docs/internal/artifacts.md b/docs/internal/artifacts.md index 1b4d518..87c1b97 100644 --- a/docs/internal/artifacts.md +++ b/docs/internal/artifacts.md @@ -1,41 +1,35 @@ # Internal: Artifacts ## Purpose -Define Narratio's artifact identity and resolution model for built-in transcript/bounds artifacts and runtime-configured analyze artifacts. +Define Narratio artifact identity, catalog, and source-resolution behavior for: +- built-in session artifacts; +- configured analyze artifacts; +- canonical previous-session artifact sources. ## Inputs and outputs Inputs: -- artifact sources from config/runtime (`pipeline.scriptorium.artifacts.*.inputs.*.source`) -- session paths and optional session manifest stage outputs -- runtime artifact catalog state for configured artifact sources +- configured input sources (`pipeline.scriptorium.artifacts.*.inputs.*.source`); +- session paths and manifest inputs/outputs; +- runtime catalog state. Outputs: -- resolved local artifact path and provenance (`ResolvedSessionArtifact`) -- runtime catalog entries for planned/executable/available artifacts -- validation errors for unsupported, missing, or invalid artifact sources +- resolved artifact path + provenance (`ResolvedSessionArtifact`); +- runtime catalog entries for built-ins and configured artifacts; +- requirement sets for canonical previous-session inputs. ## Boundaries Owns: -- built-in artifact registry and content validation rules -- runtime artifact catalog for configured artifact source IDs -- source resolution behavior for built-in and configured artifact sources +- built-in source registry and validation; +- configured artifact catalog identity (`narratio.artifact.`); +- canonical previous-session source parsing and resolution; +- previous-session requirement collection (`CollectPreviousArtifactRequirements`). Does not own: -- artifact generation (stages produce files) -- manifest transition policy -- archive promotion behavior - -## Config fields used -- `pipeline.scriptorium.artifacts..enabled` -- `pipeline.scriptorium.artifacts..output_path` -- `pipeline.scriptorium.artifacts..inputs..source` - -## External adapters used -- none - -## State and manifest behavior -Built-in registry entries: +- prepare-stage remote hydration; +- stage success/skip transitions; +- archive upload orchestration. +## Built-in IDs | Artifact ID | Canonical file | Producer stage | Output kind | | --- | --- | --- | --- | | `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | @@ -44,44 +38,58 @@ Built-in registry entries: | `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | | `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | -Runtime catalog entries include built-ins and configured `narratio.artifact.` sources. +## Source families +- built-in: `narratio.transcript.*`, `narratio.bounds.session` +- configured artifact: `narratio.artifact.` +- canonical previous-session artifact: `narratio.previous_session.artifact.` -Catalog states: -- `planned`: source is registered and known for this run -- `executable`: configured artifact is selected for analyze execution -- `available`: artifact has a usable file path (generated this run or reused from disk) +## Runtime catalog model +Catalog entries track: +- `planned`: source is registered for this run; +- `executable`: configured artifact is selected for analyze execution; +- `available`: usable local file exists (generated this run or reused from disk). -Resolution behavior: -- built-in sources resolve via manifest producer outputs first, then canonical fallback path -- configured `narratio.artifact.` sources resolve through runtime catalog availability -- configured source lookup requires catalog context - -Configured artifact provenance values: +Configured artifact provenance values include: - `generated.current_analyze_run` - `filesystem.disabled_artifact_output` -Content validation: -- transcript built-ins: JSON with top-level `segments` array -- bounds built-in: valid JSON -- configured artifacts: non-empty text file +Previous-session canonical provenance values include: +- `manifest.inputs.previous_cache` +- `current_session.previous_cache` -## Skip and resume behavior -- resolver and catalog have no direct skip/resume decisions -- stage/runner skip-resume behavior consumes catalog/resolver results +## Resolution behavior +- Built-ins resolve via manifest producer outputs first, then canonical fallback paths. +- Configured `narratio.artifact.` sources resolve through catalog availability. +- Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths. +- Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`. + +## Previous-session requirement scanning +`CollectPreviousArtifactRequirements`: +- scans enabled configured artifacts only; +- includes canonical previous-session sources only; +- deduplicates by artifact key; +- merges required/optional references (`required` wins); +- records deterministic sorted source locations for diagnostics. + +## Validation behavior +- transcript built-ins: JSON with top-level `segments` array; +- bounds built-in: valid JSON; +- configured and previous-session artifact files: non-empty text content. ## Failure behavior -- unsupported source -> source validation error -- known source unavailable -> `ErrSessionArtifactNotFound` -- configured source without catalog -> resolution error -- resolved file with invalid content -> validation error +- unsupported source or malformed canonical previous source: validation/resolution error; +- known source unavailable: `ErrSessionArtifactNotFound`; +- configured/previous canonical source without catalog: error; +- resolved invalid file content: validation error. ## Tests to inspect before changing - `internal/artifacts/artifact_resolver_test.go` - `internal/artifacts/catalog_test.go` +- `internal/artifacts/previous_requirements_test.go` +- `internal/stage/prepare_previous_test.go` - `internal/stage/analyze_test.go` -- `internal/config/scriptorium_test.go` ## Architectural invariants -- built-in IDs are static and registry-backed -- configured artifact IDs are runtime-derived (`narratio.artifact.`) and catalog-backed -- built-in/source resolution remains deterministic and validation-gated +- Built-in source IDs are static. +- Configured and previous-session source IDs are artifact-key based and validation-gated. +- Resolution behavior remains deterministic and manifest-aware. diff --git a/docs/internal/command-restore.md b/docs/internal/command-restore.md index 44be344..3d1ed30 100644 --- a/docs/internal/command-restore.md +++ b/docs/internal/command-restore.md @@ -1,11 +1,11 @@ # Internal: Command Restore ## Purpose -Define the implemented `narratio restore` command contract: committed remote-state discovery, deterministic planning, safe file installation, conflict policy, and restore reporting. +Define the implemented `narratio restore` contract: committed remote-state discovery, deterministic plan classification, safe file install semantics, and restore reporting. ## Inputs and outputs Inputs: -- CLI flags: `--config`, `--session`, `--session-id`, `--dry-run`, `--force`, `--include-audio`. +- CLI flags: `--config`, `--session`, `--session-id`, `--previous-session-id`, `--dry-run`, `--force`, `--include-audio`. - Resolved/validated `pipeline.yml` and `session.yml`. - Configured remote object store. - Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`). @@ -54,6 +54,21 @@ Does not own: - existing local manifest is preserved if restored manifest validation/install fails. - Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`. +Restore path scope: +- includes: + - `manifest.json` + - `transcripts/**` + - `artifacts/**` + - `previous/**` + - `audio/**` only when `--include-audio` is set +- excludes: + - `runs/**` + - `logs/**` + - `reports/**` + - `config/**` + - `inputs/**` + - remote `current/**` pointer files as local restore targets + ## Skip and resume behavior - Restore does not participate in stage skip/resume decisions. - Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state. @@ -80,7 +95,7 @@ Does not own: - `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files. - Local path mapping is traversal-safe and constrained to session root. - Restore scope is deterministic and path-classified: - - include `manifest.json`, `transcripts/**`, `artifacts/**` + - include `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**` - include `audio/**` only with `--include-audio` - exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**` - Command remains standalone; no implicit `run --restore` behavior. diff --git a/docs/internal/stage-analyze.md b/docs/internal/stage-analyze.md index 11fd9d5..8aed41b 100644 --- a/docs/internal/stage-analyze.md +++ b/docs/internal/stage-analyze.md @@ -3,31 +3,36 @@ ## Purpose Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths. -## Inputs and Outputs +## Inputs and outputs Inputs: -- configured artifact definitions from `pipeline.scriptorium.artifacts` -- selected artifact filter from runtime (`--artifacts`) when provided -- resolved artifact input sources declared per artifact (`inputs.*.source`) -- optional previous-session file inputs (`previous_session_artifact`) +- configured artifact definitions from `pipeline.scriptorium.artifacts`; +- selected artifact filter (`--artifacts`) when provided; +- resolved artifact sources from resolver/catalog. + +Source types used by analyze: +- built-ins: `narratio.transcript.*`, `narratio.bounds.session`; +- configured artifacts: `narratio.artifact.`; +- canonical previous-session artifacts: `narratio.previous_session.artifact.`; +- legacy path-based previous-session source: `previous_session_artifact` (uses `inputs.*.path`). Outputs: -- one promoted output file per executed configured artifact at that artifact's configured `output_path` -- stage metadata containing generated artifact entries and reused disabled-artifact entries +- promoted configured artifact files at each configured `output_path`; +- stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info). ## Boundaries Owns: -- runtime artifact catalog construction for analyze execution -- selected-artifact planning and dependency ordering -- per-artifact input resolution, var resolution, timeout/render-debug resolution -- Scriptorium run/render invocation for each selected artifact -- run-local output generation and canonical promotion +- runtime artifact catalog construction; +- selected-artifact planning and dependency ordering; +- per-input resolution and required/optional handling; +- Scriptorium render/run invocation; +- run-local output generation and canonical promotion. Does not own: -- transcript generation/processing stages -- archive promotion policy -- per-artifact resume semantics +- prepare-time previous-session hydration; +- object-store access for previous-session sources; +- archive promotion policy. -## Config Fields Used +## Config fields used - `session.session_id` - `session.campaign` - `pipeline.workspace.root` @@ -36,49 +41,41 @@ Does not own: - `pipeline.scriptorium.timeout` - `pipeline.scriptorium.render_debug` - `pipeline.scriptorium.artifacts..*` - - `enabled` - - `depends_on` - - `prompt_id` - - `profile_id` - - `timeout` - - `output_path` - - `render_debug` - - `inputs` - - `vars` -## External Adapters Used +## External adapters used - Scriptorium adapter: - - optional `RenderArtifact` (render debug) - - `RunArtifact` (artifact generation) + - optional `RenderArtifact` when render-debug is enabled; + - `RunArtifact` for artifact generation. -## State and Manifest Behavior -- If `pipeline.scriptorium` is absent, stage returns success metadata with `skipped=true`. -- If no artifacts are configured, stage returns success metadata with `skipped=true`. -- If zero artifacts are executable after `enabled` + `--artifacts` filtering, stage returns success metadata with `skipped=true`. -- Builds runtime catalog with built-ins and configured artifacts. -- Non-executable configured artifacts are marked available only when their configured output file exists and is valid on disk. -- Executes selected configured artifacts in topological order with deterministic tie-breaking. -- For each generated artifact, records metadata fields including `name`, `source_id`, `output_kind`, `path`, `prompt_id`, `profile_id`, and `provenance`. -- Reused disabled artifacts are recorded separately in `reused_artifacts` with provenance `filesystem.disabled_artifact_output`. +## State and manifest behavior +- If Scriptorium config is absent, or no artifacts are executable after filtering, analyze returns success metadata with `skipped=true`. +- Builds runtime catalog with built-ins and configured `narratio.artifact.` entries. +- Non-executable configured artifacts may still be marked available from existing canonical output files. +- Resolves canonical previous-session sources from local prepared `previous/` cache: + - prefers manifest-backed previous input paths when present; + - may fall back to current-session `previous/` filesystem paths. +- Analyze does not call object storage for canonical previous-session source resolution. +- Required canonical previous-session input missing: + - fails with guidance to run `narratio run-stage --force prepare`. +- Optional missing sources are omitted from adapter input paths. -## Skip and Resume Behavior +## Skip and resume behavior - Runner-level skip applies when analyze is already `succeeded` and `--force` is not set. -- Analyze remains stage-scoped for resume/skip; there is no per-artifact resume state. -- `--artifacts` filters which configured artifacts are executable when analyze runs; it does not imply `--force`. +- Analyze is stage-scoped for resume; no per-artifact manifest resume state. +- `--artifacts` filters executable artifacts but does not imply force rerun. -## Failure Behavior -- Fails on invalid dependency ordering, unavailable required configured inputs, invalid built-in input prerequisites, render/run adapter failures, validation-failed adapter results, or missing/empty outputs. -- Required configured dependency missing from catalog availability fails clearly before invocation. -- Optional missing inputs are omitted. +## Failure behavior +- Fails on dependency-order violations, missing required inputs, resolver validation failures, adapter errors, and missing/empty generated outputs. +- Required unavailable configured artifact source (`narratio.artifact.`) fails before invocation. +- Required canonical previous-session source fails with prepare-rerun guidance. -## Tests to Inspect Before Changing +## Tests to inspect before changing - `internal/stage/analyze_test.go` - `internal/artifacts/catalog_test.go` - `internal/artifacts/artifact_resolver_test.go` -- `internal/adapters/scriptorium/subprocess_test.go` +- `internal/app/restore_workflow_test.go` -## Architectural Invariants -- Configured artifacts are identified by `narratio.artifact.` source IDs. -- Artifact-to-artifact references rely on explicit `depends_on` declarations validated in config. -- Generated analyze outputs are treated uniformly as Scriptorium artifacts. -- Successful outputs must exist and be non-empty before promotion. +## Architectural invariants +- Canonical previous-session behavior is local-cache only during analyze. +- Generated outputs are validated and promoted before stage success is recorded. +- Resolver/catalog decisions stay deterministic and validation-gated. diff --git a/docs/internal/stage-archive.md b/docs/internal/stage-archive.md index c19471f..10ea436 100644 --- a/docs/internal/stage-archive.md +++ b/docs/internal/stage-archive.md @@ -1,17 +1,19 @@ # Stage: archive ## Purpose -Publish run records and promoted session artifacts to object storage, then atomically advance the remote current pointer. +Publish durable run/session state to object storage, then atomically advance remote current state. ## Inputs and Outputs Inputs: - session manifest and prerequisite stage records - run root contents under `runs/{run_id}/` - promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`) +- session-level `previous/**` cache files when present Outputs: - uploaded run files under `{session_prefix}/runs/{run_id}/...` - uploaded promoted artifacts under `{session_prefix}/...` +- uploaded session previous-cache files under `{session_prefix}/previous/...` when present - `{session_prefix}/current/manifest.json` - `{session_prefix}/current/run_id.txt` written last @@ -21,6 +23,7 @@ Owns: - Prerequisite stage success enforcement - Run file collection and upload (excluding `audio/`) - Promotion rule resolution and upload +- Session previous-cache file collection/upload - Commit pointer publish order Does not own: @@ -43,8 +46,10 @@ Does not own: ## State and Manifest Behavior - Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`. - Resolves bucket/prefix from manifest identity first, then config fallback. +- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists. - Writes metadata including: - upload counts/paths + - `previous_files_uploaded` and `previous_uploaded_paths` - `current_manifest_key` - `current_run_id_key` - `current_pointer_written` @@ -64,5 +69,6 @@ Does not own: ## Architectural Invariants - Run upload excludes `audio/` subtree. +- Session `previous/**` is archiveable durable input/provenance state, not run-local output. - `current/manifest.json` uploads before `current/run_id.txt`. - `current/run_id.txt` is the remote publish commit marker. diff --git a/docs/internal/stage-prepare.md b/docs/internal/stage-prepare.md index 0c77787..51effcb 100644 --- a/docs/internal/stage-prepare.md +++ b/docs/internal/stage-prepare.md @@ -1,40 +1,49 @@ # Stage: prepare ## Purpose -Materialize all required session inputs into canonical local workspace paths and record input provenance in the session manifest. +Materialize canonical current-session input state and provenance before downstream stages run. -## Inputs and Outputs +Prepare owns: +- local input file materialization (`inputs/**`); +- audio input materialization (`audio/**`); +- previous-session cache hydration (`previous/**`) for canonical previous-session artifact sources. + +## Inputs and outputs Inputs: -- `session.yml` (resolved session config) -- `pipeline.resolved.yml` (materialized from resolved pipeline config) -- `speakers.yml` -- `autocorrect.yml` -- `glossary.yml` +- resolved config/session (`pipeline.yml`, `session.yml`); +- session-local input files (`speakers`, `autocorrect`, `glossary`); - audio source: - - local (`session.inputs.audio_dir` or `session.inputs.audio_files`), or - - S3 (`session.inputs.audio_s3.prefix`) + - local: `session.inputs.audio_dir` or `session.inputs.audio_files`; + - S3: `session.inputs.audio_s3.prefix`; +- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning); +- remote previous-session current archive state when previous hydration is required. Outputs: -- `inputs/session.yml` -- `inputs/pipeline.resolved.yml` -- `inputs/speakers.yml` -- `inputs/autocorrect.yml` -- `inputs/glossary.yml` -- `audio/*.flac` in session workdir -- `manifest.Inputs` records with checksums and source metadata +- `inputs/session.yml`; +- `inputs/pipeline.resolved.yml`; +- `inputs/speakers.yml`; +- `inputs/autocorrect.yml`; +- `inputs/glossary.yml`; +- `audio/*.flac` in canonical session `audio/`; +- optional `previous/manifest.json`; +- optional `previous/artifacts/**`; +- deterministic `manifest.Inputs` records with checksums and provenance metadata. ## Boundaries Owns: -- Input path resolution and validation -- Local copy/materialization of configs and audio files -- S3 audio download to run-scoped spool, then copy into work audio dir +- input path resolution and materialization; +- S3 audio list/download/copy flow; +- previous-session artifact requirement collection from enabled configured artifacts; +- previous cache lifecycle when requirements exist (clear and rehydrate managed `previous/` state). Does not own: -- Transcript generation/processing -- Archive publish behavior +- transcript or artifact generation; +- analyze-stage source resolution; +- archive commit behavior. -## Config Fields Used +## Config fields used - `session.session_id` +- `session.previous_session_id` - `session.campaign` - `session.inputs.speakers_file` - `session.inputs.autocorrect_file` @@ -46,29 +55,56 @@ Does not own: - `pipeline.spool.root` - `pipeline.storage.s3.bucket` - `pipeline.storage.s3.root_prefix` +- `pipeline.scriptorium.artifacts..enabled` +- `pipeline.scriptorium.artifacts..inputs..source` +- `pipeline.scriptorium.artifacts..inputs..required` -## External Adapters Used -- Object storage backend (`env.ObjectStore`) for S3 audio list/download when `audio_s3` is configured. +## External adapters used +- `storage.ObjectStore` for: + - S3 audio listing/downloads; + - previous-session current pointer/manifest/artifact object checks and downloads. -## State and Manifest Behavior +## State and manifest behavior - Ensures workspace layout exists. -- Writes resolved config and input files to canonical `inputs/` paths. -- Records all prepared inputs into `manifest.Inputs` (sorted deterministically by kind/path). -- For S3 audio, records `S3Bucket`, `S3Key`, `S3Size`, `S3ETag`, and `SpoolPath` in each audio input record. +- Materializes canonical input files and audio files. +- Scans enabled configured artifact inputs for canonical sources: + - `narratio.previous_session.artifact.` +- If one or more canonical previous-session requirements exist: + - clears managed `previous/` state; + - hydrates required/optional previous artifacts from the configured previous session’s committed archive current state; + - writes `previous/manifest.json` and hydrated `previous/artifacts/**`; + - records hydrated previous inputs in `manifest.Inputs` with source `previous_session_archive.current`. +- If no canonical previous-session requirements exist, prepare does not manage `previous/`. +- `manifest.Inputs` is sorted deterministically by `(kind, path)`. -## Skip and Resume Behavior -- Runner-level skip applies when stage already `succeeded` and `--force` is not set. -- Stage itself is deterministic/idempotent for unchanged inputs (`copyFileIfChanged`, `writeBytesIfChanged`). +## Required and optional previous-session behavior +- `previous_session_id` unset: + - if any referenced previous artifact is required: fail; + - if all referenced previous artifacts are optional: continue and omit them. +- Previous session archive current pointer or manifest missing: + - if any referenced previous artifact is required: fail; + - if all referenced previous artifacts are optional: continue and omit missing ones. +- Missing required previous artifact object: fail. +- Missing optional previous artifact object: omit. +- Downloaded previous artifacts must validate as non-empty files. -## Failure Behavior -- Fails on missing required files, invalid audio source combinations, no discoverable `.flac` files, duplicate audio basenames, missing object store for S3 mode, or S3 list/download failures. +## Skip and resume behavior +- Runner-level skip remains authoritative: + - if `prepare` already succeeded and run is not forced, `prepare` does not run and no hydration/download occurs. +- If `prepare` runs (including with `--force`), it owns managed `previous/` state for canonical previous-session inputs. -## Tests to Inspect Before Changing +## Failure behavior +- Fails on missing required input files, invalid audio-source combinations, empty/duplicate audio inputs, missing object store for S3 modes, and remote access/download/validation errors. +- For required canonical previous-session inputs, analyze-time missing-input guidance is to rerun: + - `narratio run-stage --force prepare` + +## Tests to inspect before changing - `internal/stage/prepare_test.go` -- `internal/app/session_cli_test.go` -- `internal/config/load_validate_test.go` +- `internal/stage/prepare_previous_test.go` +- `internal/artifacts/previous_requirements_test.go` +- `internal/app/runner_test.go` -## Architectural Invariants +## Architectural invariants - `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive. -- Audio files must be `.flac`. -- Canonical `inputs/*` and `audio/*` paths are the durable source for downstream stages. +- Storage keys are computed by callers using archive/path helpers; storage adapter receives explicit keys. +- `prepare` is the only stage that hydrates canonical previous-session cache state. diff --git a/docs/operations.md b/docs/operations.md index b6dda00..f461ee5 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -48,7 +48,7 @@ Restore source-of-truth: - remote current manifest: `current/manifest.json` Restore default scope: -- includes `manifest.json`, `transcripts/**`, `artifacts/**` +- includes `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**` - includes `audio/**` only with `--include-audio` - excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source) @@ -67,6 +67,7 @@ Canonical session directories: - `audio/` - `transcripts/` - `artifacts/` +- `previous/` - `reports/` - `logs/` - `config/` @@ -99,6 +100,12 @@ Configured artifact source reuse: - accepted on `run`, `resume`, and `run-stage analyze`. - filters analyze execution only; does not force stage rerun. +Canonical previous-session input behavior: +- canonical sources use `narratio.previous_session.artifact.`. +- these inputs are hydrated by `prepare`, not `analyze`. +- if analyze fails due to missing canonical previous cache, rerun: + - `narratio run-stage --session-id --force prepare` + ## Remote archive layout and publish contract When archive is enabled and run upload is enabled, archive publishes under: diff --git a/docs/roadmap/previous.md b/docs/roadmap/previous.md index 45fe36f..9eaa728 100644 --- a/docs/roadmap/previous.md +++ b/docs/roadmap/previous.md @@ -2,7 +2,7 @@ ## Status -Planned. +Completed. This roadmap describes the implementation strategy for first-class previous-session artifact support in Narratio. It belongs under `docs/roadmap/previous.md` until the feature is implemented. After implementation, current behavior should be documented in the appropriate user-facing and internal documentation files, and this roadmap should be removed or marked complete according to the documentation policy. diff --git a/examples/pipeline.full.annotated.yml b/examples/pipeline.full.annotated.yml index 5d28291..b5cb7a9 100644 --- a/examples/pipeline.full.annotated.yml +++ b/examples/pipeline.full.annotated.yml @@ -133,9 +133,7 @@ scriptorium: source: narratio.transcript.trimmed required: true previous_recap: - source: previous_session_artifact - artifact: session_recap - path: "" + source: narratio.previous_session.artifact.session_recap required: false vars: session_id: true diff --git a/examples/pipeline.production.yml b/examples/pipeline.production.yml index bc928e5..e26fc52 100644 --- a/examples/pipeline.production.yml +++ b/examples/pipeline.production.yml @@ -81,8 +81,7 @@ scriptorium: source: narratio.transcript.trimmed required: true previous_recap: - source: previous_session_artifact - artifact: session_recap + source: narratio.previous_session.artifact.session_recap required: false vars: session_id: true diff --git a/internal/app/restore_workflow_test.go b/internal/app/restore_workflow_test.go index 15f001e..12ce4cc 100644 --- a/internal/app/restore_workflow_test.go +++ b/internal/app/restore_workflow_test.go @@ -155,6 +155,116 @@ func TestRestoreThenRunStageForceAnalyzeUsesRestoredDurableState(t *testing.T) { } } +func TestRestoreThenAnalyzeUsesRestoredPreviousCacheWithoutObjectStore(t *testing.T) { + workspaceRoot := t.TempDir() + pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot) + appendRestoreWorkflowScriptoriumConfig(t, pipelinePath, ` +scriptorium: + binary: scriptorium + artifacts: + session_recap: + enabled: true + prompt_id: dnd.session_recap + output_path: artifacts/session_recap.md + inputs: + transcript: + source: narratio.transcript.trimmed + required: true + previous_recap: + source: narratio.previous_session.artifact.session_recap + required: true +`) + + fakeStore := &storage.FakeBackend{} + cfg, sessionPrefix, manifestKey, runIDKey := seedRestoreCommittedState(t, fakeStore, pipelinePath, sessionPath) + seedRestoreObject(fakeStore, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n")) + seedRestoreObject(fakeStore, manifestKey, restoreWorkflowManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign)) + seedRestoreObject(fakeStore, sessionPrefix+"transcripts/trimmed.json", []byte(`{"segments":[]}`+"\n")) + seedRestoreObject(fakeStore, sessionPrefix+"previous/manifest.json", []byte(`{"session_id":"2026-04-26"}`)) + seedRestoreObject(fakeStore, sessionPrefix+"previous/artifacts/session_recap.md", []byte("# previous recap\n")) + + restoreWithStoreAndRealPhases(t, fakeStore) + + var stdout bytes.Buffer + var stderr bytes.Buffer + restoreCode := Execute( + []string{ + "restore", + "--config", pipelinePath, + "--session", sessionPath, + "--session-id", cfg.Session.SessionID, + }, + &stdout, + &stderr, + ) + if restoreCode != 0 { + t.Fatalf("restore exit code = %d, want 0; stderr=%q", restoreCode, stderr.String()) + } + if stderr.Len() != 0 { + t.Fatalf("restore stderr = %q, want empty", stderr.String()) + } + + sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID) + mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), `{"segments":[]}`+"\n") + mustReadEquals(t, filepath.Join(sessionRoot, "previous", "manifest.json"), `{"session_id":"2026-04-26"}`) + mustReadEquals(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# previous recap\n") + + scriptoriumFake := &scriptorium.FakeRunner{} + origExecuteStagesFn := executeStagesFn + origObjectStoreFn := newObjectStoreFromConfigFn + objectStoreConstructed := false + t.Cleanup(func() { + executeStagesFn = origExecuteStagesFn + newObjectStoreFromConfigFn = origObjectStoreFn + }) + executeStagesFn = func(ctx context.Context, cfg *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) { + if opts.Env == nil { + opts.Env = &Env{} + } + opts.Env.Scriptorium = scriptoriumFake + return executeStages(ctx, cfg, stages, opts) + } + newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) { + objectStoreConstructed = true + return nil, context.Canceled + } + + stdout.Reset() + stderr.Reset() + runStageCode := Execute( + []string{ + "run-stage", + "--config", pipelinePath, + "--session", sessionPath, + "--session-id", cfg.Session.SessionID, + "--force", + "--artifacts", "session_recap", + "analyze", + }, + &stdout, + &stderr, + ) + if runStageCode != 0 { + t.Fatalf("run-stage exit code = %d, want 0; stderr=%q", runStageCode, stderr.String()) + } + if stderr.Len() != 0 { + t.Fatalf("run-stage stderr = %q, want empty", stderr.String()) + } + if objectStoreConstructed { + t.Fatal("analyze run-stage should not construct object store for previous-session input resolution") + } + if len(scriptoriumFake.RunRequests) != 1 { + t.Fatalf("scriptorium run requests = %d, want 1", len(scriptoriumFake.RunRequests)) + } + req := scriptoriumFake.RunRequests[0] + if got := req.InputPaths["transcript"]; got != filepath.Join(sessionRoot, "transcripts", "trimmed.json") { + t.Fatalf("transcript input = %q, want trimmed transcript path", got) + } + if got := req.InputPaths["previous_recap"]; got != filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md") { + t.Fatalf("previous_recap input = %q, want restored previous cache path", got) + } +} + func restoreWorkflowManifestJSON(t *testing.T, sessionID, campaign string) []byte { t.Helper() store := &manifest.LocalStore{} @@ -176,3 +286,15 @@ func restoreWorkflowManifestJSON(t *testing.T, sessionID, campaign string) []byt } return data } + +func appendRestoreWorkflowScriptoriumConfig(t *testing.T, pipelinePath, extra string) { + t.Helper() + f, err := os.OpenFile(pipelinePath, os.O_APPEND|os.O_WRONLY, 0) + if err != nil { + t.Fatalf("open pipeline config for append: %v", err) + } + defer f.Close() + if _, err := f.WriteString(extra); err != nil { + t.Fatalf("append pipeline config: %v", err) + } +}