eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Finalize previous-session artifact documentation and restore-analyze continuity coverage

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-20 15:17:04 +00:00
parent 5be831eb13
commit bc2ade38d9
12 changed files with 369 additions and 155 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/cli.md
View File

@@ -196,7 +196,7 @@ Common failure cases:
### `restore` ### `restore`
Purpose: Purpose:
- Restore durable session state (`manifest.json`, `transcripts/**`, `artifacts/**`, and optional `audio/**`) from the committed remote archive current state. - Restore durable session state (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, and optional `audio/**`) from the committed remote archive current state.
Syntax: Syntax:
@@ -265,6 +265,12 @@ narratio restore --session-id 2026-04-04
narratio run-stage --session-id 2026-04-04 --force analyze narratio run-stage --session-id 2026-04-04 --force analyze
``` ```
Rehydrate canonical previous-session inputs after artifact-input changes:
```bash
narratio run-stage --session-id 2026-04-04 --force prepare
```
## Diagnostic / Recovery Commands ## Diagnostic / Recovery Commands
Inspect stage status: Inspect stage status:

26
docs/config.md
View File

@@ -73,7 +73,6 @@ Why this is sufficient:
```yaml ```yaml
session_id: "{{ session_id }}" session_id: "{{ session_id }}"
previous_session_id: "{{ previous_session_id }}"
campaign: sample-campaign campaign: sample-campaign
inputs: inputs:
audio_dir: ./audio audio_dir: ./audio
@@ -88,6 +87,23 @@ Usage:
narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03 narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03
``` ```
Previous-session-enabled variant:
```yaml
session_id: "{{ session_id }}"
previous_session_id: "{{ previous_session_id }}"
campaign: sample-campaign
inputs:
audio_dir: ./audio
speakers_file: ./examples/speakers.yml
autocorrect_file: ./examples/autocorrect.yml
glossary_file: ./examples/glossary.yml
```
```bash
narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03 --previous-session-id 2026-04-26
```
## 6. Production-oriented config ## 6. Production-oriented config
```yaml ```yaml
@@ -132,6 +148,9 @@ scriptorium:
transcript: transcript:
source: narratio.transcript.trimmed source: narratio.transcript.trimmed
required: true required: true
previous_recap:
source: narratio.previous_session.artifact.session_recap
required: false
``` ```
Operational notes: Operational notes:
@@ -248,7 +267,6 @@ Scriptorium artifact-key and dependency rules:
Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values: Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
- `previous_session_artifact`
- `narratio.previous_session.artifact.<configured_artifact_key>` - `narratio.previous_session.artifact.<configured_artifact_key>`
- `narratio.transcript.merged` - `narratio.transcript.merged`
- `narratio.transcript.polished` - `narratio.transcript.polished`
@@ -256,6 +274,7 @@ Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
- `narratio.transcript.trimmed` - `narratio.transcript.trimmed`
- `narratio.bounds.session` - `narratio.bounds.session`
- `narratio.artifact.<configured_artifact_key>` - `narratio.artifact.<configured_artifact_key>`
- `previous_session_artifact` (legacy path-based source; uses `inputs.<key>.path`)
`pipeline.archive.promote_artifacts[].source` values: `pipeline.archive.promote_artifacts[].source` values:
@@ -278,7 +297,7 @@ Archive promotion destination rules:
Restore-related implications: Restore-related implications:
- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs). - restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs).
- restore scope considers only committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, optional `audio/**`). - restore scope considers committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, optional `audio/**`).
## 8. Full session reference ## 8. Full session reference
@@ -307,6 +326,7 @@ Audio-source rule:
Previous-session rule: Previous-session rule:
- if `session.previous_session_id` is set, it must not equal `session.session_id`. - if `session.previous_session_id` is set, it must not equal `session.session_id`.
- canonical previous-session sources (`narratio.previous_session.artifact.<name>`) are hydrated during `prepare` from archive current state when required by enabled configured artifacts.
## 9. Secrets ## 9. Secrets

108
docs/internal/artifacts.md
View File

@@ -1,41 +1,35 @@
# Internal: Artifacts # Internal: Artifacts
## Purpose ## Purpose
Define Narratio's artifact identity and resolution model for built-in transcript/bounds artifacts and runtime-configured analyze artifacts. Define Narratio artifact identity, catalog, and source-resolution behavior for:
- built-in session artifacts;
- configured analyze artifacts;
- canonical previous-session artifact sources.
## Inputs and outputs ## Inputs and outputs
Inputs: Inputs:
- artifact sources from config/runtime (`pipeline.scriptorium.artifacts.*.inputs.*.source`) - configured input sources (`pipeline.scriptorium.artifacts.*.inputs.*.source`);
- session paths and optional session manifest stage outputs - session paths and manifest inputs/outputs;
- runtime artifact catalog state for configured artifact sources - runtime catalog state.
Outputs: Outputs:
- resolved local artifact path and provenance (`ResolvedSessionArtifact`) - resolved artifact path + provenance (`ResolvedSessionArtifact`);
- runtime catalog entries for planned/executable/available artifacts - runtime catalog entries for built-ins and configured artifacts;
- validation errors for unsupported, missing, or invalid artifact sources - requirement sets for canonical previous-session inputs.
## Boundaries ## Boundaries
Owns: Owns:
- built-in artifact registry and content validation rules - built-in source registry and validation;
- runtime artifact catalog for configured artifact source IDs - configured artifact catalog identity (`narratio.artifact.<name>`);
- source resolution behavior for built-in and configured artifact sources - canonical previous-session source parsing and resolution;
- previous-session requirement collection (`CollectPreviousArtifactRequirements`).
Does not own: Does not own:
- artifact generation (stages produce files) - prepare-stage remote hydration;
- manifest transition policy - stage success/skip transitions;
- archive promotion behavior - archive upload orchestration.
## Config fields used
- `pipeline.scriptorium.artifacts.<name>.enabled`
- `pipeline.scriptorium.artifacts.<name>.output_path`
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
## External adapters used
- none
## State and manifest behavior
Built-in registry entries:
## Built-in IDs
| Artifact ID | Canonical file | Producer stage | Output kind | | Artifact ID | Canonical file | Producer stage | Output kind |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | | `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` |
@@ -44,44 +38,58 @@ Built-in registry entries:
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | | `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` |
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | | `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` |
Runtime catalog entries include built-ins and configured `narratio.artifact.<name>` sources. ## Source families
- built-in: `narratio.transcript.*`, `narratio.bounds.session`
- configured artifact: `narratio.artifact.<artifact_key>`
- canonical previous-session artifact: `narratio.previous_session.artifact.<artifact_key>`
Catalog states: ## Runtime catalog model
- `planned`: source is registered and known for this run Catalog entries track:
- `executable`: configured artifact is selected for analyze execution - `planned`: source is registered for this run;
- `available`: artifact has a usable file path (generated this run or reused from disk) - `executable`: configured artifact is selected for analyze execution;
- `available`: usable local file exists (generated this run or reused from disk).
Resolution behavior: Configured artifact provenance values include:
- built-in sources resolve via manifest producer outputs first, then canonical fallback path
- configured `narratio.artifact.<name>` sources resolve through runtime catalog availability
- configured source lookup requires catalog context
Configured artifact provenance values:
- `generated.current_analyze_run` - `generated.current_analyze_run`
- `filesystem.disabled_artifact_output` - `filesystem.disabled_artifact_output`
Content validation: Previous-session canonical provenance values include:
- transcript built-ins: JSON with top-level `segments` array - `manifest.inputs.previous_cache`
- bounds built-in: valid JSON - `current_session.previous_cache`
- configured artifacts: non-empty text file
## Skip and resume behavior ## Resolution behavior
- resolver and catalog have no direct skip/resume decisions - Built-ins resolve via manifest producer outputs first, then canonical fallback paths.
- stage/runner skip-resume behavior consumes catalog/resolver results - Configured `narratio.artifact.<name>` sources resolve through catalog availability.
- Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths.
- Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`.
## Previous-session requirement scanning
`CollectPreviousArtifactRequirements`:
- scans enabled configured artifacts only;
- includes canonical previous-session sources only;
- deduplicates by artifact key;
- merges required/optional references (`required` wins);
- records deterministic sorted source locations for diagnostics.
## Validation behavior
- transcript built-ins: JSON with top-level `segments` array;
- bounds built-in: valid JSON;
- configured and previous-session artifact files: non-empty text content.
## Failure behavior ## Failure behavior
- unsupported source -> source validation error - unsupported source or malformed canonical previous source: validation/resolution error;
- known source unavailable -> `ErrSessionArtifactNotFound` - known source unavailable: `ErrSessionArtifactNotFound`;
- configured source without catalog -> resolution error - configured/previous canonical source without catalog: error;
- resolved file with invalid content -> validation error - resolved invalid file content: validation error.
## Tests to inspect before changing ## Tests to inspect before changing
- `internal/artifacts/artifact_resolver_test.go` - `internal/artifacts/artifact_resolver_test.go`
- `internal/artifacts/catalog_test.go` - `internal/artifacts/catalog_test.go`
- `internal/artifacts/previous_requirements_test.go`
- `internal/stage/prepare_previous_test.go`
- `internal/stage/analyze_test.go` - `internal/stage/analyze_test.go`
- `internal/config/scriptorium_test.go`
## Architectural invariants ## Architectural invariants
- built-in IDs are static and registry-backed - Built-in source IDs are static.
- configured artifact IDs are runtime-derived (`narratio.artifact.<name>`) and catalog-backed - Configured and previous-session source IDs are artifact-key based and validation-gated.
- built-in/source resolution remains deterministic and validation-gated - Resolution behavior remains deterministic and manifest-aware.

21
docs/internal/command-restore.md
View File

@@ -1,11 +1,11 @@
# Internal: Command Restore # Internal: Command Restore
## Purpose ## Purpose
Define the implemented `narratio restore` command contract: committed remote-state discovery, deterministic planning, safe file installation, conflict policy, and restore reporting. Define the implemented `narratio restore` contract: committed remote-state discovery, deterministic plan classification, safe file install semantics, and restore reporting.
## Inputs and outputs ## Inputs and outputs
Inputs: Inputs:
- CLI flags: `--config`, `--session`, `--session-id`, `--dry-run`, `--force`, `--include-audio`. - CLI flags: `--config`, `--session`, `--session-id`, `--previous-session-id`, `--dry-run`, `--force`, `--include-audio`.
- Resolved/validated `pipeline.yml` and `session.yml`. - Resolved/validated `pipeline.yml` and `session.yml`.
- Configured remote object store. - Configured remote object store.
- Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`). - Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`).
@@ -54,6 +54,21 @@ Does not own:
- existing local manifest is preserved if restored manifest validation/install fails. - existing local manifest is preserved if restored manifest validation/install fails.
- Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`. - Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`.
Restore path scope:
- includes:
- `manifest.json`
- `transcripts/**`
- `artifacts/**`
- `previous/**`
- `audio/**` only when `--include-audio` is set
- excludes:
- `runs/**`
- `logs/**`
- `reports/**`
- `config/**`
- `inputs/**`
- remote `current/**` pointer files as local restore targets
## Skip and resume behavior ## Skip and resume behavior
- Restore does not participate in stage skip/resume decisions. - Restore does not participate in stage skip/resume decisions.
- Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state. - Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state.
@@ -80,7 +95,7 @@ Does not own:
- `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files. - `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files.
- Local path mapping is traversal-safe and constrained to session root. - Local path mapping is traversal-safe and constrained to session root.
- Restore scope is deterministic and path-classified: - Restore scope is deterministic and path-classified:
- include `manifest.json`, `transcripts/**`, `artifacts/**` - include `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`
- include `audio/**` only with `--include-audio` - include `audio/**` only with `--include-audio`
- exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**` - exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`
- Command remains standalone; no implicit `run --restore` behavior. - Command remains standalone; no implicit `run --restore` behavior.

99
docs/internal/stage-analyze.md
View File

@@ -3,31 +3,36 @@
## Purpose ## Purpose
Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths. Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths.
## Inputs and Outputs ## Inputs and outputs
Inputs: Inputs:
- configured artifact definitions from `pipeline.scriptorium.artifacts` - configured artifact definitions from `pipeline.scriptorium.artifacts`;
- selected artifact filter from runtime (`--artifacts`) when provided - selected artifact filter (`--artifacts`) when provided;
- resolved artifact input sources declared per artifact (`inputs.*.source`) - resolved artifact sources from resolver/catalog.
- optional previous-session file inputs (`previous_session_artifact`)
Source types used by analyze:
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`;
- configured artifacts: `narratio.artifact.<artifact_key>`;
- canonical previous-session artifacts: `narratio.previous_session.artifact.<artifact_key>`;
- legacy path-based previous-session source: `previous_session_artifact` (uses `inputs.*.path`).
Outputs: Outputs:
- one promoted output file per executed configured artifact at that artifact's configured `output_path` - promoted configured artifact files at each configured `output_path`;
- stage metadata containing generated artifact entries and reused disabled-artifact entries - stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info).
## Boundaries ## Boundaries
Owns: Owns:
- runtime artifact catalog construction for analyze execution - runtime artifact catalog construction;
- selected-artifact planning and dependency ordering - selected-artifact planning and dependency ordering;
- per-artifact input resolution, var resolution, timeout/render-debug resolution - per-input resolution and required/optional handling;
- Scriptorium run/render invocation for each selected artifact - Scriptorium render/run invocation;
- run-local output generation and canonical promotion - run-local output generation and canonical promotion.
Does not own: Does not own:
- transcript generation/processing stages - prepare-time previous-session hydration;
- archive promotion policy - object-store access for previous-session sources;
- per-artifact resume semantics - archive promotion policy.
## Config Fields Used ## Config fields used
- `session.session_id` - `session.session_id`
- `session.campaign` - `session.campaign`
- `pipeline.workspace.root` - `pipeline.workspace.root`
@@ -36,49 +41,41 @@ Does not own:
- `pipeline.scriptorium.timeout` - `pipeline.scriptorium.timeout`
- `pipeline.scriptorium.render_debug` - `pipeline.scriptorium.render_debug`
- `pipeline.scriptorium.artifacts.<name>.*` - `pipeline.scriptorium.artifacts.<name>.*`
- `enabled`
- `depends_on`
- `prompt_id`
- `profile_id`
- `timeout`
- `output_path`
- `render_debug`
- `inputs`
- `vars`
## External Adapters Used ## External adapters used
- Scriptorium adapter: - Scriptorium adapter:
- optional `RenderArtifact` (render debug) - optional `RenderArtifact` when render-debug is enabled;
- `RunArtifact` (artifact generation) - `RunArtifact` for artifact generation.
## State and Manifest Behavior ## State and manifest behavior
- If `pipeline.scriptorium` is absent, stage returns success metadata with `skipped=true`. - If Scriptorium config is absent, or no artifacts are executable after filtering, analyze returns success metadata with `skipped=true`.
- If no artifacts are configured, stage returns success metadata with `skipped=true`. - Builds runtime catalog with built-ins and configured `narratio.artifact.<name>` entries.
- If zero artifacts are executable after `enabled` + `--artifacts` filtering, stage returns success metadata with `skipped=true`. - Non-executable configured artifacts may still be marked available from existing canonical output files.
- Builds runtime catalog with built-ins and configured artifacts. - Resolves canonical previous-session sources from local prepared `previous/` cache:
- Non-executable configured artifacts are marked available only when their configured output file exists and is valid on disk. - prefers manifest-backed previous input paths when present;
- Executes selected configured artifacts in topological order with deterministic tie-breaking. - may fall back to current-session `previous/` filesystem paths.
- For each generated artifact, records metadata fields including `name`, `source_id`, `output_kind`, `path`, `prompt_id`, `profile_id`, and `provenance`. - Analyze does not call object storage for canonical previous-session source resolution.
- Reused disabled artifacts are recorded separately in `reused_artifacts` with provenance `filesystem.disabled_artifact_output`. - Required canonical previous-session input missing:
- fails with guidance to run `narratio run-stage --force prepare`.
- Optional missing sources are omitted from adapter input paths.
## Skip and Resume Behavior ## Skip and resume behavior
- Runner-level skip applies when analyze is already `succeeded` and `--force` is not set. - Runner-level skip applies when analyze is already `succeeded` and `--force` is not set.
- Analyze remains stage-scoped for resume/skip; there is no per-artifact resume state. - Analyze is stage-scoped for resume; no per-artifact manifest resume state.
- `--artifacts` filters which configured artifacts are executable when analyze runs; it does not imply `--force`. - `--artifacts` filters executable artifacts but does not imply force rerun.
## Failure Behavior ## Failure behavior
- Fails on invalid dependency ordering, unavailable required configured inputs, invalid built-in input prerequisites, render/run adapter failures, validation-failed adapter results, or missing/empty outputs. - Fails on dependency-order violations, missing required inputs, resolver validation failures, adapter errors, and missing/empty generated outputs.
- Required configured dependency missing from catalog availability fails clearly before invocation. - Required unavailable configured artifact source (`narratio.artifact.<name>`) fails before invocation.
- Optional missing inputs are omitted. - Required canonical previous-session source fails with prepare-rerun guidance.
## Tests to Inspect Before Changing ## Tests to inspect before changing
- `internal/stage/analyze_test.go` - `internal/stage/analyze_test.go`
- `internal/artifacts/catalog_test.go` - `internal/artifacts/catalog_test.go`
- `internal/artifacts/artifact_resolver_test.go` - `internal/artifacts/artifact_resolver_test.go`
- `internal/adapters/scriptorium/subprocess_test.go` - `internal/app/restore_workflow_test.go`
## Architectural Invariants ## Architectural invariants
- Configured artifacts are identified by `narratio.artifact.<name>` source IDs. - Canonical previous-session behavior is local-cache only during analyze.
- Artifact-to-artifact references rely on explicit `depends_on` declarations validated in config. - Generated outputs are validated and promoted before stage success is recorded.
- Generated analyze outputs are treated uniformly as Scriptorium artifacts. - Resolver/catalog decisions stay deterministic and validation-gated.
- Successful outputs must exist and be non-empty before promotion.

8
docs/internal/stage-archive.md
View File

@@ -1,17 +1,19 @@
# Stage: archive # Stage: archive
## Purpose ## Purpose
Publish run records and promoted session artifacts to object storage, then atomically advance the remote current pointer. Publish durable run/session state to object storage, then atomically advance remote current state.
## Inputs and Outputs ## Inputs and Outputs
Inputs: Inputs:
- session manifest and prerequisite stage records - session manifest and prerequisite stage records
- run root contents under `runs/{run_id}/` - run root contents under `runs/{run_id}/`
- promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`) - promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`)
- session-level `previous/**` cache files when present
Outputs: Outputs:
- uploaded run files under `{session_prefix}/runs/{run_id}/...` - uploaded run files under `{session_prefix}/runs/{run_id}/...`
- uploaded promoted artifacts under `{session_prefix}/...` - uploaded promoted artifacts under `{session_prefix}/...`
- uploaded session previous-cache files under `{session_prefix}/previous/...` when present
- `{session_prefix}/current/manifest.json` - `{session_prefix}/current/manifest.json`
- `{session_prefix}/current/run_id.txt` written last - `{session_prefix}/current/run_id.txt` written last
@@ -21,6 +23,7 @@ Owns:
- Prerequisite stage success enforcement - Prerequisite stage success enforcement
- Run file collection and upload (excluding `audio/`) - Run file collection and upload (excluding `audio/`)
- Promotion rule resolution and upload - Promotion rule resolution and upload
- Session previous-cache file collection/upload
- Commit pointer publish order - Commit pointer publish order
Does not own: Does not own:
@@ -43,8 +46,10 @@ Does not own:
## State and Manifest Behavior ## State and Manifest Behavior
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`. - Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
- Resolves bucket/prefix from manifest identity first, then config fallback. - Resolves bucket/prefix from manifest identity first, then config fallback.
- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
- Writes metadata including: - Writes metadata including:
- upload counts/paths - upload counts/paths
- `previous_files_uploaded` and `previous_uploaded_paths`
- `current_manifest_key` - `current_manifest_key`
- `current_run_id_key` - `current_run_id_key`
- `current_pointer_written` - `current_pointer_written`
@@ -64,5 +69,6 @@ Does not own:
## Architectural Invariants ## Architectural Invariants
- Run upload excludes `audio/` subtree. - Run upload excludes `audio/` subtree.
- Session `previous/**` is archiveable durable input/provenance state, not run-local output.
- `current/manifest.json` uploads before `current/run_id.txt`. - `current/manifest.json` uploads before `current/run_id.txt`.
- `current/run_id.txt` is the remote publish commit marker. - `current/run_id.txt` is the remote publish commit marker.

114
docs/internal/stage-prepare.md
View File

@@ -1,40 +1,49 @@
# Stage: prepare # Stage: prepare
## Purpose ## Purpose
Materialize all required session inputs into canonical local workspace paths and record input provenance in the session manifest. Materialize canonical current-session input state and provenance before downstream stages run.
## Inputs and Outputs Prepare owns:
- local input file materialization (`inputs/**`);
- audio input materialization (`audio/**`);
- previous-session cache hydration (`previous/**`) for canonical previous-session artifact sources.
## Inputs and outputs
Inputs: Inputs:
- `session.yml` (resolved session config) - resolved config/session (`pipeline.yml`, `session.yml`);
- `pipeline.resolved.yml` (materialized from resolved pipeline config) - session-local input files (`speakers`, `autocorrect`, `glossary`);
- `speakers.yml`
- `autocorrect.yml`
- `glossary.yml`
- audio source: - audio source:
- local (`session.inputs.audio_dir` or `session.inputs.audio_files`), or - local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
- S3 (`session.inputs.audio_s3.prefix`) - S3: `session.inputs.audio_s3.prefix`;
- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning);
- remote previous-session current archive state when previous hydration is required.
Outputs: Outputs:
- `inputs/session.yml` - `inputs/session.yml`;
- `inputs/pipeline.resolved.yml` - `inputs/pipeline.resolved.yml`;
- `inputs/speakers.yml` - `inputs/speakers.yml`;
- `inputs/autocorrect.yml` - `inputs/autocorrect.yml`;
- `inputs/glossary.yml` - `inputs/glossary.yml`;
- `audio/*.flac` in session workdir - `audio/*.flac` in canonical session `audio/`;
- `manifest.Inputs` records with checksums and source metadata - optional `previous/manifest.json`;
- optional `previous/artifacts/**`;
- deterministic `manifest.Inputs` records with checksums and provenance metadata.
## Boundaries ## Boundaries
Owns: Owns:
- Input path resolution and validation - input path resolution and materialization;
- Local copy/materialization of configs and audio files - S3 audio list/download/copy flow;
- S3 audio download to run-scoped spool, then copy into work audio dir - previous-session artifact requirement collection from enabled configured artifacts;
- previous cache lifecycle when requirements exist (clear and rehydrate managed `previous/` state).
Does not own: Does not own:
- Transcript generation/processing - transcript or artifact generation;
- Archive publish behavior - analyze-stage source resolution;
- archive commit behavior.
## Config Fields Used ## Config fields used
- `session.session_id` - `session.session_id`
- `session.previous_session_id`
- `session.campaign` - `session.campaign`
- `session.inputs.speakers_file` - `session.inputs.speakers_file`
- `session.inputs.autocorrect_file` - `session.inputs.autocorrect_file`
@@ -46,29 +55,56 @@ Does not own:
- `pipeline.spool.root` - `pipeline.spool.root`
- `pipeline.storage.s3.bucket` - `pipeline.storage.s3.bucket`
- `pipeline.storage.s3.root_prefix` - `pipeline.storage.s3.root_prefix`
- `pipeline.scriptorium.artifacts.<name>.enabled`
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required`
## External Adapters Used ## External adapters used
- Object storage backend (`env.ObjectStore`) for S3 audio list/download when `audio_s3` is configured. - `storage.ObjectStore` for:
- S3 audio listing/downloads;
- previous-session current pointer/manifest/artifact object checks and downloads.
## State and Manifest Behavior ## State and manifest behavior
- Ensures workspace layout exists. - Ensures workspace layout exists.
- Writes resolved config and input files to canonical `inputs/` paths. - Materializes canonical input files and audio files.
- Records all prepared inputs into `manifest.Inputs` (sorted deterministically by kind/path). - Scans enabled configured artifact inputs for canonical sources:
- For S3 audio, records `S3Bucket`, `S3Key`, `S3Size`, `S3ETag`, and `SpoolPath` in each audio input record. - `narratio.previous_session.artifact.<artifact_key>`
- If one or more canonical previous-session requirements exist:
- clears managed `previous/` state;
- hydrates required/optional previous artifacts from the configured previous session’s committed archive current state;
- writes `previous/manifest.json` and hydrated `previous/artifacts/**`;
- records hydrated previous inputs in `manifest.Inputs` with source `previous_session_archive.current`.
- If no canonical previous-session requirements exist, prepare does not manage `previous/`.
- `manifest.Inputs` is sorted deterministically by `(kind, path)`.
## Skip and Resume Behavior ## Required and optional previous-session behavior
- Runner-level skip applies when stage already `succeeded` and `--force` is not set. - `previous_session_id` unset:
- Stage itself is deterministic/idempotent for unchanged inputs (`copyFileIfChanged`, `writeBytesIfChanged`). - if any referenced previous artifact is required: fail;
- if all referenced previous artifacts are optional: continue and omit them.
- Previous session archive current pointer or manifest missing:
- if any referenced previous artifact is required: fail;
- if all referenced previous artifacts are optional: continue and omit missing ones.
- Missing required previous artifact object: fail.
- Missing optional previous artifact object: omit.
- Downloaded previous artifacts must validate as non-empty files.
## Failure Behavior ## Skip and resume behavior
- Fails on missing required files, invalid audio source combinations, no discoverable `.flac` files, duplicate audio basenames, missing object store for S3 mode, or S3 list/download failures. - Runner-level skip remains authoritative:
- if `prepare` already succeeded and run is not forced, `prepare` does not run and no hydration/download occurs.
- If `prepare` runs (including with `--force`), it owns managed `previous/` state for canonical previous-session inputs.
## Tests to Inspect Before Changing ## Failure behavior
- Fails on missing required input files, invalid audio-source combinations, empty/duplicate audio inputs, missing object store for S3 modes, and remote access/download/validation errors.
- For required canonical previous-session inputs, analyze-time missing-input guidance is to rerun:
- `narratio run-stage --force prepare`
## Tests to inspect before changing
- `internal/stage/prepare_test.go` - `internal/stage/prepare_test.go`
- `internal/app/session_cli_test.go` - `internal/stage/prepare_previous_test.go`
- `internal/config/load_validate_test.go` - `internal/artifacts/previous_requirements_test.go`
- `internal/app/runner_test.go`
## Architectural Invariants ## Architectural invariants
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive. - `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
- Audio files must be `.flac`. - Storage keys are computed by callers using archive/path helpers; storage adapter receives explicit keys.
- Canonical `inputs/*` and `audio/*` paths are the durable source for downstream stages. - `prepare` is the only stage that hydrates canonical previous-session cache state.

9
docs/operations.md
View File

@@ -48,7 +48,7 @@ Restore source-of-truth:
- remote current manifest: `current/manifest.json` - remote current manifest: `current/manifest.json`
Restore default scope: Restore default scope:
- includes `manifest.json`, `transcripts/**`, `artifacts/**` - includes `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`
- includes `audio/**` only with `--include-audio` - includes `audio/**` only with `--include-audio`
- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source) - excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source)
@@ -67,6 +67,7 @@ Canonical session directories:
- `audio/` - `audio/`
- `transcripts/` - `transcripts/`
- `artifacts/` - `artifacts/`
- `previous/`
- `reports/` - `reports/`
- `logs/` - `logs/`
- `config/` - `config/`
@@ -99,6 +100,12 @@ Configured artifact source reuse:
- accepted on `run`, `resume`, and `run-stage analyze`. - accepted on `run`, `resume`, and `run-stage analyze`.
- filters analyze execution only; does not force stage rerun. - filters analyze execution only; does not force stage rerun.
Canonical previous-session input behavior:
- canonical sources use `narratio.previous_session.artifact.<artifact_key>`.
- these inputs are hydrated by `prepare`, not `analyze`.
- if analyze fails due to missing canonical previous cache, rerun:
- `narratio run-stage --session-id <id> --force prepare`
## Remote archive layout and publish contract ## Remote archive layout and publish contract
When archive is enabled and run upload is enabled, archive publishes under: When archive is enabled and run upload is enabled, archive publishes under:

2
docs/roadmap/previous.md
View File

@@ -2,7 +2,7 @@
## Status ## Status
Planned. Completed.
This roadmap describes the implementation strategy for first-class previous-session artifact support in Narratio. It belongs under `docs/roadmap/previous.md` until the feature is implemented. After implementation, current behavior should be documented in the appropriate user-facing and internal documentation files, and this roadmap should be removed or marked complete according to the documentation policy. This roadmap describes the implementation strategy for first-class previous-session artifact support in Narratio. It belongs under `docs/roadmap/previous.md` until the feature is implemented. After implementation, current behavior should be documented in the appropriate user-facing and internal documentation files, and this roadmap should be removed or marked complete according to the documentation policy.

4
examples/pipeline.full.annotated.yml
View File

@@ -133,9 +133,7 @@ scriptorium:
source: narratio.transcript.trimmed source: narratio.transcript.trimmed
required: true required: true
previous_recap: previous_recap:
source: previous_session_artifact source: narratio.previous_session.artifact.session_recap
artifact: session_recap
path: ""
required: false required: false
vars: vars:
session_id: true session_id: true

3
examples/pipeline.production.yml
View File

@@ -81,8 +81,7 @@ scriptorium:
source: narratio.transcript.trimmed source: narratio.transcript.trimmed
required: true required: true
previous_recap: previous_recap:
source: previous_session_artifact source: narratio.previous_session.artifact.session_recap
artifact: session_recap
required: false required: false
vars: vars:
session_id: true session_id: true

122
internal/app/restore_workflow_test.go
View File

@@ -155,6 +155,116 @@ func TestRestoreThenRunStageForceAnalyzeUsesRestoredDurableState(t *testing.T) {
} }
} }
func TestRestoreThenAnalyzeUsesRestoredPreviousCacheWithoutObjectStore(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
appendRestoreWorkflowScriptoriumConfig(t, pipelinePath, `
scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
inputs:
transcript:
source: narratio.transcript.trimmed
required: true
previous_recap:
source: narratio.previous_session.artifact.session_recap
required: true
`)
fakeStore := &storage.FakeBackend{}
cfg, sessionPrefix, manifestKey, runIDKey := seedRestoreCommittedState(t, fakeStore, pipelinePath, sessionPath)
seedRestoreObject(fakeStore, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
seedRestoreObject(fakeStore, manifestKey, restoreWorkflowManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
seedRestoreObject(fakeStore, sessionPrefix+"transcripts/trimmed.json", []byte(`{"segments":[]}`+"\n"))
seedRestoreObject(fakeStore, sessionPrefix+"previous/manifest.json", []byte(`{"session_id":"2026-04-26"}`))
seedRestoreObject(fakeStore, sessionPrefix+"previous/artifacts/session_recap.md", []byte("# previous recap\n"))
restoreWithStoreAndRealPhases(t, fakeStore)
var stdout bytes.Buffer
var stderr bytes.Buffer
restoreCode := Execute(
[]string{
"restore",
"--config", pipelinePath,
"--session", sessionPath,
"--session-id", cfg.Session.SessionID,
},
&stdout,
&stderr,
)
if restoreCode != 0 {
t.Fatalf("restore exit code = %d, want 0; stderr=%q", restoreCode, stderr.String())
}
if stderr.Len() != 0 {
t.Fatalf("restore stderr = %q, want empty", stderr.String())
}
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), `{"segments":[]}`+"\n")
mustReadEquals(t, filepath.Join(sessionRoot, "previous", "manifest.json"), `{"session_id":"2026-04-26"}`)
mustReadEquals(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# previous recap\n")
scriptoriumFake := &scriptorium.FakeRunner{}
origExecuteStagesFn := executeStagesFn
origObjectStoreFn := newObjectStoreFromConfigFn
objectStoreConstructed := false
t.Cleanup(func() {
executeStagesFn = origExecuteStagesFn
newObjectStoreFromConfigFn = origObjectStoreFn
})
executeStagesFn = func(ctx context.Context, cfg *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
if opts.Env == nil {
opts.Env = &Env{}
}
opts.Env.Scriptorium = scriptoriumFake
return executeStages(ctx, cfg, stages, opts)
}
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
objectStoreConstructed = true
return nil, context.Canceled
}
stdout.Reset()
stderr.Reset()
runStageCode := Execute(
[]string{
"run-stage",
"--config", pipelinePath,
"--session", sessionPath,
"--session-id", cfg.Session.SessionID,
"--force",
"--artifacts", "session_recap",
"analyze",
},
&stdout,
&stderr,
)
if runStageCode != 0 {
t.Fatalf("run-stage exit code = %d, want 0; stderr=%q", runStageCode, stderr.String())
}
if stderr.Len() != 0 {
t.Fatalf("run-stage stderr = %q, want empty", stderr.String())
}
if objectStoreConstructed {
t.Fatal("analyze run-stage should not construct object store for previous-session input resolution")
}
if len(scriptoriumFake.RunRequests) != 1 {
t.Fatalf("scriptorium run requests = %d, want 1", len(scriptoriumFake.RunRequests))
}
req := scriptoriumFake.RunRequests[0]
if got := req.InputPaths["transcript"]; got != filepath.Join(sessionRoot, "transcripts", "trimmed.json") {
t.Fatalf("transcript input = %q, want trimmed transcript path", got)
}
if got := req.InputPaths["previous_recap"]; got != filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md") {
t.Fatalf("previous_recap input = %q, want restored previous cache path", got)
}
}
func restoreWorkflowManifestJSON(t *testing.T, sessionID, campaign string) []byte { func restoreWorkflowManifestJSON(t *testing.T, sessionID, campaign string) []byte {
t.Helper() t.Helper()
store := &manifest.LocalStore{} store := &manifest.LocalStore{}
@@ -176,3 +286,15 @@ func restoreWorkflowManifestJSON(t *testing.T, sessionID, campaign string) []byt
} }
return data return data
} }
func appendRestoreWorkflowScriptoriumConfig(t *testing.T, pipelinePath, extra string) {
t.Helper()
f, err := os.OpenFile(pipelinePath, os.O_APPEND|os.O_WRONLY, 0)
if err != nil {
t.Fatalf("open pipeline config for append: %v", err)
}
defer f.Close()
if _, err := f.WriteString(extra); err != nil {
t.Fatalf("append pipeline config: %v", err)
}
}