Finalize previous-session artifact documentation and restore-analyze continuity coverage
This commit is contained in:
@@ -196,7 +196,7 @@ Common failure cases:
|
||||
### `restore`
|
||||
|
||||
Purpose:
|
||||
- Restore durable session state (`manifest.json`, `transcripts/**`, `artifacts/**`, and optional `audio/**`) from the committed remote archive current state.
|
||||
- Restore durable session state (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, and optional `audio/**`) from the committed remote archive current state.
|
||||
|
||||
Syntax:
|
||||
|
||||
@@ -265,6 +265,12 @@ narratio restore --session-id 2026-04-04
|
||||
narratio run-stage --session-id 2026-04-04 --force analyze
|
||||
```
|
||||
|
||||
Rehydrate canonical previous-session inputs after artifact-input changes:
|
||||
|
||||
```bash
|
||||
narratio run-stage --session-id 2026-04-04 --force prepare
|
||||
```
|
||||
|
||||
## Diagnostic / Recovery Commands
|
||||
|
||||
Inspect stage status:
|
||||
|
||||
@@ -73,7 +73,6 @@ Why this is sufficient:
|
||||
|
||||
```yaml
|
||||
session_id: "{{ session_id }}"
|
||||
previous_session_id: "{{ previous_session_id }}"
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
@@ -88,6 +87,23 @@ Usage:
|
||||
narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03
|
||||
```
|
||||
|
||||
Previous-session-enabled variant:
|
||||
|
||||
```yaml
|
||||
session_id: "{{ session_id }}"
|
||||
previous_session_id: "{{ previous_session_id }}"
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
```
|
||||
|
||||
```bash
|
||||
narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03 --previous-session-id 2026-04-26
|
||||
```
|
||||
|
||||
## 6. Production-oriented config
|
||||
|
||||
```yaml
|
||||
@@ -132,6 +148,9 @@ scriptorium:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: narratio.previous_session.artifact.session_recap
|
||||
required: false
|
||||
```
|
||||
|
||||
Operational notes:
|
||||
@@ -248,7 +267,6 @@ Scriptorium artifact-key and dependency rules:
|
||||
|
||||
Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
|
||||
|
||||
- `previous_session_artifact`
|
||||
- `narratio.previous_session.artifact.<configured_artifact_key>`
|
||||
- `narratio.transcript.merged`
|
||||
- `narratio.transcript.polished`
|
||||
@@ -256,6 +274,7 @@ Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
|
||||
- `narratio.transcript.trimmed`
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.<configured_artifact_key>`
|
||||
- `previous_session_artifact` (legacy path-based source; uses `inputs.<key>.path`)
|
||||
|
||||
`pipeline.archive.promote_artifacts[].source` values:
|
||||
|
||||
@@ -278,7 +297,7 @@ Archive promotion destination rules:
|
||||
Restore-related implications:
|
||||
|
||||
- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs).
|
||||
- restore scope considers only committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, optional `audio/**`).
|
||||
- restore scope considers committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, optional `audio/**`).
|
||||
|
||||
## 8. Full session reference
|
||||
|
||||
@@ -307,6 +326,7 @@ Audio-source rule:
|
||||
Previous-session rule:
|
||||
|
||||
- if `session.previous_session_id` is set, it must not equal `session.session_id`.
|
||||
- canonical previous-session sources (`narratio.previous_session.artifact.<name>`) are hydrated during `prepare` from archive current state when required by enabled configured artifacts.
|
||||
|
||||
## 9. Secrets
|
||||
|
||||
|
||||
@@ -1,41 +1,35 @@
|
||||
# Internal: Artifacts
|
||||
|
||||
## Purpose
|
||||
Define Narratio's artifact identity and resolution model for built-in transcript/bounds artifacts and runtime-configured analyze artifacts.
|
||||
Define Narratio artifact identity, catalog, and source-resolution behavior for:
|
||||
- built-in session artifacts;
|
||||
- configured analyze artifacts;
|
||||
- canonical previous-session artifact sources.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- artifact sources from config/runtime (`pipeline.scriptorium.artifacts.*.inputs.*.source`)
|
||||
- session paths and optional session manifest stage outputs
|
||||
- runtime artifact catalog state for configured artifact sources
|
||||
- configured input sources (`pipeline.scriptorium.artifacts.*.inputs.*.source`);
|
||||
- session paths and manifest inputs/outputs;
|
||||
- runtime catalog state.
|
||||
|
||||
Outputs:
|
||||
- resolved local artifact path and provenance (`ResolvedSessionArtifact`)
|
||||
- runtime catalog entries for planned/executable/available artifacts
|
||||
- validation errors for unsupported, missing, or invalid artifact sources
|
||||
- resolved artifact path + provenance (`ResolvedSessionArtifact`);
|
||||
- runtime catalog entries for built-ins and configured artifacts;
|
||||
- requirement sets for canonical previous-session inputs.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- built-in artifact registry and content validation rules
|
||||
- runtime artifact catalog for configured artifact source IDs
|
||||
- source resolution behavior for built-in and configured artifact sources
|
||||
- built-in source registry and validation;
|
||||
- configured artifact catalog identity (`narratio.artifact.<name>`);
|
||||
- canonical previous-session source parsing and resolution;
|
||||
- previous-session requirement collection (`CollectPreviousArtifactRequirements`).
|
||||
|
||||
Does not own:
|
||||
- artifact generation (stages produce files)
|
||||
- manifest transition policy
|
||||
- archive promotion behavior
|
||||
|
||||
## Config fields used
|
||||
- `pipeline.scriptorium.artifacts.<name>.enabled`
|
||||
- `pipeline.scriptorium.artifacts.<name>.output_path`
|
||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
|
||||
|
||||
## External adapters used
|
||||
- none
|
||||
|
||||
## State and manifest behavior
|
||||
Built-in registry entries:
|
||||
- prepare-stage remote hydration;
|
||||
- stage success/skip transitions;
|
||||
- archive upload orchestration.
|
||||
|
||||
## Built-in IDs
|
||||
| Artifact ID | Canonical file | Producer stage | Output kind |
|
||||
| --- | --- | --- | --- |
|
||||
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` |
|
||||
@@ -44,44 +38,58 @@ Built-in registry entries:
|
||||
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` |
|
||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` |
|
||||
|
||||
Runtime catalog entries include built-ins and configured `narratio.artifact.<name>` sources.
|
||||
## Source families
|
||||
- built-in: `narratio.transcript.*`, `narratio.bounds.session`
|
||||
- configured artifact: `narratio.artifact.<artifact_key>`
|
||||
- canonical previous-session artifact: `narratio.previous_session.artifact.<artifact_key>`
|
||||
|
||||
Catalog states:
|
||||
- `planned`: source is registered and known for this run
|
||||
- `executable`: configured artifact is selected for analyze execution
|
||||
- `available`: artifact has a usable file path (generated this run or reused from disk)
|
||||
## Runtime catalog model
|
||||
Catalog entries track:
|
||||
- `planned`: source is registered for this run;
|
||||
- `executable`: configured artifact is selected for analyze execution;
|
||||
- `available`: usable local file exists (generated this run or reused from disk).
|
||||
|
||||
Resolution behavior:
|
||||
- built-in sources resolve via manifest producer outputs first, then canonical fallback path
|
||||
- configured `narratio.artifact.<name>` sources resolve through runtime catalog availability
|
||||
- configured source lookup requires catalog context
|
||||
|
||||
Configured artifact provenance values:
|
||||
Configured artifact provenance values include:
|
||||
- `generated.current_analyze_run`
|
||||
- `filesystem.disabled_artifact_output`
|
||||
|
||||
Content validation:
|
||||
- transcript built-ins: JSON with top-level `segments` array
|
||||
- bounds built-in: valid JSON
|
||||
- configured artifacts: non-empty text file
|
||||
Previous-session canonical provenance values include:
|
||||
- `manifest.inputs.previous_cache`
|
||||
- `current_session.previous_cache`
|
||||
|
||||
## Skip and resume behavior
|
||||
- resolver and catalog have no direct skip/resume decisions
|
||||
- stage/runner skip-resume behavior consumes catalog/resolver results
|
||||
## Resolution behavior
|
||||
- Built-ins resolve via manifest producer outputs first, then canonical fallback paths.
|
||||
- Configured `narratio.artifact.<name>` sources resolve through catalog availability.
|
||||
- Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths.
|
||||
- Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`.
|
||||
|
||||
## Previous-session requirement scanning
|
||||
`CollectPreviousArtifactRequirements`:
|
||||
- scans enabled configured artifacts only;
|
||||
- includes canonical previous-session sources only;
|
||||
- deduplicates by artifact key;
|
||||
- merges required/optional references (`required` wins);
|
||||
- records deterministic sorted source locations for diagnostics.
|
||||
|
||||
## Validation behavior
|
||||
- transcript built-ins: JSON with top-level `segments` array;
|
||||
- bounds built-in: valid JSON;
|
||||
- configured and previous-session artifact files: non-empty text content.
|
||||
|
||||
## Failure behavior
|
||||
- unsupported source -> source validation error
|
||||
- known source unavailable -> `ErrSessionArtifactNotFound`
|
||||
- configured source without catalog -> resolution error
|
||||
- resolved file with invalid content -> validation error
|
||||
- unsupported source or malformed canonical previous source: validation/resolution error;
|
||||
- known source unavailable: `ErrSessionArtifactNotFound`;
|
||||
- configured/previous canonical source without catalog: error;
|
||||
- resolved invalid file content: validation error.
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/artifacts/artifact_resolver_test.go`
|
||||
- `internal/artifacts/catalog_test.go`
|
||||
- `internal/artifacts/previous_requirements_test.go`
|
||||
- `internal/stage/prepare_previous_test.go`
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/config/scriptorium_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- built-in IDs are static and registry-backed
|
||||
- configured artifact IDs are runtime-derived (`narratio.artifact.<name>`) and catalog-backed
|
||||
- built-in/source resolution remains deterministic and validation-gated
|
||||
- Built-in source IDs are static.
|
||||
- Configured and previous-session source IDs are artifact-key based and validation-gated.
|
||||
- Resolution behavior remains deterministic and manifest-aware.
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# Internal: Command Restore
|
||||
|
||||
## Purpose
|
||||
Define the implemented `narratio restore` command contract: committed remote-state discovery, deterministic planning, safe file installation, conflict policy, and restore reporting.
|
||||
Define the implemented `narratio restore` contract: committed remote-state discovery, deterministic plan classification, safe file install semantics, and restore reporting.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- CLI flags: `--config`, `--session`, `--session-id`, `--dry-run`, `--force`, `--include-audio`.
|
||||
- CLI flags: `--config`, `--session`, `--session-id`, `--previous-session-id`, `--dry-run`, `--force`, `--include-audio`.
|
||||
- Resolved/validated `pipeline.yml` and `session.yml`.
|
||||
- Configured remote object store.
|
||||
- Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`).
|
||||
@@ -54,6 +54,21 @@ Does not own:
|
||||
- existing local manifest is preserved if restored manifest validation/install fails.
|
||||
- Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`.
|
||||
|
||||
Restore path scope:
|
||||
- includes:
|
||||
- `manifest.json`
|
||||
- `transcripts/**`
|
||||
- `artifacts/**`
|
||||
- `previous/**`
|
||||
- `audio/**` only when `--include-audio` is set
|
||||
- excludes:
|
||||
- `runs/**`
|
||||
- `logs/**`
|
||||
- `reports/**`
|
||||
- `config/**`
|
||||
- `inputs/**`
|
||||
- remote `current/**` pointer files as local restore targets
|
||||
|
||||
## Skip and resume behavior
|
||||
- Restore does not participate in stage skip/resume decisions.
|
||||
- Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state.
|
||||
@@ -80,7 +95,7 @@ Does not own:
|
||||
- `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files.
|
||||
- Local path mapping is traversal-safe and constrained to session root.
|
||||
- Restore scope is deterministic and path-classified:
|
||||
- include `manifest.json`, `transcripts/**`, `artifacts/**`
|
||||
- include `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`
|
||||
- include `audio/**` only with `--include-audio`
|
||||
- exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`
|
||||
- Command remains standalone; no implicit `run --restore` behavior.
|
||||
|
||||
@@ -3,31 +3,36 @@
|
||||
## Purpose
|
||||
Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths.
|
||||
|
||||
## Inputs and Outputs
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- configured artifact definitions from `pipeline.scriptorium.artifacts`
|
||||
- selected artifact filter from runtime (`--artifacts`) when provided
|
||||
- resolved artifact input sources declared per artifact (`inputs.*.source`)
|
||||
- optional previous-session file inputs (`previous_session_artifact`)
|
||||
- configured artifact definitions from `pipeline.scriptorium.artifacts`;
|
||||
- selected artifact filter (`--artifacts`) when provided;
|
||||
- resolved artifact sources from resolver/catalog.
|
||||
|
||||
Source types used by analyze:
|
||||
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`;
|
||||
- configured artifacts: `narratio.artifact.<artifact_key>`;
|
||||
- canonical previous-session artifacts: `narratio.previous_session.artifact.<artifact_key>`;
|
||||
- legacy path-based previous-session source: `previous_session_artifact` (uses `inputs.*.path`).
|
||||
|
||||
Outputs:
|
||||
- one promoted output file per executed configured artifact at that artifact's configured `output_path`
|
||||
- stage metadata containing generated artifact entries and reused disabled-artifact entries
|
||||
- promoted configured artifact files at each configured `output_path`;
|
||||
- stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info).
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- runtime artifact catalog construction for analyze execution
|
||||
- selected-artifact planning and dependency ordering
|
||||
- per-artifact input resolution, var resolution, timeout/render-debug resolution
|
||||
- Scriptorium run/render invocation for each selected artifact
|
||||
- run-local output generation and canonical promotion
|
||||
- runtime artifact catalog construction;
|
||||
- selected-artifact planning and dependency ordering;
|
||||
- per-input resolution and required/optional handling;
|
||||
- Scriptorium render/run invocation;
|
||||
- run-local output generation and canonical promotion.
|
||||
|
||||
Does not own:
|
||||
- transcript generation/processing stages
|
||||
- archive promotion policy
|
||||
- per-artifact resume semantics
|
||||
- prepare-time previous-session hydration;
|
||||
- object-store access for previous-session sources;
|
||||
- archive promotion policy.
|
||||
|
||||
## Config Fields Used
|
||||
## Config fields used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
@@ -36,49 +41,41 @@ Does not own:
|
||||
- `pipeline.scriptorium.timeout`
|
||||
- `pipeline.scriptorium.render_debug`
|
||||
- `pipeline.scriptorium.artifacts.<name>.*`
|
||||
- `enabled`
|
||||
- `depends_on`
|
||||
- `prompt_id`
|
||||
- `profile_id`
|
||||
- `timeout`
|
||||
- `output_path`
|
||||
- `render_debug`
|
||||
- `inputs`
|
||||
- `vars`
|
||||
|
||||
## External Adapters Used
|
||||
## External adapters used
|
||||
- Scriptorium adapter:
|
||||
- optional `RenderArtifact` (render debug)
|
||||
- `RunArtifact` (artifact generation)
|
||||
- optional `RenderArtifact` when render-debug is enabled;
|
||||
- `RunArtifact` for artifact generation.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- If `pipeline.scriptorium` is absent, stage returns success metadata with `skipped=true`.
|
||||
- If no artifacts are configured, stage returns success metadata with `skipped=true`.
|
||||
- If zero artifacts are executable after `enabled` + `--artifacts` filtering, stage returns success metadata with `skipped=true`.
|
||||
- Builds runtime catalog with built-ins and configured artifacts.
|
||||
- Non-executable configured artifacts are marked available only when their configured output file exists and is valid on disk.
|
||||
- Executes selected configured artifacts in topological order with deterministic tie-breaking.
|
||||
- For each generated artifact, records metadata fields including `name`, `source_id`, `output_kind`, `path`, `prompt_id`, `profile_id`, and `provenance`.
|
||||
- Reused disabled artifacts are recorded separately in `reused_artifacts` with provenance `filesystem.disabled_artifact_output`.
|
||||
## State and manifest behavior
|
||||
- If Scriptorium config is absent, or no artifacts are executable after filtering, analyze returns success metadata with `skipped=true`.
|
||||
- Builds runtime catalog with built-ins and configured `narratio.artifact.<name>` entries.
|
||||
- Non-executable configured artifacts may still be marked available from existing canonical output files.
|
||||
- Resolves canonical previous-session sources from local prepared `previous/` cache:
|
||||
- prefers manifest-backed previous input paths when present;
|
||||
- may fall back to current-session `previous/` filesystem paths.
|
||||
- Analyze does not call object storage for canonical previous-session source resolution.
|
||||
- Required canonical previous-session input missing:
|
||||
- fails with guidance to run `narratio run-stage --force prepare`.
|
||||
- Optional missing sources are omitted from adapter input paths.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
## Skip and resume behavior
|
||||
- Runner-level skip applies when analyze is already `succeeded` and `--force` is not set.
|
||||
- Analyze remains stage-scoped for resume/skip; there is no per-artifact resume state.
|
||||
- `--artifacts` filters which configured artifacts are executable when analyze runs; it does not imply `--force`.
|
||||
- Analyze is stage-scoped for resume; no per-artifact manifest resume state.
|
||||
- `--artifacts` filters executable artifacts but does not imply force rerun.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on invalid dependency ordering, unavailable required configured inputs, invalid built-in input prerequisites, render/run adapter failures, validation-failed adapter results, or missing/empty outputs.
|
||||
- Required configured dependency missing from catalog availability fails clearly before invocation.
|
||||
- Optional missing inputs are omitted.
|
||||
## Failure behavior
|
||||
- Fails on dependency-order violations, missing required inputs, resolver validation failures, adapter errors, and missing/empty generated outputs.
|
||||
- Required unavailable configured artifact source (`narratio.artifact.<name>`) fails before invocation.
|
||||
- Required canonical previous-session source fails with prepare-rerun guidance.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
## Tests to inspect before changing
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/artifacts/catalog_test.go`
|
||||
- `internal/artifacts/artifact_resolver_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/app/restore_workflow_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Configured artifacts are identified by `narratio.artifact.<name>` source IDs.
|
||||
- Artifact-to-artifact references rely on explicit `depends_on` declarations validated in config.
|
||||
- Generated analyze outputs are treated uniformly as Scriptorium artifacts.
|
||||
- Successful outputs must exist and be non-empty before promotion.
|
||||
## Architectural invariants
|
||||
- Canonical previous-session behavior is local-cache only during analyze.
|
||||
- Generated outputs are validated and promoted before stage success is recorded.
|
||||
- Resolver/catalog decisions stay deterministic and validation-gated.
|
||||
|
||||
@@ -1,17 +1,19 @@
|
||||
# Stage: archive
|
||||
|
||||
## Purpose
|
||||
Publish run records and promoted session artifacts to object storage, then atomically advance the remote current pointer.
|
||||
Publish durable run/session state to object storage, then atomically advance remote current state.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- session manifest and prerequisite stage records
|
||||
- run root contents under `runs/{run_id}/`
|
||||
- promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`)
|
||||
- session-level `previous/**` cache files when present
|
||||
|
||||
Outputs:
|
||||
- uploaded run files under `{session_prefix}/runs/{run_id}/...`
|
||||
- uploaded promoted artifacts under `{session_prefix}/...`
|
||||
- uploaded session previous-cache files under `{session_prefix}/previous/...` when present
|
||||
- `{session_prefix}/current/manifest.json`
|
||||
- `{session_prefix}/current/run_id.txt` written last
|
||||
|
||||
@@ -21,6 +23,7 @@ Owns:
|
||||
- Prerequisite stage success enforcement
|
||||
- Run file collection and upload (excluding `audio/`)
|
||||
- Promotion rule resolution and upload
|
||||
- Session previous-cache file collection/upload
|
||||
- Commit pointer publish order
|
||||
|
||||
Does not own:
|
||||
@@ -43,8 +46,10 @@ Does not own:
|
||||
## State and Manifest Behavior
|
||||
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
|
||||
- Resolves bucket/prefix from manifest identity first, then config fallback.
|
||||
- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
|
||||
- Writes metadata including:
|
||||
- upload counts/paths
|
||||
- `previous_files_uploaded` and `previous_uploaded_paths`
|
||||
- `current_manifest_key`
|
||||
- `current_run_id_key`
|
||||
- `current_pointer_written`
|
||||
@@ -64,5 +69,6 @@ Does not own:
|
||||
|
||||
## Architectural Invariants
|
||||
- Run upload excludes `audio/` subtree.
|
||||
- Session `previous/**` is archiveable durable input/provenance state, not run-local output.
|
||||
- `current/manifest.json` uploads before `current/run_id.txt`.
|
||||
- `current/run_id.txt` is the remote publish commit marker.
|
||||
|
||||
@@ -1,40 +1,49 @@
|
||||
# Stage: prepare
|
||||
|
||||
## Purpose
|
||||
Materialize all required session inputs into canonical local workspace paths and record input provenance in the session manifest.
|
||||
Materialize canonical current-session input state and provenance before downstream stages run.
|
||||
|
||||
## Inputs and Outputs
|
||||
Prepare owns:
|
||||
- local input file materialization (`inputs/**`);
|
||||
- audio input materialization (`audio/**`);
|
||||
- previous-session cache hydration (`previous/**`) for canonical previous-session artifact sources.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- `session.yml` (resolved session config)
|
||||
- `pipeline.resolved.yml` (materialized from resolved pipeline config)
|
||||
- `speakers.yml`
|
||||
- `autocorrect.yml`
|
||||
- `glossary.yml`
|
||||
- resolved config/session (`pipeline.yml`, `session.yml`);
|
||||
- session-local input files (`speakers`, `autocorrect`, `glossary`);
|
||||
- audio source:
|
||||
- local (`session.inputs.audio_dir` or `session.inputs.audio_files`), or
|
||||
- S3 (`session.inputs.audio_s3.prefix`)
|
||||
- local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
|
||||
- S3: `session.inputs.audio_s3.prefix`;
|
||||
- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning);
|
||||
- remote previous-session current archive state when previous hydration is required.
|
||||
|
||||
Outputs:
|
||||
- `inputs/session.yml`
|
||||
- `inputs/pipeline.resolved.yml`
|
||||
- `inputs/speakers.yml`
|
||||
- `inputs/autocorrect.yml`
|
||||
- `inputs/glossary.yml`
|
||||
- `audio/*.flac` in session workdir
|
||||
- `manifest.Inputs` records with checksums and source metadata
|
||||
- `inputs/session.yml`;
|
||||
- `inputs/pipeline.resolved.yml`;
|
||||
- `inputs/speakers.yml`;
|
||||
- `inputs/autocorrect.yml`;
|
||||
- `inputs/glossary.yml`;
|
||||
- `audio/*.flac` in canonical session `audio/`;
|
||||
- optional `previous/manifest.json`;
|
||||
- optional `previous/artifacts/**`;
|
||||
- deterministic `manifest.Inputs` records with checksums and provenance metadata.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Input path resolution and validation
|
||||
- Local copy/materialization of configs and audio files
|
||||
- S3 audio download to run-scoped spool, then copy into work audio dir
|
||||
- input path resolution and materialization;
|
||||
- S3 audio list/download/copy flow;
|
||||
- previous-session artifact requirement collection from enabled configured artifacts;
|
||||
- previous cache lifecycle when requirements exist (clear and rehydrate managed `previous/` state).
|
||||
|
||||
Does not own:
|
||||
- Transcript generation/processing
|
||||
- Archive publish behavior
|
||||
- transcript or artifact generation;
|
||||
- analyze-stage source resolution;
|
||||
- archive commit behavior.
|
||||
|
||||
## Config Fields Used
|
||||
## Config fields used
|
||||
- `session.session_id`
|
||||
- `session.previous_session_id`
|
||||
- `session.campaign`
|
||||
- `session.inputs.speakers_file`
|
||||
- `session.inputs.autocorrect_file`
|
||||
@@ -46,29 +55,56 @@ Does not own:
|
||||
- `pipeline.spool.root`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
- `pipeline.scriptorium.artifacts.<name>.enabled`
|
||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
|
||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required`
|
||||
|
||||
## External Adapters Used
|
||||
- Object storage backend (`env.ObjectStore`) for S3 audio list/download when `audio_s3` is configured.
|
||||
## External adapters used
|
||||
- `storage.ObjectStore` for:
|
||||
- S3 audio listing/downloads;
|
||||
- previous-session current pointer/manifest/artifact object checks and downloads.
|
||||
|
||||
## State and Manifest Behavior
|
||||
## State and manifest behavior
|
||||
- Ensures workspace layout exists.
|
||||
- Writes resolved config and input files to canonical `inputs/` paths.
|
||||
- Records all prepared inputs into `manifest.Inputs` (sorted deterministically by kind/path).
|
||||
- For S3 audio, records `S3Bucket`, `S3Key`, `S3Size`, `S3ETag`, and `SpoolPath` in each audio input record.
|
||||
- Materializes canonical input files and audio files.
|
||||
- Scans enabled configured artifact inputs for canonical sources:
|
||||
- `narratio.previous_session.artifact.<artifact_key>`
|
||||
- If one or more canonical previous-session requirements exist:
|
||||
- clears managed `previous/` state;
|
||||
- hydrates required/optional previous artifacts from the configured previous session’s committed archive current state;
|
||||
- writes `previous/manifest.json` and hydrated `previous/artifacts/**`;
|
||||
- records hydrated previous inputs in `manifest.Inputs` with source `previous_session_archive.current`.
|
||||
- If no canonical previous-session requirements exist, prepare does not manage `previous/`.
|
||||
- `manifest.Inputs` is sorted deterministically by `(kind, path)`.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when stage already `succeeded` and `--force` is not set.
|
||||
- Stage itself is deterministic/idempotent for unchanged inputs (`copyFileIfChanged`, `writeBytesIfChanged`).
|
||||
## Required and optional previous-session behavior
|
||||
- `previous_session_id` unset:
|
||||
- if any referenced previous artifact is required: fail;
|
||||
- if all referenced previous artifacts are optional: continue and omit them.
|
||||
- Previous session archive current pointer or manifest missing:
|
||||
- if any referenced previous artifact is required: fail;
|
||||
- if all referenced previous artifacts are optional: continue and omit missing ones.
|
||||
- Missing required previous artifact object: fail.
|
||||
- Missing optional previous artifact object: omit.
|
||||
- Downloaded previous artifacts must validate as non-empty files.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing required files, invalid audio source combinations, no discoverable `.flac` files, duplicate audio basenames, missing object store for S3 mode, or S3 list/download failures.
|
||||
## Skip and resume behavior
|
||||
- Runner-level skip remains authoritative:
|
||||
- if `prepare` already succeeded and run is not forced, `prepare` does not run and no hydration/download occurs.
|
||||
- If `prepare` runs (including with `--force`), it owns managed `previous/` state for canonical previous-session inputs.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
## Failure behavior
|
||||
- Fails on missing required input files, invalid audio-source combinations, empty/duplicate audio inputs, missing object store for S3 modes, and remote access/download/validation errors.
|
||||
- For required canonical previous-session inputs, analyze-time missing-input guidance is to rerun:
|
||||
- `narratio run-stage --force prepare`
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/stage/prepare_test.go`
|
||||
- `internal/app/session_cli_test.go`
|
||||
- `internal/config/load_validate_test.go`
|
||||
- `internal/stage/prepare_previous_test.go`
|
||||
- `internal/artifacts/previous_requirements_test.go`
|
||||
- `internal/app/runner_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
## Architectural invariants
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
|
||||
- Audio files must be `.flac`.
|
||||
- Canonical `inputs/*` and `audio/*` paths are the durable source for downstream stages.
|
||||
- Storage keys are computed by callers using archive/path helpers; storage adapter receives explicit keys.
|
||||
- `prepare` is the only stage that hydrates canonical previous-session cache state.
|
||||
|
||||
@@ -48,7 +48,7 @@ Restore source-of-truth:
|
||||
- remote current manifest: `current/manifest.json`
|
||||
|
||||
Restore default scope:
|
||||
- includes `manifest.json`, `transcripts/**`, `artifacts/**`
|
||||
- includes `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`
|
||||
- includes `audio/**` only with `--include-audio`
|
||||
- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source)
|
||||
|
||||
@@ -67,6 +67,7 @@ Canonical session directories:
|
||||
- `audio/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `previous/`
|
||||
- `reports/`
|
||||
- `logs/`
|
||||
- `config/`
|
||||
@@ -99,6 +100,12 @@ Configured artifact source reuse:
|
||||
- accepted on `run`, `resume`, and `run-stage analyze`.
|
||||
- filters analyze execution only; does not force stage rerun.
|
||||
|
||||
Canonical previous-session input behavior:
|
||||
- canonical sources use `narratio.previous_session.artifact.<artifact_key>`.
|
||||
- these inputs are hydrated by `prepare`, not `analyze`.
|
||||
- if analyze fails due to missing canonical previous cache, rerun:
|
||||
- `narratio run-stage --session-id <id> --force prepare`
|
||||
|
||||
## Remote archive layout and publish contract
|
||||
|
||||
When archive is enabled and run upload is enabled, archive publishes under:
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Status
|
||||
|
||||
Planned.
|
||||
Completed.
|
||||
|
||||
This roadmap describes the implementation strategy for first-class previous-session artifact support in Narratio. It belongs under `docs/roadmap/previous.md` until the feature is implemented. After implementation, current behavior should be documented in the appropriate user-facing and internal documentation files, and this roadmap should be removed or marked complete according to the documentation policy.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user