eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Rewrite internal documentation for current stage and state contracts

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-23 12:57:59 +00:00
parent d723384888
commit 0299b128cf
15 changed files with 450 additions and 993 deletions
Download Patch File Download Diff File Expand all files Collapse all files

146
docs/internal/stage-prepare.md
View File

@@ -1,124 +1,42 @@
# Stage: prepare
## Purpose
Materialize canonical current-session input state and provenance before downstream stages run.
Materialize canonical current-session inputs before processing stages.
Prepare owns:
- local input file materialization (`inputs/**`);
- audio input materialization (`audio/**`);
- previous-session cache hydration (`previous/**`) for canonical previous-session artifact sources.
## Inputs and outputs
Inputs:
- resolved config/campaign/session (`pipeline.yml`, `campaign.yml`, `session.yml`);
- remote session provenance when `session.yml` was loaded from S3;
- campaign or session input files (`speakers`, `autocorrect`, `glossary`);
## Inputs
- resolved `campaign.yml`, `session.yml`, and pipeline config
- stable input files (`speakers`, `autocorrect`, `glossary`)
- audio source:
- local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
- S3: `session.inputs.audio_s3.prefix`;
- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning);
- remote previous-session current publish state when previous hydration is required.
- local `audio_dir`/`audio_files`, or
- S3 `audio_s3.prefix`
- enabled configured artifact input requirements for previous-session sources
Outputs:
- `inputs/campaign.yml`;
- `inputs/session.yml`;
- `inputs/pipeline.resolved.yml`;
- `inputs/speakers.yml`;
- `inputs/autocorrect.yml`;
- `inputs/glossary.yml`;
- `audio/*.flac` in canonical session `audio/`;
- optional `previous/manifest.json`;
- optional `previous/artifacts/**`;
- deterministic `manifest.Inputs` records with checksums and provenance metadata.
## Outputs
- `inputs/campaign.yml`
- `inputs/session.yml`
- `inputs/pipeline.resolved.yml`
- `inputs/speakers.yml`
- `inputs/autocorrect.yml`
- `inputs/glossary.yml`
- `audio/*.flac`
- optional `previous/manifest.json`
- optional `previous/artifacts/**`
- deterministic `manifest.inputs` entries (checksums + provenance)
## Boundaries
Owns:
- input path resolution and materialization;
- S3 audio list/download/copy flow;
- previous-session artifact requirement collection from enabled configured artifacts;
- previous cache lifecycle when requirements exist (clear and rehydrate managed `previous/` state).
Does not own:
- transcript or artifact generation;
- analyze-stage source resolution;
- publish commit behavior.
## Config fields used
- `session.session_id`
- `session.previous_session_id`
- `session.campaign`
- `session.inputs.speakers_file`
- `session.inputs.autocorrect_file`
- `session.inputs.glossary_file`
- `session.inputs.audio_dir`
- `session.inputs.audio_files`
- `session.inputs.audio_s3.prefix`
- `pipeline.workspace.root`
- `pipeline.spool.root`
- `pipeline.cache.root`
- `pipeline.cache.s3_audio`
- `pipeline.storage.s3.bucket`
- `pipeline.storage.s3.root_prefix`
- `pipeline.scriptorium.artifacts.<name>.enabled`
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required`
- `campaign.campaign_id`
- `campaign.inputs.speakers_file`
- `campaign.inputs.autocorrect_file`
- `campaign.inputs.glossary_file`
## External adapters used
- `storage.ObjectStore` for:
- S3 audio listing/downloads;
- previous-session current pointer/manifest/artifact object checks and downloads.
## State and manifest behavior
- Ensures workspace layout exists.
- Materializes canonical input files and audio files.
- For S3 audio, uses run-scoped spool for active downloads and durable cache for reusable audio files; cache hits copy directly to work audio without downloading the object again.
- Records `inputs/session.yml` provenance as local `session_config` or remote `session_config.s3`.
- Resolves campaign-provided stable input paths relative to `campaign.yml`.
- Resolves session-provided stable input overrides relative to `session.yml`.
- Scans enabled configured artifact inputs for canonical sources:
- `narratio.previous_session.artifact.<artifact_key>`
- If one or more canonical previous-session requirements exist:
## Key Behavior
- validates required config/store state.
- enforces local audio vs S3 audio mutual exclusivity.
- materializes S3 audio through spool/cache-aware logic.
- scans enabled configured artifact inputs for `narratio.previous_session.artifact.*` requirements.
- when previous requirements exist:
- clears managed `previous/` state;
- hydrates required/optional previous artifacts from the configured previous session’s committed publish current state;
- writes `previous/manifest.json` and hydrated `previous/artifacts/**`;
- stores publish-relative artifact paths such as `artifacts/session_recap.md` as `previous/artifacts/session_recap.md`, not `previous/artifacts/artifacts/session_recap.md`;
- records hydrated previous inputs in `manifest.Inputs` with source `previous_session_publish.current`.
- If no canonical previous-session requirements exist, prepare does not manage `previous/`.
- `manifest.Inputs` is sorted deterministically by `(kind, path)`.
- S3 audio `manifest.Inputs` retain S3 provenance and include `cache_path`; `spool_path` is present only when the current prepare invocation downloaded the file.
- builds previous-cache remote plan;
- downloads previous manifest/artifacts;
- records previous inputs in `manifest.inputs`.
## Required and optional previous-session behavior
- `previous_session_id` unset:
- if any referenced previous artifact is required: fail;
- if all referenced previous artifacts are optional: continue and omit them.
- Previous session publish current pointer or manifest missing:
- if any referenced previous artifact is required: fail;
- if all referenced previous artifacts are optional: continue and omit missing ones.
- Missing required previous artifact object: fail.
- Missing optional previous artifact object: omit.
- Downloaded previous artifacts must validate as non-empty files.
Required previous-session inputs fail when unavailable; optional missing inputs are skipped.
## Skip and resume behavior
- Runner-level skip remains authoritative:
- if `prepare` already succeeded and run is not forced, `prepare` does not run and no hydration/download occurs.
- If `prepare` runs (including with `--force`), it owns managed `previous/` state for canonical previous-session inputs.
## Failure behavior
- Fails on missing required input files, invalid audio-source combinations, empty/duplicate audio inputs, missing object store for S3 modes, and remote access/download/validation errors.
- For required canonical previous-session inputs, analyze-time missing-input guidance is to rerun:
- `narratio run-stage --force prepare`
## Tests to inspect before changing
- `internal/stage/prepare_test.go`
- `internal/stage/prepare_previous_test.go`
- `internal/artifacts/previous_requirements_test.go`
- `internal/app/runner_test.go`
## Architectural invariants
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
- Storage keys are computed by callers using path helpers; storage adapter receives explicit keys.
- `prepare` is the only stage that hydrates canonical previous-session cache state.
## Invariants
- only `prepare` hydrates canonical `previous/` cache state.
- managed previous artifacts are stored under `previous/artifacts/**` without duplicate `artifacts/artifacts/` nesting.
- `manifest.inputs` ordering is deterministic (`kind`, `path`).