Finalize previous-session artifact documentation and restore-analyze continuity coverage
This commit is contained in:
@@ -1,40 +1,49 @@
|
||||
# Stage: prepare
|
||||
|
||||
## Purpose
|
||||
Materialize all required session inputs into canonical local workspace paths and record input provenance in the session manifest.
|
||||
Materialize canonical current-session input state and provenance before downstream stages run.
|
||||
|
||||
## Inputs and Outputs
|
||||
Prepare owns:
|
||||
- local input file materialization (`inputs/**`);
|
||||
- audio input materialization (`audio/**`);
|
||||
- previous-session cache hydration (`previous/**`) for canonical previous-session artifact sources.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- `session.yml` (resolved session config)
|
||||
- `pipeline.resolved.yml` (materialized from resolved pipeline config)
|
||||
- `speakers.yml`
|
||||
- `autocorrect.yml`
|
||||
- `glossary.yml`
|
||||
- resolved config/session (`pipeline.yml`, `session.yml`);
|
||||
- session-local input files (`speakers`, `autocorrect`, `glossary`);
|
||||
- audio source:
|
||||
- local (`session.inputs.audio_dir` or `session.inputs.audio_files`), or
|
||||
- S3 (`session.inputs.audio_s3.prefix`)
|
||||
- local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
|
||||
- S3: `session.inputs.audio_s3.prefix`;
|
||||
- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning);
|
||||
- remote previous-session current archive state when previous hydration is required.
|
||||
|
||||
Outputs:
|
||||
- `inputs/session.yml`
|
||||
- `inputs/pipeline.resolved.yml`
|
||||
- `inputs/speakers.yml`
|
||||
- `inputs/autocorrect.yml`
|
||||
- `inputs/glossary.yml`
|
||||
- `audio/*.flac` in session workdir
|
||||
- `manifest.Inputs` records with checksums and source metadata
|
||||
- `inputs/session.yml`;
|
||||
- `inputs/pipeline.resolved.yml`;
|
||||
- `inputs/speakers.yml`;
|
||||
- `inputs/autocorrect.yml`;
|
||||
- `inputs/glossary.yml`;
|
||||
- `audio/*.flac` in canonical session `audio/`;
|
||||
- optional `previous/manifest.json`;
|
||||
- optional `previous/artifacts/**`;
|
||||
- deterministic `manifest.Inputs` records with checksums and provenance metadata.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Input path resolution and validation
|
||||
- Local copy/materialization of configs and audio files
|
||||
- S3 audio download to run-scoped spool, then copy into work audio dir
|
||||
- input path resolution and materialization;
|
||||
- S3 audio list/download/copy flow;
|
||||
- previous-session artifact requirement collection from enabled configured artifacts;
|
||||
- previous cache lifecycle when requirements exist (clear and rehydrate managed `previous/` state).
|
||||
|
||||
Does not own:
|
||||
- Transcript generation/processing
|
||||
- Archive publish behavior
|
||||
- transcript or artifact generation;
|
||||
- analyze-stage source resolution;
|
||||
- archive commit behavior.
|
||||
|
||||
## Config Fields Used
|
||||
## Config fields used
|
||||
- `session.session_id`
|
||||
- `session.previous_session_id`
|
||||
- `session.campaign`
|
||||
- `session.inputs.speakers_file`
|
||||
- `session.inputs.autocorrect_file`
|
||||
@@ -46,29 +55,56 @@ Does not own:
|
||||
- `pipeline.spool.root`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
- `pipeline.scriptorium.artifacts.<name>.enabled`
|
||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
|
||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required`
|
||||
|
||||
## External Adapters Used
|
||||
- Object storage backend (`env.ObjectStore`) for S3 audio list/download when `audio_s3` is configured.
|
||||
## External adapters used
|
||||
- `storage.ObjectStore` for:
|
||||
- S3 audio listing/downloads;
|
||||
- previous-session current pointer/manifest/artifact object checks and downloads.
|
||||
|
||||
## State and Manifest Behavior
|
||||
## State and manifest behavior
|
||||
- Ensures workspace layout exists.
|
||||
- Writes resolved config and input files to canonical `inputs/` paths.
|
||||
- Records all prepared inputs into `manifest.Inputs` (sorted deterministically by kind/path).
|
||||
- For S3 audio, records `S3Bucket`, `S3Key`, `S3Size`, `S3ETag`, and `SpoolPath` in each audio input record.
|
||||
- Materializes canonical input files and audio files.
|
||||
- Scans enabled configured artifact inputs for canonical sources:
|
||||
- `narratio.previous_session.artifact.<artifact_key>`
|
||||
- If one or more canonical previous-session requirements exist:
|
||||
- clears managed `previous/` state;
|
||||
- hydrates required/optional previous artifacts from the configured previous session’s committed archive current state;
|
||||
- writes `previous/manifest.json` and hydrated `previous/artifacts/**`;
|
||||
- records hydrated previous inputs in `manifest.Inputs` with source `previous_session_archive.current`.
|
||||
- If no canonical previous-session requirements exist, prepare does not manage `previous/`.
|
||||
- `manifest.Inputs` is sorted deterministically by `(kind, path)`.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when stage already `succeeded` and `--force` is not set.
|
||||
- Stage itself is deterministic/idempotent for unchanged inputs (`copyFileIfChanged`, `writeBytesIfChanged`).
|
||||
## Required and optional previous-session behavior
|
||||
- `previous_session_id` unset:
|
||||
- if any referenced previous artifact is required: fail;
|
||||
- if all referenced previous artifacts are optional: continue and omit them.
|
||||
- Previous session archive current pointer or manifest missing:
|
||||
- if any referenced previous artifact is required: fail;
|
||||
- if all referenced previous artifacts are optional: continue and omit missing ones.
|
||||
- Missing required previous artifact object: fail.
|
||||
- Missing optional previous artifact object: omit.
|
||||
- Downloaded previous artifacts must validate as non-empty files.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing required files, invalid audio source combinations, no discoverable `.flac` files, duplicate audio basenames, missing object store for S3 mode, or S3 list/download failures.
|
||||
## Skip and resume behavior
|
||||
- Runner-level skip remains authoritative:
|
||||
- if `prepare` already succeeded and run is not forced, `prepare` does not run and no hydration/download occurs.
|
||||
- If `prepare` runs (including with `--force`), it owns managed `previous/` state for canonical previous-session inputs.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
## Failure behavior
|
||||
- Fails on missing required input files, invalid audio-source combinations, empty/duplicate audio inputs, missing object store for S3 modes, and remote access/download/validation errors.
|
||||
- For required canonical previous-session inputs, analyze-time missing-input guidance is to rerun:
|
||||
- `narratio run-stage --force prepare`
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/stage/prepare_test.go`
|
||||
- `internal/app/session_cli_test.go`
|
||||
- `internal/config/load_validate_test.go`
|
||||
- `internal/stage/prepare_previous_test.go`
|
||||
- `internal/artifacts/previous_requirements_test.go`
|
||||
- `internal/app/runner_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
## Architectural invariants
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
|
||||
- Audio files must be `.flac`.
|
||||
- Canonical `inputs/*` and `audio/*` paths are the durable source for downstream stages.
|
||||
- Storage keys are computed by callers using archive/path helpers; storage adapter receives explicit keys.
|
||||
- `prepare` is the only stage that hydrates canonical previous-session cache state.
|
||||
|
||||
Reference in New Issue
Block a user