Add remote session loading

This commit is contained in:
2026-05-20 20:55:13 -05:00
parent b29d8eeb50
commit 3aae4bbb12
23 changed files with 587 additions and 101 deletions

View File

@@ -6,7 +6,7 @@
narratio run --session-id 2026-04-04
```
This command uses default discovery for `pipeline.yml`, `campaign.yml`, and `session.yml`; all three files must be discoverable unless you pass explicit `--config`, `--campaign`, and `--session` paths.
This command uses default discovery for `pipeline.yml`, `campaign.yml`, and local `session.yml`. If local session discovery misses and S3 storage is configured, `--session-id` can load remote `session.yml` from the canonical session prefix.
## Command Overview
@@ -110,6 +110,7 @@ Success output:
Common failure cases:
- missing default config/campaign/session paths when flags omitted.
- missing local session plus missing/unavailable remote `session.yml`.
- invalid template/rendered session mismatch.
- unknown/invalid `--artifacts` value.
- `--artifacts` with unknown configured artifact key.
@@ -132,6 +133,7 @@ Success output includes:
Common failure cases:
- same config/campaign/session discovery and validation failures as `run`.
- remote session fallback failures when local session discovery misses.
- secrets directory read failures when `pipeline.secrets.env_dir` is configured.
### `resume`

View File

@@ -6,7 +6,7 @@ Narratio loads three YAML files:
- `pipeline.yml`: pipeline-level runtime configuration.
- `campaign.yml`: stable campaign identity and campaign-level input defaults.
- `session.yml`: per-session metadata and input selection.
- `session.yml`: per-session metadata and input selection, loaded locally or from the configured S3 backend.
These commands load and validate all three files before running:
@@ -20,6 +20,7 @@ Behavior:
- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail.
- session templates render before session YAML decode.
- remote `session.yml` uses the same strict decode and template behavior as local `session.yml`.
- defaults are applied for optional pipeline fields.
- campaign-level stable input paths fill missing session input paths.
- session-level stable input paths override campaign-level input paths.
@@ -49,11 +50,14 @@ Campaign config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`:
Session config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`:
- if `--session <path>` is provided, that path is used.
- if omitted, Narratio searches in order:
- if `--session` is omitted, Narratio searches locally in order:
1. `./session.yml`
2. `/usr/local/etc/narratio/session.yml`
3. `/etc/narratio/session.yml`
- first existing file wins.
- first existing local file wins.
- if no local session file is found, `--session-id <value>` is present, storage is configured, and campaign identity is resolved, Narratio loads remote `session.yml` from:
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
- local discovery always runs before remote fallback.
Template behavior:

View File

@@ -16,6 +16,7 @@ Outputs:
- resolved artifact path + provenance (`ResolvedSessionArtifact`);
- runtime catalog entries for built-ins and configured artifacts;
- requirement sets for canonical previous-session inputs.
- canonical S3 session, run, current, session config, audio, and promoted artifact keys.
## Boundaries
Owns:
@@ -43,6 +44,14 @@ Does not own:
- configured artifact: `narratio.artifact.<artifact_key>`
- canonical previous-session artifact: `narratio.previous_session.artifact.<artifact_key>`
## S3 key helpers
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
- session config: `{session_prefix}/session.yml`
- run prefix: `{session_prefix}/runs/{run_id}/`
- audio prefix: `{session_prefix}/{session.inputs.audio_s3.prefix}`
- current manifest: `{session_prefix}/current/manifest.json`
- current run pointer: `{session_prefix}/current/run_id.txt`
## Runtime catalog model
Catalog entries track:
- `planned`: source is registered for this run;

View File

@@ -11,6 +11,7 @@ Prepare owns:
## Inputs and outputs
Inputs:
- resolved config/campaign/session (`pipeline.yml`, `campaign.yml`, `session.yml`);
- remote session provenance when `session.yml` was loaded from S3;
- campaign or session input files (`speakers`, `autocorrect`, `glossary`);
- audio source:
- local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
@@ -72,6 +73,7 @@ Does not own:
## State and manifest behavior
- Ensures workspace layout exists.
- Materializes canonical input files and audio files.
- Records `inputs/session.yml` provenance as local `session_config` or remote `session_config.s3`.
- Resolves campaign-provided stable input paths relative to `campaign.yml`.
- Resolves session-provided stable input overrides relative to `session.yml`.
- Scans enabled configured artifact inputs for canonical sources:

View File

@@ -6,7 +6,7 @@ Document Narratio's remote storage backend contracts and implementations under `
## Inputs and outputs
Inputs:
- Resolved storage config (`pipeline.storage.*`).
- Bucket-relative object keys and local file paths from stage/app orchestration.
- Bucket-relative object keys and local file paths from app/stage orchestration.
Outputs:
- Listed/downloaded/uploaded object metadata (`ObjectInfo`).
@@ -57,6 +57,7 @@ Implementations:
- `S3Backend` constructor fails when required bucket is missing or AWS client setup fails.
- CRUD operations return contextual errors (including not-found behavior via `Exists`).
- Key normalization is applied before operations (`\\` to `/`, leading slash trimmed).
- Remote session loading uses `List` to find the exact `session.yml` key and `Download` to materialize it to a local temp file.
## Tests to inspect before changing
- `internal/adapters/storage/factory_test.go`

View File

@@ -6,19 +6,21 @@ For field-level configuration, see [docs/config.md](./config.md). For full comma
## Normal workflow (S3-first path)
1. Upload session `.flac` files to object storage under the configured session audio prefix.
2. Run Narratio:
1. Upload `session.yml` to the configured session prefix, or keep a local `session.yml` available.
2. Upload session `.flac` files to object storage under the configured session audio prefix.
3. Run Narratio:
```bash
narratio run --session-id 2026-04-04
```
3. Read success output:
4. Read success output:
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
- use `manifest=<path>` with `status` for inspection.
Notes:
- default config/campaign/session discovery applies unless `--config`, `--campaign`, and `--session` are passed.
- when local `session.yml` discovery misses, `--session-id` loads remote `session.yml` from `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
## Restore workflow

View File

@@ -2,13 +2,13 @@
## Purpose
This roadmap describes planned, unimplemented work for three related feature areas:
This roadmap tracks implementation work for three related feature areas:
1. `campaign.yml` configuration for stable campaign-level inputs.
2. Remote `session.yml` loading from the existing S3 object-store backend.
3. Logical archive locks that prevent selected top-level transcript/artifact promotions from overwriting curated archive state while still preserving run-local outputs.
Treat this document as an implementation plan, not as current behavior. Keep planned behavior under `docs/roadmap/` until each phase is implemented and canonical docs are updated.
Treat future sections of this document as an implementation plan, not as current behavior. Keep planned behavior under `docs/roadmap/` until each item is implemented and canonical docs are updated.
## Current Code Facts
@@ -43,7 +43,7 @@ Implementation must preserve these constraints:
- Keep raw secrets out of configs, manifests, logs, generated configs, archive metadata, and roadmap examples.
- Update canonical user-facing docs only after behavior is implemented.
## Phase 1: Add `campaign.yml`
## Phase 1: Add `campaign.yml` (implemented)
Add campaign-level configuration for stable campaign identity and stable input files. Do not add remote campaign loading in this phase.
@@ -116,7 +116,7 @@ Update `prepare` to materialize the resolved campaign/session inputs into canoni
Continue recording deterministic `manifest.Inputs` records with checksums. If a prepared input came from `campaign.yml`, record source/provenance using the existing manifest input fields where practical; add narrow metadata only if the existing fields cannot describe it.
## Phase 2: Load Remote `session.yml`
## Phase 2: Load Remote `session.yml` (implemented)
Support running with no local session file when a remote session file exists under the canonical session prefix.