Add remote session loading
This commit is contained in:
@@ -6,7 +6,7 @@
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
This command uses default discovery for `pipeline.yml`, `campaign.yml`, and `session.yml`; all three files must be discoverable unless you pass explicit `--config`, `--campaign`, and `--session` paths.
|
||||
This command uses default discovery for `pipeline.yml`, `campaign.yml`, and local `session.yml`. If local session discovery misses and S3 storage is configured, `--session-id` can load remote `session.yml` from the canonical session prefix.
|
||||
|
||||
## Command Overview
|
||||
|
||||
@@ -110,6 +110,7 @@ Success output:
|
||||
|
||||
Common failure cases:
|
||||
- missing default config/campaign/session paths when flags omitted.
|
||||
- missing local session plus missing/unavailable remote `session.yml`.
|
||||
- invalid template/rendered session mismatch.
|
||||
- unknown/invalid `--artifacts` value.
|
||||
- `--artifacts` with unknown configured artifact key.
|
||||
@@ -132,6 +133,7 @@ Success output includes:
|
||||
|
||||
Common failure cases:
|
||||
- same config/campaign/session discovery and validation failures as `run`.
|
||||
- remote session fallback failures when local session discovery misses.
|
||||
- secrets directory read failures when `pipeline.secrets.env_dir` is configured.
|
||||
|
||||
### `resume`
|
||||
|
||||
@@ -6,7 +6,7 @@ Narratio loads three YAML files:
|
||||
|
||||
- `pipeline.yml`: pipeline-level runtime configuration.
|
||||
- `campaign.yml`: stable campaign identity and campaign-level input defaults.
|
||||
- `session.yml`: per-session metadata and input selection.
|
||||
- `session.yml`: per-session metadata and input selection, loaded locally or from the configured S3 backend.
|
||||
|
||||
These commands load and validate all three files before running:
|
||||
|
||||
@@ -20,6 +20,7 @@ Behavior:
|
||||
|
||||
- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail.
|
||||
- session templates render before session YAML decode.
|
||||
- remote `session.yml` uses the same strict decode and template behavior as local `session.yml`.
|
||||
- defaults are applied for optional pipeline fields.
|
||||
- campaign-level stable input paths fill missing session input paths.
|
||||
- session-level stable input paths override campaign-level input paths.
|
||||
@@ -49,11 +50,14 @@ Campaign config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`:
|
||||
Session config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`:
|
||||
|
||||
- if `--session <path>` is provided, that path is used.
|
||||
- if omitted, Narratio searches in order:
|
||||
- if `--session` is omitted, Narratio searches locally in order:
|
||||
1. `./session.yml`
|
||||
2. `/usr/local/etc/narratio/session.yml`
|
||||
3. `/etc/narratio/session.yml`
|
||||
- first existing file wins.
|
||||
- first existing local file wins.
|
||||
- if no local session file is found, `--session-id <value>` is present, storage is configured, and campaign identity is resolved, Narratio loads remote `session.yml` from:
|
||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
|
||||
- local discovery always runs before remote fallback.
|
||||
|
||||
Template behavior:
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@ Outputs:
|
||||
- resolved artifact path + provenance (`ResolvedSessionArtifact`);
|
||||
- runtime catalog entries for built-ins and configured artifacts;
|
||||
- requirement sets for canonical previous-session inputs.
|
||||
- canonical S3 session, run, current, session config, audio, and promoted artifact keys.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
@@ -43,6 +44,14 @@ Does not own:
|
||||
- configured artifact: `narratio.artifact.<artifact_key>`
|
||||
- canonical previous-session artifact: `narratio.previous_session.artifact.<artifact_key>`
|
||||
|
||||
## S3 key helpers
|
||||
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- session config: `{session_prefix}/session.yml`
|
||||
- run prefix: `{session_prefix}/runs/{run_id}/`
|
||||
- audio prefix: `{session_prefix}/{session.inputs.audio_s3.prefix}`
|
||||
- current manifest: `{session_prefix}/current/manifest.json`
|
||||
- current run pointer: `{session_prefix}/current/run_id.txt`
|
||||
|
||||
## Runtime catalog model
|
||||
Catalog entries track:
|
||||
- `planned`: source is registered for this run;
|
||||
|
||||
@@ -11,6 +11,7 @@ Prepare owns:
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- resolved config/campaign/session (`pipeline.yml`, `campaign.yml`, `session.yml`);
|
||||
- remote session provenance when `session.yml` was loaded from S3;
|
||||
- campaign or session input files (`speakers`, `autocorrect`, `glossary`);
|
||||
- audio source:
|
||||
- local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
|
||||
@@ -72,6 +73,7 @@ Does not own:
|
||||
## State and manifest behavior
|
||||
- Ensures workspace layout exists.
|
||||
- Materializes canonical input files and audio files.
|
||||
- Records `inputs/session.yml` provenance as local `session_config` or remote `session_config.s3`.
|
||||
- Resolves campaign-provided stable input paths relative to `campaign.yml`.
|
||||
- Resolves session-provided stable input overrides relative to `session.yml`.
|
||||
- Scans enabled configured artifact inputs for canonical sources:
|
||||
|
||||
@@ -6,7 +6,7 @@ Document Narratio's remote storage backend contracts and implementations under `
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- Resolved storage config (`pipeline.storage.*`).
|
||||
- Bucket-relative object keys and local file paths from stage/app orchestration.
|
||||
- Bucket-relative object keys and local file paths from app/stage orchestration.
|
||||
|
||||
Outputs:
|
||||
- Listed/downloaded/uploaded object metadata (`ObjectInfo`).
|
||||
@@ -57,6 +57,7 @@ Implementations:
|
||||
- `S3Backend` constructor fails when required bucket is missing or AWS client setup fails.
|
||||
- CRUD operations return contextual errors (including not-found behavior via `Exists`).
|
||||
- Key normalization is applied before operations (`\\` to `/`, leading slash trimmed).
|
||||
- Remote session loading uses `List` to find the exact `session.yml` key and `Download` to materialize it to a local temp file.
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/adapters/storage/factory_test.go`
|
||||
|
||||
@@ -6,19 +6,21 @@ For field-level configuration, see [docs/config.md](./config.md). For full comma
|
||||
|
||||
## Normal workflow (S3-first path)
|
||||
|
||||
1. Upload session `.flac` files to object storage under the configured session audio prefix.
|
||||
2. Run Narratio:
|
||||
1. Upload `session.yml` to the configured session prefix, or keep a local `session.yml` available.
|
||||
2. Upload session `.flac` files to object storage under the configured session audio prefix.
|
||||
3. Run Narratio:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
3. Read success output:
|
||||
4. Read success output:
|
||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
- use `manifest=<path>` with `status` for inspection.
|
||||
|
||||
Notes:
|
||||
- default config/campaign/session discovery applies unless `--config`, `--campaign`, and `--session` are passed.
|
||||
- when local `session.yml` discovery misses, `--session-id` loads remote `session.yml` from `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
|
||||
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
|
||||
|
||||
## Restore workflow
|
||||
|
||||
@@ -2,13 +2,13 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap describes planned, unimplemented work for three related feature areas:
|
||||
This roadmap tracks implementation work for three related feature areas:
|
||||
|
||||
1. `campaign.yml` configuration for stable campaign-level inputs.
|
||||
2. Remote `session.yml` loading from the existing S3 object-store backend.
|
||||
3. Logical archive locks that prevent selected top-level transcript/artifact promotions from overwriting curated archive state while still preserving run-local outputs.
|
||||
|
||||
Treat this document as an implementation plan, not as current behavior. Keep planned behavior under `docs/roadmap/` until each phase is implemented and canonical docs are updated.
|
||||
Treat future sections of this document as an implementation plan, not as current behavior. Keep planned behavior under `docs/roadmap/` until each item is implemented and canonical docs are updated.
|
||||
|
||||
## Current Code Facts
|
||||
|
||||
@@ -43,7 +43,7 @@ Implementation must preserve these constraints:
|
||||
- Keep raw secrets out of configs, manifests, logs, generated configs, archive metadata, and roadmap examples.
|
||||
- Update canonical user-facing docs only after behavior is implemented.
|
||||
|
||||
## Phase 1: Add `campaign.yml`
|
||||
## Phase 1: Add `campaign.yml` (implemented)
|
||||
|
||||
Add campaign-level configuration for stable campaign identity and stable input files. Do not add remote campaign loading in this phase.
|
||||
|
||||
@@ -116,7 +116,7 @@ Update `prepare` to materialize the resolved campaign/session inputs into canoni
|
||||
|
||||
Continue recording deterministic `manifest.Inputs` records with checksums. If a prepared input came from `campaign.yml`, record source/provenance using the existing manifest input fields where practical; add narrow metadata only if the existing fields cannot describe it.
|
||||
|
||||
## Phase 2: Load Remote `session.yml`
|
||||
## Phase 2: Load Remote `session.yml` (implemented)
|
||||
|
||||
Support running with no local session file when a remote session file exists under the canonical session prefix.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user