From 2dd38c791301d336d72930c86e420a64a811c46f Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 20 May 2026 20:15:05 -0500 Subject: [PATCH] Refine campaign and remote session roadmap --- docs/roadmap/remote.md | 318 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 318 insertions(+) create mode 100644 docs/roadmap/remote.md diff --git a/docs/roadmap/remote.md b/docs/roadmap/remote.md new file mode 100644 index 0000000..9d287ab --- /dev/null +++ b/docs/roadmap/remote.md @@ -0,0 +1,318 @@ +# Roadmap: Campaign State, Remote Sessions, and Archive Locks + +## Purpose + +This roadmap describes planned, unimplemented work for three related feature areas: + +1. `campaign.yml` configuration for stable campaign-level inputs. +2. Remote `session.yml` loading from the existing S3 object-store backend. +3. Logical archive locks that prevent selected top-level transcript/artifact promotions from overwriting curated archive state while still preserving run-local outputs. + +Treat this document as an implementation plan, not as current behavior. Keep planned behavior under `docs/roadmap/` until each phase is implemented and canonical docs are updated. + +## Current Code Facts + +The current codebase already settles several design choices: + +- CLI commands use short noun flags: `--config`, `--session`, `--session-id`, `--previous-session-id`, `--force`, and `--artifacts`. +- `run-stage` uses flags before the positional stage name, for example: + + narratio run-stage --session ./session.yml prepare + +- `session.yml` is represented by `config.SessionConfig` and currently owns `session_id`, `previous_session_id`, `campaign`, `date`, `title`, and `inputs`. +- Strict YAML decoding is already implemented with `yaml.Decoder.KnownFields(true)`. +- Local session discovery is already ordered as `./session.yml`, `/usr/local/etc/narratio/session.yml`, then `/etc/narratio/session.yml`. +- The canonical S3 session prefix is already: + + {root_prefix}/campaigns/{campaign}/sessions/{session_id}/ + +- Archive promotion is already source-based through `archive.promote_artifacts[].source`, with destination derivation and validation in `internal/config`. +- Storage adapters receive bucket-relative keys and do not infer campaign, session, run, or root-prefix semantics. + +## Guardrails + +Keep Narratio explicit and stage-driven. Do not introduce a generic workflow engine, broad config language, or stage behavior that reaches through adapter boundaries. + +Implementation must preserve these constraints: + +- Keep storage details behind `internal/adapters/storage`. +- Compute session, campaign, archive, and remote config keys in app/artifact/path helpers, not inside storage implementations. +- Use centralized path helpers in `internal/artifacts` or the established local path model. +- Preserve manifest-driven resume and stage status semantics. +- Keep strict YAML decoding for `pipeline.yml`, `campaign.yml`, and `session.yml`. +- Keep raw secrets out of configs, manifests, logs, generated configs, archive metadata, and roadmap examples. +- Update canonical user-facing docs only after behavior is implemented. + +## Phase 1: Add `campaign.yml` + +Add campaign-level configuration for stable campaign identity and stable input files. Do not add remote campaign loading in this phase. + +### CLI and Discovery + +Add `--campaign ` to `run`, `plan`, `resume`, `run-stage`, and `restore`. + +Examples: + + narratio run --campaign ./campaign.yml --session ./session.yml + narratio plan --campaign ./campaign.yml --session ./session.yml + narratio resume --campaign ./campaign.yml --session ./session.yml + narratio run-stage --campaign ./campaign.yml --session ./session.yml prepare + narratio restore --campaign ./campaign.yml --session ./session.yml + +Campaign config discovery order: + +1. explicit `--campaign `; +2. `./campaign.yml`; +3. `/usr/local/etc/narratio/campaign.yml`; +4. `/etc/narratio/campaign.yml`. + +Implement this in the same style as `resolvePipelineConfigPath` and `resolveSessionConfigPath`. Add default path constants and a search-path variable in `internal/config/defaults.go`. + +### Config Shape + +Initial `campaign.yml` fields: + + campaign: icewind-dale + inputs: + speakers_file: ./speakers.yml + autocorrect_file: ./autocorrect.yml + glossary_file: ./glossary.yml + +Do not add speculative campaign artifact defaults, prompt defaults, or title conventions in the first implementation. + +### Merge Behavior + +Add `CampaignConfig` and keep the final stage-facing config explicit. + +Required behavior: + +- `pipeline.yml` remains host/runtime configuration. +- `campaign.yml` supplies campaign identity and stable input file defaults. +- `session.yml` remains the source for `session_id`, `previous_session_id`, `date`, `title`, and audio input. +- Campaign-level `speakers_file`, `autocorrect_file`, and `glossary_file` fill missing session-level stable input fields. +- Session-level stable input fields override campaign-level stable input fields. +- If both `campaign.yml` and `session.yml` specify `campaign`, the values must match. +- The resolved session must satisfy the existing session validation rules before stages run. +- Unknown fields in `campaign.yml` fail strict decode. + +Path resolution must preserve source-file locality: + +- campaign-provided stable input paths resolve relative to `campaign.yml`; +- session-provided stable input overrides resolve relative to `session.yml`; +- absolute paths keep existing behavior. + +Track enough provenance in the resolved config or prepare inputs so `prepare` can copy the correct source files without guessing which file supplied each path. + +### Prepare Behavior + +Update `prepare` to materialize the resolved campaign/session inputs into canonical session input paths: + + inputs/campaign.yml + inputs/session.yml + inputs/pipeline.resolved.yml + inputs/speakers.yml + inputs/autocorrect.yml + inputs/glossary.yml + +Continue recording deterministic `manifest.Inputs` records with checksums. If a prepared input came from `campaign.yml`, record source/provenance using the existing manifest input fields where practical; add narrow metadata only if the existing fields cannot describe it. + +## Phase 2: Load Remote `session.yml` + +Support running with no local session file when a remote session file exists under the canonical session prefix. + +### Preconditions + +Build this phase after `campaign.yml`, because campaign identity is required to compute the remote session key. Do not infer campaign identity from object-store listing. + +### Loading Precedence + +Session loading order: + +1. If `--session ` is supplied, load that local file. +2. If `--session` is omitted, use existing local discovery: `./session.yml`, `/usr/local/etc/narratio/session.yml`, `/etc/narratio/session.yml`. +3. If no local session file is found, `--session-id` is present, storage is configured, and campaign identity is resolved, load remote `session.yml`. +4. If no local or remote session can be loaded, fail with a message that lists the local search paths and the remote key that was attempted when applicable. + +Do not make remote loading mask local discovery. Existing local discovery remains the local fallback before remote is attempted. Once remote loading is attempted, a missing remote object, storage init error, or malformed remote YAML fails clearly because no local session was available. + +### Remote Key Layout + +Use the existing canonical S3 layout: + + session prefix: {root_prefix}/campaigns/{campaign}/sessions/{session_id}/ + session file: {session_prefix}/session.yml + audio prefix: {session_prefix}/{session.inputs.audio_s3.prefix} + +Add a centralized helper near `internal/artifacts/s3_keys.go`: + + S3SessionConfigKey(sessionPrefix string) string + +The helper should return `{session_prefix}/session.yml` using the same key normalization style as `S3CurrentManifestKey`, `S3CurrentRunPointerKey`, and `S3PromotedArtifactKey`. + +### Decode, Template, and Provenance + +Remote `session.yml` uses the same template variables and mismatch checks as local sessions: + +- `{{session_id}}` +- `{{ session_id }}` +- `{{previous_session_id}}` +- `{{ previous_session_id }}` + +Decode remote session YAML with strict known-field validation. Reuse the current session template/render/decode path by adding a byte/string-based loader rather than duplicating YAML decode logic. + +When `prepare` materializes a remote session into `inputs/session.yml`, record that it came from S3. Preserve useful non-secret provenance when available: + +- bucket; +- key; +- ETag; +- size; +- local checksum; +- downloaded temp/materialized path. + +## Phase 3: Add Logical Archive Locks + +Add source-based archive locks under `pipeline.archive.locks`. The current promotion system is already source-based, so the first implementation must not support destination-based locks. + +### Config Shape + +Add lock entries: + + archive: + locks: + - source: narratio.transcript.polished + reason: Human-reviewed transcript; do not overwrite automatically. + - source: narratio.artifact.session_recap + reason: Final recap was manually edited. + +Validation rules: + +- `source` is required. +- `source` must be a built-in source ID or configured `narratio.artifact.` accepted by the same source validation used for `promote_artifacts`. +- `reason` is optional and non-secret. +- duplicate lock sources fail validation. +- lock entries do not support `dest` in the first implementation; unknown fields already fail strict decode. + +### Archive Behavior + +Archive must continue uploading complete run-local outputs under `runs/{run_id}/`. + +Promotion behavior: + +1. Resolve promotion source and destination using existing source-based promotion logic. +2. If the promotion source is unlocked, upload the top-level promoted object normally. +3. If the promotion source is locked, skip only the top-level promotion overwrite. +4. Treat locked required promotions as intentional successful skips by default. +5. Continue archive commit when all run-local uploads and all non-locked required promotions succeed. +6. Upload `current/manifest.json` and `current/run_id.txt` in the existing order, with `current/run_id.txt` last. + +Ordinary `--force` must not override locks. Do not implement a lock-break override in this phase. + +### Metadata + +Record locked promotion skips in archive metadata/reporting so operators can distinguish missing optional promotions from lock-protected promotions. + +Include: + +- source ID; +- destination relative path and remote key; +- reason; +- local resolved path; +- resolved provenance; +- whether the original promotion rule was required. + +Keep existing metadata such as `promoted_paths`, `skipped_optional_promotions`, `current_manifest_key`, `current_run_id_key`, and `current_pointer_written`. + +## Phase 4: Future Operator Helpers + +These commands are future work only. Do not implement them with the first campaign, remote-session, or lock changes. + +Potential helper shapes: + + narratio session validate --session-id 2026-06-07 + narratio session init --session-id 2026-06-07 --title "The Black Cabin" + narratio status --session-id 2026-06-07 + narratio locks --session-id 2026-06-07 + narratio lock narratio.artifact.session_recap --session-id 2026-06-07 + narratio unlock narratio.artifact.session_recap --session-id 2026-06-07 + +Potential behavior: + +- validate remote session config; +- check audio object availability; +- inspect committed remote current state; +- list promoted transcripts/artifacts; +- list archive lock status; +- initialize a remote session skeleton; +- publish or sync campaign assets. + +## Implementation Sequence + +Use small, reviewable commits. + +1. Campaign config types and discovery: + add `CampaignConfig`, strict loading, defaults/search paths, `--campaign` flags, and config/app tests. +2. Campaign/session merge: + implement resolved stable input merge, path provenance, validation, and prepare materialization. +3. Campaign docs after implementation: + update canonical docs and examples only for implemented behavior. +4. Remote session key and loader: + add `S3SessionConfigKey`, byte/string session loading, remote download through `ObjectStore`, and app-level precedence tests. +5. Remote session prepare provenance: + materialize downloaded session config and record S3 provenance. +6. Remote session docs after implementation: + update canonical docs and examples only after behavior exists. +7. Archive lock config: + add lock config structs, strict decode coverage, source validation, and duplicate detection. +8. Archive lock enforcement: + skip locked top-level promotions, preserve run-local uploads, record lock metadata, and protect commit ordering. +9. Final sweep: + run focused tests, then `go test ./...`; verify planned behavior remains only in roadmap docs until implemented. + +## Test Plan + +Add focused coverage in these packages: + +- `internal/config`: campaign load, strict decode, discovery constants, merge validation, campaign/session mismatch, lock validation, duplicate lock rejection. +- `internal/app`: `--campaign` parsing on `run`, `plan`, `resume`, `run-stage`, and `restore`; campaign discovery; explicit `--session` precedence; local discovery before remote; remote session fallback when local discovery misses. +- `internal/artifacts`: `S3SessionConfigKey`; canonical session prefix compatibility; source ID validation for lock sources. +- `internal/stage/prepare`: campaign/session stable input materialization; campaign-relative and session-relative path resolution; remote session provenance in `manifest.Inputs`. +- `internal/stage/archive`: locked required promotion succeeds as skipped; unlocked promotion uploads; run-local outputs upload when top-level promotion is locked; `--force` does not break locks; `current/run_id.txt` remains the last upload. +- `internal/adapters/storage`: fake object key normalization and remote session download expectations. + +Run at least: + + go test ./internal/config -v + go test ./internal/app -run TestExecute -v + go test ./internal/artifacts -v + go test ./internal/stage -run 'Prepare|Archive' -v + go test ./internal/adapters/storage -v + go test ./... + +Use fake storage for remote-session and archive-lock behavior. Ordinary tests must not require live S3. + +## Documentation Updates After Implementation + +After each phase is implemented, update only docs for behavior that exists. + +Likely files: + +- `docs/config.md` +- `docs/cli.md` +- `docs/operations.md` +- `docs/internal/stage-prepare.md` +- `docs/internal/stage-archive.md` +- `docs/internal/storage.md` +- `docs/internal/artifacts.md` +- relevant examples under `examples/` + +Do not document remote campaign loading, helper commands, or lock override flags as current behavior until implemented. + +## Remaining Open Decisions + +The codebase resolves the campaign flag name, campaign discovery order, remote session layout, local-vs-remote session precedence, source-based archive lock model, and locked required promotion policy. + +Remaining decisions: + +1. Whether `campaign.yml` should eventually be loadable from S3. Do not implement remote campaign loading in the first phase. +2. Whether a future explicit lock override command or flag is needed. Do not make ordinary `--force` break locks. +3. Whether future helper commands should be top-level commands or subcommands. Keep them out of the first implementation.