narratio/docs/roadmap/remote.md

Roadmap: Campaign State, Remote Sessions, and Archive Locks

Purpose

This roadmap tracks implementation work for three related feature areas:

1. campaign.yml configuration for stable campaign-level inputs.
2. Remote session.yml loading from the existing S3 object-store backend.
3. Logical archive locks that prevent selected top-level transcript/artifact promotions from overwriting curated archive state while still preserving run-local outputs.

Treat future sections of this document as an implementation plan, not as current behavior. Keep planned behavior under docs/roadmap/ until each item is implemented and canonical docs are updated.

Current Code Facts

The current codebase already settles several design choices:

• CLI commands use short noun flags: --config, --session, --session-id, --previous-session-id, --force, and --artifacts.

• run-stage uses flags before the positional stage name, for example:

  narratio run-stage --config ./pipeline.yml --campaign ./campaign.yml --session ./session.yml prepare
  

• session.yml is represented by config.SessionConfig and currently owns session_id, previous_session_id, campaign, date, title, and inputs.

• Strict YAML decoding is already implemented with yaml.Decoder.KnownFields(true).

• Default local config discovery uses system paths under /usr/local/etc/narratio/ and /etc/narratio/; working-directory files are used only when passed explicitly.

• The canonical S3 session prefix is already:

  {root_prefix}/campaigns/{campaign}/sessions/{session_id}/
  

• Archive promotion is already source-based through archive.promote_artifacts[].source, with destination derivation and validation in internal/config.

• Storage adapters receive bucket-relative keys and do not infer campaign, session, run, or root-prefix semantics.

Guardrails

Keep Narratio explicit and stage-driven. Do not introduce a generic workflow engine, broad config language, or stage behavior that reaches through adapter boundaries.

Implementation must preserve these constraints:

• Keep storage details behind internal/adapters/storage.
• Compute session, campaign, archive, and remote config keys in app/artifact/path helpers, not inside storage implementations.
• Use centralized path helpers in internal/artifacts or the established local path model.
• Preserve manifest-driven resume and stage status semantics.
• Keep strict YAML decoding for pipeline.yml, campaign.yml, and session.yml.
• Keep raw secrets out of configs, manifests, logs, generated configs, archive metadata, and roadmap examples.
• Update canonical user-facing docs only after behavior is implemented.

Phase 1: Add campaign.yml (implemented)

Add campaign-level configuration for stable campaign identity and stable input files. Do not add remote campaign loading in this phase.

CLI and Discovery

Add --campaign <path> to run, plan, resume, run-stage, and restore.

Examples:

  narratio run --campaign ./campaign.yml --session ./session.yml
  narratio plan --campaign ./campaign.yml --session ./session.yml
  narratio resume --campaign ./campaign.yml --session ./session.yml
  narratio run-stage --campaign ./campaign.yml --session ./session.yml prepare
  narratio restore --campaign ./campaign.yml --session ./session.yml

Campaign config discovery order:

1. explicit --campaign <path>;
2. /usr/local/etc/narratio/campaign.yml;
3. /etc/narratio/campaign.yml.

Implement this in the same style as resolvePipelineConfigPath and resolveSessionConfigPath. Add default path constants and a search-path variable in internal/config/defaults.go.

Config Shape

Initial campaign.yml fields:

  campaign: icewind-dale
  inputs:
    speakers_file: ./speakers.yml
    autocorrect_file: ./autocorrect.yml
    glossary_file: ./glossary.yml

Do not add speculative campaign artifact defaults, prompt defaults, or title conventions in the first implementation.

Merge Behavior

Add CampaignConfig and keep the final stage-facing config explicit.

Required behavior:

• pipeline.yml remains host/runtime configuration.
• campaign.yml supplies campaign identity and stable input file defaults.
• session.yml remains the source for session_id, previous_session_id, date, title, and audio input.
• Campaign-level speakers_file, autocorrect_file, and glossary_file fill missing session-level stable input fields.
• Session-level stable input fields override campaign-level stable input fields.
• If both campaign.yml and session.yml specify campaign, the values must match.
• The resolved session must satisfy the existing session validation rules before stages run.
• Unknown fields in campaign.yml fail strict decode.

Path resolution must preserve source-file locality:

• campaign-provided stable input paths resolve relative to campaign.yml;
• session-provided stable input overrides resolve relative to session.yml;
• absolute paths keep existing behavior.

Track enough provenance in the resolved config or prepare inputs so prepare can copy the correct source files without guessing which file supplied each path.

Prepare Behavior

Update prepare to materialize the resolved campaign/session inputs into canonical session input paths:

  inputs/campaign.yml
  inputs/session.yml
  inputs/pipeline.resolved.yml
  inputs/speakers.yml
  inputs/autocorrect.yml
  inputs/glossary.yml

Continue recording deterministic manifest.Inputs records with checksums. If a prepared input came from campaign.yml, record source/provenance using the existing manifest input fields where practical; add narrow metadata only if the existing fields cannot describe it.

Phase 2: Load Remote session.yml (implemented)

Support running with no local session file when a remote session file exists under the canonical session prefix.

Preconditions

Build this phase after campaign.yml, because campaign identity is required to compute the remote session key. Do not infer campaign identity from object-store listing.

Loading Precedence

Session loading order:

1. If --session <path> is supplied, load that local file.
2. If --session is omitted, use existing local discovery: /usr/local/etc/narratio/session.yml, /etc/narratio/session.yml.
3. If no local session file is found, --session-id is present, storage is configured, and campaign identity is resolved, load remote session.yml.
4. If no local or remote session can be loaded, fail with a message that lists the local search paths and the remote key that was attempted when applicable.

Do not make remote loading mask local discovery. Existing local discovery remains the local fallback before remote is attempted. Once remote loading is attempted, a missing remote object, storage init error, or malformed remote YAML fails clearly because no local session was available.

Remote Key Layout

Use the existing canonical S3 layout:

  session prefix: {root_prefix}/campaigns/{campaign}/sessions/{session_id}/
  session file:   {session_prefix}/session.yml
  audio prefix:   {session_prefix}/{session.inputs.audio_s3.prefix}

Add a centralized helper near internal/artifacts/s3_keys.go:

  S3SessionConfigKey(sessionPrefix string) string

The helper should return {session_prefix}/session.yml using the same key normalization style as S3CurrentManifestKey, S3CurrentRunPointerKey, and S3PromotedArtifactKey.

Decode, Template, and Provenance

Remote session.yml uses the same template variables and mismatch checks as local sessions:

• {{session_id}}
• {{ session_id }}
• {{previous_session_id}}
• {{ previous_session_id }}

Decode remote session YAML with strict known-field validation. Reuse the current session template/render/decode path by adding a byte/string-based loader rather than duplicating YAML decode logic.

When prepare materializes a remote session into inputs/session.yml, record that it came from S3. Preserve useful non-secret provenance when available:

• bucket;
• key;
• ETag;
• size;
• local checksum;
• downloaded temp/materialized path.

Phase 3: Add Logical Archive Locks (implemented)

Narratio supports source-based archive locks under pipeline.archive.locks. The promotion system is source-based; destination-based locks are not supported.

Config Shape

Lock entries:

  archive:
    locks:
      - source: narratio.transcript.polished
        reason: Human-reviewed transcript; do not overwrite automatically.
      - source: narratio.artifact.session_recap
        reason: Final recap was manually edited.

Validation rules:

• source is required.
• source must be a built-in source ID or configured narratio.artifact.<key> accepted by the same source validation used for promote_artifacts.
• reason is optional and non-secret.
• duplicate lock sources fail validation.
• lock entries do not support dest in the first implementation; unknown fields already fail strict decode.

Archive Behavior

Archive continues uploading complete run-local outputs under runs/{run_id}/.

Promotion behavior:

1. Resolve promotion source and destination using existing source-based promotion logic.
2. If the promotion source is unlocked, upload the top-level promoted object normally.
3. If the promotion source is locked, skip only the top-level promotion overwrite.
4. Treat locked required promotions as intentional successful skips by default.
5. Continue archive commit when all run-local uploads and all non-locked required promotions succeed.
6. Upload current/manifest.json and current/run_id.txt in the existing order, with current/run_id.txt last.

Ordinary --force does not override locks. A lock-break override remains out of scope.

Metadata

Archive records locked promotion skips in archive metadata/reporting so operators can distinguish missing optional promotions from lock-protected promotions.

Include:

• source ID;
• destination relative path and remote key;
• reason;
• local resolved path;
• resolved provenance;
• whether the original promotion rule was required.

Keep existing metadata such as promoted_paths, skipped_optional_promotions, current_manifest_key, current_run_id_key, and current_pointer_written.

Phase 4: Future Operator Helpers

These commands are future work only. Do not implement them with the first campaign, remote-session, or lock changes.

Potential helper shapes:

  narratio session validate --session-id 2026-06-07
  narratio session init --session-id 2026-06-07 --title "The Black Cabin"
  narratio status --session-id 2026-06-07
  narratio locks --session-id 2026-06-07
  narratio lock narratio.artifact.session_recap --session-id 2026-06-07
  narratio unlock narratio.artifact.session_recap --session-id 2026-06-07

Potential behavior:

• validate remote session config;
• check audio object availability;
• inspect committed remote current state;
• list promoted transcripts/artifacts;
• list archive lock status;
• initialize a remote session skeleton;
• publish or sync campaign assets.

Implementation Sequence

Use small, reviewable commits.

1. Campaign config types and discovery: add CampaignConfig, strict loading, defaults/search paths, --campaign flags, and config/app tests.
2. Campaign/session merge: implement resolved stable input merge, path provenance, validation, and prepare materialization.
3. Campaign docs after implementation: update canonical docs and examples only for implemented behavior.
4. Remote session key and loader: add S3SessionConfigKey, byte/string session loading, remote download through ObjectStore, and app-level precedence tests.
5. Remote session prepare provenance: materialize downloaded session config and record S3 provenance.
6. Remote session docs after implementation: update canonical docs and examples only after behavior exists.
7. Archive lock config: add lock config structs, strict decode coverage, source validation, and duplicate detection.
8. Archive lock enforcement: skip locked top-level promotions, preserve run-local uploads, record lock metadata, and protect commit ordering.
9. Final sweep: run focused tests, then go test ./...; verify planned behavior remains only in roadmap docs until implemented.

Test Plan

Add focused coverage in these packages:

• internal/config: campaign load, strict decode, discovery constants, merge validation, campaign/session mismatch, lock validation, duplicate lock rejection.
• internal/app: --campaign parsing on run, plan, resume, run-stage, and restore; campaign discovery; explicit --session precedence; local discovery before remote; remote session fallback when local discovery misses.
• internal/artifacts: S3SessionConfigKey; canonical session prefix compatibility; source ID validation for lock sources.
• internal/stage/prepare: campaign/session stable input materialization; campaign-relative and session-relative path resolution; remote session provenance in manifest.Inputs.
• internal/stage/archive: locked required promotion succeeds as skipped; unlocked promotion uploads; run-local outputs upload when top-level promotion is locked; --force does not break locks; current/run_id.txt remains the last upload.
• internal/adapters/storage: fake object key normalization and remote session download expectations.

Run at least:

  go test ./internal/config -v
  go test ./internal/app -run TestExecute -v
  go test ./internal/artifacts -v
  go test ./internal/stage -run 'Prepare|Archive' -v
  go test ./internal/adapters/storage -v
  go test ./...

Use fake storage for remote-session and archive-lock behavior. Ordinary tests must not require live S3.

Documentation Updates After Implementation

After each phase is implemented, update only docs for behavior that exists.

Likely files:

• docs/config.md
• docs/cli.md
• docs/operations.md
• docs/internal/stage-prepare.md
• docs/internal/stage-archive.md
• docs/internal/storage.md
• docs/internal/artifacts.md
• relevant examples under examples/

Do not document remote campaign loading, helper commands, or lock override flags as current behavior until implemented.

Remaining Open Decisions

The codebase resolves the campaign flag name, campaign discovery order, remote session layout, local-vs-remote session precedence, source-based archive lock model, and locked required promotion policy.

Remaining decisions:

1. Whether campaign.yml should eventually be loadable from S3. Do not implement remote campaign loading in the first phase.
2. Whether a future explicit lock override command or flag is needed. Do not make ordinary --force break locks.
3. Whether future helper commands should be top-level commands or subcommands. Keep them out of the first implementation.