Implemented operations helper commands for validation, locking, and status
This commit is contained in:
53
docs/roadmap/operations.md
Normal file
53
docs/roadmap/operations.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# Roadmap: Operator Helper Commands
|
||||
|
||||
## Status
|
||||
|
||||
Implemented.
|
||||
|
||||
The operator helper command set is no longer conceptual. Current behavior is documented in:
|
||||
|
||||
- `docs/cli.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/config.md`
|
||||
- `docs/internal/artifacts.md`
|
||||
- `docs/internal/stage-archive.md`
|
||||
|
||||
## Implemented Commands
|
||||
|
||||
- `narratio session validate`
|
||||
- `narratio status --manifest <path>`
|
||||
- `narratio status --session-id <id>`
|
||||
- `narratio session init --output <path>`
|
||||
- `narratio session init --remote`
|
||||
- `narratio artifacts list`
|
||||
- `narratio artifacts list --remote`
|
||||
- `narratio locks`
|
||||
- `narratio lock <source>`
|
||||
- `narratio unlock <source>`
|
||||
|
||||
## Implemented Decisions
|
||||
|
||||
- Helper output is text-only. No JSON schema exists yet.
|
||||
- `status` remains a top-level command.
|
||||
- `session validate`, `session init`, and `artifacts list` are nested helper commands.
|
||||
- `lock`, `unlock`, and `locks` are top-level commands.
|
||||
- Remote session initialization requires explicit `--remote`.
|
||||
- Local session initialization requires `--output`.
|
||||
- Remote artifact availability is opt-in with `artifacts list --remote`.
|
||||
- Mutable locks are source-based and stored at `{session_prefix}/locks.yml`.
|
||||
- The remote lock store uses strict YAML with top-level `locks`.
|
||||
- Static `pipeline.archive.locks` and remote locks are merged; static locks win on duplicate sources.
|
||||
- `unlock` removes only remote locks.
|
||||
- Ordinary execution `--force` does not override locks.
|
||||
- Remote lock writes use existence checks and `--force` for updates; there is no compare-and-swap protection.
|
||||
|
||||
## Remaining Future Enhancements
|
||||
|
||||
These are intentionally not implemented:
|
||||
|
||||
- `--json` output for helper commands.
|
||||
- Optimistic concurrency or ETag compare-and-swap for remote lock mutations.
|
||||
- Rich remote artifact availability across historical run-local objects.
|
||||
- Session-lock acquisition for remote mutation helpers.
|
||||
- Broader campaign helper commands such as `campaign validate` or `campaign publish`.
|
||||
|
||||
@@ -1,317 +0,0 @@
|
||||
# Roadmap: Campaign State, Remote Sessions, and Archive Locks
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap tracks implementation work for three related feature areas:
|
||||
|
||||
1. `campaign.yml` configuration for stable campaign-level inputs.
|
||||
2. Remote `session.yml` loading from the existing S3 object-store backend.
|
||||
3. Logical archive locks that prevent selected top-level transcript/artifact promotions from overwriting curated archive state while still preserving run-local outputs.
|
||||
|
||||
Treat future sections of this document as an implementation plan, not as current behavior. Keep planned behavior under `docs/roadmap/` until each item is implemented and canonical docs are updated.
|
||||
|
||||
## Current Code Facts
|
||||
|
||||
The current codebase already settles several design choices:
|
||||
|
||||
- CLI commands use short noun flags: `--config`, `--session`, `--session-id`, `--previous-session-id`, `--force`, and `--artifacts`.
|
||||
- `run-stage` uses flags before the positional stage name, for example:
|
||||
|
||||
narratio run-stage --config ./pipeline.yml --campaign ./campaign.yml --session ./session.yml prepare
|
||||
|
||||
- `session.yml` is represented by `config.SessionConfig` and currently owns `session_id`, `previous_session_id`, `campaign`, `date`, `title`, and `inputs`.
|
||||
- Strict YAML decoding is already implemented with `yaml.Decoder.KnownFields(true)`.
|
||||
- Default local config discovery uses system paths under `/usr/local/etc/narratio/` and `/etc/narratio/`; working-directory files are used only when passed explicitly.
|
||||
- The canonical S3 session prefix is already:
|
||||
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session_id}/
|
||||
|
||||
- Archive promotion is already source-based through `archive.promote_artifacts[].source`, with destination derivation and validation in `internal/config`.
|
||||
- Storage adapters receive bucket-relative keys and do not infer campaign, session, run, or root-prefix semantics.
|
||||
|
||||
## Guardrails
|
||||
|
||||
Keep Narratio explicit and stage-driven. Do not introduce a generic workflow engine, broad config language, or stage behavior that reaches through adapter boundaries.
|
||||
|
||||
Implementation must preserve these constraints:
|
||||
|
||||
- Keep storage details behind `internal/adapters/storage`.
|
||||
- Compute session, campaign, archive, and remote config keys in app/artifact/path helpers, not inside storage implementations.
|
||||
- Use centralized path helpers in `internal/artifacts` or the established local path model.
|
||||
- Preserve manifest-driven resume and stage status semantics.
|
||||
- Keep strict YAML decoding for `pipeline.yml`, `campaign.yml`, and `session.yml`.
|
||||
- Keep raw secrets out of configs, manifests, logs, generated configs, archive metadata, and roadmap examples.
|
||||
- Update canonical user-facing docs only after behavior is implemented.
|
||||
|
||||
## Phase 1: Add `campaign.yml` (implemented)
|
||||
|
||||
Add campaign-level configuration for stable campaign identity and stable input files. Do not add remote campaign loading in this phase.
|
||||
|
||||
### CLI and Discovery
|
||||
|
||||
Add `--campaign <path>` to `run`, `plan`, `resume`, `run-stage`, and `restore`.
|
||||
|
||||
Examples:
|
||||
|
||||
narratio run --campaign ./campaign.yml --session ./session.yml
|
||||
narratio plan --campaign ./campaign.yml --session ./session.yml
|
||||
narratio resume --campaign ./campaign.yml --session ./session.yml
|
||||
narratio run-stage --campaign ./campaign.yml --session ./session.yml prepare
|
||||
narratio restore --campaign ./campaign.yml --session ./session.yml
|
||||
|
||||
Campaign config discovery order:
|
||||
|
||||
1. explicit `--campaign <path>`;
|
||||
2. `/usr/local/etc/narratio/campaign.yml`;
|
||||
3. `/etc/narratio/campaign.yml`.
|
||||
|
||||
Implement this in the same style as `resolvePipelineConfigPath` and `resolveSessionConfigPath`. Add default path constants and a search-path variable in `internal/config/defaults.go`.
|
||||
|
||||
### Config Shape
|
||||
|
||||
Initial `campaign.yml` fields:
|
||||
|
||||
campaign: icewind-dale
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
|
||||
Do not add speculative campaign artifact defaults, prompt defaults, or title conventions in the first implementation.
|
||||
|
||||
### Merge Behavior
|
||||
|
||||
Add `CampaignConfig` and keep the final stage-facing config explicit.
|
||||
|
||||
Required behavior:
|
||||
|
||||
- `pipeline.yml` remains host/runtime configuration.
|
||||
- `campaign.yml` supplies campaign identity and stable input file defaults.
|
||||
- `session.yml` remains the source for `session_id`, `previous_session_id`, `date`, `title`, and audio input.
|
||||
- Campaign-level `speakers_file`, `autocorrect_file`, and `glossary_file` fill missing session-level stable input fields.
|
||||
- Session-level stable input fields override campaign-level stable input fields.
|
||||
- If both `campaign.yml` and `session.yml` specify `campaign`, the values must match.
|
||||
- The resolved session must satisfy the existing session validation rules before stages run.
|
||||
- Unknown fields in `campaign.yml` fail strict decode.
|
||||
|
||||
Path resolution must preserve source-file locality:
|
||||
|
||||
- campaign-provided stable input paths resolve relative to `campaign.yml`;
|
||||
- session-provided stable input overrides resolve relative to `session.yml`;
|
||||
- absolute paths keep existing behavior.
|
||||
|
||||
Track enough provenance in the resolved config or prepare inputs so `prepare` can copy the correct source files without guessing which file supplied each path.
|
||||
|
||||
### Prepare Behavior
|
||||
|
||||
Update `prepare` to materialize the resolved campaign/session inputs into canonical session input paths:
|
||||
|
||||
inputs/campaign.yml
|
||||
inputs/session.yml
|
||||
inputs/pipeline.resolved.yml
|
||||
inputs/speakers.yml
|
||||
inputs/autocorrect.yml
|
||||
inputs/glossary.yml
|
||||
|
||||
Continue recording deterministic `manifest.Inputs` records with checksums. If a prepared input came from `campaign.yml`, record source/provenance using the existing manifest input fields where practical; add narrow metadata only if the existing fields cannot describe it.
|
||||
|
||||
## Phase 2: Load Remote `session.yml` (implemented)
|
||||
|
||||
Support running with no local session file when a remote session file exists under the canonical session prefix.
|
||||
|
||||
### Preconditions
|
||||
|
||||
Build this phase after `campaign.yml`, because campaign identity is required to compute the remote session key. Do not infer campaign identity from object-store listing.
|
||||
|
||||
### Loading Precedence
|
||||
|
||||
Session loading order:
|
||||
|
||||
1. If `--session <path>` is supplied, load that local file.
|
||||
2. If `--session` is omitted, use existing local discovery: `/usr/local/etc/narratio/session.yml`, `/etc/narratio/session.yml`.
|
||||
3. If no local session file is found, `--session-id` is present, storage is configured, and campaign identity is resolved, load remote `session.yml`.
|
||||
4. If no local or remote session can be loaded, fail with a message that lists the local search paths and the remote key that was attempted when applicable.
|
||||
|
||||
Do not make remote loading mask local discovery. Existing local discovery remains the local fallback before remote is attempted. Once remote loading is attempted, a missing remote object, storage init error, or malformed remote YAML fails clearly because no local session was available.
|
||||
|
||||
### Remote Key Layout
|
||||
|
||||
Use the existing canonical S3 layout:
|
||||
|
||||
session prefix: {root_prefix}/campaigns/{campaign}/sessions/{session_id}/
|
||||
session file: {session_prefix}/session.yml
|
||||
audio prefix: {session_prefix}/{session.inputs.audio_s3.prefix}
|
||||
|
||||
Add a centralized helper near `internal/artifacts/s3_keys.go`:
|
||||
|
||||
S3SessionConfigKey(sessionPrefix string) string
|
||||
|
||||
The helper should return `{session_prefix}/session.yml` using the same key normalization style as `S3CurrentManifestKey`, `S3CurrentRunPointerKey`, and `S3PromotedArtifactKey`.
|
||||
|
||||
### Decode, Template, and Provenance
|
||||
|
||||
Remote `session.yml` uses the same template variables and mismatch checks as local sessions:
|
||||
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- `{{previous_session_id}}`
|
||||
- `{{ previous_session_id }}`
|
||||
|
||||
Decode remote session YAML with strict known-field validation. Reuse the current session template/render/decode path by adding a byte/string-based loader rather than duplicating YAML decode logic.
|
||||
|
||||
When `prepare` materializes a remote session into `inputs/session.yml`, record that it came from S3. Preserve useful non-secret provenance when available:
|
||||
|
||||
- bucket;
|
||||
- key;
|
||||
- ETag;
|
||||
- size;
|
||||
- local checksum;
|
||||
- downloaded temp/materialized path.
|
||||
|
||||
## Phase 3: Add Logical Archive Locks (implemented)
|
||||
|
||||
Narratio supports source-based archive locks under `pipeline.archive.locks`. The promotion system is source-based; destination-based locks are not supported.
|
||||
|
||||
### Config Shape
|
||||
|
||||
Lock entries:
|
||||
|
||||
archive:
|
||||
locks:
|
||||
- source: narratio.transcript.polished
|
||||
reason: Human-reviewed transcript; do not overwrite automatically.
|
||||
- source: narratio.artifact.session_recap
|
||||
reason: Final recap was manually edited.
|
||||
|
||||
Validation rules:
|
||||
|
||||
- `source` is required.
|
||||
- `source` must be a built-in source ID or configured `narratio.artifact.<key>` accepted by the same source validation used for `promote_artifacts`.
|
||||
- `reason` is optional and non-secret.
|
||||
- duplicate lock sources fail validation.
|
||||
- lock entries do not support `dest` in the first implementation; unknown fields already fail strict decode.
|
||||
|
||||
### Archive Behavior
|
||||
|
||||
Archive continues uploading complete run-local outputs under `runs/{run_id}/`.
|
||||
|
||||
Promotion behavior:
|
||||
|
||||
1. Resolve promotion source and destination using existing source-based promotion logic.
|
||||
2. If the promotion source is unlocked, upload the top-level promoted object normally.
|
||||
3. If the promotion source is locked, skip only the top-level promotion overwrite.
|
||||
4. Treat locked required promotions as intentional successful skips by default.
|
||||
5. Continue archive commit when all run-local uploads and all non-locked required promotions succeed.
|
||||
6. Upload `current/manifest.json` and `current/run_id.txt` in the existing order, with `current/run_id.txt` last.
|
||||
|
||||
Ordinary `--force` does not override locks. A lock-break override remains out of scope.
|
||||
|
||||
### Metadata
|
||||
|
||||
Archive records locked promotion skips in archive metadata/reporting so operators can distinguish missing optional promotions from lock-protected promotions.
|
||||
|
||||
Include:
|
||||
|
||||
- source ID;
|
||||
- destination relative path and remote key;
|
||||
- reason;
|
||||
- local resolved path;
|
||||
- resolved provenance;
|
||||
- whether the original promotion rule was required.
|
||||
|
||||
Keep existing metadata such as `promoted_paths`, `skipped_optional_promotions`, `current_manifest_key`, `current_run_id_key`, and `current_pointer_written`.
|
||||
|
||||
## Phase 4: Future Operator Helpers
|
||||
|
||||
These commands are future work only. Do not implement them with the first campaign, remote-session, or lock changes.
|
||||
|
||||
Potential helper shapes:
|
||||
|
||||
narratio session validate --session-id 2026-06-07
|
||||
narratio session init --session-id 2026-06-07 --title "The Black Cabin"
|
||||
narratio status --session-id 2026-06-07
|
||||
narratio locks --session-id 2026-06-07
|
||||
narratio lock narratio.artifact.session_recap --session-id 2026-06-07
|
||||
narratio unlock narratio.artifact.session_recap --session-id 2026-06-07
|
||||
|
||||
Potential behavior:
|
||||
|
||||
- validate remote session config;
|
||||
- check audio object availability;
|
||||
- inspect committed remote current state;
|
||||
- list promoted transcripts/artifacts;
|
||||
- list archive lock status;
|
||||
- initialize a remote session skeleton;
|
||||
- publish or sync campaign assets.
|
||||
|
||||
## Implementation Sequence
|
||||
|
||||
Use small, reviewable commits.
|
||||
|
||||
1. Campaign config types and discovery:
|
||||
add `CampaignConfig`, strict loading, defaults/search paths, `--campaign` flags, and config/app tests.
|
||||
2. Campaign/session merge:
|
||||
implement resolved stable input merge, path provenance, validation, and prepare materialization.
|
||||
3. Campaign docs after implementation:
|
||||
update canonical docs and examples only for implemented behavior.
|
||||
4. Remote session key and loader:
|
||||
add `S3SessionConfigKey`, byte/string session loading, remote download through `ObjectStore`, and app-level precedence tests.
|
||||
5. Remote session prepare provenance:
|
||||
materialize downloaded session config and record S3 provenance.
|
||||
6. Remote session docs after implementation:
|
||||
update canonical docs and examples only after behavior exists.
|
||||
7. Archive lock config:
|
||||
add lock config structs, strict decode coverage, source validation, and duplicate detection.
|
||||
8. Archive lock enforcement:
|
||||
skip locked top-level promotions, preserve run-local uploads, record lock metadata, and protect commit ordering.
|
||||
9. Final sweep:
|
||||
run focused tests, then `go test ./...`; verify planned behavior remains only in roadmap docs until implemented.
|
||||
|
||||
## Test Plan
|
||||
|
||||
Add focused coverage in these packages:
|
||||
|
||||
- `internal/config`: campaign load, strict decode, discovery constants, merge validation, campaign/session mismatch, lock validation, duplicate lock rejection.
|
||||
- `internal/app`: `--campaign` parsing on `run`, `plan`, `resume`, `run-stage`, and `restore`; campaign discovery; explicit `--session` precedence; local discovery before remote; remote session fallback when local discovery misses.
|
||||
- `internal/artifacts`: `S3SessionConfigKey`; canonical session prefix compatibility; source ID validation for lock sources.
|
||||
- `internal/stage/prepare`: campaign/session stable input materialization; campaign-relative and session-relative path resolution; remote session provenance in `manifest.Inputs`.
|
||||
- `internal/stage/archive`: locked required promotion succeeds as skipped; unlocked promotion uploads; run-local outputs upload when top-level promotion is locked; `--force` does not break locks; `current/run_id.txt` remains the last upload.
|
||||
- `internal/adapters/storage`: fake object key normalization and remote session download expectations.
|
||||
|
||||
Run at least:
|
||||
|
||||
go test ./internal/config -v
|
||||
go test ./internal/app -run TestExecute -v
|
||||
go test ./internal/artifacts -v
|
||||
go test ./internal/stage -run 'Prepare|Archive' -v
|
||||
go test ./internal/adapters/storage -v
|
||||
go test ./...
|
||||
|
||||
Use fake storage for remote-session and archive-lock behavior. Ordinary tests must not require live S3.
|
||||
|
||||
## Documentation Updates After Implementation
|
||||
|
||||
After each phase is implemented, update only docs for behavior that exists.
|
||||
|
||||
Likely files:
|
||||
|
||||
- `docs/config.md`
|
||||
- `docs/cli.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/stage-prepare.md`
|
||||
- `docs/internal/stage-archive.md`
|
||||
- `docs/internal/storage.md`
|
||||
- `docs/internal/artifacts.md`
|
||||
- relevant examples under `examples/`
|
||||
|
||||
Do not document remote campaign loading, helper commands, or lock override flags as current behavior until implemented.
|
||||
|
||||
## Remaining Open Decisions
|
||||
|
||||
The codebase resolves the campaign flag name, campaign discovery order, remote session layout, local-vs-remote session precedence, source-based archive lock model, and locked required promotion policy.
|
||||
|
||||
Remaining decisions:
|
||||
|
||||
1. Whether `campaign.yml` should eventually be loadable from S3. Do not implement remote campaign loading in the first phase.
|
||||
2. Whether a future explicit lock override command or flag is needed. Do not make ordinary `--force` break locks.
|
||||
3. Whether future helper commands should be top-level commands or subcommands. Keep them out of the first implementation.
|
||||
Reference in New Issue
Block a user