diff --git a/docs/roadmap/publish.md b/docs/roadmap/publish.md new file mode 100644 index 0000000..35bffcb --- /dev/null +++ b/docs/roadmap/publish.md @@ -0,0 +1,287 @@ +# Roadmap: Publish Contract + +Status: Planned + +## Problem + +Narratio currently uses several terms for one operator-facing concept: + +- `archive` is the stage that uploads run state and commits remote current + state. +- `publish` is the convenience command that force-runs the archive stage. +- `promote`, `promoted`, and `promote_artifacts` describe configured top-level + remote output writes. + +This mixed vocabulary makes the public contract harder to explain. Operators +should not need to distinguish "archive the run", "publish the run", and +"promote artifacts" when these are all part of the same publish action. + +The public model should use: + +- `publish` for the stage, command, config section, and action; +- `published` for an expected remote output that exists at its top-level + current destination; +- `publish rules` for the configured source-to-destination output rules; +- `locked` for sources whose top-level published destination must not be + overwritten; +- `run history` for immutable per-run records under `runs//`. + +## Target Model + +The public stage is `publish`. + +The convenience command: + + narratio publish + +is equivalent to: + + narratio run-stage publish --force + +Pipeline configuration uses `publish`: + + publish: + enabled: true + upload_run: true + outputs: + - source: narratio.transcript.final_trimmed + - source: narratio.artifact.session_recap + locks: + - source: narratio.artifact.session_recap + reason: Final recap was manually edited. + +Publish output rules are source-based. Each rule writes one artifact source to +a top-level remote destination. If `dest` is omitted, Narratio derives the +destination from the artifact registry or configured artifact output path. + +The mutable remote lock store remains: + + {session_prefix}/locks.yml + +Remote availability output uses `published`: + + Published: + - narratio.transcript.final_trimmed remote=published + - narratio.artifact.session_recap locked remote=published + +The remote key layout is otherwise unchanged: + +- immutable run history stays under `{session_prefix}/runs/{run_id}/`; +- current state stays under `{session_prefix}/current/manifest.json`; +- the final commit marker stays `{session_prefix}/current/run_id.txt`; +- `current/run_id.txt` is still written last. + +## Compatibility Policy + +This is a hard cutover. + +After implementation: + +- `pipeline.archive` is rejected by strict YAML decoding. +- `pipeline.archive.promote_artifacts` is rejected. +- `pipeline.workspace.cleanup_after_archive` is rejected. +- `pipeline.spool.delete_audio_after_archive` is rejected. +- `narratio run-stage archive ` is an unknown stage. +- manifests that record an `archive` stage are not migrated. +- old archive/promotion metadata keys are not read as compatibility fallbacks. + +Existing remote objects are not moved or renamed. Remote layout remains stable; +the rename changes configuration, stage names, status output, metadata, helper +names, tests, examples, and documentation. + +## Implementation Stages + +### Stage 1: Public Schema and Stage Cutover + +Status: Planned + +Switch the public config and stage contract to publish terminology. + +Implementation requirements: + +- Replace `pipeline.archive` with `pipeline.publish`. +- Replace `archive.promote_artifacts` with `publish.outputs`. +- Keep output rule fields: + - `source` + - `dest` + - `required` +- Replace `pipeline.archive.locks` with `pipeline.publish.locks`. +- Rename post-publish cleanup fields: + - `pipeline.workspace.cleanup_after_publish` + - `pipeline.spool.delete_audio_after_publish` +- Rename the registered stage from `archive` to `publish`. +- Update stage order so `publish` runs after `analyze` and before `notify`. +- Update top-level `narratio publish` to target stage `publish`. +- Keep `run-stage --artifacts publish` support. +- Reject `run-stage --artifacts ` for stages other than `analyze` and + `publish`. +- Preserve the remote commit ordering and storage adapter boundaries. + +Acceptance criteria: + +- `narratio run-stage publish ` executes the publish stage. +- `narratio publish ` force-runs the publish stage. +- `narratio run-stage archive ` fails clearly as an unknown stage. +- Old archive config fields fail strict decoding. +- New publish config fields load, default, and validate. + +### Stage 2: Runtime Terminology and Metadata Cutover + +Status: Planned + +Rename implementation concepts and runtime output to publish terminology. + +Implementation requirements: + +- Rename archive/promotion config and runtime types conceptually to + publish/output terms. +- Rename the remote key helper intent from promoted artifact to published + output while keeping generated keys unchanged. +- Change helper output: + - `Promoted:` becomes `Published:` + - `remote=promoted` becomes `remote=published` + - lock output uses `published` / `not-published` +- Rename publish-stage metadata, including: + - `promoted_paths` to `published_paths` + - `promoted_files_uploaded` to `published_files_uploaded` + - `skipped_optional_promotions` to `skipped_optional_outputs` + - `skipped_unselected_promotions` to `skipped_unselected_outputs` + - `locked_promotion_count` to `locked_output_count` + - `locked_promotions` to `locked_outputs` +- Update previous-cache and restore logic to use the `publish` stage and + `published_paths` metadata only. +- Keep run-local stage output materialization separate from remote publish + terminology. If local helper names are confusing, rename them to + materialization-oriented names rather than publish names. + +Acceptance criteria: + +- Status and artifact helper output use `Published:` and `remote=published`. +- Publish metadata contains only publish/output terminology. +- Previous-cache and restore behavior works with publish metadata and does not + depend on old archive metadata. +- Storage adapters still receive explicit keys and no AWS SDK details leak into + app or stage logic. + +### Stage 3: Documentation, Examples, and Final Cleanup + +Status: Planned + +Update implemented-behavior docs and remove stale public terminology after the +runtime cutover lands. + +Implementation requirements: + +- Update current-behavior docs: + - `docs/config.md` + - `docs/cli.md` + - `docs/operations.md` + - `docs/troubleshooting.md` + - `docs/architecture.md` + - relevant files under `docs/internal/` +- Rename `docs/internal/stage-archive.md` to + `docs/internal/stage-publish.md`. +- Update internal documentation links and references. +- Update examples to use: + - `publish.outputs` + - `publish.locks` + - `cleanup_after_publish` + - `delete_audio_after_publish` +- Update tests and final searches so old terminology remains only in this + roadmap as historical context. + +Acceptance criteria: + +- Maintained examples load and validate. +- Current-behavior docs describe only implemented publish terminology. +- Internal docs describe run history, published outputs, locks, and current + commit ordering clearly. +- Old user-facing archive/promote wording is removed except where discussing + historical behavior in this roadmap. + +## Test Guidance + +Focused tests: + +- `go test ./internal/config -v` +- `go test ./internal/app -v` +- `go test ./internal/stage -v` +- `go test ./internal/artifacts -v` + +Full validation: + +- `go test ./...` + +Config tests to add or update: + +- `publish.outputs` defaults and validates. +- `publish.outputs[].dest` derives from the artifact registry when omitted. +- `publish.locks` validates with the same source rules as publish outputs. +- old `archive` fails strict decode. +- old `promote_artifacts` fails strict decode. +- old cleanup fields fail strict decode. + +App and stage tests to add or update: + +- stage order uses `publish` before `notify`. +- `run-stage publish` succeeds. +- `run-stage archive` fails clearly. +- `narratio publish` force-runs the `publish` stage. +- `--artifacts` is accepted for `run-stage publish`. +- `--artifacts` error text names `analyze` and `publish`. +- status and artifact list output show `Published:` and `remote=published`. +- lock output says `published` or `not-published`. +- previous-cache and restore use `publish` stage metadata. + +Final searches: + +- Config/stage names: + - `pipeline.archive` + - `archive:` + - `promote_artifacts` + - `cleanup_after_archive` + - `delete_audio_after_archive` +- User-facing output: + - `Promoted:` + - `remote=promoted` + - `not-promoted` +- Runtime symbols and metadata: + - `ArchiveConfig` + - `ArchivePromotionRule` + - `S3PromotedArtifactKey` + - `promoted_paths` + - `promoted_files_uploaded` + - `locked_promotions` + +Expected remaining matches should be limited to this roadmap and narrowly +justified historical references until the roadmap is fully retired. + +## Architecture Guardrails + +- Keep Narratio explicit and stage-driven. +- Do not introduce a generic workflow or DAG abstraction. +- Keep strict YAML decoding. +- Keep remote path construction centralized. +- Keep storage details behind `storage.ObjectStore`. +- Keep AWS SDK types inside storage adapters. +- Preserve manifest-driven resume and restore behavior. +- Preserve current-state commit ordering with `current/run_id.txt` written + last. +- Keep raw secrets out of configs, manifests, logs, generated configs, and + publish metadata. +- Keep planned behavior only in this roadmap until implementation lands. + +## Assumptions + +- This is a breaking public/config/stage contract change. +- No compatibility aliases are retained. +- No migration logic is needed for in-progress local manifests. +- No migration logic is needed for old remote manifests. +- Existing remote objects are not moved or renamed. +- `publish` means uploading run history, writing configured published outputs, + and committing current state. +- `run history` is the preferred term for immutable per-run records under + `runs//`. +- `archive` remains acceptable only as a generic English concept in historical + roadmap context, not as a public Narratio command, config field, stage name, + or metadata term after implementation.