Added roadmap documentation for the upcoming refactoring of the publish stage
This commit is contained in:
287
docs/roadmap/publish.md
Normal file
287
docs/roadmap/publish.md
Normal file
@@ -0,0 +1,287 @@
|
||||
# Roadmap: Publish Contract
|
||||
|
||||
Status: Planned
|
||||
|
||||
## Problem
|
||||
|
||||
Narratio currently uses several terms for one operator-facing concept:
|
||||
|
||||
- `archive` is the stage that uploads run state and commits remote current
|
||||
state.
|
||||
- `publish` is the convenience command that force-runs the archive stage.
|
||||
- `promote`, `promoted`, and `promote_artifacts` describe configured top-level
|
||||
remote output writes.
|
||||
|
||||
This mixed vocabulary makes the public contract harder to explain. Operators
|
||||
should not need to distinguish "archive the run", "publish the run", and
|
||||
"promote artifacts" when these are all part of the same publish action.
|
||||
|
||||
The public model should use:
|
||||
|
||||
- `publish` for the stage, command, config section, and action;
|
||||
- `published` for an expected remote output that exists at its top-level
|
||||
current destination;
|
||||
- `publish rules` for the configured source-to-destination output rules;
|
||||
- `locked` for sources whose top-level published destination must not be
|
||||
overwritten;
|
||||
- `run history` for immutable per-run records under `runs/<run_id>/`.
|
||||
|
||||
## Target Model
|
||||
|
||||
The public stage is `publish`.
|
||||
|
||||
The convenience command:
|
||||
|
||||
narratio publish <session_id>
|
||||
|
||||
is equivalent to:
|
||||
|
||||
narratio run-stage publish <session_id> --force
|
||||
|
||||
Pipeline configuration uses `publish`:
|
||||
|
||||
publish:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
outputs:
|
||||
- source: narratio.transcript.final_trimmed
|
||||
- source: narratio.artifact.session_recap
|
||||
locks:
|
||||
- source: narratio.artifact.session_recap
|
||||
reason: Final recap was manually edited.
|
||||
|
||||
Publish output rules are source-based. Each rule writes one artifact source to
|
||||
a top-level remote destination. If `dest` is omitted, Narratio derives the
|
||||
destination from the artifact registry or configured artifact output path.
|
||||
|
||||
The mutable remote lock store remains:
|
||||
|
||||
{session_prefix}/locks.yml
|
||||
|
||||
Remote availability output uses `published`:
|
||||
|
||||
Published:
|
||||
- narratio.transcript.final_trimmed remote=published
|
||||
- narratio.artifact.session_recap locked remote=published
|
||||
|
||||
The remote key layout is otherwise unchanged:
|
||||
|
||||
- immutable run history stays under `{session_prefix}/runs/{run_id}/`;
|
||||
- current state stays under `{session_prefix}/current/manifest.json`;
|
||||
- the final commit marker stays `{session_prefix}/current/run_id.txt`;
|
||||
- `current/run_id.txt` is still written last.
|
||||
|
||||
## Compatibility Policy
|
||||
|
||||
This is a hard cutover.
|
||||
|
||||
After implementation:
|
||||
|
||||
- `pipeline.archive` is rejected by strict YAML decoding.
|
||||
- `pipeline.archive.promote_artifacts` is rejected.
|
||||
- `pipeline.workspace.cleanup_after_archive` is rejected.
|
||||
- `pipeline.spool.delete_audio_after_archive` is rejected.
|
||||
- `narratio run-stage archive <session_id>` is an unknown stage.
|
||||
- manifests that record an `archive` stage are not migrated.
|
||||
- old archive/promotion metadata keys are not read as compatibility fallbacks.
|
||||
|
||||
Existing remote objects are not moved or renamed. Remote layout remains stable;
|
||||
the rename changes configuration, stage names, status output, metadata, helper
|
||||
names, tests, examples, and documentation.
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
### Stage 1: Public Schema and Stage Cutover
|
||||
|
||||
Status: Planned
|
||||
|
||||
Switch the public config and stage contract to publish terminology.
|
||||
|
||||
Implementation requirements:
|
||||
|
||||
- Replace `pipeline.archive` with `pipeline.publish`.
|
||||
- Replace `archive.promote_artifacts` with `publish.outputs`.
|
||||
- Keep output rule fields:
|
||||
- `source`
|
||||
- `dest`
|
||||
- `required`
|
||||
- Replace `pipeline.archive.locks` with `pipeline.publish.locks`.
|
||||
- Rename post-publish cleanup fields:
|
||||
- `pipeline.workspace.cleanup_after_publish`
|
||||
- `pipeline.spool.delete_audio_after_publish`
|
||||
- Rename the registered stage from `archive` to `publish`.
|
||||
- Update stage order so `publish` runs after `analyze` and before `notify`.
|
||||
- Update top-level `narratio publish` to target stage `publish`.
|
||||
- Keep `run-stage --artifacts <names> publish` support.
|
||||
- Reject `run-stage --artifacts <names>` for stages other than `analyze` and
|
||||
`publish`.
|
||||
- Preserve the remote commit ordering and storage adapter boundaries.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- `narratio run-stage publish <session_id>` executes the publish stage.
|
||||
- `narratio publish <session_id>` force-runs the publish stage.
|
||||
- `narratio run-stage archive <session_id>` fails clearly as an unknown stage.
|
||||
- Old archive config fields fail strict decoding.
|
||||
- New publish config fields load, default, and validate.
|
||||
|
||||
### Stage 2: Runtime Terminology and Metadata Cutover
|
||||
|
||||
Status: Planned
|
||||
|
||||
Rename implementation concepts and runtime output to publish terminology.
|
||||
|
||||
Implementation requirements:
|
||||
|
||||
- Rename archive/promotion config and runtime types conceptually to
|
||||
publish/output terms.
|
||||
- Rename the remote key helper intent from promoted artifact to published
|
||||
output while keeping generated keys unchanged.
|
||||
- Change helper output:
|
||||
- `Promoted:` becomes `Published:`
|
||||
- `remote=promoted` becomes `remote=published`
|
||||
- lock output uses `published` / `not-published`
|
||||
- Rename publish-stage metadata, including:
|
||||
- `promoted_paths` to `published_paths`
|
||||
- `promoted_files_uploaded` to `published_files_uploaded`
|
||||
- `skipped_optional_promotions` to `skipped_optional_outputs`
|
||||
- `skipped_unselected_promotions` to `skipped_unselected_outputs`
|
||||
- `locked_promotion_count` to `locked_output_count`
|
||||
- `locked_promotions` to `locked_outputs`
|
||||
- Update previous-cache and restore logic to use the `publish` stage and
|
||||
`published_paths` metadata only.
|
||||
- Keep run-local stage output materialization separate from remote publish
|
||||
terminology. If local helper names are confusing, rename them to
|
||||
materialization-oriented names rather than publish names.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Status and artifact helper output use `Published:` and `remote=published`.
|
||||
- Publish metadata contains only publish/output terminology.
|
||||
- Previous-cache and restore behavior works with publish metadata and does not
|
||||
depend on old archive metadata.
|
||||
- Storage adapters still receive explicit keys and no AWS SDK details leak into
|
||||
app or stage logic.
|
||||
|
||||
### Stage 3: Documentation, Examples, and Final Cleanup
|
||||
|
||||
Status: Planned
|
||||
|
||||
Update implemented-behavior docs and remove stale public terminology after the
|
||||
runtime cutover lands.
|
||||
|
||||
Implementation requirements:
|
||||
|
||||
- Update current-behavior docs:
|
||||
- `docs/config.md`
|
||||
- `docs/cli.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/troubleshooting.md`
|
||||
- `docs/architecture.md`
|
||||
- relevant files under `docs/internal/`
|
||||
- Rename `docs/internal/stage-archive.md` to
|
||||
`docs/internal/stage-publish.md`.
|
||||
- Update internal documentation links and references.
|
||||
- Update examples to use:
|
||||
- `publish.outputs`
|
||||
- `publish.locks`
|
||||
- `cleanup_after_publish`
|
||||
- `delete_audio_after_publish`
|
||||
- Update tests and final searches so old terminology remains only in this
|
||||
roadmap as historical context.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Maintained examples load and validate.
|
||||
- Current-behavior docs describe only implemented publish terminology.
|
||||
- Internal docs describe run history, published outputs, locks, and current
|
||||
commit ordering clearly.
|
||||
- Old user-facing archive/promote wording is removed except where discussing
|
||||
historical behavior in this roadmap.
|
||||
|
||||
## Test Guidance
|
||||
|
||||
Focused tests:
|
||||
|
||||
- `go test ./internal/config -v`
|
||||
- `go test ./internal/app -v`
|
||||
- `go test ./internal/stage -v`
|
||||
- `go test ./internal/artifacts -v`
|
||||
|
||||
Full validation:
|
||||
|
||||
- `go test ./...`
|
||||
|
||||
Config tests to add or update:
|
||||
|
||||
- `publish.outputs` defaults and validates.
|
||||
- `publish.outputs[].dest` derives from the artifact registry when omitted.
|
||||
- `publish.locks` validates with the same source rules as publish outputs.
|
||||
- old `archive` fails strict decode.
|
||||
- old `promote_artifacts` fails strict decode.
|
||||
- old cleanup fields fail strict decode.
|
||||
|
||||
App and stage tests to add or update:
|
||||
|
||||
- stage order uses `publish` before `notify`.
|
||||
- `run-stage publish` succeeds.
|
||||
- `run-stage archive` fails clearly.
|
||||
- `narratio publish` force-runs the `publish` stage.
|
||||
- `--artifacts` is accepted for `run-stage publish`.
|
||||
- `--artifacts` error text names `analyze` and `publish`.
|
||||
- status and artifact list output show `Published:` and `remote=published`.
|
||||
- lock output says `published` or `not-published`.
|
||||
- previous-cache and restore use `publish` stage metadata.
|
||||
|
||||
Final searches:
|
||||
|
||||
- Config/stage names:
|
||||
- `pipeline.archive`
|
||||
- `archive:`
|
||||
- `promote_artifacts`
|
||||
- `cleanup_after_archive`
|
||||
- `delete_audio_after_archive`
|
||||
- User-facing output:
|
||||
- `Promoted:`
|
||||
- `remote=promoted`
|
||||
- `not-promoted`
|
||||
- Runtime symbols and metadata:
|
||||
- `ArchiveConfig`
|
||||
- `ArchivePromotionRule`
|
||||
- `S3PromotedArtifactKey`
|
||||
- `promoted_paths`
|
||||
- `promoted_files_uploaded`
|
||||
- `locked_promotions`
|
||||
|
||||
Expected remaining matches should be limited to this roadmap and narrowly
|
||||
justified historical references until the roadmap is fully retired.
|
||||
|
||||
## Architecture Guardrails
|
||||
|
||||
- Keep Narratio explicit and stage-driven.
|
||||
- Do not introduce a generic workflow or DAG abstraction.
|
||||
- Keep strict YAML decoding.
|
||||
- Keep remote path construction centralized.
|
||||
- Keep storage details behind `storage.ObjectStore`.
|
||||
- Keep AWS SDK types inside storage adapters.
|
||||
- Preserve manifest-driven resume and restore behavior.
|
||||
- Preserve current-state commit ordering with `current/run_id.txt` written
|
||||
last.
|
||||
- Keep raw secrets out of configs, manifests, logs, generated configs, and
|
||||
publish metadata.
|
||||
- Keep planned behavior only in this roadmap until implementation lands.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- This is a breaking public/config/stage contract change.
|
||||
- No compatibility aliases are retained.
|
||||
- No migration logic is needed for in-progress local manifests.
|
||||
- No migration logic is needed for old remote manifests.
|
||||
- Existing remote objects are not moved or renamed.
|
||||
- `publish` means uploading run history, writing configured published outputs,
|
||||
and committing current state.
|
||||
- `run history` is the preferred term for immutable per-run records under
|
||||
`runs/<run_id>/`.
|
||||
- `archive` remains acceptable only as a generic English concept in historical
|
||||
roadmap context, not as a public Narratio command, config field, stage name,
|
||||
or metadata term after implementation.
|
||||
Reference in New Issue
Block a user