narratio/docs/roadmap/publish.md

Roadmap: Publish Contract

Status: Planned

Problem

Narratio currently uses several terms for one operator-facing concept:

• archive is the stage that uploads run state and commits remote current state.
• publish is the convenience command that force-runs the archive stage.
• promote, promoted, and promote_artifacts describe configured top-level remote output writes.

This mixed vocabulary makes the public contract harder to explain. Operators should not need to distinguish "archive the run", "publish the run", and "promote artifacts" when these are all part of the same publish action.

The public model should use:

• publish for the stage, command, config section, and action;
• published for an expected remote output that exists at its top-level current destination;
• publish rules for the configured source-to-destination output rules;
• locked for sources whose top-level published destination must not be overwritten;
• run history for immutable per-run records under runs/<run_id>/.

Target Model

The public stage is publish.

The convenience command:

narratio publish <session_id>

is equivalent to:

narratio run-stage publish <session_id> --force

Pipeline configuration uses publish:

publish:
  enabled: true
  upload_run: true
  outputs:
    - source: narratio.transcript.final_trimmed
    - source: narratio.artifact.session_recap
  locks:
    - source: narratio.artifact.session_recap
      reason: Final recap was manually edited.

Publish output rules are source-based. Each rule writes one artifact source to a top-level remote destination. If dest is omitted, Narratio derives the destination from the artifact registry or configured artifact output path.

The mutable remote lock store remains:

{session_prefix}/locks.yml

Remote availability output uses published:

Published:
- narratio.transcript.final_trimmed remote=published
- narratio.artifact.session_recap locked remote=published

The remote key layout is otherwise unchanged:

• immutable run history stays under {session_prefix}/runs/{run_id}/;
• current state stays under {session_prefix}/current/manifest.json;
• the final commit marker stays {session_prefix}/current/run_id.txt;
• current/run_id.txt is still written last.

Compatibility Policy

This is a hard cutover.

After implementation:

• pipeline.archive is rejected by strict YAML decoding.
• pipeline.archive.promote_artifacts is rejected.
• pipeline.workspace.cleanup_after_archive is rejected.
• pipeline.spool.delete_audio_after_archive is rejected.
• narratio run-stage archive <session_id> is an unknown stage.
• manifests that record an archive stage are not migrated.
• old archive/promotion metadata keys are not read as compatibility fallbacks.

Existing remote objects are not moved or renamed. Remote layout remains stable; the rename changes configuration, stage names, status output, metadata, helper names, tests, examples, and documentation.

Implementation Stages

Stage 1: Public Schema and Stage Cutover

Status: Planned

Switch the public config and stage contract to publish terminology.

Implementation requirements:

• Replace pipeline.archive with pipeline.publish.
• Replace archive.promote_artifacts with publish.outputs.
• Keep output rule fields:
  • source
  • dest
  • required
• Replace pipeline.archive.locks with pipeline.publish.locks.
• Rename post-publish cleanup fields:
  • pipeline.workspace.cleanup_after_publish
  • pipeline.spool.delete_audio_after_publish
• Rename the registered stage from archive to publish.
• Update stage order so publish runs after analyze and before notify.
• Update top-level narratio publish to target stage publish.
• Keep run-stage --artifacts <names> publish support.
• Reject run-stage --artifacts <names> for stages other than analyze and publish.
• Preserve the remote commit ordering and storage adapter boundaries.

Acceptance criteria:

• narratio run-stage publish <session_id> executes the publish stage.
• narratio publish <session_id> force-runs the publish stage.
• narratio run-stage archive <session_id> fails clearly as an unknown stage.
• Old archive config fields fail strict decoding.
• New publish config fields load, default, and validate.

Stage 2: Runtime Terminology and Metadata Cutover

Status: Planned

Rename implementation concepts and runtime output to publish terminology.

Implementation requirements:

• Rename archive/promotion config and runtime types conceptually to publish/output terms.
• Rename the remote key helper intent from promoted artifact to published output while keeping generated keys unchanged.
• Change helper output:
  • Promoted: becomes Published:
  • remote=promoted becomes remote=published
  • lock output uses published / not-published
• Rename publish-stage metadata, including:
  • promoted_paths to published_paths
  • promoted_files_uploaded to published_files_uploaded
  • skipped_optional_promotions to skipped_optional_outputs
  • skipped_unselected_promotions to skipped_unselected_outputs
  • locked_promotion_count to locked_output_count
  • locked_promotions to locked_outputs
• Update previous-cache and restore logic to use the publish stage and published_paths metadata only.
• Keep run-local stage output materialization separate from remote publish terminology. If local helper names are confusing, rename them to materialization-oriented names rather than publish names.

Acceptance criteria:

• Status and artifact helper output use Published: and remote=published.
• Publish metadata contains only publish/output terminology.
• Previous-cache and restore behavior works with publish metadata and does not depend on old archive metadata.
• Storage adapters still receive explicit keys and no AWS SDK details leak into app or stage logic.

Stage 3: Documentation, Examples, and Final Cleanup

Status: Planned

Update implemented-behavior docs and remove stale public terminology after the runtime cutover lands.

Implementation requirements:

• Update current-behavior docs:
  • docs/config.md
  • docs/cli.md
  • docs/operations.md
  • docs/troubleshooting.md
  • docs/architecture.md
  • relevant files under docs/internal/
• Rename docs/internal/stage-archive.md to docs/internal/stage-publish.md.
• Update internal documentation links and references.
• Update examples to use:
  • publish.outputs
  • publish.locks
  • cleanup_after_publish
  • delete_audio_after_publish
• Update tests and final searches so old terminology remains only in this roadmap as historical context.

Acceptance criteria:

• Maintained examples load and validate.
• Current-behavior docs describe only implemented publish terminology.
• Internal docs describe run history, published outputs, locks, and current commit ordering clearly.
• Old user-facing archive/promote wording is removed except where discussing historical behavior in this roadmap.

Test Guidance

Focused tests:

• go test ./internal/config -v
• go test ./internal/app -v
• go test ./internal/stage -v
• go test ./internal/artifacts -v

Full validation:

• go test ./...

Config tests to add or update:

• publish.outputs defaults and validates.
• publish.outputs[].dest derives from the artifact registry when omitted.
• publish.locks validates with the same source rules as publish outputs.
• old archive fails strict decode.
• old promote_artifacts fails strict decode.
• old cleanup fields fail strict decode.

App and stage tests to add or update:

• stage order uses publish before notify.
• run-stage publish succeeds.
• run-stage archive fails clearly.
• narratio publish force-runs the publish stage.
• --artifacts is accepted for run-stage publish.
• --artifacts error text names analyze and publish.
• status and artifact list output show Published: and remote=published.
• lock output says published or not-published.
• previous-cache and restore use publish stage metadata.

Final searches:

• Config/stage names:
  • pipeline.archive
  • archive:
  • promote_artifacts
  • cleanup_after_archive
  • delete_audio_after_archive
• User-facing output:
  • Promoted:
  • remote=promoted
  • not-promoted
• Runtime symbols and metadata:
  • ArchiveConfig
  • ArchivePromotionRule
  • S3PromotedArtifactKey
  • promoted_paths
  • promoted_files_uploaded
  • locked_promotions

Expected remaining matches should be limited to this roadmap and narrowly justified historical references until the roadmap is fully retired.

Architecture Guardrails

• Keep Narratio explicit and stage-driven.
• Do not introduce a generic workflow or DAG abstraction.
• Keep strict YAML decoding.
• Keep remote path construction centralized.
• Keep storage details behind storage.ObjectStore.
• Keep AWS SDK types inside storage adapters.
• Preserve manifest-driven resume and restore behavior.
• Preserve current-state commit ordering with current/run_id.txt written last.
• Keep raw secrets out of configs, manifests, logs, generated configs, and publish metadata.
• Keep planned behavior only in this roadmap until implementation lands.

Assumptions

• This is a breaking public/config/stage contract change.
• No compatibility aliases are retained.
• No migration logic is needed for in-progress local manifests.
• No migration logic is needed for old remote manifests.
• Existing remote objects are not moved or renamed.
• publish means uploading run history, writing configured published outputs, and committing current state.
• run history is the preferred term for immutable per-run records under runs/<run_id>/.
• archive remains acceptable only as a generic English concept in historical roadmap context, not as a public Narratio command, config field, stage name, or metadata term after implementation.