narratio/docs/internal/stage-publish.md

Stage: publish

Purpose

Publish durable run/session state to object storage, then atomically advance remote current state.

Inputs and Outputs

Inputs:

• session manifest and prerequisite stage records
• run root contents under runs/{run_id}/
• publish output rules with artifact source IDs and publish dest paths (pipeline.publish.outputs)
• effective source-based publish locks from static config and remote session lock store
• session-level previous/** cache files when present

Outputs:

• uploaded run files under {session_prefix}/runs/{run_id}/...
• uploaded published outputs under {session_prefix}/...
• uploaded session previous-cache files under {session_prefix}/previous/... when present
• {session_prefix}/current/manifest.json
• {session_prefix}/current/run_id.txt written last

Boundaries

Owns:

• publish enable/disable gate behavior
• prerequisite stage success enforcement
• run file collection and upload (excluding audio/)
• publish output rule resolution and upload
• publish lock enforcement
• session previous-cache file collection/upload
• commit pointer publish order

Does not own:

• stage execution before publish
• post-publish local cleanup policy execution (handled by app cleanup logic)

Config Fields Used

• pipeline.publish.enabled
• pipeline.publish.upload_run
• pipeline.publish.outputs
• pipeline.publish.locks
• {session_prefix}/locks.yml loaded by app orchestration before publish execution
• pipeline.storage.s3.bucket
• pipeline.storage.s3.root_prefix
• pipeline.workspace.root
• session.campaign
• session.session_id

External Adapters Used

• Object storage backend (env.ObjectStore) for upload/list primitives.

State and Manifest Behavior

• Requires prepare, transcribe, merge, polish, normalize, trim, and analyze status succeeded.
• Resolves bucket/prefix from manifest identity first, then config fallback.
• Uploads session previous/** files as durable session state when the local previous/ directory exists.
• Skips top-level published output uploads for effective locked sources; run-local materialized outputs remain unchanged.
• When selected configured artifact keys are supplied, skips publish rules for unselected narratio.artifact.<key> sources; built-in transcript and bounds outputs still publish.
• Effective locks are the union of pipeline.publish.locks and remote {session_prefix}/locks.yml; static pipeline locks win on duplicate sources.
• Writes metadata including:
  • upload counts/paths
  • previous_files_uploaded and previous_uploaded_paths
  • published_files_uploaded and published_paths
  • skipped_optional_outputs
  • skipped_unselected_outputs
  • locked_output_count and locked_outputs
  • current_manifest_key
  • current_run_id_key
  • current_pointer_written
• On skipped publish path, returns metadata with skipped=true and pointer not written.

Skip and Resume Behavior

• Stage may self-skip (metadata skip) when publish disabled or run upload disabled.
• Runner-level skip also applies for previously succeeded stage unless forced.

Failure Behavior

• Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required output source, upload failures, or pointer write failures.
• Locked required outputs are intentional skips and do not fail publish.
• Pointer semantics are fail-safe: current/run_id.txt is not written if prior required uploads fail.

Tests to Inspect Before Changing

• internal/stage/archive_test.go
• internal/app/post_archive_cleanup_test.go

Architectural Invariants

• Run upload excludes audio/ subtree.
• Session previous/** is publishable durable input/provenance state, not run-local output.
• Ordinary --force does not override publish locks.
• Malformed or unreadable remote lock store fails publish-capable execution before output uploads.
• current/manifest.json uploads before current/run_id.txt.
• current/run_id.txt is the remote publish commit marker.