Stage: publish

Purpose

Upload run/session outputs to object storage and atomically advance remote current state.

Inputs

successful preceding stages from the canonical stage set
invocation-scoped run files
resolved publish output rules
effective publish locks (static + remote merged lock set), revalidated at the remote commit boundary
durable previous-session cache files when present

Outputs

uploaded invocation record and selected publish outputs;
uploaded durable previous-session cache files when present;
immutable run-scoped commit manifest; and
current commit pointer, written last.

Exact remote placement and the operator workflow belong in Operations.

Key Behavior

when publishing or run upload is disabled, completes successfully with no outputs and records explanatory metadata. This is not an explicit self-skip: both manifests record success, and an ordinary later run reuses that result until publish is forced.
validates prerequisite stage success and object-store availability.
derives a deterministic run-archive allowlist from the validated run manifest.json: declared run-local outputs, logs, generated configs, and the manifest itself. Unlisted workspace files are not archive candidates.
opens each archive candidate beneath its archive root without following symlinked ancestors or leaf entries, verifies that it is a regular file and checks a declared checksum when present, then streams the opened descriptor.
derives the durable previous-cache archive from its validated manifest using the same confinement and regular-file checks.
resolves publish output sources through runtime artifact catalog and manifest-aware resolution.
publishes extraction lanes only through explicit configured output rules; neither run-local nor durable Notarius bundles are scanned or uploaded wholesale.
selected artifact filter applies to configured artifact sources only.
locked outputs are skipped intentionally (including required ones).
optional missing outputs are skipped; required missing unlocked outputs fail.
creates one complete immutable source-to-destination mapping before upload;
uploads and verifies every declared immutable object and the commit manifest;
updates current/commit-pointer.json exactly once, last; and
does not write the legacy current/manifest.json or current/run_id.txt pair.
rechecks remote lock state immediately before the pointer update. A newly committed lock aborts selection, leaving any uploaded immutable attempt unselected.
reads the mutable remote lock document through a direct limit-plus-one read capped by MaxRemoteLockStoreBytes (1 MiB), retaining the generation returned with the opened body for conditional updates. Oversized lock documents fail before YAML decoding; published artifact payloads do not use this limit.

Metadata Signals

Includes counts/lists for:

run uploads
published output uploads
previous uploads
skipped optional outputs
skipped unselected outputs
locked outputs
remote commit and current-pointer key paths
the run identifier selected by the commit

Invariants

current/commit-pointer.json is the remote commit marker and is written last.
run files, selected outputs, previous-cache files, and the committed session manifest are all declared by an immutable commit under the run prefix.
run and previous uploads contain only manifest-declared regular files opened from verified descriptors; symlinks, special files, replacement races, and undeclared entries are rejected or ignored before uploads begin.
run-local diagnostics, including Notarius receipt and stderr files, are archived only when recorded by the run manifest.
publish locks are not overridden by --force; remote locks are revalidated immediately before current-state selection.
post-commit local cleanup is authorized by the committed publish metadata and is durably recorded by the application lifecycle before any local deletion.

The commit boundary and cleanup gate are normative architecture invariants; see Architecture.

Configuration owns output and static-lock fields.
Operations owns remote lock lifecycle and physical remote state.
Artifact Internals explains source resolution and current-state helpers.
Implementation and tests: internal/stage/publish.go, internal/stage/publish_test.go, internal/app/operator_helpers_test.go, internal/app/post_publish_cleanup_test.go

4.7 KiB Raw Blame History