Stage: publish

Purpose

Upload run/session outputs to object storage and atomically advance remote current state.

Inputs

successful preceding stages from the canonical stage set
declared run-local diagnostic files
resolved publish output rules
effective publish locks (static + remote merged lock set), revalidated at the remote commit boundary
durable previous-session cache files when present

Outputs

uploaded session manifest, declared run-local diagnostics, and selected publish outputs;
uploaded durable previous-session cache files when present;
immutable run-scoped commit manifest; and
current commit pointer, written last.

Exact remote placement and the operator workflow belong in Operations.

Key Behavior

publish.go retains the transaction-owning Run path. Source discovery, manifest-derived archive metadata, confined opening, and checksum validation live in publish_sources.go; artifact-catalog matching, selection, required and optional rules, destination resolution, and collision inputs live in publish_outputs.go.

when publishing or run upload is disabled, completes successfully with no outputs and records explanatory metadata. This is not an explicit self-skip: the session manifest records success. Enablement and upload policy are fingerprinted, so changing either automatically makes the prior result non-resumable.
validates prerequisite stage success and object-store availability.
derives a deterministic run-archive allowlist from the session manifest's declared current-run outputs, logs, and generated configurations. References outside the current run root, including canonical session files and prior-run diagnostics, are not archive candidates. No distinguished diagnostic file is read or uploaded.
opens each archive candidate beneath its archive root without following symlinked ancestors or leaf entries, verifies that it is a regular file and checks a declared checksum when present, then streams the opened descriptor.
derives the durable previous-cache archive from its validated manifest using the same confinement and regular-file checks.
resolves publish output sources through runtime artifact catalog and manifest-aware resolution. Configured Scriptorium outputs are publishable only from validated current per-artifact analyze evidence; an incidental canonical file, aggregate output, stale/failed/unselected record, or mismatched path, size, or checksum remains unavailable. Built-in and extraction sources likewise require their own current manifest evidence.
publishes extraction lanes only through explicit configured output rules; neither run-local nor durable Notarius bundles are scanned or uploaded wholesale.
selected artifact filter applies to configured artifact sources only.
locked outputs are skipped intentionally (including required ones).
optional missing outputs are skipped; required missing unlocked outputs fail.
creates one complete immutable source-to-destination mapping before upload;
uploads and verifies every declared immutable object and the commit manifest;
updates current/commit-pointer.json exactly once, last; and
writes only the selected current/commit-pointer.json control record.
rechecks remote lock state immediately before the pointer update. A newly committed lock aborts selection, leaving any uploaded immutable attempt unselected.
reads the mutable remote lock document through a direct limit-plus-one read capped by MaxRemoteLockStoreBytes (1 MiB), retaining the generation returned with the opened body for conditional updates. Oversized lock documents fail before YAML decoding; published artifact payloads do not use this limit.

Metadata Signals

Includes counts/lists for:

run uploads
published output uploads
previous uploads
skipped optional outputs
skipped unselected outputs
locked outputs
remote commit and current-pointer key paths
the run identifier selected by the commit

Invariants

current/commit-pointer.json is the remote commit marker and is written last.
run files, selected outputs, previous-cache files, and the committed session manifest are all declared by an immutable commit under the run prefix.
run and previous uploads contain only manifest-declared regular files opened from verified descriptors; symlinks, special files, replacement races, and undeclared entries are rejected or ignored before uploads begin.
run-local diagnostics, including Notarius receipt and stderr files, are archived only when recorded by the session manifest. A run with no declared current-run diagnostics legitimately uploads none.
publish locks are not overridden by --force; remote locks are revalidated immediately before current-state selection.
post-commit local cleanup is authorized by the committed publish metadata and is durably recorded by the application lifecycle before any local deletion.

Resume Semantics

The versioned semantic fingerprint covers enabled behavior, run-upload policy, normalized source/destination/required output rules, static lock policy, and the remote backend, bucket, region, endpoint, and root-prefix identity. Rule and lock ordering is canonicalized. Credential environment names, path-addressing transport mode, local workspace placement, and run identifiers are excluded. Remote locks remain mutable state and are still revalidated at the commit boundary; semantic evidence does not replace that safety check.

The commit boundary and cleanup gate are normative architecture invariants; see Architecture.

Configuration owns output and static-lock fields.
Operations owns remote lock lifecycle and physical remote state.
Artifact Internals explains source resolution and current-state helpers.
Implementation and tests: internal/stage/publish.go, internal/stage/publish_sources.go, internal/stage/publish_outputs.go, internal/stage/publish_test.go, internal/stage/semantic_contracts_delivery.go, internal/app/operator_helpers_test.go, and internal/app/post_publish_cleanup_test.go

6.3 KiB Raw Blame History