Stage: publish

Purpose

Upload run/session outputs to object storage and atomically advance remote current state.

Inputs

successful preceding stages from the canonical stage set
declared run-local diagnostic files
resolved publish output rules
independent per-source remote lock objects read at planning start
the validated prior current commit when an initially locked source may carry
durable previous-session cache files when present

Outputs

uploaded session manifest, declared run-local diagnostics, and selected publish outputs;
uploaded durable previous-session cache files when present;
immutable run-scoped commit manifest; and
current commit pointer, written last.

Exact remote placement and the operator workflow belong in Operations.

Key Behavior

publish.go retains the transaction-owning Run path. Source discovery, manifest-derived archive metadata, confined opening, and checksum validation live in publish_sources.go; artifact-catalog matching, selection, required and optional rules, destination resolution, and collision inputs live in publish_outputs.go. Copying, hashing, immutable upload, reuse checks, and remote verification live in publish_transfer.go; committed-manifest, immutable-commit, and current-pointer encoding and writes live in publish_commit.go. publish_output_plan.go owns the deterministic, I/O-free five-outcome model and its secret-free metadata projection.

when publishing or run upload is disabled, completes successfully with no outputs and records explanatory metadata. This is not an explicit self-skip: the session manifest records success. Enablement and upload policy are fingerprinted, so changing either automatically makes the prior result non-resumable.
validates prerequisite stage success and object-store availability.
derives a deterministic run-archive allowlist from the session manifest's declared current-run outputs, logs, and generated configurations. References outside the current run root, including canonical session files and prior-run diagnostics, are not archive candidates. No distinguished diagnostic file is read or uploaded.
opens each archive candidate beneath its archive root without following symlinked ancestors or leaf entries, verifies that it is a regular file and checks a declared checksum when present, then streams the opened descriptor.
derives the durable previous-cache archive from its validated manifest using the same confinement and regular-file checks.
resolves publish output sources through runtime artifact catalog and manifest-aware resolution. Configured Scriptorium outputs are publishable only from validated current per-artifact analyze evidence; an incidental canonical file, aggregate output, stale/failed/unselected record, or mismatched path, size, or checksum remains unavailable. Built-in and extraction sources likewise require their own current manifest evidence.
publishes extraction lanes only through explicit configured output rules; neither run-local nor durable Notarius bundles are scanned or uploaded wholesale.
selected artifact filter applies to configured artifact sources only.
plans every configured output into exactly one source-sorted outcome: published, carried_locked, locked_unpublished, optional_missing, or unselected.
an initially locked selected output carries its exact prior published_output declaration when one exists. The declaration may point to an earlier run in the same session and retains its source, key, checksum, size, and generation; the bytes are not copied.
verifies a carried object's exact key, size, and generation through object metadata without downloading or opening the local output source.
an initially locked source with no prior declaration remains absent even when the output rule is required. Optional missing unlocked outputs remain absent; required missing unlocked outputs fail.
creates one complete immutable source-to-destination mapping before upload;
uploads and verifies every declared immutable object and the commit manifest;
writes the immutable commit from the combined new and exact carried declarations.
rereads every initially unlocked participating source after the immutable commit. A newly present lock aborts selection, leaving the uploaded immutable attempt unselected. Removing an initial lock may conservatively produce one carry because the initial snapshot controls the plan.
updates current/commit-pointer.json exactly once, last, and writes no other mutable publication control record.

Metadata Signals

Includes counts/lists for:

run uploads
published output uploads
previous uploads
one publish_output_outcomes collection containing all five output states, exact declared keys, and key-derived origin runs where applicable
remote commit and current-pointer key paths
the run identifier selected by the commit

Invariants

current/commit-pointer.json is the remote commit marker and is written last.
run files, selected outputs, previous-cache files, and the committed session manifest are all declared by an immutable commit under the run prefix.
run and previous uploads contain only manifest-declared regular files opened from verified descriptors; symlinks, special files, replacement races, and undeclared entries are rejected or ignored before uploads begin.
run-local diagnostics, including Notarius receipt and stderr files, are archived only when recorded by the session manifest. A run with no declared current-run diagnostics legitimately uploads none.
independent publish locks are not overridden by --force; every initially unlocked relevant source is reread immediately before current-state selection.
successful uploaded publication records the exact run ID, remote commit key, and current-pointer key consumed by the later cleanup stage.

Resume Semantics

The versioned semantic fingerprint covers enabled behavior, run-upload policy, normalized source/destination/required output rules, and the remote backend, bucket, region, endpoint, and root-prefix identity. Rule ordering is canonicalized. Credential environment names, path-addressing transport mode, local workspace placement, and run identifiers are excluded. Independent remote locks remain mutable state and are read directly by publish at planning and commit boundaries; semantic evidence does not replace those safety checks.

The commit boundary and cleanup authorization are normative architecture invariants; see Architecture.

Publish never deletes local state. The standalone publish command selects only this stage; a full run selects notify and cleanup afterward.

Configuration owns output rules.
Operations owns remote lock lifecycle and physical remote state.
Artifact Internals explains source resolution and current-state helpers.
Implementation and tests: internal/stage/publish.go, internal/stage/publish_sources.go, internal/stage/publish_outputs.go, internal/stage/publish_output_plan.go, internal/stage/publish_transfer.go, internal/stage/publish_commit.go, internal/stage/publish_test.go, internal/stage/semantic_contracts_delivery.go, internal/app/operator_helpers_test.go, and internal/app/cleanup_orchestration_test.go

7.5 KiB Raw Blame History