distributor/docs/internal/publish.md

Publish Internals

Audience: developers and LLM coding agents changing internal/publish.

Purpose

internal/publish plans and executes publication for one validated source bundle and one destination bundle path. It owns output selection, URL planning, catalog action selection, managed cleanup selection, forced replacement safety, and destination state projection.

Inputs And Outputs

Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, workflow, transformer resolver, distributor version, force flag, and request time.

Output from planning is a Plan with action, reason, destination identity, selected outputs, workflow, owner scope, optional existing catalog state, optional superseded legacy marker, optional primary URL, catalog outputs to write, catalog outputs to retain, catalog outputs to delete, force metadata, and clear-root metadata. Execution writes selected source outputs, generated outputs, and schema version 4 .distributor.json for executable catalog actions.

Boundaries

The package does not parse CLI flags, load config files, open concrete adapters, discover source bundles, select fixed-path bundle candidates, register transforms, prune retained outputs, or render command output. The app layer supplies validated request data and concrete dependencies.

External destination state semantics are documented in docs/integrations/destination-state.md.

Config Fields Used

The package consumes already-defaulted config values for destination workflow, publish, transform, links, and path mapping mode. It uses config.ValidatePublishTransformPolicy for publish/transform consistency.

Adapters Used

The package depends on internal/storage.Backend for source and destination IO, and on a narrow transformer resolver interface for generated outputs. It does not import concrete storage adapters or concrete transform implementations.

State And Manifest Behavior

Planning inspects destination state through internal/state and maps catalog conditions into actions:

• publish_new: no valid state exists and the destination bundle path is empty.
• upsert_additive: additive workflow writes planned outputs and retains unrelated catalog outputs.
• replace_catalog: replacement workflow writes planned outputs and deletes omitted outputs for the current owner.
• skip_same: valid catalog metadata already matches every planned output.
• force_replace: explicit bounded replacement selected by Force.
• fail_unmanaged: unmanaged destination content blocks publication.
• fail_conflict: invalid or unsupported state blocks publication.

Superseded legacy state is identified by schema number and planned through catalog output projection. A successful publish writes schema version 4 catalog state.

Execution writes destination state after selected outputs are written. Catalog output records include owner identity, compact source identity, copied source output metadata, generated output metadata, output timestamps, and optional URL metadata.

Workflow Behavior

Additive workflow computes a write set for the planned outputs and preserves catalog outputs for unplanned paths. Existing catalog records for planned paths are replaced by the current owner and source identity. When every planned output already matches catalog metadata, planning returns skip_same and ignores unrelated retained catalog outputs for the no-op decision.

Replacement workflow computes a write set for the planned outputs, preserves other-owner outputs, and deletes omitted outputs owned by the current pipeline and destination. It does not need Force. Matching planned outputs return skip_same only when replacement would not delete omitted outputs for the current owner.

Catalog skip comparison is metadata-only. It checks pipeline id, destination id, source id, source digest, source creation timestamp, output path, kind, digest, size, generated output source path, generated output transform, and output URL metadata. It does not read destination file bytes, and skip_same execution does not write outputs, rewrite .distributor.json, delete files, or notify.

Forced replacement is explicit per request. It deletes the bounded destination bundle path before writing planned outputs and schema version 4 catalog state. Catalog planning selects force_replace only when Force is true and normal planning would otherwise fail for a non-empty no-state destination, a planned path collision with unmanaged storage content, invalid destination state, or unsupported future destination state.

Retention pruning is not part of publish execution and does not run automatically after a successful publish. The app-level prune workflow uses destination state after publication to select managed outputs for deletion.

Failure Behavior

Planning fails for incomplete requests, invalid publish/transform policy, invalid workflow, output path collisions, unresolved transforms, invalid Markdown output selection, invalid link URL planning, invalid destination state without force, unmanaged destination content without force, and unsupported future state without force.

Execution fails on delete, read, transform output, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable catalog publish or replacement actions.

Tests To Inspect

• internal/publish/*_test.go
• internal/app/run_test.go
• internal/state/*_test.go
• internal/transform/markdown/*_test.go

Architectural Invariants

• Planning is deterministic for the same request and destination state.
• Destination bundle paths are caller-supplied and backend-root-relative.
• URL generation uses URL path semantics and never infers public URLs from backend config.
• Additive workflow never adopts unmanaged storage content.
• Replacement workflow deletes only catalog outputs owned by the current pipeline and destination unless force replacement clears the bounded destination bundle path.
• Forced replacement deletes only within the supplied destination bundle path and then writes catalog state using the same output projection as normal catalog publish planning.
• Destination state is written after selected outputs are written.
• Transform resolution stays behind a caller-supplied interface.
• Unmanaged content is claimed only by explicit force.