distributor/docs/internal/publish.md

Publish Internals

Audience: developers and LLM coding agents changing internal/publish.

Purpose

internal/publish plans and executes publication for one validated source bundle and one destination bundle path. It owns destination comparison mapping, output selection, URL planning, managed cleanup selection, replacement safety, and destination state projection.

Inputs And Outputs

Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, state policy, reconciliation policy, takeover policy, transformer resolver, transfer policy, distributor version, and force flag.

Output from planning is a Plan with action, reason, destination identity, selected outputs, state mode, owner scope, reconciliation mode, optional existing single-owner or shared-root state, optional primary URL, and force metadata. Shared-root plans also expose other-owner outputs to preserve, current-owner outputs retained by merge, current-owner outputs deleted by replace, and current-owner outputs to write. Execution writes selected source outputs, generated outputs, and .distributor.json for executable publish or replacement actions.

Boundaries

The package does not parse CLI flags, load config files, open concrete adapters, discover source bundles, select fixed-path bundle candidates, register transforms, prune retained outputs, or render command output. The app layer supplies validated request data and concrete dependencies.

External destination state semantics are documented in docs/integrations/destination-state.md.

Config Fields Used

The package consumes already-defaulted config values for destination publish, transform, links, state, reconciliation, takeover, transfer, and path mapping mode. It uses config.ValidatePublishTransformPolicy for publish/transform consistency.

Adapters Used

The package depends on internal/storage.Backend for source and destination IO, and on a narrow transformer resolver interface for generated outputs. It does not import concrete storage adapters or concrete transform implementations.

State And Manifest Behavior

Planning inspects destination state through internal/state, compares it with the source manifest, and maps comparison outcomes plus transfer and takeover policy into actions: publish_new, replace_older, replace_takeover, force_replace, skip_same, skip_destination_newer, fail_conflict, or fail_unmanaged.

Single-owner destinations compare the whole destination state against the configured pipeline and destination ids. Valid managed identity and source conflicts can become replace_takeover when takeover.mode allows them. Shared-root destinations compare only the current owner scope, keyed by pipeline id and destination id. An absent shared-root owner is publishable for that owner unless a planned output collides with unmanaged storage content. Planned writes to a path owned by another shared-root owner become replace_takeover when takeover.mode allows that managed output path to move to the current owner.

Execution writes destination state after selected outputs are written. Destination state includes copied source output metadata, generated output metadata, output timestamps, embedded source manifest, reconciliation metadata, link metadata when configured, pipeline id, destination id, and publication timestamps.

Skip And Resume Behavior

skip_same and skip_destination_newer execute as no-ops. Replacement-mode single-owner updates remove managed output paths from existing state plus .distributor.json, verify the destination is empty, and write state whose outputs are exactly the new plan. replace_takeover uses managed replacement mechanics and does not retain omitted outputs through merge reconciliation. Replacement-mode shared-root updates remove only current-owner omitted outputs and preserve unrelated owners. Shared-root takeover rewrites only the taken-over output records and current owner records. Merge-mode same-source updates retain omitted managed outputs, overwrite only paths already recorded as managed, reject unmanaged destination path collisions, and write cumulative output state. Failed writes trigger cleanup where practical; same-source merge cleanup removes only newly created outputs from the failed attempt.

Shared-root execution writes schema version 3 state. It preserves unrelated owner records and outputs, updates only the publishing owner metadata, preserves root created_at, and updates root updated_at after successful state writes. Compatible single-owner state for the same pipeline and destination is converted to shared-root state on successful publish.

Forced replacement is explicit per request and deletes the bounded destination bundle path before writing new outputs and state.

Retention pruning is not part of publish execution and does not run automatically after a successful publish. The app-level prune workflow uses destination state after publication to select managed outputs for deletion.

Failure Behavior

Planning fails for incomplete requests, invalid publish/transform policy, invalid state mode, invalid reconciliation mode, output path collisions, invalid destination state, unmanaged destination content without force, shared-root owner path conflicts not allowed by takeover.mode, conflict outcomes not allowed by transfer policy, unresolved transforms, invalid Markdown output selection, and invalid link URL planning.

Execution fails on delete, read, transform output, unmanaged merge path collision, shared-root ownership conflict, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable publish or replacement actions.

Tests To Inspect

• internal/publish/*_test.go
• internal/app/run_test.go
• internal/state/*_test.go
• internal/transform/markdown/*_test.go

Architectural Invariants

• Planning is deterministic for the same request and destination state.
• Destination bundle paths are caller-supplied and backend-root-relative.
• URL generation uses URL path semantics and never infers public URLs from backend config.
• Replacement reconciliation deletes only managed paths recorded in existing state plus .distributor.json for single-owner state, and only current-owner omitted outputs for shared-root state.
• Merge reconciliation never adopts unmanaged content.
• Merge state output records are cumulative for the single owner.
• Shared-root planning is owner-scoped and preserves unrelated owner outputs.
• Shared-root execution writes owner-scoped changes without deleting unrelated owners.
• Forced replacement deletes only within the supplied destination bundle path.
• Destination state is written after selected outputs are written.
• Transform resolution stays behind a caller-supplied interface.
• Unmanaged content is claimed only by explicit force.