distributor/docs/operations.md

Distributor Operations

Normal Workflow

Validate a source bundle:

go run ./cmd/distributor validate examples/source-bundle

Preview a local publication:

go run ./cmd/distributor run --config examples/local-publish.yml --dry-run

Run the local publication:

go run ./cmd/distributor run --config examples/local-publish.yml

Run the local HTML publication:

go run ./cmd/distributor run --config examples/local-html.yml

Preview local fan-out publication:

go run ./cmd/distributor run --config examples/fan-out.yml --dry-run

Preview an environment-gated SSH destination config after editing it for an SSH/SFTP endpoint you control:

go run ./cmd/distributor run --config examples/ssh-destination.yml --dry-run

Filesystem Layout

Source bundles are discovered beneath the configured source root. Each bundle is a directory containing manifest.json.

Destination bundle paths preserve the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under daily/ publishes under daily/ at each destination.

The maintained local examples write under workspace/, which is ignored by Git.

SSH backends use the configured remote path as the backend root. Source bundle discovery and destination bundle paths are relative to that root, using the same logical path rules as local storage.

Destination State

Each published destination bundle contains .distributor.json. This file is the managed sentinel and destination state record. It stores:

• pipeline and destination identity;
• publication timestamp;
• source manifest used for publication;
• copied source output metadata;
• generated output metadata.

manifest.json from the source bundle is not copied as destination state.

Do not edit .distributor.json by hand during normal operation. If it is missing or invalid while destination files remain, distributor treats the destination as unmanaged or conflicted.

Dry Runs

--dry-run loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files or destination state.

Dry-run output is useful before publishing to confirm actions such as publish_new, replace_older, skip_same, and skip_destination_newer.

Retry and Replacement Behavior

If a destination has matching .distributor.json, publication skips it as already published.

If destination state is older than the source manifest and transfer policy allows replacement, publication deletes only managed outputs recorded in .distributor.json plus the state file, then writes the new outputs and state.

If destination state is newer than the source manifest, the default behavior is to skip. If destination state has the same source id and created timestamp but a different digest, publication fails as a conflict.

If a destination path has files but no valid .distributor.json, publication fails as unmanaged content. There is no force overwrite option.

Failure Handling

If one destination fails in a fan-out run, independent later destinations are still planned and executed. The command exits non-zero after printing the final status if any destination failed.

If a write fails during publication, distributor attempts to remove outputs written during that failed attempt so a retry does not see those partial outputs as unmanaged destination content.

After a successful publish or replacement, the internal notifier hook runs. The current default notifier is a no-op. Skipped destinations do not invoke it.

SSH Operation Notes

SSH execution uses SFTP over golang.org/x/crypto/ssh and github.com/pkg/sftp. It does not shell out to ssh, scp, or rsync.

Configure ssh_key_file, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported.

The default host key policy is accept-new. New host keys are written to known_hosts when the file path is writable. Changed host keys are fatal for both strict and accept-new. The off policy disables host key checking and run prints a warning when stdout is enabled.

Recovery boundaries are the same as local storage: replacement deletes only managed output paths recorded in .distributor.json plus the state file, and failed writes are cleaned up where practical. Distributor never performs broad recursive remote deletion.

Caveats

S3 execution, external notification adapters, and force overwrite behavior are unavailable.

For symptom-oriented fixes, see troubleshooting. For config details, see configuration. For command syntax, see CLI.