137 lines
6.6 KiB
Markdown
137 lines
6.6 KiB
Markdown
# Distributor Operations
|
|
|
|
## Normal Workflow
|
|
|
|
Validate a source bundle:
|
|
|
|
```sh
|
|
go run ./cmd/distributor validate examples/source-bundle
|
|
```
|
|
|
|
Preview a local publication:
|
|
|
|
```sh
|
|
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
|
|
```
|
|
|
|
Run the local publication:
|
|
|
|
```sh
|
|
go run ./cmd/distributor run --config examples/local-publish.yml
|
|
```
|
|
|
|
Run the local HTML publication:
|
|
|
|
```sh
|
|
go run ./cmd/distributor run --config examples/local-html.yml
|
|
```
|
|
|
|
Preview local fan-out publication:
|
|
|
|
```sh
|
|
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
|
|
```
|
|
|
|
Preview an environment-gated SSH destination config after editing it for an SSH/SFTP endpoint you control:
|
|
|
|
```sh
|
|
go run ./cmd/distributor run --config examples/ssh-destination.yml --dry-run
|
|
```
|
|
|
|
Preview an environment-gated S3 destination config after editing it for an S3-compatible endpoint and bucket you control:
|
|
|
|
```sh
|
|
go run ./cmd/distributor run --config examples/s3-destination.yml --dry-run
|
|
```
|
|
|
|
## Filesystem Layout
|
|
|
|
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
|
|
|
|
Destination bundle paths preserve the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at each destination.
|
|
|
|
The maintained local examples write under `workspace/`, which is ignored by Git.
|
|
|
|
SSH backends use the configured remote `path` as the backend root. Source bundle discovery and destination bundle paths are relative to that root, using the same logical path rules as local storage.
|
|
|
|
S3 backends use the configured bucket plus optional `prefix` as the backend root. Source bundle discovery and destination bundle paths are relative to that object-key prefix. Prefixes are object-key prefixes, not real directories.
|
|
|
|
## Destination State
|
|
|
|
Each published destination bundle contains `.distributor.json`. This file is the managed sentinel and destination state record. It stores:
|
|
|
|
- pipeline and destination identity;
|
|
- publication timestamp;
|
|
- source manifest used for publication;
|
|
- copied source output metadata;
|
|
- generated output metadata.
|
|
|
|
`manifest.json` from the source bundle is not copied as destination state.
|
|
|
|
Do not edit `.distributor.json` by hand during normal operation. If it is missing or invalid while destination files remain, `distributor` treats the destination as unmanaged or conflicted.
|
|
|
|
## Dry Runs
|
|
|
|
`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files or destination state.
|
|
|
|
Dry-run output is useful before publishing to confirm actions such as `publish_new`, `replace_older`, `skip_same`, and `skip_destination_newer`.
|
|
|
|
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication.
|
|
|
|
## Retry and Replacement Behavior
|
|
|
|
If a destination has matching `.distributor.json`, publication skips it as already published.
|
|
|
|
If destination state is older than the source manifest and transfer policy allows replacement, publication deletes only managed outputs recorded in `.distributor.json` plus the state file, then writes the new outputs and state.
|
|
|
|
If destination state is newer than the source manifest, the default behavior is to skip. If destination state has the same source id and created timestamp but a different digest, publication fails as a conflict.
|
|
|
|
If a destination path has files but no valid `.distributor.json`, publication fails as unmanaged content. There is no force overwrite option.
|
|
|
|
## Failure Handling
|
|
|
|
If one destination fails in a fan-out run, independent later destinations are still planned and executed. The command exits non-zero after printing the final status if any destination failed.
|
|
|
|
Errors include the pipeline id, destination id, destination backend, and bundle path where applicable.
|
|
|
|
If a write fails during publication, `distributor` attempts to remove outputs written during that failed attempt so a retry does not see those partial outputs as unmanaged destination content.
|
|
|
|
After a successful publish or replacement, the internal notifier hook runs. The current default notifier is a no-op. Skipped destinations do not invoke it.
|
|
|
|
## SSH Operation Notes
|
|
|
|
SSH execution uses SFTP over `golang.org/x/crypto/ssh` and `github.com/pkg/sftp`. It does not shell out to `ssh`, `scp`, or `rsync`.
|
|
|
|
Configure `ssh_key_file`, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported.
|
|
|
|
The default host key policy is `accept-new`. New host keys are written to `known_hosts` when the file path is writable. Changed host keys are fatal for both `strict` and `accept-new`. The `off` policy disables host key checking and `run` prints a warning when stdout is enabled.
|
|
|
|
Recovery boundaries are the same as local storage: replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, and failed writes are cleaned up where practical. Distributor never performs broad recursive remote deletion.
|
|
|
|
## S3 Operation Notes
|
|
|
|
S3 execution uses the AWS SDK for Go v2. Configure `endpoint`, `bucket`, optional `prefix`, optional `region`, and optional explicit credential environment variable names.
|
|
|
|
When explicit credential env names are configured, both variables must resolve to non-empty values through the real process environment or `secrets.directory`. When they are omitted, the AWS SDK default credential chain is used as-is.
|
|
|
|
Replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Distributor does not perform recursive prefix deletion and does not manage bucket versioning or delete markers.
|
|
|
|
## Secrets Directory
|
|
|
|
Configure `secrets.directory` when credential values should come from mounted files, such as deployment secrets:
|
|
|
|
```yaml
|
|
secrets:
|
|
directory: /run/secrets/distributor
|
|
```
|
|
|
|
The directory is loaded during `run` before any source or destination backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the run fails before publication work starts.
|
|
|
|
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` prints a warning naming the ignored secret file variable without printing either value. The process environment is not changed.
|
|
|
|
## Caveats
|
|
|
|
External notification adapters and force overwrite behavior are unavailable.
|
|
|
|
For symptom-oriented fixes, see [troubleshooting](troubleshooting.md). For config details, see [configuration](config.md). For command syntax, see [CLI](cli.md).
|