eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Add SSH SFTP backend support

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-31 16:53:37 +00:00
parent 1ad566264f
commit 84f77ec0d0
29 changed files with 1629 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
docs/operations.md
View File

@@ -32,13 +32,21 @@ Preview local fan-out publication:
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
```
Preview an environment-gated SSH destination config after editing it for an SSH/SFTP endpoint you control:
```sh
go run ./cmd/distributor run --config examples/ssh-destination.yml --dry-run
```
## Filesystem Layout
Source bundles are discovered beneath the configured local source root. Each bundle is a directory containing `manifest.json`.
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
Destination bundle paths preserve the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at each destination.
The maintained examples write under `workspace/`, which is ignored by Git.
The maintained local examples write under `workspace/`, which is ignored by Git.
SSH backends use the configured remote `path` as the backend root. Source bundle discovery and destination bundle paths are relative to that root, using the same logical path rules as local storage.
## Destination State
@@ -74,12 +82,22 @@ If a destination path has files but no valid `.distributor.json`, publication fa
If one destination fails in a fan-out run, independent later destinations are still planned and executed. The command exits non-zero after printing the final status if any destination failed.
If a write fails during local publication, `distributor` attempts to remove outputs written during that failed attempt so a retry does not see those partial outputs as unmanaged destination content.
If a write fails during publication, `distributor` attempts to remove outputs written during that failed attempt so a retry does not see those partial outputs as unmanaged destination content.
After a successful publish or replacement, the internal notifier hook runs. The current default notifier is a no-op. Skipped destinations do not invoke it.
## SSH Operation Notes
SSH execution uses SFTP over `golang.org/x/crypto/ssh` and `github.com/pkg/sftp`. It does not shell out to `ssh`, `scp`, or `rsync`.
Configure `ssh_key_file`, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported.
The default host key policy is `accept-new`. New host keys are written to `known_hosts` when the file path is writable. Changed host keys are fatal for both `strict` and `accept-new`. The `off` policy disables host key checking and `run` prints a warning when stdout is enabled.
Recovery boundaries are the same as local storage: replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, and failed writes are cleaned up where practical. Distributor never performs broad recursive remote deletion.
## Caveats
Only local-to-local execution is available. SSH execution, S3 execution, external notification adapters, and force overwrite behavior are unavailable.
S3 execution, external notification adapters, and force overwrite behavior are unavailable.
For symptom-oriented fixes, see [troubleshooting](troubleshooting.md). For config details, see [configuration](config.md). For command syntax, see [CLI](cli.md).