eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Add S3-compatible storage backend

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-31 17:11:53 +00:00
parent 052aa8a64a
commit 14fa9c8000
27 changed files with 1334 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
docs/operations.md
View File

@@ -38,6 +38,12 @@ Preview an environment-gated SSH destination config after editing it for an SSH/
go run ./cmd/distributor run --config examples/ssh-destination.yml --dry-run
```
Preview an environment-gated S3 destination config after editing it for an S3-compatible endpoint and bucket you control:
```sh
go run ./cmd/distributor run --config examples/s3-destination.yml --dry-run
```
## Filesystem Layout
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
@@ -48,6 +54,8 @@ The maintained local examples write under `workspace/`, which is ignored by Git.
SSH backends use the configured remote `path` as the backend root. Source bundle discovery and destination bundle paths are relative to that root, using the same logical path rules as local storage.
S3 backends use the configured bucket plus optional `prefix` as the backend root. Source bundle discovery and destination bundle paths are relative to that object-key prefix. Prefixes are object-key prefixes, not real directories.
## Destination State
Each published destination bundle contains `.distributor.json`. This file is the managed sentinel and destination state record. It stores:
@@ -96,6 +104,14 @@ The default host key policy is `accept-new`. New host keys are written to `known
Recovery boundaries are the same as local storage: replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, and failed writes are cleaned up where practical. Distributor never performs broad recursive remote deletion.
## S3 Operation Notes
S3 execution uses the AWS SDK for Go v2. Configure `endpoint`, `bucket`, optional `prefix`, optional `region`, and optional explicit credential environment variable names.
When explicit credential env names are configured, both variables must resolve to non-empty values through the real process environment or `secrets.directory`. When they are omitted, the AWS SDK default credential chain is used as-is.
Replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Distributor does not perform recursive prefix deletion and does not manage bucket versioning or delete markers.
## Secrets Directory
Configure `secrets.directory` when credential values should come from mounted files, such as deployment secrets:
@@ -111,6 +127,6 @@ Real process environment values take precedence over files with the same name. I
## Caveats
S3 execution, external notification adapters, and force overwrite behavior are unavailable.
External notification adapters and force overwrite behavior are unavailable.
For symptom-oriented fixes, see [troubleshooting](troubleshooting.md). For config details, see [configuration](config.md). For command syntax, see [CLI](cli.md).