Add S3-compatible storage backend

This commit is contained in:
2026-05-31 17:11:53 +00:00
parent 052aa8a64a
commit 14fa9c8000
27 changed files with 1334 additions and 45 deletions

View File

@@ -10,7 +10,7 @@ If `--config` is omitted, `run` uses:
/usr/local/etc/distributor/config.yml
```
Config parsing rejects unknown YAML fields. The executable backends are `local` and `ssh`. S3 config fields are accepted by config validation, but runtime execution for S3 is unavailable.
Config parsing rejects unknown YAML fields. The executable backends are `local`, `ssh`, and `s3`.
## Minimal Local Config
@@ -94,9 +94,9 @@ Source backend:
- `host_key_policy`: optional for `ssh`; defaults to `accept-new`.
- `endpoint`: required for `s3`.
- `bucket`: required for `s3`.
- `prefix`: optional for `s3`.
- `region`: optional for `s3`.
- `force_path_style`: optional for `s3`.
- `prefix`: optional for `s3`; leading and trailing slashes are trimmed.
- `region`: optional for `s3`; defaults to `us-east-1`.
- `force_path_style`: optional for `s3`; defaults to `true`. Set `false` only for services that require virtual-host addressing.
- `credentials.access_key_id_env`: optional S3 credential environment variable name.
- `credentials.secret_access_key_env`: optional S3 credential environment variable name.
@@ -112,7 +112,7 @@ Accepted backend names:
- `local`: executable; requires `path`.
- `ssh`: executable; requires `host` and `path`.
- `s3`: config validation only; execution is unavailable.
- `s3`: executable; requires `endpoint` and `bucket`.
## SSH Backend
@@ -139,6 +139,26 @@ Host key policies:
`accept-new` and `strict` use `known_hosts` when configured. If omitted, distributor uses the current service user's default OpenSSH `known_hosts` path where practical. `accept-new` fails when it needs to persist a new host key and no writable `known_hosts` path is available. It does not create a missing parent `.ssh` directory.
## S3 Backend
S3 uses the AWS SDK for Go v2 and supports S3-compatible endpoints:
```yaml
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: archive
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
```
`endpoint` and `bucket` are required. `prefix` is an optional backend root; it is treated as an object-key prefix, not a real directory. Prefixes must be clean slash-separated paths after trimming leading and trailing slashes. `http://` endpoints are allowed for explicitly configured local development or local S3-compatible test services.
If either credential environment variable name is configured, both must be configured and both referenced variables must resolve to non-empty values through the real process environment or `secrets.directory`. Explicit credentials take precedence over the AWS SDK default credential chain. If credential environment variable names are omitted, the SDK default credential chain is used and `secrets.directory` values are not injected into the process environment.
Publish policy:
- `publish.source`: publish source artifacts.
@@ -160,6 +180,8 @@ Defaults are applied after YAML decoding and before validation:
- `validation.on_digest_mismatch: fail`
- SSH `port: 22`
- SSH `host_key_policy: accept-new`
- S3 `region: us-east-1`
- S3 `force_path_style: true`
- `publish.source: true`
- `publish.html: false`
- `transfer.on_destination_same: skip`
@@ -185,8 +207,6 @@ S3 credentials may name environment variables:
- `credentials.access_key_id_env`
- `credentials.secret_access_key_env`
S3 execution is unavailable; these fields are accepted so config shape can be validated.
## Examples
Maintained examples live under [examples](../examples/):
@@ -196,3 +216,4 @@ Maintained examples live under [examples](../examples/):
- `local-html.yml`: runnable local HTML publication.
- `fan-out.yml`: runnable local fan-out publication to source and HTML destinations.
- `ssh-destination.yml`: environment-gated local-to-SSH publication example.
- `s3-destination.yml`: environment-gated local-to-S3 publication example.