Add S3-compatible storage backend
This commit is contained in:
@@ -10,7 +10,7 @@ If `--config` is omitted, `run` uses:
|
||||
/usr/local/etc/distributor/config.yml
|
||||
```
|
||||
|
||||
Config parsing rejects unknown YAML fields. The executable backends are `local` and `ssh`. S3 config fields are accepted by config validation, but runtime execution for S3 is unavailable.
|
||||
Config parsing rejects unknown YAML fields. The executable backends are `local`, `ssh`, and `s3`.
|
||||
|
||||
## Minimal Local Config
|
||||
|
||||
@@ -94,9 +94,9 @@ Source backend:
|
||||
- `host_key_policy`: optional for `ssh`; defaults to `accept-new`.
|
||||
- `endpoint`: required for `s3`.
|
||||
- `bucket`: required for `s3`.
|
||||
- `prefix`: optional for `s3`.
|
||||
- `region`: optional for `s3`.
|
||||
- `force_path_style`: optional for `s3`.
|
||||
- `prefix`: optional for `s3`; leading and trailing slashes are trimmed.
|
||||
- `region`: optional for `s3`; defaults to `us-east-1`.
|
||||
- `force_path_style`: optional for `s3`; defaults to `true`. Set `false` only for services that require virtual-host addressing.
|
||||
- `credentials.access_key_id_env`: optional S3 credential environment variable name.
|
||||
- `credentials.secret_access_key_env`: optional S3 credential environment variable name.
|
||||
|
||||
@@ -112,7 +112,7 @@ Accepted backend names:
|
||||
|
||||
- `local`: executable; requires `path`.
|
||||
- `ssh`: executable; requires `host` and `path`.
|
||||
- `s3`: config validation only; execution is unavailable.
|
||||
- `s3`: executable; requires `endpoint` and `bucket`.
|
||||
|
||||
## SSH Backend
|
||||
|
||||
@@ -139,6 +139,26 @@ Host key policies:
|
||||
|
||||
`accept-new` and `strict` use `known_hosts` when configured. If omitted, distributor uses the current service user's default OpenSSH `known_hosts` path where practical. `accept-new` fails when it needs to persist a new host key and no writable `known_hosts` path is available. It does not create a missing parent `.ssh` directory.
|
||||
|
||||
## S3 Backend
|
||||
|
||||
S3 uses the AWS SDK for Go v2 and supports S3-compatible endpoints:
|
||||
|
||||
```yaml
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
`endpoint` and `bucket` are required. `prefix` is an optional backend root; it is treated as an object-key prefix, not a real directory. Prefixes must be clean slash-separated paths after trimming leading and trailing slashes. `http://` endpoints are allowed for explicitly configured local development or local S3-compatible test services.
|
||||
|
||||
If either credential environment variable name is configured, both must be configured and both referenced variables must resolve to non-empty values through the real process environment or `secrets.directory`. Explicit credentials take precedence over the AWS SDK default credential chain. If credential environment variable names are omitted, the SDK default credential chain is used and `secrets.directory` values are not injected into the process environment.
|
||||
|
||||
Publish policy:
|
||||
|
||||
- `publish.source`: publish source artifacts.
|
||||
@@ -160,6 +180,8 @@ Defaults are applied after YAML decoding and before validation:
|
||||
- `validation.on_digest_mismatch: fail`
|
||||
- SSH `port: 22`
|
||||
- SSH `host_key_policy: accept-new`
|
||||
- S3 `region: us-east-1`
|
||||
- S3 `force_path_style: true`
|
||||
- `publish.source: true`
|
||||
- `publish.html: false`
|
||||
- `transfer.on_destination_same: skip`
|
||||
@@ -185,8 +207,6 @@ S3 credentials may name environment variables:
|
||||
- `credentials.access_key_id_env`
|
||||
- `credentials.secret_access_key_env`
|
||||
|
||||
S3 execution is unavailable; these fields are accepted so config shape can be validated.
|
||||
|
||||
## Examples
|
||||
|
||||
Maintained examples live under [examples](../examples/):
|
||||
@@ -196,3 +216,4 @@ Maintained examples live under [examples](../examples/):
|
||||
- `local-html.yml`: runnable local HTML publication.
|
||||
- `fan-out.yml`: runnable local fan-out publication to source and HTML destinations.
|
||||
- `ssh-destination.yml`: environment-gated local-to-SSH publication example.
|
||||
- `s3-destination.yml`: environment-gated local-to-S3 publication example.
|
||||
|
||||
Reference in New Issue
Block a user