eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Add S3-compatible storage backend

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-31 17:11:53 +00:00
parent 052aa8a64a
commit 14fa9c8000
27 changed files with 1334 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/cli.md
View File

@@ -23,7 +23,7 @@ distributor inspect <path>
- `validate`: validates a local source bundle directory or a local tree containing source bundles.
- `inspect`: validates local source bundles and prints normalized bundle metadata.
`validate` and `inspect` accept local paths only. `run` executes `local` and `ssh` backends. S3 config can be parsed and validated, but configured S3 execution fails with a clear unsupported-execution error.
`validate` and `inspect` accept local paths only. `run` executes `local`, `ssh`, and `s3` backends.
## Flag reference

35
docs/config.md
View File

@@ -10,7 +10,7 @@ If `--config` is omitted, `run` uses:
/usr/local/etc/distributor/config.yml
```
Config parsing rejects unknown YAML fields. The executable backends are `local` and `ssh`. S3 config fields are accepted by config validation, but runtime execution for S3 is unavailable.
Config parsing rejects unknown YAML fields. The executable backends are `local`, `ssh`, and `s3`.
## Minimal Local Config
@@ -94,9 +94,9 @@ Source backend:
- `host_key_policy`: optional for `ssh`; defaults to `accept-new`.
- `endpoint`: required for `s3`.
- `bucket`: required for `s3`.
- `prefix`: optional for `s3`.
- `region`: optional for `s3`.
- `force_path_style`: optional for `s3`.
- `prefix`: optional for `s3`; leading and trailing slashes are trimmed.
- `region`: optional for `s3`; defaults to `us-east-1`.
- `force_path_style`: optional for `s3`; defaults to `true`. Set `false` only for services that require virtual-host addressing.
- `credentials.access_key_id_env`: optional S3 credential environment variable name.
- `credentials.secret_access_key_env`: optional S3 credential environment variable name.
@@ -112,7 +112,7 @@ Accepted backend names:
- `local`: executable; requires `path`.
- `ssh`: executable; requires `host` and `path`.
- `s3`: config validation only; execution is unavailable.
- `s3`: executable; requires `endpoint` and `bucket`.
## SSH Backend
@@ -139,6 +139,26 @@ Host key policies:
`accept-new` and `strict` use `known_hosts` when configured. If omitted, distributor uses the current service user's default OpenSSH `known_hosts` path where practical. `accept-new` fails when it needs to persist a new host key and no writable `known_hosts` path is available. It does not create a missing parent `.ssh` directory.
## S3 Backend
S3 uses the AWS SDK for Go v2 and supports S3-compatible endpoints:
```yaml
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: archive
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
```
`endpoint` and `bucket` are required. `prefix` is an optional backend root; it is treated as an object-key prefix, not a real directory. Prefixes must be clean slash-separated paths after trimming leading and trailing slashes. `http://` endpoints are allowed for explicitly configured local development or local S3-compatible test services.
If either credential environment variable name is configured, both must be configured and both referenced variables must resolve to non-empty values through the real process environment or `secrets.directory`. Explicit credentials take precedence over the AWS SDK default credential chain. If credential environment variable names are omitted, the SDK default credential chain is used and `secrets.directory` values are not injected into the process environment.
Publish policy:
- `publish.source`: publish source artifacts.
@@ -160,6 +180,8 @@ Defaults are applied after YAML decoding and before validation:
- `validation.on_digest_mismatch: fail`
- SSH `port: 22`
- SSH `host_key_policy: accept-new`
- S3 `region: us-east-1`
- S3 `force_path_style: true`
- `publish.source: true`
- `publish.html: false`
- `transfer.on_destination_same: skip`
@@ -185,8 +207,6 @@ S3 credentials may name environment variables:
- `credentials.access_key_id_env`
- `credentials.secret_access_key_env`
S3 execution is unavailable; these fields are accepted so config shape can be validated.
## Examples
Maintained examples live under [examples](../examples/):
@@ -196,3 +216,4 @@ Maintained examples live under [examples](../examples/):
- `local-html.yml`: runnable local HTML publication.
- `fan-out.yml`: runnable local fan-out publication to source and HTML destinations.
- `ssh-destination.yml`: environment-gated local-to-SSH publication example.
- `s3-destination.yml`: environment-gated local-to-S3 publication example.

2
docs/internal/app.md
View File

@@ -27,7 +27,7 @@ Destination failures are collected while later destinations continue to run. Sou
## Backend and transform wiring
The app-level backend factory registers local and SSH backends for execution. Config validation accepts S3 shape, but `Run` cannot execute S3 sources or destinations.
The app-level backend factory registers local, SSH, and S3 backends for execution. S3 explicit credential references are resolved through the config environment resolver before adapter construction.
The app-level transform registry registers Markdown-to-HTML using `internal/transform/markdown`. Lower-level publish code receives a resolver and does not import concrete transform implementations.

4
docs/internal/config.md
View File

@@ -37,10 +37,12 @@ Validation requires at least one pipeline, slug-like unique pipeline ids, one so
## Executable support boundary
Config validation accepts `local`, `ssh`, and `s3` backend shapes so config files can be validated as schemas. Runtime execution opens local and SSH backends through `internal/app`; S3 remains accepted by validation but unavailable at execution.
Config validation accepts `local`, `ssh`, and `s3` backend shapes. Runtime execution opens all three through `internal/app`.
SSH config uses structured fields: `host`, optional `user`, optional `port`, `path`, optional `ssh_key_file`, optional `known_hosts`, and optional `host_key_policy`. `host_key_policy` accepts YAML booleans and strings and normalizes `true`/`strict`, `accept-new`, and `false`/`off`.
S3 config requires `endpoint` and `bucket`, normalizes optional `prefix`, defaults `region` to `us-east-1`, and defaults omitted `force_path_style` to `true` while preserving explicit `false`.
## Secrets and credential resolution
`secrets.directory` points to a directory of credential files. `LoadSecretEnvironment` reads regular files and symlinks to regular files, rejects invalid filenames, trims exactly one trailing LF or CRLF, and returns an `Environment` resolver plus conflict metadata.

6
docs/internal/publish.md
View File

@@ -22,17 +22,17 @@ Execution fails if a write, delete, state serialization, or context check fails.
## Boundaries
The current implementation publishes source files and Markdown-to-HTML sidecar outputs. Backend behavior is supplied through `internal/storage`; app runtime currently supplies local backends.
The current implementation publishes source files and Markdown-to-HTML sidecar outputs. Backend behavior is supplied through `internal/storage`; app runtime currently supplies local, SSH, and S3 backends.
The package uses `internal/state` for destination comparison, `internal/storage` for IO, and the shared `internal/config` publish/transform policy helper for request validation. It resolves transforms through a narrow resolver supplied by the caller; concrete transform registration is owned by the app layer. It does not parse CLI flags or load config files.
## Safety
Replacement deletes only outputs recorded in existing destination state plus `.distributor.json`. Failed local writes trigger cleanup of outputs written during the failed attempt.
Replacement deletes only outputs recorded in existing destination state plus `.distributor.json`. Failed writes trigger cleanup of outputs written during the failed attempt where practical.
## Tests
Before changing publish behavior, inspect tests under `internal/publish` and local run tests under `internal/app`.
Before changing publish behavior, inspect tests under `internal/publish` and run tests under `internal/app`.
## Invariants

7
docs/internal/storage.md
View File

@@ -14,7 +14,7 @@ Entries report a logical path, type, and size when available. Entry types are `f
Core packages should depend on `internal/storage`, not adapter packages. Adapter-specific path handling stays behind backend implementations.
The local adapter lives in `internal/adapters/local`. The SSH/SFTP adapter lives in `internal/adapters/ssh`. Runtime backend construction is wired through the app-level backend factory and storage registry. The fake backend lives in `internal/storage/fake` for tests and is not registered for runtime use.
The local adapter lives in `internal/adapters/local`. The SSH/SFTP adapter lives in `internal/adapters/ssh`. The S3-compatible adapter lives in `internal/adapters/s3`. Runtime backend construction is wired through the app-level backend factory and storage registry. The fake backend lives in `internal/storage/fake` for tests and is not registered for runtime use.
## Paths
@@ -30,12 +30,14 @@ Backends may wrap implementation-specific errors, but callers should receive sto
Backends expose guarded managed deletion only. `DeleteManagedBundle` may delete listed managed outputs plus `.distributor.json`; it does not provide broad recursive deletion.
## Local, SSH, and fake backends
## Local, SSH, S3, and fake backends
The local adapter maps logical paths to a configured filesystem root and keeps adapter-specific path handling behind the storage interface.
The SSH adapter maps logical paths to a configured remote SFTP root. It uses native SSH and SFTP libraries, supports SSH agent and key-file authentication, applies host-key policies, rejects unsafe logical paths, reports symlink entries from `Lstat`, and limits deletion to managed targets.
The S3 adapter maps logical paths to object keys below a configured bucket and optional prefix. It uses the AWS SDK for Go v2, treats prefixes as object trees, requires exact objects for `Stat`, paginates traversal, applies conservative overwrite checks with `HeadObject`, infers basic content types, and limits deletion to managed target objects.
The fake backend is an in-memory implementation for package tests. It is not registered for runtime use.
## Tests
@@ -46,6 +48,7 @@ Before changing storage behavior, inspect tests under:
- `internal/storage/fake`
- `internal/adapters/local`
- `internal/adapters/ssh`
- `internal/adapters/s3`
## Invariants

18
docs/operations.md
View File

@@ -38,6 +38,12 @@ Preview an environment-gated SSH destination config after editing it for an SSH/
go run ./cmd/distributor run --config examples/ssh-destination.yml --dry-run
```
Preview an environment-gated S3 destination config after editing it for an S3-compatible endpoint and bucket you control:
```sh
go run ./cmd/distributor run --config examples/s3-destination.yml --dry-run
```
## Filesystem Layout
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
@@ -48,6 +54,8 @@ The maintained local examples write under `workspace/`, which is ignored by Git.
SSH backends use the configured remote `path` as the backend root. Source bundle discovery and destination bundle paths are relative to that root, using the same logical path rules as local storage.
S3 backends use the configured bucket plus optional `prefix` as the backend root. Source bundle discovery and destination bundle paths are relative to that object-key prefix. Prefixes are object-key prefixes, not real directories.
## Destination State
Each published destination bundle contains `.distributor.json`. This file is the managed sentinel and destination state record. It stores:
@@ -96,6 +104,14 @@ The default host key policy is `accept-new`. New host keys are written to `known
Recovery boundaries are the same as local storage: replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, and failed writes are cleaned up where practical. Distributor never performs broad recursive remote deletion.
## S3 Operation Notes
S3 execution uses the AWS SDK for Go v2. Configure `endpoint`, `bucket`, optional `prefix`, optional `region`, and optional explicit credential environment variable names.
When explicit credential env names are configured, both variables must resolve to non-empty values through the real process environment or `secrets.directory`. When they are omitted, the AWS SDK default credential chain is used as-is.
Replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Distributor does not perform recursive prefix deletion and does not manage bucket versioning or delete markers.
## Secrets Directory
Configure `secrets.directory` when credential values should come from mounted files, such as deployment secrets:
@@ -111,6 +127,6 @@ Real process environment values take precedence over files with the same name. I
## Caveats
S3 execution, external notification adapters, and force overwrite behavior are unavailable.
External notification adapters and force overwrite behavior are unavailable.
For symptom-oriented fixes, see [troubleshooting](troubleshooting.md). For config details, see [configuration](config.md). For command syntax, see [CLI](cli.md).

3
docs/policy/architecture.md
View File

@@ -166,7 +166,7 @@ For example, one destination may publish source files only as a long-term archiv
## Backend Abstraction
Sources and destinations use the same storage abstraction. Current runtime execution uses the local filesystem and SSH/SFTP backends. Additional storage backends should be peer implementations behind the same interface, and any backend-specific execution limitation must be documented.
Sources and destinations use the same storage abstraction. Current runtime execution uses the local filesystem, SSH/SFTP, and S3-compatible backends. Additional storage backends should be peer implementations behind the same interface, and any backend-specific execution limitation must be documented.
Application logic must interact with storage through internal backend interfaces. Backend-specific behavior belongs in adapter packages. Pipeline, bundle, state, publish, and transform packages must not import service-specific or filesystem adapter implementation details.
@@ -195,6 +195,7 @@ Use this current layout unless the project has a documented reason to differ:
- `internal/storage`: backend interfaces, shared path/resource types, backend registry, and storage errors.
- `internal/adapters/local`: local filesystem backend.
- `internal/adapters/ssh`: SSH/SFTP backend.
- `internal/adapters/s3`: S3-compatible object storage backend.
- `internal/transform`: transform interfaces, registry, planning, and shared transform models.
- `internal/transform/markdown`: Markdown-to-HTML implementation.
- `internal/publish`: destination planning, reconciliation, safety checks, and publish execution.

10
docs/policy/development.md
View File

@@ -14,6 +14,7 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
- `internal/storage`: backend interface, registry, logical path rules, typed errors, and shared storage helpers.
- `internal/adapters/local`: local filesystem backend.
- `internal/adapters/ssh`: SSH/SFTP backend.
- `internal/adapters/s3`: S3-compatible object storage backend.
- `internal/storage/fake`: in-memory backend for tests.
- `internal/publish`: destination inspection, output planning, reconciliation, execution, and managed cleanup.
- `internal/transform`: transform interface and registry.
@@ -85,6 +86,7 @@ The project currently depends on:
- `github.com/yuin/goldmark` for Markdown rendering.
- `golang.org/x/crypto/ssh`, `golang.org/x/crypto/ssh/agent`, and `golang.org/x/crypto/ssh/knownhosts` for native SSH support.
- `github.com/pkg/sftp` for native SFTP support.
- `github.com/aws/aws-sdk-go-v2/...` packages for S3-compatible storage support.
Add external dependencies only when they materially improve correctness,
security, interoperability, or implementation complexity. Avoid dependencies
@@ -105,7 +107,7 @@ When adding or changing configuration:
Config validation may accept fields for backends that are not executable yet,
but user-facing docs and examples must clearly state execution support. At the
time of this policy, local and SSH backends are executable.
time of this policy, local, SSH, and S3 backends are executable.
Credential-consuming code must use the config-owned environment resolver for
explicit credential environment variable references. Do not call `os.Getenv`
@@ -126,7 +128,7 @@ When adding or changing commands or flags:
4. Update `docs/cli.md` if syntax, flags, output expectations, or workflows change.
`validate` and `inspect` are local path commands. `run` loads configured
pipelines and currently executes local and SSH backends.
pipelines and currently executes local, SSH, and S3 backends.
## Storage Backends
@@ -142,8 +144,8 @@ When adding a backend:
5. Add focused adapter tests and app-level wiring tests.
6. Update user docs, operations docs, examples, and internal docs only for behavior that is actually implemented.
Do not document S3 execution as available until the corresponding adapter
package and app wiring exist.
Do not document future backend execution as available until the corresponding
adapter package and app wiring exist.
## Transforms

49
docs/troubleshooting.md
View File

@@ -34,11 +34,11 @@ Diagnostic:
rg -n "backend:" <config-path>
```
Safe fix: use `backend: local` or `backend: ssh` for executable workflows. S3 config shape is accepted only for validation; runtime execution is unavailable.
Safe fix: use `backend: local`, `backend: ssh`, or `backend: s3` for executable workflows.
## `backend s3 is not implemented for execution`
## `prefix must be a clean relative slash-separated path`
Likely cause: the config validates but `run` tried to execute S3, which is not implemented.
Likely cause: S3 `prefix` contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.
Diagnostic:
@@ -46,7 +46,44 @@ Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: use `local` or `ssh` for executable workflows. See [configuration](config.md).
Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`.
## `NoSuchBucket`, `InvalidBucketName`, or `not_found`
Likely cause: the S3 bucket, endpoint, or prefix is wrong, or the configured credentials cannot see the requested object.
Diagnostic:
```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: verify `endpoint`, `bucket`, `region`, `force_path_style`, and `prefix`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing.
## `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch`
Likely cause: S3 credentials are missing, wrong, empty, or lack permission for the bucket or prefix.
Diagnostic:
```sh
env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
ls -l <secrets-directory>
```
Safe fix: provide both configured credential environment variables through the real environment or `secrets.directory`, or omit explicit credential fields to use the AWS SDK default credential chain.
## S3 endpoint connection failures
Likely cause: the endpoint URL is unreachable, uses the wrong scheme, or does not match the configured path-style mode.
Diagnostic:
```sh
curl -I <endpoint>
```
Safe fix: correct `endpoint`, network routing, TLS settings outside distributor, or `force_path_style`. Distributor does not provide insecure TLS bypass configuration.
## `load secrets directory ... no such file or directory`
@@ -92,7 +129,7 @@ Likely cause: a backend credential field references an environment variable that
Diagnostic:
```sh
printenv <variable-name>
env | cut -d= -f1 | rg '^<variable-name>$'
ls -l <secrets-directory>/<variable-name>
```
@@ -105,7 +142,7 @@ Likely cause: the real process environment and secrets directory both define the
Diagnostic:
```sh
printenv <variable-name>
env | cut -d= -f1 | rg '^<variable-name>$'
ls -l <secrets-directory>/<variable-name>
```