eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Add S3-compatible storage backend

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-31 17:11:53 +00:00
parent 052aa8a64a
commit 14fa9c8000
27 changed files with 1334 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
docs/policy/development.md
View File

@@ -14,6 +14,7 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
- `internal/storage`: backend interface, registry, logical path rules, typed errors, and shared storage helpers.
- `internal/adapters/local`: local filesystem backend.
- `internal/adapters/ssh`: SSH/SFTP backend.
- `internal/adapters/s3`: S3-compatible object storage backend.
- `internal/storage/fake`: in-memory backend for tests.
- `internal/publish`: destination inspection, output planning, reconciliation, execution, and managed cleanup.
- `internal/transform`: transform interface and registry.
@@ -85,6 +86,7 @@ The project currently depends on:
- `github.com/yuin/goldmark` for Markdown rendering.
- `golang.org/x/crypto/ssh`, `golang.org/x/crypto/ssh/agent`, and `golang.org/x/crypto/ssh/knownhosts` for native SSH support.
- `github.com/pkg/sftp` for native SFTP support.
- `github.com/aws/aws-sdk-go-v2/...` packages for S3-compatible storage support.
Add external dependencies only when they materially improve correctness,
security, interoperability, or implementation complexity. Avoid dependencies
@@ -105,7 +107,7 @@ When adding or changing configuration:
Config validation may accept fields for backends that are not executable yet,
but user-facing docs and examples must clearly state execution support. At the
time of this policy, local and SSH backends are executable.
time of this policy, local, SSH, and S3 backends are executable.
Credential-consuming code must use the config-owned environment resolver for
explicit credential environment variable references. Do not call `os.Getenv`
@@ -126,7 +128,7 @@ When adding or changing commands or flags:
4. Update `docs/cli.md` if syntax, flags, output expectations, or workflows change.
`validate` and `inspect` are local path commands. `run` loads configured
pipelines and currently executes local and SSH backends.
pipelines and currently executes local, SSH, and S3 backends.
## Storage Backends
@@ -142,8 +144,8 @@ When adding a backend:
5. Add focused adapter tests and app-level wiring tests.
6. Update user docs, operations docs, examples, and internal docs only for behavior that is actually implemented.
Do not document S3 execution as available until the corresponding adapter
package and app wiring exist.
Do not document future backend execution as available until the corresponding
adapter package and app wiring exist.
## Transforms