S3-Compatible Storage Integration

Audience: operators and maintainers configuring S3-compatible sources or destinations.

The S3 backend uses the AWS SDK for Go v2 against a configured S3-compatible endpoint.

Dependencies

Runtime S3 support uses:

github.com/aws/aws-sdk-go-v2
github.com/aws/aws-sdk-go-v2/config
github.com/aws/aws-sdk-go-v2/credentials
github.com/aws/aws-sdk-go-v2/service/s3
github.com/aws/smithy-go

Exact versions are pinned in go.mod.

Config Contract

Required fields:

backend: s3
endpoint
bucket

Optional fields:

prefix: backend root object-key prefix; leading and trailing slashes are trimmed.
region: defaults to us-east-1.
force_path_style: defaults to true through config defaults.
credentials.access_key_id_env
credentials.secret_access_key_env

Credential environment variable names must either both be configured or both be omitted. When configured, values are resolved through the process environment or secrets.directory before opening the backend. When omitted, the AWS SDK default credential chain is used.

Object Key Mapping

The configured bucket plus optional prefix is the backend root. Logical storage paths are joined under that prefix using slash-separated object keys.

Prefixes and logical paths must be clean relative slash-separated paths. Prefixes are object-key prefixes, not real directories.

Storage Behavior

The adapter uses these S3 operations:

HeadObject for stat and overwrite checks.
GetObject for reads.
PutObject for writes.
ListObjectsV2 for walks and prefix deletion planning.
DeleteObject for managed cleanup and replacement.

Writes buffer the input and set ContentLength. If no content type is supplied by the caller, the adapter infers a content type from the logical path.

Normal replacement and failed-write cleanup delete only managed output objects plus .distributor.json. Same-source merge publication retains omitted managed objects and may overwrite existing managed objects. Takeover replacement does not retain omitted outputs through merge reconciliation. Forced replacement deletes objects under the bounded destination bundle prefix. The backend does not manage bucket versioning, lifecycle rules, object lock, or delete markers.

Error Mapping

S3 not-found responses are translated into shared storage not-found errors where practical. Other service, credential, network, or endpoint errors are surfaced through storage errors with backend and logical path context.

Boundaries

The S3 integration does not create buckets, alter bucket policy, configure TLS bypass, manage public URLs, or infer website URLs from endpoint or bucket settings. Public URL metadata is configured separately with destination links.

Tests

Before changing this integration, inspect and run:

go test ./internal/adapters/s3

Live S3-compatible tests are opt-in and gated by environment variables in the adapter test package.

2.9 KiB Raw Blame History