38 lines
1.3 KiB
Markdown
38 lines
1.3 KiB
Markdown
# Internal: Storage
|
|
|
|
## Purpose
|
|
Document remote object-store contracts and S3 implementation behavior.
|
|
|
|
## Primary Contract
|
|
`storage.ObjectStore` interface:
|
|
- `List(ctx, prefix)`
|
|
- `Download(ctx, key, localPath)`
|
|
- `Upload(ctx, localPath, key, opts)`
|
|
- `Exists(ctx, key)`
|
|
|
|
Key invariant:
|
|
- callers pass full bucket-relative keys;
|
|
- storage implementations do not infer campaign/session/run prefixes.
|
|
|
|
## Configuration
|
|
`NewObjectStoreFromConfig` currently supports S3-backed stores from `pipeline.storage.*` config.
|
|
|
|
S3 constructor behavior:
|
|
- requires configured bucket;
|
|
- uses region/endpoint/path-style options when set;
|
|
- resolves credentials from configured env var names (with defaults).
|
|
|
|
## S3 Backend Behavior
|
|
- normalizes object keys.
|
|
- `List` paginates and returns normalized `ObjectInfo`.
|
|
- `Download` writes local files with parent directory creation.
|
|
- `Upload` streams local file and returns remote metadata.
|
|
- `Exists` maps not-found responses to `false`.
|
|
|
|
## Legacy Compatibility Interface
|
|
`storage.Backend` (with `ArchiveRequest`) remains as compatibility surface with `NoopBackend`; it is not used by current stage execution.
|
|
|
|
## Invariants
|
|
- storage layer is stateless regarding manifest/stage progression.
|
|
- publish ordering semantics are owned by stage/app code, not storage adapters.
|