72 lines
2.8 KiB
Markdown
72 lines
2.8 KiB
Markdown
# Internal: Storage
|
|
|
|
## Purpose
|
|
Document Narratio's remote storage backend contracts and implementations under `internal/adapters/storage`.
|
|
|
|
## Inputs and outputs
|
|
Inputs:
|
|
- Resolved storage config (`pipeline.storage.*`).
|
|
- Bucket-relative object keys and local file paths from stage/app orchestration.
|
|
|
|
Outputs:
|
|
- Listed/downloaded/uploaded object metadata (`ObjectInfo`).
|
|
- Existence checks and storage-layer errors.
|
|
|
|
## Boundaries
|
|
Owns:
|
|
- Remote object-store interface and implementation details.
|
|
- S3 client wiring and API calls.
|
|
- Object key normalization and upload/download/list primitives.
|
|
|
|
Does not own:
|
|
- Session/run prefix semantics.
|
|
- Archive commit order semantics.
|
|
- Manifest updates.
|
|
|
|
## Config fields used
|
|
- `pipeline.storage.backend`
|
|
- `pipeline.storage.s3.bucket`
|
|
- `pipeline.storage.s3.region`
|
|
- `pipeline.storage.s3.endpoint`
|
|
- `pipeline.storage.s3.force_path_style`
|
|
- `pipeline.storage.s3.access_key_id_env`
|
|
- `pipeline.storage.s3.secret_access_key_env`
|
|
|
|
## External adapters used
|
|
Storage package contracts:
|
|
- `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`.
|
|
- `Backend` (archive request boundary): currently implemented with `NoopBackend` only.
|
|
|
|
Implementations:
|
|
- `S3Backend`: AWS SDK-backed `ObjectStore` implementation.
|
|
- `FakeBackend`: deterministic test `ObjectStore` and archive backend.
|
|
- `NoopBackend`: deterministic no-op archive backend for compatibility wiring.
|
|
|
|
## State and manifest behavior
|
|
- Storage implementations are stateless with respect to manifest/session lifecycle.
|
|
- Caller supplies fully-qualified bucket-relative keys.
|
|
- Storage layer does not infer campaign/session/run/root-prefix semantics.
|
|
- Caller controls publish ordering; storage layer executes individual operations in the order invoked.
|
|
|
|
## Skip and resume behavior
|
|
- No storage-level skip/resume behavior.
|
|
- Skip/resume decisions are made by stage/app logic before storage calls occur.
|
|
|
|
## Failure behavior
|
|
- `NewObjectStoreFromConfig` fails when no remote backend is configured or required S3 config is missing.
|
|
- `S3Backend` constructor fails when required bucket is missing or AWS client setup fails.
|
|
- CRUD operations return contextual errors (including not-found behavior via `Exists`).
|
|
- Key normalization is applied before operations (`\\` to `/`, leading slash trimmed).
|
|
|
|
## Tests to inspect before changing
|
|
- `internal/adapters/storage/factory_test.go`
|
|
- `internal/adapters/storage/s3_backend_test.go`
|
|
- `internal/adapters/storage/fake_test.go`
|
|
- `internal/adapters/storage/keys_test.go`
|
|
- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `archive`)
|
|
|
|
## Architectural invariants
|
|
- Callers pass full bucket-relative keys.
|
|
- Storage backends must not prepend or infer narratio prefixes.
|
|
- Remote transport details remain isolated to storage adapter implementations.
|