Add remote storage backend
This commit is contained in:
@@ -95,11 +95,14 @@ Implemented in repository:
|
||||
- campaign/session/run local work and spool path helpers
|
||||
- manifest run/path identity fields
|
||||
- examples and tests for the above foundations
|
||||
- remote storage backend layer:
|
||||
- object-store abstraction with `List`, `Download`, `Upload`, and `Exists`
|
||||
- fake storage backend for deterministic, no-network testing
|
||||
- S3-compatible backend built from `storage.s3` config
|
||||
- backend construction helper from resolved config
|
||||
|
||||
Not implemented yet:
|
||||
|
||||
- real S3 backend
|
||||
- AWS SDK integration
|
||||
- prepare-stage S3 list/download behavior
|
||||
- archive-stage S3 upload behavior
|
||||
- promotion uploads
|
||||
@@ -849,9 +852,9 @@ Expected commit:
|
||||
Add archive storage path configuration
|
||||
```
|
||||
|
||||
### Storage Backend Interface and S3 Backend
|
||||
### Storage Backend Interface and S3 Backend (Implemented)
|
||||
|
||||
Implement:
|
||||
Implemented:
|
||||
|
||||
- storage backend interface
|
||||
- object metadata type
|
||||
@@ -859,7 +862,7 @@ Implement:
|
||||
- real S3 backend using AWS SDK or existing project dependency policy
|
||||
- backend construction from config
|
||||
|
||||
No stage behavior yet.
|
||||
No prepare/archive stage behavior yet.
|
||||
|
||||
Expected commit:
|
||||
|
||||
|
||||
@@ -12,11 +12,14 @@ This runbook documents the currently implemented storage/archive foundations and
|
||||
- promotion rule validation for safe relative paths
|
||||
- run ID generation and path/key helper functions
|
||||
- manifest run/path identity fields
|
||||
- remote storage backend layer:
|
||||
- object-store interface (`List`, `Download`, `Upload`, `Exists`)
|
||||
- fake backend for deterministic tests
|
||||
- S3-compatible backend using AWS SDK v2
|
||||
- config-based backend construction helper
|
||||
|
||||
## Not Implemented Yet
|
||||
|
||||
- real S3 backend integration
|
||||
- AWS SDK wiring
|
||||
- prepare-stage S3 object listing or download
|
||||
- archive-stage S3 upload or promotion writes
|
||||
- writing `current/manifest.json` or `current/run_id.txt` in S3
|
||||
@@ -29,4 +32,4 @@ This runbook documents the currently implemented storage/archive foundations and
|
||||
|
||||
## Next Implementation Target
|
||||
|
||||
Build the storage backend layer that can list/download/upload S3 objects through a fake-tested interface, then wire prepare/archive stages to use that backend.
|
||||
Use the storage backend layer in prepare-stage session audio discovery/download flow, while preserving local audio input support.
|
||||
|
||||
58
docs/storage-backends.md
Normal file
58
docs/storage-backends.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Storage Backends
|
||||
|
||||
This document describes the currently implemented remote object storage backend layer used by Narratio, and its intended role in later prepare/archive work.
|
||||
|
||||
## Implemented
|
||||
|
||||
Remote object store abstraction:
|
||||
|
||||
- `List(ctx, prefix)`
|
||||
- `Download(ctx, key, localPath)`
|
||||
- `Upload(ctx, localPath, key, opts)`
|
||||
- `Exists(ctx, key)`
|
||||
|
||||
Object metadata model includes:
|
||||
|
||||
- key
|
||||
- size
|
||||
- ETag (provider metadata only)
|
||||
- last modified time when available
|
||||
|
||||
Backends:
|
||||
|
||||
- fake storage backend for deterministic tests
|
||||
- S3-compatible backend implemented with AWS SDK for Go v2
|
||||
|
||||
Construction:
|
||||
|
||||
- config-based constructor builds S3 backend from `pipeline.storage.s3` values:
|
||||
- bucket
|
||||
- region
|
||||
- endpoint
|
||||
- force_path_style
|
||||
|
||||
## Key Invariant
|
||||
|
||||
- callers pass full bucket-relative object keys
|
||||
- storage backends do not prepend `root_prefix`
|
||||
- storage backends do not infer campaign/session/run paths
|
||||
|
||||
S3 session/run key builders remain separate and continue to live outside backend implementations.
|
||||
|
||||
## Security Boundary
|
||||
|
||||
- do not store AWS credentials in Narratio config
|
||||
- AWS credentials are resolved through standard AWS SDK credential chains
|
||||
- AWS SDK-specific types remain isolated to the storage adapter package
|
||||
|
||||
## Testing
|
||||
|
||||
- fake storage tests cover list/download/upload/exists and error paths
|
||||
- S3 backend tests use injected fake S3 API clients
|
||||
- tests do not require live S3 services, AWS credentials, or network access
|
||||
|
||||
## Not Implemented Yet
|
||||
|
||||
- prepare-stage S3 object listing or downloads
|
||||
- archive-stage S3 uploads or promotion writes
|
||||
- writing `current/manifest.json` or `current/run_id.txt` to S3
|
||||
Reference in New Issue
Block a user