Add remote storage backend

This commit is contained in:
2026-05-16 14:22:04 +00:00
parent 0454296c81
commit 1e6db89dd4
16 changed files with 1016 additions and 11 deletions

View File

@@ -95,11 +95,14 @@ Implemented in repository:
- campaign/session/run local work and spool path helpers
- manifest run/path identity fields
- examples and tests for the above foundations
- remote storage backend layer:
- object-store abstraction with `List`, `Download`, `Upload`, and `Exists`
- fake storage backend for deterministic, no-network testing
- S3-compatible backend built from `storage.s3` config
- backend construction helper from resolved config
Not implemented yet:
- real S3 backend
- AWS SDK integration
- prepare-stage S3 list/download behavior
- archive-stage S3 upload behavior
- promotion uploads
@@ -849,9 +852,9 @@ Expected commit:
Add archive storage path configuration
```
### Storage Backend Interface and S3 Backend
### Storage Backend Interface and S3 Backend (Implemented)
Implement:
Implemented:
- storage backend interface
- object metadata type
@@ -859,7 +862,7 @@ Implement:
- real S3 backend using AWS SDK or existing project dependency policy
- backend construction from config
No stage behavior yet.
No prepare/archive stage behavior yet.
Expected commit:

View File

@@ -12,11 +12,14 @@ This runbook documents the currently implemented storage/archive foundations and
- promotion rule validation for safe relative paths
- run ID generation and path/key helper functions
- manifest run/path identity fields
- remote storage backend layer:
- object-store interface (`List`, `Download`, `Upload`, `Exists`)
- fake backend for deterministic tests
- S3-compatible backend using AWS SDK v2
- config-based backend construction helper
## Not Implemented Yet
- real S3 backend integration
- AWS SDK wiring
- prepare-stage S3 object listing or download
- archive-stage S3 upload or promotion writes
- writing `current/manifest.json` or `current/run_id.txt` in S3
@@ -29,4 +32,4 @@ This runbook documents the currently implemented storage/archive foundations and
## Next Implementation Target
Build the storage backend layer that can list/download/upload S3 objects through a fake-tested interface, then wire prepare/archive stages to use that backend.
Use the storage backend layer in prepare-stage session audio discovery/download flow, while preserving local audio input support.

58
docs/storage-backends.md Normal file
View File

@@ -0,0 +1,58 @@
# Storage Backends
This document describes the currently implemented remote object storage backend layer used by Narratio, and its intended role in later prepare/archive work.
## Implemented
Remote object store abstraction:
- `List(ctx, prefix)`
- `Download(ctx, key, localPath)`
- `Upload(ctx, localPath, key, opts)`
- `Exists(ctx, key)`
Object metadata model includes:
- key
- size
- ETag (provider metadata only)
- last modified time when available
Backends:
- fake storage backend for deterministic tests
- S3-compatible backend implemented with AWS SDK for Go v2
Construction:
- config-based constructor builds S3 backend from `pipeline.storage.s3` values:
- bucket
- region
- endpoint
- force_path_style
## Key Invariant
- callers pass full bucket-relative object keys
- storage backends do not prepend `root_prefix`
- storage backends do not infer campaign/session/run paths
S3 session/run key builders remain separate and continue to live outside backend implementations.
## Security Boundary
- do not store AWS credentials in Narratio config
- AWS credentials are resolved through standard AWS SDK credential chains
- AWS SDK-specific types remain isolated to the storage adapter package
## Testing
- fake storage tests cover list/download/upload/exists and error paths
- S3 backend tests use injected fake S3 API clients
- tests do not require live S3 services, AWS credentials, or network access
## Not Implemented Yet
- prepare-stage S3 object listing or downloads
- archive-stage S3 uploads or promotion writes
- writing `current/manifest.json` or `current/run_id.txt` to S3