Storage Backends

This document describes the currently implemented remote object storage backend layer used by Narratio, and its intended role in later prepare/archive work.

Implemented

Remote object store abstraction:

List(ctx, prefix)
Download(ctx, key, localPath)
Upload(ctx, localPath, key, opts)
Exists(ctx, key)

Object metadata model includes:

key
size
ETag (provider metadata only)
last modified time when available

Backends:

fake storage backend for deterministic tests
S3-compatible backend implemented with AWS SDK for Go v2

Construction:

config-based constructor builds S3 backend from pipeline.storage.s3 values:
- bucket
- region
- endpoint
- force_path_style
- access_key_id_env
- secret_access_key_env

Key Invariant

callers pass full bucket-relative object keys
storage backends do not prepend root_prefix
storage backends do not infer campaign/session/run paths

S3 session/run key builders remain separate and continue to live outside backend implementations.

Security Boundary

do not store AWS credentials in Narratio config
Narratio first checks configured env-var names (access_key_id_env, secret_access_key_env); when both are present and non-empty, it uses static credentials from those values
when either configured credential value is missing, Narratio falls back to the standard AWS SDK credential chain
AWS SDK-specific types remain isolated to the storage adapter package

Testing

fake storage tests cover list/download/upload/exists and error paths
S3 backend tests use injected fake S3 API clients
tests do not require live S3 services, AWS credentials, or network access

Not Implemented Yet

prepare-stage S3 object listing or downloads
archive-stage S3 uploads or promotion writes
writing current/manifest.json or current/run_id.txt to S3

1.8 KiB Raw Permalink Blame History