narratio/docs/internal/storage.md

Internal: Storage

Purpose

Document Narratio's remote storage backend contracts and implementations under internal/adapters/storage.

Inputs and outputs

Inputs:

• Resolved storage config (pipeline.storage.*).
• Already-loaded environment variables for configured S3 credentials.
• Bucket-relative object keys and local file paths from app/stage orchestration.

Outputs:

• Listed/downloaded/uploaded object metadata (ObjectInfo).
• Existence checks and storage-layer errors.

Boundaries

Owns:

• Remote object-store interface and implementation details.
• S3 client wiring and API calls.
• Object key normalization and upload/download/list primitives.

Does not own:

• Session/run prefix semantics.
• Archive commit order semantics.
• Manifest updates.
• Filesystem secret loading from pipeline.secrets.env_dir.

Config fields used

• pipeline.storage.backend
• pipeline.storage.s3.bucket
• pipeline.storage.s3.region
• pipeline.storage.s3.endpoint
• pipeline.storage.s3.force_path_style
• pipeline.storage.s3.access_key_id_env
• pipeline.storage.s3.secret_access_key_env

External adapters used

Storage package contracts:

• ObjectStore (active remote object-store boundary): List, Download, Upload, Exists.
• Backend (legacy compatibility boundary): currently implemented with NoopBackend only.

Implementations:

• S3Backend: AWS SDK-backed ObjectStore implementation.
• FakeBackend: deterministic test ObjectStore and compatibility backend.
• NoopBackend: deterministic no-op compatibility backend for wiring/tests.

State and manifest behavior

• Storage implementations are stateless with respect to manifest/session lifecycle.
• Caller supplies fully-qualified bucket-relative keys.
• Storage layer does not infer campaign/session/run/root-prefix semantics.
• Caller controls publish ordering; storage layer executes individual operations in the order invoked.

Skip and resume behavior

• No storage-level skip/resume behavior.
• Skip/resume decisions are made by stage/app logic before storage calls occur.

Failure behavior

• NewObjectStoreFromConfig fails when no remote backend is configured or required S3 config is missing.
• S3Backend constructor fails when required bucket is missing or AWS client setup fails.
• App command orchestration loads configured filesystem secrets before calling the object-store factory.
• CRUD operations return contextual errors (including not-found behavior via Exists).
• Key normalization is applied before operations (\\ to /, leading slash trimmed).
• Remote session loading uses List to find the exact session.yml key and Download to materialize it to a local temp file.

Tests to inspect before changing

• internal/adapters/storage/factory_test.go
• internal/adapters/storage/s3_backend_test.go
• internal/adapters/storage/fake_test.go
• internal/adapters/storage/keys_test.go
• internal/adapters/storage/archive.go + consumers in stage tests (prepare, publish)

Architectural invariants

• Callers pass full bucket-relative keys.
• Storage backends must not prepend or infer narratio prefixes.
• Remote transport details remain isolated to storage adapter implementations.