distributor/docs/internal/storage.md

Storage Internals

Audience: developers and LLM coding agents changing internal/storage, storage adapters, or storage-backed callers.

Purpose

internal/storage defines backend-rooted logical file access, path validation, typed storage errors, traversal helpers, backend registration, managed deletion targets, and test fake storage behavior.

Inputs And Outputs

Inputs are contexts, logical paths or prefixes, byte slices or readers, write options, walk options, delete options, and backend open configs. Outputs are file bytes, readers, Entry metadata, walk callbacks, boolean content checks, registered backends, and typed errors.

Boundaries

Core packages depend on internal/storage, not concrete adapters. Adapter protocol behavior belongs in internal/adapters/local, internal/adapters/ssh, and internal/adapters/s3; external SSH/SFTP and S3 notes live under docs/integrations/.

Runtime backend construction and registration are owned by internal/app. The fake backend is for tests only.

Config Fields Used

The storage package does not read config directly. App adapter wiring converts config fields into backend open config values.

Adapters Used

Local, SSH/SFTP, and S3-compatible adapters implement storage.Backend. internal/storage/fake implements the same interface for tests.

State And Manifest Behavior

Storage owns .distributor.json path helpers through StateFileName, StatePath, and ManagedBundleTargets. It does not parse source manifests or destination state.

Logical paths are slash-separated and relative to a backend root. Prefix validation allows an empty prefix to mean the backend root; file path validation requires a non-empty path.

Skip And Resume Behavior

Storage has no publication skip policy. It supplies HasAny for unmanaged-content checks, DeleteManagedBundle target construction for normal replacement cleanup, and DeletePrefix semantics for explicit forced replacement.

Failure Behavior

Storage errors use typed categories: not found, already exists, not empty, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates instead of matching strings. Traversal can stop cleanly with ErrStopWalk.

Tests To Inspect

• internal/storage/*_test.go
• internal/storage/fake/*_test.go
• internal/adapters/local/*_test.go
• internal/adapters/ssh/*_test.go
• internal/adapters/s3/*_test.go

Architectural Invariants

• Logical paths are clean relative slash-separated paths confined to the backend root.
• Core packages never import concrete adapters.
• storage.List returns deterministic sorted entries.
• Managed deletion targets are recorded outputs plus .distributor.json.
• Prefix deletion is bounded to the requested logical prefix.
• Runtime registration remains app-owned.