distributor/docs/internal/storage.md

Storage Internals

Audience: developers and LLM coding agents changing internal/storage, storage adapters, or storage-backed callers.

Purpose

internal/storage defines backend-rooted logical file access, path validation, typed storage errors, traversal helpers, backend registration, managed deletion targets, and test fake storage behavior.

Inputs And Outputs

Inputs are contexts, logical paths or prefixes, byte slices or readers, write options, walk options, delete options, and backend open configs. Outputs are file bytes, readers, Entry metadata, walk callbacks, boolean content checks, registered backends, and typed errors.

Boundaries

Core packages depend on internal/storage, not concrete adapters. Adapter protocol behavior belongs in internal/adapters/local, internal/adapters/ssh, and internal/adapters/s3; external SSH/SFTP and S3 notes live under docs/integrations/.

Runtime backend construction and registration are owned by internal/app. The fake backend is for tests only.

Config Fields Used

The storage package does not read config directly. App adapter wiring converts config fields into backend open config values.

Adapters Used

Local, SSH/SFTP, and S3-compatible adapters implement storage.Backend. internal/storage/fake implements the same interface for tests.

State And Manifest Behavior

Storage owns .distributor.json path helpers through StateFileName, StatePath, ManagedOutputTargets, and ManagedBundleTargets. It does not parse source manifests or destination state.

Logical paths are slash-separated and relative to a backend root. Prefix validation allows an empty prefix to mean the backend root; file path validation requires a non-empty path.

Reconcile-state callers use Stat to check whether managed output paths still exist and bounded recursive Walk to report unmanaged entries under a selected destination root. Prune execution callers use DeleteManagedOutputs for selected managed output records so .distributor.json is not part of the deletion target set. Storage does not decide whether entries are managed; callers compare entries against destination state.

Skip And Resume Behavior

Storage has no publication skip policy. It supplies HasAny for unmanaged-content checks, Stat and Walk for state repair inspection, DeleteManagedOutputs target construction for managed output cleanup, DeleteManagedBundle target construction for normal replacement cleanup, and DeletePrefix semantics for explicit forced replacement.

Failure Behavior

Storage errors use typed categories: not found, already exists, not empty, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates instead of matching strings. Traversal can stop cleanly with ErrStopWalk.

Tests To Inspect

• internal/storage/*_test.go
• internal/storage/fake/*_test.go
• internal/app/reconcile_state_test.go
• internal/adapters/local/*_test.go
• internal/adapters/ssh/*_test.go
• internal/adapters/s3/*_test.go

Architectural Invariants

• Logical paths are clean relative slash-separated paths confined to the backend root.
• Core packages never import concrete adapters.
• storage.List returns deterministic sorted entries.
• Managed output deletion targets are recorded outputs only; managed bundle deletion targets are recorded outputs plus .distributor.json.
• Prefix deletion is bounded to the requested logical prefix.
• Runtime registration remains app-owned.