distributor/docs/internal/storage.md

Storage

Purpose

internal/storage defines backend-rooted logical file access for core packages. Callers use slash-separated paths relative to a configured backend root.

Inputs and outputs

The storage interface supports byte reads, stream reads, byte writes, stream writes, exact metadata lookup, traversal, destination emptiness checks, and guarded managed deletion.

Entries report a logical path, type, and size when available. Entry types are file, directory, symlink, and other.

Boundaries

Core packages should depend on internal/storage, not adapter packages. Adapter-specific path handling stays behind backend implementations.

The local adapter lives in internal/adapters/local. The SSH/SFTP adapter lives in internal/adapters/ssh. The S3-compatible adapter lives in internal/adapters/s3. Runtime backend construction is wired through the app-level backend factory and storage registry. The fake backend lives in internal/storage/fake for tests and is not registered for runtime use.

Paths

Logical file paths must be non-empty, relative, clean, slash-separated, and must not contain . or .. segments or backslashes. Prefix paths follow the same rules, except an empty prefix means the backend root.

Failure behavior

Storage errors use typed categories such as not found, already exists, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates rather than matching error strings.

Backends may wrap implementation-specific errors, but callers should receive storage errors where practical. Traversal can stop cleanly with ErrStopWalk.

Deletion

Backends expose guarded managed deletion only. DeleteManagedBundle may delete listed managed outputs plus .distributor.json; it does not provide broad recursive deletion.

Local, SSH, S3, and fake backends

The local adapter maps logical paths to a configured filesystem root and keeps adapter-specific path handling behind the storage interface.

The SSH adapter maps logical paths to a configured remote SFTP root. It uses native SSH and SFTP libraries, supports SSH agent and key-file authentication, applies host-key policies, rejects unsafe logical paths, reports symlink entries from Lstat, and limits deletion to managed targets.

The S3 adapter maps logical paths to object keys below a configured bucket and optional prefix. It uses the AWS SDK for Go v2, treats prefixes as object trees, requires exact objects for Stat, paginates traversal, applies conservative overwrite checks with HeadObject, infers basic content types, and limits deletion to managed target objects.

The fake backend is an in-memory implementation for package tests. It is not registered for runtime use.

Tests

Before changing storage behavior, inspect tests under:

• internal/storage
• internal/storage/fake
• internal/adapters/local
• internal/adapters/ssh
• internal/adapters/s3

Invariants

• Core packages depend on internal/storage, not concrete adapters.
• Logical paths are slash-separated and confined to the backend root.
• storage.List uses backend traversal and returns deterministic entries.
• Managed deletion is limited to recorded outputs plus .distributor.json.
• Runtime backend registration is owned by internal/app.