distributor/docs/internal/storage.md

Storage

Purpose

internal/storage defines backend-rooted logical file access for core packages. Callers use slash-separated paths relative to a configured backend root.

Inputs and outputs

The storage interface supports byte reads, stream reads, byte writes, stream writes, exact metadata lookup, traversal, destination emptiness checks, guarded managed deletion, and bounded prefix deletion for explicit forced replacement.

Entries report a logical path, type, and size when available. Entry types are file, directory, symlink, and other.

Boundaries

Core packages should depend on internal/storage, not adapter packages. Adapter-specific path handling stays behind backend implementations.

The local adapter lives in internal/adapters/local. The SSH/SFTP adapter lives in internal/adapters/ssh. The S3-compatible adapter lives in internal/adapters/s3. Runtime backend construction is wired through the app-level backend factory and storage registry. The fake backend lives in internal/storage/fake for tests and is not registered for runtime use.

Paths

Logical file paths must be non-empty, relative, clean, slash-separated, and must not contain . or .. segments or backslashes. Prefix paths follow the same rules, except an empty prefix means the backend root.

Failure behavior

Storage errors use typed categories such as not found, already exists, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates rather than matching error strings.

Backends may wrap implementation-specific errors, but callers should receive storage errors where practical. Traversal can stop cleanly with ErrStopWalk.

Traversal helpers

Backends own their traversal mechanics. The local adapter owns filesystem walking, the SSH adapter owns SFTP directory walking, and the S3 adapter owns object listing and pagination.

internal/storage owns the shared callback emission rules used by backends:

• context cancellation is checked before callback emission;
• WalkOptions.Limit bounds the number of emitted entries;
• ErrStopWalk stops traversal without becoming a caller-visible error;
• callback errors are wrapped as storage walk errors.

storage.HasAny(ctx, backend, prefix) provides the shared destination-content check. It calls Walk with non-recursive, limit-one traversal and stops after the first emitted entry.

Deletion

DeleteManagedBundle may delete listed managed outputs plus .distributor.json.

DeletePrefix removes content at and below a logical prefix for explicit forced replacement. It must not delete above the requested prefix or above the configured backend root.

Local, SSH, S3, and fake backends

The local adapter maps logical paths to a configured filesystem root and keeps adapter-specific path handling behind the storage interface.

The SSH adapter maps logical paths to a configured remote SFTP root. It uses native SSH and SFTP libraries, supports SSH agent and key-file authentication, applies host-key policies, rejects unsafe logical paths, reports symlink entries from Lstat, and limits deletion to managed targets or explicit bounded prefixes.

The S3 adapter maps logical paths to object keys below a configured bucket and optional prefix. It uses the AWS SDK for Go v2, treats prefixes as object trees, requires exact objects for Stat, paginates traversal, applies conservative overwrite checks with HeadObject, infers basic content types, and limits deletion to managed target objects or explicit bounded object-key prefixes.

The fake backend is an in-memory implementation for package tests. It is not registered for runtime use.

Tests

Before changing storage behavior, inspect tests under:

• internal/storage
• internal/storage/fake
• internal/adapters/local
• internal/adapters/ssh
• internal/adapters/s3

Invariants

• Core packages depend on internal/storage, not concrete adapters.
• Logical paths are slash-separated and confined to the backend root.
• storage.List uses backend traversal and returns deterministic entries.
• Managed deletion is limited to recorded outputs plus .distributor.json.
• Prefix deletion is limited to the requested logical prefix.
• Runtime backend registration is owned by internal/app.