distributor/docs/roadmap/storage.md

Storage Interface Roadmap

This roadmap defines the planned internal/storage contract for the distributor MVP. The goal is to give bundle validation, destination state inspection, publish planning, and backend adapters one consistent IO boundary without leaking local filesystem, SSH/SFTP, or S3-specific behavior into core packages.

Purpose and Invariants

The storage layer is responsible for safe, backend-rooted access to files, objects, prefixes, and destination bundle paths.

Core invariants:

• Backends are opened at configured roots.
• Core packages operate on backend-rooted logical paths, not absolute filesystem paths or raw object keys.
• Backend adapters translate native storage behavior into common storage entries and typed errors.
• Destructive operations remain narrow and managed.
• Staging or atomic write behavior belongs behind the storage interface where practical.
• The fake backend exists for tests only and must not be registered as a runtime backend.

Logical Path Model

Storage paths are slash-separated logical paths relative to an already configured backend root.

File paths:

• must be non-empty;
• must be relative;
• must be clean;
• must not contain . or .. segments;
• must not start with /;
• must not contain backslashes;
• must not resolve outside the backend root.

Prefix paths use the same slash-separated model. A prefix may be empty to represent the backend root for traversal and destination emptiness checks.

Prefix matching must preserve logical path boundaries. A prefix of foo matches foo and entries below foo/; it must not match a sibling path such as foobar. Backends that map logical paths to object keys must apply the same normalized boundary rule after combining configured backend prefixes with caller-provided logical prefixes.

Backends own conversion from logical paths to native paths or object keys. Core packages should not construct local filesystem paths, SFTP paths, or S3 object keys directly.

Core Interface Shape

The MVP should use a hybrid byte and stream interface:

type Backend interface {
    ReadFile(ctx context.Context, path string) ([]byte, error)
    OpenReader(ctx context.Context, path string) (io.ReadCloser, error)
    WriteFile(ctx context.Context, path string, data []byte, opts WriteOptions) (Entry, error)
    WriteFrom(ctx context.Context, path string, r io.Reader, opts WriteOptions) (Entry, error)
    Stat(ctx context.Context, path string) (Entry, error)
    Walk(ctx context.Context, prefix string, opts WalkOptions, fn WalkFunc) error
    HasAny(ctx context.Context, prefix string) (bool, error)
    DeleteManagedBundle(ctx context.Context, bundlePath string, managedOutputPaths []string, opts DeleteOptions) error
}

Byte helpers are expected to cover manifests, destination state, small source artifacts, and generated outputs. Stream methods are included from the start for backend flexibility and larger future artifacts.

Write operations should create required parent directories or prefixes as needed.

Concrete option and callback types should use this shape:

type WalkOptions struct {
    Recursive bool
    Limit     int
}

type WalkFunc func(Entry) error

var ErrStopWalk = errors.New("stop walk")

type WriteOptions struct {
    ContentType  string
    Overwrite    bool
    PreferAtomic bool
    Size         int64
    SizeKnown    bool
}

type DeleteOptions struct {
    IgnoreMissing  bool
    PruneEmptyDirs bool
}

WalkOptions.Limit == 0 means no explicit limit. SizeKnown applies primarily to WriteFrom; byte writes can infer size from the provided data.

Entries and Metadata

Storage metadata should be represented by an Entry model with at least:

• backend-relative logical path;
• entry type;
• size, where available.

Entry types:

• file: filesystem file or object-storage object;
• directory: filesystem directory or logical prefix;
• symlink: local filesystem symlink;
• other: unknown or unsupported native entry type.

Stat returns metadata for one exact logical path. It may report a real filesystem directory, symlink, file, or exact object. It must not synthesize S3-like directory metadata solely because objects exist below a prefix; callers that need prefix existence or destination emptiness must use HasAny or Walk.

Walk traverses entries below a prefix and calls a callback for each entry. WalkOptions should include:

• whether traversal is recursive;
• an optional entry limit for callers that only need to know whether content exists.

If a callback returns ErrStopWalk, traversal stops successfully and Walk returns nil. Any other callback error stops traversal and is returned with storage context where practical. If WalkOptions.Limit is greater than zero, reaching the limit stops traversal successfully.

Backends may stream or paginate traversal internally. S3-compatible adapters should not need to load a whole prefix into memory to satisfy traversal.

Raw traversal is not required to be lexically sorted. A helper that materializes walk results for bundle discovery, tests, or CLI output should sort entries lexically by logical path before returning them.

HasAny reports whether at least one entry exists below a prefix. It should stop as soon as content is found.

Source validation must reject symlink entries reported by local Stat or Walk.

Read Behavior

ReadFile reads the whole object into memory and is appropriate for MVP manifest, state, and ordinary artifact handling.

OpenReader returns a stream for callers that need to copy or hash content without requiring a second storage-specific API. Callers must close the returned reader.

Both read methods must:

• validate logical paths before backend access;
• reject directories, prefixes, symlinks, and unsupported entries;
• return typed not-found and invalid-path errors where applicable.

Write Behavior

WriteOptions should include:

• content type, when the destination backend can use it;
• overwrite permission;
• atomic or staged write preference;
• optional known size for stream writes.

Backends own staging and atomic behavior where practical:

• Local backend writes to a temporary file in the destination directory and renames or promotes into place.
• SSH/SFTP backend should use a temporary remote file and rename where available.
• S3-compatible backend treats a successful object PUT as publish-on-success and applies content type metadata.

Remote adapters may buffer or spool WriteFrom input when needed to satisfy backend requirements such as content length, multipart upload, or retry behavior. Callers that know the stream size should set SizeKnown and Size.

If overwrite is false and the target exists, writes should fail with an already-exists error.

WriteFile and WriteFrom should return the written Entry, including final path and size where available.

DeleteOptions should include:

• whether missing managed output paths are ignored;
• whether empty parent directories may be pruned for filesystem-like backends.

Managed Deletion

The storage interface should expose a guarded managed deletion operation rather than raw recursive delete.

DeleteManagedBundle(ctx, bundlePath, managedOutputPaths, opts) may delete only:

• files or objects listed in valid .distributor.json.outputs;
• .distributor.json at the destination bundle path;
• empty directories created by those files, for filesystem-like backends.

managedOutputPaths are relative to the destination bundle path. The backend validates each path and resolves it under bundlePath.

If bundlePath == "", deletion may remove explicit managed files at the destination root, but must never delete the root itself.

Prefix or recursive deletion is out of MVP scope. A future force-overwrite stage may add broader behavior, but it must remain explicit and separately documented.

Destination Emptiness

Destination emptiness should use HasAny(prefix) and typed not-found behavior. Callers that only need emptiness must not materialize a full recursive traversal.

Rules:

• A local destination bundle path is empty when the directory does not exist or exists with no entries.
• An S3-compatible prefix is empty when no objects exist below that exact destination bundle prefix.
• Entries outside the exact destination bundle path or prefix do not affect emptiness.

Error Model

Storage should expose typed error categories with wrapping context. Callers should use helper predicates rather than string matching.

Required categories:

• not found;
• already exists;
• not empty;
• invalid path;
• conflict;
• permission;
• temporary;
• unsupported;
• unknown.

Adapters should translate backend-native errors into these categories while preserving useful operation, backend, path, and cause context.

Adapter Expectations

Local

The local backend should:

• constrain all operations beneath the configured root;
• reject traversal and absolute logical paths;
• report symlinks through metadata;
• reject symlink reads for source artifacts;
• use staged writes where practical;
• perform managed deletion only for explicit managed files and .distributor.json;
• clean up empty directories created by managed outputs where safe.

Fake

The fake backend should:

• be in-memory and deterministic;
• implement the same logical path validation rules;
• support Stat, Walk, HasAny, byte reads and writes, stream reads and writes, managed deletion, and destination emptiness helper behavior;
• support configured symlink entries for validation tests;
• be used only by tests.

SSH/SFTP

The SSH/SFTP backend should:

• use native SFTP operations;
• enforce the same logical path rules as local storage;
• use temporary file plus rename for staged writes where available;
• translate remote errors into storage error categories;
• avoid exposing SSH or SFTP dependency types through internal/storage.

S3-Compatible

The S3-compatible backend should:

• treat prefixes as object trees, not real directories;
• normalize configured prefix plus logical path into object keys;
• use object PUT as publish-on-success;
• set content type from WriteOptions;
• implement traversal and emptiness by exact prefix;
• use backend pagination for traversal where available;
• allow HasAny to stop after the first matching object;
• constrain managed deletion to listed output objects and .distributor.json.

Tests and Fixtures

Storage implementation stages should test:

• path validation rejects absolute paths, traversal, empty file paths, backslashes, and dot segments;
• Walk visits backend-rooted logical paths under a prefix and supports recursive traversal;
• a materializing helper sorts walk results lexically for deterministic tests and CLI output;
• HasAny returns quickly for non-empty prefixes without requiring full traversal;
• ReadFile and OpenReader return equivalent bytes;
• WriteFile and WriteFrom honor overwrite and content-type options;
• local staged writes do not leave final files on failure where testable;
• managed deletion deletes only state-listed files and .distributor.json;
• managed deletion never deletes destination root or unlisted files;
• destination emptiness helper handles missing, empty, and non-empty local paths;
• S3-compatible traversal can use pagination without loading a whole prefix into memory;
• symlink entries are reported and rejected by source validation;
• typed errors are usable through helper predicates;
• fake backend behavior matches local backend semantics relevant to core tests.