266 lines
11 KiB
Markdown
266 lines
11 KiB
Markdown
# Storage Interface Roadmap
|
|
|
|
This roadmap defines the planned `internal/storage` contract for the `distributor` MVP. The goal is to give bundle validation, destination state inspection, publish planning, and backend adapters one consistent IO boundary without leaking local filesystem, SSH/SFTP, or S3-specific behavior into core packages.
|
|
|
|
## Purpose and Invariants
|
|
|
|
The storage layer is responsible for safe, backend-rooted access to files, objects, prefixes, and destination bundle paths.
|
|
|
|
Core invariants:
|
|
|
|
- Backends are opened at configured roots.
|
|
- Core packages operate on backend-rooted logical paths, not absolute filesystem paths or raw object keys.
|
|
- Backend adapters translate native storage behavior into common storage entries and typed errors.
|
|
- Destructive operations remain narrow and managed.
|
|
- Staging or atomic write behavior belongs behind the storage interface where practical.
|
|
- The fake backend exists for tests only and must not be registered as a runtime backend.
|
|
|
|
## Logical Path Model
|
|
|
|
Storage paths are slash-separated logical paths relative to an already configured backend root.
|
|
|
|
File paths:
|
|
|
|
- must be non-empty;
|
|
- must be relative;
|
|
- must be clean;
|
|
- must not contain `.` or `..` segments;
|
|
- must not start with `/`;
|
|
- must not contain backslashes;
|
|
- must not resolve outside the backend root.
|
|
|
|
Prefix paths use the same slash-separated model. A prefix may be empty to represent the backend root for traversal and destination emptiness checks.
|
|
|
|
Prefix matching must preserve logical path boundaries. A prefix of `foo` matches `foo` and entries below `foo/`; it must not match a sibling path such as `foobar`. Backends that map logical paths to object keys must apply the same normalized boundary rule after combining configured backend prefixes with caller-provided logical prefixes.
|
|
|
|
Backends own conversion from logical paths to native paths or object keys. Core packages should not construct local filesystem paths, SFTP paths, or S3 object keys directly.
|
|
|
|
## Core Interface Shape
|
|
|
|
The MVP should use a hybrid byte and stream interface:
|
|
|
|
```go
|
|
type Backend interface {
|
|
ReadFile(ctx context.Context, path string) ([]byte, error)
|
|
OpenReader(ctx context.Context, path string) (io.ReadCloser, error)
|
|
WriteFile(ctx context.Context, path string, data []byte, opts WriteOptions) (Entry, error)
|
|
WriteFrom(ctx context.Context, path string, r io.Reader, opts WriteOptions) (Entry, error)
|
|
Stat(ctx context.Context, path string) (Entry, error)
|
|
Walk(ctx context.Context, prefix string, opts WalkOptions, fn WalkFunc) error
|
|
HasAny(ctx context.Context, prefix string) (bool, error)
|
|
DeleteManagedBundle(ctx context.Context, bundlePath string, managedOutputPaths []string, opts DeleteOptions) error
|
|
}
|
|
```
|
|
|
|
Byte helpers are expected to cover manifests, destination state, small source artifacts, and generated outputs. Stream methods are included from the start for backend flexibility and larger future artifacts.
|
|
|
|
Write operations should create required parent directories or prefixes as needed.
|
|
|
|
Concrete option and callback types should use this shape:
|
|
|
|
```go
|
|
type WalkOptions struct {
|
|
Recursive bool
|
|
Limit int
|
|
}
|
|
|
|
type WalkFunc func(Entry) error
|
|
|
|
var ErrStopWalk = errors.New("stop walk")
|
|
|
|
type WriteOptions struct {
|
|
ContentType string
|
|
Overwrite bool
|
|
PreferAtomic bool
|
|
Size int64
|
|
SizeKnown bool
|
|
}
|
|
|
|
type DeleteOptions struct {
|
|
IgnoreMissing bool
|
|
PruneEmptyDirs bool
|
|
}
|
|
```
|
|
|
|
`WalkOptions.Limit == 0` means no explicit limit. `SizeKnown` applies primarily to `WriteFrom`; byte writes can infer size from the provided data.
|
|
|
|
## Entries and Metadata
|
|
|
|
Storage metadata should be represented by an `Entry` model with at least:
|
|
|
|
- backend-relative logical path;
|
|
- entry type;
|
|
- size, where available.
|
|
|
|
Entry types:
|
|
|
|
- `file`: filesystem file or object-storage object;
|
|
- `directory`: filesystem directory or logical prefix;
|
|
- `symlink`: local filesystem symlink;
|
|
- `other`: unknown or unsupported native entry type.
|
|
|
|
`Stat` returns metadata for one exact logical path. It may report a real filesystem directory, symlink, file, or exact object. It must not synthesize S3-like directory metadata solely because objects exist below a prefix; callers that need prefix existence or destination emptiness must use `HasAny` or `Walk`.
|
|
|
|
`Walk` traverses entries below a prefix and calls a callback for each entry. `WalkOptions` should include:
|
|
|
|
- whether traversal is recursive;
|
|
- an optional entry limit for callers that only need to know whether content exists.
|
|
|
|
If a callback returns `ErrStopWalk`, traversal stops successfully and `Walk` returns nil. Any other callback error stops traversal and is returned with storage context where practical. If `WalkOptions.Limit` is greater than zero, reaching the limit stops traversal successfully.
|
|
|
|
Backends may stream or paginate traversal internally. S3-compatible adapters should not need to load a whole prefix into memory to satisfy traversal.
|
|
|
|
Raw traversal is not required to be lexically sorted. A helper that materializes walk results for bundle discovery, tests, or CLI output should sort entries lexically by logical path before returning them.
|
|
|
|
`HasAny` reports whether at least one entry exists below a prefix. It should stop as soon as content is found.
|
|
|
|
Source validation must reject symlink entries reported by local `Stat` or `Walk`.
|
|
|
|
## Read Behavior
|
|
|
|
`ReadFile` reads the whole object into memory and is appropriate for MVP manifest, state, and ordinary artifact handling.
|
|
|
|
`OpenReader` returns a stream for callers that need to copy or hash content without requiring a second storage-specific API. Callers must close the returned reader.
|
|
|
|
Both read methods must:
|
|
|
|
- validate logical paths before backend access;
|
|
- reject directories, prefixes, symlinks, and unsupported entries;
|
|
- return typed not-found and invalid-path errors where applicable.
|
|
|
|
## Write Behavior
|
|
|
|
`WriteOptions` should include:
|
|
|
|
- content type, when the destination backend can use it;
|
|
- overwrite permission;
|
|
- atomic or staged write preference;
|
|
- optional known size for stream writes.
|
|
|
|
Backends own staging and atomic behavior where practical:
|
|
|
|
- Local backend writes to a temporary file in the destination directory and renames or promotes into place.
|
|
- SSH/SFTP backend should use a temporary remote file and rename where available.
|
|
- S3-compatible backend treats a successful object PUT as publish-on-success and applies content type metadata.
|
|
|
|
Remote adapters may buffer or spool `WriteFrom` input when needed to satisfy backend requirements such as content length, multipart upload, or retry behavior. Callers that know the stream size should set `SizeKnown` and `Size`.
|
|
|
|
If overwrite is false and the target exists, writes should fail with an already-exists error.
|
|
|
|
`WriteFile` and `WriteFrom` should return the written `Entry`, including final path and size where available.
|
|
|
|
`DeleteOptions` should include:
|
|
|
|
- whether missing managed output paths are ignored;
|
|
- whether empty parent directories may be pruned for filesystem-like backends.
|
|
|
|
## Managed Deletion
|
|
|
|
The storage interface should expose a guarded managed deletion operation rather than raw recursive delete.
|
|
|
|
`DeleteManagedBundle(ctx, bundlePath, managedOutputPaths, opts)` may delete only:
|
|
|
|
- files or objects listed in valid `.distributor.json.outputs`;
|
|
- `.distributor.json` at the destination bundle path;
|
|
- empty directories created by those files, for filesystem-like backends.
|
|
|
|
`managedOutputPaths` are relative to the destination bundle path. The backend validates each path and resolves it under `bundlePath`.
|
|
|
|
If `bundlePath == ""`, deletion may remove explicit managed files at the destination root, but must never delete the root itself.
|
|
|
|
Prefix or recursive deletion is out of MVP scope. A future force-overwrite stage may add broader behavior, but it must remain explicit and separately documented.
|
|
|
|
## Destination Emptiness
|
|
|
|
Destination emptiness should use `HasAny(prefix)` and typed not-found behavior. Callers that only need emptiness must not materialize a full recursive traversal.
|
|
|
|
Rules:
|
|
|
|
- A local destination bundle path is empty when the directory does not exist or exists with no entries.
|
|
- An S3-compatible prefix is empty when no objects exist below that exact destination bundle prefix.
|
|
- Entries outside the exact destination bundle path or prefix do not affect emptiness.
|
|
|
|
## Error Model
|
|
|
|
Storage should expose typed error categories with wrapping context. Callers should use helper predicates rather than string matching.
|
|
|
|
Required categories:
|
|
|
|
- not found;
|
|
- already exists;
|
|
- not empty;
|
|
- invalid path;
|
|
- conflict;
|
|
- permission;
|
|
- temporary;
|
|
- unsupported;
|
|
- unknown.
|
|
|
|
Adapters should translate backend-native errors into these categories while preserving useful operation, backend, path, and cause context.
|
|
|
|
## Adapter Expectations
|
|
|
|
### Local
|
|
|
|
The local backend should:
|
|
|
|
- constrain all operations beneath the configured root;
|
|
- reject traversal and absolute logical paths;
|
|
- report symlinks through metadata;
|
|
- reject symlink reads for source artifacts;
|
|
- use staged writes where practical;
|
|
- perform managed deletion only for explicit managed files and `.distributor.json`;
|
|
- clean up empty directories created by managed outputs where safe.
|
|
|
|
### Fake
|
|
|
|
The fake backend should:
|
|
|
|
- be in-memory and deterministic;
|
|
- implement the same logical path validation rules;
|
|
- support `Stat`, `Walk`, `HasAny`, byte reads and writes, stream reads and writes, managed deletion, and destination emptiness helper behavior;
|
|
- support configured symlink entries for validation tests;
|
|
- be used only by tests.
|
|
|
|
### SSH/SFTP
|
|
|
|
The SSH/SFTP backend should:
|
|
|
|
- use native SFTP operations;
|
|
- enforce the same logical path rules as local storage;
|
|
- use temporary file plus rename for staged writes where available;
|
|
- translate remote errors into storage error categories;
|
|
- avoid exposing SSH or SFTP dependency types through `internal/storage`.
|
|
|
|
### S3-Compatible
|
|
|
|
The S3-compatible backend should:
|
|
|
|
- treat prefixes as object trees, not real directories;
|
|
- normalize configured prefix plus logical path into object keys;
|
|
- use object PUT as publish-on-success;
|
|
- set content type from `WriteOptions`;
|
|
- implement traversal and emptiness by exact prefix;
|
|
- use backend pagination for traversal where available;
|
|
- allow `HasAny` to stop after the first matching object;
|
|
- constrain managed deletion to listed output objects and `.distributor.json`.
|
|
|
|
## Tests and Fixtures
|
|
|
|
Storage implementation stages should test:
|
|
|
|
- path validation rejects absolute paths, traversal, empty file paths, backslashes, and dot segments;
|
|
- `Walk` visits backend-rooted logical paths under a prefix and supports recursive traversal;
|
|
- a materializing helper sorts walk results lexically for deterministic tests and CLI output;
|
|
- `HasAny` returns quickly for non-empty prefixes without requiring full traversal;
|
|
- `ReadFile` and `OpenReader` return equivalent bytes;
|
|
- `WriteFile` and `WriteFrom` honor overwrite and content-type options;
|
|
- local staged writes do not leave final files on failure where testable;
|
|
- managed deletion deletes only state-listed files and `.distributor.json`;
|
|
- managed deletion never deletes destination root or unlisted files;
|
|
- destination emptiness helper handles missing, empty, and non-empty local paths;
|
|
- S3-compatible traversal can use pagination without loading a whole prefix into memory;
|
|
- symlink entries are reported and rejected by source validation;
|
|
- typed errors are usable through helper predicates;
|
|
- fake backend behavior matches local backend semantics relevant to core tests.
|