Update and refine the implementation roadmap
This commit is contained in:
230
docs/roadmap/storage.md
Normal file
230
docs/roadmap/storage.md
Normal file
@@ -0,0 +1,230 @@
|
||||
# Storage Interface Roadmap
|
||||
|
||||
This roadmap defines the planned `internal/storage` contract for the `distributor` MVP. The goal is to give bundle validation, destination state inspection, publish planning, and backend adapters one consistent IO boundary without leaking local filesystem, SSH/SFTP, or S3-specific behavior into core packages.
|
||||
|
||||
## Purpose and Invariants
|
||||
|
||||
The storage layer is responsible for safe, backend-rooted access to files, objects, prefixes, and destination bundle paths.
|
||||
|
||||
Core invariants:
|
||||
|
||||
- Backends are opened at configured roots.
|
||||
- Core packages operate on backend-rooted logical paths, not absolute filesystem paths or raw object keys.
|
||||
- Backend adapters translate native storage behavior into common storage entries and typed errors.
|
||||
- Destructive operations remain narrow and managed.
|
||||
- Staging or atomic write behavior belongs behind the storage interface where practical.
|
||||
- The fake backend exists for tests only and must not be registered as a runtime backend.
|
||||
|
||||
## Logical Path Model
|
||||
|
||||
Storage paths are slash-separated logical paths relative to an already configured backend root.
|
||||
|
||||
File paths:
|
||||
|
||||
- must be non-empty;
|
||||
- must be relative;
|
||||
- must be clean;
|
||||
- must not contain `.` or `..` segments;
|
||||
- must not start with `/`;
|
||||
- must not contain backslashes;
|
||||
- must not resolve outside the backend root.
|
||||
|
||||
Prefix paths use the same slash-separated model. A prefix may be empty to represent the backend root for listing and destination emptiness checks.
|
||||
|
||||
Backends own conversion from logical paths to native paths or object keys. Core packages should not construct local filesystem paths, SFTP paths, or S3 object keys directly.
|
||||
|
||||
## Core Interface Shape
|
||||
|
||||
The MVP should use a hybrid byte and stream interface:
|
||||
|
||||
```go
|
||||
type Backend interface {
|
||||
ReadFile(ctx context.Context, path string) ([]byte, error)
|
||||
OpenReader(ctx context.Context, path string) (io.ReadCloser, error)
|
||||
WriteFile(ctx context.Context, path string, data []byte, opts WriteOptions) (Entry, error)
|
||||
WriteFrom(ctx context.Context, path string, r io.Reader, opts WriteOptions) (Entry, error)
|
||||
Stat(ctx context.Context, path string) (Entry, error)
|
||||
Walk(ctx context.Context, prefix string, opts WalkOptions, fn WalkFunc) error
|
||||
HasAny(ctx context.Context, prefix string) (bool, error)
|
||||
DeleteManagedBundle(ctx context.Context, bundlePath string, managedOutputPaths []string, opts DeleteOptions) error
|
||||
}
|
||||
```
|
||||
|
||||
Byte helpers are expected to cover manifests, destination state, small source artifacts, and generated outputs. Stream methods are included from the start for backend flexibility and larger future artifacts.
|
||||
|
||||
Write operations should create required parent directories or prefixes as needed.
|
||||
|
||||
## Entries and Metadata
|
||||
|
||||
Storage metadata should be represented by an `Entry` model with at least:
|
||||
|
||||
- backend-relative logical path;
|
||||
- entry type;
|
||||
- size, where available.
|
||||
|
||||
Entry types:
|
||||
|
||||
- `file`: filesystem file or object-storage object;
|
||||
- `directory`: filesystem directory or logical prefix;
|
||||
- `symlink`: local filesystem symlink;
|
||||
- `other`: unknown or unsupported native entry type.
|
||||
|
||||
`Stat` returns metadata for one logical path.
|
||||
|
||||
`Walk` traverses entries below a prefix and calls a callback for each entry. `WalkOptions` should include:
|
||||
|
||||
- whether traversal is recursive;
|
||||
- an optional entry limit for callers that only need to know whether content exists.
|
||||
|
||||
Backends may stream or paginate traversal internally. S3-compatible adapters should not need to load a whole prefix into memory to satisfy traversal.
|
||||
|
||||
Raw traversal is not required to be lexically sorted. A helper that materializes walk results for bundle discovery, tests, or CLI output should sort entries lexically by logical path before returning them.
|
||||
|
||||
`HasAny` reports whether at least one entry exists below a prefix. It should stop as soon as content is found.
|
||||
|
||||
Source validation must reject symlink entries reported by local `Stat` or `Walk`.
|
||||
|
||||
## Read Behavior
|
||||
|
||||
`ReadFile` reads the whole object into memory and is appropriate for MVP manifest, state, and ordinary artifact handling.
|
||||
|
||||
`OpenReader` returns a stream for callers that need to copy or hash content without requiring a second storage-specific API. Callers must close the returned reader.
|
||||
|
||||
Both read methods must:
|
||||
|
||||
- validate logical paths before backend access;
|
||||
- reject directories, prefixes, symlinks, and unsupported entries;
|
||||
- return typed not-found and invalid-path errors where applicable.
|
||||
|
||||
## Write Behavior
|
||||
|
||||
`WriteOptions` should include:
|
||||
|
||||
- content type, when the destination backend can use it;
|
||||
- overwrite permission;
|
||||
- atomic or staged write preference.
|
||||
|
||||
Backends own staging and atomic behavior where practical:
|
||||
|
||||
- Local backend writes to a temporary file in the destination directory and renames or promotes into place.
|
||||
- SSH/SFTP backend should use a temporary remote file and rename where available.
|
||||
- S3-compatible backend treats a successful object PUT as publish-on-success and applies content type metadata.
|
||||
|
||||
If overwrite is false and the target exists, writes should fail with an already-exists error.
|
||||
|
||||
`WriteFile` and `WriteFrom` should return the written `Entry`, including final path and size where available.
|
||||
|
||||
`DeleteOptions` should include:
|
||||
|
||||
- whether missing managed output paths are ignored;
|
||||
- whether empty parent directories may be pruned for filesystem-like backends.
|
||||
|
||||
## Managed Deletion
|
||||
|
||||
The storage interface should expose a guarded managed deletion operation rather than raw recursive delete.
|
||||
|
||||
`DeleteManagedBundle(ctx, bundlePath, managedOutputPaths, opts)` may delete only:
|
||||
|
||||
- files or objects listed in valid `.distributor.json.outputs`;
|
||||
- `.distributor.json` at the destination bundle path;
|
||||
- empty directories created by those files, for filesystem-like backends.
|
||||
|
||||
`managedOutputPaths` are relative to the destination bundle path. The backend validates each path and resolves it under `bundlePath`.
|
||||
|
||||
If `bundlePath == ""`, deletion may remove explicit managed files at the destination root, but must never delete the root itself.
|
||||
|
||||
Prefix or recursive deletion is out of MVP scope. A future force-overwrite stage may add broader behavior, but it must remain explicit and separately documented.
|
||||
|
||||
## Destination Emptiness
|
||||
|
||||
Destination emptiness should use `HasAny(prefix)` and typed not-found behavior. Callers that only need emptiness must not materialize a full recursive traversal.
|
||||
|
||||
Rules:
|
||||
|
||||
- A local destination bundle path is empty when the directory does not exist or exists with no entries.
|
||||
- An S3-compatible prefix is empty when no objects exist below that exact destination bundle prefix.
|
||||
- Entries outside the exact destination bundle path or prefix do not affect emptiness.
|
||||
|
||||
## Error Model
|
||||
|
||||
Storage should expose typed error categories with wrapping context. Callers should use helper predicates rather than string matching.
|
||||
|
||||
Required categories:
|
||||
|
||||
- not found;
|
||||
- already exists;
|
||||
- not empty;
|
||||
- invalid path;
|
||||
- conflict;
|
||||
- permission;
|
||||
- temporary;
|
||||
- unsupported;
|
||||
- unknown.
|
||||
|
||||
Adapters should translate backend-native errors into these categories while preserving useful operation, backend, path, and cause context.
|
||||
|
||||
## Adapter Expectations
|
||||
|
||||
### Local
|
||||
|
||||
The local backend should:
|
||||
|
||||
- constrain all operations beneath the configured root;
|
||||
- reject traversal and absolute logical paths;
|
||||
- report symlinks through metadata;
|
||||
- reject symlink reads for source artifacts;
|
||||
- use staged writes where practical;
|
||||
- perform managed deletion only for explicit managed files and `.distributor.json`;
|
||||
- clean up empty directories created by managed outputs where safe.
|
||||
|
||||
### Fake
|
||||
|
||||
The fake backend should:
|
||||
|
||||
- be in-memory and deterministic;
|
||||
- implement the same logical path validation rules;
|
||||
- support `Stat`, `Walk`, `HasAny`, byte reads and writes, stream reads and writes, managed deletion, and destination emptiness helper behavior;
|
||||
- support configured symlink entries for validation tests;
|
||||
- be used only by tests.
|
||||
|
||||
### SSH/SFTP
|
||||
|
||||
The SSH/SFTP backend should:
|
||||
|
||||
- use native SFTP operations;
|
||||
- enforce the same logical path rules as local storage;
|
||||
- use temporary file plus rename for staged writes where available;
|
||||
- translate remote errors into storage error categories;
|
||||
- avoid exposing SSH or SFTP dependency types through `internal/storage`.
|
||||
|
||||
### S3-Compatible
|
||||
|
||||
The S3-compatible backend should:
|
||||
|
||||
- treat prefixes as object trees, not real directories;
|
||||
- normalize configured prefix plus logical path into object keys;
|
||||
- use object PUT as publish-on-success;
|
||||
- set content type from `WriteOptions`;
|
||||
- implement traversal and emptiness by exact prefix;
|
||||
- use backend pagination for traversal where available;
|
||||
- allow `HasAny` to stop after the first matching object;
|
||||
- constrain managed deletion to listed output objects and `.distributor.json`.
|
||||
|
||||
## Tests and Fixtures
|
||||
|
||||
Storage implementation stages should test:
|
||||
|
||||
- path validation rejects absolute paths, traversal, empty file paths, backslashes, and dot segments;
|
||||
- `Walk` visits backend-rooted logical paths under a prefix and supports recursive traversal;
|
||||
- a materializing helper sorts walk results lexically for deterministic tests and CLI output;
|
||||
- `HasAny` returns quickly for non-empty prefixes without requiring full traversal;
|
||||
- `ReadFile` and `OpenReader` return equivalent bytes;
|
||||
- `WriteFile` and `WriteFrom` honor overwrite and content-type options;
|
||||
- local staged writes do not leave final files on failure where testable;
|
||||
- managed deletion deletes only state-listed files and `.distributor.json`;
|
||||
- managed deletion never deletes destination root or unlisted files;
|
||||
- destination emptiness helper handles missing, empty, and non-empty local paths;
|
||||
- S3-compatible traversal can use pagination without loading a whole prefix into memory;
|
||||
- symlink entries are reported and rejected by source validation;
|
||||
- typed errors are usable through helper predicates;
|
||||
- fake backend behavior matches local backend semantics relevant to core tests.
|
||||
Reference in New Issue
Block a user