76 lines
4.2 KiB
Markdown
76 lines
4.2 KiB
Markdown
# Storage
|
|
|
|
## Purpose
|
|
|
|
`internal/storage` defines backend-rooted logical file access for core packages. Callers use slash-separated paths relative to a configured backend root.
|
|
|
|
## Inputs and outputs
|
|
|
|
The storage interface supports byte reads, stream reads, byte writes, stream writes, exact metadata lookup, traversal, destination emptiness checks, guarded managed deletion, and bounded prefix deletion for explicit forced replacement.
|
|
|
|
Entries report a logical path, type, and size when available. Entry types are `file`, `directory`, `symlink`, and `other`.
|
|
|
|
## Boundaries
|
|
|
|
Core packages should depend on `internal/storage`, not adapter packages. Adapter-specific path handling stays behind backend implementations.
|
|
|
|
The local adapter lives in `internal/adapters/local`. The SSH/SFTP adapter lives in `internal/adapters/ssh`. The S3-compatible adapter lives in `internal/adapters/s3`. Runtime backend construction is wired through the app-level backend factory and storage registry. The fake backend lives in `internal/storage/fake` for tests and is not registered for runtime use.
|
|
|
|
## Paths
|
|
|
|
Logical file paths must be non-empty, relative, clean, slash-separated, and must not contain `.` or `..` segments or backslashes. Prefix paths follow the same rules, except an empty prefix means the backend root.
|
|
|
|
## Failure behavior
|
|
|
|
Storage errors use typed categories such as not found, already exists, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates rather than matching error strings.
|
|
|
|
Backends may wrap implementation-specific errors, but callers should receive storage errors where practical. Traversal can stop cleanly with `ErrStopWalk`.
|
|
|
|
## Traversal helpers
|
|
|
|
Backends own their traversal mechanics. The local adapter owns filesystem walking, the SSH adapter owns SFTP directory walking, and the S3 adapter owns object listing and pagination.
|
|
|
|
`internal/storage` owns the shared callback emission rules used by backends:
|
|
|
|
- context cancellation is checked before callback emission;
|
|
- `WalkOptions.Limit` bounds the number of emitted entries;
|
|
- `ErrStopWalk` stops traversal without becoming a caller-visible error;
|
|
- callback errors are wrapped as storage walk errors.
|
|
|
|
`storage.HasAny(ctx, backend, prefix)` provides the shared destination-content check. It calls `Walk` with non-recursive, limit-one traversal and stops after the first emitted entry.
|
|
|
|
## Deletion
|
|
|
|
`DeleteManagedBundle` may delete listed managed outputs plus `.distributor.json`.
|
|
|
|
`DeletePrefix` removes content at and below a logical prefix for explicit forced replacement. It must not delete above the requested prefix or above the configured backend root.
|
|
|
|
## Local, SSH, S3, and fake backends
|
|
|
|
The local adapter maps logical paths to a configured filesystem root and keeps adapter-specific path handling behind the storage interface.
|
|
|
|
The SSH adapter maps logical paths to a configured remote SFTP root. It uses native SSH and SFTP libraries, supports SSH agent and key-file authentication, applies host-key policies, rejects unsafe logical paths, reports symlink entries from `Lstat`, and limits deletion to managed targets or explicit bounded prefixes.
|
|
|
|
The S3 adapter maps logical paths to object keys below a configured bucket and optional prefix. It uses the AWS SDK for Go v2, treats prefixes as object trees, requires exact objects for `Stat`, paginates traversal, applies conservative overwrite checks with `HeadObject`, infers basic content types, and limits deletion to managed target objects or explicit bounded object-key prefixes.
|
|
|
|
The fake backend is an in-memory implementation for package tests. It is not registered for runtime use.
|
|
|
|
## Tests
|
|
|
|
Before changing storage behavior, inspect tests under:
|
|
|
|
- `internal/storage`
|
|
- `internal/storage/fake`
|
|
- `internal/adapters/local`
|
|
- `internal/adapters/ssh`
|
|
- `internal/adapters/s3`
|
|
|
|
## Invariants
|
|
|
|
- Core packages depend on `internal/storage`, not concrete adapters.
|
|
- Logical paths are slash-separated and confined to the backend root.
|
|
- `storage.List` uses backend traversal and returns deterministic entries.
|
|
- Managed deletion is limited to recorded outputs plus `.distributor.json`.
|
|
- Prefix deletion is limited to the requested logical prefix.
|
|
- Runtime backend registration is owned by `internal/app`.
|