eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Add explicit force replacement workflow

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-31 17:29:20 +00:00
parent 7a174ce5f1
commit 48169dc8b4
28 changed files with 811 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
docs/internal/storage.md
View File

@@ -6,7 +6,7 @@
## Inputs and outputs
The storage interface supports byte reads, stream reads, byte writes, stream writes, exact metadata lookup, traversal, destination emptiness checks, and guarded managed deletion.
The storage interface supports byte reads, stream reads, byte writes, stream writes, exact metadata lookup, traversal, destination emptiness checks, guarded managed deletion, and bounded prefix deletion for explicit forced replacement.
Entries report a logical path, type, and size when available. Entry types are `file`, `directory`, `symlink`, and `other`.
@@ -28,15 +28,17 @@ Backends may wrap implementation-specific errors, but callers should receive sto
## Deletion
Backends expose guarded managed deletion only. `DeleteManagedBundle` may delete listed managed outputs plus `.distributor.json`; it does not provide broad recursive deletion.
`DeleteManagedBundle` may delete listed managed outputs plus `.distributor.json`.
`DeletePrefix` removes content at and below a logical prefix for explicit forced replacement. It must not delete above the requested prefix or above the configured backend root.
## Local, SSH, S3, and fake backends
The local adapter maps logical paths to a configured filesystem root and keeps adapter-specific path handling behind the storage interface.
The SSH adapter maps logical paths to a configured remote SFTP root. It uses native SSH and SFTP libraries, supports SSH agent and key-file authentication, applies host-key policies, rejects unsafe logical paths, reports symlink entries from `Lstat`, and limits deletion to managed targets.
The SSH adapter maps logical paths to a configured remote SFTP root. It uses native SSH and SFTP libraries, supports SSH agent and key-file authentication, applies host-key policies, rejects unsafe logical paths, reports symlink entries from `Lstat`, and limits deletion to managed targets or explicit bounded prefixes.
The S3 adapter maps logical paths to object keys below a configured bucket and optional prefix. It uses the AWS SDK for Go v2, treats prefixes as object trees, requires exact objects for `Stat`, paginates traversal, applies conservative overwrite checks with `HeadObject`, infers basic content types, and limits deletion to managed target objects.
The S3 adapter maps logical paths to object keys below a configured bucket and optional prefix. It uses the AWS SDK for Go v2, treats prefixes as object trees, requires exact objects for `Stat`, paginates traversal, applies conservative overwrite checks with `HeadObject`, infers basic content types, and limits deletion to managed target objects or explicit bounded object-key prefixes.
The fake backend is an in-memory implementation for package tests. It is not registered for runtime use.
@@ -56,4 +58,5 @@ Before changing storage behavior, inspect tests under:
- Logical paths are slash-separated and confined to the backend root.
- `storage.List` uses backend traversal and returns deterministic entries.
- Managed deletion is limited to recorded outputs plus `.distributor.json`.
- Prefix deletion is limited to the requested logical prefix.
- Runtime backend registration is owned by `internal/app`.