60 lines
3.5 KiB
Markdown
60 lines
3.5 KiB
Markdown
# Storage Internals
|
|
|
|
Audience: developers and LLM coding agents changing `internal/storage`, storage adapters, or storage-backed callers.
|
|
|
|
## Purpose
|
|
|
|
`internal/storage` defines backend-rooted logical file access, path validation, typed storage errors, traversal helpers, backend registration, managed deletion targets, and test fake storage behavior.
|
|
|
|
## Inputs And Outputs
|
|
|
|
Inputs are contexts, logical paths or prefixes, byte slices or readers, write options, walk options, delete options, and backend open configs. Outputs are file bytes, readers, `Entry` metadata, walk callbacks, boolean content checks, registered backends, and typed errors.
|
|
|
|
## Boundaries
|
|
|
|
Core packages depend on `internal/storage`, not concrete adapters. Adapter protocol behavior belongs in `internal/adapters/local`, `internal/adapters/ssh`, and `internal/adapters/s3`; external SSH/SFTP and S3 notes live under `docs/integrations/`.
|
|
|
|
Runtime backend construction and registration are owned by `internal/app`. The fake backend is for tests only.
|
|
|
|
## Config Fields Used
|
|
|
|
The storage package does not read config directly. App adapter wiring converts config fields into backend open config values.
|
|
|
|
## Adapters Used
|
|
|
|
Local, SSH/SFTP, and S3-compatible adapters implement `storage.Backend`. `internal/storage/fake` implements the same interface for tests.
|
|
|
|
## State And Manifest Behavior
|
|
|
|
Storage owns `.distributor.json` path helpers through `StateFileName`, `StatePath`, `ManagedOutputTargets`, and `ManagedBundleTargets`. It does not parse source manifests or destination state.
|
|
|
|
Logical paths are slash-separated and relative to a backend root. Prefix validation allows an empty prefix to mean the backend root; file path validation requires a non-empty path.
|
|
|
|
Reconcile-state callers use `Stat` to check whether managed output paths still exist and bounded recursive `Walk` to report unmanaged entries under a selected destination root. Prune execution callers use `DeleteManagedOutputs` for selected managed output records so `.distributor.json` is not part of the deletion target set. Storage does not decide whether entries are managed; callers compare entries against destination state.
|
|
|
|
## Skip And Resume Behavior
|
|
|
|
Storage has no publication skip policy. It supplies `HasAny` for unmanaged-content checks, `Stat` and `Walk` for state repair inspection, `DeleteManagedOutputs` target construction for managed output cleanup, `DeleteManagedBundle` target construction for normal replacement cleanup, and `DeletePrefix` semantics for explicit forced replacement.
|
|
|
|
## Failure Behavior
|
|
|
|
Storage errors use typed categories: not found, already exists, not empty, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates instead of matching strings. Traversal can stop cleanly with `ErrStopWalk`.
|
|
|
|
## Tests To Inspect
|
|
|
|
- `internal/storage/*_test.go`
|
|
- `internal/storage/fake/*_test.go`
|
|
- `internal/app/reconcile_state_test.go`
|
|
- `internal/adapters/local/*_test.go`
|
|
- `internal/adapters/ssh/*_test.go`
|
|
- `internal/adapters/s3/*_test.go`
|
|
|
|
## Architectural Invariants
|
|
|
|
- Logical paths are clean relative slash-separated paths confined to the backend root.
|
|
- Core packages never import concrete adapters.
|
|
- `storage.List` returns deterministic sorted entries.
|
|
- Managed output deletion targets are recorded outputs only; managed bundle deletion targets are recorded outputs plus `.distributor.json`.
|
|
- Prefix deletion is bounded to the requested logical prefix.
|
|
- Runtime registration remains app-owned.
|