Normalize internal component documentation
This commit is contained in:
@@ -1,75 +1,56 @@
|
||||
# Storage
|
||||
# Storage Internals
|
||||
|
||||
Audience: developers and LLM coding agents changing `internal/storage`, storage adapters, or storage-backed callers.
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/storage` defines backend-rooted logical file access for core packages. Callers use slash-separated paths relative to a configured backend root.
|
||||
`internal/storage` defines backend-rooted logical file access, path validation, typed storage errors, traversal helpers, backend registration, managed deletion targets, and test fake storage behavior.
|
||||
|
||||
## Inputs and outputs
|
||||
## Inputs And Outputs
|
||||
|
||||
The storage interface supports byte reads, stream reads, byte writes, stream writes, exact metadata lookup, traversal, destination emptiness checks, guarded managed deletion, and bounded prefix deletion for explicit forced replacement.
|
||||
|
||||
Entries report a logical path, type, and size when available. Entry types are `file`, `directory`, `symlink`, and `other`.
|
||||
Inputs are contexts, logical paths or prefixes, byte slices or readers, write options, walk options, delete options, and backend open configs. Outputs are file bytes, readers, `Entry` metadata, walk callbacks, boolean content checks, registered backends, and typed errors.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Core packages should depend on `internal/storage`, not adapter packages. Adapter-specific path handling stays behind backend implementations.
|
||||
Core packages depend on `internal/storage`, not concrete adapters. Adapter protocol behavior belongs in `internal/adapters/local`, `internal/adapters/ssh`, and `internal/adapters/s3`; external SSH/SFTP and S3 notes live under `docs/integrations/`.
|
||||
|
||||
The local adapter lives in `internal/adapters/local`. The SSH/SFTP adapter lives in `internal/adapters/ssh`. The S3-compatible adapter lives in `internal/adapters/s3`. Runtime backend construction is wired through the app-level backend factory and storage registry. The fake backend lives in `internal/storage/fake` for tests and is not registered for runtime use.
|
||||
Runtime backend construction and registration are owned by `internal/app`. The fake backend is for tests only.
|
||||
|
||||
## Paths
|
||||
## Config Fields Used
|
||||
|
||||
Logical file paths must be non-empty, relative, clean, slash-separated, and must not contain `.` or `..` segments or backslashes. Prefix paths follow the same rules, except an empty prefix means the backend root.
|
||||
The storage package does not read config directly. App adapter wiring converts config fields into backend open config values.
|
||||
|
||||
## Failure behavior
|
||||
## Adapters Used
|
||||
|
||||
Storage errors use typed categories such as not found, already exists, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates rather than matching error strings.
|
||||
Local, SSH/SFTP, and S3-compatible adapters implement `storage.Backend`. `internal/storage/fake` implements the same interface for tests.
|
||||
|
||||
Backends may wrap implementation-specific errors, but callers should receive storage errors where practical. Traversal can stop cleanly with `ErrStopWalk`.
|
||||
## State And Manifest Behavior
|
||||
|
||||
## Traversal helpers
|
||||
Storage owns `.distributor.json` path helpers through `StateFileName`, `StatePath`, and `ManagedBundleTargets`. It does not parse source manifests or destination state.
|
||||
|
||||
Backends own their traversal mechanics. The local adapter owns filesystem walking, the SSH adapter owns SFTP directory walking, and the S3 adapter owns object listing and pagination.
|
||||
Logical paths are slash-separated and relative to a backend root. Prefix validation allows an empty prefix to mean the backend root; file path validation requires a non-empty path.
|
||||
|
||||
`internal/storage` owns the shared callback emission rules used by backends:
|
||||
## Skip And Resume Behavior
|
||||
|
||||
- context cancellation is checked before callback emission;
|
||||
- `WalkOptions.Limit` bounds the number of emitted entries;
|
||||
- `ErrStopWalk` stops traversal without becoming a caller-visible error;
|
||||
- callback errors are wrapped as storage walk errors.
|
||||
Storage has no publication skip policy. It supplies `HasAny` for unmanaged-content checks, `DeleteManagedBundle` target construction for normal replacement cleanup, and `DeletePrefix` semantics for explicit forced replacement.
|
||||
|
||||
`storage.HasAny(ctx, backend, prefix)` provides the shared destination-content check. It calls `Walk` with non-recursive, limit-one traversal and stops after the first emitted entry.
|
||||
## Failure Behavior
|
||||
|
||||
## Deletion
|
||||
Storage errors use typed categories: not found, already exists, not empty, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates instead of matching strings. Traversal can stop cleanly with `ErrStopWalk`.
|
||||
|
||||
`DeleteManagedBundle` may delete listed managed outputs plus `.distributor.json`.
|
||||
## Tests To Inspect
|
||||
|
||||
`DeletePrefix` removes content at and below a logical prefix for explicit forced replacement. It must not delete above the requested prefix or above the configured backend root.
|
||||
- `internal/storage/*_test.go`
|
||||
- `internal/storage/fake/*_test.go`
|
||||
- `internal/adapters/local/*_test.go`
|
||||
- `internal/adapters/ssh/*_test.go`
|
||||
- `internal/adapters/s3/*_test.go`
|
||||
|
||||
## Local, SSH, S3, and fake backends
|
||||
## Architectural Invariants
|
||||
|
||||
The local adapter maps logical paths to a configured filesystem root and keeps adapter-specific path handling behind the storage interface.
|
||||
|
||||
The SSH adapter maps logical paths to a configured remote SFTP root. It uses native SSH and SFTP libraries, supports SSH agent and key-file authentication, applies host-key policies, rejects unsafe logical paths, reports symlink entries from `Lstat`, and limits deletion to managed targets or explicit bounded prefixes.
|
||||
|
||||
The S3 adapter maps logical paths to object keys below a configured bucket and optional prefix. It uses the AWS SDK for Go v2, treats prefixes as object trees, requires exact objects for `Stat`, paginates traversal, applies conservative overwrite checks with `HeadObject`, infers basic content types, and limits deletion to managed target objects or explicit bounded object-key prefixes.
|
||||
|
||||
The fake backend is an in-memory implementation for package tests. It is not registered for runtime use.
|
||||
|
||||
## Tests
|
||||
|
||||
Before changing storage behavior, inspect tests under:
|
||||
|
||||
- `internal/storage`
|
||||
- `internal/storage/fake`
|
||||
- `internal/adapters/local`
|
||||
- `internal/adapters/ssh`
|
||||
- `internal/adapters/s3`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Core packages depend on `internal/storage`, not concrete adapters.
|
||||
- Logical paths are slash-separated and confined to the backend root.
|
||||
- `storage.List` uses backend traversal and returns deterministic entries.
|
||||
- Managed deletion is limited to recorded outputs plus `.distributor.json`.
|
||||
- Prefix deletion is limited to the requested logical prefix.
|
||||
- Runtime backend registration is owned by `internal/app`.
|
||||
- Logical paths are clean relative slash-separated paths confined to the backend root.
|
||||
- Core packages never import concrete adapters.
|
||||
- `storage.List` returns deterministic sorted entries.
|
||||
- Managed deletion targets are recorded outputs plus `.distributor.json`.
|
||||
- Prefix deletion is bounded to the requested logical prefix.
|
||||
- Runtime registration remains app-owned.
|
||||
|
||||
Reference in New Issue
Block a user