184 lines
7.1 KiB
Markdown
184 lines
7.1 KiB
Markdown
# Prune Roadmap
|
|
|
|
This document records planned destination pruning work that is not part of the
|
|
current implementation. Current cleanup and forced replacement behavior is
|
|
documented in `docs/operations.md`.
|
|
|
|
This work should be implemented after the reconciliation work in
|
|
`docs/roadmap/reconciliation.md`, shared-root state work in
|
|
`docs/roadmap/multipipeline.md`, and reconcile-state tooling in
|
|
`docs/roadmap/reconcile-state.md`. Prune decisions should rely on accurate
|
|
managed state, explicit ownership, and timestamp metadata.
|
|
|
|
## Problem
|
|
|
|
Merge reconciliation and shared-root publishing allow long-lived destinations
|
|
to accumulate distributor-managed outputs. That is useful for archives and
|
|
date-first report layouts, but those destinations can grow without bound.
|
|
|
|
Operators need a safe way to delete old distributor-managed outputs according
|
|
to explicit retention policy without touching unmanaged files or outputs owned
|
|
by unrelated scopes.
|
|
|
|
## Goal
|
|
|
|
Add pruning support for managed destination outputs.
|
|
|
|
Initial pruning should support both of these policy families:
|
|
|
|
- keep only the most recently updated N files in a configured scope;
|
|
- delete managed files older than a configured duration.
|
|
|
|
Pruning must be based on `.distributor.json` ownership records. It should never
|
|
delete files merely because they exist under a configured prefix.
|
|
|
|
## Policy Model
|
|
|
|
Configuration shape:
|
|
|
|
```yaml
|
|
destinations:
|
|
- id: archive
|
|
backend: s3
|
|
bucket: reports
|
|
prefix: weather
|
|
reconciliation:
|
|
mode: merge
|
|
retention:
|
|
prune:
|
|
enabled: true
|
|
older_than: 90d
|
|
keep_latest: 500
|
|
```
|
|
|
|
Policy semantics:
|
|
|
|
- `enabled` defaults to `false`.
|
|
- `older_than` deletes managed outputs whose selected timestamp is older than
|
|
the duration.
|
|
- `keep_latest` keeps the most recent N managed outputs in scope and deletes
|
|
older managed outputs beyond that count.
|
|
- If both are set, preserve outputs that must be kept by `keep_latest`, then
|
|
delete remaining outputs older than `older_than`.
|
|
- Retention scope is the configured destination owner scope unless
|
|
shared-root configuration explicitly chooses a broader scope.
|
|
|
|
Timestamp basis: use per-output `updated_at` for both policies, with
|
|
`created_at` available for future created-time retention. Updated-time
|
|
retention best matches recurring reports where a stable path is overwritten.
|
|
|
|
## CLI Shape
|
|
|
|
Command shape:
|
|
|
|
```sh
|
|
distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run
|
|
distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --apply
|
|
```
|
|
|
|
The command should use configured retention policy by default. A later explicit
|
|
override mode can allow one-off retention values if that is useful, but the
|
|
initial implementation should avoid ad hoc deletion flags that bypass reviewed
|
|
config.
|
|
|
|
## Behavior
|
|
|
|
Planned behavior:
|
|
|
|
- Load and validate config normally.
|
|
- Resolve the selected destination and retention policy.
|
|
- Read and validate `.distributor.json`.
|
|
- Select managed output records in the requested owner scope.
|
|
- Sort candidates deterministically by selected timestamp and path.
|
|
- Build a delete plan from `older_than`, `keep_latest`, or both.
|
|
- In dry-run mode, report planned deletions and state changes without writing.
|
|
- In apply mode, delete only planned managed output paths.
|
|
- After successful deletes, remove deleted outputs from state and update state
|
|
timestamps.
|
|
- If a delete fails, report partial failure and preserve enough state for safe
|
|
retry. Prefer removing records only for confirmed deleted outputs.
|
|
- Never delete unmanaged storage entries.
|
|
- Delete only managed output paths initially. Local and SSH/SFTP backend helpers
|
|
may prune empty directories only where they already do so safely. Object
|
|
storage prefix markers are preserved unless the prefix marker is itself a
|
|
managed output record.
|
|
|
|
Prune should operate on output records, not source manifests. If a future state
|
|
model groups outputs into logical source publications, group-level pruning can
|
|
be added later as an explicit policy.
|
|
|
|
## Safety Rules
|
|
|
|
Required safety behavior:
|
|
|
|
- Prune is opt-in.
|
|
- Apply mode is explicit; dry-run is the safe default.
|
|
- Delete only paths recorded as managed in `.distributor.json`.
|
|
- Do not delete `.distributor.json` unless every managed output in scope is
|
|
deleted and a later design explicitly allows removing empty state.
|
|
- Do not delete unmanaged files, unknown paths, parent directories, or sibling
|
|
prefixes.
|
|
- Do not run prune against invalid or ambiguous state.
|
|
- Preserve outputs outside the selected owner scope.
|
|
- Report all destructive actions with backend and path context.
|
|
|
|
## Implementation Work
|
|
|
|
Implementation work:
|
|
|
|
- Add retention config structs, defaults, parsing, and validation.
|
|
- Add duration parsing that supports documented minute/hour/day units.
|
|
- Add app-level prune planning and execution use cases.
|
|
- Add state helpers for selecting owner-scoped output records, sorting by
|
|
timestamp, and removing deleted records.
|
|
- Add storage delete support or reuse existing managed-delete helpers.
|
|
- Add CLI parsing, help text, dry-run output, and apply output.
|
|
- Ensure local, SSH/SFTP, S3, and fake backends preserve bounded deletion
|
|
semantics.
|
|
- Ensure pruning composes with merge mode and shared-root state.
|
|
|
|
## Testing
|
|
|
|
Important tests:
|
|
|
|
- Config defaults retention disabled.
|
|
- Config rejects invalid durations, negative counts, and enabled policies with
|
|
no retention rule.
|
|
- Dry-run reports deletes without deleting files or rewriting state.
|
|
- `older_than` deletes only managed outputs older than the threshold.
|
|
- `keep_latest` preserves the newest N outputs and deletes older managed
|
|
outputs.
|
|
- Combined `older_than` and `keep_latest` behavior is deterministic.
|
|
- Shared-root pruning preserves outputs from other owners when scoped to one
|
|
owner.
|
|
- Unmanaged files under the destination root are never deleted.
|
|
- Partial delete failure preserves accurate state for confirmed deleted and
|
|
undeleted outputs.
|
|
- CLI tests cover required flags, apply gating, and report output.
|
|
- Backend tests cover local, SSH/SFTP, S3, and fake deletion behavior used by
|
|
prune.
|
|
|
|
## Documentation Work
|
|
|
|
When implemented, update current-behavior docs in the same change:
|
|
|
|
- `docs/config.md`: retention configuration, defaults, and examples.
|
|
- `docs/cli.md`: prune command syntax and workflows.
|
|
- `docs/operations.md`: pruning safety, dry-run, apply, and recovery behavior.
|
|
- `docs/troubleshooting.md`: prune conflicts and partial failures.
|
|
- `docs/integrations/destination-state.md`: timestamp fields used by pruning.
|
|
- `docs/internal/state.md`, `docs/internal/publish.md`, and
|
|
`docs/internal/app.md`: implementation boundaries and invariants.
|
|
|
|
## Boundaries
|
|
|
|
- Do not let producers choose retention policy.
|
|
- Do not use prune as a general storage cleanup tool.
|
|
- Do not delete unmanaged content.
|
|
- Do not infer retention from filenames unless a later roadmap explicitly adds
|
|
path/date parsing.
|
|
- Do not depend on backend-native lifecycle policies for correctness; those can
|
|
coexist operationally but are not distributor's managed state.
|
|
- Do not run pruning automatically after publish in the initial implementation;
|
|
pruning should start as an explicit CLI command that operators can schedule.
|