distributor/docs/roadmap/prune.md

Prune Roadmap

This document records planned destination pruning work that is not part of the current implementation. Current cleanup and forced replacement behavior is documented in docs/operations.md.

This work should be implemented after the reconciliation work in docs/roadmap/reconciliation.md, shared-root state work in docs/roadmap/multipipeline.md, and reconcile-state tooling in docs/roadmap/reconcile-state.md. Prune decisions should rely on accurate managed state, explicit ownership, and timestamp metadata.

Problem

Merge reconciliation and shared-root publishing allow long-lived destinations to accumulate distributor-managed outputs. That is useful for archives and date-first report layouts, but those destinations can grow without bound.

Operators need a safe way to delete old distributor-managed outputs according to explicit retention policy without touching unmanaged files or outputs owned by unrelated scopes.

Goal

Add pruning support for managed destination outputs.

Initial pruning should support both of these policy families:

• keep only the most recently updated N files in a configured scope;
• delete managed files older than a configured duration.

Pruning must be based on .distributor.json ownership records. It should never delete files merely because they exist under a configured prefix.

Policy Model

Configuration shape:

destinations:
  - id: archive
    backend: s3
    bucket: reports
    prefix: weather
    reconciliation:
      mode: merge
    retention:
      prune:
        enabled: true
        older_than: 90d
        keep_latest: 500

Policy semantics:

• enabled defaults to false.
• older_than deletes managed outputs whose selected timestamp is older than the duration.
• keep_latest keeps the most recent N managed outputs in scope and deletes older managed outputs beyond that count.
• If both are set, preserve outputs that must be kept by keep_latest, then delete remaining outputs older than older_than.
• Retention scope is the configured destination owner scope unless shared-root configuration explicitly chooses a broader scope.

Timestamp basis: use per-output updated_at for both policies, with created_at available for future created-time retention. Updated-time retention best matches recurring reports where a stable path is overwritten.

CLI Shape

Command shape:

distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run
distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --apply

The command should use configured retention policy by default. A later explicit override mode can allow one-off retention values if that is useful, but the initial implementation should avoid ad hoc deletion flags that bypass reviewed config.

Behavior

Planned behavior:

• Load and validate config normally.
• Resolve the selected destination and retention policy.
• Read and validate .distributor.json.
• Select managed output records in the requested owner scope.
• Sort candidates deterministically by selected timestamp and path.
• Build a delete plan from older_than, keep_latest, or both.
• In dry-run mode, report planned deletions and state changes without writing.
• In apply mode, delete only planned managed output paths.
• After successful deletes, remove deleted outputs from state and update state timestamps.
• If a delete fails, report partial failure and preserve enough state for safe retry. Prefer removing records only for confirmed deleted outputs.
• Never delete unmanaged storage entries.
• Delete only managed output paths initially. Local and SSH/SFTP backend helpers may prune empty directories only where they already do so safely. Object storage prefix markers are preserved unless the prefix marker is itself a managed output record.

Prune should operate on output records, not source manifests. If a future state model groups outputs into logical source publications, group-level pruning can be added later as an explicit policy.

Safety Rules

Required safety behavior:

• Prune is opt-in.
• Apply mode is explicit; dry-run is the safe default.
• Delete only paths recorded as managed in .distributor.json.
• Do not delete .distributor.json unless every managed output in scope is deleted and a later design explicitly allows removing empty state.
• Do not delete unmanaged files, unknown paths, parent directories, or sibling prefixes.
• Do not run prune against invalid or ambiguous state.
• Preserve outputs outside the selected owner scope.
• Report all destructive actions with backend and path context.

Implementation Work

Implementation work:

• Add retention config structs, defaults, parsing, and validation.
• Add duration parsing that supports documented minute/hour/day units.
• Add app-level prune planning and execution use cases.
• Add state helpers for selecting owner-scoped output records, sorting by timestamp, and removing deleted records.
• Add storage delete support or reuse existing managed-delete helpers.
• Add CLI parsing, help text, dry-run output, and apply output.
• Ensure local, SSH/SFTP, S3, and fake backends preserve bounded deletion semantics.
• Ensure pruning composes with merge mode and shared-root state.

Testing

Important tests:

• Config defaults retention disabled.
• Config rejects invalid durations, negative counts, and enabled policies with no retention rule.
• Dry-run reports deletes without deleting files or rewriting state.
• older_than deletes only managed outputs older than the threshold.
• keep_latest preserves the newest N outputs and deletes older managed outputs.
• Combined older_than and keep_latest behavior is deterministic.
• Shared-root pruning preserves outputs from other owners when scoped to one owner.
• Unmanaged files under the destination root are never deleted.
• Partial delete failure preserves accurate state for confirmed deleted and undeleted outputs.
• CLI tests cover required flags, apply gating, and report output.
• Backend tests cover local, SSH/SFTP, S3, and fake deletion behavior used by prune.

Documentation Work

When implemented, update current-behavior docs in the same change:

• docs/config.md: retention configuration, defaults, and examples.
• docs/cli.md: prune command syntax and workflows.
• docs/operations.md: pruning safety, dry-run, apply, and recovery behavior.
• docs/troubleshooting.md: prune conflicts and partial failures.
• docs/integrations/destination-state.md: timestamp fields used by pruning.
• docs/internal/state.md, docs/internal/publish.md, and docs/internal/app.md: implementation boundaries and invariants.

Boundaries

• Do not let producers choose retention policy.
• Do not use prune as a general storage cleanup tool.
• Do not delete unmanaged content.
• Do not infer retention from filenames unless a later roadmap explicitly adds path/date parsing.
• Do not depend on backend-native lifecycle policies for correctness; those can coexist operationally but are not distributor's managed state.
• Do not run pruning automatically after publish in the initial implementation; pruning should start as an explicit CLI command that operators can schedule.