# Prune Roadmap

This document records planned destination pruning work that is not part of the
current implementation. Current cleanup and forced replacement behavior is
documented in `docs/operations.md`.

This work should be implemented after the reconciliation work in
`docs/roadmap/reconciliation.md`, shared-root state work in
`docs/roadmap/multipipeline.md`, and reconcile-state tooling in
`docs/roadmap/reconcile-state.md`. Prune decisions should rely on accurate
managed state, explicit ownership, and timestamp metadata.

## Problem

Merge reconciliation and shared-root publishing allow long-lived destinations
to accumulate distributor-managed outputs. That is useful for archives and
date-first report layouts, but those destinations can grow without bound.

Operators need a safe way to delete old distributor-managed outputs according
to explicit retention policy without touching unmanaged files or outputs owned
by unrelated scopes.

## Goal

Add pruning support for managed destination outputs.

Initial pruning should support both of these policy families:

- keep only the most recently updated N files in a configured scope;
- delete managed files older than a configured duration.

Pruning must be based on `.distributor.json` ownership records. It should never
delete files merely because they exist under a configured prefix.

## Policy Model

Configuration shape:

```yaml
destinations:
  - id: archive
    backend: s3
    bucket: reports
    prefix: weather
    reconciliation:
      mode: merge
    retention:
      prune:
        enabled: true
        older_than: 90d
        keep_latest: 500
```

Policy semantics:

- `enabled` defaults to `false`.
- `older_than` deletes managed outputs whose selected timestamp is older than
  the duration.
- `keep_latest` keeps the most recent N managed outputs in scope and deletes
  older managed outputs beyond that count.
- If both are set, preserve outputs that must be kept by `keep_latest`, then
  delete remaining outputs older than `older_than`.
- Retention scope is the configured destination owner scope unless
  shared-root configuration explicitly chooses a broader scope.

Timestamp basis: use per-output `updated_at` for both policies, with
`created_at` available for future created-time retention. Updated-time
retention best matches recurring reports where a stable path is overwritten.

## CLI Shape

Command shape:

```sh
distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run
distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --apply
```

The command should use configured retention policy by default. A later explicit
override mode can allow one-off retention values if that is useful, but the
initial implementation should avoid ad hoc deletion flags that bypass reviewed
config.

## Behavior

Planned behavior:

- Load and validate config normally.
- Resolve the selected destination and retention policy.
- Read and validate `.distributor.json`.
- Select managed output records in the requested owner scope.
- Sort candidates deterministically by selected timestamp and path.
- Build a delete plan from `older_than`, `keep_latest`, or both.
- In dry-run mode, report planned deletions and state changes without writing.
- In apply mode, delete only planned managed output paths.
- After successful deletes, remove deleted outputs from state and update state
  timestamps.
- If a delete fails, report partial failure and preserve enough state for safe
  retry. Prefer removing records only for confirmed deleted outputs.
- Never delete unmanaged storage entries.
- Delete only managed output paths initially. Local and SSH/SFTP backend helpers
  may prune empty directories only where they already do so safely. Object
  storage prefix markers are preserved unless the prefix marker is itself a
  managed output record.

Prune should operate on output records, not source manifests. If a future state
model groups outputs into logical source publications, group-level pruning can
be added later as an explicit policy.

## Safety Rules

Required safety behavior:

- Prune is opt-in.
- Apply mode is explicit; dry-run is the safe default.
- Delete only paths recorded as managed in `.distributor.json`.
- Do not delete `.distributor.json` unless every managed output in scope is
  deleted and a later design explicitly allows removing empty state.
- Do not delete unmanaged files, unknown paths, parent directories, or sibling
  prefixes.
- Do not run prune against invalid or ambiguous state.
- Preserve outputs outside the selected owner scope.
- Report all destructive actions with backend and path context.

## Implementation Work

Implementation work:

- Add retention config structs, defaults, parsing, and validation.
- Add duration parsing that supports documented minute/hour/day units.
- Add app-level prune planning and execution use cases.
- Add state helpers for selecting owner-scoped output records, sorting by
  timestamp, and removing deleted records.
- Add storage delete support or reuse existing managed-delete helpers.
- Add CLI parsing, help text, dry-run output, and apply output.
- Ensure local, SSH/SFTP, S3, and fake backends preserve bounded deletion
  semantics.
- Ensure pruning composes with merge mode and shared-root state.

## Testing

Important tests:

- Config defaults retention disabled.
- Config rejects invalid durations, negative counts, and enabled policies with
  no retention rule.
- Dry-run reports deletes without deleting files or rewriting state.
- `older_than` deletes only managed outputs older than the threshold.
- `keep_latest` preserves the newest N outputs and deletes older managed
  outputs.
- Combined `older_than` and `keep_latest` behavior is deterministic.
- Shared-root pruning preserves outputs from other owners when scoped to one
  owner.
- Unmanaged files under the destination root are never deleted.
- Partial delete failure preserves accurate state for confirmed deleted and
  undeleted outputs.
- CLI tests cover required flags, apply gating, and report output.
- Backend tests cover local, SSH/SFTP, S3, and fake deletion behavior used by
  prune.

## Documentation Work

When implemented, update current-behavior docs in the same change:

- `docs/config.md`: retention configuration, defaults, and examples.
- `docs/cli.md`: prune command syntax and workflows.
- `docs/operations.md`: pruning safety, dry-run, apply, and recovery behavior.
- `docs/troubleshooting.md`: prune conflicts and partial failures.
- `docs/integrations/destination-state.md`: timestamp fields used by pruning.
- `docs/internal/state.md`, `docs/internal/publish.md`, and
  `docs/internal/app.md`: implementation boundaries and invariants.

## Boundaries

- Do not let producers choose retention policy.
- Do not use prune as a general storage cleanup tool.
- Do not delete unmanaged content.
- Do not infer retention from filenames unless a later roadmap explicitly adds
  path/date parsing.
- Do not depend on backend-native lifecycle policies for correctness; those can
  coexist operationally but are not distributor's managed state.
- Do not run pruning automatically after publish in the initial implementation;
  pruning should start as an explicit CLI command that operators can schedule.