# Reconcile-State Roadmap

This document records planned destination state repair tooling that is not part
of the current implementation. Current recovery behavior is documented in
`docs/operations.md`.

This work should be implemented after the reconciliation work in
`docs/roadmap/reconciliation.md` and the shared-root state work in
`docs/roadmap/multipipeline.md`. Reconcile-state tooling should understand the
final managed state model before prune behavior is added in
`docs/roadmap/prune.md`.

## Problem

`.distributor.json` is the managed ownership index for destination outputs. In
long-lived destinations, operators may delete files manually, lifecycle rules
may remove objects, failed external operations may leave state stale, or a
future shared-root destination may accumulate many independently managed
outputs.

Without a first-class repair command, operators must inspect and edit state by
hand. That is risky because destination state drives replacement, merge,
conflict, and cleanup decisions.

## Goal

Add an operator-facing reconcile-state workflow that compares
`.distributor.json` to actual destination storage and repairs stale managed
state safely.

The initial command should focus on removing state records for managed outputs
that no longer exist. It should report unmanaged files, but it should not adopt
them by default.

## CLI Shape

Command shape:

```sh
distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run
distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive
```

For shared-root destinations, the command should also support reconciling all
owners in one selected destination root. The final selector should be explicit
enough to avoid relying on destination ids being globally unique:

```sh
distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --all-owners --dry-run
```

The exact flags should follow existing CLI conventions. Dry-run should be the
explicit `--dry-run` flag; without `--dry-run`, the command applies state
repairs.

## Behavior

Planned behavior:

- Load and validate config normally.
- Open the configured destination backend.
- Resolve the destination root or destination bundle path using the same path
  mapping semantics as publish where applicable.
- Read and validate `.distributor.json`.
- For each managed output record, check whether the storage object/file exists.
- Remove state records for managed outputs that are missing unless `--dry-run`
  is set.
- Preserve state records for managed outputs that exist.
- Report storage entries that exist but are not recorded as managed.
- Preserve unmanaged entries by default.
- Update state timestamps when state is changed.
- Write state atomically where the backend supports atomic writes.
- Never delete storage objects as part of the initial reconcile-state command.
- Do not validate output digests against storage contents in the initial
  command; digest verification can be added later as a separate audit mode.

For current single-owner state, reconciliation removes missing records from the
single `outputs` array. For future shared-root state, reconciliation removes
missing output records across the selected owner scope or all owners, depending
on command flags.

## Reporting

Dry-run and apply output should be explicit enough for operators and LLM agents
to understand the result.

Reports should include:

- destination backend and root;
- state schema version;
- owner scope, if filtered;
- count of managed outputs checked;
- count and paths of missing managed outputs;
- count and paths of unmanaged existing entries, subject to sensible output
  limits;
- whether state would be changed or was changed;
- errors with enough backend/path context to recover safely.

Machine-readable output should be considered if existing CLI report formats can
support it without broad refactoring.

## Safety Rules

Required safety behavior:

- Do not delete destination files.
- Do not adopt unmanaged destination files by default.
- Do not rewrite invalid state except through a future explicit repair workflow.
- Do not repair state when the destination identity or ownership scope is
  ambiguous.
- Do not scan outside the configured backend root or destination root.
- Apply state repairs by default and make `--dry-run` the explicit read-only
  inspection mode.
- Keep secrets out of reports and errors.

## Implementation Work

Implementation work:

- Add CLI parsing and help text for the reconcile-state command.
- Add an app-level use case that loads config, opens destinations, and builds a
  reconcile-state report.
- Add state package helpers for removing missing outputs without weakening
  validation.
- Add storage-facing existence/listing helpers only if current backend
  interfaces are insufficient.
- Implement local, SSH/SFTP, S3, and fake backend support needed by the command.
- Preserve shared-root ownership metadata and timestamps when rewriting state.
- Add dry-run and applied-change output in the existing CLI reporting style.

## Testing

Important tests:

- Dry-run reports missing managed outputs and does not write state.
- Apply removes missing managed output records and writes valid state.
- Existing managed outputs are preserved.
- Unmanaged existing files are reported but not adopted or deleted.
- Invalid state fails without rewriting.
- Shared-root state can be reconciled for one owner scope.
- Shared-root state can be reconciled for all owner scopes.
- Local, SSH/SFTP, S3, and fake backend behavior is covered for the existence
  and listing operations used.
- CLI tests cover flag validation, default apply behavior, `--dry-run`
  behavior, and report output.

## Documentation Work

When implemented, update current-behavior docs in the same change:

- `docs/cli.md`: command syntax, flags, and examples.
- `docs/operations.md`: when and how to run reconcile-state safely.
- `docs/troubleshooting.md`: stale state and missing managed file entries.
- `docs/integrations/destination-state.md`: explain how tools may rewrite
  state.
- `docs/internal/state.md`, `docs/internal/storage.md`, and
  `docs/internal/app.md`: record implementation boundaries.

## Boundaries

- Do not implement prune behavior in this work.
- Do not let reconcile-state infer producer source data or regenerate outputs.
- Do not repair invalid JSON or manually edited state unless a later roadmap
  defines an explicit recovery flow.
- Do not claim unmanaged files by default.
- Do not add unmanaged-file adoption or digest validation in the initial
  command.
- Do not add whole-config reconciliation in the initial command; require enough
  scope to identify one destination root, with all-owner reconciliation allowed
  inside a configured shared-root destination.
- Do not expose a producer-facing API for this workflow.