Added four new roadmaps related to state management and a corresponding implementation plan

docs/roadmap/reconcile-state.md

@@ -0,0 +1,171 @@
+# Reconcile-State Roadmap
+
+This document records planned destination state repair tooling that is not part
+of the current implementation. Current recovery behavior is documented in
+`docs/operations.md`.
+
+This work should be implemented after the reconciliation work in
+`docs/roadmap/reconciliation.md` and the shared-root state work in
+`docs/roadmap/multipipeline.md`. Reconcile-state tooling should understand the
+final managed state model before prune behavior is added in
+`docs/roadmap/prune.md`.
+
+## Problem
+
+`.distributor.json` is the managed ownership index for destination outputs. In
+long-lived destinations, operators may delete files manually, lifecycle rules
+may remove objects, failed external operations may leave state stale, or a
+future shared-root destination may accumulate many independently managed
+outputs.
+
+Without a first-class repair command, operators must inspect and edit state by
+hand. That is risky because destination state drives replacement, merge,
+conflict, and cleanup decisions.
+
+## Goal
+
+Add an operator-facing reconcile-state workflow that compares
+`.distributor.json` to actual destination storage and repairs stale managed
+state safely.
+
+The initial command should focus on removing state records for managed outputs
+that no longer exist. It should report unmanaged files, but it should not adopt
+them by default.
+
+## CLI Shape
+
+Command shape:
+
+```sh
+distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run
+distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive
+```
+
+For shared-root destinations, the command should also support reconciling all
+owners in one selected destination root. The final selector should be explicit
+enough to avoid relying on destination ids being globally unique:
+
+```sh
+distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --all-owners --dry-run
+```
+
+The exact flags should follow existing CLI conventions. Dry-run should be the
+explicit `--dry-run` flag; without `--dry-run`, the command applies state
+repairs.
+
+## Behavior
+
+Planned behavior:
+
+- Load and validate config normally.
+- Open the configured destination backend.
+- Resolve the destination root or destination bundle path using the same path
+  mapping semantics as publish where applicable.
+- Read and validate `.distributor.json`.
+- For each managed output record, check whether the storage object/file exists.
+- Remove state records for managed outputs that are missing unless `--dry-run`
+  is set.
+- Preserve state records for managed outputs that exist.
+- Report storage entries that exist but are not recorded as managed.
+- Preserve unmanaged entries by default.
+- Update state timestamps when state is changed.
+- Write state atomically where the backend supports atomic writes.
+- Never delete storage objects as part of the initial reconcile-state command.
+- Do not validate output digests against storage contents in the initial
+  command; digest verification can be added later as a separate audit mode.
+
+For current single-owner state, reconciliation removes missing records from the
+single `outputs` array. For future shared-root state, reconciliation removes
+missing output records across the selected owner scope or all owners, depending
+on command flags.
+
+## Reporting
+
+Dry-run and apply output should be explicit enough for operators and LLM agents
+to understand the result.
+
+Reports should include:
+
+- destination backend and root;
+- state schema version;
+- owner scope, if filtered;
+- count of managed outputs checked;
+- count and paths of missing managed outputs;
+- count and paths of unmanaged existing entries, subject to sensible output
+  limits;
+- whether state would be changed or was changed;
+- errors with enough backend/path context to recover safely.
+
+Machine-readable output should be considered if existing CLI report formats can
+support it without broad refactoring.
+
+## Safety Rules
+
+Required safety behavior:
+
+- Do not delete destination files.
+- Do not adopt unmanaged destination files by default.
+- Do not rewrite invalid state except through a future explicit repair workflow.
+- Do not repair state when the destination identity or ownership scope is
+  ambiguous.
+- Do not scan outside the configured backend root or destination root.
+- Apply state repairs by default and make `--dry-run` the explicit read-only
+  inspection mode.
+- Keep secrets out of reports and errors.
+
+## Implementation Work
+
+Implementation work:
+
+- Add CLI parsing and help text for the reconcile-state command.
+- Add an app-level use case that loads config, opens destinations, and builds a
+  reconcile-state report.
+- Add state package helpers for removing missing outputs without weakening
+  validation.
+- Add storage-facing existence/listing helpers only if current backend
+  interfaces are insufficient.
+- Implement local, SSH/SFTP, S3, and fake backend support needed by the command.
+- Preserve shared-root ownership metadata and timestamps when rewriting state.
+- Add dry-run and applied-change output in the existing CLI reporting style.
+
+## Testing
+
+Important tests:
+
+- Dry-run reports missing managed outputs and does not write state.
+- Apply removes missing managed output records and writes valid state.
+- Existing managed outputs are preserved.
+- Unmanaged existing files are reported but not adopted or deleted.
+- Invalid state fails without rewriting.
+- Shared-root state can be reconciled for one owner scope.
+- Shared-root state can be reconciled for all owner scopes.
+- Local, SSH/SFTP, S3, and fake backend behavior is covered for the existence
+  and listing operations used.
+- CLI tests cover flag validation, default apply behavior, `--dry-run`
+  behavior, and report output.
+
+## Documentation Work
+
+When implemented, update current-behavior docs in the same change:
+
+- `docs/cli.md`: command syntax, flags, and examples.
+- `docs/operations.md`: when and how to run reconcile-state safely.
+- `docs/troubleshooting.md`: stale state and missing managed file entries.
+- `docs/integrations/destination-state.md`: explain how tools may rewrite
+  state.
+- `docs/internal/state.md`, `docs/internal/storage.md`, and
+  `docs/internal/app.md`: record implementation boundaries.
+
+## Boundaries
+
+- Do not implement prune behavior in this work.
+- Do not let reconcile-state infer producer source data or regenerate outputs.
+- Do not repair invalid JSON or manually edited state unless a later roadmap
+  defines an explicit recovery flow.
+- Do not claim unmanaged files by default.
+- Do not add unmanaged-file adoption or digest validation in the initial
+  command.
+- Do not add whole-config reconciliation in the initial command; require enough
+  scope to identify one destination root, with all-owner reconciliation allowed
+  inside a configured shared-root destination.
+- Do not expose a producer-facing API for this workflow.