distributor/docs/roadmap/implementation.md

# Reconciliation, Shared-Root, Reconcile-State, And Prune Implementation Plan

This document is the staged implementation plan for the feature roadmaps in:

- `docs/roadmap/reconciliation.md`
- `docs/roadmap/multipipeline.md`
- `docs/roadmap/reconcile-state.md`
- `docs/roadmap/prune.md`

Audience: LLM coding agents and maintainers implementing these features in
order.

This is roadmap guidance, not current behavior documentation. Do not describe
any stage as implemented outside `docs/roadmap/` until that stage has landed.

## Global Decisions

- Implement in this order:
  1. destination reconciliation modes;
  2. shared-root multi-pipeline state;
  3. reconcile-state tooling;
  4. pruning.
- Keep reconciliation mode per destination as `reconciliation.mode`.
- Use reconciliation modes `replace` and `merge`; default is `replace`.
- Persist reconciliation metadata in `.distributor.json`.
- Keep shared-root behavior as an explicit per-destination opt-in using
  `state.mode: single_owner|shared_root`.
- Key shared-root output ownership by both `pipeline_id` and `destination_id`.
- Do not allow implicit takeover of another owner path.
- Store compact per-output source identity fields, and keep the latest full
  source manifest once per owner where publish comparison needs it.
- `reconcile-state` applies repairs by default and uses `--dry-run` for
  read-only inspection.
- `prune` is opt-in, uses configured retention policy, and requires `--apply`
  for deletion.
- Use per-output `updated_at` for initial prune policies.
- Support both `older_than` and `keep_latest` pruning in the initial prune
  implementation.
- Write single-owner reconciliation-capable state as schema version `2`.
- Write shared-root collection state as schema version `3`.
- Do not adopt unmanaged files, validate output digests, or perform
  whole-config reconciliation in the initial reconcile-state command.
- Do not run pruning automatically after publish in the initial implementation.

## Cross-Cutting Constraints

- Follow `docs/policy/architecture.md`, `docs/policy/development.md`, and
  `docs/policy/documentation.md`.
- Keep producer manifests free of destination routing, reconciliation, state,
  retention, transform, link, and backend policy.
- Keep adapter packages thin. State comparison, reconciliation, pruning, and
  retention decisions belong in `internal/state`, `internal/publish`, and
  `internal/app`, not concrete backends.
- Keep public packages `pkg/bundle` and `pkg/upload` unaffected unless an
  implementation stage explicitly requires consumer-facing docs updates.
- Use dry-run behavior for every workflow that may write state or delete files.
- Never delete unmanaged content.
- Use `storage.StateFileName`, `storage.StatePath`,
  `storage.ManagedBundleTargets`, and storage path validation helpers instead
  of duplicating path rules.
- Update implemented-behavior docs and examples in the same change as the
  corresponding behavior. Keep future behavior in roadmap docs only.
- A feature group is not complete until its paired documentation stage lands.
  Do not move the roadmap item out of future/planned status until the behavior,
  tests, current-behavior docs, and examples for that group are complete.
- Run the stage-specific tests before moving to the next stage. Run
  `go test ./...` after each completed feature group.

## Stage 1: Reconciliation Config And State Foundation

Goal: add destination-level reconciliation configuration and a state model that
can persist reconciliation metadata and cumulative outputs for single-owner
destinations.

Implementation:

- Add config types:
  - `Destination.Reconciliation ReconciliationPolicy`
  - `ReconciliationPolicy.Mode string`
- Add constants in `internal/config/defaults.go`:
  - `ReconciliationModeReplace = "replace"`
  - `ReconciliationModeMerge = "merge"`
- Default `reconciliation.mode` to `replace` for every destination.
- Validate `reconciliation.mode` as `replace` or `merge`.
- Add focused config tests for defaults, explicit `replace`, explicit `merge`,
  and invalid modes.
- Extend `internal/state` for a new single-owner state schema that writes:
  - `schema_version`;
  - `distributor_version`;
  - `pipeline_id`;
  - `destination_id`;
  - `published_at`;
  - `created_at`;
  - `updated_at`;
  - `state.mode: "single_owner"`;
  - `reconciliation.mode`;
  - `source.manifest`;
  - `links`;
  - cumulative `outputs`.
- Add per-output `created_at` and `updated_at` fields now. For current
  single-owner output records written by this stage, set both timestamps when a
  path is first created and preserve `created_at` when a path is updated.
- Keep parsing current schema version 1 state for compatibility. When v1 state
  is read, infer:
  - `state.mode = "single_owner"`;
  - `reconciliation.mode = "replace"`;
  - `created_at = published_at`;
  - `updated_at = published_at`;
  - each output `created_at` and `updated_at` from `published_at`.
- Write only the new schema for newly written state after this stage.
- Keep validation strict: required timestamps, valid reconciliation mode, unique
  output paths, valid links, valid digests, clean paths, and valid embedded
  source manifest.
- Add state helper functions for:
  - finding output records by path;
  - merging retained and newly planned output records;
  - computing owner-scoped managed output paths for single-owner state;
  - projecting publish outputs into state outputs with timestamps.

Tests:

```sh
go test ./internal/config
go test ./internal/state
```

Keep publish semantics unchanged in this stage. Only adapt current state
construction to write schema version `2` with default `replace` metadata and
timestamps.

## Stage 2: Single-Owner Replace And Merge Publish Execution

Goal: implement `reconciliation.mode` for current single-owner destinations.

Implementation:

- Add `Reconciliation config.ReconciliationPolicy` to `publish.Request` and
  `publish.Plan`.
- Thread destination reconciliation config from `internal/app` into
  `internal/publish`.
- Keep `replace` behavior equivalent to current behavior:
  - delete prior managed outputs absent from the new plan;
  - write the new outputs;
  - write state whose `outputs` are exactly the new planned output set.
- Implement `merge` behavior:
  - inspect existing state before writing;
  - retain previous managed outputs not overwritten by the new plan;
  - allow overwrite only when the destination path is already recorded in
    existing state as managed;
  - fail if a newly planned path exists in storage but is not recorded as
    managed;
  - write new or overwritten outputs;
  - write state with the cumulative managed output set;
  - derive `links.primary_url` from newly planned outputs only;
  - clean up only outputs written by the failed attempt.
- For `merge`, update per-output timestamps:
  - new output path: `created_at = updated_at = now`;
  - overwritten managed path: preserve existing `created_at`, set
    `updated_at = now`;
  - retained output: preserve both timestamps.
- Ensure state writes use overwrite semantics appropriate for replacing the
  existing `.distributor.json`, while output writes remain narrow and explicit.
- Preserve existing force behavior. Do not add unmanaged-content adoption.

Tests:

```sh
go test ./internal/publish ./internal/state
go test ./internal/app ./internal/cli
go test ./...
```

Required coverage:

- `replace` deletes omitted managed outputs.
- `merge` retains omitted managed outputs.
- `merge` overwrites an existing managed path.
- `merge` fails on unmanaged destination path collision.
- failed merge cleanup deletes only attempted new writes.
- fixed-path destinations support both modes.
- source-only, generated-only, and source-plus-generated policies honor both
  modes.

## Stage 3: Reconciliation Documentation And Examples

Goal: make implemented reconciliation behavior visible outside roadmap docs.
The reconciliation feature group is not complete until this stage lands.

Implementation:

- Update `docs/config.md` with `reconciliation.mode`, defaults, and examples.
- Update `docs/operations.md` with replace versus merge behavior, dry-run
  expectations, failed-write cleanup, and recovery notes.
- Update `docs/integrations/destination-state.md` with the new state schema,
  reconciliation metadata, output timestamps, and v1 compatibility behavior.
- Update `docs/internal/state.md` and `docs/internal/publish.md`.
- Update relevant examples under `examples/` only if they are valid and
  executable for implemented behavior.

Tests:

```sh
go test ./internal/config
go test ./...
```

## Stage 4: Shared-Root State Model And Config

Goal: add explicit shared-root destination state support without changing
publish behavior yet.

Implementation:

- Add destination state config:
  - `Destination.State StatePolicy`
  - `StatePolicy.Mode string`
  - accepted values: `single_owner`, `shared_root`
  - default: `single_owner`
- Validate state mode values.
- Add a shared-root state schema in `internal/state`.
- The shared-root JSON must include:
  - `schema_version`;
  - `distributor_version`;
  - `created_at`;
  - `updated_at`;
  - `state.mode: "shared_root"`;
  - owner records keyed by `pipeline_id` and `destination_id`;
  - latest full source manifest per owner for comparison;
  - owner-level reconciliation metadata;
  - owner-level latest `links.primary_url`, if present;
  - output records for every managed path in the root.
- Shared-root output records must include:
  - path;
  - kind;
  - source path;
  - transform, when generated;
  - URL, when present;
  - SHA-256;
  - size;
  - owner `pipeline_id`;
  - owner `destination_id`;
  - source manifest id;
  - source manifest digest;
  - source manifest created timestamp;
  - `created_at`;
  - `updated_at`.
- Keep compact per-output source identity fields. Do not repeat full source
  manifests on every output.
- Add state helpers that can:
  - parse both single-owner and shared-root state;
  - identify the current owner scope;
  - return the latest source manifest for one owner;
  - list managed paths for one owner or all owners;
  - detect path ownership conflicts;
  - merge one owner's planned output state while preserving unrelated owners;
  - remove one owner's omitted outputs for owner-scoped `replace`.
- Do not automatically convert unrelated single-owner state to shared-root
  state. If shared-root is configured and existing single-owner state matches
  the current owner, convert it during the first successful shared-root publish.
  Otherwise fail with an actionable conflict.

Tests:

```sh
go test ./internal/config
go test ./internal/state
```

Do not enable shared-root publish behavior until Stage 5.

## Stage 5: Shared-Root Publish Planning And Comparison

Goal: teach publish planning to reason about shared-root owner scopes without
writing shared-root outputs yet.

Implementation:

- Thread destination state mode from `internal/app` into `internal/publish`.
- Update destination inspection and comparison so:
  - `single_owner` destinations preserve current single-owner comparison
    semantics;
  - `shared_root` destinations compare only the current
    `pipeline_id`/`destination_id` owner scope;
  - absent owner state behaves like destination absent for that owner, unless
    unmanaged storage content collides with a planned output;
  - outputs owned by other owners are preserved.
- Enforce ownership conflict rules:
  - same `pipeline_id` and `destination_id` may overwrite its own managed path;
  - different owner writing the same path fails as a conflict;
  - unmanaged storage path collision fails by default;
  - no implicit takeover.
- Compose reconciliation within the owner scope:
  - owner-scoped `replace` removes prior outputs for that owner that are absent
    from the new plan;
  - owner-scoped `merge` retains prior outputs for that owner that are absent
    from the new plan.
- Return plans that include the current owner scope, retained owner outputs,
  owner outputs to delete, and owner outputs to write.
- Keep execution disabled for shared-root write actions until Stage 6.

Tests:

```sh
go test ./internal/publish ./internal/state
go test ./internal/app
```

Required coverage:

- planning treats absent owner state as publishable for that owner;
- planning preserves other owners in the planned retained state;
- path conflict with another owner plans or returns a conflict failure;
- unmanaged path collision plans or returns an unmanaged-content failure;
- owner-scoped `replace` selects only that owner's omitted outputs for deletion;
- owner-scoped `merge` retains that owner's omitted outputs;
- current single-owner planning remains covered.

## Stage 6: Shared-Root Publish Execution

Goal: allow multiple pipelines to publish disjoint managed paths into one
shared destination root.

Implementation:

- Preserve unrelated owners and their outputs exactly.
- Preserve and update timestamps:
  - top-level `created_at` remains the original root creation time;
  - top-level `updated_at` changes on successful state writes;
  - output `created_at` is stable;
  - output `updated_at` changes only when that path is rewritten;
  - owner metadata updates when that owner publishes.
- Keep failed-write cleanup limited to outputs written by the failed attempt.
- Keep forced replacement bounded and explicitly reported. Forced replacement
  for a shared-root destination may delete the configured destination root, so
  retain existing dry-run visibility and do not broaden its scope.

Tests:

```sh
go test ./internal/publish ./internal/state
go test ./internal/app ./internal/cli
go test ./...
```

Required coverage:

- two pipelines publish disjoint paths into the same shared root;
- one pipeline replaces its own output without deleting another owner's output;
- one pipeline merges new outputs while preserving its own retained outputs and
  other owners' outputs;
- path conflict with another owner fails;
- unmanaged path collision fails;
- shared-root dry-run writes no outputs or state;
- current single-owner behavior remains covered.

## Stage 7: Shared-Root Documentation And Examples

Goal: document implemented shared-root behavior.
The shared-root feature group is not complete until this stage lands.

Implementation:

- Update `docs/config.md` with destination state mode config and defaults.
- Update `docs/integrations/destination-state.md` with shared-root schema,
  owners, output ownership, timestamps, and migration rules.
- Update `docs/operations.md` with shared-root publishing, conflicts, dry-run,
  and recovery behavior.
- Update `docs/troubleshooting.md` with ownership conflict and unmanaged
  collision entries.
- Update `docs/internal/state.md` and `docs/internal/publish.md`.
- Add maintained examples only when they are valid and load-tested.

Tests:

```sh
go test ./internal/config
go test ./...
```

## Stage 8: Reconcile-State Planning Core

Goal: add app-level planning and state helpers for reconcile-state without
adding CLI execution first.

Implementation:

- Add `internal/app` reconcile-state use case types:
  - options with config path, pipeline id, destination id, all-owners flag,
    dry-run flag, stdout, and output format;
  - report struct with destination identity, backend, root path, state schema,
    owner scope, checked count, missing managed outputs, unmanaged entries, and
    changed/would-change status.
- Add state helpers for removing missing managed output records:
  - single-owner scope;
  - shared-root current-owner scope;
  - shared-root all-owner scope.
- Use storage `Stat` for managed output existence checks.
- Use bounded `Walk` under the selected destination root to report unmanaged
  existing files. Exclude `.distributor.json` and all paths already recorded as
  managed.
- Do not validate output digests.
- Do not delete destination files.
- Do not adopt unmanaged files.
- Do not rewrite invalid or ambiguous state.
- Require enough scope to identify one destination root. For current
  single-owner state, require pipeline and destination. For shared-root
  all-owner mode, still require a pipeline/destination selector to identify the
  configured destination root, then apply `--all-owners` inside that root.
- Default behavior applies state repair. `DryRun` reports without writing.

Tests:

```sh
go test ./internal/app ./internal/state ./internal/storage/fake
```

Required coverage:

- dry-run reports missing managed outputs without rewriting state;
- apply removes missing records and writes valid state;
- unmanaged files are reported but not adopted or deleted;
- invalid state fails without rewrite;
- shared-root owner-scope and all-owner repair both work.

## Stage 9: Reconcile-State CLI And Docs

Goal: expose reconcile-state as an operator command.
The reconcile-state feature group is not complete until this stage lands.

Implementation:

- Add `reconcile-state` to `internal/cli/root.go` help and dispatch.
- Add CLI flags:
  - `--config <path>`;
  - `--pipeline <id>`;
  - `--destination <id>`;
  - `--all-owners`;
  - `--dry-run`;
  - `--format text|json`.
- Without `--dry-run`, apply state repairs.
- Text output should clearly report whether state was changed or would be
  changed.
- JSON output should use the existing app JSON envelope pattern.
- Update `docs/cli.md`, `docs/operations.md`, `docs/troubleshooting.md`,
  `docs/integrations/destination-state.md`, `docs/internal/state.md`,
  `docs/internal/storage.md`, and `docs/internal/app.md`.

Tests:

```sh
go test ./internal/cli ./internal/app
go test ./...
```

Required coverage:

- flag validation;
- default apply behavior;
- `--dry-run` behavior;
- text and JSON reports;
- no deletion of destination files.

## Stage 10: Prune Config And Planning Core

Goal: add retention config and pure prune planning.

Implementation:

- Add destination retention config:
  - `Destination.Retention RetentionPolicy`
  - `RetentionPolicy.Prune PrunePolicy`
  - `PrunePolicy.Enabled bool`
  - `PrunePolicy.OlderThan *Duration`
  - `PrunePolicy.KeepLatest *int`
- Default pruning disabled.
- Validate:
  - enabled prune policy must set at least one of `older_than` or
    `keep_latest`;
  - `older_than` must be positive;
  - `keep_latest` must be zero or greater;
  - documented duration units must parse consistently with existing duration
    config behavior.
- Add app/state prune planning helpers:
  - select owner-scoped managed outputs;
  - sort deterministically by `updated_at`, then path;
  - preserve the newest `keep_latest` outputs;
  - delete remaining outputs older than `older_than`;
  - if both policies are set, first preserve `keep_latest`, then delete
    remaining outputs older than `older_than`.
- Use per-output `updated_at` as the timestamp basis.
- Plan only; do not delete storage files in this stage.

Tests:

```sh
go test ./internal/config
go test ./internal/app ./internal/state
```

Required coverage:

- config defaults and validation;
- `older_than` planning;
- `keep_latest` planning;
- combined policy planning;
- deterministic tie-breaking;
- shared-root owner scope preservation.

## Stage 11: Prune Execution Core

Goal: implement safe managed-output pruning below the CLI layer.

Implementation:

- Add app prune execution:
  - load config;
  - resolve selected destination and owner scope;
  - read and validate state;
  - build prune plan from configured retention policy;
  - in dry-run, write no files and delete nothing;
  - in apply mode, delete only planned managed output paths;
  - after confirmed deletes, remove deleted records from state and update state
    timestamps;
  - preserve state records for failed deletes so retry remains accurate;
  - never delete unmanaged files;
  - do not delete `.distributor.json` unless a later roadmap explicitly permits
    empty-state removal.
- Do not add CLI dispatch in this stage.
- Return a prune report that can later be rendered by text and JSON CLI output.
- Do not add one-off retention overrides in the initial implementation.
- Let local and SSH/SFTP backend helpers prune empty directories only where
  they already do so safely. Preserve object-storage prefix markers unless they
  are managed output records.

Tests:

```sh
go test ./internal/config
go test ./internal/app ./internal/state
go test ./internal/storage/fake
```

Required coverage:

- dry-run reports deletes without deleting files or rewriting state;
- apply deletes only managed output paths;
- unmanaged files under the destination root are preserved;
- partial delete failure preserves accurate state for confirmed deleted and
  undeleted outputs;
- shared-root pruning preserves other owners when scoped to one owner.

## Stage 12: Prune CLI And Docs

Goal: expose safe managed-output pruning as an operator command. The prune
feature group is not complete until this stage lands.

Implementation:

- Add `prune` CLI dispatch and help.
- Add CLI flags:
  - `--config <path>`;
  - `--pipeline <id>`;
  - `--destination <id>`;
  - `--dry-run`;
  - `--apply`;
  - `--format text|json`.
- Require exactly one of `--dry-run` or `--apply`. `--dry-run` plans and
  reports without writing. `--apply` deletes planned managed outputs and
  rewrites state after confirmed deletes.
- JSON output should use the existing app JSON envelope pattern.
- Update `docs/config.md`, `docs/cli.md`, `docs/operations.md`,
  `docs/troubleshooting.md`, `docs/integrations/destination-state.md`,
  `docs/internal/state.md`, `docs/internal/publish.md`, and
  `docs/internal/app.md`.

Tests:

```sh
go test ./internal/config
go test ./internal/app ./internal/cli ./internal/state
go test ./internal/storage/fake
go test ./...
```

Required coverage:

- CLI requires exactly one of `--dry-run` or `--apply`;
- text and JSON reports;
- CLI apply gating and report output.

## Final Verification

After all stages land:

```sh
go test ./internal/config
go test ./internal/state ./internal/publish
go test ./internal/app ./internal/cli
go test ./internal/storage/fake
go test ./pkg/bundle ./pkg/upload
go test ./...
```

Also run:

```sh
rg -n "reconciliation|shared_root|single_owner|reconcile-state|retention|prune|older_than|keep_latest" README.md docs examples
```

Confirm:

- current-behavior docs outside `docs/roadmap/` describe only implemented
  behavior;
- roadmap docs no longer describe completed behavior as future work, or are
  revised to mark only remaining deferred work;
- examples load successfully where config tests cover them;
- producer-facing docs still keep destination routing, reconciliation, state,
  and retention policy out of producer manifests and upload requests.

## Non-Goals For This Plan

- No unmanaged-file adoption workflow.
- No digest-audit mode for reconcile-state.
- No whole-config reconcile-state command.
- No automatic post-publish pruning.
- No path/date parsing retention policy.
- No producer-facing retention, reconciliation, or destination routing API.
- No broad storage synchronization behavior.