# Reconciliation Roadmap

This document records planned destination reconciliation work that is not part
of the current implementation. Current destination replacement, state, and
recovery behavior is documented in `docs/operations.md`,
`docs/config.md`, and `docs/integrations/destination-state.md`.

This work should be implemented before the shared-root multi-pipeline state
work described in `docs/roadmap/multipipeline.md`. The merge mode described
here is still scoped to one pipeline/destination owner for a destination bundle
path; it does not by itself allow multiple pipelines to share one
`.distributor.json`.

## Destination Reconciliation Modes

Current destination replacement behavior treats each publish as the complete
desired managed output set for the destination bundle path. When a newer source
replaces an older destination state, `distributor` deletes prior managed outputs
that are no longer present in the new plan, then writes the new outputs and
state.

That behavior is useful for snapshot-style producers, but it is not ideal for
incremental producers that publish new reports without retaining prior report
state.

Planned work:

- Add a per-destination reconciliation mode.
- Keep current behavior as the default mode.
- Add a mode that writes new outputs while retaining prior managed outputs that
  are not present in the new plan.
- Keep reconciliation mode per destination. Replacement and merge behavior are
  destination safety policies, and one pipeline may need both a snapshot-style
  `latest` destination and an accumulating `archive` destination from the same
  source bundle.

Proposed configuration:

```yaml
destinations:
  - id: archive
    backend: s3
    bucket: reports
    prefix: weather/morning/archive
    reconciliation:
      mode: merge

  - id: latest
    backend: s3
    bucket: reports
    prefix: weather/morning/latest
    path_mapping:
      mode: fixed
    reconciliation:
      mode: replace
```

Accepted modes:

- `replace`: current behavior. The destination bundle path is reconciled to the
  new planned output set. Prior managed outputs not present in the new plan are
  removed.
- `merge`: new planned outputs are written into the managed destination set.
  Prior managed outputs not present in the new plan are retained.

## State Model

Destination state needs to represent cumulative managed outputs for merge mode.
The current state file already records an `outputs` array, but that array
currently represents the outputs from the most recent publication.

Planned state semantics:

- `source.manifest` continues to record the latest source manifest used for
  destination comparison.
- `outputs` records the complete currently managed output set for the
  destination bundle path.
- In `replace` mode, the new state `outputs` are exactly the newly planned
  outputs.
- In `merge` mode, the new state `outputs` are:
  - previous managed outputs not overwritten by the new plan;
  - plus newly planned outputs.
- Persist reconciliation metadata so state interpretation remains explicit after
  config changes and so later migration, reconcile-state, and prune tooling can
  explain why historical outputs were retained, for example:

```json
{
  "reconciliation": {
    "mode": "merge"
  }
}
```

If state schema changes are required, bump the destination state schema version
and keep parsing/validation rules explicit.

## Safety Rules

Merge mode must not become an unsafe overwrite path.

Required safety behavior:

- A new output may overwrite a prior output only when that path is already
  recorded as distributor-managed in the existing `.distributor.json`.
- A new output path that exists in storage but is not recorded as managed should
  fail as unmanaged content by default.
- Prior managed outputs not included in the new plan are retained in merge mode.
- Prior managed outputs not included in the new plan are deleted in replace
  mode.
- Failed-write cleanup should delete only outputs written by the failed attempt,
  not retained prior managed outputs.
- Forced replacement behavior remains explicit, bounded, dry-runnable, and
  constrained to the destination bundle path.

## Planning And Execution Work

Implementation work:

- Add destination config type `reconciliation.mode`.
- Default `reconciliation.mode` to `replace`.
- Validate mode values as `replace` or `merge`.
- Thread reconciliation mode into publish planning and execution requests.
- During replace execution, preserve current managed cleanup behavior.
- During merge execution:
  - inspect existing state outputs before writing;
  - allow overwrites only for paths already managed by existing state;
  - write new outputs with overwrite enabled only for managed existing paths;
  - retain previous managed outputs that are not overwritten;
  - write destination state with the cumulative managed output set.
- Ensure generated HTML outputs and copied source outputs use the same
  reconciliation semantics.
- Ensure link metadata and primary URL behavior remain deterministic when
  retained outputs exist. Derive `links.primary_url` from the newly planned
  outputs for the current publish, not from retained historical outputs.

## Testing

Important tests:

- Config defaults `reconciliation.mode` to `replace`.
- Config rejects unknown reconciliation modes.
- Replace mode deletes prior managed outputs omitted from the new bundle.
- Merge mode retains prior managed outputs omitted from the new bundle.
- Merge mode overwrites a path already recorded as managed.
- Merge mode fails when a new output path collides with unmanaged destination
  content.
- Merge mode state records the cumulative managed output set.
- Failed merge writes clean up only outputs from the failed attempt.
- Fixed-path destinations support both modes.
- Source-only, generated-HTML-only, and source-plus-HTML publish policies all
  honor reconciliation mode.
- S3, SSH/SFTP, local, and fake backend tests cover any backend overwrite or
  cleanup behavior that changes.

## Documentation Work

When implemented, update current-behavior docs in the same change:

- `docs/config.md`: document `reconciliation.mode`, defaults, and examples.
- `docs/operations.md`: explain replace versus merge behavior and recovery.
- `docs/integrations/destination-state.md`: document any state schema or
  semantic changes.
- `docs/internal/publish.md` and related internal docs: record planning and
  execution invariants.
- Relevant examples under `examples/`, especially archive/latest style
  configurations.

## Boundaries

- Do not add lifecycle retention policies such as keep-latest-N or delete older
  than a duration as part of this work.
- Do not let producers choose reconciliation mode through source manifests or
  upload requests.
- Do not weaken unmanaged-content safety checks.
- Do not make merge mode the default; preserve current replacement behavior for
  existing configurations.
- Do not claim unmanaged colliding files in merge mode. Claiming unmanaged
  content should remain an explicit force or repair workflow, not normal publish
  behavior.