Reconciliation Roadmap

This document records planned destination reconciliation work that is not part of the current implementation. Current destination replacement, state, and recovery behavior is documented in docs/operations.md, docs/config.md, and docs/integrations/destination-state.md.

This work should be implemented before the shared-root multi-pipeline state work described in docs/roadmap/multipipeline.md. The merge mode described here is still scoped to one pipeline/destination owner for a destination bundle path; it does not by itself allow multiple pipelines to share one .distributor.json.

Destination Reconciliation Modes

Current destination replacement behavior treats each publish as the complete desired managed output set for the destination bundle path. When a newer source replaces an older destination state, distributor deletes prior managed outputs that are no longer present in the new plan, then writes the new outputs and state.

That behavior is useful for snapshot-style producers, but it is not ideal for incremental producers that publish new reports without retaining prior report state.

Planned work:

Add a per-destination reconciliation mode.
Keep current behavior as the default mode.
Add a mode that writes new outputs while retaining prior managed outputs that are not present in the new plan.
Keep reconciliation mode per destination. Replacement and merge behavior are destination safety policies, and one pipeline may need both a snapshot-style latest destination and an accumulating archive destination from the same source bundle.

Proposed configuration:

destinations:
  - id: archive
    backend: s3
    bucket: reports
    prefix: weather/morning/archive
    reconciliation:
      mode: merge

  - id: latest
    backend: s3
    bucket: reports
    prefix: weather/morning/latest
    path_mapping:
      mode: fixed
    reconciliation:
      mode: replace

Accepted modes:

replace: current behavior. The destination bundle path is reconciled to the new planned output set. Prior managed outputs not present in the new plan are removed.
merge: new planned outputs are written into the managed destination set. Prior managed outputs not present in the new plan are retained.

State Model

Destination state needs to represent cumulative managed outputs for merge mode. The current state file already records an outputs array, but that array currently represents the outputs from the most recent publication.

Planned state semantics:

source.manifest continues to record the latest source manifest used for destination comparison.
outputs records the complete currently managed output set for the destination bundle path.
In replace mode, the new state outputs are exactly the newly planned outputs.
In merge mode, the new state outputs are:
- previous managed outputs not overwritten by the new plan;
- plus newly planned outputs.
Persist reconciliation metadata so state interpretation remains explicit after config changes and so later migration, reconcile-state, and prune tooling can explain why historical outputs were retained, for example:

{
  "reconciliation": {
    "mode": "merge"
  }
}

If state schema changes are required, bump the destination state schema version and keep parsing/validation rules explicit.

Safety Rules

Merge mode must not become an unsafe overwrite path.

Required safety behavior:

A new output may overwrite a prior output only when that path is already recorded as distributor-managed in the existing .distributor.json.
A new output path that exists in storage but is not recorded as managed should fail as unmanaged content by default.
Prior managed outputs not included in the new plan are retained in merge mode.
Prior managed outputs not included in the new plan are deleted in replace mode.
Failed-write cleanup should delete only outputs written by the failed attempt, not retained prior managed outputs.
Forced replacement behavior remains explicit, bounded, dry-runnable, and constrained to the destination bundle path.

Planning And Execution Work

Implementation work:

Add destination config type reconciliation.mode.
Default reconciliation.mode to replace.
Validate mode values as replace or merge.
Thread reconciliation mode into publish planning and execution requests.
During replace execution, preserve current managed cleanup behavior.
During merge execution:
- inspect existing state outputs before writing;
- allow overwrites only for paths already managed by existing state;
- write new outputs with overwrite enabled only for managed existing paths;
- retain previous managed outputs that are not overwritten;
- write destination state with the cumulative managed output set.
Ensure generated HTML outputs and copied source outputs use the same reconciliation semantics.
Ensure link metadata and primary URL behavior remain deterministic when retained outputs exist. Derive links.primary_url from the newly planned outputs for the current publish, not from retained historical outputs.

Testing

Important tests:

Config defaults reconciliation.mode to replace.
Config rejects unknown reconciliation modes.
Replace mode deletes prior managed outputs omitted from the new bundle.
Merge mode retains prior managed outputs omitted from the new bundle.
Merge mode overwrites a path already recorded as managed.
Merge mode fails when a new output path collides with unmanaged destination content.
Merge mode state records the cumulative managed output set.
Failed merge writes clean up only outputs from the failed attempt.
Fixed-path destinations support both modes.
Source-only, generated-HTML-only, and source-plus-HTML publish policies all honor reconciliation mode.
S3, SSH/SFTP, local, and fake backend tests cover any backend overwrite or cleanup behavior that changes.

Documentation Work

When implemented, update current-behavior docs in the same change:

docs/config.md: document reconciliation.mode, defaults, and examples.
docs/operations.md: explain replace versus merge behavior and recovery.
docs/integrations/destination-state.md: document any state schema or semantic changes.
docs/internal/publish.md and related internal docs: record planning and execution invariants.
Relevant examples under examples/, especially archive/latest style configurations.

Boundaries

Do not add lifecycle retention policies such as keep-latest-N or delete older than a duration as part of this work.
Do not let producers choose reconciliation mode through source manifests or upload requests.
Do not weaken unmanaged-content safety checks.
Do not make merge mode the default; preserve current replacement behavior for existing configurations.
Do not claim unmanaged colliding files in merge mode. Claiming unmanaged content should remain an explicit force or repair workflow, not normal publish behavior.

6.9 KiB Raw Blame History