Document reconciliation modes

This commit is contained in:
2026-06-08 18:36:07 +00:00
parent 93821ea6f9
commit ef0b6c1056
12 changed files with 161 additions and 35 deletions

View File

@@ -56,7 +56,7 @@ Destination path mapping controls where each source bundle is published beneath
Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins.
Published destination bundle paths contain `.distributor.json`. See [Destination State Contract](integrations/destination-state.md). This file is both the managed sentinel and the destination state record. It records the pipeline id, destination id, publication time, source manifest, copied outputs, generated outputs, and optional public URL metadata.
Published destination bundle paths contain `.distributor.json`. See [Destination State Contract](integrations/destination-state.md). This file is both the managed sentinel and the destination state record. It records the pipeline id, destination id, publication time, state mode, reconciliation mode, source manifest, copied outputs, generated outputs, output timestamps, and optional public URL metadata.
`manifest.json` from the source bundle is not copied as destination state.
@@ -71,15 +71,21 @@ Published destination bundle paths contain `.distributor.json`. See [Destination
- Invalid destination state, identity mismatch, different source id, or same-created digest mismatch: fail by default.
- Content without `.distributor.json`: fail as unmanaged content by default.
Normal replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, then verifies the destination bundle path is empty before writing new outputs and state.
When destination state is older than the source, `transfer.on_destination_older` controls whether publication may proceed and `reconciliation.mode` controls how managed outputs are updated.
If a write fails after some outputs were written, `distributor` attempts to delete outputs from that failed attempt so a retry does not treat partial outputs as unmanaged content. Operators should still inspect the destination after a failed write before retrying.
`reconciliation.mode: replace` is the default. It deletes only managed output paths recorded in `.distributor.json` plus the state file, verifies the destination bundle path is empty, then writes the newly planned outputs and state. The new state `outputs` array is exactly the newly planned output set.
`reconciliation.mode: merge` retains prior managed outputs that are omitted from the new plan. It overwrites planned paths only when those paths are already recorded in existing state as managed. If a newly planned path already exists in storage but is not recorded in state, publication fails as an unmanaged path collision. The new state `outputs` array is the cumulative managed output set.
For both modes, retained or overwritten paths are identified only from `.distributor.json`; unmanaged files are not adopted.
If a write fails after some outputs were written, `distributor` attempts cleanup before returning the error. In `replace` mode, cleanup removes outputs written during that failed attempt. In `merge` mode, cleanup removes only newly created outputs from that failed attempt; overwritten managed outputs are left in place because they previously belonged to the managed set. Operators should still inspect the destination after a failed write before retrying.
Fan-out destinations are independent. If one destination fails after planning or execution begins, later destinations are still attempted. The command exits non-zero if any destination failed.
## Dry Runs And Output Review
`run --dry-run` loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, `.distributor.json`, or SSH `known_hosts` entries.
`run --dry-run` loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, `.distributor.json`, or SSH `known_hosts` entries. For reconciliation, dry runs report the same high-level action labels as execution; inspect the configured destination's `reconciliation.mode` to determine whether `replace_older` will replace the managed set or merge into it.
Review these action labels before publishing:
@@ -194,7 +200,7 @@ S3 execution uses the AWS SDK for Go v2. See [S3-Compatible Storage Integration]
When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is.
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Merge publication retains omitted managed objects and may overwrite existing managed objects. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
## Secrets Operation
@@ -209,7 +215,7 @@ Use these recovery boundaries:
- For source validation failures, regenerate the source bundle and manifest together.
- For an empty or missing destination, rerun after fixing config or storage access.
- For unmanaged destination content, move unrelated files aside or use a different destination path before publishing.
- For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun `--dry-run`.
- For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun `--dry-run`. In merge mode, retained outputs may be intentional managed outputs from the prior state.
- For state conflicts, verify the source, pipeline, destination, and existing `.distributor.json` before considering `--force`.
- For HTTP upload failures, inspect `/runs/<run-id>` while retained; after expiry or restart, rely on destination state and logs/output from the publishing run.