Document reconciliation modes
This commit is contained in:
@@ -6,15 +6,23 @@ Each managed destination bundle path contains `.distributor.json`. This file is
|
||||
|
||||
## State Schema
|
||||
|
||||
Current schema version: `1`.
|
||||
Current schema version: `2`.
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"schema_version": 2,
|
||||
"distributor_version": "dev",
|
||||
"pipeline_id": "reports",
|
||||
"destination_id": "archive",
|
||||
"published_at": "2026-06-04T12:00:00Z",
|
||||
"created_at": "2026-06-04T12:00:00Z",
|
||||
"updated_at": "2026-06-04T12:00:00Z",
|
||||
"state": {
|
||||
"mode": "single_owner"
|
||||
},
|
||||
"reconciliation": {
|
||||
"mode": "replace"
|
||||
},
|
||||
"source": {
|
||||
"manifest": {
|
||||
"schema_version": 1,
|
||||
@@ -37,7 +45,9 @@ Current schema version: `1`.
|
||||
"transform": "markdown_to_html",
|
||||
"url": "https://reports.example.com/archive/report.html",
|
||||
"sha256": "sha256:...",
|
||||
"size": 2345
|
||||
"size": 2345,
|
||||
"created_at": "2026-06-04T12:00:00Z",
|
||||
"updated_at": "2026-06-04T12:00:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -45,10 +55,14 @@ Current schema version: `1`.
|
||||
|
||||
Required fields:
|
||||
|
||||
- `schema_version`: must be `1`.
|
||||
- `schema_version`: must be `2` for newly written state.
|
||||
- `pipeline_id`: configured pipeline id that wrote the state.
|
||||
- `destination_id`: configured destination id that wrote the state.
|
||||
- `published_at`: RFC3339 publication timestamp.
|
||||
- `created_at`: RFC3339 timestamp for when this destination state record was first created.
|
||||
- `updated_at`: RFC3339 timestamp for the latest state update.
|
||||
- `state.mode`: must be `single_owner`.
|
||||
- `reconciliation.mode`: `replace` or `merge`.
|
||||
- `source.manifest`: embedded source bundle manifest.
|
||||
- `outputs`: output records array; empty is allowed, but the field is required.
|
||||
|
||||
@@ -68,9 +82,24 @@ Each output record has:
|
||||
- `url`: optional absolute HTTP or HTTPS URL for the output.
|
||||
- `sha256`: lowercase `sha256:<64 hex>` digest of the output bytes.
|
||||
- `size`: output byte size, zero or greater.
|
||||
- `created_at`: RFC3339 timestamp for when this output path was first recorded as managed.
|
||||
- `updated_at`: RFC3339 timestamp for when this output path was last written or updated in state.
|
||||
|
||||
Output paths must be unique and use clean relative slash-separated path rules.
|
||||
|
||||
For replacement updates, newly planned outputs are written into state. For merge updates, retained output records preserve both timestamps, overwritten managed output records preserve `created_at` and receive a new `updated_at`, and new output records receive the current publication time for both fields.
|
||||
|
||||
## Reconciliation Semantics
|
||||
|
||||
Destination reconciliation applies when destination state is older than the source and transfer policy permits replacement.
|
||||
|
||||
- `replace`: delete managed output paths recorded in `outputs` plus `.distributor.json`, require the destination bundle path to be empty afterward, write the newly planned outputs, and write state whose `outputs` are exactly that new planned set.
|
||||
- `merge`: retain managed output paths omitted from the new plan, overwrite planned paths only when they are already recorded in existing state, fail when a newly planned path exists in storage but is not recorded as managed, and write state whose `outputs` are the cumulative managed set.
|
||||
|
||||
Top-level `links.primary_url` is selected from the newly planned outputs for the current publication. Retained outputs keep their existing per-output URL metadata.
|
||||
|
||||
If a merge publication fails after writing outputs, cleanup removes only newly created outputs from that failed attempt. Previously managed overwritten paths remain managed and are not removed by failed-attempt cleanup.
|
||||
|
||||
## Comparison Semantics
|
||||
|
||||
`distributor` compares the current source manifest to destination state before writing:
|
||||
@@ -84,7 +113,13 @@ Output paths must be unique and use clean relative slash-separated path rules.
|
||||
- Different source id, pipeline id, or destination id: conflict.
|
||||
- Invalid state JSON or invalid state fields: conflict.
|
||||
|
||||
Normal replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path.
|
||||
Normal replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Merge publication retains omitted managed outputs. Forced replacement deletes the bounded destination bundle path.
|
||||
|
||||
## Compatibility
|
||||
|
||||
`distributor` can read schema version `1` destination state for compatibility. When v1 state is read, it is treated as single-owner state with `reconciliation.mode: replace`. Missing top-level `created_at` and `updated_at` are inferred from `published_at`, and missing per-output timestamps are also inferred from `published_at`.
|
||||
|
||||
Newly written destination state uses schema version `2`.
|
||||
|
||||
## Boundaries
|
||||
|
||||
|
||||
@@ -52,7 +52,7 @@ The adapter uses these S3 operations:
|
||||
|
||||
Writes buffer the input and set `ContentLength`. If no content type is supplied by the caller, the adapter infers a content type from the logical path.
|
||||
|
||||
Normal replacement and failed-write cleanup delete only managed output objects plus `.distributor.json`. Forced replacement deletes objects under the bounded destination bundle prefix. The backend does not manage bucket versioning, lifecycle rules, object lock, or delete markers.
|
||||
Normal replacement and failed-write cleanup delete only managed output objects plus `.distributor.json`. Merge publication retains omitted managed objects and may overwrite existing managed objects. Forced replacement deletes objects under the bounded destination bundle prefix. The backend does not manage bucket versioning, lifecycle rules, object lock, or delete markers.
|
||||
|
||||
## Error Mapping
|
||||
|
||||
|
||||
@@ -54,7 +54,7 @@ The configured `path` is the backend root. All source discovery, destination pat
|
||||
|
||||
The adapter rejects symlink ancestors for reads and writes. Reads require regular files. Writes create parent directories and prefer atomic temp-file-plus-rename writes when requested. Walk output is sorted through the shared storage walker.
|
||||
|
||||
Managed cleanup and normal replacement delete only managed output paths plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path.
|
||||
Managed cleanup and normal replacement delete only managed output paths plus `.distributor.json`. Merge publication retains omitted managed paths and may overwrite existing managed paths. Forced replacement deletes the bounded destination bundle path.
|
||||
|
||||
## Boundaries
|
||||
|
||||
|
||||
Reference in New Issue
Block a user