Document reconciliation modes
This commit is contained in:
@@ -35,7 +35,7 @@ pipelines:
|
|||||||
path: /srv/reports/archive
|
path: /srv/reports/archive
|
||||||
```
|
```
|
||||||
|
|
||||||
This config publishes source files only. It uses default validation, destination path mapping, publish, transfer, and HTTP server values.
|
This config publishes source files only. It uses default validation, destination path mapping, publish, reconciliation, transfer, and HTTP server values.
|
||||||
|
|
||||||
## Production-Oriented Local Config
|
## Production-Oriented Local Config
|
||||||
|
|
||||||
@@ -66,6 +66,8 @@ pipelines:
|
|||||||
html: false
|
html: false
|
||||||
path_mapping:
|
path_mapping:
|
||||||
mode: preserve_relative
|
mode: preserve_relative
|
||||||
|
reconciliation:
|
||||||
|
mode: replace
|
||||||
transfer:
|
transfer:
|
||||||
on_destination_same: skip
|
on_destination_same: skip
|
||||||
on_destination_older: replace
|
on_destination_older: replace
|
||||||
@@ -253,7 +255,7 @@ Source bundle digest mismatches fail validation before destination writes occur.
|
|||||||
|
|
||||||
## Destination Fields
|
## Destination Fields
|
||||||
|
|
||||||
Each destination embeds a backend config at the destination level and may also configure publishing, transforms, path mapping, links, and transfer behavior.
|
Each destination embeds a backend config at the destination level and may also configure publishing, transforms, path mapping, links, reconciliation, and transfer behavior.
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
destinations:
|
destinations:
|
||||||
@@ -265,6 +267,8 @@ destinations:
|
|||||||
html: false
|
html: false
|
||||||
path_mapping:
|
path_mapping:
|
||||||
mode: preserve_relative
|
mode: preserve_relative
|
||||||
|
reconciliation:
|
||||||
|
mode: replace
|
||||||
transfer:
|
transfer:
|
||||||
on_destination_same: skip
|
on_destination_same: skip
|
||||||
on_destination_older: replace
|
on_destination_older: replace
|
||||||
@@ -278,7 +282,8 @@ destinations:
|
|||||||
- `transform`: required only when publishing generated HTML.
|
- `transform`: required only when publishing generated HTML.
|
||||||
- `path_mapping`: optional destination path mapping policy.
|
- `path_mapping`: optional destination path mapping policy.
|
||||||
- `links`: optional public URL metadata policy.
|
- `links`: optional public URL metadata policy.
|
||||||
- `transfer`: optional destination reconciliation policy.
|
- `reconciliation`: optional managed-output reconciliation policy.
|
||||||
|
- `transfer`: optional destination comparison action policy.
|
||||||
|
|
||||||
Destination ids must be unique within a pipeline.
|
Destination ids must be unique within a pipeline.
|
||||||
|
|
||||||
@@ -354,6 +359,41 @@ Primary URL policies:
|
|||||||
|
|
||||||
If no output matches the primary policy, per-output URLs may still be recorded and the top-level primary URL is omitted.
|
If no output matches the primary policy, per-output URLs may still be recorded and the top-level primary URL is omitted.
|
||||||
|
|
||||||
|
## Reconciliation Policy
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
reconciliation:
|
||||||
|
mode: replace
|
||||||
|
```
|
||||||
|
|
||||||
|
- `reconciliation.mode`: optional. Accepted values are `replace` and `merge`; default is `replace`.
|
||||||
|
|
||||||
|
Reconciliation controls how a destination with older managed state is updated after transfer policy selects `replace_older`.
|
||||||
|
|
||||||
|
`replace` deletes the prior managed outputs recorded in `.distributor.json`, deletes the state file, verifies the destination bundle path is empty, then writes only the newly planned outputs and a new state file. This is the default and is appropriate when each publication should exactly match the current publish and transform policy.
|
||||||
|
|
||||||
|
`merge` keeps prior managed outputs that are not produced by the new plan. Planned paths already recorded in existing state may be overwritten; planned paths that already exist in storage but are not recorded as managed fail as unmanaged collisions. The resulting state file records the cumulative managed output set.
|
||||||
|
|
||||||
|
Example merge destination:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
destinations:
|
||||||
|
- id: static-site
|
||||||
|
backend: local
|
||||||
|
path: /srv/reports/site
|
||||||
|
publish:
|
||||||
|
source: false
|
||||||
|
html: true
|
||||||
|
transform:
|
||||||
|
markdown_to_html:
|
||||||
|
enabled: true
|
||||||
|
mode: sidecar
|
||||||
|
reconciliation:
|
||||||
|
mode: merge
|
||||||
|
```
|
||||||
|
|
||||||
|
`links.primary_url` is selected from the newly planned outputs for the current run. Retained outputs keep their prior output metadata and timestamps.
|
||||||
|
|
||||||
## Transfer Policy
|
## Transfer Policy
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
@@ -408,6 +448,7 @@ Defaults are applied after YAML decoding and before validation:
|
|||||||
- `transform.markdown_to_html.mode: sidecar` when a Markdown transform block is present and mode is omitted
|
- `transform.markdown_to_html.mode: sidecar` when a Markdown transform block is present and mode is omitted
|
||||||
- `path_mapping.mode: preserve_relative`
|
- `path_mapping.mode: preserve_relative`
|
||||||
- `links.primary: auto` when a `links` block is present and `primary` is omitted
|
- `links.primary: auto` when a `links` block is present and `primary` is omitted
|
||||||
|
- `reconciliation.mode: replace`
|
||||||
- `transfer.on_destination_same: skip`
|
- `transfer.on_destination_same: skip`
|
||||||
- `transfer.on_destination_older: replace`
|
- `transfer.on_destination_older: replace`
|
||||||
- `transfer.on_destination_newer: skip`
|
- `transfer.on_destination_newer: skip`
|
||||||
@@ -444,6 +485,7 @@ Local examples:
|
|||||||
- `local-index.yml`: local `index.html` publication.
|
- `local-index.yml`: local `index.html` publication.
|
||||||
- `fan-out.yml`: local fan-out publication to source and HTML destinations.
|
- `fan-out.yml`: local fan-out publication to source and HTML destinations.
|
||||||
- `archive-and-latest.yml`: local archive plus fixed latest publication.
|
- `archive-and-latest.yml`: local archive plus fixed latest publication.
|
||||||
|
- `merge-reconciliation.yml`: local HTML publication using merge reconciliation.
|
||||||
- `http-upload-local.yml`: local HTTP upload server config; requires `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN` in the process environment or as a secret-file name before running `serve`.
|
- `http-upload-local.yml`: local HTTP upload server config; requires `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN` in the process environment or as a secret-file name before running `serve`.
|
||||||
|
|
||||||
Environment-gated remote examples:
|
Environment-gated remote examples:
|
||||||
|
|||||||
@@ -6,15 +6,23 @@ Each managed destination bundle path contains `.distributor.json`. This file is
|
|||||||
|
|
||||||
## State Schema
|
## State Schema
|
||||||
|
|
||||||
Current schema version: `1`.
|
Current schema version: `2`.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"schema_version": 1,
|
"schema_version": 2,
|
||||||
"distributor_version": "dev",
|
"distributor_version": "dev",
|
||||||
"pipeline_id": "reports",
|
"pipeline_id": "reports",
|
||||||
"destination_id": "archive",
|
"destination_id": "archive",
|
||||||
"published_at": "2026-06-04T12:00:00Z",
|
"published_at": "2026-06-04T12:00:00Z",
|
||||||
|
"created_at": "2026-06-04T12:00:00Z",
|
||||||
|
"updated_at": "2026-06-04T12:00:00Z",
|
||||||
|
"state": {
|
||||||
|
"mode": "single_owner"
|
||||||
|
},
|
||||||
|
"reconciliation": {
|
||||||
|
"mode": "replace"
|
||||||
|
},
|
||||||
"source": {
|
"source": {
|
||||||
"manifest": {
|
"manifest": {
|
||||||
"schema_version": 1,
|
"schema_version": 1,
|
||||||
@@ -37,7 +45,9 @@ Current schema version: `1`.
|
|||||||
"transform": "markdown_to_html",
|
"transform": "markdown_to_html",
|
||||||
"url": "https://reports.example.com/archive/report.html",
|
"url": "https://reports.example.com/archive/report.html",
|
||||||
"sha256": "sha256:...",
|
"sha256": "sha256:...",
|
||||||
"size": 2345
|
"size": 2345,
|
||||||
|
"created_at": "2026-06-04T12:00:00Z",
|
||||||
|
"updated_at": "2026-06-04T12:00:00Z"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
@@ -45,10 +55,14 @@ Current schema version: `1`.
|
|||||||
|
|
||||||
Required fields:
|
Required fields:
|
||||||
|
|
||||||
- `schema_version`: must be `1`.
|
- `schema_version`: must be `2` for newly written state.
|
||||||
- `pipeline_id`: configured pipeline id that wrote the state.
|
- `pipeline_id`: configured pipeline id that wrote the state.
|
||||||
- `destination_id`: configured destination id that wrote the state.
|
- `destination_id`: configured destination id that wrote the state.
|
||||||
- `published_at`: RFC3339 publication timestamp.
|
- `published_at`: RFC3339 publication timestamp.
|
||||||
|
- `created_at`: RFC3339 timestamp for when this destination state record was first created.
|
||||||
|
- `updated_at`: RFC3339 timestamp for the latest state update.
|
||||||
|
- `state.mode`: must be `single_owner`.
|
||||||
|
- `reconciliation.mode`: `replace` or `merge`.
|
||||||
- `source.manifest`: embedded source bundle manifest.
|
- `source.manifest`: embedded source bundle manifest.
|
||||||
- `outputs`: output records array; empty is allowed, but the field is required.
|
- `outputs`: output records array; empty is allowed, but the field is required.
|
||||||
|
|
||||||
@@ -68,9 +82,24 @@ Each output record has:
|
|||||||
- `url`: optional absolute HTTP or HTTPS URL for the output.
|
- `url`: optional absolute HTTP or HTTPS URL for the output.
|
||||||
- `sha256`: lowercase `sha256:<64 hex>` digest of the output bytes.
|
- `sha256`: lowercase `sha256:<64 hex>` digest of the output bytes.
|
||||||
- `size`: output byte size, zero or greater.
|
- `size`: output byte size, zero or greater.
|
||||||
|
- `created_at`: RFC3339 timestamp for when this output path was first recorded as managed.
|
||||||
|
- `updated_at`: RFC3339 timestamp for when this output path was last written or updated in state.
|
||||||
|
|
||||||
Output paths must be unique and use clean relative slash-separated path rules.
|
Output paths must be unique and use clean relative slash-separated path rules.
|
||||||
|
|
||||||
|
For replacement updates, newly planned outputs are written into state. For merge updates, retained output records preserve both timestamps, overwritten managed output records preserve `created_at` and receive a new `updated_at`, and new output records receive the current publication time for both fields.
|
||||||
|
|
||||||
|
## Reconciliation Semantics
|
||||||
|
|
||||||
|
Destination reconciliation applies when destination state is older than the source and transfer policy permits replacement.
|
||||||
|
|
||||||
|
- `replace`: delete managed output paths recorded in `outputs` plus `.distributor.json`, require the destination bundle path to be empty afterward, write the newly planned outputs, and write state whose `outputs` are exactly that new planned set.
|
||||||
|
- `merge`: retain managed output paths omitted from the new plan, overwrite planned paths only when they are already recorded in existing state, fail when a newly planned path exists in storage but is not recorded as managed, and write state whose `outputs` are the cumulative managed set.
|
||||||
|
|
||||||
|
Top-level `links.primary_url` is selected from the newly planned outputs for the current publication. Retained outputs keep their existing per-output URL metadata.
|
||||||
|
|
||||||
|
If a merge publication fails after writing outputs, cleanup removes only newly created outputs from that failed attempt. Previously managed overwritten paths remain managed and are not removed by failed-attempt cleanup.
|
||||||
|
|
||||||
## Comparison Semantics
|
## Comparison Semantics
|
||||||
|
|
||||||
`distributor` compares the current source manifest to destination state before writing:
|
`distributor` compares the current source manifest to destination state before writing:
|
||||||
@@ -84,7 +113,13 @@ Output paths must be unique and use clean relative slash-separated path rules.
|
|||||||
- Different source id, pipeline id, or destination id: conflict.
|
- Different source id, pipeline id, or destination id: conflict.
|
||||||
- Invalid state JSON or invalid state fields: conflict.
|
- Invalid state JSON or invalid state fields: conflict.
|
||||||
|
|
||||||
Normal replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path.
|
Normal replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Merge publication retains omitted managed outputs. Forced replacement deletes the bounded destination bundle path.
|
||||||
|
|
||||||
|
## Compatibility
|
||||||
|
|
||||||
|
`distributor` can read schema version `1` destination state for compatibility. When v1 state is read, it is treated as single-owner state with `reconciliation.mode: replace`. Missing top-level `created_at` and `updated_at` are inferred from `published_at`, and missing per-output timestamps are also inferred from `published_at`.
|
||||||
|
|
||||||
|
Newly written destination state uses schema version `2`.
|
||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
|
|||||||
@@ -52,7 +52,7 @@ The adapter uses these S3 operations:
|
|||||||
|
|
||||||
Writes buffer the input and set `ContentLength`. If no content type is supplied by the caller, the adapter infers a content type from the logical path.
|
Writes buffer the input and set `ContentLength`. If no content type is supplied by the caller, the adapter infers a content type from the logical path.
|
||||||
|
|
||||||
Normal replacement and failed-write cleanup delete only managed output objects plus `.distributor.json`. Forced replacement deletes objects under the bounded destination bundle prefix. The backend does not manage bucket versioning, lifecycle rules, object lock, or delete markers.
|
Normal replacement and failed-write cleanup delete only managed output objects plus `.distributor.json`. Merge publication retains omitted managed objects and may overwrite existing managed objects. Forced replacement deletes objects under the bounded destination bundle prefix. The backend does not manage bucket versioning, lifecycle rules, object lock, or delete markers.
|
||||||
|
|
||||||
## Error Mapping
|
## Error Mapping
|
||||||
|
|
||||||
|
|||||||
@@ -54,7 +54,7 @@ The configured `path` is the backend root. All source discovery, destination pat
|
|||||||
|
|
||||||
The adapter rejects symlink ancestors for reads and writes. Reads require regular files. Writes create parent directories and prefer atomic temp-file-plus-rename writes when requested. Walk output is sorted through the shared storage walker.
|
The adapter rejects symlink ancestors for reads and writes. Reads require regular files. Writes create parent directories and prefer atomic temp-file-plus-rename writes when requested. Walk output is sorted through the shared storage walker.
|
||||||
|
|
||||||
Managed cleanup and normal replacement delete only managed output paths plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path.
|
Managed cleanup and normal replacement delete only managed output paths plus `.distributor.json`. Merge publication retains omitted managed paths and may overwrite existing managed paths. Forced replacement deletes the bounded destination bundle path.
|
||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ User-facing command parsing stays in `internal/cli`. User-facing config referenc
|
|||||||
|
|
||||||
## Config Fields Used
|
## Config Fields Used
|
||||||
|
|
||||||
The package consumes the loaded `config.Config`: `server.http`, `secrets.directory`, pipeline ids, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, and transfer policy.
|
The package consumes the loaded `config.Config`: `server.http`, `secrets.directory`, pipeline ids, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, reconciliation policy, and transfer policy.
|
||||||
|
|
||||||
Config fields are validated and defaulted by `internal/config` before app workflows use them.
|
Config fields are validated and defaulted by `internal/config` before app workflows use them.
|
||||||
|
|
||||||
|
|||||||
@@ -8,9 +8,9 @@ Audience: developers and LLM coding agents changing `internal/publish`.
|
|||||||
|
|
||||||
## Inputs And Outputs
|
## Inputs And Outputs
|
||||||
|
|
||||||
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, transformer resolver, transfer policy, distributor version, and force flag.
|
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, reconciliation policy, transformer resolver, transfer policy, distributor version, and force flag.
|
||||||
|
|
||||||
Output from planning is a `Plan` with action, reason, destination identity, selected outputs, optional existing state, optional primary URL, and force metadata. Execution writes selected source outputs, generated outputs, and `.distributor.json` for executable publish or replacement actions.
|
Output from planning is a `Plan` with action, reason, destination identity, selected outputs, reconciliation mode, optional existing state, optional primary URL, and force metadata. Execution writes selected source outputs, generated outputs, and `.distributor.json` for executable publish or replacement actions.
|
||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
@@ -20,7 +20,7 @@ External destination state semantics are documented in `docs/integrations/destin
|
|||||||
|
|
||||||
## Config Fields Used
|
## Config Fields Used
|
||||||
|
|
||||||
The package consumes already-defaulted config values for destination `publish`, `transform`, `links`, `transfer`, and path mapping mode. It uses `config.ValidatePublishTransformPolicy` for publish/transform consistency.
|
The package consumes already-defaulted config values for destination `publish`, `transform`, `links`, `reconciliation`, `transfer`, and path mapping mode. It uses `config.ValidatePublishTransformPolicy` for publish/transform consistency.
|
||||||
|
|
||||||
## Adapters Used
|
## Adapters Used
|
||||||
|
|
||||||
@@ -30,19 +30,19 @@ The package depends on `internal/storage.Backend` for source and destination IO,
|
|||||||
|
|
||||||
Planning inspects destination state through `internal/state`, compares it with the source manifest, and maps comparison outcomes plus transfer policy into actions: `publish_new`, `replace_older`, `force_replace`, `skip_same`, `skip_destination_newer`, `fail_conflict`, or `fail_unmanaged`.
|
Planning inspects destination state through `internal/state`, compares it with the source manifest, and maps comparison outcomes plus transfer policy into actions: `publish_new`, `replace_older`, `force_replace`, `skip_same`, `skip_destination_newer`, `fail_conflict`, or `fail_unmanaged`.
|
||||||
|
|
||||||
Execution writes destination state after selected outputs are written. Destination state includes copied source output metadata, generated output metadata, embedded source manifest, link metadata when configured, pipeline id, destination id, and publication timestamp.
|
Execution writes destination state after selected outputs are written. Destination state includes copied source output metadata, generated output metadata, output timestamps, embedded source manifest, reconciliation metadata, link metadata when configured, pipeline id, destination id, and publication timestamps.
|
||||||
|
|
||||||
## Skip And Resume Behavior
|
## Skip And Resume Behavior
|
||||||
|
|
||||||
`skip_same` and `skip_destination_newer` execute as no-ops. Normal replacement removes only managed output paths from existing state plus `.distributor.json`; this allows retries without broad deletion. Failed writes trigger cleanup of outputs written during that failed attempt where practical.
|
`skip_same` and `skip_destination_newer` execute as no-ops. Replacement-mode updates remove only managed output paths from existing state plus `.distributor.json`, verify the destination is empty, and write state whose outputs are exactly the new plan. Merge-mode updates retain omitted managed outputs, overwrite only paths already recorded as managed, reject unmanaged destination path collisions, and write cumulative output state. Failed writes trigger cleanup where practical; merge cleanup removes only newly created outputs from the failed attempt.
|
||||||
|
|
||||||
Forced replacement is explicit per request and deletes the bounded destination bundle path before writing new outputs and state.
|
Forced replacement is explicit per request and deletes the bounded destination bundle path before writing new outputs and state.
|
||||||
|
|
||||||
## Failure Behavior
|
## Failure Behavior
|
||||||
|
|
||||||
Planning fails for incomplete requests, invalid publish/transform policy, output path collisions, invalid destination state, unmanaged destination content without force, conflict outcomes not allowed by transfer policy, unresolved transforms, invalid Markdown output selection, and invalid link URL planning.
|
Planning fails for incomplete requests, invalid publish/transform policy, invalid reconciliation mode, output path collisions, invalid destination state, unmanaged destination content without force, conflict outcomes not allowed by transfer policy, unresolved transforms, invalid Markdown output selection, and invalid link URL planning.
|
||||||
|
|
||||||
Execution fails on delete, read, transform output, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable publish or replacement actions.
|
Execution fails on delete, read, transform output, unmanaged merge path collision, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable publish or replacement actions.
|
||||||
|
|
||||||
## Tests To Inspect
|
## Tests To Inspect
|
||||||
|
|
||||||
@@ -56,7 +56,9 @@ Execution fails on delete, read, transform output, write, state validation, stat
|
|||||||
- Planning is deterministic for the same request and destination state.
|
- Planning is deterministic for the same request and destination state.
|
||||||
- Destination bundle paths are caller-supplied and backend-root-relative.
|
- Destination bundle paths are caller-supplied and backend-root-relative.
|
||||||
- URL generation uses URL path semantics and never infers public URLs from backend config.
|
- URL generation uses URL path semantics and never infers public URLs from backend config.
|
||||||
- Normal replacement deletes only managed paths recorded in existing state plus `.distributor.json`.
|
- Replacement reconciliation deletes only managed paths recorded in existing state plus `.distributor.json`.
|
||||||
|
- Merge reconciliation never adopts unmanaged content.
|
||||||
|
- Merge state output records are cumulative for the single owner.
|
||||||
- Forced replacement deletes only within the supplied destination bundle path.
|
- Forced replacement deletes only within the supplied destination bundle path.
|
||||||
- Destination state is written after selected outputs are written.
|
- Destination state is written after selected outputs are written.
|
||||||
- Transform resolution stays behind a caller-supplied interface.
|
- Transform resolution stays behind a caller-supplied interface.
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ The external destination state contract is documented in `docs/integrations/dest
|
|||||||
|
|
||||||
## Config Fields Used
|
## Config Fields Used
|
||||||
|
|
||||||
None directly. Destination ids, pipeline ids, and link URLs originate from config but are supplied as values by callers.
|
`internal/state` uses config reconciliation mode constants for destination state validation and legacy state normalization. Destination ids, pipeline ids, and link URLs originate from config but are supplied as values by callers.
|
||||||
|
|
||||||
## Adapters Used
|
## Adapters Used
|
||||||
|
|
||||||
@@ -26,9 +26,13 @@ None.
|
|||||||
|
|
||||||
## State And Manifest Behavior
|
## State And Manifest Behavior
|
||||||
|
|
||||||
`.distributor.json` schema version is `1`. Required fields are `pipeline_id`, `destination_id`, `published_at`, `source.manifest`, and `outputs`. `distributor_version` and `links` are optional.
|
`.distributor.json` schema version is `2` for newly written single-owner state. Required fields are `pipeline_id`, `destination_id`, `published_at`, `created_at`, `updated_at`, `state.mode`, `reconciliation.mode`, `source.manifest`, and `outputs`. `distributor_version` and `links` are optional.
|
||||||
|
|
||||||
Embedded source manifests are parsed and validated through `internal/bundle`, which delegates source manifest semantics to `pkg/bundle`. Output records require clean paths, `source` or `generated` kind, valid source paths, lowercase SHA-256 digests, non-negative sizes, and transform ids for generated outputs. Stored URLs must pass `internal/link` validation.
|
Schema version `1` state remains readable. Parsing infers `state.mode: single_owner`, `reconciliation.mode: replace`, top-level `created_at` and `updated_at` from `published_at`, and per-output timestamps from `published_at`.
|
||||||
|
|
||||||
|
Embedded source manifests are parsed and validated through `internal/bundle`, which delegates source manifest semantics to `pkg/bundle`. Output records require clean paths, `source` or `generated` kind, valid source paths, lowercase SHA-256 digests, non-negative sizes, created and updated timestamps, and transform ids for generated outputs. Stored URLs must pass `internal/link` validation.
|
||||||
|
|
||||||
|
The package also provides helpers for finding output records by path, projecting planned publish outputs into timestamped state outputs, merging retained and newly planned output records, and computing managed output paths from single-owner state.
|
||||||
|
|
||||||
## Skip And Resume Behavior
|
## Skip And Resume Behavior
|
||||||
|
|
||||||
@@ -36,7 +40,7 @@ Comparison is pure. It returns outcomes for absent state, unmanaged content, inv
|
|||||||
|
|
||||||
## Failure Behavior
|
## Failure Behavior
|
||||||
|
|
||||||
Parsing rejects invalid JSON, trailing data, missing required fields, invalid timestamps, invalid embedded manifests, duplicate outputs, invalid output paths, unsupported output kinds, missing generated transforms, invalid URLs, invalid digests, and negative sizes.
|
Parsing rejects invalid JSON, trailing data, missing required fields, invalid timestamps, invalid state mode, invalid reconciliation mode, invalid embedded manifests, duplicate outputs, invalid output paths, unsupported output kinds, missing generated transforms, invalid URLs, invalid digests, and negative sizes.
|
||||||
|
|
||||||
## Tests To Inspect
|
## Tests To Inspect
|
||||||
|
|
||||||
@@ -49,6 +53,9 @@ Parsing rejects invalid JSON, trailing data, missing required fields, invalid ti
|
|||||||
- `.distributor.json` is the destination sentinel and state record.
|
- `.distributor.json` is the destination sentinel and state record.
|
||||||
- Comparison does not mutate storage.
|
- Comparison does not mutate storage.
|
||||||
- Embedded source manifests use the source bundle contract.
|
- Embedded source manifests use the source bundle contract.
|
||||||
|
- Newly written single-owner state uses schema version `2`.
|
||||||
|
- Schema version `1` state remains readable as replacement-mode single-owner state.
|
||||||
- Generated outputs always record a transform id.
|
- Generated outputs always record a transform id.
|
||||||
|
- Output records always carry created and updated timestamps after parsing.
|
||||||
- Stored URLs are optional and must be absolute HTTP or HTTPS URLs when present.
|
- Stored URLs are optional and must be absolute HTTP or HTTPS URLs when present.
|
||||||
- `distributor_version` is diagnostic metadata, not a comparison key.
|
- `distributor_version` is diagnostic metadata, not a comparison key.
|
||||||
|
|||||||
@@ -56,7 +56,7 @@ Destination path mapping controls where each source bundle is published beneath
|
|||||||
|
|
||||||
Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins.
|
Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins.
|
||||||
|
|
||||||
Published destination bundle paths contain `.distributor.json`. See [Destination State Contract](integrations/destination-state.md). This file is both the managed sentinel and the destination state record. It records the pipeline id, destination id, publication time, source manifest, copied outputs, generated outputs, and optional public URL metadata.
|
Published destination bundle paths contain `.distributor.json`. See [Destination State Contract](integrations/destination-state.md). This file is both the managed sentinel and the destination state record. It records the pipeline id, destination id, publication time, state mode, reconciliation mode, source manifest, copied outputs, generated outputs, output timestamps, and optional public URL metadata.
|
||||||
|
|
||||||
`manifest.json` from the source bundle is not copied as destination state.
|
`manifest.json` from the source bundle is not copied as destination state.
|
||||||
|
|
||||||
@@ -71,15 +71,21 @@ Published destination bundle paths contain `.distributor.json`. See [Destination
|
|||||||
- Invalid destination state, identity mismatch, different source id, or same-created digest mismatch: fail by default.
|
- Invalid destination state, identity mismatch, different source id, or same-created digest mismatch: fail by default.
|
||||||
- Content without `.distributor.json`: fail as unmanaged content by default.
|
- Content without `.distributor.json`: fail as unmanaged content by default.
|
||||||
|
|
||||||
Normal replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, then verifies the destination bundle path is empty before writing new outputs and state.
|
When destination state is older than the source, `transfer.on_destination_older` controls whether publication may proceed and `reconciliation.mode` controls how managed outputs are updated.
|
||||||
|
|
||||||
If a write fails after some outputs were written, `distributor` attempts to delete outputs from that failed attempt so a retry does not treat partial outputs as unmanaged content. Operators should still inspect the destination after a failed write before retrying.
|
`reconciliation.mode: replace` is the default. It deletes only managed output paths recorded in `.distributor.json` plus the state file, verifies the destination bundle path is empty, then writes the newly planned outputs and state. The new state `outputs` array is exactly the newly planned output set.
|
||||||
|
|
||||||
|
`reconciliation.mode: merge` retains prior managed outputs that are omitted from the new plan. It overwrites planned paths only when those paths are already recorded in existing state as managed. If a newly planned path already exists in storage but is not recorded in state, publication fails as an unmanaged path collision. The new state `outputs` array is the cumulative managed output set.
|
||||||
|
|
||||||
|
For both modes, retained or overwritten paths are identified only from `.distributor.json`; unmanaged files are not adopted.
|
||||||
|
|
||||||
|
If a write fails after some outputs were written, `distributor` attempts cleanup before returning the error. In `replace` mode, cleanup removes outputs written during that failed attempt. In `merge` mode, cleanup removes only newly created outputs from that failed attempt; overwritten managed outputs are left in place because they previously belonged to the managed set. Operators should still inspect the destination after a failed write before retrying.
|
||||||
|
|
||||||
Fan-out destinations are independent. If one destination fails after planning or execution begins, later destinations are still attempted. The command exits non-zero if any destination failed.
|
Fan-out destinations are independent. If one destination fails after planning or execution begins, later destinations are still attempted. The command exits non-zero if any destination failed.
|
||||||
|
|
||||||
## Dry Runs And Output Review
|
## Dry Runs And Output Review
|
||||||
|
|
||||||
`run --dry-run` loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, `.distributor.json`, or SSH `known_hosts` entries.
|
`run --dry-run` loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, `.distributor.json`, or SSH `known_hosts` entries. For reconciliation, dry runs report the same high-level action labels as execution; inspect the configured destination's `reconciliation.mode` to determine whether `replace_older` will replace the managed set or merge into it.
|
||||||
|
|
||||||
Review these action labels before publishing:
|
Review these action labels before publishing:
|
||||||
|
|
||||||
@@ -194,7 +200,7 @@ S3 execution uses the AWS SDK for Go v2. See [S3-Compatible Storage Integration]
|
|||||||
|
|
||||||
When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is.
|
When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is.
|
||||||
|
|
||||||
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
|
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Merge publication retains omitted managed objects and may overwrite existing managed objects. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
|
||||||
|
|
||||||
## Secrets Operation
|
## Secrets Operation
|
||||||
|
|
||||||
@@ -209,7 +215,7 @@ Use these recovery boundaries:
|
|||||||
- For source validation failures, regenerate the source bundle and manifest together.
|
- For source validation failures, regenerate the source bundle and manifest together.
|
||||||
- For an empty or missing destination, rerun after fixing config or storage access.
|
- For an empty or missing destination, rerun after fixing config or storage access.
|
||||||
- For unmanaged destination content, move unrelated files aside or use a different destination path before publishing.
|
- For unmanaged destination content, move unrelated files aside or use a different destination path before publishing.
|
||||||
- For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun `--dry-run`.
|
- For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun `--dry-run`. In merge mode, retained outputs may be intentional managed outputs from the prior state.
|
||||||
- For state conflicts, verify the source, pipeline, destination, and existing `.distributor.json` before considering `--force`.
|
- For state conflicts, verify the source, pipeline, destination, and existing `.distributor.json` before considering `--force`.
|
||||||
- For HTTP upload failures, inspect `/runs/<run-id>` while retained; after expiry or restart, rely on destination state and logs/output from the publishing run.
|
- For HTTP upload failures, inspect `/runs/<run-id>` while retained; after expiry or restart, rely on destination state and logs/output from the publishing run.
|
||||||
|
|
||||||
|
|||||||
@@ -100,6 +100,9 @@ Each destination bundle path is managed by `.distributor.json`. This file is bot
|
|||||||
- pipeline id;
|
- pipeline id;
|
||||||
- destination id;
|
- destination id;
|
||||||
- publication timestamp;
|
- publication timestamp;
|
||||||
|
- state creation and update timestamps;
|
||||||
|
- single-owner state mode;
|
||||||
|
- reconciliation mode;
|
||||||
- the normalized source manifest used for publication;
|
- the normalized source manifest used for publication;
|
||||||
- metadata for copied source outputs;
|
- metadata for copied source outputs;
|
||||||
- metadata for generated outputs, such as HTML files;
|
- metadata for generated outputs, such as HTML files;
|
||||||
@@ -110,11 +113,19 @@ A representative destination state file is:
|
|||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"schema_version": 1,
|
"schema_version": 2,
|
||||||
"distributor_version": "0.1.0",
|
"distributor_version": "0.1.0",
|
||||||
"pipeline_id": "weather-daily",
|
"pipeline_id": "weather-daily",
|
||||||
"destination_id": "static-html",
|
"destination_id": "static-html",
|
||||||
"published_at": "2026-05-30T11:12:00Z",
|
"published_at": "2026-05-30T11:12:00Z",
|
||||||
|
"created_at": "2026-05-30T11:12:00Z",
|
||||||
|
"updated_at": "2026-05-30T11:12:00Z",
|
||||||
|
"state": {
|
||||||
|
"mode": "single_owner"
|
||||||
|
},
|
||||||
|
"reconciliation": {
|
||||||
|
"mode": "replace"
|
||||||
|
},
|
||||||
"source": {
|
"source": {
|
||||||
"manifest": {
|
"manifest": {
|
||||||
"schema_version": 1,
|
"schema_version": 1,
|
||||||
@@ -141,7 +152,9 @@ A representative destination state file is:
|
|||||||
"transform": "markdown_to_html",
|
"transform": "markdown_to_html",
|
||||||
"sha256": "sha256:...",
|
"sha256": "sha256:...",
|
||||||
"size": 23456,
|
"size": 23456,
|
||||||
"url": "https://reports.example.com/weather-daily/"
|
"url": "https://reports.example.com/weather-daily/",
|
||||||
|
"created_at": "2026-05-30T11:12:00Z",
|
||||||
|
"updated_at": "2026-05-30T11:12:00Z"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
@@ -156,6 +169,8 @@ Destination comparison rules are based on `.distributor.json`:
|
|||||||
- Existing state has the same source id and same `created` but different digest: fail as a conflict.
|
- Existing state has the same source id and same `created` but different digest: fail as a conflict.
|
||||||
- Existing state has a different source id: fail as a conflict.
|
- Existing state has a different source id: fail as a conflict.
|
||||||
|
|
||||||
|
For older destination state, destination `reconciliation.mode` controls output updates. `replace` rewrites the managed output set to match the new plan. `merge` retains omitted managed outputs, overwrites only existing managed paths, and rejects planned paths that collide with unmanaged storage content.
|
||||||
|
|
||||||
## Publication and Transform Policy
|
## Publication and Transform Policy
|
||||||
|
|
||||||
Source files are canonical bundle artifacts. Transform outputs are derived publication artifacts.
|
Source files are canonical bundle artifacts. Transform outputs are derived publication artifacts.
|
||||||
|
|||||||
@@ -216,9 +216,9 @@ Reference: [Operations](operations.md#forced-replacement-workflow).
|
|||||||
|
|
||||||
## Output Path Collision
|
## Output Path Collision
|
||||||
|
|
||||||
Symptom: `destination output path collision`.
|
Symptom: `destination output path collision` or `merge output path ... exists but is not managed by destination state`.
|
||||||
|
|
||||||
Likely cause: publication would write two outputs to the same destination path, such as copying `report.html` while also generating `report.html` from `report.md`.
|
Likely cause: publication would write two outputs to the same destination path, such as copying `report.html` while also generating `report.html` from `report.md`. For merge reconciliation, it can also mean a planned output path already exists in storage but is not recorded in `.distributor.json` as managed.
|
||||||
|
|
||||||
Diagnostic:
|
Diagnostic:
|
||||||
|
|
||||||
@@ -226,7 +226,7 @@ Diagnostic:
|
|||||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide.
|
Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide. For merge reconciliation, move unmanaged content aside, choose another destination path, or use replacement/forced replacement only when deleting the destination bundle path is intended.
|
||||||
|
|
||||||
Reference: [Configuration](config.md#publish-and-transform-policy).
|
Reference: [Configuration](config.md#publish-and-transform-policy).
|
||||||
|
|
||||||
@@ -242,7 +242,7 @@ Diagnostic:
|
|||||||
find <destination-path> -maxdepth 2 -print
|
find <destination-path> -maxdepth 2 -print
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. Rerun `--dry-run` before publishing again.
|
Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. In merge mode, previously managed retained or overwritten outputs may remain intentionally. Rerun `--dry-run` before publishing again.
|
||||||
|
|
||||||
Reference: [Operations](operations.md#destination-state-and-retry-behavior).
|
Reference: [Operations](operations.md#destination-state-and-retry-behavior).
|
||||||
|
|
||||||
|
|||||||
18
examples/merge-reconciliation.yml
Normal file
18
examples/merge-reconciliation.yml
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
pipelines:
|
||||||
|
- id: example-merge-reconciliation
|
||||||
|
source:
|
||||||
|
backend: local
|
||||||
|
path: examples/source-bundle
|
||||||
|
destinations:
|
||||||
|
- id: local-merge-html
|
||||||
|
backend: local
|
||||||
|
path: workspace/published/merge-reconciliation
|
||||||
|
publish:
|
||||||
|
source: false
|
||||||
|
html: true
|
||||||
|
transform:
|
||||||
|
markdown_to_html:
|
||||||
|
enabled: true
|
||||||
|
mode: sidecar
|
||||||
|
reconciliation:
|
||||||
|
mode: merge
|
||||||
@@ -875,6 +875,7 @@ func TestExampleConfigsLoad(t *testing.T) {
|
|||||||
"../../examples/local-index.yml",
|
"../../examples/local-index.yml",
|
||||||
"../../examples/fan-out.yml",
|
"../../examples/fan-out.yml",
|
||||||
"../../examples/archive-and-latest.yml",
|
"../../examples/archive-and-latest.yml",
|
||||||
|
"../../examples/merge-reconciliation.yml",
|
||||||
"../../examples/http-upload-local.yml",
|
"../../examples/http-upload-local.yml",
|
||||||
"../../examples/ssh-destination.yml",
|
"../../examples/ssh-destination.yml",
|
||||||
"../../examples/s3-destination.yml",
|
"../../examples/s3-destination.yml",
|
||||||
|
|||||||
Reference in New Issue
Block a user