eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Synchronize catalog documentation and examples

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-19 16:50:40 +00:00
parent 9a2eaf8e5e
commit 484fda2514
16 changed files with 296 additions and 871 deletions
Download Patch File Download Diff File Expand all files Collapse all files

279
docs/integrations/destination-state.md
View File

@@ -2,197 +2,37 @@
Audience: operators, integrators, and maintainers who inspect or reason about destination `.distributor.json` files.
Each managed destination bundle path contains `.distributor.json`. This file is the destination sentinel and state record used for comparison, skip, replacement, and recovery decisions.
Each managed destination bundle path contains `.distributor.json`. This file is the destination sentinel and state record used for catalog planning, managed replacement, retention pruning, repair, and recovery.
## Single-Owner State Schema
## Catalog State Schema
State written by `run` for `state.mode: single_owner` uses schema version `2`.
Publish execution writes catalog state with `schema_version` `4`.
```json
{
"schema_version": 2,
"distributor_version": "dev",
"pipeline_id": "reports",
"destination_id": "archive",
"published_at": "2026-06-04T12:00:00Z",
"created_at": "2026-06-04T12:00:00Z",
"updated_at": "2026-06-04T12:00:00Z",
"state": {
"mode": "single_owner"
},
"reconciliation": {
"mode": "replace"
},
"source": {
"manifest": {
"schema_version": 1,
"id": "reports.example.2026-06-04",
"digest": "sha256:...",
"created": "2026-06-04T11:55:00Z",
"files": [
{"path": "report.md", "sha256": "sha256:...", "size": 1234}
]
}
},
"links": {
"primary_url": "https://reports.example.com/archive/report.html"
},
"outputs": [
{
"path": "report.html",
"kind": "generated",
"source_path": "report.md",
"transform": "markdown_to_html",
"url": "https://reports.example.com/archive/report.html",
"sha256": "sha256:...",
"size": 2345,
"created_at": "2026-06-04T12:00:00Z",
"updated_at": "2026-06-04T12:00:00Z"
}
]
}
```
Required fields:
- `schema_version`: must be `2` for newly written state.
- `pipeline_id`: configured pipeline id that wrote the state.
- `destination_id`: configured destination id that wrote the state.
- `published_at`: RFC3339 publication timestamp.
- `created_at`: RFC3339 timestamp for when this destination state record was first created.
- `updated_at`: RFC3339 timestamp for the latest state update.
- `state.mode`: must be `single_owner`.
- `reconciliation.mode`: `replace` or `merge`.
- `source.manifest`: embedded source bundle manifest.
- `outputs`: output records array; empty is allowed, but the field is required.
Optional fields:
- `distributor_version`: application version string when available.
- `links.primary_url`: absolute HTTP or HTTPS URL selected by destination link policy.
## Output Records
Each output record has:
- `path`: destination-relative output path.
- `kind`: `source` or `generated`.
- `source_path`: source manifest path used for the output.
- `transform`: required for `generated` outputs; omitted for copied source outputs.
- `url`: optional absolute HTTP or HTTPS URL for the output.
- `sha256`: lowercase `sha256:<64 hex>` digest of the output bytes.
- `size`: output byte size, zero or greater.
- `created_at`: RFC3339 timestamp for when this output path was first recorded as managed.
- `updated_at`: RFC3339 timestamp for when this output path was last written or updated in state.
Output paths must be unique and use clean relative slash-separated path rules.
For replacement updates, newly planned outputs are written into state. For merge updates, retained output records preserve both timestamps, overwritten managed output records preserve `created_at` and receive a new `updated_at`, and new output records receive the current publication time for both fields.
## Reconciliation Semantics
Destination reconciliation applies when destination state is older than the source and transfer policy permits replacement.
- `replace`: for single-owner state, delete managed output paths recorded in `outputs` plus `.distributor.json`, require the destination bundle path to be empty afterward, write the newly planned outputs, and write state whose `outputs` are exactly that new planned set. For shared-root state, delete only the current owner's omitted outputs and preserve unrelated owners.
- `merge`: retain managed output paths omitted from the new plan, overwrite planned paths only when they are already recorded in existing state, fail when a newly planned path exists in storage but is not recorded as managed, and write state whose `outputs` are the cumulative managed set.
Takeover replacement applies when valid managed state has an identity, source, or shared-root output-owner mismatch and destination `takeover.mode` permits the current publication to take ownership. It uses bounded managed replacement behavior. For shared-root state, only taken-over output records and the current owner records are rewritten; unrelated owner records and non-conflicting outputs remain managed by their existing owners.
Top-level `links.primary_url` is selected from the newly planned outputs for the current publication. Retained outputs keep their existing per-output URL metadata.
If a same-source merge publication fails after writing outputs, cleanup removes only newly created outputs from that failed attempt. Previously managed overwritten paths remain managed and are not removed by failed-attempt cleanup. Takeover replacement does not retain omitted outputs through merge reconciliation.
## Comparison Semantics
`distributor` compares the current source manifest to destination state before writing:
- No state and no content: publish new outputs.
- No state and existing content: treat the destination as unmanaged.
- Shared-root state without the current owner: publish new outputs for that owner if planned paths do not collide with unmanaged content or with other owners that `takeover.mode` does not permit.
- Matching embedded source manifest: skip.
- Same source id with older `created`: replace if policy allows.
- Same source id with newer `created`: skip by default, or replace when `transfer.on_destination_newer: replace` is configured.
- Same source id and same `created` with different digest: conflict by default, or replace when `transfer.on_conflict: replace` is configured.
- Different source id, pipeline id, or destination id in single-owner state: conflict unless `takeover.mode` permits managed ownership transfer or `transfer.on_conflict: replace` is configured.
- Shared-root output path owned by a different owner: conflict unless `takeover.mode` permits managed ownership transfer or `transfer.on_conflict: replace` is configured.
- Invalid state JSON or invalid state fields: conflict.
Normal managed replacement deletes only managed output paths recorded in `.distributor.json`. Forced replacement deletes the bounded destination bundle path, then writes planned outputs and schema version `4` catalog state.
## State Repair Semantics
`distributor reconcile-state` can remove managed output records for files that no longer exist in destination storage. It uses the configured pipeline and destination selector to open one destination root and reads that root's `.distributor.json`.
For single-owner state, the state `pipeline_id` and `destination_id` must match the selected pipeline and destination. The command checks output paths recorded in `outputs`, reports missing managed outputs, reports unmanaged entries under the destination root, and removes missing output records from valid state unless `--dry-run` is set.
For shared-root state, repair is scoped to the selected owner by default. With `--all-owners`, it checks and repairs missing output records for every owner in the selected shared-root state file.
State repair does not validate output digests, delete destination files, adopt unmanaged entries, or rewrite invalid or mismatched state.
## Prune Semantics
`distributor prune` can delete managed outputs selected by the configured destination retention policy. It uses the configured pipeline and destination selector to open one destination root and reads that root's `.distributor.json`.
For single-owner state, the state `pipeline_id` and `destination_id` must match the selected pipeline and destination. For shared-root state, pruning is scoped to the selected owner and preserves other owners.
Prune planning uses output `updated_at` timestamps. `prune --dry-run` reports planned managed-output deletes without deleting files or rewriting state. `prune --apply` deletes only planned managed output paths, removes confirmed deleted records from valid state, and updates the state timestamp. It does not delete unmanaged files or `.distributor.json`.
## Compatibility
`distributor` can read schema version `1` destination state for compatibility. When v1 state is read, it is treated as single-owner state with `reconciliation.mode: replace`. Missing top-level `created_at` and `updated_at` are inferred from `published_at`, and missing per-output timestamps are also inferred from `published_at`.
Newly written single-owner destination state from publish execution uses schema version `2`.
## Shared-Root State Schema
State written by `run` for `state.mode: shared_root` uses schema version `3`.
```json
{
"schema_version": 3,
"schema_version": 4,
"distributor_version": "dev",
"created_at": "2026-06-04T12:00:00Z",
"updated_at": "2026-06-04T12:10:00Z",
"state": {
"mode": "shared_root"
"mode": "catalog"
},
"owners": [
{
"pipeline_id": "reports",
"destination_id": "archive",
"reconciliation": {
"mode": "merge"
},
"source": {
"manifest": {
"schema_version": 1,
"id": "reports.example.2026-06-04",
"digest": "sha256:...",
"created": "2026-06-04T11:55:00Z",
"files": [
{"path": "report.md", "sha256": "sha256:...", "size": 1234}
]
}
},
"links": {
"primary_url": "https://reports.example.com/archive/report.html"
}
}
],
"outputs": [
{
"path": "report.html",
"pipeline_id": "reports",
"destination_id": "static-site",
"source": {
"id": "reports.example.2026-06-04",
"digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
"created": "2026-06-04T11:55:00Z"
},
"kind": "generated",
"source_path": "report.md",
"transform": "markdown_to_html",
"url": "https://reports.example.com/archive/report.html",
"sha256": "sha256:...",
"sha256": "sha256:abcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcd",
"size": 2345,
"pipeline_id": "reports",
"destination_id": "archive",
"source_id": "reports.example.2026-06-04",
"source_digest": "sha256:...",
"source_created": "2026-06-04T11:55:00Z",
"created_at": "2026-06-04T12:00:00Z",
"updated_at": "2026-06-04T12:10:00Z"
}
@@ -200,55 +40,88 @@ State written by `run` for `state.mode: shared_root` uses schema version `3`.
}
```
Shared-root required fields:
Top-level fields:
- `schema_version`: must be `3`.
- `created_at`: RFC3339 timestamp for when this shared-root state record was first created.
- `updated_at`: RFC3339 timestamp for the latest shared-root state update.
- `state.mode`: must be `shared_root`.
- `owners`: owner records keyed by `pipeline_id` and `destination_id`; each owner records its latest source manifest and reconciliation mode.
- `outputs`: output records for every managed path under the shared destination root.
- `schema_version`: required value `4`.
- `distributor_version`: optional application version string.
- `created_at`: RFC3339 timestamp for when this catalog record was first created.
- `updated_at`: RFC3339 timestamp for the latest catalog update.
- `state.mode`: required value `catalog`.
- `outputs`: required array of catalog output records. Empty is valid.
Shared-root optional fields:
## Output Records
- `distributor_version`: application version string when available.
- `owners[].links.primary_url`: absolute HTTP or HTTPS URL selected by that owner's destination link policy.
Each output record has:
Shared-root output records carry the same `path`, `kind`, `source_path`, `transform`, `url`, `sha256`, `size`, `created_at`, and `updated_at` fields as single-owner outputs. They also include the owner `pipeline_id` and `destination_id`, plus compact source identity fields `source_id`, `source_digest`, and `source_created`.
- `path`: destination-bundle-relative output path.
- `pipeline_id`: configured pipeline id that manages the output path.
- `destination_id`: configured destination id that manages the output path.
- `source`: compact source identity for the output.
- `kind`: `source` or `generated`.
- `source_path`: generated outputs only; source manifest path used to derive the output.
- `transform`: generated outputs only; transform id, currently `markdown_to_html`.
- `url`: optional absolute HTTP or HTTPS URL for the output.
- `sha256`: lowercase `sha256:<64 hex>` digest of the output bytes.
- `size`: output byte size, zero or greater.
- `created_at`: RFC3339 timestamp for when this output path was first recorded as managed.
- `updated_at`: RFC3339 timestamp for when this output path was last written or updated in state.
## Shared-Root Ownership
Output paths must be unique and use clean relative slash-separated path rules. `pipeline_id` and `destination_id` must be slug-like identifiers.
Shared-root state is owner-scoped by `pipeline_id` and `destination_id`.
For copied source outputs, `source_path` and `transform` are omitted. For generated outputs, both fields are required.
- One output path may be owned by only one owner.
- The same owner may overwrite its own managed paths.
- A different owner planning an already owned path fails as a conflict.
- A planned path that exists in storage but is not recorded in state fails as unmanaged content unless forced replacement is explicitly selected.
## Source Identity
When an owner publishes, unrelated owner records and output records are preserved. The publishing owner's record is updated with the latest source manifest, reconciliation mode, and latest primary URL when present.
Each output records the source identity that produced it:
## Shared-Root Timestamps
- `source.id`: source manifest id.
- `source.digest`: source manifest digest.
- `source.created`: source manifest creation timestamp.
For shared-root state:
The full source manifest is not embedded in catalog state. The source bundle's `manifest.json` remains the producer contract, and `.distributor.json` records only the source identity needed for catalog ownership and later maintenance workflows.
- top-level `created_at` remains the original shared-root state creation time;
- top-level `updated_at` changes after a successful state write;
- output `created_at` remains stable for an existing managed path;
- output `updated_at` changes only when that path is rewritten;
- newly managed output paths receive the publication time for both output timestamps.
## Workflow Semantics
## Shared-Root Migration
Destination `workflow` is runtime configuration and is not persisted in `.distributor.json`.
If `state.mode: shared_root` is configured and existing state is a compatible single-owner `.distributor.json` for the same pipeline id and destination id, the next successful publish writes schema version `3` shared-root state for that owner.
`workflow: additive` writes planned outputs and retains unrelated catalog-managed outputs. If a planned path already has a catalog output record, the current publication replaces that record and overwrites the file. If a planned path exists in storage but is not recorded in valid catalog state, planning fails as unmanaged unless `run --force` selects `force_replace`.
If `state.mode: shared_root` is configured and existing single-owner state belongs to a different pipeline or destination, publish fails as a conflict. `distributor` does not implicitly convert unrelated single-owner state or take over unmanaged files during shared-root migration.
`workflow: replacement` writes planned outputs for the current pipeline and destination and removes omitted outputs owned by that same pipeline and destination. Outputs owned by other pipeline/destination pairs remain catalog-managed. This is normal managed replacement and does not require `--force`.
`force_replace` is an explicit per-run recovery path. It deletes only the resolved destination bundle path, then writes planned outputs and fresh catalog state. It can replace unmanaged content, planned unmanaged path collisions, invalid destination state, and unsupported future destination state after dry-run review.
## Publish Planning Outcomes
Current run reports use these destination action labels:
- `publish_new`: no valid state exists and the destination bundle path is empty.
- `upsert_additive`: valid catalog state exists and additive workflow will write the planned outputs.
- `replace_catalog`: valid catalog state exists and replacement workflow will write the planned outputs and remove omitted outputs for the current owner.
- `skip_same`: no-op action value in the run output vocabulary.
- `force_replace`: explicit bounded destructive replacement selected by `--force`.
- `fail_unmanaged`: unmanaged destination content prevents publication.
- `fail_conflict`: invalid state or unsupported state prevents publication without explicit force.
Schema versions older than `4` are superseded legacy state for publish planning. Normal catalog planning may publish over superseded legacy state according to the configured workflow, while invalid state and unsupported future schema versions fail unless `--force` is explicitly selected.
## Repair Semantics
`distributor reconcile-state` removes catalog output records for files that no longer exist in destination storage. It uses the configured pipeline and destination selector to open one destination root and reads that root's `.distributor.json`.
By default, repair is scoped to output records whose `pipeline_id` and `destination_id` match the selected pipeline and destination. With `--all-owners`, it checks every catalog output record in the selected root.
The command reports missing managed outputs and unmanaged storage entries. Without `--dry-run`, it removes missing managed output records from valid catalog state and rewrites `.distributor.json`. It does not delete destination files, adopt unmanaged entries, validate output digests, or rewrite invalid state.
## Prune Semantics
`distributor prune` deletes catalog output paths selected by the configured destination `retention.prune` policy. It uses the configured pipeline and destination selector to open one destination root and reads that root's `.distributor.json`.
Prune planning is scoped to output records whose `pipeline_id` and `destination_id` match the selected pipeline and destination. It uses output `updated_at` timestamps. `prune --dry-run` reports planned managed-output deletes without deleting files or rewriting state. `prune --apply` deletes only planned managed output paths, removes confirmed deleted records from valid catalog state, and updates the state timestamp. It does not delete unmanaged files or `.distributor.json`.
## Boundaries
Destination state is internal managed state written by `distributor`. Operators may inspect it during recovery, but normal workflows should not edit it by hand. Source `manifest.json` is not copied as destination state.
## Tests
Before changing this contract, inspect and run:
```sh

2
docs/integrations/markdown.md
View File

@@ -50,7 +50,7 @@ Destination state stores generated outputs with `kind: generated`, `source_path`
## Boundaries
Markdown rendering does not mutate source bundles, publish files, write `.distributor.json`, select destination actions, or choose transfer policy. Publish planning decides whether generated HTML is selected for a destination and destination state records the generated output metadata.
Markdown rendering does not mutate source bundles, publish files, write `.distributor.json`, select destination actions, or choose catalog workflow behavior. Publish planning decides whether generated HTML is selected for a destination and destination state records the generated output metadata.
## Tests