eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Document shared-root publishing

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-08 19:06:23 +00:00
parent 9afb3550c4
commit cb9502f790
8 changed files with 154 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
docs/config.md
View File

@@ -371,10 +371,48 @@ state:
- `state.mode`: optional. Accepted values are `single_owner` and `shared_root`; default is `single_owner`.
`single_owner` state records one pipeline/destination owner for each destination bundle path and is the state mode written by `run`.
`single_owner` state records one pipeline/destination owner for each destination bundle path and is the default state mode.
`shared_root` records multiple pipeline/destination owners in one destination root. Publish execution preserves unrelated owners, rejects path ownership conflicts, and writes shared-root destination state.
Use `shared_root` when multiple configured destinations intentionally write disjoint output paths into the same backend root:
```yaml
pipelines:
- id: reports-source
source:
backend: local
path: /var/spool/distributor/reports
destinations:
- id: shared-root
backend: local
path: /srv/reports/shared
state:
mode: shared_root
publish:
source: true
html: false
- id: reports-html
source:
backend: local
path: /var/spool/distributor/reports
destinations:
- id: shared-root
backend: local
path: /srv/reports/shared
state:
mode: shared_root
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
Every output path in a shared root belongs to exactly one `pipeline_id` and `destination_id`. A different owner planning the same path fails as a conflict.
## Reconciliation Policy
```yaml
@@ -503,6 +541,7 @@ Local examples:
- `fan-out.yml`: local fan-out publication to source and HTML destinations.
- `archive-and-latest.yml`: local archive plus fixed latest publication.
- `merge-reconciliation.yml`: local HTML publication using merge reconciliation.
- `shared-root.yml`: two local pipelines publishing disjoint outputs into one shared destination root.
- `http-upload-local.yml`: local HTTP upload server config; requires `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN` in the process environment or as a secret-file name before running `serve`.
Environment-gated remote examples:

32
docs/integrations/destination-state.md
View File

@@ -93,7 +93,7 @@ For replacement updates, newly planned outputs are written into state. For merge
Destination reconciliation applies when destination state is older than the source and transfer policy permits replacement.
- `replace`: delete managed output paths recorded in `outputs` plus `.distributor.json`, require the destination bundle path to be empty afterward, write the newly planned outputs, and write state whose `outputs` are exactly that new planned set.
- `replace`: for single-owner state, delete managed output paths recorded in `outputs` plus `.distributor.json`, require the destination bundle path to be empty afterward, write the newly planned outputs, and write state whose `outputs` are exactly that new planned set. For shared-root state, delete only the current owner's omitted outputs and preserve unrelated owners.
- `merge`: retain managed output paths omitted from the new plan, overwrite planned paths only when they are already recorded in existing state, fail when a newly planned path exists in storage but is not recorded as managed, and write state whose `outputs` are the cumulative managed set.
Top-level `links.primary_url` is selected from the newly planned outputs for the current publication. Retained outputs keep their existing per-output URL metadata.
@@ -106,6 +106,7 @@ If a merge publication fails after writing outputs, cleanup removes only newly c
- No state and no content: publish new outputs.
- No state and existing content: treat the destination as unmanaged.
- Shared-root state without the current owner: publish new outputs for that owner if planned paths do not collide with other owners or unmanaged content.
- Matching embedded source manifest: skip.
- Same source id with older `created`: replace if policy allows.
- Same source id with newer `created`: skip by default.
@@ -113,7 +114,7 @@ If a merge publication fails after writing outputs, cleanup removes only newly c
- Different source id, pipeline id, or destination id: conflict.
- Invalid state JSON or invalid state fields: conflict.
Normal replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Merge publication retains omitted managed outputs. Forced replacement deletes the bounded destination bundle path.
Normal single-owner replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Shared-root replacement deletes only omitted outputs for the current owner. Merge publication retains omitted managed outputs. Forced replacement deletes the bounded destination bundle path.
## Compatibility
@@ -194,6 +195,33 @@ Shared-root optional fields:
Shared-root output records carry the same `path`, `kind`, `source_path`, `transform`, `url`, `sha256`, `size`, `created_at`, and `updated_at` fields as single-owner outputs. They also include the owner `pipeline_id` and `destination_id`, plus compact source identity fields `source_id`, `source_digest`, and `source_created`.
## Shared-Root Ownership
Shared-root state is owner-scoped by `pipeline_id` and `destination_id`.
- One output path may be owned by only one owner.
- The same owner may overwrite its own managed paths.
- A different owner planning an already owned path fails as a conflict.
- A planned path that exists in storage but is not recorded in state fails as unmanaged content unless forced replacement is explicitly selected.
When an owner publishes, unrelated owner records and output records are preserved. The publishing owner's record is updated with the latest source manifest, reconciliation mode, and latest primary URL when present.
## Shared-Root Timestamps
For shared-root state:
- top-level `created_at` remains the original shared-root state creation time;
- top-level `updated_at` changes after a successful state write;
- output `created_at` remains stable for an existing managed path;
- output `updated_at` changes only when that path is rewritten;
- newly managed output paths receive the publication time for both output timestamps.
## Shared-Root Migration
If `state.mode: shared_root` is configured and existing state is a compatible single-owner `.distributor.json` for the same pipeline id and destination id, the next successful publish writes schema version `3` shared-root state for that owner.
If existing single-owner state belongs to a different pipeline or destination, publish fails as a conflict. `distributor` does not implicitly take over unrelated state or unmanaged files.
## Boundaries
Destination state is internal managed state written by `distributor`. Operators may inspect it during recovery, but normal workflows should not edit it by hand. Source `manifest.json` is not copied as destination state.

6
docs/internal/publish.md
View File

@@ -10,7 +10,7 @@ Audience: developers and LLM coding agents changing `internal/publish`.
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, state policy, reconciliation policy, transformer resolver, transfer policy, distributor version, and force flag.
Output from planning is a `Plan` with action, reason, destination identity, selected outputs, state mode, owner scope, reconciliation mode, optional existing single-owner or shared-root state, optional primary URL, and force metadata. Shared-root plans also expose other-owner outputs to preserve, current-owner outputs retained by merge, current-owner outputs deleted by replace, and current-owner outputs to write. Execution writes selected source outputs, generated outputs, and `.distributor.json` for executable single-owner publish or replacement actions.
Output from planning is a `Plan` with action, reason, destination identity, selected outputs, state mode, owner scope, reconciliation mode, optional existing single-owner or shared-root state, optional primary URL, and force metadata. Shared-root plans also expose other-owner outputs to preserve, current-owner outputs retained by merge, current-owner outputs deleted by replace, and current-owner outputs to write. Execution writes selected source outputs, generated outputs, and `.distributor.json` for executable publish or replacement actions.
## Boundaries
@@ -38,7 +38,7 @@ Execution writes destination state after selected outputs are written. Destinati
`skip_same` and `skip_destination_newer` execute as no-ops. Replacement-mode single-owner updates remove managed output paths from existing state plus `.distributor.json`, verify the destination is empty, and write state whose outputs are exactly the new plan. Replacement-mode shared-root updates remove only current-owner omitted outputs and preserve unrelated owners. Merge-mode updates retain omitted managed outputs, overwrite only paths already recorded as managed, reject unmanaged destination path collisions, and write cumulative output state. Failed writes trigger cleanup where practical; merge cleanup removes only newly created outputs from the failed attempt.
Shared-root execution writes schema version `3` state. It preserves unrelated owner records and outputs, updates only the publishing owner metadata, preserves root `created_at`, and updates root `updated_at` after successful state writes.
Shared-root execution writes schema version `3` state. It preserves unrelated owner records and outputs, updates only the publishing owner metadata, preserves root `created_at`, and updates root `updated_at` after successful state writes. Compatible single-owner state for the same pipeline and destination is converted to shared-root state on successful publish.
Forced replacement is explicit per request and deletes the bounded destination bundle path before writing new outputs and state.
@@ -46,7 +46,7 @@ Forced replacement is explicit per request and deletes the bounded destination b
Planning fails for incomplete requests, invalid publish/transform policy, invalid state mode, invalid reconciliation mode, output path collisions, invalid destination state, unmanaged destination content without force, shared-root owner path conflicts, conflict outcomes not allowed by transfer policy, unresolved transforms, invalid Markdown output selection, and invalid link URL planning.
Execution fails on delete, read, transform output, unmanaged merge path collision, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable publish or replacement actions.
Execution fails on delete, read, transform output, unmanaged merge path collision, shared-root ownership conflict, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable publish or replacement actions.
## Tests To Inspect

4
docs/internal/state.md
View File

@@ -32,12 +32,16 @@ Schema version `1` state remains readable. Parsing infers `state.mode: single_ow
Schema version `3` is shared-root state. It records `state.mode: shared_root`, shared state timestamps, owner records keyed by pipeline id and destination id, each owner's latest source manifest and reconciliation metadata, optional owner primary links, and output records for every managed path. Shared-root output records include owner ids and compact source identity fields for source id, digest, and creation time.
Shared-root publish conversion is explicit. Compatible single-owner state for the same pipeline and destination can be projected into the current owner scope by publish execution. Unrelated single-owner state remains a conflict.
Embedded source manifests are parsed and validated through `internal/bundle`, which delegates source manifest semantics to `pkg/bundle`. Output records require clean paths, `source` or `generated` kind, valid source paths, lowercase SHA-256 digests, non-negative sizes, created and updated timestamps, and transform ids for generated outputs. Stored URLs must pass `internal/link` validation.
The package also provides helpers for finding output records by path, projecting planned publish outputs into timestamped state outputs, merging retained and newly planned output records, and computing managed output paths from single-owner state.
For shared-root state, helpers parse either state shape, identify the current owner scope, return an owner's latest source manifest, list managed paths for one owner or all owners, detect path ownership conflicts, project planned owner outputs, merge one owner's planned outputs while preserving unrelated owners, and replace one owner's outputs by removing that owner's omitted outputs.
Shared-root helper projections preserve output `created_at` for existing managed paths and use the current publication time for rewritten `updated_at`. Root-level `created_at` preservation is owned by publish execution.
## Skip And Resume Behavior
Comparison is pure. It returns outcomes for absent state, unmanaged content, invalid state, pipeline/destination mismatch, same source manifest, older destination, newer destination, same-created digest conflict, and different source id conflict. It does not decide whether to skip, replace, force, or fail; publish planning maps outcomes to actions.

13
docs/operations.md
View File

@@ -79,6 +79,10 @@ When destination state is older than the source, `transfer.on_destination_older`
For both modes, retained or overwritten paths are identified only from `.distributor.json`; unmanaged files are not adopted.
For `state.mode: shared_root`, one destination root may contain outputs from multiple pipeline/destination owners. Comparisons, replacement, and merge retention are scoped to the current owner. Outputs owned by other owners are preserved. A planned output path owned by another owner fails as a conflict, and a planned path that exists in storage but is not recorded in state fails as unmanaged content by default.
If `state.mode: shared_root` is configured on a destination whose existing single-owner state belongs to the same pipeline and destination, the next successful publish converts that state file to shared-root schema. Existing single-owner state for a different pipeline or destination remains a conflict.
If a write fails after some outputs were written, `distributor` attempts cleanup before returning the error. In `replace` mode, cleanup removes outputs written during that failed attempt. In `merge` mode, cleanup removes only newly created outputs from that failed attempt; overwritten managed outputs are left in place because they previously belonged to the managed set. Operators should still inspect the destination after a failed write before retrying.
Fan-out destinations are independent. If one destination fails after planning or execution begins, later destinations are still attempted. The command exits non-zero if any destination failed.
@@ -87,9 +91,11 @@ Fan-out destinations are independent. If one destination fails after planning or
`run --dry-run` loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, `.distributor.json`, or SSH `known_hosts` entries. For reconciliation, dry runs report the same high-level action labels as execution; inspect the configured destination's `reconciliation.mode` to determine whether `replace_older` will replace the managed set or merge into it.
For shared-root destinations, dry runs are owner-scoped. A `replace_older` action replaces or merges only the current owner according to `reconciliation.mode`; unrelated owners remain managed by the shared-root state.
Review these action labels before publishing:
- `publish_new`: destination is empty and unmanaged.
- `publish_new`: destination state is absent, or a shared-root owner is absent and planned paths are publishable.
- `replace_older`: destination state is older than the source.
- `skip_same`: destination state already matches the source.
- `skip_destination_newer`: destination state is newer than the source and is skipped.
@@ -116,6 +122,8 @@ Forced replacement can claim unmanaged non-empty destination paths. State confli
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete parent paths, sibling paths, or storage outside the destination bundle path. For fixed destinations, the destination bundle path is the backend root, so a forced replacement can clear that configured root.
For shared-root destinations, forced replacement also deletes the configured destination bundle path before writing new shared-root state. This removes unrelated owners inside that destination root. Preview with `--dry-run --force` and confirm the destination path before applying.
`--force` applies only to the current invocation. There is no config field that enables forced replacement by default.
## HTTP Upload Operation
@@ -200,7 +208,7 @@ S3 execution uses the AWS SDK for Go v2. See [S3-Compatible Storage Integration]
When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is.
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Merge publication retains omitted managed objects and may overwrite existing managed objects. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
Normal single-owner replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Shared-root replacement deletes only current-owner omitted output objects and rewrites the shared state object. Merge publication retains omitted managed objects and may overwrite existing managed objects. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
## Secrets Operation
@@ -215,6 +223,7 @@ Use these recovery boundaries:
- For source validation failures, regenerate the source bundle and manifest together.
- For an empty or missing destination, rerun after fixing config or storage access.
- For unmanaged destination content, move unrelated files aside or use a different destination path before publishing.
- For shared-root ownership conflicts, change one owner so it writes a different destination path, or use a separate destination root.
- For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun `--dry-run`. In merge mode, retained outputs may be intentional managed outputs from the prior state.
- For state conflicts, verify the source, pipeline, destination, and existing `.distributor.json` before considering `--force`.
- For HTTP upload failures, inspect `/runs/<run-id>` while retained; after expiry or restart, rely on destination state and logs/output from the publishing run.

34
docs/troubleshooting.md
View File

@@ -230,6 +230,40 @@ Safe fix: adjust source files or publish/transform policy so copied and generate
Reference: [Configuration](config.md#publish-and-transform-policy).
## Shared-Root Ownership Conflict
Symptom: `fail_conflict` with a reason like `destination output path ... is owned by <pipeline>/<destination>`.
Likely cause: a `state.mode: shared_root` destination planned an output path already recorded in `.distributor.json` for another pipeline/destination owner.
Diagnostic:
```sh
cat <destination-path>/.distributor.json
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: change one owner so it writes a different output path, use a separate destination root, or intentionally replace the whole destination root only after previewing with `--dry-run --force`.
Reference: [Operations](operations.md#destination-state-and-retry-behavior).
## Shared-Root Unmanaged Path Collision
Symptom: `fail_unmanaged` with a reason like `destination output path ... exists but is not managed by destination state`.
Likely cause: a `state.mode: shared_root` destination planned a new output path that already exists in storage but is not recorded as managed in `.distributor.json`.
Diagnostic:
```sh
find <destination-path> -maxdepth 2 -print
cat <destination-path>/.distributor.json
```
Safe fix: move the unmanaged file aside, change the planned output path, or use forced replacement only when deleting the configured destination root is intended.
Reference: [Operations](operations.md#forced-replacement-workflow).
## Run Failed After Writing Some Files
Symptom: a destination write failed and the command exited non-zero after partial work.