From cb9502f7900897092b41bfdcbfc615e804126e8b Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 8 Jun 2026 19:06:23 +0000 Subject: [PATCH] Document shared-root publishing --- docs/config.md | 41 +++++++++++++++++++++++++- docs/integrations/destination-state.md | 32 ++++++++++++++++++-- docs/internal/publish.md | 6 ++-- docs/internal/state.md | 4 +++ docs/operations.md | 13 ++++++-- docs/troubleshooting.md | 34 +++++++++++++++++++++ examples/shared-root.yml | 31 +++++++++++++++++++ internal/config/load_test.go | 1 + 8 files changed, 154 insertions(+), 8 deletions(-) create mode 100644 examples/shared-root.yml diff --git a/docs/config.md b/docs/config.md index 2e94be0..42e1aa3 100644 --- a/docs/config.md +++ b/docs/config.md @@ -371,10 +371,48 @@ state: - `state.mode`: optional. Accepted values are `single_owner` and `shared_root`; default is `single_owner`. -`single_owner` state records one pipeline/destination owner for each destination bundle path and is the state mode written by `run`. +`single_owner` state records one pipeline/destination owner for each destination bundle path and is the default state mode. `shared_root` records multiple pipeline/destination owners in one destination root. Publish execution preserves unrelated owners, rejects path ownership conflicts, and writes shared-root destination state. +Use `shared_root` when multiple configured destinations intentionally write disjoint output paths into the same backend root: + +```yaml +pipelines: + - id: reports-source + source: + backend: local + path: /var/spool/distributor/reports + destinations: + - id: shared-root + backend: local + path: /srv/reports/shared + state: + mode: shared_root + publish: + source: true + html: false + - id: reports-html + source: + backend: local + path: /var/spool/distributor/reports + destinations: + - id: shared-root + backend: local + path: /srv/reports/shared + state: + mode: shared_root + publish: + source: false + html: true + transform: + markdown_to_html: + enabled: true + mode: sidecar +``` + +Every output path in a shared root belongs to exactly one `pipeline_id` and `destination_id`. A different owner planning the same path fails as a conflict. + ## Reconciliation Policy ```yaml @@ -503,6 +541,7 @@ Local examples: - `fan-out.yml`: local fan-out publication to source and HTML destinations. - `archive-and-latest.yml`: local archive plus fixed latest publication. - `merge-reconciliation.yml`: local HTML publication using merge reconciliation. +- `shared-root.yml`: two local pipelines publishing disjoint outputs into one shared destination root. - `http-upload-local.yml`: local HTTP upload server config; requires `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN` in the process environment or as a secret-file name before running `serve`. Environment-gated remote examples: diff --git a/docs/integrations/destination-state.md b/docs/integrations/destination-state.md index 58f478f..bd0da89 100644 --- a/docs/integrations/destination-state.md +++ b/docs/integrations/destination-state.md @@ -93,7 +93,7 @@ For replacement updates, newly planned outputs are written into state. For merge Destination reconciliation applies when destination state is older than the source and transfer policy permits replacement. -- `replace`: delete managed output paths recorded in `outputs` plus `.distributor.json`, require the destination bundle path to be empty afterward, write the newly planned outputs, and write state whose `outputs` are exactly that new planned set. +- `replace`: for single-owner state, delete managed output paths recorded in `outputs` plus `.distributor.json`, require the destination bundle path to be empty afterward, write the newly planned outputs, and write state whose `outputs` are exactly that new planned set. For shared-root state, delete only the current owner's omitted outputs and preserve unrelated owners. - `merge`: retain managed output paths omitted from the new plan, overwrite planned paths only when they are already recorded in existing state, fail when a newly planned path exists in storage but is not recorded as managed, and write state whose `outputs` are the cumulative managed set. Top-level `links.primary_url` is selected from the newly planned outputs for the current publication. Retained outputs keep their existing per-output URL metadata. @@ -106,6 +106,7 @@ If a merge publication fails after writing outputs, cleanup removes only newly c - No state and no content: publish new outputs. - No state and existing content: treat the destination as unmanaged. +- Shared-root state without the current owner: publish new outputs for that owner if planned paths do not collide with other owners or unmanaged content. - Matching embedded source manifest: skip. - Same source id with older `created`: replace if policy allows. - Same source id with newer `created`: skip by default. @@ -113,7 +114,7 @@ If a merge publication fails after writing outputs, cleanup removes only newly c - Different source id, pipeline id, or destination id: conflict. - Invalid state JSON or invalid state fields: conflict. -Normal replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Merge publication retains omitted managed outputs. Forced replacement deletes the bounded destination bundle path. +Normal single-owner replacement deletes only managed output paths recorded in `outputs` plus `.distributor.json`. Shared-root replacement deletes only omitted outputs for the current owner. Merge publication retains omitted managed outputs. Forced replacement deletes the bounded destination bundle path. ## Compatibility @@ -194,6 +195,33 @@ Shared-root optional fields: Shared-root output records carry the same `path`, `kind`, `source_path`, `transform`, `url`, `sha256`, `size`, `created_at`, and `updated_at` fields as single-owner outputs. They also include the owner `pipeline_id` and `destination_id`, plus compact source identity fields `source_id`, `source_digest`, and `source_created`. +## Shared-Root Ownership + +Shared-root state is owner-scoped by `pipeline_id` and `destination_id`. + +- One output path may be owned by only one owner. +- The same owner may overwrite its own managed paths. +- A different owner planning an already owned path fails as a conflict. +- A planned path that exists in storage but is not recorded in state fails as unmanaged content unless forced replacement is explicitly selected. + +When an owner publishes, unrelated owner records and output records are preserved. The publishing owner's record is updated with the latest source manifest, reconciliation mode, and latest primary URL when present. + +## Shared-Root Timestamps + +For shared-root state: + +- top-level `created_at` remains the original shared-root state creation time; +- top-level `updated_at` changes after a successful state write; +- output `created_at` remains stable for an existing managed path; +- output `updated_at` changes only when that path is rewritten; +- newly managed output paths receive the publication time for both output timestamps. + +## Shared-Root Migration + +If `state.mode: shared_root` is configured and existing state is a compatible single-owner `.distributor.json` for the same pipeline id and destination id, the next successful publish writes schema version `3` shared-root state for that owner. + +If existing single-owner state belongs to a different pipeline or destination, publish fails as a conflict. `distributor` does not implicitly take over unrelated state or unmanaged files. + ## Boundaries Destination state is internal managed state written by `distributor`. Operators may inspect it during recovery, but normal workflows should not edit it by hand. Source `manifest.json` is not copied as destination state. diff --git a/docs/internal/publish.md b/docs/internal/publish.md index a29ac2e..9e81804 100644 --- a/docs/internal/publish.md +++ b/docs/internal/publish.md @@ -10,7 +10,7 @@ Audience: developers and LLM coding agents changing `internal/publish`. Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, state policy, reconciliation policy, transformer resolver, transfer policy, distributor version, and force flag. -Output from planning is a `Plan` with action, reason, destination identity, selected outputs, state mode, owner scope, reconciliation mode, optional existing single-owner or shared-root state, optional primary URL, and force metadata. Shared-root plans also expose other-owner outputs to preserve, current-owner outputs retained by merge, current-owner outputs deleted by replace, and current-owner outputs to write. Execution writes selected source outputs, generated outputs, and `.distributor.json` for executable single-owner publish or replacement actions. +Output from planning is a `Plan` with action, reason, destination identity, selected outputs, state mode, owner scope, reconciliation mode, optional existing single-owner or shared-root state, optional primary URL, and force metadata. Shared-root plans also expose other-owner outputs to preserve, current-owner outputs retained by merge, current-owner outputs deleted by replace, and current-owner outputs to write. Execution writes selected source outputs, generated outputs, and `.distributor.json` for executable publish or replacement actions. ## Boundaries @@ -38,7 +38,7 @@ Execution writes destination state after selected outputs are written. Destinati `skip_same` and `skip_destination_newer` execute as no-ops. Replacement-mode single-owner updates remove managed output paths from existing state plus `.distributor.json`, verify the destination is empty, and write state whose outputs are exactly the new plan. Replacement-mode shared-root updates remove only current-owner omitted outputs and preserve unrelated owners. Merge-mode updates retain omitted managed outputs, overwrite only paths already recorded as managed, reject unmanaged destination path collisions, and write cumulative output state. Failed writes trigger cleanup where practical; merge cleanup removes only newly created outputs from the failed attempt. -Shared-root execution writes schema version `3` state. It preserves unrelated owner records and outputs, updates only the publishing owner metadata, preserves root `created_at`, and updates root `updated_at` after successful state writes. +Shared-root execution writes schema version `3` state. It preserves unrelated owner records and outputs, updates only the publishing owner metadata, preserves root `created_at`, and updates root `updated_at` after successful state writes. Compatible single-owner state for the same pipeline and destination is converted to shared-root state on successful publish. Forced replacement is explicit per request and deletes the bounded destination bundle path before writing new outputs and state. @@ -46,7 +46,7 @@ Forced replacement is explicit per request and deletes the bounded destination b Planning fails for incomplete requests, invalid publish/transform policy, invalid state mode, invalid reconciliation mode, output path collisions, invalid destination state, unmanaged destination content without force, shared-root owner path conflicts, conflict outcomes not allowed by transfer policy, unresolved transforms, invalid Markdown output selection, and invalid link URL planning. -Execution fails on delete, read, transform output, unmanaged merge path collision, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable publish or replacement actions. +Execution fails on delete, read, transform output, unmanaged merge path collision, shared-root ownership conflict, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable publish or replacement actions. ## Tests To Inspect diff --git a/docs/internal/state.md b/docs/internal/state.md index 7a4523d..25c3129 100644 --- a/docs/internal/state.md +++ b/docs/internal/state.md @@ -32,12 +32,16 @@ Schema version `1` state remains readable. Parsing infers `state.mode: single_ow Schema version `3` is shared-root state. It records `state.mode: shared_root`, shared state timestamps, owner records keyed by pipeline id and destination id, each owner's latest source manifest and reconciliation metadata, optional owner primary links, and output records for every managed path. Shared-root output records include owner ids and compact source identity fields for source id, digest, and creation time. +Shared-root publish conversion is explicit. Compatible single-owner state for the same pipeline and destination can be projected into the current owner scope by publish execution. Unrelated single-owner state remains a conflict. + Embedded source manifests are parsed and validated through `internal/bundle`, which delegates source manifest semantics to `pkg/bundle`. Output records require clean paths, `source` or `generated` kind, valid source paths, lowercase SHA-256 digests, non-negative sizes, created and updated timestamps, and transform ids for generated outputs. Stored URLs must pass `internal/link` validation. The package also provides helpers for finding output records by path, projecting planned publish outputs into timestamped state outputs, merging retained and newly planned output records, and computing managed output paths from single-owner state. For shared-root state, helpers parse either state shape, identify the current owner scope, return an owner's latest source manifest, list managed paths for one owner or all owners, detect path ownership conflicts, project planned owner outputs, merge one owner's planned outputs while preserving unrelated owners, and replace one owner's outputs by removing that owner's omitted outputs. +Shared-root helper projections preserve output `created_at` for existing managed paths and use the current publication time for rewritten `updated_at`. Root-level `created_at` preservation is owned by publish execution. + ## Skip And Resume Behavior Comparison is pure. It returns outcomes for absent state, unmanaged content, invalid state, pipeline/destination mismatch, same source manifest, older destination, newer destination, same-created digest conflict, and different source id conflict. It does not decide whether to skip, replace, force, or fail; publish planning maps outcomes to actions. diff --git a/docs/operations.md b/docs/operations.md index a388525..b1751d7 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -79,6 +79,10 @@ When destination state is older than the source, `transfer.on_destination_older` For both modes, retained or overwritten paths are identified only from `.distributor.json`; unmanaged files are not adopted. +For `state.mode: shared_root`, one destination root may contain outputs from multiple pipeline/destination owners. Comparisons, replacement, and merge retention are scoped to the current owner. Outputs owned by other owners are preserved. A planned output path owned by another owner fails as a conflict, and a planned path that exists in storage but is not recorded in state fails as unmanaged content by default. + +If `state.mode: shared_root` is configured on a destination whose existing single-owner state belongs to the same pipeline and destination, the next successful publish converts that state file to shared-root schema. Existing single-owner state for a different pipeline or destination remains a conflict. + If a write fails after some outputs were written, `distributor` attempts cleanup before returning the error. In `replace` mode, cleanup removes outputs written during that failed attempt. In `merge` mode, cleanup removes only newly created outputs from that failed attempt; overwritten managed outputs are left in place because they previously belonged to the managed set. Operators should still inspect the destination after a failed write before retrying. Fan-out destinations are independent. If one destination fails after planning or execution begins, later destinations are still attempted. The command exits non-zero if any destination failed. @@ -87,9 +91,11 @@ Fan-out destinations are independent. If one destination fails after planning or `run --dry-run` loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, `.distributor.json`, or SSH `known_hosts` entries. For reconciliation, dry runs report the same high-level action labels as execution; inspect the configured destination's `reconciliation.mode` to determine whether `replace_older` will replace the managed set or merge into it. +For shared-root destinations, dry runs are owner-scoped. A `replace_older` action replaces or merges only the current owner according to `reconciliation.mode`; unrelated owners remain managed by the shared-root state. + Review these action labels before publishing: -- `publish_new`: destination is empty and unmanaged. +- `publish_new`: destination state is absent, or a shared-root owner is absent and planned paths are publishable. - `replace_older`: destination state is older than the source. - `skip_same`: destination state already matches the source. - `skip_destination_newer`: destination state is newer than the source and is skipped. @@ -116,6 +122,8 @@ Forced replacement can claim unmanaged non-empty destination paths. State confli Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete parent paths, sibling paths, or storage outside the destination bundle path. For fixed destinations, the destination bundle path is the backend root, so a forced replacement can clear that configured root. +For shared-root destinations, forced replacement also deletes the configured destination bundle path before writing new shared-root state. This removes unrelated owners inside that destination root. Preview with `--dry-run --force` and confirm the destination path before applying. + `--force` applies only to the current invocation. There is no config field that enables forced replacement by default. ## HTTP Upload Operation @@ -200,7 +208,7 @@ S3 execution uses the AWS SDK for Go v2. See [S3-Compatible Storage Integration] When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is. -Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Merge publication retains omitted managed objects and may overwrite existing managed objects. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers. +Normal single-owner replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Shared-root replacement deletes only current-owner omitted output objects and rewrites the shared state object. Merge publication retains omitted managed objects and may overwrite existing managed objects. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers. ## Secrets Operation @@ -215,6 +223,7 @@ Use these recovery boundaries: - For source validation failures, regenerate the source bundle and manifest together. - For an empty or missing destination, rerun after fixing config or storage access. - For unmanaged destination content, move unrelated files aside or use a different destination path before publishing. +- For shared-root ownership conflicts, change one owner so it writes a different destination path, or use a separate destination root. - For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun `--dry-run`. In merge mode, retained outputs may be intentional managed outputs from the prior state. - For state conflicts, verify the source, pipeline, destination, and existing `.distributor.json` before considering `--force`. - For HTTP upload failures, inspect `/runs/` while retained; after expiry or restart, rely on destination state and logs/output from the publishing run. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 45f3cf0..e138314 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -230,6 +230,40 @@ Safe fix: adjust source files or publish/transform policy so copied and generate Reference: [Configuration](config.md#publish-and-transform-policy). +## Shared-Root Ownership Conflict + +Symptom: `fail_conflict` with a reason like `destination output path ... is owned by /`. + +Likely cause: a `state.mode: shared_root` destination planned an output path already recorded in `.distributor.json` for another pipeline/destination owner. + +Diagnostic: + +```sh +cat /.distributor.json +go run ./cmd/distributor run --config --dry-run +``` + +Safe fix: change one owner so it writes a different output path, use a separate destination root, or intentionally replace the whole destination root only after previewing with `--dry-run --force`. + +Reference: [Operations](operations.md#destination-state-and-retry-behavior). + +## Shared-Root Unmanaged Path Collision + +Symptom: `fail_unmanaged` with a reason like `destination output path ... exists but is not managed by destination state`. + +Likely cause: a `state.mode: shared_root` destination planned a new output path that already exists in storage but is not recorded as managed in `.distributor.json`. + +Diagnostic: + +```sh +find -maxdepth 2 -print +cat /.distributor.json +``` + +Safe fix: move the unmanaged file aside, change the planned output path, or use forced replacement only when deleting the configured destination root is intended. + +Reference: [Operations](operations.md#forced-replacement-workflow). + ## Run Failed After Writing Some Files Symptom: a destination write failed and the command exited non-zero after partial work. diff --git a/examples/shared-root.yml b/examples/shared-root.yml new file mode 100644 index 0000000..a7aa326 --- /dev/null +++ b/examples/shared-root.yml @@ -0,0 +1,31 @@ +pipelines: + - id: example-shared-root-source + source: + backend: local + path: examples/source-bundle + destinations: + - id: shared-local-root + backend: local + path: workspace/published/shared-root + state: + mode: shared_root + publish: + source: true + html: false + - id: example-shared-root-html + source: + backend: local + path: examples/source-bundle + destinations: + - id: shared-local-root + backend: local + path: workspace/published/shared-root + state: + mode: shared_root + publish: + source: false + html: true + transform: + markdown_to_html: + enabled: true + mode: sidecar diff --git a/internal/config/load_test.go b/internal/config/load_test.go index 9622ac8..ded5c55 100644 --- a/internal/config/load_test.go +++ b/internal/config/load_test.go @@ -908,6 +908,7 @@ func TestExampleConfigsLoad(t *testing.T) { "../../examples/fan-out.yml", "../../examples/archive-and-latest.yml", "../../examples/merge-reconciliation.yml", + "../../examples/shared-root.yml", "../../examples/http-upload-local.yml", "../../examples/ssh-destination.yml", "../../examples/s3-destination.yml",