eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Synchronize catalog documentation and examples

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-19 16:50:40 +00:00
parent 9a2eaf8e5e
commit 484fda2514
16 changed files with 296 additions and 871 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
docs/internal/publish.md
View File

@@ -4,13 +4,13 @@ Audience: developers and LLM coding agents changing `internal/publish`.
## Purpose
`internal/publish` plans and executes publication for one validated source bundle and one destination bundle path. It owns destination comparison mapping, output selection, URL planning, managed cleanup selection, replacement safety, and destination state projection.
`internal/publish` plans and executes publication for one validated source bundle and one destination bundle path. It owns output selection, URL planning, catalog action selection, managed cleanup selection, forced replacement safety, and destination state projection.
## Inputs And Outputs
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, state policy, reconciliation policy, takeover policy, transformer resolver, transfer policy, distributor version, and force flag.
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, workflow, transformer resolver, distributor version, force flag, and request time.
Output from planning is a `Plan` with action, reason, destination identity, selected outputs, state mode, owner scope, reconciliation mode, takeover mode, optional existing single-owner or shared-root state, optional primary URL, and force metadata. Shared-root plans also expose other-owner outputs to preserve, taken-over outputs, current-owner outputs retained by merge, current-owner outputs deleted by replace or takeover, and current-owner outputs to write. Execution writes selected source outputs, generated outputs, and `.distributor.json` for executable publish or replacement actions.
Output from planning is a `Plan` with action, reason, destination identity, selected outputs, workflow, owner scope, optional existing catalog state, optional superseded legacy marker, optional primary URL, catalog outputs to write, catalog outputs to retain, catalog outputs to delete, force metadata, and clear-root metadata. Execution writes selected source outputs, generated outputs, and schema version `4` `.distributor.json` for executable catalog actions.
## Boundaries
@@ -20,7 +20,7 @@ External destination state semantics are documented in `docs/integrations/destin
## Config Fields Used
The package consumes already-defaulted config values for destination `publish`, `transform`, `links`, `state`, `reconciliation`, `takeover`, `transfer`, and path mapping mode. It uses `config.ValidatePublishTransformPolicy` for publish/transform consistency.
The package consumes already-defaulted config values for destination `workflow`, `publish`, `transform`, `links`, and path mapping mode. It uses `config.ValidatePublishTransformPolicy` for publish/transform consistency.
## Adapters Used
@@ -28,27 +28,34 @@ The package depends on `internal/storage.Backend` for source and destination IO,
## State And Manifest Behavior
Planning inspects destination state through `internal/state`, compares it with the source manifest, and maps comparison outcomes plus transfer and takeover policy into actions. Catalog planning uses `publish_new`, `upsert_additive`, `replace_catalog`, `force_replace`, `fail_conflict`, or `fail_unmanaged`; legacy action labels remain only for older planning and removal work scheduled separately.
Planning inspects destination state through `internal/state` and maps catalog conditions into actions:
Single-owner destinations compare the whole destination state against the configured pipeline and destination ids. Valid managed identity and source conflicts can become `replace_takeover` when `takeover.mode` allows them. Shared-root destinations compare only the current owner scope, keyed by pipeline id and destination id. An absent shared-root owner is publishable for that owner unless a planned output collides with unmanaged storage content. Planned writes to a path owned by another shared-root owner become `replace_takeover` when `takeover.mode` allows that managed output path to move to the current owner.
- `publish_new`: no valid state exists and the destination bundle path is empty.
- `upsert_additive`: additive workflow writes planned outputs and retains unrelated catalog outputs.
- `replace_catalog`: replacement workflow writes planned outputs and deletes omitted outputs for the current owner.
- `force_replace`: explicit bounded replacement selected by `Force`.
- `fail_unmanaged`: unmanaged destination content blocks publication.
- `fail_conflict`: invalid or unsupported state blocks publication.
Execution writes destination state after selected outputs are written. Destination state includes copied source output metadata, generated output metadata, output timestamps, embedded source manifest, reconciliation metadata, link metadata when configured, pipeline id, destination id, and publication timestamps.
Superseded legacy state is identified by schema number and planned through catalog output projection. A successful publish writes schema version `4` catalog state.
## Skip And Resume Behavior
Execution writes destination state after selected outputs are written. Catalog output records include owner identity, compact source identity, copied source output metadata, generated output metadata, output timestamps, and optional URL metadata.
`skip_same` and `skip_destination_newer` execute as no-ops. Replacement-mode single-owner updates remove managed output paths from existing state plus `.distributor.json`, verify the destination is empty, and write state whose outputs are exactly the new plan. `replace_conflict` uses managed replacement mechanics and does not retain omitted outputs through merge reconciliation. `replace_newer` follows the same managed replacement and merge-retention rules as `replace_older`. `replace_takeover` uses managed replacement mechanics and does not retain omitted outputs through merge reconciliation. Replacement-mode shared-root updates remove only current-owner omitted outputs and preserve unrelated owners. Shared-root takeover rewrites only the taken-over output records and current owner records. Merge-mode same-source updates retain omitted managed outputs, overwrite only paths already recorded as managed, reject unmanaged destination path collisions, and write cumulative output state. Failed writes trigger cleanup where practical; same-source merge cleanup removes only newly created outputs from the failed attempt.
## Workflow Behavior
Shared-root execution writes schema version `3` state. It preserves unrelated owner records and outputs, updates only the publishing owner metadata, preserves root `created_at`, and updates root `updated_at` after successful state writes. Compatible single-owner state for the same pipeline and destination is converted to shared-root state on successful publish.
Additive workflow computes a write set for the planned outputs and preserves catalog outputs for unplanned paths. Existing catalog records for planned paths are replaced by the current owner and source identity.
Forced replacement is explicit per request and deletes the bounded destination bundle path before writing planned outputs and schema version `4` catalog state. Catalog planning selects `force_replace` only when `Force` is true and normal planning would otherwise fail for a non-empty no-state destination, a planned path collision with unmanaged storage content, invalid destination state, or unsupported future destination state. Valid catalog-managed additive upserts and replacement workflow publishes are normal managed actions; they are not forced replacement actions.
Replacement workflow computes a write set for the planned outputs, preserves other-owner outputs, and deletes omitted outputs owned by the current pipeline and destination. It does not need `Force`.
Forced replacement is explicit per request. It deletes the bounded destination bundle path before writing planned outputs and schema version `4` catalog state. Catalog planning selects `force_replace` only when `Force` is true and normal planning would otherwise fail for a non-empty no-state destination, a planned path collision with unmanaged storage content, invalid destination state, or unsupported future destination state.
Retention pruning is not part of publish execution and does not run automatically after a successful publish. The app-level prune workflow uses destination state after publication to select managed outputs for deletion.
## Failure Behavior
Planning fails for incomplete requests, invalid publish/transform policy, invalid state mode, invalid reconciliation mode, output path collisions, invalid destination state, unmanaged destination content without force, shared-root owner path conflicts not allowed by `takeover.mode`, conflict outcomes not allowed by transfer policy, unresolved transforms, invalid Markdown output selection, and invalid link URL planning.
Planning fails for incomplete requests, invalid publish/transform policy, invalid workflow, output path collisions, unresolved transforms, invalid Markdown output selection, invalid link URL planning, invalid destination state without force, unmanaged destination content without force, and unsupported future state without force.
Execution fails on delete, read, transform output, unmanaged merge path collision, shared-root ownership conflict, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable publish or replacement actions.
Execution fails on delete, read, transform output, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable catalog publish or replacement actions.
## Tests To Inspect
@@ -62,11 +69,8 @@ Execution fails on delete, read, transform output, unmanaged merge path collisio
- Planning is deterministic for the same request and destination state.
- Destination bundle paths are caller-supplied and backend-root-relative.
- URL generation uses URL path semantics and never infers public URLs from backend config.
- Replacement reconciliation deletes only managed paths recorded in existing state plus `.distributor.json` for single-owner state, and only current-owner omitted outputs for shared-root state.
- Merge reconciliation never adopts unmanaged content.
- Merge state output records are cumulative for the single owner.
- Shared-root planning is owner-scoped, preserves unrelated owner outputs, and records taken-over managed output paths separately from unrelated owners.
- Shared-root execution writes owner-scoped changes without deleting unrelated owners.
- Additive workflow never adopts unmanaged storage content.
- Replacement workflow deletes only catalog outputs owned by the current pipeline and destination unless force replacement clears the bounded destination bundle path.
- Forced replacement deletes only within the supplied destination bundle path and then writes catalog state using the same output projection as normal catalog publish planning.
- Destination state is written after selected outputs are written.
- Transform resolution stays behind a caller-supplied interface.