eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Synchronize catalog documentation and examples

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-19 16:50:40 +00:00
parent 9a2eaf8e5e
commit 484fda2514
16 changed files with 296 additions and 871 deletions
Download Patch File Download Diff File Expand all files Collapse all files

222
docs/config.md
View File

@@ -2,11 +2,11 @@
Audience: administrators, operators, and advanced users who write YAML configuration for `distributor`.
This document is the canonical user-facing configuration reference. CLI syntax lives in [CLI](cli.md), operating procedures live in [Operations](operations.md), symptom-oriented recovery lives in [Troubleshooting](troubleshooting.md), and external contracts live under [Integrations](integrations/source-bundle.md).
This is the canonical user-facing configuration reference. CLI syntax lives in [CLI](cli.md), operations guidance lives in [Operations](operations.md), recovery guidance lives in [Troubleshooting](troubleshooting.md), and file-format contracts live under [Integrations](integrations/source-bundle.md).
## Config File Loading
`distributor run --config <path>` and `distributor serve --config <path>` load the YAML file at `<path>`. If `--config` is omitted, both commands use:
`distributor run --config <path>` and `distributor serve --config <path>` load the YAML file at `<path>`. If `--config` is omitted, commands use:
```text
/usr/local/etc/distributor/config.yml
@@ -21,7 +21,7 @@ Runtime backend support is command-specific:
- `serve` uses `http_upload` sources through the HTTP upload API and publishes to configured `local`, `ssh`, and `s3` destinations.
- `http_upload` is valid only as a source backend.
## Minimal Local Config
## Minimal Working Config
```yaml
pipelines:
@@ -35,9 +35,9 @@ pipelines:
path: /srv/reports/archive
```
This config publishes source files only. It uses default validation, destination path mapping, publish, state, reconciliation, retention, transfer, and HTTP server values.
This publishes source files only. It uses default validation, additive workflow, preserve-relative path mapping, source-only publish policy, disabled pruning, and default HTTP server values.
## Production-Oriented Local Config
## Production-Oriented Config
```yaml
server:
@@ -61,23 +61,20 @@ pipelines:
- id: archive
backend: local
path: /srv/reports/archive
workflow: additive
publish:
source: true
html: false
path_mapping:
mode: preserve_relative
reconciliation:
mode: replace
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
retention:
prune:
enabled: false
```
## HTTP Upload Source Config
HTTP upload sources are configured on pipelines and are served by `distributor serve`. Upload tokens are resolved from the process environment or `secrets.directory`; literal bearer tokens are not configured in YAML.
HTTP upload sources are configured on pipelines and served by `distributor serve`. Upload tokens are resolved from the process environment or `secrets.directory`; literal bearer tokens are not configured in YAML.
```yaml
server:
@@ -97,15 +94,13 @@ pipelines:
- id: weather-daily
source:
backend: http_upload
staging_path: /var/spool/distributor/weather-daily
max_upload_size: 20MB
destinations:
- id: archive
backend: local
path: /srv/reports/archive
```
`upload_tokens` is required when any pipeline source uses `http_upload`. Each token record resolves its bearer token value from the process environment or `secrets.directory`. `allow_pipelines` lists configured upload pipeline ids that the token may submit to. One token may authorize multiple upload pipelines, and multiple tokens may authorize the same upload pipeline.
`upload_tokens` is required when any pipeline source uses `http_upload`. Each token record resolves its bearer token value from the process environment or `secrets.directory`. `allow_pipelines` lists configured upload pipeline ids that the token may submit to.
For `http_upload` sources, `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`. `max_upload_size` defaults to `server.http.max_upload_size`.
@@ -136,7 +131,7 @@ See [Secrets](#secrets) for resolution rules.
Each token has:
- `id`: required unique slug-like identifier for the token record. It must start with a letter or number and may contain letters, numbers, `.`, `_`, and `-`.
- `id`: required unique slug-like identifier for the token record.
- `token_env`: required environment variable or secret-file name containing the bearer token value.
- `allow_pipelines`: required non-empty list of configured pipeline ids whose source backend is `http_upload`.
@@ -148,12 +143,12 @@ Token values must resolve to non-empty strings and must be unique across token r
Each pipeline has:
- `id`: required unique slug-like identifier. It must start with a letter or number and may contain letters, numbers, `.`, `_`, and `-`.
- `id`: required unique slug-like identifier.
- `source`: required source backend config.
- `validation`: optional validation policy.
- `destinations`: required non-empty destination list.
Pipeline ids must be unique across the config.
Slug-like identifiers must start with a letter or number and may contain letters, numbers, `.`, `_`, and `-`.
## Backend Reference
@@ -171,7 +166,7 @@ path: /srv/distributor/archive
### SSH/SFTP Backend
SSH backends use native SFTP and can be used as sources and destinations. Adapter protocol behavior is documented in [SSH/SFTP Integration](integrations/ssh-sftp.md).
SSH backends use native SFTP and can be used as sources and destinations. Adapter behavior is documented in [SSH/SFTP Integration](integrations/ssh-sftp.md).
```yaml
backend: ssh
@@ -193,17 +188,11 @@ host_key_policy: strict
- `known_hosts`: optional OpenSSH `known_hosts` path.
- `host_key_policy`: optional host key policy. Default: `accept-new`.
Accepted host key policy values:
- `strict` or boolean `true`: require a matching known host key.
- `accept-new`: accept and persist a new host key, but reject changed known keys.
- `off` or boolean `false`: disable host key checking.
Authentication uses SSH agent identities when `SSH_AUTH_SOCK` is available, then `ssh_key_file` when configured. Password authentication is not configured in YAML.
Accepted host key policy values are `strict` or boolean `true`, `accept-new`, and `off` or boolean `false`. Authentication uses SSH agent identities when `SSH_AUTH_SOCK` is available, then `ssh_key_file` when configured. Password authentication is not configured in YAML.
### S3-Compatible Backend
S3 backends can be used as sources and destinations. Adapter protocol behavior is documented in [S3-Compatible Storage Integration](integrations/s3.md).
S3 backends can be used as sources and destinations. Adapter behavior is documented in [S3-Compatible Storage Integration](integrations/s3.md).
```yaml
backend: s3
@@ -255,42 +244,48 @@ Source bundle digest mismatches fail validation before destination writes occur.
## Destination Fields
Each destination embeds a backend config at the destination level and may also configure publishing, transforms, path mapping, links, state, reconciliation, retention, and transfer behavior.
Each destination embeds a backend config at the destination level and may also configure workflow, publishing, transforms, path mapping, links, and retention.
```yaml
destinations:
- id: archive
backend: local
path: /srv/reports/archive
workflow: additive
publish:
source: true
html: false
path_mapping:
mode: preserve_relative
state:
mode: single_owner
reconciliation:
mode: replace
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
```
- `id`: required unique slug-like identifier within the pipeline.
- Backend fields: required according to the selected destination backend.
- `workflow`: optional catalog update workflow. Default: `additive`.
- `publish`: optional publish policy. Default: source-only publication.
- `transform`: required only when publishing generated HTML.
- `path_mapping`: optional destination path mapping policy.
- `links`: optional public URL metadata policy.
- `state`: optional destination state ownership policy.
- `reconciliation`: optional managed-output reconciliation policy.
- `retention`: optional managed-output retention policy.
- `transfer`: optional destination comparison action policy.
Destination ids must be unique within a pipeline.
Legacy destination policy keys `state`, `reconciliation`, `takeover`, and `transfer` are not accepted config fields. YAML files containing those keys fail during config loading.
## Destination Workflow
```yaml
workflow: additive
```
- `workflow`: optional. Accepted values are `additive` and `replacement`; default is `additive`.
`additive` writes planned outputs into the catalog and retains unrelated catalog-managed outputs in the same destination bundle path. Existing catalog records for planned paths are replaced by the current publication. A planned path that exists in storage but is not recorded in valid catalog state fails as unmanaged content unless `run --force` selects `force_replace`.
`replacement` writes planned outputs for the current pipeline and destination, and removes catalog outputs owned by the same pipeline and destination when those outputs are omitted from the new plan. Outputs owned by other pipeline/destination pairs remain catalog-managed. Replacement workflow is normal managed behavior and does not require `--force`.
Use `replacement` for stable latest-style destinations where the current owner should publish exactly the currently planned output set. Use `additive` when a destination root intentionally accumulates outputs over time or receives disjoint outputs from multiple configured destinations.
## Publish And Transform Policy
### Source-Only Publication
@@ -365,113 +360,7 @@ Primary URL policies:
- `html`: use the first generated HTML output.
- `source`: use the first copied source output.
If no output matches the primary policy, per-output URLs may still be recorded and the top-level primary URL is omitted.
## Destination State Policy
```yaml
state:
mode: single_owner
```
- `state.mode`: optional. Accepted values are `single_owner` and `shared_root`; default is `single_owner`.
`single_owner` state records one pipeline/destination owner for each destination bundle path and is the default state mode.
`shared_root` records multiple pipeline/destination owners in one destination root. Publish execution preserves unrelated owners, rejects path ownership conflicts, and writes shared-root destination state.
Use `shared_root` when multiple configured destinations intentionally write disjoint output paths into the same backend root:
```yaml
pipelines:
- id: reports-source
source:
backend: local
path: /var/spool/distributor/reports
destinations:
- id: shared-root
backend: local
path: /srv/reports/shared
state:
mode: shared_root
publish:
source: true
html: false
- id: reports-html
source:
backend: local
path: /var/spool/distributor/reports
destinations:
- id: shared-root
backend: local
path: /srv/reports/shared
state:
mode: shared_root
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
Every output path in a shared root belongs to exactly one `pipeline_id` and `destination_id`. A different owner planning the same path fails as a conflict unless `takeover.mode` allows the managed output path to move to the current owner.
## Takeover Policy
```yaml
takeover:
mode: same_pipeline
```
- `takeover.mode`: optional. Accepted values are `same_pipeline`, `same_source`, `any_managed`, and `never`; default is `same_pipeline`.
Takeover controls when a destination may normally replace valid distributor-managed state whose pipeline, destination, source identity, or shared-root output owner differs from the current publication.
- `same_pipeline`: replace managed state owned by the same pipeline. The previous destination id and source id may differ.
- `same_source`: replace managed state only when the existing source manifest id matches the current source id.
- `any_managed`: replace any valid distributor-managed state at the selected destination bundle path.
- `never`: do not replace identity or source conflicts without the explicit forced replacement workflow.
For shared-root state, `same_pipeline` permits taking over output paths owned by another destination in the same pipeline, `same_source` permits taking over output paths whose owner records the same source manifest id, and `any_managed` permits taking over output paths owned by any valid shared-root owner. Unrelated owner records and non-conflicting outputs remain managed by their existing owners.
Takeover does not apply to unmanaged content, invalid destination state, same-created digest conflicts, or same-source destination-newer comparisons.
## Reconciliation Policy
```yaml
reconciliation:
mode: replace
```
- `reconciliation.mode`: optional. Accepted values are `replace` and `merge`; default is `replace`.
Reconciliation controls how a destination with older managed state is updated after transfer policy selects `replace_older`.
`replace` deletes the prior managed outputs recorded in `.distributor.json`, deletes the state file, verifies the destination bundle path is empty, then writes only the newly planned outputs and a new state file. This is the default and is appropriate when each publication should exactly match the current publish and transform policy.
`merge` keeps prior managed outputs that are not produced by the new plan. Planned paths already recorded in existing state may be overwritten; planned paths that already exist in storage but are not recorded as managed fail as unmanaged collisions. The resulting state file records the cumulative managed output set.
Example merge destination:
```yaml
destinations:
- id: static-site
backend: local
path: /srv/reports/site
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
reconciliation:
mode: merge
```
`links.primary_url` is selected from the newly planned outputs for the current run. Retained outputs keep their prior output metadata and timestamps.
If no output matches the primary policy, per-output URLs may still be recorded and no primary URL is reported for the run.
## Retention Policy
@@ -489,28 +378,9 @@ retention:
When `retention.prune.enabled` is `true`, at least one of `older_than` or `keep_latest` is required. `older_than` must be greater than zero, and `keep_latest` must be zero or greater.
Pruning uses managed output `updated_at` timestamps from destination state. If both `keep_latest` and `older_than` are set, the newest `keep_latest` outputs are preserved first, then age-based pruning is applied to the remaining managed outputs.
Pruning uses catalog output `updated_at` timestamps from destination state. If both `keep_latest` and `older_than` are set, the newest `keep_latest` outputs are preserved first, then age-based pruning is applied to the remaining managed outputs.
The `prune` command is owner-scoped for shared-root state. `prune --dry-run` reports selected managed outputs without writing. `prune --apply` deletes only selected managed output paths and rewrites destination state after confirmed deletes. It does not delete unmanaged files or `.distributor.json`, and it does not run automatically after `run`.
## Transfer Policy
```yaml
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
```
Transfer fields and accepted values:
- `transfer.on_destination_same`: `skip` or `fail`. Default: `skip`.
- `transfer.on_destination_older`: `replace` or `fail`. Default: `replace`.
- `transfer.on_destination_newer`: `skip`, `replace`, or `fail`. Default: `skip`.
- `transfer.on_conflict`: `fail` or `replace`. Default: `fail`.
`replace` for `on_destination_newer` and `on_conflict` applies only to valid distributor-managed destination state. Unmanaged destination content and invalid destination state still require the explicit `run --force` workflow where supported.
The `prune` command is scoped to the selected pipeline and destination owner. `prune --dry-run` reports selected managed outputs without writing. `prune --apply` deletes only selected managed output paths and rewrites destination state after confirmed deletes. It does not delete unmanaged files or `.distributor.json`, and it does not run automatically after `run`.
## Size And Duration Values
@@ -543,18 +413,12 @@ Defaults are applied after YAML decoding and before validation:
- S3 `force_path_style: true`
- `http_upload` source `staging_path: <server.http.staging_root>/<pipeline id>`
- `http_upload` source `max_upload_size: server.http.max_upload_size`
- `workflow: additive`
- `publish.source: true` and `publish.html: false`
- `transform.markdown_to_html.mode: sidecar` when a Markdown transform block is present and mode is omitted
- `path_mapping.mode: preserve_relative`
- `links.primary: auto` when a `links` block is present and `primary` is omitted
- `state.mode: single_owner`
- `reconciliation.mode: replace`
- `takeover.mode: same_pipeline`
- `retention.prune.enabled: false`
- `transfer.on_destination_same: skip`
- `transfer.on_destination_older: replace`
- `transfer.on_destination_newer: skip`
- `transfer.on_conflict: fail`
## Secrets
@@ -577,7 +441,7 @@ Fields resolved through this resolver:
## Maintained Examples
Maintained examples live under [examples](../examples/). Config tests load every file listed here.
Maintained examples live under [examples](../examples/). Config tests load these YAML files.
Local examples:
@@ -587,8 +451,8 @@ Local examples:
- `local-index.yml`: local `index.html` publication.
- `fan-out.yml`: local fan-out publication to source and HTML destinations.
- `archive-and-latest.yml`: local archive plus fixed latest publication.
- `merge-reconciliation.yml`: local HTML publication using merge reconciliation.
- `shared-root.yml`: two local pipelines publishing disjoint outputs into one shared destination root.
- `additive-workflow.yml`: two destinations publishing disjoint outputs into one catalog-managed root.
- `replacement-workflow.yml`: fixed-path replacement workflow for a stable latest-style output set.
- `http-upload-local.yml`: local HTTP upload server config; requires `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN` in the process environment or as a secret-file name before running `serve`.
Environment-gated remote examples: