Files

Eric Rakestraw c67ecf86a9 Add prune retention planning

2026-06-08 19:26:48 +00:00

22 KiB

Raw Blame History

Configuration Reference

Audience: administrators, operators, and advanced users who write YAML configuration for distributor.

This document is the canonical user-facing configuration reference. CLI syntax lives in CLI, operating procedures live in Operations, symptom-oriented recovery lives in Troubleshooting, and external contracts live under Integrations.

Config File Loading

distributor run --config <path> and distributor serve --config <path> load the YAML file at <path>. If --config is omitted, both commands use:

/usr/local/etc/distributor/config.yml

YAML decoding rejects unknown fields. Defaults are applied after decoding and before validation.

Runtime backend support is command-specific:

run, validate --config, and inspect --config execute local, ssh, and s3 sources.
run executes local, ssh, and s3 destinations.
serve uses http_upload sources through the HTTP upload API and publishes to configured local, ssh, and s3 destinations.
http_upload is valid only as a source backend.

Minimal Local Config

pipelines:
  - id: reports
    source:
      backend: local
      path: /var/spool/distributor/reports
    destinations:
      - id: archive
        backend: local
        path: /srv/reports/archive

This config publishes source files only. It uses default validation, destination path mapping, publish, reconciliation, transfer, and HTTP server values.

Production-Oriented Local Config

server:
  http:
    bind: 127.0.0.1:8080
    staging_root: /var/spool/distributor
    max_upload_size: 20MB
    queue_size: 16
    max_concurrency: 1
    retention: 24h
secrets:
  directory: /run/secrets/distributor
pipelines:
  - id: reports
    source:
      backend: local
      path: /var/spool/distributor/reports
    validation:
      on_digest_mismatch: fail
    destinations:
      - id: archive
        backend: local
        path: /srv/reports/archive
        publish:
          source: true
          html: false
        path_mapping:
          mode: preserve_relative
        reconciliation:
          mode: replace
        transfer:
          on_destination_same: skip
          on_destination_older: replace
          on_destination_newer: skip
          on_conflict: fail

HTTP Upload Source Config

HTTP upload sources are configured on pipelines and are served by distributor serve. Upload tokens are resolved from the process environment or secrets.directory; literal bearer tokens are not configured in YAML.

server:
  http:
    bind: 127.0.0.1:8080
    staging_root: /var/spool/distributor
    max_upload_size: 20MB
    queue_size: 16
    max_concurrency: 1
    retention: 24h
upload_tokens:
  - id: weather-reporter
    token_env: WEATHER_UPLOAD_TOKEN
    allow_pipelines:
      - weather-daily
pipelines:
  - id: weather-daily
    source:
      backend: http_upload
      staging_path: /var/spool/distributor/weather-daily
      max_upload_size: 20MB
    destinations:
      - id: archive
        backend: local
        path: /srv/reports/archive

upload_tokens is required when any pipeline source uses http_upload. Each token record resolves its bearer token value from the process environment or secrets.directory. allow_pipelines lists configured upload pipeline ids that the token may submit to. One token may authorize multiple upload pipelines, and multiple tokens may authorize the same upload pipeline.

For http_upload sources, staging_path defaults to <server.http.staging_root>/<pipeline id>. max_upload_size defaults to server.http.max_upload_size.

Top-Level Fields

`server.http`

server.http controls the HTTP upload server used by serve.

bind: optional TCP bind address. Default: 127.0.0.1:8080.
staging_root: optional root used to default http_upload source staging paths. Default: /var/spool/distributor.
max_upload_size: optional default upload limit for HTTP upload sources. Default: 20MB.
queue_size: optional upload admission queue size. Default: 16.
max_concurrency: optional upload worker concurrency. Default: 1.
retention: optional in-memory completed-run retention duration. Default: 24h.

Numeric server values and durations must be greater than zero after defaults are applied.

`secrets`

directory: optional directory of secret files used by the config-owned credential resolver.

See Secrets for resolution rules.

`upload_tokens`

upload_tokens configures bearer tokens for distributor serve. It is required when any pipeline source backend is http_upload and is invalid when no upload pipelines are configured.

Each token has:

id: required unique slug-like identifier for the token record. It must start with a letter or number and may contain letters, numbers, ., _, and -.
token_env: required environment variable or secret-file name containing the bearer token value.
allow_pipelines: required non-empty list of configured pipeline ids whose source backend is http_upload.

Token values must resolve to non-empty strings and must be unique across token records. Every configured upload pipeline must be allowed by at least one token.

`pipelines`

pipelines is required and must contain at least one pipeline.

Each pipeline has:

id: required unique slug-like identifier. It must start with a letter or number and may contain letters, numbers, ., _, and -.
source: required source backend config.
validation: optional validation policy.
destinations: required non-empty destination list.

Pipeline ids must be unique across the config.

Backend Reference

Local Backend

Local backends can be used as sources and destinations.

backend: local
path: /srv/distributor/archive

backend: required value local.
path: required local filesystem root for this backend.

SSH/SFTP Backend

SSH backends use native SFTP and can be used as sources and destinations. Adapter protocol behavior is documented in SSH/SFTP Integration.

backend: ssh
host: ssh.example.com
user: distributor
port: 22
path: /srv/distributor/archive
ssh_key_file: /home/distributor/.ssh/id_ed25519
known_hosts: /home/distributor/.ssh/known_hosts
host_key_policy: strict

backend: required value ssh.
host: required SSH host.
path: required remote root path.
user: optional SSH username. If omitted, the adapter uses the current OS user when available.
port: optional TCP port. Default: 22.
ssh_key_file: optional private key path.
known_hosts: optional OpenSSH known_hosts path.
host_key_policy: optional host key policy. Default: accept-new.

Accepted host key policy values:

strict or boolean true: require a matching known host key.
accept-new: accept and persist a new host key, but reject changed known keys.
off or boolean false: disable host key checking.

Authentication uses SSH agent identities when SSH_AUTH_SOCK is available, then ssh_key_file when configured. Password authentication is not configured in YAML.

S3-Compatible Backend

S3 backends can be used as sources and destinations. Adapter protocol behavior is documented in S3-Compatible Storage Integration.

backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: distributor/archive
region: us-east-1
force_path_style: true
credentials:
  access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
  secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY

backend: required value s3.
endpoint: required S3-compatible endpoint URL.
bucket: required bucket name.
prefix: optional backend root prefix. Leading and trailing slashes are trimmed; the remaining value must be a clean relative slash-separated path.
region: optional region. Default: us-east-1.
force_path_style: optional addressing mode toggle. Default: true.
credentials.access_key_id_env: optional environment variable or secret-file name for the access key id.
credentials.secret_access_key_env: optional environment variable or secret-file name for the secret access key.

The S3 credential variable names must either both be configured or both be omitted. When omitted, the AWS SDK default credential chain is used. When configured, both values must resolve to non-empty strings through the process environment or secrets.directory.

HTTP Upload Source Backend

HTTP upload backends are valid only as pipeline sources and are served by distributor serve. The API contract is documented in HTTP Upload API Contract.

backend: http_upload
staging_path: /var/spool/distributor/weather-daily
max_upload_size: 20MB

backend: required value http_upload.
staging_path: optional staging path. Default: <server.http.staging_root>/<pipeline id>.
max_upload_size: optional per-source upload limit. Default: server.http.max_upload_size.

Validation Policy

validation:
  on_digest_mismatch: fail

validation.on_digest_mismatch: optional. Default and only accepted value: fail.

Source bundle digest mismatches fail validation before destination writes occur. The manifest file-format contract is documented in Source Bundle Contract.

Destination Fields

Each destination embeds a backend config at the destination level and may also configure publishing, transforms, path mapping, links, state, reconciliation, and transfer behavior.

destinations:
  - id: archive
    backend: local
    path: /srv/reports/archive
    publish:
      source: true
      html: false
    path_mapping:
      mode: preserve_relative
    state:
      mode: single_owner
    reconciliation:
      mode: replace
    transfer:
      on_destination_same: skip
      on_destination_older: replace
      on_destination_newer: skip
      on_conflict: fail

id: required unique slug-like identifier within the pipeline.
Backend fields: required according to the selected destination backend.
publish: optional publish policy. Default: source-only publication.
transform: required only when publishing generated HTML.
path_mapping: optional destination path mapping policy.
links: optional public URL metadata policy.
state: optional destination state ownership policy.
reconciliation: optional managed-output reconciliation policy.
retention: optional managed-output retention policy.
transfer: optional destination comparison action policy.

Destination ids must be unique within a pipeline.

Publish And Transform Policy

Source-Only Publication

publish:
  source: true
  html: false

publish.source controls whether source manifest files are copied to the destination.

Markdown-To-HTML Publication

publish:
  source: false
  html: true
transform:
  markdown_to_html:
    enabled: true
    mode: sidecar

publish.html controls whether generated HTML outputs are published. When publish.html is true, transform.markdown_to_html.enabled must also be true.

Markdown transform fields:

transform.markdown_to_html.enabled: enables Markdown-to-HTML generation for this destination.
transform.markdown_to_html.mode: optional. Accepted values are sidecar and index; default is sidecar when a Markdown transform block is present.
transform.markdown_to_html.input: optional source manifest path for index mode only.

sidecar mode renders every manifest-listed .md file to a same-directory .html output. index mode renders one Markdown source to index.html at the destination bundle path. If index mode omits input, the selected source bundle must contain exactly one Markdown file.

At least one output type must be enabled. Enabled Markdown transforms are rejected when publish.html is false, and input is rejected unless mode is index.

Destination Path Mapping

path_mapping:
  mode: preserve_relative

path_mapping.mode: optional. Accepted values are preserve_relative and fixed; default is preserve_relative.

preserve_relative publishes each discovered source bundle at the same path relative to the destination backend root.

fixed publishes one selected source bundle directly at the destination backend root. Fixed destinations select the newest discovered source bundle by manifest created timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins.

Fixed mapping is useful for stable latest-style paths. Preview fixed destinations with run --dry-run, especially before using --force.

Destination Links

links:
  base_url: https://reports.example.com/archive
  primary: auto

links.base_url: required when links is present. It must be an absolute http or https URL with a host and no query string or fragment.
links.primary: optional. Accepted values are auto, html, and source; default is auto when links is present.

distributor does not infer public URLs from backend config. Destination state URL fields are documented in Destination State Contract. Output URLs are built from links.base_url, the destination bundle path, and output paths using URL path semantics. index.html outputs produce directory-style URLs that omit the filename.

Primary URL policies:

auto: prefer index.html, then generated HTML, then source outputs.
html: use the first generated HTML output.
source: use the first copied source output.

If no output matches the primary policy, per-output URLs may still be recorded and the top-level primary URL is omitted.

Destination State Policy

state:
  mode: single_owner

state.mode: optional. Accepted values are single_owner and shared_root; default is single_owner.

single_owner state records one pipeline/destination owner for each destination bundle path and is the default state mode.

shared_root records multiple pipeline/destination owners in one destination root. Publish execution preserves unrelated owners, rejects path ownership conflicts, and writes shared-root destination state.

Use shared_root when multiple configured destinations intentionally write disjoint output paths into the same backend root:

pipelines:
  - id: reports-source
    source:
      backend: local
      path: /var/spool/distributor/reports
    destinations:
      - id: shared-root
        backend: local
        path: /srv/reports/shared
        state:
          mode: shared_root
        publish:
          source: true
          html: false
  - id: reports-html
    source:
      backend: local
      path: /var/spool/distributor/reports
    destinations:
      - id: shared-root
        backend: local
        path: /srv/reports/shared
        state:
          mode: shared_root
        publish:
          source: false
          html: true
        transform:
          markdown_to_html:
            enabled: true
            mode: sidecar

Every output path in a shared root belongs to exactly one pipeline_id and destination_id. A different owner planning the same path fails as a conflict.

Reconciliation Policy

reconciliation:
  mode: replace

reconciliation.mode: optional. Accepted values are replace and merge; default is replace.

Reconciliation controls how a destination with older managed state is updated after transfer policy selects replace_older.

replace deletes the prior managed outputs recorded in .distributor.json, deletes the state file, verifies the destination bundle path is empty, then writes only the newly planned outputs and a new state file. This is the default and is appropriate when each publication should exactly match the current publish and transform policy.

merge keeps prior managed outputs that are not produced by the new plan. Planned paths already recorded in existing state may be overwritten; planned paths that already exist in storage but are not recorded as managed fail as unmanaged collisions. The resulting state file records the cumulative managed output set.

Example merge destination:

destinations:
  - id: static-site
    backend: local
    path: /srv/reports/site
    publish:
      source: false
      html: true
    transform:
      markdown_to_html:
        enabled: true
        mode: sidecar
    reconciliation:
      mode: merge

links.primary_url is selected from the newly planned outputs for the current run. Retained outputs keep their prior output metadata and timestamps.

Retention Policy

retention:
  prune:
    enabled: false
    older_than: 168h
    keep_latest: 3

retention.prune.enabled: optional boolean. Default is false.
retention.prune.older_than: optional duration. When pruning is enabled, outputs older than this duration are eligible in prune planning.
retention.prune.keep_latest: optional non-negative integer. When pruning is enabled, this many newest managed outputs are preserved before age-based pruning is considered.

When retention.prune.enabled is true, at least one of older_than or keep_latest is required. older_than must be greater than zero, and keep_latest must be zero or greater.

Prune planning uses managed output updated_at timestamps from destination state. It is owner-scoped for shared-root state. The current implementation parses, validates, and plans pruning from this policy; it does not delete destination files.

Transfer Policy

transfer:
  on_destination_same: skip
  on_destination_older: replace
  on_destination_newer: skip
  on_conflict: fail

Transfer fields and accepted values:

transfer.on_destination_same: skip or fail. Default: skip.
transfer.on_destination_older: replace or fail. Default: replace.
transfer.on_destination_newer: skip, replace, or fail. Default: skip.
transfer.on_conflict: fail or replace. Default: fail.

replace for on_destination_newer and on_conflict is honored only when run --force is supplied. There is no config field that enables forced replacement by default.

Size And Duration Values

Upload size fields must be YAML strings with an integer and one of these suffixes:

B
KB
MB
GB

Suffix multipliers use powers of 1024. Values must be greater than zero after defaults are applied.

Duration fields must be YAML strings accepted by Go duration parsing, such as 24h, 90m, or 168h. Values must be greater than zero after defaults are applied.

Defaults

Defaults are applied after YAML decoding and before validation:

server.http.bind: 127.0.0.1:8080
server.http.staging_root: /var/spool/distributor
server.http.max_upload_size: 20MB
server.http.queue_size: 16
server.http.max_concurrency: 1
server.http.retention: 24h
validation.on_digest_mismatch: fail
SSH port: 22
SSH host_key_policy: accept-new
S3 region: us-east-1
S3 prefix: leading and trailing slashes trimmed
S3 force_path_style: true
http_upload source staging_path: <server.http.staging_root>/<pipeline id>
http_upload source max_upload_size: server.http.max_upload_size
publish.source: true and publish.html: false
transform.markdown_to_html.mode: sidecar when a Markdown transform block is present and mode is omitted
path_mapping.mode: preserve_relative
links.primary: auto when a links block is present and primary is omitted
state.mode: single_owner
reconciliation.mode: replace
retention.prune.enabled: false
transfer.on_destination_same: skip
transfer.on_destination_older: replace
transfer.on_destination_newer: skip
transfer.on_conflict: fail

Secrets

Do not put literal secrets in config files. secrets.directory lets deployments provide credential values as files:

secrets:
  directory: /run/secrets/distributor

Each regular file in the directory becomes an internal credential environment value named by the filename. Valid filenames match [A-Za-z_][A-Za-z0-9_]*. Directories are ignored. Symlinks to regular files are followed. Exactly one trailing LF or CRLF is trimmed from each file; other whitespace is preserved.

Credential resolution checks the real process environment first, then secrets.directory. If both define the same name with different values, run emits a warning with the variable name and uses the real environment value. Secret values are not printed. The process environment is not modified, so SDK default credential chains see only real process environment variables.

Fields resolved through this resolver:

credentials.access_key_id_env
credentials.secret_access_key_env
upload_tokens[].token_env

Maintained Examples

Maintained examples live under examples. Config tests load every file listed here.

Local examples:

local-to-local.yml: minimal local-to-local config using absolute sample paths; load-tested, but paths should be adapted before running.
local-publish.yml: runnable local source publication used by the README quickstart.
local-html.yml: local sidecar HTML publication.
local-index.yml: local index.html publication.
fan-out.yml: local fan-out publication to source and HTML destinations.
archive-and-latest.yml: local archive plus fixed latest publication.
merge-reconciliation.yml: local HTML publication using merge reconciliation.
shared-root.yml: two local pipelines publishing disjoint outputs into one shared destination root.
http-upload-local.yml: local HTTP upload server config; requires DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN in the process environment or as a secret-file name before running serve.

Environment-gated remote examples:

ssh-destination.yml: local-to-SSH publication; replace host, user, path, key, and known-host values for an SSH/SFTP endpoint you control.
s3-destination.yml: local-to-S3 publication; replace endpoint, bucket, prefix, region, and credential variable names for an S3-compatible service you control.

22 KiB Raw Blame History