distributor/docs/config.md

Configuration Reference

Config File Location

distributor run --config <path> and distributor serve --config <path> load the YAML config at the provided path.

If --config is omitted, both commands use:

/usr/local/etc/distributor/config.yml

Config parsing rejects unknown YAML fields. The executable run backends are local, ssh, and s3. The serve command executes http_upload sources through the HTTP upload API and normal destination fan-out.

Minimal Local Config

pipelines:
  - id: reports
    source:
      backend: local
      path: /var/spool/distributor/reports
    destinations:
      - id: archive
        backend: local
        path: /srv/reports/archive

This publishes source files only. It uses the default validation and transfer policies.

Production-Oriented Local Config

server:
  http:
    bind: 127.0.0.1:8080
    staging_root: /var/spool/distributor
    max_upload_size: 20MB
    queue_size: 16
    max_concurrency: 1
    retention: 24h
pipelines:
  - id: reports
    source:
      backend: local
      path: /var/spool/distributor/reports
    validation:
      on_digest_mismatch: fail
    destinations:
      - id: archive
        backend: local
        path: /srv/reports/archive
        publish:
          source: true
          html: false
        transfer:
          on_destination_same: skip
          on_destination_older: replace
          on_destination_newer: skip
          on_conflict: fail

HTTP Upload Source Configuration

HTTP upload sources are configured as pipeline sources only. They are not valid destination backends. distributor serve maps each configured upload token to exactly one http_upload pipeline.

server:
  http:
    bind: 127.0.0.1:8080
    staging_root: /var/spool/distributor
    max_upload_size: 20MB
    queue_size: 16
    max_concurrency: 1
    retention: 24h
pipelines:
  - id: weather-daily
    source:
      backend: http_upload
      token_env: WEATHER_DAILY_UPLOAD_TOKEN
      staging_path: /var/spool/distributor/weather-daily
      max_upload_size: 20MB
    destinations:
      - id: archive
        backend: local
        path: /srv/reports/archive

source.token_env is required and names the environment variable or secrets.directory file that provides the bearer token. Literal upload tokens are not supported in YAML.

source.staging_path is optional. When omitted, it defaults to <server.http.staging_root>/<pipeline id>.

source.max_upload_size is optional. When omitted, it defaults to server.http.max_upload_size.

The server resolves each token_env through the real process environment and the configured secrets.directory resolver. Startup fails if any configured upload token is missing, empty, or resolves to the same value as another upload pipeline. Token values are not read from YAML and are not printed in API responses.

HTTP Upload API

distributor serve binds to server.http.bind, which defaults to 127.0.0.1:8080.

Routes:

• GET /healthz: returns readiness status after config and upload tokens load.
• POST /upload: accepts one tar or tar.gz source bundle archive.
• GET /runs/<run_id>: returns an in-memory upload status record, or 404 if the run id is unknown or expired.

POST /upload authenticates with:

Authorization: Bearer <token>

The token selects the configured http_upload pipeline. Producers do not send a pipeline id. Requests with a submitted pipeline or pipeline_id query value are rejected.

Accepted upload content types:

• application/x-tar
• application/gzip
• application/x-gzip

Accepted uploads return:

{"run_id":"<id>","status":"accepted"}

The run id can be queried through GET /runs/<run_id> while the status record is retained in memory. Completed records expire after server.http.retention; expiration also removes committed staged bundle directories for completed uploads.

HTML Publication

To publish generated sidecar HTML from Markdown files:

publish:
  source: false
  html: true
transform:
  markdown_to_html:
    enabled: true
    mode: sidecar

Sidecar generation writes report.html for report.md. It does not mutate the source bundle.

To publish a single Markdown file as index.html:

publish:
  source: false
  html: true
transform:
  markdown_to_html:
    enabled: true
    mode: index
    input: report.md

When mode: index omits input, the source manifest must list exactly one Markdown file.

Destination Path Mapping

Each destination chooses how source bundle paths map into that destination:

path_mapping:
  mode: preserve_relative

preserve_relative is the default. It publishes each discovered source bundle at the same path relative to the destination backend root. A source bundle at daily/2026-06-01 publishes below daily/2026-06-01 for that destination.

fixed publishes one selected source bundle directly at the destination backend root:

destinations:
  - id: latest-html
    backend: local
    path: /srv/www/reports/latest
    path_mapping:
      mode: fixed
    publish:
      source: false
      html: true
    transform:
      markdown_to_html:
        enabled: true
        mode: index
        input: report.md

Fixed destinations select the newest discovered source bundle by manifest created timestamp. If multiple candidates have the same timestamp, the source-root-relative bundle path in ascending order wins. Older candidates are not planned or written for that destination.

Fixed mapping is useful for stable latest-style paths. It is more destructive than archive-style publication because successive source bundles target the same destination root. Preview fixed destinations with run --dry-run, especially before using --force.

Destination Links

Destinations can record public URLs for published outputs:

links:
  base_url: https://reports.example.com/archive
  primary: auto

links.base_url is an absolute http or https URL corresponding to the destination backend root. It may include a path prefix, but it must not include a query string or fragment. Distributor does not infer public URLs from backend config.

links.primary selects the top-level primary URL stored in destination state:

• auto: prefer index.html, then generated HTML, then source outputs.
• html: use the first generated HTML output.
• source: use the first copied source output.

If a destination has no links block, no URL metadata is generated. If a primary policy has no matching output, per-output URLs are still recorded and the top-level primary URL is omitted.

Output URLs are built from links.base_url, the destination bundle path, and the output path using URL path semantics. index.html outputs produce directory-style URLs that omit the filename.

Reference

Top level:

• server.http.bind: optional HTTP bind address; defaults to 127.0.0.1:8080.
• server.http.staging_root: optional root for default HTTP upload staging paths; defaults to /var/spool/distributor.
• server.http.max_upload_size: optional default upload size limit; defaults to 20MB.
• server.http.queue_size: optional HTTP upload admission queue size; defaults to 16.
• server.http.max_concurrency: optional HTTP upload worker concurrency; defaults to 1.
• server.http.retention: optional completed upload retention duration; defaults to 24h.
• secrets.directory: optional credential secrets directory.
• pipelines: required non-empty list.

Pipeline:

• id: required unique slug-like identifier.
• source: required backend config.
• validation.on_digest_mismatch: optional; defaults to fail; only fail is supported.
• destinations: required non-empty destination list.

Source backend:

• backend: required.
• path: required for local and ssh.
• host: required for ssh.
• user: optional for ssh; defaults to the current OS user when available.
• port: optional for ssh; defaults to 22.
• ssh_key_file: optional for ssh.
• known_hosts: optional for ssh; defaults to the service user's OpenSSH known_hosts path when available.
• host_key_policy: optional for ssh; defaults to accept-new.
• endpoint: required for s3.
• bucket: required for s3.
• prefix: optional for s3; leading and trailing slashes are trimmed.
• region: optional for s3; defaults to us-east-1.
• force_path_style: optional for s3; defaults to true. Set false only for services that require virtual-host addressing.
• credentials.access_key_id_env: optional S3 credential environment variable name.
• credentials.secret_access_key_env: optional S3 credential environment variable name.
• token_env: required for http_upload; names the token environment variable or secret-file name.
• staging_path: optional for http_upload; defaults below server.http.staging_root using the pipeline id.
• max_upload_size: optional for http_upload; defaults to server.http.max_upload_size.

Destination:

• id: required unique slug-like identifier within the pipeline.
• Backend fields: same accepted shape as source backends, with destination fields at the destination level.
• publish: optional; defaults to source-only publication.
• transform: required only for generated HTML publication.
• path_mapping.mode: optional; defaults to preserve_relative. Accepted values are preserve_relative and fixed.
• links.base_url: optional links block; when present, base_url is required and must be an absolute HTTP or HTTPS URL without query string or fragment.
• links.primary: optional; defaults to auto. Accepted values are auto, html, and source.
• transfer: optional; defaults described below.

Accepted backend names:

• local: executable; requires path.
• ssh: executable; requires host and path.
• s3: executable; requires endpoint and bucket.
• http_upload: source-only configuration; requires token_env.

Size And Duration Values

Upload size fields use an integer plus one of the supported binary-size suffixes:

• B
• KB
• MB
• GB

Suffix multipliers use powers of 1024. Size values must be greater than zero after defaults are applied.

HTTP retention uses Go-style duration strings such as 24h, 90m, or 168h. Retention must be greater than zero after defaults are applied.

SSH Backend

SSH uses native SFTP. It can be used for sources, destinations, or both:

backend: ssh
host: example.com
user: distributor
port: 2222
path: /remote/root
ssh_key_file: /home/distributor/.ssh/id_ed25519
known_hosts: /home/distributor/.ssh/known_hosts
host_key_policy: accept-new

Authentication uses SSH agent identities first when SSH_AUTH_SOCK is set, then ssh_key_file if configured. Password authentication in YAML is not supported.

Host key policies:

• strict, true, and "true" require a matching known host key.
• accept-new accepts and persists a new host key, but fails if an existing key changed. During run --dry-run, new host keys are accepted only for the current connection and are not persisted.
• off, false, and "false" disable host key checking and are insecure.

accept-new and strict use known_hosts when configured. If omitted, distributor uses the current service user's default OpenSSH known_hosts path where practical. accept-new fails when it needs to persist a new host key and no writable known_hosts path is available. It does not create a missing parent .ssh directory.

S3 Backend

S3 uses the AWS SDK for Go v2 and supports S3-compatible endpoints:

backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: archive
region: us-east-1
force_path_style: true
credentials:
  access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
  secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY

endpoint and bucket are required. prefix is an optional backend root; it is treated as an object-key prefix, not a real directory. Prefixes must be clean slash-separated paths after trimming leading and trailing slashes. http:// endpoints are allowed for explicitly configured local development or local S3-compatible test services.

If either credential environment variable name is configured, both must be configured and both referenced variables must resolve to non-empty values through the real process environment or secrets.directory. Explicit credentials take precedence over the AWS SDK default credential chain. If credential environment variable names are omitted, the SDK default credential chain is used and secrets.directory values are not injected into the process environment.

Publish policy:

• publish.source: publish source artifacts.
• publish.html: publish generated HTML artifacts from Markdown source files.

At least one output type must be enabled. When publish.html is true, transform.markdown_to_html.enabled must be true.

Markdown-to-HTML transform:

• transform.markdown_to_html.enabled: enables Markdown-to-HTML generation for destinations with publish.html: true.
• transform.markdown_to_html.mode: optional; defaults to sidecar. Accepted values are sidecar and index.
• transform.markdown_to_html.input: optional source manifest path for index mode. It must identify a listed Markdown file.

sidecar mode renders each manifest-listed .md file to a same-directory .html output. index mode renders one selected Markdown file to index.html at the destination bundle path. Enabled Markdown-to-HTML config is rejected when publish.html is false, and input is valid only with mode: index.

Transfer policy:

• transfer.on_destination_same: skip or fail; defaults to skip.
• transfer.on_destination_older: replace or fail; defaults to replace.
• transfer.on_destination_newer: skip, replace, or fail; defaults to skip.
• transfer.on_conflict: fail or replace; defaults to fail.

replace for on_destination_newer and on_conflict is honored only when run --force is used for that invocation. Force is CLI-only; there is no persistent config field that enables forced replacement by default.

Defaults

Defaults are applied after YAML decoding and before validation:

• validation.on_digest_mismatch: fail
• SSH port: 22
• SSH host_key_policy: accept-new
• S3 region: us-east-1
• S3 force_path_style: true
• server.http.bind: 127.0.0.1:8080
• server.http.staging_root: /var/spool/distributor
• server.http.max_upload_size: 20MB
• server.http.queue_size: 16
• server.http.max_concurrency: 1
• server.http.retention: 24h
• source.staging_path: /var/spool/distributor/<pipeline id> for http_upload
• source.max_upload_size: server.http.max_upload_size for http_upload
• transform.markdown_to_html.mode: sidecar when a Markdown-to-HTML transform block is present and mode is omitted
• publish.source: true
• publish.html: false
• path_mapping.mode: preserve_relative
• links.primary: auto when a links block is present and primary is omitted
• transfer.on_destination_same: skip
• transfer.on_destination_older: replace
• transfer.on_destination_newer: skip
• transfer.on_conflict: fail

Secrets

Do not put literal secrets in config files. secrets.directory lets deployments provide credential values as files:

secrets:
  directory: /run/secrets/distributor

Each regular file in the directory becomes an internal credential environment value named by the filename. Valid filenames must match [A-Za-z_][A-Za-z0-9_]*. Directories are ignored, and symlinks to regular files are followed. Exactly one trailing LF or CRLF is trimmed from each file; other whitespace is preserved.

The resolver checks the real process environment first, then the secrets directory. If both define the same variable with different values, run prints a warning with the variable name and uses the real environment value. Secret values are not printed. The process environment is not modified, so SDK default credential chains see only real environment variables.

S3 credentials may name environment variables:

• credentials.access_key_id_env
• credentials.secret_access_key_env

HTTP upload tokens name one environment variable or secret-file name:

• source.token_env

Examples

Maintained examples live under examples:

• local-to-local.yml: minimal local config.
• local-publish.yml: runnable local source publication.
• local-html.yml: runnable local HTML publication.
• local-index.yml: runnable local index.html publication.
• fan-out.yml: runnable local fan-out publication to source and HTML destinations.
• archive-and-latest.yml: runnable local fan-out publication to an archive destination and a fixed latest destination.
• http-upload-local.yml: local HTTP upload server example with a token environment variable reference.
• ssh-destination.yml: environment-gated local-to-SSH publication example.
• s3-destination.yml: environment-gated local-to-S3 publication example.