distributor/docs/config.md

Distributor Configuration

Config File Location

distributor run --config <path> loads the YAML config at the provided path.

If --config is omitted, run uses:

/usr/local/etc/distributor/config.yml

Config parsing rejects unknown YAML fields. The executable backends are local and ssh. S3 config fields are accepted by config validation, but runtime execution for S3 is unavailable.

Minimal Local Config

pipelines:
  - id: reports
    source:
      backend: local
      path: /var/spool/distributor/reports
    destinations:
      - id: archive
        backend: local
        path: /srv/reports/archive

This publishes source files only. It uses the default validation and transfer policies.

Production-Oriented Local Config

pipelines:
  - id: reports
    source:
      backend: local
      path: /var/spool/distributor/reports
    validation:
      on_digest_mismatch: fail
    destinations:
      - id: archive
        backend: local
        path: /srv/reports/archive
        publish:
          source: true
          html: false
        transfer:
          on_destination_same: skip
          on_destination_older: replace
          on_destination_newer: skip
          on_conflict: fail

HTML Publication

To publish generated HTML from Markdown files:

publish:
  source: false
  html: true
transform:
  markdown_to_html:
    enabled: true
    mode: sidecar

Sidecar generation writes report.html for report.md. It does not mutate the source bundle.

Reference

Top level:

• secrets.directory: optional credential secrets directory.
• pipelines: required non-empty list.

Pipeline:

• id: required unique slug-like identifier.
• source: required backend config.
• validation.on_digest_mismatch: optional; defaults to fail; only fail is supported.
• destinations: required non-empty destination list.

Source backend:

• backend: required.
• path: required for local and ssh.
• host: required for ssh.
• user: optional for ssh; defaults to the current OS user when available.
• port: optional for ssh; defaults to 22.
• ssh_key_file: optional for ssh.
• known_hosts: optional for ssh; defaults to the service user's OpenSSH known_hosts path when available.
• host_key_policy: optional for ssh; defaults to accept-new.
• endpoint: required for s3.
• bucket: required for s3.
• prefix: optional for s3.
• region: optional for s3.
• force_path_style: optional for s3.
• credentials.access_key_id_env: optional S3 credential environment variable name.
• credentials.secret_access_key_env: optional S3 credential environment variable name.

Destination:

• id: required unique slug-like identifier within the pipeline.
• Backend fields: same accepted shape as source backends, with destination fields at the destination level.
• publish: optional; defaults to source-only publication.
• transform: required only for generated HTML publication.
• transfer: optional; defaults described below.

Accepted backend names:

• local: executable; requires path.
• ssh: executable; requires host and path.
• s3: config validation only; execution is unavailable.

SSH Backend

SSH uses native SFTP. It can be used for sources, destinations, or both:

backend: ssh
host: example.com
user: distributor
port: 2222
path: /remote/root
ssh_key_file: /home/distributor/.ssh/id_ed25519
known_hosts: /home/distributor/.ssh/known_hosts
host_key_policy: accept-new

Authentication uses SSH agent identities first when SSH_AUTH_SOCK is set, then ssh_key_file if configured. Password authentication in YAML is not supported.

Host key policies:

• strict, true, and "true" require a matching known host key.
• accept-new accepts and persists a new host key, but fails if an existing key changed.
• off, false, and "false" disable host key checking and are insecure.

accept-new and strict use known_hosts when configured. If omitted, distributor uses the current service user's default OpenSSH known_hosts path where practical. accept-new fails when it needs to persist a new host key and no writable known_hosts path is available. It does not create a missing parent .ssh directory.

Publish policy:

• publish.source: publish source artifacts.
• publish.html: publish generated HTML artifacts from Markdown source files.

At least one output type must be enabled. When publish.html is true, transform.markdown_to_html.enabled must be true and transform.markdown_to_html.mode must be sidecar.

Transfer policy:

• transfer.on_destination_same: skip or fail; defaults to skip.
• transfer.on_destination_older: replace or fail; defaults to replace.
• transfer.on_destination_newer: skip or fail; defaults to skip.
• transfer.on_conflict: only fail; defaults to fail.

Defaults

Defaults are applied after YAML decoding and before validation:

• validation.on_digest_mismatch: fail
• SSH port: 22
• SSH host_key_policy: accept-new
• publish.source: true
• publish.html: false
• transfer.on_destination_same: skip
• transfer.on_destination_older: replace
• transfer.on_destination_newer: skip
• transfer.on_conflict: fail

Secrets

Do not put literal secrets in config files. secrets.directory lets deployments provide credential values as files:

secrets:
  directory: /run/secrets/distributor

Each regular file in the directory becomes an internal credential environment value named by the filename. Valid filenames must match [A-Za-z_][A-Za-z0-9_]*. Directories are ignored, and symlinks to regular files are followed. Exactly one trailing LF or CRLF is trimmed from each file; other whitespace is preserved.

The resolver checks the real process environment first, then the secrets directory. If both define the same variable with different values, run prints a warning with the variable name and uses the real environment value. Secret values are not printed. The process environment is not modified, so SDK default credential chains see only real environment variables.

S3 credentials may name environment variables:

• credentials.access_key_id_env
• credentials.secret_access_key_env

S3 execution is unavailable; these fields are accepted so config shape can be validated.

Examples

Maintained examples live under examples:

• local-to-local.yml: minimal local config.
• local-publish.yml: runnable local source publication.
• local-html.yml: runnable local HTML publication.
• fan-out.yml: runnable local fan-out publication to source and HTML destinations.
• ssh-destination.yml: environment-gated local-to-SSH publication example.