439 lines
17 KiB
Markdown
439 lines
17 KiB
Markdown
# Configuration Reference
|
|
|
|
Audience: administrators, operators, and advanced users who write YAML configuration for `distributor`.
|
|
|
|
This document is the canonical user-facing configuration reference. CLI syntax lives in [CLI](cli.md), operating procedures live in [Operations](operations.md), symptom-oriented recovery lives in [Troubleshooting](troubleshooting.md), and external contracts live under [Integrations](integrations/source-bundle.md).
|
|
|
|
## Config File Loading
|
|
|
|
`distributor run --config <path>` and `distributor serve --config <path>` load the YAML file at `<path>`. If `--config` is omitted, both commands use:
|
|
|
|
```text
|
|
/usr/local/etc/distributor/config.yml
|
|
```
|
|
|
|
YAML decoding rejects unknown fields. Defaults are applied after decoding and before validation.
|
|
|
|
Runtime backend support is command-specific:
|
|
|
|
- `run`, `validate --config`, and `inspect --config` execute `local`, `ssh`, and `s3` sources.
|
|
- `run` executes `local`, `ssh`, and `s3` destinations.
|
|
- `serve` uses `http_upload` sources through the HTTP upload API and publishes to configured `local`, `ssh`, and `s3` destinations.
|
|
- `http_upload` is valid only as a source backend.
|
|
|
|
## Minimal Local Config
|
|
|
|
```yaml
|
|
pipelines:
|
|
- id: reports
|
|
source:
|
|
backend: local
|
|
path: /var/spool/distributor/reports
|
|
destinations:
|
|
- id: archive
|
|
backend: local
|
|
path: /srv/reports/archive
|
|
```
|
|
|
|
This config publishes source files only. It uses default validation, destination path mapping, publish, transfer, and HTTP server values.
|
|
|
|
## Production-Oriented Local Config
|
|
|
|
```yaml
|
|
server:
|
|
http:
|
|
bind: 127.0.0.1:8080
|
|
staging_root: /var/spool/distributor
|
|
max_upload_size: 20MB
|
|
queue_size: 16
|
|
max_concurrency: 1
|
|
retention: 24h
|
|
secrets:
|
|
directory: /run/secrets/distributor
|
|
pipelines:
|
|
- id: reports
|
|
source:
|
|
backend: local
|
|
path: /var/spool/distributor/reports
|
|
validation:
|
|
on_digest_mismatch: fail
|
|
destinations:
|
|
- id: archive
|
|
backend: local
|
|
path: /srv/reports/archive
|
|
publish:
|
|
source: true
|
|
html: false
|
|
path_mapping:
|
|
mode: preserve_relative
|
|
transfer:
|
|
on_destination_same: skip
|
|
on_destination_older: replace
|
|
on_destination_newer: skip
|
|
on_conflict: fail
|
|
```
|
|
|
|
## HTTP Upload Source Config
|
|
|
|
HTTP upload sources are configured on pipelines and are served by `distributor serve`. Upload tokens are resolved from the process environment or `secrets.directory`; literal bearer tokens are not configured in YAML.
|
|
|
|
```yaml
|
|
server:
|
|
http:
|
|
bind: 127.0.0.1:8080
|
|
staging_root: /var/spool/distributor
|
|
max_upload_size: 20MB
|
|
queue_size: 16
|
|
max_concurrency: 1
|
|
retention: 24h
|
|
pipelines:
|
|
- id: weather-daily
|
|
source:
|
|
backend: http_upload
|
|
token_env: WEATHER_DAILY_UPLOAD_TOKEN
|
|
staging_path: /var/spool/distributor/weather-daily
|
|
max_upload_size: 20MB
|
|
destinations:
|
|
- id: archive
|
|
backend: local
|
|
path: /srv/reports/archive
|
|
```
|
|
|
|
`token_env` is required for `http_upload` sources. `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`. `max_upload_size` defaults to `server.http.max_upload_size`.
|
|
|
|
`serve` maps each resolved bearer token to exactly one `http_upload` pipeline. Startup fails when a token is missing, empty, or duplicates another upload pipeline token.
|
|
|
|
## Top-Level Fields
|
|
|
|
### `server.http`
|
|
|
|
`server.http` controls the HTTP upload server used by `serve`.
|
|
|
|
- `bind`: optional TCP bind address. Default: `127.0.0.1:8080`.
|
|
- `staging_root`: optional root used to default `http_upload` source staging paths. Default: `/var/spool/distributor`.
|
|
- `max_upload_size`: optional default upload limit for HTTP upload sources. Default: `20MB`.
|
|
- `queue_size`: optional upload admission queue size. Default: `16`.
|
|
- `max_concurrency`: optional upload worker concurrency. Default: `1`.
|
|
- `retention`: optional in-memory completed-run retention duration. Default: `24h`.
|
|
|
|
Numeric server values and durations must be greater than zero after defaults are applied.
|
|
|
|
### `secrets`
|
|
|
|
- `directory`: optional directory of secret files used by the config-owned credential resolver.
|
|
|
|
See [Secrets](#secrets) for resolution rules.
|
|
|
|
### `pipelines`
|
|
|
|
`pipelines` is required and must contain at least one pipeline.
|
|
|
|
Each pipeline has:
|
|
|
|
- `id`: required unique slug-like identifier. It must start with a letter or number and may contain letters, numbers, `.`, `_`, and `-`.
|
|
- `source`: required source backend config.
|
|
- `validation`: optional validation policy.
|
|
- `destinations`: required non-empty destination list.
|
|
|
|
Pipeline ids must be unique across the config.
|
|
|
|
## Backend Reference
|
|
|
|
### Local Backend
|
|
|
|
Local backends can be used as sources and destinations.
|
|
|
|
```yaml
|
|
backend: local
|
|
path: /srv/distributor/archive
|
|
```
|
|
|
|
- `backend`: required value `local`.
|
|
- `path`: required local filesystem root for this backend.
|
|
|
|
### SSH/SFTP Backend
|
|
|
|
SSH backends use native SFTP and can be used as sources and destinations. Adapter protocol behavior is documented in [SSH/SFTP Integration](integrations/ssh-sftp.md).
|
|
|
|
```yaml
|
|
backend: ssh
|
|
host: ssh.example.com
|
|
user: distributor
|
|
port: 22
|
|
path: /srv/distributor/archive
|
|
ssh_key_file: /home/distributor/.ssh/id_ed25519
|
|
known_hosts: /home/distributor/.ssh/known_hosts
|
|
host_key_policy: strict
|
|
```
|
|
|
|
- `backend`: required value `ssh`.
|
|
- `host`: required SSH host.
|
|
- `path`: required remote root path.
|
|
- `user`: optional SSH username. If omitted, the adapter uses the current OS user when available.
|
|
- `port`: optional TCP port. Default: `22`.
|
|
- `ssh_key_file`: optional private key path.
|
|
- `known_hosts`: optional OpenSSH `known_hosts` path.
|
|
- `host_key_policy`: optional host key policy. Default: `accept-new`.
|
|
|
|
Accepted host key policy values:
|
|
|
|
- `strict` or boolean `true`: require a matching known host key.
|
|
- `accept-new`: accept and persist a new host key, but reject changed known keys.
|
|
- `off` or boolean `false`: disable host key checking.
|
|
|
|
Authentication uses SSH agent identities when `SSH_AUTH_SOCK` is available, then `ssh_key_file` when configured. Password authentication is not configured in YAML.
|
|
|
|
### S3-Compatible Backend
|
|
|
|
S3 backends can be used as sources and destinations. Adapter protocol behavior is documented in [S3-Compatible Storage Integration](integrations/s3.md).
|
|
|
|
```yaml
|
|
backend: s3
|
|
endpoint: https://s3.example.com
|
|
bucket: reports
|
|
prefix: distributor/archive
|
|
region: us-east-1
|
|
force_path_style: true
|
|
credentials:
|
|
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
|
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
|
```
|
|
|
|
- `backend`: required value `s3`.
|
|
- `endpoint`: required S3-compatible endpoint URL.
|
|
- `bucket`: required bucket name.
|
|
- `prefix`: optional backend root prefix. Leading and trailing slashes are trimmed; the remaining value must be a clean relative slash-separated path.
|
|
- `region`: optional region. Default: `us-east-1`.
|
|
- `force_path_style`: optional addressing mode toggle. Default: `true`.
|
|
- `credentials.access_key_id_env`: optional environment variable or secret-file name for the access key id.
|
|
- `credentials.secret_access_key_env`: optional environment variable or secret-file name for the secret access key.
|
|
|
|
The S3 credential variable names must either both be configured or both be omitted. When omitted, the AWS SDK default credential chain is used. When configured, both values must resolve to non-empty strings through the process environment or `secrets.directory`.
|
|
|
|
### HTTP Upload Source Backend
|
|
|
|
HTTP upload backends are valid only as pipeline sources and are served by `distributor serve`. The API contract is documented in [HTTP Upload API Contract](integrations/http-upload.md).
|
|
|
|
```yaml
|
|
backend: http_upload
|
|
token_env: WEATHER_DAILY_UPLOAD_TOKEN
|
|
staging_path: /var/spool/distributor/weather-daily
|
|
max_upload_size: 20MB
|
|
```
|
|
|
|
- `backend`: required value `http_upload`.
|
|
- `token_env`: required environment variable or secret-file name containing the bearer token.
|
|
- `staging_path`: optional staging path. Default: `<server.http.staging_root>/<pipeline id>`.
|
|
- `max_upload_size`: optional per-source upload limit. Default: `server.http.max_upload_size`.
|
|
|
|
## Validation Policy
|
|
|
|
```yaml
|
|
validation:
|
|
on_digest_mismatch: fail
|
|
```
|
|
|
|
- `validation.on_digest_mismatch`: optional. Default and only accepted value: `fail`.
|
|
|
|
Source bundle digest mismatches fail validation before destination writes occur. The manifest file-format contract is documented in [Source Bundle Contract](integrations/source-bundle.md).
|
|
|
|
## Destination Fields
|
|
|
|
Each destination embeds a backend config at the destination level and may also configure publishing, transforms, path mapping, links, and transfer behavior.
|
|
|
|
```yaml
|
|
destinations:
|
|
- id: archive
|
|
backend: local
|
|
path: /srv/reports/archive
|
|
publish:
|
|
source: true
|
|
html: false
|
|
path_mapping:
|
|
mode: preserve_relative
|
|
transfer:
|
|
on_destination_same: skip
|
|
on_destination_older: replace
|
|
on_destination_newer: skip
|
|
on_conflict: fail
|
|
```
|
|
|
|
- `id`: required unique slug-like identifier within the pipeline.
|
|
- Backend fields: required according to the selected destination backend.
|
|
- `publish`: optional publish policy. Default: source-only publication.
|
|
- `transform`: required only when publishing generated HTML.
|
|
- `path_mapping`: optional destination path mapping policy.
|
|
- `links`: optional public URL metadata policy.
|
|
- `transfer`: optional destination reconciliation policy.
|
|
|
|
Destination ids must be unique within a pipeline.
|
|
|
|
## Publish And Transform Policy
|
|
|
|
### Source-Only Publication
|
|
|
|
```yaml
|
|
publish:
|
|
source: true
|
|
html: false
|
|
```
|
|
|
|
`publish.source` controls whether source manifest files are copied to the destination.
|
|
|
|
### Markdown-To-HTML Publication
|
|
|
|
```yaml
|
|
publish:
|
|
source: false
|
|
html: true
|
|
transform:
|
|
markdown_to_html:
|
|
enabled: true
|
|
mode: sidecar
|
|
```
|
|
|
|
`publish.html` controls whether generated HTML outputs are published. When `publish.html` is `true`, `transform.markdown_to_html.enabled` must also be `true`.
|
|
|
|
Markdown transform fields:
|
|
|
|
- `transform.markdown_to_html.enabled`: enables Markdown-to-HTML generation for this destination.
|
|
- `transform.markdown_to_html.mode`: optional. Accepted values are `sidecar` and `index`; default is `sidecar` when a Markdown transform block is present.
|
|
- `transform.markdown_to_html.input`: optional source manifest path for `index` mode only.
|
|
|
|
`sidecar` mode renders every manifest-listed `.md` file to a same-directory `.html` output. `index` mode renders one Markdown source to `index.html` at the destination bundle path. If `index` mode omits `input`, the selected source bundle must contain exactly one Markdown file.
|
|
|
|
At least one output type must be enabled. Enabled Markdown transforms are rejected when `publish.html` is `false`, and `input` is rejected unless `mode` is `index`.
|
|
|
|
## Destination Path Mapping
|
|
|
|
```yaml
|
|
path_mapping:
|
|
mode: preserve_relative
|
|
```
|
|
|
|
- `path_mapping.mode`: optional. Accepted values are `preserve_relative` and `fixed`; default is `preserve_relative`.
|
|
|
|
`preserve_relative` publishes each discovered source bundle at the same path relative to the destination backend root.
|
|
|
|
`fixed` publishes one selected source bundle directly at the destination backend root. Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins.
|
|
|
|
Fixed mapping is useful for stable latest-style paths. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
|
|
|
|
## Destination Links
|
|
|
|
```yaml
|
|
links:
|
|
base_url: https://reports.example.com/archive
|
|
primary: auto
|
|
```
|
|
|
|
- `links.base_url`: required when `links` is present. It must be an absolute `http` or `https` URL with a host and no query string or fragment.
|
|
- `links.primary`: optional. Accepted values are `auto`, `html`, and `source`; default is `auto` when `links` is present.
|
|
|
|
`distributor` does not infer public URLs from backend config. Destination state URL fields are documented in [Destination State Contract](integrations/destination-state.md). Output URLs are built from `links.base_url`, the destination bundle path, and output paths using URL path semantics. `index.html` outputs produce directory-style URLs that omit the filename.
|
|
|
|
Primary URL policies:
|
|
|
|
- `auto`: prefer `index.html`, then generated HTML, then source outputs.
|
|
- `html`: use the first generated HTML output.
|
|
- `source`: use the first copied source output.
|
|
|
|
If no output matches the primary policy, per-output URLs may still be recorded and the top-level primary URL is omitted.
|
|
|
|
## Transfer Policy
|
|
|
|
```yaml
|
|
transfer:
|
|
on_destination_same: skip
|
|
on_destination_older: replace
|
|
on_destination_newer: skip
|
|
on_conflict: fail
|
|
```
|
|
|
|
Transfer fields and accepted values:
|
|
|
|
- `transfer.on_destination_same`: `skip` or `fail`. Default: `skip`.
|
|
- `transfer.on_destination_older`: `replace` or `fail`. Default: `replace`.
|
|
- `transfer.on_destination_newer`: `skip`, `replace`, or `fail`. Default: `skip`.
|
|
- `transfer.on_conflict`: `fail` or `replace`. Default: `fail`.
|
|
|
|
`replace` for `on_destination_newer` and `on_conflict` is honored only when `run --force` is supplied. There is no config field that enables forced replacement by default.
|
|
|
|
## Size And Duration Values
|
|
|
|
Upload size fields must be YAML strings with an integer and one of these suffixes:
|
|
|
|
- `B`
|
|
- `KB`
|
|
- `MB`
|
|
- `GB`
|
|
|
|
Suffix multipliers use powers of 1024. Values must be greater than zero after defaults are applied.
|
|
|
|
Duration fields must be YAML strings accepted by Go duration parsing, such as `24h`, `90m`, or `168h`. Values must be greater than zero after defaults are applied.
|
|
|
|
## Defaults
|
|
|
|
Defaults are applied after YAML decoding and before validation:
|
|
|
|
- `server.http.bind: 127.0.0.1:8080`
|
|
- `server.http.staging_root: /var/spool/distributor`
|
|
- `server.http.max_upload_size: 20MB`
|
|
- `server.http.queue_size: 16`
|
|
- `server.http.max_concurrency: 1`
|
|
- `server.http.retention: 24h`
|
|
- `validation.on_digest_mismatch: fail`
|
|
- SSH `port: 22`
|
|
- SSH `host_key_policy: accept-new`
|
|
- S3 `region: us-east-1`
|
|
- S3 `prefix`: leading and trailing slashes trimmed
|
|
- S3 `force_path_style: true`
|
|
- `http_upload` source `staging_path: <server.http.staging_root>/<pipeline id>`
|
|
- `http_upload` source `max_upload_size: server.http.max_upload_size`
|
|
- `publish.source: true` and `publish.html: false`
|
|
- `transform.markdown_to_html.mode: sidecar` when a Markdown transform block is present and mode is omitted
|
|
- `path_mapping.mode: preserve_relative`
|
|
- `links.primary: auto` when a `links` block is present and `primary` is omitted
|
|
- `transfer.on_destination_same: skip`
|
|
- `transfer.on_destination_older: replace`
|
|
- `transfer.on_destination_newer: skip`
|
|
- `transfer.on_conflict: fail`
|
|
|
|
## Secrets
|
|
|
|
Do not put literal secrets in config files. `secrets.directory` lets deployments provide credential values as files:
|
|
|
|
```yaml
|
|
secrets:
|
|
directory: /run/secrets/distributor
|
|
```
|
|
|
|
Each regular file in the directory becomes an internal credential environment value named by the filename. Valid filenames match `[A-Za-z_][A-Za-z0-9_]*`. Directories are ignored. Symlinks to regular files are followed. Exactly one trailing LF or CRLF is trimmed from each file; other whitespace is preserved.
|
|
|
|
Credential resolution checks the real process environment first, then `secrets.directory`. If both define the same name with different values, `run` emits a warning with the variable name and uses the real environment value. Secret values are not printed. The process environment is not modified, so SDK default credential chains see only real process environment variables.
|
|
|
|
Fields resolved through this resolver:
|
|
|
|
- `credentials.access_key_id_env`
|
|
- `credentials.secret_access_key_env`
|
|
- `source.token_env` for `http_upload` sources
|
|
|
|
## Maintained Examples
|
|
|
|
Maintained examples live under [examples](../examples/). Config tests load every file listed here.
|
|
|
|
Local examples:
|
|
|
|
- `local-to-local.yml`: minimal local-to-local config using absolute sample paths; load-tested, but paths should be adapted before running.
|
|
- `local-publish.yml`: runnable local source publication used by the README quickstart.
|
|
- `local-html.yml`: local sidecar HTML publication.
|
|
- `local-index.yml`: local `index.html` publication.
|
|
- `fan-out.yml`: local fan-out publication to source and HTML destinations.
|
|
- `archive-and-latest.yml`: local archive plus fixed latest publication.
|
|
- `http-upload-local.yml`: local HTTP upload server config; requires `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN` in the process environment or as a secret-file name before running `serve`.
|
|
|
|
Environment-gated remote examples:
|
|
|
|
- `ssh-destination.yml`: local-to-SSH publication; replace host, user, path, key, and known-host values for an SSH/SFTP endpoint you control.
|
|
- `s3-destination.yml`: local-to-S3 publication; replace endpoint, bucket, prefix, region, and credential variable names for an S3-compatible service you control.
|