429 lines
16 KiB
Markdown
429 lines
16 KiB
Markdown
# Configuration Reference
|
|
|
|
## Config File Location
|
|
|
|
`distributor run --config <path>` and `distributor serve --config <path>` load
|
|
the YAML config at the provided path.
|
|
|
|
If `--config` is omitted, both commands use:
|
|
|
|
```text
|
|
/usr/local/etc/distributor/config.yml
|
|
```
|
|
|
|
Config parsing rejects unknown YAML fields. The executable `run` backends are
|
|
`local`, `ssh`, and `s3`. The `serve` command executes `http_upload` sources
|
|
through the HTTP upload API and normal destination fan-out.
|
|
|
|
## Minimal Local Config
|
|
|
|
```yaml
|
|
pipelines:
|
|
- id: reports
|
|
source:
|
|
backend: local
|
|
path: /var/spool/distributor/reports
|
|
destinations:
|
|
- id: archive
|
|
backend: local
|
|
path: /srv/reports/archive
|
|
```
|
|
|
|
This publishes source files only. It uses the default validation and transfer policies.
|
|
|
|
## Production-Oriented Local Config
|
|
|
|
```yaml
|
|
server:
|
|
http:
|
|
bind: 127.0.0.1:8080
|
|
staging_root: /var/spool/distributor
|
|
max_upload_size: 20MB
|
|
queue_size: 16
|
|
max_concurrency: 1
|
|
retention: 24h
|
|
pipelines:
|
|
- id: reports
|
|
source:
|
|
backend: local
|
|
path: /var/spool/distributor/reports
|
|
validation:
|
|
on_digest_mismatch: fail
|
|
destinations:
|
|
- id: archive
|
|
backend: local
|
|
path: /srv/reports/archive
|
|
publish:
|
|
source: true
|
|
html: false
|
|
transfer:
|
|
on_destination_same: skip
|
|
on_destination_older: replace
|
|
on_destination_newer: skip
|
|
on_conflict: fail
|
|
```
|
|
|
|
## HTTP Upload Source Configuration
|
|
|
|
HTTP upload sources are configured as pipeline sources only. They are not valid
|
|
destination backends. `distributor serve` maps each configured upload token to
|
|
exactly one `http_upload` pipeline.
|
|
|
|
```yaml
|
|
server:
|
|
http:
|
|
bind: 127.0.0.1:8080
|
|
staging_root: /var/spool/distributor
|
|
max_upload_size: 20MB
|
|
queue_size: 16
|
|
max_concurrency: 1
|
|
retention: 24h
|
|
pipelines:
|
|
- id: weather-daily
|
|
source:
|
|
backend: http_upload
|
|
token_env: WEATHER_DAILY_UPLOAD_TOKEN
|
|
staging_path: /var/spool/distributor/weather-daily
|
|
max_upload_size: 20MB
|
|
destinations:
|
|
- id: archive
|
|
backend: local
|
|
path: /srv/reports/archive
|
|
```
|
|
|
|
`source.token_env` is required and names the environment variable or `secrets.directory` file that provides the bearer token. Literal upload tokens are not supported in YAML.
|
|
|
|
`source.staging_path` is optional. When omitted, it defaults to `<server.http.staging_root>/<pipeline id>`.
|
|
|
|
`source.max_upload_size` is optional. When omitted, it defaults to `server.http.max_upload_size`.
|
|
|
|
The server resolves each `token_env` through the real process environment and
|
|
the configured `secrets.directory` resolver. Startup fails if any configured
|
|
upload token is missing, empty, or resolves to the same value as another upload
|
|
pipeline. Token values are not read from YAML and are not printed in API
|
|
responses.
|
|
|
|
## HTTP Upload API
|
|
|
|
`distributor serve` binds to `server.http.bind`, which defaults to
|
|
`127.0.0.1:8080`.
|
|
|
|
Routes:
|
|
|
|
- `GET /healthz`: returns readiness status after config and upload tokens load.
|
|
- `POST /upload`: accepts one tar or tar.gz source bundle archive.
|
|
- `GET /runs/<run_id>`: returns an in-memory upload status record, or `404` if the run id is unknown or expired.
|
|
|
|
`POST /upload` authenticates with:
|
|
|
|
```text
|
|
Authorization: Bearer <token>
|
|
```
|
|
|
|
The token selects the configured `http_upload` pipeline. Producers do not send a
|
|
pipeline id. Requests with a submitted `pipeline` or `pipeline_id` query value
|
|
are rejected.
|
|
|
|
Accepted upload content types:
|
|
|
|
- `application/x-tar`
|
|
- `application/gzip`
|
|
- `application/x-gzip`
|
|
|
|
Accepted uploads return:
|
|
|
|
```json
|
|
{"run_id":"<id>","status":"accepted"}
|
|
```
|
|
|
|
The run id can be queried through `GET /runs/<run_id>` while the status record
|
|
is retained in memory. Completed records expire after `server.http.retention`;
|
|
expiration also removes committed staged bundle directories for completed
|
|
uploads.
|
|
|
|
## HTML Publication
|
|
|
|
To publish generated sidecar HTML from Markdown files:
|
|
|
|
```yaml
|
|
publish:
|
|
source: false
|
|
html: true
|
|
transform:
|
|
markdown_to_html:
|
|
enabled: true
|
|
mode: sidecar
|
|
```
|
|
|
|
Sidecar generation writes `report.html` for `report.md`. It does not mutate the source bundle.
|
|
|
|
To publish a single Markdown file as `index.html`:
|
|
|
|
```yaml
|
|
publish:
|
|
source: false
|
|
html: true
|
|
transform:
|
|
markdown_to_html:
|
|
enabled: true
|
|
mode: index
|
|
input: report.md
|
|
```
|
|
|
|
When `mode: index` omits `input`, the source manifest must list exactly one Markdown file.
|
|
|
|
## Destination Path Mapping
|
|
|
|
Each destination chooses how source bundle paths map into that destination:
|
|
|
|
```yaml
|
|
path_mapping:
|
|
mode: preserve_relative
|
|
```
|
|
|
|
`preserve_relative` is the default. It publishes each discovered source bundle at the same path relative to the destination backend root. A source bundle at `daily/2026-06-01` publishes below `daily/2026-06-01` for that destination.
|
|
|
|
`fixed` publishes one selected source bundle directly at the destination backend root:
|
|
|
|
```yaml
|
|
destinations:
|
|
- id: latest-html
|
|
backend: local
|
|
path: /srv/www/reports/latest
|
|
path_mapping:
|
|
mode: fixed
|
|
publish:
|
|
source: false
|
|
html: true
|
|
transform:
|
|
markdown_to_html:
|
|
enabled: true
|
|
mode: index
|
|
input: report.md
|
|
```
|
|
|
|
Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple candidates have the same timestamp, the source-root-relative bundle path in ascending order wins. Older candidates are not planned or written for that destination.
|
|
|
|
Fixed mapping is useful for stable latest-style paths. It is more destructive than archive-style publication because successive source bundles target the same destination root. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
|
|
|
|
## Destination Links
|
|
|
|
Destinations can record public URLs for published outputs:
|
|
|
|
```yaml
|
|
links:
|
|
base_url: https://reports.example.com/archive
|
|
primary: auto
|
|
```
|
|
|
|
`links.base_url` is an absolute `http` or `https` URL corresponding to the destination backend root. It may include a path prefix, but it must not include a query string or fragment. Distributor does not infer public URLs from backend config.
|
|
|
|
`links.primary` selects the top-level primary URL stored in destination state:
|
|
|
|
- `auto`: prefer `index.html`, then generated HTML, then source outputs.
|
|
- `html`: use the first generated HTML output.
|
|
- `source`: use the first copied source output.
|
|
|
|
If a destination has no `links` block, no URL metadata is generated. If a primary policy has no matching output, per-output URLs are still recorded and the top-level primary URL is omitted.
|
|
|
|
Output URLs are built from `links.base_url`, the destination bundle path, and the output path using URL path semantics. `index.html` outputs produce directory-style URLs that omit the filename.
|
|
|
|
## Reference
|
|
|
|
Top level:
|
|
|
|
- `server.http.bind`: optional HTTP bind address; defaults to `127.0.0.1:8080`.
|
|
- `server.http.staging_root`: optional root for default HTTP upload staging paths; defaults to `/var/spool/distributor`.
|
|
- `server.http.max_upload_size`: optional default upload size limit; defaults to `20MB`.
|
|
- `server.http.queue_size`: optional HTTP upload admission queue size; defaults to `16`.
|
|
- `server.http.max_concurrency`: optional HTTP upload worker concurrency; defaults to `1`.
|
|
- `server.http.retention`: optional completed upload retention duration; defaults to `24h`.
|
|
- `secrets.directory`: optional credential secrets directory.
|
|
- `pipelines`: required non-empty list.
|
|
|
|
Pipeline:
|
|
|
|
- `id`: required unique slug-like identifier.
|
|
- `source`: required backend config.
|
|
- `validation.on_digest_mismatch`: optional; defaults to `fail`; only `fail` is supported.
|
|
- `destinations`: required non-empty destination list.
|
|
|
|
Source backend:
|
|
|
|
- `backend`: required.
|
|
- `path`: required for `local` and `ssh`.
|
|
- `host`: required for `ssh`.
|
|
- `user`: optional for `ssh`; defaults to the current OS user when available.
|
|
- `port`: optional for `ssh`; defaults to `22`.
|
|
- `ssh_key_file`: optional for `ssh`.
|
|
- `known_hosts`: optional for `ssh`; defaults to the service user's OpenSSH `known_hosts` path when available.
|
|
- `host_key_policy`: optional for `ssh`; defaults to `accept-new`.
|
|
- `endpoint`: required for `s3`.
|
|
- `bucket`: required for `s3`.
|
|
- `prefix`: optional for `s3`; leading and trailing slashes are trimmed.
|
|
- `region`: optional for `s3`; defaults to `us-east-1`.
|
|
- `force_path_style`: optional for `s3`; defaults to `true`. Set `false` only for services that require virtual-host addressing.
|
|
- `credentials.access_key_id_env`: optional S3 credential environment variable name.
|
|
- `credentials.secret_access_key_env`: optional S3 credential environment variable name.
|
|
- `token_env`: required for `http_upload`; names the token environment variable or secret-file name.
|
|
- `staging_path`: optional for `http_upload`; defaults below `server.http.staging_root` using the pipeline id.
|
|
- `max_upload_size`: optional for `http_upload`; defaults to `server.http.max_upload_size`.
|
|
|
|
Destination:
|
|
|
|
- `id`: required unique slug-like identifier within the pipeline.
|
|
- Backend fields: same accepted shape as source backends, with destination fields at the destination level.
|
|
- `publish`: optional; defaults to source-only publication.
|
|
- `transform`: required only for generated HTML publication.
|
|
- `path_mapping.mode`: optional; defaults to `preserve_relative`. Accepted values are `preserve_relative` and `fixed`.
|
|
- `links.base_url`: optional links block; when present, `base_url` is required and must be an absolute HTTP or HTTPS URL without query string or fragment.
|
|
- `links.primary`: optional; defaults to `auto`. Accepted values are `auto`, `html`, and `source`.
|
|
- `transfer`: optional; defaults described below.
|
|
|
|
Accepted backend names:
|
|
|
|
- `local`: executable; requires `path`.
|
|
- `ssh`: executable; requires `host` and `path`.
|
|
- `s3`: executable; requires `endpoint` and `bucket`.
|
|
- `http_upload`: source-only configuration; requires `token_env`.
|
|
|
|
## Size And Duration Values
|
|
|
|
Upload size fields use an integer plus one of the supported binary-size suffixes:
|
|
|
|
- `B`
|
|
- `KB`
|
|
- `MB`
|
|
- `GB`
|
|
|
|
Suffix multipliers use powers of 1024. Size values must be greater than zero after defaults are applied.
|
|
|
|
HTTP retention uses Go-style duration strings such as `24h`, `90m`, or `168h`. Retention must be greater than zero after defaults are applied.
|
|
|
|
## SSH Backend
|
|
|
|
SSH uses native SFTP. It can be used for sources, destinations, or both:
|
|
|
|
```yaml
|
|
backend: ssh
|
|
host: example.com
|
|
user: distributor
|
|
port: 2222
|
|
path: /remote/root
|
|
ssh_key_file: /home/distributor/.ssh/id_ed25519
|
|
known_hosts: /home/distributor/.ssh/known_hosts
|
|
host_key_policy: accept-new
|
|
```
|
|
|
|
Authentication uses SSH agent identities first when `SSH_AUTH_SOCK` is set, then `ssh_key_file` if configured. Password authentication in YAML is not supported.
|
|
|
|
Host key policies:
|
|
|
|
- `strict`, `true`, and `"true"` require a matching known host key.
|
|
- `accept-new` accepts and persists a new host key, but fails if an existing key changed. During `run --dry-run`, new host keys are accepted only for the current connection and are not persisted.
|
|
- `off`, `false`, and `"false"` disable host key checking and are insecure.
|
|
|
|
`accept-new` and `strict` use `known_hosts` when configured. If omitted, distributor uses the current service user's default OpenSSH `known_hosts` path where practical. `accept-new` fails when it needs to persist a new host key and no writable `known_hosts` path is available. It does not create a missing parent `.ssh` directory.
|
|
|
|
## S3 Backend
|
|
|
|
S3 uses the AWS SDK for Go v2 and supports S3-compatible endpoints:
|
|
|
|
```yaml
|
|
backend: s3
|
|
endpoint: https://s3.example.com
|
|
bucket: reports
|
|
prefix: archive
|
|
region: us-east-1
|
|
force_path_style: true
|
|
credentials:
|
|
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
|
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
|
```
|
|
|
|
`endpoint` and `bucket` are required. `prefix` is an optional backend root; it is treated as an object-key prefix, not a real directory. Prefixes must be clean slash-separated paths after trimming leading and trailing slashes. `http://` endpoints are allowed for explicitly configured local development or local S3-compatible test services.
|
|
|
|
If either credential environment variable name is configured, both must be configured and both referenced variables must resolve to non-empty values through the real process environment or `secrets.directory`. Explicit credentials take precedence over the AWS SDK default credential chain. If credential environment variable names are omitted, the SDK default credential chain is used and `secrets.directory` values are not injected into the process environment.
|
|
|
|
Publish policy:
|
|
|
|
- `publish.source`: publish source artifacts.
|
|
- `publish.html`: publish generated HTML artifacts from Markdown source files.
|
|
|
|
At least one output type must be enabled. When `publish.html` is true, `transform.markdown_to_html.enabled` must be `true`.
|
|
|
|
Markdown-to-HTML transform:
|
|
|
|
- `transform.markdown_to_html.enabled`: enables Markdown-to-HTML generation for destinations with `publish.html: true`.
|
|
- `transform.markdown_to_html.mode`: optional; defaults to `sidecar`. Accepted values are `sidecar` and `index`.
|
|
- `transform.markdown_to_html.input`: optional source manifest path for `index` mode. It must identify a listed Markdown file.
|
|
|
|
`sidecar` mode renders each manifest-listed `.md` file to a same-directory `.html` output. `index` mode renders one selected Markdown file to `index.html` at the destination bundle path. Enabled Markdown-to-HTML config is rejected when `publish.html` is false, and `input` is valid only with `mode: index`.
|
|
|
|
Transfer policy:
|
|
|
|
- `transfer.on_destination_same`: `skip` or `fail`; defaults to `skip`.
|
|
- `transfer.on_destination_older`: `replace` or `fail`; defaults to `replace`.
|
|
- `transfer.on_destination_newer`: `skip`, `replace`, or `fail`; defaults to `skip`.
|
|
- `transfer.on_conflict`: `fail` or `replace`; defaults to `fail`.
|
|
|
|
`replace` for `on_destination_newer` and `on_conflict` is honored only when `run --force` is used for that invocation. Force is CLI-only; there is no persistent config field that enables forced replacement by default.
|
|
|
|
## Defaults
|
|
|
|
Defaults are applied after YAML decoding and before validation:
|
|
|
|
- `validation.on_digest_mismatch: fail`
|
|
- SSH `port: 22`
|
|
- SSH `host_key_policy: accept-new`
|
|
- S3 `region: us-east-1`
|
|
- S3 `force_path_style: true`
|
|
- `server.http.bind: 127.0.0.1:8080`
|
|
- `server.http.staging_root: /var/spool/distributor`
|
|
- `server.http.max_upload_size: 20MB`
|
|
- `server.http.queue_size: 16`
|
|
- `server.http.max_concurrency: 1`
|
|
- `server.http.retention: 24h`
|
|
- `source.staging_path: /var/spool/distributor/<pipeline id>` for `http_upload`
|
|
- `source.max_upload_size: server.http.max_upload_size` for `http_upload`
|
|
- `transform.markdown_to_html.mode: sidecar` when a Markdown-to-HTML transform block is present and mode is omitted
|
|
- `publish.source: true`
|
|
- `publish.html: false`
|
|
- `path_mapping.mode: preserve_relative`
|
|
- `links.primary: auto` when a `links` block is present and `primary` is omitted
|
|
- `transfer.on_destination_same: skip`
|
|
- `transfer.on_destination_older: replace`
|
|
- `transfer.on_destination_newer: skip`
|
|
- `transfer.on_conflict: fail`
|
|
|
|
## Secrets
|
|
|
|
Do not put literal secrets in config files. `secrets.directory` lets deployments provide credential values as files:
|
|
|
|
```yaml
|
|
secrets:
|
|
directory: /run/secrets/distributor
|
|
```
|
|
|
|
Each regular file in the directory becomes an internal credential environment value named by the filename. Valid filenames must match `[A-Za-z_][A-Za-z0-9_]*`. Directories are ignored, and symlinks to regular files are followed. Exactly one trailing LF or CRLF is trimmed from each file; other whitespace is preserved.
|
|
|
|
The resolver checks the real process environment first, then the secrets directory. If both define the same variable with different values, `run` prints a warning with the variable name and uses the real environment value. Secret values are not printed. The process environment is not modified, so SDK default credential chains see only real environment variables.
|
|
|
|
S3 credentials may name environment variables:
|
|
|
|
- `credentials.access_key_id_env`
|
|
- `credentials.secret_access_key_env`
|
|
|
|
## Examples
|
|
|
|
Maintained examples live under [examples](../examples/):
|
|
|
|
- `local-to-local.yml`: minimal local config.
|
|
- `local-publish.yml`: runnable local source publication.
|
|
- `local-html.yml`: runnable local HTML publication.
|
|
- `local-index.yml`: runnable local `index.html` publication.
|
|
- `fan-out.yml`: runnable local fan-out publication to source and HTML destinations.
|
|
- `archive-and-latest.yml`: runnable local fan-out publication to an archive destination and a fixed latest destination.
|
|
- `ssh-destination.yml`: environment-gated local-to-SSH publication example.
|
|
- `s3-destination.yml`: environment-gated local-to-S3 publication example.
|