eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Refresh configuration reference documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-04 12:07:41 +00:00
parent b19128b77e
commit 18bba116f2
1 changed files with 246 additions and 244 deletions
Download Patch File Download Diff File Expand all files Collapse all files

490
docs/config.md
View File

@@ -1,19 +1,25 @@
# Configuration Reference # Configuration Reference
## Config File Location Audience: administrators, operators, and advanced users who write YAML configuration for `distributor`.
`distributor run --config <path>` and `distributor serve --config <path>` load This document is the canonical user-facing configuration reference. CLI syntax lives in [CLI](cli.md), operating procedures live in [Operations](operations.md), and symptom-oriented recovery lives in [Troubleshooting](troubleshooting.md).
the YAML config at the provided path.
If `--config` is omitted, both commands use: ## Config File Loading
`distributor run --config <path>` and `distributor serve --config <path>` load the YAML file at `<path>`. If `--config` is omitted, both commands use:
```text ```text
/usr/local/etc/distributor/config.yml /usr/local/etc/distributor/config.yml
``` ```
Config parsing rejects unknown YAML fields. The executable `run` backends are YAML decoding rejects unknown fields. Defaults are applied after decoding and before validation.
`local`, `ssh`, and `s3`. The `serve` command executes `http_upload` sources
through the HTTP upload API and normal destination fan-out. Runtime backend support is command-specific:
- `run`, `validate --config`, and `inspect --config` execute `local`, `ssh`, and `s3` sources.
- `run` executes `local`, `ssh`, and `s3` destinations.
- `serve` uses `http_upload` sources through the HTTP upload API and publishes to configured `local`, `ssh`, and `s3` destinations.
- `http_upload` is valid only as a source backend.
## Minimal Local Config ## Minimal Local Config
@@ -29,7 +35,7 @@ pipelines:
path: /srv/reports/archive path: /srv/reports/archive
``` ```
This publishes source files only. It uses the default validation and transfer policies. This config publishes source files only. It uses default validation, destination path mapping, publish, transfer, and HTTP server values.
## Production-Oriented Local Config ## Production-Oriented Local Config
@@ -42,6 +48,8 @@ server:
queue_size: 16 queue_size: 16
max_concurrency: 1 max_concurrency: 1
retention: 24h retention: 24h
secrets:
directory: /run/secrets/distributor
pipelines: pipelines:
- id: reports - id: reports
source: source:
@@ -56,6 +64,8 @@ pipelines:
publish: publish:
source: true source: true
html: false html: false
path_mapping:
mode: preserve_relative
transfer: transfer:
on_destination_same: skip on_destination_same: skip
on_destination_older: replace on_destination_older: replace
@@ -63,11 +73,9 @@ pipelines:
on_conflict: fail on_conflict: fail
``` ```
## HTTP Upload Source Configuration ## HTTP Upload Source Config
HTTP upload sources are configured as pipeline sources only. They are not valid HTTP upload sources are configured on pipelines and are served by `distributor serve`. Upload tokens are resolved from the process environment or `secrets.directory`; literal bearer tokens are not configured in YAML.
destination backends. `distributor serve` maps each configured upload token to
exactly one `http_upload` pipeline.
```yaml ```yaml
server: server:
@@ -91,62 +99,188 @@ pipelines:
path: /srv/reports/archive path: /srv/reports/archive
``` ```
`source.token_env` is required and names the environment variable or `secrets.directory` file that provides the bearer token. Literal upload tokens are not supported in YAML. `token_env` is required for `http_upload` sources. `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`. `max_upload_size` defaults to `server.http.max_upload_size`.
`source.staging_path` is optional. When omitted, it defaults to `<server.http.staging_root>/<pipeline id>`. `serve` maps each resolved bearer token to exactly one `http_upload` pipeline. Startup fails when a token is missing, empty, or duplicates another upload pipeline token.
`source.max_upload_size` is optional. When omitted, it defaults to `server.http.max_upload_size`. ## Top-Level Fields
The server resolves each `token_env` through the real process environment and ### `server.http`
the configured `secrets.directory` resolver. Startup fails if any configured
upload token is missing, empty, or resolves to the same value as another upload
pipeline. Token values are not read from YAML and are not printed in API
responses.
## HTTP Upload API `server.http` controls the HTTP upload server used by `serve`.
`distributor serve` binds to `server.http.bind`, which defaults to - `bind`: optional TCP bind address. Default: `127.0.0.1:8080`.
`127.0.0.1:8080`. - `staging_root`: optional root used to default `http_upload` source staging paths. Default: `/var/spool/distributor`.
- `max_upload_size`: optional default upload limit for HTTP upload sources. Default: `20MB`.
- `queue_size`: optional upload admission queue size. Default: `16`.
- `max_concurrency`: optional upload worker concurrency. Default: `1`.
- `retention`: optional in-memory completed-run retention duration. Default: `24h`.
Routes: Numeric server values and durations must be greater than zero after defaults are applied.
- `GET /healthz`: returns readiness status after config and upload tokens load. ### `secrets`
- `POST /upload`: accepts one tar or tar.gz source bundle archive.
- `GET /runs/<run_id>`: returns an in-memory upload status record, or `404` if the run id is unknown or expired.
`POST /upload` authenticates with: - `directory`: optional directory of secret files used by the config-owned credential resolver.
```text See [Secrets](#secrets) for resolution rules.
Authorization: Bearer <token>
### `pipelines`
`pipelines` is required and must contain at least one pipeline.
Each pipeline has:
- `id`: required unique slug-like identifier. It must start with a letter or number and may contain letters, numbers, `.`, `_`, and `-`.
- `source`: required source backend config.
- `validation`: optional validation policy.
- `destinations`: required non-empty destination list.
Pipeline ids must be unique across the config.
## Backend Reference
### Local Backend
Local backends can be used as sources and destinations.
```yaml
backend: local
path: /srv/distributor/archive
``` ```
The token selects the configured `http_upload` pipeline. Producers do not send a - `backend`: required value `local`.
pipeline id. Requests with a submitted `pipeline` or `pipeline_id` query value - `path`: required local filesystem root for this backend.
are rejected.
Accepted upload content types: ### SSH/SFTP Backend
- `application/x-tar` SSH backends use native SFTP and can be used as sources and destinations.
- `application/gzip`
- `application/x-gzip`
Accepted uploads return after the archive is staged and validated: ```yaml
backend: ssh
```json host: ssh.example.com
{"run_id":"<id>","status":"accepted"} user: distributor
port: 22
path: /srv/distributor/archive
ssh_key_file: /home/distributor/.ssh/id_ed25519
known_hosts: /home/distributor/.ssh/known_hosts
host_key_policy: strict
``` ```
Malformed tar or gzip content and invalid staged bundles are rejected before a - `backend`: required value `ssh`.
run id is issued. - `host`: required SSH host.
- `path`: required remote root path.
- `user`: optional SSH username. If omitted, the adapter uses the current OS user when available.
- `port`: optional TCP port. Default: `22`.
- `ssh_key_file`: optional private key path.
- `known_hosts`: optional OpenSSH `known_hosts` path.
- `host_key_policy`: optional host key policy. Default: `accept-new`.
The run id can be queried through `GET /runs/<run_id>` while the status record Accepted host key policy values:
is retained in memory. Completed records expire after `server.http.retention`;
expiration also removes committed staged bundle directories for completed
uploads.
## HTML Publication - `strict` or boolean `true`: require a matching known host key.
- `accept-new`: accept and persist a new host key, but reject changed known keys.
- `off` or boolean `false`: disable host key checking.
To publish generated sidecar HTML from Markdown files: Authentication uses SSH agent identities when `SSH_AUTH_SOCK` is available, then `ssh_key_file` when configured. Password authentication is not configured in YAML.
### S3-Compatible Backend
S3 backends can be used as sources and destinations.
```yaml
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: distributor/archive
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
```
- `backend`: required value `s3`.
- `endpoint`: required S3-compatible endpoint URL.
- `bucket`: required bucket name.
- `prefix`: optional backend root prefix. Leading and trailing slashes are trimmed; the remaining value must be a clean relative slash-separated path.
- `region`: optional region. Default: `us-east-1`.
- `force_path_style`: optional addressing mode toggle. Default: `true`.
- `credentials.access_key_id_env`: optional environment variable or secret-file name for the access key id.
- `credentials.secret_access_key_env`: optional environment variable or secret-file name for the secret access key.
The S3 credential variable names must either both be configured or both be omitted. When omitted, the AWS SDK default credential chain is used. When configured, both values must resolve to non-empty strings through the process environment or `secrets.directory`.
### HTTP Upload Source Backend
HTTP upload backends are valid only as pipeline sources and are served by `distributor serve`.
```yaml
backend: http_upload
token_env: WEATHER_DAILY_UPLOAD_TOKEN
staging_path: /var/spool/distributor/weather-daily
max_upload_size: 20MB
```
- `backend`: required value `http_upload`.
- `token_env`: required environment variable or secret-file name containing the bearer token.
- `staging_path`: optional staging path. Default: `<server.http.staging_root>/<pipeline id>`.
- `max_upload_size`: optional per-source upload limit. Default: `server.http.max_upload_size`.
## Validation Policy
```yaml
validation:
on_digest_mismatch: fail
```
- `validation.on_digest_mismatch`: optional. Default and only accepted value: `fail`.
Source bundle digest mismatches fail validation before destination writes occur.
## Destination Fields
Each destination embeds a backend config at the destination level and may also configure publishing, transforms, path mapping, links, and transfer behavior.
```yaml
destinations:
- id: archive
backend: local
path: /srv/reports/archive
publish:
source: true
html: false
path_mapping:
mode: preserve_relative
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
```
- `id`: required unique slug-like identifier within the pipeline.
- Backend fields: required according to the selected destination backend.
- `publish`: optional publish policy. Default: source-only publication.
- `transform`: required only when publishing generated HTML.
- `path_mapping`: optional destination path mapping policy.
- `links`: optional public URL metadata policy.
- `transfer`: optional destination reconciliation policy.
Destination ids must be unique within a pipeline.
## Publish And Transform Policy
### Source-Only Publication
```yaml
publish:
source: true
html: false
```
`publish.source` controls whether source manifest files are copied to the destination.
### Markdown-To-HTML Publication
```yaml ```yaml
publish: publish:
@@ -158,240 +292,106 @@ transform:
mode: sidecar mode: sidecar
``` ```
Sidecar generation writes `report.html` for `report.md`. It does not mutate the source bundle. `publish.html` controls whether generated HTML outputs are published. When `publish.html` is `true`, `transform.markdown_to_html.enabled` must also be `true`.
To publish a single Markdown file as `index.html`: Markdown transform fields:
```yaml - `transform.markdown_to_html.enabled`: enables Markdown-to-HTML generation for this destination.
publish: - `transform.markdown_to_html.mode`: optional. Accepted values are `sidecar` and `index`; default is `sidecar` when a Markdown transform block is present.
source: false - `transform.markdown_to_html.input`: optional source manifest path for `index` mode only.
html: true
transform:
markdown_to_html:
enabled: true
mode: index
input: report.md
```
When `mode: index` omits `input`, the source manifest must list exactly one Markdown file. `sidecar` mode renders every manifest-listed `.md` file to a same-directory `.html` output. `index` mode renders one Markdown source to `index.html` at the destination bundle path. If `index` mode omits `input`, the selected source bundle must contain exactly one Markdown file.
At least one output type must be enabled. Enabled Markdown transforms are rejected when `publish.html` is `false`, and `input` is rejected unless `mode` is `index`.
## Destination Path Mapping ## Destination Path Mapping
Each destination chooses how source bundle paths map into that destination:
```yaml ```yaml
path_mapping: path_mapping:
mode: preserve_relative mode: preserve_relative
``` ```
`preserve_relative` is the default. It publishes each discovered source bundle at the same path relative to the destination backend root. A source bundle at `daily/2026-06-01` publishes below `daily/2026-06-01` for that destination. - `path_mapping.mode`: optional. Accepted values are `preserve_relative` and `fixed`; default is `preserve_relative`.
`fixed` publishes one selected source bundle directly at the destination backend root: `preserve_relative` publishes each discovered source bundle at the same path relative to the destination backend root.
```yaml `fixed` publishes one selected source bundle directly at the destination backend root. Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins.
destinations:
- id: latest-html
backend: local
path: /srv/www/reports/latest
path_mapping:
mode: fixed
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: index
input: report.md
```
Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple candidates have the same timestamp, the source-root-relative bundle path in ascending order wins. Older candidates are not planned or written for that destination. Fixed mapping is useful for stable latest-style paths. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
Fixed mapping is useful for stable latest-style paths. It is more destructive than archive-style publication because successive source bundles target the same destination root. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
## Destination Links ## Destination Links
Destinations can record public URLs for published outputs:
```yaml ```yaml
links: links:
base_url: https://reports.example.com/archive base_url: https://reports.example.com/archive
primary: auto primary: auto
``` ```
`links.base_url` is an absolute `http` or `https` URL corresponding to the destination backend root. It may include a path prefix, but it must not include a query string or fragment. Distributor does not infer public URLs from backend config. - `links.base_url`: required when `links` is present. It must be an absolute `http` or `https` URL with a host and no query string or fragment.
- `links.primary`: optional. Accepted values are `auto`, `html`, and `source`; default is `auto` when `links` is present.
`links.primary` selects the top-level primary URL stored in destination state: `distributor` does not infer public URLs from backend config. Output URLs are built from `links.base_url`, the destination bundle path, and output paths using URL path semantics. `index.html` outputs produce directory-style URLs that omit the filename.
Primary URL policies:
- `auto`: prefer `index.html`, then generated HTML, then source outputs. - `auto`: prefer `index.html`, then generated HTML, then source outputs.
- `html`: use the first generated HTML output. - `html`: use the first generated HTML output.
- `source`: use the first copied source output. - `source`: use the first copied source output.
If a destination has no `links` block, no URL metadata is generated. If a primary policy has no matching output, per-output URLs are still recorded and the top-level primary URL is omitted. If no output matches the primary policy, per-output URLs may still be recorded and the top-level primary URL is omitted.
Output URLs are built from `links.base_url`, the destination bundle path, and the output path using URL path semantics. `index.html` outputs produce directory-style URLs that omit the filename. ## Transfer Policy
## Reference ```yaml
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
```
Top level: Transfer fields and accepted values:
- `server.http.bind`: optional HTTP bind address; defaults to `127.0.0.1:8080`. - `transfer.on_destination_same`: `skip` or `fail`. Default: `skip`.
- `server.http.staging_root`: optional root for default HTTP upload staging paths; defaults to `/var/spool/distributor`. - `transfer.on_destination_older`: `replace` or `fail`. Default: `replace`.
- `server.http.max_upload_size`: optional default upload size limit; defaults to `20MB`. - `transfer.on_destination_newer`: `skip`, `replace`, or `fail`. Default: `skip`.
- `server.http.queue_size`: optional HTTP upload admission queue size; defaults to `16`. - `transfer.on_conflict`: `fail` or `replace`. Default: `fail`.
- `server.http.max_concurrency`: optional HTTP upload worker concurrency; defaults to `1`.
- `server.http.retention`: optional completed upload retention duration; defaults to `24h`.
- `secrets.directory`: optional credential secrets directory.
- `pipelines`: required non-empty list.
Pipeline: `replace` for `on_destination_newer` and `on_conflict` is honored only when `run --force` is supplied. There is no config field that enables forced replacement by default.
- `id`: required unique slug-like identifier.
- `source`: required backend config.
- `validation.on_digest_mismatch`: optional; defaults to `fail`; only `fail` is supported.
- `destinations`: required non-empty destination list.
Source backend:
- `backend`: required.
- `path`: required for `local` and `ssh`.
- `host`: required for `ssh`.
- `user`: optional for `ssh`; defaults to the current OS user when available.
- `port`: optional for `ssh`; defaults to `22`.
- `ssh_key_file`: optional for `ssh`.
- `known_hosts`: optional for `ssh`; defaults to the service user's OpenSSH `known_hosts` path when available.
- `host_key_policy`: optional for `ssh`; defaults to `accept-new`.
- `endpoint`: required for `s3`.
- `bucket`: required for `s3`.
- `prefix`: optional for `s3`; leading and trailing slashes are trimmed.
- `region`: optional for `s3`; defaults to `us-east-1`.
- `force_path_style`: optional for `s3`; defaults to `true`. Set `false` only for services that require virtual-host addressing.
- `credentials.access_key_id_env`: optional S3 credential environment variable name.
- `credentials.secret_access_key_env`: optional S3 credential environment variable name.
- `token_env`: required for `http_upload`; names the token environment variable or secret-file name.
- `staging_path`: optional for `http_upload`; defaults below `server.http.staging_root` using the pipeline id.
- `max_upload_size`: optional for `http_upload`; defaults to `server.http.max_upload_size`.
Destination:
- `id`: required unique slug-like identifier within the pipeline.
- Backend fields: same accepted shape as source backends, with destination fields at the destination level.
- `publish`: optional; defaults to source-only publication.
- `transform`: required only for generated HTML publication.
- `path_mapping.mode`: optional; defaults to `preserve_relative`. Accepted values are `preserve_relative` and `fixed`.
- `links.base_url`: optional links block; when present, `base_url` is required and must be an absolute HTTP or HTTPS URL without query string or fragment.
- `links.primary`: optional; defaults to `auto`. Accepted values are `auto`, `html`, and `source`.
- `transfer`: optional; defaults described below.
Accepted backend names:
- `local`: executable; requires `path`.
- `ssh`: executable; requires `host` and `path`.
- `s3`: executable; requires `endpoint` and `bucket`.
- `http_upload`: source-only configuration; requires `token_env`.
## Size And Duration Values ## Size And Duration Values
Upload size fields use an integer plus one of the supported binary-size suffixes: Upload size fields must be YAML strings with an integer and one of these suffixes:
- `B` - `B`
- `KB` - `KB`
- `MB` - `MB`
- `GB` - `GB`
Suffix multipliers use powers of 1024. Size values must be greater than zero after defaults are applied. Suffix multipliers use powers of 1024. Values must be greater than zero after defaults are applied.
HTTP retention uses Go-style duration strings such as `24h`, `90m`, or `168h`. Retention must be greater than zero after defaults are applied. Duration fields must be YAML strings accepted by Go duration parsing, such as `24h`, `90m`, or `168h`. Values must be greater than zero after defaults are applied.
## SSH Backend
SSH uses native SFTP. It can be used for sources, destinations, or both:
```yaml
backend: ssh
host: example.com
user: distributor
port: 2222
path: /remote/root
ssh_key_file: /home/distributor/.ssh/id_ed25519
known_hosts: /home/distributor/.ssh/known_hosts
host_key_policy: accept-new
```
Authentication uses SSH agent identities first when `SSH_AUTH_SOCK` is set, then `ssh_key_file` if configured. Password authentication in YAML is not supported.
Host key policies:
- `strict`, `true`, and `"true"` require a matching known host key.
- `accept-new` accepts and persists a new host key, but fails if an existing key changed. During `run --dry-run`, new host keys are accepted only for the current connection and are not persisted.
- `off`, `false`, and `"false"` disable host key checking and are insecure.
`accept-new` and `strict` use `known_hosts` when configured. If omitted, distributor uses the current service user's default OpenSSH `known_hosts` path where practical. `accept-new` fails when it needs to persist a new host key and no writable `known_hosts` path is available. It does not create a missing parent `.ssh` directory.
## S3 Backend
S3 uses the AWS SDK for Go v2 and supports S3-compatible endpoints:
```yaml
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: archive
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
```
`endpoint` and `bucket` are required. `prefix` is an optional backend root; it is treated as an object-key prefix, not a real directory. Prefixes must be clean slash-separated paths after trimming leading and trailing slashes. `http://` endpoints are allowed for explicitly configured local development or local S3-compatible test services.
If either credential environment variable name is configured, both must be configured and both referenced variables must resolve to non-empty values through the real process environment or `secrets.directory`. Explicit credentials take precedence over the AWS SDK default credential chain. If credential environment variable names are omitted, the SDK default credential chain is used and `secrets.directory` values are not injected into the process environment.
Publish policy:
- `publish.source`: publish source artifacts.
- `publish.html`: publish generated HTML artifacts from Markdown source files.
At least one output type must be enabled. When `publish.html` is true, `transform.markdown_to_html.enabled` must be `true`.
Markdown-to-HTML transform:
- `transform.markdown_to_html.enabled`: enables Markdown-to-HTML generation for destinations with `publish.html: true`.
- `transform.markdown_to_html.mode`: optional; defaults to `sidecar`. Accepted values are `sidecar` and `index`.
- `transform.markdown_to_html.input`: optional source manifest path for `index` mode. It must identify a listed Markdown file.
`sidecar` mode renders each manifest-listed `.md` file to a same-directory `.html` output. `index` mode renders one selected Markdown file to `index.html` at the destination bundle path. Enabled Markdown-to-HTML config is rejected when `publish.html` is false, and `input` is valid only with `mode: index`.
Transfer policy:
- `transfer.on_destination_same`: `skip` or `fail`; defaults to `skip`.
- `transfer.on_destination_older`: `replace` or `fail`; defaults to `replace`.
- `transfer.on_destination_newer`: `skip`, `replace`, or `fail`; defaults to `skip`.
- `transfer.on_conflict`: `fail` or `replace`; defaults to `fail`.
`replace` for `on_destination_newer` and `on_conflict` is honored only when `run --force` is used for that invocation. Force is CLI-only; there is no persistent config field that enables forced replacement by default.
## Defaults ## Defaults
Defaults are applied after YAML decoding and before validation: Defaults are applied after YAML decoding and before validation:
- `validation.on_digest_mismatch: fail`
- SSH `port: 22`
- SSH `host_key_policy: accept-new`
- S3 `region: us-east-1`
- S3 `force_path_style: true`
- `server.http.bind: 127.0.0.1:8080` - `server.http.bind: 127.0.0.1:8080`
- `server.http.staging_root: /var/spool/distributor` - `server.http.staging_root: /var/spool/distributor`
- `server.http.max_upload_size: 20MB` - `server.http.max_upload_size: 20MB`
- `server.http.queue_size: 16` - `server.http.queue_size: 16`
- `server.http.max_concurrency: 1` - `server.http.max_concurrency: 1`
- `server.http.retention: 24h` - `server.http.retention: 24h`
- `source.staging_path: /var/spool/distributor/<pipeline id>` for `http_upload` - `validation.on_digest_mismatch: fail`
- `source.max_upload_size: server.http.max_upload_size` for `http_upload` - SSH `port: 22`
- `transform.markdown_to_html.mode: sidecar` when a Markdown-to-HTML transform block is present and mode is omitted - SSH `host_key_policy: accept-new`
- `publish.source: true` - S3 `region: us-east-1`
- `publish.html: false` - S3 `prefix`: leading and trailing slashes trimmed
- S3 `force_path_style: true`
- `http_upload` source `staging_path: <server.http.staging_root>/<pipeline id>`
- `http_upload` source `max_upload_size: server.http.max_upload_size`
- `publish.source: true` and `publish.html: false`
- `transform.markdown_to_html.mode: sidecar` when a Markdown transform block is present and mode is omitted
- `path_mapping.mode: preserve_relative` - `path_mapping.mode: preserve_relative`
- `links.primary: auto` when a `links` block is present and `primary` is omitted - `links.primary: auto` when a `links` block is present and `primary` is omitted
- `transfer.on_destination_same: skip` - `transfer.on_destination_same: skip`
@@ -408,29 +408,31 @@ secrets:
directory: /run/secrets/distributor directory: /run/secrets/distributor
``` ```
Each regular file in the directory becomes an internal credential environment value named by the filename. Valid filenames must match `[A-Za-z_][A-Za-z0-9_]*`. Directories are ignored, and symlinks to regular files are followed. Exactly one trailing LF or CRLF is trimmed from each file; other whitespace is preserved. Each regular file in the directory becomes an internal credential environment value named by the filename. Valid filenames match `[A-Za-z_][A-Za-z0-9_]*`. Directories are ignored. Symlinks to regular files are followed. Exactly one trailing LF or CRLF is trimmed from each file; other whitespace is preserved.
The resolver checks the real process environment first, then the secrets directory. If both define the same variable with different values, `run` prints a warning with the variable name and uses the real environment value. Secret values are not printed. The process environment is not modified, so SDK default credential chains see only real environment variables. Credential resolution checks the real process environment first, then `secrets.directory`. If both define the same name with different values, `run` emits a warning with the variable name and uses the real environment value. Secret values are not printed. The process environment is not modified, so SDK default credential chains see only real process environment variables.
S3 credentials may name environment variables: Fields resolved through this resolver:
- `credentials.access_key_id_env` - `credentials.access_key_id_env`
- `credentials.secret_access_key_env` - `credentials.secret_access_key_env`
- `source.token_env` for `http_upload` sources
HTTP upload tokens name one environment variable or secret-file name: ## Maintained Examples
- `source.token_env` Maintained examples live under [examples](../examples/). Config tests load every file listed here.
## Examples Local examples:
Maintained examples live under [examples](../examples/): - `local-to-local.yml`: minimal local-to-local config using absolute sample paths; load-tested, but paths should be adapted before running.
- `local-publish.yml`: runnable local source publication used by the README quickstart.
- `local-html.yml`: local sidecar HTML publication.
- `local-index.yml`: local `index.html` publication.
- `fan-out.yml`: local fan-out publication to source and HTML destinations.
- `archive-and-latest.yml`: local archive plus fixed latest publication.
- `http-upload-local.yml`: local HTTP upload server config; requires `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN` in the process environment or as a secret-file name before running `serve`.
- `local-to-local.yml`: minimal local config. Environment-gated remote examples:
- `local-publish.yml`: runnable local source publication.
- `local-html.yml`: runnable local HTML publication. - `ssh-destination.yml`: local-to-SSH publication; replace host, user, path, key, and known-host values for an SSH/SFTP endpoint you control.
- `local-index.yml`: runnable local `index.html` publication. - `s3-destination.yml`: local-to-S3 publication; replace endpoint, bucket, prefix, region, and credential variable names for an S3-compatible service you control.
- `fan-out.yml`: runnable local fan-out publication to source and HTML destinations.
- `archive-and-latest.yml`: runnable local fan-out publication to an archive destination and a fixed latest destination.
- `http-upload-local.yml`: local HTTP upload server example with a token environment variable reference.
- `ssh-destination.yml`: environment-gated local-to-SSH publication example.
- `s3-destination.yml`: environment-gated local-to-S3 publication example.