Document pipeline-scoped upload behavior

This commit is contained in:
2026-06-08 04:46:22 +00:00
parent bd5892d1f2
commit 29f01da37b
7 changed files with 103 additions and 201 deletions

View File

@@ -86,11 +86,15 @@ server:
queue_size: 16
max_concurrency: 1
retention: 24h
upload_tokens:
- id: weather-reporter
token_env: WEATHER_UPLOAD_TOKEN
allow_pipelines:
- weather-daily
pipelines:
- id: weather-daily
source:
backend: http_upload
token_env: WEATHER_DAILY_UPLOAD_TOKEN
staging_path: /var/spool/distributor/weather-daily
max_upload_size: 20MB
destinations:
@@ -99,9 +103,9 @@ pipelines:
path: /srv/reports/archive
```
`token_env` is required for `http_upload` sources. `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`. `max_upload_size` defaults to `server.http.max_upload_size`.
`upload_tokens` is required when any pipeline source uses `http_upload`. Each token record resolves its bearer token value from the process environment or `secrets.directory`. `allow_pipelines` lists configured upload pipeline ids that the token may submit to. One token may authorize multiple upload pipelines, and multiple tokens may authorize the same upload pipeline.
`serve` maps each resolved bearer token to exactly one `http_upload` pipeline. Startup fails when a token is missing, empty, or duplicates another upload pipeline token.
For `http_upload` sources, `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`. `max_upload_size` defaults to `server.http.max_upload_size`.
## Top-Level Fields
@@ -124,6 +128,18 @@ Numeric server values and durations must be greater than zero after defaults are
See [Secrets](#secrets) for resolution rules.
### `upload_tokens`
`upload_tokens` configures bearer tokens for `distributor serve`. It is required when any pipeline source backend is `http_upload` and is invalid when no upload pipelines are configured.
Each token has:
- `id`: required unique slug-like identifier for the token record. It must start with a letter or number and may contain letters, numbers, `.`, `_`, and `-`.
- `token_env`: required environment variable or secret-file name containing the bearer token value.
- `allow_pipelines`: required non-empty list of configured pipeline ids whose source backend is `http_upload`.
Token values must resolve to non-empty strings and must be unique across token records. Every configured upload pipeline must be allowed by at least one token.
### `pipelines`
`pipelines` is required and must contain at least one pipeline.
@@ -216,13 +232,11 @@ HTTP upload backends are valid only as pipeline sources and are served by `distr
```yaml
backend: http_upload
token_env: WEATHER_DAILY_UPLOAD_TOKEN
staging_path: /var/spool/distributor/weather-daily
max_upload_size: 20MB
```
- `backend`: required value `http_upload`.
- `token_env`: required environment variable or secret-file name containing the bearer token.
- `staging_path`: optional staging path. Default: `<server.http.staging_root>/<pipeline id>`.
- `max_upload_size`: optional per-source upload limit. Default: `server.http.max_upload_size`.
@@ -416,7 +430,7 @@ Fields resolved through this resolver:
- `credentials.access_key_id_env`
- `credentials.secret_access_key_env`
- `source.token_env` for `http_upload` sources
- `upload_tokens[].token_env`
## Maintained Examples