Add HTTP upload configuration support

This commit is contained in:
2026-06-03 15:01:01 +00:00
parent 28eb5e07a0
commit 35c5237dfc
7 changed files with 469 additions and 8 deletions

View File

@@ -10,7 +10,7 @@ If `--config` is omitted, `run` uses:
/usr/local/etc/distributor/config.yml
```
Config parsing rejects unknown YAML fields. The executable backends are `local`, `ssh`, and `s3`.
Config parsing rejects unknown YAML fields. The executable `run` backends are `local`, `ssh`, and `s3`. The schema also accepts `http_upload` as a source-only ingestion backend configuration.
## Minimal Local Config
@@ -31,6 +31,14 @@ This publishes source files only. It uses the default validation and transfer po
## Production-Oriented Local Config
```yaml
server:
http:
bind: 127.0.0.1:8080
staging_root: /var/spool/distributor
max_upload_size: 20MB
queue_size: 16
max_concurrency: 1
retention: 24h
pipelines:
- id: reports
source:
@@ -52,6 +60,38 @@ pipelines:
on_conflict: fail
```
## HTTP Upload Source Configuration
HTTP upload sources are configured as pipeline sources only. They are not valid destination backends.
```yaml
server:
http:
bind: 127.0.0.1:8080
staging_root: /var/spool/distributor
max_upload_size: 20MB
queue_size: 16
max_concurrency: 1
retention: 24h
pipelines:
- id: weather-daily
source:
backend: http_upload
token_env: WEATHER_DAILY_UPLOAD_TOKEN
staging_path: /var/spool/distributor/weather-daily
max_upload_size: 20MB
destinations:
- id: archive
backend: local
path: /srv/reports/archive
```
`source.token_env` is required and names the environment variable or `secrets.directory` file that provides the bearer token. Literal upload tokens are not supported in YAML.
`source.staging_path` is optional. When omitted, it defaults to `<server.http.staging_root>/<pipeline id>`.
`source.max_upload_size` is optional. When omitted, it defaults to `server.http.max_upload_size`.
## HTML Publication
To publish generated sidecar HTML from Markdown files:
@@ -143,6 +183,12 @@ Output URLs are built from `links.base_url`, the destination bundle path, and th
Top level:
- `server.http.bind`: optional HTTP bind address; defaults to `127.0.0.1:8080`.
- `server.http.staging_root`: optional root for default HTTP upload staging paths; defaults to `/var/spool/distributor`.
- `server.http.max_upload_size`: optional default upload size limit; defaults to `20MB`.
- `server.http.queue_size`: optional HTTP upload admission queue size; defaults to `16`.
- `server.http.max_concurrency`: optional HTTP upload worker concurrency; defaults to `1`.
- `server.http.retention`: optional completed upload retention duration; defaults to `24h`.
- `secrets.directory`: optional credential secrets directory.
- `pipelines`: required non-empty list.
@@ -170,6 +216,9 @@ Source backend:
- `force_path_style`: optional for `s3`; defaults to `true`. Set `false` only for services that require virtual-host addressing.
- `credentials.access_key_id_env`: optional S3 credential environment variable name.
- `credentials.secret_access_key_env`: optional S3 credential environment variable name.
- `token_env`: required for `http_upload`; names the token environment variable or secret-file name.
- `staging_path`: optional for `http_upload`; defaults below `server.http.staging_root` using the pipeline id.
- `max_upload_size`: optional for `http_upload`; defaults to `server.http.max_upload_size`.
Destination:
@@ -187,6 +236,20 @@ Accepted backend names:
- `local`: executable; requires `path`.
- `ssh`: executable; requires `host` and `path`.
- `s3`: executable; requires `endpoint` and `bucket`.
- `http_upload`: source-only configuration; requires `token_env`.
## Size And Duration Values
Upload size fields use an integer plus one of the supported binary-size suffixes:
- `B`
- `KB`
- `MB`
- `GB`
Suffix multipliers use powers of 1024. Size values must be greater than zero after defaults are applied.
HTTP retention uses Go-style duration strings such as `24h`, `90m`, or `168h`. Retention must be greater than zero after defaults are applied.
## SSH Backend
@@ -266,6 +329,14 @@ Defaults are applied after YAML decoding and before validation:
- SSH `host_key_policy: accept-new`
- S3 `region: us-east-1`
- S3 `force_path_style: true`
- `server.http.bind: 127.0.0.1:8080`
- `server.http.staging_root: /var/spool/distributor`
- `server.http.max_upload_size: 20MB`
- `server.http.queue_size: 16`
- `server.http.max_concurrency: 1`
- `server.http.retention: 24h`
- `source.staging_path: /var/spool/distributor/<pipeline id>` for `http_upload`
- `source.max_upload_size: server.http.max_upload_size` for `http_upload`
- `transform.markdown_to_html.mode: sidecar` when a Markdown-to-HTML transform block is present and mode is omitted
- `publish.source: true`
- `publish.html: false`