Add HTTP upload configuration support
This commit is contained in:
@@ -10,7 +10,7 @@ If `--config` is omitted, `run` uses:
|
||||
/usr/local/etc/distributor/config.yml
|
||||
```
|
||||
|
||||
Config parsing rejects unknown YAML fields. The executable backends are `local`, `ssh`, and `s3`.
|
||||
Config parsing rejects unknown YAML fields. The executable `run` backends are `local`, `ssh`, and `s3`. The schema also accepts `http_upload` as a source-only ingestion backend configuration.
|
||||
|
||||
## Minimal Local Config
|
||||
|
||||
@@ -31,6 +31,14 @@ This publishes source files only. It uses the default validation and transfer po
|
||||
## Production-Oriented Local Config
|
||||
|
||||
```yaml
|
||||
server:
|
||||
http:
|
||||
bind: 127.0.0.1:8080
|
||||
staging_root: /var/spool/distributor
|
||||
max_upload_size: 20MB
|
||||
queue_size: 16
|
||||
max_concurrency: 1
|
||||
retention: 24h
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
@@ -52,6 +60,38 @@ pipelines:
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
## HTTP Upload Source Configuration
|
||||
|
||||
HTTP upload sources are configured as pipeline sources only. They are not valid destination backends.
|
||||
|
||||
```yaml
|
||||
server:
|
||||
http:
|
||||
bind: 127.0.0.1:8080
|
||||
staging_root: /var/spool/distributor
|
||||
max_upload_size: 20MB
|
||||
queue_size: 16
|
||||
max_concurrency: 1
|
||||
retention: 24h
|
||||
pipelines:
|
||||
- id: weather-daily
|
||||
source:
|
||||
backend: http_upload
|
||||
token_env: WEATHER_DAILY_UPLOAD_TOKEN
|
||||
staging_path: /var/spool/distributor/weather-daily
|
||||
max_upload_size: 20MB
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: /srv/reports/archive
|
||||
```
|
||||
|
||||
`source.token_env` is required and names the environment variable or `secrets.directory` file that provides the bearer token. Literal upload tokens are not supported in YAML.
|
||||
|
||||
`source.staging_path` is optional. When omitted, it defaults to `<server.http.staging_root>/<pipeline id>`.
|
||||
|
||||
`source.max_upload_size` is optional. When omitted, it defaults to `server.http.max_upload_size`.
|
||||
|
||||
## HTML Publication
|
||||
|
||||
To publish generated sidecar HTML from Markdown files:
|
||||
@@ -143,6 +183,12 @@ Output URLs are built from `links.base_url`, the destination bundle path, and th
|
||||
|
||||
Top level:
|
||||
|
||||
- `server.http.bind`: optional HTTP bind address; defaults to `127.0.0.1:8080`.
|
||||
- `server.http.staging_root`: optional root for default HTTP upload staging paths; defaults to `/var/spool/distributor`.
|
||||
- `server.http.max_upload_size`: optional default upload size limit; defaults to `20MB`.
|
||||
- `server.http.queue_size`: optional HTTP upload admission queue size; defaults to `16`.
|
||||
- `server.http.max_concurrency`: optional HTTP upload worker concurrency; defaults to `1`.
|
||||
- `server.http.retention`: optional completed upload retention duration; defaults to `24h`.
|
||||
- `secrets.directory`: optional credential secrets directory.
|
||||
- `pipelines`: required non-empty list.
|
||||
|
||||
@@ -170,6 +216,9 @@ Source backend:
|
||||
- `force_path_style`: optional for `s3`; defaults to `true`. Set `false` only for services that require virtual-host addressing.
|
||||
- `credentials.access_key_id_env`: optional S3 credential environment variable name.
|
||||
- `credentials.secret_access_key_env`: optional S3 credential environment variable name.
|
||||
- `token_env`: required for `http_upload`; names the token environment variable or secret-file name.
|
||||
- `staging_path`: optional for `http_upload`; defaults below `server.http.staging_root` using the pipeline id.
|
||||
- `max_upload_size`: optional for `http_upload`; defaults to `server.http.max_upload_size`.
|
||||
|
||||
Destination:
|
||||
|
||||
@@ -187,6 +236,20 @@ Accepted backend names:
|
||||
- `local`: executable; requires `path`.
|
||||
- `ssh`: executable; requires `host` and `path`.
|
||||
- `s3`: executable; requires `endpoint` and `bucket`.
|
||||
- `http_upload`: source-only configuration; requires `token_env`.
|
||||
|
||||
## Size And Duration Values
|
||||
|
||||
Upload size fields use an integer plus one of the supported binary-size suffixes:
|
||||
|
||||
- `B`
|
||||
- `KB`
|
||||
- `MB`
|
||||
- `GB`
|
||||
|
||||
Suffix multipliers use powers of 1024. Size values must be greater than zero after defaults are applied.
|
||||
|
||||
HTTP retention uses Go-style duration strings such as `24h`, `90m`, or `168h`. Retention must be greater than zero after defaults are applied.
|
||||
|
||||
## SSH Backend
|
||||
|
||||
@@ -266,6 +329,14 @@ Defaults are applied after YAML decoding and before validation:
|
||||
- SSH `host_key_policy: accept-new`
|
||||
- S3 `region: us-east-1`
|
||||
- S3 `force_path_style: true`
|
||||
- `server.http.bind: 127.0.0.1:8080`
|
||||
- `server.http.staging_root: /var/spool/distributor`
|
||||
- `server.http.max_upload_size: 20MB`
|
||||
- `server.http.queue_size: 16`
|
||||
- `server.http.max_concurrency: 1`
|
||||
- `server.http.retention: 24h`
|
||||
- `source.staging_path: /var/spool/distributor/<pipeline id>` for `http_upload`
|
||||
- `source.max_upload_size: server.http.max_upload_size` for `http_upload`
|
||||
- `transform.markdown_to_html.mode: sidecar` when a Markdown-to-HTML transform block is present and mode is omitted
|
||||
- `publish.source: true`
|
||||
- `publish.html: false`
|
||||
|
||||
Reference in New Issue
Block a user