eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Add HTTP upload configuration support

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-03 15:01:01 +00:00
parent 28eb5e07a0
commit 35c5237dfc
7 changed files with 469 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/internal/config.md
View File

@@ -6,7 +6,7 @@
## Inputs and outputs
Input is a YAML file containing optional `secrets` and required `pipelines`. Output is a `Config` value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.
Input is a YAML file containing optional `server`, optional `secrets`, and required `pipelines`. Output is a `Config` value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.
## Loading flow
@@ -20,6 +20,14 @@ Known-field checking rejects misspelled or unknown YAML keys before defaults and
Defaults are applied in `ApplyDefaults`:
- HTTP server `bind` defaults to `127.0.0.1:8080`;
- HTTP server `staging_root` defaults to `/var/spool/distributor`;
- HTTP server `max_upload_size` defaults to `20MB`;
- HTTP server `queue_size` defaults to `16`;
- HTTP server `max_concurrency` defaults to `1`;
- HTTP server `retention` defaults to `24h`;
- `http_upload` source `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`;
- `http_upload` source `max_upload_size` defaults to `server.http.max_upload_size`;
- pipeline validation defaults `on_digest_mismatch` to `fail`;
- SSH backend `port` defaults to `22`;
- SSH backend `host_key_policy` defaults to `accept-new`;
@@ -34,7 +42,11 @@ Defaults are applied in `ApplyDefaults`:
## Validation responsibilities
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.
Validation requires positive HTTP server limits and retention, at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.
HTTP upload sources require `token_env`, a staging path after defaults, and a positive maximum upload size. Literal token fields are not part of the YAML schema. The `http_upload` backend is accepted only for sources and rejected for destinations.
Upload size values are parsed from strings with `B`, `KB`, `MB`, or `GB` suffixes using 1024 multipliers. Retention values are parsed with `time.ParseDuration`. Explicit zero values fail validation; omitted values receive defaults before validation.
Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
@@ -46,12 +58,14 @@ Destination links are optional. When a `links` block is present, `base_url` is r
## Executable support boundary
Config validation accepts `local`, `ssh`, and `s3` backend shapes. Runtime execution opens all three through `internal/app`.
Config validation accepts `local`, `ssh`, `s3`, and source-only `http_upload` backend shapes. Runtime execution opens `local`, `ssh`, and `s3` through `internal/app`.
SSH config uses structured fields: `host`, optional `user`, optional `port`, `path`, optional `ssh_key_file`, optional `known_hosts`, and optional `host_key_policy`. `host_key_policy` accepts YAML booleans and strings and normalizes `true`/`strict`, `accept-new`, and `false`/`off`.
S3 config requires `endpoint` and `bucket`, normalizes optional `prefix`, defaults `region` to `us-east-1`, and defaults omitted `force_path_style` to `true` while preserving explicit `false`.
HTTP upload config is source-only. Config owns its YAML shape, defaulting, size and duration parsing, and validation. The config package does not resolve `token_env`, authenticate requests, stage uploads, or execute HTTP upload sources.
## Secrets and credential resolution
`secrets.directory` points to a directory of credential files. `LoadSecretEnvironment` reads regular files and symlinks to regular files, rejects invalid filenames, trims exactly one trailing LF or CRLF, and returns an `Environment` resolver plus conflict metadata.