Add HTTP upload configuration support

docs/config.md

@@ -10,7 +10,7 @@ If `--config` is omitted, `run` uses:
 /usr/local/etc/distributor/config.yml
 ```
 
-Config parsing rejects unknown YAML fields. The executable backends are `local`, `ssh`, and `s3`.
+Config parsing rejects unknown YAML fields. The executable `run` backends are `local`, `ssh`, and `s3`. The schema also accepts `http_upload` as a source-only ingestion backend configuration.
 
 ## Minimal Local Config
 
@@ -31,6 +31,14 @@ This publishes source files only. It uses the default validation and transfer po
 ## Production-Oriented Local Config
 
 ```yaml
+server:
+  http:
+    bind: 127.0.0.1:8080
+    staging_root: /var/spool/distributor
+    max_upload_size: 20MB
+    queue_size: 16
+    max_concurrency: 1
+    retention: 24h
 pipelines:
   - id: reports
     source:
@@ -52,6 +60,38 @@ pipelines:
           on_conflict: fail
 ```
 
+## HTTP Upload Source Configuration
+
+HTTP upload sources are configured as pipeline sources only. They are not valid destination backends.
+
+```yaml
+server:
+  http:
+    bind: 127.0.0.1:8080
+    staging_root: /var/spool/distributor
+    max_upload_size: 20MB
+    queue_size: 16
+    max_concurrency: 1
+    retention: 24h
+pipelines:
+  - id: weather-daily
+    source:
+      backend: http_upload
+      token_env: WEATHER_DAILY_UPLOAD_TOKEN
+      staging_path: /var/spool/distributor/weather-daily
+      max_upload_size: 20MB
+    destinations:
+      - id: archive
+        backend: local
+        path: /srv/reports/archive
+```
+
+`source.token_env` is required and names the environment variable or `secrets.directory` file that provides the bearer token. Literal upload tokens are not supported in YAML.
+
+`source.staging_path` is optional. When omitted, it defaults to `<server.http.staging_root>/<pipeline id>`.
+
+`source.max_upload_size` is optional. When omitted, it defaults to `server.http.max_upload_size`.
+
 ## HTML Publication
 
 To publish generated sidecar HTML from Markdown files:
@@ -143,6 +183,12 @@ Output URLs are built from `links.base_url`, the destination bundle path, and th
 
 Top level:
 
+- `server.http.bind`: optional HTTP bind address; defaults to `127.0.0.1:8080`.
+- `server.http.staging_root`: optional root for default HTTP upload staging paths; defaults to `/var/spool/distributor`.
+- `server.http.max_upload_size`: optional default upload size limit; defaults to `20MB`.
+- `server.http.queue_size`: optional HTTP upload admission queue size; defaults to `16`.
+- `server.http.max_concurrency`: optional HTTP upload worker concurrency; defaults to `1`.
+- `server.http.retention`: optional completed upload retention duration; defaults to `24h`.
 - `secrets.directory`: optional credential secrets directory.
 - `pipelines`: required non-empty list.
 
@@ -170,6 +216,9 @@ Source backend:
 - `force_path_style`: optional for `s3`; defaults to `true`. Set `false` only for services that require virtual-host addressing.
 - `credentials.access_key_id_env`: optional S3 credential environment variable name.
 - `credentials.secret_access_key_env`: optional S3 credential environment variable name.
+- `token_env`: required for `http_upload`; names the token environment variable or secret-file name.
+- `staging_path`: optional for `http_upload`; defaults below `server.http.staging_root` using the pipeline id.
+- `max_upload_size`: optional for `http_upload`; defaults to `server.http.max_upload_size`.
 
 Destination:
 
@@ -187,6 +236,20 @@ Accepted backend names:
 - `local`: executable; requires `path`.
 - `ssh`: executable; requires `host` and `path`.
 - `s3`: executable; requires `endpoint` and `bucket`.
+- `http_upload`: source-only configuration; requires `token_env`.
+
+## Size And Duration Values
+
+Upload size fields use an integer plus one of the supported binary-size suffixes:
+
+- `B`
+- `KB`
+- `MB`
+- `GB`
+
+Suffix multipliers use powers of 1024. Size values must be greater than zero after defaults are applied.
+
+HTTP retention uses Go-style duration strings such as `24h`, `90m`, or `168h`. Retention must be greater than zero after defaults are applied.
 
 ## SSH Backend
 
@@ -266,6 +329,14 @@ Defaults are applied after YAML decoding and before validation:
 - SSH `host_key_policy: accept-new`
 - S3 `region: us-east-1`
 - S3 `force_path_style: true`
+- `server.http.bind: 127.0.0.1:8080`
+- `server.http.staging_root: /var/spool/distributor`
+- `server.http.max_upload_size: 20MB`
+- `server.http.queue_size: 16`
+- `server.http.max_concurrency: 1`
+- `server.http.retention: 24h`
+- `source.staging_path: /var/spool/distributor/<pipeline id>` for `http_upload`
+- `source.max_upload_size: server.http.max_upload_size` for `http_upload`
 - `transform.markdown_to_html.mode: sidecar` when a Markdown-to-HTML transform block is present and mode is omitted
 - `publish.source: true`
 - `publish.html: false`

docs/internal/config.md

@@ -6,7 +6,7 @@
 
 ## Inputs and outputs
 
-Input is a YAML file containing optional `secrets` and required `pipelines`. Output is a `Config` value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.
+Input is a YAML file containing optional `server`, optional `secrets`, and required `pipelines`. Output is a `Config` value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.
 
 ## Loading flow
 
@@ -20,6 +20,14 @@ Known-field checking rejects misspelled or unknown YAML keys before defaults and
 
 Defaults are applied in `ApplyDefaults`:
 
+- HTTP server `bind` defaults to `127.0.0.1:8080`;
+- HTTP server `staging_root` defaults to `/var/spool/distributor`;
+- HTTP server `max_upload_size` defaults to `20MB`;
+- HTTP server `queue_size` defaults to `16`;
+- HTTP server `max_concurrency` defaults to `1`;
+- HTTP server `retention` defaults to `24h`;
+- `http_upload` source `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`;
+- `http_upload` source `max_upload_size` defaults to `server.http.max_upload_size`;
 - pipeline validation defaults `on_digest_mismatch` to `fail`;
 - SSH backend `port` defaults to `22`;
 - SSH backend `host_key_policy` defaults to `accept-new`;
@@ -34,7 +42,11 @@ Defaults are applied in `ApplyDefaults`:
 
 ## Validation responsibilities
 
-Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.
+Validation requires positive HTTP server limits and retention, at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.
+
+HTTP upload sources require `token_env`, a staging path after defaults, and a positive maximum upload size. Literal token fields are not part of the YAML schema. The `http_upload` backend is accepted only for sources and rejected for destinations.
+
+Upload size values are parsed from strings with `B`, `KB`, `MB`, or `GB` suffixes using 1024 multipliers. Retention values are parsed with `time.ParseDuration`. Explicit zero values fail validation; omitted values receive defaults before validation.
 
 Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
 
@@ -46,12 +58,14 @@ Destination links are optional. When a `links` block is present, `base_url` is r
 
 ## Executable support boundary
 
-Config validation accepts `local`, `ssh`, and `s3` backend shapes. Runtime execution opens all three through `internal/app`.
+Config validation accepts `local`, `ssh`, `s3`, and source-only `http_upload` backend shapes. Runtime execution opens `local`, `ssh`, and `s3` through `internal/app`.
 
 SSH config uses structured fields: `host`, optional `user`, optional `port`, `path`, optional `ssh_key_file`, optional `known_hosts`, and optional `host_key_policy`. `host_key_policy` accepts YAML booleans and strings and normalizes `true`/`strict`, `accept-new`, and `false`/`off`.
 
 S3 config requires `endpoint` and `bucket`, normalizes optional `prefix`, defaults `region` to `us-east-1`, and defaults omitted `force_path_style` to `true` while preserving explicit `false`.
 
+HTTP upload config is source-only. Config owns its YAML shape, defaulting, size and duration parsing, and validation. The config package does not resolve `token_env`, authenticate requests, stage uploads, or execute HTTP upload sources.
+
 ## Secrets and credential resolution
 
 `secrets.directory` points to a directory of credential files. `LoadSecretEnvironment` reads regular files and symlinks to regular files, rejects invalid filenames, trims exactly one trailing LF or CRLF, and returns an `Environment` resolver plus conflict metadata.