Add HTTP upload server and serve command

2026-06-03 15:22:52 +00:00
parent f0c10210eb
commit 6beef58dbf
12 changed files with 942 additions and 10 deletions
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -14,6 +14,7 @@ This discovers the example source bundle and publishes source files to `workspac
 distributor [--help]
 distributor version [--format text|json]
 distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
+distributor serve [--config <path>]
 distributor validate [--format text|json] <path>
 distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
 distributor inspect [--format text|json] <path>
@@ -23,11 +24,12 @@ distributor manifest create <bundle-path> --id <bundle-id> [options]

 - `version`: prints the application name and version. Development builds print `distributor dev`.
 - `run`: loads a YAML config, discovers source bundles, plans each configured destination, writes selected outputs unless `--dry-run` is set, and prints a final status summary.
+- `serve`: loads a YAML config, resolves HTTP upload bearer tokens, and runs the HTTP upload API.
 - `validate`: validates a local source bundle directory, a local source bundle tree, or one configured pipeline source.
 - `inspect`: validates source bundles and prints normalized bundle metadata for a local path or one configured pipeline source.
 - `manifest create`: creates `manifest.json` for a local source bundle directory.

-`validate` and `inspect` have two mutually exclusive modes: a local path shortcut, or configured source mode with `--config <path> --pipeline <id>`. Configured source mode opens only the selected pipeline source and supports configured `local`, `ssh`, and `s3` sources. It does not open destinations. `run` executes configured sources and destinations.
+`validate` and `inspect` have two mutually exclusive modes: a local path shortcut, or configured source mode with `--config <path> --pipeline <id>`. Configured source mode opens only the selected pipeline source and supports configured `local`, `ssh`, and `s3` sources. It does not open destinations. `run` executes configured `local`, `ssh`, and `s3` sources and destinations. `serve` executes configured `http_upload` sources through the upload API and normal destination fan-out.

 ## Flag reference

@@ -49,6 +51,10 @@ Output-producing subcommands:
 - `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write output files, destination state, or SSH `known_hosts` entries.
 - `--force`: allow explicit destructive replacement for supported conflict cases in this run only.

+`serve` flags:
+
+- `--config <path>`: config file to load. If omitted, `serve` uses `/usr/local/etc/distributor/config.yml`.
+
 `validate` and `inspect` configured source flags:

 - `--config <path>`: config file to load for source validation or inspection. Required in configured source mode.
@@ -127,6 +133,39 @@ Publish the local HTML example:
 go run ./cmd/distributor run --config examples/local-html.yml
 ```

+Start the HTTP upload API:
+
+```sh
+go run ./cmd/distributor serve --config <config-path>
+```
+
+Upload an archive to the configured `http_upload` pipeline associated with a bearer token:
+
+```sh
+curl -X POST http://127.0.0.1:8080/upload \
+  -H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \
+  -H "Content-Type: application/gzip" \
+  --data-binary @bundle.tar.gz
+```
+
+The upload response is accepted asynchronously:
+
+```json
+{"run_id":"reports.20260603T120000Z.abcdef12","status":"accepted"}
+```
+
+Check upload status:
+
+```sh
+curl http://127.0.0.1:8080/runs/<run-id>
+```
+
+Check server readiness:
+
+```sh
+curl http://127.0.0.1:8080/healthz
+```
+
 Preview local fan-out publication:

 ```sh
--- a/docs/config.md
+++ b/docs/config.md
@@ -2,15 +2,18 @@

 ## Config File Location

-`distributor run --config <path>` loads the YAML config at the provided path.
+`distributor run --config <path>` and `distributor serve --config <path>` load
+the YAML config at the provided path.

-If `--config` is omitted, `run` uses:
+If `--config` is omitted, both commands use:

 ```text
 /usr/local/etc/distributor/config.yml
 ```

-Config parsing rejects unknown YAML fields. The executable `run` backends are `local`, `ssh`, and `s3`. The schema also accepts `http_upload` as a source-only ingestion backend configuration.
+Config parsing rejects unknown YAML fields. The executable `run` backends are
+`local`, `ssh`, and `s3`. The `serve` command executes `http_upload` sources
+through the HTTP upload API and normal destination fan-out.

 ## Minimal Local Config

@@ -62,7 +65,9 @@ pipelines:

 ## HTTP Upload Source Configuration

-HTTP upload sources are configured as pipeline sources only. They are not valid destination backends.
+HTTP upload sources are configured as pipeline sources only. They are not valid
+destination backends. `distributor serve` maps each configured upload token to
+exactly one `http_upload` pipeline.

 ```yaml
 server:
@@ -92,6 +97,50 @@ pipelines:

 `source.max_upload_size` is optional. When omitted, it defaults to `server.http.max_upload_size`.

+The server resolves each `token_env` through the real process environment and
+the configured `secrets.directory` resolver. Startup fails if any configured
+upload token is missing, empty, or resolves to the same value as another upload
+pipeline. Token values are not read from YAML and are not printed in API
+responses.
+
+## HTTP Upload API
+
+`distributor serve` binds to `server.http.bind`, which defaults to
+`127.0.0.1:8080`.
+
+Routes:
+
+- `GET /healthz`: returns readiness status after config and upload tokens load.
+- `POST /upload`: accepts one tar or tar.gz source bundle archive.
+- `GET /runs/<run_id>`: returns an in-memory upload status record, or `404` if the run id is unknown or expired.
+
+`POST /upload` authenticates with:
+
+```text
+Authorization: Bearer <token>
+```
+
+The token selects the configured `http_upload` pipeline. Producers do not send a
+pipeline id. Requests with a submitted `pipeline` or `pipeline_id` query value
+are rejected.
+
+Accepted upload content types:
+
+- `application/x-tar`
+- `application/gzip`
+- `application/x-gzip`
+
+Accepted uploads return:
+
+```json
+{"run_id":"<id>","status":"accepted"}
+```
+
+The run id can be queried through `GET /runs/<run_id>` while the status record
+is retained in memory. Completed records expire after `server.http.retention`;
+expiration also removes committed staged bundle directories for completed
+uploads.
+
 ## HTML Publication

 To publish generated sidecar HTML from Markdown files:
--- a/docs/internal/app.md
+++ b/docs/internal/app.md
@@ -34,6 +34,11 @@ same destination fan-out path as normal runs.
 source. They share source backend construction with run workflows and never open
 destination backends.

+`Serve` is the CLI-facing HTTP upload server entrypoint. It loads config,
+loads the configured secrets directory, resolves upload bearer tokens for
+configured `http_upload` sources, creates an `UploadCoordinator`, binds
+`server.http.bind`, and serves the upload API until its context is cancelled.
+
 ## Run Reports

 `RunReport` is the structured result model for run workflows. It includes
@@ -116,6 +121,32 @@ Completed records retain the final run report or error text until
 their committed staged bundle directories. The coordinator is memory-only and
 does not persist queue state, status records, or run reports.

+## HTTP Upload Server
+
+The HTTP upload server is app-layer transport wiring around
+`UploadCoordinator`. It owns request authentication, route dispatch, HTTP status
+mapping, and JSON response projection. Bundle staging and publication remain in
+the coordinator and staged-source run path.
+
+Server startup resolves every configured `http_upload` source `token_env`
+through the config-owned environment resolver after `secrets.directory` has
+been loaded. Startup fails when a token is missing, empty, or duplicates another
+upload pipeline token. Error messages identify environment variable names and
+pipeline ids, but not token values.
+
+Routes:
+
+- `GET /healthz`: returns `200` after config, secrets, tokens, coordinator, and route setup succeed.
+- `POST /upload`: accepts authenticated tar and tar.gz archives and returns an accepted run id.
+- `GET /runs/<run_id>`: returns the current in-memory upload status record or `404`.
+
+The upload token maps to exactly one configured pipeline. Producers do not
+submit pipeline ids, and submitted `pipeline` or `pipeline_id` query values are
+rejected. Full queues are rejected before the request body is read. Oversized
+uploads, unsupported content types, invalid bearer tokens, full queues, and
+unknown status records are mapped to stable HTTP status codes without returning
+secret token values.
+
 ## Coordination

 `PipelineRunCoordinator` wraps `RunPipeline` with in-memory admission control.
@@ -171,6 +202,8 @@ Run helpers are grouped by responsibility:
 - `run_notify.go`: notification event projection and action filtering.
 - `run_coordinator.go`: in-memory run admission, run IDs, status records, and duplicate-run errors.
 - `upload_coordinator.go`: in-memory upload admission, queueing, status tracking, staging handoff, and staged-source execution.
+- `upload_http.go`: HTTP upload authentication, routes, JSON response projection, and HTTP error mapping.
+- `serve.go`: config/secrets loading and HTTP server startup.
 - `backends.go`: app-level backend factory wiring.
 - `transforms.go`: app-level transform registry wiring.
 - `source_select.go`: configured-source selection shared by `validate` and `inspect`.
--- a/docs/internal/config.md
+++ b/docs/internal/config.md
@@ -14,7 +14,8 @@ Input is a YAML file containing optional `server`, optional `secrets`, and requi

 Known-field checking rejects misspelled or unknown YAML keys before defaults and validation run.

-`LoadFile` does not read secret files. `Run` loads the configured secrets directory after config validation and before backend construction.
+`LoadFile` does not read secret files. App entrypoints load the configured
+secrets directory after config validation and before credential-consuming work.

 ## Defaults

@@ -58,13 +59,13 @@ Destination links are optional. When a `links` block is present, `base_url` is r

 ## Executable support boundary

-Config validation accepts `local`, `ssh`, `s3`, and source-only `http_upload` backend shapes. Runtime execution opens `local`, `ssh`, and `s3` through `internal/app`.
+Config validation accepts `local`, `ssh`, `s3`, and source-only `http_upload` backend shapes. Runtime `run`, `validate`, and `inspect` workflows open `local`, `ssh`, and `s3` through `internal/app`. Runtime `serve` workflows execute `http_upload` sources through the app upload coordinator and HTTP server.

 SSH config uses structured fields: `host`, optional `user`, optional `port`, `path`, optional `ssh_key_file`, optional `known_hosts`, and optional `host_key_policy`. `host_key_policy` accepts YAML booleans and strings and normalizes `true`/`strict`, `accept-new`, and `false`/`off`.

 S3 config requires `endpoint` and `bucket`, normalizes optional `prefix`, defaults `region` to `us-east-1`, and defaults omitted `force_path_style` to `true` while preserving explicit `false`.

-HTTP upload config is source-only. Config owns its YAML shape, defaulting, size and duration parsing, and validation. The config package does not resolve `token_env`, authenticate requests, stage uploads, or execute HTTP upload sources.
+HTTP upload config is source-only. Config owns its YAML shape, defaulting, size and duration parsing, and validation. The config package does not authenticate requests, stage uploads, or execute HTTP upload sources. The app layer resolves `token_env` through the config-owned environment resolver before starting the HTTP server.

 ## Secrets and credential resolution

--- a/docs/troubleshooting.md
+++ b/docs/troubleshooting.md
@@ -26,7 +26,10 @@ Safe fix: compare the file to the reference in [configuration](config.md) and re

 ## `validate config ... backend ... is unsupported`

-Likely cause: a source or destination uses a backend name other than `local`, `ssh`, or `s3`.
+Likely cause: a source or destination uses an unsupported backend name, or a
+command is trying to execute a backend that is valid only for another workflow.
+`run`, `validate`, and `inspect` execute `local`, `ssh`, and `s3` sources.
+`serve` executes `http_upload` sources.

 Diagnostic:

@@ -34,7 +37,127 @@ Diagnostic:
 rg -n "backend:" <config-path>
 ```

-Safe fix: use `backend: local`, `backend: ssh`, or `backend: s3` for executable workflows.
+Safe fix: use `backend: local`, `backend: ssh`, or `backend: s3` for normal
+source and destination workflows. Use `backend: http_upload` only for sources
+handled by `distributor serve`.
+
+## `bind HTTP server ... address already in use`
+
+Likely cause: another process is already listening on `server.http.bind`.
+
+Diagnostic:
+
+```sh
+ss -ltnp | rg '<port>'
+```
+
+Safe fix: stop the conflicting process or configure a different
+`server.http.bind` value. The default bind address is `127.0.0.1:8080`.
+
+## `upload token environment variable ... is not set`
+
+Likely cause: a configured `http_upload` source references `token_env`, but the
+variable is absent from both the real process environment and
+`secrets.directory`.
+
+Diagnostic:
+
+```sh
+env | cut -d= -f1 | rg '^<token-variable>$'
+ls -l <secrets-directory>/<token-variable>
+```
+
+Safe fix: set the real environment variable or create a readable
+secrets-directory file with the same name. Do not place literal token values in
+YAML.
+
+## `upload token environment variables ... resolve to the same value`
+
+Likely cause: two configured `http_upload` pipelines resolve to the same bearer
+token value.
+
+Diagnostic:
+
+```sh
+rg -n 'token_env:' <config-path>
+```
+
+Safe fix: assign a distinct non-empty token value to each `http_upload`
+pipeline. Distributor does not print the duplicate token value.
+
+## `POST /upload` returns `401`
+
+Likely cause: the request is missing `Authorization: Bearer <token>` or the
+token does not match any configured `http_upload` pipeline.
+
+Diagnostic:
+
+```sh
+curl -i -X POST http://127.0.0.1:8080/upload \
+  -H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \
+  -H "Content-Type: application/x-tar" \
+  --data-binary @bundle.tar
+```
+
+Safe fix: use the token value resolved by the configured `token_env`. Do not
+include token values in logs or tickets.
+
+## `POST /upload` returns `413`
+
+Likely cause: the request body exceeds the selected pipeline's
+`source.max_upload_size` or the default `server.http.max_upload_size`.
+
+Diagnostic:
+
+```sh
+ls -lh bundle.tar bundle.tar.gz
+rg -n 'max_upload_size:' <config-path>
+```
+
+Safe fix: upload a smaller archive, remove unnecessary files from the source
+bundle, or raise the configured upload size limit.
+
+## `POST /upload` returns `415`
+
+Likely cause: the upload uses an unsupported content type. The server accepts
+uncompressed tar and gzip-compressed tar archives only.
+
+Diagnostic:
+
+```sh
+file bundle.tar.gz
+```
+
+Safe fix: send `Content-Type: application/x-tar`, `application/gzip`, or
+`application/x-gzip`, matching the archive format.
+
+## `POST /upload` returns `503`
+
+Likely cause: the in-memory upload queue is full.
+
+Diagnostic:
+
+```sh
+rg -n 'queue_size|max_concurrency' <config-path>
+```
+
+Safe fix: retry after active uploads finish, or increase `server.http.queue_size`
+for the deployment.
+
+## `GET /runs/<run_id>` returns `404`
+
+Likely cause: the run id is wrong, the process restarted, or the completed
+status record expired after `server.http.retention`.
+
+Diagnostic:
+
+```sh
+curl -i http://127.0.0.1:8080/runs/<run-id>
+rg -n 'retention:' <config-path>
+```
+
+Safe fix: use the exact `run_id` returned by `POST /upload`. If status retention
+is too short for operators, increase `server.http.retention`.

 ## `--format: format must be text or json`