Add HTTP upload server and serve command

This commit is contained in:
2026-06-03 15:22:52 +00:00
parent f0c10210eb
commit 6beef58dbf
12 changed files with 942 additions and 10 deletions

View File

@@ -34,6 +34,11 @@ same destination fan-out path as normal runs.
source. They share source backend construction with run workflows and never open
destination backends.
`Serve` is the CLI-facing HTTP upload server entrypoint. It loads config,
loads the configured secrets directory, resolves upload bearer tokens for
configured `http_upload` sources, creates an `UploadCoordinator`, binds
`server.http.bind`, and serves the upload API until its context is cancelled.
## Run Reports
`RunReport` is the structured result model for run workflows. It includes
@@ -116,6 +121,32 @@ Completed records retain the final run report or error text until
their committed staged bundle directories. The coordinator is memory-only and
does not persist queue state, status records, or run reports.
## HTTP Upload Server
The HTTP upload server is app-layer transport wiring around
`UploadCoordinator`. It owns request authentication, route dispatch, HTTP status
mapping, and JSON response projection. Bundle staging and publication remain in
the coordinator and staged-source run path.
Server startup resolves every configured `http_upload` source `token_env`
through the config-owned environment resolver after `secrets.directory` has
been loaded. Startup fails when a token is missing, empty, or duplicates another
upload pipeline token. Error messages identify environment variable names and
pipeline ids, but not token values.
Routes:
- `GET /healthz`: returns `200` after config, secrets, tokens, coordinator, and route setup succeed.
- `POST /upload`: accepts authenticated tar and tar.gz archives and returns an accepted run id.
- `GET /runs/<run_id>`: returns the current in-memory upload status record or `404`.
The upload token maps to exactly one configured pipeline. Producers do not
submit pipeline ids, and submitted `pipeline` or `pipeline_id` query values are
rejected. Full queues are rejected before the request body is read. Oversized
uploads, unsupported content types, invalid bearer tokens, full queues, and
unknown status records are mapped to stable HTTP status codes without returning
secret token values.
## Coordination
`PipelineRunCoordinator` wraps `RunPipeline` with in-memory admission control.
@@ -171,6 +202,8 @@ Run helpers are grouped by responsibility:
- `run_notify.go`: notification event projection and action filtering.
- `run_coordinator.go`: in-memory run admission, run IDs, status records, and duplicate-run errors.
- `upload_coordinator.go`: in-memory upload admission, queueing, status tracking, staging handoff, and staged-source execution.
- `upload_http.go`: HTTP upload authentication, routes, JSON response projection, and HTTP error mapping.
- `serve.go`: config/secrets loading and HTTP server startup.
- `backends.go`: app-level backend factory wiring.
- `transforms.go`: app-level transform registry wiring.
- `source_select.go`: configured-source selection shared by `validate` and `inspect`.