Add HTTP upload server and serve command
This commit is contained in:
@@ -34,6 +34,11 @@ same destination fan-out path as normal runs.
|
||||
source. They share source backend construction with run workflows and never open
|
||||
destination backends.
|
||||
|
||||
`Serve` is the CLI-facing HTTP upload server entrypoint. It loads config,
|
||||
loads the configured secrets directory, resolves upload bearer tokens for
|
||||
configured `http_upload` sources, creates an `UploadCoordinator`, binds
|
||||
`server.http.bind`, and serves the upload API until its context is cancelled.
|
||||
|
||||
## Run Reports
|
||||
|
||||
`RunReport` is the structured result model for run workflows. It includes
|
||||
@@ -116,6 +121,32 @@ Completed records retain the final run report or error text until
|
||||
their committed staged bundle directories. The coordinator is memory-only and
|
||||
does not persist queue state, status records, or run reports.
|
||||
|
||||
## HTTP Upload Server
|
||||
|
||||
The HTTP upload server is app-layer transport wiring around
|
||||
`UploadCoordinator`. It owns request authentication, route dispatch, HTTP status
|
||||
mapping, and JSON response projection. Bundle staging and publication remain in
|
||||
the coordinator and staged-source run path.
|
||||
|
||||
Server startup resolves every configured `http_upload` source `token_env`
|
||||
through the config-owned environment resolver after `secrets.directory` has
|
||||
been loaded. Startup fails when a token is missing, empty, or duplicates another
|
||||
upload pipeline token. Error messages identify environment variable names and
|
||||
pipeline ids, but not token values.
|
||||
|
||||
Routes:
|
||||
|
||||
- `GET /healthz`: returns `200` after config, secrets, tokens, coordinator, and route setup succeed.
|
||||
- `POST /upload`: accepts authenticated tar and tar.gz archives and returns an accepted run id.
|
||||
- `GET /runs/<run_id>`: returns the current in-memory upload status record or `404`.
|
||||
|
||||
The upload token maps to exactly one configured pipeline. Producers do not
|
||||
submit pipeline ids, and submitted `pipeline` or `pipeline_id` query values are
|
||||
rejected. Full queues are rejected before the request body is read. Oversized
|
||||
uploads, unsupported content types, invalid bearer tokens, full queues, and
|
||||
unknown status records are mapped to stable HTTP status codes without returning
|
||||
secret token values.
|
||||
|
||||
## Coordination
|
||||
|
||||
`PipelineRunCoordinator` wraps `RunPipeline` with in-memory admission control.
|
||||
@@ -171,6 +202,8 @@ Run helpers are grouped by responsibility:
|
||||
- `run_notify.go`: notification event projection and action filtering.
|
||||
- `run_coordinator.go`: in-memory run admission, run IDs, status records, and duplicate-run errors.
|
||||
- `upload_coordinator.go`: in-memory upload admission, queueing, status tracking, staging handoff, and staged-source execution.
|
||||
- `upload_http.go`: HTTP upload authentication, routes, JSON response projection, and HTTP error mapping.
|
||||
- `serve.go`: config/secrets loading and HTTP server startup.
|
||||
- `backends.go`: app-level backend factory wiring.
|
||||
- `transforms.go`: app-level transform registry wiring.
|
||||
- `source_select.go`: configured-source selection shared by `validate` and `inspect`.
|
||||
|
||||
Reference in New Issue
Block a user