Stage uploads before accepting HTTP runs
This commit is contained in:
@@ -87,10 +87,10 @@ prepared.
|
||||
## Upload Coordination
|
||||
|
||||
`UploadCoordinator` owns in-memory coordination for asynchronous upload
|
||||
processing. It admits uploads for configured `http_upload` pipelines, generates
|
||||
run IDs, tracks status records, stages accepted archives through
|
||||
`internal/ingest`, and executes the selected pipeline through
|
||||
`RunPipelineWithLocalSource`.
|
||||
processing. It admits uploads for configured `http_upload` pipelines, reserves
|
||||
queue capacity before request-body staging, stages and validates archives
|
||||
through `internal/ingest`, tracks accepted status records, and executes the
|
||||
selected pipeline through `RunPipelineWithLocalSource`.
|
||||
|
||||
Upload run IDs use:
|
||||
|
||||
@@ -111,10 +111,11 @@ The coordinator records these statuses:
|
||||
- `expired`
|
||||
|
||||
Admission is bounded by `server.http.queue_size`. Full queues are rejected
|
||||
before the upload body is staged. Execution is bounded by
|
||||
before the upload body is read. Successfully reserved uploads are staged and
|
||||
validated before an accepted run record is created. Execution is bounded by
|
||||
`server.http.max_concurrency`, and only one upload for a given pipeline may run
|
||||
at a time. Later uploads for the same pipeline remain queued until the active
|
||||
run finishes.
|
||||
at a time. Later accepted uploads for the same pipeline remain queued until the
|
||||
active run finishes.
|
||||
|
||||
Completed records retain the final run report or error text until
|
||||
`server.http.retention` elapses. Expiration removes completed status records and
|
||||
@@ -137,15 +138,16 @@ pipeline ids, but not token values.
|
||||
Routes:
|
||||
|
||||
- `GET /healthz`: returns `200` after config, secrets, tokens, coordinator, and route setup succeed.
|
||||
- `POST /upload`: accepts authenticated tar and tar.gz archives and returns an accepted run id.
|
||||
- `POST /upload`: stages and validates an authenticated tar or tar.gz archive, then returns an accepted run id.
|
||||
- `GET /runs/<run_id>`: returns the current in-memory upload status record or `404`.
|
||||
|
||||
The upload token maps to exactly one configured pipeline. Producers do not
|
||||
submit pipeline ids, and submitted `pipeline` or `pipeline_id` query values are
|
||||
rejected. Full queues are rejected before the request body is read. Oversized
|
||||
uploads, unsupported content types, invalid bearer tokens, full queues, and
|
||||
unknown status records are mapped to stable HTTP status codes without returning
|
||||
secret token values.
|
||||
rejected. Full queues are rejected before the request body is read. Malformed
|
||||
archives and invalid staged bundles are rejected before a run id is issued.
|
||||
Oversized uploads, unsupported content types, invalid bearer tokens, full
|
||||
queues, and unknown status records are mapped to stable HTTP status codes
|
||||
without returning secret token values.
|
||||
|
||||
## Coordination
|
||||
|
||||
@@ -201,7 +203,7 @@ Run helpers are grouped by responsibility:
|
||||
- `run_warnings.go`: secret and SSH warning records.
|
||||
- `run_notify.go`: notification event projection and action filtering.
|
||||
- `run_coordinator.go`: in-memory run admission, run IDs, status records, and duplicate-run errors.
|
||||
- `upload_coordinator.go`: in-memory upload admission, queueing, status tracking, staging handoff, and staged-source execution.
|
||||
- `upload_coordinator.go`: in-memory upload admission, queue reservation, staging handoff, status tracking, queueing, and staged-source execution.
|
||||
- `upload_http.go`: HTTP upload authentication, routes, JSON response projection, and HTTP error mapping.
|
||||
- `serve.go`: config/secrets loading and HTTP server startup.
|
||||
- `backends.go`: app-level backend factory wiring.
|
||||
|
||||
Reference in New Issue
Block a user