eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Stage uploads before accepting HTTP runs

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-04 00:14:32 +00:00
parent 1340418a2b
commit 0d346dcdf5
12 changed files with 248 additions and 162 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
docs/internal/app.md
View File

@@ -87,10 +87,10 @@ prepared.
## Upload Coordination
`UploadCoordinator` owns in-memory coordination for asynchronous upload
processing. It admits uploads for configured `http_upload` pipelines, generates
run IDs, tracks status records, stages accepted archives through
`internal/ingest`, and executes the selected pipeline through
`RunPipelineWithLocalSource`.
processing. It admits uploads for configured `http_upload` pipelines, reserves
queue capacity before request-body staging, stages and validates archives
through `internal/ingest`, tracks accepted status records, and executes the
selected pipeline through `RunPipelineWithLocalSource`.
Upload run IDs use:
@@ -111,10 +111,11 @@ The coordinator records these statuses:
- `expired`
Admission is bounded by `server.http.queue_size`. Full queues are rejected
before the upload body is staged. Execution is bounded by
before the upload body is read. Successfully reserved uploads are staged and
validated before an accepted run record is created. Execution is bounded by
`server.http.max_concurrency`, and only one upload for a given pipeline may run
at a time. Later uploads for the same pipeline remain queued until the active
run finishes.
at a time. Later accepted uploads for the same pipeline remain queued until the
active run finishes.
Completed records retain the final run report or error text until
`server.http.retention` elapses. Expiration removes completed status records and
@@ -137,15 +138,16 @@ pipeline ids, but not token values.
Routes:
- `GET /healthz`: returns `200` after config, secrets, tokens, coordinator, and route setup succeed.
- `POST /upload`: accepts authenticated tar and tar.gz archives and returns an accepted run id.
- `POST /upload`: stages and validates an authenticated tar or tar.gz archive, then returns an accepted run id.
- `GET /runs/<run_id>`: returns the current in-memory upload status record or `404`.
The upload token maps to exactly one configured pipeline. Producers do not
submit pipeline ids, and submitted `pipeline` or `pipeline_id` query values are
rejected. Full queues are rejected before the request body is read. Oversized
uploads, unsupported content types, invalid bearer tokens, full queues, and
unknown status records are mapped to stable HTTP status codes without returning
secret token values.
rejected. Full queues are rejected before the request body is read. Malformed
archives and invalid staged bundles are rejected before a run id is issued.
Oversized uploads, unsupported content types, invalid bearer tokens, full
queues, and unknown status records are mapped to stable HTTP status codes
without returning secret token values.
## Coordination
@@ -201,7 +203,7 @@ Run helpers are grouped by responsibility:
- `run_warnings.go`: secret and SSH warning records.
- `run_notify.go`: notification event projection and action filtering.
- `run_coordinator.go`: in-memory run admission, run IDs, status records, and duplicate-run errors.
- `upload_coordinator.go`: in-memory upload admission, queueing, status tracking, staging handoff, and staged-source execution.
- `upload_coordinator.go`: in-memory upload admission, queue reservation, staging handoff, status tracking, queueing, and staged-source execution.
- `upload_http.go`: HTTP upload authentication, routes, JSON response projection, and HTTP error mapping.
- `serve.go`: config/secrets loading and HTTP server startup.
- `backends.go`: app-level backend factory wiring.

3
docs/internal/ingest.md
View File

@@ -6,6 +6,9 @@
## Archive staging
`ValidateContentType` owns accepted upload content-type policy for archive
staging callers.
`StageArchive` accepts one upload body, content type, pipeline staging path, run id, and explicit size and file-count limits. It writes the request body to temporary storage while enforcing the configured upload size limit, extracts the archive into temporary local storage, validates the extracted source bundle, and then commits the validated bundle to:
```text