Add async upload coordination

This commit is contained in:
2026-06-03 15:15:22 +00:00
parent f9436a7423
commit f0c10210eb
3 changed files with 793 additions and 0 deletions

View File

@@ -79,6 +79,43 @@ destination planning and execution loop. Destination code receives the normal
storage backend and bundle values and does not depend on how the source root was
prepared.
## Upload Coordination
`UploadCoordinator` owns in-memory coordination for asynchronous upload
processing. It admits uploads for configured `http_upload` pipelines, generates
run IDs, tracks status records, stages accepted archives through
`internal/ingest`, and executes the selected pipeline through
`RunPipelineWithLocalSource`.
Upload run IDs use:
```text
<pipeline id>.<UTC timestamp>.<random suffix>
```
The timestamp uses `YYYYMMDDThhmmssZ` UTC format and the suffix is filesystem
safe.
The coordinator records these statuses:
- `accepted`
- `queued`
- `running`
- `succeeded`
- `failed`
- `expired`
Admission is bounded by `server.http.queue_size`. Full queues are rejected
before the upload body is staged. Execution is bounded by
`server.http.max_concurrency`, and only one upload for a given pipeline may run
at a time. Later uploads for the same pipeline remain queued until the active
run finishes.
Completed records retain the final run report or error text until
`server.http.retention` elapses. Expiration removes completed status records and
their committed staged bundle directories. The coordinator is memory-only and
does not persist queue state, status records, or run reports.
## Coordination
`PipelineRunCoordinator` wraps `RunPipeline` with in-memory admission control.
@@ -133,6 +170,7 @@ Run helpers are grouped by responsibility:
- `run_warnings.go`: secret and SSH warning records.
- `run_notify.go`: notification event projection and action filtering.
- `run_coordinator.go`: in-memory run admission, run IDs, status records, and duplicate-run errors.
- `upload_coordinator.go`: in-memory upload admission, queueing, status tracking, staging handoff, and staged-source execution.
- `backends.go`: app-level backend factory wiring.
- `transforms.go`: app-level transform registry wiring.
- `source_select.go`: configured-source selection shared by `validate` and `inspect`.