Add async upload coordination
This commit is contained in:
@@ -79,6 +79,43 @@ destination planning and execution loop. Destination code receives the normal
|
||||
storage backend and bundle values and does not depend on how the source root was
|
||||
prepared.
|
||||
|
||||
## Upload Coordination
|
||||
|
||||
`UploadCoordinator` owns in-memory coordination for asynchronous upload
|
||||
processing. It admits uploads for configured `http_upload` pipelines, generates
|
||||
run IDs, tracks status records, stages accepted archives through
|
||||
`internal/ingest`, and executes the selected pipeline through
|
||||
`RunPipelineWithLocalSource`.
|
||||
|
||||
Upload run IDs use:
|
||||
|
||||
```text
|
||||
<pipeline id>.<UTC timestamp>.<random suffix>
|
||||
```
|
||||
|
||||
The timestamp uses `YYYYMMDDThhmmssZ` UTC format and the suffix is filesystem
|
||||
safe.
|
||||
|
||||
The coordinator records these statuses:
|
||||
|
||||
- `accepted`
|
||||
- `queued`
|
||||
- `running`
|
||||
- `succeeded`
|
||||
- `failed`
|
||||
- `expired`
|
||||
|
||||
Admission is bounded by `server.http.queue_size`. Full queues are rejected
|
||||
before the upload body is staged. Execution is bounded by
|
||||
`server.http.max_concurrency`, and only one upload for a given pipeline may run
|
||||
at a time. Later uploads for the same pipeline remain queued until the active
|
||||
run finishes.
|
||||
|
||||
Completed records retain the final run report or error text until
|
||||
`server.http.retention` elapses. Expiration removes completed status records and
|
||||
their committed staged bundle directories. The coordinator is memory-only and
|
||||
does not persist queue state, status records, or run reports.
|
||||
|
||||
## Coordination
|
||||
|
||||
`PipelineRunCoordinator` wraps `RunPipeline` with in-memory admission control.
|
||||
@@ -133,6 +170,7 @@ Run helpers are grouped by responsibility:
|
||||
- `run_warnings.go`: secret and SSH warning records.
|
||||
- `run_notify.go`: notification event projection and action filtering.
|
||||
- `run_coordinator.go`: in-memory run admission, run IDs, status records, and duplicate-run errors.
|
||||
- `upload_coordinator.go`: in-memory upload admission, queueing, status tracking, staging handoff, and staged-source execution.
|
||||
- `backends.go`: app-level backend factory wiring.
|
||||
- `transforms.go`: app-level transform registry wiring.
|
||||
- `source_select.go`: configured-source selection shared by `validate` and `inspect`.
|
||||
|
||||
Reference in New Issue
Block a user