eric/weatherreporter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Implement the distributor v0.5 PipelineID update

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-08 07:04:31 -05:00
parent 9bc8156615
commit d71c7e4d28
24 changed files with 264 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
docs/integrations/distributor/pkg-upload.md
View File

@@ -8,7 +8,7 @@ Import path:
import "gitea.maximumdirect.net/eric/distributor/pkg/upload"
```
`pkg/upload` is the producer-facing HTTP upload client. It builds on `pkg/bundle`, packages valid source bundles as gzip-compressed tar archives, sends bearer authentication, includes idempotency keys, and exposes a status polling helper.
`pkg/upload` is the producer-facing HTTP upload client. It builds on `pkg/bundle`, packages valid source bundles as gzip-compressed tar archives, sends bearer authentication, routes uploads to a configured pipeline, includes idempotency keys, and exposes a status polling helper.
`UploadFiles` examples also use:
@@ -30,7 +30,7 @@ if err != nil {
}
```
`Endpoint` is the distributor server base URL. The client derives `/upload` and `/runs/<run-id>`. `Token` is required and is sent as `Authorization: Bearer <token>`. Token values are redacted from client errors.
`Endpoint` is the distributor server base URL. The client derives `/v1/pipelines/<pipeline-id>/upload` and `/runs/<run-id>`. `Token` is required and is sent as `Authorization: Bearer <token>`. Token values are redacted from client errors.
`HTTPClient` and `Retry` are optional. Defaults use a 30 second HTTP timeout and safe retry settings.
@@ -40,8 +40,9 @@ Use `UploadFiles` when the producer has generated output files but has not assem
```go
result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
ID: "weather.hourly.brentwood.2026-06-07T15",
IdempotencyKey: "weather.hourly.brentwood.2026-06-07T15",
PipelineID: "weather-hourly",
ID: "weather.hourly.brentwood",
IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
Files: []bundle.BundleFile{
{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
@@ -53,7 +54,7 @@ if err != nil {
_ = result.RunID
```
`UploadFiles` creates a temporary bundle, writes and validates a manifest, uploads the archive, and removes temporary files when the call returns. It does not write into producer source directories.
`PipelineID` is required and selects the configured distributor workflow for this upload. `ID` is the source manifest id and identifies the logical artifact inside that workflow. `UploadFiles` creates a temporary bundle, writes and validates a manifest, uploads the archive, and removes temporary files when the call returns. It does not write into producer source directories.
## Upload An Existing Bundle
@@ -61,8 +62,9 @@ Use `UploadBundle` when the producer already has a complete local bundle root co
```go
result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
PipelineID: "weather-hourly",
Root: "/var/spool/weather/hourly-2026-06-07T15",
IdempotencyKey: "weather.hourly.brentwood.2026-06-07T15",
IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
})
if err != nil {
return err
@@ -70,7 +72,7 @@ if err != nil {
_ = result.RunID
```
`UploadBundle` validates the local bundle by default and uploads only `manifest.json` plus manifest-listed files. Unlisted files are not uploaded.
`PipelineID` is required for existing bundles too. `UploadBundle` validates the local bundle by default and uploads only `manifest.json` plus manifest-listed files. Unlisted files are not uploaded.
## Result And Status
@@ -94,7 +96,9 @@ Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Co
Every upload request includes `Idempotency-Key`.
If `IdempotencyKey` is omitted, the client generates a random 128-bit lowercase hexadecimal key for that upload operation and reuses it for retries within the same call. For cross-process retry safety, producers should pass a stable key derived from the producer job or report id.
If `IdempotencyKey` is omitted, the client generates a random 128-bit lowercase hexadecimal key for that upload operation and reuses it for retries within the same call. For cross-process retry safety, producers should pass a key derived from the producer run, such as `<bundle-id>.<run-id>`.
Do not reuse the same idempotency key for multiple distinct report generations. Reuse it only when retrying the exact same run with the same token, pipeline id, and source manifest. A repeated key with the same manifest in that scope returns the original accepted run instead of enqueueing another run; a repeated key with different content returns an idempotency conflict.
The client retries only safe cases:
@@ -102,7 +106,7 @@ The client retries only safe cases:
- temporary network errors;
- ambiguous mid-upload failures.
It does not retry after `202 Accepted` and does not retry `400`, `401`, `409`, `413`, or `415`.
It does not retry after `202 Accepted` and does not retry `400`, `401`, `403`, `404`, `409`, `413`, or `415`.
Detect conflicting key reuse with `errors.As`: