eric/weatherreporter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Refresh Distributor integration guides

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-31 01:17:37 +00:00
parent ff2e664c62
commit 1130d807dc
3 changed files with 119 additions and 313 deletions
Download Patch File Download Diff File Expand all files Collapse all files

150
docs/integrations/distributor/pkg-upload.md
View File

@@ -1,123 +1,51 @@
# `pkg/upload`
# Distributor Upload Client Contract
Audience: upstream Go producer developers and LLM coding agents submitting bundles to `distributor serve`.
Weatherreporter uses `gitea.maximumdirect.net/eric/distributor/pkg/upload` at
the pinned module version `v0.5.0`. It constructs one client per notification
attempt and calls `UploadFiles`, followed by `Status` for the accepted run.
Import path:
## Client And Upload
```go
import "gitea.maximumdirect.net/eric/distributor/pkg/upload"
```
The adapter constructs the client with the configured endpoint, bearer token,
and an HTTP client whose timeout is the configured Distributor timeout. It
passes no custom retry options, so the pinned client's defaults apply: three
attempts, 100 ms base delay, and one-second maximum delay.
`pkg/upload` is the producer-facing HTTP upload client. It builds on `pkg/bundle`, packages valid source bundles as gzip-compressed tar archives, sends bearer authentication, routes uploads to a configured pipeline, includes idempotency keys, and exposes a status polling helper.
For each notification, Weatherreporter calls `UploadFiles` with:
`UploadFiles` examples also use:
- the rendered pipeline ID;
- the rendered bundle ID as the source manifest ID;
- the report or batch generation time as `Created`;
- the managed-report-to-bundle-path mappings described in the
[bundle mapping contract](pkg-bundle.md); and
- a rendered idempotency key.
```go
import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
```
It leaves bundle validation enabled. `UploadFiles` creates the temporary source
bundle and sends it as a gzip-compressed tar archive; Weatherreporter does not
call `UploadBundle` or submit prebuilt bundle roots.
The canonical HTTP wire contract is `docs/integrations/http-upload.md` in the
Distributor repository.
## Retry, Conflict, And Status
## Client Construction
The pinned upload client retries only `503 Service Unavailable` and retryable
network failures. It does not retry successful `202` responses or other HTTP
errors. Because every Weatherreporter request supplies an idempotency key, a
retry keeps the same upload identity.
```go
client, err := upload.NewClient(upload.ClientOptions{
Endpoint: "https://distributor.example.com",
Token: token,
})
if err != nil {
return err
}
```
The client decodes the accepted upload result (`run_id`, `status`) and the run
status record. A `409` response is an upstream idempotency conflict; the
adapter translates it to its own conflict error without exposing the token.
`Endpoint` is the distributor server base URL. The client derives `/v1/pipelines/<pipeline-id>/upload` and `/runs/<run-id>`. `Token` is required and is sent as `Authorization: Bearer <token>`. Token values are redacted from client errors.
The adapter then calls `Status` for the accepted run. A terminal `failed`
status is a notification failure. A status lookup failure or a timeout before a
terminal status remains attached to the otherwise accepted upload as diagnostic
status information. Polling cadence, final failure handling, redaction, and
notification artifact persistence are internal behavior documented in the
[Distributor adapter](../../internal/distributor-adapter.md) and
[application orchestration](../../internal/app-orchestration.md).
`HTTPClient` and `Retry` are optional. Defaults use a 30 second HTTP timeout and safe retry settings.
## Compatibility Reference
## Upload Producer Files
Use `UploadFiles` when the producer has generated output files but has not assembled a bundle directory.
```go
result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
PipelineID: "weather-hourly",
ID: "weather.hourly.brentwood",
IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
Files: []bundle.BundleFile{
{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
},
})
if err != nil {
return err
}
_ = result.RunID
```
`PipelineID` is required and selects the configured distributor workflow for this upload. `ID` is the source manifest id and identifies the logical artifact inside that workflow. `UploadFiles` creates a temporary bundle, writes and validates a manifest, uploads the archive, and removes temporary files when the call returns. It does not write into producer source directories.
## Upload An Existing Bundle
Use `UploadBundle` when the producer already has a complete local bundle root containing `manifest.json`.
```go
result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
PipelineID: "weather-hourly",
Root: "/var/spool/weather/hourly-2026-06-07T15",
IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
})
if err != nil {
return err
}
_ = result.RunID
```
`PipelineID` is required for existing bundles too. `UploadBundle` validates the local bundle by default and uploads only `manifest.json` plus manifest-listed files. Unlisted files are not uploaded.
## Result And Status
Upload success means the server returned `202 Accepted` after staging and validating the upload. It does not mean all configured destinations have published.
Poll status while the server retains the in-memory run record:
```go
status, err := client.Status(ctx, result.RunID)
if err != nil {
return err
}
if status.Status == "failed" {
return fmt.Errorf("distributor run failed: %s", status.Error)
}
```
Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to `server.http.retention`; server restart clears run status and idempotency records.
## Idempotency And Retry
Every upload request includes `Idempotency-Key`.
If `IdempotencyKey` is omitted, the client generates a random 128-bit lowercase hexadecimal key for that upload operation and reuses it for retries within the same call. For cross-process retry safety, producers should pass a key derived from the producer run, such as `<bundle-id>.<run-id>`.
Do not reuse the same idempotency key for multiple distinct report generations. Reuse it only when retrying the exact same run with the same token, pipeline id, and source manifest. A repeated key with the same manifest in that scope returns the original accepted run instead of enqueueing another run; a repeated key with different content returns an idempotency conflict.
The client retries only safe cases:
- `503 Service Unavailable`;
- temporary network errors;
- ambiguous mid-upload failures.
It does not retry after `202 Accepted` and does not retry `400`, `401`, `403`, `404`, `409`, `413`, or `415`.
Detect conflicting key reuse with `errors.As`:
```go
var conflict *upload.IdempotencyConflictError
if errors.As(err, &conflict) {
return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
}
```
## Boundaries
`pkg/upload` does not configure server pipelines, choose destinations, wait for publication completion automatically, persist client queues, provide durable idempotency across server restarts, or expose destination state. It submits complete source bundles to the configured HTTP upload API.
The upstream package workflow is documented in
`docs/consumers/pkg-upload.md` in the Distributor repository. Weatherreporter
uses only the client construction, `UploadFiles`, retry/conflict behavior, and
`Status` operations described here.