Document producer upload client behavior

This commit is contained in:
2026-06-04 14:15:27 +00:00
parent d637949db4
commit f9142fded4
10 changed files with 124 additions and 515 deletions

View File

@@ -106,6 +106,28 @@ Archive entry rules:
The uploaded archive size and extracted bundle size are bounded by the selected pipeline's `source.max_upload_size`. Extracted file count is also bounded by the implementation.
## Go Producer Helper
Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/upload` to build or validate source bundles, package them as gzip-compressed tar archives, and submit them to this API:
```go
client, err := upload.NewClient(upload.ClientOptions{
Endpoint: "http://127.0.0.1:8080",
Token: token,
})
if err != nil {
return err
}
result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
Root: "examples/source-bundle",
IdempotencyKey: "reports.example.20260604T120000Z",
})
```
`Endpoint` is the server base URL; the package derives `/upload` and `/runs/<run-id>`. `UploadBundle` validates a local bundle by default and uploads only `manifest.json` plus manifest-listed files. `UploadFiles` creates a temporary bundle from explicit `bundle.BundleFile` values before uploading. When `IdempotencyKey` is omitted, the package generates one random 128-bit lowercase hex key for the upload operation and reuses it across retries.
The helper retries only safe cases: `503 Service Unavailable`, temporary network errors, and ambiguous mid-upload failures. It does not retry after `202 Accepted` and does not retry `400`, `401`, `409`, `413`, or `415`. Bearer token values are redacted from returned errors.
## Queue And Retention
`server.http.queue_size` bounds accepted-but-not-started uploads plus uploads being staged. `server.http.max_concurrency` bounds publishing concurrency. The coordinator does not run two uploads for the same pipeline concurrently.
@@ -123,5 +145,5 @@ The HTTP API does not expose pipeline selection by request parameter, TLS, publi
Before changing this contract, inspect and run:
```sh
go test ./internal/app ./internal/ingest
go test ./internal/app ./internal/ingest ./pkg/upload
```

View File

@@ -68,6 +68,8 @@ Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to bu
- `LoadManifest`, `ParseManifest`, `ValidateManifest`, and `ValidateBundle`: parse and validate local bundles.
- `FileDigest`, `BundleDigest`, and `ValidateDigest`: digest helpers.
Go producers that submit bundles to `distributor serve` can use `gitea.maximumdirect.net/eric/distributor/pkg/upload`. It builds on `pkg/bundle`, packages valid bundles as gzip-compressed tar uploads, sends bearer authentication, and includes idempotency keys for safe retry behavior. See [HTTP Upload API Contract](http-upload.md).
CLI producers can use:
```sh
@@ -88,5 +90,5 @@ The source bundle manifest does not configure routing, destination selection, pu
Before changing this contract, inspect and run:
```sh
go test ./pkg/bundle ./internal/bundle
go test ./pkg/bundle ./pkg/upload ./internal/bundle
```