119 lines
4.1 KiB
Markdown
119 lines
4.1 KiB
Markdown
# `pkg/upload`
|
|
|
|
Audience: upstream Go producer developers and LLM coding agents submitting bundles to `distributor serve`.
|
|
|
|
Import path:
|
|
|
|
```go
|
|
import "gitea.maximumdirect.net/eric/distributor/pkg/upload"
|
|
```
|
|
|
|
`pkg/upload` is the producer-facing HTTP upload client. It builds on `pkg/bundle`, packages valid source bundles as gzip-compressed tar archives, sends bearer authentication, includes idempotency keys, and exposes a status polling helper.
|
|
|
|
`UploadFiles` examples also use:
|
|
|
|
```go
|
|
import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
|
|
```
|
|
|
|
The canonical HTTP wire contract is [HTTP Upload API Contract](../integrations/http-upload.md).
|
|
|
|
## Client Construction
|
|
|
|
```go
|
|
client, err := upload.NewClient(upload.ClientOptions{
|
|
Endpoint: "https://distributor.example.com",
|
|
Token: token,
|
|
})
|
|
if err != nil {
|
|
return err
|
|
}
|
|
```
|
|
|
|
`Endpoint` is the distributor server base URL. The client derives `/upload` and `/runs/<run-id>`. `Token` is required and is sent as `Authorization: Bearer <token>`. Token values are redacted from client errors.
|
|
|
|
`HTTPClient` and `Retry` are optional. Defaults use a 30 second HTTP timeout and safe retry settings.
|
|
|
|
## Upload Producer Files
|
|
|
|
Use `UploadFiles` when the producer has generated output files but has not assembled a bundle directory.
|
|
|
|
```go
|
|
result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
|
|
ID: "weather.hourly.brentwood.2026-06-07T15",
|
|
IdempotencyKey: "weather.hourly.brentwood.2026-06-07T15",
|
|
Files: []bundle.BundleFile{
|
|
{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
|
|
{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
|
|
},
|
|
})
|
|
if err != nil {
|
|
return err
|
|
}
|
|
_ = result.RunID
|
|
```
|
|
|
|
`UploadFiles` creates a temporary bundle, writes and validates a manifest, uploads the archive, and removes temporary files when the call returns. It does not write into producer source directories.
|
|
|
|
## Upload An Existing Bundle
|
|
|
|
Use `UploadBundle` when the producer already has a complete local bundle root containing `manifest.json`.
|
|
|
|
```go
|
|
result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
|
|
Root: "/var/spool/weather/hourly-2026-06-07T15",
|
|
IdempotencyKey: "weather.hourly.brentwood.2026-06-07T15",
|
|
})
|
|
if err != nil {
|
|
return err
|
|
}
|
|
_ = result.RunID
|
|
```
|
|
|
|
`UploadBundle` validates the local bundle by default and uploads only `manifest.json` plus manifest-listed files. Unlisted files are not uploaded.
|
|
|
|
## Result And Status
|
|
|
|
Upload success means the server returned `202 Accepted` after staging and validating the upload. It does not mean all configured destinations have published.
|
|
|
|
Poll status while the server retains the in-memory run record:
|
|
|
|
```go
|
|
status, err := client.Status(ctx, result.RunID)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if status.Status == "failed" {
|
|
return fmt.Errorf("distributor run failed: %s", status.Error)
|
|
}
|
|
```
|
|
|
|
Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to `server.http.retention`; server restart clears run status and idempotency records.
|
|
|
|
## Idempotency And Retry
|
|
|
|
Every upload request includes `Idempotency-Key`.
|
|
|
|
If `IdempotencyKey` is omitted, the client generates a random 128-bit lowercase hexadecimal key for that upload operation and reuses it for retries within the same call. For cross-process retry safety, producers should pass a stable key derived from the producer job or report id.
|
|
|
|
The client retries only safe cases:
|
|
|
|
- `503 Service Unavailable`;
|
|
- temporary network errors;
|
|
- ambiguous mid-upload failures.
|
|
|
|
It does not retry after `202 Accepted` and does not retry `400`, `401`, `409`, `413`, or `415`.
|
|
|
|
Detect conflicting key reuse with `errors.As`:
|
|
|
|
```go
|
|
var conflict *upload.IdempotencyConflictError
|
|
if errors.As(err, &conflict) {
|
|
return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
|
|
}
|
|
```
|
|
|
|
## Boundaries
|
|
|
|
`pkg/upload` does not configure server pipelines, choose destinations, wait for publication completion automatically, persist client queues, provide durable idempotency across server restarts, or expose destination state. It submits complete source bundles to the configured HTTP upload API.
|