`pkg/upload`

Audience: upstream Go producer developers and LLM coding agents submitting bundles to distributor serve.

Import path:

import "gitea.maximumdirect.net/eric/distributor/pkg/upload"

pkg/upload is the producer-facing HTTP upload client. It builds on pkg/bundle, packages valid source bundles as gzip-compressed tar archives, sends bearer authentication, routes uploads to a configured pipeline, includes idempotency keys, and exposes a status polling helper.

UploadFiles examples also use:

import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"

The canonical HTTP wire contract is docs/integrations/http-upload.md in the Distributor repository.

Client Construction

client, err := upload.NewClient(upload.ClientOptions{
	Endpoint: "https://distributor.example.com",
	Token:    token,
})
if err != nil {
	return err
}

Endpoint is the distributor server base URL. The client derives /v1/pipelines/<pipeline-id>/upload and /runs/<run-id>. Token is required and is sent as Authorization: Bearer <token>. Token values are redacted from client errors.

HTTPClient and Retry are optional. Defaults use a 30 second HTTP timeout and safe retry settings.

Upload Producer Files

Use UploadFiles when the producer has generated output files but has not assembled a bundle directory.

result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
	PipelineID:     "weather-hourly",
	ID:             "weather.hourly.brentwood",
	IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
	Files: []bundle.BundleFile{
		{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
		{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
	},
})
if err != nil {
	return err
}
_ = result.RunID

PipelineID is required and selects the configured distributor workflow for this upload. ID is the source manifest id and identifies the logical artifact inside that workflow. UploadFiles creates a temporary bundle, writes and validates a manifest, uploads the archive, and removes temporary files when the call returns. It does not write into producer source directories.

Upload An Existing Bundle

Use UploadBundle when the producer already has a complete local bundle root containing manifest.json.

result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
	PipelineID:     "weather-hourly",
	Root:           "/var/spool/weather/hourly-2026-06-07T15",
	IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
})
if err != nil {
	return err
}
_ = result.RunID

PipelineID is required for existing bundles too. UploadBundle validates the local bundle by default and uploads only manifest.json plus manifest-listed files. Unlisted files are not uploaded.

Result And Status

Upload success means the server returned 202 Accepted after staging and validating the upload. It does not mean all configured destinations have published.

Poll status while the server retains the in-memory run record:

status, err := client.Status(ctx, result.RunID)
if err != nil {
	return err
}
if status.Status == "failed" {
	return fmt.Errorf("distributor run failed: %s", status.Error)
}

Status values are accepted, queued, running, succeeded, and failed. Completed records expire according to server.http.retention; server restart clears run status and idempotency records.

Idempotency And Retry

Every upload request includes Idempotency-Key.

If IdempotencyKey is omitted, the client generates a random 128-bit lowercase hexadecimal key for that upload operation and reuses it for retries within the same call. For cross-process retry safety, producers should pass a key derived from the producer run, such as <bundle-id>.<run-id>.

Do not reuse the same idempotency key for multiple distinct report generations. Reuse it only when retrying the exact same run with the same token, pipeline id, and source manifest. A repeated key with the same manifest in that scope returns the original accepted run instead of enqueueing another run; a repeated key with different content returns an idempotency conflict.

The client retries only safe cases:

503 Service Unavailable;
temporary network errors;
ambiguous mid-upload failures.

It does not retry after 202 Accepted and does not retry 400, 401, 403, 404, 409, 413, or 415.

Detect conflicting key reuse with errors.As:

var conflict *upload.IdempotencyConflictError
if errors.As(err, &conflict) {
	return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
}

Boundaries

pkg/upload does not configure server pipelines, choose destinations, wait for publication completion automatically, persist client queues, provide durable idempotency across server restarts, or expose destination state. It submits complete source bundles to the configured HTTP upload API.

4.8 KiB Raw Blame History

pkg/upload