distributor/docs/consumers/api.md

Upstream Producer Integration

Audience: developers and LLM coding agents adding distributor support to an upstream Go producer application.

This document is the copyable implementation guide for submitting producer outputs to a distributor pipeline whose source backend is http_upload.

Required Inputs

The upstream application needs these values from deployment or operator configuration:

• distributor endpoint: the HTTP server base URL, such as https://distributor.example.com;
• upload token: bearer token for exactly one configured http_upload pipeline;
• generated files: regular local files to include in the source bundle;
• bundle id: stable identifier for this producer output;
• idempotency key: stable key for retrying the same producer operation.

Do not put destination routing, public URLs, transform settings, or credentials in the source manifest. Those belong in the distributor pipeline configuration.

Recommended Workflow

Use gitea.maximumdirect.net/eric/distributor/pkg/upload.

For most producers, use UploadFiles. It accepts producer-generated files, builds a temporary valid source bundle with pkg/bundle, uploads a gzip-compressed tar archive, and removes temporary files when the call returns.

Use UploadBundle only when the producer already assembled a complete bundle directory containing manifest.json.

Add the dependency from the upstream application:

go get gitea.maximumdirect.net/eric/distributor

Minimal Go Example

package reports

import (
	"context"
	"errors"
	"fmt"
	"os"
	"time"

	"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
	"gitea.maximumdirect.net/eric/distributor/pkg/upload"
)

func SubmitReport(reportPath, summaryPath string) error {
	endpoint := os.Getenv("DISTRIBUTOR_UPLOAD_ENDPOINT")
	token := os.Getenv("DISTRIBUTOR_UPLOAD_TOKEN")
	if endpoint == "" || token == "" {
		return fmt.Errorf("distributor endpoint and token are required")
	}

	reportID := "weather.hourly.brentwood.2026-06-07T15"
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()

	client, err := upload.NewClient(upload.ClientOptions{
		Endpoint: endpoint,
		Token:    token,
	})
	if err != nil {
		return err
	}

	result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
		ID:             reportID,
		IdempotencyKey: reportID,
		Files: []bundle.BundleFile{
			{SourcePath: reportPath, Path: "report.md"},
			{SourcePath: summaryPath, Path: "summary.txt"},
		},
	})
	if err != nil {
		var conflict *upload.IdempotencyConflictError
		if errors.As(err, &conflict) {
			return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
		}
		return err
	}

	fmt.Printf("distributor accepted run %s\n", result.RunID)
	return nil
}

Producer Responsibilities

• Use a stable bundle id for the producer output, such as a report type plus logical timestamp.
• Use a stable idempotency key for cross-process retries of the same producer operation.
• Map each generated file to a clean slash-separated bundle path, such as report.md or assets/chart.png.
• Include only regular files. Symlinks, directories as files, devices, FIFOs, and sockets are rejected.
• Keep file contents stable after upload inputs are selected. Bundle digests are calculated from file bytes.
• Treat upload success as admission only. UploadFiles and UploadBundle return after the server accepts and validates the upload, not after all destinations publish.

Valid bundle paths are relative slash paths. They must not be empty, absolute, contain backslashes, contain . or .. path segments, contain empty path segments, or use reserved basenames manifest.json or .distributor.json.

Idempotency And Status

pkg/upload sends Idempotency-Key on every upload. If the caller omits one, the package generates a random key for that call and reuses it for in-process retries. That is enough for transient network retry within one process.

For producer jobs that may retry after process restart, supply a stable key derived from the producer operation, such as the report id or job id. Reusing the same key with the same normalized source manifest returns the original accepted run. Reusing the same key with different source content returns a conflict.

Status polls /runs/<run-id> while the distributor server retains the in-memory status record. Status values are accepted, queued, running, succeeded, and failed. Completed records expire according to the server's server.http.retention setting, and server restart clears status and idempotency records.

Optional status check:

status, err := client.Status(ctx, result.RunID)
if err != nil {
	return err
}
if status.Status == "failed" {
	return fmt.Errorf("distributor run failed: %s", status.Error)
}

References

In the distributor source tree:

• docs/consumers/pkg-upload.md: Go upload package workflow.
• docs/consumers/pkg-bundle.md: Go bundle package workflow.
• docs/integrations/http-upload.md: canonical HTTP upload wire contract.
• docs/integrations/source-bundle.md: canonical source bundle file-format contract.