distributor/docs/integrations/http-upload.md

HTTP Upload API Contract

Audience: producers, operators, and maintainers integrating with distributor serve.

distributor serve exposes a local HTTP upload API for pipelines whose source backend is http_upload. Bearer tokens authenticate producers, and the upload path selects the configured pipeline. The selected token must be allowed for the requested pipeline.

Authentication

Uploads authenticate with:

Authorization: Bearer <token>

Token values are resolved from top-level upload_tokens records through the process environment or secrets.directory. Tokens are not configured as YAML literal values.

Requests that include pipeline or pipeline_id query parameters are rejected. Use the pipeline id in the upload path.

Endpoints

GET /healthz

Returns 200 OK when the server is running:

{"status":"ok"}

POST /v1/pipelines/{pipeline_id}/upload

Accepts one source bundle archive and returns after the archive is staged and validated.

Producers may include:

Idempotency-Key: <key>

pipeline_id must name a configured pipeline whose source backend is http_upload, and the authenticated token must allow that pipeline. Idempotency keys are scoped to token id, pipeline id, and key. Valid keys are non-empty ASCII strings up to 128 bytes using letters, digits, ., _, -, and :. Invalid keys return 400.

Accepted content types:

• application/x-tar
• application/gzip
• application/x-gzip

Successful admission returns 202 Accepted:

{"run_id":"reports.20260604T120000Z.abcdef12","status":"accepted"}

Common error responses:

• 400: pipeline query supplied, invalid idempotency key, archive rejected, malformed archive, or invalid staged source bundle.
• 401: missing, empty, or unknown bearer token.
• 403: bearer token is valid but is not allowed for the requested pipeline.
• 404: upload path is unknown or the requested upload pipeline is not configured.
• 409: repeated idempotency key conflicts with another source manifest, or the same key is already being staged.
• 413: upload body exceeds the selected pipeline size limit.
• 415: unsupported content type.
• 503: upload queue is full.

Error bodies use:

{"error":"<message>"}

Retryable idempotency conflicts include:

{"error":"upload idempotency key is already being processed","retryable":true}

When Idempotency-Key is omitted, upload admission preserves the raw HTTP behavior: every valid accepted upload receives its own run id. When a key is supplied, the server records the accepted run after archive staging and source bundle validation succeed. Reusing the same key for the same token id, pipeline id, and normalized source manifest returns the original 202 Accepted response and does not enqueue another run. Reusing the same key for a different normalized source manifest within that scope returns 409 Conflict. Producers should use a fresh key for each distinct producer run and reuse a key only for retries of that same run.

GET /runs/<run-id>

Returns an in-memory status record while retained:

{
  "run_id": "reports.20260604T120000Z.abcdef12",
  "pipeline_id": "reports",
  "status": "succeeded",
  "accepted_at": "2026-06-04T12:00:00Z",
  "started_at": "2026-06-04T12:00:01Z",
  "finished_at": "2026-06-04T12:00:02Z",
  "report": {}
}

Status values are accepted, queued, running, succeeded, and failed. Failed records include error. Succeeded and failed records may include a run report.

Unknown, malformed, expired, or process-lost run ids return 404.

Archive Contract

Upload archives must be uncompressed tar or gzip-compressed tar. The archive must contain exactly one root-level manifest.json and all manifest-listed files.

Archive entry rules:

• Paths must be clean relative slash-separated paths.
• Absolute paths, backslashes, . and .. segments, duplicate files, and nested manifest.json entries are rejected.
• Only directories and regular files are accepted.
• Symlinks, hardlinks, devices, FIFOs, sockets, and other entry types are rejected.

The uploaded archive size and extracted bundle size are bounded by the selected pipeline's source.max_upload_size. Extracted file count is also bounded by the implementation.

Go Producer Helper

Go producers can use gitea.maximumdirect.net/eric/distributor/pkg/upload to build or validate source bundles, package them as gzip-compressed tar archives, and submit them to this API. See Upstream Producer Integration for the copyable upstream implementation guide and pkg/upload for package-specific workflow guidance.

client, err := upload.NewClient(upload.ClientOptions{
	Endpoint: "http://127.0.0.1:8080",
	Token:    token,
})
if err != nil {
	return err
}
result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
	PipelineID:     "reports",
	Root:           "examples/source-bundle",
	IdempotencyKey: "reports.example.20260604T120000Z",
})

Endpoint is the server base URL; the package derives /v1/pipelines/<pipeline-id>/upload and /runs/<run-id>. PipelineID is required and selects the configured distributor workflow. UploadBundle validates a local bundle by default and uploads only manifest.json plus manifest-listed files. UploadFiles creates a temporary bundle from explicit bundle.BundleFile values before uploading. When IdempotencyKey is omitted, the package generates one random 128-bit lowercase hex key for the upload operation and reuses it across retries.

The helper retries only safe cases: 503 Service Unavailable, temporary network errors, and ambiguous mid-upload failures. It does not retry after 202 Accepted and does not retry 400, 401, 403, 404, 409, 413, or 415. Bearer token values are redacted from returned errors.

Queue And Retention

server.http.queue_size bounds accepted-but-not-started uploads plus uploads being staged. server.http.max_concurrency bounds publishing concurrency. The coordinator does not run two uploads for the same pipeline concurrently.

Completed status records expire after server.http.retention; expiration removes committed staged bundle directories for completed uploads. Server restart clears queue state and status records.

Idempotency records are memory-only, expire with completed upload status records, and are cleared by server restart.

Boundaries

The HTTP API does not expose pipeline selection by query parameter, TLS, public routing policy, or durable status storage. Put public access controls, TLS termination, and rate limiting in deployment infrastructure.

Tests

Before changing this contract, inspect and run:

go test ./internal/app ./internal/ingest ./pkg/upload