distributor/docs/roadmap/producer.md

Roadmap: Producer HTTP Upload Package

Purpose

Add a second public producer-facing package that lets Go producer applications build or validate a compliant source bundle, package it as a gzip-compressed tar archive, and submit it to the HTTP upload API with safe retry support.

The current public producer package, pkg/bundle, owns source manifest semantics, digest calculation, path validation, local manifest building, local bundle writing, and local bundle validation. The new package must build on that contract instead of reimplementing it.

This roadmap also adds server-side producer idempotency keys to the HTTP upload API. Idempotency is required for the producer upload package's retry behavior: the client can safely retry an upload with the same key, and the server can collapse duplicate accepted uploads into the original run.

Goals

• Make the common Go producer workflow small and hard to misuse.
• Reuse pkg/bundle for manifest generation, path normalization, SHA-256 calculation, digest calculation, and local validation.
• Create upload archives that match the server's source bundle archive contract.
• Add server-side idempotency records scoped to the authenticated pipeline.
• Send idempotency keys from the public upload package by default.
• Handle bearer authentication without logging or returning token values.
• Parse successful, duplicate, conflict, and error responses into typed producer-side results.
• Retry safely using idempotency keys and bounded backoff.
• Keep the package dependency-light and usable from ordinary Go producer applications.

Non-Goals

• Do not expose internal/app, internal/ingest, storage backends, server config, or destination state types through the public package.
• Do not add durable idempotency storage, durable client queues, background workers, or database-backed retry processing.
• Do not add zstd, multipart upload, resumable upload, or non-tar archive formats.
• Do not require producers to know distributor pipeline ids; server-side token mapping remains authoritative.
• Do not make the package a replacement for the existing CLI or server API documentation.

Implementation Sequence

Implement this feature in three stages:

1. Server-side HTTP idempotency keys.
2. Public pkg/upload client package.
3. Current-behavior documentation and examples.

Server idempotency should land first so the public upload package can rely on the final retry contract from its first release.

Stage 1: Server Idempotency Keys

Goal:

Extend distributor serve so POST /upload can safely accept retried producer uploads without creating duplicate accepted runs.

HTTP contract:

• Producers may send Idempotency-Key: <key> with POST /upload.
• The public upload package must always send this header.
• Raw HTTP clients may omit it; omitted keys preserve current behavior.
• Keys are scoped to the authenticated pipeline selected by bearer token.
• Valid keys are non-empty ASCII strings up to 128 bytes using letters, digits, ., _, -, and :.
• Invalid keys return 400.

Server behavior:

• After archive staging and source bundle validation succeeds, record the idempotency key with the accepted run id and the normalized source manifest identity.
• If the same pipeline receives the same key and the staged upload has the same normalized source manifest identity, return the original accepted response instead of enqueueing another run.
• If the same pipeline receives the same key and the staged upload has a different normalized source manifest identity, return 409 Conflict.
• If the same key is already being processed concurrently for the same pipeline before a manifest identity is available, return a retryable conflict response without accepting a new run.
• Idempotency records are memory-only and expire with the existing HTTP upload retention window.
• Server restart clears idempotency records, matching the current memory-only status and queue behavior.

Manifest identity:

• Compare normalized source manifest semantics, not raw archive bytes.
• At minimum, compare manifest schema version, id, created timestamp, bundle digest, and ordered file records.
• Different tar metadata or gzip encoding for the same source bundle should not create a conflict.

Tests:

• go test ./internal/app ./internal/ingest
• Same token, same key, same staged bundle returns the original run id and does not enqueue a second run.
• Same token, same key, different staged bundle returns 409.
• Same key under different authenticated pipelines does not conflict.
• Missing idempotency key preserves existing raw HTTP behavior.
• Invalid key syntax returns 400.
• Idempotency records expire with completed run status retention.
• Secret tokens and idempotency keys are not logged in errors beyond the key value itself where required for diagnostics.

Completion criteria:

• The HTTP API has an implemented, tested idempotency contract.
• Existing clients without Idempotency-Key continue to work.
• Duplicate idempotent uploads cannot create duplicate accepted runs.

Stage 2: Public pkg/upload Client

Goal:

Add a new public pkg/upload package that submits compliant bundles to distributor serve using bearer authentication and idempotency keys.

Package name:

• Use pkg/upload.
• Rationale: pkg/bundle owns bundle construction and validation; pkg/upload owns submission to the distributor HTTP upload API.

API shape:

Use options-struct APIs rather than one large positional function. Initial APIs must cover two producer workflows:

• upload an existing local bundle root;
• build a temporary bundle from explicit producer files and upload it.

Representative API shape:

package upload

type Client struct {
    // unexported fields
}

type ClientOptions struct {
    Endpoint   string
    Token      string
    HTTPClient *http.Client
    Retry      RetryOptions
}

type RetryOptions struct {
    MaxAttempts int
    BaseDelay   time.Duration
    MaxDelay    time.Duration
}

type UploadBundleOptions struct {
    Root           string
    Validate       bool
    IdempotencyKey string
}

type UploadFilesOptions struct {
    ID             string
    Created        time.Time
    Files          []bundle.BundleFile
    Validate       bool
    TempDir        string
    IdempotencyKey string
}

type Result struct {
    RunID  string
    Status string
}

func NewClient(opts ClientOptions) (*Client, error)
func (c *Client) UploadBundle(ctx context.Context, opts UploadBundleOptions) (Result, error)
func (c *Client) UploadFiles(ctx context.Context, opts UploadFilesOptions) (Result, error)
func (c *Client) Status(ctx context.Context, runID string) (RunStatus, error)

Required API semantics:

• Endpoint is a distributor server base URL. The client derives /upload and /runs/<run-id> internally.
• Token is required and is sent as Authorization: Bearer <token>.
• HTTPClient is optional; when omitted, use a client with conservative timeouts.
• UploadBundle reads and packages an existing local source bundle.
• UploadFiles creates a temporary complete bundle through pkg/bundle, then packages and uploads it.
• Status is optional for callers and never required by UploadBundle or UploadFiles.
• Result represents upload admission, not final publication success.

Idempotency key behavior:

• The client must send Idempotency-Key on every upload.
• If the caller supplies IdempotencyKey, use it.
• If omitted, generate a random 128-bit lowercase hex key once for that upload operation and reuse it for all retries from that call.
• Generated keys are not stable across process restarts or separate calls.
• Producers that need cross-process retry safety must supply their own stable key.
• Validate caller-supplied keys before making a request.

Bundle and archive behavior:

• UploadBundle loads manifest.json from the bundle root and validates the local bundle by default.
• UploadBundle includes manifest.json and every manifest-listed file in the tar.gz archive, and does not include unlisted files.
• UploadFiles requires a non-empty bundle id and non-empty file list.
• UploadFiles uses pkg/bundle.WriteBundle or equivalent public bundle APIs in a temporary directory and preserves explicit file order.
• Zero Created follows pkg/bundle defaulting behavior.
• Validation is enabled by default and may be explicitly disabled only for callers that already performed equivalent validation.
• Tar entry names are slash-separated bundle-relative paths.
• The package must not write into producer source directories.

Archive and retry strategy:

• Create a replayable upload body for each upload operation.
• The implementation may either create a temporary .tar.gz file or regenerate the tar.gz body from the validated staged bundle for each attempt.
• Clean up all temporary bundles and archive files created by the package.
• Use Content-Type: application/gzip.

Retry policy:

• Defaults should be safe and modest, for example three total attempts with bounded exponential backoff.
• Retry 503 Service Unavailable because the upload was not accepted.
• Retry temporary network errors and ambiguous mid-upload failures using the same idempotency key and replayable body.
• Do not retry 400, 401, 409, 413, or 415.
• Do not retry after 202 Accepted.
• Respect caller context cancellation before waiting and before each retry.
• Redact the bearer token from all errors.

HTTP response handling:

• Parse 202 Accepted responses into Result.
• Parse JSON error bodies where available.
• Include HTTP status codes and response messages in typed errors.
• Treat duplicate idempotent 202 responses the same as first acceptance.
• Treat 409 Conflict as an idempotency conflict error.
• Close response bodies on every attempt.

Tests:

• go test ./pkg/bundle ./pkg/upload
• Client construction with valid and invalid base endpoints.
• Missing token rejection and token redaction in errors.
• Caller-supplied and generated idempotency keys.
• Uploading an existing valid bundle root.
• Building and uploading from bundle.BundleFile values.
• Local validation failures before any HTTP request.
• Tar.gz entry names, manifest inclusion, and exclusion of unlisted files.
• 202, 400, 401, 409, 413, 415, 503, non-JSON errors, and unexpected status response parsing.
• Safe retry with the same idempotency key for 503 and retryable network failures.
• No retry for non-retryable statuses.
• Context cancellation during retry backoff.
• Custom *http.Client behavior through httptest.Server.

Completion criteria:

• Producer applications can build or validate a bundle and upload it with one package.
• All uploads include idempotency keys.
• Retry behavior is safe under the server idempotency contract.

Stage 3: Documentation And Examples

Goal:

Document the implemented producer upload package and idempotency behavior only after the server and public package exist.

Current-behavior documentation updates:

• README.md: mention the new producer upload package briefly.
• docs/integrations/source-bundle.md: link from producer APIs to upload helpers.
• docs/integrations/http-upload.md: document Idempotency-Key and add a short Go producer helper section.
• docs/operations.md: add a concise producer-side example if useful.
• docs/internal/app.md: document server-side idempotency record behavior.
• docs/policy/development.md: document the pkg/upload boundary and test expectations.

Tests and checks:

go test ./...
rg -n "pkg/upload|Idempotency-Key|UploadBundle|UploadFiles" README.md docs examples

Completion criteria:

• Current docs describe implemented behavior.
• Future-only behavior remains under docs/roadmap/.

Deferred Work

• Durable idempotency records across server restarts.
• Producer-supplied idempotency keys integrated with a database-backed queue.
• UploadAndWait or long-polling helpers.
• Run cancellation, retry, or listing endpoints.
• Zstandard-compressed tar archives.
• Multipart, resumable, or streaming object upload support.
• URL-token authentication for constrained clients.