distributor/docs/roadmap/implementation.md

Producer HTTP Upload Package Implementation Roadmap

Current Baseline

distributor serve and the http_upload source backend are implemented. Producers can already submit complete tar or gzip-compressed tar source bundles to POST /upload with bearer authentication, and each bearer token maps to one configured upload pipeline.

The public pkg/bundle package already provides producer-side source manifest semantics, digest calculation, path validation, local manifest building, local bundle writing, and local bundle validation helpers.

The current implementation does not have a server-side producer idempotency contract, and it does not provide a public pkg/upload helper package.

Future behavior remains under docs/roadmap/ until implemented. Do not update README, current user docs, examples, or current-behavior internal docs until the corresponding stage has been implemented.

This active roadmap implements the accepted producer upload package plan in docs/roadmap/producer.md. It supersedes the older docs/roadmap/http.md deferred note for producer-supplied idempotency keys; producer idempotency is now active roadmap work.

Active Roadmap

Stage 1: Server-Side Upload Idempotency

Goal:

Add Idempotency-Key support to POST /upload so safe producer retries do not create duplicate accepted runs.

Implementation scope:

• Validate optional Idempotency-Key headers using the syntax defined in docs/roadmap/producer.md.
• Scope keys by the authenticated pipeline selected through bearer-token mapping.
• Record accepted keys after archive staging and source bundle validation succeed.
• Compare normalized source manifest identity, not raw archive bytes.
• Return the original accepted run response for the same pipeline, same key, and same manifest identity.
• Return 409 Conflict for the same pipeline and key with a different manifest identity.
• Return a retryable conflict response when the same key is already being processed concurrently for the same pipeline before manifest identity is known.
• Expire idempotency records with existing upload status retention.
• Keep idempotency records memory-only; server restart clears them.
• Preserve current raw HTTP behavior when no idempotency key is supplied.

Current-behavior documentation updates after implementation:

• docs/integrations/http-upload.md
• docs/operations.md
• docs/internal/app.md
• docs/troubleshooting.md

Tests:

• Same key and same bundle returns the original run id and does not enqueue a second run.
• Same key and different bundle returns 409.
• Same key under different authenticated pipelines does not conflict.
• Missing key preserves current raw HTTP behavior.
• Invalid key syntax returns 400.
• Expiration removes idempotency records.
• Tokens are never leaked; idempotency keys appear only where needed for diagnostics.

Completion criteria:

• Existing upload clients continue to work.
• The HTTP API has an implemented, tested idempotency contract.
• Idempotent retries cannot create duplicate accepted runs.

Stage 2: Public pkg/upload API And Client

Goal:

Add a producer-facing upload package that builds or validates bundles, archives them, and submits them to the HTTP upload API.

Implementation scope:

• Add public pkg/upload.
• Use pkg/bundle for manifest generation, digest and path semantics, local validation, and temporary bundle creation.
• Implement options-struct APIs matching docs/roadmap/producer.md: ClientOptions, RetryOptions, UploadBundleOptions, UploadFilesOptions, Result, RunStatus, NewClient, UploadBundle, UploadFiles, and Status.
• Treat Endpoint as the distributor server base URL, deriving /upload and /runs/<run-id> internally.
• Require bearer token authentication and redact token values from all errors.
• Always send Idempotency-Key.
• Use caller-supplied idempotency keys when provided.
• When no key is supplied, generate one random 128-bit lowercase hex key per upload operation and reuse it across retries from that call.
• Support uploading an existing local bundle root.
• Support building a temporary bundle from explicit bundle.BundleFile values and uploading it.
• Create replayable gzip-compressed tar uploads with Content-Type: application/gzip.
• Retry only safe cases: 503 Service Unavailable, temporary network errors, and ambiguous mid-upload failures, using the same idempotency key and replayable body.
• Do not retry 400, 401, 409, 413, or 415.
• Do not retry after 202 Accepted.
• Respect context cancellation before waiting and before each retry.
• Close response bodies on every attempt.

Current-behavior documentation updates after implementation:

• Update pkg/bundle integration references only as needed once pkg/upload exists.
• Keep full user-facing docs and examples for Stage 3.

Tests:

• Client construction validates endpoint and token requirements.
• Token values are redacted from errors.
• Caller-supplied and generated idempotency keys are sent correctly.
• Existing bundle upload includes only manifest.json and manifest-listed files.
• File-based upload builds a compliant temporary bundle without touching producer source directories.
• Local validation failures prevent HTTP requests.
• Response parsing covers 202, 400, 401, 409, 413, 415, 503, non-JSON errors, and unexpected statuses.
• Retry uses the same idempotency key and stops correctly.
• Context cancellation during retry backoff is honored.
• Custom *http.Client behavior is covered with httptest.

Completion criteria:

• Go producers can upload valid bundles through pkg/upload.
• Safe retry behavior relies on the implemented server idempotency contract.
• Public package tests prove upload behavior does not duplicate or drift from pkg/bundle semantics.

Stage 3: Documentation And Examples

Goal:

Document implemented producer upload and idempotency behavior after the server contract and public package exist.

Implementation scope:

• Update current-behavior docs only after Stages 1 and 2 are implemented.
• Add secret-free examples where they are safe, copyable, and describe implemented behavior.
• Keep deferred items out of current docs.

Docs to update:

• README.md
• docs/integrations/source-bundle.md
• docs/integrations/http-upload.md
• docs/operations.md
• docs/internal/app.md
• docs/policy/development.md
• examples/, only if examples are safe, copyable, and implemented

Tests and checks:

go test ./...
rg -n "idempotency|Idempotency-Key|pkg/upload|UploadBundle|UploadFiles" README.md docs examples

Completion criteria:

• Current docs describe the implemented server idempotency and pkg/upload API.
• Completed behavior is not documented only as future work.
• docs/roadmap/ contains only future or deferred producer-upload work.

Deferred Work

• Durable idempotency storage across server restarts.
• Database-backed queues or durable producer retry processing.
• UploadAndWait, long polling, run cancellation, run retry, or run listing helpers.
• Zstandard archives.
• Multipart, resumable, or streaming upload protocols.
• URL-token authentication.
• Browser UI or public exposure defaults.

Validation

For this documentation pass:

rg -n "Idempotency-Key|pkg/upload|UploadBundle|UploadFiles|Stage 1: Server-Side Upload Idempotency" docs/roadmap/implementation.md
rg -n "Producer-supplied idempotency keys|Producer Coordination" docs/roadmap/http.md docs/roadmap/implementation.md
rg -n "pkg/upload|UploadBundle|UploadFiles|Idempotency-Key" README.md docs examples --glob '!docs/roadmap/**'
git status --short
git diff -- docs/roadmap/implementation.md

Expected result:

• New future behavior appears only under docs/roadmap/.
• Existing unrelated worktree changes, including any current docs/roadmap/documentation.md deletion, are not touched.
• This pass changes only docs/roadmap/implementation.md.