distributor/docs/roadmap/implementation.md

# Producer HTTP Upload Package Implementation Roadmap

## Current Baseline

`distributor serve` and the `http_upload` source backend are implemented.
Producers can already submit complete tar or gzip-compressed tar source bundles
to `POST /upload` with bearer authentication, and each bearer token maps to one
configured upload pipeline.

The public `pkg/bundle` package already provides producer-side source manifest
semantics, digest calculation, path validation, local manifest building, local
bundle writing, and local bundle validation helpers.

The current implementation does not have a server-side producer idempotency
contract, and it does not provide a public `pkg/upload` helper package.

Future behavior remains under `docs/roadmap/` until implemented. Do not update
README, current user docs, examples, or current-behavior internal docs until the
corresponding stage has been implemented.

This active roadmap implements the accepted producer upload package plan in
`docs/roadmap/producer.md`. It supersedes the older `docs/roadmap/http.md`
deferred note for producer-supplied idempotency keys; producer idempotency is
now active roadmap work.

## Active Roadmap

## Stage 1: Server-Side Upload Idempotency

Goal:

Add `Idempotency-Key` support to `POST /upload` so safe producer retries do not
create duplicate accepted runs.

Implementation scope:

- Validate optional `Idempotency-Key` headers using the syntax defined in
  `docs/roadmap/producer.md`.
- Scope keys by the authenticated pipeline selected through bearer-token
  mapping.
- Record accepted keys after archive staging and source bundle validation
  succeed.
- Compare normalized source manifest identity, not raw archive bytes.
- Return the original accepted run response for the same pipeline, same key,
  and same manifest identity.
- Return `409 Conflict` for the same pipeline and key with a different manifest
  identity.
- Return a retryable conflict response when the same key is already being
  processed concurrently for the same pipeline before manifest identity is
  known.
- Expire idempotency records with existing upload status retention.
- Keep idempotency records memory-only; server restart clears them.
- Preserve current raw HTTP behavior when no idempotency key is supplied.

Current-behavior documentation updates after implementation:

- `docs/integrations/http-upload.md`
- `docs/operations.md`
- `docs/internal/app.md`
- `docs/troubleshooting.md`

Tests:

- Same key and same bundle returns the original run id and does not enqueue a
  second run.
- Same key and different bundle returns `409`.
- Same key under different authenticated pipelines does not conflict.
- Missing key preserves current raw HTTP behavior.
- Invalid key syntax returns `400`.
- Expiration removes idempotency records.
- Tokens are never leaked; idempotency keys appear only where needed for
  diagnostics.

Completion criteria:

- Existing upload clients continue to work.
- The HTTP API has an implemented, tested idempotency contract.
- Idempotent retries cannot create duplicate accepted runs.

## Stage 2: Public `pkg/upload` API And Client

Goal:

Add a producer-facing upload package that builds or validates bundles, archives
them, and submits them to the HTTP upload API.

Implementation scope:

- Add public `pkg/upload`.
- Use `pkg/bundle` for manifest generation, digest and path semantics, local
  validation, and temporary bundle creation.
- Implement options-struct APIs matching `docs/roadmap/producer.md`:
  `ClientOptions`, `RetryOptions`, `UploadBundleOptions`,
  `UploadFilesOptions`, `Result`, `RunStatus`, `NewClient`, `UploadBundle`,
  `UploadFiles`, and `Status`.
- Treat `Endpoint` as the distributor server base URL, deriving `/upload` and
  `/runs/<run-id>` internally.
- Require bearer token authentication and redact token values from all errors.
- Always send `Idempotency-Key`.
- Use caller-supplied idempotency keys when provided.
- When no key is supplied, generate one random 128-bit lowercase hex key per
  upload operation and reuse it across retries from that call.
- Support uploading an existing local bundle root.
- Support building a temporary bundle from explicit `bundle.BundleFile` values
  and uploading it.
- Create replayable gzip-compressed tar uploads with
  `Content-Type: application/gzip`.
- Retry only safe cases: `503 Service Unavailable`, temporary network errors,
  and ambiguous mid-upload failures, using the same idempotency key and
  replayable body.
- Do not retry `400`, `401`, `409`, `413`, or `415`.
- Do not retry after `202 Accepted`.
- Respect context cancellation before waiting and before each retry.
- Close response bodies on every attempt.

Current-behavior documentation updates after implementation:

- Update `pkg/bundle` integration references only as needed once `pkg/upload`
  exists.
- Keep full user-facing docs and examples for Stage 3.

Tests:

- Client construction validates endpoint and token requirements.
- Token values are redacted from errors.
- Caller-supplied and generated idempotency keys are sent correctly.
- Existing bundle upload includes only `manifest.json` and manifest-listed
  files.
- File-based upload builds a compliant temporary bundle without touching
  producer source directories.
- Local validation failures prevent HTTP requests.
- Response parsing covers `202`, `400`, `401`, `409`, `413`, `415`, `503`,
  non-JSON errors, and unexpected statuses.
- Retry uses the same idempotency key and stops correctly.
- Context cancellation during retry backoff is honored.
- Custom `*http.Client` behavior is covered with `httptest`.

Completion criteria:

- Go producers can upload valid bundles through `pkg/upload`.
- Safe retry behavior relies on the implemented server idempotency contract.
- Public package tests prove upload behavior does not duplicate or drift from
  `pkg/bundle` semantics.

## Stage 3: Documentation And Examples

Goal:

Document implemented producer upload and idempotency behavior after the server
contract and public package exist.

Implementation scope:

- Update current-behavior docs only after Stages 1 and 2 are implemented.
- Add secret-free examples where they are safe, copyable, and describe
  implemented behavior.
- Keep deferred items out of current docs.

Docs to update:

- `README.md`
- `docs/integrations/source-bundle.md`
- `docs/integrations/http-upload.md`
- `docs/operations.md`
- `docs/internal/app.md`
- `docs/policy/development.md`
- `examples/`, only if examples are safe, copyable, and implemented

Tests and checks:

```sh
go test ./...
rg -n "idempotency|Idempotency-Key|pkg/upload|UploadBundle|UploadFiles" README.md docs examples
```

Completion criteria:

- Current docs describe the implemented server idempotency and `pkg/upload`
  API.
- Completed behavior is not documented only as future work.
- `docs/roadmap/` contains only future or deferred producer-upload work.

## Deferred Work

- Durable idempotency storage across server restarts.
- Database-backed queues or durable producer retry processing.
- `UploadAndWait`, long polling, run cancellation, run retry, or run listing
  helpers.
- Zstandard archives.
- Multipart, resumable, or streaming upload protocols.
- URL-token authentication.
- Browser UI or public exposure defaults.

## Validation

For this documentation pass:

```sh
rg -n "Idempotency-Key|pkg/upload|UploadBundle|UploadFiles|Stage 1: Server-Side Upload Idempotency" docs/roadmap/implementation.md
rg -n "Producer-supplied idempotency keys|Producer Coordination" docs/roadmap/http.md docs/roadmap/implementation.md
rg -n "pkg/upload|UploadBundle|UploadFiles|Idempotency-Key" README.md docs examples --glob '!docs/roadmap/**'
git status --short
git diff -- docs/roadmap/implementation.md
```

Expected result:

- New future behavior appears only under `docs/roadmap/`.
- Existing unrelated worktree changes, including any current
  `docs/roadmap/documentation.md` deletion, are not touched.
- This pass changes only `docs/roadmap/implementation.md`.