Revise and solidify the implementation roadmap

README.md

@@ -10,11 +10,12 @@ Run the maintained local example:
 go run ./cmd/distributor run --config examples/local-publish.yml
 ```
 
-Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate local source bundles with the same manifest contract used by the CLI. See [Source bundle contract](docs/integrations/source-bundle.md).
+Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/upload` and `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to submit compatible bundles to `distributor serve`. See [Upstream producer integration](docs/consumers/api.md).
 
 - [CLI reference](docs/cli.md)
 - [Configuration reference](docs/config.md)
 - [Operations guide](docs/operations.md)
+- [Consumer API guide](docs/consumers/api.md)
 - [Troubleshooting](docs/troubleshooting.md)
 - [Integration contracts](docs/integrations/source-bundle.md)
 - [Development architecture](docs/policy/architecture.md)

docs/config.md

@@ -86,11 +86,15 @@ server:
     queue_size: 16
     max_concurrency: 1
     retention: 24h
+upload_tokens:
+  - id: weather-reporter
+    token_env: WEATHER_UPLOAD_TOKEN
+    allow_pipelines:
+      - weather-daily
 pipelines:
   - id: weather-daily
     source:
       backend: http_upload
-      token_env: WEATHER_DAILY_UPLOAD_TOKEN
       staging_path: /var/spool/distributor/weather-daily
       max_upload_size: 20MB
     destinations:
@@ -99,9 +103,9 @@ pipelines:
         path: /srv/reports/archive
 ```
 
-`token_env` is required for `http_upload` sources. `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`. `max_upload_size` defaults to `server.http.max_upload_size`.
+`upload_tokens` is required when any pipeline source uses `http_upload`. Each token record resolves its bearer token value from the process environment or `secrets.directory`. `allow_pipelines` lists configured upload pipeline ids that the token may submit to. One token may authorize multiple upload pipelines, and multiple tokens may authorize the same upload pipeline.
 
-`serve` maps each resolved bearer token to exactly one `http_upload` pipeline. Startup fails when a token is missing, empty, or duplicates another upload pipeline token.
+For `http_upload` sources, `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`. `max_upload_size` defaults to `server.http.max_upload_size`.
 
 ## Top-Level Fields
 
@@ -124,6 +128,18 @@ Numeric server values and durations must be greater than zero after defaults are
 
 See [Secrets](#secrets) for resolution rules.
 
+### `upload_tokens`
+
+`upload_tokens` configures bearer tokens for `distributor serve`. It is required when any pipeline source backend is `http_upload` and is invalid when no upload pipelines are configured.
+
+Each token has:
+
+- `id`: required unique slug-like identifier for the token record. It must start with a letter or number and may contain letters, numbers, `.`, `_`, and `-`.
+- `token_env`: required environment variable or secret-file name containing the bearer token value.
+- `allow_pipelines`: required non-empty list of configured pipeline ids whose source backend is `http_upload`.
+
+Token values must resolve to non-empty strings and must be unique across token records. Every configured upload pipeline must be allowed by at least one token.
+
 ### `pipelines`
 
 `pipelines` is required and must contain at least one pipeline.
@@ -216,13 +232,11 @@ HTTP upload backends are valid only as pipeline sources and are served by `distr
 
 ```yaml
 backend: http_upload
-token_env: WEATHER_DAILY_UPLOAD_TOKEN
 staging_path: /var/spool/distributor/weather-daily
 max_upload_size: 20MB
 ```
 
 - `backend`: required value `http_upload`.
-- `token_env`: required environment variable or secret-file name containing the bearer token.
 - `staging_path`: optional staging path. Default: `<server.http.staging_root>/<pipeline id>`.
 - `max_upload_size`: optional per-source upload limit. Default: `server.http.max_upload_size`.
 
@@ -416,7 +430,7 @@ Fields resolved through this resolver:
 
 - `credentials.access_key_id_env`
 - `credentials.secret_access_key_env`
-- `source.token_env` for `http_upload` sources
+- `upload_tokens[].token_env`
 
 ## Maintained Examples
 

docs/consumers/api.md

@@ -0,0 +1,136 @@
+# Upstream Producer Integration
+
+Audience: developers and LLM coding agents adding `distributor` support to an upstream Go producer application.
+
+This document is the copyable implementation guide for submitting producer outputs to a `distributor` pipeline whose source backend is `http_upload`.
+
+## Required Inputs
+
+The upstream application needs these values from deployment or operator configuration:
+
+- distributor endpoint: the HTTP server base URL, such as `https://distributor.example.com`;
+- upload token: bearer token that authenticates the producer;
+- pipeline id: configured `http_upload` pipeline that should process this upload;
+- generated files: regular local files to include in the source bundle;
+- bundle id: stable identifier for the logical report stream or artifact;
+- idempotency key: unique key for one producer run, reused only when retrying that same run.
+
+Do not put destination routing, public URLs, transform settings, or credentials in the source manifest. Those belong in the `distributor` pipeline configuration.
+
+The token, pipeline id, bundle id, and idempotency key have different jobs. The token authenticates the producer. The pipeline id selects the configured distributor workflow, including destinations and publishing policy. The bundle id tells `distributor` whether a new upload is a newer version of the same source; keep it stable across runs that should replace the same managed destination artifact. The idempotency key tells `distributor` whether an upload request is a retry; change it for each distinct producer run so new content is enqueued.
+
+## Recommended Workflow
+
+Use `gitea.maximumdirect.net/eric/distributor/pkg/upload`.
+
+For most producers, use `UploadFiles`. It accepts producer-generated files, builds a temporary valid source bundle with `pkg/bundle`, uploads a gzip-compressed tar archive, and removes temporary files when the call returns.
+
+Use `UploadBundle` only when the producer already assembled a complete bundle directory containing `manifest.json`.
+
+Add the dependency from the upstream application:
+
+```sh
+go get gitea.maximumdirect.net/eric/distributor
+```
+
+## Minimal Go Example
+
+```go
+package reports
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"time"
+
+	"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+	"gitea.maximumdirect.net/eric/distributor/pkg/upload"
+)
+
+func SubmitReport(reportPath, summaryPath string) error {
+	endpoint := os.Getenv("DISTRIBUTOR_UPLOAD_ENDPOINT")
+	token := os.Getenv("DISTRIBUTOR_UPLOAD_TOKEN")
+	if endpoint == "" || token == "" {
+		return fmt.Errorf("distributor endpoint and token are required")
+	}
+
+	pipelineID := "weather-hourly"
+	reportID := "weather.hourly.brentwood"
+	runID := time.Now().UTC().Format("20060102T150405.000000000Z")
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+
+	client, err := upload.NewClient(upload.ClientOptions{
+		Endpoint: endpoint,
+		Token:    token,
+	})
+	if err != nil {
+		return err
+	}
+
+	result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
+		PipelineID:     pipelineID,
+		ID:             reportID,
+		IdempotencyKey: reportID + "." + runID,
+		Files: []bundle.BundleFile{
+			{SourcePath: reportPath, Path: "report.md"},
+			{SourcePath: summaryPath, Path: "summary.txt"},
+		},
+	})
+	if err != nil {
+		var conflict *upload.IdempotencyConflictError
+		if errors.As(err, &conflict) {
+			return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
+		}
+		return err
+	}
+
+	fmt.Printf("distributor accepted run %s\n", result.RunID)
+	return nil
+}
+```
+
+## Producer Responsibilities
+
+- Use a stable bundle id for the logical producer output that should replace the same destination artifact, such as `weather.hourly.brentwood`.
+- Set `PipelineID` to the configured upload pipeline that should process the bundle.
+- Do not include per-run timestamps, random values, or job ids in the bundle id unless each run should be treated as a different source.
+- Use an idempotency key that changes for every distinct producer run, such as `<bundle-id>.<run-id>`.
+- Reuse the same idempotency key only when retrying the exact same producer run with the same source manifest.
+- Map each generated file to a clean slash-separated bundle path, such as `report.md` or `assets/chart.png`.
+- Include only regular files. Symlinks, directories as files, devices, FIFOs, and sockets are rejected.
+- Keep file contents stable after upload inputs are selected. Bundle digests are calculated from file bytes.
+- Treat upload success as admission only. `UploadFiles` and `UploadBundle` return after the server accepts and validates the upload, not after all destinations publish.
+
+Valid bundle paths are relative slash paths. They must not be empty, absolute, contain backslashes, contain `.` or `..` path segments, contain empty path segments, or use reserved basenames `manifest.json` or `.distributor.json`.
+
+## Idempotency And Status
+
+`pkg/upload` sends `Idempotency-Key` on every upload. If the caller omits one, the package generates a random key for that call and reuses it for in-process retries. That is enough for transient network retry within one process, but it does not give cross-process retry identity.
+
+For producer jobs that may retry after process restart, supply a key derived from the producer run, such as `<bundle-id>.<run-id>`. Reusing the same key with the same token, pipeline id, and normalized source manifest returns the original accepted run. Reusing the same key with different source content in that scope returns a conflict. Reusing one key across multiple distinct report generations prevents those generations from being treated as new uploads.
+
+`Status` polls `/runs/<run-id>` while the distributor server retains the in-memory status record. Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to the server's `server.http.retention` setting, and server restart clears status and idempotency records.
+
+Optional status check:
+
+```go
+status, err := client.Status(ctx, result.RunID)
+if err != nil {
+	return err
+}
+if status.Status == "failed" {
+	return fmt.Errorf("distributor run failed: %s", status.Error)
+}
+```
+
+## References
+
+In the `distributor` source tree:
+
+- `docs/consumers/pkg-upload.md`: Go upload package workflow.
+- `docs/consumers/pkg-bundle.md`: Go bundle package workflow.
+- `docs/integrations/http-upload.md`: canonical HTTP upload wire contract.
+- `docs/integrations/source-bundle.md`: canonical source bundle file-format contract.

docs/consumers/pkg-bundle.md

@@ -0,0 +1,90 @@
+# `pkg/bundle`
+
+Audience: upstream Go producer developers and LLM coding agents using `distributor` source bundle helpers.
+
+Import path:
+
+```go
+import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+```
+
+`pkg/bundle` builds, writes, parses, and validates local source bundles. Use it directly when a producer writes bundles for `distributor` to discover, or when a producer wants to assemble and validate a bundle before using another transport.
+
+The canonical source bundle file-format contract is [Source Bundle Contract](../integrations/source-bundle.md).
+
+## Preferred Complete-Bundle Workflow
+
+Use `WriteBundle` when producer-generated files live outside the final bundle root.
+
+```go
+manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{
+	Root: "/var/spool/distributor/weather/hourly-2026-06-07T15",
+	ID:   "weather.hourly.brentwood",
+	Files: []bundle.BundleFile{
+		{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
+		{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
+	},
+})
+if err != nil {
+	return err
+}
+_ = manifest
+```
+
+`WriteBundle` copies each source file into a staged bundle root, writes `manifest.json`, validates the staged bundle, and promotes it into place. Set `Overwrite: true` only when the producer intentionally replaces an existing bundle root.
+
+## Existing Bundle Root Workflow
+
+Use `BuildManifest` and `WriteManifest` when files are already staged under the final bundle root.
+
+```go
+root := "/var/spool/distributor/weather/hourly-2026-06-07T15"
+manifest, err := bundle.BuildManifest(bundle.BuildOptions{
+	Root:  root,
+	ID:    "weather.hourly.brentwood",
+	Files: []string{"report.md", "summary.txt"},
+})
+if err != nil {
+	return err
+}
+if err := bundle.WriteManifest(root, manifest, bundle.WriteManifestOptions{}); err != nil {
+	return err
+}
+if err := bundle.ValidateBundle(root, manifest); err != nil {
+	return err
+}
+```
+
+Use `Scan: true` instead of `Files` only when every valid regular file under the root should be included. Scan mode includes dotfiles, skips reserved metadata files, rejects symlinks, and sorts paths lexically.
+
+## Paths And Ordering
+
+Bundle paths are slash-separated paths relative to the bundle root.
+
+Invalid paths include:
+
+- empty paths;
+- absolute paths;
+- paths containing backslashes;
+- `.` or `..` path segments;
+- empty path segments;
+- any basename of `manifest.json` or `.distributor.json`.
+
+Explicit file lists preserve caller order. File order is part of the bundle digest, so producers should choose it deliberately and keep it stable.
+
+The manifest `ID` is the logical source identity used by `distributor` destination comparison. Keep it stable for runs that should replace the same managed destination artifact. If every run uses a different manifest `ID`, `distributor` treats those runs as different sources and may report a destination conflict instead of replacing older output.
+
+## Validation And Digest Helpers
+
+Use `ValidateBundle` before handing an existing local bundle to another process. It verifies manifest semantics, file existence, regular-file type, file size, per-file SHA-256 digests, and bundle digest.
+
+Useful helpers:
+
+- `LoadManifest`: read `manifest.json` from a bundle root.
+- `ParseManifest` and `MarshalManifest`: parse or write manifest bytes.
+- `ValidateManifest`: validate manifest-only semantics.
+- `FileDigest`, `BundleDigest`, and `ValidateDigest`: digest helpers for diagnostics and tests.
+
+## Boundaries
+
+`pkg/bundle` does not upload bundles, publish destinations, transform Markdown, select pipelines, configure credentials, or write destination state. Those concerns belong to `pkg/upload` or the `distributor` application.

docs/consumers/pkg-upload.md

@@ -0,0 +1,122 @@
+# `pkg/upload`
+
+Audience: upstream Go producer developers and LLM coding agents submitting bundles to `distributor serve`.
+
+Import path:
+
+```go
+import "gitea.maximumdirect.net/eric/distributor/pkg/upload"
+```
+
+`pkg/upload` is the producer-facing HTTP upload client. It builds on `pkg/bundle`, packages valid source bundles as gzip-compressed tar archives, sends bearer authentication, routes uploads to a configured pipeline, includes idempotency keys, and exposes a status polling helper.
+
+`UploadFiles` examples also use:
+
+```go
+import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+```
+
+The canonical HTTP wire contract is [HTTP Upload API Contract](../integrations/http-upload.md).
+
+## Client Construction
+
+```go
+client, err := upload.NewClient(upload.ClientOptions{
+	Endpoint: "https://distributor.example.com",
+	Token:    token,
+})
+if err != nil {
+	return err
+}
+```
+
+`Endpoint` is the distributor server base URL. The client derives `/v1/pipelines/<pipeline-id>/upload` and `/runs/<run-id>`. `Token` is required and is sent as `Authorization: Bearer <token>`. Token values are redacted from client errors.
+
+`HTTPClient` and `Retry` are optional. Defaults use a 30 second HTTP timeout and safe retry settings.
+
+## Upload Producer Files
+
+Use `UploadFiles` when the producer has generated output files but has not assembled a bundle directory.
+
+```go
+result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
+	PipelineID:     "weather-hourly",
+	ID:             "weather.hourly.brentwood",
+	IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
+	Files: []bundle.BundleFile{
+		{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
+		{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
+	},
+})
+if err != nil {
+	return err
+}
+_ = result.RunID
+```
+
+`PipelineID` is required and selects the configured distributor workflow for this upload. `ID` is the source manifest id and identifies the logical artifact inside that workflow. `UploadFiles` creates a temporary bundle, writes and validates a manifest, uploads the archive, and removes temporary files when the call returns. It does not write into producer source directories.
+
+## Upload An Existing Bundle
+
+Use `UploadBundle` when the producer already has a complete local bundle root containing `manifest.json`.
+
+```go
+result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
+	PipelineID:     "weather-hourly",
+	Root:           "/var/spool/weather/hourly-2026-06-07T15",
+	IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
+})
+if err != nil {
+	return err
+}
+_ = result.RunID
+```
+
+`PipelineID` is required for existing bundles too. `UploadBundle` validates the local bundle by default and uploads only `manifest.json` plus manifest-listed files. Unlisted files are not uploaded.
+
+## Result And Status
+
+Upload success means the server returned `202 Accepted` after staging and validating the upload. It does not mean all configured destinations have published.
+
+Poll status while the server retains the in-memory run record:
+
+```go
+status, err := client.Status(ctx, result.RunID)
+if err != nil {
+	return err
+}
+if status.Status == "failed" {
+	return fmt.Errorf("distributor run failed: %s", status.Error)
+}
+```
+
+Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to `server.http.retention`; server restart clears run status and idempotency records.
+
+## Idempotency And Retry
+
+Every upload request includes `Idempotency-Key`.
+
+If `IdempotencyKey` is omitted, the client generates a random 128-bit lowercase hexadecimal key for that upload operation and reuses it for retries within the same call. For cross-process retry safety, producers should pass a key derived from the producer run, such as `<bundle-id>.<run-id>`.
+
+Do not reuse the same idempotency key for multiple distinct report generations. Reuse it only when retrying the exact same run with the same token, pipeline id, and source manifest. A repeated key with the same manifest in that scope returns the original accepted run instead of enqueueing another run; a repeated key with different content returns an idempotency conflict.
+
+The client retries only safe cases:
+
+- `503 Service Unavailable`;
+- temporary network errors;
+- ambiguous mid-upload failures.
+
+It does not retry after `202 Accepted` and does not retry `400`, `401`, `403`, `404`, `409`, `413`, or `415`.
+
+Detect conflicting key reuse with `errors.As`:
+
+```go
+var conflict *upload.IdempotencyConflictError
+if errors.As(err, &conflict) {
+	return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
+}
+```
+
+## Boundaries
+
+`pkg/upload` does not configure server pipelines, choose destinations, wait for publication completion automatically, persist client queues, provide durable idempotency across server restarts, or expose destination state. It submits complete source bundles to the configured HTTP upload API.

docs/integrations/http-upload.md

@@ -2,7 +2,7 @@
 
 Audience: producers, operators, and maintainers integrating with `distributor serve`.
 
-`distributor serve` exposes a local HTTP upload API for pipelines whose source backend is `http_upload`. Each bearer token maps to exactly one configured pipeline.
+`distributor serve` exposes a local HTTP upload API for pipelines whose source backend is `http_upload`. Bearer tokens authenticate producers, and the upload path selects the configured pipeline. The selected token must be allowed for the requested pipeline.
 
 ## Authentication
 
@@ -12,9 +12,9 @@ Uploads authenticate with:
 Authorization: Bearer <token>
 ```
 
-Token values are resolved from the configured `source.token_env` through the process environment or `secrets.directory`. Tokens are not configured as YAML literal values.
+Token values are resolved from top-level `upload_tokens` records through the process environment or `secrets.directory`. Tokens are not configured as YAML literal values.
 
-Requests that include `pipeline` or `pipeline_id` query parameters are rejected. The bearer token selects the pipeline.
+Requests that include `pipeline` or `pipeline_id` query parameters are rejected. Use the pipeline id in the upload path.
 
 ## Endpoints
 
@@ -26,10 +26,18 @@ Returns `200 OK` when the server is running:
 {"status":"ok"}
 ```
 
-### `POST /upload`
+### `POST /v1/pipelines/{pipeline_id}/upload`
 
 Accepts one source bundle archive and returns after the archive is staged and validated.
 
+Producers may include:
+
+```text
+Idempotency-Key: <key>
+```
+
+`pipeline_id` must name a configured pipeline whose source backend is `http_upload`, and the authenticated token must allow that pipeline. Idempotency keys are scoped to token id, pipeline id, and key. Valid keys are non-empty ASCII strings up to 128 bytes using letters, digits, `.`, `_`, `-`, and `:`. Invalid keys return `400`.
+
 Accepted content types:
 
 - `application/x-tar`
@@ -44,8 +52,11 @@ Successful admission returns `202 Accepted`:
 
 Common error responses:
 
-- `400`: pipeline query supplied, archive rejected, malformed archive, or invalid staged source bundle.
+- `400`: pipeline query supplied, invalid idempotency key, archive rejected, malformed archive, or invalid staged source bundle.
 - `401`: missing, empty, or unknown bearer token.
+- `403`: bearer token is valid but is not allowed for the requested pipeline.
+- `404`: upload path is unknown or the requested upload pipeline is not configured.
+- `409`: repeated idempotency key conflicts with another source manifest, or the same key is already being staged.
 - `413`: upload body exceeds the selected pipeline size limit.
 - `415`: unsupported content type.
 - `503`: upload queue is full.
@@ -56,6 +67,14 @@ Error bodies use:
 {"error":"<message>"}
 ```
 
+Retryable idempotency conflicts include:
+
+```json
+{"error":"upload idempotency key is already being processed","retryable":true}
+```
+
+When `Idempotency-Key` is omitted, upload admission preserves the raw HTTP behavior: every valid accepted upload receives its own run id. When a key is supplied, the server records the accepted run after archive staging and source bundle validation succeed. Reusing the same key for the same token id, pipeline id, and normalized source manifest returns the original `202 Accepted` response and does not enqueue another run. Reusing the same key for a different normalized source manifest within that scope returns `409 Conflict`. Producers should use a fresh key for each distinct producer run and reuse a key only for retries of that same run.
+
 ### `GET /runs/<run-id>`
 
 Returns an in-memory status record while retained:
@@ -89,20 +108,45 @@ Archive entry rules:
 
 The uploaded archive size and extracted bundle size are bounded by the selected pipeline's `source.max_upload_size`. Extracted file count is also bounded by the implementation.
 
+## Go Producer Helper
+
+Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/upload` to build or validate source bundles, package them as gzip-compressed tar archives, and submit them to this API. See [Upstream Producer Integration](../consumers/api.md) for the copyable upstream implementation guide and [`pkg/upload`](../consumers/pkg-upload.md) for package-specific workflow guidance.
+
+```go
+client, err := upload.NewClient(upload.ClientOptions{
+	Endpoint: "http://127.0.0.1:8080",
+	Token:    token,
+})
+if err != nil {
+	return err
+}
+result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
+	PipelineID:     "reports",
+	Root:           "examples/source-bundle",
+	IdempotencyKey: "reports.example.20260604T120000Z",
+})
+```
+
+`Endpoint` is the server base URL; the package derives `/v1/pipelines/<pipeline-id>/upload` and `/runs/<run-id>`. `PipelineID` is required and selects the configured distributor workflow. `UploadBundle` validates a local bundle by default and uploads only `manifest.json` plus manifest-listed files. `UploadFiles` creates a temporary bundle from explicit `bundle.BundleFile` values before uploading. When `IdempotencyKey` is omitted, the package generates one random 128-bit lowercase hex key for the upload operation and reuses it across retries.
+
+The helper retries only safe cases: `503 Service Unavailable`, temporary network errors, and ambiguous mid-upload failures. It does not retry after `202 Accepted` and does not retry `400`, `401`, `403`, `404`, `409`, `413`, or `415`. Bearer token values are redacted from returned errors.
+
 ## Queue And Retention
 
 `server.http.queue_size` bounds accepted-but-not-started uploads plus uploads being staged. `server.http.max_concurrency` bounds publishing concurrency. The coordinator does not run two uploads for the same pipeline concurrently.
 
 Completed status records expire after `server.http.retention`; expiration removes committed staged bundle directories for completed uploads. Server restart clears queue state and status records.
 
+Idempotency records are memory-only, expire with completed upload status records, and are cleared by server restart.
+
 ## Boundaries
 
-The HTTP API does not expose pipeline selection by request parameter, TLS, public routing policy, or durable status storage. Put public access controls, TLS termination, and rate limiting in deployment infrastructure.
+The HTTP API does not expose pipeline selection by query parameter, TLS, public routing policy, or durable status storage. Put public access controls, TLS termination, and rate limiting in deployment infrastructure.
 
 ## Tests
 
 Before changing this contract, inspect and run:
 
 ```sh
-go test ./internal/app ./internal/ingest
+go test ./internal/app ./internal/ingest ./pkg/upload
 ```

docs/integrations/source-bundle.md

@@ -27,7 +27,7 @@ Current schema version: `1`.
 Required manifest fields:
 
 - `schema_version`: must be `1`.
-- `id`: non-empty bundle identifier.
+- `id`: non-empty bundle identifier. For replacement workflows, keep this stable for the logical source that should update the same managed destination artifact.
 - `digest`: lowercase `sha256:<64 hex>` digest of the ordered `files` list.
 - `created`: RFC3339 timestamp.
 - `files`: non-empty ordered list of file records.
@@ -60,7 +60,7 @@ File order is significant. Explicit file lists preserve caller order. Scan mode
 
 ## Producer APIs
 
-Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build and validate this contract:
+Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build and validate this contract. See [`pkg/bundle`](../consumers/pkg-bundle.md) for producer workflow guidance.
 
 - `BuildManifest`: builds a manifest from explicit file paths or scan mode.
 - `WriteManifest`: writes `manifest.json`, optionally replacing an existing manifest.
@@ -68,6 +68,8 @@ Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to bu
 - `LoadManifest`, `ParseManifest`, `ValidateManifest`, and `ValidateBundle`: parse and validate local bundles.
 - `FileDigest`, `BundleDigest`, and `ValidateDigest`: digest helpers.
 
+Go producers that submit bundles to `distributor serve` can use `gitea.maximumdirect.net/eric/distributor/pkg/upload`. See [Upstream Producer Integration](../consumers/api.md) and [HTTP Upload API Contract](http-upload.md).
+
 CLI producers can use:
 
 ```sh
@@ -88,5 +90,5 @@ The source bundle manifest does not configure routing, destination selection, pu
 Before changing this contract, inspect and run:
 
 ```sh
-go test ./pkg/bundle ./internal/bundle
+go test ./pkg/bundle ./pkg/upload ./internal/bundle
 ```

docs/internal/app.md

@@ -34,19 +34,21 @@ The app layer registers default transforms, including Markdown-to-HTML, and supp
 
 Run workflows discover and validate source bundles through `internal/bundle`. Destination state actions are prepared and written through `internal/publish` and `internal/state`; the app layer records report projections of those actions and results.
 
-HTTP uploads stage and validate archives before enqueueing a pipeline run with a local staged source root.
+HTTP uploads stage and validate archives before enqueueing a pipeline run with a local staged source root. Go producers can use the public `pkg/upload` package to create client-side gzip tar uploads for this server contract; `internal/app` remains the server-side orchestration boundary and does not import that producer package.
+
+Upload idempotency is owned by the upload coordinator. Optional `Idempotency-Key` values are scoped to token id, pipeline id, and key. The coordinator reserves a key while staging is in progress, records the accepted run id with the validated source manifest identity after staging succeeds, returns the original accepted record for the same scoped key and same manifest, and rejects the same scoped key with a different manifest as a conflict.
 
 ## Skip And Resume Behavior
 
 Fan-out destinations are independent. A destination failure is recorded and does not prevent later destinations from being attempted. Dry-run builds plans and reports without destination writes, destination state writes, notifier calls, or SSH known-host persistence.
 
-HTTP upload status is in memory. Accepted jobs move through accepted, queued, running, succeeded, or failed states and expire after configured retention.
+HTTP upload status is in memory. Accepted jobs move through accepted, queued, running, succeeded, or failed states and expire after configured retention. Upload idempotency records are also memory-only, expire with the completed status record for their accepted run, and are cleared by process restart.
 
 ## Failure Behavior
 
 Runtime setup fails for config load, config validation, secret loading, or credential resolution errors. Source setup failures stop the affected run before destination planning. Destination open, planning, execution, and notification failures are recorded as destination failures where a partial result exists.
 
-HTTP upload startup fails if upload tokens are missing, empty, or duplicated. Upload requests can fail during authentication, content-type validation, queue admission, archive staging, source validation, or later publish execution.
+HTTP upload startup fails if upload tokens are missing, empty, or duplicated. Upload requests can fail during authentication, idempotency-key validation, content-type validation, idempotency conflict checks, queue admission, archive staging, source validation, or later publish execution.
 
 ## Tests To Inspect
 
@@ -63,4 +65,5 @@ HTTP upload startup fails if upload tokens are missing, empty, or duplicated. Up
 - Fan-out destinations remain independent after a destination-scoped failure.
 - Secret values are never printed; warnings may name variables only.
 - Upload admission stages and validates a bundle before returning a run id.
+- Idempotent upload retries compare normalized source manifest identity, not archive bytes.
 - Runtime backend registration remains app-owned.

docs/operations.md

@@ -114,7 +114,7 @@ Forced replacement deletes the current destination bundle path before writing ou
 
 ## HTTP Upload Operation
 
-The [HTTP Upload API Contract](integrations/http-upload.md) defines request and response details. `distributor serve` runs the HTTP upload API for pipelines whose source backend is `http_upload`. Each bearer token maps to exactly one configured upload pipeline. Token values come from the process environment or `secrets.directory`, not from YAML literal values.
+The [HTTP Upload API Contract](integrations/http-upload.md) defines request and response details. `distributor serve` runs the HTTP upload API for pipelines whose source backend is `http_upload`. Top-level `upload_tokens` authenticate producers and allow one or more upload pipelines. Token values come from the process environment or `secrets.directory`, not from YAML literal values.
 
 Start the maintained local example:
 
@@ -132,12 +132,30 @@ curl http://127.0.0.1:8080/healthz
 Upload one tar or tar.gz source bundle archive:
 
 ```sh
-curl -X POST http://127.0.0.1:8080/upload \
+curl -X POST http://127.0.0.1:8080/v1/pipelines/example-http-upload/upload \
   -H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \
   -H "Content-Type: application/gzip" \
   --data-binary @bundle.tar.gz
 ```
 
+For safe producer retries, include an idempotency key that is stable for the same producer run and different for each distinct run:
+
+```sh
+curl -X POST http://127.0.0.1:8080/v1/pipelines/example-http-upload/upload \
+  -H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \
+  -H "Content-Type: application/gzip" \
+  -H "Idempotency-Key: producer.run.20260604T120000Z" \
+  --data-binary @bundle.tar.gz
+```
+
+Go producer applications can use `pkg/upload` instead of constructing archives and HTTP requests directly. See [Upstream Producer Integration](consumers/api.md) for the copyable producer implementation guide.
+
+The maintained example client uses the local upload server, reads the token from `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN`, and defaults the pipeline id to `example-http-upload`. Set `DISTRIBUTOR_EXAMPLE_UPLOAD_PIPELINE_ID` or pass a second argument to use another configured upload pipeline. It generates an idempotency key by default; set `DISTRIBUTOR_EXAMPLE_UPLOAD_IDEMPOTENCY_KEY` when retrying the same producer run across separate process runs.
+
+```sh
+go run ./examples/upload-client
+```
+
 Accepted uploads return after the archive is staged and validated:
 
 ```json
@@ -154,6 +172,8 @@ Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Co
 
 Upload admission is bounded by `server.http.queue_size`. Publication concurrency is bounded by `server.http.max_concurrency`, and the coordinator does not run two uploads for the same pipeline at the same time.
 
+`Idempotency-Key` is optional for raw HTTP clients. When present, it is scoped to the token id, pipeline id, and key. Reusing the same key with the same normalized source manifest in that scope returns the original accepted run response and does not enqueue another run. Reusing the key with a different source manifest returns `409 Conflict`. If another request with the same key is still being staged before its manifest is known, the server returns a retryable `409 Conflict`. Idempotency records are memory-only and expire with completed upload status records.
+
 The upload server accepts `application/x-tar`, `application/gzip`, and `application/x-gzip`. Archives are extracted into a temporary staging directory, must contain exactly one root-level `manifest.json`, and must validate as one complete source bundle before a run id is issued. Per-source `max_upload_size` bounds both uploaded archive size and extracted bundle size. The implementation also caps extracted file count.
 
 The default bind address is private loopback. Put TLS, public routing, rate limiting, and external access policy in a reverse proxy or deployment layer.

docs/policy/architecture.md

@@ -197,6 +197,7 @@ Use this current layout unless the project has a documented reason to differ:
 
 - `cmd/distributor`: application entrypoint only.
 - `pkg/bundle`: public producer-facing source manifest model, digest logic, parsing, manifest building, complete local bundle writing, and local validation helpers.
+- `pkg/upload`: public producer-facing HTTP upload client built on `pkg/bundle`.
 - `internal/app`: application orchestration and top-level use cases.
 - `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
 - `internal/config`: configuration structs, defaults, loading, precedence, and validation.
@@ -323,9 +324,9 @@ Important tests include:
 
 Documentation should follow the project documentation policy. Keep user docs focused on implemented behavior. Put future, planned, or aspirational work only under `docs/roadmap/`.
 
-When changing architecture, config, CLI behavior, adapters, manifest/state contracts, transform behavior, publish behavior, or component contracts, update the relevant docs and examples in the same change.
+When changing architecture, config, CLI behavior, adapters, manifest/state contracts, transform behavior, publish behavior, public package/API behavior, or component contracts, update the relevant docs and examples in the same change.
 
-The source manifest and destination `.distributor.json` schemas should have canonical documentation once implemented. Example configs should be valid and load-tested where practical.
+The source manifest and destination `.distributor.json` schemas should have canonical documentation once implemented. Producer-facing package and API workflows belong under `docs/consumers/`. Example configs should be valid and load-tested where practical.
 
 ## Non-Goals
 

docs/policy/development.md

@@ -7,6 +7,7 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
 
 - `cmd/distributor`: executable entrypoint only.
 - `pkg/bundle`: public producer-facing source manifest and local bundle writer helpers.
+- `pkg/upload`: public producer-facing HTTP upload client built on `pkg/bundle`.
 - `internal/app`: top-level use cases for `run`, `validate`, and `inspect`.
 - `internal/cli`: standard-library command parsing, flags, help text, and command wiring.
 - `internal/config`: YAML configuration structs, loading, defaults, and validation.
@@ -22,13 +23,13 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
 - `internal/transform/markdown`: Markdown-to-HTML transform.
 - `internal/notify`: notification interface and current no-op notifier.
 - `internal/testutil`: shared test fixtures. Production code must not import this package.
-- `docs`: current user, operator, policy, internal, and roadmap documentation.
+- `docs`: current user, operator, consumer, integration, policy, internal, and roadmap documentation.
 - `examples`: copyable example configs and source bundles.
 
 Do not create new top-level package families such as public `pkg/...` packages
-beyond `pkg/bundle`, generic workflow containers, or service-specific adapter
-directories unless the architecture policy or a current roadmap explicitly
-calls for them.
+beyond `pkg/bundle` and `pkg/upload`, generic workflow containers, or
+service-specific adapter directories unless the architecture policy or a
+current roadmap explicitly calls for them.
 
 ## Common Commands
 
@@ -45,6 +46,7 @@ go test ./internal/config
 go test ./internal/cli ./internal/app
 go test ./internal/publish ./internal/state
 go test ./internal/transform/markdown
+go test ./pkg/bundle ./pkg/upload
 ```
 
 Run the CLI against an example config:
@@ -76,6 +78,7 @@ GOCACHE=/private/tmp/distributor-gocache GOMODCACHE=/private/tmp/distributor-gom
 - Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and implemented backend behavior unless the current task explicitly changes them.
 - Use `storage.DisplayPath`, `storage.StateFileName`, `storage.StatePath`, and `storage.ManagedBundleTargets` instead of duplicating those conventions.
 - Use `pkg/bundle` for normalized source manifest semantics. Internal packages should reach those rules through `internal/bundle` when they also need storage-backed bundle discovery or validation.
+- Keep `pkg/upload` as a producer-facing HTTP client. It should depend on `pkg/bundle` and standard HTTP/archive primitives, not on `internal/app`, `internal/ingest`, server config, storage backends, or destination state types.
 - Use `config.ValidatePublishTransformPolicy` for publish and transform policy combinations.
 - Do not import concrete transform implementations from `internal/publish`; app-level wiring owns transform registration.
 - Do not import `internal/testutil` from production code.
@@ -171,6 +174,7 @@ Test close to the behavior being changed:
 - Use `internal/app` and `internal/cli` tests for user-facing workflows.
 - Use `internal/testutil` for shared valid fixtures only; keep edge cases near the package under test.
 - Run `go test ./...` after cross-package changes or documentation/example changes tied to tests.
+- Run `go test ./pkg/bundle ./pkg/upload` after changing producer-facing bundle or upload APIs.
 
 Live integration tests must be opt-in and skipped during normal `go test ./...`
 unless their required environment variables are set. Test-only environment
@@ -207,5 +211,7 @@ Follow `docs/policy/documentation.md`.
 - Keep `docs/config.md` canonical for user-facing config reference.
 - Keep `docs/cli.md` canonical for command syntax and workflows.
 - Keep `docs/operations.md` canonical for operational and recovery behavior.
+- Keep `docs/consumers/` canonical for public package and consumer API workflows.
+- Keep `docs/integrations/` canonical for external file-format and wire-protocol contracts.
 - Keep `docs/internal/` focused on implemented package contracts.
-- Update docs in the same change as behavior when public behavior, config, CLI, examples, or internal contracts change.
+- Update docs in the same change as behavior when public behavior, public packages/APIs, config, CLI, examples, or internal contracts change.

docs/policy/documentation.md

@@ -2,12 +2,13 @@
 
 ## Purpose
 
-Project documentation must help four audiences:
+Project documentation must help five audiences:
 
 1. users who need to run the application;
 2. administrators/operators who need to configure and operate it;
 3. developers who need to understand and change it safely;
-4. LLM coding agents that need clear scope, boundaries, and invariants.
+4. LLM coding agents that need clear scope, boundaries, and invariants;
+5. developers and LLM coding agents integrating this project from another codebase.
 
 Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
 
@@ -46,7 +47,9 @@ Canonical homes:
 - CLI reference: `docs/cli.md`
 - operations and recovery: `docs/operations.md`
 - troubleshooting: `docs/troubleshooting.md`
+- public API/package consumer guidance: `docs/consumers/`
 - implemented internals: `docs/internal/`
+- external protocol, service, and file-format contracts: `docs/integrations/`
 - future work: `docs/roadmap/`
 - contributor workflow: `docs/policy/development.md`
 - copyable examples: `examples/`
@@ -119,6 +122,15 @@ Recommended:
 - `docs/troubleshooting.md`
 - validated examples under `examples/`
 
+### Project with public packages or consumer APIs
+
+Required:
+- `docs/consumers/api.md`
+- one `docs/consumers/pkg-<name>.md` file per public package, if public packages exist
+
+Recommended:
+- copyable consumer examples under `examples/`, if practical
+
 ## Required Documents
 
 ### README.md
@@ -244,6 +256,33 @@ Each entry should include:
 - safe fix;
 - relevant links.
 
+### docs/consumers/
+
+**Audience:** developers and LLM coding agents integrating this project from another codebase
+
+Required for projects with public packages, SDKs, client APIs, plugin APIs, or other application-facing integration surfaces.
+
+This directory describes how an external codebase should consume the project's public API. It should be task-oriented and copyable where useful. It is not the place for internal implementation details or operator procedures.
+
+`docs/consumers/api.md` should provide the consumer-facing overview and primary implementation workflow. It should include:
+
+1. intended consumer audience and use cases;
+2. required inputs supplied by operators or deployment configuration;
+3. recommended public package or API workflow;
+4. minimal copyable example;
+5. consumer responsibilities and boundaries;
+6. retry, idempotency, or status behavior, if applicable;
+7. links to package-specific docs and canonical integration contracts.
+
+Package-specific docs should be named `pkg-<name>.md` and should include:
+
+1. import path;
+2. intended use cases;
+3. primary types and functions needed by consumers;
+4. minimal examples;
+5. validation, error, retry, and boundary behavior;
+6. links to canonical file-format or wire-protocol contracts.
+
 ### docs/internal/
 
 **Audience:** developers, LLM coding agents
@@ -289,7 +328,7 @@ Roadmap docs should not be confused with current behavior.
 
 Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
 
-This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
+This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses or exposes.
 
 Use one file per integration where useful.
 
@@ -348,6 +387,7 @@ Before merging documentation changes, verify:
 - `docs/policy/architecture.md` describes development principles.
 - Future work appears only under `docs/roadmap/`.
 - User-facing docs avoid unnecessary internals.
+- Consumer-facing docs explain public APIs without duplicating integration contracts.
 - Developer-facing docs preserve boundaries and invariants.
 - Config examples match the schema.
 - CLI examples match real commands and flags.

docs/roadmap/api.md

@@ -0,0 +1,25 @@
+# API Roadmap
+
+This document records API work that is not part of the current implementation. Current HTTP upload behavior is documented in `docs/integrations/http-upload.md`, current configuration behavior is documented in `docs/config.md`, and current producer package usage is documented under `docs/consumers/`.
+
+## Deferred Upload API Work
+
+- Durable upload status storage.
+- Durable idempotency records across server restarts.
+- Run listing, retry, and cancellation endpoints.
+- Long-polling or wait-for-completion workflows.
+- Multipart, resumable, or streaming upload protocols.
+- Additional archive content negotiation beyond tar and gzip-compressed tar.
+- URL-token authentication for constrained clients.
+- Upload token lifecycle tooling.
+- Mutual TLS or other in-app identity mechanisms.
+- In-app TLS termination.
+- In-app public exposure policy.
+- In-app upload rate limiting.
+
+## Boundaries
+
+- Producers do not choose destination ids, destination paths, transforms, links, publish policy, transfer policy, or storage backends through upload requests.
+- Source manifests remain free of routing, destination, transform, and credential data.
+- `http_upload` remains source-only unless a future design changes that contract.
+- Public access policy, TLS termination, and rate limiting belong in deployment infrastructure unless a future design changes that boundary.

docs/roadmap/documentation.md

@@ -1,97 +0,0 @@
-# Documentation Roadmap
-
-## Purpose
-
-This roadmap tracks the remaining work required to verify that project documentation complies with `docs/policy/documentation.md` and accurately reflects the current implementation.
-
-The documentation migration has rewritten the current user, operator, integration, and internal component docs. This file now records only remaining validation work. Current behavior belongs outside `docs/roadmap/`; deferred or unimplemented work belongs under `docs/roadmap/`.
-
-## Current Documentation Set
-
-Current documentation outside roadmap:
-
-- `README.md`: concise project orientation and quickstart.
-- `docs/cli.md`: canonical CLI command, flag, workflow, and output reference.
-- `docs/config.md`: canonical YAML configuration reference.
-- `docs/operations.md`: operating, safety, state, upload, and recovery guidance.
-- `docs/troubleshooting.md`: symptom-oriented diagnostic and safe-fix guide.
-- `docs/policy/architecture.md`: architecture and invariant policy.
-- `docs/policy/development.md`: contributor and coding workflow policy.
-- `docs/policy/documentation.md`: controlling documentation policy.
-- `docs/integrations/*.md`: implemented external/file-format/protocol contracts.
-- `docs/internal/*.md`: implemented internal component contracts.
-- `examples/*.yml` and `examples/source-bundle/*`: maintained example configs and source bundle fixture.
-
-Current roadmap files:
-
-- `docs/roadmap/documentation.md`: this remaining documentation validation plan.
-- `docs/roadmap/http.md`: deferred HTTP upload extensions only.
-
-Removed completed roadmap artifacts:
-
-- `docs/roadmap/audit.md`
-- `docs/roadmap/cleanup.md`
-- `docs/roadmap/implementation.md`
-
-## Remaining Documentation Validation
-
-Goal: verify the rewritten docs against tests, examples, code, links, and the documentation policy checklist.
-
-Files to create, update, delete, or move: fixes only if validation finds gaps.
-
-Repository areas to inspect:
-
-- `README.md`
-- `docs/cli.md`
-- `docs/config.md`
-- `docs/operations.md`
-- `docs/troubleshooting.md`
-- `docs/internal/`
-- `docs/integrations/`
-- `docs/policy/`
-- `examples/`
-- CLI parser code under `internal/cli`
-- config loading/defaulting/validation under `internal/config`
-- app/report/upload behavior under `internal/app`
-- source bundle, state, publish, storage, adapter, and transform packages
-
-Acceptance criteria:
-
-- Tests pass for the full repository.
-- Maintained example configs load.
-- CLI examples and flags match parser behavior.
-- Config fields and defaults match `internal/config`.
-- Operations and troubleshooting docs describe implemented behavior only.
-- Internal docs preserve package boundaries and policy-required sections.
-- Integration docs describe only implemented contracts.
-- Roadmap files contain only remaining or deferred work.
-- Links resolve.
-- No secrets or private data are present.
-
-Suggested validation commands:
-
-```sh
-go test ./...
-go test ./internal/config ./internal/cli ./internal/app
-go test ./pkg/bundle ./internal/bundle ./internal/state ./internal/publish ./internal/storage ./internal/storage/fake
-go test ./internal/adapters/local ./internal/adapters/ssh ./internal/adapters/s3 ./internal/ingest ./internal/transform/markdown
-rg -n -i "future|planned|deferred|experimental|deprecated|not implemented|old behavior" README.md docs --glob '!docs/roadmap/**' --glob '!docs/policy/**'
-rg -n "\\b(Stages?|Phases?)\\b" README.md docs --glob '!docs/roadmap/**' --glob '!docs/policy/**'
-rg -n -- "--config|--dry-run|--force|--format|--pipeline|--bundle|--id|--file|--created|--overwrite" docs/cli.md internal/cli
-rg -n "examples/" README.md docs examples internal/config/load_test.go
-```
-
-Manual review items:
-
-- Confirm README remains concise and orientation-focused.
-- Confirm `docs/config.md` is the only full config field/default reference.
-- Confirm `docs/cli.md` is the only full command/flag reference.
-- Confirm `docs/operations.md` focuses on operating and recovery.
-- Confirm `docs/troubleshooting.md` remains symptom-first.
-- Confirm `docs/internal/` describes implemented component contracts and boundaries.
-- Confirm integration docs do not claim support for unimplemented external features.
-- Confirm examples contain no secrets and distinguish local runnable examples from environment-gated remote examples.
-
-## Open Questions
-
-No open questions block the remaining validation work.

docs/roadmap/future.md

@@ -0,0 +1,80 @@
+# Future Roadmap
+
+This document records planned or deferred work that is not part of the current
+implementation. Current behavior is documented outside roadmap files in the
+README, integration contracts, operations guide, troubleshooting guide, and
+internal docs.
+
+## Durability And Recovery
+
+- Durable upload status persistence across process restarts.
+- Durable idempotency records across server restarts.
+- Database-backed upload queueing.
+- Recovery semantics for queued or running uploads after a restart.
+- Durable producer retry processing.
+
+## Producer Client Workflows
+
+- Durable client queues or background producer workers.
+- `UploadAndWait` helper.
+- Long-polling helper or equivalent wait workflow.
+
+## Run Control APIs
+
+- Run retry endpoints.
+- Run cancellation endpoints.
+- Run listing endpoints.
+
+## Archive And Transport Protocols
+
+- Zstandard-compressed tar archives.
+- Additional content negotiation rules for future archive formats.
+- Multipart upload support.
+- Resumable upload support.
+- Streaming upload protocols.
+
+## Destination Backends
+
+- GitHub Gist destination backend support.
+- Authentication and secret handling for GitHub API tokens.
+- Gist file mapping, update, replacement, and conflict semantics.
+- Rate-limit handling and retry behavior for GitHub API responses.
+
+## Docker Image Support
+
+- Official container image build and release workflow.
+- Runtime filesystem layout for config, secrets, staging, and local outputs.
+- Container-oriented examples for `run` and `serve`.
+- Image tagging, versioning, and upgrade guidance.
+
+## Notifications And Hooks
+
+- Email notification support for completed, failed, or partially failed
+  distribution runs.
+- SMTP configuration, authentication, secret handling, and recipient policy.
+- ntfy notification support for completed, failed, or partially failed
+  distribution runs.
+- ntfy topic, server, token, priority, and action configuration.
+- General post-distribution hook support.
+- Hook payload contract that can pass run status, summaries, destination
+  outcomes, output metadata, and public links to external tools.
+- Local executable hook adapter with bounded arguments, environment, stdin,
+  timeout, exit-code handling, and secret-redaction behavior.
+
+## Authentication And Deployment Surface
+
+- URL-token authentication for constrained clients.
+- Additional token lifecycle tooling.
+- Mutual TLS or other in-app identity mechanisms.
+- In-app TLS.
+- Public exposure defaults.
+- Browser UI.
+
+## Boundaries
+
+- `http_upload` remains source-only unless a future implementation changes that
+  contract.
+- Current upload status, queue, and idempotency state are memory-only.
+- Producers submit complete tar or gzip-compressed tar source bundles today.
+- Public access policy, TLS termination, and rate limiting belong outside
+  `distributor` unless a future implementation changes that boundary.

docs/roadmap/http.md

@@ -1,61 +0,0 @@
-# Roadmap: HTTP Upload Extensions
-
-## Purpose
-
-The HTTP upload API is implemented. Current behavior is documented in:
-
-- [CLI](../cli.md)
-- [Configuration](../config.md)
-- [Operations](../operations.md)
-- [Troubleshooting](../troubleshooting.md)
-- [HTTP upload contract](../integrations/http-upload.md)
-- [Application internals](../internal/app.md)
-- [Ingestion internals](../internal/ingest.md)
-
-This roadmap records HTTP upload extensions that are intentionally not part of
-the current implementation.
-
-## Deferred Extensions
-
-### Authentication
-
-- URL-token authentication for constrained clients.
-- Additional token lifecycle tooling.
-- Mutual TLS or other in-app identity mechanisms.
-
-### Archive Formats
-
-- Zstandard-compressed tar archives.
-- Additional content negotiation rules for future archive formats.
-
-### Status And Queue Durability
-
-- Durable status persistence across process restarts.
-- Database-backed queueing.
-- Recovery semantics for queued or running uploads after a restart.
-
-### Producer Coordination
-
-- Producer-supplied idempotency keys.
-- Run retry endpoints.
-- Run cancellation endpoints.
-- Run listing endpoints.
-
-### Deployment Surface
-
-- In-app TLS.
-- Public exposure defaults.
-- Browser UI.
-
-## Boundaries
-
-Current HTTP upload behavior remains intentionally small:
-
-- `http_upload` is source-only and is not a durable storage backend.
-- Upload status is memory-only.
-- Producers submit complete tar or gzip-compressed tar source bundles.
-- Producers authenticate with `Authorization: Bearer <token>`.
-- Public access policy, TLS termination, and rate limiting belong outside
-  `distributor` unless a future roadmap explicitly changes that boundary.
-
-Do not document deferred extensions as available outside `docs/roadmap/`.

docs/roadmap/implementation.md

@@ -0,0 +1,628 @@
+# Reconciliation, Shared-Root, Reconcile-State, And Prune Implementation Plan
+
+This document is the staged implementation plan for the feature roadmaps in:
+
+- `docs/roadmap/reconciliation.md`
+- `docs/roadmap/multipipeline.md`
+- `docs/roadmap/reconcile-state.md`
+- `docs/roadmap/prune.md`
+
+Audience: LLM coding agents and maintainers implementing these features in
+order.
+
+This is roadmap guidance, not current behavior documentation. Do not describe
+any stage as implemented outside `docs/roadmap/` until that stage has landed.
+
+## Global Decisions
+
+- Implement in this order:
+  1. destination reconciliation modes;
+  2. shared-root multi-pipeline state;
+  3. reconcile-state tooling;
+  4. pruning.
+- Keep reconciliation mode per destination as `reconciliation.mode`.
+- Use reconciliation modes `replace` and `merge`; default is `replace`.
+- Persist reconciliation metadata in `.distributor.json`.
+- Keep shared-root behavior as an explicit per-destination opt-in using
+  `state.mode: single_owner|shared_root`.
+- Key shared-root output ownership by both `pipeline_id` and `destination_id`.
+- Do not allow implicit takeover of another owner path.
+- Store compact per-output source identity fields, and keep the latest full
+  source manifest once per owner where publish comparison needs it.
+- `reconcile-state` applies repairs by default and uses `--dry-run` for
+  read-only inspection.
+- `prune` is opt-in, uses configured retention policy, and requires `--apply`
+  for deletion.
+- Use per-output `updated_at` for initial prune policies.
+- Support both `older_than` and `keep_latest` pruning in the initial prune
+  implementation.
+- Write single-owner reconciliation-capable state as schema version `2`.
+- Write shared-root collection state as schema version `3`.
+- Do not adopt unmanaged files, validate output digests, or perform
+  whole-config reconciliation in the initial reconcile-state command.
+- Do not run pruning automatically after publish in the initial implementation.
+
+## Cross-Cutting Constraints
+
+- Follow `docs/policy/architecture.md`, `docs/policy/development.md`, and
+  `docs/policy/documentation.md`.
+- Keep producer manifests free of destination routing, reconciliation, state,
+  retention, transform, link, and backend policy.
+- Keep adapter packages thin. State comparison, reconciliation, pruning, and
+  retention decisions belong in `internal/state`, `internal/publish`, and
+  `internal/app`, not concrete backends.
+- Keep public packages `pkg/bundle` and `pkg/upload` unaffected unless an
+  implementation stage explicitly requires consumer-facing docs updates.
+- Use dry-run behavior for every workflow that may write state or delete files.
+- Never delete unmanaged content.
+- Use `storage.StateFileName`, `storage.StatePath`,
+  `storage.ManagedBundleTargets`, and storage path validation helpers instead
+  of duplicating path rules.
+- Update implemented-behavior docs and examples in the same change as the
+  corresponding behavior. Keep future behavior in roadmap docs only.
+- A feature group is not complete until its paired documentation stage lands.
+  Do not move the roadmap item out of future/planned status until the behavior,
+  tests, current-behavior docs, and examples for that group are complete.
+- Run the stage-specific tests before moving to the next stage. Run
+  `go test ./...` after each completed feature group.
+
+## Stage 1: Reconciliation Config And State Foundation
+
+Goal: add destination-level reconciliation configuration and a state model that
+can persist reconciliation metadata and cumulative outputs for single-owner
+destinations.
+
+Implementation:
+
+- Add config types:
+  - `Destination.Reconciliation ReconciliationPolicy`
+  - `ReconciliationPolicy.Mode string`
+- Add constants in `internal/config/defaults.go`:
+  - `ReconciliationModeReplace = "replace"`
+  - `ReconciliationModeMerge = "merge"`
+- Default `reconciliation.mode` to `replace` for every destination.
+- Validate `reconciliation.mode` as `replace` or `merge`.
+- Add focused config tests for defaults, explicit `replace`, explicit `merge`,
+  and invalid modes.
+- Extend `internal/state` for a new single-owner state schema that writes:
+  - `schema_version`;
+  - `distributor_version`;
+  - `pipeline_id`;
+  - `destination_id`;
+  - `published_at`;
+  - `created_at`;
+  - `updated_at`;
+  - `state.mode: "single_owner"`;
+  - `reconciliation.mode`;
+  - `source.manifest`;
+  - `links`;
+  - cumulative `outputs`.
+- Add per-output `created_at` and `updated_at` fields now. For current
+  single-owner output records written by this stage, set both timestamps when a
+  path is first created and preserve `created_at` when a path is updated.
+- Keep parsing current schema version 1 state for compatibility. When v1 state
+  is read, infer:
+  - `state.mode = "single_owner"`;
+  - `reconciliation.mode = "replace"`;
+  - `created_at = published_at`;
+  - `updated_at = published_at`;
+  - each output `created_at` and `updated_at` from `published_at`.
+- Write only the new schema for newly written state after this stage.
+- Keep validation strict: required timestamps, valid reconciliation mode, unique
+  output paths, valid links, valid digests, clean paths, and valid embedded
+  source manifest.
+- Add state helper functions for:
+  - finding output records by path;
+  - merging retained and newly planned output records;
+  - computing owner-scoped managed output paths for single-owner state;
+  - projecting publish outputs into state outputs with timestamps.
+
+Tests:
+
+```sh
+go test ./internal/config
+go test ./internal/state
+```
+
+Keep publish semantics unchanged in this stage. Only adapt current state
+construction to write schema version `2` with default `replace` metadata and
+timestamps.
+
+## Stage 2: Single-Owner Replace And Merge Publish Execution
+
+Goal: implement `reconciliation.mode` for current single-owner destinations.
+
+Implementation:
+
+- Add `Reconciliation config.ReconciliationPolicy` to `publish.Request` and
+  `publish.Plan`.
+- Thread destination reconciliation config from `internal/app` into
+  `internal/publish`.
+- Keep `replace` behavior equivalent to current behavior:
+  - delete prior managed outputs absent from the new plan;
+  - write the new outputs;
+  - write state whose `outputs` are exactly the new planned output set.
+- Implement `merge` behavior:
+  - inspect existing state before writing;
+  - retain previous managed outputs not overwritten by the new plan;
+  - allow overwrite only when the destination path is already recorded in
+    existing state as managed;
+  - fail if a newly planned path exists in storage but is not recorded as
+    managed;
+  - write new or overwritten outputs;
+  - write state with the cumulative managed output set;
+  - derive `links.primary_url` from newly planned outputs only;
+  - clean up only outputs written by the failed attempt.
+- For `merge`, update per-output timestamps:
+  - new output path: `created_at = updated_at = now`;
+  - overwritten managed path: preserve existing `created_at`, set
+    `updated_at = now`;
+  - retained output: preserve both timestamps.
+- Ensure state writes use overwrite semantics appropriate for replacing the
+  existing `.distributor.json`, while output writes remain narrow and explicit.
+- Preserve existing force behavior. Do not add unmanaged-content adoption.
+
+Tests:
+
+```sh
+go test ./internal/publish ./internal/state
+go test ./internal/app ./internal/cli
+go test ./...
+```
+
+Required coverage:
+
+- `replace` deletes omitted managed outputs.
+- `merge` retains omitted managed outputs.
+- `merge` overwrites an existing managed path.
+- `merge` fails on unmanaged destination path collision.
+- failed merge cleanup deletes only attempted new writes.
+- fixed-path destinations support both modes.
+- source-only, generated-only, and source-plus-generated policies honor both
+  modes.
+
+## Stage 3: Reconciliation Documentation And Examples
+
+Goal: make implemented reconciliation behavior visible outside roadmap docs.
+The reconciliation feature group is not complete until this stage lands.
+
+Implementation:
+
+- Update `docs/config.md` with `reconciliation.mode`, defaults, and examples.
+- Update `docs/operations.md` with replace versus merge behavior, dry-run
+  expectations, failed-write cleanup, and recovery notes.
+- Update `docs/integrations/destination-state.md` with the new state schema,
+  reconciliation metadata, output timestamps, and v1 compatibility behavior.
+- Update `docs/internal/state.md` and `docs/internal/publish.md`.
+- Update relevant examples under `examples/` only if they are valid and
+  executable for implemented behavior.
+
+Tests:
+
+```sh
+go test ./internal/config
+go test ./...
+```
+
+## Stage 4: Shared-Root State Model And Config
+
+Goal: add explicit shared-root destination state support without changing
+publish behavior yet.
+
+Implementation:
+
+- Add destination state config:
+  - `Destination.State StatePolicy`
+  - `StatePolicy.Mode string`
+  - accepted values: `single_owner`, `shared_root`
+  - default: `single_owner`
+- Validate state mode values.
+- Add a shared-root state schema in `internal/state`.
+- The shared-root JSON must include:
+  - `schema_version`;
+  - `distributor_version`;
+  - `created_at`;
+  - `updated_at`;
+  - `state.mode: "shared_root"`;
+  - owner records keyed by `pipeline_id` and `destination_id`;
+  - latest full source manifest per owner for comparison;
+  - owner-level reconciliation metadata;
+  - owner-level latest `links.primary_url`, if present;
+  - output records for every managed path in the root.
+- Shared-root output records must include:
+  - path;
+  - kind;
+  - source path;
+  - transform, when generated;
+  - URL, when present;
+  - SHA-256;
+  - size;
+  - owner `pipeline_id`;
+  - owner `destination_id`;
+  - source manifest id;
+  - source manifest digest;
+  - source manifest created timestamp;
+  - `created_at`;
+  - `updated_at`.
+- Keep compact per-output source identity fields. Do not repeat full source
+  manifests on every output.
+- Add state helpers that can:
+  - parse both single-owner and shared-root state;
+  - identify the current owner scope;
+  - return the latest source manifest for one owner;
+  - list managed paths for one owner or all owners;
+  - detect path ownership conflicts;
+  - merge one owner's planned output state while preserving unrelated owners;
+  - remove one owner's omitted outputs for owner-scoped `replace`.
+- Do not automatically convert unrelated single-owner state to shared-root
+  state. If shared-root is configured and existing single-owner state matches
+  the current owner, convert it during the first successful shared-root publish.
+  Otherwise fail with an actionable conflict.
+
+Tests:
+
+```sh
+go test ./internal/config
+go test ./internal/state
+```
+
+Do not enable shared-root publish behavior until Stage 5.
+
+## Stage 5: Shared-Root Publish Planning And Comparison
+
+Goal: teach publish planning to reason about shared-root owner scopes without
+writing shared-root outputs yet.
+
+Implementation:
+
+- Thread destination state mode from `internal/app` into `internal/publish`.
+- Update destination inspection and comparison so:
+  - `single_owner` destinations preserve current single-owner comparison
+    semantics;
+  - `shared_root` destinations compare only the current
+    `pipeline_id`/`destination_id` owner scope;
+  - absent owner state behaves like destination absent for that owner, unless
+    unmanaged storage content collides with a planned output;
+  - outputs owned by other owners are preserved.
+- Enforce ownership conflict rules:
+  - same `pipeline_id` and `destination_id` may overwrite its own managed path;
+  - different owner writing the same path fails as a conflict;
+  - unmanaged storage path collision fails by default;
+  - no implicit takeover.
+- Compose reconciliation within the owner scope:
+  - owner-scoped `replace` removes prior outputs for that owner that are absent
+    from the new plan;
+  - owner-scoped `merge` retains prior outputs for that owner that are absent
+    from the new plan.
+- Return plans that include the current owner scope, retained owner outputs,
+  owner outputs to delete, and owner outputs to write.
+- Keep execution disabled for shared-root write actions until Stage 6.
+
+Tests:
+
+```sh
+go test ./internal/publish ./internal/state
+go test ./internal/app
+```
+
+Required coverage:
+
+- planning treats absent owner state as publishable for that owner;
+- planning preserves other owners in the planned retained state;
+- path conflict with another owner plans or returns a conflict failure;
+- unmanaged path collision plans or returns an unmanaged-content failure;
+- owner-scoped `replace` selects only that owner's omitted outputs for deletion;
+- owner-scoped `merge` retains that owner's omitted outputs;
+- current single-owner planning remains covered.
+
+## Stage 6: Shared-Root Publish Execution
+
+Goal: allow multiple pipelines to publish disjoint managed paths into one
+shared destination root.
+
+Implementation:
+
+- Preserve unrelated owners and their outputs exactly.
+- Preserve and update timestamps:
+  - top-level `created_at` remains the original root creation time;
+  - top-level `updated_at` changes on successful state writes;
+  - output `created_at` is stable;
+  - output `updated_at` changes only when that path is rewritten;
+  - owner metadata updates when that owner publishes.
+- Keep failed-write cleanup limited to outputs written by the failed attempt.
+- Keep forced replacement bounded and explicitly reported. Forced replacement
+  for a shared-root destination may delete the configured destination root, so
+  retain existing dry-run visibility and do not broaden its scope.
+
+Tests:
+
+```sh
+go test ./internal/publish ./internal/state
+go test ./internal/app ./internal/cli
+go test ./...
+```
+
+Required coverage:
+
+- two pipelines publish disjoint paths into the same shared root;
+- one pipeline replaces its own output without deleting another owner's output;
+- one pipeline merges new outputs while preserving its own retained outputs and
+  other owners' outputs;
+- path conflict with another owner fails;
+- unmanaged path collision fails;
+- shared-root dry-run writes no outputs or state;
+- current single-owner behavior remains covered.
+
+## Stage 7: Shared-Root Documentation And Examples
+
+Goal: document implemented shared-root behavior.
+The shared-root feature group is not complete until this stage lands.
+
+Implementation:
+
+- Update `docs/config.md` with destination state mode config and defaults.
+- Update `docs/integrations/destination-state.md` with shared-root schema,
+  owners, output ownership, timestamps, and migration rules.
+- Update `docs/operations.md` with shared-root publishing, conflicts, dry-run,
+  and recovery behavior.
+- Update `docs/troubleshooting.md` with ownership conflict and unmanaged
+  collision entries.
+- Update `docs/internal/state.md` and `docs/internal/publish.md`.
+- Add maintained examples only when they are valid and load-tested.
+
+Tests:
+
+```sh
+go test ./internal/config
+go test ./...
+```
+
+## Stage 8: Reconcile-State Planning Core
+
+Goal: add app-level planning and state helpers for reconcile-state without
+adding CLI execution first.
+
+Implementation:
+
+- Add `internal/app` reconcile-state use case types:
+  - options with config path, pipeline id, destination id, all-owners flag,
+    dry-run flag, stdout, and output format;
+  - report struct with destination identity, backend, root path, state schema,
+    owner scope, checked count, missing managed outputs, unmanaged entries, and
+    changed/would-change status.
+- Add state helpers for removing missing managed output records:
+  - single-owner scope;
+  - shared-root current-owner scope;
+  - shared-root all-owner scope.
+- Use storage `Stat` for managed output existence checks.
+- Use bounded `Walk` under the selected destination root to report unmanaged
+  existing files. Exclude `.distributor.json` and all paths already recorded as
+  managed.
+- Do not validate output digests.
+- Do not delete destination files.
+- Do not adopt unmanaged files.
+- Do not rewrite invalid or ambiguous state.
+- Require enough scope to identify one destination root. For current
+  single-owner state, require pipeline and destination. For shared-root
+  all-owner mode, still require a pipeline/destination selector to identify the
+  configured destination root, then apply `--all-owners` inside that root.
+- Default behavior applies state repair. `DryRun` reports without writing.
+
+Tests:
+
+```sh
+go test ./internal/app ./internal/state ./internal/storage/fake
+```
+
+Required coverage:
+
+- dry-run reports missing managed outputs without rewriting state;
+- apply removes missing records and writes valid state;
+- unmanaged files are reported but not adopted or deleted;
+- invalid state fails without rewrite;
+- shared-root owner-scope and all-owner repair both work.
+
+## Stage 9: Reconcile-State CLI And Docs
+
+Goal: expose reconcile-state as an operator command.
+The reconcile-state feature group is not complete until this stage lands.
+
+Implementation:
+
+- Add `reconcile-state` to `internal/cli/root.go` help and dispatch.
+- Add CLI flags:
+  - `--config <path>`;
+  - `--pipeline <id>`;
+  - `--destination <id>`;
+  - `--all-owners`;
+  - `--dry-run`;
+  - `--format text|json`.
+- Without `--dry-run`, apply state repairs.
+- Text output should clearly report whether state was changed or would be
+  changed.
+- JSON output should use the existing app JSON envelope pattern.
+- Update `docs/cli.md`, `docs/operations.md`, `docs/troubleshooting.md`,
+  `docs/integrations/destination-state.md`, `docs/internal/state.md`,
+  `docs/internal/storage.md`, and `docs/internal/app.md`.
+
+Tests:
+
+```sh
+go test ./internal/cli ./internal/app
+go test ./...
+```
+
+Required coverage:
+
+- flag validation;
+- default apply behavior;
+- `--dry-run` behavior;
+- text and JSON reports;
+- no deletion of destination files.
+
+## Stage 10: Prune Config And Planning Core
+
+Goal: add retention config and pure prune planning.
+
+Implementation:
+
+- Add destination retention config:
+  - `Destination.Retention RetentionPolicy`
+  - `RetentionPolicy.Prune PrunePolicy`
+  - `PrunePolicy.Enabled bool`
+  - `PrunePolicy.OlderThan *Duration`
+  - `PrunePolicy.KeepLatest *int`
+- Default pruning disabled.
+- Validate:
+  - enabled prune policy must set at least one of `older_than` or
+    `keep_latest`;
+  - `older_than` must be positive;
+  - `keep_latest` must be zero or greater;
+  - documented duration units must parse consistently with existing duration
+    config behavior.
+- Add app/state prune planning helpers:
+  - select owner-scoped managed outputs;
+  - sort deterministically by `updated_at`, then path;
+  - preserve the newest `keep_latest` outputs;
+  - delete remaining outputs older than `older_than`;
+  - if both policies are set, first preserve `keep_latest`, then delete
+    remaining outputs older than `older_than`.
+- Use per-output `updated_at` as the timestamp basis.
+- Plan only; do not delete storage files in this stage.
+
+Tests:
+
+```sh
+go test ./internal/config
+go test ./internal/app ./internal/state
+```
+
+Required coverage:
+
+- config defaults and validation;
+- `older_than` planning;
+- `keep_latest` planning;
+- combined policy planning;
+- deterministic tie-breaking;
+- shared-root owner scope preservation.
+
+## Stage 11: Prune Execution Core
+
+Goal: implement safe managed-output pruning below the CLI layer.
+
+Implementation:
+
+- Add app prune execution:
+  - load config;
+  - resolve selected destination and owner scope;
+  - read and validate state;
+  - build prune plan from configured retention policy;
+  - in dry-run, write no files and delete nothing;
+  - in apply mode, delete only planned managed output paths;
+  - after confirmed deletes, remove deleted records from state and update state
+    timestamps;
+  - preserve state records for failed deletes so retry remains accurate;
+  - never delete unmanaged files;
+  - do not delete `.distributor.json` unless a later roadmap explicitly permits
+    empty-state removal.
+- Do not add CLI dispatch in this stage.
+- Return a prune report that can later be rendered by text and JSON CLI output.
+- Do not add one-off retention overrides in the initial implementation.
+- Let local and SSH/SFTP backend helpers prune empty directories only where
+  they already do so safely. Preserve object-storage prefix markers unless they
+  are managed output records.
+
+Tests:
+
+```sh
+go test ./internal/config
+go test ./internal/app ./internal/state
+go test ./internal/storage/fake
+```
+
+Required coverage:
+
+- dry-run reports deletes without deleting files or rewriting state;
+- apply deletes only managed output paths;
+- unmanaged files under the destination root are preserved;
+- partial delete failure preserves accurate state for confirmed deleted and
+  undeleted outputs;
+- shared-root pruning preserves other owners when scoped to one owner.
+
+## Stage 12: Prune CLI And Docs
+
+Goal: expose safe managed-output pruning as an operator command. The prune
+feature group is not complete until this stage lands.
+
+Implementation:
+
+- Add `prune` CLI dispatch and help.
+- Add CLI flags:
+  - `--config <path>`;
+  - `--pipeline <id>`;
+  - `--destination <id>`;
+  - `--dry-run`;
+  - `--apply`;
+  - `--format text|json`.
+- Require exactly one of `--dry-run` or `--apply`. `--dry-run` plans and
+  reports without writing. `--apply` deletes planned managed outputs and
+  rewrites state after confirmed deletes.
+- JSON output should use the existing app JSON envelope pattern.
+- Update `docs/config.md`, `docs/cli.md`, `docs/operations.md`,
+  `docs/troubleshooting.md`, `docs/integrations/destination-state.md`,
+  `docs/internal/state.md`, `docs/internal/publish.md`, and
+  `docs/internal/app.md`.
+
+Tests:
+
+```sh
+go test ./internal/config
+go test ./internal/app ./internal/cli ./internal/state
+go test ./internal/storage/fake
+go test ./...
+```
+
+Required coverage:
+
+- CLI requires exactly one of `--dry-run` or `--apply`;
+- text and JSON reports;
+- CLI apply gating and report output.
+
+## Final Verification
+
+After all stages land:
+
+```sh
+go test ./internal/config
+go test ./internal/state ./internal/publish
+go test ./internal/app ./internal/cli
+go test ./internal/storage/fake
+go test ./pkg/bundle ./pkg/upload
+go test ./...
+```
+
+Also run:
+
+```sh
+rg -n "reconciliation|shared_root|single_owner|reconcile-state|retention|prune|older_than|keep_latest" README.md docs examples
+```
+
+Confirm:
+
+- current-behavior docs outside `docs/roadmap/` describe only implemented
+  behavior;
+- roadmap docs no longer describe completed behavior as future work, or are
+  revised to mark only remaining deferred work;
+- examples load successfully where config tests cover them;
+- producer-facing docs still keep destination routing, reconciliation, state,
+  and retention policy out of producer manifests and upload requests.
+
+## Non-Goals For This Plan
+
+- No unmanaged-file adoption workflow.
+- No digest-audit mode for reconcile-state.
+- No whole-config reconcile-state command.
+- No automatic post-publish pruning.
+- No path/date parsing retention policy.
+- No producer-facing retention, reconciliation, or destination routing API.
+- No broad storage synchronization behavior.

docs/roadmap/multipipeline.md

@@ -0,0 +1,195 @@
+# Multi-Pipeline Shared Root Roadmap
+
+This document records planned shared-root destination state work that is not
+part of the current implementation. Current destination state behavior is
+documented in `docs/integrations/destination-state.md`.
+
+This work should not be implemented until the reconciliation work in
+`docs/roadmap/reconciliation.md` is present and complete. Shared-root
+multi-pipeline publishing should then be implemented before reconcile-state
+tooling (`docs/roadmap/reconcile-state.md`) and pruning
+(`docs/roadmap/prune.md`) because both tools should operate on the final shared
+state model.
+
+## Problem
+
+Current `.distributor.json` state is a single-owner destination bundle record.
+It stores one `pipeline_id`, one `destination_id`, one latest source manifest,
+and one output set. That works when each pipeline writes to a separate
+destination bundle path.
+
+It does not work well when multiple pipelines intentionally publish into one
+shared destination root, for example:
+
+```text
+2026-06-08/daily/report.md
+2026-06-08/event/watch-123.md
+```
+
+If both pipelines resolve to the same destination bundle path, the second
+pipeline encounters a state identity mismatch even when its output paths are
+disjoint. Separating roots avoids the conflict, but can make retention,
+cleanup, linking, and human browsing less natural.
+
+## Goal
+
+Support multiple configured pipelines writing to one shared destination root
+while preserving distributor ownership, unmanaged-content safety, dry-run
+behavior, and deterministic conflict handling.
+
+The desired shape is a collection-oriented `.distributor.json` that records
+managed outputs from multiple owners under one state file. Producers still
+submit source bundles only. Producers must not choose destinations,
+reconciliation mode, retention policy, transforms, links, or storage backends.
+
+Shared-root behavior should be an explicit per-destination state mode, for
+example `state.mode: single_owner|shared_root` or a similarly named field under
+the destination. Shared-root behavior changes the meaning of `.distributor.json`,
+so it should be visible at the destination boundary rather than inferred from
+other configured pipelines.
+
+## State Model
+
+Add a new destination state schema that can represent a managed collection
+rather than one latest source bundle.
+
+State shape:
+
+- top-level `schema_version`;
+- top-level `distributor_version`;
+- top-level `created_at` for the first time this managed root was created;
+- top-level `updated_at` for the last successful state update;
+- top-level collection metadata identifying the state as shared-root capable;
+- an `outputs` array containing all managed output records in the shared root;
+- per-output owner metadata:
+  - `pipeline_id`;
+  - `destination_id`;
+  - source manifest id;
+  - source manifest digest;
+  - source manifest created timestamp;
+  - output kind, source path, transform, URL, SHA-256, and size;
+  - `created_at` for first publication of that output path;
+  - `updated_at` for the most recent write of that output path.
+- optional source-manifest history keyed by source digest, if compact per-output
+  source identity fields are not enough for later diagnostics or tooling.
+
+Output ownership is keyed by both `pipeline_id` and `destination_id`. The
+architecture treats destinations as independent policy boundaries, and the same
+pipeline may publish different output sets to different destinations.
+
+The state should be able to answer these questions without scanning the whole
+destination:
+
+- Which paths are distributor-managed?
+- Which pipeline/destination/source currently owns each managed path?
+- Which outputs belong to a specific pipeline/destination publish operation?
+- Which outputs are old enough, or least recently updated enough, for later
+  pruning decisions?
+- Which paths are absent from storage and should be removed by reconcile-state
+  tooling?
+
+## Ownership And Conflict Rules
+
+Shared-root support must keep ownership explicit.
+
+Required rules:
+
+- A publish may write only output paths planned from its configured pipeline and
+  destination.
+- A publish may overwrite an existing managed output path only when ownership
+  rules allow it.
+- A new output path that exists in storage but is not recorded in
+  `.distributor.json` remains unmanaged content and fails by default.
+- Disjoint outputs from different pipelines can coexist in one state file.
+- State updates must preserve unrelated managed outputs from other pipelines.
+- Failed-write cleanup must delete only paths written by the failed attempt.
+- Forced replacement must remain explicit, dry-runnable, and bounded to the
+  configured destination root.
+
+Overwrite rule: an output path may be overwritten by the same `pipeline_id` and
+`destination_id`; a different pipeline or destination writing the same path
+fails as an ownership conflict unless a later explicit operator takeover
+workflow is implemented.
+
+## Planning And Execution Work
+
+Implementation work:
+
+- Introduce a new state schema and parser that can represent shared ownership.
+- Keep current schema parsing available long enough to migrate or reject old
+  state with clear errors.
+- Add explicit per-destination configuration that opts a destination into
+  shared-root behavior.
+- Thread shared-root state semantics through destination inspection,
+  comparison, publish planning, and execution.
+- Make comparison operate on the subset of state owned by the current
+  pipeline/destination where appropriate.
+- Make output collision detection check both newly planned outputs and existing
+  managed outputs from other owners.
+- Write state atomically with preserved unrelated owners and updated records for
+  the current publish.
+- Preserve merge and replace semantics within the current owner scope:
+  - owner-scoped `replace` removes prior outputs for that owner that are absent
+    from the new plan;
+  - owner-scoped `merge` retains prior outputs for that owner that are absent
+    from the new plan.
+- Ensure local, SSH/SFTP, S3, and fake backends support the necessary state read,
+  write, overwrite, and cleanup paths.
+
+## Migration And Compatibility
+
+The initial implementation should prefer explicit safety over automatic
+conversion.
+
+Migration behavior:
+
+- A destination configured for shared-root behavior may read current single-owner
+  state and convert it only when the existing state identity matches the current
+  pipeline/destination or when an explicit migration command/flag is used.
+- A destination not configured for shared-root behavior continues to use the
+  current single-owner state semantics.
+- Invalid or ambiguous state fails with an actionable error and a dry-run report
+  path.
+
+## Testing
+
+Important tests:
+
+- Two pipelines publish disjoint paths into the same destination root.
+- One pipeline can replace its own older output without deleting another
+  pipeline's output.
+- One pipeline can merge new outputs while retaining its prior outputs and
+  another pipeline's outputs.
+- A pipeline cannot silently overwrite another pipeline's managed path.
+- A pipeline cannot claim unmanaged storage content by default.
+- State timestamps are set on creation and updated on subsequent writes.
+- Per-output timestamps preserve original `created_at` and update `updated_at`
+  only when that output path is rewritten.
+- Failed writes clean up only attempted outputs.
+- Dry-run reports shared-root actions without writing outputs or state.
+- Forced replacement remains bounded and explicitly reported.
+- Current single-owner state behavior remains covered for non-shared
+  destinations.
+
+## Documentation Work
+
+When implemented, update current-behavior docs in the same change:
+
+- `docs/integrations/destination-state.md`: document the new schema and
+  ownership rules.
+- `docs/config.md`: document the shared-root opt-in configuration.
+- `docs/operations.md`: explain shared-root publishing, conflicts, and recovery.
+- `docs/troubleshooting.md`: add shared-root ownership conflict entries.
+- `docs/internal/state.md` and `docs/internal/publish.md`: record the new
+  planning and state invariants.
+- Maintained examples that demonstrate date-first shared roots only after the
+  behavior is implemented.
+
+## Boundaries
+
+- Do not let producers route to destinations or choose shared-root behavior.
+- Do not add prune behavior in this work.
+- Do not add broad unmanaged-content adoption in this work.
+- Do not make shared-root behavior implicit for all destinations.
+- Do not use storage listings as the primary source of truth during normal
+  publishing; `.distributor.json` remains the managed ownership index.

docs/roadmap/prune.md

@@ -0,0 +1,183 @@
+# Prune Roadmap
+
+This document records planned destination pruning work that is not part of the
+current implementation. Current cleanup and forced replacement behavior is
+documented in `docs/operations.md`.
+
+This work should be implemented after the reconciliation work in
+`docs/roadmap/reconciliation.md`, shared-root state work in
+`docs/roadmap/multipipeline.md`, and reconcile-state tooling in
+`docs/roadmap/reconcile-state.md`. Prune decisions should rely on accurate
+managed state, explicit ownership, and timestamp metadata.
+
+## Problem
+
+Merge reconciliation and shared-root publishing allow long-lived destinations
+to accumulate distributor-managed outputs. That is useful for archives and
+date-first report layouts, but those destinations can grow without bound.
+
+Operators need a safe way to delete old distributor-managed outputs according
+to explicit retention policy without touching unmanaged files or outputs owned
+by unrelated scopes.
+
+## Goal
+
+Add pruning support for managed destination outputs.
+
+Initial pruning should support both of these policy families:
+
+- keep only the most recently updated N files in a configured scope;
+- delete managed files older than a configured duration.
+
+Pruning must be based on `.distributor.json` ownership records. It should never
+delete files merely because they exist under a configured prefix.
+
+## Policy Model
+
+Configuration shape:
+
+```yaml
+destinations:
+  - id: archive
+    backend: s3
+    bucket: reports
+    prefix: weather
+    reconciliation:
+      mode: merge
+    retention:
+      prune:
+        enabled: true
+        older_than: 90d
+        keep_latest: 500
+```
+
+Policy semantics:
+
+- `enabled` defaults to `false`.
+- `older_than` deletes managed outputs whose selected timestamp is older than
+  the duration.
+- `keep_latest` keeps the most recent N managed outputs in scope and deletes
+  older managed outputs beyond that count.
+- If both are set, preserve outputs that must be kept by `keep_latest`, then
+  delete remaining outputs older than `older_than`.
+- Retention scope is the configured destination owner scope unless
+  shared-root configuration explicitly chooses a broader scope.
+
+Timestamp basis: use per-output `updated_at` for both policies, with
+`created_at` available for future created-time retention. Updated-time
+retention best matches recurring reports where a stable path is overwritten.
+
+## CLI Shape
+
+Command shape:
+
+```sh
+distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run
+distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --apply
+```
+
+The command should use configured retention policy by default. A later explicit
+override mode can allow one-off retention values if that is useful, but the
+initial implementation should avoid ad hoc deletion flags that bypass reviewed
+config.
+
+## Behavior
+
+Planned behavior:
+
+- Load and validate config normally.
+- Resolve the selected destination and retention policy.
+- Read and validate `.distributor.json`.
+- Select managed output records in the requested owner scope.
+- Sort candidates deterministically by selected timestamp and path.
+- Build a delete plan from `older_than`, `keep_latest`, or both.
+- In dry-run mode, report planned deletions and state changes without writing.
+- In apply mode, delete only planned managed output paths.
+- After successful deletes, remove deleted outputs from state and update state
+  timestamps.
+- If a delete fails, report partial failure and preserve enough state for safe
+  retry. Prefer removing records only for confirmed deleted outputs.
+- Never delete unmanaged storage entries.
+- Delete only managed output paths initially. Local and SSH/SFTP backend helpers
+  may prune empty directories only where they already do so safely. Object
+  storage prefix markers are preserved unless the prefix marker is itself a
+  managed output record.
+
+Prune should operate on output records, not source manifests. If a future state
+model groups outputs into logical source publications, group-level pruning can
+be added later as an explicit policy.
+
+## Safety Rules
+
+Required safety behavior:
+
+- Prune is opt-in.
+- Apply mode is explicit; dry-run is the safe default.
+- Delete only paths recorded as managed in `.distributor.json`.
+- Do not delete `.distributor.json` unless every managed output in scope is
+  deleted and a later design explicitly allows removing empty state.
+- Do not delete unmanaged files, unknown paths, parent directories, or sibling
+  prefixes.
+- Do not run prune against invalid or ambiguous state.
+- Preserve outputs outside the selected owner scope.
+- Report all destructive actions with backend and path context.
+
+## Implementation Work
+
+Implementation work:
+
+- Add retention config structs, defaults, parsing, and validation.
+- Add duration parsing that supports documented minute/hour/day units.
+- Add app-level prune planning and execution use cases.
+- Add state helpers for selecting owner-scoped output records, sorting by
+  timestamp, and removing deleted records.
+- Add storage delete support or reuse existing managed-delete helpers.
+- Add CLI parsing, help text, dry-run output, and apply output.
+- Ensure local, SSH/SFTP, S3, and fake backends preserve bounded deletion
+  semantics.
+- Ensure pruning composes with merge mode and shared-root state.
+
+## Testing
+
+Important tests:
+
+- Config defaults retention disabled.
+- Config rejects invalid durations, negative counts, and enabled policies with
+  no retention rule.
+- Dry-run reports deletes without deleting files or rewriting state.
+- `older_than` deletes only managed outputs older than the threshold.
+- `keep_latest` preserves the newest N outputs and deletes older managed
+  outputs.
+- Combined `older_than` and `keep_latest` behavior is deterministic.
+- Shared-root pruning preserves outputs from other owners when scoped to one
+  owner.
+- Unmanaged files under the destination root are never deleted.
+- Partial delete failure preserves accurate state for confirmed deleted and
+  undeleted outputs.
+- CLI tests cover required flags, apply gating, and report output.
+- Backend tests cover local, SSH/SFTP, S3, and fake deletion behavior used by
+  prune.
+
+## Documentation Work
+
+When implemented, update current-behavior docs in the same change:
+
+- `docs/config.md`: retention configuration, defaults, and examples.
+- `docs/cli.md`: prune command syntax and workflows.
+- `docs/operations.md`: pruning safety, dry-run, apply, and recovery behavior.
+- `docs/troubleshooting.md`: prune conflicts and partial failures.
+- `docs/integrations/destination-state.md`: timestamp fields used by pruning.
+- `docs/internal/state.md`, `docs/internal/publish.md`, and
+  `docs/internal/app.md`: implementation boundaries and invariants.
+
+## Boundaries
+
+- Do not let producers choose retention policy.
+- Do not use prune as a general storage cleanup tool.
+- Do not delete unmanaged content.
+- Do not infer retention from filenames unless a later roadmap explicitly adds
+  path/date parsing.
+- Do not depend on backend-native lifecycle policies for correctness; those can
+  coexist operationally but are not distributor's managed state.
+- Do not run pruning automatically after publish in the initial implementation;
+  pruning should start as an explicit CLI command that operators can schedule.

docs/roadmap/reconcile-state.md

@@ -0,0 +1,171 @@
+# Reconcile-State Roadmap
+
+This document records planned destination state repair tooling that is not part
+of the current implementation. Current recovery behavior is documented in
+`docs/operations.md`.
+
+This work should be implemented after the reconciliation work in
+`docs/roadmap/reconciliation.md` and the shared-root state work in
+`docs/roadmap/multipipeline.md`. Reconcile-state tooling should understand the
+final managed state model before prune behavior is added in
+`docs/roadmap/prune.md`.
+
+## Problem
+
+`.distributor.json` is the managed ownership index for destination outputs. In
+long-lived destinations, operators may delete files manually, lifecycle rules
+may remove objects, failed external operations may leave state stale, or a
+future shared-root destination may accumulate many independently managed
+outputs.
+
+Without a first-class repair command, operators must inspect and edit state by
+hand. That is risky because destination state drives replacement, merge,
+conflict, and cleanup decisions.
+
+## Goal
+
+Add an operator-facing reconcile-state workflow that compares
+`.distributor.json` to actual destination storage and repairs stale managed
+state safely.
+
+The initial command should focus on removing state records for managed outputs
+that no longer exist. It should report unmanaged files, but it should not adopt
+them by default.
+
+## CLI Shape
+
+Command shape:
+
+```sh
+distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run
+distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive
+```
+
+For shared-root destinations, the command should also support reconciling all
+owners in one selected destination root. The final selector should be explicit
+enough to avoid relying on destination ids being globally unique:
+
+```sh
+distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --all-owners --dry-run
+```
+
+The exact flags should follow existing CLI conventions. Dry-run should be the
+explicit `--dry-run` flag; without `--dry-run`, the command applies state
+repairs.
+
+## Behavior
+
+Planned behavior:
+
+- Load and validate config normally.
+- Open the configured destination backend.
+- Resolve the destination root or destination bundle path using the same path
+  mapping semantics as publish where applicable.
+- Read and validate `.distributor.json`.
+- For each managed output record, check whether the storage object/file exists.
+- Remove state records for managed outputs that are missing unless `--dry-run`
+  is set.
+- Preserve state records for managed outputs that exist.
+- Report storage entries that exist but are not recorded as managed.
+- Preserve unmanaged entries by default.
+- Update state timestamps when state is changed.
+- Write state atomically where the backend supports atomic writes.
+- Never delete storage objects as part of the initial reconcile-state command.
+- Do not validate output digests against storage contents in the initial
+  command; digest verification can be added later as a separate audit mode.
+
+For current single-owner state, reconciliation removes missing records from the
+single `outputs` array. For future shared-root state, reconciliation removes
+missing output records across the selected owner scope or all owners, depending
+on command flags.
+
+## Reporting
+
+Dry-run and apply output should be explicit enough for operators and LLM agents
+to understand the result.
+
+Reports should include:
+
+- destination backend and root;
+- state schema version;
+- owner scope, if filtered;
+- count of managed outputs checked;
+- count and paths of missing managed outputs;
+- count and paths of unmanaged existing entries, subject to sensible output
+  limits;
+- whether state would be changed or was changed;
+- errors with enough backend/path context to recover safely.
+
+Machine-readable output should be considered if existing CLI report formats can
+support it without broad refactoring.
+
+## Safety Rules
+
+Required safety behavior:
+
+- Do not delete destination files.
+- Do not adopt unmanaged destination files by default.
+- Do not rewrite invalid state except through a future explicit repair workflow.
+- Do not repair state when the destination identity or ownership scope is
+  ambiguous.
+- Do not scan outside the configured backend root or destination root.
+- Apply state repairs by default and make `--dry-run` the explicit read-only
+  inspection mode.
+- Keep secrets out of reports and errors.
+
+## Implementation Work
+
+Implementation work:
+
+- Add CLI parsing and help text for the reconcile-state command.
+- Add an app-level use case that loads config, opens destinations, and builds a
+  reconcile-state report.
+- Add state package helpers for removing missing outputs without weakening
+  validation.
+- Add storage-facing existence/listing helpers only if current backend
+  interfaces are insufficient.
+- Implement local, SSH/SFTP, S3, and fake backend support needed by the command.
+- Preserve shared-root ownership metadata and timestamps when rewriting state.
+- Add dry-run and applied-change output in the existing CLI reporting style.
+
+## Testing
+
+Important tests:
+
+- Dry-run reports missing managed outputs and does not write state.
+- Apply removes missing managed output records and writes valid state.
+- Existing managed outputs are preserved.
+- Unmanaged existing files are reported but not adopted or deleted.
+- Invalid state fails without rewriting.
+- Shared-root state can be reconciled for one owner scope.
+- Shared-root state can be reconciled for all owner scopes.
+- Local, SSH/SFTP, S3, and fake backend behavior is covered for the existence
+  and listing operations used.
+- CLI tests cover flag validation, default apply behavior, `--dry-run`
+  behavior, and report output.
+
+## Documentation Work
+
+When implemented, update current-behavior docs in the same change:
+
+- `docs/cli.md`: command syntax, flags, and examples.
+- `docs/operations.md`: when and how to run reconcile-state safely.
+- `docs/troubleshooting.md`: stale state and missing managed file entries.
+- `docs/integrations/destination-state.md`: explain how tools may rewrite
+  state.
+- `docs/internal/state.md`, `docs/internal/storage.md`, and
+  `docs/internal/app.md`: record implementation boundaries.
+
+## Boundaries
+
+- Do not implement prune behavior in this work.
+- Do not let reconcile-state infer producer source data or regenerate outputs.
+- Do not repair invalid JSON or manually edited state unless a later roadmap
+  defines an explicit recovery flow.
+- Do not claim unmanaged files by default.
+- Do not add unmanaged-file adoption or digest validation in the initial
+  command.
+- Do not add whole-config reconciliation in the initial command; require enough
+  scope to identify one destination root, with all-owner reconciliation allowed
+  inside a configured shared-root destination.
+- Do not expose a producer-facing API for this workflow.

docs/roadmap/reconciliation.md

@@ -0,0 +1,180 @@
+# Reconciliation Roadmap
+
+This document records planned destination reconciliation work that is not part
+of the current implementation. Current destination replacement, state, and
+recovery behavior is documented in `docs/operations.md`,
+`docs/config.md`, and `docs/integrations/destination-state.md`.
+
+This work should be implemented before the shared-root multi-pipeline state
+work described in `docs/roadmap/multipipeline.md`. The merge mode described
+here is still scoped to one pipeline/destination owner for a destination bundle
+path; it does not by itself allow multiple pipelines to share one
+`.distributor.json`.
+
+## Destination Reconciliation Modes
+
+Current destination replacement behavior treats each publish as the complete
+desired managed output set for the destination bundle path. When a newer source
+replaces an older destination state, `distributor` deletes prior managed outputs
+that are no longer present in the new plan, then writes the new outputs and
+state.
+
+That behavior is useful for snapshot-style producers, but it is not ideal for
+incremental producers that publish new reports without retaining prior report
+state.
+
+Planned work:
+
+- Add a per-destination reconciliation mode.
+- Keep current behavior as the default mode.
+- Add a mode that writes new outputs while retaining prior managed outputs that
+  are not present in the new plan.
+- Keep reconciliation mode per destination. Replacement and merge behavior are
+  destination safety policies, and one pipeline may need both a snapshot-style
+  `latest` destination and an accumulating `archive` destination from the same
+  source bundle.
+
+Proposed configuration:
+
+```yaml
+destinations:
+  - id: archive
+    backend: s3
+    bucket: reports
+    prefix: weather/morning/archive
+    reconciliation:
+      mode: merge
+
+  - id: latest
+    backend: s3
+    bucket: reports
+    prefix: weather/morning/latest
+    path_mapping:
+      mode: fixed
+    reconciliation:
+      mode: replace
+```
+
+Accepted modes:
+
+- `replace`: current behavior. The destination bundle path is reconciled to the
+  new planned output set. Prior managed outputs not present in the new plan are
+  removed.
+- `merge`: new planned outputs are written into the managed destination set.
+  Prior managed outputs not present in the new plan are retained.
+
+## State Model
+
+Destination state needs to represent cumulative managed outputs for merge mode.
+The current state file already records an `outputs` array, but that array
+currently represents the outputs from the most recent publication.
+
+Planned state semantics:
+
+- `source.manifest` continues to record the latest source manifest used for
+  destination comparison.
+- `outputs` records the complete currently managed output set for the
+  destination bundle path.
+- In `replace` mode, the new state `outputs` are exactly the newly planned
+  outputs.
+- In `merge` mode, the new state `outputs` are:
+  - previous managed outputs not overwritten by the new plan;
+  - plus newly planned outputs.
+- Persist reconciliation metadata so state interpretation remains explicit after
+  config changes and so later migration, reconcile-state, and prune tooling can
+  explain why historical outputs were retained, for example:
+
+```json
+{
+  "reconciliation": {
+    "mode": "merge"
+  }
+}
+```
+
+If state schema changes are required, bump the destination state schema version
+and keep parsing/validation rules explicit.
+
+## Safety Rules
+
+Merge mode must not become an unsafe overwrite path.
+
+Required safety behavior:
+
+- A new output may overwrite a prior output only when that path is already
+  recorded as distributor-managed in the existing `.distributor.json`.
+- A new output path that exists in storage but is not recorded as managed should
+  fail as unmanaged content by default.
+- Prior managed outputs not included in the new plan are retained in merge mode.
+- Prior managed outputs not included in the new plan are deleted in replace
+  mode.
+- Failed-write cleanup should delete only outputs written by the failed attempt,
+  not retained prior managed outputs.
+- Forced replacement behavior remains explicit, bounded, dry-runnable, and
+  constrained to the destination bundle path.
+
+## Planning And Execution Work
+
+Implementation work:
+
+- Add destination config type `reconciliation.mode`.
+- Default `reconciliation.mode` to `replace`.
+- Validate mode values as `replace` or `merge`.
+- Thread reconciliation mode into publish planning and execution requests.
+- During replace execution, preserve current managed cleanup behavior.
+- During merge execution:
+  - inspect existing state outputs before writing;
+  - allow overwrites only for paths already managed by existing state;
+  - write new outputs with overwrite enabled only for managed existing paths;
+  - retain previous managed outputs that are not overwritten;
+  - write destination state with the cumulative managed output set.
+- Ensure generated HTML outputs and copied source outputs use the same
+  reconciliation semantics.
+- Ensure link metadata and primary URL behavior remain deterministic when
+  retained outputs exist. Derive `links.primary_url` from the newly planned
+  outputs for the current publish, not from retained historical outputs.
+
+## Testing
+
+Important tests:
+
+- Config defaults `reconciliation.mode` to `replace`.
+- Config rejects unknown reconciliation modes.
+- Replace mode deletes prior managed outputs omitted from the new bundle.
+- Merge mode retains prior managed outputs omitted from the new bundle.
+- Merge mode overwrites a path already recorded as managed.
+- Merge mode fails when a new output path collides with unmanaged destination
+  content.
+- Merge mode state records the cumulative managed output set.
+- Failed merge writes clean up only outputs from the failed attempt.
+- Fixed-path destinations support both modes.
+- Source-only, generated-HTML-only, and source-plus-HTML publish policies all
+  honor reconciliation mode.
+- S3, SSH/SFTP, local, and fake backend tests cover any backend overwrite or
+  cleanup behavior that changes.
+
+## Documentation Work
+
+When implemented, update current-behavior docs in the same change:
+
+- `docs/config.md`: document `reconciliation.mode`, defaults, and examples.
+- `docs/operations.md`: explain replace versus merge behavior and recovery.
+- `docs/integrations/destination-state.md`: document any state schema or
+  semantic changes.
+- `docs/internal/publish.md` and related internal docs: record planning and
+  execution invariants.
+- Relevant examples under `examples/`, especially archive/latest style
+  configurations.
+
+## Boundaries
+
+- Do not add lifecycle retention policies such as keep-latest-N or delete older
+  than a duration as part of this work.
+- Do not let producers choose reconciliation mode through source manifests or
+  upload requests.
+- Do not weaken unmanaged-content safety checks.
+- Do not make merge mode the default; preserve current replacement behavior for
+  existing configurations.
+- Do not claim unmanaged colliding files in merge mode. Claiming unmanaged
+  content should remain an explicit force or repair workflow, not normal publish
+  behavior.

docs/troubleshooting.md

@@ -400,7 +400,7 @@ Reference: [Configuration](config.md#serverhttp).
 
 Symptom: `upload token environment variable ... is not set`, `... is empty`, or `upload token environment variables ... resolve to the same value`.
 
-Likely cause: an `http_upload` source references a missing/empty `token_env`, or two upload pipelines resolve to the same bearer token.
+Likely cause: a top-level upload token record references a missing or empty `token_env`, or two token records resolve to the same bearer token.
 
 Diagnostic:
 
@@ -410,20 +410,20 @@ env | cut -d= -f1 | rg '^<token-variable>$'
 ls -l <secrets-directory>/<token-variable>
 ```
 
-Safe fix: provide one distinct non-empty token value per upload pipeline through the process environment or `secrets.directory`. Do not put literal tokens in YAML.
+Safe fix: provide one distinct non-empty token value per upload token record through the process environment or `secrets.directory`. Do not put literal tokens in YAML.
 
-Reference: [Configuration](config.md#http-upload-source-backend).
+Reference: [Configuration](config.md#upload_tokens).
 
 ## Upload Request Is Unauthorized
 
-Symptom: `POST /upload` returns `401`.
+Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `401`.
 
-Likely cause: the request lacks `Authorization: Bearer <token>`, has an empty token, or uses a token that does not match any configured upload pipeline.
+Likely cause: the request lacks `Authorization: Bearer <token>`, has an empty token, or uses a token that does not match any configured upload token record.
 
 Diagnostic:
 
 ```sh
-curl -i -X POST http://127.0.0.1:8080/upload \
+curl -i -X POST http://127.0.0.1:8080/v1/pipelines/<pipeline-id>/upload \
   -H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \
   -H "Content-Type: application/x-tar" \
   --data-binary @bundle.tar
@@ -433,11 +433,27 @@ Safe fix: use the token value resolved by the configured `token_env`. Do not inc
 
 Reference: [Operations](operations.md#http-upload-operation).
 
+## Upload Request Is Forbidden
+
+Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `403`.
+
+Likely cause: the bearer token is valid, but its configured `allow_pipelines` list does not include the requested upload pipeline.
+
+Diagnostic:
+
+```sh
+rg -n 'upload_tokens:|allow_pipelines:|id:' <config-path>
+```
+
+Safe fix: request the intended pipeline id, or update the token allowlist to include the configured `http_upload` pipeline that this producer may submit to.
+
+Reference: [Configuration](config.md#upload_tokens).
+
 ## Upload Request Is Rejected Before A Run ID
 
-Symptom: `POST /upload` returns `400`, `413`, `415`, or `503`.
+Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `400`, `413`, `415`, or `503`.
 
-Likely cause: the request included a `pipeline` or `pipeline_id` query, archive content is malformed, the body exceeds size limits, content type is unsupported, or the in-memory upload queue is full.
+Likely cause: the request path has an invalid pipeline id, included a `pipeline` or `pipeline_id` query, archive content is malformed, the body exceeds size limits, content type is unsupported, or the in-memory upload queue is full.
 
 Diagnostic:
 
@@ -447,10 +463,30 @@ tar -tzf bundle.tar.gz
 rg -n 'max_upload_size|queue_size|max_concurrency' <config-path>
 ```
 
-Safe fix: send one valid tar or tar.gz source bundle archive with `Content-Type: application/x-tar`, `application/gzip`, or `application/x-gzip`; remove pipeline query parameters; reduce archive size or raise the configured limit; retry after queue pressure drops.
+Safe fix: send one valid tar or tar.gz source bundle archive to `/v1/pipelines/<pipeline-id>/upload` with `Content-Type: application/x-tar`, `application/gzip`, or `application/x-gzip`; remove pipeline query parameters; reduce archive size or raise the configured limit; retry after queue pressure drops.
 
 Reference: [Operations](operations.md#http-upload-operation).
 
+## Upload Idempotency Conflict
+
+Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `409`.
+
+Likely cause: the request reused an `Idempotency-Key` for the same token id and pipeline id with a different source manifest, or another request with the same key is still being staged before its manifest is known.
+
+Diagnostic:
+
+```sh
+curl -i -X POST http://127.0.0.1:8080/v1/pipelines/<pipeline-id>/upload \
+  -H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \
+  -H "Content-Type: application/gzip" \
+  -H "Idempotency-Key: <key>" \
+  --data-binary @bundle.tar.gz
+```
+
+Safe fix: if the response includes `"retryable":true`, retry the same upload later with the same key. Otherwise, inspect the producer operation and use the same key only for the same source bundle.
+
+Reference: [HTTP Upload API Contract](integrations/http-upload.md).
+
 ## Upload Status Is Missing
 
 Symptom: `GET /runs/<run_id>` returns `404`.

examples/http-upload-local.yml

@@ -9,11 +9,15 @@ server:
     queue_size: 16
     max_concurrency: 1
     retention: 24h
+upload_tokens:
+  - id: example-uploader
+    token_env: DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN
+    allow_pipelines:
+      - example-http-upload
 pipelines:
   - id: example-http-upload
     source:
       backend: http_upload
-      token_env: DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN
     destinations:
       - id: local-archive
         backend: local
@@ -21,4 +25,3 @@ pipelines:
         publish:
           source: true
           html: false
-

examples/upload-client/main.go

@@ -0,0 +1,57 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"os"
+	"time"
+
+	"gitea.maximumdirect.net/eric/distributor/pkg/upload"
+)
+
+func main() {
+	token := os.Getenv("DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN")
+	if token == "" {
+		log.Fatal("set DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN before running this example")
+	}
+	endpoint := os.Getenv("DISTRIBUTOR_EXAMPLE_UPLOAD_ENDPOINT")
+	if endpoint == "" {
+		endpoint = "http://127.0.0.1:8080"
+	}
+	bundleRoot := "examples/source-bundle"
+	if len(os.Args) > 1 {
+		bundleRoot = os.Args[1]
+	}
+	pipelineID := os.Getenv("DISTRIBUTOR_EXAMPLE_UPLOAD_PIPELINE_ID")
+	if pipelineID == "" {
+		pipelineID = "example-http-upload"
+	}
+	if len(os.Args) > 2 {
+		pipelineID = os.Args[2]
+	}
+	idempotencyKey := os.Getenv("DISTRIBUTOR_EXAMPLE_UPLOAD_IDEMPOTENCY_KEY")
+
+	client, err := upload.NewClient(upload.ClientOptions{
+		Endpoint: endpoint,
+		Token:    token,
+	})
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+	opts := upload.UploadBundleOptions{
+		PipelineID: pipelineID,
+		Root:       bundleRoot,
+	}
+	if idempotencyKey != "" {
+		opts.IdempotencyKey = idempotencyKey
+	}
+	result, err := client.UploadBundle(ctx, opts)
+	if err != nil {
+		log.Fatal(err)
+	}
+	fmt.Printf("accepted run %s with status %s\n", result.RunID, result.Status)
+}

internal/app/run_test.go

@@ -292,7 +292,6 @@ func TestRunPipelineWithLocalSourcePublishesToRegisteredDestinationBackends(t *t
 			ID: "reports",
 			Source: config.Backend{
 				Backend: config.BackendHTTPUpload,
-				Upload:  config.HTTPUpload{TokenEnv: "UPLOAD_TOKEN"},
 			},
 			Destinations: []config.Destination{
 				{
@@ -309,6 +308,11 @@ func TestRunPipelineWithLocalSourcePublishesToRegisteredDestinationBackends(t *t
 				},
 			},
 		}},
+		UploadTokens: []config.UploadToken{{
+			ID:             "reporter",
+			TokenEnv:       "UPLOAD_TOKEN",
+			AllowPipelines: []string{"reports"},
+		}},
 	}
 	config.ApplyDefaults(&cfg)
 	provider := fakeBackendFactoryProvider(t, map[string]storage.Backend{
@@ -1674,11 +1678,15 @@ func writeFanoutConfig(t *testing.T, sourceRoot, firstDestination, secondDestina
 func writeUploadPipelineConfig(t *testing.T, destinationRoot string) string {
 	t.Helper()
 	return writeConfigFile(t, `
+upload_tokens:
+  - id: reporter
+    token_env: UPLOAD_TOKEN
+    allow_pipelines:
+      - reports
 pipelines:
   - id: reports
     source:
       backend: http_upload
-      token_env: UPLOAD_TOKEN
     destinations:
       - id: archive
         backend: local

internal/app/serve_test.go

@@ -70,14 +70,24 @@ func writeServeUploadConfig(t *testing.T, tokenEnvs []string) string {
 server:
   http:
     bind: 127.0.0.1:0
-pipelines:
+upload_tokens:
 `
 	for index, tokenEnv := range tokenEnvs {
 		body += `
+  - id: reporter-` + string(rune('a'+index)) + `
+    token_env: ` + tokenEnv + `
+    allow_pipelines:
+      - reports-` + string(rune('a'+index)) + `
+`
+	}
+	body += `
+pipelines:
+`
+	for index := range tokenEnvs {
+		body += `
   - id: reports-` + string(rune('a'+index)) + `
     source:
       backend: http_upload
-      token_env: ` + tokenEnv + `
     destinations:
       - id: archive
         backend: local

internal/app/upload_coordinator.go

@@ -13,6 +13,7 @@ import (
 
 	"gitea.maximumdirect.net/eric/distributor/internal/config"
 	"gitea.maximumdirect.net/eric/distributor/internal/ingest"
+	sourcebundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
 )
 
 const DefaultUploadMaxFileCount = 4096
@@ -43,12 +44,14 @@ type UploadRunRecord struct {
 }
 
 type UploadRequest struct {
-	PipelineID   string
-	ContentType  string
-	Body         io.Reader
-	DryRun       bool
-	Force        bool
-	MaxFileCount int
+	TokenID        string
+	PipelineID     string
+	ContentType    string
+	Body           io.Reader
+	IdempotencyKey string
+	DryRun         bool
+	Force          bool
+	MaxFileCount   int
 }
 
 type UploadQueueFullError struct {
@@ -64,6 +67,22 @@ func IsUploadQueueFull(err error) bool {
 	return errors.As(err, &full)
 }
 
+type UploadIdempotencyConflictError struct {
+	Retryable bool
+}
+
+func (err UploadIdempotencyConflictError) Error() string {
+	if err.Retryable {
+		return "upload idempotency key is already being processed"
+	}
+	return "upload idempotency key conflicts with a different source manifest"
+}
+
+func IsUploadIdempotencyConflict(err error) bool {
+	var conflict UploadIdempotencyConflictError
+	return errors.As(err, &conflict)
+}
+
 type UploadCoordinator struct {
 	ctx          context.Context
 	cfg          config.Config
@@ -82,6 +101,7 @@ type UploadCoordinator struct {
 	activePipeline map[string]bool
 	pending        []*uploadJob
 	records        map[UploadRunID]UploadRunRecord
+	idempotency    map[uploadIdempotencyScope]uploadIdempotencyRecord
 }
 
 type uploadStageFunc func(context.Context, ingest.StageOptions) (ingest.StagedBundle, error)
@@ -95,6 +115,18 @@ type uploadJob struct {
 	stagedRoot string
 }
 
+type uploadIdempotencyScope struct {
+	TokenID    string
+	PipelineID string
+	Key        string
+}
+
+type uploadIdempotencyRecord struct {
+	RunID    UploadRunID
+	Manifest sourcebundle.Manifest
+	Pending  bool
+}
+
 type uploadCoordinatorHooks struct {
 	stage        uploadStageFunc
 	run          uploadRunFunc
@@ -140,6 +172,7 @@ func newUploadCoordinator(ctx context.Context, cfg config.Config, hooks uploadCo
 		maxConcurrency: cfg.Server.HTTP.MaxConcurrency,
 		activePipeline: map[string]bool{},
 		records:        map[UploadRunID]UploadRunRecord{},
+		idempotency:    map[uploadIdempotencyScope]uploadIdempotencyRecord{},
 	}
 	go coordinator.dispatchLoop()
 	return coordinator
@@ -169,14 +202,26 @@ func (coordinator *UploadCoordinator) Submit(ctx context.Context, request Upload
 	if err := ingest.ValidateContentType(request.ContentType); err != nil {
 		return UploadRunRecord{}, err
 	}
+	scope, hasKey := uploadRequestIdempotencyScope(request.TokenID, pipeline.ID, request.IdempotencyKey)
 
 	coordinator.mu.Lock()
 	coordinator.expireLocked(coordinator.now().UTC())
-	if coordinator.queueFullLocked() {
+	existingIdempotency, hasExistingIdempotency := coordinator.idempotency[scope]
+	if hasKey && hasExistingIdempotency && existingIdempotency.Pending {
 		coordinator.mu.Unlock()
-		return UploadRunRecord{}, UploadQueueFullError{QueueSize: coordinator.queueSize}
+		return UploadRunRecord{}, UploadIdempotencyConflictError{Retryable: true}
+	}
+	needsReservation := !hasKey || !hasExistingIdempotency
+	if needsReservation {
+		if coordinator.queueFullLocked() {
+			coordinator.mu.Unlock()
+			return UploadRunRecord{}, UploadQueueFullError{QueueSize: coordinator.queueSize}
+		}
+		coordinator.reservedCount++
+		if hasKey {
+			coordinator.idempotency[scope] = uploadIdempotencyRecord{Pending: true}
+		}
 	}
-	coordinator.reservedCount++
 	coordinator.mu.Unlock()
 
 	staged, err := coordinator.stage(ctx, ingest.StageOptions{
@@ -189,13 +234,36 @@ func (coordinator *UploadCoordinator) Submit(ctx context.Context, request Upload
 		MaxFileCount:        uploadMaxFileCount(request.MaxFileCount),
 	})
 	if err != nil {
-		coordinator.releaseReservation()
+		if needsReservation {
+			coordinator.releaseReservation(scope, hasKey)
+		}
 		return UploadRunRecord{}, err
 	}
 
 	coordinator.mu.Lock()
 	defer coordinator.mu.Unlock()
-	coordinator.reservedCount--
+	if needsReservation {
+		coordinator.reservedCount--
+	}
+	if hasKey {
+		existingIdempotency, hasExistingIdempotency = coordinator.idempotency[scope]
+		if hasExistingIdempotency && !existingIdempotency.Pending {
+			if uploadManifestsEqual(existingIdempotency.Manifest, staged.Manifest) {
+				_ = os.RemoveAll(staged.Root)
+				record, ok := coordinator.records[existingIdempotency.RunID]
+				if !ok {
+					return UploadRunRecord{}, fmt.Errorf("idempotency record references missing run")
+				}
+				return record, nil
+			}
+			_ = os.RemoveAll(staged.Root)
+			return UploadRunRecord{}, UploadIdempotencyConflictError{}
+		}
+		if !hasExistingIdempotency && !needsReservation && coordinator.queueFullLocked() {
+			_ = os.RemoveAll(staged.Root)
+			return UploadRunRecord{}, UploadQueueFullError{QueueSize: coordinator.queueSize}
+		}
+	}
 	record := UploadRunRecord{
 		ID:         runID,
 		PipelineID: pipeline.ID,
@@ -204,6 +272,12 @@ func (coordinator *UploadCoordinator) Submit(ctx context.Context, request Upload
 		StagedRoot: staged.Root,
 	}
 	coordinator.records[runID] = record
+	if hasKey {
+		coordinator.idempotency[scope] = uploadIdempotencyRecord{
+			RunID:    runID,
+			Manifest: staged.Manifest,
+		}
+	}
 	coordinator.pending = append(coordinator.pending, &uploadJob{
 		recordID:   runID,
 		request:    request,
@@ -214,6 +288,13 @@ func (coordinator *UploadCoordinator) Submit(ctx context.Context, request Upload
 	return record, nil
 }
 
+func uploadRequestIdempotencyScope(tokenID, pipelineID, key string) (uploadIdempotencyScope, bool) {
+	if key == "" {
+		return uploadIdempotencyScope{}, false
+	}
+	return uploadIdempotencyScope{TokenID: tokenID, PipelineID: pipelineID, Key: key}, true
+}
+
 func (coordinator *UploadCoordinator) Status(runID UploadRunID) (UploadRunRecord, bool) {
 	coordinator.mu.Lock()
 	defer coordinator.mu.Unlock()
@@ -326,10 +407,15 @@ func (coordinator *UploadCoordinator) runJob(job *uploadJob) {
 	coordinator.complete(job, &report, err)
 }
 
-func (coordinator *UploadCoordinator) releaseReservation() {
+func (coordinator *UploadCoordinator) releaseReservation(scope uploadIdempotencyScope, hasKey bool) {
 	coordinator.mu.Lock()
 	defer coordinator.mu.Unlock()
 	coordinator.reservedCount--
+	if hasKey {
+		if record, ok := coordinator.idempotency[scope]; ok && record.Pending {
+			delete(coordinator.idempotency, scope)
+		}
+	}
 }
 
 func (coordinator *UploadCoordinator) queueFullLocked() bool {
@@ -379,10 +465,31 @@ func (coordinator *UploadCoordinator) expireLocked(now time.Time) []UploadRunRec
 		record.Error = ""
 		expired = append(expired, record)
 		delete(coordinator.records, runID)
+		for scope, idempotencyRecord := range coordinator.idempotency {
+			if idempotencyRecord.RunID == runID {
+				delete(coordinator.idempotency, scope)
+			}
+		}
 	}
 	return expired
 }
 
+func uploadManifestsEqual(a, b sourcebundle.Manifest) bool {
+	if a.SchemaVersion != b.SchemaVersion ||
+		a.ID != b.ID ||
+		a.Digest != b.Digest ||
+		!a.Created.Equal(b.Created) ||
+		len(a.Files) != len(b.Files) {
+		return false
+	}
+	for index := range a.Files {
+		if a.Files[index] != b.Files[index] {
+			return false
+		}
+	}
+	return true
+}
+
 func (coordinator *UploadCoordinator) notify() {
 	select {
 	case coordinator.signal <- struct{}{}:

internal/app/upload_coordinator_test.go

@@ -15,6 +15,7 @@ import (
 
 	"gitea.maximumdirect.net/eric/distributor/internal/config"
 	"gitea.maximumdirect.net/eric/distributor/internal/ingest"
+	sourcebundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
 )
 
 func TestUploadCoordinatorGeneratesRunIDAndAcceptedStatus(t *testing.T) {
@@ -254,6 +255,330 @@ func TestUploadCoordinatorExpiresCompletedRecordsAndStagingDirectories(t *testin
 	}
 }
 
+func TestUploadCoordinatorIdempotencyReturnsOriginalRunForSameManifest(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	var runCount atomic.Int64
+	coordinator := newUploadCoordinator(ctx, uploadCoordinatorConfig(t, uploadCoordinatorConfigOptions{
+		pipelineIDs: []string{"reports"},
+	}), uploadCoordinatorHooks{
+		randomSuffix: uploadTestSuffixes("00000001", "00000002"),
+		stage:        manifestUploadStage,
+		run: func(ctx context.Context, cfg config.Config, options RunPipelineWithLocalSourceOptions) (RunReport, error) {
+			runCount.Add(1)
+			return RunReport{}, nil
+		},
+	})
+
+	first, err := coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:        "reporter-a",
+		PipelineID:     "reports",
+		ContentType:    ingest.ContentTypeTar,
+		Body:           strings.NewReader("same"),
+		IdempotencyKey: "producer.retry:20260603",
+	})
+	if err != nil {
+		t.Fatalf("first Submit() error = %v", err)
+	}
+	waitForUploadStatus(t, coordinator, first.ID, UploadStatusSucceeded)
+
+	second, err := coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:        "reporter-a",
+		PipelineID:     "reports",
+		ContentType:    ingest.ContentTypeTar,
+		Body:           strings.NewReader("same"),
+		IdempotencyKey: "producer.retry:20260603",
+	})
+	if err != nil {
+		t.Fatalf("second Submit() error = %v", err)
+	}
+	if second.ID != first.ID {
+		t.Fatalf("second run id = %q, want original %q", second.ID, first.ID)
+	}
+	if got := runCount.Load(); got != 1 {
+		t.Fatalf("run count = %d, want 1", got)
+	}
+}
+
+func TestUploadCoordinatorIdempotencyConflictsForDifferentManifest(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	coordinator := newUploadCoordinator(ctx, uploadCoordinatorConfig(t, uploadCoordinatorConfigOptions{
+		pipelineIDs: []string{"reports"},
+	}), uploadCoordinatorHooks{
+		randomSuffix: uploadTestSuffixes("00000001", "00000002"),
+		stage:        manifestUploadStage,
+		run:          successfulUploadRun,
+	})
+
+	first, err := coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:        "reporter-a",
+		PipelineID:     "reports",
+		ContentType:    ingest.ContentTypeTar,
+		Body:           strings.NewReader("one"),
+		IdempotencyKey: "same-key",
+	})
+	if err != nil {
+		t.Fatalf("first Submit() error = %v", err)
+	}
+	waitForUploadStatus(t, coordinator, first.ID, UploadStatusSucceeded)
+
+	_, err = coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:        "reporter-a",
+		PipelineID:     "reports",
+		ContentType:    ingest.ContentTypeTar,
+		Body:           strings.NewReader("two"),
+		IdempotencyKey: "same-key",
+	})
+	if err == nil || !IsUploadIdempotencyConflict(err) {
+		t.Fatalf("second Submit() error = %v, want idempotency conflict", err)
+	}
+}
+
+func TestUploadCoordinatorIdempotencyIsScopedByToken(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	coordinator := newUploadCoordinator(ctx, uploadCoordinatorConfig(t, uploadCoordinatorConfigOptions{
+		pipelineIDs: []string{"reports"},
+	}), uploadCoordinatorHooks{
+		randomSuffix: uploadTestSuffixes("00000001", "00000002"),
+		stage:        manifestUploadStage,
+		run:          successfulUploadRun,
+	})
+
+	first, err := coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:        "reporter-a",
+		PipelineID:     "reports",
+		ContentType:    ingest.ContentTypeTar,
+		Body:           strings.NewReader("one"),
+		IdempotencyKey: "shared-key",
+	})
+	if err != nil {
+		t.Fatalf("first Submit() error = %v", err)
+	}
+	second, err := coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:        "reporter-b",
+		PipelineID:     "reports",
+		ContentType:    ingest.ContentTypeTar,
+		Body:           strings.NewReader("two"),
+		IdempotencyKey: "shared-key",
+	})
+	if err != nil {
+		t.Fatalf("second Submit() error = %v", err)
+	}
+	if second.ID == first.ID {
+		t.Fatalf("run ids matched across tokens: %q", second.ID)
+	}
+}
+
+func TestUploadCoordinatorIdempotencyIsScopedByPipeline(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	coordinator := newUploadCoordinator(ctx, uploadCoordinatorConfig(t, uploadCoordinatorConfigOptions{
+		pipelineIDs: []string{"reports-one", "reports-two"},
+	}), uploadCoordinatorHooks{
+		randomSuffix: uploadTestSuffixes("00000001", "00000002"),
+		stage:        manifestUploadStage,
+		run:          successfulUploadRun,
+	})
+
+	first, err := coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:        "reporter-a",
+		PipelineID:     "reports-one",
+		ContentType:    ingest.ContentTypeTar,
+		Body:           strings.NewReader("one"),
+		IdempotencyKey: "shared-key",
+	})
+	if err != nil {
+		t.Fatalf("first Submit() error = %v", err)
+	}
+	second, err := coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:        "reporter-a",
+		PipelineID:     "reports-two",
+		ContentType:    ingest.ContentTypeTar,
+		Body:           strings.NewReader("two"),
+		IdempotencyKey: "shared-key",
+	})
+	if err != nil {
+		t.Fatalf("second Submit() error = %v", err)
+	}
+	if second.ID == first.ID {
+		t.Fatalf("run ids matched across pipelines: %q", second.ID)
+	}
+}
+
+func TestUploadCoordinatorWithoutIdempotencyKeyAcceptsDuplicateBodies(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	coordinator := newUploadCoordinator(ctx, uploadCoordinatorConfig(t, uploadCoordinatorConfigOptions{
+		pipelineIDs: []string{"reports"},
+	}), uploadCoordinatorHooks{
+		randomSuffix: uploadTestSuffixes("00000001", "00000002"),
+		stage:        manifestUploadStage,
+		run:          successfulUploadRun,
+	})
+
+	first, err := coordinator.Submit(context.Background(), UploadRequest{PipelineID: "reports", ContentType: ingest.ContentTypeTar, Body: strings.NewReader("same")})
+	if err != nil {
+		t.Fatalf("first Submit() error = %v", err)
+	}
+	second, err := coordinator.Submit(context.Background(), UploadRequest{PipelineID: "reports", ContentType: ingest.ContentTypeTar, Body: strings.NewReader("same")})
+	if err != nil {
+		t.Fatalf("second Submit() error = %v", err)
+	}
+	if second.ID == first.ID {
+		t.Fatalf("second run id = %q, want distinct run", second.ID)
+	}
+}
+
+func TestUploadCoordinatorIdempotencyReturnsRetryableConflictWhileStaging(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	entered := make(chan struct{})
+	release := make(chan struct{})
+	coordinator := newUploadCoordinator(ctx, uploadCoordinatorConfig(t, uploadCoordinatorConfigOptions{
+		pipelineIDs: []string{"reports"},
+	}), uploadCoordinatorHooks{
+		randomSuffix: uploadTestSuffixes("00000001", "00000002"),
+		stage: func(ctx context.Context, opts ingest.StageOptions) (ingest.StagedBundle, error) {
+			close(entered)
+			<-release
+			return manifestUploadStage(ctx, opts)
+		},
+		run: successfulUploadRun,
+	})
+	firstErr := make(chan error, 1)
+	go func() {
+		_, err := coordinator.Submit(context.Background(), UploadRequest{
+			TokenID:        "reporter-a",
+			PipelineID:     "reports",
+			ContentType:    ingest.ContentTypeTar,
+			Body:           strings.NewReader("same"),
+			IdempotencyKey: "in-flight",
+		})
+		firstErr <- err
+	}()
+	<-entered
+
+	var reads atomic.Int64
+	_, err := coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:     "reporter-a",
+		PipelineID:  "reports",
+		ContentType: ingest.ContentTypeTar,
+		Body: readerFunc(func(data []byte) (int, error) {
+			reads.Add(1)
+			return 0, io.EOF
+		}),
+		IdempotencyKey: "in-flight",
+	})
+	var conflict UploadIdempotencyConflictError
+	if err == nil || !errors.As(err, &conflict) || !conflict.Retryable {
+		t.Fatalf("second Submit() error = %v, want retryable idempotency conflict", err)
+	}
+	if got := reads.Load(); got != 0 {
+		t.Fatalf("retryable conflict body reads = %d, want 0", got)
+	}
+	close(release)
+	if err := <-firstErr; err != nil {
+		t.Fatalf("first Submit() error = %v", err)
+	}
+}
+
+func TestUploadCoordinatorIdempotencyPendingScopeIncludesToken(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	var calls atomic.Int64
+	entered := make(chan struct{})
+	release := make(chan struct{})
+	coordinator := newUploadCoordinator(ctx, uploadCoordinatorConfig(t, uploadCoordinatorConfigOptions{
+		pipelineIDs: []string{"reports"},
+	}), uploadCoordinatorHooks{
+		randomSuffix: uploadTestSuffixes("00000001", "00000002"),
+		stage: func(ctx context.Context, opts ingest.StageOptions) (ingest.StagedBundle, error) {
+			if calls.Add(1) == 1 {
+				close(entered)
+				<-release
+			}
+			return manifestUploadStage(ctx, opts)
+		},
+		run: successfulUploadRun,
+	})
+	firstErr := make(chan error, 1)
+	go func() {
+		_, err := coordinator.Submit(context.Background(), UploadRequest{
+			TokenID:        "reporter-a",
+			PipelineID:     "reports",
+			ContentType:    ingest.ContentTypeTar,
+			Body:           strings.NewReader("same"),
+			IdempotencyKey: "in-flight",
+		})
+		firstErr <- err
+	}()
+	<-entered
+
+	second, err := coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:        "reporter-b",
+		PipelineID:     "reports",
+		ContentType:    ingest.ContentTypeTar,
+		Body:           strings.NewReader("same"),
+		IdempotencyKey: "in-flight",
+	})
+	if err != nil {
+		t.Fatalf("second Submit() error = %v", err)
+	}
+	if second.ID == "" {
+		t.Fatal("second run id is empty, want accepted run")
+	}
+	close(release)
+	if err := <-firstErr; err != nil {
+		t.Fatalf("first Submit() error = %v", err)
+	}
+}
+
+func TestUploadCoordinatorIdempotencyExpiresWithCompletedStatus(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	clock := newUploadTestClock(time.Date(2026, 6, 3, 12, 0, 0, 0, time.UTC))
+	coordinator := newUploadCoordinator(ctx, uploadCoordinatorConfig(t, uploadCoordinatorConfigOptions{
+		pipelineIDs: []string{"reports"},
+		retention:   time.Second,
+	}), uploadCoordinatorHooks{
+		now:          clock.Now,
+		randomSuffix: uploadTestSuffixes("00000001", "00000002", "00000003"),
+		stage:        manifestUploadStage,
+		run:          successfulUploadRun,
+	})
+
+	first, err := coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:        "reporter-a",
+		PipelineID:     "reports",
+		ContentType:    ingest.ContentTypeTar,
+		Body:           strings.NewReader("same"),
+		IdempotencyKey: "expires",
+	})
+	if err != nil {
+		t.Fatalf("first Submit() error = %v", err)
+	}
+	waitForUploadStatus(t, coordinator, first.ID, UploadStatusSucceeded)
+
+	clock.Advance(2 * time.Second)
+	coordinator.Expire()
+
+	second, err := coordinator.Submit(context.Background(), UploadRequest{
+		TokenID:        "reporter-a",
+		PipelineID:     "reports",
+		ContentType:    ingest.ContentTypeTar,
+		Body:           strings.NewReader("same"),
+		IdempotencyKey: "expires",
+	})
+	if err != nil {
+		t.Fatalf("second Submit() error = %v", err)
+	}
+	if second.ID == first.ID {
+		t.Fatalf("second run id = %q, want new run after expiry", second.ID)
+	}
+}
+
 type readerFunc func([]byte) (int, error)
 
 func (fn readerFunc) Read(data []byte) (int, error) {
@@ -268,6 +593,37 @@ func successfulUploadStage(ctx context.Context, opts ingest.StageOptions) (inges
 	return ingest.StagedBundle{Root: root}, nil
 }
 
+func manifestUploadStage(ctx context.Context, opts ingest.StageOptions) (ingest.StagedBundle, error) {
+	data, err := io.ReadAll(opts.Body)
+	if err != nil {
+		return ingest.StagedBundle{}, err
+	}
+	root := filepath.Join(opts.PipelineStagingPath, opts.RunID)
+	if err := os.MkdirAll(root, 0o755); err != nil {
+		return ingest.StagedBundle{}, err
+	}
+	return ingest.StagedBundle{
+		Root:     root,
+		Manifest: uploadTestManifest(string(data)),
+	}, nil
+}
+
+func uploadTestManifest(id string) sourcebundle.Manifest {
+	created := time.Date(2026, 6, 3, 12, 0, 0, 0, time.UTC)
+	file := sourcebundle.ManifestFile{
+		Path:   "report.md",
+		SHA256: sourcebundle.FileDigest([]byte(id)),
+		Size:   int64(len(id)),
+	}
+	return sourcebundle.Manifest{
+		SchemaVersion: sourcebundle.SchemaVersion,
+		ID:            id,
+		Created:       created,
+		Files:         []sourcebundle.ManifestFile{file},
+		Digest:        sourcebundle.BundleDigest([]sourcebundle.ManifestFile{file}),
+	}
+}
+
 func successfulUploadRun(ctx context.Context, cfg config.Config, options RunPipelineWithLocalSourceOptions) (RunReport, error) {
 	return RunReport{}, nil
 }
@@ -305,11 +661,11 @@ func uploadCoordinatorConfig(t *testing.T, opts uploadCoordinatorConfigOptions)
 		}},
 	}
 	for _, pipelineID := range opts.pipelineIDs {
+		tokenEnv := strings.ToUpper(strings.ReplaceAll(pipelineID, "-", "_")) + "_TOKEN"
 		cfg.Pipelines = append(cfg.Pipelines, config.Pipeline{
 			ID: pipelineID,
 			Source: config.Backend{
 				Backend: config.BackendHTTPUpload,
-				Upload:  config.HTTPUpload{TokenEnv: strings.ToUpper(strings.ReplaceAll(pipelineID, "-", "_")) + "_TOKEN"},
 			},
 			Destinations: []config.Destination{{
 				ID:      "archive",
@@ -317,6 +673,11 @@ func uploadCoordinatorConfig(t *testing.T, opts uploadCoordinatorConfigOptions)
 				Path:    t.TempDir(),
 			}},
 		})
+		cfg.UploadTokens = append(cfg.UploadTokens, config.UploadToken{
+			ID:             pipelineID + "-reporter",
+			TokenEnv:       tokenEnv,
+			AllowPipelines: []string{pipelineID},
+		})
 	}
 	return cfg
 }

internal/app/upload_http.go

@@ -19,8 +19,15 @@ type uploadCoordinator interface {
 }
 
 type uploadHTTPHandler struct {
-	coordinator uploadCoordinator
-	tokens      map[string]string
+	coordinator     uploadCoordinator
+	tokens          map[string]resolvedUploadToken
+	uploadPipelines map[string]struct{}
+}
+
+type resolvedUploadToken struct {
+	ID               string
+	Value            string
+	AllowedPipelines map[string]struct{}
 }
 
 type uploadAcceptedResponse struct {
@@ -29,9 +36,12 @@ type uploadAcceptedResponse struct {
 }
 
 type httpErrorResponse struct {
-	Error string `json:"error"`
+	Error     string `json:"error"`
+	Retryable bool   `json:"retryable,omitempty"`
 }
 
+const idempotencyKeyHeader = "Idempotency-Key"
+
 func newUploadHTTPHandler(ctx context.Context, cfg config.Config, environment config.Environment) (http.Handler, error) {
 	config.ApplyDefaults(&cfg)
 	tokens, err := resolveUploadTokens(cfg, environment)
@@ -39,38 +49,57 @@ func newUploadHTTPHandler(ctx context.Context, cfg config.Config, environment co
 		return nil, err
 	}
 	return uploadHTTPHandler{
-		coordinator: NewUploadCoordinator(ctx, cfg),
-		tokens:      tokens,
+		coordinator:     NewUploadCoordinator(ctx, cfg),
+		tokens:          tokens,
+		uploadPipelines: uploadPipelineSet(cfg),
 	}, nil
 }
 
-func resolveUploadTokens(cfg config.Config, environment config.Environment) (map[string]string, error) {
-	tokens := make(map[string]string)
-	for _, pipeline := range cfg.Pipelines {
-		if pipeline.Source.Backend != config.BackendHTTPUpload {
-			continue
-		}
-		tokenName := pipeline.Source.Upload.TokenEnv
-		token, ok := environment.Lookup(tokenName)
+func resolveUploadTokens(cfg config.Config, environment config.Environment) (map[string]resolvedUploadToken, error) {
+	tokens := make(map[string]resolvedUploadToken)
+	for _, uploadToken := range cfg.UploadTokens {
+		token, ok := environment.Lookup(uploadToken.TokenEnv)
 		if !ok {
-			return nil, fmt.Errorf("upload token environment variable %s is not set", tokenName)
+			return nil, fmt.Errorf("upload token %s environment variable %s is not set", uploadToken.ID, uploadToken.TokenEnv)
 		}
 		if token == "" {
-			return nil, fmt.Errorf("upload token environment variable %s is empty", tokenName)
+			return nil, fmt.Errorf("upload token %s environment variable %s is empty", uploadToken.ID, uploadToken.TokenEnv)
 		}
 		if existing, exists := tokens[token]; exists {
-			return nil, fmt.Errorf("upload token environment variables for pipelines %s and %s resolve to the same value", existing, pipeline.ID)
+			return nil, fmt.Errorf("upload token environment variables for tokens %s and %s resolve to the same value", existing.ID, uploadToken.ID)
+		}
+		tokens[token] = resolvedUploadToken{
+			ID:               uploadToken.ID,
+			Value:            token,
+			AllowedPipelines: pipelineIDSet(uploadToken.AllowPipelines),
 		}
-		tokens[token] = pipeline.ID
 	}
 	return tokens, nil
 }
 
+func uploadPipelineSet(cfg config.Config) map[string]struct{} {
+	pipelines := make(map[string]struct{})
+	for _, pipeline := range cfg.Pipelines {
+		if pipeline.Source.Backend == config.BackendHTTPUpload {
+			pipelines[pipeline.ID] = struct{}{}
+		}
+	}
+	return pipelines
+}
+
+func pipelineIDSet(ids []string) map[string]struct{} {
+	set := make(map[string]struct{}, len(ids))
+	for _, id := range ids {
+		set[id] = struct{}{}
+	}
+	return set
+}
+
 func (handler uploadHTTPHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	switch {
 	case r.Method == http.MethodGet && r.URL.Path == "/healthz":
 		handler.handleHealth(w)
-	case r.Method == http.MethodPost && r.URL.Path == "/upload":
+	case r.Method == http.MethodPost && strings.HasPrefix(r.URL.Path, "/v1/pipelines/"):
 		handler.handleUpload(w, r)
 	case r.Method == http.MethodGet && strings.HasPrefix(r.URL.Path, "/runs/"):
 		handler.handleRunStatus(w, r)
@@ -88,24 +117,44 @@ func (handler uploadHTTPHandler) handleUpload(w http.ResponseWriter, r *http.Req
 		writeHTTPError(w, http.StatusBadRequest, "pipeline id is not accepted")
 		return
 	}
-	pipelineID, ok := handler.authenticate(r.Header.Get("Authorization"))
+	pipelineID, ok := uploadPipelineIDFromPath(r.URL.Path)
+	if !ok {
+		writeHTTPError(w, http.StatusNotFound, "not found")
+		return
+	}
+	if !config.IsSlugLikeID(pipelineID) {
+		writeHTTPError(w, http.StatusBadRequest, "invalid pipeline id")
+		return
+	}
+	token, ok := handler.authenticate(r.Header.Get("Authorization"))
 	if !ok {
 		writeHTTPError(w, http.StatusUnauthorized, "unauthorized")
 		return
 	}
+	if _, ok := handler.uploadPipelines[pipelineID]; !ok {
+		writeHTTPError(w, http.StatusNotFound, "upload pipeline not found")
+		return
+	}
+	if _, ok := token.AllowedPipelines[pipelineID]; !ok {
+		writeHTTPError(w, http.StatusForbidden, "forbidden")
+		return
+	}
 	contentType := r.Header.Get("Content-Type")
 	if err := ingest.ValidateContentType(contentType); err != nil {
 		writeHTTPError(w, http.StatusUnsupportedMediaType, "unsupported content type")
 		return
 	}
-	if !handler.coordinator.CanAccept() {
-		writeHTTPError(w, http.StatusServiceUnavailable, "upload queue is full")
+	idempotencyKey, err := uploadIdempotencyKey(r.Header)
+	if err != nil {
+		writeHTTPError(w, http.StatusBadRequest, "invalid idempotency key")
 		return
 	}
 	record, err := handler.coordinator.Submit(r.Context(), UploadRequest{
-		PipelineID:  pipelineID,
-		ContentType: contentType,
-		Body:        r.Body,
+		TokenID:        token.ID,
+		PipelineID:     pipelineID,
+		ContentType:    contentType,
+		Body:           r.Body,
+		IdempotencyKey: idempotencyKey,
 	})
 	if err != nil {
 		writeUploadSubmitError(w, err)
@@ -117,6 +166,19 @@ func (handler uploadHTTPHandler) handleUpload(w http.ResponseWriter, r *http.Req
 	})
 }
 
+func uploadPipelineIDFromPath(path string) (string, bool) {
+	const prefix = "/v1/pipelines/"
+	const suffix = "/upload"
+	if !strings.HasPrefix(path, prefix) || !strings.HasSuffix(path, suffix) {
+		return "", false
+	}
+	pipelineID := strings.TrimSuffix(strings.TrimPrefix(path, prefix), suffix)
+	if pipelineID == "" || strings.Contains(pipelineID, "/") {
+		return "", false
+	}
+	return pipelineID, true
+}
+
 func (handler uploadHTTPHandler) handleRunStatus(w http.ResponseWriter, r *http.Request) {
 	rawRunID := strings.TrimPrefix(r.URL.Path, "/runs/")
 	if rawRunID == "" || strings.Contains(rawRunID, "/") {
@@ -131,23 +193,57 @@ func (handler uploadHTTPHandler) handleRunStatus(w http.ResponseWriter, r *http.
 	writeJSON(w, http.StatusOK, record)
 }
 
-func (handler uploadHTTPHandler) authenticate(header string) (string, bool) {
+func (handler uploadHTTPHandler) authenticate(header string) (resolvedUploadToken, bool) {
 	const prefix = "Bearer "
 	if !strings.HasPrefix(header, prefix) {
-		return "", false
+		return resolvedUploadToken{}, false
 	}
 	token := strings.TrimSpace(strings.TrimPrefix(header, prefix))
 	if token == "" {
-		return "", false
+		return resolvedUploadToken{}, false
 	}
-	pipelineID, ok := handler.tokens[token]
-	return pipelineID, ok
+	resolved, ok := handler.tokens[token]
+	return resolved, ok
+}
+
+func uploadIdempotencyKey(header http.Header) (string, error) {
+	values := header.Values(idempotencyKeyHeader)
+	if len(values) == 0 {
+		return "", nil
+	}
+	if len(values) != 1 {
+		return "", fmt.Errorf("idempotency key must appear at most once")
+	}
+	key := values[0]
+	if key == "" {
+		return "", fmt.Errorf("idempotency key is required when header is present")
+	}
+	if len(key) > 128 {
+		return "", fmt.Errorf("idempotency key must be at most 128 bytes")
+	}
+	for index := 0; index < len(key); index++ {
+		character := key[index]
+		if character >= 'a' && character <= 'z' ||
+			character >= 'A' && character <= 'Z' ||
+			character >= '0' && character <= '9' ||
+			character == '.' ||
+			character == '_' ||
+			character == '-' ||
+			character == ':' {
+			continue
+		}
+		return "", fmt.Errorf("idempotency key contains unsupported character")
+	}
+	return key, nil
 }
 
 func writeUploadSubmitError(w http.ResponseWriter, err error) {
+	var idempotencyConflict UploadIdempotencyConflictError
 	switch {
 	case IsUploadQueueFull(err):
 		writeHTTPError(w, http.StatusServiceUnavailable, "upload queue is full")
+	case errors.As(err, &idempotencyConflict):
+		writeHTTPErrorRetryable(w, http.StatusConflict, idempotencyConflict.Error(), idempotencyConflict.Retryable)
 	case errors.Is(err, ingest.ErrUploadTooLarge):
 		writeHTTPError(w, http.StatusRequestEntityTooLarge, "upload exceeds maximum size")
 	case errors.Is(err, ingest.ErrUnsupportedContentType):
@@ -158,7 +254,11 @@ func writeUploadSubmitError(w http.ResponseWriter, err error) {
 }
 
 func writeHTTPError(w http.ResponseWriter, status int, message string) {
-	writeJSON(w, status, httpErrorResponse{Error: message})
+	writeHTTPErrorRetryable(w, status, message, false)
+}
+
+func writeHTTPErrorRetryable(w http.ResponseWriter, status int, message string, retryable bool) {
+	writeJSON(w, status, httpErrorResponse{Error: message, Retryable: retryable})
 }
 
 func writeJSON(w http.ResponseWriter, status int, value any) {

internal/app/upload_http_integration_test.go

@@ -21,6 +21,7 @@ import (
 	"gitea.maximumdirect.net/eric/distributor/internal/ingest"
 	"gitea.maximumdirect.net/eric/distributor/internal/storage"
 	"gitea.maximumdirect.net/eric/distributor/internal/testutil"
+	clientupload "gitea.maximumdirect.net/eric/distributor/pkg/upload"
 )
 
 func TestHTTPUploadPublishesTarAndGzipFanout(t *testing.T) {
@@ -78,15 +79,16 @@ func TestHTTPUploadInvalidArchiveIsRejectedWithoutRunID(t *testing.T) {
 		destinations: []string{destination},
 	}}, 4, 1))
 	handler := uploadHTTPHandler{
-		coordinator: coordinator,
-		tokens:      map[string]string{"reports-secret": "reports"},
+		coordinator:     coordinator,
+		tokens:          map[string]resolvedUploadToken{"reports-secret": uploadHTTPTestToken("reports-reporter", "reports-secret", "reports")},
+		uploadPipelines: pipelineIDSet([]string{"reports"}),
 	}
 	server := httptest.NewServer(handler)
 	defer server.Close()
 
 	status, body := postHTTPUpload(t, server, "reports-secret", ingest.ContentTypeTar, []byte("not a tar archive"))
 	if status != http.StatusBadRequest {
-		t.Fatalf("POST /upload status = %d, want %d; body = %s", status, http.StatusBadRequest, body)
+		t.Fatalf("POST upload status = %d, want %d; body = %s", status, http.StatusBadRequest, body)
 	}
 	if strings.Contains(body, "run_id") || strings.Contains(body, "reports-secret") {
 		t.Fatalf("invalid archive response exposed run id or token: %s", body)
@@ -97,6 +99,64 @@ func TestHTTPUploadInvalidArchiveIsRejectedWithoutRunID(t *testing.T) {
 	assertDirectoryEmpty(t, destination)
 }
 
+func TestHTTPUploadIdempotencyReturnsOriginalRunForSameBundle(t *testing.T) {
+	destination := t.TempDir()
+	coordinator := NewUploadCoordinator(context.Background(), httpUploadIntegrationConfig(t, []httpUploadPipelineSpec{{
+		id:           "reports",
+		tokenEnv:     "REPORTS_TOKEN",
+		stagingPath:  filepath.Join(t.TempDir(), "reports"),
+		destinations: []string{destination},
+	}}, 4, 1))
+	handler := uploadHTTPHandler{
+		coordinator:     coordinator,
+		tokens:          map[string]resolvedUploadToken{"reports-secret": uploadHTTPTestToken("reports-reporter", "reports-secret", "reports")},
+		uploadPipelines: pipelineIDSet([]string{"reports"}),
+	}
+	server := httptest.NewServer(handler)
+	defer server.Close()
+
+	firstRunID := submitHTTPUploadWithKey(t, server, "reports-secret", ingest.ContentTypeTar, "same-key", bundleArchive(t, false, testutil.BundleOptions{}))
+	waitForHTTPUploadStatus(t, server, firstRunID, UploadStatusSucceeded)
+
+	secondRunID := submitHTTPUploadWithKey(t, server, "reports-secret", ingest.ContentTypeGzip, "same-key", bundleArchive(t, true, testutil.BundleOptions{}))
+	if secondRunID != firstRunID {
+		t.Fatalf("second run id = %q, want original %q", secondRunID, firstRunID)
+	}
+	if got := coordinator.QueueDepth(); got != 0 {
+		t.Fatalf("queue depth = %d, want no duplicate run queued", got)
+	}
+}
+
+func TestHTTPUploadIdempotencyReturnsConflictForDifferentBundle(t *testing.T) {
+	destination := t.TempDir()
+	coordinator := NewUploadCoordinator(context.Background(), httpUploadIntegrationConfig(t, []httpUploadPipelineSpec{{
+		id:           "reports",
+		tokenEnv:     "REPORTS_TOKEN",
+		stagingPath:  filepath.Join(t.TempDir(), "reports"),
+		destinations: []string{destination},
+	}}, 4, 1))
+	handler := uploadHTTPHandler{
+		coordinator:     coordinator,
+		tokens:          map[string]resolvedUploadToken{"reports-secret": uploadHTTPTestToken("reports-reporter", "reports-secret", "reports")},
+		uploadPipelines: pipelineIDSet([]string{"reports"}),
+	}
+	server := httptest.NewServer(handler)
+	defer server.Close()
+
+	firstRunID := submitHTTPUploadWithKey(t, server, "reports-secret", ingest.ContentTypeTar, "same-key", bundleArchive(t, false, testutil.BundleOptions{}))
+	waitForHTTPUploadStatus(t, server, firstRunID, UploadStatusSucceeded)
+
+	status, body := postHTTPUploadWithKey(t, server, "reports-secret", ingest.ContentTypeTar, "same-key", bundleArchive(t, false, testutil.BundleOptions{
+		ID: "weather.daily.brentwood.2026-05-31",
+	}))
+	if status != http.StatusConflict {
+		t.Fatalf("POST upload status = %d, want %d; body = %s", status, http.StatusConflict, body)
+	}
+	if strings.Contains(body, "reports-secret") {
+		t.Fatalf("conflict response exposed token: %s", body)
+	}
+}
+
 func TestHTTPUploadOversizedArchiveIsRejectedWithoutRunID(t *testing.T) {
 	destination := t.TempDir()
 	stagingPath := filepath.Join(t.TempDir(), "reports")
@@ -111,15 +171,16 @@ func TestHTTPUploadOversizedArchiveIsRejectedWithoutRunID(t *testing.T) {
 	cfg.Pipelines[0].Source.Upload.MaxUploadSize = &size
 	coordinator := NewUploadCoordinator(context.Background(), cfg)
 	handler := uploadHTTPHandler{
-		coordinator: coordinator,
-		tokens:      map[string]string{"reports-secret": "reports"},
+		coordinator:     coordinator,
+		tokens:          map[string]resolvedUploadToken{"reports-secret": uploadHTTPTestToken("reports-reporter", "reports-secret", "reports")},
+		uploadPipelines: pipelineIDSet([]string{"reports"}),
 	}
 	server := httptest.NewServer(handler)
 	defer server.Close()
 
 	status, body := postHTTPUpload(t, server, "reports-secret", ingest.ContentTypeTar, bundleArchive(t, false, testutil.BundleOptions{}))
 	if status != http.StatusRequestEntityTooLarge {
-		t.Fatalf("POST /upload status = %d, want %d; body = %s", status, http.StatusRequestEntityTooLarge, body)
+		t.Fatalf("POST upload status = %d, want %d; body = %s", status, http.StatusRequestEntityTooLarge, body)
 	}
 	if strings.Contains(body, "run_id") || strings.Contains(body, "reports-secret") {
 		t.Fatalf("oversized response exposed run id or token: %s", body)
@@ -154,8 +215,9 @@ func TestHTTPUploadSamePipelineRequestsSerialize(t *testing.T) {
 		},
 	})
 	handler := uploadHTTPHandler{
-		coordinator: coordinator,
-		tokens:      map[string]string{"reports-secret": "reports"},
+		coordinator:     coordinator,
+		tokens:          map[string]resolvedUploadToken{"reports-secret": uploadHTTPTestToken("reports-reporter", "reports-secret", "reports")},
+		uploadPipelines: pipelineIDSet([]string{"reports"}),
 	}
 	server := httptest.NewServer(handler)
 	defer server.Close()
@@ -206,16 +268,17 @@ func TestHTTPUploadDifferentPipelinesRunConcurrently(t *testing.T) {
 	})
 	handler := uploadHTTPHandler{
 		coordinator: coordinator,
-		tokens: map[string]string{
-			"one-secret": "reports-one",
-			"two-secret": "reports-two",
+		tokens: map[string]resolvedUploadToken{
+			"one-secret": uploadHTTPTestToken("reports-one-reporter", "one-secret", "reports-one"),
+			"two-secret": uploadHTTPTestToken("reports-two-reporter", "two-secret", "reports-two"),
 		},
+		uploadPipelines: pipelineIDSet([]string{"reports-one", "reports-two"}),
 	}
 	server := httptest.NewServer(handler)
 	defer server.Close()
 
-	firstRunID := submitHTTPUpload(t, server, "one-secret", ingest.ContentTypeTar, []byte("first"))
-	secondRunID := submitHTTPUpload(t, server, "two-secret", ingest.ContentTypeTar, []byte("second"))
+	firstRunID := submitHTTPUploadToPipeline(t, server, "reports-one", "one-secret", ingest.ContentTypeTar, []byte("first"))
+	secondRunID := submitHTTPUploadToPipeline(t, server, "reports-two", "two-secret", ingest.ContentTypeTar, []byte("second"))
 	waitForStartedPipelines(t, started, "reports-one", "reports-two")
 	waitForHTTPUploadStatus(t, server, firstRunID, UploadStatusRunning)
 	waitForHTTPUploadStatus(t, server, secondRunID, UploadStatusRunning)
@@ -228,6 +291,199 @@ func TestHTTPUploadDifferentPipelinesRunConcurrently(t *testing.T) {
 	waitForHTTPUploadStatus(t, server, secondRunID, UploadStatusSucceeded)
 }
 
+func TestHTTPUploadOneTokenCanUploadToMultiplePipelines(t *testing.T) {
+	firstDestination := t.TempDir()
+	secondDestination := t.TempDir()
+	cfg := httpUploadIntegrationConfig(t, []httpUploadPipelineSpec{
+		{
+			id:           "reports-one",
+			tokenEnv:     "SHARED_UPLOAD_TOKEN",
+			stagingPath:  filepath.Join(t.TempDir(), "reports-one"),
+			destinations: []string{firstDestination},
+		},
+		{
+			id:           "reports-two",
+			tokenEnv:     "SHARED_UPLOAD_TOKEN",
+			stagingPath:  filepath.Join(t.TempDir(), "reports-two"),
+			destinations: []string{secondDestination},
+		},
+	}, 4, 1)
+	cfg.UploadTokens = []config.UploadToken{{
+		ID:             "shared-reporter",
+		TokenEnv:       "SHARED_UPLOAD_TOKEN",
+		AllowPipelines: []string{"reports-one", "reports-two"},
+	}}
+	handler, err := newUploadHTTPHandler(context.Background(), cfg, uploadHTTPTestEnvironment(map[string]string{
+		"SHARED_UPLOAD_TOKEN": "shared-secret",
+	}))
+	if err != nil {
+		t.Fatalf("newUploadHTTPHandler() error = %v", err)
+	}
+	server := httptest.NewServer(handler)
+	defer server.Close()
+
+	firstRunID := submitHTTPUploadToPipeline(t, server, "reports-one", "shared-secret", ingest.ContentTypeTar, bundleArchive(t, false, testutil.BundleOptions{
+		ID: "reports.one.2026-06-08",
+	}))
+	secondRunID := submitHTTPUploadToPipeline(t, server, "reports-two", "shared-secret", ingest.ContentTypeTar, bundleArchive(t, false, testutil.BundleOptions{
+		ID: "reports.two.2026-06-08",
+	}))
+
+	first := waitForHTTPUploadStatus(t, server, firstRunID, UploadStatusSucceeded)
+	second := waitForHTTPUploadStatus(t, server, secondRunID, UploadStatusSucceeded)
+	if first.PipelineID != "reports-one" || second.PipelineID != "reports-two" {
+		t.Fatalf("statuses pipeline = %q/%q, want reports-one/reports-two", first.PipelineID, second.PipelineID)
+	}
+	assertPublishedBundle(t, firstDestination)
+	assertPublishedBundle(t, secondDestination)
+}
+
+func TestHTTPUploadMultipleTokensCanUploadToOnePipeline(t *testing.T) {
+	destination := t.TempDir()
+	cfg := httpUploadIntegrationConfig(t, []httpUploadPipelineSpec{{
+		id:           "reports",
+		tokenEnv:     "FIRST_UPLOAD_TOKEN",
+		stagingPath:  filepath.Join(t.TempDir(), "reports"),
+		destinations: []string{destination},
+	}}, 4, 1)
+	cfg.UploadTokens = []config.UploadToken{
+		{ID: "first-reporter", TokenEnv: "FIRST_UPLOAD_TOKEN", AllowPipelines: []string{"reports"}},
+		{ID: "second-reporter", TokenEnv: "SECOND_UPLOAD_TOKEN", AllowPipelines: []string{"reports"}},
+	}
+	handler, err := newUploadHTTPHandler(context.Background(), cfg, uploadHTTPTestEnvironment(map[string]string{
+		"FIRST_UPLOAD_TOKEN":  "first-secret",
+		"SECOND_UPLOAD_TOKEN": "second-secret",
+	}))
+	if err != nil {
+		t.Fatalf("newUploadHTTPHandler() error = %v", err)
+	}
+	server := httptest.NewServer(handler)
+	defer server.Close()
+
+	firstRunID := submitHTTPUpload(t, server, "first-secret", ingest.ContentTypeTar, bundleArchive(t, false, testutil.BundleOptions{}))
+	secondRunID := submitHTTPUpload(t, server, "second-secret", ingest.ContentTypeTar, bundleArchive(t, false, testutil.BundleOptions{}))
+
+	first := waitForHTTPUploadStatus(t, server, firstRunID, UploadStatusSucceeded)
+	second := waitForHTTPUploadStatus(t, server, secondRunID, UploadStatusSucceeded)
+	if first.PipelineID != "reports" || second.PipelineID != "reports" {
+		t.Fatalf("statuses pipeline = %q/%q, want reports/reports", first.PipelineID, second.PipelineID)
+	}
+	assertPublishedBundle(t, destination)
+}
+
+func TestHTTPUploadRejectsDisallowedPipelineAndLegacyUploadWithoutQueueing(t *testing.T) {
+	coordinator := NewUploadCoordinator(context.Background(), httpUploadIntegrationConfig(t, []httpUploadPipelineSpec{
+		{
+			id:           "reports",
+			tokenEnv:     "REPORTS_TOKEN",
+			stagingPath:  filepath.Join(t.TempDir(), "reports"),
+			destinations: []string{t.TempDir()},
+		},
+		{
+			id:           "private",
+			tokenEnv:     "PRIVATE_TOKEN",
+			stagingPath:  filepath.Join(t.TempDir(), "private"),
+			destinations: []string{t.TempDir()},
+		},
+	}, 4, 1))
+	handler := uploadHTTPHandler{
+		coordinator: coordinator,
+		tokens: map[string]resolvedUploadToken{
+			"reports-secret": uploadHTTPTestToken("reports-reporter", "reports-secret", "reports"),
+		},
+		uploadPipelines: pipelineIDSet([]string{"reports", "private"}),
+	}
+	server := httptest.NewServer(handler)
+	defer server.Close()
+
+	status, body := postHTTPUploadToPipeline(t, server, "private", "reports-secret", ingest.ContentTypeTar, []byte("archive"))
+	if status != http.StatusForbidden {
+		t.Fatalf("disallowed upload status = %d, want %d; body = %s", status, http.StatusForbidden, body)
+	}
+	status, body = postLegacyHTTPUpload(t, server, "reports-secret", ingest.ContentTypeTar, []byte("archive"))
+	if status != http.StatusNotFound {
+		t.Fatalf("legacy upload status = %d, want %d; body = %s", status, http.StatusNotFound, body)
+	}
+	if got := coordinator.QueueDepth(); got != 0 {
+		t.Fatalf("queue depth = %d, want 0", got)
+	}
+}
+
+func TestHTTPUploadPublishesThroughSelectedPipeline(t *testing.T) {
+	firstDestination := t.TempDir()
+	secondDestination := t.TempDir()
+	cfg := httpUploadIntegrationConfig(t, []httpUploadPipelineSpec{
+		{
+			id:           "reports-one",
+			tokenEnv:     "SHARED_UPLOAD_TOKEN",
+			stagingPath:  filepath.Join(t.TempDir(), "reports-one"),
+			destinations: []string{firstDestination},
+		},
+		{
+			id:           "reports-two",
+			tokenEnv:     "SHARED_UPLOAD_TOKEN",
+			stagingPath:  filepath.Join(t.TempDir(), "reports-two"),
+			destinations: []string{secondDestination},
+		},
+	}, 4, 1)
+	cfg.UploadTokens = []config.UploadToken{{
+		ID:             "shared-reporter",
+		TokenEnv:       "SHARED_UPLOAD_TOKEN",
+		AllowPipelines: []string{"reports-one", "reports-two"},
+	}}
+	handler, err := newUploadHTTPHandler(context.Background(), cfg, uploadHTTPTestEnvironment(map[string]string{
+		"SHARED_UPLOAD_TOKEN": "shared-secret",
+	}))
+	if err != nil {
+		t.Fatalf("newUploadHTTPHandler() error = %v", err)
+	}
+	server := httptest.NewServer(handler)
+	defer server.Close()
+
+	bundleRoot := t.TempDir()
+	testutil.WriteSourceBundle(t, bundleRoot, "", testutil.BundleOptions{
+		ID: "reports.selected.2026-06-08",
+	})
+	client, err := clientupload.NewClient(clientupload.ClientOptions{
+		Endpoint:   server.URL,
+		Token:      "shared-secret",
+		HTTPClient: server.Client(),
+	})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	result, err := client.UploadBundle(context.Background(), clientupload.UploadBundleOptions{
+		PipelineID: "reports-two",
+		Root:       bundleRoot,
+	})
+	if err != nil {
+		t.Fatalf("UploadBundle() error = %v", err)
+	}
+	runID := UploadRunID(result.RunID)
+	record := waitForHTTPUploadStatus(t, server, runID, UploadStatusSucceeded)
+
+	if record.PipelineID != "reports-two" {
+		t.Fatalf("record pipeline = %q, want reports-two", record.PipelineID)
+	}
+	if record.Report == nil {
+		t.Fatal("completed status report = nil, want run report")
+	}
+	if got, want := len(record.Report.Pipelines), 1; got != want {
+		t.Fatalf("report pipeline count = %d, want %d", got, want)
+	}
+	if record.Report.Pipelines[0].ID != "reports-two" {
+		t.Fatalf("report pipeline = %q, want reports-two", record.Report.Pipelines[0].ID)
+	}
+	if got, want := len(record.Report.Actions), 1; got != want {
+		t.Fatalf("report action count = %d, want %d", got, want)
+	}
+	if record.Report.Actions[0].PipelineID != "reports-two" {
+		t.Fatalf("action pipeline = %q, want reports-two", record.Report.Actions[0].PipelineID)
+	}
+	assertDirectoryEmpty(t, firstDestination)
+	assertPublishedBundle(t, secondDestination)
+}
+
 type httpUploadPipelineSpec struct {
 	id           string
 	tokenEnv     string
@@ -255,7 +511,6 @@ func httpUploadIntegrationConfig(t *testing.T, pipelines []httpUploadPipelineSpe
 			Source: config.Backend{
 				Backend: config.BackendHTTPUpload,
 				Upload: config.HTTPUpload{
-					TokenEnv:      spec.tokenEnv,
 					StagingPath:   spec.stagingPath,
 					MaxUploadSize: &size,
 				},
@@ -270,6 +525,11 @@ func httpUploadIntegrationConfig(t *testing.T, pipelines []httpUploadPipelineSpe
 			})
 		}
 		cfg.Pipelines = append(cfg.Pipelines, pipeline)
+		cfg.UploadTokens = append(cfg.UploadTokens, config.UploadToken{
+			ID:             spec.id + "-reporter",
+			TokenEnv:       spec.tokenEnv,
+			AllowPipelines: []string{spec.id},
+		})
 	}
 	config.ApplyDefaults(&cfg)
 	return cfg
@@ -277,9 +537,25 @@ func httpUploadIntegrationConfig(t *testing.T, pipelines []httpUploadPipelineSpe
 
 func submitHTTPUpload(t *testing.T, server *httptest.Server, token, contentType string, body []byte) UploadRunID {
 	t.Helper()
-	status, responseBody := postHTTPUpload(t, server, token, contentType, body)
+	return submitHTTPUploadToPipeline(t, server, "reports", token, contentType, body)
+}
+
+func submitHTTPUploadToPipeline(t *testing.T, server *httptest.Server, pipelineID, token, contentType string, body []byte) UploadRunID {
+	t.Helper()
+	status, responseBody := postHTTPUploadToPipeline(t, server, pipelineID, token, contentType, body)
+	return decodeAcceptedHTTPUpload(t, status, responseBody)
+}
+
+func submitHTTPUploadWithKey(t *testing.T, server *httptest.Server, token, contentType, key string, body []byte) UploadRunID {
+	t.Helper()
+	status, responseBody := postHTTPUploadWithKey(t, server, token, contentType, key, body)
+	return decodeAcceptedHTTPUpload(t, status, responseBody)
+}
+
+func decodeAcceptedHTTPUpload(t *testing.T, status int, responseBody string) UploadRunID {
+	t.Helper()
 	if status != http.StatusAccepted {
-		t.Fatalf("POST /upload status = %d, want %d; body = %s", status, http.StatusAccepted, responseBody)
+		t.Fatalf("POST upload status = %d, want %d; body = %s", status, http.StatusAccepted, responseBody)
 	}
 	var accepted uploadAcceptedResponse
 	if err := json.Unmarshal([]byte(responseBody), &accepted); err != nil {
@@ -292,6 +568,44 @@ func submitHTTPUpload(t *testing.T, server *httptest.Server, token, contentType
 }
 
 func postHTTPUpload(t *testing.T, server *httptest.Server, token, contentType string, body []byte) (int, string) {
+	t.Helper()
+	return postHTTPUploadToPipeline(t, server, "reports", token, contentType, body)
+}
+
+func postHTTPUploadToPipeline(t *testing.T, server *httptest.Server, pipelineID, token, contentType string, body []byte) (int, string) {
+	t.Helper()
+	return postHTTPUploadWithKeyToPipeline(t, server, pipelineID, token, contentType, "", body)
+}
+
+func postHTTPUploadWithKey(t *testing.T, server *httptest.Server, token, contentType, key string, body []byte) (int, string) {
+	t.Helper()
+	return postHTTPUploadWithKeyToPipeline(t, server, "reports", token, contentType, key, body)
+}
+
+func postHTTPUploadWithKeyToPipeline(t *testing.T, server *httptest.Server, pipelineID, token, contentType, key string, body []byte) (int, string) {
+	t.Helper()
+	request, err := http.NewRequest(http.MethodPost, server.URL+"/v1/pipelines/"+pipelineID+"/upload", bytes.NewReader(body))
+	if err != nil {
+		t.Fatalf("NewRequest() error = %v", err)
+	}
+	request.Header.Set("Authorization", "Bearer "+token)
+	request.Header.Set("Content-Type", contentType)
+	if key != "" {
+		request.Header.Set("Idempotency-Key", key)
+	}
+	response, err := server.Client().Do(request)
+	if err != nil {
+		t.Fatalf("POST upload error = %v", err)
+	}
+	defer response.Body.Close()
+	data, err := io.ReadAll(response.Body)
+	if err != nil {
+		t.Fatalf("read response body: %v", err)
+	}
+	return response.StatusCode, string(data)
+}
+
+func postLegacyHTTPUpload(t *testing.T, server *httptest.Server, token, contentType string, body []byte) (int, string) {
 	t.Helper()
 	request, err := http.NewRequest(http.MethodPost, server.URL+"/upload", bytes.NewReader(body))
 	if err != nil {
@@ -301,7 +615,7 @@ func postHTTPUpload(t *testing.T, server *httptest.Server, token, contentType st
 	request.Header.Set("Content-Type", contentType)
 	response, err := server.Client().Do(request)
 	if err != nil {
-		t.Fatalf("POST /upload error = %v", err)
+		t.Fatalf("POST legacy upload error = %v", err)
 	}
 	defer response.Body.Close()
 	data, err := io.ReadAll(response.Body)

internal/app/upload_http_test.go

@@ -53,10 +53,14 @@ func TestResolveUploadTokensFailsForMissingAndDuplicateTokens(t *testing.T) {
 		ID: "weekly",
 		Source: config.Backend{
 			Backend: config.BackendHTTPUpload,
-			Upload:  config.HTTPUpload{TokenEnv: "OTHER_UPLOAD_TOKEN"},
 		},
 		Destinations: cfg.Pipelines[0].Destinations,
 	})
+	cfg.UploadTokens = append(cfg.UploadTokens, config.UploadToken{
+		ID:             "weekly-reporter",
+		TokenEnv:       "OTHER_UPLOAD_TOKEN",
+		AllowPipelines: []string{"weekly"},
+	})
 	config.ApplyDefaults(&cfg)
 	secret := "super-secret-token"
 	_, err = resolveUploadTokens(cfg, uploadHTTPTestEnvironment(map[string]string{
@@ -71,6 +75,39 @@ func TestResolveUploadTokensFailsForMissingAndDuplicateTokens(t *testing.T) {
 	}
 }
 
+func TestResolveUploadTokensAllowsMultiplePipelines(t *testing.T) {
+	cfg := uploadHTTPTestConfig()
+	cfg.Pipelines = append(cfg.Pipelines, config.Pipeline{
+		ID: "weekly",
+		Source: config.Backend{
+			Backend: config.BackendHTTPUpload,
+		},
+		Destinations: cfg.Pipelines[0].Destinations,
+	})
+	cfg.UploadTokens[0].AllowPipelines = []string{"reports", "weekly"}
+	config.ApplyDefaults(&cfg)
+
+	tokens, err := resolveUploadTokens(cfg, uploadHTTPTestEnvironment(map[string]string{
+		"UPLOAD_TOKEN": "secret",
+	}))
+	if err != nil {
+		t.Fatalf("resolveUploadTokens() error = %v", err)
+	}
+	token, ok := tokens["secret"]
+	if !ok {
+		t.Fatal("resolved token missing")
+	}
+	if token.ID != "reporter" || token.Value != "secret" {
+		t.Fatalf("resolved token = %#v, want id and value", token)
+	}
+	if _, ok := token.AllowedPipelines["reports"]; !ok {
+		t.Fatalf("allowed pipelines = %#v, want reports", token.AllowedPipelines)
+	}
+	if _, ok := token.AllowedPipelines["weekly"]; !ok {
+		t.Fatalf("allowed pipelines = %#v, want weekly", token.AllowedPipelines)
+	}
+}
+
 func TestNewUploadHTTPHandlerAcceptsDefaultedConfig(t *testing.T) {
 	cfg := uploadHTTPTestConfig()
 	cfg.Server.HTTP.Bind = ""
@@ -97,7 +134,6 @@ func TestUploadHTTPHandlerAuthenticatesAndAcceptsUpload(t *testing.T) {
 	var submitted UploadRequest
 	handler := uploadHTTPHandler{
 		coordinator: fakeUploadCoordinator{
-			canAccept: true,
 			submit: func(_ context.Context, request UploadRequest) (UploadRunRecord, error) {
 				submitted = request
 				body, err := io.ReadAll(request.Body)
@@ -110,12 +146,14 @@ func TestUploadHTTPHandlerAuthenticatesAndAcceptsUpload(t *testing.T) {
 				return UploadRunRecord{ID: "reports.20260603T120000Z.abcdef12", Status: UploadStatusAccepted}, nil
 			},
 		},
-		tokens: map[string]string{"valid-token": "reports"},
+		tokens:          map[string]resolvedUploadToken{"valid-token": uploadHTTPTestToken("reporter", "valid-token", "reports")},
+		uploadPipelines: pipelineIDSet([]string{"reports"}),
 	}
 	recorder := httptest.NewRecorder()
-	request := httptest.NewRequest(http.MethodPost, "/upload", strings.NewReader("archive"))
+	request := httptest.NewRequest(http.MethodPost, "/v1/pipelines/reports/upload", strings.NewReader("archive"))
 	request.Header.Set("Authorization", "Bearer valid-token")
 	request.Header.Set("Content-Type", "application/x-tar")
+	request.Header.Set("Idempotency-Key", "producer.retry:20260603")
 
 	handler.ServeHTTP(recorder, request)
 
@@ -125,6 +163,12 @@ func TestUploadHTTPHandlerAuthenticatesAndAcceptsUpload(t *testing.T) {
 	if submitted.PipelineID != "reports" {
 		t.Fatalf("submitted pipeline = %q, want reports", submitted.PipelineID)
 	}
+	if submitted.TokenID != "reporter" {
+		t.Fatalf("submitted token id = %q, want reporter", submitted.TokenID)
+	}
+	if submitted.IdempotencyKey != "producer.retry:20260603" {
+		t.Fatalf("submitted idempotency key = %q, want producer.retry:20260603", submitted.IdempotencyKey)
+	}
 	var response uploadAcceptedResponse
 	if err := json.Unmarshal(recorder.Body.Bytes(), &response); err != nil {
 		t.Fatalf("decode response: %v", err)
@@ -139,13 +183,14 @@ func TestUploadHTTPHandlerAuthenticatesAndAcceptsUpload(t *testing.T) {
 
 func TestUploadHTTPHandlerRejectsUnauthorizedRequests(t *testing.T) {
 	handler := uploadHTTPHandler{
-		coordinator: fakeUploadCoordinator{canAccept: true},
-		tokens:      map[string]string{"valid-token": "reports"},
+		coordinator:     fakeUploadCoordinator{},
+		tokens:          map[string]resolvedUploadToken{"valid-token": uploadHTTPTestToken("reporter", "valid-token", "reports")},
+		uploadPipelines: pipelineIDSet([]string{"reports"}),
 	}
 
-	for _, authHeader := range []string{"", "Bearer wrong-token"} {
+	for _, authHeader := range []string{"", "Basic valid-token", "Bearer", "Bearer wrong-token"} {
 		recorder := httptest.NewRecorder()
-		request := httptest.NewRequest(http.MethodPost, "/upload", strings.NewReader("archive"))
+		request := httptest.NewRequest(http.MethodPost, "/v1/pipelines/reports/upload", strings.NewReader("archive"))
 		request.Header.Set("Authorization", authHeader)
 		request.Header.Set("Content-Type", "application/x-tar")
 
@@ -160,36 +205,117 @@ func TestUploadHTTPHandlerRejectsUnauthorizedRequests(t *testing.T) {
 	}
 }
 
-func TestUploadHTTPHandlerRejectsUnsupportedOversizedFullQueueAndPipelineID(t *testing.T) {
+func TestUploadHTTPHandlerRejectsForbiddenPipeline(t *testing.T) {
+	handler := uploadHTTPHandler{
+		coordinator: fakeUploadCoordinator{
+			submit: func(context.Context, UploadRequest) (UploadRunRecord, error) {
+				t.Fatal("Submit should not be called")
+				return UploadRunRecord{}, nil
+			},
+		},
+		tokens:          map[string]resolvedUploadToken{"valid-token": uploadHTTPTestToken("reporter", "valid-token", "reports")},
+		uploadPipelines: pipelineIDSet([]string{"reports", "private"}),
+	}
+	recorder := httptest.NewRecorder()
+	request := httptest.NewRequest(http.MethodPost, "/v1/pipelines/private/upload", strings.NewReader("archive"))
+	request.Header.Set("Authorization", "Bearer valid-token")
+	request.Header.Set("Content-Type", "application/x-tar")
+
+	handler.ServeHTTP(recorder, request)
+
+	if recorder.Code != http.StatusForbidden {
+		t.Fatalf("status = %d, want %d; body = %q", recorder.Code, http.StatusForbidden, recorder.Body.String())
+	}
+	if strings.Contains(recorder.Body.String(), "valid-token") {
+		t.Fatalf("forbidden response exposed token: %q", recorder.Body.String())
+	}
+}
+
+func TestUploadHTTPHandlerRejectsInvalidPathAndRemovedLegacyUpload(t *testing.T) {
+	handler := uploadHTTPHandler{
+		coordinator: fakeUploadCoordinator{
+			submit: func(context.Context, UploadRequest) (UploadRunRecord, error) {
+				t.Fatal("Submit should not be called")
+				return UploadRunRecord{}, nil
+			},
+		},
+		tokens:          map[string]resolvedUploadToken{"valid-token": uploadHTTPTestToken("reporter", "valid-token", "reports")},
+		uploadPipelines: pipelineIDSet([]string{"reports"}),
+	}
+	tests := []struct {
+		name       string
+		url        string
+		wantStatus int
+	}{
+		{name: "legacy upload", url: "/upload", wantStatus: http.StatusNotFound},
+		{name: "missing pipeline", url: "/v1/pipelines//upload", wantStatus: http.StatusNotFound},
+		{name: "extra segment", url: "/v1/pipelines/reports/upload/extra", wantStatus: http.StatusNotFound},
+		{name: "invalid pipeline id", url: "/v1/pipelines/.reports/upload", wantStatus: http.StatusBadRequest},
+		{name: "pipeline query", url: "/v1/pipelines/reports/upload?pipeline=other", wantStatus: http.StatusBadRequest},
+		{name: "pipeline id query", url: "/v1/pipelines/reports/upload?pipeline_id=other", wantStatus: http.StatusBadRequest},
+		{name: "unknown upload pipeline", url: "/v1/pipelines/missing/upload", wantStatus: http.StatusNotFound},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			recorder := httptest.NewRecorder()
+			request := httptest.NewRequest(http.MethodPost, tt.url, strings.NewReader("archive"))
+			request.Header.Set("Authorization", "Bearer valid-token")
+			request.Header.Set("Content-Type", "application/x-tar")
+
+			handler.ServeHTTP(recorder, request)
+
+			if recorder.Code != tt.wantStatus {
+				t.Fatalf("status = %d, want %d; body = %q", recorder.Code, tt.wantStatus, recorder.Body.String())
+			}
+		})
+	}
+}
+
+func TestUploadHTTPHandlerRejectsUnsupportedContentTypeAndInvalidKey(t *testing.T) {
 	tests := []struct {
 		name        string
-		canAccept   bool
 		url         string
 		contentType string
+		keyValues   []string
 		body        io.Reader
 		wantStatus  int
 	}{
 		{
 			name:        "unsupported content type",
-			canAccept:   true,
-			url:         "/upload",
+			url:         "/v1/pipelines/reports/upload",
 			contentType: "application/zip",
 			body:        strings.NewReader("archive"),
 			wantStatus:  http.StatusUnsupportedMediaType,
 		},
 		{
-			name:        "full queue",
-			canAccept:   false,
-			url:         "/upload",
+			name:        "invalid key syntax",
+			url:         "/v1/pipelines/reports/upload",
 			contentType: "application/x-tar",
-			body:        &countingReader{reader: strings.NewReader("archive")},
-			wantStatus:  http.StatusServiceUnavailable,
+			keyValues:   []string{"bad key"},
+			body:        strings.NewReader("archive"),
+			wantStatus:  http.StatusBadRequest,
 		},
 		{
-			name:        "submitted pipeline id",
-			canAccept:   true,
-			url:         "/upload?pipeline_id=reports",
+			name:        "empty key",
+			url:         "/v1/pipelines/reports/upload",
 			contentType: "application/x-tar",
+			keyValues:   []string{""},
+			body:        strings.NewReader("archive"),
+			wantStatus:  http.StatusBadRequest,
+		},
+		{
+			name:        "too long key",
+			url:         "/v1/pipelines/reports/upload",
+			contentType: "application/x-tar",
+			keyValues:   []string{strings.Repeat("a", 129)},
+			body:        strings.NewReader("archive"),
+			wantStatus:  http.StatusBadRequest,
+		},
+		{
+			name:        "multiple keys",
+			url:         "/v1/pipelines/reports/upload",
+			contentType: "application/x-tar",
+			keyValues:   []string{"one", "two"},
 			body:        strings.NewReader("archive"),
 			wantStatus:  http.StatusBadRequest,
 		},
@@ -198,27 +324,27 @@ func TestUploadHTTPHandlerRejectsUnsupportedOversizedFullQueueAndPipelineID(t *t
 		t.Run(tt.name, func(t *testing.T) {
 			handler := uploadHTTPHandler{
 				coordinator: fakeUploadCoordinator{
-					canAccept: tt.canAccept,
 					submit: func(context.Context, UploadRequest) (UploadRunRecord, error) {
 						t.Fatal("Submit should not be called")
 						return UploadRunRecord{}, nil
 					},
 				},
-				tokens: map[string]string{"valid-token": "reports"},
+				tokens:          map[string]resolvedUploadToken{"valid-token": uploadHTTPTestToken("reporter", "valid-token", "reports")},
+				uploadPipelines: pipelineIDSet([]string{"reports"}),
 			}
 			recorder := httptest.NewRecorder()
 			request := httptest.NewRequest(http.MethodPost, tt.url, tt.body)
 			request.Header.Set("Authorization", "Bearer valid-token")
 			request.Header.Set("Content-Type", tt.contentType)
+			for _, value := range tt.keyValues {
+				request.Header.Add("Idempotency-Key", value)
+			}
 
 			handler.ServeHTTP(recorder, request)
 
 			if recorder.Code != tt.wantStatus {
 				t.Fatalf("status = %d, want %d; body = %q", recorder.Code, tt.wantStatus, recorder.Body.String())
 			}
-			if reader, ok := tt.body.(*countingReader); ok && reader.reads != 0 {
-				t.Fatalf("full queue read body %d time(s), want zero", reader.reads)
-			}
 		})
 	}
 }
@@ -228,9 +354,13 @@ func TestUploadHTTPHandlerMapsSubmitErrors(t *testing.T) {
 		name       string
 		err        error
 		wantStatus int
+		wantBody   string
 	}{
 		{name: "oversized", err: ingest.ErrUploadTooLarge, wantStatus: http.StatusRequestEntityTooLarge},
 		{name: "unsupported", err: ingest.ErrUnsupportedContentType, wantStatus: http.StatusUnsupportedMediaType},
+		{name: "full queue", err: UploadQueueFullError{QueueSize: 1}, wantStatus: http.StatusServiceUnavailable},
+		{name: "idempotency conflict", err: UploadIdempotencyConflictError{}, wantStatus: http.StatusConflict, wantBody: "different source manifest"},
+		{name: "idempotency in progress", err: UploadIdempotencyConflictError{Retryable: true}, wantStatus: http.StatusConflict, wantBody: `"retryable":true`},
 		{name: "malformed", err: errors.New("malformed archive"), wantStatus: http.StatusBadRequest},
 	}
 	for _, tt := range tests {
@@ -242,10 +372,11 @@ func TestUploadHTTPHandlerMapsSubmitErrors(t *testing.T) {
 						return UploadRunRecord{}, tt.err
 					},
 				},
-				tokens: map[string]string{"valid-token": "reports"},
+				tokens:          map[string]resolvedUploadToken{"valid-token": uploadHTTPTestToken("reporter", "valid-token", "reports")},
+				uploadPipelines: pipelineIDSet([]string{"reports"}),
 			}
 			recorder := httptest.NewRecorder()
-			request := httptest.NewRequest(http.MethodPost, "/upload", strings.NewReader("archive"))
+			request := httptest.NewRequest(http.MethodPost, "/v1/pipelines/reports/upload", strings.NewReader("archive"))
 			request.Header.Set("Authorization", "Bearer valid-token")
 			request.Header.Set("Content-Type", "application/x-tar")
 
@@ -254,6 +385,9 @@ func TestUploadHTTPHandlerMapsSubmitErrors(t *testing.T) {
 			if recorder.Code != tt.wantStatus {
 				t.Fatalf("status = %d, want %d; body = %q", recorder.Code, tt.wantStatus, recorder.Body.String())
 			}
+			if tt.wantBody != "" && !strings.Contains(recorder.Body.String(), tt.wantBody) {
+				t.Fatalf("body = %q, want substring %q", recorder.Body.String(), tt.wantBody)
+			}
 		})
 	}
 }
@@ -275,7 +409,8 @@ func TestUploadHTTPHandlerRunStatusAndHealth(t *testing.T) {
 				}, true
 			},
 		},
-		tokens: map[string]string{"valid-token": "reports"},
+		tokens:          map[string]resolvedUploadToken{"valid-token": uploadHTTPTestToken("reporter", "valid-token", "reports")},
+		uploadPipelines: pipelineIDSet([]string{"reports"}),
 	}
 
 	recorder := httptest.NewRecorder()
@@ -331,7 +466,6 @@ func uploadHTTPTestConfig() config.Config {
 			Source: config.Backend{
 				Backend: config.BackendHTTPUpload,
 				Upload: config.HTTPUpload{
-					TokenEnv:      "UPLOAD_TOKEN",
 					StagingPath:   "/tmp/distributor-test/reports",
 					MaxUploadSize: &size,
 				},
@@ -343,6 +477,11 @@ func uploadHTTPTestConfig() config.Config {
 				Publish: &config.PublishPolicy{Source: true},
 			}},
 		}},
+		UploadTokens: []config.UploadToken{{
+			ID:             "reporter",
+			TokenEnv:       "UPLOAD_TOKEN",
+			AllowPipelines: []string{"reports"},
+		}},
 	}
 	config.ApplyDefaults(&cfg)
 	return cfg
@@ -353,3 +492,11 @@ func uploadHTTPTestEnvironment(values map[string]string) config.Environment {
 		return "", false
 	})
 }
+
+func uploadHTTPTestToken(id, value string, pipelines ...string) resolvedUploadToken {
+	return resolvedUploadToken{
+		ID:               id,
+		Value:            value,
+		AllowedPipelines: pipelineIDSet(pipelines),
+	}
+}

internal/config/backend_view_test.go

@@ -53,12 +53,15 @@ func TestBackendViewValidationKeepsHTTPUploadSourceOnly(t *testing.T) {
 		ID: "reports",
 		Source: Backend{
 			Backend: BackendHTTPUpload,
-			Upload:  HTTPUpload{TokenEnv: "UPLOAD_TOKEN"},
 		},
 		Destinations: []Destination{{
 			ID:      "archive",
 			Backend: BackendHTTPUpload,
 		}},
+	}}, UploadTokens: []UploadToken{{
+		ID:             "reporter",
+		TokenEnv:       "UPLOAD_TOKEN",
+		AllowPipelines: []string{"reports"},
 	}}}
 	ApplyDefaults(&cfg)
 

internal/config/config.go

@@ -1,9 +1,10 @@
 package config
 
 type Config struct {
-	Server    Server     `yaml:"server"`
-	Secrets   Secrets    `yaml:"secrets"`
-	Pipelines []Pipeline `yaml:"pipelines"`
+	Server       Server        `yaml:"server"`
+	Secrets      Secrets       `yaml:"secrets"`
+	UploadTokens []UploadToken `yaml:"upload_tokens"`
+	Pipelines    []Pipeline    `yaml:"pipelines"`
 }
 
 type Server struct {
@@ -23,6 +24,12 @@ type Secrets struct {
 	Directory string `yaml:"directory"`
 }
 
+type UploadToken struct {
+	ID             string   `yaml:"id"`
+	TokenEnv       string   `yaml:"token_env"`
+	AllowPipelines []string `yaml:"allow_pipelines"`
+}
+
 type Pipeline struct {
 	ID           string           `yaml:"id"`
 	Source       Backend          `yaml:"source"`
@@ -68,7 +75,6 @@ type Backend struct {
 }
 
 type HTTPUpload struct {
-	TokenEnv      string    `yaml:"token_env"`
 	StagingPath   string    `yaml:"staging_path"`
 	MaxUploadSize *ByteSize `yaml:"max_upload_size"`
 }

internal/config/load_test.go

@@ -256,13 +256,17 @@ pipelines:
   - id: weather-daily
     source:
       backend: http_upload
-      token_env: WEATHER_DAILY_UPLOAD_TOKEN
       staging_path: /srv/distributor/staging/weather-daily
       max_upload_size: 32MB
     destinations:
       - id: archive
         backend: local
         path: /archive
+upload_tokens:
+  - id: weather-reporter
+    token_env: WEATHER_DAILY_UPLOAD_TOKEN
+    allow_pipelines:
+      - weather-daily
 `)
 
 	server := cfg.Server.HTTP
@@ -289,9 +293,6 @@ pipelines:
 	if got, want := source.Backend, BackendHTTPUpload; got != want {
 		t.Fatalf("source.backend = %q, want %q", got, want)
 	}
-	if got, want := source.Upload.TokenEnv, "WEATHER_DAILY_UPLOAD_TOKEN"; got != want {
-		t.Fatalf("source.token_env = %q, want %q", got, want)
-	}
 	if got, want := source.Upload.StagingPath, "/srv/distributor/staging/weather-daily"; got != want {
 		t.Fatalf("source.staging_path = %q, want %q", got, want)
 	}
@@ -309,11 +310,15 @@ pipelines:
   - id: weather-daily
     source:
       backend: http_upload
-      token_env: WEATHER_DAILY_UPLOAD_TOKEN
     destinations:
       - id: archive
         backend: local
         path: /archive
+upload_tokens:
+  - id: weather-reporter
+    token_env: WEATHER_DAILY_UPLOAD_TOKEN
+    allow_pipelines:
+      - weather-daily
 `)
 
 	source := cfg.Pipelines[0].Source
@@ -325,6 +330,120 @@ pipelines:
 	}
 }
 
+func TestLoadFileAcceptsHTTPUploadTokens(t *testing.T) {
+	tests := map[string]string{
+		"valid multi pipeline token": `
+pipelines:
+  - id: weather-daily
+    source:
+      backend: http_upload
+    destinations:
+      - id: archive
+        backend: local
+        path: /archive/weather
+  - id: calendar-daily
+    source:
+      backend: http_upload
+    destinations:
+      - id: archive
+        backend: local
+        path: /archive/calendar
+upload_tokens:
+  - id: reporter
+    token_env: REPORTER_UPLOAD_TOKEN
+    allow_pipelines:
+      - weather-daily
+      - calendar-daily
+`,
+		"multiple tokens for one pipeline": `
+pipelines:
+  - id: reports
+    source:
+      backend: http_upload
+    destinations:
+      - id: archive
+        backend: local
+        path: /archive
+upload_tokens:
+  - id: reporter-a
+    token_env: REPORTER_A_UPLOAD_TOKEN
+    allow_pipelines:
+      - reports
+  - id: reporter-b
+    token_env: REPORTER_B_UPLOAD_TOKEN
+    allow_pipelines:
+      - reports
+`,
+	}
+	for name, body := range tests {
+		t.Run(name, func(t *testing.T) {
+			loadConfig(t, body)
+		})
+	}
+}
+
+func TestLoadFileRejectsInvalidUploadTokens(t *testing.T) {
+	tests := map[string]struct {
+		body string
+		want string
+	}{
+		"missing token list": {
+			body: `pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+			want: "upload_tokens is required",
+		},
+		"duplicate token ids": {
+			body: `upload_tokens: [{id: reporter, token_env: ONE_UPLOAD_TOKEN, allow_pipelines: [reports]}, {id: reporter, token_env: TWO_UPLOAD_TOKEN, allow_pipelines: [reports]}]
+pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+			want: "upload token id reporter is duplicated",
+		},
+		"duplicate allowlist entries": {
+			body: `upload_tokens: [{id: reporter, token_env: UPLOAD_TOKEN, allow_pipelines: [reports, reports]}]
+pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+			want: "allow_pipelines contains duplicate pipeline id reports",
+		},
+		"unknown allowed pipeline id": {
+			body: `upload_tokens: [{id: reporter, token_env: UPLOAD_TOKEN, allow_pipelines: [missing]}]
+pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+			want: "references unknown pipeline missing",
+		},
+		"non upload allowed pipeline id": {
+			body: `upload_tokens: [{id: reporter, token_env: UPLOAD_TOKEN, allow_pipelines: [reports]}, {id: uploader, token_env: OTHER_UPLOAD_TOKEN, allow_pipelines: [upload]}]
+pipelines: [{id: reports, source: {backend: local, path: /source}, destinations: [{id: archive, backend: local, path: /archive}]}, {id: upload, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive-upload}]}]`,
+			want: "references non-http_upload pipeline reports",
+		},
+		"upload pipeline not allowed": {
+			body: `upload_tokens: [{id: reporter, token_env: UPLOAD_TOKEN, allow_pipelines: [reports]}]
+pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}, {id: other, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive-other}]}]`,
+			want: "http_upload pipeline other is not allowed by any upload token",
+		},
+		"missing token id": {
+			body: `upload_tokens: [{token_env: UPLOAD_TOKEN, allow_pipelines: [reports]}]
+pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+			want: "upload_tokens[0].id is required",
+		},
+		"invalid token id": {
+			body: `upload_tokens: [{id: ".reporter", token_env: UPLOAD_TOKEN, allow_pipelines: [reports]}]
+pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+			want: "upload_tokens[0].id must be a slug-like identifier",
+		},
+		"missing token env": {
+			body: `upload_tokens: [{id: reporter, allow_pipelines: [reports]}]
+pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+			want: "upload_tokens[0].token_env is required",
+		},
+		"missing allowlist": {
+			body: `upload_tokens: [{id: reporter, token_env: UPLOAD_TOKEN}]
+pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+			want: "upload_tokens[0].allow_pipelines is required",
+		},
+	}
+	for name, tt := range tests {
+		t.Run(name, func(t *testing.T) {
+			assertLoadError(t, tt.body, tt.want)
+		})
+	}
+}
+
 func TestLoadFileValidBackendConfigs(t *testing.T) {
 	tests := map[string]string{
 		"local": `
@@ -515,16 +634,22 @@ func TestLoadFileRejectsInvalidS3Config(t *testing.T) {
 
 func TestLoadFileRejectsInvalidHTTPUploadConfig(t *testing.T) {
 	tests := map[string]string{
-		"server size":             `server: {http: {max_upload_size: 20XB}}`,
-		"source size":             `pipelines: [{id: reports, source: {backend: http_upload, token_env: UPLOAD_TOKEN, max_upload_size: 20XB}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
-		"zero source size":        `pipelines: [{id: reports, source: {backend: http_upload, token_env: UPLOAD_TOKEN, max_upload_size: 0B}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+		"server size": `server: {http: {max_upload_size: 20XB}}`,
+		"source size": `upload_tokens: [{id: reporter, token_env: UPLOAD_TOKEN, allow_pipelines: [reports]}]
+pipelines: [{id: reports, source: {backend: http_upload, max_upload_size: 20XB}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+		"zero source size": `upload_tokens: [{id: reporter, token_env: UPLOAD_TOKEN, allow_pipelines: [reports]}]
+pipelines: [{id: reports, source: {backend: http_upload, max_upload_size: 0B}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
 		"server duration":         `server: {http: {retention: forever}}`,
 		"zero server duration":    `server: {http: {retention: 0s}}`,
-		"missing token env":       `pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+		"missing upload tokens":   `pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
 		"destination http upload": `pipelines: [{id: reports, source: {backend: local, path: /source}, destinations: [{id: ingest, backend: http_upload}]}]`,
-		"literal token":           `pipelines: [{id: reports, source: {backend: http_upload, token: secret, token_env: UPLOAD_TOKEN}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
-		"unknown server field":    `server: {http: {surprise: true}}`,
-		"unknown source field":    `pipelines: [{id: reports, source: {backend: http_upload, token_env: UPLOAD_TOKEN, surprise: true}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+		"literal token": `upload_tokens: [{id: reporter, token: secret, token_env: UPLOAD_TOKEN, allow_pipelines: [reports]}]
+pipelines: [{id: reports, source: {backend: http_upload}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+		"unknown server field": `server: {http: {surprise: true}}`,
+		"legacy source token env": `upload_tokens: [{id: reporter, token_env: UPLOAD_TOKEN, allow_pipelines: [reports]}]
+pipelines: [{id: reports, source: {backend: http_upload, token_env: UPLOAD_TOKEN}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
+		"unknown source field": `upload_tokens: [{id: reporter, token_env: UPLOAD_TOKEN, allow_pipelines: [reports]}]
+pipelines: [{id: reports, source: {backend: http_upload, surprise: true}, destinations: [{id: archive, backend: local, path: /archive}]}]`,
 	}
 	for name, body := range tests {
 		t.Run(name, func(t *testing.T) {

internal/config/validate.go

@@ -10,6 +10,10 @@ import (
 
 var idPattern = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._-]*$`)
 
+func IsSlugLikeID(value string) bool {
+	return idPattern.MatchString(value)
+}
+
 type ValidationErrors []string
 
 func (e ValidationErrors) Error() string {
@@ -29,11 +33,12 @@ func Validate(cfg Config) error {
 	}
 
 	pipelineIDs := make(map[string]struct{}, len(cfg.Pipelines))
+	uploadPipelineIDs := make(map[string]struct{})
 	for pipelineIndex, pipeline := range cfg.Pipelines {
 		pipelineContext := fmt.Sprintf("pipelines[%d]", pipelineIndex)
 		if pipeline.ID == "" {
 			errs = append(errs, pipelineContext+".id is required")
-		} else if !idPattern.MatchString(pipeline.ID) {
+		} else if !IsSlugLikeID(pipeline.ID) {
 			errs = append(errs, pipelineContext+".id must be a slug-like identifier")
 		} else if _, exists := pipelineIDs[pipeline.ID]; exists {
 			errs = append(errs, "pipeline id "+pipeline.ID+" is duplicated")
@@ -42,6 +47,9 @@ func Validate(cfg Config) error {
 		}
 
 		errs = validateSourceBackend(errs, pipelineContext+".source", pipeline.Source)
+		if pipeline.Source.Backend == BackendHTTPUpload && pipeline.ID != "" {
+			uploadPipelineIDs[pipeline.ID] = struct{}{}
+		}
 		errs = validateValidationPolicy(errs, pipelineContext+".validation", pipeline.Validation)
 		if len(pipeline.Destinations) == 0 {
 			errs = append(errs, pipelineContext+".destinations is required")
@@ -52,7 +60,7 @@ func Validate(cfg Config) error {
 			destinationContext := fmt.Sprintf("%s.destinations[%d]", pipelineContext, destinationIndex)
 			if destination.ID == "" {
 				errs = append(errs, destinationContext+".id is required")
-			} else if !idPattern.MatchString(destination.ID) {
+			} else if !IsSlugLikeID(destination.ID) {
 				errs = append(errs, destinationContext+".id must be a slug-like identifier")
 			} else if _, exists := destinationIDs[destination.ID]; exists {
 				errs = append(errs, "destination id "+destination.ID+" is duplicated in pipeline "+pipeline.ID)
@@ -68,6 +76,8 @@ func Validate(cfg Config) error {
 		}
 	}
 
+	errs = validateUploadTokens(errs, cfg.UploadTokens, pipelineIDs, uploadPipelineIDs)
+
 	if len(errs) > 0 {
 		return errs
 	}
@@ -112,9 +122,6 @@ func validateDestinationBackend(errs ValidationErrors, context string, destinati
 }
 
 func validateHTTPUploadSource(errs ValidationErrors, context string, upload HTTPUpload) ValidationErrors {
-	if upload.TokenEnv == "" {
-		errs = append(errs, context+".token_env is required for http_upload backend")
-	}
 	if upload.StagingPath == "" {
 		errs = append(errs, context+".staging_path is required for http_upload backend")
 	}
@@ -124,6 +131,70 @@ func validateHTTPUploadSource(errs ValidationErrors, context string, upload HTTP
 	return errs
 }
 
+func validateUploadTokens(errs ValidationErrors, tokens []UploadToken, pipelineIDs, uploadPipelineIDs map[string]struct{}) ValidationErrors {
+	if len(uploadPipelineIDs) == 0 {
+		if len(tokens) > 0 {
+			errs = append(errs, "upload_tokens must reference configured http_upload pipelines")
+		}
+		return errs
+	}
+	if len(tokens) == 0 {
+		return append(errs, "upload_tokens is required when any pipeline source backend is http_upload")
+	}
+
+	tokenIDs := make(map[string]struct{}, len(tokens))
+	allowedUploadPipelineIDs := make(map[string]struct{}, len(uploadPipelineIDs))
+	for tokenIndex, token := range tokens {
+		context := fmt.Sprintf("upload_tokens[%d]", tokenIndex)
+		if token.ID == "" {
+			errs = append(errs, context+".id is required")
+		} else if !IsSlugLikeID(token.ID) {
+			errs = append(errs, context+".id must be a slug-like identifier")
+		} else if _, exists := tokenIDs[token.ID]; exists {
+			errs = append(errs, "upload token id "+token.ID+" is duplicated")
+		} else {
+			tokenIDs[token.ID] = struct{}{}
+		}
+
+		if token.TokenEnv == "" {
+			errs = append(errs, context+".token_env is required")
+		}
+		if len(token.AllowPipelines) == 0 {
+			errs = append(errs, context+".allow_pipelines is required")
+		}
+
+		seenAllowed := make(map[string]struct{}, len(token.AllowPipelines))
+		for allowIndex, pipelineID := range token.AllowPipelines {
+			allowContext := fmt.Sprintf("%s.allow_pipelines[%d]", context, allowIndex)
+			if pipelineID == "" {
+				errs = append(errs, allowContext+" is required")
+				continue
+			}
+			if _, exists := seenAllowed[pipelineID]; exists {
+				errs = append(errs, context+".allow_pipelines contains duplicate pipeline id "+pipelineID)
+				continue
+			}
+			seenAllowed[pipelineID] = struct{}{}
+			if _, exists := pipelineIDs[pipelineID]; !exists {
+				errs = append(errs, allowContext+" references unknown pipeline "+pipelineID)
+				continue
+			}
+			if _, exists := uploadPipelineIDs[pipelineID]; !exists {
+				errs = append(errs, allowContext+" references non-http_upload pipeline "+pipelineID)
+				continue
+			}
+			allowedUploadPipelineIDs[pipelineID] = struct{}{}
+		}
+	}
+
+	for pipelineID := range uploadPipelineIDs {
+		if _, exists := allowedUploadPipelineIDs[pipelineID]; !exists {
+			errs = append(errs, "http_upload pipeline "+pipelineID+" is not allowed by any upload token")
+		}
+	}
+	return errs
+}
+
 func validateBackend(errs ValidationErrors, context string, backend backendView) ValidationErrors {
 	switch backend.Backend {
 	case "":

pkg/bundle/doc.go

@@ -1,9 +1,104 @@
 // Package bundle provides producer-facing helpers for distributor source
-// bundle manifests.
+// bundles.
 //
-// A source bundle is a local directory containing a manifest.json file and the
-// files listed by that manifest. This package owns the public manifest model,
-// digest calculation, path validation, manifest parsing, manifest building,
-// local bundle writing, and local bundle validation used by Go producer
-// applications.
+// A source bundle is a local directory containing manifest.json and the files
+// listed by that manifest. Producer applications use this package when they
+// need to generate manifests, validate bundles locally, or write complete
+// bundle directories for distributor to discover, upload, or publish.
+//
+// # Bundle Contract
+//
+// The source manifest is the producer-to-distributor contract. It is named by
+// ManifestName, currently "manifest.json", and uses SchemaVersion, currently 1.
+// A Manifest contains:
+//
+//   - SchemaVersion: the source manifest schema version.
+//   - ID: the producer's stable bundle identifier.
+//   - Digest: the canonical digest of the ordered file records.
+//   - Created: an RFC3339 timestamp when marshaled to JSON.
+//   - Files: an ordered list of ManifestFile records.
+//
+// Each ManifestFile records a slash-separated bundle-relative Path, a lowercase
+// sha256:<64 hex> SHA256 digest, and a byte Size. File order is significant for
+// the bundle digest and should be chosen deliberately by the producer. Explicit
+// file lists preserve caller order; scan mode sorts by slash-separated path.
+//
+// # Path Rules
+//
+// Public bundle paths are always slash-separated and relative to the bundle
+// root. ValidateSourcePath rejects empty paths, absolute paths, path traversal,
+// dot segments, backslashes, and reserved manifest/state paths. Source files
+// must be regular files; symlinks and other special files are rejected.
+//
+// BuildManifest with Scan true recursively scans Root, includes regular files
+// including dotfiles, excludes manifest.json and .distributor.json, rejects
+// symlinks, and sorts paths lexically. BuildManifest with Files uses exactly
+// the caller-provided paths and preserves their order. Exactly one selection
+// mode must be used.
+//
+// # Manifest Workflows
+//
+// BuildManifest reads existing files under a local root, calculates each
+// ManifestFile, defaults a zero Created value to the current UTC time, calculates
+// the bundle digest, and validates the result. WriteManifest writes
+// manifest.json and fails if it already exists unless WriteManifestOptions has
+// Overwrite set. LoadManifest reads and parses manifest.json. ParseManifest and
+// MarshalManifest are useful when an application stores or transmits manifest
+// bytes directly; MarshalManifest validates before writing deterministic,
+// indented JSON with a trailing newline.
+//
+// ValidateManifest checks manifest-only semantics, including schema version,
+// required fields, path safety, duplicate file paths, digest syntax, file sizes,
+// and bundle digest. ValidateBundle checks a supplied Manifest against local
+// files under a root, including existence, regular-file type, size, SHA-256
+// digest, path safety, and bundle digest.
+//
+// # Complete Bundle Writing
+//
+// WriteBundle is the most convenient producer workflow when source files live
+// outside the final bundle directory. It copies each BundleFile.SourcePath into
+// a staged bundle at BundleFile.Path, builds and writes a compliant manifest,
+// validates the staged bundle, and promotes it to WriteBundleOptions.Root.
+// Overwrite permits replacement of an existing bundle root using a best-effort
+// sibling temporary and backup strategy.
+//
+// # Digest Helpers
+//
+// FileDigest returns the sha256:<64 hex> digest for file bytes. BundleDigest
+// returns the canonical bundle digest for an ordered []ManifestFile.
+// CanonicalFilePayload returns the JSON payload used by BundleDigest, which is
+// mainly useful for tests and diagnostics. ValidateDigest checks digest syntax.
+//
+// Example: build and write a manifest for files already under a bundle root.
+//
+//	root := "/var/lib/reports/daily-2026-06-06"
+//	manifest, err := bundle.BuildManifest(bundle.BuildOptions{
+//		Root:  root,
+//		ID:    "reports.daily.2026-06-06",
+//		Files: []string{"report.md", "summary.txt"},
+//	})
+//	if err != nil {
+//		return err
+//	}
+//	if err := bundle.WriteManifest(root, manifest, bundle.WriteManifestOptions{}); err != nil {
+//		return err
+//	}
+//	if err := bundle.ValidateBundle(root, manifest); err != nil {
+//		return err
+//	}
+//
+// Example: create a complete bundle from producer-generated files.
+//
+//	manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{
+//		Root: "/var/lib/distributor-source/daily-2026-06-06",
+//		ID:   "reports.daily.2026-06-06",
+//		Files: []bundle.BundleFile{
+//			{SourcePath: "/tmp/report.md", Path: "report.md"},
+//			{SourcePath: "/tmp/summary.txt", Path: "summary.txt"},
+//		},
+//	})
+//	if err != nil {
+//		return err
+//	}
+//	_ = manifest
 package bundle

pkg/upload/archive.go

@@ -0,0 +1,104 @@
+package upload
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"time"
+
+	"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+)
+
+func archiveBundle(root string, manifest bundle.Manifest) ([]byte, error) {
+	var output bytes.Buffer
+	gzipWriter := gzip.NewWriter(&output)
+	tarWriter := tar.NewWriter(gzipWriter)
+
+	manifestData, err := bundle.MarshalManifest(manifest)
+	if err != nil {
+		return nil, err
+	}
+	if err := writeTarEntry(tarWriter, bundle.ManifestName, manifestData, 0o600, manifest.Created); err != nil {
+		_ = tarWriter.Close()
+		_ = gzipWriter.Close()
+		return nil, err
+	}
+	for _, manifestFile := range manifest.Files {
+		fullPath := filepath.Join(root, filepath.FromSlash(manifestFile.Path))
+		info, err := os.Lstat(fullPath)
+		if err != nil {
+			_ = tarWriter.Close()
+			_ = gzipWriter.Close()
+			return nil, fmt.Errorf("file %q stat: %w", manifestFile.Path, err)
+		}
+		if !info.Mode().IsRegular() {
+			_ = tarWriter.Close()
+			_ = gzipWriter.Close()
+			return nil, fmt.Errorf("file %q must be a regular file", manifestFile.Path)
+		}
+		file, err := os.Open(fullPath)
+		if err != nil {
+			_ = tarWriter.Close()
+			_ = gzipWriter.Close()
+			return nil, fmt.Errorf("file %q open: %w", manifestFile.Path, err)
+		}
+		if err := writeTarFile(tarWriter, manifestFile.Path, file, info.Mode().Perm(), info.ModTime(), info.Size()); err != nil {
+			_ = file.Close()
+			_ = tarWriter.Close()
+			_ = gzipWriter.Close()
+			return nil, err
+		}
+		if err := file.Close(); err != nil {
+			_ = tarWriter.Close()
+			_ = gzipWriter.Close()
+			return nil, fmt.Errorf("file %q close: %w", manifestFile.Path, err)
+		}
+	}
+	if err := tarWriter.Close(); err != nil {
+		_ = gzipWriter.Close()
+		return nil, fmt.Errorf("close tar archive: %w", err)
+	}
+	if err := gzipWriter.Close(); err != nil {
+		return nil, fmt.Errorf("close gzip archive: %w", err)
+	}
+	return output.Bytes(), nil
+}
+
+func writeTarEntry(writer *tar.Writer, name string, data []byte, mode int64, modTime time.Time) error {
+	header := &tar.Header{
+		Name:    name,
+		Mode:    mode,
+		Size:    int64(len(data)),
+		ModTime: modTime,
+	}
+	if err := writer.WriteHeader(header); err != nil {
+		return fmt.Errorf("write tar header %q: %w", name, err)
+	}
+	if _, err := writer.Write(data); err != nil {
+		return fmt.Errorf("write tar entry %q: %w", name, err)
+	}
+	return nil
+}
+
+func writeTarFile(writer *tar.Writer, name string, file *os.File, mode os.FileMode, modTime time.Time, size int64) error {
+	if mode == 0 {
+		mode = 0o600
+	}
+	header := &tar.Header{
+		Name:    name,
+		Mode:    int64(mode),
+		Size:    size,
+		ModTime: modTime,
+	}
+	if err := writer.WriteHeader(header); err != nil {
+		return fmt.Errorf("write tar header %q: %w", name, err)
+	}
+	if _, err := io.Copy(writer, file); err != nil {
+		return fmt.Errorf("write tar entry %q: %w", name, err)
+	}
+	return nil
+}

pkg/upload/client.go

@@ -0,0 +1,444 @@
+package upload
+
+import (
+	"bytes"
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+	"net/url"
+	"os"
+	"path"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+
+	"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+)
+
+const (
+	runsPath              = "runs"
+	idempotencyKeyHeader  = "Idempotency-Key"
+	defaultHTTPTimeout    = 30 * time.Second
+	defaultRetryAttempts  = 3
+	defaultRetryBaseDelay = 100 * time.Millisecond
+	defaultRetryMaxDelay  = time.Second
+	uploadContentTypeGzip = "application/gzip"
+	authorizationPrefix   = "Bearer "
+	redactedSecret        = "[redacted]"
+)
+
+var pipelineIDPattern = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._-]*$`)
+
+func NewClient(opts ClientOptions) (*Client, error) {
+	endpoint, err := cleanEndpoint(opts.Endpoint)
+	if err != nil {
+		return nil, err
+	}
+	if opts.Token == "" {
+		return nil, fmt.Errorf("token is required")
+	}
+	retry, err := cleanRetryOptions(opts.Retry)
+	if err != nil {
+		return nil, err
+	}
+	httpClient := opts.HTTPClient
+	if httpClient == nil {
+		httpClient = &http.Client{Timeout: defaultHTTPTimeout}
+	}
+	return &Client{
+		endpoint:   endpoint,
+		token:      opts.Token,
+		httpClient: httpClient,
+		retry:      retry,
+	}, nil
+}
+
+func (c *Client) UploadBundle(ctx context.Context, opts UploadBundleOptions) (Result, error) {
+	if c == nil {
+		return Result{}, fmt.Errorf("client is nil")
+	}
+	if opts.Validate && opts.DisableValidation {
+		return Result{}, fmt.Errorf("validate and disable validation cannot both be set")
+	}
+	if err := validatePipelineID(opts.PipelineID); err != nil {
+		return Result{}, err
+	}
+	if opts.Root == "" {
+		return Result{}, fmt.Errorf("root is required")
+	}
+	manifest, err := bundle.LoadManifest(opts.Root)
+	if err != nil {
+		return Result{}, c.redactError(err)
+	}
+	if shouldValidateBundle(opts.Validate, opts.DisableValidation) {
+		if err := bundle.ValidateBundle(opts.Root, manifest); err != nil {
+			return Result{}, c.redactError(err)
+		}
+	}
+	archive, err := archiveBundle(opts.Root, manifest)
+	if err != nil {
+		return Result{}, c.redactError(err)
+	}
+	key, err := uploadIdempotencyKey(opts.IdempotencyKey)
+	if err != nil {
+		return Result{}, err
+	}
+	return c.uploadArchive(ctx, opts.PipelineID, archive, key)
+}
+
+func (c *Client) UploadFiles(ctx context.Context, opts UploadFilesOptions) (Result, error) {
+	if c == nil {
+		return Result{}, fmt.Errorf("client is nil")
+	}
+	if opts.Validate && opts.DisableValidation {
+		return Result{}, fmt.Errorf("validate and disable validation cannot both be set")
+	}
+	if err := validatePipelineID(opts.PipelineID); err != nil {
+		return Result{}, err
+	}
+	if opts.ID == "" {
+		return Result{}, fmt.Errorf("id is required")
+	}
+	if len(opts.Files) == 0 {
+		return Result{}, fmt.Errorf("files is required")
+	}
+	tempRoot, err := os.MkdirTemp(opts.TempDir, "distributor-upload-*")
+	if err != nil {
+		return Result{}, c.redactError(fmt.Errorf("create temporary bundle root: %w", err))
+	}
+	defer func() {
+		_ = os.RemoveAll(tempRoot)
+	}()
+	localBundleRoot := filepath.Join(tempRoot, "bundle")
+	manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{
+		Root:    localBundleRoot,
+		ID:      opts.ID,
+		Created: opts.Created,
+		Files:   opts.Files,
+	})
+	if err != nil {
+		return Result{}, c.redactError(err)
+	}
+	if shouldValidateBundle(opts.Validate, opts.DisableValidation) {
+		if err := bundle.ValidateBundle(localBundleRoot, manifest); err != nil {
+			return Result{}, c.redactError(err)
+		}
+	}
+	archive, err := archiveBundle(localBundleRoot, manifest)
+	if err != nil {
+		return Result{}, c.redactError(err)
+	}
+	key, err := uploadIdempotencyKey(opts.IdempotencyKey)
+	if err != nil {
+		return Result{}, err
+	}
+	return c.uploadArchive(ctx, opts.PipelineID, archive, key)
+}
+
+func (c *Client) Status(ctx context.Context, runID string) (RunStatus, error) {
+	if c == nil {
+		return RunStatus{}, fmt.Errorf("client is nil")
+	}
+	if runID == "" {
+		return RunStatus{}, fmt.Errorf("run id is required")
+	}
+	if ctx == nil {
+		ctx = context.Background()
+	}
+	if err := ctx.Err(); err != nil {
+		return RunStatus{}, err
+	}
+	request, err := http.NewRequestWithContext(ctx, http.MethodGet, c.statusURL(runID), nil)
+	if err != nil {
+		return RunStatus{}, c.redactError(err)
+	}
+	c.authorize(request)
+	response, err := c.httpClient.Do(request)
+	if err != nil {
+		return RunStatus{}, c.redactError(err)
+	}
+	defer response.Body.Close()
+	if response.StatusCode != http.StatusOK {
+		return RunStatus{}, c.responseError(response)
+	}
+	var status RunStatus
+	if err := json.NewDecoder(response.Body).Decode(&status); err != nil {
+		return RunStatus{}, c.redactError(fmt.Errorf("decode run status: %w", err))
+	}
+	return status, nil
+}
+
+func (c *Client) uploadArchive(ctx context.Context, pipelineID string, archive []byte, idempotencyKey string) (Result, error) {
+	if ctx == nil {
+		ctx = context.Background()
+	}
+	var lastErr error
+	for attempt := 1; attempt <= c.retry.MaxAttempts; attempt++ {
+		if err := ctx.Err(); err != nil {
+			return Result{}, err
+		}
+		result, retry, err := c.uploadAttempt(ctx, pipelineID, archive, idempotencyKey)
+		if err == nil {
+			return result, nil
+		}
+		lastErr = err
+		if !retry || attempt == c.retry.MaxAttempts {
+			return Result{}, err
+		}
+		if err := waitForRetry(ctx, retryDelay(c.retry, attempt)); err != nil {
+			return Result{}, err
+		}
+	}
+	return Result{}, lastErr
+}
+
+func (c *Client) uploadAttempt(ctx context.Context, pipelineID string, archive []byte, idempotencyKey string) (Result, bool, error) {
+	request, err := http.NewRequestWithContext(ctx, http.MethodPost, c.uploadURL(pipelineID), bytes.NewReader(archive))
+	if err != nil {
+		return Result{}, false, c.redactError(err)
+	}
+	c.authorize(request)
+	request.Header.Set("Content-Type", uploadContentTypeGzip)
+	request.Header.Set(idempotencyKeyHeader, idempotencyKey)
+
+	response, err := c.httpClient.Do(request)
+	if err != nil {
+		if ctxErr := ctx.Err(); ctxErr != nil {
+			return Result{}, false, ctxErr
+		}
+		return Result{}, isRetryableNetworkError(err), c.redactError(err)
+	}
+	defer response.Body.Close()
+
+	if response.StatusCode == http.StatusAccepted {
+		var result Result
+		if err := json.NewDecoder(response.Body).Decode(&result); err != nil {
+			return Result{}, false, c.redactError(fmt.Errorf("decode upload response: %w", err))
+		}
+		if result.RunID == "" {
+			return Result{}, false, fmt.Errorf("upload response run_id is required")
+		}
+		return result, false, nil
+	}
+	err = c.responseError(response)
+	return Result{}, response.StatusCode == http.StatusServiceUnavailable, err
+}
+
+func (c *Client) authorize(request *http.Request) {
+	request.Header.Set("Authorization", authorizationPrefix+c.token)
+}
+
+func (c *Client) uploadURL(pipelineID string) string {
+	return joinEndpointPath(c.endpoint, "v1", "pipelines", pipelineID, "upload")
+}
+
+func (c *Client) statusURL(runID string) string {
+	return joinEndpointPath(c.endpoint, runsPath, runID)
+}
+
+func (c *Client) responseError(response *http.Response) error {
+	body, readErr := io.ReadAll(response.Body)
+	message := http.StatusText(response.StatusCode)
+	retryable := false
+	if readErr == nil && len(body) > 0 {
+		var decoded struct {
+			Error     string `json:"error"`
+			Retryable bool   `json:"retryable"`
+		}
+		if err := json.Unmarshal(body, &decoded); err == nil && decoded.Error != "" {
+			message = decoded.Error
+			retryable = decoded.Retryable
+		} else if trimmed := strings.TrimSpace(string(body)); trimmed != "" {
+			message = trimmed
+		}
+	}
+	message = c.redactString(message)
+	status := c.redactString(response.Status)
+	httpErr := HTTPError{
+		StatusCode: response.StatusCode,
+		Status:     status,
+		Message:    message,
+		Retryable:  retryable,
+	}
+	if response.StatusCode == http.StatusConflict {
+		return &IdempotencyConflictError{HTTPError: httpErr}
+	}
+	return &httpErr
+}
+
+func (c *Client) redactError(err error) error {
+	if err == nil {
+		return nil
+	}
+	message := c.redactString(err.Error())
+	if message == err.Error() {
+		return err
+	}
+	return errors.New(message)
+}
+
+func (c *Client) redactString(value string) string {
+	if c == nil || c.token == "" {
+		return value
+	}
+	return strings.ReplaceAll(value, c.token, redactedSecret)
+}
+
+func cleanEndpoint(value string) (string, error) {
+	if value == "" {
+		return "", fmt.Errorf("endpoint is required")
+	}
+	parsed, err := url.Parse(value)
+	if err != nil {
+		return "", fmt.Errorf("endpoint is invalid: %w", err)
+	}
+	if parsed.Scheme != "http" && parsed.Scheme != "https" {
+		return "", fmt.Errorf("endpoint scheme must be http or https")
+	}
+	if parsed.Host == "" {
+		return "", fmt.Errorf("endpoint host is required")
+	}
+	if parsed.User != nil {
+		return "", fmt.Errorf("endpoint userinfo is not supported")
+	}
+	if parsed.RawQuery != "" || parsed.Fragment != "" {
+		return "", fmt.Errorf("endpoint must not include query or fragment")
+	}
+	parsed.Path = strings.TrimRight(parsed.Path, "/")
+	parsed.RawPath = ""
+	return parsed.String(), nil
+}
+
+func cleanRetryOptions(opts RetryOptions) (RetryOptions, error) {
+	if opts.MaxAttempts < 0 {
+		return RetryOptions{}, fmt.Errorf("retry max attempts must be non-negative")
+	}
+	if opts.BaseDelay < 0 {
+		return RetryOptions{}, fmt.Errorf("retry base delay must be non-negative")
+	}
+	if opts.MaxDelay < 0 {
+		return RetryOptions{}, fmt.Errorf("retry max delay must be non-negative")
+	}
+	if opts.MaxAttempts == 0 {
+		opts.MaxAttempts = defaultRetryAttempts
+	}
+	if opts.BaseDelay == 0 {
+		opts.BaseDelay = defaultRetryBaseDelay
+	}
+	if opts.MaxDelay == 0 {
+		opts.MaxDelay = defaultRetryMaxDelay
+	}
+	if opts.MaxDelay < opts.BaseDelay {
+		return RetryOptions{}, fmt.Errorf("retry max delay must be greater than or equal to base delay")
+	}
+	return opts, nil
+}
+
+func uploadIdempotencyKey(value string) (string, error) {
+	if value == "" {
+		return randomIdempotencyKey()
+	}
+	if err := validateIdempotencyKey(value); err != nil {
+		return "", err
+	}
+	return value, nil
+}
+
+func validatePipelineID(value string) error {
+	if value == "" {
+		return fmt.Errorf("pipeline id is required")
+	}
+	if !pipelineIDPattern.MatchString(value) {
+		return fmt.Errorf("pipeline id must be a slug-like identifier")
+	}
+	return nil
+}
+
+func validateIdempotencyKey(value string) error {
+	if value == "" {
+		return fmt.Errorf("idempotency key is required")
+	}
+	if len(value) > 128 {
+		return fmt.Errorf("idempotency key must be at most 128 bytes")
+	}
+	for index := 0; index < len(value); index++ {
+		character := value[index]
+		if character >= 'a' && character <= 'z' ||
+			character >= 'A' && character <= 'Z' ||
+			character >= '0' && character <= '9' ||
+			character == '.' ||
+			character == '_' ||
+			character == '-' ||
+			character == ':' {
+			continue
+		}
+		return fmt.Errorf("idempotency key contains unsupported character")
+	}
+	return nil
+}
+
+func randomIdempotencyKey() (string, error) {
+	var data [16]byte
+	if _, err := rand.Read(data[:]); err != nil {
+		return "", fmt.Errorf("generate idempotency key: %w", err)
+	}
+	return hex.EncodeToString(data[:]), nil
+}
+
+func shouldValidateBundle(validate, disable bool) bool {
+	return validate || !disable
+}
+
+func retryDelay(opts RetryOptions, attempt int) time.Duration {
+	delay := opts.BaseDelay
+	for index := 1; index < attempt; index++ {
+		delay *= 2
+		if delay >= opts.MaxDelay {
+			return opts.MaxDelay
+		}
+	}
+	return delay
+}
+
+func waitForRetry(ctx context.Context, delay time.Duration) error {
+	timer := time.NewTimer(delay)
+	defer timer.Stop()
+	select {
+	case <-ctx.Done():
+		return ctx.Err()
+	case <-timer.C:
+		return nil
+	}
+}
+
+func isRetryableNetworkError(err error) bool {
+	if err == nil {
+		return false
+	}
+	var netErr net.Error
+	if errors.As(err, &netErr) {
+		return netErr.Timeout() || netErr.Temporary()
+	}
+	return true
+}
+
+func joinEndpointPath(endpoint string, elements ...string) string {
+	parsed, err := url.Parse(endpoint)
+	if err != nil {
+		return endpoint
+	}
+	parts := []string{}
+	if parsed.Path != "" && parsed.Path != "/" {
+		parts = append(parts, strings.Trim(parsed.Path, "/"))
+	}
+	parts = append(parts, elements...)
+	parsed.Path = "/" + path.Join(parts...)
+	return parsed.String()
+}

pkg/upload/client_test.go

@@ -0,0 +1,641 @@
+package upload
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"context"
+	"encoding/json"
+	"errors"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	sourcebundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+)
+
+func TestNewClientValidatesOptions(t *testing.T) {
+	tests := []struct {
+		name string
+		opts ClientOptions
+	}{
+		{name: "missing endpoint", opts: ClientOptions{Token: "secret"}},
+		{name: "missing token", opts: ClientOptions{Endpoint: "http://127.0.0.1:8080"}},
+		{name: "bad scheme", opts: ClientOptions{Endpoint: "ftp://127.0.0.1:8080", Token: "secret"}},
+		{name: "missing host", opts: ClientOptions{Endpoint: "http:///upload", Token: "secret"}},
+		{name: "query", opts: ClientOptions{Endpoint: "http://127.0.0.1:8080?x=1", Token: "secret"}},
+		{name: "userinfo", opts: ClientOptions{Endpoint: "http://user@127.0.0.1:8080", Token: "secret"}},
+		{name: "negative attempts", opts: ClientOptions{Endpoint: "http://127.0.0.1:8080", Token: "secret", Retry: RetryOptions{MaxAttempts: -1}}},
+		{name: "negative delay", opts: ClientOptions{Endpoint: "http://127.0.0.1:8080", Token: "secret", Retry: RetryOptions{BaseDelay: -1}}},
+		{name: "max below base", opts: ClientOptions{Endpoint: "http://127.0.0.1:8080", Token: "secret", Retry: RetryOptions{BaseDelay: time.Second, MaxDelay: time.Millisecond}}},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if _, err := NewClient(tt.opts); err == nil {
+				t.Fatal("NewClient() error = nil, want error")
+			}
+		})
+	}
+
+	client, err := NewClient(ClientOptions{Endpoint: "http://127.0.0.1:8080/base/", Token: "secret"})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	if got, want := client.uploadURL("reports.daily"), "http://127.0.0.1:8080/base/v1/pipelines/reports.daily/upload"; got != want {
+		t.Fatalf("upload URL = %q, want %q", got, want)
+	}
+	if client.httpClient == nil || client.httpClient.Timeout == 0 {
+		t.Fatalf("default HTTP client = %#v, want timeout", client.httpClient)
+	}
+}
+
+func TestUploadBundleSendsCallerKeyAndManifestArchive(t *testing.T) {
+	root := writeTestBundle(t, "reports.daily", []testFile{
+		{path: "report.md", data: "# Report\n"},
+		{path: "nested/summary.txt", data: "Summary\n"},
+	})
+	if err := os.WriteFile(filepath.Join(root, "unlisted.txt"), []byte("nope"), 0o600); err != nil {
+		t.Fatalf("write unlisted file: %v", err)
+	}
+
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if got, want := r.URL.Path, "/v1/pipelines/reports.daily/upload"; got != want {
+			t.Fatalf("path = %q, want %q", got, want)
+		}
+		if got, want := r.Header.Get("Authorization"), "Bearer secret-token"; got != want {
+			t.Fatalf("authorization = %q, want %q", got, want)
+		}
+		if got, want := r.Header.Get("Content-Type"), uploadContentTypeGzip; got != want {
+			t.Fatalf("content type = %q, want %q", got, want)
+		}
+		if got, want := r.Header.Get(idempotencyKeyHeader), "producer.retry:one"; got != want {
+			t.Fatalf("idempotency key = %q, want %q", got, want)
+		}
+		entries := readArchiveEntries(t, r.Body)
+		if got, want := strings.Join(entryNames(entries), ","), "manifest.json,report.md,nested/summary.txt"; got != want {
+			t.Fatalf("archive entries = %q, want %q", got, want)
+		}
+		if _, ok := entries["unlisted.txt"]; ok {
+			t.Fatal("archive included unlisted file")
+		}
+		writeAccepted(t, w, "reports.20260604T120000Z.abcdef12")
+	}))
+	defer server.Close()
+
+	client, err := NewClient(ClientOptions{Endpoint: server.URL, Token: "secret-token", HTTPClient: server.Client()})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	result, err := client.UploadBundle(context.Background(), UploadBundleOptions{
+		PipelineID:     "reports.daily",
+		Root:           root,
+		IdempotencyKey: "producer.retry:one",
+	})
+	if err != nil {
+		t.Fatalf("UploadBundle() error = %v", err)
+	}
+	if result.RunID != "reports.20260604T120000Z.abcdef12" || result.Status != "accepted" {
+		t.Fatalf("result = %#v, want accepted run", result)
+	}
+}
+
+func TestUploadFilesBuildsTemporaryBundleWithoutTouchingSources(t *testing.T) {
+	sourceRoot := t.TempDir()
+	sourcePath := filepath.Join(sourceRoot, "producer-output.md")
+	if err := os.WriteFile(sourcePath, []byte("producer data\n"), 0o600); err != nil {
+		t.Fatalf("write source: %v", err)
+	}
+	tempDir := t.TempDir()
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if got, want := r.URL.Path, "/v1/pipelines/reports.files/upload"; got != want {
+			t.Fatalf("path = %q, want %q", got, want)
+		}
+		entries := readArchiveEntries(t, r.Body)
+		if got := string(entries["manifest.json"]); !strings.Contains(got, `"id": "reports.from.files"`) {
+			t.Fatalf("manifest = %s, want uploaded id", got)
+		}
+		if got, want := string(entries["reports/report.md"]), "producer data\n"; got != want {
+			t.Fatalf("uploaded file = %q, want %q", got, want)
+		}
+		if _, ok := entries["producer-output.md"]; ok {
+			t.Fatal("archive used producer source path instead of bundle path")
+		}
+		writeAccepted(t, w, "reports.20260604T120000Z.abcdef12")
+	}))
+	defer server.Close()
+
+	client, err := NewClient(ClientOptions{Endpoint: server.URL, Token: "secret", HTTPClient: server.Client()})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	_, err = client.UploadFiles(context.Background(), UploadFilesOptions{
+		PipelineID: "reports.files",
+		ID:         "reports.from.files",
+		Files: []sourcebundle.BundleFile{{
+			SourcePath: sourcePath,
+			Path:       "reports/report.md",
+		}},
+		TempDir: tempDir,
+	})
+	if err != nil {
+		t.Fatalf("UploadFiles() error = %v", err)
+	}
+	if _, err := os.Stat(filepath.Join(sourceRoot, sourcebundle.ManifestName)); !os.IsNotExist(err) {
+		t.Fatalf("producer source manifest stat = %v, want not exist", err)
+	}
+	entries, err := os.ReadDir(tempDir)
+	if err != nil {
+		t.Fatalf("read temp dir: %v", err)
+	}
+	if len(entries) != 0 {
+		t.Fatalf("temp dir entries = %d, want cleanup", len(entries))
+	}
+}
+
+func TestUploadMethodsRequirePipelineIDBeforeLocalWork(t *testing.T) {
+	var requests atomic.Int64
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		requests.Add(1)
+		t.Fatal("server should not receive request")
+	}))
+	defer server.Close()
+
+	client, err := NewClient(ClientOptions{Endpoint: server.URL, Token: "secret", HTTPClient: server.Client()})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+
+	missingRoot := filepath.Join(t.TempDir(), "missing")
+	if _, err := client.UploadBundle(context.Background(), UploadBundleOptions{Root: missingRoot}); err == nil || !strings.Contains(err.Error(), "pipeline id is required") {
+		t.Fatalf("UploadBundle() error = %v, want missing pipeline id", err)
+	}
+
+	tempDir := t.TempDir()
+	sourcePath := filepath.Join(t.TempDir(), "report.md")
+	if err := os.WriteFile(sourcePath, []byte("data"), 0o600); err != nil {
+		t.Fatalf("write source: %v", err)
+	}
+	if _, err := client.UploadFiles(context.Background(), UploadFilesOptions{
+		ID: "reports.from.files",
+		Files: []sourcebundle.BundleFile{{
+			SourcePath: sourcePath,
+			Path:       "report.md",
+		}},
+		TempDir: tempDir,
+	}); err == nil || !strings.Contains(err.Error(), "pipeline id is required") {
+		t.Fatalf("UploadFiles() error = %v, want missing pipeline id", err)
+	}
+	entries, err := os.ReadDir(tempDir)
+	if err != nil {
+		t.Fatalf("read temp dir: %v", err)
+	}
+	if len(entries) != 0 {
+		t.Fatalf("temp dir entries = %d, want no local bundle work", len(entries))
+	}
+	if got := requests.Load(); got != 0 {
+		t.Fatalf("requests = %d, want 0", got)
+	}
+}
+
+func TestUploadMethodsRejectInvalidPipelineIDBeforeHTTPRequest(t *testing.T) {
+	root := writeTestBundle(t, "reports.daily", []testFile{{path: "report.md", data: "data"}})
+	var requests atomic.Int64
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		requests.Add(1)
+		t.Fatal("server should not receive request")
+	}))
+	defer server.Close()
+
+	client, err := NewClient(ClientOptions{Endpoint: server.URL, Token: "secret", HTTPClient: server.Client()})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	for _, pipelineID := range []string{".reports", "reports/daily", "reports daily"} {
+		t.Run(pipelineID, func(t *testing.T) {
+			_, err := client.UploadBundle(context.Background(), UploadBundleOptions{PipelineID: pipelineID, Root: root})
+			if err == nil || !strings.Contains(err.Error(), "pipeline id must be a slug-like identifier") {
+				t.Fatalf("UploadBundle() error = %v, want invalid pipeline id", err)
+			}
+		})
+	}
+	if got := requests.Load(); got != 0 {
+		t.Fatalf("requests = %d, want 0", got)
+	}
+}
+
+func TestUploadBundleValidationFailurePreventsHTTPRequest(t *testing.T) {
+	root := writeTestBundle(t, "reports.daily", []testFile{{path: "report.md", data: "original"}})
+	if err := os.WriteFile(filepath.Join(root, "report.md"), []byte("changed"), 0o600); err != nil {
+		t.Fatalf("mutate bundle file: %v", err)
+	}
+	var requests atomic.Int64
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		requests.Add(1)
+		t.Fatal("server should not receive request")
+	}))
+	defer server.Close()
+
+	client, err := NewClient(ClientOptions{Endpoint: server.URL, Token: "secret", HTTPClient: server.Client()})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	if _, err := client.UploadBundle(context.Background(), UploadBundleOptions{PipelineID: "reports", Root: root}); err == nil {
+		t.Fatal("UploadBundle() error = nil, want validation error")
+	}
+	if got := requests.Load(); got != 0 {
+		t.Fatalf("requests = %d, want 0", got)
+	}
+}
+
+func TestUploadBundleCanDisableLocalValidation(t *testing.T) {
+	root := writeTestBundle(t, "reports.daily", []testFile{{path: "report.md", data: "original"}})
+	if err := os.WriteFile(filepath.Join(root, "report.md"), []byte("changed"), 0o600); err != nil {
+		t.Fatalf("mutate bundle file: %v", err)
+	}
+	var requests atomic.Int64
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		requests.Add(1)
+		writeAccepted(t, w, "reports.20260604T120000Z.abcdef12")
+	}))
+	defer server.Close()
+
+	client, err := NewClient(ClientOptions{Endpoint: server.URL, Token: "secret", HTTPClient: server.Client()})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	if _, err := client.UploadBundle(context.Background(), UploadBundleOptions{PipelineID: "reports", Root: root, DisableValidation: true}); err != nil {
+		t.Fatalf("UploadBundle() error = %v", err)
+	}
+	if got := requests.Load(); got != 1 {
+		t.Fatalf("requests = %d, want 1", got)
+	}
+}
+
+func TestGeneratedIdempotencyKeyIsReusedAcrossRetry(t *testing.T) {
+	root := writeTestBundle(t, "reports.daily", []testFile{{path: "report.md", data: "data"}})
+	var attempts atomic.Int64
+	var keys []string
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if got, want := r.URL.Path, "/v1/pipelines/reports/upload"; got != want {
+			t.Fatalf("path = %q, want %q", got, want)
+		}
+		keys = append(keys, r.Header.Get(idempotencyKeyHeader))
+		if attempts.Add(1) == 1 {
+			writeJSONError(w, http.StatusServiceUnavailable, "busy", false)
+			return
+		}
+		writeAccepted(t, w, "reports.20260604T120000Z.abcdef12")
+	}))
+	defer server.Close()
+
+	client, err := NewClient(ClientOptions{
+		Endpoint:   server.URL,
+		Token:      "secret",
+		HTTPClient: server.Client(),
+		Retry:      RetryOptions{MaxAttempts: 2, BaseDelay: time.Millisecond, MaxDelay: time.Millisecond},
+	})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	if _, err := client.UploadBundle(context.Background(), UploadBundleOptions{PipelineID: "reports", Root: root}); err != nil {
+		t.Fatalf("UploadBundle() error = %v", err)
+	}
+	if got, want := attempts.Load(), int64(2); got != want {
+		t.Fatalf("attempts = %d, want %d", got, want)
+	}
+	if len(keys) != 2 || keys[0] == "" || keys[0] != keys[1] {
+		t.Fatalf("idempotency keys = %#v, want same generated key", keys)
+	}
+	if !regexp.MustCompile(`^[0-9a-f]{32}$`).MatchString(keys[0]) {
+		t.Fatalf("generated key = %q, want 128-bit lowercase hex", keys[0])
+	}
+}
+
+func TestUploadResponseParsingAndNoRetryStatuses(t *testing.T) {
+	root := writeTestBundle(t, "reports.daily", []testFile{{path: "report.md", data: "data"}})
+	tests := []struct {
+		name          string
+		status        int
+		body          string
+		wantConflict  bool
+		wantMessage   string
+		wantRetryable bool
+	}{
+		{name: "bad request", status: http.StatusBadRequest, body: `{"error":"bad bundle"}`, wantMessage: "bad bundle"},
+		{name: "unauthorized", status: http.StatusUnauthorized, body: `{"error":"bad token"}`, wantMessage: "bad token"},
+		{name: "conflict", status: http.StatusConflict, body: `{"error":"different manifest","retryable":true}`, wantConflict: true, wantMessage: "different manifest", wantRetryable: true},
+		{name: "too large", status: http.StatusRequestEntityTooLarge, body: `{"error":"too large"}`, wantMessage: "too large"},
+		{name: "unsupported", status: http.StatusUnsupportedMediaType, body: `{"error":"unsupported"}`, wantMessage: "unsupported"},
+		{name: "service unavailable", status: http.StatusServiceUnavailable, body: `{"error":"busy"}`, wantMessage: "busy"},
+		{name: "non json", status: http.StatusBadRequest, body: `plain failure`, wantMessage: "plain failure"},
+		{name: "unexpected", status: http.StatusTeapot, body: ``, wantMessage: "I'm a teapot"},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			var attempts atomic.Int64
+			server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+				attempts.Add(1)
+				w.WriteHeader(tt.status)
+				_, _ = w.Write([]byte(tt.body))
+			}))
+			defer server.Close()
+
+			client, err := NewClient(ClientOptions{
+				Endpoint:   server.URL,
+				Token:      "secret",
+				HTTPClient: server.Client(),
+				Retry:      RetryOptions{MaxAttempts: 1},
+			})
+			if err != nil {
+				t.Fatalf("NewClient() error = %v", err)
+			}
+			_, err = client.UploadBundle(context.Background(), UploadBundleOptions{PipelineID: "reports", Root: root, IdempotencyKey: "key"})
+			if err == nil {
+				t.Fatal("UploadBundle() error = nil, want error")
+			}
+			var httpErr *HTTPError
+			if !errors.As(err, &httpErr) {
+				t.Fatalf("error = %T %v, want HTTPError", err, err)
+			}
+			if httpErr.StatusCode != tt.status || !strings.Contains(httpErr.Message, tt.wantMessage) || httpErr.Retryable != tt.wantRetryable {
+				t.Fatalf("HTTPError = %#v, want status %d message %q retryable %t", httpErr, tt.status, tt.wantMessage, tt.wantRetryable)
+			}
+			var conflict *IdempotencyConflictError
+			if got := errors.As(err, &conflict); got != tt.wantConflict {
+				t.Fatalf("conflict error = %t, want %t", got, tt.wantConflict)
+			}
+			if got := attempts.Load(); got != 1 {
+				t.Fatalf("attempts = %d, want 1", got)
+			}
+		})
+	}
+}
+
+func TestTokenRedactedFromHTTPError(t *testing.T) {
+	root := writeTestBundle(t, "reports.daily", []testFile{{path: "report.md", data: "data"}})
+	token := "super-secret-token"
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		writeJSONError(w, http.StatusBadRequest, "token "+token+" rejected", false)
+	}))
+	defer server.Close()
+	client, err := NewClient(ClientOptions{Endpoint: server.URL, Token: token, HTTPClient: server.Client(), Retry: RetryOptions{MaxAttempts: 1}})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	_, err = client.UploadBundle(context.Background(), UploadBundleOptions{PipelineID: "reports", Root: root, IdempotencyKey: "key"})
+	if err == nil {
+		t.Fatal("UploadBundle() error = nil, want error")
+	}
+	if strings.Contains(err.Error(), token) {
+		t.Fatalf("error exposed token: %v", err)
+	}
+	if !strings.Contains(err.Error(), redactedSecret) {
+		t.Fatalf("error = %v, want redaction marker", err)
+	}
+}
+
+func TestNetworkRetryUsesSameIdempotencyKey(t *testing.T) {
+	root := writeTestBundle(t, "reports.daily", []testFile{{path: "report.md", data: "data"}})
+	var attempts atomic.Int64
+	var keys []string
+	client, err := NewClient(ClientOptions{
+		Endpoint: "http://upload.example",
+		Token:    "secret",
+		HTTPClient: &http.Client{Transport: roundTripFunc(func(request *http.Request) (*http.Response, error) {
+			keys = append(keys, request.Header.Get(idempotencyKeyHeader))
+			if got, want := request.URL.Path, "/v1/pipelines/reports/upload"; got != want {
+				t.Fatalf("path = %q, want %q", got, want)
+			}
+			if attempts.Add(1) == 1 {
+				return nil, temporaryNetworkError{}
+			}
+			return &http.Response{
+				StatusCode: http.StatusAccepted,
+				Status:     "202 Accepted",
+				Header:     make(http.Header),
+				Body:       io.NopCloser(strings.NewReader(`{"run_id":"reports.20260604T120000Z.abcdef12","status":"accepted"}`)),
+				Request:    request,
+			}, nil
+		})},
+		Retry: RetryOptions{MaxAttempts: 2, BaseDelay: time.Millisecond, MaxDelay: time.Millisecond},
+	})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	result, err := client.UploadBundle(context.Background(), UploadBundleOptions{PipelineID: "reports", Root: root, IdempotencyKey: "network-retry"})
+	if err != nil {
+		t.Fatalf("UploadBundle() error = %v", err)
+	}
+	if result.RunID == "" {
+		t.Fatalf("result = %#v, want run id", result)
+	}
+	if got, want := attempts.Load(), int64(2); got != want {
+		t.Fatalf("attempts = %d, want %d", got, want)
+	}
+	if got, want := strings.Join(keys, ","), "network-retry,network-retry"; got != want {
+		t.Fatalf("keys = %q, want %q", got, want)
+	}
+}
+
+func TestContextCancellationDuringRetryBackoff(t *testing.T) {
+	root := writeTestBundle(t, "reports.daily", []testFile{{path: "report.md", data: "data"}})
+	ctx, cancel := context.WithCancel(context.Background())
+	var attempts atomic.Int64
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		attempts.Add(1)
+		cancel()
+		writeJSONError(w, http.StatusServiceUnavailable, "busy", false)
+	}))
+	defer server.Close()
+
+	client, err := NewClient(ClientOptions{
+		Endpoint:   server.URL,
+		Token:      "secret",
+		HTTPClient: server.Client(),
+		Retry:      RetryOptions{MaxAttempts: 2, BaseDelay: time.Hour, MaxDelay: time.Hour},
+	})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	_, err = client.UploadBundle(ctx, UploadBundleOptions{PipelineID: "reports", Root: root, IdempotencyKey: "cancel"})
+	if !errors.Is(err, context.Canceled) {
+		t.Fatalf("UploadBundle() error = %v, want context.Canceled", err)
+	}
+	if got := attempts.Load(); got != 1 {
+		t.Fatalf("attempts = %d, want 1", got)
+	}
+}
+
+func TestStatusParsesRunStatusAndErrors(t *testing.T) {
+	acceptedAt := time.Date(2026, 6, 4, 12, 0, 0, 0, time.UTC)
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if got, want := r.URL.Path, "/runs/reports.20260604T120000Z.abcdef12"; got != want {
+			t.Fatalf("path = %q, want %q", got, want)
+		}
+		if got, want := r.Header.Get("Authorization"), "Bearer secret"; got != want {
+			t.Fatalf("authorization = %q, want %q", got, want)
+		}
+		_ = json.NewEncoder(w).Encode(RunStatus{
+			RunID:      "reports.20260604T120000Z.abcdef12",
+			PipelineID: "reports",
+			Status:     "succeeded",
+			AcceptedAt: acceptedAt,
+		})
+	}))
+	defer server.Close()
+
+	client, err := NewClient(ClientOptions{Endpoint: server.URL, Token: "secret", HTTPClient: server.Client()})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	status, err := client.Status(context.Background(), "reports.20260604T120000Z.abcdef12")
+	if err != nil {
+		t.Fatalf("Status() error = %v", err)
+	}
+	if status.RunID != "reports.20260604T120000Z.abcdef12" || status.PipelineID != "reports" || status.Status != "succeeded" || !status.AcceptedAt.Equal(acceptedAt) {
+		t.Fatalf("status = %#v, want succeeded run", status)
+	}
+
+	errorServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		writeJSONError(w, http.StatusNotFound, "run not found", false)
+	}))
+	defer errorServer.Close()
+	client, err = NewClient(ClientOptions{Endpoint: errorServer.URL, Token: "secret", HTTPClient: errorServer.Client()})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	_, err = client.Status(context.Background(), "missing")
+	var httpErr *HTTPError
+	if err == nil || !errors.As(err, &httpErr) || httpErr.StatusCode != http.StatusNotFound {
+		t.Fatalf("Status() error = %v, want 404 HTTPError", err)
+	}
+}
+
+func TestInvalidCallerIdempotencyKeyPreventsHTTPRequest(t *testing.T) {
+	root := writeTestBundle(t, "reports.daily", []testFile{{path: "report.md", data: "data"}})
+	var requests atomic.Int64
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		requests.Add(1)
+	}))
+	defer server.Close()
+	client, err := NewClient(ClientOptions{Endpoint: server.URL, Token: "secret", HTTPClient: server.Client()})
+	if err != nil {
+		t.Fatalf("NewClient() error = %v", err)
+	}
+	if _, err := client.UploadBundle(context.Background(), UploadBundleOptions{PipelineID: "reports", Root: root, IdempotencyKey: "bad key"}); err == nil {
+		t.Fatal("UploadBundle() error = nil, want invalid key error")
+	}
+	if got := requests.Load(); got != 0 {
+		t.Fatalf("requests = %d, want 0", got)
+	}
+}
+
+type testFile struct {
+	path string
+	data string
+}
+
+func writeTestBundle(t *testing.T, id string, files []testFile) string {
+	t.Helper()
+	sourceRoot := t.TempDir()
+	bundleFiles := make([]sourcebundle.BundleFile, 0, len(files))
+	for _, file := range files {
+		sourcePath := filepath.Join(sourceRoot, filepath.FromSlash(file.path))
+		if err := os.MkdirAll(filepath.Dir(sourcePath), 0o755); err != nil {
+			t.Fatalf("mkdir source parent: %v", err)
+		}
+		if err := os.WriteFile(sourcePath, []byte(file.data), 0o600); err != nil {
+			t.Fatalf("write source file: %v", err)
+		}
+		bundleFiles = append(bundleFiles, sourcebundle.BundleFile{
+			SourcePath: sourcePath,
+			Path:       file.path,
+		})
+	}
+	root := filepath.Join(t.TempDir(), "bundle")
+	if _, err := sourcebundle.WriteBundle(sourcebundle.WriteBundleOptions{
+		Root:    root,
+		ID:      id,
+		Created: time.Date(2026, 6, 4, 12, 0, 0, 0, time.UTC),
+		Files:   bundleFiles,
+	}); err != nil {
+		t.Fatalf("WriteBundle() error = %v", err)
+	}
+	return root
+}
+
+func readArchiveEntries(t *testing.T, body io.Reader) map[string][]byte {
+	t.Helper()
+	gzipReader, err := gzip.NewReader(body)
+	if err != nil {
+		t.Fatalf("open gzip archive: %v", err)
+	}
+	defer gzipReader.Close()
+	tarReader := tar.NewReader(gzipReader)
+	entries := map[string][]byte{}
+	for {
+		header, err := tarReader.Next()
+		if errors.Is(err, io.EOF) {
+			return entries
+		}
+		if err != nil {
+			t.Fatalf("read tar archive: %v", err)
+		}
+		data, err := io.ReadAll(tarReader)
+		if err != nil {
+			t.Fatalf("read tar entry %q: %v", header.Name, err)
+		}
+		entries[header.Name] = data
+	}
+}
+
+func entryNames(entries map[string][]byte) []string {
+	ordered := []string{}
+	for _, name := range []string{"manifest.json", "report.md", "nested/summary.txt", "reports/report.md", "unlisted.txt"} {
+		if _, ok := entries[name]; ok {
+			ordered = append(ordered, name)
+		}
+	}
+	return ordered
+}
+
+func writeAccepted(t *testing.T, w http.ResponseWriter, runID string) {
+	t.Helper()
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(http.StatusAccepted)
+	if err := json.NewEncoder(w).Encode(Result{RunID: runID, Status: "accepted"}); err != nil {
+		t.Fatalf("write accepted response: %v", err)
+	}
+}
+
+func writeJSONError(w http.ResponseWriter, status int, message string, retryable bool) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(status)
+	_ = json.NewEncoder(w).Encode(map[string]any{"error": message, "retryable": retryable})
+}
+
+type roundTripFunc func(*http.Request) (*http.Response, error)
+
+func (fn roundTripFunc) RoundTrip(request *http.Request) (*http.Response, error) {
+	return fn(request)
+}
+
+type temporaryNetworkError struct{}
+
+func (temporaryNetworkError) Error() string {
+	return "temporary network failure"
+}
+
+func (temporaryNetworkError) Timeout() bool {
+	return false
+}
+
+func (temporaryNetworkError) Temporary() bool {
+	return true
+}

pkg/upload/doc.go

@@ -0,0 +1,136 @@
+// Package upload provides producer-facing helpers for submitting distributor
+// source bundles to the HTTP upload API.
+//
+// The package is intended for Go producer applications that already create
+// reports or other Markdown bundle contents and want to hand those bundles to a
+// running distributor server. It builds on pkg/bundle for manifest generation,
+// path validation, digest calculation, local bundle writing, and local bundle
+// validation. It does not expose distributor internals, server configuration,
+// storage backends, destination state, or publish behavior.
+//
+// # Client Construction
+//
+// NewClient creates a Client from ClientOptions. Endpoint is required and must
+// be an http or https distributor server base URL without userinfo, query, or
+// fragment. The client derives /v1/pipelines/<pipeline-id>/upload for
+// submissions and /runs/<run-id> for status checks. Token is required and is
+// sent as Authorization: Bearer <token>. Token values are redacted from errors
+// produced by the client.
+//
+// HTTPClient is optional. When omitted, the package uses a client with a
+// conservative timeout. Retry is optional; zero values select safe defaults.
+// RetryOptions.MaxAttempts, BaseDelay, and MaxDelay must not be negative, and
+// MaxDelay must be greater than or equal to BaseDelay.
+//
+// # Upload Workflows
+//
+// UploadBundle uploads an existing local source bundle root to the configured
+// PipelineID. PipelineID is required and must match the server's slug-like
+// pipeline id syntax. The root must contain manifest.json. By default,
+// UploadBundle loads the manifest and validates the complete local bundle with
+// pkg/bundle before making any HTTP request. The generated gzip-compressed tar
+// archive contains manifest.json and exactly the manifest-listed files;
+// unlisted files are not uploaded.
+//
+// UploadFiles is the convenience workflow for producer applications that have
+// generated files but have not yet assembled a bundle directory. PipelineID is
+// required and selects the configured distributor workflow. UploadFiles uses
+// pkg/bundle to create a temporary complete bundle from explicit
+// bundle.BundleFile values, validates it by default, archives it, uploads it,
+// and removes temporary files when the call returns. UploadFiles does not write
+// into producer source directories. A zero Created timestamp follows
+// pkg/bundle defaulting behavior.
+//
+// The producer contract has four separate identifiers: the bearer token
+// authenticates the client, PipelineID selects the distributor workflow, the
+// source manifest ID identifies the logical artifact within that workflow, and
+// IdempotencyKey identifies one producer run and retry group.
+//
+// Validation is enabled by default. Set DisableValidation when the application
+// has already performed equivalent local validation and wants to skip the
+// package's validation step. Validate and DisableValidation must not both be
+// true.
+//
+// # Idempotency And Retry
+//
+// Every upload request includes Idempotency-Key. If UploadBundleOptions or
+// UploadFilesOptions provides IdempotencyKey, the client validates and uses
+// that value. Otherwise, it generates a random 128-bit lowercase hexadecimal
+// key once for that upload operation and reuses it for all retries from that
+// call.
+//
+// Generated idempotency keys are useful for retrying transient failures within
+// a single process call. Producers that need cross-process retry safety should
+// provide their own stable key, such as a key derived from the producer job id
+// or report id. Valid keys are non-empty ASCII strings up to 128 bytes using
+// letters, digits, '.', '_', '-', and ':'.
+//
+// The client retries only safe cases: 503 Service Unavailable, temporary
+// network errors, and ambiguous mid-upload failures. Retries use the same
+// idempotency key and replayable gzip archive body. The client does not retry
+// 400, 401, 409, 413, 415, or any response after 202 Accepted. Context
+// cancellation is honored before each attempt and while waiting between
+// retries.
+//
+// # Results And Errors
+//
+// Result represents upload admission. A successful UploadBundle or UploadFiles
+// call means the server accepted the upload and returned a run id; it does not
+// mean the asynchronous distribution run has finished successfully.
+//
+// Status fetches the current server status for a run id and returns RunStatus.
+// This is a separate polling helper; upload calls do not wait for publication
+// completion.
+//
+// Non-2xx upload and status responses return *HTTPError when the server status
+// can be represented as an HTTP failure. HTTPError includes the numeric status
+// code, HTTP status string, response message, and server retryable flag when
+// present. A 409 Conflict response is returned as *IdempotencyConflictError,
+// which wraps HTTPError and can be detected with errors.As.
+//
+// Example: upload producer files with a stable idempotency key.
+//
+//	ctx := context.Background()
+//	client, err := upload.NewClient(upload.ClientOptions{
+//		Endpoint: "https://distributor.example.com",
+//		Token:    os.Getenv("DISTRIBUTOR_UPLOAD_TOKEN"),
+//	})
+//	if err != nil {
+//		return err
+//	}
+//
+//	result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
+//		PipelineID:     "reports.daily",
+//		ID:             "reports.daily.2026-06-06",
+//		IdempotencyKey: "reports.daily.2026-06-06",
+//		Files: []bundle.BundleFile{
+//			{SourcePath: "/tmp/report.md", Path: "report.md"},
+//			{SourcePath: "/tmp/summary.txt", Path: "summary.txt"},
+//		},
+//	})
+//	if err != nil {
+//		var conflict *upload.IdempotencyConflictError
+//		if errors.As(err, &conflict) {
+//			return fmt.Errorf("upload conflicts with an earlier different bundle: %w", err)
+//		}
+//		return err
+//	}
+//
+//	status, err := client.Status(ctx, result.RunID)
+//	if err != nil {
+//		return err
+//	}
+//	_ = status
+//
+// Example: upload an existing bundle root.
+//
+//	result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
+//		PipelineID:     "reports.daily",
+//		Root:           "/var/lib/reports/daily-2026-06-06",
+//		IdempotencyKey: "reports.daily.2026-06-06",
+//	})
+//	if err != nil {
+//		return err
+//	}
+//	_ = result
+package upload

pkg/upload/types.go

@@ -0,0 +1,94 @@
+package upload
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"time"
+
+	"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+)
+
+type Client struct {
+	endpoint   string
+	token      string
+	httpClient *http.Client
+	retry      RetryOptions
+}
+
+type ClientOptions struct {
+	Endpoint   string
+	Token      string
+	HTTPClient *http.Client
+	Retry      RetryOptions
+}
+
+type RetryOptions struct {
+	MaxAttempts int
+	BaseDelay   time.Duration
+	MaxDelay    time.Duration
+}
+
+type UploadBundleOptions struct {
+	PipelineID        string
+	Root              string
+	Validate          bool
+	DisableValidation bool
+	IdempotencyKey    string
+}
+
+type UploadFilesOptions struct {
+	PipelineID        string
+	ID                string
+	Created           time.Time
+	Files             []bundle.BundleFile
+	Validate          bool
+	DisableValidation bool
+	TempDir           string
+	IdempotencyKey    string
+}
+
+type Result struct {
+	RunID  string `json:"run_id"`
+	Status string `json:"status"`
+}
+
+type RunStatus struct {
+	RunID      string          `json:"run_id"`
+	PipelineID string          `json:"pipeline_id"`
+	Status     string          `json:"status"`
+	AcceptedAt time.Time       `json:"accepted_at"`
+	StartedAt  *time.Time      `json:"started_at,omitempty"`
+	FinishedAt *time.Time      `json:"finished_at,omitempty"`
+	Report     json.RawMessage `json:"report,omitempty"`
+	Error      string          `json:"error,omitempty"`
+}
+
+type HTTPError struct {
+	StatusCode int
+	Status     string
+	Message    string
+	Retryable  bool
+}
+
+func (err *HTTPError) Error() string {
+	if err == nil {
+		return "<nil>"
+	}
+	if err.Message == "" {
+		return fmt.Sprintf("upload request failed: %s", err.Status)
+	}
+	return fmt.Sprintf("upload request failed: %s: %s", err.Status, err.Message)
+}
+
+type IdempotencyConflictError struct {
+	HTTPError
+}
+
+func (err *IdempotencyConflictError) Error() string {
+	return (*HTTPError)(&err.HTTPError).Error()
+}
+
+func (err *IdempotencyConflictError) Unwrap() error {
+	return &err.HTTPError
+}