Document pipeline-scoped upload behavior

This commit is contained in:
2026-06-08 04:46:22 +00:00
parent bd5892d1f2
commit 29f01da37b
7 changed files with 103 additions and 201 deletions

View File

@@ -2,7 +2,7 @@
Audience: producers, operators, and maintainers integrating with `distributor serve`.
`distributor serve` exposes a local HTTP upload API for pipelines whose source backend is `http_upload`. Each bearer token maps to exactly one configured pipeline.
`distributor serve` exposes a local HTTP upload API for pipelines whose source backend is `http_upload`. Bearer tokens authenticate producers, and the upload path selects the configured pipeline. The selected token must be allowed for the requested pipeline.
## Authentication
@@ -12,9 +12,9 @@ Uploads authenticate with:
Authorization: Bearer <token>
```
Token values are resolved from the configured `source.token_env` through the process environment or `secrets.directory`. Tokens are not configured as YAML literal values.
Token values are resolved from top-level `upload_tokens` records through the process environment or `secrets.directory`. Tokens are not configured as YAML literal values.
Requests that include `pipeline` or `pipeline_id` query parameters are rejected. The bearer token selects the pipeline.
Requests that include `pipeline` or `pipeline_id` query parameters are rejected. Use the pipeline id in the upload path.
## Endpoints
@@ -26,7 +26,7 @@ Returns `200 OK` when the server is running:
{"status":"ok"}
```
### `POST /upload`
### `POST /v1/pipelines/{pipeline_id}/upload`
Accepts one source bundle archive and returns after the archive is staged and validated.
@@ -36,7 +36,7 @@ Producers may include:
Idempotency-Key: <key>
```
Idempotency keys are scoped to the authenticated pipeline selected by the bearer token. Valid keys are non-empty ASCII strings up to 128 bytes using letters, digits, `.`, `_`, `-`, and `:`. Invalid keys return `400`.
`pipeline_id` must name a configured pipeline whose source backend is `http_upload`, and the authenticated token must allow that pipeline. Idempotency keys are scoped to token id, pipeline id, and key. Valid keys are non-empty ASCII strings up to 128 bytes using letters, digits, `.`, `_`, `-`, and `:`. Invalid keys return `400`.
Accepted content types:
@@ -54,6 +54,8 @@ Common error responses:
- `400`: pipeline query supplied, invalid idempotency key, archive rejected, malformed archive, or invalid staged source bundle.
- `401`: missing, empty, or unknown bearer token.
- `403`: bearer token is valid but is not allowed for the requested pipeline.
- `404`: upload path is unknown or the requested upload pipeline is not configured.
- `409`: repeated idempotency key conflicts with another source manifest, or the same key is already being staged.
- `413`: upload body exceeds the selected pipeline size limit.
- `415`: unsupported content type.
@@ -71,7 +73,7 @@ Retryable idempotency conflicts include:
{"error":"upload idempotency key is already being processed","retryable":true}
```
When `Idempotency-Key` is omitted, upload admission preserves the raw HTTP behavior: every valid accepted upload receives its own run id. When a key is supplied, the server records the accepted run after archive staging and source bundle validation succeed. Reusing the same key for the same authenticated pipeline and the same normalized source manifest returns the original `202 Accepted` response and does not enqueue another run. Reusing the same key for a different normalized source manifest returns `409 Conflict`. Producers should use a fresh key for each distinct producer run and reuse a key only for retries of that same run.
When `Idempotency-Key` is omitted, upload admission preserves the raw HTTP behavior: every valid accepted upload receives its own run id. When a key is supplied, the server records the accepted run after archive staging and source bundle validation succeed. Reusing the same key for the same token id, pipeline id, and normalized source manifest returns the original `202 Accepted` response and does not enqueue another run. Reusing the same key for a different normalized source manifest within that scope returns `409 Conflict`. Producers should use a fresh key for each distinct producer run and reuse a key only for retries of that same run.
### `GET /runs/<run-id>`
@@ -119,14 +121,15 @@ if err != nil {
return err
}
result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
PipelineID: "reports",
Root: "examples/source-bundle",
IdempotencyKey: "reports.example.20260604T120000Z",
})
```
`Endpoint` is the server base URL; the package derives `/upload` and `/runs/<run-id>`. `UploadBundle` validates a local bundle by default and uploads only `manifest.json` plus manifest-listed files. `UploadFiles` creates a temporary bundle from explicit `bundle.BundleFile` values before uploading. When `IdempotencyKey` is omitted, the package generates one random 128-bit lowercase hex key for the upload operation and reuses it across retries.
`Endpoint` is the server base URL; the package derives `/v1/pipelines/<pipeline-id>/upload` and `/runs/<run-id>`. `PipelineID` is required and selects the configured distributor workflow. `UploadBundle` validates a local bundle by default and uploads only `manifest.json` plus manifest-listed files. `UploadFiles` creates a temporary bundle from explicit `bundle.BundleFile` values before uploading. When `IdempotencyKey` is omitted, the package generates one random 128-bit lowercase hex key for the upload operation and reuses it across retries.
The helper retries only safe cases: `503 Service Unavailable`, temporary network errors, and ambiguous mid-upload failures. It does not retry after `202 Accepted` and does not retry `400`, `401`, `409`, `413`, or `415`. Bearer token values are redacted from returned errors.
The helper retries only safe cases: `503 Service Unavailable`, temporary network errors, and ambiguous mid-upload failures. It does not retry after `202 Accepted` and does not retry `400`, `401`, `403`, `404`, `409`, `413`, or `415`. Bearer token values are redacted from returned errors.
## Queue And Retention
@@ -138,7 +141,7 @@ Idempotency records are memory-only, expire with completed upload status records
## Boundaries
The HTTP API does not expose pipeline selection by request parameter, TLS, public routing policy, or durable status storage. Put public access controls, TLS termination, and rate limiting in deployment infrastructure.
The HTTP API does not expose pipeline selection by query parameter, TLS, public routing policy, or durable status storage. Put public access controls, TLS termination, and rate limiting in deployment infrastructure.
## Tests