distributor/docs/roadmap/implementation.md

Pipeline-Scoped Upload API Implementation Plan

This roadmap is for an LLM coding agent implementing the planned API work in docs/roadmap/api.md. It describes future work only. Do not update current-behavior docs outside docs/roadmap/ until the corresponding stage is implemented.

Before implementation, read:

• docs/policy/architecture.md
• docs/policy/development.md
• docs/policy/documentation.md
• docs/roadmap/api.md

Target Behavior

The final implementation is a breaking v1 HTTP upload API change:

• Upload authentication is configured with top-level upload_tokens.
• Pipeline routing is selected by POST /v1/pipelines/{pipeline_id}/upload.
• Producers must set PipelineID in pkg/upload upload options.
• Legacy per-source source.token_env is removed.
• Legacy POST /upload no longer accepts uploads.
• Idempotency is scoped by token id, pipeline id, and idempotency key.
• /healthz and /runs/<run-id> continue to work.

The producer contract is:

• token authenticates the producer/client;
• PipelineID selects the configured distributor workflow;
• source manifest id identifies the logical artifact within that workflow;
• idempotency key identifies one producer run and retry group.

Stage 1: Config Model And Validation

Goal: make the YAML schema express upload authentication separately from pipeline source configuration.

Implementation:

• Add UploadTokens []UploadToken to internal/config.Config with YAML key upload_tokens.
• Add UploadToken with fields:
  • ID string as id;
  • TokenEnv string as token_env;
  • AllowPipelines []string as allow_pipelines.
• Remove TokenEnv from HTTPUpload; keep StagingPath and MaxUploadSize.
• Keep http_upload source defaults for staging_path and max_upload_size.
• Update validation:
  • upload_tokens is required when any pipeline uses source.backend: http_upload.
  • upload_tokens is invalid when it references no configured upload pipelines.
  • token id is required, slug-like, and unique.
  • token token_env is required.
  • token allow_pipelines is required.
  • each token's allow_pipelines entries are unique.
  • each allowed pipeline id exists and names a pipeline whose source backend is http_upload.
  • every configured http_upload pipeline is allowed by at least one token.
  • http_upload sources no longer require token_env.
• Remove tests and fixtures that expect source.token_env.
• Add config load/validation tests for valid multi-pipeline tokens, multiple tokens for one pipeline, missing token list, duplicate token ids, duplicate allowlist entries, unknown allowed pipeline id, non-upload allowed pipeline id, and upload pipeline not allowed by any token.

Acceptance:

• go test ./internal/config passes.
• YAML containing source.token_env fails as an unknown field.
• YAML using top-level upload_tokens and http_upload sources without source.token_env loads and validates.

Stage 2: Upload Token Resolution And HTTP Routing

Goal: authenticate by bearer token, authorize by token allowlist, and route by URL pipeline id.

Implementation:

• Replace the current token-to-single-pipeline map with resolved upload token records containing:
  • token id;
  • resolved token value;
  • allowed pipeline id set.
• Resolve token values through the existing config environment resolver so process environment and secrets.directory behavior remains consistent.
• Startup must fail when a token env is missing, empty, or resolves to the same token value as another upload token. Error messages may name token ids and env var names, but must not print token values.
• Change uploadHTTPHandler routes:
  • keep GET /healthz;
  • keep GET /runs/<run-id>;
  • add POST /v1/pipelines/{pipeline_id}/upload;
  • remove legacy POST /upload.
• Path parsing rules:
  • match exactly /v1/pipelines/<pipeline-id>/upload;
  • reject missing pipeline id, extra path segments, and query-based pipeline or pipeline_id routing;
  • validate requested pipeline id using the same slug-like id policy used for configured pipeline ids.
• Request handling rules:
  • missing, malformed, or unknown bearer token returns 401;
  • valid token not allowed for requested pipeline returns 403;
  • requested pipeline must be configured with source.backend: http_upload;
  • content type and idempotency key validation remain unchanged;
  • successful requests submit the requested pipeline id and token id to the upload coordinator.
• Update HTTP handler tests for accepted v1 upload, unauthorized upload, forbidden pipeline, invalid pipeline path, removed /upload, invalid content type, invalid idempotency key, and no token leakage.

Acceptance:

• go test ./internal/app passes for handler tests touched in this stage.
• POST /upload returns not found or another non-accepting error and does not call Submit.
• POST /v1/pipelines/{pipeline_id}/upload routes only when token authorization allows that pipeline.

Stage 3: Coordinator Idempotency Scope

Goal: prevent idempotency collisions between distinct authorized clients and between pipelines.

Implementation:

• Add TokenID string to UploadRequest.
• Update idempotency scope to include token id, pipeline id, and key.
• Preserve existing behavior inside one idempotency scope:
  • same key and same normalized manifest returns the original accepted run;
  • same key and different normalized manifest returns conflict;
  • same key while staging returns retryable conflict.
• Keep requests without idempotency keys unscoped and always admitted according to queue capacity.
• Update coordinator tests:
  • same token, same pipeline, same key returns original run;
  • same token, same pipeline, same key with changed manifest conflicts;
  • different token ids can use the same key for the same pipeline without collision;
  • same token id can use the same key for different pipelines without collision;
  • pending-key retryable conflict still applies only within the same token and pipeline scope.

Acceptance:

• go test ./internal/app passes.
• Existing queueing, retention, status, and same-pipeline serialization behavior remains unchanged.

Stage 4: Public pkg/upload API

Goal: make producer clients route uploads through the v1 pipeline-scoped API.

Implementation:

• Add PipelineID string to UploadFilesOptions.
• Add PipelineID string to UploadBundleOptions.
• Require non-empty valid PipelineID in both upload methods before local bundle staging, validation, archiving, or HTTP requests.
• Use the same accepted pipeline id syntax as server config ids: starts with an ASCII letter or digit and then contains ASCII letters, digits, ., _, or -.
• Build upload URLs as /v1/pipelines/{pipeline_id}/upload.
• Keep Status(ctx, runID) unchanged.
• Update package tests:
  • missing PipelineID fails before local file work or HTTP request;
  • invalid PipelineID fails before HTTP request;
  • UploadFiles and UploadBundle post to the v1 route;
  • retry behavior reuses the same idempotency key and v1 route;
  • token redaction still works.
• Update pkg/upload package documentation to explain the four-part producer contract: token, PipelineID, manifest ID, idempotency key.

Acceptance:

• go test ./pkg/upload ./pkg/bundle passes.
• Existing Status tests pass without endpoint changes.

Stage 5: Examples And End-To-End App Coverage

Goal: prove the new server and client API work together across realistic upload pipelines.

Implementation:

• Update examples/http-upload-local.yml to use top-level upload_tokens.
• Update examples/upload-client to require or default a pipeline id and pass it to pkg/upload.
• Update app integration tests:
  • one token authorized for two upload pipelines can upload to both;
  • two tokens authorized for one upload pipeline can upload to that pipeline;
  • a valid token rejected for a disallowed pipeline returns 403;
  • removed /upload endpoint does not enqueue work;
  • full upload publishes through the selected pipeline and records the selected pipeline id in status/report output.
• Update helper config builders in tests to use upload_tokens.

Acceptance:

• go test ./internal/app passes.
• Example config loading tests pass.
• go run ./examples/upload-client remains documented as an example that targets a configured pipeline.

Stage 6: Implemented-Behavior Documentation

Goal: move the feature from roadmap-only language into current-behavior docs after the code is implemented.

Implementation:

• Update docs/config.md:
  • document top-level upload_tokens;
  • remove source.token_env;
  • document http_upload source fields staging_path and max_upload_size;
  • document token allowlist semantics.
• Update docs/integrations/http-upload.md:
  • replace POST /upload with POST /v1/pipelines/{pipeline_id}/upload;
  • document 401 versus 403;
  • document idempotency scope by token id and pipeline id.
• Update docs/consumers/api.md and docs/consumers/pkg-upload.md:
  • show PipelineID in UploadFiles and UploadBundle;
  • explain token authentication versus pipeline routing.
• Update docs/operations.md and docs/troubleshooting.md for the new curl path, upload token config, and forbidden-pipeline diagnosis.
• Update README.md only if its producer integration links or summary become stale.
• Revise docs/roadmap/api.md so it no longer presents implemented behavior as future work. Either mark the pipeline-scoped upload API as implemented and leave only deferred ideas, or remove implemented sections.

Acceptance:

• rg -n "POST /upload|source.token_env|token_env.*http_upload|maps each resolved bearer token to exactly one" README.md docs examples returns no stale current-behavior references outside historical roadmap context.
• Documentation outside docs/roadmap/ describes only implemented behavior.

Final Verification

Run these commands after all stages are implemented:

go test ./internal/config
go test ./internal/app
go test ./pkg/bundle ./pkg/upload
go test ./...

Also run:

rg -n "POST /upload|source.token_env|/v1/pipelines|upload_tokens|PipelineID" README.md docs examples internal pkg

Review the output for stale references, missing docs, and tests that still expect legacy upload routing.

Non-Goals

• Do not let producers specify destination ids, destination paths, transforms, links, publish policy, transfer policy, or storage backends through upload requests.
• Do not add durable upload status, durable idempotency, retry endpoints, cancellation endpoints, long polling, or UploadAndWait.
• Do not add in-app TLS, public exposure policy, or rate limiting.
• Do not introduce new public package families beyond existing pkg/bundle and pkg/upload.