Clean up upload API documentation roadmap

docs/internal/app.md

@@ -36,7 +36,7 @@ Run workflows discover and validate source bundles through `internal/bundle`. De
 
 HTTP uploads stage and validate archives before enqueueing a pipeline run with a local staged source root. Go producers can use the public `pkg/upload` package to create client-side gzip tar uploads for this server contract; `internal/app` remains the server-side orchestration boundary and does not import that producer package.
 
-Upload idempotency is owned by the upload coordinator. Optional `Idempotency-Key` values are scoped to the authenticated pipeline. The coordinator reserves a key while staging is in progress, records the accepted run id with the validated source manifest identity after staging succeeds, returns the original accepted record for the same key and same manifest, and rejects the same key with a different manifest as a conflict.
+Upload idempotency is owned by the upload coordinator. Optional `Idempotency-Key` values are scoped to token id, pipeline id, and key. The coordinator reserves a key while staging is in progress, records the accepted run id with the validated source manifest identity after staging succeeds, returns the original accepted record for the same scoped key and same manifest, and rejects the same scoped key with a different manifest as a conflict.
 
 ## Skip And Resume Behavior
 

docs/roadmap/api.md

@@ -2,34 +2,24 @@
 
 This document records API work that is not part of the current implementation. Current HTTP upload behavior is documented in `docs/integrations/http-upload.md`, current configuration behavior is documented in `docs/config.md`, and current producer package usage is documented under `docs/consumers/`.
 
-## Implemented HTTP Upload Behavior
+## Deferred Upload API Work
 
-The pipeline-scoped HTTP upload API is implemented. The current contract is:
+- Durable upload status storage.
-
+- Durable idempotency records across server restarts.
-- upload authentication is configured with top-level `upload_tokens`;
+- Run listing, retry, and cancellation endpoints.
-- token values resolve through the process environment or `secrets.directory`, never YAML literals;
+- Long-polling or wait-for-completion workflows.
-- bearer tokens authenticate producers and authorize configured upload pipelines through `allow_pipelines`;
+- Multipart, resumable, or streaming upload protocols.
-- upload requests use `POST /v1/pipelines/{pipeline_id}/upload`;
+- Additional archive content negotiation beyond tar and gzip-compressed tar.
-- missing, malformed, or unknown bearer tokens return `401 Unauthorized`;
+- URL-token authentication for constrained clients.
-- valid tokens that are not allowed for the requested pipeline return `403 Forbidden`;
+- Upload token lifecycle tooling.
-- `pkg/upload` upload options require `PipelineID`;
+- Mutual TLS or other in-app identity mechanisms.
-- idempotency records are scoped by token id, pipeline id, and idempotency key;
+- In-app TLS termination.
-- source manifests remain free of routing, destination, transform, and credential data.
+- In-app public exposure policy.
-
+- In-app upload rate limiting.
-The unscoped upload route no longer accepts uploads. Pipeline query parameters are rejected.
-
-## Deferred API Work
-
-The following topics remain separate roadmap items:
-
-- durable upload status storage;
-- durable idempotency records across server restarts;
-- in-app public exposure policy;
-- built-in TLS termination;
-- built-in upload rate limiting.
 
 ## Boundaries
 
-- Do not add destination selection to producer manifests.
+- Producers do not choose destination ids, destination paths, transforms, links, publish policy, transfer policy, or storage backends through upload requests.
-- Do not let producers specify destination ids, transforms, links, publish policy, or transfer policy through the upload API.
+- Source manifests remain free of routing, destination, transform, and credential data.
-- Keep `http_upload` source-only. A selected pipeline owns destination configuration, transforms, links, transfer policy, and publication behavior.
+- `http_upload` remains source-only unless a future design changes that contract.
+- Public access policy, TLS termination, and rate limiting belong in deployment infrastructure unless a future design changes that boundary.

docs/roadmap/implementation.md

@@ -1,275 +1,22 @@
-# Pipeline-Scoped Upload API Implementation Plan
+# Upload API Implementation Notes
 
-This roadmap is for an LLM coding agent implementing the planned API work in
+This file has no active implementation tasks for the pipeline-scoped upload API.
-`docs/roadmap/api.md`. It describes future work only. Do not update
-current-behavior docs outside `docs/roadmap/` until the corresponding stage is
-implemented.
 
-Before implementation, read:
+Current behavior is documented in:
 
-- `docs/policy/architecture.md`
+- `docs/config.md`
-- `docs/policy/development.md`
+- `docs/integrations/http-upload.md`
-- `docs/policy/documentation.md`
+- `docs/consumers/api.md`
-- `docs/roadmap/api.md`
+- `docs/consumers/pkg-upload.md`
+- `docs/operations.md`
+- `docs/troubleshooting.md`
 
-## Target Behavior
+Deferred API work is tracked in `docs/roadmap/api.md` and broader deferred work is tracked in `docs/roadmap/future.md`.
 
-The final implementation is a breaking v1 HTTP upload API change:
+## Boundaries
 
-- Upload authentication is configured with top-level `upload_tokens`.
+- Keep producer routing, destination selection, transform policy, publish policy, transfer policy, and backend credentials out of source manifests.
-- Pipeline routing is selected by `POST /v1/pipelines/{pipeline_id}/upload`.
+- Keep `pkg/upload` focused on producer-side bundle submission and status polling.
-- Producers must set `PipelineID` in `pkg/upload` upload options.
+- Keep server-side upload authentication, authorization, queueing, status, and publish orchestration in `internal/app`.
-- Legacy per-source `source.token_env` is removed.
+- Keep archive extraction and staged bundle validation in `internal/ingest`.
-- Legacy `POST /upload` no longer accepts uploads.
+- Keep durable status, durable idempotency, retry endpoints, cancellation endpoints, and wait helpers out of the current implementation until a new roadmap item defines them.
-- Idempotency is scoped by token id, pipeline id, and idempotency key.
-- `/healthz` and `/runs/<run-id>` continue to work.
-
-The producer contract is:
-
-- token authenticates the producer/client;
-- `PipelineID` selects the configured distributor workflow;
-- source manifest `id` identifies the logical artifact within that workflow;
-- idempotency key identifies one producer run and retry group.
-
-## Stage 1: Config Model And Validation
-
-Goal: make the YAML schema express upload authentication separately from
-pipeline source configuration.
-
-Implementation:
-
-- Add `UploadTokens []UploadToken` to `internal/config.Config` with YAML key
-  `upload_tokens`.
-- Add `UploadToken` with fields:
-  - `ID string` as `id`;
-  - `TokenEnv string` as `token_env`;
-  - `AllowPipelines []string` as `allow_pipelines`.
-- Remove `TokenEnv` from `HTTPUpload`; keep `StagingPath` and
-  `MaxUploadSize`.
-- Keep `http_upload` source defaults for `staging_path` and `max_upload_size`.
-- Update validation:
-  - `upload_tokens` is required when any pipeline uses
-    `source.backend: http_upload`.
-  - `upload_tokens` is invalid when it references no configured upload
-    pipelines.
-  - token `id` is required, slug-like, and unique.
-  - token `token_env` is required.
-  - token `allow_pipelines` is required.
-  - each token's `allow_pipelines` entries are unique.
-  - each allowed pipeline id exists and names a pipeline whose source backend
-    is `http_upload`.
-  - every configured `http_upload` pipeline is allowed by at least one token.
-  - `http_upload` sources no longer require `token_env`.
-- Remove tests and fixtures that expect `source.token_env`.
-- Add config load/validation tests for valid multi-pipeline tokens, multiple
-  tokens for one pipeline, missing token list, duplicate token ids, duplicate
-  allowlist entries, unknown allowed pipeline id, non-upload allowed pipeline
-  id, and upload pipeline not allowed by any token.
-
-Acceptance:
-
-- `go test ./internal/config` passes.
-- YAML containing `source.token_env` fails as an unknown field.
-- YAML using top-level `upload_tokens` and `http_upload` sources without
-  `source.token_env` loads and validates.
-
-## Stage 2: Upload Token Resolution And HTTP Routing
-
-Goal: authenticate by bearer token, authorize by token allowlist, and route by
-URL pipeline id.
-
-Implementation:
-
-- Replace the current token-to-single-pipeline map with resolved upload token
-  records containing:
-  - token id;
-  - resolved token value;
-  - allowed pipeline id set.
-- Resolve token values through the existing config environment resolver so
-  process environment and `secrets.directory` behavior remains consistent.
-- Startup must fail when a token env is missing, empty, or resolves to the same
-  token value as another upload token. Error messages may name token ids and
-  env var names, but must not print token values.
-- Change `uploadHTTPHandler` routes:
-  - keep `GET /healthz`;
-  - keep `GET /runs/<run-id>`;
-  - add `POST /v1/pipelines/{pipeline_id}/upload`;
-  - remove legacy `POST /upload`.
-- Path parsing rules:
-  - match exactly `/v1/pipelines/<pipeline-id>/upload`;
-  - reject missing pipeline id, extra path segments, and query-based
-    `pipeline` or `pipeline_id` routing;
-  - validate requested pipeline id using the same slug-like id policy used for
-    configured pipeline ids.
-- Request handling rules:
-  - missing, malformed, or unknown bearer token returns `401`;
-  - valid token not allowed for requested pipeline returns `403`;
-  - requested pipeline must be configured with `source.backend: http_upload`;
-  - content type and idempotency key validation remain unchanged;
-  - successful requests submit the requested pipeline id and token id to the
-    upload coordinator.
-- Update HTTP handler tests for accepted v1 upload, unauthorized upload,
-  forbidden pipeline, invalid pipeline path, removed `/upload`, invalid
-  content type, invalid idempotency key, and no token leakage.
-
-Acceptance:
-
-- `go test ./internal/app` passes for handler tests touched in this stage.
-- `POST /upload` returns not found or another non-accepting error and does not
-  call `Submit`.
-- `POST /v1/pipelines/{pipeline_id}/upload` routes only when token
-  authorization allows that pipeline.
-
-## Stage 3: Coordinator Idempotency Scope
-
-Goal: prevent idempotency collisions between distinct authorized clients and
-between pipelines.
-
-Implementation:
-
-- Add `TokenID string` to `UploadRequest`.
-- Update idempotency scope to include token id, pipeline id, and key.
-- Preserve existing behavior inside one idempotency scope:
-  - same key and same normalized manifest returns the original accepted run;
-  - same key and different normalized manifest returns conflict;
-  - same key while staging returns retryable conflict.
-- Keep requests without idempotency keys unscoped and always admitted according
-  to queue capacity.
-- Update coordinator tests:
-  - same token, same pipeline, same key returns original run;
-  - same token, same pipeline, same key with changed manifest conflicts;
-  - different token ids can use the same key for the same pipeline without
-    collision;
-  - same token id can use the same key for different pipelines without
-    collision;
-  - pending-key retryable conflict still applies only within the same token and
-    pipeline scope.
-
-Acceptance:
-
-- `go test ./internal/app` passes.
-- Existing queueing, retention, status, and same-pipeline serialization behavior
-  remains unchanged.
-
-## Stage 4: Public `pkg/upload` API
-
-Goal: make producer clients route uploads through the v1 pipeline-scoped API.
-
-Implementation:
-
-- Add `PipelineID string` to `UploadFilesOptions`.
-- Add `PipelineID string` to `UploadBundleOptions`.
-- Require non-empty valid `PipelineID` in both upload methods before local
-  bundle staging, validation, archiving, or HTTP requests.
-- Use the same accepted pipeline id syntax as server config ids: starts with an
-  ASCII letter or digit and then contains ASCII letters, digits, `.`, `_`, or
-  `-`.
-- Build upload URLs as
-  `/v1/pipelines/{pipeline_id}/upload`.
-- Keep `Status(ctx, runID)` unchanged.
-- Update package tests:
-  - missing `PipelineID` fails before local file work or HTTP request;
-  - invalid `PipelineID` fails before HTTP request;
-  - `UploadFiles` and `UploadBundle` post to the v1 route;
-  - retry behavior reuses the same idempotency key and v1 route;
-  - token redaction still works.
-- Update `pkg/upload` package documentation to explain the four-part producer
-  contract: token, `PipelineID`, manifest `ID`, idempotency key.
-
-Acceptance:
-
-- `go test ./pkg/upload ./pkg/bundle` passes.
-- Existing `Status` tests pass without endpoint changes.
-
-## Stage 5: Examples And End-To-End App Coverage
-
-Goal: prove the new server and client API work together across realistic upload
-pipelines.
-
-Implementation:
-
-- Update `examples/http-upload-local.yml` to use top-level `upload_tokens`.
-- Update `examples/upload-client` to require or default a pipeline id and pass
-  it to `pkg/upload`.
-- Update app integration tests:
-  - one token authorized for two upload pipelines can upload to both;
-  - two tokens authorized for one upload pipeline can upload to that pipeline;
-  - a valid token rejected for a disallowed pipeline returns `403`;
-  - removed `/upload` endpoint does not enqueue work;
-  - full upload publishes through the selected pipeline and records the
-    selected pipeline id in status/report output.
-- Update helper config builders in tests to use `upload_tokens`.
-
-Acceptance:
-
-- `go test ./internal/app` passes.
-- Example config loading tests pass.
-- `go run ./examples/upload-client` remains documented as an example that
-  targets a configured pipeline.
-
-## Stage 6: Implemented-Behavior Documentation
-
-Goal: move the feature from roadmap-only language into current-behavior docs
-after the code is implemented.
-
-Implementation:
-
-- Update `docs/config.md`:
-  - document top-level `upload_tokens`;
-  - remove `source.token_env`;
-  - document `http_upload` source fields `staging_path` and
-    `max_upload_size`;
-  - document token allowlist semantics.
-- Update `docs/integrations/http-upload.md`:
-  - replace `POST /upload` with
-    `POST /v1/pipelines/{pipeline_id}/upload`;
-  - document `401` versus `403`;
-  - document idempotency scope by token id and pipeline id.
-- Update `docs/consumers/api.md` and `docs/consumers/pkg-upload.md`:
-  - show `PipelineID` in `UploadFiles` and `UploadBundle`;
-  - explain token authentication versus pipeline routing.
-- Update `docs/operations.md` and `docs/troubleshooting.md` for the new curl
-  path, upload token config, and forbidden-pipeline diagnosis.
-- Update `README.md` only if its producer integration links or summary become
-  stale.
-- Revise `docs/roadmap/api.md` so it no longer presents implemented behavior as
-  future work. Either mark the pipeline-scoped upload API as implemented and
-  leave only deferred ideas, or remove implemented sections.
-
-Acceptance:
-
-- `rg -n "POST /upload|source.token_env|token_env.*http_upload|maps each resolved bearer token to exactly one" README.md docs examples`
-  returns no stale current-behavior references outside historical roadmap
-  context.
-- Documentation outside `docs/roadmap/` describes only implemented behavior.
-
-## Final Verification
-
-Run these commands after all stages are implemented:
-
-```sh
-go test ./internal/config
-go test ./internal/app
-go test ./pkg/bundle ./pkg/upload
-go test ./...
-```
-
-Also run:
-
-```sh
-rg -n "POST /upload|source.token_env|/v1/pipelines|upload_tokens|PipelineID" README.md docs examples internal pkg
-```
-
-Review the output for stale references, missing docs, and tests that still
-expect legacy upload routing.
-
-## Non-Goals
-
-- Do not let producers specify destination ids, destination paths, transforms,
-  links, publish policy, transfer policy, or storage backends through upload
-  requests.
-- Do not add durable upload status, durable idempotency, retry endpoints,
-  cancellation endpoints, long polling, or `UploadAndWait`.
-- Do not add in-app TLS, public exposure policy, or rate limiting.
-- Do not introduce new public package families beyond existing `pkg/bundle` and
-  `pkg/upload`.