From 4fa7d1ebb57ef0691c061fb260fe7067fabed7f6 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 7 Jun 2026 23:19:26 -0500 Subject: [PATCH] Add roadmap and an implementation plan for HTTP API upgrades --- docs/roadmap/api.md | 172 +++++++++++++++++++++ docs/roadmap/implementation.md | 275 +++++++++++++++++++++++++++++++++ 2 files changed, 447 insertions(+) create mode 100644 docs/roadmap/api.md create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/api.md b/docs/roadmap/api.md new file mode 100644 index 0000000..de3512c --- /dev/null +++ b/docs/roadmap/api.md @@ -0,0 +1,172 @@ +# API Roadmap + +This document records planned API work that is not part of the current +implementation. Current HTTP upload behavior is documented in +`docs/integrations/http-upload.md` and current producer package usage is +documented under `docs/consumers/`. + +## Pipeline-Scoped HTTP Upload API + +Current `http_upload` behavior uses one bearer token to both authenticate a +producer and select exactly one pipeline. That is simple, but it does not scale +well for producer applications that generate multiple report types on different +schedules. + +Planned work: + +- Separate upload authentication from pipeline routing. +- Add token records that can authorize one producer/client for one or more + upload pipelines. +- Add a pipeline-scoped upload endpoint: + +```text +POST /v1/pipelines/{pipeline_id}/upload +``` + +- Keep the source manifest free of routing, destination, transform, and + credential data. +- Keep `http_upload` source-only. A selected pipeline still owns destination + configuration, transforms, links, transfer policy, and publication behavior. + +## Proposed Configuration Shape + +Move bearer token configuration out of individual pipeline sources and into a +top-level upload token list: + +```yaml +upload_tokens: + - id: weatherreporter-prod + token_env: WEATHERREPORTER_UPLOAD_TOKEN + allow_pipelines: + - weather.morning + - weather.weekend + - weather.next_6_hours + - weather.storm + - weather.event +``` + +Pipeline sources would continue to use `http_upload`, but would no longer need +one unique token per pipeline: + +```yaml +pipelines: + - id: weather.morning + source: + backend: http_upload + destinations: + - id: archive + backend: s3 + bucket: reports + prefix: weather/morning/archive + - id: latest + backend: s3 + bucket: reports + prefix: weather/morning/latest + path_mapping: + mode: fixed +``` + +Per-pipeline upload settings such as `staging_path` and `max_upload_size` should +remain on the `http_upload` source. + +## Authorization Semantics + +- Missing, malformed, or unknown bearer tokens should return `401 Unauthorized`. +- Valid tokens that are not allowed for the requested pipeline should return + `403 Forbidden`. +- Requested pipeline ids must name configured pipelines whose source backend is + `http_upload`. +- Multiple upload tokens may authorize the same pipeline. +- One upload token may authorize multiple pipelines. +- Token values must continue to resolve through the process environment or + `secrets.directory`, not YAML literal values. + +Idempotency records should be scoped by token id, pipeline id, and idempotency +key. This avoids collisions when multiple authorized producers submit to the +same pipeline. + +## Producer Package Changes + +Add `PipelineID` to producer upload options: + +```go +result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{ + PipelineID: "weather.morning", + ID: "weather.morning.brentwood", + IdempotencyKey: "weather.morning.brentwood.20260607T050000Z", + Files: []bundle.BundleFile{ + {SourcePath: reportPath, Path: "report.md"}, + {SourcePath: dataPath, Path: "data.json"}, + }, +}) +``` + +`pkg/upload` should derive `/v1/pipelines/{pipeline_id}/upload` when +`PipelineID` is set. Status lookup can continue to use run ids returned by the +server. + +The producer contract should remain: + +- token identifies and authenticates the producer/client; +- `PipelineID` selects the configured distributor workflow; +- source manifest `id` identifies the logical artifact within that workflow; +- idempotency key identifies one producer run and retry group. + +## Compatibility Plan + +Prefer a transition period: + +- Keep current `POST /upload` behavior for legacy configs where one token maps + to exactly one `http_upload` pipeline. +- Reject legacy `/upload` routing when a token is authorized for multiple + pipelines, because routing would be ambiguous. +- Keep rejecting `pipeline` and `pipeline_id` query parameters. +- Document `/v1/pipelines/{pipeline_id}/upload` as the preferred endpoint for + new clients. + +After the transition period, consider deprecating or removing legacy `/upload` +if the compatibility burden is no longer useful. + +## Implementation Work + +- Add `upload_tokens` config structs, defaults, validation, and secret + resolution. +- Update upload token resolution to produce token identities and pipeline + allowlists instead of a token-to-single-pipeline map. +- Add the `/v1/pipelines/{pipeline_id}/upload` HTTP route and pipeline id path + validation. +- Preserve `/healthz` and `/runs/` behavior. +- Pass token identity into upload admission so idempotency can be scoped by + token id, pipeline id, and key. +- Add `PipelineID` to `pkg/upload` upload option structs and endpoint + construction. +- Update configuration, operation, integration, consumer, and troubleshooting + docs for implemented behavior. + +## Tests + +Important tests: + +- Config loading and validation for `upload_tokens`. +- Startup failure for missing, empty, duplicated, or invalid upload token + records. +- `401` for missing or unknown bearer token. +- `403` for valid token not allowed for requested pipeline. +- Successful upload to two different pipelines with one token. +- Successful upload to one pipeline from two different authorized tokens. +- Idempotency isolation across token ids and pipeline ids. +- Legacy `/upload` compatibility for one-token-one-pipeline routing. +- Legacy `/upload` rejection when routing is ambiguous. +- `pkg/upload` endpoint construction with `PipelineID`. +- Producer package tests for missing or invalid `PipelineID`. + +## Boundaries + +- Do not add destination selection to producer manifests. +- Do not let producers specify destination ids, transforms, links, publish + policy, or transfer policy through the upload API. +- Do not add durable upload status or durable idempotency as part of this work; + those remain separate roadmap items. +- Do not add in-app public exposure policy, TLS, or rate limiting as part of + this work; those remain deployment-layer concerns unless a future + implementation changes that boundary. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..c231a2e --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,275 @@ +# Pipeline-Scoped Upload API Implementation Plan + +This roadmap is for an LLM coding agent implementing the planned API work in +`docs/roadmap/api.md`. It describes future work only. Do not update +current-behavior docs outside `docs/roadmap/` until the corresponding stage is +implemented. + +Before implementation, read: + +- `docs/policy/architecture.md` +- `docs/policy/development.md` +- `docs/policy/documentation.md` +- `docs/roadmap/api.md` + +## Target Behavior + +The final implementation is a breaking v1 HTTP upload API change: + +- Upload authentication is configured with top-level `upload_tokens`. +- Pipeline routing is selected by `POST /v1/pipelines/{pipeline_id}/upload`. +- Producers must set `PipelineID` in `pkg/upload` upload options. +- Legacy per-source `source.token_env` is removed. +- Legacy `POST /upload` no longer accepts uploads. +- Idempotency is scoped by token id, pipeline id, and idempotency key. +- `/healthz` and `/runs/` continue to work. + +The producer contract is: + +- token authenticates the producer/client; +- `PipelineID` selects the configured distributor workflow; +- source manifest `id` identifies the logical artifact within that workflow; +- idempotency key identifies one producer run and retry group. + +## Stage 1: Config Model And Validation + +Goal: make the YAML schema express upload authentication separately from +pipeline source configuration. + +Implementation: + +- Add `UploadTokens []UploadToken` to `internal/config.Config` with YAML key + `upload_tokens`. +- Add `UploadToken` with fields: + - `ID string` as `id`; + - `TokenEnv string` as `token_env`; + - `AllowPipelines []string` as `allow_pipelines`. +- Remove `TokenEnv` from `HTTPUpload`; keep `StagingPath` and + `MaxUploadSize`. +- Keep `http_upload` source defaults for `staging_path` and `max_upload_size`. +- Update validation: + - `upload_tokens` is required when any pipeline uses + `source.backend: http_upload`. + - `upload_tokens` is invalid when it references no configured upload + pipelines. + - token `id` is required, slug-like, and unique. + - token `token_env` is required. + - token `allow_pipelines` is required. + - each token's `allow_pipelines` entries are unique. + - each allowed pipeline id exists and names a pipeline whose source backend + is `http_upload`. + - every configured `http_upload` pipeline is allowed by at least one token. + - `http_upload` sources no longer require `token_env`. +- Remove tests and fixtures that expect `source.token_env`. +- Add config load/validation tests for valid multi-pipeline tokens, multiple + tokens for one pipeline, missing token list, duplicate token ids, duplicate + allowlist entries, unknown allowed pipeline id, non-upload allowed pipeline + id, and upload pipeline not allowed by any token. + +Acceptance: + +- `go test ./internal/config` passes. +- YAML containing `source.token_env` fails as an unknown field. +- YAML using top-level `upload_tokens` and `http_upload` sources without + `source.token_env` loads and validates. + +## Stage 2: Upload Token Resolution And HTTP Routing + +Goal: authenticate by bearer token, authorize by token allowlist, and route by +URL pipeline id. + +Implementation: + +- Replace the current token-to-single-pipeline map with resolved upload token + records containing: + - token id; + - resolved token value; + - allowed pipeline id set. +- Resolve token values through the existing config environment resolver so + process environment and `secrets.directory` behavior remains consistent. +- Startup must fail when a token env is missing, empty, or resolves to the same + token value as another upload token. Error messages may name token ids and + env var names, but must not print token values. +- Change `uploadHTTPHandler` routes: + - keep `GET /healthz`; + - keep `GET /runs/`; + - add `POST /v1/pipelines/{pipeline_id}/upload`; + - remove legacy `POST /upload`. +- Path parsing rules: + - match exactly `/v1/pipelines//upload`; + - reject missing pipeline id, extra path segments, and query-based + `pipeline` or `pipeline_id` routing; + - validate requested pipeline id using the same slug-like id policy used for + configured pipeline ids. +- Request handling rules: + - missing, malformed, or unknown bearer token returns `401`; + - valid token not allowed for requested pipeline returns `403`; + - requested pipeline must be configured with `source.backend: http_upload`; + - content type and idempotency key validation remain unchanged; + - successful requests submit the requested pipeline id and token id to the + upload coordinator. +- Update HTTP handler tests for accepted v1 upload, unauthorized upload, + forbidden pipeline, invalid pipeline path, removed `/upload`, invalid + content type, invalid idempotency key, and no token leakage. + +Acceptance: + +- `go test ./internal/app` passes for handler tests touched in this stage. +- `POST /upload` returns not found or another non-accepting error and does not + call `Submit`. +- `POST /v1/pipelines/{pipeline_id}/upload` routes only when token + authorization allows that pipeline. + +## Stage 3: Coordinator Idempotency Scope + +Goal: prevent idempotency collisions between distinct authorized clients and +between pipelines. + +Implementation: + +- Add `TokenID string` to `UploadRequest`. +- Update idempotency scope to include token id, pipeline id, and key. +- Preserve existing behavior inside one idempotency scope: + - same key and same normalized manifest returns the original accepted run; + - same key and different normalized manifest returns conflict; + - same key while staging returns retryable conflict. +- Keep requests without idempotency keys unscoped and always admitted according + to queue capacity. +- Update coordinator tests: + - same token, same pipeline, same key returns original run; + - same token, same pipeline, same key with changed manifest conflicts; + - different token ids can use the same key for the same pipeline without + collision; + - same token id can use the same key for different pipelines without + collision; + - pending-key retryable conflict still applies only within the same token and + pipeline scope. + +Acceptance: + +- `go test ./internal/app` passes. +- Existing queueing, retention, status, and same-pipeline serialization behavior + remains unchanged. + +## Stage 4: Public `pkg/upload` API + +Goal: make producer clients route uploads through the v1 pipeline-scoped API. + +Implementation: + +- Add `PipelineID string` to `UploadFilesOptions`. +- Add `PipelineID string` to `UploadBundleOptions`. +- Require non-empty valid `PipelineID` in both upload methods before local + bundle staging, validation, archiving, or HTTP requests. +- Use the same accepted pipeline id syntax as server config ids: starts with an + ASCII letter or digit and then contains ASCII letters, digits, `.`, `_`, or + `-`. +- Build upload URLs as + `/v1/pipelines/{pipeline_id}/upload`. +- Keep `Status(ctx, runID)` unchanged. +- Update package tests: + - missing `PipelineID` fails before local file work or HTTP request; + - invalid `PipelineID` fails before HTTP request; + - `UploadFiles` and `UploadBundle` post to the v1 route; + - retry behavior reuses the same idempotency key and v1 route; + - token redaction still works. +- Update `pkg/upload` package documentation to explain the four-part producer + contract: token, `PipelineID`, manifest `ID`, idempotency key. + +Acceptance: + +- `go test ./pkg/upload ./pkg/bundle` passes. +- Existing `Status` tests pass without endpoint changes. + +## Stage 5: Examples And End-To-End App Coverage + +Goal: prove the new server and client API work together across realistic upload +pipelines. + +Implementation: + +- Update `examples/http-upload-local.yml` to use top-level `upload_tokens`. +- Update `examples/upload-client` to require or default a pipeline id and pass + it to `pkg/upload`. +- Update app integration tests: + - one token authorized for two upload pipelines can upload to both; + - two tokens authorized for one upload pipeline can upload to that pipeline; + - a valid token rejected for a disallowed pipeline returns `403`; + - removed `/upload` endpoint does not enqueue work; + - full upload publishes through the selected pipeline and records the + selected pipeline id in status/report output. +- Update helper config builders in tests to use `upload_tokens`. + +Acceptance: + +- `go test ./internal/app` passes. +- Example config loading tests pass. +- `go run ./examples/upload-client` remains documented as an example that + targets a configured pipeline. + +## Stage 6: Implemented-Behavior Documentation + +Goal: move the feature from roadmap-only language into current-behavior docs +after the code is implemented. + +Implementation: + +- Update `docs/config.md`: + - document top-level `upload_tokens`; + - remove `source.token_env`; + - document `http_upload` source fields `staging_path` and + `max_upload_size`; + - document token allowlist semantics. +- Update `docs/integrations/http-upload.md`: + - replace `POST /upload` with + `POST /v1/pipelines/{pipeline_id}/upload`; + - document `401` versus `403`; + - document idempotency scope by token id and pipeline id. +- Update `docs/consumers/api.md` and `docs/consumers/pkg-upload.md`: + - show `PipelineID` in `UploadFiles` and `UploadBundle`; + - explain token authentication versus pipeline routing. +- Update `docs/operations.md` and `docs/troubleshooting.md` for the new curl + path, upload token config, and forbidden-pipeline diagnosis. +- Update `README.md` only if its producer integration links or summary become + stale. +- Revise `docs/roadmap/api.md` so it no longer presents implemented behavior as + future work. Either mark the pipeline-scoped upload API as implemented and + leave only deferred ideas, or remove implemented sections. + +Acceptance: + +- `rg -n "POST /upload|source.token_env|token_env.*http_upload|maps each resolved bearer token to exactly one" README.md docs examples` + returns no stale current-behavior references outside historical roadmap + context. +- Documentation outside `docs/roadmap/` describes only implemented behavior. + +## Final Verification + +Run these commands after all stages are implemented: + +```sh +go test ./internal/config +go test ./internal/app +go test ./pkg/bundle ./pkg/upload +go test ./... +``` + +Also run: + +```sh +rg -n "POST /upload|source.token_env|/v1/pipelines|upload_tokens|PipelineID" README.md docs examples internal pkg +``` + +Review the output for stale references, missing docs, and tests that still +expect legacy upload routing. + +## Non-Goals + +- Do not let producers specify destination ids, destination paths, transforms, + links, publish policy, transfer policy, or storage backends through upload + requests. +- Do not add durable upload status, durable idempotency, retry endpoints, + cancellation endpoints, long polling, or `UploadAndWait`. +- Do not add in-app TLS, public exposure policy, or rate limiting. +- Do not introduce new public package families beyond existing `pkg/bundle` and + `pkg/upload`.