Add roadmap and an implementation plan for HTTP API upgrades
This commit is contained in:
172
docs/roadmap/api.md
Normal file
172
docs/roadmap/api.md
Normal file
@@ -0,0 +1,172 @@
|
||||
# API Roadmap
|
||||
|
||||
This document records planned API work that is not part of the current
|
||||
implementation. Current HTTP upload behavior is documented in
|
||||
`docs/integrations/http-upload.md` and current producer package usage is
|
||||
documented under `docs/consumers/`.
|
||||
|
||||
## Pipeline-Scoped HTTP Upload API
|
||||
|
||||
Current `http_upload` behavior uses one bearer token to both authenticate a
|
||||
producer and select exactly one pipeline. That is simple, but it does not scale
|
||||
well for producer applications that generate multiple report types on different
|
||||
schedules.
|
||||
|
||||
Planned work:
|
||||
|
||||
- Separate upload authentication from pipeline routing.
|
||||
- Add token records that can authorize one producer/client for one or more
|
||||
upload pipelines.
|
||||
- Add a pipeline-scoped upload endpoint:
|
||||
|
||||
```text
|
||||
POST /v1/pipelines/{pipeline_id}/upload
|
||||
```
|
||||
|
||||
- Keep the source manifest free of routing, destination, transform, and
|
||||
credential data.
|
||||
- Keep `http_upload` source-only. A selected pipeline still owns destination
|
||||
configuration, transforms, links, transfer policy, and publication behavior.
|
||||
|
||||
## Proposed Configuration Shape
|
||||
|
||||
Move bearer token configuration out of individual pipeline sources and into a
|
||||
top-level upload token list:
|
||||
|
||||
```yaml
|
||||
upload_tokens:
|
||||
- id: weatherreporter-prod
|
||||
token_env: WEATHERREPORTER_UPLOAD_TOKEN
|
||||
allow_pipelines:
|
||||
- weather.morning
|
||||
- weather.weekend
|
||||
- weather.next_6_hours
|
||||
- weather.storm
|
||||
- weather.event
|
||||
```
|
||||
|
||||
Pipeline sources would continue to use `http_upload`, but would no longer need
|
||||
one unique token per pipeline:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: weather.morning
|
||||
source:
|
||||
backend: http_upload
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: s3
|
||||
bucket: reports
|
||||
prefix: weather/morning/archive
|
||||
- id: latest
|
||||
backend: s3
|
||||
bucket: reports
|
||||
prefix: weather/morning/latest
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
```
|
||||
|
||||
Per-pipeline upload settings such as `staging_path` and `max_upload_size` should
|
||||
remain on the `http_upload` source.
|
||||
|
||||
## Authorization Semantics
|
||||
|
||||
- Missing, malformed, or unknown bearer tokens should return `401 Unauthorized`.
|
||||
- Valid tokens that are not allowed for the requested pipeline should return
|
||||
`403 Forbidden`.
|
||||
- Requested pipeline ids must name configured pipelines whose source backend is
|
||||
`http_upload`.
|
||||
- Multiple upload tokens may authorize the same pipeline.
|
||||
- One upload token may authorize multiple pipelines.
|
||||
- Token values must continue to resolve through the process environment or
|
||||
`secrets.directory`, not YAML literal values.
|
||||
|
||||
Idempotency records should be scoped by token id, pipeline id, and idempotency
|
||||
key. This avoids collisions when multiple authorized producers submit to the
|
||||
same pipeline.
|
||||
|
||||
## Producer Package Changes
|
||||
|
||||
Add `PipelineID` to producer upload options:
|
||||
|
||||
```go
|
||||
result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
|
||||
PipelineID: "weather.morning",
|
||||
ID: "weather.morning.brentwood",
|
||||
IdempotencyKey: "weather.morning.brentwood.20260607T050000Z",
|
||||
Files: []bundle.BundleFile{
|
||||
{SourcePath: reportPath, Path: "report.md"},
|
||||
{SourcePath: dataPath, Path: "data.json"},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
`pkg/upload` should derive `/v1/pipelines/{pipeline_id}/upload` when
|
||||
`PipelineID` is set. Status lookup can continue to use run ids returned by the
|
||||
server.
|
||||
|
||||
The producer contract should remain:
|
||||
|
||||
- token identifies and authenticates the producer/client;
|
||||
- `PipelineID` selects the configured distributor workflow;
|
||||
- source manifest `id` identifies the logical artifact within that workflow;
|
||||
- idempotency key identifies one producer run and retry group.
|
||||
|
||||
## Compatibility Plan
|
||||
|
||||
Prefer a transition period:
|
||||
|
||||
- Keep current `POST /upload` behavior for legacy configs where one token maps
|
||||
to exactly one `http_upload` pipeline.
|
||||
- Reject legacy `/upload` routing when a token is authorized for multiple
|
||||
pipelines, because routing would be ambiguous.
|
||||
- Keep rejecting `pipeline` and `pipeline_id` query parameters.
|
||||
- Document `/v1/pipelines/{pipeline_id}/upload` as the preferred endpoint for
|
||||
new clients.
|
||||
|
||||
After the transition period, consider deprecating or removing legacy `/upload`
|
||||
if the compatibility burden is no longer useful.
|
||||
|
||||
## Implementation Work
|
||||
|
||||
- Add `upload_tokens` config structs, defaults, validation, and secret
|
||||
resolution.
|
||||
- Update upload token resolution to produce token identities and pipeline
|
||||
allowlists instead of a token-to-single-pipeline map.
|
||||
- Add the `/v1/pipelines/{pipeline_id}/upload` HTTP route and pipeline id path
|
||||
validation.
|
||||
- Preserve `/healthz` and `/runs/<run-id>` behavior.
|
||||
- Pass token identity into upload admission so idempotency can be scoped by
|
||||
token id, pipeline id, and key.
|
||||
- Add `PipelineID` to `pkg/upload` upload option structs and endpoint
|
||||
construction.
|
||||
- Update configuration, operation, integration, consumer, and troubleshooting
|
||||
docs for implemented behavior.
|
||||
|
||||
## Tests
|
||||
|
||||
Important tests:
|
||||
|
||||
- Config loading and validation for `upload_tokens`.
|
||||
- Startup failure for missing, empty, duplicated, or invalid upload token
|
||||
records.
|
||||
- `401` for missing or unknown bearer token.
|
||||
- `403` for valid token not allowed for requested pipeline.
|
||||
- Successful upload to two different pipelines with one token.
|
||||
- Successful upload to one pipeline from two different authorized tokens.
|
||||
- Idempotency isolation across token ids and pipeline ids.
|
||||
- Legacy `/upload` compatibility for one-token-one-pipeline routing.
|
||||
- Legacy `/upload` rejection when routing is ambiguous.
|
||||
- `pkg/upload` endpoint construction with `PipelineID`.
|
||||
- Producer package tests for missing or invalid `PipelineID`.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Do not add destination selection to producer manifests.
|
||||
- Do not let producers specify destination ids, transforms, links, publish
|
||||
policy, or transfer policy through the upload API.
|
||||
- Do not add durable upload status or durable idempotency as part of this work;
|
||||
those remain separate roadmap items.
|
||||
- Do not add in-app public exposure policy, TLS, or rate limiting as part of
|
||||
this work; those remain deployment-layer concerns unless a future
|
||||
implementation changes that boundary.
|
||||
275
docs/roadmap/implementation.md
Normal file
275
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,275 @@
|
||||
# Pipeline-Scoped Upload API Implementation Plan
|
||||
|
||||
This roadmap is for an LLM coding agent implementing the planned API work in
|
||||
`docs/roadmap/api.md`. It describes future work only. Do not update
|
||||
current-behavior docs outside `docs/roadmap/` until the corresponding stage is
|
||||
implemented.
|
||||
|
||||
Before implementation, read:
|
||||
|
||||
- `docs/policy/architecture.md`
|
||||
- `docs/policy/development.md`
|
||||
- `docs/policy/documentation.md`
|
||||
- `docs/roadmap/api.md`
|
||||
|
||||
## Target Behavior
|
||||
|
||||
The final implementation is a breaking v1 HTTP upload API change:
|
||||
|
||||
- Upload authentication is configured with top-level `upload_tokens`.
|
||||
- Pipeline routing is selected by `POST /v1/pipelines/{pipeline_id}/upload`.
|
||||
- Producers must set `PipelineID` in `pkg/upload` upload options.
|
||||
- Legacy per-source `source.token_env` is removed.
|
||||
- Legacy `POST /upload` no longer accepts uploads.
|
||||
- Idempotency is scoped by token id, pipeline id, and idempotency key.
|
||||
- `/healthz` and `/runs/<run-id>` continue to work.
|
||||
|
||||
The producer contract is:
|
||||
|
||||
- token authenticates the producer/client;
|
||||
- `PipelineID` selects the configured distributor workflow;
|
||||
- source manifest `id` identifies the logical artifact within that workflow;
|
||||
- idempotency key identifies one producer run and retry group.
|
||||
|
||||
## Stage 1: Config Model And Validation
|
||||
|
||||
Goal: make the YAML schema express upload authentication separately from
|
||||
pipeline source configuration.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Add `UploadTokens []UploadToken` to `internal/config.Config` with YAML key
|
||||
`upload_tokens`.
|
||||
- Add `UploadToken` with fields:
|
||||
- `ID string` as `id`;
|
||||
- `TokenEnv string` as `token_env`;
|
||||
- `AllowPipelines []string` as `allow_pipelines`.
|
||||
- Remove `TokenEnv` from `HTTPUpload`; keep `StagingPath` and
|
||||
`MaxUploadSize`.
|
||||
- Keep `http_upload` source defaults for `staging_path` and `max_upload_size`.
|
||||
- Update validation:
|
||||
- `upload_tokens` is required when any pipeline uses
|
||||
`source.backend: http_upload`.
|
||||
- `upload_tokens` is invalid when it references no configured upload
|
||||
pipelines.
|
||||
- token `id` is required, slug-like, and unique.
|
||||
- token `token_env` is required.
|
||||
- token `allow_pipelines` is required.
|
||||
- each token's `allow_pipelines` entries are unique.
|
||||
- each allowed pipeline id exists and names a pipeline whose source backend
|
||||
is `http_upload`.
|
||||
- every configured `http_upload` pipeline is allowed by at least one token.
|
||||
- `http_upload` sources no longer require `token_env`.
|
||||
- Remove tests and fixtures that expect `source.token_env`.
|
||||
- Add config load/validation tests for valid multi-pipeline tokens, multiple
|
||||
tokens for one pipeline, missing token list, duplicate token ids, duplicate
|
||||
allowlist entries, unknown allowed pipeline id, non-upload allowed pipeline
|
||||
id, and upload pipeline not allowed by any token.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `go test ./internal/config` passes.
|
||||
- YAML containing `source.token_env` fails as an unknown field.
|
||||
- YAML using top-level `upload_tokens` and `http_upload` sources without
|
||||
`source.token_env` loads and validates.
|
||||
|
||||
## Stage 2: Upload Token Resolution And HTTP Routing
|
||||
|
||||
Goal: authenticate by bearer token, authorize by token allowlist, and route by
|
||||
URL pipeline id.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Replace the current token-to-single-pipeline map with resolved upload token
|
||||
records containing:
|
||||
- token id;
|
||||
- resolved token value;
|
||||
- allowed pipeline id set.
|
||||
- Resolve token values through the existing config environment resolver so
|
||||
process environment and `secrets.directory` behavior remains consistent.
|
||||
- Startup must fail when a token env is missing, empty, or resolves to the same
|
||||
token value as another upload token. Error messages may name token ids and
|
||||
env var names, but must not print token values.
|
||||
- Change `uploadHTTPHandler` routes:
|
||||
- keep `GET /healthz`;
|
||||
- keep `GET /runs/<run-id>`;
|
||||
- add `POST /v1/pipelines/{pipeline_id}/upload`;
|
||||
- remove legacy `POST /upload`.
|
||||
- Path parsing rules:
|
||||
- match exactly `/v1/pipelines/<pipeline-id>/upload`;
|
||||
- reject missing pipeline id, extra path segments, and query-based
|
||||
`pipeline` or `pipeline_id` routing;
|
||||
- validate requested pipeline id using the same slug-like id policy used for
|
||||
configured pipeline ids.
|
||||
- Request handling rules:
|
||||
- missing, malformed, or unknown bearer token returns `401`;
|
||||
- valid token not allowed for requested pipeline returns `403`;
|
||||
- requested pipeline must be configured with `source.backend: http_upload`;
|
||||
- content type and idempotency key validation remain unchanged;
|
||||
- successful requests submit the requested pipeline id and token id to the
|
||||
upload coordinator.
|
||||
- Update HTTP handler tests for accepted v1 upload, unauthorized upload,
|
||||
forbidden pipeline, invalid pipeline path, removed `/upload`, invalid
|
||||
content type, invalid idempotency key, and no token leakage.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `go test ./internal/app` passes for handler tests touched in this stage.
|
||||
- `POST /upload` returns not found or another non-accepting error and does not
|
||||
call `Submit`.
|
||||
- `POST /v1/pipelines/{pipeline_id}/upload` routes only when token
|
||||
authorization allows that pipeline.
|
||||
|
||||
## Stage 3: Coordinator Idempotency Scope
|
||||
|
||||
Goal: prevent idempotency collisions between distinct authorized clients and
|
||||
between pipelines.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Add `TokenID string` to `UploadRequest`.
|
||||
- Update idempotency scope to include token id, pipeline id, and key.
|
||||
- Preserve existing behavior inside one idempotency scope:
|
||||
- same key and same normalized manifest returns the original accepted run;
|
||||
- same key and different normalized manifest returns conflict;
|
||||
- same key while staging returns retryable conflict.
|
||||
- Keep requests without idempotency keys unscoped and always admitted according
|
||||
to queue capacity.
|
||||
- Update coordinator tests:
|
||||
- same token, same pipeline, same key returns original run;
|
||||
- same token, same pipeline, same key with changed manifest conflicts;
|
||||
- different token ids can use the same key for the same pipeline without
|
||||
collision;
|
||||
- same token id can use the same key for different pipelines without
|
||||
collision;
|
||||
- pending-key retryable conflict still applies only within the same token and
|
||||
pipeline scope.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `go test ./internal/app` passes.
|
||||
- Existing queueing, retention, status, and same-pipeline serialization behavior
|
||||
remains unchanged.
|
||||
|
||||
## Stage 4: Public `pkg/upload` API
|
||||
|
||||
Goal: make producer clients route uploads through the v1 pipeline-scoped API.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Add `PipelineID string` to `UploadFilesOptions`.
|
||||
- Add `PipelineID string` to `UploadBundleOptions`.
|
||||
- Require non-empty valid `PipelineID` in both upload methods before local
|
||||
bundle staging, validation, archiving, or HTTP requests.
|
||||
- Use the same accepted pipeline id syntax as server config ids: starts with an
|
||||
ASCII letter or digit and then contains ASCII letters, digits, `.`, `_`, or
|
||||
`-`.
|
||||
- Build upload URLs as
|
||||
`/v1/pipelines/{pipeline_id}/upload`.
|
||||
- Keep `Status(ctx, runID)` unchanged.
|
||||
- Update package tests:
|
||||
- missing `PipelineID` fails before local file work or HTTP request;
|
||||
- invalid `PipelineID` fails before HTTP request;
|
||||
- `UploadFiles` and `UploadBundle` post to the v1 route;
|
||||
- retry behavior reuses the same idempotency key and v1 route;
|
||||
- token redaction still works.
|
||||
- Update `pkg/upload` package documentation to explain the four-part producer
|
||||
contract: token, `PipelineID`, manifest `ID`, idempotency key.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `go test ./pkg/upload ./pkg/bundle` passes.
|
||||
- Existing `Status` tests pass without endpoint changes.
|
||||
|
||||
## Stage 5: Examples And End-To-End App Coverage
|
||||
|
||||
Goal: prove the new server and client API work together across realistic upload
|
||||
pipelines.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Update `examples/http-upload-local.yml` to use top-level `upload_tokens`.
|
||||
- Update `examples/upload-client` to require or default a pipeline id and pass
|
||||
it to `pkg/upload`.
|
||||
- Update app integration tests:
|
||||
- one token authorized for two upload pipelines can upload to both;
|
||||
- two tokens authorized for one upload pipeline can upload to that pipeline;
|
||||
- a valid token rejected for a disallowed pipeline returns `403`;
|
||||
- removed `/upload` endpoint does not enqueue work;
|
||||
- full upload publishes through the selected pipeline and records the
|
||||
selected pipeline id in status/report output.
|
||||
- Update helper config builders in tests to use `upload_tokens`.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `go test ./internal/app` passes.
|
||||
- Example config loading tests pass.
|
||||
- `go run ./examples/upload-client` remains documented as an example that
|
||||
targets a configured pipeline.
|
||||
|
||||
## Stage 6: Implemented-Behavior Documentation
|
||||
|
||||
Goal: move the feature from roadmap-only language into current-behavior docs
|
||||
after the code is implemented.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Update `docs/config.md`:
|
||||
- document top-level `upload_tokens`;
|
||||
- remove `source.token_env`;
|
||||
- document `http_upload` source fields `staging_path` and
|
||||
`max_upload_size`;
|
||||
- document token allowlist semantics.
|
||||
- Update `docs/integrations/http-upload.md`:
|
||||
- replace `POST /upload` with
|
||||
`POST /v1/pipelines/{pipeline_id}/upload`;
|
||||
- document `401` versus `403`;
|
||||
- document idempotency scope by token id and pipeline id.
|
||||
- Update `docs/consumers/api.md` and `docs/consumers/pkg-upload.md`:
|
||||
- show `PipelineID` in `UploadFiles` and `UploadBundle`;
|
||||
- explain token authentication versus pipeline routing.
|
||||
- Update `docs/operations.md` and `docs/troubleshooting.md` for the new curl
|
||||
path, upload token config, and forbidden-pipeline diagnosis.
|
||||
- Update `README.md` only if its producer integration links or summary become
|
||||
stale.
|
||||
- Revise `docs/roadmap/api.md` so it no longer presents implemented behavior as
|
||||
future work. Either mark the pipeline-scoped upload API as implemented and
|
||||
leave only deferred ideas, or remove implemented sections.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `rg -n "POST /upload|source.token_env|token_env.*http_upload|maps each resolved bearer token to exactly one" README.md docs examples`
|
||||
returns no stale current-behavior references outside historical roadmap
|
||||
context.
|
||||
- Documentation outside `docs/roadmap/` describes only implemented behavior.
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run these commands after all stages are implemented:
|
||||
|
||||
```sh
|
||||
go test ./internal/config
|
||||
go test ./internal/app
|
||||
go test ./pkg/bundle ./pkg/upload
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Also run:
|
||||
|
||||
```sh
|
||||
rg -n "POST /upload|source.token_env|/v1/pipelines|upload_tokens|PipelineID" README.md docs examples internal pkg
|
||||
```
|
||||
|
||||
Review the output for stale references, missing docs, and tests that still
|
||||
expect legacy upload routing.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not let producers specify destination ids, destination paths, transforms,
|
||||
links, publish policy, transfer policy, or storage backends through upload
|
||||
requests.
|
||||
- Do not add durable upload status, durable idempotency, retry endpoints,
|
||||
cancellation endpoints, long polling, or `UploadAndWait`.
|
||||
- Do not add in-app TLS, public exposure policy, or rate limiting.
|
||||
- Do not introduce new public package families beyond existing `pkg/bundle` and
|
||||
`pkg/upload`.
|
||||
Reference in New Issue
Block a user