Document pipeline-scoped upload behavior
This commit is contained in:
@@ -1,172 +1,35 @@
|
||||
# API Roadmap
|
||||
|
||||
This document records planned API work that is not part of the current
|
||||
implementation. Current HTTP upload behavior is documented in
|
||||
`docs/integrations/http-upload.md` and current producer package usage is
|
||||
documented under `docs/consumers/`.
|
||||
This document records API work that is not part of the current implementation. Current HTTP upload behavior is documented in `docs/integrations/http-upload.md`, current configuration behavior is documented in `docs/config.md`, and current producer package usage is documented under `docs/consumers/`.
|
||||
|
||||
## Pipeline-Scoped HTTP Upload API
|
||||
## Implemented HTTP Upload Behavior
|
||||
|
||||
Current `http_upload` behavior uses one bearer token to both authenticate a
|
||||
producer and select exactly one pipeline. That is simple, but it does not scale
|
||||
well for producer applications that generate multiple report types on different
|
||||
schedules.
|
||||
The pipeline-scoped HTTP upload API is implemented. The current contract is:
|
||||
|
||||
Planned work:
|
||||
- upload authentication is configured with top-level `upload_tokens`;
|
||||
- token values resolve through the process environment or `secrets.directory`, never YAML literals;
|
||||
- bearer tokens authenticate producers and authorize configured upload pipelines through `allow_pipelines`;
|
||||
- upload requests use `POST /v1/pipelines/{pipeline_id}/upload`;
|
||||
- missing, malformed, or unknown bearer tokens return `401 Unauthorized`;
|
||||
- valid tokens that are not allowed for the requested pipeline return `403 Forbidden`;
|
||||
- `pkg/upload` upload options require `PipelineID`;
|
||||
- idempotency records are scoped by token id, pipeline id, and idempotency key;
|
||||
- source manifests remain free of routing, destination, transform, and credential data.
|
||||
|
||||
- Separate upload authentication from pipeline routing.
|
||||
- Add token records that can authorize one producer/client for one or more
|
||||
upload pipelines.
|
||||
- Add a pipeline-scoped upload endpoint:
|
||||
The unscoped upload route no longer accepts uploads. Pipeline query parameters are rejected.
|
||||
|
||||
```text
|
||||
POST /v1/pipelines/{pipeline_id}/upload
|
||||
```
|
||||
## Deferred API Work
|
||||
|
||||
- Keep the source manifest free of routing, destination, transform, and
|
||||
credential data.
|
||||
- Keep `http_upload` source-only. A selected pipeline still owns destination
|
||||
configuration, transforms, links, transfer policy, and publication behavior.
|
||||
The following topics remain separate roadmap items:
|
||||
|
||||
## Proposed Configuration Shape
|
||||
|
||||
Move bearer token configuration out of individual pipeline sources and into a
|
||||
top-level upload token list:
|
||||
|
||||
```yaml
|
||||
upload_tokens:
|
||||
- id: weatherreporter-prod
|
||||
token_env: WEATHERREPORTER_UPLOAD_TOKEN
|
||||
allow_pipelines:
|
||||
- weather.morning
|
||||
- weather.weekend
|
||||
- weather.next_6_hours
|
||||
- weather.storm
|
||||
- weather.event
|
||||
```
|
||||
|
||||
Pipeline sources would continue to use `http_upload`, but would no longer need
|
||||
one unique token per pipeline:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: weather.morning
|
||||
source:
|
||||
backend: http_upload
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: s3
|
||||
bucket: reports
|
||||
prefix: weather/morning/archive
|
||||
- id: latest
|
||||
backend: s3
|
||||
bucket: reports
|
||||
prefix: weather/morning/latest
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
```
|
||||
|
||||
Per-pipeline upload settings such as `staging_path` and `max_upload_size` should
|
||||
remain on the `http_upload` source.
|
||||
|
||||
## Authorization Semantics
|
||||
|
||||
- Missing, malformed, or unknown bearer tokens should return `401 Unauthorized`.
|
||||
- Valid tokens that are not allowed for the requested pipeline should return
|
||||
`403 Forbidden`.
|
||||
- Requested pipeline ids must name configured pipelines whose source backend is
|
||||
`http_upload`.
|
||||
- Multiple upload tokens may authorize the same pipeline.
|
||||
- One upload token may authorize multiple pipelines.
|
||||
- Token values must continue to resolve through the process environment or
|
||||
`secrets.directory`, not YAML literal values.
|
||||
|
||||
Idempotency records should be scoped by token id, pipeline id, and idempotency
|
||||
key. This avoids collisions when multiple authorized producers submit to the
|
||||
same pipeline.
|
||||
|
||||
## Producer Package Changes
|
||||
|
||||
Add `PipelineID` to producer upload options:
|
||||
|
||||
```go
|
||||
result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
|
||||
PipelineID: "weather.morning",
|
||||
ID: "weather.morning.brentwood",
|
||||
IdempotencyKey: "weather.morning.brentwood.20260607T050000Z",
|
||||
Files: []bundle.BundleFile{
|
||||
{SourcePath: reportPath, Path: "report.md"},
|
||||
{SourcePath: dataPath, Path: "data.json"},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
`pkg/upload` should derive `/v1/pipelines/{pipeline_id}/upload` when
|
||||
`PipelineID` is set. Status lookup can continue to use run ids returned by the
|
||||
server.
|
||||
|
||||
The producer contract should remain:
|
||||
|
||||
- token identifies and authenticates the producer/client;
|
||||
- `PipelineID` selects the configured distributor workflow;
|
||||
- source manifest `id` identifies the logical artifact within that workflow;
|
||||
- idempotency key identifies one producer run and retry group.
|
||||
|
||||
## Compatibility Plan
|
||||
|
||||
Prefer a transition period:
|
||||
|
||||
- Keep current `POST /upload` behavior for legacy configs where one token maps
|
||||
to exactly one `http_upload` pipeline.
|
||||
- Reject legacy `/upload` routing when a token is authorized for multiple
|
||||
pipelines, because routing would be ambiguous.
|
||||
- Keep rejecting `pipeline` and `pipeline_id` query parameters.
|
||||
- Document `/v1/pipelines/{pipeline_id}/upload` as the preferred endpoint for
|
||||
new clients.
|
||||
|
||||
After the transition period, consider deprecating or removing legacy `/upload`
|
||||
if the compatibility burden is no longer useful.
|
||||
|
||||
## Implementation Work
|
||||
|
||||
- Add `upload_tokens` config structs, defaults, validation, and secret
|
||||
resolution.
|
||||
- Update upload token resolution to produce token identities and pipeline
|
||||
allowlists instead of a token-to-single-pipeline map.
|
||||
- Add the `/v1/pipelines/{pipeline_id}/upload` HTTP route and pipeline id path
|
||||
validation.
|
||||
- Preserve `/healthz` and `/runs/<run-id>` behavior.
|
||||
- Pass token identity into upload admission so idempotency can be scoped by
|
||||
token id, pipeline id, and key.
|
||||
- Add `PipelineID` to `pkg/upload` upload option structs and endpoint
|
||||
construction.
|
||||
- Update configuration, operation, integration, consumer, and troubleshooting
|
||||
docs for implemented behavior.
|
||||
|
||||
## Tests
|
||||
|
||||
Important tests:
|
||||
|
||||
- Config loading and validation for `upload_tokens`.
|
||||
- Startup failure for missing, empty, duplicated, or invalid upload token
|
||||
records.
|
||||
- `401` for missing or unknown bearer token.
|
||||
- `403` for valid token not allowed for requested pipeline.
|
||||
- Successful upload to two different pipelines with one token.
|
||||
- Successful upload to one pipeline from two different authorized tokens.
|
||||
- Idempotency isolation across token ids and pipeline ids.
|
||||
- Legacy `/upload` compatibility for one-token-one-pipeline routing.
|
||||
- Legacy `/upload` rejection when routing is ambiguous.
|
||||
- `pkg/upload` endpoint construction with `PipelineID`.
|
||||
- Producer package tests for missing or invalid `PipelineID`.
|
||||
- durable upload status storage;
|
||||
- durable idempotency records across server restarts;
|
||||
- in-app public exposure policy;
|
||||
- built-in TLS termination;
|
||||
- built-in upload rate limiting.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Do not add destination selection to producer manifests.
|
||||
- Do not let producers specify destination ids, transforms, links, publish
|
||||
policy, or transfer policy through the upload API.
|
||||
- Do not add durable upload status or durable idempotency as part of this work;
|
||||
those remain separate roadmap items.
|
||||
- Do not add in-app public exposure policy, TLS, or rate limiting as part of
|
||||
this work; those remain deployment-layer concerns unless a future
|
||||
implementation changes that boundary.
|
||||
- Do not let producers specify destination ids, transforms, links, publish policy, or transfer policy through the upload API.
|
||||
- Keep `http_upload` source-only. A selected pipeline owns destination configuration, transforms, links, transfer policy, and publication behavior.
|
||||
|
||||
Reference in New Issue
Block a user