API Roadmap

This document records planned API work that is not part of the current implementation. Current HTTP upload behavior is documented in docs/integrations/http-upload.md and current producer package usage is documented under docs/consumers/.

Pipeline-Scoped HTTP Upload API

Current http_upload behavior uses one bearer token to both authenticate a producer and select exactly one pipeline. That is simple, but it does not scale well for producer applications that generate multiple report types on different schedules.

Planned work:

Separate upload authentication from pipeline routing.
Add token records that can authorize one producer/client for one or more upload pipelines.
Add a pipeline-scoped upload endpoint:

POST /v1/pipelines/{pipeline_id}/upload

Keep the source manifest free of routing, destination, transform, and credential data.
Keep http_upload source-only. A selected pipeline still owns destination configuration, transforms, links, transfer policy, and publication behavior.

Proposed Configuration Shape

Move bearer token configuration out of individual pipeline sources and into a top-level upload token list:

upload_tokens:
  - id: weatherreporter-prod
    token_env: WEATHERREPORTER_UPLOAD_TOKEN
    allow_pipelines:
      - weather.morning
      - weather.weekend
      - weather.next_6_hours
      - weather.storm
      - weather.event

Pipeline sources would continue to use http_upload, but would no longer need one unique token per pipeline:

pipelines:
  - id: weather.morning
    source:
      backend: http_upload
    destinations:
      - id: archive
        backend: s3
        bucket: reports
        prefix: weather/morning/archive
      - id: latest
        backend: s3
        bucket: reports
        prefix: weather/morning/latest
        path_mapping:
          mode: fixed

Per-pipeline upload settings such as staging_path and max_upload_size should remain on the http_upload source.

Authorization Semantics

Missing, malformed, or unknown bearer tokens should return 401 Unauthorized.
Valid tokens that are not allowed for the requested pipeline should return 403 Forbidden.
Requested pipeline ids must name configured pipelines whose source backend is http_upload.
Multiple upload tokens may authorize the same pipeline.
One upload token may authorize multiple pipelines.
Token values must continue to resolve through the process environment or secrets.directory, not YAML literal values.

Idempotency records should be scoped by token id, pipeline id, and idempotency key. This avoids collisions when multiple authorized producers submit to the same pipeline.

Producer Package Changes

Add PipelineID to producer upload options:

result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
	PipelineID:     "weather.morning",
	ID:             "weather.morning.brentwood",
	IdempotencyKey: "weather.morning.brentwood.20260607T050000Z",
	Files: []bundle.BundleFile{
		{SourcePath: reportPath, Path: "report.md"},
		{SourcePath: dataPath, Path: "data.json"},
	},
})

pkg/upload should derive /v1/pipelines/{pipeline_id}/upload when PipelineID is set. Status lookup can continue to use run ids returned by the server.

The producer contract should remain:

token identifies and authenticates the producer/client;
PipelineID selects the configured distributor workflow;
source manifest id identifies the logical artifact within that workflow;
idempotency key identifies one producer run and retry group.

Compatibility Plan

Prefer a transition period:

Keep current POST /upload behavior for legacy configs where one token maps to exactly one http_upload pipeline.
Reject legacy /upload routing when a token is authorized for multiple pipelines, because routing would be ambiguous.
Keep rejecting pipeline and pipeline_id query parameters.
Document /v1/pipelines/{pipeline_id}/upload as the preferred endpoint for new clients.

After the transition period, consider deprecating or removing legacy /upload if the compatibility burden is no longer useful.

Implementation Work

Add upload_tokens config structs, defaults, validation, and secret resolution.
Update upload token resolution to produce token identities and pipeline allowlists instead of a token-to-single-pipeline map.
Add the /v1/pipelines/{pipeline_id}/upload HTTP route and pipeline id path validation.
Preserve /healthz and /runs/<run-id> behavior.
Pass token identity into upload admission so idempotency can be scoped by token id, pipeline id, and key.
Add PipelineID to pkg/upload upload option structs and endpoint construction.
Update configuration, operation, integration, consumer, and troubleshooting docs for implemented behavior.

Tests

Important tests:

Config loading and validation for upload_tokens.
Startup failure for missing, empty, duplicated, or invalid upload token records.
401 for missing or unknown bearer token.
403 for valid token not allowed for requested pipeline.
Successful upload to two different pipelines with one token.
Successful upload to one pipeline from two different authorized tokens.
Idempotency isolation across token ids and pipeline ids.
Legacy /upload compatibility for one-token-one-pipeline routing.
Legacy /upload rejection when routing is ambiguous.
pkg/upload endpoint construction with PipelineID.
Producer package tests for missing or invalid PipelineID.

Boundaries

Do not add destination selection to producer manifests.
Do not let producers specify destination ids, transforms, links, publish policy, or transfer policy through the upload API.
Do not add durable upload status or durable idempotency as part of this work; those remain separate roadmap items.
Do not add in-app public exposure policy, TLS, or rate limiting as part of this work; those remain deployment-layer concerns unless a future implementation changes that boundary.

6.0 KiB Raw Blame History