distributor/docs/roadmap/api.md

# API Roadmap

This document records planned API work that is not part of the current
implementation. Current HTTP upload behavior is documented in
`docs/integrations/http-upload.md` and current producer package usage is
documented under `docs/consumers/`.

## Pipeline-Scoped HTTP Upload API

Current `http_upload` behavior uses one bearer token to both authenticate a
producer and select exactly one pipeline. That is simple, but it does not scale
well for producer applications that generate multiple report types on different
schedules.

Planned work:

- Separate upload authentication from pipeline routing.
- Add token records that can authorize one producer/client for one or more
  upload pipelines.
- Add a pipeline-scoped upload endpoint:

```text
POST /v1/pipelines/{pipeline_id}/upload
```

- Keep the source manifest free of routing, destination, transform, and
  credential data.
- Keep `http_upload` source-only. A selected pipeline still owns destination
  configuration, transforms, links, transfer policy, and publication behavior.

## Proposed Configuration Shape

Move bearer token configuration out of individual pipeline sources and into a
top-level upload token list:

```yaml
upload_tokens:
  - id: weatherreporter-prod
    token_env: WEATHERREPORTER_UPLOAD_TOKEN
    allow_pipelines:
      - weather.morning
      - weather.weekend
      - weather.next_6_hours
      - weather.storm
      - weather.event
```

Pipeline sources would continue to use `http_upload`, but would no longer need
one unique token per pipeline:

```yaml
pipelines:
  - id: weather.morning
    source:
      backend: http_upload
    destinations:
      - id: archive
        backend: s3
        bucket: reports
        prefix: weather/morning/archive
      - id: latest
        backend: s3
        bucket: reports
        prefix: weather/morning/latest
        path_mapping:
          mode: fixed
```

Per-pipeline upload settings such as `staging_path` and `max_upload_size` should
remain on the `http_upload` source.

## Authorization Semantics

- Missing, malformed, or unknown bearer tokens should return `401 Unauthorized`.
- Valid tokens that are not allowed for the requested pipeline should return
  `403 Forbidden`.
- Requested pipeline ids must name configured pipelines whose source backend is
  `http_upload`.
- Multiple upload tokens may authorize the same pipeline.
- One upload token may authorize multiple pipelines.
- Token values must continue to resolve through the process environment or
  `secrets.directory`, not YAML literal values.

Idempotency records should be scoped by token id, pipeline id, and idempotency
key. This avoids collisions when multiple authorized producers submit to the
same pipeline.

## Producer Package Changes

Add `PipelineID` to producer upload options:

```go
result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
	PipelineID:     "weather.morning",
	ID:             "weather.morning.brentwood",
	IdempotencyKey: "weather.morning.brentwood.20260607T050000Z",
	Files: []bundle.BundleFile{
		{SourcePath: reportPath, Path: "report.md"},
		{SourcePath: dataPath, Path: "data.json"},
	},
})
```

`pkg/upload` should derive `/v1/pipelines/{pipeline_id}/upload` when
`PipelineID` is set. Status lookup can continue to use run ids returned by the
server.

The producer contract should remain:

- token identifies and authenticates the producer/client;
- `PipelineID` selects the configured distributor workflow;
- source manifest `id` identifies the logical artifact within that workflow;
- idempotency key identifies one producer run and retry group.

## Compatibility Plan

Prefer a transition period:

- Keep current `POST /upload` behavior for legacy configs where one token maps
  to exactly one `http_upload` pipeline.
- Reject legacy `/upload` routing when a token is authorized for multiple
  pipelines, because routing would be ambiguous.
- Keep rejecting `pipeline` and `pipeline_id` query parameters.
- Document `/v1/pipelines/{pipeline_id}/upload` as the preferred endpoint for
  new clients.

After the transition period, consider deprecating or removing legacy `/upload`
if the compatibility burden is no longer useful.

## Implementation Work

- Add `upload_tokens` config structs, defaults, validation, and secret
  resolution.
- Update upload token resolution to produce token identities and pipeline
  allowlists instead of a token-to-single-pipeline map.
- Add the `/v1/pipelines/{pipeline_id}/upload` HTTP route and pipeline id path
  validation.
- Preserve `/healthz` and `/runs/<run-id>` behavior.
- Pass token identity into upload admission so idempotency can be scoped by
  token id, pipeline id, and key.
- Add `PipelineID` to `pkg/upload` upload option structs and endpoint
  construction.
- Update configuration, operation, integration, consumer, and troubleshooting
  docs for implemented behavior.

## Tests

Important tests:

- Config loading and validation for `upload_tokens`.
- Startup failure for missing, empty, duplicated, or invalid upload token
  records.
- `401` for missing or unknown bearer token.
- `403` for valid token not allowed for requested pipeline.
- Successful upload to two different pipelines with one token.
- Successful upload to one pipeline from two different authorized tokens.
- Idempotency isolation across token ids and pipeline ids.
- Legacy `/upload` compatibility for one-token-one-pipeline routing.
- Legacy `/upload` rejection when routing is ambiguous.
- `pkg/upload` endpoint construction with `PipelineID`.
- Producer package tests for missing or invalid `PipelineID`.

## Boundaries

- Do not add destination selection to producer manifests.
- Do not let producers specify destination ids, transforms, links, publish
  policy, or transfer policy through the upload API.
- Do not add durable upload status or durable idempotency as part of this work;
  those remain separate roadmap items.
- Do not add in-app public exposure policy, TLS, or rate limiting as part of
  this work; those remain deployment-layer concerns unless a future
  implementation changes that boundary.