# Pipeline-Scoped Upload API Implementation Plan

This roadmap is for an LLM coding agent implementing the planned API work in
`docs/roadmap/api.md`. It describes future work only. Do not update
current-behavior docs outside `docs/roadmap/` until the corresponding stage is
implemented.

Before implementation, read:

- `docs/policy/architecture.md`
- `docs/policy/development.md`
- `docs/policy/documentation.md`
- `docs/roadmap/api.md`

## Target Behavior

The final implementation is a breaking v1 HTTP upload API change:

- Upload authentication is configured with top-level `upload_tokens`.
- Pipeline routing is selected by `POST /v1/pipelines/{pipeline_id}/upload`.
- Producers must set `PipelineID` in `pkg/upload` upload options.
- Legacy per-source `source.token_env` is removed.
- Legacy `POST /upload` no longer accepts uploads.
- Idempotency is scoped by token id, pipeline id, and idempotency key.
- `/healthz` and `/runs/<run-id>` continue to work.

The producer contract is:

- token authenticates the producer/client;
- `PipelineID` selects the configured distributor workflow;
- source manifest `id` identifies the logical artifact within that workflow;
- idempotency key identifies one producer run and retry group.

## Stage 1: Config Model And Validation

Goal: make the YAML schema express upload authentication separately from
pipeline source configuration.

Implementation:

- Add `UploadTokens []UploadToken` to `internal/config.Config` with YAML key
  `upload_tokens`.
- Add `UploadToken` with fields:
  - `ID string` as `id`;
  - `TokenEnv string` as `token_env`;
  - `AllowPipelines []string` as `allow_pipelines`.
- Remove `TokenEnv` from `HTTPUpload`; keep `StagingPath` and
  `MaxUploadSize`.
- Keep `http_upload` source defaults for `staging_path` and `max_upload_size`.
- Update validation:
  - `upload_tokens` is required when any pipeline uses
    `source.backend: http_upload`.
  - `upload_tokens` is invalid when it references no configured upload
    pipelines.
  - token `id` is required, slug-like, and unique.
  - token `token_env` is required.
  - token `allow_pipelines` is required.
  - each token's `allow_pipelines` entries are unique.
  - each allowed pipeline id exists and names a pipeline whose source backend
    is `http_upload`.
  - every configured `http_upload` pipeline is allowed by at least one token.
  - `http_upload` sources no longer require `token_env`.
- Remove tests and fixtures that expect `source.token_env`.
- Add config load/validation tests for valid multi-pipeline tokens, multiple
  tokens for one pipeline, missing token list, duplicate token ids, duplicate
  allowlist entries, unknown allowed pipeline id, non-upload allowed pipeline
  id, and upload pipeline not allowed by any token.

Acceptance:

- `go test ./internal/config` passes.
- YAML containing `source.token_env` fails as an unknown field.
- YAML using top-level `upload_tokens` and `http_upload` sources without
  `source.token_env` loads and validates.

## Stage 2: Upload Token Resolution And HTTP Routing

Goal: authenticate by bearer token, authorize by token allowlist, and route by
URL pipeline id.

Implementation:

- Replace the current token-to-single-pipeline map with resolved upload token
  records containing:
  - token id;
  - resolved token value;
  - allowed pipeline id set.
- Resolve token values through the existing config environment resolver so
  process environment and `secrets.directory` behavior remains consistent.
- Startup must fail when a token env is missing, empty, or resolves to the same
  token value as another upload token. Error messages may name token ids and
  env var names, but must not print token values.
- Change `uploadHTTPHandler` routes:
  - keep `GET /healthz`;
  - keep `GET /runs/<run-id>`;
  - add `POST /v1/pipelines/{pipeline_id}/upload`;
  - remove legacy `POST /upload`.
- Path parsing rules:
  - match exactly `/v1/pipelines/<pipeline-id>/upload`;
  - reject missing pipeline id, extra path segments, and query-based
    `pipeline` or `pipeline_id` routing;
  - validate requested pipeline id using the same slug-like id policy used for
    configured pipeline ids.
- Request handling rules:
  - missing, malformed, or unknown bearer token returns `401`;
  - valid token not allowed for requested pipeline returns `403`;
  - requested pipeline must be configured with `source.backend: http_upload`;
  - content type and idempotency key validation remain unchanged;
  - successful requests submit the requested pipeline id and token id to the
    upload coordinator.
- Update HTTP handler tests for accepted v1 upload, unauthorized upload,
  forbidden pipeline, invalid pipeline path, removed `/upload`, invalid
  content type, invalid idempotency key, and no token leakage.

Acceptance:

- `go test ./internal/app` passes for handler tests touched in this stage.
- `POST /upload` returns not found or another non-accepting error and does not
  call `Submit`.
- `POST /v1/pipelines/{pipeline_id}/upload` routes only when token
  authorization allows that pipeline.

## Stage 3: Coordinator Idempotency Scope

Goal: prevent idempotency collisions between distinct authorized clients and
between pipelines.

Implementation:

- Add `TokenID string` to `UploadRequest`.
- Update idempotency scope to include token id, pipeline id, and key.
- Preserve existing behavior inside one idempotency scope:
  - same key and same normalized manifest returns the original accepted run;
  - same key and different normalized manifest returns conflict;
  - same key while staging returns retryable conflict.
- Keep requests without idempotency keys unscoped and always admitted according
  to queue capacity.
- Update coordinator tests:
  - same token, same pipeline, same key returns original run;
  - same token, same pipeline, same key with changed manifest conflicts;
  - different token ids can use the same key for the same pipeline without
    collision;
  - same token id can use the same key for different pipelines without
    collision;
  - pending-key retryable conflict still applies only within the same token and
    pipeline scope.

Acceptance:

- `go test ./internal/app` passes.
- Existing queueing, retention, status, and same-pipeline serialization behavior
  remains unchanged.

## Stage 4: Public `pkg/upload` API

Goal: make producer clients route uploads through the v1 pipeline-scoped API.

Implementation:

- Add `PipelineID string` to `UploadFilesOptions`.
- Add `PipelineID string` to `UploadBundleOptions`.
- Require non-empty valid `PipelineID` in both upload methods before local
  bundle staging, validation, archiving, or HTTP requests.
- Use the same accepted pipeline id syntax as server config ids: starts with an
  ASCII letter or digit and then contains ASCII letters, digits, `.`, `_`, or
  `-`.
- Build upload URLs as
  `/v1/pipelines/{pipeline_id}/upload`.
- Keep `Status(ctx, runID)` unchanged.
- Update package tests:
  - missing `PipelineID` fails before local file work or HTTP request;
  - invalid `PipelineID` fails before HTTP request;
  - `UploadFiles` and `UploadBundle` post to the v1 route;
  - retry behavior reuses the same idempotency key and v1 route;
  - token redaction still works.
- Update `pkg/upload` package documentation to explain the four-part producer
  contract: token, `PipelineID`, manifest `ID`, idempotency key.

Acceptance:

- `go test ./pkg/upload ./pkg/bundle` passes.
- Existing `Status` tests pass without endpoint changes.

## Stage 5: Examples And End-To-End App Coverage

Goal: prove the new server and client API work together across realistic upload
pipelines.

Implementation:

- Update `examples/http-upload-local.yml` to use top-level `upload_tokens`.
- Update `examples/upload-client` to require or default a pipeline id and pass
  it to `pkg/upload`.
- Update app integration tests:
  - one token authorized for two upload pipelines can upload to both;
  - two tokens authorized for one upload pipeline can upload to that pipeline;
  - a valid token rejected for a disallowed pipeline returns `403`;
  - removed `/upload` endpoint does not enqueue work;
  - full upload publishes through the selected pipeline and records the
    selected pipeline id in status/report output.
- Update helper config builders in tests to use `upload_tokens`.

Acceptance:

- `go test ./internal/app` passes.
- Example config loading tests pass.
- `go run ./examples/upload-client` remains documented as an example that
  targets a configured pipeline.

## Stage 6: Implemented-Behavior Documentation

Goal: move the feature from roadmap-only language into current-behavior docs
after the code is implemented.

Implementation:

- Update `docs/config.md`:
  - document top-level `upload_tokens`;
  - remove `source.token_env`;
  - document `http_upload` source fields `staging_path` and
    `max_upload_size`;
  - document token allowlist semantics.
- Update `docs/integrations/http-upload.md`:
  - replace `POST /upload` with
    `POST /v1/pipelines/{pipeline_id}/upload`;
  - document `401` versus `403`;
  - document idempotency scope by token id and pipeline id.
- Update `docs/consumers/api.md` and `docs/consumers/pkg-upload.md`:
  - show `PipelineID` in `UploadFiles` and `UploadBundle`;
  - explain token authentication versus pipeline routing.
- Update `docs/operations.md` and `docs/troubleshooting.md` for the new curl
  path, upload token config, and forbidden-pipeline diagnosis.
- Update `README.md` only if its producer integration links or summary become
  stale.
- Revise `docs/roadmap/api.md` so it no longer presents implemented behavior as
  future work. Either mark the pipeline-scoped upload API as implemented and
  leave only deferred ideas, or remove implemented sections.

Acceptance:

- `rg -n "POST /upload|source.token_env|token_env.*http_upload|maps each resolved bearer token to exactly one" README.md docs examples`
  returns no stale current-behavior references outside historical roadmap
  context.
- Documentation outside `docs/roadmap/` describes only implemented behavior.

## Final Verification

Run these commands after all stages are implemented:

```sh
go test ./internal/config
go test ./internal/app
go test ./pkg/bundle ./pkg/upload
go test ./...
```

Also run:

```sh
rg -n "POST /upload|source.token_env|/v1/pipelines|upload_tokens|PipelineID" README.md docs examples internal pkg
```

Review the output for stale references, missing docs, and tests that still
expect legacy upload routing.

## Non-Goals

- Do not let producers specify destination ids, destination paths, transforms,
  links, publish policy, transfer policy, or storage backends through upload
  requests.
- Do not add durable upload status, durable idempotency, retry endpoints,
  cancellation endpoints, long polling, or `UploadAndWait`.
- Do not add in-app TLS, public exposure policy, or rate limiting.
- Do not introduce new public package families beyond existing `pkg/bundle` and
  `pkg/upload`.