276 lines
11 KiB
Markdown
276 lines
11 KiB
Markdown
# Pipeline-Scoped Upload API Implementation Plan
|
|
|
|
This roadmap is for an LLM coding agent implementing the planned API work in
|
|
`docs/roadmap/api.md`. It describes future work only. Do not update
|
|
current-behavior docs outside `docs/roadmap/` until the corresponding stage is
|
|
implemented.
|
|
|
|
Before implementation, read:
|
|
|
|
- `docs/policy/architecture.md`
|
|
- `docs/policy/development.md`
|
|
- `docs/policy/documentation.md`
|
|
- `docs/roadmap/api.md`
|
|
|
|
## Target Behavior
|
|
|
|
The final implementation is a breaking v1 HTTP upload API change:
|
|
|
|
- Upload authentication is configured with top-level `upload_tokens`.
|
|
- Pipeline routing is selected by `POST /v1/pipelines/{pipeline_id}/upload`.
|
|
- Producers must set `PipelineID` in `pkg/upload` upload options.
|
|
- Legacy per-source `source.token_env` is removed.
|
|
- Legacy `POST /upload` no longer accepts uploads.
|
|
- Idempotency is scoped by token id, pipeline id, and idempotency key.
|
|
- `/healthz` and `/runs/<run-id>` continue to work.
|
|
|
|
The producer contract is:
|
|
|
|
- token authenticates the producer/client;
|
|
- `PipelineID` selects the configured distributor workflow;
|
|
- source manifest `id` identifies the logical artifact within that workflow;
|
|
- idempotency key identifies one producer run and retry group.
|
|
|
|
## Stage 1: Config Model And Validation
|
|
|
|
Goal: make the YAML schema express upload authentication separately from
|
|
pipeline source configuration.
|
|
|
|
Implementation:
|
|
|
|
- Add `UploadTokens []UploadToken` to `internal/config.Config` with YAML key
|
|
`upload_tokens`.
|
|
- Add `UploadToken` with fields:
|
|
- `ID string` as `id`;
|
|
- `TokenEnv string` as `token_env`;
|
|
- `AllowPipelines []string` as `allow_pipelines`.
|
|
- Remove `TokenEnv` from `HTTPUpload`; keep `StagingPath` and
|
|
`MaxUploadSize`.
|
|
- Keep `http_upload` source defaults for `staging_path` and `max_upload_size`.
|
|
- Update validation:
|
|
- `upload_tokens` is required when any pipeline uses
|
|
`source.backend: http_upload`.
|
|
- `upload_tokens` is invalid when it references no configured upload
|
|
pipelines.
|
|
- token `id` is required, slug-like, and unique.
|
|
- token `token_env` is required.
|
|
- token `allow_pipelines` is required.
|
|
- each token's `allow_pipelines` entries are unique.
|
|
- each allowed pipeline id exists and names a pipeline whose source backend
|
|
is `http_upload`.
|
|
- every configured `http_upload` pipeline is allowed by at least one token.
|
|
- `http_upload` sources no longer require `token_env`.
|
|
- Remove tests and fixtures that expect `source.token_env`.
|
|
- Add config load/validation tests for valid multi-pipeline tokens, multiple
|
|
tokens for one pipeline, missing token list, duplicate token ids, duplicate
|
|
allowlist entries, unknown allowed pipeline id, non-upload allowed pipeline
|
|
id, and upload pipeline not allowed by any token.
|
|
|
|
Acceptance:
|
|
|
|
- `go test ./internal/config` passes.
|
|
- YAML containing `source.token_env` fails as an unknown field.
|
|
- YAML using top-level `upload_tokens` and `http_upload` sources without
|
|
`source.token_env` loads and validates.
|
|
|
|
## Stage 2: Upload Token Resolution And HTTP Routing
|
|
|
|
Goal: authenticate by bearer token, authorize by token allowlist, and route by
|
|
URL pipeline id.
|
|
|
|
Implementation:
|
|
|
|
- Replace the current token-to-single-pipeline map with resolved upload token
|
|
records containing:
|
|
- token id;
|
|
- resolved token value;
|
|
- allowed pipeline id set.
|
|
- Resolve token values through the existing config environment resolver so
|
|
process environment and `secrets.directory` behavior remains consistent.
|
|
- Startup must fail when a token env is missing, empty, or resolves to the same
|
|
token value as another upload token. Error messages may name token ids and
|
|
env var names, but must not print token values.
|
|
- Change `uploadHTTPHandler` routes:
|
|
- keep `GET /healthz`;
|
|
- keep `GET /runs/<run-id>`;
|
|
- add `POST /v1/pipelines/{pipeline_id}/upload`;
|
|
- remove legacy `POST /upload`.
|
|
- Path parsing rules:
|
|
- match exactly `/v1/pipelines/<pipeline-id>/upload`;
|
|
- reject missing pipeline id, extra path segments, and query-based
|
|
`pipeline` or `pipeline_id` routing;
|
|
- validate requested pipeline id using the same slug-like id policy used for
|
|
configured pipeline ids.
|
|
- Request handling rules:
|
|
- missing, malformed, or unknown bearer token returns `401`;
|
|
- valid token not allowed for requested pipeline returns `403`;
|
|
- requested pipeline must be configured with `source.backend: http_upload`;
|
|
- content type and idempotency key validation remain unchanged;
|
|
- successful requests submit the requested pipeline id and token id to the
|
|
upload coordinator.
|
|
- Update HTTP handler tests for accepted v1 upload, unauthorized upload,
|
|
forbidden pipeline, invalid pipeline path, removed `/upload`, invalid
|
|
content type, invalid idempotency key, and no token leakage.
|
|
|
|
Acceptance:
|
|
|
|
- `go test ./internal/app` passes for handler tests touched in this stage.
|
|
- `POST /upload` returns not found or another non-accepting error and does not
|
|
call `Submit`.
|
|
- `POST /v1/pipelines/{pipeline_id}/upload` routes only when token
|
|
authorization allows that pipeline.
|
|
|
|
## Stage 3: Coordinator Idempotency Scope
|
|
|
|
Goal: prevent idempotency collisions between distinct authorized clients and
|
|
between pipelines.
|
|
|
|
Implementation:
|
|
|
|
- Add `TokenID string` to `UploadRequest`.
|
|
- Update idempotency scope to include token id, pipeline id, and key.
|
|
- Preserve existing behavior inside one idempotency scope:
|
|
- same key and same normalized manifest returns the original accepted run;
|
|
- same key and different normalized manifest returns conflict;
|
|
- same key while staging returns retryable conflict.
|
|
- Keep requests without idempotency keys unscoped and always admitted according
|
|
to queue capacity.
|
|
- Update coordinator tests:
|
|
- same token, same pipeline, same key returns original run;
|
|
- same token, same pipeline, same key with changed manifest conflicts;
|
|
- different token ids can use the same key for the same pipeline without
|
|
collision;
|
|
- same token id can use the same key for different pipelines without
|
|
collision;
|
|
- pending-key retryable conflict still applies only within the same token and
|
|
pipeline scope.
|
|
|
|
Acceptance:
|
|
|
|
- `go test ./internal/app` passes.
|
|
- Existing queueing, retention, status, and same-pipeline serialization behavior
|
|
remains unchanged.
|
|
|
|
## Stage 4: Public `pkg/upload` API
|
|
|
|
Goal: make producer clients route uploads through the v1 pipeline-scoped API.
|
|
|
|
Implementation:
|
|
|
|
- Add `PipelineID string` to `UploadFilesOptions`.
|
|
- Add `PipelineID string` to `UploadBundleOptions`.
|
|
- Require non-empty valid `PipelineID` in both upload methods before local
|
|
bundle staging, validation, archiving, or HTTP requests.
|
|
- Use the same accepted pipeline id syntax as server config ids: starts with an
|
|
ASCII letter or digit and then contains ASCII letters, digits, `.`, `_`, or
|
|
`-`.
|
|
- Build upload URLs as
|
|
`/v1/pipelines/{pipeline_id}/upload`.
|
|
- Keep `Status(ctx, runID)` unchanged.
|
|
- Update package tests:
|
|
- missing `PipelineID` fails before local file work or HTTP request;
|
|
- invalid `PipelineID` fails before HTTP request;
|
|
- `UploadFiles` and `UploadBundle` post to the v1 route;
|
|
- retry behavior reuses the same idempotency key and v1 route;
|
|
- token redaction still works.
|
|
- Update `pkg/upload` package documentation to explain the four-part producer
|
|
contract: token, `PipelineID`, manifest `ID`, idempotency key.
|
|
|
|
Acceptance:
|
|
|
|
- `go test ./pkg/upload ./pkg/bundle` passes.
|
|
- Existing `Status` tests pass without endpoint changes.
|
|
|
|
## Stage 5: Examples And End-To-End App Coverage
|
|
|
|
Goal: prove the new server and client API work together across realistic upload
|
|
pipelines.
|
|
|
|
Implementation:
|
|
|
|
- Update `examples/http-upload-local.yml` to use top-level `upload_tokens`.
|
|
- Update `examples/upload-client` to require or default a pipeline id and pass
|
|
it to `pkg/upload`.
|
|
- Update app integration tests:
|
|
- one token authorized for two upload pipelines can upload to both;
|
|
- two tokens authorized for one upload pipeline can upload to that pipeline;
|
|
- a valid token rejected for a disallowed pipeline returns `403`;
|
|
- removed `/upload` endpoint does not enqueue work;
|
|
- full upload publishes through the selected pipeline and records the
|
|
selected pipeline id in status/report output.
|
|
- Update helper config builders in tests to use `upload_tokens`.
|
|
|
|
Acceptance:
|
|
|
|
- `go test ./internal/app` passes.
|
|
- Example config loading tests pass.
|
|
- `go run ./examples/upload-client` remains documented as an example that
|
|
targets a configured pipeline.
|
|
|
|
## Stage 6: Implemented-Behavior Documentation
|
|
|
|
Goal: move the feature from roadmap-only language into current-behavior docs
|
|
after the code is implemented.
|
|
|
|
Implementation:
|
|
|
|
- Update `docs/config.md`:
|
|
- document top-level `upload_tokens`;
|
|
- remove `source.token_env`;
|
|
- document `http_upload` source fields `staging_path` and
|
|
`max_upload_size`;
|
|
- document token allowlist semantics.
|
|
- Update `docs/integrations/http-upload.md`:
|
|
- replace `POST /upload` with
|
|
`POST /v1/pipelines/{pipeline_id}/upload`;
|
|
- document `401` versus `403`;
|
|
- document idempotency scope by token id and pipeline id.
|
|
- Update `docs/consumers/api.md` and `docs/consumers/pkg-upload.md`:
|
|
- show `PipelineID` in `UploadFiles` and `UploadBundle`;
|
|
- explain token authentication versus pipeline routing.
|
|
- Update `docs/operations.md` and `docs/troubleshooting.md` for the new curl
|
|
path, upload token config, and forbidden-pipeline diagnosis.
|
|
- Update `README.md` only if its producer integration links or summary become
|
|
stale.
|
|
- Revise `docs/roadmap/api.md` so it no longer presents implemented behavior as
|
|
future work. Either mark the pipeline-scoped upload API as implemented and
|
|
leave only deferred ideas, or remove implemented sections.
|
|
|
|
Acceptance:
|
|
|
|
- `rg -n "POST /upload|source.token_env|token_env.*http_upload|maps each resolved bearer token to exactly one" README.md docs examples`
|
|
returns no stale current-behavior references outside historical roadmap
|
|
context.
|
|
- Documentation outside `docs/roadmap/` describes only implemented behavior.
|
|
|
|
## Final Verification
|
|
|
|
Run these commands after all stages are implemented:
|
|
|
|
```sh
|
|
go test ./internal/config
|
|
go test ./internal/app
|
|
go test ./pkg/bundle ./pkg/upload
|
|
go test ./...
|
|
```
|
|
|
|
Also run:
|
|
|
|
```sh
|
|
rg -n "POST /upload|source.token_env|/v1/pipelines|upload_tokens|PipelineID" README.md docs examples internal pkg
|
|
```
|
|
|
|
Review the output for stale references, missing docs, and tests that still
|
|
expect legacy upload routing.
|
|
|
|
## Non-Goals
|
|
|
|
- Do not let producers specify destination ids, destination paths, transforms,
|
|
links, publish policy, transfer policy, or storage backends through upload
|
|
requests.
|
|
- Do not add durable upload status, durable idempotency, retry endpoints,
|
|
cancellation endpoints, long polling, or `UploadAndWait`.
|
|
- Do not add in-app TLS, public exposure policy, or rate limiting.
|
|
- Do not introduce new public package families beyond existing `pkg/bundle` and
|
|
`pkg/upload`.
|