Clean up upload API documentation roadmap
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
This commit is contained in:
@@ -2,34 +2,24 @@
|
||||
|
||||
This document records API work that is not part of the current implementation. Current HTTP upload behavior is documented in `docs/integrations/http-upload.md`, current configuration behavior is documented in `docs/config.md`, and current producer package usage is documented under `docs/consumers/`.
|
||||
|
||||
## Implemented HTTP Upload Behavior
|
||||
## Deferred Upload API Work
|
||||
|
||||
The pipeline-scoped HTTP upload API is implemented. The current contract is:
|
||||
|
||||
- upload authentication is configured with top-level `upload_tokens`;
|
||||
- token values resolve through the process environment or `secrets.directory`, never YAML literals;
|
||||
- bearer tokens authenticate producers and authorize configured upload pipelines through `allow_pipelines`;
|
||||
- upload requests use `POST /v1/pipelines/{pipeline_id}/upload`;
|
||||
- missing, malformed, or unknown bearer tokens return `401 Unauthorized`;
|
||||
- valid tokens that are not allowed for the requested pipeline return `403 Forbidden`;
|
||||
- `pkg/upload` upload options require `PipelineID`;
|
||||
- idempotency records are scoped by token id, pipeline id, and idempotency key;
|
||||
- source manifests remain free of routing, destination, transform, and credential data.
|
||||
|
||||
The unscoped upload route no longer accepts uploads. Pipeline query parameters are rejected.
|
||||
|
||||
## Deferred API Work
|
||||
|
||||
The following topics remain separate roadmap items:
|
||||
|
||||
- durable upload status storage;
|
||||
- durable idempotency records across server restarts;
|
||||
- in-app public exposure policy;
|
||||
- built-in TLS termination;
|
||||
- built-in upload rate limiting.
|
||||
- Durable upload status storage.
|
||||
- Durable idempotency records across server restarts.
|
||||
- Run listing, retry, and cancellation endpoints.
|
||||
- Long-polling or wait-for-completion workflows.
|
||||
- Multipart, resumable, or streaming upload protocols.
|
||||
- Additional archive content negotiation beyond tar and gzip-compressed tar.
|
||||
- URL-token authentication for constrained clients.
|
||||
- Upload token lifecycle tooling.
|
||||
- Mutual TLS or other in-app identity mechanisms.
|
||||
- In-app TLS termination.
|
||||
- In-app public exposure policy.
|
||||
- In-app upload rate limiting.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Do not add destination selection to producer manifests.
|
||||
- Do not let producers specify destination ids, transforms, links, publish policy, or transfer policy through the upload API.
|
||||
- Keep `http_upload` source-only. A selected pipeline owns destination configuration, transforms, links, transfer policy, and publication behavior.
|
||||
- Producers do not choose destination ids, destination paths, transforms, links, publish policy, transfer policy, or storage backends through upload requests.
|
||||
- Source manifests remain free of routing, destination, transform, and credential data.
|
||||
- `http_upload` remains source-only unless a future design changes that contract.
|
||||
- Public access policy, TLS termination, and rate limiting belong in deployment infrastructure unless a future design changes that boundary.
|
||||
|
||||
@@ -1,275 +1,22 @@
|
||||
# Pipeline-Scoped Upload API Implementation Plan
|
||||
# Upload API Implementation Notes
|
||||
|
||||
This roadmap is for an LLM coding agent implementing the planned API work in
|
||||
`docs/roadmap/api.md`. It describes future work only. Do not update
|
||||
current-behavior docs outside `docs/roadmap/` until the corresponding stage is
|
||||
implemented.
|
||||
This file has no active implementation tasks for the pipeline-scoped upload API.
|
||||
|
||||
Before implementation, read:
|
||||
Current behavior is documented in:
|
||||
|
||||
- `docs/policy/architecture.md`
|
||||
- `docs/policy/development.md`
|
||||
- `docs/policy/documentation.md`
|
||||
- `docs/roadmap/api.md`
|
||||
- `docs/config.md`
|
||||
- `docs/integrations/http-upload.md`
|
||||
- `docs/consumers/api.md`
|
||||
- `docs/consumers/pkg-upload.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/troubleshooting.md`
|
||||
|
||||
## Target Behavior
|
||||
Deferred API work is tracked in `docs/roadmap/api.md` and broader deferred work is tracked in `docs/roadmap/future.md`.
|
||||
|
||||
The final implementation is a breaking v1 HTTP upload API change:
|
||||
## Boundaries
|
||||
|
||||
- Upload authentication is configured with top-level `upload_tokens`.
|
||||
- Pipeline routing is selected by `POST /v1/pipelines/{pipeline_id}/upload`.
|
||||
- Producers must set `PipelineID` in `pkg/upload` upload options.
|
||||
- Legacy per-source `source.token_env` is removed.
|
||||
- Legacy `POST /upload` no longer accepts uploads.
|
||||
- Idempotency is scoped by token id, pipeline id, and idempotency key.
|
||||
- `/healthz` and `/runs/<run-id>` continue to work.
|
||||
|
||||
The producer contract is:
|
||||
|
||||
- token authenticates the producer/client;
|
||||
- `PipelineID` selects the configured distributor workflow;
|
||||
- source manifest `id` identifies the logical artifact within that workflow;
|
||||
- idempotency key identifies one producer run and retry group.
|
||||
|
||||
## Stage 1: Config Model And Validation
|
||||
|
||||
Goal: make the YAML schema express upload authentication separately from
|
||||
pipeline source configuration.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Add `UploadTokens []UploadToken` to `internal/config.Config` with YAML key
|
||||
`upload_tokens`.
|
||||
- Add `UploadToken` with fields:
|
||||
- `ID string` as `id`;
|
||||
- `TokenEnv string` as `token_env`;
|
||||
- `AllowPipelines []string` as `allow_pipelines`.
|
||||
- Remove `TokenEnv` from `HTTPUpload`; keep `StagingPath` and
|
||||
`MaxUploadSize`.
|
||||
- Keep `http_upload` source defaults for `staging_path` and `max_upload_size`.
|
||||
- Update validation:
|
||||
- `upload_tokens` is required when any pipeline uses
|
||||
`source.backend: http_upload`.
|
||||
- `upload_tokens` is invalid when it references no configured upload
|
||||
pipelines.
|
||||
- token `id` is required, slug-like, and unique.
|
||||
- token `token_env` is required.
|
||||
- token `allow_pipelines` is required.
|
||||
- each token's `allow_pipelines` entries are unique.
|
||||
- each allowed pipeline id exists and names a pipeline whose source backend
|
||||
is `http_upload`.
|
||||
- every configured `http_upload` pipeline is allowed by at least one token.
|
||||
- `http_upload` sources no longer require `token_env`.
|
||||
- Remove tests and fixtures that expect `source.token_env`.
|
||||
- Add config load/validation tests for valid multi-pipeline tokens, multiple
|
||||
tokens for one pipeline, missing token list, duplicate token ids, duplicate
|
||||
allowlist entries, unknown allowed pipeline id, non-upload allowed pipeline
|
||||
id, and upload pipeline not allowed by any token.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `go test ./internal/config` passes.
|
||||
- YAML containing `source.token_env` fails as an unknown field.
|
||||
- YAML using top-level `upload_tokens` and `http_upload` sources without
|
||||
`source.token_env` loads and validates.
|
||||
|
||||
## Stage 2: Upload Token Resolution And HTTP Routing
|
||||
|
||||
Goal: authenticate by bearer token, authorize by token allowlist, and route by
|
||||
URL pipeline id.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Replace the current token-to-single-pipeline map with resolved upload token
|
||||
records containing:
|
||||
- token id;
|
||||
- resolved token value;
|
||||
- allowed pipeline id set.
|
||||
- Resolve token values through the existing config environment resolver so
|
||||
process environment and `secrets.directory` behavior remains consistent.
|
||||
- Startup must fail when a token env is missing, empty, or resolves to the same
|
||||
token value as another upload token. Error messages may name token ids and
|
||||
env var names, but must not print token values.
|
||||
- Change `uploadHTTPHandler` routes:
|
||||
- keep `GET /healthz`;
|
||||
- keep `GET /runs/<run-id>`;
|
||||
- add `POST /v1/pipelines/{pipeline_id}/upload`;
|
||||
- remove legacy `POST /upload`.
|
||||
- Path parsing rules:
|
||||
- match exactly `/v1/pipelines/<pipeline-id>/upload`;
|
||||
- reject missing pipeline id, extra path segments, and query-based
|
||||
`pipeline` or `pipeline_id` routing;
|
||||
- validate requested pipeline id using the same slug-like id policy used for
|
||||
configured pipeline ids.
|
||||
- Request handling rules:
|
||||
- missing, malformed, or unknown bearer token returns `401`;
|
||||
- valid token not allowed for requested pipeline returns `403`;
|
||||
- requested pipeline must be configured with `source.backend: http_upload`;
|
||||
- content type and idempotency key validation remain unchanged;
|
||||
- successful requests submit the requested pipeline id and token id to the
|
||||
upload coordinator.
|
||||
- Update HTTP handler tests for accepted v1 upload, unauthorized upload,
|
||||
forbidden pipeline, invalid pipeline path, removed `/upload`, invalid
|
||||
content type, invalid idempotency key, and no token leakage.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `go test ./internal/app` passes for handler tests touched in this stage.
|
||||
- `POST /upload` returns not found or another non-accepting error and does not
|
||||
call `Submit`.
|
||||
- `POST /v1/pipelines/{pipeline_id}/upload` routes only when token
|
||||
authorization allows that pipeline.
|
||||
|
||||
## Stage 3: Coordinator Idempotency Scope
|
||||
|
||||
Goal: prevent idempotency collisions between distinct authorized clients and
|
||||
between pipelines.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Add `TokenID string` to `UploadRequest`.
|
||||
- Update idempotency scope to include token id, pipeline id, and key.
|
||||
- Preserve existing behavior inside one idempotency scope:
|
||||
- same key and same normalized manifest returns the original accepted run;
|
||||
- same key and different normalized manifest returns conflict;
|
||||
- same key while staging returns retryable conflict.
|
||||
- Keep requests without idempotency keys unscoped and always admitted according
|
||||
to queue capacity.
|
||||
- Update coordinator tests:
|
||||
- same token, same pipeline, same key returns original run;
|
||||
- same token, same pipeline, same key with changed manifest conflicts;
|
||||
- different token ids can use the same key for the same pipeline without
|
||||
collision;
|
||||
- same token id can use the same key for different pipelines without
|
||||
collision;
|
||||
- pending-key retryable conflict still applies only within the same token and
|
||||
pipeline scope.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `go test ./internal/app` passes.
|
||||
- Existing queueing, retention, status, and same-pipeline serialization behavior
|
||||
remains unchanged.
|
||||
|
||||
## Stage 4: Public `pkg/upload` API
|
||||
|
||||
Goal: make producer clients route uploads through the v1 pipeline-scoped API.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Add `PipelineID string` to `UploadFilesOptions`.
|
||||
- Add `PipelineID string` to `UploadBundleOptions`.
|
||||
- Require non-empty valid `PipelineID` in both upload methods before local
|
||||
bundle staging, validation, archiving, or HTTP requests.
|
||||
- Use the same accepted pipeline id syntax as server config ids: starts with an
|
||||
ASCII letter or digit and then contains ASCII letters, digits, `.`, `_`, or
|
||||
`-`.
|
||||
- Build upload URLs as
|
||||
`/v1/pipelines/{pipeline_id}/upload`.
|
||||
- Keep `Status(ctx, runID)` unchanged.
|
||||
- Update package tests:
|
||||
- missing `PipelineID` fails before local file work or HTTP request;
|
||||
- invalid `PipelineID` fails before HTTP request;
|
||||
- `UploadFiles` and `UploadBundle` post to the v1 route;
|
||||
- retry behavior reuses the same idempotency key and v1 route;
|
||||
- token redaction still works.
|
||||
- Update `pkg/upload` package documentation to explain the four-part producer
|
||||
contract: token, `PipelineID`, manifest `ID`, idempotency key.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `go test ./pkg/upload ./pkg/bundle` passes.
|
||||
- Existing `Status` tests pass without endpoint changes.
|
||||
|
||||
## Stage 5: Examples And End-To-End App Coverage
|
||||
|
||||
Goal: prove the new server and client API work together across realistic upload
|
||||
pipelines.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Update `examples/http-upload-local.yml` to use top-level `upload_tokens`.
|
||||
- Update `examples/upload-client` to require or default a pipeline id and pass
|
||||
it to `pkg/upload`.
|
||||
- Update app integration tests:
|
||||
- one token authorized for two upload pipelines can upload to both;
|
||||
- two tokens authorized for one upload pipeline can upload to that pipeline;
|
||||
- a valid token rejected for a disallowed pipeline returns `403`;
|
||||
- removed `/upload` endpoint does not enqueue work;
|
||||
- full upload publishes through the selected pipeline and records the
|
||||
selected pipeline id in status/report output.
|
||||
- Update helper config builders in tests to use `upload_tokens`.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `go test ./internal/app` passes.
|
||||
- Example config loading tests pass.
|
||||
- `go run ./examples/upload-client` remains documented as an example that
|
||||
targets a configured pipeline.
|
||||
|
||||
## Stage 6: Implemented-Behavior Documentation
|
||||
|
||||
Goal: move the feature from roadmap-only language into current-behavior docs
|
||||
after the code is implemented.
|
||||
|
||||
Implementation:
|
||||
|
||||
- Update `docs/config.md`:
|
||||
- document top-level `upload_tokens`;
|
||||
- remove `source.token_env`;
|
||||
- document `http_upload` source fields `staging_path` and
|
||||
`max_upload_size`;
|
||||
- document token allowlist semantics.
|
||||
- Update `docs/integrations/http-upload.md`:
|
||||
- replace `POST /upload` with
|
||||
`POST /v1/pipelines/{pipeline_id}/upload`;
|
||||
- document `401` versus `403`;
|
||||
- document idempotency scope by token id and pipeline id.
|
||||
- Update `docs/consumers/api.md` and `docs/consumers/pkg-upload.md`:
|
||||
- show `PipelineID` in `UploadFiles` and `UploadBundle`;
|
||||
- explain token authentication versus pipeline routing.
|
||||
- Update `docs/operations.md` and `docs/troubleshooting.md` for the new curl
|
||||
path, upload token config, and forbidden-pipeline diagnosis.
|
||||
- Update `README.md` only if its producer integration links or summary become
|
||||
stale.
|
||||
- Revise `docs/roadmap/api.md` so it no longer presents implemented behavior as
|
||||
future work. Either mark the pipeline-scoped upload API as implemented and
|
||||
leave only deferred ideas, or remove implemented sections.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- `rg -n "POST /upload|source.token_env|token_env.*http_upload|maps each resolved bearer token to exactly one" README.md docs examples`
|
||||
returns no stale current-behavior references outside historical roadmap
|
||||
context.
|
||||
- Documentation outside `docs/roadmap/` describes only implemented behavior.
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run these commands after all stages are implemented:
|
||||
|
||||
```sh
|
||||
go test ./internal/config
|
||||
go test ./internal/app
|
||||
go test ./pkg/bundle ./pkg/upload
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Also run:
|
||||
|
||||
```sh
|
||||
rg -n "POST /upload|source.token_env|/v1/pipelines|upload_tokens|PipelineID" README.md docs examples internal pkg
|
||||
```
|
||||
|
||||
Review the output for stale references, missing docs, and tests that still
|
||||
expect legacy upload routing.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not let producers specify destination ids, destination paths, transforms,
|
||||
links, publish policy, transfer policy, or storage backends through upload
|
||||
requests.
|
||||
- Do not add durable upload status, durable idempotency, retry endpoints,
|
||||
cancellation endpoints, long polling, or `UploadAndWait`.
|
||||
- Do not add in-app TLS, public exposure policy, or rate limiting.
|
||||
- Do not introduce new public package families beyond existing `pkg/bundle` and
|
||||
`pkg/upload`.
|
||||
- Keep producer routing, destination selection, transform policy, publish policy, transfer policy, and backend credentials out of source manifests.
|
||||
- Keep `pkg/upload` focused on producer-side bundle submission and status polling.
|
||||
- Keep server-side upload authentication, authorization, queueing, status, and publish orchestration in `internal/app`.
|
||||
- Keep archive extraction and staged bundle validation in `internal/ingest`.
|
||||
- Keep durable status, durable idempotency, retry endpoints, cancellation endpoints, and wait helpers out of the current implementation until a new roadmap item defines them.
|
||||
|
||||
Reference in New Issue
Block a user