Added a roadmap and an implentation plan for a new public http upload helper package
This commit is contained in:
211
docs/roadmap/implementation.md
Normal file
211
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,211 @@
|
||||
# Producer HTTP Upload Package Implementation Roadmap
|
||||
|
||||
## Current Baseline
|
||||
|
||||
`distributor serve` and the `http_upload` source backend are implemented.
|
||||
Producers can already submit complete tar or gzip-compressed tar source bundles
|
||||
to `POST /upload` with bearer authentication, and each bearer token maps to one
|
||||
configured upload pipeline.
|
||||
|
||||
The public `pkg/bundle` package already provides producer-side source manifest
|
||||
semantics, digest calculation, path validation, local manifest building, local
|
||||
bundle writing, and local bundle validation helpers.
|
||||
|
||||
The current implementation does not have a server-side producer idempotency
|
||||
contract, and it does not provide a public `pkg/upload` helper package.
|
||||
|
||||
Future behavior remains under `docs/roadmap/` until implemented. Do not update
|
||||
README, current user docs, examples, or current-behavior internal docs until the
|
||||
corresponding stage has been implemented.
|
||||
|
||||
This active roadmap implements the accepted producer upload package plan in
|
||||
`docs/roadmap/producer.md`. It supersedes the older `docs/roadmap/http.md`
|
||||
deferred note for producer-supplied idempotency keys; producer idempotency is
|
||||
now active roadmap work.
|
||||
|
||||
## Active Roadmap
|
||||
|
||||
## Stage 1: Server-Side Upload Idempotency
|
||||
|
||||
Goal:
|
||||
|
||||
Add `Idempotency-Key` support to `POST /upload` so safe producer retries do not
|
||||
create duplicate accepted runs.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Validate optional `Idempotency-Key` headers using the syntax defined in
|
||||
`docs/roadmap/producer.md`.
|
||||
- Scope keys by the authenticated pipeline selected through bearer-token
|
||||
mapping.
|
||||
- Record accepted keys after archive staging and source bundle validation
|
||||
succeed.
|
||||
- Compare normalized source manifest identity, not raw archive bytes.
|
||||
- Return the original accepted run response for the same pipeline, same key,
|
||||
and same manifest identity.
|
||||
- Return `409 Conflict` for the same pipeline and key with a different manifest
|
||||
identity.
|
||||
- Return a retryable conflict response when the same key is already being
|
||||
processed concurrently for the same pipeline before manifest identity is
|
||||
known.
|
||||
- Expire idempotency records with existing upload status retention.
|
||||
- Keep idempotency records memory-only; server restart clears them.
|
||||
- Preserve current raw HTTP behavior when no idempotency key is supplied.
|
||||
|
||||
Current-behavior documentation updates after implementation:
|
||||
|
||||
- `docs/integrations/http-upload.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/app.md`
|
||||
- `docs/troubleshooting.md`
|
||||
|
||||
Tests:
|
||||
|
||||
- Same key and same bundle returns the original run id and does not enqueue a
|
||||
second run.
|
||||
- Same key and different bundle returns `409`.
|
||||
- Same key under different authenticated pipelines does not conflict.
|
||||
- Missing key preserves current raw HTTP behavior.
|
||||
- Invalid key syntax returns `400`.
|
||||
- Expiration removes idempotency records.
|
||||
- Tokens are never leaked; idempotency keys appear only where needed for
|
||||
diagnostics.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Existing upload clients continue to work.
|
||||
- The HTTP API has an implemented, tested idempotency contract.
|
||||
- Idempotent retries cannot create duplicate accepted runs.
|
||||
|
||||
## Stage 2: Public `pkg/upload` API And Client
|
||||
|
||||
Goal:
|
||||
|
||||
Add a producer-facing upload package that builds or validates bundles, archives
|
||||
them, and submits them to the HTTP upload API.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add public `pkg/upload`.
|
||||
- Use `pkg/bundle` for manifest generation, digest and path semantics, local
|
||||
validation, and temporary bundle creation.
|
||||
- Implement options-struct APIs matching `docs/roadmap/producer.md`:
|
||||
`ClientOptions`, `RetryOptions`, `UploadBundleOptions`,
|
||||
`UploadFilesOptions`, `Result`, `RunStatus`, `NewClient`, `UploadBundle`,
|
||||
`UploadFiles`, and `Status`.
|
||||
- Treat `Endpoint` as the distributor server base URL, deriving `/upload` and
|
||||
`/runs/<run-id>` internally.
|
||||
- Require bearer token authentication and redact token values from all errors.
|
||||
- Always send `Idempotency-Key`.
|
||||
- Use caller-supplied idempotency keys when provided.
|
||||
- When no key is supplied, generate one random 128-bit lowercase hex key per
|
||||
upload operation and reuse it across retries from that call.
|
||||
- Support uploading an existing local bundle root.
|
||||
- Support building a temporary bundle from explicit `bundle.BundleFile` values
|
||||
and uploading it.
|
||||
- Create replayable gzip-compressed tar uploads with
|
||||
`Content-Type: application/gzip`.
|
||||
- Retry only safe cases: `503 Service Unavailable`, temporary network errors,
|
||||
and ambiguous mid-upload failures, using the same idempotency key and
|
||||
replayable body.
|
||||
- Do not retry `400`, `401`, `409`, `413`, or `415`.
|
||||
- Do not retry after `202 Accepted`.
|
||||
- Respect context cancellation before waiting and before each retry.
|
||||
- Close response bodies on every attempt.
|
||||
|
||||
Current-behavior documentation updates after implementation:
|
||||
|
||||
- Update `pkg/bundle` integration references only as needed once `pkg/upload`
|
||||
exists.
|
||||
- Keep full user-facing docs and examples for Stage 3.
|
||||
|
||||
Tests:
|
||||
|
||||
- Client construction validates endpoint and token requirements.
|
||||
- Token values are redacted from errors.
|
||||
- Caller-supplied and generated idempotency keys are sent correctly.
|
||||
- Existing bundle upload includes only `manifest.json` and manifest-listed
|
||||
files.
|
||||
- File-based upload builds a compliant temporary bundle without touching
|
||||
producer source directories.
|
||||
- Local validation failures prevent HTTP requests.
|
||||
- Response parsing covers `202`, `400`, `401`, `409`, `413`, `415`, `503`,
|
||||
non-JSON errors, and unexpected statuses.
|
||||
- Retry uses the same idempotency key and stops correctly.
|
||||
- Context cancellation during retry backoff is honored.
|
||||
- Custom `*http.Client` behavior is covered with `httptest`.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Go producers can upload valid bundles through `pkg/upload`.
|
||||
- Safe retry behavior relies on the implemented server idempotency contract.
|
||||
- Public package tests prove upload behavior does not duplicate or drift from
|
||||
`pkg/bundle` semantics.
|
||||
|
||||
## Stage 3: Documentation And Examples
|
||||
|
||||
Goal:
|
||||
|
||||
Document implemented producer upload and idempotency behavior after the server
|
||||
contract and public package exist.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Update current-behavior docs only after Stages 1 and 2 are implemented.
|
||||
- Add secret-free examples where they are safe, copyable, and describe
|
||||
implemented behavior.
|
||||
- Keep deferred items out of current docs.
|
||||
|
||||
Docs to update:
|
||||
|
||||
- `README.md`
|
||||
- `docs/integrations/source-bundle.md`
|
||||
- `docs/integrations/http-upload.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/app.md`
|
||||
- `docs/policy/development.md`
|
||||
- `examples/`, only if examples are safe, copyable, and implemented
|
||||
|
||||
Tests and checks:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
rg -n "idempotency|Idempotency-Key|pkg/upload|UploadBundle|UploadFiles" README.md docs examples
|
||||
```
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Current docs describe the implemented server idempotency and `pkg/upload`
|
||||
API.
|
||||
- Completed behavior is not documented only as future work.
|
||||
- `docs/roadmap/` contains only future or deferred producer-upload work.
|
||||
|
||||
## Deferred Work
|
||||
|
||||
- Durable idempotency storage across server restarts.
|
||||
- Database-backed queues or durable producer retry processing.
|
||||
- `UploadAndWait`, long polling, run cancellation, run retry, or run listing
|
||||
helpers.
|
||||
- Zstandard archives.
|
||||
- Multipart, resumable, or streaming upload protocols.
|
||||
- URL-token authentication.
|
||||
- Browser UI or public exposure defaults.
|
||||
|
||||
## Validation
|
||||
|
||||
For this documentation pass:
|
||||
|
||||
```sh
|
||||
rg -n "Idempotency-Key|pkg/upload|UploadBundle|UploadFiles|Stage 1: Server-Side Upload Idempotency" docs/roadmap/implementation.md
|
||||
rg -n "Producer-supplied idempotency keys|Producer Coordination" docs/roadmap/http.md docs/roadmap/implementation.md
|
||||
rg -n "pkg/upload|UploadBundle|UploadFiles|Idempotency-Key" README.md docs examples --glob '!docs/roadmap/**'
|
||||
git status --short
|
||||
git diff -- docs/roadmap/implementation.md
|
||||
```
|
||||
|
||||
Expected result:
|
||||
|
||||
- New future behavior appears only under `docs/roadmap/`.
|
||||
- Existing unrelated worktree changes, including any current
|
||||
`docs/roadmap/documentation.md` deletion, are not touched.
|
||||
- This pass changes only `docs/roadmap/implementation.md`.
|
||||
Reference in New Issue
Block a user