eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Document producer upload client behavior

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-04 14:15:27 +00:00
parent d637949db4
commit f9142fded4
10 changed files with 124 additions and 515 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -10,7 +10,7 @@ Run the maintained local example:
go run ./cmd/distributor run --config examples/local-publish.yml
```
Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate local source bundles with the same manifest contract used by the CLI. See [Source bundle contract](docs/integrations/source-bundle.md).
Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate local source bundles with the same manifest contract used by the CLI. They can use `gitea.maximumdirect.net/eric/distributor/pkg/upload` to build or validate a bundle and submit it to `distributor serve` with bearer authentication and idempotency keys. See [Source bundle contract](docs/integrations/source-bundle.md) and [HTTP upload contract](docs/integrations/http-upload.md).
- [CLI reference](docs/cli.md)
- [Configuration reference](docs/config.md)

24
docs/integrations/http-upload.md
View File

@@ -106,6 +106,28 @@ Archive entry rules:
The uploaded archive size and extracted bundle size are bounded by the selected pipeline's `source.max_upload_size`. Extracted file count is also bounded by the implementation.
## Go Producer Helper
Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/upload` to build or validate source bundles, package them as gzip-compressed tar archives, and submit them to this API:
```go
client, err := upload.NewClient(upload.ClientOptions{
Endpoint: "http://127.0.0.1:8080",
Token: token,
})
if err != nil {
return err
}
result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
Root: "examples/source-bundle",
IdempotencyKey: "reports.example.20260604T120000Z",
})
```
`Endpoint` is the server base URL; the package derives `/upload` and `/runs/<run-id>`. `UploadBundle` validates a local bundle by default and uploads only `manifest.json` plus manifest-listed files. `UploadFiles` creates a temporary bundle from explicit `bundle.BundleFile` values before uploading. When `IdempotencyKey` is omitted, the package generates one random 128-bit lowercase hex key for the upload operation and reuses it across retries.
The helper retries only safe cases: `503 Service Unavailable`, temporary network errors, and ambiguous mid-upload failures. It does not retry after `202 Accepted` and does not retry `400`, `401`, `409`, `413`, or `415`. Bearer token values are redacted from returned errors.
## Queue And Retention
`server.http.queue_size` bounds accepted-but-not-started uploads plus uploads being staged. `server.http.max_concurrency` bounds publishing concurrency. The coordinator does not run two uploads for the same pipeline concurrently.
@@ -123,5 +145,5 @@ The HTTP API does not expose pipeline selection by request parameter, TLS, publi
Before changing this contract, inspect and run:
```sh
go test ./internal/app ./internal/ingest
go test ./internal/app ./internal/ingest ./pkg/upload
```

4
docs/integrations/source-bundle.md
View File

@@ -68,6 +68,8 @@ Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to bu
- `LoadManifest`, `ParseManifest`, `ValidateManifest`, and `ValidateBundle`: parse and validate local bundles.
- `FileDigest`, `BundleDigest`, and `ValidateDigest`: digest helpers.
Go producers that submit bundles to `distributor serve` can use `gitea.maximumdirect.net/eric/distributor/pkg/upload`. It builds on `pkg/bundle`, packages valid bundles as gzip-compressed tar uploads, sends bearer authentication, and includes idempotency keys for safe retry behavior. See [HTTP Upload API Contract](http-upload.md).
CLI producers can use:
```sh
@@ -88,5 +90,5 @@ The source bundle manifest does not configure routing, destination selection, pu
Before changing this contract, inspect and run:
```sh
go test ./pkg/bundle ./internal/bundle
go test ./pkg/bundle ./pkg/upload ./internal/bundle
```

2
docs/internal/app.md
View File

@@ -34,7 +34,7 @@ The app layer registers default transforms, including Markdown-to-HTML, and supp
Run workflows discover and validate source bundles through `internal/bundle`. Destination state actions are prepared and written through `internal/publish` and `internal/state`; the app layer records report projections of those actions and results.
HTTP uploads stage and validate archives before enqueueing a pipeline run with a local staged source root.
HTTP uploads stage and validate archives before enqueueing a pipeline run with a local staged source root. Go producers can use the public `pkg/upload` package to create client-side gzip tar uploads for this server contract; `internal/app` remains the server-side orchestration boundary and does not import that producer package.
Upload idempotency is owned by the upload coordinator. Optional `Idempotency-Key` values are scoped to the authenticated pipeline. The coordinator reserves a key while staging is in progress, records the accepted run id with the validated source manifest identity after staging succeeds, returns the original accepted record for the same key and same manifest, and rejects the same key with a different manifest as a conflict.

19
docs/operations.md
View File

@@ -148,6 +148,25 @@ curl -X POST http://127.0.0.1:8080/upload \
--data-binary @bundle.tar.gz
```
Go producer applications can use `pkg/upload` instead of constructing archives and HTTP requests directly. The package sends `Idempotency-Key` on every upload, derives `/upload` from the configured endpoint, and reuses the same key and replayable request body for safe retries:
```go
client, err := upload.NewClient(upload.ClientOptions{
Endpoint: "http://127.0.0.1:8080",
Token: token,
})
result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
Root: "examples/source-bundle",
IdempotencyKey: "producer.run.20260604T120000Z",
})
```
The maintained example client uses the local upload server and reads the token from `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN`. It generates an idempotency key by default; set `DISTRIBUTOR_EXAMPLE_UPLOAD_IDEMPOTENCY_KEY` when a retry must be stable across separate process runs.
```sh
go run ./examples/upload-client
```
Accepted uploads return after the archive is staged and validated:
```json

10
docs/policy/development.md
View File

@@ -7,6 +7,7 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
- `cmd/distributor`: executable entrypoint only.
- `pkg/bundle`: public producer-facing source manifest and local bundle writer helpers.
- `pkg/upload`: public producer-facing HTTP upload client built on `pkg/bundle`.
- `internal/app`: top-level use cases for `run`, `validate`, and `inspect`.
- `internal/cli`: standard-library command parsing, flags, help text, and command wiring.
- `internal/config`: YAML configuration structs, loading, defaults, and validation.
@@ -26,9 +27,9 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
- `examples`: copyable example configs and source bundles.
Do not create new top-level package families such as public `pkg/...` packages
beyond `pkg/bundle`, generic workflow containers, or service-specific adapter
directories unless the architecture policy or a current roadmap explicitly
calls for them.
beyond `pkg/bundle` and `pkg/upload`, generic workflow containers, or
service-specific adapter directories unless the architecture policy or a
current roadmap explicitly calls for them.
## Common Commands
@@ -45,6 +46,7 @@ go test ./internal/config
go test ./internal/cli ./internal/app
go test ./internal/publish ./internal/state
go test ./internal/transform/markdown
go test ./pkg/bundle ./pkg/upload
```
Run the CLI against an example config:
@@ -76,6 +78,7 @@ GOCACHE=/private/tmp/distributor-gocache GOMODCACHE=/private/tmp/distributor-gom
- Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and implemented backend behavior unless the current task explicitly changes them.
- Use `storage.DisplayPath`, `storage.StateFileName`, `storage.StatePath`, and `storage.ManagedBundleTargets` instead of duplicating those conventions.
- Use `pkg/bundle` for normalized source manifest semantics. Internal packages should reach those rules through `internal/bundle` when they also need storage-backed bundle discovery or validation.
- Keep `pkg/upload` as a producer-facing HTTP client. It should depend on `pkg/bundle` and standard HTTP/archive primitives, not on `internal/app`, `internal/ingest`, server config, storage backends, or destination state types.
- Use `config.ValidatePublishTransformPolicy` for publish and transform policy combinations.
- Do not import concrete transform implementations from `internal/publish`; app-level wiring owns transform registration.
- Do not import `internal/testutil` from production code.
@@ -171,6 +174,7 @@ Test close to the behavior being changed:
- Use `internal/app` and `internal/cli` tests for user-facing workflows.
- Use `internal/testutil` for shared valid fixtures only; keep edge cases near the package under test.
- Run `go test ./...` after cross-package changes or documentation/example changes tied to tests.
- Run `go test ./pkg/bundle ./pkg/upload` after changing producer-facing bundle or upload APIs.
Live integration tests must be opt-in and skipped during normal `go test ./...`
unless their required environment variables are set. Test-only environment

1
docs/roadmap/http.md
View File

@@ -36,7 +36,6 @@ the current implementation.
### Producer Coordination
- Producer-supplied idempotency keys.
- Run retry endpoints.
- Run cancellation endpoints.
- Run listing endpoints.

209
docs/roadmap/implementation.md
View File

@@ -1,184 +1,18 @@
# Producer HTTP Upload Package Implementation Roadmap
# Producer Upload Implementation Roadmap
## Current Baseline
The producer upload implementation described by this roadmap is complete.
Current behavior is documented outside roadmap files:
`distributor serve` and the `http_upload` source backend are implemented.
Producers can already submit complete tar or gzip-compressed tar source bundles
to `POST /upload` with bearer authentication, and each bearer token maps to one
configured upload pipeline.
- [README](../../README.md)
- [Source bundle contract](../integrations/source-bundle.md)
- [HTTP upload contract](../integrations/http-upload.md)
- [Operations](../operations.md)
- [Troubleshooting](../troubleshooting.md)
- [Application internals](../internal/app.md)
- [Development policy](../policy/development.md)
The public `pkg/bundle` package already provides producer-side source manifest
semantics, digest calculation, path validation, local manifest building, local
bundle writing, and local bundle validation helpers.
The current implementation does not have a server-side producer idempotency
contract, and it does not provide a public `pkg/upload` helper package.
Future behavior remains under `docs/roadmap/` until implemented. Do not update
README, current user docs, examples, or current-behavior internal docs until the
corresponding stage has been implemented.
This active roadmap implements the accepted producer upload package plan in
`docs/roadmap/producer.md`. It supersedes the older `docs/roadmap/http.md`
deferred note for producer-supplied idempotency keys; producer idempotency is
now active roadmap work.
## Active Roadmap
## Stage 1: Server-Side Upload Idempotency
Goal:
Add `Idempotency-Key` support to `POST /upload` so safe producer retries do not
create duplicate accepted runs.
Implementation scope:
- Validate optional `Idempotency-Key` headers using the syntax defined in
`docs/roadmap/producer.md`.
- Scope keys by the authenticated pipeline selected through bearer-token
mapping.
- Record accepted keys after archive staging and source bundle validation
succeed.
- Compare normalized source manifest identity, not raw archive bytes.
- Return the original accepted run response for the same pipeline, same key,
and same manifest identity.
- Return `409 Conflict` for the same pipeline and key with a different manifest
identity.
- Return a retryable conflict response when the same key is already being
processed concurrently for the same pipeline before manifest identity is
known.
- Expire idempotency records with existing upload status retention.
- Keep idempotency records memory-only; server restart clears them.
- Preserve current raw HTTP behavior when no idempotency key is supplied.
Current-behavior documentation updates after implementation:
- `docs/integrations/http-upload.md`
- `docs/operations.md`
- `docs/internal/app.md`
- `docs/troubleshooting.md`
Tests:
- Same key and same bundle returns the original run id and does not enqueue a
second run.
- Same key and different bundle returns `409`.
- Same key under different authenticated pipelines does not conflict.
- Missing key preserves current raw HTTP behavior.
- Invalid key syntax returns `400`.
- Expiration removes idempotency records.
- Tokens are never leaked; idempotency keys appear only where needed for
diagnostics.
Completion criteria:
- Existing upload clients continue to work.
- The HTTP API has an implemented, tested idempotency contract.
- Idempotent retries cannot create duplicate accepted runs.
## Stage 2: Public `pkg/upload` API And Client
Goal:
Add a producer-facing upload package that builds or validates bundles, archives
them, and submits them to the HTTP upload API.
Implementation scope:
- Add public `pkg/upload`.
- Use `pkg/bundle` for manifest generation, digest and path semantics, local
validation, and temporary bundle creation.
- Implement options-struct APIs matching `docs/roadmap/producer.md`:
`ClientOptions`, `RetryOptions`, `UploadBundleOptions`,
`UploadFilesOptions`, `Result`, `RunStatus`, `NewClient`, `UploadBundle`,
`UploadFiles`, and `Status`.
- Treat `Endpoint` as the distributor server base URL, deriving `/upload` and
`/runs/<run-id>` internally.
- Require bearer token authentication and redact token values from all errors.
- Always send `Idempotency-Key`.
- Use caller-supplied idempotency keys when provided.
- When no key is supplied, generate one random 128-bit lowercase hex key per
upload operation and reuse it across retries from that call.
- Support uploading an existing local bundle root.
- Support building a temporary bundle from explicit `bundle.BundleFile` values
and uploading it.
- Create replayable gzip-compressed tar uploads with
`Content-Type: application/gzip`.
- Retry only safe cases: `503 Service Unavailable`, temporary network errors,
and ambiguous mid-upload failures, using the same idempotency key and
replayable body.
- Do not retry `400`, `401`, `409`, `413`, or `415`.
- Do not retry after `202 Accepted`.
- Respect context cancellation before waiting and before each retry.
- Close response bodies on every attempt.
Current-behavior documentation updates after implementation:
- Update `pkg/bundle` integration references only as needed once `pkg/upload`
exists.
- Keep full user-facing docs and examples for Stage 3.
Tests:
- Client construction validates endpoint and token requirements.
- Token values are redacted from errors.
- Caller-supplied and generated idempotency keys are sent correctly.
- Existing bundle upload includes only `manifest.json` and manifest-listed
files.
- File-based upload builds a compliant temporary bundle without touching
producer source directories.
- Local validation failures prevent HTTP requests.
- Response parsing covers `202`, `400`, `401`, `409`, `413`, `415`, `503`,
non-JSON errors, and unexpected statuses.
- Retry uses the same idempotency key and stops correctly.
- Context cancellation during retry backoff is honored.
- Custom `*http.Client` behavior is covered with `httptest`.
Completion criteria:
- Go producers can upload valid bundles through `pkg/upload`.
- Safe retry behavior relies on the implemented server idempotency contract.
- Public package tests prove upload behavior does not duplicate or drift from
`pkg/bundle` semantics.
## Stage 3: Documentation And Examples
Goal:
Document implemented producer upload and idempotency behavior after the server
contract and public package exist.
Implementation scope:
- Update current-behavior docs only after Stages 1 and 2 are implemented.
- Add secret-free examples where they are safe, copyable, and describe
implemented behavior.
- Keep deferred items out of current docs.
Docs to update:
- `README.md`
- `docs/integrations/source-bundle.md`
- `docs/integrations/http-upload.md`
- `docs/operations.md`
- `docs/internal/app.md`
- `docs/policy/development.md`
- `examples/`, only if examples are safe, copyable, and implemented
Tests and checks:
```sh
go test ./...
rg -n "idempotency|Idempotency-Key|pkg/upload|UploadBundle|UploadFiles" README.md docs examples
```
Completion criteria:
- Current docs describe the implemented server idempotency and `pkg/upload`
API.
- Completed behavior is not documented only as future work.
- `docs/roadmap/` contains only future or deferred producer-upload work.
This file now tracks only producer-upload work that remains outside the current
implementation.
## Deferred Work
@@ -190,22 +24,3 @@ Completion criteria:
- Multipart, resumable, or streaming upload protocols.
- URL-token authentication.
- Browser UI or public exposure defaults.
## Validation
For this documentation pass:
```sh
rg -n "Idempotency-Key|pkg/upload|UploadBundle|UploadFiles|Stage 1: Server-Side Upload Idempotency" docs/roadmap/implementation.md
rg -n "Producer-supplied idempotency keys|Producer Coordination" docs/roadmap/http.md docs/roadmap/implementation.md
rg -n "pkg/upload|UploadBundle|UploadFiles|Idempotency-Key" README.md docs examples --glob '!docs/roadmap/**'
git status --short
git diff -- docs/roadmap/implementation.md
```
Expected result:
- New future behavior appears only under `docs/roadmap/`.
- Existing unrelated worktree changes, including any current
`docs/roadmap/documentation.md` deletion, are not touched.
- This pass changes only `docs/roadmap/implementation.md`.

321
docs/roadmap/producer.md
View File

@@ -1,320 +1,21 @@
# Roadmap: Producer HTTP Upload Package
# Roadmap: Producer Upload Deferred Work
## Purpose
The producer HTTP upload package plan has been implemented for the current
server and client scope. Current behavior is documented outside roadmap files:
Add a second public producer-facing package that lets Go producer applications
build or validate a compliant source bundle, package it as a gzip-compressed tar
archive, and submit it to the HTTP upload API with safe retry support.
- [Source bundle contract](../integrations/source-bundle.md)
- [HTTP upload contract](../integrations/http-upload.md)
- [Operations](../operations.md)
- [Development policy](../policy/development.md)
The current public producer package, `pkg/bundle`, owns source manifest
semantics, digest calculation, path validation, local manifest building, local
bundle writing, and local bundle validation. The new package must build on that
contract instead of reimplementing it.
This roadmap also adds server-side producer idempotency keys to the HTTP upload
API. Idempotency is required for the producer upload package's retry behavior:
the client can safely retry an upload with the same key, and the server can
collapse duplicate accepted uploads into the original run.
## Goals
- Make the common Go producer workflow small and hard to misuse.
- Reuse `pkg/bundle` for manifest generation, path normalization, SHA-256
calculation, digest calculation, and local validation.
- Create upload archives that match the server's source bundle archive contract.
- Add server-side idempotency records scoped to the authenticated pipeline.
- Send idempotency keys from the public upload package by default.
- Handle bearer authentication without logging or returning token values.
- Parse successful, duplicate, conflict, and error responses into typed
producer-side results.
- Retry safely using idempotency keys and bounded backoff.
- Keep the package dependency-light and usable from ordinary Go producer
applications.
## Non-Goals
- Do not expose `internal/app`, `internal/ingest`, storage backends, server
config, or destination state types through the public package.
- Do not add durable idempotency storage, durable client queues, background
workers, or database-backed retry processing.
- Do not add zstd, multipart upload, resumable upload, or non-tar archive
formats.
- Do not require producers to know distributor pipeline ids; server-side token
mapping remains authoritative.
- Do not make the package a replacement for the existing CLI or server API
documentation.
## Implementation Sequence
Implement this feature in three stages:
1. Server-side HTTP idempotency keys.
2. Public `pkg/upload` client package.
3. Current-behavior documentation and examples.
Server idempotency should land first so the public upload package can rely on
the final retry contract from its first release.
## Stage 1: Server Idempotency Keys
Goal:
Extend `distributor serve` so `POST /upload` can safely accept retried producer
uploads without creating duplicate accepted runs.
HTTP contract:
- Producers may send `Idempotency-Key: <key>` with `POST /upload`.
- The public upload package must always send this header.
- Raw HTTP clients may omit it; omitted keys preserve current behavior.
- Keys are scoped to the authenticated pipeline selected by bearer token.
- Valid keys are non-empty ASCII strings up to 128 bytes using
letters, digits, `.`, `_`, `-`, and `:`.
- Invalid keys return `400`.
Server behavior:
- After archive staging and source bundle validation succeeds, record the
idempotency key with the accepted run id and the normalized source manifest
identity.
- If the same pipeline receives the same key and the staged upload has the same
normalized source manifest identity, return the original accepted response
instead of enqueueing another run.
- If the same pipeline receives the same key and the staged upload has a
different normalized source manifest identity, return `409 Conflict`.
- If the same key is already being processed concurrently for the same pipeline
before a manifest identity is available, return a retryable conflict response
without accepting a new run.
- Idempotency records are memory-only and expire with the existing HTTP upload
retention window.
- Server restart clears idempotency records, matching the current memory-only
status and queue behavior.
Manifest identity:
- Compare normalized source manifest semantics, not raw archive bytes.
- At minimum, compare manifest schema version, id, created timestamp, bundle
digest, and ordered file records.
- Different tar metadata or gzip encoding for the same source bundle should not
create a conflict.
Tests:
- `go test ./internal/app ./internal/ingest`
- Same token, same key, same staged bundle returns the original run id and does
not enqueue a second run.
- Same token, same key, different staged bundle returns `409`.
- Same key under different authenticated pipelines does not conflict.
- Missing idempotency key preserves existing raw HTTP behavior.
- Invalid key syntax returns `400`.
- Idempotency records expire with completed run status retention.
- Secret tokens and idempotency keys are not logged in errors beyond the key
value itself where required for diagnostics.
Completion criteria:
- The HTTP API has an implemented, tested idempotency contract.
- Existing clients without `Idempotency-Key` continue to work.
- Duplicate idempotent uploads cannot create duplicate accepted runs.
## Stage 2: Public `pkg/upload` Client
Goal:
Add a new public `pkg/upload` package that submits compliant bundles to
`distributor serve` using bearer authentication and idempotency keys.
Package name:
- Use `pkg/upload`.
- Rationale: `pkg/bundle` owns bundle construction and validation; `pkg/upload`
owns submission to the distributor HTTP upload API.
API shape:
Use options-struct APIs rather than one large positional function. Initial APIs
must cover two producer workflows:
- upload an existing local bundle root;
- build a temporary bundle from explicit producer files and upload it.
Representative API shape:
```go
package upload
type Client struct {
// unexported fields
}
type ClientOptions struct {
Endpoint string
Token string
HTTPClient *http.Client
Retry RetryOptions
}
type RetryOptions struct {
MaxAttempts int
BaseDelay time.Duration
MaxDelay time.Duration
}
type UploadBundleOptions struct {
Root string
Validate bool
IdempotencyKey string
}
type UploadFilesOptions struct {
ID string
Created time.Time
Files []bundle.BundleFile
Validate bool
TempDir string
IdempotencyKey string
}
type Result struct {
RunID string
Status string
}
func NewClient(opts ClientOptions) (*Client, error)
func (c *Client) UploadBundle(ctx context.Context, opts UploadBundleOptions) (Result, error)
func (c *Client) UploadFiles(ctx context.Context, opts UploadFilesOptions) (Result, error)
func (c *Client) Status(ctx context.Context, runID string) (RunStatus, error)
```
Required API semantics:
- `Endpoint` is a distributor server base URL. The client derives `/upload` and
`/runs/<run-id>` internally.
- `Token` is required and is sent as `Authorization: Bearer <token>`.
- `HTTPClient` is optional; when omitted, use a client with conservative
timeouts.
- `UploadBundle` reads and packages an existing local source bundle.
- `UploadFiles` creates a temporary complete bundle through `pkg/bundle`, then
packages and uploads it.
- `Status` is optional for callers and never required by `UploadBundle` or
`UploadFiles`.
- `Result` represents upload admission, not final publication success.
Idempotency key behavior:
- The client must send `Idempotency-Key` on every upload.
- If the caller supplies `IdempotencyKey`, use it.
- If omitted, generate a random 128-bit lowercase hex key once for that upload
operation and reuse it for all retries from that call.
- Generated keys are not stable across process restarts or separate calls.
- Producers that need cross-process retry safety must supply their own stable
key.
- Validate caller-supplied keys before making a request.
Bundle and archive behavior:
- `UploadBundle` loads `manifest.json` from the bundle root and validates the
local bundle by default.
- `UploadBundle` includes `manifest.json` and every manifest-listed file in the
tar.gz archive, and does not include unlisted files.
- `UploadFiles` requires a non-empty bundle id and non-empty file list.
- `UploadFiles` uses `pkg/bundle.WriteBundle` or equivalent public bundle APIs
in a temporary directory and preserves explicit file order.
- Zero `Created` follows `pkg/bundle` defaulting behavior.
- Validation is enabled by default and may be explicitly disabled only for
callers that already performed equivalent validation.
- Tar entry names are slash-separated bundle-relative paths.
- The package must not write into producer source directories.
Archive and retry strategy:
- Create a replayable upload body for each upload operation.
- The implementation may either create a temporary `.tar.gz` file or regenerate
the tar.gz body from the validated staged bundle for each attempt.
- Clean up all temporary bundles and archive files created by the package.
- Use `Content-Type: application/gzip`.
Retry policy:
- Defaults should be safe and modest, for example three total attempts with
bounded exponential backoff.
- Retry `503 Service Unavailable` because the upload was not accepted.
- Retry temporary network errors and ambiguous mid-upload failures using the
same idempotency key and replayable body.
- Do not retry `400`, `401`, `409`, `413`, or `415`.
- Do not retry after `202 Accepted`.
- Respect caller context cancellation before waiting and before each retry.
- Redact the bearer token from all errors.
HTTP response handling:
- Parse `202 Accepted` responses into `Result`.
- Parse JSON error bodies where available.
- Include HTTP status codes and response messages in typed errors.
- Treat duplicate idempotent `202` responses the same as first acceptance.
- Treat `409 Conflict` as an idempotency conflict error.
- Close response bodies on every attempt.
Tests:
- `go test ./pkg/bundle ./pkg/upload`
- Client construction with valid and invalid base endpoints.
- Missing token rejection and token redaction in errors.
- Caller-supplied and generated idempotency keys.
- Uploading an existing valid bundle root.
- Building and uploading from `bundle.BundleFile` values.
- Local validation failures before any HTTP request.
- Tar.gz entry names, manifest inclusion, and exclusion of unlisted files.
- `202`, `400`, `401`, `409`, `413`, `415`, `503`, non-JSON errors, and
unexpected status response parsing.
- Safe retry with the same idempotency key for `503` and retryable network
failures.
- No retry for non-retryable statuses.
- Context cancellation during retry backoff.
- Custom `*http.Client` behavior through `httptest.Server`.
Completion criteria:
- Producer applications can build or validate a bundle and upload it with one
package.
- All uploads include idempotency keys.
- Retry behavior is safe under the server idempotency contract.
## Stage 3: Documentation And Examples
Goal:
Document the implemented producer upload package and idempotency behavior only
after the server and public package exist.
Current-behavior documentation updates:
- `README.md`: mention the new producer upload package briefly.
- `docs/integrations/source-bundle.md`: link from producer APIs to upload
helpers.
- `docs/integrations/http-upload.md`: document `Idempotency-Key` and add a
short Go producer helper section.
- `docs/operations.md`: add a concise producer-side example if useful.
- `docs/internal/app.md`: document server-side idempotency record behavior.
- `docs/policy/development.md`: document the `pkg/upload` boundary and test
expectations.
Tests and checks:
```sh
go test ./...
rg -n "pkg/upload|Idempotency-Key|UploadBundle|UploadFiles" README.md docs examples
```
Completion criteria:
- Current docs describe implemented behavior.
- Future-only behavior remains under `docs/roadmap/`.
This roadmap records producer-upload extensions that are intentionally not part
of the current implementation.
## Deferred Work
- Durable idempotency records across server restarts.
- Producer-supplied idempotency keys integrated with a database-backed queue.
- Producer coordination integrated with a database-backed queue.
- Durable client queues or background producer workers.
- `UploadAndWait` or long-polling helpers.
- Run cancellation, retry, or listing endpoints.
- Zstandard-compressed tar archives.

47
examples/upload-client/main.go Normal file
View File

@@ -0,0 +1,47 @@
package main
import (
"context"
"fmt"
"log"
"os"
"time"
"gitea.maximumdirect.net/eric/distributor/pkg/upload"
)
func main() {
token := os.Getenv("DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN")
if token == "" {
log.Fatal("set DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN before running this example")
}
endpoint := os.Getenv("DISTRIBUTOR_EXAMPLE_UPLOAD_ENDPOINT")
if endpoint == "" {
endpoint = "http://127.0.0.1:8080"
}
bundleRoot := "examples/source-bundle"
if len(os.Args) > 1 {
bundleRoot = os.Args[1]
}
idempotencyKey := os.Getenv("DISTRIBUTOR_EXAMPLE_UPLOAD_IDEMPOTENCY_KEY")
client, err := upload.NewClient(upload.ClientOptions{
Endpoint: endpoint,
Token: token,
})
if err != nil {
log.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
opts := upload.UploadBundleOptions{Root: bundleRoot}
if idempotencyKey != "" {
opts.IdempotencyKey = idempotencyKey
}
result, err := client.UploadBundle(ctx, opts)
if err != nil {
log.Fatal(err)
}
fmt.Printf("accepted run %s with status %s\n", result.RunID, result.Status)
}