From 1a402e6cfa4c97537244b0fa6a94da0624673fd8 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 4 Jun 2026 08:55:43 -0500 Subject: [PATCH] Added a roadmap and an implentation plan for a new public http upload helper package --- docs/roadmap/documentation.md | 97 ---------- docs/roadmap/implementation.md | 211 +++++++++++++++++++++ docs/roadmap/producer.md | 322 +++++++++++++++++++++++++++++++++ 3 files changed, 533 insertions(+), 97 deletions(-) delete mode 100644 docs/roadmap/documentation.md create mode 100644 docs/roadmap/implementation.md create mode 100644 docs/roadmap/producer.md diff --git a/docs/roadmap/documentation.md b/docs/roadmap/documentation.md deleted file mode 100644 index 654de4c..0000000 --- a/docs/roadmap/documentation.md +++ /dev/null @@ -1,97 +0,0 @@ -# Documentation Roadmap - -## Purpose - -This roadmap tracks the remaining work required to verify that project documentation complies with `docs/policy/documentation.md` and accurately reflects the current implementation. - -The documentation migration has rewritten the current user, operator, integration, and internal component docs. This file now records only remaining validation work. Current behavior belongs outside `docs/roadmap/`; deferred or unimplemented work belongs under `docs/roadmap/`. - -## Current Documentation Set - -Current documentation outside roadmap: - -- `README.md`: concise project orientation and quickstart. -- `docs/cli.md`: canonical CLI command, flag, workflow, and output reference. -- `docs/config.md`: canonical YAML configuration reference. -- `docs/operations.md`: operating, safety, state, upload, and recovery guidance. -- `docs/troubleshooting.md`: symptom-oriented diagnostic and safe-fix guide. -- `docs/policy/architecture.md`: architecture and invariant policy. -- `docs/policy/development.md`: contributor and coding workflow policy. -- `docs/policy/documentation.md`: controlling documentation policy. -- `docs/integrations/*.md`: implemented external/file-format/protocol contracts. -- `docs/internal/*.md`: implemented internal component contracts. -- `examples/*.yml` and `examples/source-bundle/*`: maintained example configs and source bundle fixture. - -Current roadmap files: - -- `docs/roadmap/documentation.md`: this remaining documentation validation plan. -- `docs/roadmap/http.md`: deferred HTTP upload extensions only. - -Removed completed roadmap artifacts: - -- `docs/roadmap/audit.md` -- `docs/roadmap/cleanup.md` -- `docs/roadmap/implementation.md` - -## Remaining Documentation Validation - -Goal: verify the rewritten docs against tests, examples, code, links, and the documentation policy checklist. - -Files to create, update, delete, or move: fixes only if validation finds gaps. - -Repository areas to inspect: - -- `README.md` -- `docs/cli.md` -- `docs/config.md` -- `docs/operations.md` -- `docs/troubleshooting.md` -- `docs/internal/` -- `docs/integrations/` -- `docs/policy/` -- `examples/` -- CLI parser code under `internal/cli` -- config loading/defaulting/validation under `internal/config` -- app/report/upload behavior under `internal/app` -- source bundle, state, publish, storage, adapter, and transform packages - -Acceptance criteria: - -- Tests pass for the full repository. -- Maintained example configs load. -- CLI examples and flags match parser behavior. -- Config fields and defaults match `internal/config`. -- Operations and troubleshooting docs describe implemented behavior only. -- Internal docs preserve package boundaries and policy-required sections. -- Integration docs describe only implemented contracts. -- Roadmap files contain only remaining or deferred work. -- Links resolve. -- No secrets or private data are present. - -Suggested validation commands: - -```sh -go test ./... -go test ./internal/config ./internal/cli ./internal/app -go test ./pkg/bundle ./internal/bundle ./internal/state ./internal/publish ./internal/storage ./internal/storage/fake -go test ./internal/adapters/local ./internal/adapters/ssh ./internal/adapters/s3 ./internal/ingest ./internal/transform/markdown -rg -n -i "future|planned|deferred|experimental|deprecated|not implemented|old behavior" README.md docs --glob '!docs/roadmap/**' --glob '!docs/policy/**' -rg -n "\\b(Stages?|Phases?)\\b" README.md docs --glob '!docs/roadmap/**' --glob '!docs/policy/**' -rg -n -- "--config|--dry-run|--force|--format|--pipeline|--bundle|--id|--file|--created|--overwrite" docs/cli.md internal/cli -rg -n "examples/" README.md docs examples internal/config/load_test.go -``` - -Manual review items: - -- Confirm README remains concise and orientation-focused. -- Confirm `docs/config.md` is the only full config field/default reference. -- Confirm `docs/cli.md` is the only full command/flag reference. -- Confirm `docs/operations.md` focuses on operating and recovery. -- Confirm `docs/troubleshooting.md` remains symptom-first. -- Confirm `docs/internal/` describes implemented component contracts and boundaries. -- Confirm integration docs do not claim support for unimplemented external features. -- Confirm examples contain no secrets and distinguish local runnable examples from environment-gated remote examples. - -## Open Questions - -No open questions block the remaining validation work. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..7c44041 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,211 @@ +# Producer HTTP Upload Package Implementation Roadmap + +## Current Baseline + +`distributor serve` and the `http_upload` source backend are implemented. +Producers can already submit complete tar or gzip-compressed tar source bundles +to `POST /upload` with bearer authentication, and each bearer token maps to one +configured upload pipeline. + +The public `pkg/bundle` package already provides producer-side source manifest +semantics, digest calculation, path validation, local manifest building, local +bundle writing, and local bundle validation helpers. + +The current implementation does not have a server-side producer idempotency +contract, and it does not provide a public `pkg/upload` helper package. + +Future behavior remains under `docs/roadmap/` until implemented. Do not update +README, current user docs, examples, or current-behavior internal docs until the +corresponding stage has been implemented. + +This active roadmap implements the accepted producer upload package plan in +`docs/roadmap/producer.md`. It supersedes the older `docs/roadmap/http.md` +deferred note for producer-supplied idempotency keys; producer idempotency is +now active roadmap work. + +## Active Roadmap + +## Stage 1: Server-Side Upload Idempotency + +Goal: + +Add `Idempotency-Key` support to `POST /upload` so safe producer retries do not +create duplicate accepted runs. + +Implementation scope: + +- Validate optional `Idempotency-Key` headers using the syntax defined in + `docs/roadmap/producer.md`. +- Scope keys by the authenticated pipeline selected through bearer-token + mapping. +- Record accepted keys after archive staging and source bundle validation + succeed. +- Compare normalized source manifest identity, not raw archive bytes. +- Return the original accepted run response for the same pipeline, same key, + and same manifest identity. +- Return `409 Conflict` for the same pipeline and key with a different manifest + identity. +- Return a retryable conflict response when the same key is already being + processed concurrently for the same pipeline before manifest identity is + known. +- Expire idempotency records with existing upload status retention. +- Keep idempotency records memory-only; server restart clears them. +- Preserve current raw HTTP behavior when no idempotency key is supplied. + +Current-behavior documentation updates after implementation: + +- `docs/integrations/http-upload.md` +- `docs/operations.md` +- `docs/internal/app.md` +- `docs/troubleshooting.md` + +Tests: + +- Same key and same bundle returns the original run id and does not enqueue a + second run. +- Same key and different bundle returns `409`. +- Same key under different authenticated pipelines does not conflict. +- Missing key preserves current raw HTTP behavior. +- Invalid key syntax returns `400`. +- Expiration removes idempotency records. +- Tokens are never leaked; idempotency keys appear only where needed for + diagnostics. + +Completion criteria: + +- Existing upload clients continue to work. +- The HTTP API has an implemented, tested idempotency contract. +- Idempotent retries cannot create duplicate accepted runs. + +## Stage 2: Public `pkg/upload` API And Client + +Goal: + +Add a producer-facing upload package that builds or validates bundles, archives +them, and submits them to the HTTP upload API. + +Implementation scope: + +- Add public `pkg/upload`. +- Use `pkg/bundle` for manifest generation, digest and path semantics, local + validation, and temporary bundle creation. +- Implement options-struct APIs matching `docs/roadmap/producer.md`: + `ClientOptions`, `RetryOptions`, `UploadBundleOptions`, + `UploadFilesOptions`, `Result`, `RunStatus`, `NewClient`, `UploadBundle`, + `UploadFiles`, and `Status`. +- Treat `Endpoint` as the distributor server base URL, deriving `/upload` and + `/runs/` internally. +- Require bearer token authentication and redact token values from all errors. +- Always send `Idempotency-Key`. +- Use caller-supplied idempotency keys when provided. +- When no key is supplied, generate one random 128-bit lowercase hex key per + upload operation and reuse it across retries from that call. +- Support uploading an existing local bundle root. +- Support building a temporary bundle from explicit `bundle.BundleFile` values + and uploading it. +- Create replayable gzip-compressed tar uploads with + `Content-Type: application/gzip`. +- Retry only safe cases: `503 Service Unavailable`, temporary network errors, + and ambiguous mid-upload failures, using the same idempotency key and + replayable body. +- Do not retry `400`, `401`, `409`, `413`, or `415`. +- Do not retry after `202 Accepted`. +- Respect context cancellation before waiting and before each retry. +- Close response bodies on every attempt. + +Current-behavior documentation updates after implementation: + +- Update `pkg/bundle` integration references only as needed once `pkg/upload` + exists. +- Keep full user-facing docs and examples for Stage 3. + +Tests: + +- Client construction validates endpoint and token requirements. +- Token values are redacted from errors. +- Caller-supplied and generated idempotency keys are sent correctly. +- Existing bundle upload includes only `manifest.json` and manifest-listed + files. +- File-based upload builds a compliant temporary bundle without touching + producer source directories. +- Local validation failures prevent HTTP requests. +- Response parsing covers `202`, `400`, `401`, `409`, `413`, `415`, `503`, + non-JSON errors, and unexpected statuses. +- Retry uses the same idempotency key and stops correctly. +- Context cancellation during retry backoff is honored. +- Custom `*http.Client` behavior is covered with `httptest`. + +Completion criteria: + +- Go producers can upload valid bundles through `pkg/upload`. +- Safe retry behavior relies on the implemented server idempotency contract. +- Public package tests prove upload behavior does not duplicate or drift from + `pkg/bundle` semantics. + +## Stage 3: Documentation And Examples + +Goal: + +Document implemented producer upload and idempotency behavior after the server +contract and public package exist. + +Implementation scope: + +- Update current-behavior docs only after Stages 1 and 2 are implemented. +- Add secret-free examples where they are safe, copyable, and describe + implemented behavior. +- Keep deferred items out of current docs. + +Docs to update: + +- `README.md` +- `docs/integrations/source-bundle.md` +- `docs/integrations/http-upload.md` +- `docs/operations.md` +- `docs/internal/app.md` +- `docs/policy/development.md` +- `examples/`, only if examples are safe, copyable, and implemented + +Tests and checks: + +```sh +go test ./... +rg -n "idempotency|Idempotency-Key|pkg/upload|UploadBundle|UploadFiles" README.md docs examples +``` + +Completion criteria: + +- Current docs describe the implemented server idempotency and `pkg/upload` + API. +- Completed behavior is not documented only as future work. +- `docs/roadmap/` contains only future or deferred producer-upload work. + +## Deferred Work + +- Durable idempotency storage across server restarts. +- Database-backed queues or durable producer retry processing. +- `UploadAndWait`, long polling, run cancellation, run retry, or run listing + helpers. +- Zstandard archives. +- Multipart, resumable, or streaming upload protocols. +- URL-token authentication. +- Browser UI or public exposure defaults. + +## Validation + +For this documentation pass: + +```sh +rg -n "Idempotency-Key|pkg/upload|UploadBundle|UploadFiles|Stage 1: Server-Side Upload Idempotency" docs/roadmap/implementation.md +rg -n "Producer-supplied idempotency keys|Producer Coordination" docs/roadmap/http.md docs/roadmap/implementation.md +rg -n "pkg/upload|UploadBundle|UploadFiles|Idempotency-Key" README.md docs examples --glob '!docs/roadmap/**' +git status --short +git diff -- docs/roadmap/implementation.md +``` + +Expected result: + +- New future behavior appears only under `docs/roadmap/`. +- Existing unrelated worktree changes, including any current + `docs/roadmap/documentation.md` deletion, are not touched. +- This pass changes only `docs/roadmap/implementation.md`. diff --git a/docs/roadmap/producer.md b/docs/roadmap/producer.md new file mode 100644 index 0000000..d2d041f --- /dev/null +++ b/docs/roadmap/producer.md @@ -0,0 +1,322 @@ +# Roadmap: Producer HTTP Upload Package + +## Purpose + +Add a second public producer-facing package that lets Go producer applications +build or validate a compliant source bundle, package it as a gzip-compressed tar +archive, and submit it to the HTTP upload API with safe retry support. + +The current public producer package, `pkg/bundle`, owns source manifest +semantics, digest calculation, path validation, local manifest building, local +bundle writing, and local bundle validation. The new package must build on that +contract instead of reimplementing it. + +This roadmap also adds server-side producer idempotency keys to the HTTP upload +API. Idempotency is required for the producer upload package's retry behavior: +the client can safely retry an upload with the same key, and the server can +collapse duplicate accepted uploads into the original run. + +## Goals + +- Make the common Go producer workflow small and hard to misuse. +- Reuse `pkg/bundle` for manifest generation, path normalization, SHA-256 + calculation, digest calculation, and local validation. +- Create upload archives that match the server's source bundle archive contract. +- Add server-side idempotency records scoped to the authenticated pipeline. +- Send idempotency keys from the public upload package by default. +- Handle bearer authentication without logging or returning token values. +- Parse successful, duplicate, conflict, and error responses into typed + producer-side results. +- Retry safely using idempotency keys and bounded backoff. +- Keep the package dependency-light and usable from ordinary Go producer + applications. + +## Non-Goals + +- Do not expose `internal/app`, `internal/ingest`, storage backends, server + config, or destination state types through the public package. +- Do not add durable idempotency storage, durable client queues, background + workers, or database-backed retry processing. +- Do not add zstd, multipart upload, resumable upload, or non-tar archive + formats. +- Do not require producers to know distributor pipeline ids; server-side token + mapping remains authoritative. +- Do not make the package a replacement for the existing CLI or server API + documentation. + +## Implementation Sequence + +Implement this feature in three stages: + +1. Server-side HTTP idempotency keys. +2. Public `pkg/upload` client package. +3. Current-behavior documentation and examples. + +Server idempotency should land first so the public upload package can rely on +the final retry contract from its first release. + +## Stage 1: Server Idempotency Keys + +Goal: + +Extend `distributor serve` so `POST /upload` can safely accept retried producer +uploads without creating duplicate accepted runs. + +HTTP contract: + +- Producers may send `Idempotency-Key: ` with `POST /upload`. +- The public upload package must always send this header. +- Raw HTTP clients may omit it; omitted keys preserve current behavior. +- Keys are scoped to the authenticated pipeline selected by bearer token. +- Valid keys are non-empty ASCII strings up to 128 bytes using + letters, digits, `.`, `_`, `-`, and `:`. +- Invalid keys return `400`. + +Server behavior: + +- After archive staging and source bundle validation succeeds, record the + idempotency key with the accepted run id and the normalized source manifest + identity. +- If the same pipeline receives the same key and the staged upload has the same + normalized source manifest identity, return the original accepted response + instead of enqueueing another run. +- If the same pipeline receives the same key and the staged upload has a + different normalized source manifest identity, return `409 Conflict`. +- If the same key is already being processed concurrently for the same pipeline + before a manifest identity is available, return a retryable conflict response + without accepting a new run. +- Idempotency records are memory-only and expire with the existing HTTP upload + retention window. +- Server restart clears idempotency records, matching the current memory-only + status and queue behavior. + +Manifest identity: + +- Compare normalized source manifest semantics, not raw archive bytes. +- At minimum, compare manifest schema version, id, created timestamp, bundle + digest, and ordered file records. +- Different tar metadata or gzip encoding for the same source bundle should not + create a conflict. + +Tests: + +- `go test ./internal/app ./internal/ingest` +- Same token, same key, same staged bundle returns the original run id and does + not enqueue a second run. +- Same token, same key, different staged bundle returns `409`. +- Same key under different authenticated pipelines does not conflict. +- Missing idempotency key preserves existing raw HTTP behavior. +- Invalid key syntax returns `400`. +- Idempotency records expire with completed run status retention. +- Secret tokens and idempotency keys are not logged in errors beyond the key + value itself where required for diagnostics. + +Completion criteria: + +- The HTTP API has an implemented, tested idempotency contract. +- Existing clients without `Idempotency-Key` continue to work. +- Duplicate idempotent uploads cannot create duplicate accepted runs. + +## Stage 2: Public `pkg/upload` Client + +Goal: + +Add a new public `pkg/upload` package that submits compliant bundles to +`distributor serve` using bearer authentication and idempotency keys. + +Package name: + +- Use `pkg/upload`. +- Rationale: `pkg/bundle` owns bundle construction and validation; `pkg/upload` + owns submission to the distributor HTTP upload API. + +API shape: + +Use options-struct APIs rather than one large positional function. Initial APIs +must cover two producer workflows: + +- upload an existing local bundle root; +- build a temporary bundle from explicit producer files and upload it. + +Representative API shape: + +```go +package upload + +type Client struct { + // unexported fields +} + +type ClientOptions struct { + Endpoint string + Token string + HTTPClient *http.Client + Retry RetryOptions +} + +type RetryOptions struct { + MaxAttempts int + BaseDelay time.Duration + MaxDelay time.Duration +} + +type UploadBundleOptions struct { + Root string + Validate bool + IdempotencyKey string +} + +type UploadFilesOptions struct { + ID string + Created time.Time + Files []bundle.BundleFile + Validate bool + TempDir string + IdempotencyKey string +} + +type Result struct { + RunID string + Status string +} + +func NewClient(opts ClientOptions) (*Client, error) +func (c *Client) UploadBundle(ctx context.Context, opts UploadBundleOptions) (Result, error) +func (c *Client) UploadFiles(ctx context.Context, opts UploadFilesOptions) (Result, error) +func (c *Client) Status(ctx context.Context, runID string) (RunStatus, error) +``` + +Required API semantics: + +- `Endpoint` is a distributor server base URL. The client derives `/upload` and + `/runs/` internally. +- `Token` is required and is sent as `Authorization: Bearer `. +- `HTTPClient` is optional; when omitted, use a client with conservative + timeouts. +- `UploadBundle` reads and packages an existing local source bundle. +- `UploadFiles` creates a temporary complete bundle through `pkg/bundle`, then + packages and uploads it. +- `Status` is optional for callers and never required by `UploadBundle` or + `UploadFiles`. +- `Result` represents upload admission, not final publication success. + +Idempotency key behavior: + +- The client must send `Idempotency-Key` on every upload. +- If the caller supplies `IdempotencyKey`, use it. +- If omitted, generate a random 128-bit lowercase hex key once for that upload + operation and reuse it for all retries from that call. +- Generated keys are not stable across process restarts or separate calls. +- Producers that need cross-process retry safety must supply their own stable + key. +- Validate caller-supplied keys before making a request. + +Bundle and archive behavior: + +- `UploadBundle` loads `manifest.json` from the bundle root and validates the + local bundle by default. +- `UploadBundle` includes `manifest.json` and every manifest-listed file in the + tar.gz archive, and does not include unlisted files. +- `UploadFiles` requires a non-empty bundle id and non-empty file list. +- `UploadFiles` uses `pkg/bundle.WriteBundle` or equivalent public bundle APIs + in a temporary directory and preserves explicit file order. +- Zero `Created` follows `pkg/bundle` defaulting behavior. +- Validation is enabled by default and may be explicitly disabled only for + callers that already performed equivalent validation. +- Tar entry names are slash-separated bundle-relative paths. +- The package must not write into producer source directories. + +Archive and retry strategy: + +- Create a replayable upload body for each upload operation. +- The implementation may either create a temporary `.tar.gz` file or regenerate + the tar.gz body from the validated staged bundle for each attempt. +- Clean up all temporary bundles and archive files created by the package. +- Use `Content-Type: application/gzip`. + +Retry policy: + +- Defaults should be safe and modest, for example three total attempts with + bounded exponential backoff. +- Retry `503 Service Unavailable` because the upload was not accepted. +- Retry temporary network errors and ambiguous mid-upload failures using the + same idempotency key and replayable body. +- Do not retry `400`, `401`, `409`, `413`, or `415`. +- Do not retry after `202 Accepted`. +- Respect caller context cancellation before waiting and before each retry. +- Redact the bearer token from all errors. + +HTTP response handling: + +- Parse `202 Accepted` responses into `Result`. +- Parse JSON error bodies where available. +- Include HTTP status codes and response messages in typed errors. +- Treat duplicate idempotent `202` responses the same as first acceptance. +- Treat `409 Conflict` as an idempotency conflict error. +- Close response bodies on every attempt. + +Tests: + +- `go test ./pkg/bundle ./pkg/upload` +- Client construction with valid and invalid base endpoints. +- Missing token rejection and token redaction in errors. +- Caller-supplied and generated idempotency keys. +- Uploading an existing valid bundle root. +- Building and uploading from `bundle.BundleFile` values. +- Local validation failures before any HTTP request. +- Tar.gz entry names, manifest inclusion, and exclusion of unlisted files. +- `202`, `400`, `401`, `409`, `413`, `415`, `503`, non-JSON errors, and + unexpected status response parsing. +- Safe retry with the same idempotency key for `503` and retryable network + failures. +- No retry for non-retryable statuses. +- Context cancellation during retry backoff. +- Custom `*http.Client` behavior through `httptest.Server`. + +Completion criteria: + +- Producer applications can build or validate a bundle and upload it with one + package. +- All uploads include idempotency keys. +- Retry behavior is safe under the server idempotency contract. + +## Stage 3: Documentation And Examples + +Goal: + +Document the implemented producer upload package and idempotency behavior only +after the server and public package exist. + +Current-behavior documentation updates: + +- `README.md`: mention the new producer upload package briefly. +- `docs/integrations/source-bundle.md`: link from producer APIs to upload + helpers. +- `docs/integrations/http-upload.md`: document `Idempotency-Key` and add a + short Go producer helper section. +- `docs/operations.md`: add a concise producer-side example if useful. +- `docs/internal/app.md`: document server-side idempotency record behavior. +- `docs/policy/development.md`: document the `pkg/upload` boundary and test + expectations. + +Tests and checks: + +```sh +go test ./... +rg -n "pkg/upload|Idempotency-Key|UploadBundle|UploadFiles" README.md docs examples +``` + +Completion criteria: + +- Current docs describe implemented behavior. +- Future-only behavior remains under `docs/roadmap/`. + +## Deferred Work + +- Durable idempotency records across server restarts. +- Producer-supplied idempotency keys integrated with a database-backed queue. +- `UploadAndWait` or long-polling helpers. +- Run cancellation, retry, or listing endpoints. +- Zstandard-compressed tar archives. +- Multipart, resumable, or streaming object upload support. +- URL-token authentication for constrained clients.