diff --git a/README.md b/README.md index 52c0afd..397c8c6 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,10 @@ `distributor` validates manifested report bundles and publishes selected source or generated artifacts to configured destinations. -It is a local-first CLI with SSH/SFTP and S3-compatible storage support: source bundles can be read from local or remote storage, destinations can be local directories or remote paths, and Markdown files can be rendered to HTML sidecars or `index.html`. +It is a local-first CLI with SSH/SFTP, S3-compatible storage, and HTTP upload +support: source bundles can be read from local or remote storage, pushed to the +upload API, published to local directories or remote paths, and rendered from +Markdown to HTML sidecars or `index.html`. Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate complete local source bundles with the same manifest contract used by `distributor`. diff --git a/docs/cli.md b/docs/cli.md index 7ec1a56..f206f05 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -136,14 +136,15 @@ go run ./cmd/distributor run --config examples/local-html.yml Start the HTTP upload API: ```sh -go run ./cmd/distributor serve --config +DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN= \ + go run ./cmd/distributor serve --config examples/http-upload-local.yml ``` Upload an archive to the configured `http_upload` pipeline associated with a bearer token: ```sh curl -X POST http://127.0.0.1:8080/upload \ - -H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \ + -H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \ -H "Content-Type: application/gzip" \ --data-binary @bundle.tar.gz ``` diff --git a/docs/config.md b/docs/config.md index d8633b8..1c32d14 100644 --- a/docs/config.md +++ b/docs/config.md @@ -414,6 +414,10 @@ S3 credentials may name environment variables: - `credentials.access_key_id_env` - `credentials.secret_access_key_env` +HTTP upload tokens name one environment variable or secret-file name: + +- `source.token_env` + ## Examples Maintained examples live under [examples](../examples/): @@ -424,5 +428,6 @@ Maintained examples live under [examples](../examples/): - `local-index.yml`: runnable local `index.html` publication. - `fan-out.yml`: runnable local fan-out publication to source and HTML destinations. - `archive-and-latest.yml`: runnable local fan-out publication to an archive destination and a fixed latest destination. +- `http-upload-local.yml`: local HTTP upload server example with a token environment variable reference. - `ssh-destination.yml`: environment-gated local-to-SSH publication example. - `s3-destination.yml`: environment-gated local-to-S3 publication example. diff --git a/docs/operations.md b/docs/operations.md index c4d5dec..7b6c20a 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -92,28 +92,29 @@ pipelines: - id: reports source: backend: http_upload - token_env: REPORTS_UPLOAD_TOKEN + token_env: DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN destinations: - id: archive backend: local path: /srv/reports/archive ``` -Create `/run/secrets/distributor/REPORTS_UPLOAD_TOKEN` or set the real process -environment variable before starting the server. Distributor does not read -literal upload tokens from YAML. +Create `/run/secrets/distributor/DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN` or set the +real process environment variable before starting the server. Distributor does +not read literal upload tokens from YAML. -Start the server: +Start the maintained local example: ```sh -go run ./cmd/distributor serve --config +DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN= \ + go run ./cmd/distributor serve --config examples/http-upload-local.yml ``` Submit a tar or tar.gz source bundle: ```sh curl -X POST http://127.0.0.1:8080/upload \ - -H "Authorization: Bearer $REPORTS_UPLOAD_TOKEN" \ + -H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \ -H "Content-Type: application/gzip" \ --data-binary @bundle.tar.gz ``` diff --git a/docs/roadmap/http.md b/docs/roadmap/http.md index 9b263ae..552140a 100644 --- a/docs/roadmap/http.md +++ b/docs/roadmap/http.md @@ -1,349 +1,60 @@ -# Roadmap: HTTP Upload API +# Roadmap: HTTP Upload Extensions ## Purpose -Add an HTTP upload API that lets producer applications push complete source -bundles into `distributor`. +The HTTP upload API is implemented. Current behavior is documented in: -The current application already supports local, SSH/SFTP, and S3-compatible -source and destination backends. Those source backends are pull-oriented: -`distributor` opens configured storage, discovers `manifest.json`, validates -bundles, and fans out selected outputs to configured destinations. +- [CLI](../cli.md) +- [Configuration](../config.md) +- [Operations](../operations.md) +- [Troubleshooting](../troubleshooting.md) +- [Application internals](../internal/app.md) +- [Ingestion internals](../internal/ingest.md) -`http_upload` is different because it is push-oriented. A producer sends one -complete bundle to `distributor`, and `distributor` stages that upload before -running normal validation and fan-out. The HTTP layer should be an ingestion -layer over the existing app, bundle, storage, publish, transform, link, state, -and notification behavior. +This roadmap records HTTP upload extensions that are intentionally not part of +the current implementation. -## Current Implementation Grounding +## Deferred Extensions -Implemented behavior already provides the core pieces this feature should reuse: +### Authentication -- source bundle validation through the bundle package; -- local, SSH/SFTP, and S3-compatible storage backends; -- destination fan-out through publish planning and execution; -- single-pipeline app execution through the app layer; -- structured run reports and JSON-capable CLI output; -- secrets-directory environment resolution for credential material. +- URL-token authentication for constrained clients. +- Additional token lifecycle tooling. +- Mutual TLS or other in-app identity mechanisms. -The HTTP implementation should not duplicate bundle validation or publication -logic. Once an upload is staged, it should proceed through the same validation -and fan-out behavior as any other source bundle. +### Archive Formats -## Accepted Direction +- Zstandard-compressed tar archives. +- Additional content negotiation rules for future archive formats. -Add `http_upload` as a source backend option for configured pipelines. +### Status And Queue Durability -An `http_upload` source is not a normal durable storage backend. It represents -an HTTP ingestion endpoint that receives an uploaded bundle, writes it into -pipeline-local staging storage, validates it, and then dispatches the existing -pipeline fan-out flow. +- Durable status persistence across process restarts. +- Database-backed queueing. +- Recovery semantics for queued or running uploads after a restart. -Accepted behavior: +### Producer Coordination -- producer applications upload a compliant source bundle manifest and all - referenced files; -- uploads are asynchronous; -- each accepted upload receives a generated run id and an initial `accepted` - status; -- clients can query run status later by run id; -- authentication uses a static token associated with the selected - `http_upload` pipeline; -- upload requests send that token with `Authorization: Bearer `; -- tokens may be supplied through `secrets.directory` using the existing - internal environment resolver; -- each `http_upload` pipeline has a configurable upload staging directory; -- default staging root is `/var/spool/distributor`; -- default pipeline staging directory is `/var/spool/distributor/`; -- default maximum upload size is 20 MB; -- upload archives may be uncompressed tar or gzip-compressed tar; -- the server must prevent multiple simultaneous active runs of the same - pipeline; -- the server uses an internal bounded queue and a global `max_concurrency` - setting for accepted uploads; -- completed status records and staged run directories use time-based retention; -- every accepted upload is a new run with a `distributor`-generated run id. +- Producer-supplied idempotency keys. +- Run retry endpoints. +- Run cancellation endpoints. +- Run listing endpoints. -## Configuration Shape +### Deployment Surface -Add `http_upload` as a source backend only. It should not be valid as a -destination backend. +- In-app TLS. +- Public exposure defaults. +- Browser UI. -Add top-level HTTP server configuration for cross-pipeline server behavior: +## Boundaries -```yaml -server: - http: - bind: 127.0.0.1:8080 - staging_root: /var/spool/distributor - max_upload_size: 20MB - queue_size: 16 - max_concurrency: 1 - retention: 24h -``` +Current HTTP upload behavior remains intentionally small: -Defaults: +- `http_upload` is source-only and is not a durable storage backend. +- Upload status is memory-only. +- Producers submit complete tar or gzip-compressed tar source bundles. +- Producers authenticate with `Authorization: Bearer `. +- Public access policy, TLS termination, and rate limiting belong outside + `distributor` unless a future roadmap explicitly changes that boundary. -- `bind`: `127.0.0.1:8080`; -- `staging_root`: `/var/spool/distributor`; -- `max_upload_size`: `20MB`; -- `queue_size`: `16`; -- `max_concurrency`: `1`; -- `retention`: `24h`. - -Pipeline shape: - -```yaml -pipelines: - - id: weather-daily - source: - backend: http_upload - token_env: WEATHER_DAILY_UPLOAD_TOKEN - staging_path: /var/spool/distributor/weather-daily - max_upload_size: 20MB - destinations: - - id: archive - backend: s3 - ... -``` - -`token_env` is required for the first implementation. Literal token values in -YAML are not supported. Real environment variables or `secrets.directory` files -provide the token without putting secret values in config. - -If `staging_path` is omitted, default it to: - -```text -/var/spool/distributor/ -``` - -If source-level `max_upload_size` is omitted, use the top-level -`server.http.max_upload_size` value. - -Server-level queue configuration is separate from per-pipeline source -configuration. Per-pipeline fields may override staging path and upload size; -bind address, queue size, concurrency, and retention are server-level settings. - -## Upload Model - -The uploaded request should contain one complete source bundle. - -Use archive upload rather than multipart fields for each file. `distributor` -should extract the archive into a per-run staging directory under the pipeline -staging path, then validate the extracted bundle. - -The first implementation supports: - -- uncompressed tar; -- gzip-compressed tar. - -The server should accept uncompressed tar as `application/x-tar`. It should -accept gzip-compressed tar as `application/gzip` or `application/x-gzip`. - -Archive extraction must be conservative: - -- reject absolute paths; -- reject path traversal and backslash paths; -- reject symlinks, hardlinks, devices, sockets, and other special entries; -- require exactly one root-level `manifest.json`; -- require all manifest-listed files to be present as regular files; -- reject extra nested manifests; -- enforce upload size limits before extraction; -- enforce extracted size and file-count limits after extraction; -- clean up failed extraction directories. - -After extraction, validation should use the existing source bundle validation -contract. A bundle that fails manifest, path, size, digest, or file validation -must not be published. - -## Async Run Flow - -HTTP upload processing should be asynchronous: - -1. Authenticate the request token and map it to exactly one `http_upload` - pipeline. -2. Admit or reject the request according to queue capacity and per-pipeline - active-run rules. -3. Create a run id and per-run staging directory. -4. Record initial in-memory run status as `accepted`. -5. Return an acceptance response without waiting for fan-out to complete. -6. In a worker, extract the archive, validate the staged bundle, and run the - pipeline fan-out using the staged bundle as the effective source. -7. Record final success or failure status. - -Initial acceptance response shape: - -```json -{ - "run_id": "weather-daily.20260603T120000Z.ab12cd34", - "status": "accepted" -} -``` - -Run ids should be generated by `distributor`. Use the pipeline id, a -filesystem-safe UTC timestamp, and a short random suffix to avoid collisions. - -Suggested status values: - -- `accepted`; -- `queued`; -- `running`; -- `succeeded`; -- `failed`; -- `expired`. - -Status records should include run id, pipeline id, status, timestamps, and the -completed run report or error details when available. Status records are kept -in memory for the first implementation and expire after the configured -retention period. - -Every accepted upload is treated as a new run. The first implementation does -not accept producer-supplied idempotency keys. - -## Queue And Concurrency - -The server must not run more than one upload-triggered execution for the same -pipeline at the same time. - -Use two controls: - -- a global worker limit, configured as `max_concurrency`; -- a bounded admission queue, configured as `queue_size`; -- a per-pipeline single-active-run guard. - -If a run for the same pipeline is already active, later accepted uploads for -that pipeline should wait in queue rather than starting concurrently. - -If the queue is full, the upload request should fail before consuming and -staging the request body. The server must not allow unbounded memory or disk -growth. - -## Authentication - -Each `http_upload` pipeline has one static upload token. - -Upload requests send the token in an HTTP bearer header: - -```http -Authorization: Bearer -``` - -Authentication maps the incoming bearer token to exactly one configured -pipeline. If no pipeline matches, the request fails. If more than one pipeline -resolves to the same token, config validation should fail before the server -starts. - -Secret values must never be logged, returned in responses, or included in run -status. - -## HTTP Server Boundary - -Add a server mode rather than trying to make `run` poll an HTTP source. - -The likely CLI shape is: - -```sh -distributor serve --config -``` - -The HTTP API should default to a private bind address. Operators that need -public access, TLS termination, or mTLS should place `distributor` behind a -reverse proxy or private network boundary unless a later roadmap explicitly -adds in-app TLS. - -The server should expose: - -- `POST /upload`; -- `GET /runs/`; -- `GET /healthz`. - -`GET /healthz` should return success after the server has loaded and validated -configuration and is ready to accept requests. - -## Relationship To Existing Architecture - -`http_upload` should reuse existing code paths after staging: - -- upload staging should produce a local staged bundle tree; -- staged bundle validation should use the existing bundle validation contract; -- fan-out should use the app-layer single-pipeline run behavior where possible; -- destination handling should remain backend-agnostic; -- publish, transform, link, state, and notification behavior should not know - that the source arrived over HTTP. - -If the current app-layer single-pipeline runner assumes it can open and walk the -configured source backend, the HTTP implementation should add a narrow app-layer -entry point for "run this pipeline using this already-staged source backend" -rather than bending `http_upload` into a fake durable storage backend. - -## Non-Goals - -The first HTTP upload implementation should not add: - -- destination-side HTTP upload; -- browser UI; -- producer execution; -- source manifest schema changes; -- warning-only digest mismatch behavior; -- durable database-backed queueing; -- producer-supplied idempotency keys; -- zstd archive support; -- URL-token authentication; -- public network exposure defaults; -- in-app TLS; -- general-purpose workflow orchestration. - -## Testing Expectations - -Suggested coverage: - -- config validation accepts `http_upload` sources and rejects `http_upload` - destinations; -- default staging path becomes `/var/spool/distributor/`; -- token env references resolve through real environment and `secrets.directory`; -- duplicate token values across pipelines fail validation; -- upload size limit defaults to 20 MB and is enforced; -- archive extraction rejects unsafe paths, symlinks, hardlinks, devices, - missing manifest, missing manifest-listed files, and nested manifests; -- valid uploaded bundles validate through the existing bundle contract; -- upload admission returns `accepted` and a generated run id; -- status lookup reports queued, running, succeeded, failed, and expired states; -- per-pipeline runs do not execute concurrently; -- global `max_concurrency` is honored; -- full queues reject uploads before request-body staging; -- `GET /runs/` returns in-memory status records until retention expiry; -- failed extraction and failed runs clean up or retain staging data according to - the configured retention policy; -- secret values never appear in logs, responses, or status records; -- normal local, SSH/SFTP, and S3 source behavior remains unchanged. - -## Documentation Updates After Implementation - -- Update `docs/config.md` with `http_upload` source fields and defaults. -- Update `docs/cli.md` with `serve` syntax and HTTP behavior. -- Update `docs/operations.md` with upload, queue, status, and staging - workflows. -- Update `docs/troubleshooting.md` for authentication, archive extraction, - validation, queue, and fan-out failures. -- Add examples only if they are secret-free and safe to run locally. -- Update internal docs for any new app, HTTP, queue, or ingestion packages. - -Keep this roadmap under `docs/roadmap/` until implemented. - -## Future Work - -The first implementation intentionally defers: - -- URL-token authentication; -- zstd-compressed archive support; -- durable status persistence across process restarts; -- database-backed queueing; -- producer-supplied idempotency keys; -- run listing, cancellation, or retry endpoints; -- in-app TLS; -- public network exposure defaults; -- browser UI. - -These items should remain out of current-behavior docs until a later roadmap -selects and specifies them. +Do not document deferred extensions as available outside `docs/roadmap/`. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index bd5819c..1262de8 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,328 +1,34 @@ -# HTTP Upload API Implementation Roadmap +# HTTP Upload Deferred Work ## Purpose -This roadmap is the canonical staged implementation plan for -`docs/roadmap/http.md`. - -The current application supports CLI-driven distribution using configured -local, SSH/SFTP, and S3-compatible source and destination backends. It does not -yet implement an HTTP server, a `serve` command, `http_upload` source -configuration, upload authentication, archive ingestion, async upload status, -or HTTP routes. - -Future, planned, or aspirational behavior belongs under `docs/roadmap/` until -it is implemented. Current-behavior docs must be updated only after the -corresponding stage is complete. - -## Implementation Principles - -- Treat `docs/roadmap/http.md` as the source of truth for accepted HTTP upload - policy. -- Prefer standard-library HTTP, tar, gzip, and sync primitives. -- Do not add external dependencies for the first HTTP upload implementation. -- Do not register `http_upload` as a durable storage backend. -- Reuse existing bundle validation, destination fan-out, transform, link, - state, notification, and run-report behavior after upload staging. -- Keep deferred items out of current-behavior docs: URL-token auth, zstd, - durable status, database queues, producer idempotency keys, run - cancellation/listing, in-app TLS, public exposure defaults, and browser UI. - -## Stage 1: HTTP Upload Configuration - -Goal: add config support for HTTP upload sources and server settings without -adding HTTP runtime behavior. - -Implementation scope: - -- add top-level `server.http` config with defaults: - - `bind: 127.0.0.1:8080`; - - `staging_root: /var/spool/distributor`; - - `max_upload_size: 20MB`; - - `queue_size: 16`; - - `max_concurrency: 1`; - - `retention: 24h`; -- add source-only `backend: http_upload`; -- add `http_upload` source fields: - - required `token_env`; - - optional `staging_path`; - - optional `max_upload_size`; -- default omitted `staging_path` to - `/var/spool/distributor/`; -- default omitted source `max_upload_size` to - `server.http.max_upload_size`; -- reject `http_upload` as a destination backend; -- parse sizes with `B`, `KB`, `MB`, and `GB` suffixes using 1024 multipliers; -- parse `retention` with `time.ParseDuration`; -- keep literal upload tokens out of YAML. - -Documentation updates after implementation: - -- update `docs/config.md` for implemented config fields and defaults; -- update `docs/internal/config.md` for config ownership and validation rules; -- do not document HTTP routes yet. - -Tests: - -- config loading accepts valid `server.http` and `http_upload` source config; -- defaults apply for bind, staging root, staging path, upload size, queue size, - concurrency, and retention; -- known-field decoding rejects unknown fields; -- invalid sizes, invalid durations, missing token env, missing pipeline ids, - duplicate pipeline ids, and `http_upload` destinations fail validation; -- existing local, SSH/SFTP, and S3 config behavior remains unchanged. - -Completion criteria: `go test ./internal/config` passes and no runtime code -attempts to execute `http_upload`. - -## Stage 2: Upload Archive Staging - -Goal: stage uploaded tar or tar.gz archives into a validated local source -bundle tree. - -Implementation scope: - -- add an internal ingestion package for upload archive staging; -- support uncompressed tar and gzip-compressed tar only; -- accept content types: - - `application/x-tar`; - - `application/gzip`; - - `application/x-gzip`; -- stream request bodies to a per-run archive or staging path while enforcing - max upload size; -- extract into a per-run staging directory under the pipeline staging path; -- reject absolute paths, path traversal, backslash paths, symlinks, hardlinks, - devices, sockets, and other special entries; -- require exactly one root-level `manifest.json`; -- reject nested manifests; -- require all manifest-listed files to exist as regular files; -- enforce extracted size and file-count limits; -- clean up failed extraction directories; -- validate staged bundles through the existing source bundle contract. - -Documentation updates after implementation: - -- update relevant `docs/internal/` docs for the new ingestion package; -- keep user-facing HTTP docs out of current-behavior docs until the server - stage is implemented. - -Tests: - -- valid tar and tar.gz uploads stage successfully; -- unsupported content types fail; -- max upload size is enforced while streaming; -- unsafe archive entries are rejected; -- missing manifest, nested manifest, missing listed file, digest mismatch, and - non-regular manifest-listed files fail validation; -- failed extraction cleans up staging data according to the package contract. - -Completion criteria: ingestion can produce a validated staged local bundle and -does not publish anything. - -## Stage 3: Staged Source Pipeline Execution - -Goal: run one configured pipeline using an already-staged local source bundle -root. - -Implementation scope: - -- add a narrow app-layer entry point for executing one pipeline with a staged - local source backend/root; -- bypass normal source backend opening only for this staged-source entry point; -- do not register `http_upload` as a normal `storage.Backend`; -- reuse existing destination fan-out, transform, link, state, notification, and - run-report behavior; -- ensure publish, transform, link, and state code do not know the source - arrived over HTTP. - -Documentation updates after implementation: - -- update `docs/internal/app.md` for the staged-source app entry point; -- update `docs/internal/bundle.md` only if source validation behavior changes. - -Tests: - -- staged valid bundles publish through configured local destinations; -- invalid staged bundles fail before destination writes; -- configured destination behavior for local, SSH/SFTP, and S3 remains - backend-agnostic; -- run reports match existing app report semantics. - -Completion criteria: app-level tests prove staged local bundles can run through -normal fan-out without HTTP server code. - -## Stage 4: Async Upload Queue And Status - -Goal: add in-memory async upload coordination, queueing, and status tracking. - -Implementation scope: - -- add an in-memory upload coordinator; -- generate run ids shaped like - `..`; -- use statuses: - - `accepted`; - - `queued`; - - `running`; - - `succeeded`; - - `failed`; - - `expired`; -- enforce global `max_concurrency`; -- enforce bounded `queue_size`; -- enforce one active running upload per pipeline; -- queue later accepted uploads for the same pipeline instead of running them - concurrently; -- reject uploads before consuming or staging the request body when the queue is - full; -- keep status in memory; -- apply time-based retention to completed status and staged run directories; -- treat every accepted upload as a new run; -- do not support producer-supplied idempotency keys. - -Documentation updates after implementation: - -- update internal docs for the upload coordinator; -- do not add user-facing HTTP docs until the server stage is implemented. - -Tests: - -- run id format includes pipeline id, filesystem-safe UTC timestamp, and random - suffix; -- queue size is bounded; -- full queues reject admission before staging; -- same-pipeline uploads serialize; -- different pipelines run concurrently up to `max_concurrency`; -- status transitions cover accepted, queued, running, succeeded, failed, and - expired; -- final status retains run report or error details until retention expiry; -- staging directories are retained or cleaned according to retention policy. - -Completion criteria: coordinator tests pass without starting an HTTP server. - -## Stage 5: HTTP Server And `serve` CLI - -Goal: expose the upload coordinator through HTTP and add the server CLI command. - -Implementation scope: - -- add `distributor serve --config `; -- load config and `secrets.directory` before starting the server; -- resolve each `http_upload` `token_env` through the config-owned environment - resolver; -- fail startup if any configured token is missing or duplicated; -- bind to `server.http.bind`, defaulting to `127.0.0.1:8080`; -- implement `POST /upload`; -- implement `GET /runs/`; -- implement `GET /healthz`; -- authenticate uploads with `Authorization: Bearer `; -- map each token to exactly one configured `http_upload` pipeline; -- do not require or accept a producer-submitted pipeline id; -- return `202 Accepted` with: - - ```json - {"run_id":"","status":"accepted"} - ``` - -- return `401` for missing or invalid bearer token; -- return `413` for oversized upload; -- return `415` for unsupported archive content type; -- return `503` for full queue; -- return `404` for unknown run status; -- never log or return secret token values. - -Documentation updates after implementation: - -- update `docs/cli.md` with `serve` syntax; -- update `docs/config.md` with HTTP upload source and server config; -- update `docs/troubleshooting.md` for startup, auth, upload, and status - failures; -- update relevant internal docs for HTTP server package boundaries. - -Tests: - -- CLI parsing recognizes `serve --config `; -- private bind default is applied; -- startup fails for missing or duplicate tokens; -- auth accepts valid bearer tokens and rejects missing or invalid tokens; -- routes return the expected status codes and JSON response shapes; -- health succeeds after configuration is loaded and the server is ready; -- responses, logs, and status records do not expose secret token values. - -Completion criteria: `httptest` route tests and CLI tests pass, and no current -local/SSH/S3 CLI behavior regresses. - -## Stage 6: End-To-End HTTP Upload Flow - -Goal: prove the complete async HTTP upload path publishes valid bundles and -rejects invalid ones safely. - -Implementation scope: - -- add integration-style tests using `httptest`; -- submit valid tar and tar.gz bundles; -- poll `GET /runs/` until completion; -- verify fan-out reaches configured local destinations; -- verify invalid archives fail without publishing; -- verify same-pipeline uploads serialize; -- verify different pipelines can run up to `max_concurrency`; -- verify status includes completed run report or error details. - -Documentation updates after implementation: - -- update `docs/operations.md` with an HTTP upload workflow; -- add safe local examples only if they are secret-free and testable. - -Tests: - -- valid upload returns `202 Accepted`, then `succeeded`; -- invalid archive returns an accepted run only when admission succeeds, then - transitions to `failed`; -- successful fan-out writes expected destination outputs and state; -- failed upload does not write destination outputs; -- same-pipeline and cross-pipeline concurrency follow configured policy; -- run focused package tests and `go test ./...`. - -Completion criteria: end-to-end HTTP upload tests pass and full test suite -passes. - -## Stage 7: Documentation And Roadmap Closeout - -Goal: document implemented HTTP upload behavior and remove completed roadmap -drift. - -Implementation scope: - -- update current-behavior docs for implemented HTTP upload behavior: - - `docs/config.md`; - - `docs/cli.md`; - - `docs/operations.md`; - - `docs/troubleshooting.md`; - - relevant `docs/internal/` docs; -- add only secret-free, safe local examples; -- keep deferred items out of current docs; -- remove or rewrite completed roadmap material once behavior is fully - documented. - -Deferred items: - -- URL-token authentication; -- zstd archive support; -- durable status persistence; -- database-backed queues; -- producer idempotency keys; -- run cancellation or listing; -- in-app TLS; -- public exposure defaults; -- browser UI. - -Tests: - -- documentation consistency checks show implemented HTTP behavior is no longer - described only as future work; -- current-behavior docs do not describe deferred behavior as available; -- examples are valid and secret-free; -- run focused tests for docs/examples backed by tests and `go test ./...` if - examples or behavior docs changed with code. - -Completion criteria: current docs describe implemented behavior accurately, and -`docs/roadmap/` contains only future or deferred HTTP work. +HTTP upload behavior is implemented and documented in the current-behavior +manuals: + +- [CLI](../cli.md) +- [Configuration](../config.md) +- [Operations](../operations.md) +- [Troubleshooting](../troubleshooting.md) +- [Application internals](../internal/app.md) +- [Configuration internals](../internal/config.md) +- [Ingestion internals](../internal/ingest.md) + +This file tracks only HTTP upload work that is not implemented. + +## Deferred Work + +- URL-token authentication. +- Zstandard-compressed archive support. +- Durable status persistence across process restarts. +- Database-backed queueing. +- Producer-supplied idempotency keys. +- Run listing, cancellation, and retry endpoints. +- In-app TLS. +- Public network exposure defaults. +- Browser UI. + +## Documentation Rule + +Deferred behavior belongs under `docs/roadmap/` until implemented. Current +behavior docs must describe only the active HTTP upload API, configuration, +operation, troubleshooting, and internal package contracts. diff --git a/examples/http-upload-local.yml b/examples/http-upload-local.yml new file mode 100644 index 0000000..eca21cc --- /dev/null +++ b/examples/http-upload-local.yml @@ -0,0 +1,24 @@ +# Local HTTP upload example. +# Set DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN in the process environment or provide a +# secrets-directory file with that name before running `distributor serve`. +server: + http: + bind: 127.0.0.1:8080 + staging_root: workspace/http-upload/staging + max_upload_size: 20MB + queue_size: 16 + max_concurrency: 1 + retention: 24h +pipelines: + - id: example-http-upload + source: + backend: http_upload + token_env: DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN + destinations: + - id: local-archive + backend: local + path: workspace/published/http-upload + publish: + source: true + html: false + diff --git a/internal/config/load_test.go b/internal/config/load_test.go index 213e805..f9dc3ba 100644 --- a/internal/config/load_test.go +++ b/internal/config/load_test.go @@ -718,6 +718,7 @@ func TestExampleConfigsLoad(t *testing.T) { "../../examples/local-index.yml", "../../examples/fan-out.yml", "../../examples/archive-and-latest.yml", + "../../examples/http-upload-local.yml", "../../examples/ssh-destination.yml", "../../examples/s3-destination.yml", } {