eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Close out HTTP upload documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-03 15:30:02 +00:00
parent 0f1ef9e622
commit dc1f1f11f9
8 changed files with 116 additions and 664 deletions
Download Patch File Download Diff File Expand all files Collapse all files

369
docs/roadmap/http.md
View File

@@ -1,349 +1,60 @@
# Roadmap: HTTP Upload API
# Roadmap: HTTP Upload Extensions
## Purpose
Add an HTTP upload API that lets producer applications push complete source
bundles into `distributor`.
The HTTP upload API is implemented. Current behavior is documented in:
The current application already supports local, SSH/SFTP, and S3-compatible
source and destination backends. Those source backends are pull-oriented:
`distributor` opens configured storage, discovers `manifest.json`, validates
bundles, and fans out selected outputs to configured destinations.
- [CLI](../cli.md)
- [Configuration](../config.md)
- [Operations](../operations.md)
- [Troubleshooting](../troubleshooting.md)
- [Application internals](../internal/app.md)
- [Ingestion internals](../internal/ingest.md)
`http_upload` is different because it is push-oriented. A producer sends one
complete bundle to `distributor`, and `distributor` stages that upload before
running normal validation and fan-out. The HTTP layer should be an ingestion
layer over the existing app, bundle, storage, publish, transform, link, state,
and notification behavior.
This roadmap records HTTP upload extensions that are intentionally not part of
the current implementation.
## Current Implementation Grounding
## Deferred Extensions
Implemented behavior already provides the core pieces this feature should reuse:
### Authentication
- source bundle validation through the bundle package;
- local, SSH/SFTP, and S3-compatible storage backends;
- destination fan-out through publish planning and execution;
- single-pipeline app execution through the app layer;
- structured run reports and JSON-capable CLI output;
- secrets-directory environment resolution for credential material.
- URL-token authentication for constrained clients.
- Additional token lifecycle tooling.
- Mutual TLS or other in-app identity mechanisms.
The HTTP implementation should not duplicate bundle validation or publication
logic. Once an upload is staged, it should proceed through the same validation
and fan-out behavior as any other source bundle.
### Archive Formats
## Accepted Direction
- Zstandard-compressed tar archives.
- Additional content negotiation rules for future archive formats.
Add `http_upload` as a source backend option for configured pipelines.
### Status And Queue Durability
An `http_upload` source is not a normal durable storage backend. It represents
an HTTP ingestion endpoint that receives an uploaded bundle, writes it into
pipeline-local staging storage, validates it, and then dispatches the existing
pipeline fan-out flow.
- Durable status persistence across process restarts.
- Database-backed queueing.
- Recovery semantics for queued or running uploads after a restart.
Accepted behavior:
### Producer Coordination
- producer applications upload a compliant source bundle manifest and all
referenced files;
- uploads are asynchronous;
- each accepted upload receives a generated run id and an initial `accepted`
status;
- clients can query run status later by run id;
- authentication uses a static token associated with the selected
`http_upload` pipeline;
- upload requests send that token with `Authorization: Bearer <token>`;
- tokens may be supplied through `secrets.directory` using the existing
internal environment resolver;
- each `http_upload` pipeline has a configurable upload staging directory;
- default staging root is `/var/spool/distributor`;
- default pipeline staging directory is `/var/spool/distributor/<pipeline_id>`;
- default maximum upload size is 20 MB;
- upload archives may be uncompressed tar or gzip-compressed tar;
- the server must prevent multiple simultaneous active runs of the same
pipeline;
- the server uses an internal bounded queue and a global `max_concurrency`
setting for accepted uploads;
- completed status records and staged run directories use time-based retention;
- every accepted upload is a new run with a `distributor`-generated run id.
- Producer-supplied idempotency keys.
- Run retry endpoints.
- Run cancellation endpoints.
- Run listing endpoints.
## Configuration Shape
### Deployment Surface
Add `http_upload` as a source backend only. It should not be valid as a
destination backend.
- In-app TLS.
- Public exposure defaults.
- Browser UI.
Add top-level HTTP server configuration for cross-pipeline server behavior:
## Boundaries
```yaml
server:
http:
bind: 127.0.0.1:8080
staging_root: /var/spool/distributor
max_upload_size: 20MB
queue_size: 16
max_concurrency: 1
retention: 24h
```
Current HTTP upload behavior remains intentionally small:
Defaults:
- `http_upload` is source-only and is not a durable storage backend.
- Upload status is memory-only.
- Producers submit complete tar or gzip-compressed tar source bundles.
- Producers authenticate with `Authorization: Bearer <token>`.
- Public access policy, TLS termination, and rate limiting belong outside
`distributor` unless a future roadmap explicitly changes that boundary.
- `bind`: `127.0.0.1:8080`;
- `staging_root`: `/var/spool/distributor`;
- `max_upload_size`: `20MB`;
- `queue_size`: `16`;
- `max_concurrency`: `1`;
- `retention`: `24h`.
Pipeline shape:
```yaml
pipelines:
- id: weather-daily
source:
backend: http_upload
token_env: WEATHER_DAILY_UPLOAD_TOKEN
staging_path: /var/spool/distributor/weather-daily
max_upload_size: 20MB
destinations:
- id: archive
backend: s3
...
```
`token_env` is required for the first implementation. Literal token values in
YAML are not supported. Real environment variables or `secrets.directory` files
provide the token without putting secret values in config.
If `staging_path` is omitted, default it to:
```text
/var/spool/distributor/<pipeline_id>
```
If source-level `max_upload_size` is omitted, use the top-level
`server.http.max_upload_size` value.
Server-level queue configuration is separate from per-pipeline source
configuration. Per-pipeline fields may override staging path and upload size;
bind address, queue size, concurrency, and retention are server-level settings.
## Upload Model
The uploaded request should contain one complete source bundle.
Use archive upload rather than multipart fields for each file. `distributor`
should extract the archive into a per-run staging directory under the pipeline
staging path, then validate the extracted bundle.
The first implementation supports:
- uncompressed tar;
- gzip-compressed tar.
The server should accept uncompressed tar as `application/x-tar`. It should
accept gzip-compressed tar as `application/gzip` or `application/x-gzip`.
Archive extraction must be conservative:
- reject absolute paths;
- reject path traversal and backslash paths;
- reject symlinks, hardlinks, devices, sockets, and other special entries;
- require exactly one root-level `manifest.json`;
- require all manifest-listed files to be present as regular files;
- reject extra nested manifests;
- enforce upload size limits before extraction;
- enforce extracted size and file-count limits after extraction;
- clean up failed extraction directories.
After extraction, validation should use the existing source bundle validation
contract. A bundle that fails manifest, path, size, digest, or file validation
must not be published.
## Async Run Flow
HTTP upload processing should be asynchronous:
1. Authenticate the request token and map it to exactly one `http_upload`
pipeline.
2. Admit or reject the request according to queue capacity and per-pipeline
active-run rules.
3. Create a run id and per-run staging directory.
4. Record initial in-memory run status as `accepted`.
5. Return an acceptance response without waiting for fan-out to complete.
6. In a worker, extract the archive, validate the staged bundle, and run the
pipeline fan-out using the staged bundle as the effective source.
7. Record final success or failure status.
Initial acceptance response shape:
```json
{
"run_id": "weather-daily.20260603T120000Z.ab12cd34",
"status": "accepted"
}
```
Run ids should be generated by `distributor`. Use the pipeline id, a
filesystem-safe UTC timestamp, and a short random suffix to avoid collisions.
Suggested status values:
- `accepted`;
- `queued`;
- `running`;
- `succeeded`;
- `failed`;
- `expired`.
Status records should include run id, pipeline id, status, timestamps, and the
completed run report or error details when available. Status records are kept
in memory for the first implementation and expire after the configured
retention period.
Every accepted upload is treated as a new run. The first implementation does
not accept producer-supplied idempotency keys.
## Queue And Concurrency
The server must not run more than one upload-triggered execution for the same
pipeline at the same time.
Use two controls:
- a global worker limit, configured as `max_concurrency`;
- a bounded admission queue, configured as `queue_size`;
- a per-pipeline single-active-run guard.
If a run for the same pipeline is already active, later accepted uploads for
that pipeline should wait in queue rather than starting concurrently.
If the queue is full, the upload request should fail before consuming and
staging the request body. The server must not allow unbounded memory or disk
growth.
## Authentication
Each `http_upload` pipeline has one static upload token.
Upload requests send the token in an HTTP bearer header:
```http
Authorization: Bearer <token>
```
Authentication maps the incoming bearer token to exactly one configured
pipeline. If no pipeline matches, the request fails. If more than one pipeline
resolves to the same token, config validation should fail before the server
starts.
Secret values must never be logged, returned in responses, or included in run
status.
## HTTP Server Boundary
Add a server mode rather than trying to make `run` poll an HTTP source.
The likely CLI shape is:
```sh
distributor serve --config <path>
```
The HTTP API should default to a private bind address. Operators that need
public access, TLS termination, or mTLS should place `distributor` behind a
reverse proxy or private network boundary unless a later roadmap explicitly
adds in-app TLS.
The server should expose:
- `POST /upload`;
- `GET /runs/<run_id>`;
- `GET /healthz`.
`GET /healthz` should return success after the server has loaded and validated
configuration and is ready to accept requests.
## Relationship To Existing Architecture
`http_upload` should reuse existing code paths after staging:
- upload staging should produce a local staged bundle tree;
- staged bundle validation should use the existing bundle validation contract;
- fan-out should use the app-layer single-pipeline run behavior where possible;
- destination handling should remain backend-agnostic;
- publish, transform, link, state, and notification behavior should not know
that the source arrived over HTTP.
If the current app-layer single-pipeline runner assumes it can open and walk the
configured source backend, the HTTP implementation should add a narrow app-layer
entry point for "run this pipeline using this already-staged source backend"
rather than bending `http_upload` into a fake durable storage backend.
## Non-Goals
The first HTTP upload implementation should not add:
- destination-side HTTP upload;
- browser UI;
- producer execution;
- source manifest schema changes;
- warning-only digest mismatch behavior;
- durable database-backed queueing;
- producer-supplied idempotency keys;
- zstd archive support;
- URL-token authentication;
- public network exposure defaults;
- in-app TLS;
- general-purpose workflow orchestration.
## Testing Expectations
Suggested coverage:
- config validation accepts `http_upload` sources and rejects `http_upload`
destinations;
- default staging path becomes `/var/spool/distributor/<pipeline_id>`;
- token env references resolve through real environment and `secrets.directory`;
- duplicate token values across pipelines fail validation;
- upload size limit defaults to 20 MB and is enforced;
- archive extraction rejects unsafe paths, symlinks, hardlinks, devices,
missing manifest, missing manifest-listed files, and nested manifests;
- valid uploaded bundles validate through the existing bundle contract;
- upload admission returns `accepted` and a generated run id;
- status lookup reports queued, running, succeeded, failed, and expired states;
- per-pipeline runs do not execute concurrently;
- global `max_concurrency` is honored;
- full queues reject uploads before request-body staging;
- `GET /runs/<run_id>` returns in-memory status records until retention expiry;
- failed extraction and failed runs clean up or retain staging data according to
the configured retention policy;
- secret values never appear in logs, responses, or status records;
- normal local, SSH/SFTP, and S3 source behavior remains unchanged.
## Documentation Updates After Implementation
- Update `docs/config.md` with `http_upload` source fields and defaults.
- Update `docs/cli.md` with `serve` syntax and HTTP behavior.
- Update `docs/operations.md` with upload, queue, status, and staging
workflows.
- Update `docs/troubleshooting.md` for authentication, archive extraction,
validation, queue, and fan-out failures.
- Add examples only if they are secret-free and safe to run locally.
- Update internal docs for any new app, HTTP, queue, or ingestion packages.
Keep this roadmap under `docs/roadmap/` until implemented.
## Future Work
The first implementation intentionally defers:
- URL-token authentication;
- zstd-compressed archive support;
- durable status persistence across process restarts;
- database-backed queueing;
- producer-supplied idempotency keys;
- run listing, cancellation, or retry endpoints;
- in-app TLS;
- public network exposure defaults;
- browser UI.
These items should remain out of current-behavior docs until a later roadmap
selects and specifies them.
Do not document deferred extensions as available outside `docs/roadmap/`.

356
docs/roadmap/implementation.md
View File

@@ -1,328 +1,34 @@
# HTTP Upload API Implementation Roadmap
# HTTP Upload Deferred Work
## Purpose
This roadmap is the canonical staged implementation plan for
`docs/roadmap/http.md`.
The current application supports CLI-driven distribution using configured
local, SSH/SFTP, and S3-compatible source and destination backends. It does not
yet implement an HTTP server, a `serve` command, `http_upload` source
configuration, upload authentication, archive ingestion, async upload status,
or HTTP routes.
Future, planned, or aspirational behavior belongs under `docs/roadmap/` until
it is implemented. Current-behavior docs must be updated only after the
corresponding stage is complete.
## Implementation Principles
- Treat `docs/roadmap/http.md` as the source of truth for accepted HTTP upload
policy.
- Prefer standard-library HTTP, tar, gzip, and sync primitives.
- Do not add external dependencies for the first HTTP upload implementation.
- Do not register `http_upload` as a durable storage backend.
- Reuse existing bundle validation, destination fan-out, transform, link,
state, notification, and run-report behavior after upload staging.
- Keep deferred items out of current-behavior docs: URL-token auth, zstd,
durable status, database queues, producer idempotency keys, run
cancellation/listing, in-app TLS, public exposure defaults, and browser UI.
## Stage 1: HTTP Upload Configuration
Goal: add config support for HTTP upload sources and server settings without
adding HTTP runtime behavior.
Implementation scope:
- add top-level `server.http` config with defaults:
- `bind: 127.0.0.1:8080`;
- `staging_root: /var/spool/distributor`;
- `max_upload_size: 20MB`;
- `queue_size: 16`;
- `max_concurrency: 1`;
- `retention: 24h`;
- add source-only `backend: http_upload`;
- add `http_upload` source fields:
- required `token_env`;
- optional `staging_path`;
- optional `max_upload_size`;
- default omitted `staging_path` to
`/var/spool/distributor/<pipeline_id>`;
- default omitted source `max_upload_size` to
`server.http.max_upload_size`;
- reject `http_upload` as a destination backend;
- parse sizes with `B`, `KB`, `MB`, and `GB` suffixes using 1024 multipliers;
- parse `retention` with `time.ParseDuration`;
- keep literal upload tokens out of YAML.
Documentation updates after implementation:
- update `docs/config.md` for implemented config fields and defaults;
- update `docs/internal/config.md` for config ownership and validation rules;
- do not document HTTP routes yet.
Tests:
- config loading accepts valid `server.http` and `http_upload` source config;
- defaults apply for bind, staging root, staging path, upload size, queue size,
concurrency, and retention;
- known-field decoding rejects unknown fields;
- invalid sizes, invalid durations, missing token env, missing pipeline ids,
duplicate pipeline ids, and `http_upload` destinations fail validation;
- existing local, SSH/SFTP, and S3 config behavior remains unchanged.
Completion criteria: `go test ./internal/config` passes and no runtime code
attempts to execute `http_upload`.
## Stage 2: Upload Archive Staging
Goal: stage uploaded tar or tar.gz archives into a validated local source
bundle tree.
Implementation scope:
- add an internal ingestion package for upload archive staging;
- support uncompressed tar and gzip-compressed tar only;
- accept content types:
- `application/x-tar`;
- `application/gzip`;
- `application/x-gzip`;
- stream request bodies to a per-run archive or staging path while enforcing
max upload size;
- extract into a per-run staging directory under the pipeline staging path;
- reject absolute paths, path traversal, backslash paths, symlinks, hardlinks,
devices, sockets, and other special entries;
- require exactly one root-level `manifest.json`;
- reject nested manifests;
- require all manifest-listed files to exist as regular files;
- enforce extracted size and file-count limits;
- clean up failed extraction directories;
- validate staged bundles through the existing source bundle contract.
Documentation updates after implementation:
- update relevant `docs/internal/` docs for the new ingestion package;
- keep user-facing HTTP docs out of current-behavior docs until the server
stage is implemented.
Tests:
- valid tar and tar.gz uploads stage successfully;
- unsupported content types fail;
- max upload size is enforced while streaming;
- unsafe archive entries are rejected;
- missing manifest, nested manifest, missing listed file, digest mismatch, and
non-regular manifest-listed files fail validation;
- failed extraction cleans up staging data according to the package contract.
Completion criteria: ingestion can produce a validated staged local bundle and
does not publish anything.
## Stage 3: Staged Source Pipeline Execution
Goal: run one configured pipeline using an already-staged local source bundle
root.
Implementation scope:
- add a narrow app-layer entry point for executing one pipeline with a staged
local source backend/root;
- bypass normal source backend opening only for this staged-source entry point;
- do not register `http_upload` as a normal `storage.Backend`;
- reuse existing destination fan-out, transform, link, state, notification, and
run-report behavior;
- ensure publish, transform, link, and state code do not know the source
arrived over HTTP.
Documentation updates after implementation:
- update `docs/internal/app.md` for the staged-source app entry point;
- update `docs/internal/bundle.md` only if source validation behavior changes.
Tests:
- staged valid bundles publish through configured local destinations;
- invalid staged bundles fail before destination writes;
- configured destination behavior for local, SSH/SFTP, and S3 remains
backend-agnostic;
- run reports match existing app report semantics.
Completion criteria: app-level tests prove staged local bundles can run through
normal fan-out without HTTP server code.
## Stage 4: Async Upload Queue And Status
Goal: add in-memory async upload coordination, queueing, and status tracking.
Implementation scope:
- add an in-memory upload coordinator;
- generate run ids shaped like
`<pipeline_id>.<utc_timestamp>.<random_suffix>`;
- use statuses:
- `accepted`;
- `queued`;
- `running`;
- `succeeded`;
- `failed`;
- `expired`;
- enforce global `max_concurrency`;
- enforce bounded `queue_size`;
- enforce one active running upload per pipeline;
- queue later accepted uploads for the same pipeline instead of running them
concurrently;
- reject uploads before consuming or staging the request body when the queue is
full;
- keep status in memory;
- apply time-based retention to completed status and staged run directories;
- treat every accepted upload as a new run;
- do not support producer-supplied idempotency keys.
Documentation updates after implementation:
- update internal docs for the upload coordinator;
- do not add user-facing HTTP docs until the server stage is implemented.
Tests:
- run id format includes pipeline id, filesystem-safe UTC timestamp, and random
suffix;
- queue size is bounded;
- full queues reject admission before staging;
- same-pipeline uploads serialize;
- different pipelines run concurrently up to `max_concurrency`;
- status transitions cover accepted, queued, running, succeeded, failed, and
expired;
- final status retains run report or error details until retention expiry;
- staging directories are retained or cleaned according to retention policy.
Completion criteria: coordinator tests pass without starting an HTTP server.
## Stage 5: HTTP Server And `serve` CLI
Goal: expose the upload coordinator through HTTP and add the server CLI command.
Implementation scope:
- add `distributor serve --config <path>`;
- load config and `secrets.directory` before starting the server;
- resolve each `http_upload` `token_env` through the config-owned environment
resolver;
- fail startup if any configured token is missing or duplicated;
- bind to `server.http.bind`, defaulting to `127.0.0.1:8080`;
- implement `POST /upload`;
- implement `GET /runs/<run_id>`;
- implement `GET /healthz`;
- authenticate uploads with `Authorization: Bearer <token>`;
- map each token to exactly one configured `http_upload` pipeline;
- do not require or accept a producer-submitted pipeline id;
- return `202 Accepted` with:
```json
{"run_id":"<id>","status":"accepted"}
```
- return `401` for missing or invalid bearer token;
- return `413` for oversized upload;
- return `415` for unsupported archive content type;
- return `503` for full queue;
- return `404` for unknown run status;
- never log or return secret token values.
Documentation updates after implementation:
- update `docs/cli.md` with `serve` syntax;
- update `docs/config.md` with HTTP upload source and server config;
- update `docs/troubleshooting.md` for startup, auth, upload, and status
failures;
- update relevant internal docs for HTTP server package boundaries.
Tests:
- CLI parsing recognizes `serve --config <path>`;
- private bind default is applied;
- startup fails for missing or duplicate tokens;
- auth accepts valid bearer tokens and rejects missing or invalid tokens;
- routes return the expected status codes and JSON response shapes;
- health succeeds after configuration is loaded and the server is ready;
- responses, logs, and status records do not expose secret token values.
Completion criteria: `httptest` route tests and CLI tests pass, and no current
local/SSH/S3 CLI behavior regresses.
## Stage 6: End-To-End HTTP Upload Flow
Goal: prove the complete async HTTP upload path publishes valid bundles and
rejects invalid ones safely.
Implementation scope:
- add integration-style tests using `httptest`;
- submit valid tar and tar.gz bundles;
- poll `GET /runs/<run_id>` until completion;
- verify fan-out reaches configured local destinations;
- verify invalid archives fail without publishing;
- verify same-pipeline uploads serialize;
- verify different pipelines can run up to `max_concurrency`;
- verify status includes completed run report or error details.
Documentation updates after implementation:
- update `docs/operations.md` with an HTTP upload workflow;
- add safe local examples only if they are secret-free and testable.
Tests:
- valid upload returns `202 Accepted`, then `succeeded`;
- invalid archive returns an accepted run only when admission succeeds, then
transitions to `failed`;
- successful fan-out writes expected destination outputs and state;
- failed upload does not write destination outputs;
- same-pipeline and cross-pipeline concurrency follow configured policy;
- run focused package tests and `go test ./...`.
Completion criteria: end-to-end HTTP upload tests pass and full test suite
passes.
## Stage 7: Documentation And Roadmap Closeout
Goal: document implemented HTTP upload behavior and remove completed roadmap
drift.
Implementation scope:
- update current-behavior docs for implemented HTTP upload behavior:
- `docs/config.md`;
- `docs/cli.md`;
- `docs/operations.md`;
- `docs/troubleshooting.md`;
- relevant `docs/internal/` docs;
- add only secret-free, safe local examples;
- keep deferred items out of current docs;
- remove or rewrite completed roadmap material once behavior is fully
documented.
Deferred items:
- URL-token authentication;
- zstd archive support;
- durable status persistence;
- database-backed queues;
- producer idempotency keys;
- run cancellation or listing;
- in-app TLS;
- public exposure defaults;
- browser UI.
Tests:
- documentation consistency checks show implemented HTTP behavior is no longer
described only as future work;
- current-behavior docs do not describe deferred behavior as available;
- examples are valid and secret-free;
- run focused tests for docs/examples backed by tests and `go test ./...` if
examples or behavior docs changed with code.
Completion criteria: current docs describe implemented behavior accurately, and
`docs/roadmap/` contains only future or deferred HTTP work.
HTTP upload behavior is implemented and documented in the current-behavior
manuals:
- [CLI](../cli.md)
- [Configuration](../config.md)
- [Operations](../operations.md)
- [Troubleshooting](../troubleshooting.md)
- [Application internals](../internal/app.md)
- [Configuration internals](../internal/config.md)
- [Ingestion internals](../internal/ingest.md)
This file tracks only HTTP upload work that is not implemented.
## Deferred Work
- URL-token authentication.
- Zstandard-compressed archive support.
- Durable status persistence across process restarts.
- Database-backed queueing.
- Producer-supplied idempotency keys.
- Run listing, cancellation, and retry endpoints.
- In-app TLS.
- Public network exposure defaults.
- Browser UI.
## Documentation Rule
Deferred behavior belongs under `docs/roadmap/` until implemented. Current
behavior docs must describe only the active HTTP upload API, configuration,
operation, troubleshooting, and internal package contracts.