distributor/docs/roadmap/implementation.md

# HTTP Upload API Implementation Roadmap

## Purpose

This roadmap is the canonical staged implementation plan for
`docs/roadmap/http.md`.

The current application supports CLI-driven distribution using configured
local, SSH/SFTP, and S3-compatible source and destination backends. It does not
yet implement an HTTP server, a `serve` command, `http_upload` source
configuration, upload authentication, archive ingestion, async upload status,
or HTTP routes.

Future, planned, or aspirational behavior belongs under `docs/roadmap/` until
it is implemented. Current-behavior docs must be updated only after the
corresponding stage is complete.

## Implementation Principles

- Treat `docs/roadmap/http.md` as the source of truth for accepted HTTP upload
  policy.
- Prefer standard-library HTTP, tar, gzip, and sync primitives.
- Do not add external dependencies for the first HTTP upload implementation.
- Do not register `http_upload` as a durable storage backend.
- Reuse existing bundle validation, destination fan-out, transform, link,
  state, notification, and run-report behavior after upload staging.
- Keep deferred items out of current-behavior docs: URL-token auth, zstd,
  durable status, database queues, producer idempotency keys, run
  cancellation/listing, in-app TLS, public exposure defaults, and browser UI.

## Stage 1: HTTP Upload Configuration

Goal: add config support for HTTP upload sources and server settings without
adding HTTP runtime behavior.

Implementation scope:

- add top-level `server.http` config with defaults:
  - `bind: 127.0.0.1:8080`;
  - `staging_root: /var/spool/distributor`;
  - `max_upload_size: 20MB`;
  - `queue_size: 16`;
  - `max_concurrency: 1`;
  - `retention: 24h`;
- add source-only `backend: http_upload`;
- add `http_upload` source fields:
  - required `token_env`;
  - optional `staging_path`;
  - optional `max_upload_size`;
- default omitted `staging_path` to
  `/var/spool/distributor/<pipeline_id>`;
- default omitted source `max_upload_size` to
  `server.http.max_upload_size`;
- reject `http_upload` as a destination backend;
- parse sizes with `B`, `KB`, `MB`, and `GB` suffixes using 1024 multipliers;
- parse `retention` with `time.ParseDuration`;
- keep literal upload tokens out of YAML.

Documentation updates after implementation:

- update `docs/config.md` for implemented config fields and defaults;
- update `docs/internal/config.md` for config ownership and validation rules;
- do not document HTTP routes yet.

Tests:

- config loading accepts valid `server.http` and `http_upload` source config;
- defaults apply for bind, staging root, staging path, upload size, queue size,
  concurrency, and retention;
- known-field decoding rejects unknown fields;
- invalid sizes, invalid durations, missing token env, missing pipeline ids,
  duplicate pipeline ids, and `http_upload` destinations fail validation;
- existing local, SSH/SFTP, and S3 config behavior remains unchanged.

Completion criteria: `go test ./internal/config` passes and no runtime code
attempts to execute `http_upload`.

## Stage 2: Upload Archive Staging

Goal: stage uploaded tar or tar.gz archives into a validated local source
bundle tree.

Implementation scope:

- add an internal ingestion package for upload archive staging;
- support uncompressed tar and gzip-compressed tar only;
- accept content types:
  - `application/x-tar`;
  - `application/gzip`;
  - `application/x-gzip`;
- stream request bodies to a per-run archive or staging path while enforcing
  max upload size;
- extract into a per-run staging directory under the pipeline staging path;
- reject absolute paths, path traversal, backslash paths, symlinks, hardlinks,
  devices, sockets, and other special entries;
- require exactly one root-level `manifest.json`;
- reject nested manifests;
- require all manifest-listed files to exist as regular files;
- enforce extracted size and file-count limits;
- clean up failed extraction directories;
- validate staged bundles through the existing source bundle contract.

Documentation updates after implementation:

- update relevant `docs/internal/` docs for the new ingestion package;
- keep user-facing HTTP docs out of current-behavior docs until the server
  stage is implemented.

Tests:

- valid tar and tar.gz uploads stage successfully;
- unsupported content types fail;
- max upload size is enforced while streaming;
- unsafe archive entries are rejected;
- missing manifest, nested manifest, missing listed file, digest mismatch, and
  non-regular manifest-listed files fail validation;
- failed extraction cleans up staging data according to the package contract.

Completion criteria: ingestion can produce a validated staged local bundle and
does not publish anything.

## Stage 3: Staged Source Pipeline Execution

Goal: run one configured pipeline using an already-staged local source bundle
root.

Implementation scope:

- add a narrow app-layer entry point for executing one pipeline with a staged
  local source backend/root;
- bypass normal source backend opening only for this staged-source entry point;
- do not register `http_upload` as a normal `storage.Backend`;
- reuse existing destination fan-out, transform, link, state, notification, and
  run-report behavior;
- ensure publish, transform, link, and state code do not know the source
  arrived over HTTP.

Documentation updates after implementation:

- update `docs/internal/app.md` for the staged-source app entry point;
- update `docs/internal/bundle.md` only if source validation behavior changes.

Tests:

- staged valid bundles publish through configured local destinations;
- invalid staged bundles fail before destination writes;
- configured destination behavior for local, SSH/SFTP, and S3 remains
  backend-agnostic;
- run reports match existing app report semantics.

Completion criteria: app-level tests prove staged local bundles can run through
normal fan-out without HTTP server code.

## Stage 4: Async Upload Queue And Status

Goal: add in-memory async upload coordination, queueing, and status tracking.

Implementation scope:

- add an in-memory upload coordinator;
- generate run ids shaped like
  `<pipeline_id>.<utc_timestamp>.<random_suffix>`;
- use statuses:
  - `accepted`;
  - `queued`;
  - `running`;
  - `succeeded`;
  - `failed`;
  - `expired`;
- enforce global `max_concurrency`;
- enforce bounded `queue_size`;
- enforce one active running upload per pipeline;
- queue later accepted uploads for the same pipeline instead of running them
  concurrently;
- reject uploads before consuming or staging the request body when the queue is
  full;
- keep status in memory;
- apply time-based retention to completed status and staged run directories;
- treat every accepted upload as a new run;
- do not support producer-supplied idempotency keys.

Documentation updates after implementation:

- update internal docs for the upload coordinator;
- do not add user-facing HTTP docs until the server stage is implemented.

Tests:

- run id format includes pipeline id, filesystem-safe UTC timestamp, and random
  suffix;
- queue size is bounded;
- full queues reject admission before staging;
- same-pipeline uploads serialize;
- different pipelines run concurrently up to `max_concurrency`;
- status transitions cover accepted, queued, running, succeeded, failed, and
  expired;
- final status retains run report or error details until retention expiry;
- staging directories are retained or cleaned according to retention policy.

Completion criteria: coordinator tests pass without starting an HTTP server.

## Stage 5: HTTP Server And `serve` CLI

Goal: expose the upload coordinator through HTTP and add the server CLI command.

Implementation scope:

- add `distributor serve --config <path>`;
- load config and `secrets.directory` before starting the server;
- resolve each `http_upload` `token_env` through the config-owned environment
  resolver;
- fail startup if any configured token is missing or duplicated;
- bind to `server.http.bind`, defaulting to `127.0.0.1:8080`;
- implement `POST /upload`;
- implement `GET /runs/<run_id>`;
- implement `GET /healthz`;
- authenticate uploads with `Authorization: Bearer <token>`;
- map each token to exactly one configured `http_upload` pipeline;
- do not require or accept a producer-submitted pipeline id;
- return `202 Accepted` with:

  ```json
  {"run_id":"<id>","status":"accepted"}
  ```

- return `401` for missing or invalid bearer token;
- return `413` for oversized upload;
- return `415` for unsupported archive content type;
- return `503` for full queue;
- return `404` for unknown run status;
- never log or return secret token values.

Documentation updates after implementation:

- update `docs/cli.md` with `serve` syntax;
- update `docs/config.md` with HTTP upload source and server config;
- update `docs/troubleshooting.md` for startup, auth, upload, and status
  failures;
- update relevant internal docs for HTTP server package boundaries.

Tests:

- CLI parsing recognizes `serve --config <path>`;
- private bind default is applied;
- startup fails for missing or duplicate tokens;
- auth accepts valid bearer tokens and rejects missing or invalid tokens;
- routes return the expected status codes and JSON response shapes;
- health succeeds after configuration is loaded and the server is ready;
- responses, logs, and status records do not expose secret token values.

Completion criteria: `httptest` route tests and CLI tests pass, and no current
local/SSH/S3 CLI behavior regresses.

## Stage 6: End-To-End HTTP Upload Flow

Goal: prove the complete async HTTP upload path publishes valid bundles and
rejects invalid ones safely.

Implementation scope:

- add integration-style tests using `httptest`;
- submit valid tar and tar.gz bundles;
- poll `GET /runs/<run_id>` until completion;
- verify fan-out reaches configured local destinations;
- verify invalid archives fail without publishing;
- verify same-pipeline uploads serialize;
- verify different pipelines can run up to `max_concurrency`;
- verify status includes completed run report or error details.

Documentation updates after implementation:

- update `docs/operations.md` with an HTTP upload workflow;
- add safe local examples only if they are secret-free and testable.

Tests:

- valid upload returns `202 Accepted`, then `succeeded`;
- invalid archive returns an accepted run only when admission succeeds, then
  transitions to `failed`;
- successful fan-out writes expected destination outputs and state;
- failed upload does not write destination outputs;
- same-pipeline and cross-pipeline concurrency follow configured policy;
- run focused package tests and `go test ./...`.

Completion criteria: end-to-end HTTP upload tests pass and full test suite
passes.

## Stage 7: Documentation And Roadmap Closeout

Goal: document implemented HTTP upload behavior and remove completed roadmap
drift.

Implementation scope:

- update current-behavior docs for implemented HTTP upload behavior:
  - `docs/config.md`;
  - `docs/cli.md`;
  - `docs/operations.md`;
  - `docs/troubleshooting.md`;
  - relevant `docs/internal/` docs;
- add only secret-free, safe local examples;
- keep deferred items out of current docs;
- remove or rewrite completed roadmap material once behavior is fully
  documented.

Deferred items:

- URL-token authentication;
- zstd archive support;
- durable status persistence;
- database-backed queues;
- producer idempotency keys;
- run cancellation or listing;
- in-app TLS;
- public exposure defaults;
- browser UI.

Tests:

- documentation consistency checks show implemented HTTP behavior is no longer
  described only as future work;
- current-behavior docs do not describe deferred behavior as available;
- examples are valid and secret-free;
- run focused tests for docs/examples backed by tests and `go test ./...` if
  examples or behavior docs changed with code.

Completion criteria: current docs describe implemented behavior accurately, and
`docs/roadmap/` contains only future or deferred HTTP work.