distributor/docs/roadmap/implementation.md

HTTP Upload API Implementation Roadmap

Purpose

This roadmap is the canonical staged implementation plan for docs/roadmap/http.md.

The current application supports CLI-driven distribution using configured local, SSH/SFTP, and S3-compatible source and destination backends. It does not yet implement an HTTP server, a serve command, http_upload source configuration, upload authentication, archive ingestion, async upload status, or HTTP routes.

Future, planned, or aspirational behavior belongs under docs/roadmap/ until it is implemented. Current-behavior docs must be updated only after the corresponding stage is complete.

Implementation Principles

• Treat docs/roadmap/http.md as the source of truth for accepted HTTP upload policy.
• Prefer standard-library HTTP, tar, gzip, and sync primitives.
• Do not add external dependencies for the first HTTP upload implementation.
• Do not register http_upload as a durable storage backend.
• Reuse existing bundle validation, destination fan-out, transform, link, state, notification, and run-report behavior after upload staging.
• Keep deferred items out of current-behavior docs: URL-token auth, zstd, durable status, database queues, producer idempotency keys, run cancellation/listing, in-app TLS, public exposure defaults, and browser UI.

Stage 1: HTTP Upload Configuration

Goal: add config support for HTTP upload sources and server settings without adding HTTP runtime behavior.

Implementation scope:

• add top-level server.http config with defaults:
  • bind: 127.0.0.1:8080;
  • staging_root: /var/spool/distributor;
  • max_upload_size: 20MB;
  • queue_size: 16;
  • max_concurrency: 1;
  • retention: 24h;
• add source-only backend: http_upload;
• add http_upload source fields:
  • required token_env;
  • optional staging_path;
  • optional max_upload_size;
• default omitted staging_path to /var/spool/distributor/<pipeline_id>;
• default omitted source max_upload_size to server.http.max_upload_size;
• reject http_upload as a destination backend;
• parse sizes with B, KB, MB, and GB suffixes using 1024 multipliers;
• parse retention with time.ParseDuration;
• keep literal upload tokens out of YAML.

Documentation updates after implementation:

• update docs/config.md for implemented config fields and defaults;
• update docs/internal/config.md for config ownership and validation rules;
• do not document HTTP routes yet.

Tests:

• config loading accepts valid server.http and http_upload source config;
• defaults apply for bind, staging root, staging path, upload size, queue size, concurrency, and retention;
• known-field decoding rejects unknown fields;
• invalid sizes, invalid durations, missing token env, missing pipeline ids, duplicate pipeline ids, and http_upload destinations fail validation;
• existing local, SSH/SFTP, and S3 config behavior remains unchanged.

Completion criteria: go test ./internal/config passes and no runtime code attempts to execute http_upload.

Stage 2: Upload Archive Staging

Goal: stage uploaded tar or tar.gz archives into a validated local source bundle tree.

Implementation scope:

• add an internal ingestion package for upload archive staging;
• support uncompressed tar and gzip-compressed tar only;
• accept content types:
  • application/x-tar;
  • application/gzip;
  • application/x-gzip;
• stream request bodies to a per-run archive or staging path while enforcing max upload size;
• extract into a per-run staging directory under the pipeline staging path;
• reject absolute paths, path traversal, backslash paths, symlinks, hardlinks, devices, sockets, and other special entries;
• require exactly one root-level manifest.json;
• reject nested manifests;
• require all manifest-listed files to exist as regular files;
• enforce extracted size and file-count limits;
• clean up failed extraction directories;
• validate staged bundles through the existing source bundle contract.

Documentation updates after implementation:

• update relevant docs/internal/ docs for the new ingestion package;
• keep user-facing HTTP docs out of current-behavior docs until the server stage is implemented.

Tests:

• valid tar and tar.gz uploads stage successfully;
• unsupported content types fail;
• max upload size is enforced while streaming;
• unsafe archive entries are rejected;
• missing manifest, nested manifest, missing listed file, digest mismatch, and non-regular manifest-listed files fail validation;
• failed extraction cleans up staging data according to the package contract.

Completion criteria: ingestion can produce a validated staged local bundle and does not publish anything.

Stage 3: Staged Source Pipeline Execution

Goal: run one configured pipeline using an already-staged local source bundle root.

Implementation scope:

• add a narrow app-layer entry point for executing one pipeline with a staged local source backend/root;
• bypass normal source backend opening only for this staged-source entry point;
• do not register http_upload as a normal storage.Backend;
• reuse existing destination fan-out, transform, link, state, notification, and run-report behavior;
• ensure publish, transform, link, and state code do not know the source arrived over HTTP.

Documentation updates after implementation:

• update docs/internal/app.md for the staged-source app entry point;
• update docs/internal/bundle.md only if source validation behavior changes.

Tests:

• staged valid bundles publish through configured local destinations;
• invalid staged bundles fail before destination writes;
• configured destination behavior for local, SSH/SFTP, and S3 remains backend-agnostic;
• run reports match existing app report semantics.

Completion criteria: app-level tests prove staged local bundles can run through normal fan-out without HTTP server code.

Stage 4: Async Upload Queue And Status

Goal: add in-memory async upload coordination, queueing, and status tracking.

Implementation scope:

• add an in-memory upload coordinator;
• generate run ids shaped like <pipeline_id>.<utc_timestamp>.<random_suffix>;
• use statuses:
  • accepted;
  • queued;
  • running;
  • succeeded;
  • failed;
  • expired;
• enforce global max_concurrency;
• enforce bounded queue_size;
• enforce one active running upload per pipeline;
• queue later accepted uploads for the same pipeline instead of running them concurrently;
• reject uploads before consuming or staging the request body when the queue is full;
• keep status in memory;
• apply time-based retention to completed status and staged run directories;
• treat every accepted upload as a new run;
• do not support producer-supplied idempotency keys.

Documentation updates after implementation:

• update internal docs for the upload coordinator;
• do not add user-facing HTTP docs until the server stage is implemented.

Tests:

• run id format includes pipeline id, filesystem-safe UTC timestamp, and random suffix;
• queue size is bounded;
• full queues reject admission before staging;
• same-pipeline uploads serialize;
• different pipelines run concurrently up to max_concurrency;
• status transitions cover accepted, queued, running, succeeded, failed, and expired;
• final status retains run report or error details until retention expiry;
• staging directories are retained or cleaned according to retention policy.

Completion criteria: coordinator tests pass without starting an HTTP server.

Stage 5: HTTP Server And serve CLI

Goal: expose the upload coordinator through HTTP and add the server CLI command.

Implementation scope:

• add distributor serve --config <path>;

• load config and secrets.directory before starting the server;

• resolve each http_upload token_env through the config-owned environment resolver;

• fail startup if any configured token is missing or duplicated;

• bind to server.http.bind, defaulting to 127.0.0.1:8080;

• implement POST /upload;

• implement GET /runs/<run_id>;

• implement GET /healthz;

• authenticate uploads with Authorization: Bearer <token>;

• map each token to exactly one configured http_upload pipeline;

• do not require or accept a producer-submitted pipeline id;

• return 202 Accepted with:

  {"run_id":"<id>","status":"accepted"}
  

• return 401 for missing or invalid bearer token;

• return 413 for oversized upload;

• return 415 for unsupported archive content type;

• return 503 for full queue;

• return 404 for unknown run status;

• never log or return secret token values.

Documentation updates after implementation:

• update docs/cli.md with serve syntax;
• update docs/config.md with HTTP upload source and server config;
• update docs/troubleshooting.md for startup, auth, upload, and status failures;
• update relevant internal docs for HTTP server package boundaries.

Tests:

• CLI parsing recognizes serve --config <path>;
• private bind default is applied;
• startup fails for missing or duplicate tokens;
• auth accepts valid bearer tokens and rejects missing or invalid tokens;
• routes return the expected status codes and JSON response shapes;
• health succeeds after configuration is loaded and the server is ready;
• responses, logs, and status records do not expose secret token values.

Completion criteria: httptest route tests and CLI tests pass, and no current local/SSH/S3 CLI behavior regresses.

Stage 6: End-To-End HTTP Upload Flow

Goal: prove the complete async HTTP upload path publishes valid bundles and rejects invalid ones safely.

Implementation scope:

• add integration-style tests using httptest;
• submit valid tar and tar.gz bundles;
• poll GET /runs/<run_id> until completion;
• verify fan-out reaches configured local destinations;
• verify invalid archives fail without publishing;
• verify same-pipeline uploads serialize;
• verify different pipelines can run up to max_concurrency;
• verify status includes completed run report or error details.

Documentation updates after implementation:

• update docs/operations.md with an HTTP upload workflow;
• add safe local examples only if they are secret-free and testable.

Tests:

• valid upload returns 202 Accepted, then succeeded;
• invalid archive returns an accepted run only when admission succeeds, then transitions to failed;
• successful fan-out writes expected destination outputs and state;
• failed upload does not write destination outputs;
• same-pipeline and cross-pipeline concurrency follow configured policy;
• run focused package tests and go test ./....

Completion criteria: end-to-end HTTP upload tests pass and full test suite passes.

Stage 7: Documentation And Roadmap Closeout

Goal: document implemented HTTP upload behavior and remove completed roadmap drift.

Implementation scope:

• update current-behavior docs for implemented HTTP upload behavior:
  • docs/config.md;
  • docs/cli.md;
  • docs/operations.md;
  • docs/troubleshooting.md;
  • relevant docs/internal/ docs;
• add only secret-free, safe local examples;
• keep deferred items out of current docs;
• remove or rewrite completed roadmap material once behavior is fully documented.

Deferred items:

• URL-token authentication;
• zstd archive support;
• durable status persistence;
• database-backed queues;
• producer idempotency keys;
• run cancellation or listing;
• in-app TLS;
• public exposure defaults;
• browser UI.

Tests:

• documentation consistency checks show implemented HTTP behavior is no longer described only as future work;
• current-behavior docs do not describe deferred behavior as available;
• examples are valid and secret-free;
• run focused tests for docs/examples backed by tests and go test ./... if examples or behavior docs changed with code.

Completion criteria: current docs describe implemented behavior accurately, and docs/roadmap/ contains only future or deferred HTTP work.