Add implementation roadmap for HTTP API

docs/roadmap/implementation.md

@@ -1,91 +1,328 @@
-# HTTP API Boundary Roadmap
+# HTTP Upload API Implementation Roadmap
 
 ## Purpose
 
-This roadmap records the accepted boundary for a future HTTP API. The current
-application exposes CLI commands and internal app-layer run contracts; it does
-not implement an HTTP server, HTTP routes, a `serve` command, app-level
-authentication, or in-app TLS.
+This roadmap is the canonical staged implementation plan for
+`docs/roadmap/http.md`.
 
-Implemented internal run contracts are documented in `docs/internal/app.md`.
-This file is the canonical home for future HTTP boundary decisions until an
-HTTP implementation roadmap replaces it.
+The current application supports CLI-driven distribution using configured
+local, SSH/SFTP, and S3-compatible source and destination backends. It does not
+yet implement an HTTP server, a `serve` command, `http_upload` source
+configuration, upload authentication, archive ingestion, async upload status,
+or HTTP routes.
 
-## Accepted Direction
+Future, planned, or aspirational behavior belongs under `docs/roadmap/` until
+it is implemented. Current-behavior docs must be updated only after the
+corresponding stage is complete.
 
-The future HTTP API should trigger configured distributor pipelines through the
-existing app-layer single-pipeline run path and in-memory coordinator.
+## Implementation Principles
 
-The HTTP API is intentionally narrow:
+- Treat `docs/roadmap/http.md` as the source of truth for accepted HTTP upload
+  policy.
+- Prefer standard-library HTTP, tar, gzip, and sync primitives.
+- Do not add external dependencies for the first HTTP upload implementation.
+- Do not register `http_upload` as a durable storage backend.
+- Reuse existing bundle validation, destination fan-out, transform, link,
+  state, notification, and run-report behavior after upload staging.
+- Keep deferred items out of current-behavior docs: URL-token auth, zstd,
+  durable status, database queues, producer idempotency keys, run
+  cancellation/listing, in-app TLS, public exposure defaults, and browser UI.
 
-- A trigger request accepts only a pipeline ID as application input.
-- A trigger request starts work asynchronously and returns a run ID after
-  admission.
-- Run status is read through a separate status endpoint keyed by run ID.
-- Status records expose run ID, pipeline ID, current status, timestamps, and
-  completed report or error details when available.
+## Stage 1: HTTP Upload Configuration
 
-The application remains a bundle distribution tool. The HTTP API must not turn
-`distributor` into a workflow engine, CMS, report generator, or public web
-authoring service.
+Goal: add config support for HTTP upload sources and server settings without
+adding HTTP runtime behavior.
 
-## Error Mapping
+Implementation scope:
 
-Future transport code should map app-layer errors without changing app-layer
-error ownership:
+- add top-level `server.http` config with defaults:
+  - `bind: 127.0.0.1:8080`;
+  - `staging_root: /var/spool/distributor`;
+  - `max_upload_size: 20MB`;
+  - `queue_size: 16`;
+  - `max_concurrency: 1`;
+  - `retention: 24h`;
+- add source-only `backend: http_upload`;
+- add `http_upload` source fields:
+  - required `token_env`;
+  - optional `staging_path`;
+  - optional `max_upload_size`;
+- default omitted `staging_path` to
+  `/var/spool/distributor/<pipeline_id>`;
+- default omitted source `max_upload_size` to
+  `server.http.max_upload_size`;
+- reject `http_upload` as a destination backend;
+- parse sizes with `B`, `KB`, `MB`, and `GB` suffixes using 1024 multipliers;
+- parse `retention` with `time.ParseDuration`;
+- keep literal upload tokens out of YAML.
 
-- Unknown pipeline IDs map to `404 Not Found`.
-- Duplicate in-flight runs for the same pipeline ID map to `409 Conflict`.
-- Validation, config, source, destination, publish, transform, and notification
-  errors map to transport errors according to their app-layer context.
+Documentation updates after implementation:
 
-Duplicate runs must not be queued. Run state remains in memory unless a later
-roadmap explicitly adds durable run storage.
+- update `docs/config.md` for implemented config fields and defaults;
+- update `docs/internal/config.md` for config ownership and validation rules;
+- do not document HTTP routes yet.
 
-## Context And Lifetime
+Tests:
 
-The request context guards admission. Once a run is admitted, execution is tied
-to the server or coordinator lifetime context rather than to the client request
-lifetime.
+- config loading accepts valid `server.http` and `http_upload` source config;
+- defaults apply for bind, staging root, staging path, upload size, queue size,
+  concurrency, and retention;
+- known-field decoding rejects unknown fields;
+- invalid sizes, invalid durations, missing token env, missing pipeline ids,
+  duplicate pipeline ids, and `http_upload` destinations fail validation;
+- existing local, SSH/SFTP, and S3 config behavior remains unchanged.
 
-This split allows a client disconnect or request timeout to stop waiting for
-admission without canceling a run that has already been accepted.
+Completion criteria: `go test ./internal/config` passes and no runtime code
+attempts to execute `http_upload`.
 
-## Security Boundary
+## Stage 2: Upload Archive Staging
 
-The first HTTP server should default to private binding, such as `127.0.0.1`.
-Operators should expose it through a reverse proxy, private network, or external
-mTLS when transport security or remote access is required.
+Goal: stage uploaded tar or tar.gz archives into a validated local source
+bundle tree.
 
-The first HTTP implementation should not include:
+Implementation scope:
 
-- bearer-token authentication;
-- in-app TLS configuration;
-- public-network exposure defaults.
+- add an internal ingestion package for upload archive staging;
+- support uncompressed tar and gzip-compressed tar only;
+- accept content types:
+  - `application/x-tar`;
+  - `application/gzip`;
+  - `application/x-gzip`;
+- stream request bodies to a per-run archive or staging path while enforcing
+  max upload size;
+- extract into a per-run staging directory under the pipeline staging path;
+- reject absolute paths, path traversal, backslash paths, symlinks, hardlinks,
+  devices, sockets, and other special entries;
+- require exactly one root-level `manifest.json`;
+- reject nested manifests;
+- require all manifest-listed files to exist as regular files;
+- enforce extracted size and file-count limits;
+- clean up failed extraction directories;
+- validate staged bundles through the existing source bundle contract.
 
-A later roadmap must explicitly change this security decision before any of
-those features are added.
+Documentation updates after implementation:
 
-## Non-Goals
+- update relevant `docs/internal/` docs for the new ingestion package;
+- keep user-facing HTTP docs out of current-behavior docs until the server
+  stage is implemented.
 
-The future HTTP API should not add:
+Tests:
 
-- public CLI flags for selecting one pipeline during `distributor run`;
-- a CLI framework;
-- a generic workflow engine;
-- plugin execution;
-- durable run storage;
-- app-level authentication;
-- in-app TLS.
+- valid tar and tar.gz uploads stage successfully;
+- unsupported content types fail;
+- max upload size is enforced while streaming;
+- unsafe archive entries are rejected;
+- missing manifest, nested manifest, missing listed file, digest mismatch, and
+  non-regular manifest-listed files fail validation;
+- failed extraction cleans up staging data according to the package contract.
 
-## Verification Expectations
+Completion criteria: ingestion can produce a validated staged local bundle and
+does not publish anything.
 
-Any later HTTP implementation should preserve existing CLI behavior and keep
-the app-layer run contracts tested. At minimum, it should cover:
+## Stage 3: Staged Source Pipeline Execution
 
-- trigger requests with known and unknown pipeline IDs;
-- duplicate in-flight trigger requests;
-- asynchronous acceptance and status lookup;
-- private bind defaults;
-- request-context admission behavior;
-- coordinator-lifetime run execution.
+Goal: run one configured pipeline using an already-staged local source bundle
+root.
+
+Implementation scope:
+
+- add a narrow app-layer entry point for executing one pipeline with a staged
+  local source backend/root;
+- bypass normal source backend opening only for this staged-source entry point;
+- do not register `http_upload` as a normal `storage.Backend`;
+- reuse existing destination fan-out, transform, link, state, notification, and
+  run-report behavior;
+- ensure publish, transform, link, and state code do not know the source
+  arrived over HTTP.
+
+Documentation updates after implementation:
+
+- update `docs/internal/app.md` for the staged-source app entry point;
+- update `docs/internal/bundle.md` only if source validation behavior changes.
+
+Tests:
+
+- staged valid bundles publish through configured local destinations;
+- invalid staged bundles fail before destination writes;
+- configured destination behavior for local, SSH/SFTP, and S3 remains
+  backend-agnostic;
+- run reports match existing app report semantics.
+
+Completion criteria: app-level tests prove staged local bundles can run through
+normal fan-out without HTTP server code.
+
+## Stage 4: Async Upload Queue And Status
+
+Goal: add in-memory async upload coordination, queueing, and status tracking.
+
+Implementation scope:
+
+- add an in-memory upload coordinator;
+- generate run ids shaped like
+  `<pipeline_id>.<utc_timestamp>.<random_suffix>`;
+- use statuses:
+  - `accepted`;
+  - `queued`;
+  - `running`;
+  - `succeeded`;
+  - `failed`;
+  - `expired`;
+- enforce global `max_concurrency`;
+- enforce bounded `queue_size`;
+- enforce one active running upload per pipeline;
+- queue later accepted uploads for the same pipeline instead of running them
+  concurrently;
+- reject uploads before consuming or staging the request body when the queue is
+  full;
+- keep status in memory;
+- apply time-based retention to completed status and staged run directories;
+- treat every accepted upload as a new run;
+- do not support producer-supplied idempotency keys.
+
+Documentation updates after implementation:
+
+- update internal docs for the upload coordinator;
+- do not add user-facing HTTP docs until the server stage is implemented.
+
+Tests:
+
+- run id format includes pipeline id, filesystem-safe UTC timestamp, and random
+  suffix;
+- queue size is bounded;
+- full queues reject admission before staging;
+- same-pipeline uploads serialize;
+- different pipelines run concurrently up to `max_concurrency`;
+- status transitions cover accepted, queued, running, succeeded, failed, and
+  expired;
+- final status retains run report or error details until retention expiry;
+- staging directories are retained or cleaned according to retention policy.
+
+Completion criteria: coordinator tests pass without starting an HTTP server.
+
+## Stage 5: HTTP Server And `serve` CLI
+
+Goal: expose the upload coordinator through HTTP and add the server CLI command.
+
+Implementation scope:
+
+- add `distributor serve --config <path>`;
+- load config and `secrets.directory` before starting the server;
+- resolve each `http_upload` `token_env` through the config-owned environment
+  resolver;
+- fail startup if any configured token is missing or duplicated;
+- bind to `server.http.bind`, defaulting to `127.0.0.1:8080`;
+- implement `POST /upload`;
+- implement `GET /runs/<run_id>`;
+- implement `GET /healthz`;
+- authenticate uploads with `Authorization: Bearer <token>`;
+- map each token to exactly one configured `http_upload` pipeline;
+- do not require or accept a producer-submitted pipeline id;
+- return `202 Accepted` with:
+
+  ```json
+  {"run_id":"<id>","status":"accepted"}
+  ```
+
+- return `401` for missing or invalid bearer token;
+- return `413` for oversized upload;
+- return `415` for unsupported archive content type;
+- return `503` for full queue;
+- return `404` for unknown run status;
+- never log or return secret token values.
+
+Documentation updates after implementation:
+
+- update `docs/cli.md` with `serve` syntax;
+- update `docs/config.md` with HTTP upload source and server config;
+- update `docs/troubleshooting.md` for startup, auth, upload, and status
+  failures;
+- update relevant internal docs for HTTP server package boundaries.
+
+Tests:
+
+- CLI parsing recognizes `serve --config <path>`;
+- private bind default is applied;
+- startup fails for missing or duplicate tokens;
+- auth accepts valid bearer tokens and rejects missing or invalid tokens;
+- routes return the expected status codes and JSON response shapes;
+- health succeeds after configuration is loaded and the server is ready;
+- responses, logs, and status records do not expose secret token values.
+
+Completion criteria: `httptest` route tests and CLI tests pass, and no current
+local/SSH/S3 CLI behavior regresses.
+
+## Stage 6: End-To-End HTTP Upload Flow
+
+Goal: prove the complete async HTTP upload path publishes valid bundles and
+rejects invalid ones safely.
+
+Implementation scope:
+
+- add integration-style tests using `httptest`;
+- submit valid tar and tar.gz bundles;
+- poll `GET /runs/<run_id>` until completion;
+- verify fan-out reaches configured local destinations;
+- verify invalid archives fail without publishing;
+- verify same-pipeline uploads serialize;
+- verify different pipelines can run up to `max_concurrency`;
+- verify status includes completed run report or error details.
+
+Documentation updates after implementation:
+
+- update `docs/operations.md` with an HTTP upload workflow;
+- add safe local examples only if they are secret-free and testable.
+
+Tests:
+
+- valid upload returns `202 Accepted`, then `succeeded`;
+- invalid archive returns an accepted run only when admission succeeds, then
+  transitions to `failed`;
+- successful fan-out writes expected destination outputs and state;
+- failed upload does not write destination outputs;
+- same-pipeline and cross-pipeline concurrency follow configured policy;
+- run focused package tests and `go test ./...`.
+
+Completion criteria: end-to-end HTTP upload tests pass and full test suite
+passes.
+
+## Stage 7: Documentation And Roadmap Closeout
+
+Goal: document implemented HTTP upload behavior and remove completed roadmap
+drift.
+
+Implementation scope:
+
+- update current-behavior docs for implemented HTTP upload behavior:
+  - `docs/config.md`;
+  - `docs/cli.md`;
+  - `docs/operations.md`;
+  - `docs/troubleshooting.md`;
+  - relevant `docs/internal/` docs;
+- add only secret-free, safe local examples;
+- keep deferred items out of current docs;
+- remove or rewrite completed roadmap material once behavior is fully
+  documented.
+
+Deferred items:
+
+- URL-token authentication;
+- zstd archive support;
+- durable status persistence;
+- database-backed queues;
+- producer idempotency keys;
+- run cancellation or listing;
+- in-app TLS;
+- public exposure defaults;
+- browser UI.
+
+Tests:
+
+- documentation consistency checks show implemented HTTP behavior is no longer
+  described only as future work;
+- current-behavior docs do not describe deferred behavior as available;
+- examples are valid and secret-free;
+- run focused tests for docs/examples backed by tests and `go test ./...` if
+  examples or behavior docs changed with code.
+
+Completion criteria: current docs describe implemented behavior accurately, and
+`docs/roadmap/` contains only future or deferred HTTP work.