diff --git a/docs/roadmap/http.md b/docs/roadmap/http.md new file mode 100644 index 0000000..9b263ae --- /dev/null +++ b/docs/roadmap/http.md @@ -0,0 +1,349 @@ +# Roadmap: HTTP Upload API + +## Purpose + +Add an HTTP upload API that lets producer applications push complete source +bundles into `distributor`. + +The current application already supports local, SSH/SFTP, and S3-compatible +source and destination backends. Those source backends are pull-oriented: +`distributor` opens configured storage, discovers `manifest.json`, validates +bundles, and fans out selected outputs to configured destinations. + +`http_upload` is different because it is push-oriented. A producer sends one +complete bundle to `distributor`, and `distributor` stages that upload before +running normal validation and fan-out. The HTTP layer should be an ingestion +layer over the existing app, bundle, storage, publish, transform, link, state, +and notification behavior. + +## Current Implementation Grounding + +Implemented behavior already provides the core pieces this feature should reuse: + +- source bundle validation through the bundle package; +- local, SSH/SFTP, and S3-compatible storage backends; +- destination fan-out through publish planning and execution; +- single-pipeline app execution through the app layer; +- structured run reports and JSON-capable CLI output; +- secrets-directory environment resolution for credential material. + +The HTTP implementation should not duplicate bundle validation or publication +logic. Once an upload is staged, it should proceed through the same validation +and fan-out behavior as any other source bundle. + +## Accepted Direction + +Add `http_upload` as a source backend option for configured pipelines. + +An `http_upload` source is not a normal durable storage backend. It represents +an HTTP ingestion endpoint that receives an uploaded bundle, writes it into +pipeline-local staging storage, validates it, and then dispatches the existing +pipeline fan-out flow. + +Accepted behavior: + +- producer applications upload a compliant source bundle manifest and all + referenced files; +- uploads are asynchronous; +- each accepted upload receives a generated run id and an initial `accepted` + status; +- clients can query run status later by run id; +- authentication uses a static token associated with the selected + `http_upload` pipeline; +- upload requests send that token with `Authorization: Bearer `; +- tokens may be supplied through `secrets.directory` using the existing + internal environment resolver; +- each `http_upload` pipeline has a configurable upload staging directory; +- default staging root is `/var/spool/distributor`; +- default pipeline staging directory is `/var/spool/distributor/`; +- default maximum upload size is 20 MB; +- upload archives may be uncompressed tar or gzip-compressed tar; +- the server must prevent multiple simultaneous active runs of the same + pipeline; +- the server uses an internal bounded queue and a global `max_concurrency` + setting for accepted uploads; +- completed status records and staged run directories use time-based retention; +- every accepted upload is a new run with a `distributor`-generated run id. + +## Configuration Shape + +Add `http_upload` as a source backend only. It should not be valid as a +destination backend. + +Add top-level HTTP server configuration for cross-pipeline server behavior: + +```yaml +server: + http: + bind: 127.0.0.1:8080 + staging_root: /var/spool/distributor + max_upload_size: 20MB + queue_size: 16 + max_concurrency: 1 + retention: 24h +``` + +Defaults: + +- `bind`: `127.0.0.1:8080`; +- `staging_root`: `/var/spool/distributor`; +- `max_upload_size`: `20MB`; +- `queue_size`: `16`; +- `max_concurrency`: `1`; +- `retention`: `24h`. + +Pipeline shape: + +```yaml +pipelines: + - id: weather-daily + source: + backend: http_upload + token_env: WEATHER_DAILY_UPLOAD_TOKEN + staging_path: /var/spool/distributor/weather-daily + max_upload_size: 20MB + destinations: + - id: archive + backend: s3 + ... +``` + +`token_env` is required for the first implementation. Literal token values in +YAML are not supported. Real environment variables or `secrets.directory` files +provide the token without putting secret values in config. + +If `staging_path` is omitted, default it to: + +```text +/var/spool/distributor/ +``` + +If source-level `max_upload_size` is omitted, use the top-level +`server.http.max_upload_size` value. + +Server-level queue configuration is separate from per-pipeline source +configuration. Per-pipeline fields may override staging path and upload size; +bind address, queue size, concurrency, and retention are server-level settings. + +## Upload Model + +The uploaded request should contain one complete source bundle. + +Use archive upload rather than multipart fields for each file. `distributor` +should extract the archive into a per-run staging directory under the pipeline +staging path, then validate the extracted bundle. + +The first implementation supports: + +- uncompressed tar; +- gzip-compressed tar. + +The server should accept uncompressed tar as `application/x-tar`. It should +accept gzip-compressed tar as `application/gzip` or `application/x-gzip`. + +Archive extraction must be conservative: + +- reject absolute paths; +- reject path traversal and backslash paths; +- reject symlinks, hardlinks, devices, sockets, and other special entries; +- require exactly one root-level `manifest.json`; +- require all manifest-listed files to be present as regular files; +- reject extra nested manifests; +- enforce upload size limits before extraction; +- enforce extracted size and file-count limits after extraction; +- clean up failed extraction directories. + +After extraction, validation should use the existing source bundle validation +contract. A bundle that fails manifest, path, size, digest, or file validation +must not be published. + +## Async Run Flow + +HTTP upload processing should be asynchronous: + +1. Authenticate the request token and map it to exactly one `http_upload` + pipeline. +2. Admit or reject the request according to queue capacity and per-pipeline + active-run rules. +3. Create a run id and per-run staging directory. +4. Record initial in-memory run status as `accepted`. +5. Return an acceptance response without waiting for fan-out to complete. +6. In a worker, extract the archive, validate the staged bundle, and run the + pipeline fan-out using the staged bundle as the effective source. +7. Record final success or failure status. + +Initial acceptance response shape: + +```json +{ + "run_id": "weather-daily.20260603T120000Z.ab12cd34", + "status": "accepted" +} +``` + +Run ids should be generated by `distributor`. Use the pipeline id, a +filesystem-safe UTC timestamp, and a short random suffix to avoid collisions. + +Suggested status values: + +- `accepted`; +- `queued`; +- `running`; +- `succeeded`; +- `failed`; +- `expired`. + +Status records should include run id, pipeline id, status, timestamps, and the +completed run report or error details when available. Status records are kept +in memory for the first implementation and expire after the configured +retention period. + +Every accepted upload is treated as a new run. The first implementation does +not accept producer-supplied idempotency keys. + +## Queue And Concurrency + +The server must not run more than one upload-triggered execution for the same +pipeline at the same time. + +Use two controls: + +- a global worker limit, configured as `max_concurrency`; +- a bounded admission queue, configured as `queue_size`; +- a per-pipeline single-active-run guard. + +If a run for the same pipeline is already active, later accepted uploads for +that pipeline should wait in queue rather than starting concurrently. + +If the queue is full, the upload request should fail before consuming and +staging the request body. The server must not allow unbounded memory or disk +growth. + +## Authentication + +Each `http_upload` pipeline has one static upload token. + +Upload requests send the token in an HTTP bearer header: + +```http +Authorization: Bearer +``` + +Authentication maps the incoming bearer token to exactly one configured +pipeline. If no pipeline matches, the request fails. If more than one pipeline +resolves to the same token, config validation should fail before the server +starts. + +Secret values must never be logged, returned in responses, or included in run +status. + +## HTTP Server Boundary + +Add a server mode rather than trying to make `run` poll an HTTP source. + +The likely CLI shape is: + +```sh +distributor serve --config +``` + +The HTTP API should default to a private bind address. Operators that need +public access, TLS termination, or mTLS should place `distributor` behind a +reverse proxy or private network boundary unless a later roadmap explicitly +adds in-app TLS. + +The server should expose: + +- `POST /upload`; +- `GET /runs/`; +- `GET /healthz`. + +`GET /healthz` should return success after the server has loaded and validated +configuration and is ready to accept requests. + +## Relationship To Existing Architecture + +`http_upload` should reuse existing code paths after staging: + +- upload staging should produce a local staged bundle tree; +- staged bundle validation should use the existing bundle validation contract; +- fan-out should use the app-layer single-pipeline run behavior where possible; +- destination handling should remain backend-agnostic; +- publish, transform, link, state, and notification behavior should not know + that the source arrived over HTTP. + +If the current app-layer single-pipeline runner assumes it can open and walk the +configured source backend, the HTTP implementation should add a narrow app-layer +entry point for "run this pipeline using this already-staged source backend" +rather than bending `http_upload` into a fake durable storage backend. + +## Non-Goals + +The first HTTP upload implementation should not add: + +- destination-side HTTP upload; +- browser UI; +- producer execution; +- source manifest schema changes; +- warning-only digest mismatch behavior; +- durable database-backed queueing; +- producer-supplied idempotency keys; +- zstd archive support; +- URL-token authentication; +- public network exposure defaults; +- in-app TLS; +- general-purpose workflow orchestration. + +## Testing Expectations + +Suggested coverage: + +- config validation accepts `http_upload` sources and rejects `http_upload` + destinations; +- default staging path becomes `/var/spool/distributor/`; +- token env references resolve through real environment and `secrets.directory`; +- duplicate token values across pipelines fail validation; +- upload size limit defaults to 20 MB and is enforced; +- archive extraction rejects unsafe paths, symlinks, hardlinks, devices, + missing manifest, missing manifest-listed files, and nested manifests; +- valid uploaded bundles validate through the existing bundle contract; +- upload admission returns `accepted` and a generated run id; +- status lookup reports queued, running, succeeded, failed, and expired states; +- per-pipeline runs do not execute concurrently; +- global `max_concurrency` is honored; +- full queues reject uploads before request-body staging; +- `GET /runs/` returns in-memory status records until retention expiry; +- failed extraction and failed runs clean up or retain staging data according to + the configured retention policy; +- secret values never appear in logs, responses, or status records; +- normal local, SSH/SFTP, and S3 source behavior remains unchanged. + +## Documentation Updates After Implementation + +- Update `docs/config.md` with `http_upload` source fields and defaults. +- Update `docs/cli.md` with `serve` syntax and HTTP behavior. +- Update `docs/operations.md` with upload, queue, status, and staging + workflows. +- Update `docs/troubleshooting.md` for authentication, archive extraction, + validation, queue, and fan-out failures. +- Add examples only if they are secret-free and safe to run locally. +- Update internal docs for any new app, HTTP, queue, or ingestion packages. + +Keep this roadmap under `docs/roadmap/` until implemented. + +## Future Work + +The first implementation intentionally defers: + +- URL-token authentication; +- zstd-compressed archive support; +- durable status persistence across process restarts; +- database-backed queueing; +- producer-supplied idempotency keys; +- run listing, cancellation, or retry endpoints; +- in-app TLS; +- public network exposure defaults; +- browser UI. + +These items should remain out of current-behavior docs until a later roadmap +selects and specifies them. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 403d7ee..bd5819c 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,91 +1,328 @@ -# HTTP API Boundary Roadmap +# HTTP Upload API Implementation Roadmap ## Purpose -This roadmap records the accepted boundary for a future HTTP API. The current -application exposes CLI commands and internal app-layer run contracts; it does -not implement an HTTP server, HTTP routes, a `serve` command, app-level -authentication, or in-app TLS. +This roadmap is the canonical staged implementation plan for +`docs/roadmap/http.md`. -Implemented internal run contracts are documented in `docs/internal/app.md`. -This file is the canonical home for future HTTP boundary decisions until an -HTTP implementation roadmap replaces it. +The current application supports CLI-driven distribution using configured +local, SSH/SFTP, and S3-compatible source and destination backends. It does not +yet implement an HTTP server, a `serve` command, `http_upload` source +configuration, upload authentication, archive ingestion, async upload status, +or HTTP routes. -## Accepted Direction +Future, planned, or aspirational behavior belongs under `docs/roadmap/` until +it is implemented. Current-behavior docs must be updated only after the +corresponding stage is complete. -The future HTTP API should trigger configured distributor pipelines through the -existing app-layer single-pipeline run path and in-memory coordinator. +## Implementation Principles -The HTTP API is intentionally narrow: +- Treat `docs/roadmap/http.md` as the source of truth for accepted HTTP upload + policy. +- Prefer standard-library HTTP, tar, gzip, and sync primitives. +- Do not add external dependencies for the first HTTP upload implementation. +- Do not register `http_upload` as a durable storage backend. +- Reuse existing bundle validation, destination fan-out, transform, link, + state, notification, and run-report behavior after upload staging. +- Keep deferred items out of current-behavior docs: URL-token auth, zstd, + durable status, database queues, producer idempotency keys, run + cancellation/listing, in-app TLS, public exposure defaults, and browser UI. -- A trigger request accepts only a pipeline ID as application input. -- A trigger request starts work asynchronously and returns a run ID after - admission. -- Run status is read through a separate status endpoint keyed by run ID. -- Status records expose run ID, pipeline ID, current status, timestamps, and - completed report or error details when available. +## Stage 1: HTTP Upload Configuration -The application remains a bundle distribution tool. The HTTP API must not turn -`distributor` into a workflow engine, CMS, report generator, or public web -authoring service. +Goal: add config support for HTTP upload sources and server settings without +adding HTTP runtime behavior. -## Error Mapping +Implementation scope: -Future transport code should map app-layer errors without changing app-layer -error ownership: +- add top-level `server.http` config with defaults: + - `bind: 127.0.0.1:8080`; + - `staging_root: /var/spool/distributor`; + - `max_upload_size: 20MB`; + - `queue_size: 16`; + - `max_concurrency: 1`; + - `retention: 24h`; +- add source-only `backend: http_upload`; +- add `http_upload` source fields: + - required `token_env`; + - optional `staging_path`; + - optional `max_upload_size`; +- default omitted `staging_path` to + `/var/spool/distributor/`; +- default omitted source `max_upload_size` to + `server.http.max_upload_size`; +- reject `http_upload` as a destination backend; +- parse sizes with `B`, `KB`, `MB`, and `GB` suffixes using 1024 multipliers; +- parse `retention` with `time.ParseDuration`; +- keep literal upload tokens out of YAML. -- Unknown pipeline IDs map to `404 Not Found`. -- Duplicate in-flight runs for the same pipeline ID map to `409 Conflict`. -- Validation, config, source, destination, publish, transform, and notification - errors map to transport errors according to their app-layer context. +Documentation updates after implementation: -Duplicate runs must not be queued. Run state remains in memory unless a later -roadmap explicitly adds durable run storage. +- update `docs/config.md` for implemented config fields and defaults; +- update `docs/internal/config.md` for config ownership and validation rules; +- do not document HTTP routes yet. -## Context And Lifetime +Tests: -The request context guards admission. Once a run is admitted, execution is tied -to the server or coordinator lifetime context rather than to the client request -lifetime. +- config loading accepts valid `server.http` and `http_upload` source config; +- defaults apply for bind, staging root, staging path, upload size, queue size, + concurrency, and retention; +- known-field decoding rejects unknown fields; +- invalid sizes, invalid durations, missing token env, missing pipeline ids, + duplicate pipeline ids, and `http_upload` destinations fail validation; +- existing local, SSH/SFTP, and S3 config behavior remains unchanged. -This split allows a client disconnect or request timeout to stop waiting for -admission without canceling a run that has already been accepted. +Completion criteria: `go test ./internal/config` passes and no runtime code +attempts to execute `http_upload`. -## Security Boundary +## Stage 2: Upload Archive Staging -The first HTTP server should default to private binding, such as `127.0.0.1`. -Operators should expose it through a reverse proxy, private network, or external -mTLS when transport security or remote access is required. +Goal: stage uploaded tar or tar.gz archives into a validated local source +bundle tree. -The first HTTP implementation should not include: +Implementation scope: -- bearer-token authentication; -- in-app TLS configuration; -- public-network exposure defaults. +- add an internal ingestion package for upload archive staging; +- support uncompressed tar and gzip-compressed tar only; +- accept content types: + - `application/x-tar`; + - `application/gzip`; + - `application/x-gzip`; +- stream request bodies to a per-run archive or staging path while enforcing + max upload size; +- extract into a per-run staging directory under the pipeline staging path; +- reject absolute paths, path traversal, backslash paths, symlinks, hardlinks, + devices, sockets, and other special entries; +- require exactly one root-level `manifest.json`; +- reject nested manifests; +- require all manifest-listed files to exist as regular files; +- enforce extracted size and file-count limits; +- clean up failed extraction directories; +- validate staged bundles through the existing source bundle contract. -A later roadmap must explicitly change this security decision before any of -those features are added. +Documentation updates after implementation: -## Non-Goals +- update relevant `docs/internal/` docs for the new ingestion package; +- keep user-facing HTTP docs out of current-behavior docs until the server + stage is implemented. -The future HTTP API should not add: +Tests: -- public CLI flags for selecting one pipeline during `distributor run`; -- a CLI framework; -- a generic workflow engine; -- plugin execution; -- durable run storage; -- app-level authentication; -- in-app TLS. +- valid tar and tar.gz uploads stage successfully; +- unsupported content types fail; +- max upload size is enforced while streaming; +- unsafe archive entries are rejected; +- missing manifest, nested manifest, missing listed file, digest mismatch, and + non-regular manifest-listed files fail validation; +- failed extraction cleans up staging data according to the package contract. -## Verification Expectations +Completion criteria: ingestion can produce a validated staged local bundle and +does not publish anything. -Any later HTTP implementation should preserve existing CLI behavior and keep -the app-layer run contracts tested. At minimum, it should cover: +## Stage 3: Staged Source Pipeline Execution -- trigger requests with known and unknown pipeline IDs; -- duplicate in-flight trigger requests; -- asynchronous acceptance and status lookup; -- private bind defaults; -- request-context admission behavior; -- coordinator-lifetime run execution. +Goal: run one configured pipeline using an already-staged local source bundle +root. + +Implementation scope: + +- add a narrow app-layer entry point for executing one pipeline with a staged + local source backend/root; +- bypass normal source backend opening only for this staged-source entry point; +- do not register `http_upload` as a normal `storage.Backend`; +- reuse existing destination fan-out, transform, link, state, notification, and + run-report behavior; +- ensure publish, transform, link, and state code do not know the source + arrived over HTTP. + +Documentation updates after implementation: + +- update `docs/internal/app.md` for the staged-source app entry point; +- update `docs/internal/bundle.md` only if source validation behavior changes. + +Tests: + +- staged valid bundles publish through configured local destinations; +- invalid staged bundles fail before destination writes; +- configured destination behavior for local, SSH/SFTP, and S3 remains + backend-agnostic; +- run reports match existing app report semantics. + +Completion criteria: app-level tests prove staged local bundles can run through +normal fan-out without HTTP server code. + +## Stage 4: Async Upload Queue And Status + +Goal: add in-memory async upload coordination, queueing, and status tracking. + +Implementation scope: + +- add an in-memory upload coordinator; +- generate run ids shaped like + `..`; +- use statuses: + - `accepted`; + - `queued`; + - `running`; + - `succeeded`; + - `failed`; + - `expired`; +- enforce global `max_concurrency`; +- enforce bounded `queue_size`; +- enforce one active running upload per pipeline; +- queue later accepted uploads for the same pipeline instead of running them + concurrently; +- reject uploads before consuming or staging the request body when the queue is + full; +- keep status in memory; +- apply time-based retention to completed status and staged run directories; +- treat every accepted upload as a new run; +- do not support producer-supplied idempotency keys. + +Documentation updates after implementation: + +- update internal docs for the upload coordinator; +- do not add user-facing HTTP docs until the server stage is implemented. + +Tests: + +- run id format includes pipeline id, filesystem-safe UTC timestamp, and random + suffix; +- queue size is bounded; +- full queues reject admission before staging; +- same-pipeline uploads serialize; +- different pipelines run concurrently up to `max_concurrency`; +- status transitions cover accepted, queued, running, succeeded, failed, and + expired; +- final status retains run report or error details until retention expiry; +- staging directories are retained or cleaned according to retention policy. + +Completion criteria: coordinator tests pass without starting an HTTP server. + +## Stage 5: HTTP Server And `serve` CLI + +Goal: expose the upload coordinator through HTTP and add the server CLI command. + +Implementation scope: + +- add `distributor serve --config `; +- load config and `secrets.directory` before starting the server; +- resolve each `http_upload` `token_env` through the config-owned environment + resolver; +- fail startup if any configured token is missing or duplicated; +- bind to `server.http.bind`, defaulting to `127.0.0.1:8080`; +- implement `POST /upload`; +- implement `GET /runs/`; +- implement `GET /healthz`; +- authenticate uploads with `Authorization: Bearer `; +- map each token to exactly one configured `http_upload` pipeline; +- do not require or accept a producer-submitted pipeline id; +- return `202 Accepted` with: + + ```json + {"run_id":"","status":"accepted"} + ``` + +- return `401` for missing or invalid bearer token; +- return `413` for oversized upload; +- return `415` for unsupported archive content type; +- return `503` for full queue; +- return `404` for unknown run status; +- never log or return secret token values. + +Documentation updates after implementation: + +- update `docs/cli.md` with `serve` syntax; +- update `docs/config.md` with HTTP upload source and server config; +- update `docs/troubleshooting.md` for startup, auth, upload, and status + failures; +- update relevant internal docs for HTTP server package boundaries. + +Tests: + +- CLI parsing recognizes `serve --config `; +- private bind default is applied; +- startup fails for missing or duplicate tokens; +- auth accepts valid bearer tokens and rejects missing or invalid tokens; +- routes return the expected status codes and JSON response shapes; +- health succeeds after configuration is loaded and the server is ready; +- responses, logs, and status records do not expose secret token values. + +Completion criteria: `httptest` route tests and CLI tests pass, and no current +local/SSH/S3 CLI behavior regresses. + +## Stage 6: End-To-End HTTP Upload Flow + +Goal: prove the complete async HTTP upload path publishes valid bundles and +rejects invalid ones safely. + +Implementation scope: + +- add integration-style tests using `httptest`; +- submit valid tar and tar.gz bundles; +- poll `GET /runs/` until completion; +- verify fan-out reaches configured local destinations; +- verify invalid archives fail without publishing; +- verify same-pipeline uploads serialize; +- verify different pipelines can run up to `max_concurrency`; +- verify status includes completed run report or error details. + +Documentation updates after implementation: + +- update `docs/operations.md` with an HTTP upload workflow; +- add safe local examples only if they are secret-free and testable. + +Tests: + +- valid upload returns `202 Accepted`, then `succeeded`; +- invalid archive returns an accepted run only when admission succeeds, then + transitions to `failed`; +- successful fan-out writes expected destination outputs and state; +- failed upload does not write destination outputs; +- same-pipeline and cross-pipeline concurrency follow configured policy; +- run focused package tests and `go test ./...`. + +Completion criteria: end-to-end HTTP upload tests pass and full test suite +passes. + +## Stage 7: Documentation And Roadmap Closeout + +Goal: document implemented HTTP upload behavior and remove completed roadmap +drift. + +Implementation scope: + +- update current-behavior docs for implemented HTTP upload behavior: + - `docs/config.md`; + - `docs/cli.md`; + - `docs/operations.md`; + - `docs/troubleshooting.md`; + - relevant `docs/internal/` docs; +- add only secret-free, safe local examples; +- keep deferred items out of current docs; +- remove or rewrite completed roadmap material once behavior is fully + documented. + +Deferred items: + +- URL-token authentication; +- zstd archive support; +- durable status persistence; +- database-backed queues; +- producer idempotency keys; +- run cancellation or listing; +- in-app TLS; +- public exposure defaults; +- browser UI. + +Tests: + +- documentation consistency checks show implemented HTTP behavior is no longer + described only as future work; +- current-behavior docs do not describe deferred behavior as available; +- examples are valid and secret-free; +- run focused tests for docs/examples backed by tests and `go test ./...` if + examples or behavior docs changed with code. + +Completion criteria: current docs describe implemented behavior accurately, and +`docs/roadmap/` contains only future or deferred HTTP work.