Add implementation roadmap for HTTP API
This commit is contained in:
349
docs/roadmap/http.md
Normal file
349
docs/roadmap/http.md
Normal file
@@ -0,0 +1,349 @@
|
||||
# Roadmap: HTTP Upload API
|
||||
|
||||
## Purpose
|
||||
|
||||
Add an HTTP upload API that lets producer applications push complete source
|
||||
bundles into `distributor`.
|
||||
|
||||
The current application already supports local, SSH/SFTP, and S3-compatible
|
||||
source and destination backends. Those source backends are pull-oriented:
|
||||
`distributor` opens configured storage, discovers `manifest.json`, validates
|
||||
bundles, and fans out selected outputs to configured destinations.
|
||||
|
||||
`http_upload` is different because it is push-oriented. A producer sends one
|
||||
complete bundle to `distributor`, and `distributor` stages that upload before
|
||||
running normal validation and fan-out. The HTTP layer should be an ingestion
|
||||
layer over the existing app, bundle, storage, publish, transform, link, state,
|
||||
and notification behavior.
|
||||
|
||||
## Current Implementation Grounding
|
||||
|
||||
Implemented behavior already provides the core pieces this feature should reuse:
|
||||
|
||||
- source bundle validation through the bundle package;
|
||||
- local, SSH/SFTP, and S3-compatible storage backends;
|
||||
- destination fan-out through publish planning and execution;
|
||||
- single-pipeline app execution through the app layer;
|
||||
- structured run reports and JSON-capable CLI output;
|
||||
- secrets-directory environment resolution for credential material.
|
||||
|
||||
The HTTP implementation should not duplicate bundle validation or publication
|
||||
logic. Once an upload is staged, it should proceed through the same validation
|
||||
and fan-out behavior as any other source bundle.
|
||||
|
||||
## Accepted Direction
|
||||
|
||||
Add `http_upload` as a source backend option for configured pipelines.
|
||||
|
||||
An `http_upload` source is not a normal durable storage backend. It represents
|
||||
an HTTP ingestion endpoint that receives an uploaded bundle, writes it into
|
||||
pipeline-local staging storage, validates it, and then dispatches the existing
|
||||
pipeline fan-out flow.
|
||||
|
||||
Accepted behavior:
|
||||
|
||||
- producer applications upload a compliant source bundle manifest and all
|
||||
referenced files;
|
||||
- uploads are asynchronous;
|
||||
- each accepted upload receives a generated run id and an initial `accepted`
|
||||
status;
|
||||
- clients can query run status later by run id;
|
||||
- authentication uses a static token associated with the selected
|
||||
`http_upload` pipeline;
|
||||
- upload requests send that token with `Authorization: Bearer <token>`;
|
||||
- tokens may be supplied through `secrets.directory` using the existing
|
||||
internal environment resolver;
|
||||
- each `http_upload` pipeline has a configurable upload staging directory;
|
||||
- default staging root is `/var/spool/distributor`;
|
||||
- default pipeline staging directory is `/var/spool/distributor/<pipeline_id>`;
|
||||
- default maximum upload size is 20 MB;
|
||||
- upload archives may be uncompressed tar or gzip-compressed tar;
|
||||
- the server must prevent multiple simultaneous active runs of the same
|
||||
pipeline;
|
||||
- the server uses an internal bounded queue and a global `max_concurrency`
|
||||
setting for accepted uploads;
|
||||
- completed status records and staged run directories use time-based retention;
|
||||
- every accepted upload is a new run with a `distributor`-generated run id.
|
||||
|
||||
## Configuration Shape
|
||||
|
||||
Add `http_upload` as a source backend only. It should not be valid as a
|
||||
destination backend.
|
||||
|
||||
Add top-level HTTP server configuration for cross-pipeline server behavior:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
http:
|
||||
bind: 127.0.0.1:8080
|
||||
staging_root: /var/spool/distributor
|
||||
max_upload_size: 20MB
|
||||
queue_size: 16
|
||||
max_concurrency: 1
|
||||
retention: 24h
|
||||
```
|
||||
|
||||
Defaults:
|
||||
|
||||
- `bind`: `127.0.0.1:8080`;
|
||||
- `staging_root`: `/var/spool/distributor`;
|
||||
- `max_upload_size`: `20MB`;
|
||||
- `queue_size`: `16`;
|
||||
- `max_concurrency`: `1`;
|
||||
- `retention`: `24h`.
|
||||
|
||||
Pipeline shape:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: weather-daily
|
||||
source:
|
||||
backend: http_upload
|
||||
token_env: WEATHER_DAILY_UPLOAD_TOKEN
|
||||
staging_path: /var/spool/distributor/weather-daily
|
||||
max_upload_size: 20MB
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: s3
|
||||
...
|
||||
```
|
||||
|
||||
`token_env` is required for the first implementation. Literal token values in
|
||||
YAML are not supported. Real environment variables or `secrets.directory` files
|
||||
provide the token without putting secret values in config.
|
||||
|
||||
If `staging_path` is omitted, default it to:
|
||||
|
||||
```text
|
||||
/var/spool/distributor/<pipeline_id>
|
||||
```
|
||||
|
||||
If source-level `max_upload_size` is omitted, use the top-level
|
||||
`server.http.max_upload_size` value.
|
||||
|
||||
Server-level queue configuration is separate from per-pipeline source
|
||||
configuration. Per-pipeline fields may override staging path and upload size;
|
||||
bind address, queue size, concurrency, and retention are server-level settings.
|
||||
|
||||
## Upload Model
|
||||
|
||||
The uploaded request should contain one complete source bundle.
|
||||
|
||||
Use archive upload rather than multipart fields for each file. `distributor`
|
||||
should extract the archive into a per-run staging directory under the pipeline
|
||||
staging path, then validate the extracted bundle.
|
||||
|
||||
The first implementation supports:
|
||||
|
||||
- uncompressed tar;
|
||||
- gzip-compressed tar.
|
||||
|
||||
The server should accept uncompressed tar as `application/x-tar`. It should
|
||||
accept gzip-compressed tar as `application/gzip` or `application/x-gzip`.
|
||||
|
||||
Archive extraction must be conservative:
|
||||
|
||||
- reject absolute paths;
|
||||
- reject path traversal and backslash paths;
|
||||
- reject symlinks, hardlinks, devices, sockets, and other special entries;
|
||||
- require exactly one root-level `manifest.json`;
|
||||
- require all manifest-listed files to be present as regular files;
|
||||
- reject extra nested manifests;
|
||||
- enforce upload size limits before extraction;
|
||||
- enforce extracted size and file-count limits after extraction;
|
||||
- clean up failed extraction directories.
|
||||
|
||||
After extraction, validation should use the existing source bundle validation
|
||||
contract. A bundle that fails manifest, path, size, digest, or file validation
|
||||
must not be published.
|
||||
|
||||
## Async Run Flow
|
||||
|
||||
HTTP upload processing should be asynchronous:
|
||||
|
||||
1. Authenticate the request token and map it to exactly one `http_upload`
|
||||
pipeline.
|
||||
2. Admit or reject the request according to queue capacity and per-pipeline
|
||||
active-run rules.
|
||||
3. Create a run id and per-run staging directory.
|
||||
4. Record initial in-memory run status as `accepted`.
|
||||
5. Return an acceptance response without waiting for fan-out to complete.
|
||||
6. In a worker, extract the archive, validate the staged bundle, and run the
|
||||
pipeline fan-out using the staged bundle as the effective source.
|
||||
7. Record final success or failure status.
|
||||
|
||||
Initial acceptance response shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"run_id": "weather-daily.20260603T120000Z.ab12cd34",
|
||||
"status": "accepted"
|
||||
}
|
||||
```
|
||||
|
||||
Run ids should be generated by `distributor`. Use the pipeline id, a
|
||||
filesystem-safe UTC timestamp, and a short random suffix to avoid collisions.
|
||||
|
||||
Suggested status values:
|
||||
|
||||
- `accepted`;
|
||||
- `queued`;
|
||||
- `running`;
|
||||
- `succeeded`;
|
||||
- `failed`;
|
||||
- `expired`.
|
||||
|
||||
Status records should include run id, pipeline id, status, timestamps, and the
|
||||
completed run report or error details when available. Status records are kept
|
||||
in memory for the first implementation and expire after the configured
|
||||
retention period.
|
||||
|
||||
Every accepted upload is treated as a new run. The first implementation does
|
||||
not accept producer-supplied idempotency keys.
|
||||
|
||||
## Queue And Concurrency
|
||||
|
||||
The server must not run more than one upload-triggered execution for the same
|
||||
pipeline at the same time.
|
||||
|
||||
Use two controls:
|
||||
|
||||
- a global worker limit, configured as `max_concurrency`;
|
||||
- a bounded admission queue, configured as `queue_size`;
|
||||
- a per-pipeline single-active-run guard.
|
||||
|
||||
If a run for the same pipeline is already active, later accepted uploads for
|
||||
that pipeline should wait in queue rather than starting concurrently.
|
||||
|
||||
If the queue is full, the upload request should fail before consuming and
|
||||
staging the request body. The server must not allow unbounded memory or disk
|
||||
growth.
|
||||
|
||||
## Authentication
|
||||
|
||||
Each `http_upload` pipeline has one static upload token.
|
||||
|
||||
Upload requests send the token in an HTTP bearer header:
|
||||
|
||||
```http
|
||||
Authorization: Bearer <token>
|
||||
```
|
||||
|
||||
Authentication maps the incoming bearer token to exactly one configured
|
||||
pipeline. If no pipeline matches, the request fails. If more than one pipeline
|
||||
resolves to the same token, config validation should fail before the server
|
||||
starts.
|
||||
|
||||
Secret values must never be logged, returned in responses, or included in run
|
||||
status.
|
||||
|
||||
## HTTP Server Boundary
|
||||
|
||||
Add a server mode rather than trying to make `run` poll an HTTP source.
|
||||
|
||||
The likely CLI shape is:
|
||||
|
||||
```sh
|
||||
distributor serve --config <path>
|
||||
```
|
||||
|
||||
The HTTP API should default to a private bind address. Operators that need
|
||||
public access, TLS termination, or mTLS should place `distributor` behind a
|
||||
reverse proxy or private network boundary unless a later roadmap explicitly
|
||||
adds in-app TLS.
|
||||
|
||||
The server should expose:
|
||||
|
||||
- `POST /upload`;
|
||||
- `GET /runs/<run_id>`;
|
||||
- `GET /healthz`.
|
||||
|
||||
`GET /healthz` should return success after the server has loaded and validated
|
||||
configuration and is ready to accept requests.
|
||||
|
||||
## Relationship To Existing Architecture
|
||||
|
||||
`http_upload` should reuse existing code paths after staging:
|
||||
|
||||
- upload staging should produce a local staged bundle tree;
|
||||
- staged bundle validation should use the existing bundle validation contract;
|
||||
- fan-out should use the app-layer single-pipeline run behavior where possible;
|
||||
- destination handling should remain backend-agnostic;
|
||||
- publish, transform, link, state, and notification behavior should not know
|
||||
that the source arrived over HTTP.
|
||||
|
||||
If the current app-layer single-pipeline runner assumes it can open and walk the
|
||||
configured source backend, the HTTP implementation should add a narrow app-layer
|
||||
entry point for "run this pipeline using this already-staged source backend"
|
||||
rather than bending `http_upload` into a fake durable storage backend.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
The first HTTP upload implementation should not add:
|
||||
|
||||
- destination-side HTTP upload;
|
||||
- browser UI;
|
||||
- producer execution;
|
||||
- source manifest schema changes;
|
||||
- warning-only digest mismatch behavior;
|
||||
- durable database-backed queueing;
|
||||
- producer-supplied idempotency keys;
|
||||
- zstd archive support;
|
||||
- URL-token authentication;
|
||||
- public network exposure defaults;
|
||||
- in-app TLS;
|
||||
- general-purpose workflow orchestration.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Suggested coverage:
|
||||
|
||||
- config validation accepts `http_upload` sources and rejects `http_upload`
|
||||
destinations;
|
||||
- default staging path becomes `/var/spool/distributor/<pipeline_id>`;
|
||||
- token env references resolve through real environment and `secrets.directory`;
|
||||
- duplicate token values across pipelines fail validation;
|
||||
- upload size limit defaults to 20 MB and is enforced;
|
||||
- archive extraction rejects unsafe paths, symlinks, hardlinks, devices,
|
||||
missing manifest, missing manifest-listed files, and nested manifests;
|
||||
- valid uploaded bundles validate through the existing bundle contract;
|
||||
- upload admission returns `accepted` and a generated run id;
|
||||
- status lookup reports queued, running, succeeded, failed, and expired states;
|
||||
- per-pipeline runs do not execute concurrently;
|
||||
- global `max_concurrency` is honored;
|
||||
- full queues reject uploads before request-body staging;
|
||||
- `GET /runs/<run_id>` returns in-memory status records until retention expiry;
|
||||
- failed extraction and failed runs clean up or retain staging data according to
|
||||
the configured retention policy;
|
||||
- secret values never appear in logs, responses, or status records;
|
||||
- normal local, SSH/SFTP, and S3 source behavior remains unchanged.
|
||||
|
||||
## Documentation Updates After Implementation
|
||||
|
||||
- Update `docs/config.md` with `http_upload` source fields and defaults.
|
||||
- Update `docs/cli.md` with `serve` syntax and HTTP behavior.
|
||||
- Update `docs/operations.md` with upload, queue, status, and staging
|
||||
workflows.
|
||||
- Update `docs/troubleshooting.md` for authentication, archive extraction,
|
||||
validation, queue, and fan-out failures.
|
||||
- Add examples only if they are secret-free and safe to run locally.
|
||||
- Update internal docs for any new app, HTTP, queue, or ingestion packages.
|
||||
|
||||
Keep this roadmap under `docs/roadmap/` until implemented.
|
||||
|
||||
## Future Work
|
||||
|
||||
The first implementation intentionally defers:
|
||||
|
||||
- URL-token authentication;
|
||||
- zstd-compressed archive support;
|
||||
- durable status persistence across process restarts;
|
||||
- database-backed queueing;
|
||||
- producer-supplied idempotency keys;
|
||||
- run listing, cancellation, or retry endpoints;
|
||||
- in-app TLS;
|
||||
- public network exposure defaults;
|
||||
- browser UI.
|
||||
|
||||
These items should remain out of current-behavior docs until a later roadmap
|
||||
selects and specifies them.
|
||||
Reference in New Issue
Block a user