distributor/docs/roadmap/http.md

Roadmap: HTTP Upload API

Purpose

Add an HTTP upload API that lets producer applications push complete source bundles into distributor.

The current application already supports local, SSH/SFTP, and S3-compatible source and destination backends. Those source backends are pull-oriented: distributor opens configured storage, discovers manifest.json, validates bundles, and fans out selected outputs to configured destinations.

http_upload is different because it is push-oriented. A producer sends one complete bundle to distributor, and distributor stages that upload before running normal validation and fan-out. The HTTP layer should be an ingestion layer over the existing app, bundle, storage, publish, transform, link, state, and notification behavior.

Current Implementation Grounding

Implemented behavior already provides the core pieces this feature should reuse:

• source bundle validation through the bundle package;
• local, SSH/SFTP, and S3-compatible storage backends;
• destination fan-out through publish planning and execution;
• single-pipeline app execution through the app layer;
• structured run reports and JSON-capable CLI output;
• secrets-directory environment resolution for credential material.

The HTTP implementation should not duplicate bundle validation or publication logic. Once an upload is staged, it should proceed through the same validation and fan-out behavior as any other source bundle.

Accepted Direction

Add http_upload as a source backend option for configured pipelines.

An http_upload source is not a normal durable storage backend. It represents an HTTP ingestion endpoint that receives an uploaded bundle, writes it into pipeline-local staging storage, validates it, and then dispatches the existing pipeline fan-out flow.

Accepted behavior:

• producer applications upload a compliant source bundle manifest and all referenced files;
• uploads are asynchronous;
• each accepted upload receives a generated run id and an initial accepted status;
• clients can query run status later by run id;
• authentication uses a static token associated with the selected http_upload pipeline;
• upload requests send that token with Authorization: Bearer <token>;
• tokens may be supplied through secrets.directory using the existing internal environment resolver;
• each http_upload pipeline has a configurable upload staging directory;
• default staging root is /var/spool/distributor;
• default pipeline staging directory is /var/spool/distributor/<pipeline_id>;
• default maximum upload size is 20 MB;
• upload archives may be uncompressed tar or gzip-compressed tar;
• the server must prevent multiple simultaneous active runs of the same pipeline;
• the server uses an internal bounded queue and a global max_concurrency setting for accepted uploads;
• completed status records and staged run directories use time-based retention;
• every accepted upload is a new run with a distributor-generated run id.

Configuration Shape

Add http_upload as a source backend only. It should not be valid as a destination backend.

Add top-level HTTP server configuration for cross-pipeline server behavior:

server:
  http:
    bind: 127.0.0.1:8080
    staging_root: /var/spool/distributor
    max_upload_size: 20MB
    queue_size: 16
    max_concurrency: 1
    retention: 24h

Defaults:

• bind: 127.0.0.1:8080;
• staging_root: /var/spool/distributor;
• max_upload_size: 20MB;
• queue_size: 16;
• max_concurrency: 1;
• retention: 24h.

Pipeline shape:

pipelines:
  - id: weather-daily
    source:
      backend: http_upload
      token_env: WEATHER_DAILY_UPLOAD_TOKEN
      staging_path: /var/spool/distributor/weather-daily
      max_upload_size: 20MB
    destinations:
      - id: archive
        backend: s3
        ...

token_env is required for the first implementation. Literal token values in YAML are not supported. Real environment variables or secrets.directory files provide the token without putting secret values in config.

If staging_path is omitted, default it to:

/var/spool/distributor/<pipeline_id>

If source-level max_upload_size is omitted, use the top-level server.http.max_upload_size value.

Server-level queue configuration is separate from per-pipeline source configuration. Per-pipeline fields may override staging path and upload size; bind address, queue size, concurrency, and retention are server-level settings.

Upload Model

The uploaded request should contain one complete source bundle.

Use archive upload rather than multipart fields for each file. distributor should extract the archive into a per-run staging directory under the pipeline staging path, then validate the extracted bundle.

The first implementation supports:

• uncompressed tar;
• gzip-compressed tar.

The server should accept uncompressed tar as application/x-tar. It should accept gzip-compressed tar as application/gzip or application/x-gzip.

Archive extraction must be conservative:

• reject absolute paths;
• reject path traversal and backslash paths;
• reject symlinks, hardlinks, devices, sockets, and other special entries;
• require exactly one root-level manifest.json;
• require all manifest-listed files to be present as regular files;
• reject extra nested manifests;
• enforce upload size limits before extraction;
• enforce extracted size and file-count limits after extraction;
• clean up failed extraction directories.

After extraction, validation should use the existing source bundle validation contract. A bundle that fails manifest, path, size, digest, or file validation must not be published.

Async Run Flow

HTTP upload processing should be asynchronous:

1. Authenticate the request token and map it to exactly one http_upload pipeline.
2. Admit or reject the request according to queue capacity and per-pipeline active-run rules.
3. Create a run id and per-run staging directory.
4. Record initial in-memory run status as accepted.
5. Return an acceptance response without waiting for fan-out to complete.
6. In a worker, extract the archive, validate the staged bundle, and run the pipeline fan-out using the staged bundle as the effective source.
7. Record final success or failure status.

Initial acceptance response shape:

{
  "run_id": "weather-daily.20260603T120000Z.ab12cd34",
  "status": "accepted"
}

Run ids should be generated by distributor. Use the pipeline id, a filesystem-safe UTC timestamp, and a short random suffix to avoid collisions.

Suggested status values:

• accepted;
• queued;
• running;
• succeeded;
• failed;
• expired.

Status records should include run id, pipeline id, status, timestamps, and the completed run report or error details when available. Status records are kept in memory for the first implementation and expire after the configured retention period.

Every accepted upload is treated as a new run. The first implementation does not accept producer-supplied idempotency keys.

Queue And Concurrency

The server must not run more than one upload-triggered execution for the same pipeline at the same time.

Use two controls:

• a global worker limit, configured as max_concurrency;
• a bounded admission queue, configured as queue_size;
• a per-pipeline single-active-run guard.

If a run for the same pipeline is already active, later accepted uploads for that pipeline should wait in queue rather than starting concurrently.

If the queue is full, the upload request should fail before consuming and staging the request body. The server must not allow unbounded memory or disk growth.

Authentication

Each http_upload pipeline has one static upload token.

Upload requests send the token in an HTTP bearer header:

Authorization: Bearer <token>

Authentication maps the incoming bearer token to exactly one configured pipeline. If no pipeline matches, the request fails. If more than one pipeline resolves to the same token, config validation should fail before the server starts.

Secret values must never be logged, returned in responses, or included in run status.

HTTP Server Boundary

Add a server mode rather than trying to make run poll an HTTP source.

The likely CLI shape is:

distributor serve --config <path>

The HTTP API should default to a private bind address. Operators that need public access, TLS termination, or mTLS should place distributor behind a reverse proxy or private network boundary unless a later roadmap explicitly adds in-app TLS.

The server should expose:

• POST /upload;
• GET /runs/<run_id>;
• GET /healthz.

GET /healthz should return success after the server has loaded and validated configuration and is ready to accept requests.

Relationship To Existing Architecture

http_upload should reuse existing code paths after staging:

• upload staging should produce a local staged bundle tree;
• staged bundle validation should use the existing bundle validation contract;
• fan-out should use the app-layer single-pipeline run behavior where possible;
• destination handling should remain backend-agnostic;
• publish, transform, link, state, and notification behavior should not know that the source arrived over HTTP.

If the current app-layer single-pipeline runner assumes it can open and walk the configured source backend, the HTTP implementation should add a narrow app-layer entry point for "run this pipeline using this already-staged source backend" rather than bending http_upload into a fake durable storage backend.

Non-Goals

The first HTTP upload implementation should not add:

• destination-side HTTP upload;
• browser UI;
• producer execution;
• source manifest schema changes;
• warning-only digest mismatch behavior;
• durable database-backed queueing;
• producer-supplied idempotency keys;
• zstd archive support;
• URL-token authentication;
• public network exposure defaults;
• in-app TLS;
• general-purpose workflow orchestration.

Testing Expectations

Suggested coverage:

• config validation accepts http_upload sources and rejects http_upload destinations;
• default staging path becomes /var/spool/distributor/<pipeline_id>;
• token env references resolve through real environment and secrets.directory;
• duplicate token values across pipelines fail validation;
• upload size limit defaults to 20 MB and is enforced;
• archive extraction rejects unsafe paths, symlinks, hardlinks, devices, missing manifest, missing manifest-listed files, and nested manifests;
• valid uploaded bundles validate through the existing bundle contract;
• upload admission returns accepted and a generated run id;
• status lookup reports queued, running, succeeded, failed, and expired states;
• per-pipeline runs do not execute concurrently;
• global max_concurrency is honored;
• full queues reject uploads before request-body staging;
• GET /runs/<run_id> returns in-memory status records until retention expiry;
• failed extraction and failed runs clean up or retain staging data according to the configured retention policy;
• secret values never appear in logs, responses, or status records;
• normal local, SSH/SFTP, and S3 source behavior remains unchanged.

Documentation Updates After Implementation

• Update docs/config.md with http_upload source fields and defaults.
• Update docs/cli.md with serve syntax and HTTP behavior.
• Update docs/operations.md with upload, queue, status, and staging workflows.
• Update docs/troubleshooting.md for authentication, archive extraction, validation, queue, and fan-out failures.
• Add examples only if they are secret-free and safe to run locally.
• Update internal docs for any new app, HTTP, queue, or ingestion packages.

Keep this roadmap under docs/roadmap/ until implemented.

Future Work

The first implementation intentionally defers:

• URL-token authentication;
• zstd-compressed archive support;
• durable status persistence across process restarts;
• database-backed queueing;
• producer-supplied idempotency keys;
• run listing, cancellation, or retry endpoints;
• in-app TLS;
• public network exposure defaults;
• browser UI.

These items should remain out of current-behavior docs until a later roadmap selects and specifies them.