distributor/docs/roadmap/http.md

# Roadmap: HTTP Upload API

## Purpose

Add an HTTP upload API that lets producer applications push complete source
bundles into `distributor`.

The current application already supports local, SSH/SFTP, and S3-compatible
source and destination backends. Those source backends are pull-oriented:
`distributor` opens configured storage, discovers `manifest.json`, validates
bundles, and fans out selected outputs to configured destinations.

`http_upload` is different because it is push-oriented. A producer sends one
complete bundle to `distributor`, and `distributor` stages that upload before
running normal validation and fan-out. The HTTP layer should be an ingestion
layer over the existing app, bundle, storage, publish, transform, link, state,
and notification behavior.

## Current Implementation Grounding

Implemented behavior already provides the core pieces this feature should reuse:

- source bundle validation through the bundle package;
- local, SSH/SFTP, and S3-compatible storage backends;
- destination fan-out through publish planning and execution;
- single-pipeline app execution through the app layer;
- structured run reports and JSON-capable CLI output;
- secrets-directory environment resolution for credential material.

The HTTP implementation should not duplicate bundle validation or publication
logic. Once an upload is staged, it should proceed through the same validation
and fan-out behavior as any other source bundle.

## Accepted Direction

Add `http_upload` as a source backend option for configured pipelines.

An `http_upload` source is not a normal durable storage backend. It represents
an HTTP ingestion endpoint that receives an uploaded bundle, writes it into
pipeline-local staging storage, validates it, and then dispatches the existing
pipeline fan-out flow.

Accepted behavior:

- producer applications upload a compliant source bundle manifest and all
  referenced files;
- uploads are asynchronous;
- each accepted upload receives a generated run id and an initial `accepted`
  status;
- clients can query run status later by run id;
- authentication uses a static token associated with the selected
  `http_upload` pipeline;
- upload requests send that token with `Authorization: Bearer <token>`;
- tokens may be supplied through `secrets.directory` using the existing
  internal environment resolver;
- each `http_upload` pipeline has a configurable upload staging directory;
- default staging root is `/var/spool/distributor`;
- default pipeline staging directory is `/var/spool/distributor/<pipeline_id>`;
- default maximum upload size is 20 MB;
- upload archives may be uncompressed tar or gzip-compressed tar;
- the server must prevent multiple simultaneous active runs of the same
  pipeline;
- the server uses an internal bounded queue and a global `max_concurrency`
  setting for accepted uploads;
- completed status records and staged run directories use time-based retention;
- every accepted upload is a new run with a `distributor`-generated run id.

## Configuration Shape

Add `http_upload` as a source backend only. It should not be valid as a
destination backend.

Add top-level HTTP server configuration for cross-pipeline server behavior:

```yaml
server:
  http:
    bind: 127.0.0.1:8080
    staging_root: /var/spool/distributor
    max_upload_size: 20MB
    queue_size: 16
    max_concurrency: 1
    retention: 24h
```

Defaults:

- `bind`: `127.0.0.1:8080`;
- `staging_root`: `/var/spool/distributor`;
- `max_upload_size`: `20MB`;
- `queue_size`: `16`;
- `max_concurrency`: `1`;
- `retention`: `24h`.

Pipeline shape:

```yaml
pipelines:
  - id: weather-daily
    source:
      backend: http_upload
      token_env: WEATHER_DAILY_UPLOAD_TOKEN
      staging_path: /var/spool/distributor/weather-daily
      max_upload_size: 20MB
    destinations:
      - id: archive
        backend: s3
        ...
```

`token_env` is required for the first implementation. Literal token values in
YAML are not supported. Real environment variables or `secrets.directory` files
provide the token without putting secret values in config.

If `staging_path` is omitted, default it to:

```text
/var/spool/distributor/<pipeline_id>
```

If source-level `max_upload_size` is omitted, use the top-level
`server.http.max_upload_size` value.

Server-level queue configuration is separate from per-pipeline source
configuration. Per-pipeline fields may override staging path and upload size;
bind address, queue size, concurrency, and retention are server-level settings.

## Upload Model

The uploaded request should contain one complete source bundle.

Use archive upload rather than multipart fields for each file. `distributor`
should extract the archive into a per-run staging directory under the pipeline
staging path, then validate the extracted bundle.

The first implementation supports:

- uncompressed tar;
- gzip-compressed tar.

The server should accept uncompressed tar as `application/x-tar`. It should
accept gzip-compressed tar as `application/gzip` or `application/x-gzip`.

Archive extraction must be conservative:

- reject absolute paths;
- reject path traversal and backslash paths;
- reject symlinks, hardlinks, devices, sockets, and other special entries;
- require exactly one root-level `manifest.json`;
- require all manifest-listed files to be present as regular files;
- reject extra nested manifests;
- enforce upload size limits before extraction;
- enforce extracted size and file-count limits after extraction;
- clean up failed extraction directories.

After extraction, validation should use the existing source bundle validation
contract. A bundle that fails manifest, path, size, digest, or file validation
must not be published.

## Async Run Flow

HTTP upload processing should be asynchronous:

1. Authenticate the request token and map it to exactly one `http_upload`
   pipeline.
2. Admit or reject the request according to queue capacity and per-pipeline
   active-run rules.
3. Create a run id and per-run staging directory.
4. Record initial in-memory run status as `accepted`.
5. Return an acceptance response without waiting for fan-out to complete.
6. In a worker, extract the archive, validate the staged bundle, and run the
   pipeline fan-out using the staged bundle as the effective source.
7. Record final success or failure status.

Initial acceptance response shape:

```json
{
  "run_id": "weather-daily.20260603T120000Z.ab12cd34",
  "status": "accepted"
}
```

Run ids should be generated by `distributor`. Use the pipeline id, a
filesystem-safe UTC timestamp, and a short random suffix to avoid collisions.

Suggested status values:

- `accepted`;
- `queued`;
- `running`;
- `succeeded`;
- `failed`;
- `expired`.

Status records should include run id, pipeline id, status, timestamps, and the
completed run report or error details when available. Status records are kept
in memory for the first implementation and expire after the configured
retention period.

Every accepted upload is treated as a new run. The first implementation does
not accept producer-supplied idempotency keys.

## Queue And Concurrency

The server must not run more than one upload-triggered execution for the same
pipeline at the same time.

Use two controls:

- a global worker limit, configured as `max_concurrency`;
- a bounded admission queue, configured as `queue_size`;
- a per-pipeline single-active-run guard.

If a run for the same pipeline is already active, later accepted uploads for
that pipeline should wait in queue rather than starting concurrently.

If the queue is full, the upload request should fail before consuming and
staging the request body. The server must not allow unbounded memory or disk
growth.

## Authentication

Each `http_upload` pipeline has one static upload token.

Upload requests send the token in an HTTP bearer header:

```http
Authorization: Bearer <token>
```

Authentication maps the incoming bearer token to exactly one configured
pipeline. If no pipeline matches, the request fails. If more than one pipeline
resolves to the same token, config validation should fail before the server
starts.

Secret values must never be logged, returned in responses, or included in run
status.

## HTTP Server Boundary

Add a server mode rather than trying to make `run` poll an HTTP source.

The likely CLI shape is:

```sh
distributor serve --config <path>
```

The HTTP API should default to a private bind address. Operators that need
public access, TLS termination, or mTLS should place `distributor` behind a
reverse proxy or private network boundary unless a later roadmap explicitly
adds in-app TLS.

The server should expose:

- `POST /upload`;
- `GET /runs/<run_id>`;
- `GET /healthz`.

`GET /healthz` should return success after the server has loaded and validated
configuration and is ready to accept requests.

## Relationship To Existing Architecture

`http_upload` should reuse existing code paths after staging:

- upload staging should produce a local staged bundle tree;
- staged bundle validation should use the existing bundle validation contract;
- fan-out should use the app-layer single-pipeline run behavior where possible;
- destination handling should remain backend-agnostic;
- publish, transform, link, state, and notification behavior should not know
  that the source arrived over HTTP.

If the current app-layer single-pipeline runner assumes it can open and walk the
configured source backend, the HTTP implementation should add a narrow app-layer
entry point for "run this pipeline using this already-staged source backend"
rather than bending `http_upload` into a fake durable storage backend.

## Non-Goals

The first HTTP upload implementation should not add:

- destination-side HTTP upload;
- browser UI;
- producer execution;
- source manifest schema changes;
- warning-only digest mismatch behavior;
- durable database-backed queueing;
- producer-supplied idempotency keys;
- zstd archive support;
- URL-token authentication;
- public network exposure defaults;
- in-app TLS;
- general-purpose workflow orchestration.

## Testing Expectations

Suggested coverage:

- config validation accepts `http_upload` sources and rejects `http_upload`
  destinations;
- default staging path becomes `/var/spool/distributor/<pipeline_id>`;
- token env references resolve through real environment and `secrets.directory`;
- duplicate token values across pipelines fail validation;
- upload size limit defaults to 20 MB and is enforced;
- archive extraction rejects unsafe paths, symlinks, hardlinks, devices,
  missing manifest, missing manifest-listed files, and nested manifests;
- valid uploaded bundles validate through the existing bundle contract;
- upload admission returns `accepted` and a generated run id;
- status lookup reports queued, running, succeeded, failed, and expired states;
- per-pipeline runs do not execute concurrently;
- global `max_concurrency` is honored;
- full queues reject uploads before request-body staging;
- `GET /runs/<run_id>` returns in-memory status records until retention expiry;
- failed extraction and failed runs clean up or retain staging data according to
  the configured retention policy;
- secret values never appear in logs, responses, or status records;
- normal local, SSH/SFTP, and S3 source behavior remains unchanged.

## Documentation Updates After Implementation

- Update `docs/config.md` with `http_upload` source fields and defaults.
- Update `docs/cli.md` with `serve` syntax and HTTP behavior.
- Update `docs/operations.md` with upload, queue, status, and staging
  workflows.
- Update `docs/troubleshooting.md` for authentication, archive extraction,
  validation, queue, and fan-out failures.
- Add examples only if they are secret-free and safe to run locally.
- Update internal docs for any new app, HTTP, queue, or ingestion packages.

Keep this roadmap under `docs/roadmap/` until implemented.

## Future Work

The first implementation intentionally defers:

- URL-token authentication;
- zstd-compressed archive support;
- durable status persistence across process restarts;
- database-backed queueing;
- producer-supplied idempotency keys;
- run listing, cancellation, or retry endpoints;
- in-app TLS;
- public network exposure defaults;
- browser UI.

These items should remain out of current-behavior docs until a later roadmap
selects and specifies them.