Add implementation roadmap for HTTP API

docs/roadmap/http.md

@@ -0,0 +1,349 @@
+# Roadmap: HTTP Upload API
+
+## Purpose
+
+Add an HTTP upload API that lets producer applications push complete source
+bundles into `distributor`.
+
+The current application already supports local, SSH/SFTP, and S3-compatible
+source and destination backends. Those source backends are pull-oriented:
+`distributor` opens configured storage, discovers `manifest.json`, validates
+bundles, and fans out selected outputs to configured destinations.
+
+`http_upload` is different because it is push-oriented. A producer sends one
+complete bundle to `distributor`, and `distributor` stages that upload before
+running normal validation and fan-out. The HTTP layer should be an ingestion
+layer over the existing app, bundle, storage, publish, transform, link, state,
+and notification behavior.
+
+## Current Implementation Grounding
+
+Implemented behavior already provides the core pieces this feature should reuse:
+
+- source bundle validation through the bundle package;
+- local, SSH/SFTP, and S3-compatible storage backends;
+- destination fan-out through publish planning and execution;
+- single-pipeline app execution through the app layer;
+- structured run reports and JSON-capable CLI output;
+- secrets-directory environment resolution for credential material.
+
+The HTTP implementation should not duplicate bundle validation or publication
+logic. Once an upload is staged, it should proceed through the same validation
+and fan-out behavior as any other source bundle.
+
+## Accepted Direction
+
+Add `http_upload` as a source backend option for configured pipelines.
+
+An `http_upload` source is not a normal durable storage backend. It represents
+an HTTP ingestion endpoint that receives an uploaded bundle, writes it into
+pipeline-local staging storage, validates it, and then dispatches the existing
+pipeline fan-out flow.
+
+Accepted behavior:
+
+- producer applications upload a compliant source bundle manifest and all
+  referenced files;
+- uploads are asynchronous;
+- each accepted upload receives a generated run id and an initial `accepted`
+  status;
+- clients can query run status later by run id;
+- authentication uses a static token associated with the selected
+  `http_upload` pipeline;
+- upload requests send that token with `Authorization: Bearer <token>`;
+- tokens may be supplied through `secrets.directory` using the existing
+  internal environment resolver;
+- each `http_upload` pipeline has a configurable upload staging directory;
+- default staging root is `/var/spool/distributor`;
+- default pipeline staging directory is `/var/spool/distributor/<pipeline_id>`;
+- default maximum upload size is 20 MB;
+- upload archives may be uncompressed tar or gzip-compressed tar;
+- the server must prevent multiple simultaneous active runs of the same
+  pipeline;
+- the server uses an internal bounded queue and a global `max_concurrency`
+  setting for accepted uploads;
+- completed status records and staged run directories use time-based retention;
+- every accepted upload is a new run with a `distributor`-generated run id.
+
+## Configuration Shape
+
+Add `http_upload` as a source backend only. It should not be valid as a
+destination backend.
+
+Add top-level HTTP server configuration for cross-pipeline server behavior:
+
+```yaml
+server:
+  http:
+    bind: 127.0.0.1:8080
+    staging_root: /var/spool/distributor
+    max_upload_size: 20MB
+    queue_size: 16
+    max_concurrency: 1
+    retention: 24h
+```
+
+Defaults:
+
+- `bind`: `127.0.0.1:8080`;
+- `staging_root`: `/var/spool/distributor`;
+- `max_upload_size`: `20MB`;
+- `queue_size`: `16`;
+- `max_concurrency`: `1`;
+- `retention`: `24h`.
+
+Pipeline shape:
+
+```yaml
+pipelines:
+  - id: weather-daily
+    source:
+      backend: http_upload
+      token_env: WEATHER_DAILY_UPLOAD_TOKEN
+      staging_path: /var/spool/distributor/weather-daily
+      max_upload_size: 20MB
+    destinations:
+      - id: archive
+        backend: s3
+        ...
+```
+
+`token_env` is required for the first implementation. Literal token values in
+YAML are not supported. Real environment variables or `secrets.directory` files
+provide the token without putting secret values in config.
+
+If `staging_path` is omitted, default it to:
+
+```text
+/var/spool/distributor/<pipeline_id>
+```
+
+If source-level `max_upload_size` is omitted, use the top-level
+`server.http.max_upload_size` value.
+
+Server-level queue configuration is separate from per-pipeline source
+configuration. Per-pipeline fields may override staging path and upload size;
+bind address, queue size, concurrency, and retention are server-level settings.
+
+## Upload Model
+
+The uploaded request should contain one complete source bundle.
+
+Use archive upload rather than multipart fields for each file. `distributor`
+should extract the archive into a per-run staging directory under the pipeline
+staging path, then validate the extracted bundle.
+
+The first implementation supports:
+
+- uncompressed tar;
+- gzip-compressed tar.
+
+The server should accept uncompressed tar as `application/x-tar`. It should
+accept gzip-compressed tar as `application/gzip` or `application/x-gzip`.
+
+Archive extraction must be conservative:
+
+- reject absolute paths;
+- reject path traversal and backslash paths;
+- reject symlinks, hardlinks, devices, sockets, and other special entries;
+- require exactly one root-level `manifest.json`;
+- require all manifest-listed files to be present as regular files;
+- reject extra nested manifests;
+- enforce upload size limits before extraction;
+- enforce extracted size and file-count limits after extraction;
+- clean up failed extraction directories.
+
+After extraction, validation should use the existing source bundle validation
+contract. A bundle that fails manifest, path, size, digest, or file validation
+must not be published.
+
+## Async Run Flow
+
+HTTP upload processing should be asynchronous:
+
+1. Authenticate the request token and map it to exactly one `http_upload`
+   pipeline.
+2. Admit or reject the request according to queue capacity and per-pipeline
+   active-run rules.
+3. Create a run id and per-run staging directory.
+4. Record initial in-memory run status as `accepted`.
+5. Return an acceptance response without waiting for fan-out to complete.
+6. In a worker, extract the archive, validate the staged bundle, and run the
+   pipeline fan-out using the staged bundle as the effective source.
+7. Record final success or failure status.
+
+Initial acceptance response shape:
+
+```json
+{
+  "run_id": "weather-daily.20260603T120000Z.ab12cd34",
+  "status": "accepted"
+}
+```
+
+Run ids should be generated by `distributor`. Use the pipeline id, a
+filesystem-safe UTC timestamp, and a short random suffix to avoid collisions.
+
+Suggested status values:
+
+- `accepted`;
+- `queued`;
+- `running`;
+- `succeeded`;
+- `failed`;
+- `expired`.
+
+Status records should include run id, pipeline id, status, timestamps, and the
+completed run report or error details when available. Status records are kept
+in memory for the first implementation and expire after the configured
+retention period.
+
+Every accepted upload is treated as a new run. The first implementation does
+not accept producer-supplied idempotency keys.
+
+## Queue And Concurrency
+
+The server must not run more than one upload-triggered execution for the same
+pipeline at the same time.
+
+Use two controls:
+
+- a global worker limit, configured as `max_concurrency`;
+- a bounded admission queue, configured as `queue_size`;
+- a per-pipeline single-active-run guard.
+
+If a run for the same pipeline is already active, later accepted uploads for
+that pipeline should wait in queue rather than starting concurrently.
+
+If the queue is full, the upload request should fail before consuming and
+staging the request body. The server must not allow unbounded memory or disk
+growth.
+
+## Authentication
+
+Each `http_upload` pipeline has one static upload token.
+
+Upload requests send the token in an HTTP bearer header:
+
+```http
+Authorization: Bearer <token>
+```
+
+Authentication maps the incoming bearer token to exactly one configured
+pipeline. If no pipeline matches, the request fails. If more than one pipeline
+resolves to the same token, config validation should fail before the server
+starts.
+
+Secret values must never be logged, returned in responses, or included in run
+status.
+
+## HTTP Server Boundary
+
+Add a server mode rather than trying to make `run` poll an HTTP source.
+
+The likely CLI shape is:
+
+```sh
+distributor serve --config <path>
+```
+
+The HTTP API should default to a private bind address. Operators that need
+public access, TLS termination, or mTLS should place `distributor` behind a
+reverse proxy or private network boundary unless a later roadmap explicitly
+adds in-app TLS.
+
+The server should expose:
+
+- `POST /upload`;
+- `GET /runs/<run_id>`;
+- `GET /healthz`.
+
+`GET /healthz` should return success after the server has loaded and validated
+configuration and is ready to accept requests.
+
+## Relationship To Existing Architecture
+
+`http_upload` should reuse existing code paths after staging:
+
+- upload staging should produce a local staged bundle tree;
+- staged bundle validation should use the existing bundle validation contract;
+- fan-out should use the app-layer single-pipeline run behavior where possible;
+- destination handling should remain backend-agnostic;
+- publish, transform, link, state, and notification behavior should not know
+  that the source arrived over HTTP.
+
+If the current app-layer single-pipeline runner assumes it can open and walk the
+configured source backend, the HTTP implementation should add a narrow app-layer
+entry point for "run this pipeline using this already-staged source backend"
+rather than bending `http_upload` into a fake durable storage backend.
+
+## Non-Goals
+
+The first HTTP upload implementation should not add:
+
+- destination-side HTTP upload;
+- browser UI;
+- producer execution;
+- source manifest schema changes;
+- warning-only digest mismatch behavior;
+- durable database-backed queueing;
+- producer-supplied idempotency keys;
+- zstd archive support;
+- URL-token authentication;
+- public network exposure defaults;
+- in-app TLS;
+- general-purpose workflow orchestration.
+
+## Testing Expectations
+
+Suggested coverage:
+
+- config validation accepts `http_upload` sources and rejects `http_upload`
+  destinations;
+- default staging path becomes `/var/spool/distributor/<pipeline_id>`;
+- token env references resolve through real environment and `secrets.directory`;
+- duplicate token values across pipelines fail validation;
+- upload size limit defaults to 20 MB and is enforced;
+- archive extraction rejects unsafe paths, symlinks, hardlinks, devices,
+  missing manifest, missing manifest-listed files, and nested manifests;
+- valid uploaded bundles validate through the existing bundle contract;
+- upload admission returns `accepted` and a generated run id;
+- status lookup reports queued, running, succeeded, failed, and expired states;
+- per-pipeline runs do not execute concurrently;
+- global `max_concurrency` is honored;
+- full queues reject uploads before request-body staging;
+- `GET /runs/<run_id>` returns in-memory status records until retention expiry;
+- failed extraction and failed runs clean up or retain staging data according to
+  the configured retention policy;
+- secret values never appear in logs, responses, or status records;
+- normal local, SSH/SFTP, and S3 source behavior remains unchanged.
+
+## Documentation Updates After Implementation
+
+- Update `docs/config.md` with `http_upload` source fields and defaults.
+- Update `docs/cli.md` with `serve` syntax and HTTP behavior.
+- Update `docs/operations.md` with upload, queue, status, and staging
+  workflows.
+- Update `docs/troubleshooting.md` for authentication, archive extraction,
+  validation, queue, and fan-out failures.
+- Add examples only if they are secret-free and safe to run locally.
+- Update internal docs for any new app, HTTP, queue, or ingestion packages.
+
+Keep this roadmap under `docs/roadmap/` until implemented.
+
+## Future Work
+
+The first implementation intentionally defers:
+
+- URL-token authentication;
+- zstd-compressed archive support;
+- durable status persistence across process restarts;
+- database-backed queueing;
+- producer-supplied idempotency keys;
+- run listing, cancellation, or retry endpoints;
+- in-app TLS;
+- public network exposure defaults;
+- browser UI.
+
+These items should remain out of current-behavior docs until a later roadmap
+selects and specifies them.

docs/roadmap/implementation.md

@@ -1,91 +1,328 @@
-# HTTP API Boundary Roadmap
+# HTTP Upload API Implementation Roadmap
 
 ## Purpose
 
-This roadmap records the accepted boundary for a future HTTP API. The current
-application exposes CLI commands and internal app-layer run contracts; it does
-not implement an HTTP server, HTTP routes, a `serve` command, app-level
-authentication, or in-app TLS.
+This roadmap is the canonical staged implementation plan for
+`docs/roadmap/http.md`.
 
-Implemented internal run contracts are documented in `docs/internal/app.md`.
-This file is the canonical home for future HTTP boundary decisions until an
-HTTP implementation roadmap replaces it.
+The current application supports CLI-driven distribution using configured
+local, SSH/SFTP, and S3-compatible source and destination backends. It does not
+yet implement an HTTP server, a `serve` command, `http_upload` source
+configuration, upload authentication, archive ingestion, async upload status,
+or HTTP routes.
 
-## Accepted Direction
+Future, planned, or aspirational behavior belongs under `docs/roadmap/` until
+it is implemented. Current-behavior docs must be updated only after the
+corresponding stage is complete.
 
-The future HTTP API should trigger configured distributor pipelines through the
-existing app-layer single-pipeline run path and in-memory coordinator.
+## Implementation Principles
 
-The HTTP API is intentionally narrow:
+- Treat `docs/roadmap/http.md` as the source of truth for accepted HTTP upload
+  policy.
+- Prefer standard-library HTTP, tar, gzip, and sync primitives.
+- Do not add external dependencies for the first HTTP upload implementation.
+- Do not register `http_upload` as a durable storage backend.
+- Reuse existing bundle validation, destination fan-out, transform, link,
+  state, notification, and run-report behavior after upload staging.
+- Keep deferred items out of current-behavior docs: URL-token auth, zstd,
+  durable status, database queues, producer idempotency keys, run
+  cancellation/listing, in-app TLS, public exposure defaults, and browser UI.
 
-- A trigger request accepts only a pipeline ID as application input.
-- A trigger request starts work asynchronously and returns a run ID after
-  admission.
-- Run status is read through a separate status endpoint keyed by run ID.
-- Status records expose run ID, pipeline ID, current status, timestamps, and
-  completed report or error details when available.
+## Stage 1: HTTP Upload Configuration
 
-The application remains a bundle distribution tool. The HTTP API must not turn
-`distributor` into a workflow engine, CMS, report generator, or public web
-authoring service.
+Goal: add config support for HTTP upload sources and server settings without
+adding HTTP runtime behavior.
 
-## Error Mapping
+Implementation scope:
 
-Future transport code should map app-layer errors without changing app-layer
-error ownership:
+- add top-level `server.http` config with defaults:
+  - `bind: 127.0.0.1:8080`;
+  - `staging_root: /var/spool/distributor`;
+  - `max_upload_size: 20MB`;
+  - `queue_size: 16`;
+  - `max_concurrency: 1`;
+  - `retention: 24h`;
+- add source-only `backend: http_upload`;
+- add `http_upload` source fields:
+  - required `token_env`;
+  - optional `staging_path`;
+  - optional `max_upload_size`;
+- default omitted `staging_path` to
+  `/var/spool/distributor/<pipeline_id>`;
+- default omitted source `max_upload_size` to
+  `server.http.max_upload_size`;
+- reject `http_upload` as a destination backend;
+- parse sizes with `B`, `KB`, `MB`, and `GB` suffixes using 1024 multipliers;
+- parse `retention` with `time.ParseDuration`;
+- keep literal upload tokens out of YAML.
 
-- Unknown pipeline IDs map to `404 Not Found`.
-- Duplicate in-flight runs for the same pipeline ID map to `409 Conflict`.
-- Validation, config, source, destination, publish, transform, and notification
-  errors map to transport errors according to their app-layer context.
+Documentation updates after implementation:
 
-Duplicate runs must not be queued. Run state remains in memory unless a later
-roadmap explicitly adds durable run storage.
+- update `docs/config.md` for implemented config fields and defaults;
+- update `docs/internal/config.md` for config ownership and validation rules;
+- do not document HTTP routes yet.
 
-## Context And Lifetime
+Tests:
 
-The request context guards admission. Once a run is admitted, execution is tied
-to the server or coordinator lifetime context rather than to the client request
-lifetime.
+- config loading accepts valid `server.http` and `http_upload` source config;
+- defaults apply for bind, staging root, staging path, upload size, queue size,
+  concurrency, and retention;
+- known-field decoding rejects unknown fields;
+- invalid sizes, invalid durations, missing token env, missing pipeline ids,
+  duplicate pipeline ids, and `http_upload` destinations fail validation;
+- existing local, SSH/SFTP, and S3 config behavior remains unchanged.
 
-This split allows a client disconnect or request timeout to stop waiting for
-admission without canceling a run that has already been accepted.
+Completion criteria: `go test ./internal/config` passes and no runtime code
+attempts to execute `http_upload`.
 
-## Security Boundary
+## Stage 2: Upload Archive Staging
 
-The first HTTP server should default to private binding, such as `127.0.0.1`.
-Operators should expose it through a reverse proxy, private network, or external
-mTLS when transport security or remote access is required.
+Goal: stage uploaded tar or tar.gz archives into a validated local source
+bundle tree.
 
-The first HTTP implementation should not include:
+Implementation scope:
 
-- bearer-token authentication;
-- in-app TLS configuration;
-- public-network exposure defaults.
+- add an internal ingestion package for upload archive staging;
+- support uncompressed tar and gzip-compressed tar only;
+- accept content types:
+  - `application/x-tar`;
+  - `application/gzip`;
+  - `application/x-gzip`;
+- stream request bodies to a per-run archive or staging path while enforcing
+  max upload size;
+- extract into a per-run staging directory under the pipeline staging path;
+- reject absolute paths, path traversal, backslash paths, symlinks, hardlinks,
+  devices, sockets, and other special entries;
+- require exactly one root-level `manifest.json`;
+- reject nested manifests;
+- require all manifest-listed files to exist as regular files;
+- enforce extracted size and file-count limits;
+- clean up failed extraction directories;
+- validate staged bundles through the existing source bundle contract.
 
-A later roadmap must explicitly change this security decision before any of
-those features are added.
+Documentation updates after implementation:
 
-## Non-Goals
+- update relevant `docs/internal/` docs for the new ingestion package;
+- keep user-facing HTTP docs out of current-behavior docs until the server
+  stage is implemented.
 
-The future HTTP API should not add:
+Tests:
 
-- public CLI flags for selecting one pipeline during `distributor run`;
-- a CLI framework;
-- a generic workflow engine;
-- plugin execution;
-- durable run storage;
-- app-level authentication;
-- in-app TLS.
+- valid tar and tar.gz uploads stage successfully;
+- unsupported content types fail;
+- max upload size is enforced while streaming;
+- unsafe archive entries are rejected;
+- missing manifest, nested manifest, missing listed file, digest mismatch, and
+  non-regular manifest-listed files fail validation;
+- failed extraction cleans up staging data according to the package contract.
 
-## Verification Expectations
+Completion criteria: ingestion can produce a validated staged local bundle and
+does not publish anything.
 
-Any later HTTP implementation should preserve existing CLI behavior and keep
-the app-layer run contracts tested. At minimum, it should cover:
+## Stage 3: Staged Source Pipeline Execution
 
-- trigger requests with known and unknown pipeline IDs;
-- duplicate in-flight trigger requests;
-- asynchronous acceptance and status lookup;
-- private bind defaults;
-- request-context admission behavior;
-- coordinator-lifetime run execution.
+Goal: run one configured pipeline using an already-staged local source bundle
+root.
+
+Implementation scope:
+
+- add a narrow app-layer entry point for executing one pipeline with a staged
+  local source backend/root;
+- bypass normal source backend opening only for this staged-source entry point;
+- do not register `http_upload` as a normal `storage.Backend`;
+- reuse existing destination fan-out, transform, link, state, notification, and
+  run-report behavior;
+- ensure publish, transform, link, and state code do not know the source
+  arrived over HTTP.
+
+Documentation updates after implementation:
+
+- update `docs/internal/app.md` for the staged-source app entry point;
+- update `docs/internal/bundle.md` only if source validation behavior changes.
+
+Tests:
+
+- staged valid bundles publish through configured local destinations;
+- invalid staged bundles fail before destination writes;
+- configured destination behavior for local, SSH/SFTP, and S3 remains
+  backend-agnostic;
+- run reports match existing app report semantics.
+
+Completion criteria: app-level tests prove staged local bundles can run through
+normal fan-out without HTTP server code.
+
+## Stage 4: Async Upload Queue And Status
+
+Goal: add in-memory async upload coordination, queueing, and status tracking.
+
+Implementation scope:
+
+- add an in-memory upload coordinator;
+- generate run ids shaped like
+  `<pipeline_id>.<utc_timestamp>.<random_suffix>`;
+- use statuses:
+  - `accepted`;
+  - `queued`;
+  - `running`;
+  - `succeeded`;
+  - `failed`;
+  - `expired`;
+- enforce global `max_concurrency`;
+- enforce bounded `queue_size`;
+- enforce one active running upload per pipeline;
+- queue later accepted uploads for the same pipeline instead of running them
+  concurrently;
+- reject uploads before consuming or staging the request body when the queue is
+  full;
+- keep status in memory;
+- apply time-based retention to completed status and staged run directories;
+- treat every accepted upload as a new run;
+- do not support producer-supplied idempotency keys.
+
+Documentation updates after implementation:
+
+- update internal docs for the upload coordinator;
+- do not add user-facing HTTP docs until the server stage is implemented.
+
+Tests:
+
+- run id format includes pipeline id, filesystem-safe UTC timestamp, and random
+  suffix;
+- queue size is bounded;
+- full queues reject admission before staging;
+- same-pipeline uploads serialize;
+- different pipelines run concurrently up to `max_concurrency`;
+- status transitions cover accepted, queued, running, succeeded, failed, and
+  expired;
+- final status retains run report or error details until retention expiry;
+- staging directories are retained or cleaned according to retention policy.
+
+Completion criteria: coordinator tests pass without starting an HTTP server.
+
+## Stage 5: HTTP Server And `serve` CLI
+
+Goal: expose the upload coordinator through HTTP and add the server CLI command.
+
+Implementation scope:
+
+- add `distributor serve --config <path>`;
+- load config and `secrets.directory` before starting the server;
+- resolve each `http_upload` `token_env` through the config-owned environment
+  resolver;
+- fail startup if any configured token is missing or duplicated;
+- bind to `server.http.bind`, defaulting to `127.0.0.1:8080`;
+- implement `POST /upload`;
+- implement `GET /runs/<run_id>`;
+- implement `GET /healthz`;
+- authenticate uploads with `Authorization: Bearer <token>`;
+- map each token to exactly one configured `http_upload` pipeline;
+- do not require or accept a producer-submitted pipeline id;
+- return `202 Accepted` with:
+
+  ```json
+  {"run_id":"<id>","status":"accepted"}
+  ```
+
+- return `401` for missing or invalid bearer token;
+- return `413` for oversized upload;
+- return `415` for unsupported archive content type;
+- return `503` for full queue;
+- return `404` for unknown run status;
+- never log or return secret token values.
+
+Documentation updates after implementation:
+
+- update `docs/cli.md` with `serve` syntax;
+- update `docs/config.md` with HTTP upload source and server config;
+- update `docs/troubleshooting.md` for startup, auth, upload, and status
+  failures;
+- update relevant internal docs for HTTP server package boundaries.
+
+Tests:
+
+- CLI parsing recognizes `serve --config <path>`;
+- private bind default is applied;
+- startup fails for missing or duplicate tokens;
+- auth accepts valid bearer tokens and rejects missing or invalid tokens;
+- routes return the expected status codes and JSON response shapes;
+- health succeeds after configuration is loaded and the server is ready;
+- responses, logs, and status records do not expose secret token values.
+
+Completion criteria: `httptest` route tests and CLI tests pass, and no current
+local/SSH/S3 CLI behavior regresses.
+
+## Stage 6: End-To-End HTTP Upload Flow
+
+Goal: prove the complete async HTTP upload path publishes valid bundles and
+rejects invalid ones safely.
+
+Implementation scope:
+
+- add integration-style tests using `httptest`;
+- submit valid tar and tar.gz bundles;
+- poll `GET /runs/<run_id>` until completion;
+- verify fan-out reaches configured local destinations;
+- verify invalid archives fail without publishing;
+- verify same-pipeline uploads serialize;
+- verify different pipelines can run up to `max_concurrency`;
+- verify status includes completed run report or error details.
+
+Documentation updates after implementation:
+
+- update `docs/operations.md` with an HTTP upload workflow;
+- add safe local examples only if they are secret-free and testable.
+
+Tests:
+
+- valid upload returns `202 Accepted`, then `succeeded`;
+- invalid archive returns an accepted run only when admission succeeds, then
+  transitions to `failed`;
+- successful fan-out writes expected destination outputs and state;
+- failed upload does not write destination outputs;
+- same-pipeline and cross-pipeline concurrency follow configured policy;
+- run focused package tests and `go test ./...`.
+
+Completion criteria: end-to-end HTTP upload tests pass and full test suite
+passes.
+
+## Stage 7: Documentation And Roadmap Closeout
+
+Goal: document implemented HTTP upload behavior and remove completed roadmap
+drift.
+
+Implementation scope:
+
+- update current-behavior docs for implemented HTTP upload behavior:
+  - `docs/config.md`;
+  - `docs/cli.md`;
+  - `docs/operations.md`;
+  - `docs/troubleshooting.md`;
+  - relevant `docs/internal/` docs;
+- add only secret-free, safe local examples;
+- keep deferred items out of current docs;
+- remove or rewrite completed roadmap material once behavior is fully
+  documented.
+
+Deferred items:
+
+- URL-token authentication;
+- zstd archive support;
+- durable status persistence;
+- database-backed queues;
+- producer idempotency keys;
+- run cancellation or listing;
+- in-app TLS;
+- public exposure defaults;
+- browser UI.
+
+Tests:
+
+- documentation consistency checks show implemented HTTP behavior is no longer
+  described only as future work;
+- current-behavior docs do not describe deferred behavior as available;
+- examples are valid and secret-free;
+- run focused tests for docs/examples backed by tests and `go test ./...` if
+  examples or behavior docs changed with code.
+
+Completion criteria: current docs describe implemented behavior accurately, and
+`docs/roadmap/` contains only future or deferred HTTP work.