Add HTTP upload end-to-end coverage

This commit is contained in:
2026-06-03 15:26:49 +00:00
parent 6beef58dbf
commit 0f1ef9e622
2 changed files with 500 additions and 1 deletions

View File

@@ -68,6 +68,83 @@ Validate one configured source without opening destinations:
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
```
## HTTP Upload Workflow
`distributor serve` runs the HTTP upload API for pipelines whose source backend
is `http_upload`. Each upload token maps to one configured pipeline, and each
accepted archive is staged, validated, and published through the same
destination fan-out path used by local source runs.
Minimal local HTTP upload configuration:
```yaml
server:
http:
bind: 127.0.0.1:8080
staging_root: /var/spool/distributor
max_upload_size: 20MB
queue_size: 16
max_concurrency: 1
retention: 24h
secrets:
directory: /run/secrets/distributor
pipelines:
- id: reports
source:
backend: http_upload
token_env: REPORTS_UPLOAD_TOKEN
destinations:
- id: archive
backend: local
path: /srv/reports/archive
```
Create `/run/secrets/distributor/REPORTS_UPLOAD_TOKEN` or set the real process
environment variable before starting the server. Distributor does not read
literal upload tokens from YAML.
Start the server:
```sh
go run ./cmd/distributor serve --config <config-path>
```
Submit a tar or tar.gz source bundle:
```sh
curl -X POST http://127.0.0.1:8080/upload \
-H "Authorization: Bearer $REPORTS_UPLOAD_TOKEN" \
-H "Content-Type: application/gzip" \
--data-binary @bundle.tar.gz
```
Successful admission returns a run id:
```json
{"run_id":"reports.20260603T120000Z.abcdef12","status":"accepted"}
```
Poll status until it reaches `succeeded` or `failed`:
```sh
curl http://127.0.0.1:8080/runs/<run-id>
```
The status record includes the completed run report on successful publication
or error details on failure. Status is memory-only and expires after
`server.http.retention`; completed staged bundle directories are removed on
expiry. Restarting the process clears upload status and queue state.
Use `GET /healthz` for readiness after config and tokens load:
```sh
curl http://127.0.0.1:8080/healthz
```
The default bind address is private loopback. Put TLS, public routing,
rate-limiting, and external access policy in a reverse proxy or deployment
layer.
## Filesystem Layout
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
@@ -322,7 +399,10 @@ secrets:
directory: /run/secrets/distributor
```
The directory is loaded during `run` and configured-source `validate` or `inspect` before any backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts.
The directory is loaded during `run`, `serve`, and configured-source `validate`
or `inspect` before credential-consuming work starts. If the directory is
missing, unreadable, or contains an invalid secret filename, the command fails
before storage work starts.
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` and configured-source diagnostics print a warning naming the ignored secret file variable without printing either value. The process environment is not changed.