Add HTTP upload end-to-end coverage
This commit is contained in:
@@ -68,6 +68,83 @@ Validate one configured source without opening destinations:
|
||||
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
|
||||
```
|
||||
|
||||
## HTTP Upload Workflow
|
||||
|
||||
`distributor serve` runs the HTTP upload API for pipelines whose source backend
|
||||
is `http_upload`. Each upload token maps to one configured pipeline, and each
|
||||
accepted archive is staged, validated, and published through the same
|
||||
destination fan-out path used by local source runs.
|
||||
|
||||
Minimal local HTTP upload configuration:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
http:
|
||||
bind: 127.0.0.1:8080
|
||||
staging_root: /var/spool/distributor
|
||||
max_upload_size: 20MB
|
||||
queue_size: 16
|
||||
max_concurrency: 1
|
||||
retention: 24h
|
||||
secrets:
|
||||
directory: /run/secrets/distributor
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: http_upload
|
||||
token_env: REPORTS_UPLOAD_TOKEN
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: /srv/reports/archive
|
||||
```
|
||||
|
||||
Create `/run/secrets/distributor/REPORTS_UPLOAD_TOKEN` or set the real process
|
||||
environment variable before starting the server. Distributor does not read
|
||||
literal upload tokens from YAML.
|
||||
|
||||
Start the server:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor serve --config <config-path>
|
||||
```
|
||||
|
||||
Submit a tar or tar.gz source bundle:
|
||||
|
||||
```sh
|
||||
curl -X POST http://127.0.0.1:8080/upload \
|
||||
-H "Authorization: Bearer $REPORTS_UPLOAD_TOKEN" \
|
||||
-H "Content-Type: application/gzip" \
|
||||
--data-binary @bundle.tar.gz
|
||||
```
|
||||
|
||||
Successful admission returns a run id:
|
||||
|
||||
```json
|
||||
{"run_id":"reports.20260603T120000Z.abcdef12","status":"accepted"}
|
||||
```
|
||||
|
||||
Poll status until it reaches `succeeded` or `failed`:
|
||||
|
||||
```sh
|
||||
curl http://127.0.0.1:8080/runs/<run-id>
|
||||
```
|
||||
|
||||
The status record includes the completed run report on successful publication
|
||||
or error details on failure. Status is memory-only and expires after
|
||||
`server.http.retention`; completed staged bundle directories are removed on
|
||||
expiry. Restarting the process clears upload status and queue state.
|
||||
|
||||
Use `GET /healthz` for readiness after config and tokens load:
|
||||
|
||||
```sh
|
||||
curl http://127.0.0.1:8080/healthz
|
||||
```
|
||||
|
||||
The default bind address is private loopback. Put TLS, public routing,
|
||||
rate-limiting, and external access policy in a reverse proxy or deployment
|
||||
layer.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
|
||||
@@ -322,7 +399,10 @@ secrets:
|
||||
directory: /run/secrets/distributor
|
||||
```
|
||||
|
||||
The directory is loaded during `run` and configured-source `validate` or `inspect` before any backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts.
|
||||
The directory is loaded during `run`, `serve`, and configured-source `validate`
|
||||
or `inspect` before credential-consuming work starts. If the directory is
|
||||
missing, unreadable, or contains an invalid secret filename, the command fails
|
||||
before storage work starts.
|
||||
|
||||
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` and configured-source diagnostics print a warning naming the ignored secret file variable without printing either value. The process environment is not changed.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user