distributor/docs/operations.md

Distributor Operations

Audience: administrators and operators who run distributor, publish bundles, operate the HTTP upload service, or recover from failed runs.

This document covers operating workflows, storage layout, safety behavior, and recovery. Command syntax lives in CLI, configuration fields live in Configuration, symptom-specific fixes live in Troubleshooting, and external contracts live under Integrations.

Normal Workflow

Validate a producer bundle before publishing:

go run ./cmd/distributor validate examples/source-bundle

Preview a configured run before writing destination content:

go run ./cmd/distributor run --config examples/local-publish.yml --dry-run

Publish after reviewing the preview:

go run ./cmd/distributor run --config examples/local-publish.yml

Use JSON output for automation:

go run ./cmd/distributor run --config examples/fan-out.yml --dry-run --format json

Use configured source diagnostics when the source is defined in YAML and may be local, SSH/SFTP, or S3-compatible storage:

go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
go run ./cmd/distributor inspect --config examples/local-publish.yml --pipeline example-source-bundle --format json

Remote examples under examples/ssh-destination.yml and examples/s3-destination.yml are load-tested templates. Edit their endpoint, path, key, bucket, prefix, and credential values for storage you control before running them.

Filesystem And Storage Layout

A source bundle is a directory containing manifest.json and every file listed in that manifest. See Source Bundle Contract. Source discovery walks beneath the configured source backend root and finds bundle directories.

Each destination has its own backend root:

• Local destinations use the configured local path.
• SSH/SFTP destinations use the configured remote path.
• S3-compatible destinations use the configured bucket plus optional prefix.

Destination path mapping controls where each source bundle is published beneath the destination root:

• preserve_relative publishes each source bundle at the same source-root-relative path.
• fixed publishes one selected source bundle at the destination root.

Fixed destinations select the newest discovered source bundle by manifest created timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins.

Published destination bundle paths contain .distributor.json. See Destination State Contract. This file is both the managed sentinel and the destination state record. It records the pipeline id, destination id, publication time, source manifest, copied outputs, generated outputs, and optional public URL metadata.

manifest.json from the source bundle is not copied as destination state.

Destination State And Retry Behavior

distributor compares the source manifest to destination .distributor.json before writing:

• No destination state and no destination content: publish new outputs.
• Matching destination state: skip as already published.
• Older destination state for the same source id: replace if transfer policy allows it.
• Newer destination state: skip by default.
• Invalid destination state, identity mismatch, different source id, or same-created digest mismatch: fail by default.
• Content without .distributor.json: fail as unmanaged content by default.

Normal replacement deletes only managed output paths recorded in .distributor.json plus the state file, then verifies the destination bundle path is empty before writing new outputs and state.

If a write fails after some outputs were written, distributor attempts to delete outputs from that failed attempt so a retry does not treat partial outputs as unmanaged content. Operators should still inspect the destination after a failed write before retrying.

Fan-out destinations are independent. If one destination fails after planning or execution begins, later destinations are still attempted. The command exits non-zero if any destination failed.

Dry Runs And Output Review

run --dry-run loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, .distributor.json, or SSH known_hosts entries.

Review these action labels before publishing:

• publish_new: destination is empty and unmanaged.
• replace_older: destination state is older than the source.
• skip_same: destination state already matches the source.
• skip_destination_newer: destination state is newer than the source and is skipped.
• force_replace: destructive replacement selected because --force is present and policy permits it.
• error: planning or execution failed for that destination.

Fixed destinations add fixed-path warnings during dry runs, including the selected source bundle and replacement warnings when the destination root would be replaced.

JSON output includes warnings, pipeline summaries, destination action records, output records, URLs when configured, final counters, and partial failure details. Fatal setup failures such as unreadable config or invalid secrets do not produce a JSON result document.

Forced Replacement Workflow

Use --force only after a dry run shows the intended bounded force_replace action:

go run ./cmd/distributor run --config <config-path> --dry-run --force
go run ./cmd/distributor run --config <config-path> --force

Forced replacement can claim unmanaged non-empty destination paths. State conflicts require both --force and transfer policy that permits replacement:

• newer destination state requires transfer.on_destination_newer: replace;
• conflict outcomes require transfer.on_conflict: replace.

Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete parent paths, sibling paths, or storage outside the destination bundle path. For fixed destinations, the destination bundle path is the backend root, so a forced replacement can clear that configured root.

--force applies only to the current invocation. There is no config field that enables forced replacement by default.

HTTP Upload Operation

The HTTP Upload API Contract defines request and response details. distributor serve runs the HTTP upload API for pipelines whose source backend is http_upload. Each bearer token maps to exactly one configured upload pipeline. Token values come from the process environment or secrets.directory, not from YAML literal values.

Start the maintained local example:

DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN=<token> \
  go run ./cmd/distributor serve --config examples/http-upload-local.yml

Readiness:

curl http://127.0.0.1:8080/healthz

Upload one tar or tar.gz source bundle archive:

curl -X POST http://127.0.0.1:8080/upload \
  -H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \
  -H "Content-Type: application/gzip" \
  --data-binary @bundle.tar.gz

Accepted uploads return after the archive is staged and validated:

{"run_id":"example-http-upload.20260604T120000Z.abcdef12","status":"accepted"}

Poll status while the in-memory record is retained:

curl http://127.0.0.1:8080/runs/<run-id>

Status values are accepted, queued, running, succeeded, and failed. Completed records expire after server.http.retention. Expiration removes committed staged bundle directories for completed uploads. Restarting the process clears upload status, queue state, and in-memory records.

Upload admission is bounded by server.http.queue_size. Publication concurrency is bounded by server.http.max_concurrency, and the coordinator does not run two uploads for the same pipeline at the same time.

The upload server accepts application/x-tar, application/gzip, and application/x-gzip. Archives are extracted into a temporary staging directory, must contain exactly one root-level manifest.json, and must validate as one complete source bundle before a run id is issued. Per-source max_upload_size bounds both uploaded archive size and extracted bundle size. The implementation also caps extracted file count.

The default bind address is private loopback. Put TLS, public routing, rate limiting, and external access policy in a reverse proxy or deployment layer.

Remote Backend Notes

SSH/SFTP

SSH execution uses native SFTP. See SSH/SFTP Integration. It does not shell out to ssh, scp, or rsync.

Configure ssh_key_file, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported.

The default host key policy is accept-new. During dry runs, unknown host keys may be accepted for the current connection but are not persisted. Changed host keys are fatal for strict and accept-new. host_key_policy: off disables host key checking and should be limited to controlled test environments.

S3-Compatible Storage

S3 execution uses the AWS SDK for Go v2. See S3-Compatible Storage Integration. Configure an endpoint, bucket, optional prefix, optional region, optional path-style setting, and optional explicit credential variable names.

When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or secrets.directory. When omitted, the AWS SDK default credential chain is used as-is.

Normal replacement and failed-write cleanup delete only managed output objects recorded in .distributor.json plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.

Secrets Operation

secrets.directory is loaded during run, serve, and configured-source validate or inspect before credential-consuming work starts. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts.

Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, commands emit a warning naming the ignored secret variable without printing either value. The process environment is not modified.

Cleanup And Recovery

Use these recovery boundaries:

• For source validation failures, regenerate the source bundle and manifest together.
• For an empty or missing destination, rerun after fixing config or storage access.
• For unmanaged destination content, move unrelated files aside or use a different destination path before publishing.
• For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun --dry-run.
• For state conflicts, verify the source, pipeline, destination, and existing .distributor.json before considering --force.
• For HTTP upload failures, inspect /runs/<run-id> while retained; after expiry or restart, rely on destination state and logs/output from the publishing run.

Do not edit .distributor.json during normal recovery. Treat it as the managed state record used for comparison and safe cleanup.

For symptom-specific fixes, see Troubleshooting.