17 KiB
Distributor Troubleshooting
Audience: administrators and operators diagnosing distributor command, configuration, publishing, storage, or HTTP upload failures.
Each entry lists the symptom, likely cause, diagnostic step, safe fix, and relevant reference link. Command syntax lives in CLI, configuration fields live in Configuration, and operating procedures live in Operations.
Config File Is Missing
Symptom: load config ... no such file or directory.
Likely cause: --config points to a missing file, or --config was omitted and /usr/local/etc/distributor/config.yml is not installed.
Diagnostic:
ls -l <config-path>
Safe fix: pass an existing file with --config, or install a config at the default path.
Reference: Configuration.
Config Contains An Unknown Field
Symptom: parse config ... field not found.
Likely cause: the YAML contains a key that is not part of the implemented config schema.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
Safe fix: remove or rename unsupported fields using the canonical config reference.
Reference: Configuration.
Backend Name Or Placement Is Invalid
Symptom: backend ... is unsupported or http_upload is only supported for sources.
Likely cause: a backend name is misspelled, not executable, or configured in the wrong role.
Diagnostic:
rg -n "backend:" <config-path>
Safe fix: use local, ssh, or s3 for executable sources and destinations. Use http_upload only as a source served by distributor serve.
Reference: Configuration.
CLI Arguments Select The Wrong Source Mode
Symptom: configured source mode requires --pipeline, does not accept a local path with --config, --pipeline, or --bundle, validate command requires a path, or inspect command requires a path.
Likely cause: validate or inspect mixed local path mode with configured source mode, or omitted the required source selector.
Diagnostic:
go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help
Safe fix: use either distributor validate <path> / distributor inspect <path>, or use --config <path> --pipeline <id> with optional --bundle <path>.
Reference: CLI.
Output Format Is Invalid
Symptom: format must be text or json.
Likely cause: an unsupported value was passed to --format.
Diagnostic:
go run ./cmd/distributor run --help
Safe fix: use --format text or --format json.
Reference: CLI.
JSON Mode Wrote No JSON Document
Symptom: --format json exits non-zero and stdout has no JSON result.
Likely cause: the command failed before it could construct a result, such as invalid arguments, missing config, unreadable secrets, or source setup failure.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --format json
Safe fix: read stderr, fix the setup problem, then rerun. Partial destination failures during run can produce JSON; fatal setup failures do not.
Reference: CLI.
Source Pipeline Is Not Found
Symptom: pipeline "<id>" not found.
Likely cause: configured source diagnostics or upload processing selected a pipeline id that is absent from the loaded config.
Diagnostic:
rg -n "id:" <config-path>
Safe fix: pass an existing --pipeline value or correct the pipeline id in config.
Reference: Configuration.
Source Bundles Are Not Found
Symptom: no bundles found, no bundles found under ".", stat ssh ... not_found, or stat s3 ... not_found.
Likely cause: the source root, source-root-relative bundle path, S3 prefix, SSH path, or permissions do not expose a directory containing manifest.json.
Diagnostic:
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
Safe fix: correct the configured source backend root, permissions, prefix, or --bundle path. The selected bundle directory must contain manifest.json.
Reference: Operations.
Source Manifest Or Files Fail Validation
Symptom: sha256 mismatch, size mismatch, digest mismatch, missing manifest fields, or unsafe source paths.
Likely cause: files changed after manifest.json was written, the manifest digest is stale, or the producer wrote invalid bundle paths.
Diagnostic:
go run ./cmd/distributor validate <source-root>
Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source validation failures.
Reference: Operations.
Destination Has Unmanaged Content
Symptom: destination has content but no distributor state or a plan reason containing fail_unmanaged.
Likely cause: the destination bundle path contains files but no valid .distributor.json, so distributor will not claim it by default.
Diagnostic:
find <destination-path> -maxdepth 2 -print
Safe fix: choose an empty destination path, move unrelated files aside, or preview run --dry-run --force only after confirming the reported bundle path is safe to replace.
Reference: Operations.
Destination State Conflicts With Source
Symptom: fail_conflict, destination source id differs, same id and created time but different digest, pipeline id ... does not match, or destination id ... does not match.
Likely cause: .distributor.json belongs to a different pipeline, destination, source id, or same-created source with different content.
Diagnostic:
cat <destination-path>/.distributor.json
go run ./cmd/distributor inspect <source-root>
Safe fix: verify the source and destination are intended to match. Use a separate destination path for unrelated content. To replace the existing state, configure transfer.on_conflict: replace, preview with --dry-run --force, then publish with --force.
Reference: Operations.
Destination Is Newer Than Source
Symptom: skip_destination_newer or destination is newer and replacement requires --force.
Likely cause: the destination state records a source manifest with a later created timestamp than the current source.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run --format json
Safe fix: keep the default skip behavior unless replacement is intentional. To replace newer state, configure transfer.on_destination_newer: replace, preview with --dry-run --force, then publish with --force.
Reference: Operations.
Forced Replacement Appears In A Plan
Symptom: dry-run output includes force_replace.
Likely cause: the run used --force, and planning selected a supported destructive replacement.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run --force
Safe fix: inspect the pipeline id, destination id, backend, and bundle path. Proceed only if deleting everything inside that destination bundle path is intended.
Reference: Operations.
Output Path Collision
Symptom: destination output path collision.
Likely cause: publication would write two outputs to the same destination path, such as copying report.html while also generating report.html from report.md.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide.
Reference: Configuration.
Run Failed After Writing Some Files
Symptom: a destination write failed and the command exited non-zero after partial work.
Likely cause: storage write failure, permission issue, network interruption, or object-store error during publish execution.
Diagnostic:
find <destination-path> -maxdepth 2 -print
Safe fix: inspect the destination bundle path printed in the error. distributor attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. Rerun --dry-run before publishing again.
Reference: Operations.
JSON Run Result Has ok: false
Symptom: run --format json exits non-zero with a JSON result where ok is false.
Likely cause: at least one destination failed after planning or execution began, while other destination results were still available.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --format json
Safe fix: inspect the top-level errors array, destination actions, output errors, and summary. Fix failed destinations, then preview with --dry-run --format json.
Reference: CLI.
Secrets Directory Is Missing Or Unreadable
Symptom: load secrets directory ... no such file or directory, permission denied, or secret filename ... is invalid.
Likely cause: secrets.directory points to a missing or unreadable directory, or it contains a regular file whose name is not a valid credential variable name.
Diagnostic:
ls -ld <secrets-directory>
find <secrets-directory> -maxdepth 1 -type f -printf '%f\n'
Safe fix: mount or create the directory, adjust permissions for the service user, or rename/remove invalid secret files. Secret filenames must match [A-Za-z_][A-Za-z0-9_]*.
Reference: Configuration.
Credential Variable Is Missing Or Empty
Symptom: credential environment variable ... is not set, credential environment variable ... is empty, or S3 authentication errors such as AccessDenied, InvalidAccessKeyId, or SignatureDoesNotMatch.
Likely cause: configured S3 credential variable names are not available through the process environment or secrets.directory, are empty, or do not authorize the requested bucket/prefix.
Diagnostic:
env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
ls -l <secrets-directory>
Safe fix: provide both configured S3 credential values, correct IAM/service permissions, or omit explicit credential fields to use the AWS SDK default credential chain.
Reference: Configuration.
Secret File Is Ignored In Favor Of Environment
Symptom: secret ... ignored because the real environment already has that variable.
Likely cause: the same credential name exists in the process environment and secrets.directory with different values.
Diagnostic:
env | cut -d= -f1 | rg '^<variable-name>$'
ls -l <secrets-directory>/<variable-name>
Safe fix: remove one source of the credential or intentionally keep the process environment value. distributor does not print either value.
Reference: Operations.
SSH Auth Is Not Configured
Symptom: no SSH auth methods configured.
Likely cause: no SSH agent is available and ssh_key_file is missing or unreadable.
Diagnostic:
test -n "$SSH_AUTH_SOCK" && ssh-add -l
ls -l <ssh-key-file>
Safe fix: start an SSH agent with an appropriate key loaded, or configure a readable private key with ssh_key_file.
Reference: Configuration.
SSH Host Key Fails
Symptom: host key ... is unknown, known_hosts is required, or host key ... has changed.
Likely cause: strict host key checking has no trusted key, accept-new cannot persist a new key, or the remote host key differs from the stored key.
Diagnostic:
ls -l <known-hosts-path>
ssh-keygen -F <host> -f <known-hosts-path>
Safe fix: pre-populate known_hosts for strict, configure a writable known_hosts path for accept-new, or verify the server identity before updating a changed key. Do not disable host key checking to bypass an unexpected changed key.
Reference: Operations.
S3 Prefix Is Invalid
Symptom: prefix must be a clean relative slash-separated path.
Likely cause: the S3 prefix contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
Safe fix: use a clean relative prefix such as reports/archive, or omit prefix.
Reference: Configuration.
S3 Location Or Connectivity Fails
Symptom: NoSuchBucket, InvalidBucketName, not_found, endpoint connection failures, or TLS/network errors.
Likely cause: endpoint, bucket, prefix, region, path-style mode, network routing, or credentials are wrong for the service.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
curl -I <endpoint>
Safe fix: verify endpoint, bucket, region, prefix, and force_path_style. For S3-compatible services, keep force_path_style: true unless the service requires virtual-host addressing. Distributor does not provide an insecure TLS bypass setting.
Reference: Operations.
HTTP Server Cannot Bind
Symptom: bind HTTP server ... address already in use.
Likely cause: another process is listening on server.http.bind.
Diagnostic:
ss -ltnp | rg '<port>'
Safe fix: stop the conflicting process or configure a different bind address.
Reference: Configuration.
HTTP Upload Token Is Missing Or Duplicated
Symptom: upload token environment variable ... is not set, ... is empty, or upload token environment variables ... resolve to the same value.
Likely cause: an http_upload source references a missing/empty token_env, or two upload pipelines resolve to the same bearer token.
Diagnostic:
rg -n 'token_env:' <config-path>
env | cut -d= -f1 | rg '^<token-variable>$'
ls -l <secrets-directory>/<token-variable>
Safe fix: provide one distinct non-empty token value per upload pipeline through the process environment or secrets.directory. Do not put literal tokens in YAML.
Reference: Configuration.
Upload Request Is Unauthorized
Symptom: POST /upload returns 401.
Likely cause: the request lacks Authorization: Bearer <token>, has an empty token, or uses a token that does not match any configured upload pipeline.
Diagnostic:
curl -i -X POST http://127.0.0.1:8080/upload \
-H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \
-H "Content-Type: application/x-tar" \
--data-binary @bundle.tar
Safe fix: use the token value resolved by the configured token_env. Do not include token values in logs or tickets.
Reference: Operations.
Upload Request Is Rejected Before A Run ID
Symptom: POST /upload returns 400, 413, 415, or 503.
Likely cause: the request included a pipeline or pipeline_id query, archive content is malformed, the body exceeds size limits, content type is unsupported, or the in-memory upload queue is full.
Diagnostic:
tar -tf bundle.tar
tar -tzf bundle.tar.gz
rg -n 'max_upload_size|queue_size|max_concurrency' <config-path>
Safe fix: send one valid tar or tar.gz source bundle archive with Content-Type: application/x-tar, application/gzip, or application/x-gzip; remove pipeline query parameters; reduce archive size or raise the configured limit; retry after queue pressure drops.
Reference: Operations.
Upload Idempotency Conflict
Symptom: POST /upload returns 409.
Likely cause: the request reused an Idempotency-Key for the same authenticated pipeline with a different source manifest, or another request with the same key is still being staged before its manifest is known.
Diagnostic:
curl -i -X POST http://127.0.0.1:8080/upload \
-H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \
-H "Content-Type: application/gzip" \
-H "Idempotency-Key: <key>" \
--data-binary @bundle.tar.gz
Safe fix: if the response includes "retryable":true, retry the same upload later with the same key. Otherwise, inspect the producer operation and use the same key only for the same source bundle.
Reference: HTTP Upload API Contract.
Upload Status Is Missing
Symptom: GET /runs/<run_id> returns 404.
Likely cause: the run id is wrong, the process restarted, or the retained status record expired after server.http.retention.
Diagnostic:
curl -i http://127.0.0.1:8080/runs/<run-id>
rg -n 'retention:' <config-path>
Safe fix: use the exact run_id returned by upload admission. Increase retention if operators need a longer status window.
Reference: Operations.