# Distributor Troubleshooting Audience: administrators and operators diagnosing `distributor` command, configuration, publishing, storage, or HTTP upload failures. Each entry lists the symptom, likely cause, diagnostic step, safe fix, and relevant reference link. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), and operating procedures live in [Operations](operations.md). ## Config File Is Missing Symptom: `load config ... no such file or directory`. Likely cause: `--config` points to a missing file, or `--config` was omitted and `/usr/local/etc/distributor/config.yml` is not installed. Diagnostic: ```sh ls -l ``` Safe fix: pass an existing file with `--config`, or install a config at the default path. Reference: [Configuration](config.md#config-file-loading). ## Config Contains An Unknown Field Symptom: `parse config ... field not found`. Likely cause: the YAML contains a key that is not part of the implemented config schema. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run ``` Safe fix: remove or rename unsupported fields using the canonical config reference. Reference: [Configuration](config.md). ## Backend Name Or Placement Is Invalid Symptom: `backend ... is unsupported` or `http_upload is only supported for sources`. Likely cause: a backend name is misspelled, not executable, or configured in the wrong role. Diagnostic: ```sh rg -n "backend:" ``` Safe fix: use `local`, `ssh`, or `s3` for executable sources and destinations. Use `http_upload` only as a source served by `distributor serve`. Reference: [Configuration](config.md#backend-reference). ## CLI Arguments Select The Wrong Source Mode Symptom: `configured source mode requires --pipeline`, `does not accept a local path with --config, --pipeline, or --bundle`, `validate command requires a path`, or `inspect command requires a path`. Likely cause: `validate` or `inspect` mixed local path mode with configured source mode, or omitted the required source selector. Diagnostic: ```sh go run ./cmd/distributor validate --help go run ./cmd/distributor inspect --help ``` Safe fix: use either `distributor validate ` / `distributor inspect `, or use `--config --pipeline ` with optional `--bundle `. Reference: [CLI](cli.md#validate). ## Reconcile-State Selector Is Missing Or Wrong Symptom: `reconcile-state requires --config`, `requires --pipeline`, `requires --destination`, `pipeline "" not found`, `destination not found`, or `state owner is ... not ...`. Likely cause: the command did not identify one configured destination root, or the selected root contains state for a different single-owner pipeline/destination. Diagnostic: ```sh go run ./cmd/distributor reconcile-state --help rg -n 'pipelines:|destinations:|id:' cat /.distributor.json ``` Safe fix: pass the configured `--config`, `--pipeline`, and `--destination` values that identify the destination root containing the state file. For unrelated single-owner state, use the correct config selector or a separate destination root; `reconcile-state` will not take over mismatched state. Reference: [CLI](cli.md#reconcile-state). ## Output Format Is Invalid Symptom: `format must be text or json`. Likely cause: an unsupported value was passed to `--format`. Diagnostic: ```sh go run ./cmd/distributor run --help ``` Safe fix: use `--format text` or `--format json`. Reference: [CLI](cli.md#common-output-format). ## JSON Mode Wrote No JSON Document Symptom: `--format json` exits non-zero and stdout has no JSON result. Likely cause: the command failed before it could construct a result, such as invalid arguments, missing config, unreadable secrets, or source setup failure. Diagnostic: ```sh go run ./cmd/distributor run --config --format json ``` Safe fix: read stderr, fix the setup problem, then rerun. Partial destination failures during `run` can produce JSON; fatal setup failures do not. Reference: [CLI](cli.md#output-and-exit-behavior). ## Source Pipeline Is Not Found Symptom: `pipeline "" not found`. Likely cause: configured source diagnostics or upload processing selected a pipeline id that is absent from the loaded config. Diagnostic: ```sh rg -n "id:" ``` Safe fix: pass an existing `--pipeline` value or correct the pipeline id in config. Reference: [Configuration](config.md#pipelines). ## Source Bundles Are Not Found Symptom: `no bundles found`, `no bundles found under "."`, `stat ssh ... not_found`, or `stat s3 ... not_found`. Likely cause: the source root, source-root-relative bundle path, S3 prefix, SSH path, or permissions do not expose a directory containing `manifest.json`. Diagnostic: ```sh go run ./cmd/distributor validate --config --pipeline ``` Safe fix: correct the configured source backend root, permissions, prefix, or `--bundle` path. The selected bundle directory must contain `manifest.json`. Reference: [Operations](operations.md#filesystem-and-storage-layout). ## Source Manifest Or Files Fail Validation Symptom: `sha256 mismatch`, `size mismatch`, `digest mismatch`, missing manifest fields, or unsafe source paths. Likely cause: files changed after `manifest.json` was written, the manifest digest is stale, or the producer wrote invalid bundle paths. Diagnostic: ```sh go run ./cmd/distributor validate ``` Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source validation failures. Reference: [Operations](operations.md#cleanup-and-recovery). ## Destination Has Unmanaged Content Symptom: `destination has content but no distributor state` or a plan reason containing `fail_unmanaged`. Likely cause: the destination bundle path contains files but no valid `.distributor.json`, so `distributor` will not claim it by default. Diagnostic: ```sh find -maxdepth 2 -print ``` Safe fix: choose an empty destination path, move unrelated files aside, or preview `run --dry-run --force` only after confirming the reported bundle path is safe to replace. Reference: [Operations](operations.md#forced-replacement-workflow). ## Destination State Conflicts With Source Symptom: `fail_conflict`, `destination source id differs`, `same id and created time but different digest`, `pipeline id ... does not match`, or `destination id ... does not match`. Likely cause: `.distributor.json` belongs to a different pipeline, destination, source id, or same-created source with different content. Diagnostic: ```sh cat /.distributor.json go run ./cmd/distributor inspect ``` Safe fix: verify the source and destination are intended to match. Use a separate destination path for unrelated content. To replace the existing state, configure `transfer.on_conflict: replace`, preview with `--dry-run --force`, then publish with `--force`. Reference: [Operations](operations.md#destination-state-and-retry-behavior). ## Destination State References Missing Managed Outputs Symptom: `reconcile-state --dry-run` reports `status=would_change` or JSON `missing_managed_outputs` entries. Likely cause: files that were recorded as managed in `.distributor.json` were removed outside `distributor`, or a previous external cleanup removed destination files without updating state. Diagnostic: ```sh go run ./cmd/distributor reconcile-state \ --config \ --pipeline \ --destination \ --dry-run ``` Safe fix: if the missing files should no longer be managed, rerun the same command without `--dry-run` to remove only the missing managed output records from `.distributor.json`. The command does not delete destination files or adopt unmanaged entries. If the files should exist, restore them from backup or republish instead. Reference: [Operations](operations.md#destination-state-repair). ## Destination Is Newer Than Source Symptom: `skip_destination_newer` or `destination is newer and replacement requires --force`. Likely cause: the destination state records a source manifest with a later `created` timestamp than the current source. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run --format json ``` Safe fix: keep the default skip behavior unless replacement is intentional. To replace newer state, configure `transfer.on_destination_newer: replace`, preview with `--dry-run --force`, then publish with `--force`. Reference: [Operations](operations.md#forced-replacement-workflow). ## Forced Replacement Appears In A Plan Symptom: dry-run output includes `force_replace`. Likely cause: the run used `--force`, and planning selected a supported destructive replacement. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run --force ``` Safe fix: inspect the pipeline id, destination id, backend, and bundle path. Proceed only if deleting everything inside that destination bundle path is intended. Reference: [Operations](operations.md#forced-replacement-workflow). ## Output Path Collision Symptom: `destination output path collision` or `merge output path ... exists but is not managed by destination state`. Likely cause: publication would write two outputs to the same destination path, such as copying `report.html` while also generating `report.html` from `report.md`. For merge reconciliation, it can also mean a planned output path already exists in storage but is not recorded in `.distributor.json` as managed. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run ``` Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide. For merge reconciliation, move unmanaged content aside, choose another destination path, or use replacement/forced replacement only when deleting the destination bundle path is intended. Reference: [Configuration](config.md#publish-and-transform-policy). ## Shared-Root Ownership Conflict Symptom: `fail_conflict` with a reason like `destination output path ... is owned by /`. Likely cause: a `state.mode: shared_root` destination planned an output path already recorded in `.distributor.json` for another pipeline/destination owner. Diagnostic: ```sh cat /.distributor.json go run ./cmd/distributor run --config --dry-run ``` Safe fix: change one owner so it writes a different output path, use a separate destination root, or intentionally replace the whole destination root only after previewing with `--dry-run --force`. Reference: [Operations](operations.md#destination-state-and-retry-behavior). ## Shared-Root Unmanaged Path Collision Symptom: `fail_unmanaged` with a reason like `destination output path ... exists but is not managed by destination state`. Likely cause: a `state.mode: shared_root` destination planned a new output path that already exists in storage but is not recorded as managed in `.distributor.json`. Diagnostic: ```sh find -maxdepth 2 -print cat /.distributor.json ``` Safe fix: move the unmanaged file aside, change the planned output path, or use forced replacement only when deleting the configured destination root is intended. Reference: [Operations](operations.md#forced-replacement-workflow). ## Run Failed After Writing Some Files Symptom: a destination write failed and the command exited non-zero after partial work. Likely cause: storage write failure, permission issue, network interruption, or object-store error during publish execution. Diagnostic: ```sh find -maxdepth 2 -print ``` Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. In merge mode, previously managed retained or overwritten outputs may remain intentionally. Rerun `--dry-run` before publishing again. Reference: [Operations](operations.md#destination-state-and-retry-behavior). ## JSON Run Result Has `ok: false` Symptom: `run --format json` exits non-zero with a JSON result where `ok` is `false`. Likely cause: at least one destination failed after planning or execution began, while other destination results were still available. Diagnostic: ```sh go run ./cmd/distributor run --config --format json ``` Safe fix: inspect the top-level `errors` array, destination actions, output errors, and summary. Fix failed destinations, then preview with `--dry-run --format json`. Reference: [CLI](cli.md#output-and-exit-behavior). ## Secrets Directory Is Missing Or Unreadable Symptom: `load secrets directory ... no such file or directory`, `permission denied`, or `secret filename ... is invalid`. Likely cause: `secrets.directory` points to a missing or unreadable directory, or it contains a regular file whose name is not a valid credential variable name. Diagnostic: ```sh ls -ld find -maxdepth 1 -type f -printf '%f\n' ``` Safe fix: mount or create the directory, adjust permissions for the service user, or rename/remove invalid secret files. Secret filenames must match `[A-Za-z_][A-Za-z0-9_]*`. Reference: [Configuration](config.md#secrets). ## Credential Variable Is Missing Or Empty Symptom: `credential environment variable ... is not set`, `credential environment variable ... is empty`, or S3 authentication errors such as `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch`. Likely cause: configured S3 credential variable names are not available through the process environment or `secrets.directory`, are empty, or do not authorize the requested bucket/prefix. Diagnostic: ```sh env | cut -d= -f1 | rg '^(|)$' ls -l ``` Safe fix: provide both configured S3 credential values, correct IAM/service permissions, or omit explicit credential fields to use the AWS SDK default credential chain. Reference: [Configuration](config.md#s3-compatible-backend). ## Secret File Is Ignored In Favor Of Environment Symptom: `secret ... ignored because the real environment already has that variable`. Likely cause: the same credential name exists in the process environment and `secrets.directory` with different values. Diagnostic: ```sh env | cut -d= -f1 | rg '^$' ls -l / ``` Safe fix: remove one source of the credential or intentionally keep the process environment value. `distributor` does not print either value. Reference: [Operations](operations.md#secrets-operation). ## SSH Auth Is Not Configured Symptom: `no SSH auth methods configured`. Likely cause: no SSH agent is available and `ssh_key_file` is missing or unreadable. Diagnostic: ```sh test -n "$SSH_AUTH_SOCK" && ssh-add -l ls -l ``` Safe fix: start an SSH agent with an appropriate key loaded, or configure a readable private key with `ssh_key_file`. Reference: [Configuration](config.md#sshsftp-backend). ## SSH Host Key Fails Symptom: `host key ... is unknown`, `known_hosts is required`, or `host key ... has changed`. Likely cause: strict host key checking has no trusted key, `accept-new` cannot persist a new key, or the remote host key differs from the stored key. Diagnostic: ```sh ls -l ssh-keygen -F -f ``` Safe fix: pre-populate `known_hosts` for `strict`, configure a writable `known_hosts` path for `accept-new`, or verify the server identity before updating a changed key. Do not disable host key checking to bypass an unexpected changed key. Reference: [Operations](operations.md#sshsftp). ## S3 Prefix Is Invalid Symptom: `prefix must be a clean relative slash-separated path`. Likely cause: the S3 prefix contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run ``` Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`. Reference: [Configuration](config.md#s3-compatible-backend). ## S3 Location Or Connectivity Fails Symptom: `NoSuchBucket`, `InvalidBucketName`, `not_found`, endpoint connection failures, or TLS/network errors. Likely cause: endpoint, bucket, prefix, region, path-style mode, network routing, or credentials are wrong for the service. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run curl -I ``` Safe fix: verify `endpoint`, `bucket`, `region`, `prefix`, and `force_path_style`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing. Distributor does not provide an insecure TLS bypass setting. Reference: [Operations](operations.md#s3-compatible-storage). ## HTTP Server Cannot Bind Symptom: `bind HTTP server ... address already in use`. Likely cause: another process is listening on `server.http.bind`. Diagnostic: ```sh ss -ltnp | rg '' ``` Safe fix: stop the conflicting process or configure a different bind address. Reference: [Configuration](config.md#serverhttp). ## HTTP Upload Token Is Missing Or Duplicated Symptom: `upload token environment variable ... is not set`, `... is empty`, or `upload token environment variables ... resolve to the same value`. Likely cause: a top-level upload token record references a missing or empty `token_env`, or two token records resolve to the same bearer token. Diagnostic: ```sh rg -n 'token_env:' env | cut -d= -f1 | rg '^$' ls -l / ``` Safe fix: provide one distinct non-empty token value per upload token record through the process environment or `secrets.directory`. Do not put literal tokens in YAML. Reference: [Configuration](config.md#upload_tokens). ## Upload Request Is Unauthorized Symptom: `POST /v1/pipelines//upload` returns `401`. Likely cause: the request lacks `Authorization: Bearer `, has an empty token, or uses a token that does not match any configured upload token record. Diagnostic: ```sh curl -i -X POST http://127.0.0.1:8080/v1/pipelines//upload \ -H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \ -H "Content-Type: application/x-tar" \ --data-binary @bundle.tar ``` Safe fix: use the token value resolved by the configured `token_env`. Do not include token values in logs or tickets. Reference: [Operations](operations.md#http-upload-operation). ## Upload Request Is Forbidden Symptom: `POST /v1/pipelines//upload` returns `403`. Likely cause: the bearer token is valid, but its configured `allow_pipelines` list does not include the requested upload pipeline. Diagnostic: ```sh rg -n 'upload_tokens:|allow_pipelines:|id:' ``` Safe fix: request the intended pipeline id, or update the token allowlist to include the configured `http_upload` pipeline that this producer may submit to. Reference: [Configuration](config.md#upload_tokens). ## Upload Request Is Rejected Before A Run ID Symptom: `POST /v1/pipelines//upload` returns `400`, `413`, `415`, or `503`. Likely cause: the request path has an invalid pipeline id, included a `pipeline` or `pipeline_id` query, archive content is malformed, the body exceeds size limits, content type is unsupported, or the in-memory upload queue is full. Diagnostic: ```sh tar -tf bundle.tar tar -tzf bundle.tar.gz rg -n 'max_upload_size|queue_size|max_concurrency' ``` Safe fix: send one valid tar or tar.gz source bundle archive to `/v1/pipelines//upload` with `Content-Type: application/x-tar`, `application/gzip`, or `application/x-gzip`; remove pipeline query parameters; reduce archive size or raise the configured limit; retry after queue pressure drops. Reference: [Operations](operations.md#http-upload-operation). ## Upload Idempotency Conflict Symptom: `POST /v1/pipelines//upload` returns `409`. Likely cause: the request reused an `Idempotency-Key` for the same token id and pipeline id with a different source manifest, or another request with the same key is still being staged before its manifest is known. Diagnostic: ```sh curl -i -X POST http://127.0.0.1:8080/v1/pipelines//upload \ -H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \ -H "Content-Type: application/gzip" \ -H "Idempotency-Key: " \ --data-binary @bundle.tar.gz ``` Safe fix: if the response includes `"retryable":true`, retry the same upload later with the same key. Otherwise, inspect the producer operation and use the same key only for the same source bundle. Reference: [HTTP Upload API Contract](integrations/http-upload.md). ## Upload Status Is Missing Symptom: `GET /runs/` returns `404`. Likely cause: the run id is wrong, the process restarted, or the retained status record expired after `server.http.retention`. Diagnostic: ```sh curl -i http://127.0.0.1:8080/runs/ rg -n 'retention:' ``` Safe fix: use the exact `run_id` returned by upload admission. Increase retention if operators need a longer status window. Reference: [Operations](operations.md#http-upload-operation).