# Distributor Troubleshooting ## `load config ... no such file or directory` Likely cause: `run` could not find the config path. If `--config` is omitted, the default path is `/usr/local/etc/distributor/config.yml`. Diagnostic: ```sh ls -l ``` Safe fix: pass an existing config path with `--config`, or install a config at the default path. See [configuration](config.md). ## `parse config ... field not found` Likely cause: the YAML contains an unknown field. Config loading rejects unknown keys. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run ``` Safe fix: compare the file to the reference in [configuration](config.md) and remove or rename unsupported fields. ## `validate config ... backend ... is unsupported` Likely cause: a source or destination uses a backend name other than `local`, `ssh`, or `s3`. Diagnostic: ```sh rg -n "backend:" ``` Safe fix: use `backend: local`, `backend: ssh`, or `backend: s3` for executable workflows. ## `prefix must be a clean relative slash-separated path` Likely cause: S3 `prefix` contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run ``` Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`. ## `NoSuchBucket`, `InvalidBucketName`, or `not_found` Likely cause: the S3 bucket, endpoint, or prefix is wrong, or the configured credentials cannot see the requested object. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run ``` Safe fix: verify `endpoint`, `bucket`, `region`, `force_path_style`, and `prefix`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing. ## `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch` Likely cause: S3 credentials are missing, wrong, empty, or lack permission for the bucket or prefix. Diagnostic: ```sh env | cut -d= -f1 | rg '^(|)$' ls -l ``` Safe fix: provide both configured credential environment variables through the real environment or `secrets.directory`, or omit explicit credential fields to use the AWS SDK default credential chain. ## S3 endpoint connection failures Likely cause: the endpoint URL is unreachable, uses the wrong scheme, or does not match the configured path-style mode. Diagnostic: ```sh curl -I ``` Safe fix: correct `endpoint`, network routing, TLS settings outside distributor, or `force_path_style`. Distributor does not provide insecure TLS bypass configuration. ## `load secrets directory ... no such file or directory` Likely cause: `secrets.directory` points to a missing directory. Diagnostic: ```sh ls -ld ``` Safe fix: create or mount the directory before running, or remove `secrets.directory` if no credential files are needed. ## `load secrets directory ... permission denied` Likely cause: the service user cannot read the configured secrets directory. Diagnostic: ```sh ls -ld namei -l ``` Safe fix: adjust the directory path or deployment permissions so the service user can read the directory. Distributor does not enforce owner, group, or mode policy beyond OS read access. ## `secret filename ... is invalid` Likely cause: a regular file in `secrets.directory` does not match `[A-Za-z_][A-Za-z0-9_]*`. Diagnostic: ```sh find -maxdepth 1 -type f -printf '%f\n' ``` Safe fix: rename the file to a valid credential environment variable name, or remove it from the secrets directory. ## `credential environment variable ... is not set` Likely cause: a backend credential field references an environment variable that is absent from both the real process environment and the configured secrets directory. Diagnostic: ```sh env | cut -d= -f1 | rg '^$' ls -l / ``` Safe fix: set the real environment variable or create a readable secrets-directory file with the same name. ## `secret ... ignored because the real environment already has that variable` Likely cause: the real process environment and secrets directory both define the variable with different values. Diagnostic: ```sh env | cut -d= -f1 | rg '^$' ls -l / ``` Safe fix: remove one source of the credential or make the deployment intentionally prefer the real environment value. Distributor does not print either value. ## `host is required for ssh backend` Likely cause: SSH config is missing the structured `host` field, or an old URL-style SSH config is still in use. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run ``` Safe fix: configure SSH with `host`, optional `user` and `port`, and `path`. SSH URLs are not part of the active config schema. ## `no SSH auth methods configured` Likely cause: neither an SSH agent nor `ssh_key_file` is available. Diagnostic: ```sh test -n "$SSH_AUTH_SOCK" && ssh-add -l ls -l ``` Safe fix: start an SSH agent with an appropriate key loaded, or configure `ssh_key_file` with a readable private key. ## `host key ... is unknown` or `known_hosts is required` Likely cause: strict host key checking has no known host key, or `accept-new` cannot persist a new key. Diagnostic: ```sh ls -l ssh-keygen -F -f ``` Safe fix: configure a writable `known_hosts` path for `accept-new`, pre-populate `known_hosts` for `strict`, or explicitly use `host_key_policy: off` only for insecure test environments. ## `host key ... has changed` Likely cause: the remote server presented a different host key than the one recorded in `known_hosts`. Diagnostic: ```sh ssh-keygen -F -f ``` Safe fix: verify the server identity out of band before updating `known_hosts`. Do not switch to `host_key_policy: off` to bypass an unexpected changed key. ## `stat ssh ... not_found` or `no bundles found` Likely cause: the configured SSH `path` is wrong, unreadable, or does not contain source bundles. Diagnostic: ```sh sftp @ ``` Safe fix: correct the remote root `path`, permissions, or source bundle location. ## `validate command requires a path` or `inspect command requires a path` Likely cause: `validate` or `inspect` was run without a path. Diagnostic: ```sh go run ./cmd/distributor validate --help go run ./cmd/distributor inspect --help ``` Safe fix: pass a local source bundle directory or a local tree containing source bundles. ## `no bundles found under "."` Likely cause: the selected source root does not contain a `manifest.json` source bundle. Diagnostic: ```sh find -name manifest.json -print ``` Safe fix: point the command or config at the directory containing the source bundle, or write a valid `manifest.json` and listed files. See [CLI](cli.md). ## `sha256 mismatch`, `size mismatch`, or `digest mismatch` Likely cause: a listed source file changed after `manifest.json` was created, or the manifest digest does not match its file list. Diagnostic: ```sh go run ./cmd/distributor validate ``` Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source digest failures. ## `destination has content but no distributor state` Likely cause: the destination path is not empty and has no `.distributor.json` state file, so `distributor` will not claim it as managed. Diagnostic: ```sh find -maxdepth 2 -print ``` Safe fix: choose an empty destination path or move existing files aside after confirming they are not needed. If the destination should be claimed by distributor, preview with `run --dry-run --force` and publish with `run --force` only after confirming the reported `force_replace` action is bounded to the intended bundle path. ## `fail_conflict` Likely cause: existing `.distributor.json` belongs to a different pipeline, a different destination, a different source id, or a same-created source with a different digest. Diagnostic: ```sh cat /.distributor.json go run ./cmd/distributor inspect ``` Safe fix: verify you are publishing the intended source to the intended destination. Use a separate destination path for unrelated content. If the existing state should be replaced, configure `transfer.on_conflict: replace`, preview with `run --dry-run --force`, then publish with `run --force`. ## `destination is newer and replacement requires --force` Likely cause: config explicitly allows newer-destination replacement, but the current run did not include `--force`. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run --force ``` Safe fix: prefer the default `transfer.on_destination_newer: skip` unless replacing newer destination state is intentional. To replace it, keep `transfer.on_destination_newer: replace`, confirm the dry-run output shows `force_replace`, then run with `--force`. ## `force_replace` Likely cause: the current run used `--force` and publish planning selected a supported destructive replacement. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run --force ``` Safe fix: inspect the printed pipeline id, destination id, backend, and bundle path. Proceed only if deleting all content within that destination bundle path is intended. ## `destination output path collision` Likely cause: configured publication would write two outputs to the same destination path, such as publishing a source `report.html` while also generating `report.html` from `report.md`. Diagnostic: ```sh go run ./cmd/distributor run --config --dry-run ``` Safe fix: adjust the source bundle contents or publish policy so source and generated outputs do not collide. ## A run failed after writing some files Likely cause: a write failed partway through publication. Local, SSH, and S3 execution attempt to clean up outputs written during the failed attempt. Diagnostic: ```sh find -maxdepth 2 -print ``` Safe fix: use the pipeline id, destination id, backend, and bundle path printed in the run error to inspect the destination before retrying. If only unrelated unmanaged files remain, move them aside or choose a clean destination. Re-run with `--dry-run` before publishing again. See [operations](operations.md).