10 KiB
Distributor Troubleshooting
load config ... no such file or directory
Likely cause: run could not find the config path. If --config is omitted, the default path is /usr/local/etc/distributor/config.yml.
Diagnostic:
ls -l <config-path>
Safe fix: pass an existing config path with --config, or install a config at the default path. See configuration.
parse config ... field not found
Likely cause: the YAML contains an unknown field. Config loading rejects unknown keys.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
Safe fix: compare the file to the reference in configuration and remove or rename unsupported fields.
validate config ... backend ... is unsupported
Likely cause: a source or destination uses a backend name other than local, ssh, or s3.
Diagnostic:
rg -n "backend:" <config-path>
Safe fix: use backend: local, backend: ssh, or backend: s3 for executable workflows.
prefix must be a clean relative slash-separated path
Likely cause: S3 prefix contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
Safe fix: use a clean relative prefix such as reports/archive, or omit prefix.
NoSuchBucket, InvalidBucketName, or not_found
Likely cause: the S3 bucket, endpoint, or prefix is wrong, or the configured credentials cannot see the requested object.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
Safe fix: verify endpoint, bucket, region, force_path_style, and prefix. For S3-compatible services, keep force_path_style: true unless the service requires virtual-host addressing.
AccessDenied, InvalidAccessKeyId, or SignatureDoesNotMatch
Likely cause: S3 credentials are missing, wrong, empty, or lack permission for the bucket or prefix.
Diagnostic:
env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
ls -l <secrets-directory>
Safe fix: provide both configured credential environment variables through the real environment or secrets.directory, or omit explicit credential fields to use the AWS SDK default credential chain.
S3 endpoint connection failures
Likely cause: the endpoint URL is unreachable, uses the wrong scheme, or does not match the configured path-style mode.
Diagnostic:
curl -I <endpoint>
Safe fix: correct endpoint, network routing, TLS settings outside distributor, or force_path_style. Distributor does not provide insecure TLS bypass configuration.
load secrets directory ... no such file or directory
Likely cause: secrets.directory points to a missing directory.
Diagnostic:
ls -ld <secrets-directory>
Safe fix: create or mount the directory before running, or remove secrets.directory if no credential files are needed.
load secrets directory ... permission denied
Likely cause: the service user cannot read the configured secrets directory.
Diagnostic:
ls -ld <secrets-directory>
namei -l <secrets-directory>
Safe fix: adjust the directory path or deployment permissions so the service user can read the directory. Distributor does not enforce owner, group, or mode policy beyond OS read access.
secret filename ... is invalid
Likely cause: a regular file in secrets.directory does not match [A-Za-z_][A-Za-z0-9_]*.
Diagnostic:
find <secrets-directory> -maxdepth 1 -type f -printf '%f\n'
Safe fix: rename the file to a valid credential environment variable name, or remove it from the secrets directory.
credential environment variable ... is not set
Likely cause: a backend credential field references an environment variable that is absent from both the real process environment and the configured secrets directory.
Diagnostic:
env | cut -d= -f1 | rg '^<variable-name>$'
ls -l <secrets-directory>/<variable-name>
Safe fix: set the real environment variable or create a readable secrets-directory file with the same name.
secret ... ignored because the real environment already has that variable
Likely cause: the real process environment and secrets directory both define the variable with different values.
Diagnostic:
env | cut -d= -f1 | rg '^<variable-name>$'
ls -l <secrets-directory>/<variable-name>
Safe fix: remove one source of the credential or make the deployment intentionally prefer the real environment value. Distributor does not print either value.
host is required for ssh backend
Likely cause: SSH config is missing the structured host field, or an old URI-based SSH config is still in use.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
Safe fix: configure SSH with host, optional user and port, and path. The uri field is not used for SSH execution.
no SSH auth methods configured
Likely cause: neither an SSH agent nor ssh_key_file is available.
Diagnostic:
test -n "$SSH_AUTH_SOCK" && ssh-add -l
ls -l <ssh-key-file>
Safe fix: start an SSH agent with an appropriate key loaded, or configure ssh_key_file with a readable private key.
host key ... is unknown or known_hosts is required
Likely cause: strict host key checking has no known host key, or accept-new cannot persist a new key.
Diagnostic:
ls -l <known-hosts-path>
ssh-keygen -F <host> -f <known-hosts-path>
Safe fix: configure a writable known_hosts path for accept-new, pre-populate known_hosts for strict, or explicitly use host_key_policy: off only for insecure test environments.
host key ... has changed
Likely cause: the remote server presented a different host key than the one recorded in known_hosts.
Diagnostic:
ssh-keygen -F <host> -f <known-hosts-path>
Safe fix: verify the server identity out of band before updating known_hosts. Do not switch to host_key_policy: off to bypass an unexpected changed key.
stat ssh ... not_found or no bundles found
Likely cause: the configured SSH path is wrong, unreadable, or does not contain source bundles.
Diagnostic:
sftp <user>@<host>
Safe fix: correct the remote root path, permissions, or source bundle location.
validate command requires a path or inspect command requires a path
Likely cause: validate or inspect was run without a path.
Diagnostic:
go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help
Safe fix: pass a local source bundle directory or a local tree containing source bundles.
no bundles found under "."
Likely cause: the selected source root does not contain a manifest.json source bundle.
Diagnostic:
find <source-root> -name manifest.json -print
Safe fix: point the command or config at the directory containing the source bundle, or write a valid manifest.json and listed files. See CLI.
sha256 mismatch, size mismatch, or digest mismatch
Likely cause: a listed source file changed after manifest.json was created, or the manifest digest does not match its file list.
Diagnostic:
go run ./cmd/distributor validate <source-root>
Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source digest failures.
destination has content but no distributor state
Likely cause: the destination path is not empty and has no .distributor.json state file, so distributor will not claim it as managed.
Diagnostic:
find <destination-path> -maxdepth 2 -print
Safe fix: choose an empty destination path or move existing files aside after confirming they are not needed. If the destination should be claimed by distributor, preview with run --dry-run --force and publish with run --force only after confirming the reported force_replace action is bounded to the intended bundle path.
fail_conflict
Likely cause: existing .distributor.json belongs to a different pipeline, a different destination, a different source id, or a same-created source with a different digest.
Diagnostic:
cat <destination-path>/.distributor.json
go run ./cmd/distributor inspect <source-root>
Safe fix: verify you are publishing the intended source to the intended destination. Use a separate destination path for unrelated content. If the existing state should be replaced, configure transfer.on_conflict: replace, preview with run --dry-run --force, then publish with run --force.
destination is newer and replacement requires --force
Likely cause: config explicitly allows newer-destination replacement, but the current run did not include --force.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run --force
Safe fix: prefer the default transfer.on_destination_newer: skip unless replacing newer destination state is intentional. To replace it, keep transfer.on_destination_newer: replace, confirm the dry-run output shows force_replace, then run with --force.
force_replace
Likely cause: the current run used --force and publish planning selected a supported destructive replacement.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run --force
Safe fix: inspect the printed pipeline id, destination id, backend, and bundle path. Proceed only if deleting all content within that destination bundle path is intended.
destination output path collision
Likely cause: configured publication would write two outputs to the same destination path, such as publishing a source report.html while also generating report.html from report.md.
Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
Safe fix: adjust the source bundle contents or publish policy so source and generated outputs do not collide.
A run failed after writing some files
Likely cause: a write failed partway through publication. Local, SSH, and S3 execution attempt to clean up outputs written during the failed attempt.
Diagnostic:
find <destination-path> -maxdepth 2 -print
Safe fix: use the pipeline id, destination id, backend, and bundle path printed in the run error to inspect the destination before retrying. If only unrelated unmanaged files remain, move them aside or choose a clean destination. Re-run with --dry-run before publishing again. See operations.