distributor/docs/troubleshooting.md

Distributor Troubleshooting

load config ... no such file or directory

Likely cause: run could not find the config path. If --config is omitted, the default path is /usr/local/etc/distributor/config.yml.

Diagnostic:

ls -l <config-path>

Safe fix: pass an existing config path with --config, or install a config at the default path. See configuration.

parse config ... field not found

Likely cause: the YAML contains an unknown field. Config loading rejects unknown keys.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --dry-run

Safe fix: compare the file to the reference in configuration and remove or rename unsupported fields.

validate config ... backend ... is unsupported

Likely cause: a source or destination uses a backend name other than local, ssh, or s3.

Diagnostic:

rg -n "backend:" <config-path>

Safe fix: use backend: local, backend: ssh, or backend: s3 for executable workflows.

prefix must be a clean relative slash-separated path

Likely cause: S3 prefix contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --dry-run

Safe fix: use a clean relative prefix such as reports/archive, or omit prefix.

NoSuchBucket, InvalidBucketName, or not_found

Likely cause: the S3 bucket, endpoint, or prefix is wrong, or the configured credentials cannot see the requested object.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --dry-run

Safe fix: verify endpoint, bucket, region, force_path_style, and prefix. For S3-compatible services, keep force_path_style: true unless the service requires virtual-host addressing.

AccessDenied, InvalidAccessKeyId, or SignatureDoesNotMatch

Likely cause: S3 credentials are missing, wrong, empty, or lack permission for the bucket or prefix.

Diagnostic:

env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
ls -l <secrets-directory>

Safe fix: provide both configured credential environment variables through the real environment or secrets.directory, or omit explicit credential fields to use the AWS SDK default credential chain.

S3 endpoint connection failures

Likely cause: the endpoint URL is unreachable, uses the wrong scheme, or does not match the configured path-style mode.

Diagnostic:

curl -I <endpoint>

Safe fix: correct endpoint, network routing, TLS settings outside distributor, or force_path_style. Distributor does not provide insecure TLS bypass configuration.

load secrets directory ... no such file or directory

Likely cause: secrets.directory points to a missing directory.

Diagnostic:

ls -ld <secrets-directory>

Safe fix: create or mount the directory before running, or remove secrets.directory if no credential files are needed.

load secrets directory ... permission denied

Likely cause: the service user cannot read the configured secrets directory.

Diagnostic:

ls -ld <secrets-directory>
namei -l <secrets-directory>

Safe fix: adjust the directory path or deployment permissions so the service user can read the directory. Distributor does not enforce owner, group, or mode policy beyond OS read access.

secret filename ... is invalid

Likely cause: a regular file in secrets.directory does not match [A-Za-z_][A-Za-z0-9_]*.

Diagnostic:

find <secrets-directory> -maxdepth 1 -type f -printf '%f\n'

Safe fix: rename the file to a valid credential environment variable name, or remove it from the secrets directory.

credential environment variable ... is not set

Likely cause: a backend credential field references an environment variable that is absent from both the real process environment and the configured secrets directory.

Diagnostic:

env | cut -d= -f1 | rg '^<variable-name>$'
ls -l <secrets-directory>/<variable-name>

Safe fix: set the real environment variable or create a readable secrets-directory file with the same name.

secret ... ignored because the real environment already has that variable

Likely cause: the real process environment and secrets directory both define the variable with different values.

Diagnostic:

env | cut -d= -f1 | rg '^<variable-name>$'
ls -l <secrets-directory>/<variable-name>

Safe fix: remove one source of the credential or make the deployment intentionally prefer the real environment value. Distributor does not print either value.

host is required for ssh backend

Likely cause: SSH config is missing the structured host field, or an old URL-style SSH config is still in use.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --dry-run

Safe fix: configure SSH with host, optional user and port, and path. SSH URLs are not part of the active config schema.

no SSH auth methods configured

Likely cause: neither an SSH agent nor ssh_key_file is available.

Diagnostic:

test -n "$SSH_AUTH_SOCK" && ssh-add -l
ls -l <ssh-key-file>

Safe fix: start an SSH agent with an appropriate key loaded, or configure ssh_key_file with a readable private key.

host key ... is unknown or known_hosts is required

Likely cause: strict host key checking has no known host key, or accept-new cannot persist a new key.

Diagnostic:

ls -l <known-hosts-path>
ssh-keygen -F <host> -f <known-hosts-path>

Safe fix: configure a writable known_hosts path for accept-new, pre-populate known_hosts for strict, or explicitly use host_key_policy: off only for insecure test environments.

host key ... has changed

Likely cause: the remote server presented a different host key than the one recorded in known_hosts.

Diagnostic:

ssh-keygen -F <host> -f <known-hosts-path>

Safe fix: verify the server identity out of band before updating known_hosts. Do not switch to host_key_policy: off to bypass an unexpected changed key.

stat ssh ... not_found or no bundles found

Likely cause: the configured SSH path is wrong, unreadable, or does not contain source bundles.

Diagnostic:

sftp <user>@<host>

Safe fix: correct the remote root path, permissions, or source bundle location.

validate command requires a path or inspect command requires a path

Likely cause: validate or inspect was run without a path.

Diagnostic:

go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help

Safe fix: pass a local source bundle directory or a local tree containing source bundles.

no bundles found under "."

Likely cause: the selected source root does not contain a manifest.json source bundle.

Diagnostic:

find <source-root> -name manifest.json -print

Safe fix: point the command or config at the directory containing the source bundle, or write a valid manifest.json and listed files. See CLI.

sha256 mismatch, size mismatch, or digest mismatch

Likely cause: a listed source file changed after manifest.json was created, or the manifest digest does not match its file list.

Diagnostic:

go run ./cmd/distributor validate <source-root>

Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source digest failures.

destination has content but no distributor state

Likely cause: the destination path is not empty and has no .distributor.json state file, so distributor will not claim it as managed.

Diagnostic:

find <destination-path> -maxdepth 2 -print

Safe fix: choose an empty destination path or move existing files aside after confirming they are not needed. If the destination should be claimed by distributor, preview with run --dry-run --force and publish with run --force only after confirming the reported force_replace action is bounded to the intended bundle path.

fail_conflict

Likely cause: existing .distributor.json belongs to a different pipeline, a different destination, a different source id, or a same-created source with a different digest.

Diagnostic:

cat <destination-path>/.distributor.json
go run ./cmd/distributor inspect <source-root>

Safe fix: verify you are publishing the intended source to the intended destination. Use a separate destination path for unrelated content. If the existing state should be replaced, configure transfer.on_conflict: replace, preview with run --dry-run --force, then publish with run --force.

destination is newer and replacement requires --force

Likely cause: config explicitly allows newer-destination replacement, but the current run did not include --force.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --dry-run --force

Safe fix: prefer the default transfer.on_destination_newer: skip unless replacing newer destination state is intentional. To replace it, keep transfer.on_destination_newer: replace, confirm the dry-run output shows force_replace, then run with --force.

force_replace

Likely cause: the current run used --force and publish planning selected a supported destructive replacement.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --dry-run --force

Safe fix: inspect the printed pipeline id, destination id, backend, and bundle path. Proceed only if deleting all content within that destination bundle path is intended.

destination output path collision

Likely cause: configured publication would write two outputs to the same destination path, such as publishing a source report.html while also generating report.html from report.md.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --dry-run

Safe fix: adjust the source bundle contents or publish policy so source and generated outputs do not collide.

A run failed after writing some files

Likely cause: a write failed partway through publication. Local, SSH, and S3 execution attempt to clean up outputs written during the failed attempt.

Diagnostic:

find <destination-path> -maxdepth 2 -print

Safe fix: use the pipeline id, destination id, backend, and bundle path printed in the run error to inspect the destination before retrying. If only unrelated unmanaged files remain, move them aside or choose a clean destination. Re-run with --dry-run before publishing again. See operations.