298 lines
9.0 KiB
Markdown
298 lines
9.0 KiB
Markdown
# Distributor Troubleshooting
|
|
|
|
## `load config ... no such file or directory`
|
|
|
|
Likely cause: `run` could not find the config path. If `--config` is omitted, the default path is `/usr/local/etc/distributor/config.yml`.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
ls -l <config-path>
|
|
```
|
|
|
|
Safe fix: pass an existing config path with `--config`, or install a config at the default path. See [configuration](config.md).
|
|
|
|
## `parse config ... field not found`
|
|
|
|
Likely cause: the YAML contains an unknown field. Config loading rejects unknown keys.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
go run ./cmd/distributor run --config <config-path> --dry-run
|
|
```
|
|
|
|
Safe fix: compare the file to the reference in [configuration](config.md) and remove or rename unsupported fields.
|
|
|
|
## `validate config ... backend ... is unsupported`
|
|
|
|
Likely cause: a source or destination uses a backend name other than `local`, `ssh`, or `s3`.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
rg -n "backend:" <config-path>
|
|
```
|
|
|
|
Safe fix: use `backend: local`, `backend: ssh`, or `backend: s3` for executable workflows.
|
|
|
|
## `prefix must be a clean relative slash-separated path`
|
|
|
|
Likely cause: S3 `prefix` contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
go run ./cmd/distributor run --config <config-path> --dry-run
|
|
```
|
|
|
|
Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`.
|
|
|
|
## `NoSuchBucket`, `InvalidBucketName`, or `not_found`
|
|
|
|
Likely cause: the S3 bucket, endpoint, or prefix is wrong, or the configured credentials cannot see the requested object.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
go run ./cmd/distributor run --config <config-path> --dry-run
|
|
```
|
|
|
|
Safe fix: verify `endpoint`, `bucket`, `region`, `force_path_style`, and `prefix`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing.
|
|
|
|
## `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch`
|
|
|
|
Likely cause: S3 credentials are missing, wrong, empty, or lack permission for the bucket or prefix.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
|
|
ls -l <secrets-directory>
|
|
```
|
|
|
|
Safe fix: provide both configured credential environment variables through the real environment or `secrets.directory`, or omit explicit credential fields to use the AWS SDK default credential chain.
|
|
|
|
## S3 endpoint connection failures
|
|
|
|
Likely cause: the endpoint URL is unreachable, uses the wrong scheme, or does not match the configured path-style mode.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
curl -I <endpoint>
|
|
```
|
|
|
|
Safe fix: correct `endpoint`, network routing, TLS settings outside distributor, or `force_path_style`. Distributor does not provide insecure TLS bypass configuration.
|
|
|
|
## `load secrets directory ... no such file or directory`
|
|
|
|
Likely cause: `secrets.directory` points to a missing directory.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
ls -ld <secrets-directory>
|
|
```
|
|
|
|
Safe fix: create or mount the directory before running, or remove `secrets.directory` if no credential files are needed.
|
|
|
|
## `load secrets directory ... permission denied`
|
|
|
|
Likely cause: the service user cannot read the configured secrets directory.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
ls -ld <secrets-directory>
|
|
namei -l <secrets-directory>
|
|
```
|
|
|
|
Safe fix: adjust the directory path or deployment permissions so the service user can read the directory. Distributor does not enforce owner, group, or mode policy beyond OS read access.
|
|
|
|
## `secret filename ... is invalid`
|
|
|
|
Likely cause: a regular file in `secrets.directory` does not match `[A-Za-z_][A-Za-z0-9_]*`.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
find <secrets-directory> -maxdepth 1 -type f -printf '%f\n'
|
|
```
|
|
|
|
Safe fix: rename the file to a valid credential environment variable name, or remove it from the secrets directory.
|
|
|
|
## `credential environment variable ... is not set`
|
|
|
|
Likely cause: a backend credential field references an environment variable that is absent from both the real process environment and the configured secrets directory.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
env | cut -d= -f1 | rg '^<variable-name>$'
|
|
ls -l <secrets-directory>/<variable-name>
|
|
```
|
|
|
|
Safe fix: set the real environment variable or create a readable secrets-directory file with the same name.
|
|
|
|
## `secret ... ignored because the real environment already has that variable`
|
|
|
|
Likely cause: the real process environment and secrets directory both define the variable with different values.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
env | cut -d= -f1 | rg '^<variable-name>$'
|
|
ls -l <secrets-directory>/<variable-name>
|
|
```
|
|
|
|
Safe fix: remove one source of the credential or make the deployment intentionally prefer the real environment value. Distributor does not print either value.
|
|
|
|
## `host is required for ssh backend`
|
|
|
|
Likely cause: SSH config is missing the structured `host` field, or an old URI-based SSH config is still in use.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
go run ./cmd/distributor run --config <config-path> --dry-run
|
|
```
|
|
|
|
Safe fix: configure SSH with `host`, optional `user` and `port`, and `path`. The `uri` field is not used for SSH execution.
|
|
|
|
## `no SSH auth methods configured`
|
|
|
|
Likely cause: neither an SSH agent nor `ssh_key_file` is available.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
test -n "$SSH_AUTH_SOCK" && ssh-add -l
|
|
ls -l <ssh-key-file>
|
|
```
|
|
|
|
Safe fix: start an SSH agent with an appropriate key loaded, or configure `ssh_key_file` with a readable private key.
|
|
|
|
## `host key ... is unknown` or `known_hosts is required`
|
|
|
|
Likely cause: strict host key checking has no known host key, or `accept-new` cannot persist a new key.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
ls -l <known-hosts-path>
|
|
ssh-keygen -F <host> -f <known-hosts-path>
|
|
```
|
|
|
|
Safe fix: configure a writable `known_hosts` path for `accept-new`, pre-populate `known_hosts` for `strict`, or explicitly use `host_key_policy: off` only for insecure test environments.
|
|
|
|
## `host key ... has changed`
|
|
|
|
Likely cause: the remote server presented a different host key than the one recorded in `known_hosts`.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
ssh-keygen -F <host> -f <known-hosts-path>
|
|
```
|
|
|
|
Safe fix: verify the server identity out of band before updating `known_hosts`. Do not switch to `host_key_policy: off` to bypass an unexpected changed key.
|
|
|
|
## `stat ssh ... not_found` or `no bundles found`
|
|
|
|
Likely cause: the configured SSH `path` is wrong, unreadable, or does not contain source bundles.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
sftp <user>@<host>
|
|
```
|
|
|
|
Safe fix: correct the remote root `path`, permissions, or source bundle location.
|
|
|
|
## `validate command requires a path` or `inspect command requires a path`
|
|
|
|
Likely cause: `validate` or `inspect` was run without a path.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
go run ./cmd/distributor validate --help
|
|
go run ./cmd/distributor inspect --help
|
|
```
|
|
|
|
Safe fix: pass a local source bundle directory or a local tree containing source bundles.
|
|
|
|
## `no bundles found under "."`
|
|
|
|
Likely cause: the selected source root does not contain a `manifest.json` source bundle.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
find <source-root> -name manifest.json -print
|
|
```
|
|
|
|
Safe fix: point the command or config at the directory containing the source bundle, or write a valid `manifest.json` and listed files. See [CLI](cli.md).
|
|
|
|
## `sha256 mismatch`, `size mismatch`, or `digest mismatch`
|
|
|
|
Likely cause: a listed source file changed after `manifest.json` was created, or the manifest digest does not match its file list.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
go run ./cmd/distributor validate <source-root>
|
|
```
|
|
|
|
Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source digest failures.
|
|
|
|
## `destination has content but no distributor state`
|
|
|
|
Likely cause: the destination path is not empty and has no `.distributor.json` state file, so `distributor` will not claim it as managed.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
find <destination-path> -maxdepth 2 -print
|
|
```
|
|
|
|
Safe fix: choose an empty destination path or move existing files aside after confirming they are not needed. There is no force overwrite option.
|
|
|
|
## `fail_conflict`
|
|
|
|
Likely cause: existing `.distributor.json` belongs to a different pipeline, a different destination, a different source id, or a same-created source with a different digest.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
cat <destination-path>/.distributor.json
|
|
go run ./cmd/distributor inspect <source-root>
|
|
```
|
|
|
|
Safe fix: verify you are publishing the intended source to the intended destination. Use a separate destination path for unrelated content.
|
|
|
|
## `destination output path collision`
|
|
|
|
Likely cause: configured publication would write two outputs to the same destination path, such as publishing a source `report.html` while also generating `report.html` from `report.md`.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
go run ./cmd/distributor run --config <config-path> --dry-run
|
|
```
|
|
|
|
Safe fix: adjust the source bundle contents or publish policy so source and generated outputs do not collide.
|
|
|
|
## A run failed after writing some files
|
|
|
|
Likely cause: a write failed partway through publication. Local execution attempts to clean up outputs written during the failed attempt.
|
|
|
|
Diagnostic:
|
|
|
|
```sh
|
|
find <destination-path> -maxdepth 2 -print
|
|
```
|
|
|
|
Safe fix: inspect the destination before retrying. If only unrelated unmanaged files remain, move them aside or choose a clean destination. Re-run with `--dry-run` before publishing again. See [operations](operations.md).
|