Refresh operations and troubleshooting documentation
This commit is contained in:
@@ -1,8 +1,14 @@
|
||||
# Distributor Troubleshooting
|
||||
|
||||
## `load config ... no such file or directory`
|
||||
Audience: administrators and operators diagnosing `distributor` command, configuration, publishing, storage, or HTTP upload failures.
|
||||
|
||||
Likely cause: `run` could not find the config path. If `--config` is omitted, the default path is `/usr/local/etc/distributor/config.yml`.
|
||||
Each entry lists the symptom, likely cause, diagnostic step, safe fix, and relevant reference link. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), and operating procedures live in [Operations](operations.md).
|
||||
|
||||
## Config File Is Missing
|
||||
|
||||
Symptom: `load config ... no such file or directory`.
|
||||
|
||||
Likely cause: `--config` points to a missing file, or `--config` was omitted and `/usr/local/etc/distributor/config.yml` is not installed.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
@@ -10,11 +16,15 @@ Diagnostic:
|
||||
ls -l <config-path>
|
||||
```
|
||||
|
||||
Safe fix: pass an existing config path with `--config`, or install a config at the default path. See [configuration](config.md).
|
||||
Safe fix: pass an existing file with `--config`, or install a config at the default path.
|
||||
|
||||
## `parse config ... field not found`
|
||||
Reference: [Configuration](config.md#config-file-loading).
|
||||
|
||||
Likely cause: the YAML contains an unknown field. Config loading rejects unknown keys.
|
||||
## Config Contains An Unknown Field
|
||||
|
||||
Symptom: `parse config ... field not found`.
|
||||
|
||||
Likely cause: the YAML contains a key that is not part of the implemented config schema.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
@@ -22,14 +32,15 @@ Diagnostic:
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||
```
|
||||
|
||||
Safe fix: compare the file to the reference in [configuration](config.md) and remove or rename unsupported fields.
|
||||
Safe fix: remove or rename unsupported fields using the canonical config reference.
|
||||
|
||||
## `validate config ... backend ... is unsupported`
|
||||
Reference: [Configuration](config.md).
|
||||
|
||||
Likely cause: a source or destination uses an unsupported backend name, or a
|
||||
command is trying to execute a backend that is valid only for another workflow.
|
||||
`run`, `validate`, and `inspect` execute `local`, `ssh`, and `s3` sources.
|
||||
`serve` executes `http_upload` sources.
|
||||
## Backend Name Or Placement Is Invalid
|
||||
|
||||
Symptom: `backend ... is unsupported` or `http_upload is only supported for sources`.
|
||||
|
||||
Likely cause: a backend name is misspelled, not executable, or configured in the wrong role.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
@@ -37,13 +48,343 @@ Diagnostic:
|
||||
rg -n "backend:" <config-path>
|
||||
```
|
||||
|
||||
Safe fix: use `backend: local`, `backend: ssh`, or `backend: s3` for normal
|
||||
source and destination workflows. Use `backend: http_upload` only for sources
|
||||
handled by `distributor serve`.
|
||||
Safe fix: use `local`, `ssh`, or `s3` for executable sources and destinations. Use `http_upload` only as a source served by `distributor serve`.
|
||||
|
||||
## `bind HTTP server ... address already in use`
|
||||
Reference: [Configuration](config.md#backend-reference).
|
||||
|
||||
Likely cause: another process is already listening on `server.http.bind`.
|
||||
## CLI Arguments Select The Wrong Source Mode
|
||||
|
||||
Symptom: `configured source mode requires --pipeline`, `does not accept a local path with --config, --pipeline, or --bundle`, `validate command requires a path`, or `inspect command requires a path`.
|
||||
|
||||
Likely cause: `validate` or `inspect` mixed local path mode with configured source mode, or omitted the required source selector.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate --help
|
||||
go run ./cmd/distributor inspect --help
|
||||
```
|
||||
|
||||
Safe fix: use either `distributor validate <path>` / `distributor inspect <path>`, or use `--config <path> --pipeline <id>` with optional `--bundle <path>`.
|
||||
|
||||
Reference: [CLI](cli.md#validate).
|
||||
|
||||
## Output Format Is Invalid
|
||||
|
||||
Symptom: `format must be text or json`.
|
||||
|
||||
Likely cause: an unsupported value was passed to `--format`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --help
|
||||
```
|
||||
|
||||
Safe fix: use `--format text` or `--format json`.
|
||||
|
||||
Reference: [CLI](cli.md#common-output-format).
|
||||
|
||||
## JSON Mode Wrote No JSON Document
|
||||
|
||||
Symptom: `--format json` exits non-zero and stdout has no JSON result.
|
||||
|
||||
Likely cause: the command failed before it could construct a result, such as invalid arguments, missing config, unreadable secrets, or source setup failure.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --format json
|
||||
```
|
||||
|
||||
Safe fix: read stderr, fix the setup problem, then rerun. Partial destination failures during `run` can produce JSON; fatal setup failures do not.
|
||||
|
||||
Reference: [CLI](cli.md#output-and-exit-behavior).
|
||||
|
||||
## Source Pipeline Is Not Found
|
||||
|
||||
Symptom: `pipeline "<id>" not found`.
|
||||
|
||||
Likely cause: configured source diagnostics or upload processing selected a pipeline id that is absent from the loaded config.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
rg -n "id:" <config-path>
|
||||
```
|
||||
|
||||
Safe fix: pass an existing `--pipeline` value or correct the pipeline id in config.
|
||||
|
||||
Reference: [Configuration](config.md#pipelines).
|
||||
|
||||
## Source Bundles Are Not Found
|
||||
|
||||
Symptom: `no bundles found`, `no bundles found under "."`, `stat ssh ... not_found`, or `stat s3 ... not_found`.
|
||||
|
||||
Likely cause: the source root, source-root-relative bundle path, S3 prefix, SSH path, or permissions do not expose a directory containing `manifest.json`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
|
||||
```
|
||||
|
||||
Safe fix: correct the configured source backend root, permissions, prefix, or `--bundle` path. The selected bundle directory must contain `manifest.json`.
|
||||
|
||||
Reference: [Operations](operations.md#filesystem-and-storage-layout).
|
||||
|
||||
## Source Manifest Or Files Fail Validation
|
||||
|
||||
Symptom: `sha256 mismatch`, `size mismatch`, `digest mismatch`, missing manifest fields, or unsafe source paths.
|
||||
|
||||
Likely cause: files changed after `manifest.json` was written, the manifest digest is stale, or the producer wrote invalid bundle paths.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate <source-root>
|
||||
```
|
||||
|
||||
Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source validation failures.
|
||||
|
||||
Reference: [Operations](operations.md#cleanup-and-recovery).
|
||||
|
||||
## Destination Has Unmanaged Content
|
||||
|
||||
Symptom: `destination has content but no distributor state` or a plan reason containing `fail_unmanaged`.
|
||||
|
||||
Likely cause: the destination bundle path contains files but no valid `.distributor.json`, so `distributor` will not claim it by default.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
find <destination-path> -maxdepth 2 -print
|
||||
```
|
||||
|
||||
Safe fix: choose an empty destination path, move unrelated files aside, or preview `run --dry-run --force` only after confirming the reported bundle path is safe to replace.
|
||||
|
||||
Reference: [Operations](operations.md#forced-replacement-workflow).
|
||||
|
||||
## Destination State Conflicts With Source
|
||||
|
||||
Symptom: `fail_conflict`, `destination source id differs`, `same id and created time but different digest`, `pipeline id ... does not match`, or `destination id ... does not match`.
|
||||
|
||||
Likely cause: `.distributor.json` belongs to a different pipeline, destination, source id, or same-created source with different content.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
cat <destination-path>/.distributor.json
|
||||
go run ./cmd/distributor inspect <source-root>
|
||||
```
|
||||
|
||||
Safe fix: verify the source and destination are intended to match. Use a separate destination path for unrelated content. To replace the existing state, configure `transfer.on_conflict: replace`, preview with `--dry-run --force`, then publish with `--force`.
|
||||
|
||||
Reference: [Operations](operations.md#destination-state-and-retry-behavior).
|
||||
|
||||
## Destination Is Newer Than Source
|
||||
|
||||
Symptom: `skip_destination_newer` or `destination is newer and replacement requires --force`.
|
||||
|
||||
Likely cause: the destination state records a source manifest with a later `created` timestamp than the current source.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run --format json
|
||||
```
|
||||
|
||||
Safe fix: keep the default skip behavior unless replacement is intentional. To replace newer state, configure `transfer.on_destination_newer: replace`, preview with `--dry-run --force`, then publish with `--force`.
|
||||
|
||||
Reference: [Operations](operations.md#forced-replacement-workflow).
|
||||
|
||||
## Forced Replacement Appears In A Plan
|
||||
|
||||
Symptom: dry-run output includes `force_replace`.
|
||||
|
||||
Likely cause: the run used `--force`, and planning selected a supported destructive replacement.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run --force
|
||||
```
|
||||
|
||||
Safe fix: inspect the pipeline id, destination id, backend, and bundle path. Proceed only if deleting everything inside that destination bundle path is intended.
|
||||
|
||||
Reference: [Operations](operations.md#forced-replacement-workflow).
|
||||
|
||||
## Output Path Collision
|
||||
|
||||
Symptom: `destination output path collision`.
|
||||
|
||||
Likely cause: publication would write two outputs to the same destination path, such as copying `report.html` while also generating `report.html` from `report.md`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||
```
|
||||
|
||||
Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide.
|
||||
|
||||
Reference: [Configuration](config.md#publish-and-transform-policy).
|
||||
|
||||
## Run Failed After Writing Some Files
|
||||
|
||||
Symptom: a destination write failed and the command exited non-zero after partial work.
|
||||
|
||||
Likely cause: storage write failure, permission issue, network interruption, or object-store error during publish execution.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
find <destination-path> -maxdepth 2 -print
|
||||
```
|
||||
|
||||
Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. Rerun `--dry-run` before publishing again.
|
||||
|
||||
Reference: [Operations](operations.md#destination-state-and-retry-behavior).
|
||||
|
||||
## JSON Run Result Has `ok: false`
|
||||
|
||||
Symptom: `run --format json` exits non-zero with a JSON result where `ok` is `false`.
|
||||
|
||||
Likely cause: at least one destination failed after planning or execution began, while other destination results were still available.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --format json
|
||||
```
|
||||
|
||||
Safe fix: inspect the top-level `errors` array, destination actions, output errors, and summary. Fix failed destinations, then preview with `--dry-run --format json`.
|
||||
|
||||
Reference: [CLI](cli.md#output-and-exit-behavior).
|
||||
|
||||
## Secrets Directory Is Missing Or Unreadable
|
||||
|
||||
Symptom: `load secrets directory ... no such file or directory`, `permission denied`, or `secret filename ... is invalid`.
|
||||
|
||||
Likely cause: `secrets.directory` points to a missing or unreadable directory, or it contains a regular file whose name is not a valid credential variable name.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
ls -ld <secrets-directory>
|
||||
find <secrets-directory> -maxdepth 1 -type f -printf '%f\n'
|
||||
```
|
||||
|
||||
Safe fix: mount or create the directory, adjust permissions for the service user, or rename/remove invalid secret files. Secret filenames must match `[A-Za-z_][A-Za-z0-9_]*`.
|
||||
|
||||
Reference: [Configuration](config.md#secrets).
|
||||
|
||||
## Credential Variable Is Missing Or Empty
|
||||
|
||||
Symptom: `credential environment variable ... is not set`, `credential environment variable ... is empty`, or S3 authentication errors such as `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch`.
|
||||
|
||||
Likely cause: configured S3 credential variable names are not available through the process environment or `secrets.directory`, are empty, or do not authorize the requested bucket/prefix.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
|
||||
ls -l <secrets-directory>
|
||||
```
|
||||
|
||||
Safe fix: provide both configured S3 credential values, correct IAM/service permissions, or omit explicit credential fields to use the AWS SDK default credential chain.
|
||||
|
||||
Reference: [Configuration](config.md#s3-compatible-backend).
|
||||
|
||||
## Secret File Is Ignored In Favor Of Environment
|
||||
|
||||
Symptom: `secret ... ignored because the real environment already has that variable`.
|
||||
|
||||
Likely cause: the same credential name exists in the process environment and `secrets.directory` with different values.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
env | cut -d= -f1 | rg '^<variable-name>$'
|
||||
ls -l <secrets-directory>/<variable-name>
|
||||
```
|
||||
|
||||
Safe fix: remove one source of the credential or intentionally keep the process environment value. `distributor` does not print either value.
|
||||
|
||||
Reference: [Operations](operations.md#secrets-operation).
|
||||
|
||||
## SSH Auth Is Not Configured
|
||||
|
||||
Symptom: `no SSH auth methods configured`.
|
||||
|
||||
Likely cause: no SSH agent is available and `ssh_key_file` is missing or unreadable.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
test -n "$SSH_AUTH_SOCK" && ssh-add -l
|
||||
ls -l <ssh-key-file>
|
||||
```
|
||||
|
||||
Safe fix: start an SSH agent with an appropriate key loaded, or configure a readable private key with `ssh_key_file`.
|
||||
|
||||
Reference: [Configuration](config.md#sshsftp-backend).
|
||||
|
||||
## SSH Host Key Fails
|
||||
|
||||
Symptom: `host key ... is unknown`, `known_hosts is required`, or `host key ... has changed`.
|
||||
|
||||
Likely cause: strict host key checking has no trusted key, `accept-new` cannot persist a new key, or the remote host key differs from the stored key.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
ls -l <known-hosts-path>
|
||||
ssh-keygen -F <host> -f <known-hosts-path>
|
||||
```
|
||||
|
||||
Safe fix: pre-populate `known_hosts` for `strict`, configure a writable `known_hosts` path for `accept-new`, or verify the server identity before updating a changed key. Do not disable host key checking to bypass an unexpected changed key.
|
||||
|
||||
Reference: [Operations](operations.md#sshsftp).
|
||||
|
||||
## S3 Prefix Is Invalid
|
||||
|
||||
Symptom: `prefix must be a clean relative slash-separated path`.
|
||||
|
||||
Likely cause: the S3 prefix contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||
```
|
||||
|
||||
Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`.
|
||||
|
||||
Reference: [Configuration](config.md#s3-compatible-backend).
|
||||
|
||||
## S3 Location Or Connectivity Fails
|
||||
|
||||
Symptom: `NoSuchBucket`, `InvalidBucketName`, `not_found`, endpoint connection failures, or TLS/network errors.
|
||||
|
||||
Likely cause: endpoint, bucket, prefix, region, path-style mode, network routing, or credentials are wrong for the service.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||
curl -I <endpoint>
|
||||
```
|
||||
|
||||
Safe fix: verify `endpoint`, `bucket`, `region`, `prefix`, and `force_path_style`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing. Distributor does not provide an insecure TLS bypass setting.
|
||||
|
||||
Reference: [Operations](operations.md#s3-compatible-storage).
|
||||
|
||||
## HTTP Server Cannot Bind
|
||||
|
||||
Symptom: `bind HTTP server ... address already in use`.
|
||||
|
||||
Likely cause: another process is listening on `server.http.bind`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
@@ -51,44 +392,33 @@ Diagnostic:
|
||||
ss -ltnp | rg '<port>'
|
||||
```
|
||||
|
||||
Safe fix: stop the conflicting process or configure a different
|
||||
`server.http.bind` value. The default bind address is `127.0.0.1:8080`.
|
||||
Safe fix: stop the conflicting process or configure a different bind address.
|
||||
|
||||
## `upload token environment variable ... is not set`
|
||||
Reference: [Configuration](config.md#serverhttp).
|
||||
|
||||
Likely cause: a configured `http_upload` source references `token_env`, but the
|
||||
variable is absent from both the real process environment and
|
||||
`secrets.directory`.
|
||||
## HTTP Upload Token Is Missing Or Duplicated
|
||||
|
||||
Diagnostic:
|
||||
Symptom: `upload token environment variable ... is not set`, `... is empty`, or `upload token environment variables ... resolve to the same value`.
|
||||
|
||||
```sh
|
||||
env | cut -d= -f1 | rg '^<token-variable>$'
|
||||
ls -l <secrets-directory>/<token-variable>
|
||||
```
|
||||
|
||||
Safe fix: set the real environment variable or create a readable
|
||||
secrets-directory file with the same name. Do not place literal token values in
|
||||
YAML.
|
||||
|
||||
## `upload token environment variables ... resolve to the same value`
|
||||
|
||||
Likely cause: two configured `http_upload` pipelines resolve to the same bearer
|
||||
token value.
|
||||
Likely cause: an `http_upload` source references a missing/empty `token_env`, or two upload pipelines resolve to the same bearer token.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
rg -n 'token_env:' <config-path>
|
||||
env | cut -d= -f1 | rg '^<token-variable>$'
|
||||
ls -l <secrets-directory>/<token-variable>
|
||||
```
|
||||
|
||||
Safe fix: assign a distinct non-empty token value to each `http_upload`
|
||||
pipeline. Distributor does not print the duplicate token value.
|
||||
Safe fix: provide one distinct non-empty token value per upload pipeline through the process environment or `secrets.directory`. Do not put literal tokens in YAML.
|
||||
|
||||
## `POST /upload` returns `401`
|
||||
Reference: [Configuration](config.md#http-upload-source-backend).
|
||||
|
||||
Likely cause: the request is missing `Authorization: Bearer <token>` or the
|
||||
token does not match any configured `http_upload` pipeline.
|
||||
## Upload Request Is Unauthorized
|
||||
|
||||
Symptom: `POST /upload` returns `401`.
|
||||
|
||||
Likely cause: the request lacks `Authorization: Bearer <token>`, has an empty token, or uses a token that does not match any configured upload pipeline.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
@@ -99,73 +429,33 @@ curl -i -X POST http://127.0.0.1:8080/upload \
|
||||
--data-binary @bundle.tar
|
||||
```
|
||||
|
||||
Safe fix: use the token value resolved by the configured `token_env`. Do not
|
||||
include token values in logs or tickets.
|
||||
Safe fix: use the token value resolved by the configured `token_env`. Do not include token values in logs or tickets.
|
||||
|
||||
## `POST /upload` returns `400`
|
||||
Reference: [Operations](operations.md#http-upload-operation).
|
||||
|
||||
Likely cause: the archive content is malformed, the gzip body is invalid, the
|
||||
tar body cannot be extracted safely, or the extracted source bundle fails
|
||||
manifest and file validation.
|
||||
## Upload Request Is Rejected Before A Run ID
|
||||
|
||||
Symptom: `POST /upload` returns `400`, `413`, `415`, or `503`.
|
||||
|
||||
Likely cause: the request included a `pipeline` or `pipeline_id` query, archive content is malformed, the body exceeds size limits, content type is unsupported, or the in-memory upload queue is full.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
tar -tf bundle.tar
|
||||
tar -tzf bundle.tar.gz
|
||||
go run ./cmd/distributor validate <extracted-bundle-root>
|
||||
rg -n 'max_upload_size|queue_size|max_concurrency' <config-path>
|
||||
```
|
||||
|
||||
Safe fix: rebuild the tar or tar.gz archive from one complete source bundle
|
||||
root. The archive must contain exactly one root-level `manifest.json`, and every
|
||||
manifest-listed file must exist as a regular file with matching size and digest.
|
||||
Safe fix: send one valid tar or tar.gz source bundle archive with `Content-Type: application/x-tar`, `application/gzip`, or `application/x-gzip`; remove pipeline query parameters; reduce archive size or raise the configured limit; retry after queue pressure drops.
|
||||
|
||||
## `POST /upload` returns `413`
|
||||
Reference: [Operations](operations.md#http-upload-operation).
|
||||
|
||||
Likely cause: the request body exceeds the selected pipeline's
|
||||
`source.max_upload_size` or the default `server.http.max_upload_size`.
|
||||
## Upload Status Is Missing
|
||||
|
||||
Diagnostic:
|
||||
Symptom: `GET /runs/<run_id>` returns `404`.
|
||||
|
||||
```sh
|
||||
ls -lh bundle.tar bundle.tar.gz
|
||||
rg -n 'max_upload_size:' <config-path>
|
||||
```
|
||||
|
||||
Safe fix: upload a smaller archive, remove unnecessary files from the source
|
||||
bundle, or raise the configured upload size limit.
|
||||
|
||||
## `POST /upload` returns `415`
|
||||
|
||||
Likely cause: the upload uses an unsupported content type. The server accepts
|
||||
uncompressed tar and gzip-compressed tar archives only.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
file bundle.tar.gz
|
||||
```
|
||||
|
||||
Safe fix: send `Content-Type: application/x-tar`, `application/gzip`, or
|
||||
`application/x-gzip`, matching the archive format.
|
||||
|
||||
## `POST /upload` returns `503`
|
||||
|
||||
Likely cause: the in-memory upload queue is full.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
rg -n 'queue_size|max_concurrency' <config-path>
|
||||
```
|
||||
|
||||
Safe fix: retry after active uploads finish, or increase `server.http.queue_size`
|
||||
for the deployment.
|
||||
|
||||
## `GET /runs/<run_id>` returns `404`
|
||||
|
||||
Likely cause: the run id is wrong, the process restarted, or the completed
|
||||
status record expired after `server.http.retention`.
|
||||
Likely cause: the run id is wrong, the process restarted, or the retained status record expired after `server.http.retention`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
@@ -174,362 +464,6 @@ curl -i http://127.0.0.1:8080/runs/<run-id>
|
||||
rg -n 'retention:' <config-path>
|
||||
```
|
||||
|
||||
Safe fix: use the exact `run_id` returned by `POST /upload`. If status retention
|
||||
is too short for operators, increase `server.http.retention`.
|
||||
Safe fix: use the exact `run_id` returned by upload admission. Increase retention if operators need a longer status window.
|
||||
|
||||
## `--format: format must be text or json`
|
||||
|
||||
Likely cause: a command was run with an unsupported output format.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --help
|
||||
```
|
||||
|
||||
Safe fix: use `--format text` or `--format json`. Help and usage output are always text.
|
||||
|
||||
## `--format json` wrote no JSON output
|
||||
|
||||
Likely cause: the command failed before it could construct a result, such as a missing config file, invalid arguments, unreadable secrets directory, or source setup failure.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --format json
|
||||
```
|
||||
|
||||
Safe fix: read the stderr error and fix the setup problem. JSON mode writes a document only after the command has enough information to construct a result.
|
||||
|
||||
## `configured source mode requires --pipeline`
|
||||
|
||||
Likely cause: `validate` or `inspect` was run with `--config` but without an explicit pipeline id.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate --help
|
||||
go run ./cmd/distributor inspect --help
|
||||
```
|
||||
|
||||
Safe fix: add `--pipeline <pipeline-id>`. Configured source diagnostics require an explicit pipeline even when the config contains one pipeline.
|
||||
|
||||
## `does not accept a local path with --config, --pipeline, or --bundle`
|
||||
|
||||
Likely cause: local-path mode and configured source mode were mixed in one `validate` or `inspect` command.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor inspect --help
|
||||
```
|
||||
|
||||
Safe fix: use either `distributor inspect <local-path>` or `distributor inspect --config <path> --pipeline <id>`, not both.
|
||||
|
||||
## `--format json` exited non-zero with `ok: false`
|
||||
|
||||
Likely cause: `run` began planning or executing destinations, and at least one destination failed while other destination results were still available.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --format json
|
||||
```
|
||||
|
||||
Safe fix: inspect the top-level `errors` array, `result.actions`, and `result.summary`. Fix the failed destination, then preview with `--dry-run --format json` before retrying.
|
||||
|
||||
## `prefix must be a clean relative slash-separated path`
|
||||
|
||||
Likely cause: S3 `prefix` contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||
```
|
||||
|
||||
Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`.
|
||||
|
||||
## `NoSuchBucket`, `InvalidBucketName`, or `not_found`
|
||||
|
||||
Likely cause: the S3 bucket, endpoint, or prefix is wrong, or the configured credentials cannot see the requested object.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||
```
|
||||
|
||||
Safe fix: verify `endpoint`, `bucket`, `region`, `force_path_style`, and `prefix`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing.
|
||||
|
||||
## `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch`
|
||||
|
||||
Likely cause: S3 credentials are missing, wrong, empty, or lack permission for the bucket or prefix.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
|
||||
ls -l <secrets-directory>
|
||||
```
|
||||
|
||||
Safe fix: provide both configured credential environment variables through the real environment or `secrets.directory`, or omit explicit credential fields to use the AWS SDK default credential chain.
|
||||
|
||||
## S3 endpoint connection failures
|
||||
|
||||
Likely cause: the endpoint URL is unreachable, uses the wrong scheme, or does not match the configured path-style mode.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
curl -I <endpoint>
|
||||
```
|
||||
|
||||
Safe fix: correct `endpoint`, network routing, TLS settings outside distributor, or `force_path_style`. Distributor does not provide insecure TLS bypass configuration.
|
||||
|
||||
## `load secrets directory ... no such file or directory`
|
||||
|
||||
Likely cause: `secrets.directory` points to a missing directory.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
ls -ld <secrets-directory>
|
||||
```
|
||||
|
||||
Safe fix: create or mount the directory before running, or remove `secrets.directory` if no credential files are needed.
|
||||
|
||||
## `load secrets directory ... permission denied`
|
||||
|
||||
Likely cause: the service user cannot read the configured secrets directory.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
ls -ld <secrets-directory>
|
||||
namei -l <secrets-directory>
|
||||
```
|
||||
|
||||
Safe fix: adjust the directory path or deployment permissions so the service user can read the directory. Distributor does not enforce owner, group, or mode policy beyond OS read access.
|
||||
|
||||
## `secret filename ... is invalid`
|
||||
|
||||
Likely cause: a regular file in `secrets.directory` does not match `[A-Za-z_][A-Za-z0-9_]*`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
find <secrets-directory> -maxdepth 1 -type f -printf '%f\n'
|
||||
```
|
||||
|
||||
Safe fix: rename the file to a valid credential environment variable name, or remove it from the secrets directory.
|
||||
|
||||
## `credential environment variable ... is not set`
|
||||
|
||||
Likely cause: a backend credential field references an environment variable that is absent from both the real process environment and the configured secrets directory.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
env | cut -d= -f1 | rg '^<variable-name>$'
|
||||
ls -l <secrets-directory>/<variable-name>
|
||||
```
|
||||
|
||||
Safe fix: set the real environment variable or create a readable secrets-directory file with the same name.
|
||||
|
||||
## `secret ... ignored because the real environment already has that variable`
|
||||
|
||||
Likely cause: the real process environment and secrets directory both define the variable with different values.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
env | cut -d= -f1 | rg '^<variable-name>$'
|
||||
ls -l <secrets-directory>/<variable-name>
|
||||
```
|
||||
|
||||
Safe fix: remove one source of the credential or make the deployment intentionally prefer the real environment value. Distributor does not print either value.
|
||||
|
||||
## `host is required for ssh backend`
|
||||
|
||||
Likely cause: SSH config is missing the structured `host` field, or an old URL-style SSH config is still in use.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||
```
|
||||
|
||||
Safe fix: configure SSH with `host`, optional `user` and `port`, and `path`. SSH URLs are not part of the active config schema.
|
||||
|
||||
## `no SSH auth methods configured`
|
||||
|
||||
Likely cause: neither an SSH agent nor `ssh_key_file` is available.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
test -n "$SSH_AUTH_SOCK" && ssh-add -l
|
||||
ls -l <ssh-key-file>
|
||||
```
|
||||
|
||||
Safe fix: start an SSH agent with an appropriate key loaded, or configure `ssh_key_file` with a readable private key.
|
||||
|
||||
## `host key ... is unknown` or `known_hosts is required`
|
||||
|
||||
Likely cause: strict host key checking has no known host key, or `accept-new` cannot persist a new key.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
ls -l <known-hosts-path>
|
||||
ssh-keygen -F <host> -f <known-hosts-path>
|
||||
```
|
||||
|
||||
Safe fix: configure a writable `known_hosts` path for `accept-new`, pre-populate `known_hosts` for `strict`, or explicitly use `host_key_policy: off` only for insecure test environments.
|
||||
|
||||
## `host key ... has changed`
|
||||
|
||||
Likely cause: the remote server presented a different host key than the one recorded in `known_hosts`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
ssh-keygen -F <host> -f <known-hosts-path>
|
||||
```
|
||||
|
||||
Safe fix: verify the server identity out of band before updating `known_hosts`. Do not switch to `host_key_policy: off` to bypass an unexpected changed key.
|
||||
|
||||
## `pipeline "<id>" not found`
|
||||
|
||||
Likely cause: configured source validation or inspection requested a pipeline id that is not present in the config file.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
rg -n "id:" <config-path>
|
||||
```
|
||||
|
||||
Safe fix: pass an existing pipeline id with `--pipeline`, or update the config.
|
||||
|
||||
## `stat ssh ... not_found`, `stat s3 ... not_found`, or `no bundles found`
|
||||
|
||||
Likely cause: the configured source root is wrong, unreadable, or does not contain source bundles.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
|
||||
```
|
||||
|
||||
Safe fix: correct the configured source root, S3 prefix, permissions, or source bundle location. Use `--bundle <path>` only with a source-root-relative bundle directory that contains `manifest.json`.
|
||||
|
||||
## `validate command requires a path` or `inspect command requires a path`
|
||||
|
||||
Likely cause: `validate` or `inspect` was run without a local path and without configured source mode.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate --help
|
||||
go run ./cmd/distributor inspect --help
|
||||
```
|
||||
|
||||
Safe fix: pass a local source bundle directory or local tree, or pass both `--config <path>` and `--pipeline <id>`.
|
||||
|
||||
## `no bundles found under "."`
|
||||
|
||||
Likely cause: the selected source root does not contain a `manifest.json` source bundle.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
find <source-root> -name manifest.json -print
|
||||
```
|
||||
|
||||
Safe fix: point the command or config at the directory containing the source bundle, or write a valid `manifest.json` and listed files. See [CLI](cli.md).
|
||||
|
||||
## `sha256 mismatch`, `size mismatch`, or `digest mismatch`
|
||||
|
||||
Likely cause: a listed source file changed after `manifest.json` was created, or the manifest digest does not match its file list.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate <source-root>
|
||||
```
|
||||
|
||||
Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source digest failures.
|
||||
|
||||
## `destination has content but no distributor state`
|
||||
|
||||
Likely cause: the destination path is not empty and has no `.distributor.json` state file, so `distributor` will not claim it as managed.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
find <destination-path> -maxdepth 2 -print
|
||||
```
|
||||
|
||||
Safe fix: choose an empty destination path or move existing files aside after confirming they are not needed. If the destination should be claimed by distributor, preview with `run --dry-run --force` and publish with `run --force` only after confirming the reported `force_replace` action is bounded to the intended bundle path.
|
||||
|
||||
## `fail_conflict`
|
||||
|
||||
Likely cause: existing `.distributor.json` belongs to a different pipeline, a different destination, a different source id, or a same-created source with a different digest.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
cat <destination-path>/.distributor.json
|
||||
go run ./cmd/distributor inspect <source-root>
|
||||
```
|
||||
|
||||
Safe fix: verify you are publishing the intended source to the intended destination. Use a separate destination path for unrelated content. If the existing state should be replaced, configure `transfer.on_conflict: replace`, preview with `run --dry-run --force`, then publish with `run --force`.
|
||||
|
||||
## `destination is newer and replacement requires --force`
|
||||
|
||||
Likely cause: config explicitly allows newer-destination replacement, but the current run did not include `--force`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run --force
|
||||
```
|
||||
|
||||
Safe fix: prefer the default `transfer.on_destination_newer: skip` unless replacing newer destination state is intentional. To replace it, keep `transfer.on_destination_newer: replace`, confirm the dry-run output shows `force_replace`, then run with `--force`.
|
||||
|
||||
## `force_replace`
|
||||
|
||||
Likely cause: the current run used `--force` and publish planning selected a supported destructive replacement.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run --force
|
||||
```
|
||||
|
||||
Safe fix: inspect the printed pipeline id, destination id, backend, and bundle path. Proceed only if deleting all content within that destination bundle path is intended.
|
||||
|
||||
## `destination output path collision`
|
||||
|
||||
Likely cause: configured publication would write two outputs to the same destination path, such as publishing a source `report.html` while also generating `report.html` from `report.md`.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config <config-path> --dry-run
|
||||
```
|
||||
|
||||
Safe fix: adjust the source bundle contents or publish policy so source and generated outputs do not collide.
|
||||
|
||||
## A run failed after writing some files
|
||||
|
||||
Likely cause: a write failed partway through publication. Local, SSH, and S3 execution attempt to clean up outputs written during the failed attempt.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
find <destination-path> -maxdepth 2 -print
|
||||
```
|
||||
|
||||
Safe fix: use the pipeline id, destination id, backend, and bundle path printed in the run error to inspect the destination before retrying. If only unrelated unmanaged files remain, move them aside or choose a clean destination. Re-run with `--dry-run` before publishing again. See [operations](operations.md).
|
||||
Reference: [Operations](operations.md#http-upload-operation).
|
||||
|
||||
Reference in New Issue
Block a user