distributor/docs/troubleshooting.md

# Distributor Troubleshooting

Audience: administrators and operators diagnosing `distributor` command, configuration, publishing, storage, or HTTP upload failures.

Each entry lists the symptom, likely cause, diagnostic step, safe fix, and relevant reference link. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), and operating procedures live in [Operations](operations.md).

## Config File Is Missing

Symptom: `load config ... no such file or directory`.

Likely cause: `--config` points to a missing file, or `--config` was omitted and `/usr/local/etc/distributor/config.yml` is not installed.

Diagnostic:

```sh
ls -l <config-path>
```

Safe fix: pass an existing file with `--config`, or install a config at the default path.

Reference: [Configuration](config.md#config-file-loading).

## Config Contains An Unknown Field

Symptom: `parse config ... field not found`.

Likely cause: the YAML contains a key that is not part of the implemented config schema.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: remove or rename unsupported fields using the canonical config reference.

Reference: [Configuration](config.md).

## Backend Name Or Placement Is Invalid

Symptom: `backend ... is unsupported` or `http_upload is only supported for sources`.

Likely cause: a backend name is misspelled, not executable, or configured in the wrong role.

Diagnostic:

```sh
rg -n "backend:" <config-path>
```

Safe fix: use `local`, `ssh`, or `s3` for executable sources and destinations. Use `http_upload` only as a source served by `distributor serve`.

Reference: [Configuration](config.md#backend-reference).

## CLI Arguments Select The Wrong Source Mode

Symptom: `configured source mode requires --pipeline`, `does not accept a local path with --config, --pipeline, or --bundle`, `validate command requires a path`, or `inspect command requires a path`.

Likely cause: `validate` or `inspect` mixed local path mode with configured source mode, or omitted the required source selector.

Diagnostic:

```sh
go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help
```

Safe fix: use either `distributor validate <path>` / `distributor inspect <path>`, or use `--config <path> --pipeline <id>` with optional `--bundle <path>`.

Reference: [CLI](cli.md#validate).

## Reconcile-State Selector Is Missing Or Wrong

Symptom: `reconcile-state requires --config`, `requires --pipeline`, `requires --destination`, `pipeline "<id>" not found`, `destination <id> not found`, or `state owner is ... not ...`.

Likely cause: the command did not identify one configured destination root, or the selected root contains state for a different single-owner pipeline/destination.

Diagnostic:

```sh
go run ./cmd/distributor reconcile-state --help
rg -n 'pipelines:|destinations:|id:' <config-path>
cat <destination-path>/.distributor.json
```

Safe fix: pass the configured `--config`, `--pipeline`, and `--destination` values that identify the destination root containing the state file. For unrelated single-owner state, use the correct config selector or a separate destination root; `reconcile-state` will not take over mismatched state.

Reference: [CLI](cli.md#reconcile-state).

## Prune Selector Or Mode Is Missing Or Wrong

Symptom: `prune requires --config`, `requires --pipeline`, `requires --destination`, `requires exactly one of --dry-run or --apply`, `pipeline "<id>" not found`, `destination <id> not found`, or `state owner is ... not ...`.

Likely cause: the command did not identify one configured destination root, did not choose exactly one execution mode, or the selected root contains state for a different single-owner pipeline/destination.

Diagnostic:

```sh
go run ./cmd/distributor prune --help
rg -n 'retention:|prune:|pipelines:|destinations:|id:' <config-path>
cat <destination-path>/.distributor.json
```

Safe fix: pass the configured `--config`, `--pipeline`, and `--destination` values that identify the destination root containing the state file. Use `--dry-run` for read-only review or `--apply` for deletion, but not both. For unrelated single-owner state, use the correct config selector or a separate destination root; `prune` will not take over mismatched state.

Reference: [CLI](cli.md#prune).

## Output Format Is Invalid

Symptom: `format must be text or json`.

Likely cause: an unsupported value was passed to `--format`.

Diagnostic:

```sh
go run ./cmd/distributor run --help
```

Safe fix: use `--format text` or `--format json`.

Reference: [CLI](cli.md#common-output-format).

## JSON Mode Wrote No JSON Document

Symptom: `--format json` exits non-zero and stdout has no JSON result.

Likely cause: the command failed before it could construct a result, such as invalid arguments, missing config, unreadable secrets, or source setup failure.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --format json
```

Safe fix: read stderr, fix the setup problem, then rerun. Partial destination failures during `run` can produce JSON; fatal setup failures do not.

Reference: [CLI](cli.md#output-and-exit-behavior).

## Source Pipeline Is Not Found

Symptom: `pipeline "<id>" not found`.

Likely cause: configured source diagnostics or upload processing selected a pipeline id that is absent from the loaded config.

Diagnostic:

```sh
rg -n "id:" <config-path>
```

Safe fix: pass an existing `--pipeline` value or correct the pipeline id in config.

Reference: [Configuration](config.md#pipelines).

## Source Bundles Are Not Found

Symptom: `no bundles found`, `no bundles found under "."`, `stat ssh ... not_found`, or `stat s3 ... not_found`.

Likely cause: the source root, source-root-relative bundle path, S3 prefix, SSH path, or permissions do not expose a directory containing `manifest.json`.

Diagnostic:

```sh
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
```

Safe fix: correct the configured source backend root, permissions, prefix, or `--bundle` path. The selected bundle directory must contain `manifest.json`.

Reference: [Operations](operations.md#filesystem-and-storage-layout).

## Source Manifest Or Files Fail Validation

Symptom: `sha256 mismatch`, `size mismatch`, `digest mismatch`, missing manifest fields, or unsafe source paths.

Likely cause: files changed after `manifest.json` was written, the manifest digest is stale, or the producer wrote invalid bundle paths.

Diagnostic:

```sh
go run ./cmd/distributor validate <source-root>
```

Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source validation failures.

Reference: [Operations](operations.md#cleanup-and-recovery).

## Destination Has Unmanaged Content

Symptom: `destination has content but no distributor state`, `destination output path ... exists but is not managed by catalog state`, or a plan reason containing `fail_unmanaged`.

Likely cause: the destination bundle path contains files but no valid `.distributor.json`, or a planned output path collides with storage content that valid catalog state does not record. `distributor` will not claim unmanaged content by default.

Diagnostic:

```sh
find <destination-path> -maxdepth 2 -print
```

Safe fix: choose an empty destination path, move unrelated files aside, or preview `run --dry-run --force` only after confirming the reported destination bundle path is safe to replace.

Reference: [Operations](operations.md#forced-replacement-workflow).

## Destination State Conflicts With Source

Symptom: `fail_conflict`, `destination source id differs`, `same id and created time but different digest`, `pipeline id ... does not match`, or `destination id ... does not match`.

Likely cause: `.distributor.json` belongs to a different pipeline, destination, source id, shared-root output owner, or same-created source with different content. Valid managed identity, source, and shared-root output-owner mismatches can publish as `replace_takeover` when destination `takeover.mode` allows them, or as `replace_conflict` when `transfer.on_conflict: replace` allows managed conflict replacement.

Diagnostic:

```sh
cat <destination-path>/.distributor.json
go run ./cmd/distributor inspect <source-root>
```

Safe fix: verify the source and destination are intended to match. Use a separate destination path for unrelated content. Use `--force` only for exceptional replacement of unmanaged content, invalid state, unsupported future state, or other cases that `run --dry-run --force` reports as `force_replace`.

Reference: [Operations](operations.md#destination-state-and-retry-behavior).

## Destination State References Missing Managed Outputs

Symptom: `reconcile-state --dry-run` reports `status=would_change` or JSON `missing_managed_outputs` entries.

Likely cause: files that were recorded as managed in `.distributor.json` were removed outside `distributor`, or a previous external cleanup removed destination files without updating state.

Diagnostic:

```sh
go run ./cmd/distributor reconcile-state \
  --config <config-path> \
  --pipeline <pipeline-id> \
  --destination <destination-id> \
  --dry-run
```

Safe fix: if the missing files should no longer be managed, rerun the same command without `--dry-run` to remove only the missing managed output records from `.distributor.json`. The command does not delete destination files or adopt unmanaged entries. If the files should exist, restore them from backup or republish instead.

Reference: [Operations](operations.md#destination-state-repair).

## Prune Reports No Planned Deletes

Symptom: `prune --dry-run` reports `planned=0` or JSON `planned_outputs: []`.

Likely cause: pruning is disabled for the selected destination, every managed output is preserved by `keep_latest`, no managed output is older than `older_than`, or the selected shared-root owner has no eligible outputs.

Diagnostic:

```sh
rg -n 'retention:|prune:|older_than:|keep_latest:' <config-path>
go run ./cmd/distributor prune \
  --config <config-path> \
  --pipeline <pipeline-id> \
  --destination <destination-id> \
  --dry-run \
  --format json
```

Safe fix: verify the selected destination's retention policy and the managed output `updated_at` timestamps in `.distributor.json`. Adjust config if the policy is too conservative, then rerun `--dry-run` before using `--apply`.

Reference: [Operations](operations.md#managed-output-pruning).

## Destination Is Newer Than Source

Symptom: `skip_destination_newer`.

Likely cause: the destination state records a source manifest with a later `created` timestamp than the current source.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run --format json
```

Safe fix: keep the default skip behavior unless replacement is intentional. To replace newer valid managed state, configure `transfer.on_destination_newer: replace`, preview with `--dry-run`, then publish without `--force`.

Reference: [Operations](operations.md#forced-replacement-workflow).

## Forced Replacement Appears In A Plan

Symptom: dry-run output includes `force_replace`.

Likely cause: the run used `--force`, and catalog planning selected a supported destructive replacement for unmanaged destination content, a planned unmanaged path collision, invalid destination state, or unsupported future destination state.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run --force
```

Safe fix: inspect the pipeline id, destination id, backend, and bundle path. Confirm the `destination_path` in JSON output, or the fixed-path `target=.` text output, before applying. Proceed only if deleting everything inside that destination bundle path is intended; `force_replace` then writes planned outputs and schema version `4` catalog state.

Reference: [Operations](operations.md#forced-replacement-workflow).

## Output Path Collision

Symptom: `destination output path collision` or `merge output path ... exists but is not managed by destination state`.

Likely cause: publication would write two outputs to the same destination path, such as copying `report.html` while also generating `report.html` from `report.md`. For merge reconciliation, it can also mean a planned output path already exists in storage but is not recorded in `.distributor.json` as managed.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide. For merge reconciliation, move unmanaged content aside, choose another destination path, or use replacement/forced replacement only when deleting the destination bundle path is intended.

Reference: [Configuration](config.md#publish-and-transform-policy).

## Shared-Root Ownership Conflict

Symptom: `fail_conflict` with a reason like `destination output path ... is owned by <pipeline>/<destination>`.

Likely cause: a `state.mode: shared_root` destination planned an output path already recorded in `.distributor.json` for another pipeline/destination owner, and `takeover.mode` does not allow that managed path to move to the current owner.

Diagnostic:

```sh
cat <destination-path>/.distributor.json
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: change one owner so it writes a different output path, use a separate destination root, or intentionally replace the whole destination root only after previewing with `--dry-run --force`.

Reference: [Operations](operations.md#destination-state-and-retry-behavior).

## Shared-Root Unmanaged Path Collision

Symptom: `fail_unmanaged` with a reason like `destination output path ... exists but is not managed by catalog state`.

Likely cause: a `state.mode: shared_root` destination planned a new output path that already exists in storage but is not recorded as managed in `.distributor.json`.

Diagnostic:

```sh
find <destination-path> -maxdepth 2 -print
cat <destination-path>/.distributor.json
```

Safe fix: move the unmanaged file aside, change the planned output path, or use forced replacement only when deleting the configured destination root is intended.

Reference: [Operations](operations.md#forced-replacement-workflow).

## Run Failed After Writing Some Files

Symptom: a destination write failed and the command exited non-zero after partial work.

Likely cause: storage write failure, permission issue, network interruption, or object-store error during publish execution.

Diagnostic:

```sh
find <destination-path> -maxdepth 2 -print
```

Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. In same-source merge mode, previously managed retained or overwritten outputs may remain intentionally. Rerun `--dry-run` before publishing again.

Reference: [Operations](operations.md#destination-state-and-retry-behavior).

## JSON Run Result Has `ok: false`

Symptom: `run --format json` exits non-zero with a JSON result where `ok` is `false`.

Likely cause: at least one destination failed after planning or execution began, while other destination results were still available.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --format json
```

Safe fix: inspect the top-level `errors` array, destination actions, output errors, and summary. Fix failed destinations, then preview with `--dry-run --format json`.

Reference: [CLI](cli.md#output-and-exit-behavior).

## Secrets Directory Is Missing Or Unreadable

Symptom: `load secrets directory ... no such file or directory`, `permission denied`, or `secret filename ... is invalid`.

Likely cause: `secrets.directory` points to a missing or unreadable directory, or it contains a regular file whose name is not a valid credential variable name.

Diagnostic:

```sh
ls -ld <secrets-directory>
find <secrets-directory> -maxdepth 1 -type f -printf '%f\n'
```

Safe fix: mount or create the directory, adjust permissions for the service user, or rename/remove invalid secret files. Secret filenames must match `[A-Za-z_][A-Za-z0-9_]*`.

Reference: [Configuration](config.md#secrets).

## Credential Variable Is Missing Or Empty

Symptom: `credential environment variable ... is not set`, `credential environment variable ... is empty`, or S3 authentication errors such as `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch`.

Likely cause: configured S3 credential variable names are not available through the process environment or `secrets.directory`, are empty, or do not authorize the requested bucket/prefix.

Diagnostic:

```sh
env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
ls -l <secrets-directory>
```

Safe fix: provide both configured S3 credential values, correct IAM/service permissions, or omit explicit credential fields to use the AWS SDK default credential chain.

Reference: [Configuration](config.md#s3-compatible-backend).

## Secret File Is Ignored In Favor Of Environment

Symptom: `secret ... ignored because the real environment already has that variable`.

Likely cause: the same credential name exists in the process environment and `secrets.directory` with different values.

Diagnostic:

```sh
env | cut -d= -f1 | rg '^<variable-name>$'
ls -l <secrets-directory>/<variable-name>
```

Safe fix: remove one source of the credential or intentionally keep the process environment value. `distributor` does not print either value.

Reference: [Operations](operations.md#secrets-operation).

## SSH Auth Is Not Configured

Symptom: `no SSH auth methods configured`.

Likely cause: no SSH agent is available and `ssh_key_file` is missing or unreadable.

Diagnostic:

```sh
test -n "$SSH_AUTH_SOCK" && ssh-add -l
ls -l <ssh-key-file>
```

Safe fix: start an SSH agent with an appropriate key loaded, or configure a readable private key with `ssh_key_file`.

Reference: [Configuration](config.md#sshsftp-backend).

## SSH Host Key Fails

Symptom: `host key ... is unknown`, `known_hosts is required`, or `host key ... has changed`.

Likely cause: strict host key checking has no trusted key, `accept-new` cannot persist a new key, or the remote host key differs from the stored key.

Diagnostic:

```sh
ls -l <known-hosts-path>
ssh-keygen -F <host> -f <known-hosts-path>
```

Safe fix: pre-populate `known_hosts` for `strict`, configure a writable `known_hosts` path for `accept-new`, or verify the server identity before updating a changed key. Do not disable host key checking to bypass an unexpected changed key.

Reference: [Operations](operations.md#sshsftp).

## S3 Prefix Is Invalid

Symptom: `prefix must be a clean relative slash-separated path`.

Likely cause: the S3 prefix contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`.

Reference: [Configuration](config.md#s3-compatible-backend).

## S3 Location Or Connectivity Fails

Symptom: `NoSuchBucket`, `InvalidBucketName`, `not_found`, endpoint connection failures, or TLS/network errors.

Likely cause: endpoint, bucket, prefix, region, path-style mode, network routing, or credentials are wrong for the service.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run
curl -I <endpoint>
```

Safe fix: verify `endpoint`, `bucket`, `region`, `prefix`, and `force_path_style`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing. Distributor does not provide an insecure TLS bypass setting.

Reference: [Operations](operations.md#s3-compatible-storage).

## HTTP Server Cannot Bind

Symptom: `bind HTTP server ... address already in use`.

Likely cause: another process is listening on `server.http.bind`.

Diagnostic:

```sh
ss -ltnp | rg '<port>'
```

Safe fix: stop the conflicting process or configure a different bind address.

Reference: [Configuration](config.md#serverhttp).

## HTTP Upload Token Is Missing Or Duplicated

Symptom: `upload token environment variable ... is not set`, `... is empty`, or `upload token environment variables ... resolve to the same value`.

Likely cause: a top-level upload token record references a missing or empty `token_env`, or two token records resolve to the same bearer token.

Diagnostic:

```sh
rg -n 'token_env:' <config-path>
env | cut -d= -f1 | rg '^<token-variable>$'
ls -l <secrets-directory>/<token-variable>
```

Safe fix: provide one distinct non-empty token value per upload token record through the process environment or `secrets.directory`. Do not put literal tokens in YAML.

Reference: [Configuration](config.md#upload_tokens).

## Upload Request Is Unauthorized

Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `401`.

Likely cause: the request lacks `Authorization: Bearer <token>`, has an empty token, or uses a token that does not match any configured upload token record.

Diagnostic:

```sh
curl -i -X POST http://127.0.0.1:8080/v1/pipelines/<pipeline-id>/upload \
  -H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \
  -H "Content-Type: application/x-tar" \
  --data-binary @bundle.tar
```

Safe fix: use the token value resolved by the configured `token_env`. Do not include token values in logs or tickets.

Reference: [Operations](operations.md#http-upload-operation).

## Upload Request Is Forbidden

Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `403`.

Likely cause: the bearer token is valid, but its configured `allow_pipelines` list does not include the requested upload pipeline.

Diagnostic:

```sh
rg -n 'upload_tokens:|allow_pipelines:|id:' <config-path>
```

Safe fix: request the intended pipeline id, or update the token allowlist to include the configured `http_upload` pipeline that this producer may submit to.

Reference: [Configuration](config.md#upload_tokens).

## Upload Request Is Rejected Before A Run ID

Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `400`, `413`, `415`, or `503`.

Likely cause: the request path has an invalid pipeline id, included a `pipeline` or `pipeline_id` query, archive content is malformed, the body exceeds size limits, content type is unsupported, or the in-memory upload queue is full.

Diagnostic:

```sh
tar -tf bundle.tar
tar -tzf bundle.tar.gz
rg -n 'max_upload_size|queue_size|max_concurrency' <config-path>
```

Safe fix: send one valid tar or tar.gz source bundle archive to `/v1/pipelines/<pipeline-id>/upload` with `Content-Type: application/x-tar`, `application/gzip`, or `application/x-gzip`; remove pipeline query parameters; reduce archive size or raise the configured limit; retry after queue pressure drops.

Reference: [Operations](operations.md#http-upload-operation).

## Upload Idempotency Conflict

Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `409`.

Likely cause: the request reused an `Idempotency-Key` for the same token id and pipeline id with a different source manifest, or another request with the same key is still being staged before its manifest is known.

Diagnostic:

```sh
curl -i -X POST http://127.0.0.1:8080/v1/pipelines/<pipeline-id>/upload \
  -H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \
  -H "Content-Type: application/gzip" \
  -H "Idempotency-Key: <key>" \
  --data-binary @bundle.tar.gz
```

Safe fix: if the response includes `"retryable":true`, retry the same upload later with the same key. Otherwise, inspect the producer operation and use the same key only for the same source bundle.

Reference: [HTTP Upload API Contract](integrations/http-upload.md).

## Upload Status Is Missing

Symptom: `GET /runs/<run_id>` returns `404`.

Likely cause: the run id is wrong, the process restarted, or the retained status record expired after `server.http.retention`.

Diagnostic:

```sh
curl -i http://127.0.0.1:8080/runs/<run-id>
rg -n 'retention:' <config-path>
```

Safe fix: use the exact `run_id` returned by upload admission. Increase retention if operators need a longer status window.

Reference: [Operations](operations.md#http-upload-operation).