distributor/docs/troubleshooting.md

Distributor Troubleshooting

Audience: administrators and operators diagnosing distributor command, configuration, publishing, storage, or HTTP upload failures.

Each entry lists the symptom, likely cause, diagnostic step, safe fix, and relevant reference link. Command syntax lives in CLI, configuration fields live in Configuration, and operating procedures live in Operations.

Config File Is Missing

Symptom: load config ... no such file or directory.

Likely cause: --config points to a missing file, or --config was omitted and /usr/local/etc/distributor/config.yml is not installed.

Diagnostic:

ls -l <config-path>

Safe fix: pass an existing file with --config, or install a config at the default path.

Reference: Configuration.

Config Contains An Unknown Field

Symptom: parse config ... field not found.

Likely cause: the YAML contains a key that is not part of the implemented config schema. Pre-workflow destination policy keys for state mode, conflict handling, ownership adoption, or per-comparison copy decisions are rejected.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --dry-run

Safe fix: remove unsupported fields using the canonical config reference. Destination behavior is configured with workflow, publish, transform, path_mapping, links, and retention.

Reference: Configuration.

Backend Name Or Placement Is Invalid

Symptom: backend ... is unsupported or http_upload is only supported for sources.

Likely cause: a backend name is misspelled, not executable, or configured in the wrong role.

Diagnostic:

rg -n "backend:" <config-path>

Safe fix: use local, ssh, or s3 for executable sources and destinations. Use http_upload only as a source served by distributor serve.

Reference: Configuration.

CLI Arguments Select The Wrong Source Mode

Symptom: configured source mode requires --pipeline, does not accept a local path with --config, --pipeline, or --bundle, validate command requires a path, or inspect command requires a path.

Likely cause: validate or inspect mixed local path mode with configured source mode, or omitted the required source selector.

Diagnostic:

go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help

Safe fix: use either distributor validate <path> / distributor inspect <path>, or use --config <path> --pipeline <id> with optional --bundle <path>.

Reference: CLI.

Reconcile-State Selector Is Missing Or Wrong

Symptom: reconcile-state requires --config, requires --pipeline, requires --destination, pipeline "<id>" not found, or destination <id> not found.

Likely cause: the command did not identify one configured destination root.

Diagnostic:

go run ./cmd/distributor reconcile-state --help
rg -n 'pipelines:|destinations:|id:' <config-path>
cat <destination-path>/.distributor.json

Safe fix: pass the configured --config, --pipeline, and --destination values that identify the destination root containing the state file.

Reference: CLI.

Prune Selector Or Mode Is Missing Or Wrong

Symptom: prune requires --config, requires --pipeline, requires --destination, requires exactly one of --dry-run or --apply, pipeline "<id>" not found, or destination <id> not found.

Likely cause: the command did not identify one configured destination root or did not choose exactly one execution mode.

Diagnostic:

go run ./cmd/distributor prune --help
rg -n 'retention:|prune:|pipelines:|destinations:|id:' <config-path>
cat <destination-path>/.distributor.json

Safe fix: pass the configured --config, --pipeline, and --destination values that identify the destination root containing the state file. Use --dry-run for read-only review or --apply for deletion, but not both.

Reference: CLI.

Output Format Is Invalid

Symptom: format must be text or json.

Likely cause: an unsupported value was passed to --format.

Diagnostic:

go run ./cmd/distributor run --help

Safe fix: use --format text or --format json.

Reference: CLI.

JSON Mode Wrote No JSON Document

Symptom: --format json exits non-zero and stdout has no JSON result.

Likely cause: the command failed before it could construct a result, such as invalid arguments, missing config, unreadable secrets, or source setup failure.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --format json

Safe fix: read stderr, fix the setup problem, then rerun. Partial destination failures during run can produce JSON; fatal setup failures do not.

Reference: CLI.

Source Pipeline Is Not Found

Symptom: pipeline "<id>" not found.

Likely cause: configured source diagnostics or upload processing selected a pipeline id that is absent from the loaded config.

Diagnostic:

rg -n "id:" <config-path>

Safe fix: pass an existing --pipeline value or correct the pipeline id in config.

Reference: Configuration.

Source Bundles Are Not Found

Symptom: no bundles found, no bundles found under ".", stat ssh ... not_found, or stat s3 ... not_found.

Likely cause: the source root, source-root-relative bundle path, S3 prefix, SSH path, or permissions do not expose a directory containing manifest.json.

Diagnostic:

go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>

Safe fix: correct the configured source backend root, permissions, prefix, or --bundle path. The selected bundle directory must contain manifest.json.

Reference: Operations.

Source Manifest Or Files Fail Validation

Symptom: sha256 mismatch, size mismatch, digest mismatch, missing manifest fields, or unsafe source paths.

Likely cause: files changed after manifest.json was written, the manifest digest is stale, or the producer wrote invalid bundle paths.

Diagnostic:

go run ./cmd/distributor validate <source-root>

Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source validation failures.

Reference: Operations.

Destination Has Unmanaged Content

Symptom: destination has content but no distributor state, destination output path ... exists but is not managed by catalog state, or a plan reason containing fail_unmanaged.

Likely cause: the destination bundle path contains files but no valid .distributor.json, or a planned output path collides with storage content that valid catalog state does not record. distributor will not claim unmanaged content by default.

Diagnostic:

find <destination-path> -maxdepth 2 -print

Safe fix: choose an empty destination path, move unrelated files aside, or preview run --dry-run --force only after confirming the reported destination bundle path is safe to replace.

Reference: Operations.

Destination State Is Invalid Or Unsupported

Symptom: fail_conflict, parse distributor state, state schema_version must be 4, or unsupported future destination state.

Likely cause: .distributor.json is invalid JSON, has invalid catalog fields, or uses an unsupported future schema.

Diagnostic:

cat <destination-path>/.distributor.json
go run ./cmd/distributor run --config <config-path> --dry-run --format json

Safe fix: restore a valid catalog state file from backup, choose a different destination path, or use --force only after run --dry-run --force reports the intended bounded force_replace.

Reference: Operations.

Destination Uses Superseded Legacy State

Symptom: dry-run reports a normal catalog action against an older .distributor.json, or reconcile-state / prune reports that the destination state schema is superseded.

Likely cause: the destination contains a state file written by an older implementation. Publish planning can replace it with catalog state on successful run, but maintenance commands only operate on current catalog state.

Diagnostic:

cat <destination-path>/.distributor.json
go run ./cmd/distributor run --config <config-path> --dry-run

Safe fix: preview the publish plan, then run publication if the destination path is correct. The successful run writes schema version 4 catalog state.

Reference: Destination State Contract.

Destination State References Missing Managed Outputs

Symptom: reconcile-state --dry-run reports status=would_change or JSON missing_managed_outputs entries.

Likely cause: files that were recorded as managed in .distributor.json were removed outside distributor, or a previous external cleanup removed destination files without updating state.

Diagnostic:

go run ./cmd/distributor reconcile-state \
  --config <config-path> \
  --pipeline <pipeline-id> \
  --destination <destination-id> \
  --dry-run

Safe fix: if the missing files should no longer be managed, rerun the same command without --dry-run to remove only the missing managed output records from .distributor.json. The command does not delete destination files or adopt unmanaged entries. If the files should exist, restore them from backup or republish instead.

Reference: Operations.

Prune Reports No Planned Deletes

Symptom: prune --dry-run reports planned=0 or JSON planned_outputs: [].

Likely cause: pruning is disabled for the selected destination, every managed output is preserved by keep_latest, no managed output is older than older_than, or the selected pipeline/destination owner has no eligible outputs.

Diagnostic:

rg -n 'retention:|prune:|older_than:|keep_latest:' <config-path>
go run ./cmd/distributor prune \
  --config <config-path> \
  --pipeline <pipeline-id> \
  --destination <destination-id> \
  --dry-run \
  --format json

Safe fix: verify the selected destination's retention policy and the managed output updated_at timestamps in .distributor.json. Adjust config if the policy is too conservative, then rerun --dry-run before using --apply.

Reference: Operations.

Forced Replacement Appears In A Plan

Symptom: dry-run output includes force_replace.

Likely cause: the run used --force, and catalog planning selected a supported destructive replacement for unmanaged destination content, a planned unmanaged path collision, invalid destination state, or unsupported future destination state.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --dry-run --force

Safe fix: inspect the pipeline id, destination id, backend, and bundle path. Confirm the destination_path in JSON output, or the fixed-path target=. text output, before applying. Proceed only if deleting everything inside that destination bundle path is intended; force_replace then writes planned outputs and schema version 4 catalog state.

Reference: Operations.

Output Path Collision

Symptom: destination output path collision or destination output path ... exists but is not managed by catalog state.

Likely cause: publication would write two outputs to the same destination path, such as copying report.html while also generating report.html from report.md, or a planned output path already exists in storage but is not catalog-managed.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --dry-run

Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide. For unmanaged storage collisions, move the unmanaged file aside, choose another destination path, or use forced replacement only when deleting the destination bundle path is intended.

Reference: Configuration.

Run Failed After Writing Some Files

Symptom: a destination write failed and the command exited non-zero after partial work.

Likely cause: storage write failure, permission issue, network interruption, or object-store error during publish execution.

Diagnostic:

find <destination-path> -maxdepth 2 -print

Safe fix: inspect the destination bundle path printed in the error. distributor attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. Rerun --dry-run before publishing again.

Reference: Operations.

JSON Run Result Has ok: false

Symptom: run --format json exits non-zero with a JSON result where ok is false.

Likely cause: at least one destination failed after planning or execution began, while other destination results were still available.

Diagnostic:

go run ./cmd/distributor run --config <config-path> --format json

Safe fix: inspect the top-level errors array, destination actions, output errors, and summary. Fix failed destinations, then preview with --dry-run --format json.

Reference: CLI.

Secrets Directory Is Missing Or Unreadable

Symptom: load secrets directory ... no such file or directory, permission denied, or secret filename ... is invalid.

Likely cause: secrets.directory points to a missing or unreadable directory, or contains a filename that cannot be used as a credential variable name.

Diagnostic:

ls -la <secrets-directory>

Safe fix: create the directory, fix permissions, or rename secret files to valid environment-variable-style names.

Reference: Configuration.