eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Synchronize catalog documentation and examples

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-19 16:50:40 +00:00
parent 9a2eaf8e5e
commit 484fda2514
16 changed files with 296 additions and 871 deletions
Download Patch File Download Diff File Expand all files Collapse all files

341
docs/troubleshooting.md
View File

@@ -24,7 +24,7 @@ Reference: [Configuration](config.md#config-file-loading).
Symptom: `parse config ... field not found`.
Likely cause: the YAML contains a key that is not part of the implemented config schema.
Likely cause: the YAML contains a key that is not part of the implemented config schema. Legacy destination policy keys `state`, `reconciliation`, `takeover`, and `transfer` are rejected.
Diagnostic:
@@ -32,9 +32,9 @@ Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: remove or rename unsupported fields using the canonical config reference.
Safe fix: remove unsupported fields using the canonical config reference. Destination behavior is configured with `workflow`, `publish`, `transform`, `path_mapping`, `links`, and `retention`.
Reference: [Configuration](config.md).
Reference: [Configuration](config.md#destination-fields).
## Backend Name Or Placement Is Invalid
@@ -71,9 +71,9 @@ Reference: [CLI](cli.md#validate).
## Reconcile-State Selector Is Missing Or Wrong
Symptom: `reconcile-state requires --config`, `requires --pipeline`, `requires --destination`, `pipeline "<id>" not found`, `destination <id> not found`, or `state owner is ... not ...`.
Symptom: `reconcile-state requires --config`, `requires --pipeline`, `requires --destination`, `pipeline "<id>" not found`, or `destination <id> not found`.
Likely cause: the command did not identify one configured destination root, or the selected root contains state for a different single-owner pipeline/destination.
Likely cause: the command did not identify one configured destination root.
Diagnostic:
@@ -83,15 +83,15 @@ rg -n 'pipelines:|destinations:|id:' <config-path>
cat <destination-path>/.distributor.json
```
Safe fix: pass the configured `--config`, `--pipeline`, and `--destination` values that identify the destination root containing the state file. For unrelated single-owner state, use the correct config selector or a separate destination root; `reconcile-state` will not take over mismatched state.
Safe fix: pass the configured `--config`, `--pipeline`, and `--destination` values that identify the destination root containing the state file.
Reference: [CLI](cli.md#reconcile-state).
## Prune Selector Or Mode Is Missing Or Wrong
Symptom: `prune requires --config`, `requires --pipeline`, `requires --destination`, `requires exactly one of --dry-run or --apply`, `pipeline "<id>" not found`, `destination <id> not found`, or `state owner is ... not ...`.
Symptom: `prune requires --config`, `requires --pipeline`, `requires --destination`, `requires exactly one of --dry-run or --apply`, `pipeline "<id>" not found`, or `destination <id> not found`.
Likely cause: the command did not identify one configured destination root, did not choose exactly one execution mode, or the selected root contains state for a different single-owner pipeline/destination.
Likely cause: the command did not identify one configured destination root or did not choose exactly one execution mode.
Diagnostic:
@@ -101,7 +101,7 @@ rg -n 'retention:|prune:|pipelines:|destinations:|id:' <config-path>
cat <destination-path>/.distributor.json
```
Safe fix: pass the configured `--config`, `--pipeline`, and `--destination` values that identify the destination root containing the state file. Use `--dry-run` for read-only review or `--apply` for deletion, but not both. For unrelated single-owner state, use the correct config selector or a separate destination root; `prune` will not take over mismatched state.
Safe fix: pass the configured `--config`, `--pipeline`, and `--destination` values that identify the destination root containing the state file. Use `--dry-run` for read-only review or `--apply` for deletion, but not both.
Reference: [CLI](cli.md#prune).
@@ -201,22 +201,39 @@ Safe fix: choose an empty destination path, move unrelated files aside, or previ
Reference: [Operations](operations.md#forced-replacement-workflow).
## Destination State Conflicts With Source
## Destination State Is Invalid Or Unsupported
Symptom: `fail_conflict`, `destination source id differs`, `same id and created time but different digest`, `pipeline id ... does not match`, or `destination id ... does not match`.
Symptom: `fail_conflict`, `parse distributor state`, `state schema_version must be 4`, or `unsupported future destination state`.
Likely cause: `.distributor.json` belongs to a different pipeline, destination, source id, shared-root output owner, or same-created source with different content. Valid managed identity, source, and shared-root output-owner mismatches can publish as `replace_takeover` when destination `takeover.mode` allows them, or as `replace_conflict` when `transfer.on_conflict: replace` allows managed conflict replacement.
Likely cause: `.distributor.json` is invalid JSON, has invalid catalog fields, or uses an unsupported future schema.
Diagnostic:
```sh
cat <destination-path>/.distributor.json
go run ./cmd/distributor inspect <source-root>
go run ./cmd/distributor run --config <config-path> --dry-run --format json
```
Safe fix: verify the source and destination are intended to match. Use a separate destination path for unrelated content. Use `--force` only for exceptional replacement of unmanaged content, invalid state, unsupported future state, or other cases that `run --dry-run --force` reports as `force_replace`.
Safe fix: restore a valid catalog state file from backup, choose a different destination path, or use `--force` only after `run --dry-run --force` reports the intended bounded `force_replace`.
Reference: [Operations](operations.md#destination-state-and-retry-behavior).
Reference: [Operations](operations.md#forced-replacement-workflow).
## Destination Uses Superseded Legacy State
Symptom: dry-run reports a normal catalog action against an older `.distributor.json`, or `reconcile-state` / `prune` reports that the destination state schema is superseded.
Likely cause: the destination contains a state file written by an older implementation. Publish planning can replace it with catalog state on successful `run`, but maintenance commands only operate on current catalog state.
Diagnostic:
```sh
cat <destination-path>/.distributor.json
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: preview the publish plan, then run publication if the destination path is correct. The successful run writes schema version `4` catalog state.
Reference: [Destination State Contract](integrations/destination-state.md).
## Destination State References Missing Managed Outputs
@@ -242,7 +259,7 @@ Reference: [Operations](operations.md#destination-state-repair).
Symptom: `prune --dry-run` reports `planned=0` or JSON `planned_outputs: []`.
Likely cause: pruning is disabled for the selected destination, every managed output is preserved by `keep_latest`, no managed output is older than `older_than`, or the selected shared-root owner has no eligible outputs.
Likely cause: pruning is disabled for the selected destination, every managed output is preserved by `keep_latest`, no managed output is older than `older_than`, or the selected pipeline/destination owner has no eligible outputs.
Diagnostic:
@@ -260,22 +277,6 @@ Safe fix: verify the selected destination's retention policy and the managed out
Reference: [Operations](operations.md#managed-output-pruning).
## Destination Is Newer Than Source
Symptom: `skip_destination_newer`.
Likely cause: the destination state records a source manifest with a later `created` timestamp than the current source.
Diagnostic:
```sh
go run ./cmd/distributor run --config <config-path> --dry-run --format json
```
Safe fix: keep the default skip behavior unless replacement is intentional. To replace newer valid managed state, configure `transfer.on_destination_newer: replace`, preview with `--dry-run`, then publish without `--force`.
Reference: [Operations](operations.md#forced-replacement-workflow).
## Forced Replacement Appears In A Plan
Symptom: dry-run output includes `force_replace`.
@@ -294,9 +295,9 @@ Reference: [Operations](operations.md#forced-replacement-workflow).
## Output Path Collision
Symptom: `destination output path collision` or `merge output path ... exists but is not managed by destination state`.
Symptom: `destination output path collision` or `destination output path ... exists but is not managed by catalog state`.
Likely cause: publication would write two outputs to the same destination path, such as copying `report.html` while also generating `report.html` from `report.md`. For merge reconciliation, it can also mean a planned output path already exists in storage but is not recorded in `.distributor.json` as managed.
Likely cause: publication would write two outputs to the same destination path, such as copying `report.html` while also generating `report.html` from `report.md`, or a planned output path already exists in storage but is not catalog-managed.
Diagnostic:
@@ -304,44 +305,10 @@ Diagnostic:
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide. For merge reconciliation, move unmanaged content aside, choose another destination path, or use replacement/forced replacement only when deleting the destination bundle path is intended.
Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide. For unmanaged storage collisions, move the unmanaged file aside, choose another destination path, or use forced replacement only when deleting the destination bundle path is intended.
Reference: [Configuration](config.md#publish-and-transform-policy).
## Shared-Root Ownership Conflict
Symptom: `fail_conflict` with a reason like `destination output path ... is owned by <pipeline>/<destination>`.
Likely cause: a `state.mode: shared_root` destination planned an output path already recorded in `.distributor.json` for another pipeline/destination owner, and `takeover.mode` does not allow that managed path to move to the current owner.
Diagnostic:
```sh
cat <destination-path>/.distributor.json
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: change one owner so it writes a different output path, use a separate destination root, or intentionally replace the whole destination root only after previewing with `--dry-run --force`.
Reference: [Operations](operations.md#destination-state-and-retry-behavior).
## Shared-Root Unmanaged Path Collision
Symptom: `fail_unmanaged` with a reason like `destination output path ... exists but is not managed by catalog state`.
Likely cause: a `state.mode: shared_root` destination planned a new output path that already exists in storage but is not recorded as managed in `.distributor.json`.
Diagnostic:
```sh
find <destination-path> -maxdepth 2 -print
cat <destination-path>/.distributor.json
```
Safe fix: move the unmanaged file aside, change the planned output path, or use forced replacement only when deleting the configured destination root is intended.
Reference: [Operations](operations.md#forced-replacement-workflow).
## Run Failed After Writing Some Files
Symptom: a destination write failed and the command exited non-zero after partial work.
@@ -354,9 +321,9 @@ Diagnostic:
find <destination-path> -maxdepth 2 -print
```
Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. In same-source merge mode, previously managed retained or overwritten outputs may remain intentionally. Rerun `--dry-run` before publishing again.
Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. Rerun `--dry-run` before publishing again.
Reference: [Operations](operations.md#destination-state-and-retry-behavior).
Reference: [Operations](operations.md#catalog-publish-behavior).
## JSON Run Result Has `ok: false`
@@ -378,240 +345,14 @@ Reference: [CLI](cli.md#output-and-exit-behavior).
Symptom: `load secrets directory ... no such file or directory`, `permission denied`, or `secret filename ... is invalid`.
Likely cause: `secrets.directory` points to a missing or unreadable directory, or it contains a regular file whose name is not a valid credential variable name.
Likely cause: `secrets.directory` points to a missing or unreadable directory, or contains a filename that cannot be used as a credential variable name.
Diagnostic:
```sh
ls -ld <secrets-directory>
find <secrets-directory> -maxdepth 1 -type f -printf '%f\n'
ls -la <secrets-directory>
```
Safe fix: mount or create the directory, adjust permissions for the service user, or rename/remove invalid secret files. Secret filenames must match `[A-Za-z_][A-Za-z0-9_]*`.
Safe fix: create the directory, fix permissions, or rename secret files to valid environment-variable-style names.
Reference: [Configuration](config.md#secrets).
## Credential Variable Is Missing Or Empty
Symptom: `credential environment variable ... is not set`, `credential environment variable ... is empty`, or S3 authentication errors such as `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch`.
Likely cause: configured S3 credential variable names are not available through the process environment or `secrets.directory`, are empty, or do not authorize the requested bucket/prefix.
Diagnostic:
```sh
env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
ls -l <secrets-directory>
```
Safe fix: provide both configured S3 credential values, correct IAM/service permissions, or omit explicit credential fields to use the AWS SDK default credential chain.
Reference: [Configuration](config.md#s3-compatible-backend).
## Secret File Is Ignored In Favor Of Environment
Symptom: `secret ... ignored because the real environment already has that variable`.
Likely cause: the same credential name exists in the process environment and `secrets.directory` with different values.
Diagnostic:
```sh
env | cut -d= -f1 | rg '^<variable-name>$'
ls -l <secrets-directory>/<variable-name>
```
Safe fix: remove one source of the credential or intentionally keep the process environment value. `distributor` does not print either value.
Reference: [Operations](operations.md#secrets-operation).
## SSH Auth Is Not Configured
Symptom: `no SSH auth methods configured`.
Likely cause: no SSH agent is available and `ssh_key_file` is missing or unreadable.
Diagnostic:
```sh
test -n "$SSH_AUTH_SOCK" && ssh-add -l
ls -l <ssh-key-file>
```
Safe fix: start an SSH agent with an appropriate key loaded, or configure a readable private key with `ssh_key_file`.
Reference: [Configuration](config.md#sshsftp-backend).
## SSH Host Key Fails
Symptom: `host key ... is unknown`, `known_hosts is required`, or `host key ... has changed`.
Likely cause: strict host key checking has no trusted key, `accept-new` cannot persist a new key, or the remote host key differs from the stored key.
Diagnostic:
```sh
ls -l <known-hosts-path>
ssh-keygen -F <host> -f <known-hosts-path>
```
Safe fix: pre-populate `known_hosts` for `strict`, configure a writable `known_hosts` path for `accept-new`, or verify the server identity before updating a changed key. Do not disable host key checking to bypass an unexpected changed key.
Reference: [Operations](operations.md#sshsftp).
## S3 Prefix Is Invalid
Symptom: `prefix must be a clean relative slash-separated path`.
Likely cause: the S3 prefix contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.
Diagnostic:
```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`.
Reference: [Configuration](config.md#s3-compatible-backend).
## S3 Location Or Connectivity Fails
Symptom: `NoSuchBucket`, `InvalidBucketName`, `not_found`, endpoint connection failures, or TLS/network errors.
Likely cause: endpoint, bucket, prefix, region, path-style mode, network routing, or credentials are wrong for the service.
Diagnostic:
```sh
go run ./cmd/distributor run --config <config-path> --dry-run
curl -I <endpoint>
```
Safe fix: verify `endpoint`, `bucket`, `region`, `prefix`, and `force_path_style`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing. Distributor does not provide an insecure TLS bypass setting.
Reference: [Operations](operations.md#s3-compatible-storage).
## HTTP Server Cannot Bind
Symptom: `bind HTTP server ... address already in use`.
Likely cause: another process is listening on `server.http.bind`.
Diagnostic:
```sh
ss -ltnp | rg '<port>'
```
Safe fix: stop the conflicting process or configure a different bind address.
Reference: [Configuration](config.md#serverhttp).
## HTTP Upload Token Is Missing Or Duplicated
Symptom: `upload token environment variable ... is not set`, `... is empty`, or `upload token environment variables ... resolve to the same value`.
Likely cause: a top-level upload token record references a missing or empty `token_env`, or two token records resolve to the same bearer token.
Diagnostic:
```sh
rg -n 'token_env:' <config-path>
env | cut -d= -f1 | rg '^<token-variable>$'
ls -l <secrets-directory>/<token-variable>
```
Safe fix: provide one distinct non-empty token value per upload token record through the process environment or `secrets.directory`. Do not put literal tokens in YAML.
Reference: [Configuration](config.md#upload_tokens).
## Upload Request Is Unauthorized
Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `401`.
Likely cause: the request lacks `Authorization: Bearer <token>`, has an empty token, or uses a token that does not match any configured upload token record.
Diagnostic:
```sh
curl -i -X POST http://127.0.0.1:8080/v1/pipelines/<pipeline-id>/upload \
-H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \
-H "Content-Type: application/x-tar" \
--data-binary @bundle.tar
```
Safe fix: use the token value resolved by the configured `token_env`. Do not include token values in logs or tickets.
Reference: [Operations](operations.md#http-upload-operation).
## Upload Request Is Forbidden
Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `403`.
Likely cause: the bearer token is valid, but its configured `allow_pipelines` list does not include the requested upload pipeline.
Diagnostic:
```sh
rg -n 'upload_tokens:|allow_pipelines:|id:' <config-path>
```
Safe fix: request the intended pipeline id, or update the token allowlist to include the configured `http_upload` pipeline that this producer may submit to.
Reference: [Configuration](config.md#upload_tokens).
## Upload Request Is Rejected Before A Run ID
Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `400`, `413`, `415`, or `503`.
Likely cause: the request path has an invalid pipeline id, included a `pipeline` or `pipeline_id` query, archive content is malformed, the body exceeds size limits, content type is unsupported, or the in-memory upload queue is full.
Diagnostic:
```sh
tar -tf bundle.tar
tar -tzf bundle.tar.gz
rg -n 'max_upload_size|queue_size|max_concurrency' <config-path>
```
Safe fix: send one valid tar or tar.gz source bundle archive to `/v1/pipelines/<pipeline-id>/upload` with `Content-Type: application/x-tar`, `application/gzip`, or `application/x-gzip`; remove pipeline query parameters; reduce archive size or raise the configured limit; retry after queue pressure drops.
Reference: [Operations](operations.md#http-upload-operation).
## Upload Idempotency Conflict
Symptom: `POST /v1/pipelines/<pipeline-id>/upload` returns `409`.
Likely cause: the request reused an `Idempotency-Key` for the same token id and pipeline id with a different source manifest, or another request with the same key is still being staged before its manifest is known.
Diagnostic:
```sh
curl -i -X POST http://127.0.0.1:8080/v1/pipelines/<pipeline-id>/upload \
-H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \
-H "Content-Type: application/gzip" \
-H "Idempotency-Key: <key>" \
--data-binary @bundle.tar.gz
```
Safe fix: if the response includes `"retryable":true`, retry the same upload later with the same key. Otherwise, inspect the producer operation and use the same key only for the same source bundle.
Reference: [HTTP Upload API Contract](integrations/http-upload.md).
## Upload Status Is Missing
Symptom: `GET /runs/<run_id>` returns `404`.
Likely cause: the run id is wrong, the process restarted, or the retained status record expired after `server.http.retention`.
Diagnostic:
```sh
curl -i http://127.0.0.1:8080/runs/<run-id>
rg -n 'retention:' <config-path>
```
Safe fix: use the exact `run_id` returned by upload admission. Increase retention if operators need a longer status window.
Reference: [Operations](operations.md#http-upload-operation).