From ecc5254e6ba0d7ea516ba0224d41a0e9bc41b5e2 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 4 Jun 2026 12:10:50 +0000 Subject: [PATCH] Refresh operations and troubleshooting documentation --- docs/operations.md | 455 ++++++---------------- docs/troubleshooting.md | 838 ++++++++++++++++++---------------------- 2 files changed, 504 insertions(+), 789 deletions(-) diff --git a/docs/operations.md b/docs/operations.md index c2e6324..991ccd6 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,107 +1,120 @@ # Distributor Operations +Audience: administrators and operators who run `distributor`, publish bundles, operate the HTTP upload service, or recover from failed runs. + +This document covers operating workflows, storage layout, safety behavior, and recovery. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), and symptom-specific fixes live in [Troubleshooting](troubleshooting.md). + ## Normal Workflow -Validate a source bundle: +Validate a producer bundle before publishing: ```sh go run ./cmd/distributor validate examples/source-bundle ``` -Preview a local publication: +Preview a configured run before writing destination content: ```sh go run ./cmd/distributor run --config examples/local-publish.yml --dry-run ``` -Run the local publication: +Publish after reviewing the preview: ```sh go run ./cmd/distributor run --config examples/local-publish.yml ``` -Run the local HTML publication: - -```sh -go run ./cmd/distributor run --config examples/local-html.yml -``` - -Run the local `index.html` publication: - -```sh -go run ./cmd/distributor run --config examples/local-index.yml -``` - -Preview local fan-out publication: - -```sh -go run ./cmd/distributor run --config examples/fan-out.yml --dry-run -``` - -Preview local archive-plus-latest publication: - -```sh -go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run -``` - -Preview a run for automation: +Use JSON output for automation: ```sh go run ./cmd/distributor run --config examples/fan-out.yml --dry-run --format json ``` -Preview an environment-gated SSH destination config after editing it for an SSH/SFTP endpoint you control: - -```sh -go run ./cmd/distributor run --config examples/ssh-destination.yml --dry-run -``` - -Preview an environment-gated S3 destination config after editing it for an S3-compatible endpoint and bucket you control: - -```sh -go run ./cmd/distributor run --config examples/s3-destination.yml --dry-run -``` - -Validate one configured source without opening destinations: +Use configured source diagnostics when the source is defined in YAML and may be local, SSH/SFTP, or S3-compatible storage: ```sh go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle +go run ./cmd/distributor inspect --config examples/local-publish.yml --pipeline example-source-bundle --format json ``` -## HTTP Upload Workflow +Remote examples under `examples/ssh-destination.yml` and `examples/s3-destination.yml` are load-tested templates. Edit their endpoint, path, key, bucket, prefix, and credential values for storage you control before running them. -`distributor serve` runs the HTTP upload API for pipelines whose source backend -is `http_upload`. Each upload token maps to one configured pipeline. A valid -archive is staged and validated before a run id is returned, then published -through the same destination fan-out path used by local source runs. +## Filesystem And Storage Layout -Minimal local HTTP upload configuration: +A source bundle is a directory containing `manifest.json` and every file listed in that manifest. Source discovery walks beneath the configured source backend root and finds bundle directories. -```yaml -server: - http: - bind: 127.0.0.1:8080 - staging_root: /var/spool/distributor - max_upload_size: 20MB - queue_size: 16 - max_concurrency: 1 - retention: 24h -secrets: - directory: /run/secrets/distributor -pipelines: - - id: reports - source: - backend: http_upload - token_env: DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN - destinations: - - id: archive - backend: local - path: /srv/reports/archive +Each destination has its own backend root: + +- Local destinations use the configured local `path`. +- SSH/SFTP destinations use the configured remote `path`. +- S3-compatible destinations use the configured `bucket` plus optional `prefix`. + +Destination path mapping controls where each source bundle is published beneath the destination root: + +- `preserve_relative` publishes each source bundle at the same source-root-relative path. +- `fixed` publishes one selected source bundle at the destination root. + +Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins. + +Published destination bundle paths contain `.distributor.json`. This file is both the managed sentinel and the destination state record. It records the pipeline id, destination id, publication time, source manifest, copied outputs, generated outputs, and optional public URL metadata. + +`manifest.json` from the source bundle is not copied as destination state. + +## Destination State And Retry Behavior + +`distributor` compares the source manifest to destination `.distributor.json` before writing: + +- No destination state and no destination content: publish new outputs. +- Matching destination state: skip as already published. +- Older destination state for the same source id: replace if transfer policy allows it. +- Newer destination state: skip by default. +- Invalid destination state, identity mismatch, different source id, or same-created digest mismatch: fail by default. +- Content without `.distributor.json`: fail as unmanaged content by default. + +Normal replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, then verifies the destination bundle path is empty before writing new outputs and state. + +If a write fails after some outputs were written, `distributor` attempts to delete outputs from that failed attempt so a retry does not treat partial outputs as unmanaged content. Operators should still inspect the destination after a failed write before retrying. + +Fan-out destinations are independent. If one destination fails after planning or execution begins, later destinations are still attempted. The command exits non-zero if any destination failed. + +## Dry Runs And Output Review + +`run --dry-run` loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, `.distributor.json`, or SSH `known_hosts` entries. + +Review these action labels before publishing: + +- `publish_new`: destination is empty and unmanaged. +- `replace_older`: destination state is older than the source. +- `skip_same`: destination state already matches the source. +- `skip_destination_newer`: destination state is newer than the source and is skipped. +- `force_replace`: destructive replacement selected because `--force` is present and policy permits it. +- `error`: planning or execution failed for that destination. + +Fixed destinations add fixed-path warnings during dry runs, including the selected source bundle and replacement warnings when the destination root would be replaced. + +JSON output includes warnings, pipeline summaries, destination action records, output records, URLs when configured, final counters, and partial failure details. Fatal setup failures such as unreadable config or invalid secrets do not produce a JSON result document. + +## Forced Replacement Workflow + +Use `--force` only after a dry run shows the intended bounded `force_replace` action: + +```sh +go run ./cmd/distributor run --config --dry-run --force +go run ./cmd/distributor run --config --force ``` -Create `/run/secrets/distributor/DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN` or set the -real process environment variable before starting the server. Distributor does -not read literal upload tokens from YAML. +Forced replacement can claim unmanaged non-empty destination paths. State conflicts require both `--force` and transfer policy that permits replacement: + +- newer destination state requires `transfer.on_destination_newer: replace`; +- conflict outcomes require `transfer.on_conflict: replace`. + +Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete parent paths, sibling paths, or storage outside the destination bundle path. For fixed destinations, the destination bundle path is the backend root, so a forced replacement can clear that configured root. + +`--force` applies only to the current invocation. There is no config field that enables forced replacement by default. + +## HTTP Upload Operation + +`distributor serve` runs the HTTP upload API for pipelines whose source backend is `http_upload`. Each bearer token maps to exactly one configured upload pipeline. Token values come from the process environment or `secrets.directory`, not from YAML literal values. Start the maintained local example: @@ -110,7 +123,13 @@ DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN= \ go run ./cmd/distributor serve --config examples/http-upload-local.yml ``` -Submit a tar or tar.gz source bundle: +Readiness: + +```sh +curl http://127.0.0.1:8080/healthz +``` + +Upload one tar or tar.gz source bundle archive: ```sh curl -X POST http://127.0.0.1:8080/upload \ @@ -119,299 +138,61 @@ curl -X POST http://127.0.0.1:8080/upload \ --data-binary @bundle.tar.gz ``` -Successful staging and admission returns a run id: +Accepted uploads return after the archive is staged and validated: ```json -{"run_id":"reports.20260603T120000Z.abcdef12","status":"accepted"} +{"run_id":"example-http-upload.20260604T120000Z.abcdef12","status":"accepted"} ``` -Poll status until it reaches `succeeded` or `failed`: +Poll status while the in-memory record is retained: ```sh curl http://127.0.0.1:8080/runs/ ``` -The status record includes the completed run report on successful publication -or error details on failure. Status is memory-only and expires after -`server.http.retention`; completed staged bundle directories are removed on -expiry. Restarting the process clears upload status and queue state. +Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire after `server.http.retention`. Expiration removes committed staged bundle directories for completed uploads. Restarting the process clears upload status, queue state, and in-memory records. -Malformed archives and invalid source bundles are rejected by `POST /upload` -before a run id is issued. +Upload admission is bounded by `server.http.queue_size`. Publication concurrency is bounded by `server.http.max_concurrency`, and the coordinator does not run two uploads for the same pipeline at the same time. -Use `GET /healthz` for readiness after config and tokens load: +The upload server accepts `application/x-tar`, `application/gzip`, and `application/x-gzip`. Archives are extracted into a temporary staging directory, must contain exactly one root-level `manifest.json`, and must validate as one complete source bundle before a run id is issued. Per-source `max_upload_size` bounds both uploaded archive size and extracted bundle size. The implementation also caps extracted file count. -```sh -curl http://127.0.0.1:8080/healthz -``` +The default bind address is private loopback. Put TLS, public routing, rate limiting, and external access policy in a reverse proxy or deployment layer. -The default bind address is private loopback. Put TLS, public routing, -rate-limiting, and external access policy in a reverse proxy or deployment -layer. +## Remote Backend Notes -## Filesystem Layout +### SSH/SFTP -Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`. - -Destination bundle paths are configured per destination with `path_mapping.mode`. - -The default mode, `preserve_relative`, preserves the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at that destination. - -The `fixed` mode publishes one selected source bundle at the destination backend root. A fixed destination with local `path: /srv/www/reports/latest` writes outputs and `.distributor.json` directly under `/srv/www/reports/latest`. Fixed destinations select the newest discovered source bundle by manifest `created` timestamp, with the source-root-relative bundle path as the deterministic tie-breaker. - -The maintained local examples write under `workspace/`, which is ignored by Git. - -SSH backends use the configured remote `path` as the backend root. Source bundle discovery and destination bundle paths are relative to that root, using the same logical path rules as local storage. - -S3 backends use the configured bucket plus optional `prefix` as the backend root. Source bundle discovery and destination bundle paths are relative to that object-key prefix. Prefixes are object-key prefixes, not real directories. - -## Destination State - -Each published destination bundle contains `.distributor.json`. This file is the managed sentinel and destination state record. It stores: - -- pipeline and destination identity; -- publication timestamp; -- source manifest used for publication; -- copied source output metadata; -- generated output metadata; -- optional public URL metadata when destination links are configured. - -`manifest.json` from the source bundle is not copied as destination state. - -Do not edit `.distributor.json` by hand during normal operation. If it is missing or invalid while destination files remain, `distributor` treats the destination as unmanaged or conflicted. - -## Go Producer Bundles - -Go producer applications can import `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to create complete local source bundles with the same path, digest, timestamp, and validation rules used by `distributor`. The package also exposes digest helpers, including `ValidateDigest`, for producer code that needs to validate lowercase `sha256:<64 hex>` strings before writing manifests. - -Minimal producer-side bundle creation: - -```go -manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{ - Root: outputDir, - ID: "reports.example.2026-05-30", - Files: []bundle.BundleFile{ - {SourcePath: reportPath, Path: "report.md"}, - {SourcePath: summaryPath, Path: "summary.txt"}, - }, -}) -if err != nil { - return err -} -``` - -`WriteBundle` copies local producer files into a sibling temporary directory, writes `manifest.json`, validates the result, and promotes the completed bundle into place. It fails if `Root` already exists unless `Overwrite` is true. With overwrite enabled, it builds and validates the replacement before moving the existing root aside. - -Use `BuildManifest` and `WriteManifest` when a producer already wrote all bundle files into the final root. `BuildManifest` can preserve an explicit file order, or `Scan: true` can recursively include regular files under `Root` in deterministic slash-path order. Scan mode includes dotfiles, excludes files named `manifest.json` or `.distributor.json`, and rejects symlinks. - -Shell producers can create the same manifest through the CLI after writing bundle files: - -```sh -go run ./cmd/distributor manifest create --id reports.example.2026-05-30 -go run ./cmd/distributor validate -``` - -Use repeated `--file` flags to preserve a specific file order. If no `--file` flags are provided, the command scans the bundle directory recursively using the same filtering rules as `pkg/bundle.BuildManifest`. - -## Static HTML Publication - -Markdown-to-HTML publication can write sidecar files or a fixed `index.html`. - -Use sidecar mode when each Markdown source should keep a matching HTML filename: - -```yaml -publish: - source: false - html: true -transform: - markdown_to_html: - enabled: true - mode: sidecar -``` - -Use index mode for static-site destinations that should serve a bundle through `index.html`: - -```yaml -publish: - source: false - html: true -transform: - markdown_to_html: - enabled: true - mode: index - input: report.md -``` - -If `input` is omitted in index mode, the source manifest must list exactly one Markdown file. Generated HTML is recorded in `.distributor.json` with `kind: generated`, `source_path`, `transform: markdown_to_html`, digest, and size metadata. - -## Archive And Latest Fan-Out - -A pipeline can publish the same source to an archive destination and a stable latest destination: - -```yaml -pipelines: - - id: reports - source: - backend: local - path: /var/spool/distributor/reports - destinations: - - id: archive - backend: local - path: /srv/reports/archive - path_mapping: - mode: preserve_relative - publish: - source: true - html: false - - id: latest-html - backend: local - path: /srv/www/reports/latest - path_mapping: - mode: fixed - links: - base_url: https://reports.example.com/latest - primary: auto - publish: - source: false - html: true - transform: - markdown_to_html: - enabled: true - mode: index - input: report.md -``` - -The archive destination plans every discovered source bundle at its source-relative path. The fixed latest destination plans only the newest discovered bundle and writes `index.html` plus `.distributor.json` at its backend root. - -## Static Site URLs - -Use destination `links` when a destination backend root corresponds to a public HTTP or HTTPS URL: - -```yaml -links: - base_url: https://reports.example.com/archive - primary: auto -``` - -Distributor records URLs in `.distributor.json`; it does not publish notifications or infer URLs from local, SSH, or S3 backend fields. - -For archive-style destinations, URLs include the destination bundle path. A source bundle under `daily/brentwood/2026-06-01` with `base_url: https://reports.example.com/archive` can produce: - -```text -https://reports.example.com/archive/daily/brentwood/2026-06-01/report.html -``` - -For fixed destinations, URLs are rooted at `links.base_url`. A fixed HTML index destination with `base_url: https://reports.example.com/latest` records: - -```text -https://reports.example.com/latest/ -``` - -`index.html` outputs use directory-style URLs. Other outputs include their filename. The primary URL is selected from the published outputs using the destination `links.primary` policy. - -## Source Validation and Inspection - -`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations. - -Configured source validation is useful when producers write directly to SSH or S3 storage: - -```sh -go run ./cmd/distributor validate --config --pipeline -go run ./cmd/distributor inspect --config --pipeline -``` - -Use `--bundle ` to validate or inspect one source-root-relative bundle directory: - -```sh -go run ./cmd/distributor validate \ - --config \ - --pipeline \ - --bundle daily/2026-06-01 -``` - -For configured SSH sources, host key and authentication behavior matches `run`. For configured S3 sources, endpoint, bucket, prefix, region, path-style, explicit credential environment variables, and `secrets.directory` handling match `run`. - -## Dry Runs - -`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH `known_hosts` entries. - -Dry-run output is useful before publishing to confirm actions such as `publish_new`, `replace_older`, `force_replace`, `skip_same`, and `skip_destination_newer`. - -Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication. Fixed path destinations add `path_mapping=fixed target=.` to planned action lines. Dry-run also prints a warning with the fixed destination candidate count and selected source bundle; destructive fixed replacements print an additional warning that the destination root would be replaced. - -Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, destination bundle paths, path mapping modes, optional link URLs, output records, final counters, and partial failure records. The summary includes `fixed_path`. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero. - -## Retry and Replacement Behavior - -If a destination has matching `.distributor.json`, publication skips it as already published. - -If destination state is older than the source manifest and transfer policy allows replacement, publication deletes only managed outputs recorded in `.distributor.json` plus the state file, then writes the new outputs and state. - -If destination state is newer than the source manifest, the default behavior is to skip. If destination state has the same source id and created timestamp but a different digest, publication fails as a conflict. - -If a destination path has files but no valid `.distributor.json`, publication fails as unmanaged content unless the current run explicitly uses `--force`. - -## Force Workflow - -Use `--force` only after a dry run shows the intended `force_replace` action: - -```sh -go run ./cmd/distributor run --config --dry-run --force -go run ./cmd/distributor run --config --force -``` - -Forced replacement can overwrite unmanaged non-empty destination paths. Destination state conflicts require `transfer.on_conflict: replace` plus `--force`. Newer destination state requires `transfer.on_destination_newer: replace` plus `--force`. - -Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For fixed destinations, the destination bundle path is the backend root, so forced replacement may clear that configured root but not its parent path, sibling directories, or anything outside the configured S3 bucket and prefix. Force is per run only and has no config default. - -## Failure Handling - -If one destination fails in a fan-out run, independent later destinations are still planned and executed. The command exits non-zero after printing the final status if any destination failed. - -Errors include the pipeline id, destination id, destination backend, and bundle path where applicable. - -In JSON mode, destination failures after planning or execution begins are reported in the top-level `errors` array and in the run result while preserving a non-zero exit code. Fatal setup errors such as an unreadable config or invalid secrets directory write no JSON document. - -If a write fails during publication, `distributor` attempts to remove outputs written during that failed attempt so a retry does not see those partial outputs as unmanaged destination content. - -After a successful publish or replacement, the internal notifier hook runs. The current default notifier is a no-op. Skipped destinations do not invoke it. - -## SSH Operation Notes - -SSH execution uses SFTP over `golang.org/x/crypto/ssh` and `github.com/pkg/sftp`. It does not shell out to `ssh`, `scp`, or `rsync`. +SSH execution uses native SFTP. It does not shell out to `ssh`, `scp`, or `rsync`. Configure `ssh_key_file`, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported. -The default host key policy is `accept-new`. New host keys are written to `known_hosts` when the file path is writable. During `--dry-run`, unknown host keys may be accepted for the current connection but are not written to `known_hosts`; a later non-dry-run may persist the same key. Changed host keys are fatal for both `strict` and `accept-new`. The `off` policy disables host key checking and `run` prints a warning when stdout is enabled. +The default host key policy is `accept-new`. During dry runs, unknown host keys may be accepted for the current connection but are not persisted. Changed host keys are fatal for `strict` and `accept-new`. `host_key_policy: off` disables host key checking and should be limited to controlled test environments. -Recovery boundaries are the same as local storage: replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, and failed writes are cleaned up where practical. Distributor never performs broad recursive remote deletion. +### S3-Compatible Storage -## S3 Operation Notes +S3 execution uses the AWS SDK for Go v2. Configure an endpoint, bucket, optional prefix, optional region, optional path-style setting, and optional explicit credential variable names. -S3 execution uses the AWS SDK for Go v2. Configure `endpoint`, `bucket`, optional `prefix`, optional `region`, and optional explicit credential environment variable names. +When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is. -When explicit credential env names are configured, both variables must resolve to non-empty values through the real process environment or `secrets.directory`. When they are omitted, the AWS SDK default credential chain is used as-is. +Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers. -Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. For fixed destinations, that prefix is the configured bucket plus optional `prefix`. Distributor does not manage bucket versioning or delete markers. +## Secrets Operation -## Secrets Directory +`secrets.directory` is loaded during `run`, `serve`, and configured-source `validate` or `inspect` before credential-consuming work starts. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts. -Configure `secrets.directory` when credential values should come from mounted files, such as deployment secrets: +Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, commands emit a warning naming the ignored secret variable without printing either value. The process environment is not modified. -```yaml -secrets: - directory: /run/secrets/distributor -``` +## Cleanup And Recovery -The directory is loaded during `run`, `serve`, and configured-source `validate` -or `inspect` before credential-consuming work starts. If the directory is -missing, unreadable, or contains an invalid secret filename, the command fails -before storage work starts. +Use these recovery boundaries: -Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` and configured-source diagnostics print a warning naming the ignored secret file variable without printing either value. The process environment is not changed. +- For source validation failures, regenerate the source bundle and manifest together. +- For an empty or missing destination, rerun after fixing config or storage access. +- For unmanaged destination content, move unrelated files aside or use a different destination path before publishing. +- For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun `--dry-run`. +- For state conflicts, verify the source, pipeline, destination, and existing `.distributor.json` before considering `--force`. +- For HTTP upload failures, inspect `/runs/` while retained; after expiry or restart, rely on destination state and logs/output from the publishing run. -## Caveats +Do not edit `.distributor.json` during normal recovery. Treat it as the managed state record used for comparison and safe cleanup. -External notification adapters are unavailable. Force overwrite behavior is available only through the explicit `run --force` workflow. - -For symptom-oriented fixes, see [troubleshooting](troubleshooting.md). For config details, see [configuration](config.md). For command syntax, see [CLI](cli.md). +For symptom-specific fixes, see [Troubleshooting](troubleshooting.md). diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 83b08e5..4080841 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,8 +1,14 @@ # Distributor Troubleshooting -## `load config ... no such file or directory` +Audience: administrators and operators diagnosing `distributor` command, configuration, publishing, storage, or HTTP upload failures. -Likely cause: `run` could not find the config path. If `--config` is omitted, the default path is `/usr/local/etc/distributor/config.yml`. +Each entry lists the symptom, likely cause, diagnostic step, safe fix, and relevant reference link. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), and operating procedures live in [Operations](operations.md). + +## Config File Is Missing + +Symptom: `load config ... no such file or directory`. + +Likely cause: `--config` points to a missing file, or `--config` was omitted and `/usr/local/etc/distributor/config.yml` is not installed. Diagnostic: @@ -10,11 +16,15 @@ Diagnostic: ls -l ``` -Safe fix: pass an existing config path with `--config`, or install a config at the default path. See [configuration](config.md). +Safe fix: pass an existing file with `--config`, or install a config at the default path. -## `parse config ... field not found` +Reference: [Configuration](config.md#config-file-loading). -Likely cause: the YAML contains an unknown field. Config loading rejects unknown keys. +## Config Contains An Unknown Field + +Symptom: `parse config ... field not found`. + +Likely cause: the YAML contains a key that is not part of the implemented config schema. Diagnostic: @@ -22,14 +32,15 @@ Diagnostic: go run ./cmd/distributor run --config --dry-run ``` -Safe fix: compare the file to the reference in [configuration](config.md) and remove or rename unsupported fields. +Safe fix: remove or rename unsupported fields using the canonical config reference. -## `validate config ... backend ... is unsupported` +Reference: [Configuration](config.md). -Likely cause: a source or destination uses an unsupported backend name, or a -command is trying to execute a backend that is valid only for another workflow. -`run`, `validate`, and `inspect` execute `local`, `ssh`, and `s3` sources. -`serve` executes `http_upload` sources. +## Backend Name Or Placement Is Invalid + +Symptom: `backend ... is unsupported` or `http_upload is only supported for sources`. + +Likely cause: a backend name is misspelled, not executable, or configured in the wrong role. Diagnostic: @@ -37,13 +48,343 @@ Diagnostic: rg -n "backend:" ``` -Safe fix: use `backend: local`, `backend: ssh`, or `backend: s3` for normal -source and destination workflows. Use `backend: http_upload` only for sources -handled by `distributor serve`. +Safe fix: use `local`, `ssh`, or `s3` for executable sources and destinations. Use `http_upload` only as a source served by `distributor serve`. -## `bind HTTP server ... address already in use` +Reference: [Configuration](config.md#backend-reference). -Likely cause: another process is already listening on `server.http.bind`. +## CLI Arguments Select The Wrong Source Mode + +Symptom: `configured source mode requires --pipeline`, `does not accept a local path with --config, --pipeline, or --bundle`, `validate command requires a path`, or `inspect command requires a path`. + +Likely cause: `validate` or `inspect` mixed local path mode with configured source mode, or omitted the required source selector. + +Diagnostic: + +```sh +go run ./cmd/distributor validate --help +go run ./cmd/distributor inspect --help +``` + +Safe fix: use either `distributor validate ` / `distributor inspect `, or use `--config --pipeline ` with optional `--bundle `. + +Reference: [CLI](cli.md#validate). + +## Output Format Is Invalid + +Symptom: `format must be text or json`. + +Likely cause: an unsupported value was passed to `--format`. + +Diagnostic: + +```sh +go run ./cmd/distributor run --help +``` + +Safe fix: use `--format text` or `--format json`. + +Reference: [CLI](cli.md#common-output-format). + +## JSON Mode Wrote No JSON Document + +Symptom: `--format json` exits non-zero and stdout has no JSON result. + +Likely cause: the command failed before it could construct a result, such as invalid arguments, missing config, unreadable secrets, or source setup failure. + +Diagnostic: + +```sh +go run ./cmd/distributor run --config --format json +``` + +Safe fix: read stderr, fix the setup problem, then rerun. Partial destination failures during `run` can produce JSON; fatal setup failures do not. + +Reference: [CLI](cli.md#output-and-exit-behavior). + +## Source Pipeline Is Not Found + +Symptom: `pipeline "" not found`. + +Likely cause: configured source diagnostics or upload processing selected a pipeline id that is absent from the loaded config. + +Diagnostic: + +```sh +rg -n "id:" +``` + +Safe fix: pass an existing `--pipeline` value or correct the pipeline id in config. + +Reference: [Configuration](config.md#pipelines). + +## Source Bundles Are Not Found + +Symptom: `no bundles found`, `no bundles found under "."`, `stat ssh ... not_found`, or `stat s3 ... not_found`. + +Likely cause: the source root, source-root-relative bundle path, S3 prefix, SSH path, or permissions do not expose a directory containing `manifest.json`. + +Diagnostic: + +```sh +go run ./cmd/distributor validate --config --pipeline +``` + +Safe fix: correct the configured source backend root, permissions, prefix, or `--bundle` path. The selected bundle directory must contain `manifest.json`. + +Reference: [Operations](operations.md#filesystem-and-storage-layout). + +## Source Manifest Or Files Fail Validation + +Symptom: `sha256 mismatch`, `size mismatch`, `digest mismatch`, missing manifest fields, or unsafe source paths. + +Likely cause: files changed after `manifest.json` was written, the manifest digest is stale, or the producer wrote invalid bundle paths. + +Diagnostic: + +```sh +go run ./cmd/distributor validate +``` + +Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source validation failures. + +Reference: [Operations](operations.md#cleanup-and-recovery). + +## Destination Has Unmanaged Content + +Symptom: `destination has content but no distributor state` or a plan reason containing `fail_unmanaged`. + +Likely cause: the destination bundle path contains files but no valid `.distributor.json`, so `distributor` will not claim it by default. + +Diagnostic: + +```sh +find -maxdepth 2 -print +``` + +Safe fix: choose an empty destination path, move unrelated files aside, or preview `run --dry-run --force` only after confirming the reported bundle path is safe to replace. + +Reference: [Operations](operations.md#forced-replacement-workflow). + +## Destination State Conflicts With Source + +Symptom: `fail_conflict`, `destination source id differs`, `same id and created time but different digest`, `pipeline id ... does not match`, or `destination id ... does not match`. + +Likely cause: `.distributor.json` belongs to a different pipeline, destination, source id, or same-created source with different content. + +Diagnostic: + +```sh +cat /.distributor.json +go run ./cmd/distributor inspect +``` + +Safe fix: verify the source and destination are intended to match. Use a separate destination path for unrelated content. To replace the existing state, configure `transfer.on_conflict: replace`, preview with `--dry-run --force`, then publish with `--force`. + +Reference: [Operations](operations.md#destination-state-and-retry-behavior). + +## Destination Is Newer Than Source + +Symptom: `skip_destination_newer` or `destination is newer and replacement requires --force`. + +Likely cause: the destination state records a source manifest with a later `created` timestamp than the current source. + +Diagnostic: + +```sh +go run ./cmd/distributor run --config --dry-run --format json +``` + +Safe fix: keep the default skip behavior unless replacement is intentional. To replace newer state, configure `transfer.on_destination_newer: replace`, preview with `--dry-run --force`, then publish with `--force`. + +Reference: [Operations](operations.md#forced-replacement-workflow). + +## Forced Replacement Appears In A Plan + +Symptom: dry-run output includes `force_replace`. + +Likely cause: the run used `--force`, and planning selected a supported destructive replacement. + +Diagnostic: + +```sh +go run ./cmd/distributor run --config --dry-run --force +``` + +Safe fix: inspect the pipeline id, destination id, backend, and bundle path. Proceed only if deleting everything inside that destination bundle path is intended. + +Reference: [Operations](operations.md#forced-replacement-workflow). + +## Output Path Collision + +Symptom: `destination output path collision`. + +Likely cause: publication would write two outputs to the same destination path, such as copying `report.html` while also generating `report.html` from `report.md`. + +Diagnostic: + +```sh +go run ./cmd/distributor run --config --dry-run +``` + +Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide. + +Reference: [Configuration](config.md#publish-and-transform-policy). + +## Run Failed After Writing Some Files + +Symptom: a destination write failed and the command exited non-zero after partial work. + +Likely cause: storage write failure, permission issue, network interruption, or object-store error during publish execution. + +Diagnostic: + +```sh +find -maxdepth 2 -print +``` + +Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. Rerun `--dry-run` before publishing again. + +Reference: [Operations](operations.md#destination-state-and-retry-behavior). + +## JSON Run Result Has `ok: false` + +Symptom: `run --format json` exits non-zero with a JSON result where `ok` is `false`. + +Likely cause: at least one destination failed after planning or execution began, while other destination results were still available. + +Diagnostic: + +```sh +go run ./cmd/distributor run --config --format json +``` + +Safe fix: inspect the top-level `errors` array, destination actions, output errors, and summary. Fix failed destinations, then preview with `--dry-run --format json`. + +Reference: [CLI](cli.md#output-and-exit-behavior). + +## Secrets Directory Is Missing Or Unreadable + +Symptom: `load secrets directory ... no such file or directory`, `permission denied`, or `secret filename ... is invalid`. + +Likely cause: `secrets.directory` points to a missing or unreadable directory, or it contains a regular file whose name is not a valid credential variable name. + +Diagnostic: + +```sh +ls -ld +find -maxdepth 1 -type f -printf '%f\n' +``` + +Safe fix: mount or create the directory, adjust permissions for the service user, or rename/remove invalid secret files. Secret filenames must match `[A-Za-z_][A-Za-z0-9_]*`. + +Reference: [Configuration](config.md#secrets). + +## Credential Variable Is Missing Or Empty + +Symptom: `credential environment variable ... is not set`, `credential environment variable ... is empty`, or S3 authentication errors such as `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch`. + +Likely cause: configured S3 credential variable names are not available through the process environment or `secrets.directory`, are empty, or do not authorize the requested bucket/prefix. + +Diagnostic: + +```sh +env | cut -d= -f1 | rg '^(|)$' +ls -l +``` + +Safe fix: provide both configured S3 credential values, correct IAM/service permissions, or omit explicit credential fields to use the AWS SDK default credential chain. + +Reference: [Configuration](config.md#s3-compatible-backend). + +## Secret File Is Ignored In Favor Of Environment + +Symptom: `secret ... ignored because the real environment already has that variable`. + +Likely cause: the same credential name exists in the process environment and `secrets.directory` with different values. + +Diagnostic: + +```sh +env | cut -d= -f1 | rg '^$' +ls -l / +``` + +Safe fix: remove one source of the credential or intentionally keep the process environment value. `distributor` does not print either value. + +Reference: [Operations](operations.md#secrets-operation). + +## SSH Auth Is Not Configured + +Symptom: `no SSH auth methods configured`. + +Likely cause: no SSH agent is available and `ssh_key_file` is missing or unreadable. + +Diagnostic: + +```sh +test -n "$SSH_AUTH_SOCK" && ssh-add -l +ls -l +``` + +Safe fix: start an SSH agent with an appropriate key loaded, or configure a readable private key with `ssh_key_file`. + +Reference: [Configuration](config.md#sshsftp-backend). + +## SSH Host Key Fails + +Symptom: `host key ... is unknown`, `known_hosts is required`, or `host key ... has changed`. + +Likely cause: strict host key checking has no trusted key, `accept-new` cannot persist a new key, or the remote host key differs from the stored key. + +Diagnostic: + +```sh +ls -l +ssh-keygen -F -f +``` + +Safe fix: pre-populate `known_hosts` for `strict`, configure a writable `known_hosts` path for `accept-new`, or verify the server identity before updating a changed key. Do not disable host key checking to bypass an unexpected changed key. + +Reference: [Operations](operations.md#sshsftp). + +## S3 Prefix Is Invalid + +Symptom: `prefix must be a clean relative slash-separated path`. + +Likely cause: the S3 prefix contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed. + +Diagnostic: + +```sh +go run ./cmd/distributor run --config --dry-run +``` + +Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`. + +Reference: [Configuration](config.md#s3-compatible-backend). + +## S3 Location Or Connectivity Fails + +Symptom: `NoSuchBucket`, `InvalidBucketName`, `not_found`, endpoint connection failures, or TLS/network errors. + +Likely cause: endpoint, bucket, prefix, region, path-style mode, network routing, or credentials are wrong for the service. + +Diagnostic: + +```sh +go run ./cmd/distributor run --config --dry-run +curl -I +``` + +Safe fix: verify `endpoint`, `bucket`, `region`, `prefix`, and `force_path_style`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing. Distributor does not provide an insecure TLS bypass setting. + +Reference: [Operations](operations.md#s3-compatible-storage). + +## HTTP Server Cannot Bind + +Symptom: `bind HTTP server ... address already in use`. + +Likely cause: another process is listening on `server.http.bind`. Diagnostic: @@ -51,44 +392,33 @@ Diagnostic: ss -ltnp | rg '' ``` -Safe fix: stop the conflicting process or configure a different -`server.http.bind` value. The default bind address is `127.0.0.1:8080`. +Safe fix: stop the conflicting process or configure a different bind address. -## `upload token environment variable ... is not set` +Reference: [Configuration](config.md#serverhttp). -Likely cause: a configured `http_upload` source references `token_env`, but the -variable is absent from both the real process environment and -`secrets.directory`. +## HTTP Upload Token Is Missing Or Duplicated -Diagnostic: +Symptom: `upload token environment variable ... is not set`, `... is empty`, or `upload token environment variables ... resolve to the same value`. -```sh -env | cut -d= -f1 | rg '^$' -ls -l / -``` - -Safe fix: set the real environment variable or create a readable -secrets-directory file with the same name. Do not place literal token values in -YAML. - -## `upload token environment variables ... resolve to the same value` - -Likely cause: two configured `http_upload` pipelines resolve to the same bearer -token value. +Likely cause: an `http_upload` source references a missing/empty `token_env`, or two upload pipelines resolve to the same bearer token. Diagnostic: ```sh rg -n 'token_env:' +env | cut -d= -f1 | rg '^$' +ls -l / ``` -Safe fix: assign a distinct non-empty token value to each `http_upload` -pipeline. Distributor does not print the duplicate token value. +Safe fix: provide one distinct non-empty token value per upload pipeline through the process environment or `secrets.directory`. Do not put literal tokens in YAML. -## `POST /upload` returns `401` +Reference: [Configuration](config.md#http-upload-source-backend). -Likely cause: the request is missing `Authorization: Bearer ` or the -token does not match any configured `http_upload` pipeline. +## Upload Request Is Unauthorized + +Symptom: `POST /upload` returns `401`. + +Likely cause: the request lacks `Authorization: Bearer `, has an empty token, or uses a token that does not match any configured upload pipeline. Diagnostic: @@ -99,73 +429,33 @@ curl -i -X POST http://127.0.0.1:8080/upload \ --data-binary @bundle.tar ``` -Safe fix: use the token value resolved by the configured `token_env`. Do not -include token values in logs or tickets. +Safe fix: use the token value resolved by the configured `token_env`. Do not include token values in logs or tickets. -## `POST /upload` returns `400` +Reference: [Operations](operations.md#http-upload-operation). -Likely cause: the archive content is malformed, the gzip body is invalid, the -tar body cannot be extracted safely, or the extracted source bundle fails -manifest and file validation. +## Upload Request Is Rejected Before A Run ID + +Symptom: `POST /upload` returns `400`, `413`, `415`, or `503`. + +Likely cause: the request included a `pipeline` or `pipeline_id` query, archive content is malformed, the body exceeds size limits, content type is unsupported, or the in-memory upload queue is full. Diagnostic: ```sh tar -tf bundle.tar tar -tzf bundle.tar.gz -go run ./cmd/distributor validate +rg -n 'max_upload_size|queue_size|max_concurrency' ``` -Safe fix: rebuild the tar or tar.gz archive from one complete source bundle -root. The archive must contain exactly one root-level `manifest.json`, and every -manifest-listed file must exist as a regular file with matching size and digest. +Safe fix: send one valid tar or tar.gz source bundle archive with `Content-Type: application/x-tar`, `application/gzip`, or `application/x-gzip`; remove pipeline query parameters; reduce archive size or raise the configured limit; retry after queue pressure drops. -## `POST /upload` returns `413` +Reference: [Operations](operations.md#http-upload-operation). -Likely cause: the request body exceeds the selected pipeline's -`source.max_upload_size` or the default `server.http.max_upload_size`. +## Upload Status Is Missing -Diagnostic: +Symptom: `GET /runs/` returns `404`. -```sh -ls -lh bundle.tar bundle.tar.gz -rg -n 'max_upload_size:' -``` - -Safe fix: upload a smaller archive, remove unnecessary files from the source -bundle, or raise the configured upload size limit. - -## `POST /upload` returns `415` - -Likely cause: the upload uses an unsupported content type. The server accepts -uncompressed tar and gzip-compressed tar archives only. - -Diagnostic: - -```sh -file bundle.tar.gz -``` - -Safe fix: send `Content-Type: application/x-tar`, `application/gzip`, or -`application/x-gzip`, matching the archive format. - -## `POST /upload` returns `503` - -Likely cause: the in-memory upload queue is full. - -Diagnostic: - -```sh -rg -n 'queue_size|max_concurrency' -``` - -Safe fix: retry after active uploads finish, or increase `server.http.queue_size` -for the deployment. - -## `GET /runs/` returns `404` - -Likely cause: the run id is wrong, the process restarted, or the completed -status record expired after `server.http.retention`. +Likely cause: the run id is wrong, the process restarted, or the retained status record expired after `server.http.retention`. Diagnostic: @@ -174,362 +464,6 @@ curl -i http://127.0.0.1:8080/runs/ rg -n 'retention:' ``` -Safe fix: use the exact `run_id` returned by `POST /upload`. If status retention -is too short for operators, increase `server.http.retention`. +Safe fix: use the exact `run_id` returned by upload admission. Increase retention if operators need a longer status window. -## `--format: format must be text or json` - -Likely cause: a command was run with an unsupported output format. - -Diagnostic: - -```sh -go run ./cmd/distributor run --help -``` - -Safe fix: use `--format text` or `--format json`. Help and usage output are always text. - -## `--format json` wrote no JSON output - -Likely cause: the command failed before it could construct a result, such as a missing config file, invalid arguments, unreadable secrets directory, or source setup failure. - -Diagnostic: - -```sh -go run ./cmd/distributor run --config --format json -``` - -Safe fix: read the stderr error and fix the setup problem. JSON mode writes a document only after the command has enough information to construct a result. - -## `configured source mode requires --pipeline` - -Likely cause: `validate` or `inspect` was run with `--config` but without an explicit pipeline id. - -Diagnostic: - -```sh -go run ./cmd/distributor validate --help -go run ./cmd/distributor inspect --help -``` - -Safe fix: add `--pipeline `. Configured source diagnostics require an explicit pipeline even when the config contains one pipeline. - -## `does not accept a local path with --config, --pipeline, or --bundle` - -Likely cause: local-path mode and configured source mode were mixed in one `validate` or `inspect` command. - -Diagnostic: - -```sh -go run ./cmd/distributor inspect --help -``` - -Safe fix: use either `distributor inspect ` or `distributor inspect --config --pipeline `, not both. - -## `--format json` exited non-zero with `ok: false` - -Likely cause: `run` began planning or executing destinations, and at least one destination failed while other destination results were still available. - -Diagnostic: - -```sh -go run ./cmd/distributor run --config --format json -``` - -Safe fix: inspect the top-level `errors` array, `result.actions`, and `result.summary`. Fix the failed destination, then preview with `--dry-run --format json` before retrying. - -## `prefix must be a clean relative slash-separated path` - -Likely cause: S3 `prefix` contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed. - -Diagnostic: - -```sh -go run ./cmd/distributor run --config --dry-run -``` - -Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`. - -## `NoSuchBucket`, `InvalidBucketName`, or `not_found` - -Likely cause: the S3 bucket, endpoint, or prefix is wrong, or the configured credentials cannot see the requested object. - -Diagnostic: - -```sh -go run ./cmd/distributor run --config --dry-run -``` - -Safe fix: verify `endpoint`, `bucket`, `region`, `force_path_style`, and `prefix`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing. - -## `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch` - -Likely cause: S3 credentials are missing, wrong, empty, or lack permission for the bucket or prefix. - -Diagnostic: - -```sh -env | cut -d= -f1 | rg '^(|)$' -ls -l -``` - -Safe fix: provide both configured credential environment variables through the real environment or `secrets.directory`, or omit explicit credential fields to use the AWS SDK default credential chain. - -## S3 endpoint connection failures - -Likely cause: the endpoint URL is unreachable, uses the wrong scheme, or does not match the configured path-style mode. - -Diagnostic: - -```sh -curl -I -``` - -Safe fix: correct `endpoint`, network routing, TLS settings outside distributor, or `force_path_style`. Distributor does not provide insecure TLS bypass configuration. - -## `load secrets directory ... no such file or directory` - -Likely cause: `secrets.directory` points to a missing directory. - -Diagnostic: - -```sh -ls -ld -``` - -Safe fix: create or mount the directory before running, or remove `secrets.directory` if no credential files are needed. - -## `load secrets directory ... permission denied` - -Likely cause: the service user cannot read the configured secrets directory. - -Diagnostic: - -```sh -ls -ld -namei -l -``` - -Safe fix: adjust the directory path or deployment permissions so the service user can read the directory. Distributor does not enforce owner, group, or mode policy beyond OS read access. - -## `secret filename ... is invalid` - -Likely cause: a regular file in `secrets.directory` does not match `[A-Za-z_][A-Za-z0-9_]*`. - -Diagnostic: - -```sh -find -maxdepth 1 -type f -printf '%f\n' -``` - -Safe fix: rename the file to a valid credential environment variable name, or remove it from the secrets directory. - -## `credential environment variable ... is not set` - -Likely cause: a backend credential field references an environment variable that is absent from both the real process environment and the configured secrets directory. - -Diagnostic: - -```sh -env | cut -d= -f1 | rg '^$' -ls -l / -``` - -Safe fix: set the real environment variable or create a readable secrets-directory file with the same name. - -## `secret ... ignored because the real environment already has that variable` - -Likely cause: the real process environment and secrets directory both define the variable with different values. - -Diagnostic: - -```sh -env | cut -d= -f1 | rg '^$' -ls -l / -``` - -Safe fix: remove one source of the credential or make the deployment intentionally prefer the real environment value. Distributor does not print either value. - -## `host is required for ssh backend` - -Likely cause: SSH config is missing the structured `host` field, or an old URL-style SSH config is still in use. - -Diagnostic: - -```sh -go run ./cmd/distributor run --config --dry-run -``` - -Safe fix: configure SSH with `host`, optional `user` and `port`, and `path`. SSH URLs are not part of the active config schema. - -## `no SSH auth methods configured` - -Likely cause: neither an SSH agent nor `ssh_key_file` is available. - -Diagnostic: - -```sh -test -n "$SSH_AUTH_SOCK" && ssh-add -l -ls -l -``` - -Safe fix: start an SSH agent with an appropriate key loaded, or configure `ssh_key_file` with a readable private key. - -## `host key ... is unknown` or `known_hosts is required` - -Likely cause: strict host key checking has no known host key, or `accept-new` cannot persist a new key. - -Diagnostic: - -```sh -ls -l -ssh-keygen -F -f -``` - -Safe fix: configure a writable `known_hosts` path for `accept-new`, pre-populate `known_hosts` for `strict`, or explicitly use `host_key_policy: off` only for insecure test environments. - -## `host key ... has changed` - -Likely cause: the remote server presented a different host key than the one recorded in `known_hosts`. - -Diagnostic: - -```sh -ssh-keygen -F -f -``` - -Safe fix: verify the server identity out of band before updating `known_hosts`. Do not switch to `host_key_policy: off` to bypass an unexpected changed key. - -## `pipeline "" not found` - -Likely cause: configured source validation or inspection requested a pipeline id that is not present in the config file. - -Diagnostic: - -```sh -rg -n "id:" -``` - -Safe fix: pass an existing pipeline id with `--pipeline`, or update the config. - -## `stat ssh ... not_found`, `stat s3 ... not_found`, or `no bundles found` - -Likely cause: the configured source root is wrong, unreadable, or does not contain source bundles. - -Diagnostic: - -```sh -go run ./cmd/distributor validate --config --pipeline -``` - -Safe fix: correct the configured source root, S3 prefix, permissions, or source bundle location. Use `--bundle ` only with a source-root-relative bundle directory that contains `manifest.json`. - -## `validate command requires a path` or `inspect command requires a path` - -Likely cause: `validate` or `inspect` was run without a local path and without configured source mode. - -Diagnostic: - -```sh -go run ./cmd/distributor validate --help -go run ./cmd/distributor inspect --help -``` - -Safe fix: pass a local source bundle directory or local tree, or pass both `--config ` and `--pipeline `. - -## `no bundles found under "."` - -Likely cause: the selected source root does not contain a `manifest.json` source bundle. - -Diagnostic: - -```sh -find -name manifest.json -print -``` - -Safe fix: point the command or config at the directory containing the source bundle, or write a valid `manifest.json` and listed files. See [CLI](cli.md). - -## `sha256 mismatch`, `size mismatch`, or `digest mismatch` - -Likely cause: a listed source file changed after `manifest.json` was created, or the manifest digest does not match its file list. - -Diagnostic: - -```sh -go run ./cmd/distributor validate -``` - -Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source digest failures. - -## `destination has content but no distributor state` - -Likely cause: the destination path is not empty and has no `.distributor.json` state file, so `distributor` will not claim it as managed. - -Diagnostic: - -```sh -find -maxdepth 2 -print -``` - -Safe fix: choose an empty destination path or move existing files aside after confirming they are not needed. If the destination should be claimed by distributor, preview with `run --dry-run --force` and publish with `run --force` only after confirming the reported `force_replace` action is bounded to the intended bundle path. - -## `fail_conflict` - -Likely cause: existing `.distributor.json` belongs to a different pipeline, a different destination, a different source id, or a same-created source with a different digest. - -Diagnostic: - -```sh -cat /.distributor.json -go run ./cmd/distributor inspect -``` - -Safe fix: verify you are publishing the intended source to the intended destination. Use a separate destination path for unrelated content. If the existing state should be replaced, configure `transfer.on_conflict: replace`, preview with `run --dry-run --force`, then publish with `run --force`. - -## `destination is newer and replacement requires --force` - -Likely cause: config explicitly allows newer-destination replacement, but the current run did not include `--force`. - -Diagnostic: - -```sh -go run ./cmd/distributor run --config --dry-run --force -``` - -Safe fix: prefer the default `transfer.on_destination_newer: skip` unless replacing newer destination state is intentional. To replace it, keep `transfer.on_destination_newer: replace`, confirm the dry-run output shows `force_replace`, then run with `--force`. - -## `force_replace` - -Likely cause: the current run used `--force` and publish planning selected a supported destructive replacement. - -Diagnostic: - -```sh -go run ./cmd/distributor run --config --dry-run --force -``` - -Safe fix: inspect the printed pipeline id, destination id, backend, and bundle path. Proceed only if deleting all content within that destination bundle path is intended. - -## `destination output path collision` - -Likely cause: configured publication would write two outputs to the same destination path, such as publishing a source `report.html` while also generating `report.html` from `report.md`. - -Diagnostic: - -```sh -go run ./cmd/distributor run --config --dry-run -``` - -Safe fix: adjust the source bundle contents or publish policy so source and generated outputs do not collide. - -## A run failed after writing some files - -Likely cause: a write failed partway through publication. Local, SSH, and S3 execution attempt to clean up outputs written during the failed attempt. - -Diagnostic: - -```sh -find -maxdepth 2 -print -``` - -Safe fix: use the pipeline id, destination id, backend, and bundle path printed in the run error to inspect the destination before retrying. If only unrelated unmanaged files remain, move them aside or choose a clean destination. Re-run with `--dry-run` before publishing again. See [operations](operations.md). +Reference: [Operations](operations.md#http-upload-operation).