distributor/docs/operations.md

# Distributor Operations

Audience: administrators and operators who run `distributor`, publish bundles, operate the HTTP upload service, or recover from failed runs.

This document covers operating workflows, storage layout, safety behavior, and recovery. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), symptom-specific fixes live in [Troubleshooting](troubleshooting.md), and external contracts live under [Integrations](integrations/source-bundle.md).

## Normal Workflow

Validate a producer bundle before publishing:

```sh
go run ./cmd/distributor validate examples/source-bundle
```

Preview a configured run before writing destination content:

```sh
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
```

Publish after reviewing the preview:

```sh
go run ./cmd/distributor run --config examples/local-publish.yml
```

Use JSON output for automation:

```sh
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run --format json
```

Use configured source diagnostics when the source is defined in YAML and may be local, SSH/SFTP, or S3-compatible storage:

```sh
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
go run ./cmd/distributor inspect --config examples/local-publish.yml --pipeline example-source-bundle --format json
```

Remote examples under `examples/ssh-destination.yml` and `examples/s3-destination.yml` are load-tested templates. Edit their endpoint, path, key, bucket, prefix, and credential values for storage you control before running them.

## Filesystem And Storage Layout

A source bundle is a directory containing `manifest.json` and every file listed in that manifest. See [Source Bundle Contract](integrations/source-bundle.md). Source discovery walks beneath the configured source backend root and finds bundle directories.

Each destination has its own backend root:

- Local destinations use the configured local `path`.
- SSH/SFTP destinations use the configured remote `path`.
- S3-compatible destinations use the configured `bucket` plus optional `prefix`.

Destination path mapping controls where each source bundle is published beneath the destination root:

- `preserve_relative` publishes each source bundle at the same source-root-relative path.
- `fixed` publishes one selected source bundle at the destination root.

Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins.

Published destination bundle paths contain `.distributor.json`. See [Destination State Contract](integrations/destination-state.md). This file is both the managed sentinel and the destination state record. It records the pipeline id, destination id, publication time, state mode, reconciliation mode, source manifest, copied outputs, generated outputs, output timestamps, and optional public URL metadata.

`manifest.json` from the source bundle is not copied as destination state.

## Destination State And Retry Behavior

`distributor` compares the source manifest to destination `.distributor.json` before writing:

- No destination state and no destination content: publish new outputs.
- Matching destination state: skip as already published.
- Older destination state for the same source id: replace if transfer policy allows it.
- Newer destination state for the same source id: skip by default.
- Valid managed state with an identity, source, or shared-root output-owner mismatch: replace only when destination `takeover.mode` allows it.
- Invalid destination state, identity, source, or shared-root output-owner mismatches not allowed by `takeover.mode`, or same-created digest mismatch: fail by default.
- Content without `.distributor.json`: fail as unmanaged content by default.

When destination state is older than the source, `transfer.on_destination_older` controls whether publication may proceed and `reconciliation.mode` controls how managed outputs are updated.

For takeover replacement, `reconciliation.mode: merge` does not retain omitted outputs from the previous source identity. The destination is rewritten as a managed replacement for the current source or shared-root owner.

`reconciliation.mode: replace` is the default. It deletes only managed output paths recorded in `.distributor.json` plus the state file, verifies the destination bundle path is empty, then writes the newly planned outputs and state. The new state `outputs` array is exactly the newly planned output set.

`reconciliation.mode: merge` retains prior managed outputs that are omitted from the new plan. It overwrites planned paths only when those paths are already recorded in existing state as managed. If a newly planned path already exists in storage but is not recorded in state, publication fails as an unmanaged path collision. The new state `outputs` array is the cumulative managed output set.

For both modes, retained or overwritten paths are identified only from `.distributor.json`; unmanaged files are not adopted.

For `state.mode: shared_root`, one destination root may contain outputs from multiple pipeline/destination owners. Comparisons, replacement, and merge retention are scoped to the current owner. Outputs owned by other owners are preserved unless a planned output path is owned by another valid owner and `takeover.mode` allows moving that path to the current owner. A planned path that exists in storage but is not recorded in state fails as unmanaged content by default.

If `state.mode: shared_root` is configured on a destination whose existing single-owner state belongs to the same pipeline and destination, the next successful publish converts that state file to shared-root schema. Existing single-owner state for a different pipeline or destination remains a conflict.

If a write fails after some outputs were written, `distributor` attempts cleanup before returning the error. In `replace` mode and takeover replacement, cleanup removes outputs written during that failed attempt. In same-source `merge` mode, cleanup removes only newly created outputs from that failed attempt; overwritten managed outputs are left in place because they previously belonged to the managed set. Operators should still inspect the destination after a failed write before retrying.

Fan-out destinations are independent. If one destination fails after planning or execution begins, later destinations are still attempted. The command exits non-zero if any destination failed.

## Destination State Repair

Use `reconcile-state` when `.distributor.json` still records managed outputs that no longer exist in destination storage. This repairs the state record only; it does not restore missing files.

Preview the repair first:

```sh
go run ./cmd/distributor reconcile-state \
  --config <config-path> \
  --pipeline <pipeline-id> \
  --destination <destination-id> \
  --dry-run
```

Apply after reviewing the report:

```sh
go run ./cmd/distributor reconcile-state \
  --config <config-path> \
  --pipeline <pipeline-id> \
  --destination <destination-id>
```

The command opens the configured destination root selected by `--pipeline` and `--destination`, reads the root `.distributor.json`, checks each managed output path with storage metadata, reports missing managed outputs, and reports unmanaged entries under that root. It excludes `.distributor.json` and already managed paths from unmanaged reporting.

Without `--dry-run`, it removes missing managed output records from valid state and rewrites `.distributor.json`. It never deletes destination files, adopts unmanaged files, validates output digests, or rewrites invalid or mismatched state.

For single-owner state, the state owner must match the selected pipeline and destination. For shared-root state, repair is scoped to the selected owner by default. Add `--all-owners` only when every owner in the selected shared-root state should have missing managed output records removed.

## Managed Output Pruning

Use `prune` when a destination config has `retention.prune.enabled: true` and old managed outputs should be removed according to that configured policy. Pruning is never automatic after publish.

Preview selected managed outputs first:

```sh
go run ./cmd/distributor prune \
  --config <config-path> \
  --pipeline <pipeline-id> \
  --destination <destination-id> \
  --dry-run
```

Apply after reviewing the report:

```sh
go run ./cmd/distributor prune \
  --config <config-path> \
  --pipeline <pipeline-id> \
  --destination <destination-id> \
  --apply
```

The command opens the configured destination root selected by `--pipeline` and `--destination`, reads the root `.distributor.json`, and plans from the selected destination's `retention.prune` policy. It uses managed output `updated_at` timestamps. When both `keep_latest` and `older_than` are configured, it preserves the newest `keep_latest` outputs before applying the age policy.

`--dry-run` does not delete outputs or rewrite state. `--apply` deletes only planned managed output paths, preserves unmanaged files, preserves `.distributor.json`, removes confirmed deleted records from state, and updates the state timestamp. If a delete fails after earlier deletes succeed, state is rewritten only for confirmed deletions; failed and unattempted output records remain so retry remains accurate.

For single-owner state, the state owner must match the selected pipeline and destination. For shared-root state, pruning is scoped to the selected owner and preserves other owners.

## Dry Runs And Output Review

`run --dry-run` loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, `.distributor.json`, or SSH `known_hosts` entries. For reconciliation, dry runs report the same high-level action labels as execution; inspect the configured destination's `reconciliation.mode` to determine whether `replace_older` will replace the managed set or merge into it.

For shared-root destinations, dry runs are owner-scoped. A `replace_older` action replaces or merges only the current owner according to `reconciliation.mode`; unrelated owners remain managed by the shared-root state. Paths owned by another owner fail as conflicts unless `takeover.mode` allows ownership transfer.

Review these action labels before publishing:

- `publish_new`: destination state is absent, or a shared-root owner is absent and planned paths are publishable.
- `replace_older`: destination state is older than the source.
- `replace_takeover`: destination state is valid managed state and `takeover.mode` allows replacement across an identity, source, or shared-root output-owner mismatch.
- `skip_same`: destination state already matches the source.
- `skip_destination_newer`: destination state is newer than the source and is skipped.
- `force_replace`: destructive replacement selected because `--force` is present and policy permits it.
- `error`: planning or execution failed for that destination.

Fixed destinations add fixed-path warnings during dry runs, including the selected source bundle and replacement warnings when the destination root would be replaced.

Text and JSON summaries count `publish_new`, `replace_older`, `replace_takeover`, `force_replace`, skipped, and failed destinations separately. Takeover action records include the configured takeover mode and the conflict reason. JSON output includes warnings, pipeline summaries, destination action records, output records, URLs when configured, final counters, and partial failure details. Fatal setup failures such as unreadable config or invalid secrets do not produce a JSON result document.

## Forced Replacement Workflow

Use `--force` only after a dry run shows the intended bounded `force_replace` action:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run --force
go run ./cmd/distributor run --config <config-path> --force
```

Forced replacement can claim unmanaged non-empty destination paths. State conflicts require both `--force` and transfer policy that permits replacement:

- newer destination state requires `transfer.on_destination_newer: replace`;
- conflict outcomes require `transfer.on_conflict: replace`.

Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete parent paths, sibling paths, or storage outside the destination bundle path. For fixed destinations, the destination bundle path is the backend root, so a forced replacement can clear that configured root.

For shared-root destinations, forced replacement also deletes the configured destination bundle path before writing new shared-root state. This removes unrelated owners inside that destination root. Preview with `--dry-run --force` and confirm the destination path before applying.

`--force` applies only to the current invocation. There is no config field that enables forced replacement by default.

## HTTP Upload Operation

The [HTTP Upload API Contract](integrations/http-upload.md) defines request and response details. `distributor serve` runs the HTTP upload API for pipelines whose source backend is `http_upload`. Top-level `upload_tokens` authenticate producers and allow one or more upload pipelines. Token values come from the process environment or `secrets.directory`, not from YAML literal values.

Start the maintained local example:

```sh
DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN=<token> \
  go run ./cmd/distributor serve --config examples/http-upload-local.yml
```

Readiness:

```sh
curl http://127.0.0.1:8080/healthz
```

Upload one tar or tar.gz source bundle archive:

```sh
curl -X POST http://127.0.0.1:8080/v1/pipelines/example-http-upload/upload \
  -H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \
  -H "Content-Type: application/gzip" \
  --data-binary @bundle.tar.gz
```

For safe producer retries, include an idempotency key that is stable for the same producer run and different for each distinct run:

```sh
curl -X POST http://127.0.0.1:8080/v1/pipelines/example-http-upload/upload \
  -H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \
  -H "Content-Type: application/gzip" \
  -H "Idempotency-Key: producer.run.20260604T120000Z" \
  --data-binary @bundle.tar.gz
```

Go producer applications can use `pkg/upload` instead of constructing archives and HTTP requests directly. See [Upstream Producer Integration](consumers/api.md) for the copyable producer implementation guide.

The maintained example client uses the local upload server, reads the token from `DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN`, and defaults the pipeline id to `example-http-upload`. Set `DISTRIBUTOR_EXAMPLE_UPLOAD_PIPELINE_ID` or pass a second argument to use another configured upload pipeline. It generates an idempotency key by default; set `DISTRIBUTOR_EXAMPLE_UPLOAD_IDEMPOTENCY_KEY` when retrying the same producer run across separate process runs.

```sh
go run ./examples/upload-client
```

Accepted uploads return after the archive is staged and validated:

```json
{"run_id":"example-http-upload.20260604T120000Z.abcdef12","status":"accepted"}
```

Poll status while the in-memory record is retained:

```sh
curl http://127.0.0.1:8080/runs/<run-id>
```

Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire after `server.http.retention`. Expiration removes committed staged bundle directories for completed uploads. Restarting the process clears upload status, queue state, and in-memory records.

Upload admission is bounded by `server.http.queue_size`. Publication concurrency is bounded by `server.http.max_concurrency`, and the coordinator does not run two uploads for the same pipeline at the same time.

`Idempotency-Key` is optional for raw HTTP clients. When present, it is scoped to the token id, pipeline id, and key. Reusing the same key with the same normalized source manifest in that scope returns the original accepted run response and does not enqueue another run. Reusing the key with a different source manifest returns `409 Conflict`. If another request with the same key is still being staged before its manifest is known, the server returns a retryable `409 Conflict`. Idempotency records are memory-only and expire with completed upload status records.

The upload server accepts `application/x-tar`, `application/gzip`, and `application/x-gzip`. Archives are extracted into a temporary staging directory, must contain exactly one root-level `manifest.json`, and must validate as one complete source bundle before a run id is issued. Per-source `max_upload_size` bounds both uploaded archive size and extracted bundle size. The implementation also caps extracted file count.

The default bind address is private loopback. Put TLS, public routing, rate limiting, and external access policy in a reverse proxy or deployment layer.

## Remote Backend Notes

### SSH/SFTP

SSH execution uses native SFTP. See [SSH/SFTP Integration](integrations/ssh-sftp.md). It does not shell out to `ssh`, `scp`, or `rsync`.

Configure `ssh_key_file`, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported.

The default host key policy is `accept-new`. During dry runs, unknown host keys may be accepted for the current connection but are not persisted. Changed host keys are fatal for `strict` and `accept-new`. `host_key_policy: off` disables host key checking and should be limited to controlled test environments.

### S3-Compatible Storage

S3 execution uses the AWS SDK for Go v2. See [S3-Compatible Storage Integration](integrations/s3.md). Configure an endpoint, bucket, optional prefix, optional region, optional path-style setting, and optional explicit credential variable names.

When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is.

Normal single-owner replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Shared-root replacement deletes only current-owner omitted output objects and rewrites the shared state object. Same-source merge publication retains omitted managed objects and may overwrite existing managed objects. Takeover replacement does not retain omitted outputs through merge reconciliation. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.

## Secrets Operation

`secrets.directory` is loaded during `run`, `serve`, and configured-source `validate` or `inspect` before credential-consuming work starts. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts.

Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, commands emit a warning naming the ignored secret variable without printing either value. The process environment is not modified.

## Cleanup And Recovery

Use these recovery boundaries:

- For source validation failures, regenerate the source bundle and manifest together.
- For an empty or missing destination, rerun after fixing config or storage access.
- For unmanaged destination content, move unrelated files aside or use a different destination path before publishing.
- For shared-root ownership conflicts, change one owner so it writes a different destination path, use a separate destination root, or configure `takeover.mode` when the current owner should take over valid managed output paths.
- For missing managed output files recorded in state, run `reconcile-state --dry-run`, then apply `reconcile-state` if the missing files should no longer be considered managed.
- For configured retention cleanup, run `prune --dry-run`, then apply `prune --apply` after reviewing the managed output list.
- For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun `--dry-run`. In merge mode, retained outputs may be intentional managed outputs from the prior state.
- For state conflicts, verify the source, pipeline, destination, and existing `.distributor.json` before considering `--force`.
- For HTTP upload failures, inspect `/runs/<run-id>` while retained; after expiry or restart, rely on destination state and logs/output from the publishing run.

Do not edit `.distributor.json` during normal recovery. Treat it as the managed state record used for comparison and safe cleanup.

For symptom-specific fixes, see [Troubleshooting](troubleshooting.md).