distributor/docs/operations.md

# Distributor Operations

## Normal Workflow

Validate a source bundle:

```sh
go run ./cmd/distributor validate examples/source-bundle
```

Preview a local publication:

```sh
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
```

Run the local publication:

```sh
go run ./cmd/distributor run --config examples/local-publish.yml
```

Run the local HTML publication:

```sh
go run ./cmd/distributor run --config examples/local-html.yml
```

Preview local fan-out publication:

```sh
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
```

Preview a run for automation:

```sh
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run --format json
```

Preview an environment-gated SSH destination config after editing it for an SSH/SFTP endpoint you control:

```sh
go run ./cmd/distributor run --config examples/ssh-destination.yml --dry-run
```

Preview an environment-gated S3 destination config after editing it for an S3-compatible endpoint and bucket you control:

```sh
go run ./cmd/distributor run --config examples/s3-destination.yml --dry-run
```

## Filesystem Layout

Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.

Destination bundle paths preserve the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at each destination.

The maintained local examples write under `workspace/`, which is ignored by Git.

SSH backends use the configured remote `path` as the backend root. Source bundle discovery and destination bundle paths are relative to that root, using the same logical path rules as local storage.

S3 backends use the configured bucket plus optional `prefix` as the backend root. Source bundle discovery and destination bundle paths are relative to that object-key prefix. Prefixes are object-key prefixes, not real directories.

## Destination State

Each published destination bundle contains `.distributor.json`. This file is the managed sentinel and destination state record. It stores:

- pipeline and destination identity;
- publication timestamp;
- source manifest used for publication;
- copied source output metadata;
- generated output metadata.

`manifest.json` from the source bundle is not copied as destination state.

Do not edit `.distributor.json` by hand during normal operation. If it is missing or invalid while destination files remain, `distributor` treats the destination as unmanaged or conflicted.

## Go Producer Manifests

Go producer applications can import `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to create source manifests with the same path, digest, timestamp, and validation rules used by `distributor`.

Minimal producer-side manifest creation:

```go
manifest, err := bundle.BuildManifest(bundle.BuildOptions{
    Root:  outputDir,
    ID:    "reports.example.2026-05-30",
    Files: []string{"report.md", "summary.txt"},
})
if err != nil {
    return err
}
if err := bundle.WriteManifest(outputDir, manifest, bundle.WriteManifestOptions{}); err != nil {
    return err
}
```

Use explicit `Files` to preserve caller order, or `Scan: true` to recursively include regular files under `Root` in deterministic slash-path order. Scan mode includes dotfiles, excludes files named `manifest.json` or `.distributor.json`, and rejects symlinks.

## Dry Runs

`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH `known_hosts` entries.

Dry-run output is useful before publishing to confirm actions such as `publish_new`, `replace_older`, `force_replace`, `skip_same`, and `skip_destination_newer`.

Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication.

Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, output records, final counters, and partial failure records. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.

## Retry and Replacement Behavior

If a destination has matching `.distributor.json`, publication skips it as already published.

If destination state is older than the source manifest and transfer policy allows replacement, publication deletes only managed outputs recorded in `.distributor.json` plus the state file, then writes the new outputs and state.

If destination state is newer than the source manifest, the default behavior is to skip. If destination state has the same source id and created timestamp but a different digest, publication fails as a conflict.

If a destination path has files but no valid `.distributor.json`, publication fails as unmanaged content unless the current run explicitly uses `--force`.

## Force Workflow

Use `--force` only after a dry run shows the intended `force_replace` action:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run --force
go run ./cmd/distributor run --config <config-path> --force
```

Forced replacement can overwrite unmanaged non-empty destination paths. Destination state conflicts require `transfer.on_conflict: replace` plus `--force`. Newer destination state requires `transfer.on_destination_newer: replace` plus `--force`.

Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For S3 destinations, deletion is constrained to the configured bucket and prefix plus the destination bundle prefix. Force is per run only and has no config default.

## Failure Handling

If one destination fails in a fan-out run, independent later destinations are still planned and executed. The command exits non-zero after printing the final status if any destination failed.

Errors include the pipeline id, destination id, destination backend, and bundle path where applicable.

In JSON mode, destination failures after planning or execution begins are reported in the top-level `errors` array and in the run result while preserving a non-zero exit code. Fatal setup errors such as an unreadable config or invalid secrets directory write no JSON document.

If a write fails during publication, `distributor` attempts to remove outputs written during that failed attempt so a retry does not see those partial outputs as unmanaged destination content.

After a successful publish or replacement, the internal notifier hook runs. The current default notifier is a no-op. Skipped destinations do not invoke it.

## SSH Operation Notes

SSH execution uses SFTP over `golang.org/x/crypto/ssh` and `github.com/pkg/sftp`. It does not shell out to `ssh`, `scp`, or `rsync`.

Configure `ssh_key_file`, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported.

The default host key policy is `accept-new`. New host keys are written to `known_hosts` when the file path is writable. During `--dry-run`, unknown host keys may be accepted for the current connection but are not written to `known_hosts`; a later non-dry-run may persist the same key. Changed host keys are fatal for both `strict` and `accept-new`. The `off` policy disables host key checking and `run` prints a warning when stdout is enabled.

Recovery boundaries are the same as local storage: replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, and failed writes are cleaned up where practical. Distributor never performs broad recursive remote deletion.

## S3 Operation Notes

S3 execution uses the AWS SDK for Go v2. Configure `endpoint`, `bucket`, optional `prefix`, optional `region`, and optional explicit credential environment variable names.

When explicit credential env names are configured, both variables must resolve to non-empty values through the real process environment or `secrets.directory`. When they are omitted, the AWS SDK default credential chain is used as-is.

Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.

## Secrets Directory

Configure `secrets.directory` when credential values should come from mounted files, such as deployment secrets:

```yaml
secrets:
  directory: /run/secrets/distributor
```

The directory is loaded during `run` before any source or destination backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the run fails before publication work starts.

Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` prints a warning naming the ignored secret file variable without printing either value. The process environment is not changed.

## Caveats

External notification adapters are unavailable. Force overwrite behavior is available only through the explicit `run --force` workflow.

For symptom-oriented fixes, see [troubleshooting](troubleshooting.md). For config details, see [configuration](config.md). For command syntax, see [CLI](cli.md).