From b19128b77ecd3874b1591c20446f1954b0062a4b Mon Sep 17 00:00:00 2001
From: Eric Rakestraw <eric@rakestraw.us>
Date: Thu, 4 Jun 2026 12:04:42 +0000
Subject: [PATCH] Refresh README and CLI documentation

---
 README.md   |  19 ++--
 docs/cli.md | 280 +++++++++++++++++++++++++---------------------------
 2 files changed, 143 insertions(+), 156 deletions(-)

diff --git a/README.md b/README.md
index 397c8c6..15584c2 100644
--- a/README.md
+++ b/README.md
@@ -1,18 +1,19 @@
 # distributor
 
-`distributor` validates manifested report bundles and publishes selected source or generated artifacts to configured destinations.
+`distributor` validates manifested report bundles, plans destination updates, and publishes selected source files or generated HTML outputs to configured destinations.
 
-It is a local-first CLI with SSH/SFTP, S3-compatible storage, and HTTP upload
-support: source bundles can be read from local or remote storage, pushed to the
-upload API, published to local directories or remote paths, and rendered from
-Markdown to HTML sidecars or `index.html`.
+It is a local-first Go CLI for report distribution. A pipeline reads one source bundle tree, validates `manifest.json`, fans out to one or more local, SSH/SFTP, or S3-compatible destinations, records destination state in `.distributor.json`, and can also accept authenticated tar or tar.gz uploads through the HTTP upload server.
 
-Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate complete local source bundles with the same manifest contract used by `distributor`.
-
-Run the local example pipeline:
+Run the maintained local example:
 
 ```sh
 go run ./cmd/distributor run --config examples/local-publish.yml
 ```
 
-See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), [docs/operations.md](docs/operations.md), and [docs/troubleshooting.md](docs/troubleshooting.md) for the implemented CLI, configuration, operating notes, and common failure modes. Future and deferred work lives under `docs/roadmap/`.
+Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate local source bundles with the same manifest contract used by the CLI.
+
+- [CLI reference](docs/cli.md)
+- [Configuration reference](docs/config.md)
+- [Operations guide](docs/operations.md)
+- [Troubleshooting](docs/troubleshooting.md)
+- [Development architecture](docs/policy/architecture.md)
diff --git a/docs/cli.md b/docs/cli.md
index 504fbbf..f22b056 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -1,17 +1,24 @@
 # Distributor CLI
 
-## Shortest useful command
+Audience: operators, integrators, and developers who run `distributor` from a shell or automation.
+
+This document is the canonical command and flag reference. Configuration schema details live in [Configuration](config.md), operational recovery guidance lives in [Operations](operations.md), and failure diagnosis lives in [Troubleshooting](troubleshooting.md).
+
+## Shortest Useful Command
+
+Run the maintained local publishing example from the repository root:
 
 ```sh
 go run ./cmd/distributor run --config examples/local-publish.yml
 ```
 
-This discovers the example source bundle and publishes source files to `workspace/published/source-bundle`.
+The example reads `examples/source-bundle/manifest.json`, publishes the configured files into `workspace/published/source-bundle`, and writes destination state metadata beside the published output.
 
-## Command overview
+## Command Overview
 
-```sh
+```text
 distributor [--help]
+distributor help
 distributor version [--format text|json]
 distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
 distributor serve [--config <path>]
@@ -19,193 +26,176 @@ distributor validate [--format text|json] <path>
 distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
 distributor inspect [--format text|json] <path>
 distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
+distributor manifest
 distributor manifest create <bundle-path> --id <bundle-id> [options]
+distributor manifest create --id <bundle-id> [options] <bundle-path>
 ```
 
-- `version`: prints the application name and version. Development builds print `distributor dev`.
-- `run`: loads a YAML config, discovers source bundles, plans each configured destination, writes selected outputs unless `--dry-run` is set, and prints a final status summary.
-- `serve`: loads a YAML config, resolves HTTP upload bearer tokens, and runs the HTTP upload API.
-- `validate`: validates a local source bundle directory, a local source bundle tree, or one configured pipeline source.
-- `inspect`: validates source bundles and prints normalized bundle metadata for a local path or one configured pipeline source.
-- `manifest create`: creates `manifest.json` for a local source bundle directory.
+- `version` prints the application name and version.
+- `run` executes configured pipelines against their destinations.
+- `serve` starts the authenticated HTTP upload API defined by the configuration file.
+- `validate` checks a local bundle path or a configured source bundle.
+- `inspect` reports manifest and file metadata for a local bundle path or a configured source bundle.
+- `manifest create` writes a `manifest.json` file for an existing bundle directory.
 
-`validate` and `inspect` have two mutually exclusive modes: a local path shortcut, or configured source mode with `--config <path> --pipeline <id>`. Configured source mode opens only the selected pipeline source and supports configured `local`, `ssh`, and `s3` sources. It does not open destinations. `run` executes configured `local`, `ssh`, and `s3` sources and destinations. `serve` executes configured `http_upload` sources through the upload API and normal destination fan-out.
+## Flag Reference
 
-## Flag reference
+### Help
 
-Root command:
+`distributor`, `distributor --help`, `distributor -h`, `distributor help`, and `distributor manifest` print command help. Unknown commands and invalid argument combinations print usage guidance and exit non-zero.
 
-- `--help`, `-h`, or `help`: print root help.
+### Common Output Format
 
-All subcommands:
+`--format text|json` is supported by `version`, `run`, `validate`, `inspect`, and `manifest create`.
 
-- `--help`, `-h`: print command-specific help.
+- `text` is the default human-readable output.
+- `json` emits one JSON document for successful command execution.
+- Invalid formats are rejected before command execution.
 
-Output-producing subcommands:
+### `version`
 
-- `--format text|json`: output format. `text` is the default. Help and usage output are always text.
+```sh
+distributor version [--format text|json]
+```
 
-`run` flags:
+`version` accepts no positional arguments. Text output prints the application name and version; JSON output includes `application` and `version` fields.
 
-- `--config <path>`: config file to load. If omitted, `run` uses `/usr/local/etc/distributor/config.yml`.
-- `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write output files, destination state, or SSH `known_hosts` entries.
-- `--force`: allow explicit destructive replacement for supported conflict cases in this run only.
+### `run`
 
-`serve` flags:
+```sh
+distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
+```
 
-- `--config <path>`: config file to load. If omitted, `serve` uses `/usr/local/etc/distributor/config.yml`.
+- `--config <path>` loads the pipeline configuration. If omitted, the application uses `/usr/local/etc/distributor/config.yml`.
+- `--dry-run` validates inputs and reports destination actions without applying changes.
+- `--force` permits a run when destination state indicates a conservative safety check would otherwise block it.
+- `--format text|json` selects human-readable or machine-readable output.
 
-`validate` and `inspect` configured source flags:
+`run` accepts no positional arguments.
 
-- `--config <path>`: config file to load for source validation or inspection. Required in configured source mode.
-- `--pipeline <id>`: pipeline source to validate or inspect. Required in configured source mode.
-- `--bundle <path>`: source-root-relative bundle directory to validate or inspect instead of discovering every bundle under the source root.
+### `serve`
 
-`manifest create` flags:
+```sh
+distributor serve [--config <path>]
+```
 
-- `--id <bundle-id>`: source bundle id. Required.
-- `--file <path>`: bundle-relative file to include. Repeatable. If omitted, files are scanned recursively.
-- `--created <time>`: RFC3339 source created timestamp. If omitted, the current UTC time is used.
-- `--overwrite`: replace an existing `manifest.json`.
+- `--config <path>` loads HTTP, source, destination, and pipeline configuration. If omitted, the application uses `/usr/local/etc/distributor/config.yml`.
 
-`run` does not accept positional arguments. `validate` and `inspect` accept at most one path in local mode. Local paths cannot be combined with `--config`, `--pipeline`, or `--bundle`.
+`serve` accepts no positional arguments and runs until interrupted or until the server exits with an error.
 
-## Common workflows
+### `validate`
 
-Validate a source bundle:
+```sh
+distributor validate [--format text|json] <path>
+distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
+```
+
+`validate` has two source modes:
+
+- Local path mode validates the bundle at `<path>`.
+- Configured source mode resolves the source from `--config <path>` and `--pipeline <id>`.
+
+Configured source flags:
+
+- `--config <path>` loads the configuration file.
+- `--pipeline <id>` selects the configured pipeline source to validate.
+- `--bundle <path>` overrides the configured source bundle path for the selected pipeline.
+- `--format text|json` selects output format.
+
+A local positional path cannot be combined with `--config`, `--pipeline`, or `--bundle`. When any configured source flag is used, both `--config` and `--pipeline` are required.
+
+### `inspect`
+
+```sh
+distributor inspect [--format text|json] <path>
+distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
+```
+
+`inspect` uses the same source mode rules as `validate`, then reports bundle metadata instead of only validation status. Local path mode requires exactly one bundle path. Configured source mode requires both `--config` and `--pipeline`; `--bundle` may override the selected pipeline source path.
+
+### `manifest create`
+
+```sh
+distributor manifest create <bundle-path> --id <bundle-id> [options]
+distributor manifest create --id <bundle-id> [options] <bundle-path>
+```
+
+Flags may appear before or after the bundle path. Both `--flag value` and `--flag=value` forms are accepted.
+
+- `--id <bundle-id>` sets the manifest bundle identifier and is required.
+- `--created <timestamp>` sets the manifest creation timestamp. If omitted, the current UTC time is used.
+- `--file <relative-path>` includes one file in the manifest. The flag may be repeated.
+- `--overwrite` allows replacing an existing `manifest.json` file.
+- `--format text|json` selects output format.
+
+If no `--file` flags are provided, `manifest create` scans the bundle directory recursively. The command requires exactly one bundle path, refuses unsafe manifest paths, and writes `manifest.json` at the bundle root.
+
+## Common Workflows
+
+### Validate Or Inspect A Local Bundle
 
 ```sh
 go run ./cmd/distributor validate examples/source-bundle
+go run ./cmd/distributor inspect --format json examples/source-bundle
 ```
 
-Inspect a source bundle:
-
-```sh
-go run ./cmd/distributor inspect examples/source-bundle
-```
-
-Validate a configured source without opening destinations:
+### Validate Or Inspect A Configured Source
 
 ```sh
 go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
+go run ./cmd/distributor inspect --config examples/local-publish.yml --pipeline example-source-bundle --format json
 ```
 
-Inspect one configured source bundle:
+Use `--bundle <path>` with configured source mode when automation needs to validate or inspect an alternate bundle path through the selected pipeline configuration.
+
+### Create A Manifest
 
 ```sh
-go run ./cmd/distributor inspect \
-  --config <config-path> \
-  --pipeline <pipeline-id> \
-  --bundle daily/2026-06-01
+go run ./cmd/distributor manifest create examples/source-bundle --id example-source-bundle --overwrite
+go run ./cmd/distributor manifest create --id example-source-bundle --overwrite examples/source-bundle
 ```
 
-Create a manifest for a local producer bundle:
+Use repeated `--file` flags when the manifest should include an explicit file list instead of the recursive directory scan:
 
 ```sh
-go run ./cmd/distributor manifest create <bundle-path> --id <bundle-id>
-```
-
-Create a manifest with explicit file order:
-
-```sh
-go run ./cmd/distributor manifest create <bundle-path> \
-  --id <bundle-id> \
-  --created 2026-06-01T11:00:00Z \
+go run ./cmd/distributor manifest create examples/source-bundle \
+  --id example-source-bundle \
   --file report.md \
-  --file summary.txt
+  --file summary.txt \
+  --overwrite
 ```
 
-Preview local publication without writing:
+### Preview Or Publish A Pipeline
 
 ```sh
 go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
-```
-
-Publish the local source example:
-
-```sh
 go run ./cmd/distributor run --config examples/local-publish.yml
 ```
 
-Publish the local HTML example:
-
-```sh
-go run ./cmd/distributor run --config examples/local-html.yml
-```
-
-Start the HTTP upload API:
-
-```sh
-DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN=<token> \
-  go run ./cmd/distributor serve --config examples/http-upload-local.yml
-```
-
-Upload an archive to the configured `http_upload` pipeline associated with a bearer token:
-
-```sh
-curl -X POST http://127.0.0.1:8080/upload \
-  -H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \
-  -H "Content-Type: application/gzip" \
-  --data-binary @bundle.tar.gz
-```
-
-The upload response is returned after the archive is staged and validated; the
-destination fan-out continues asynchronously:
-
-```json
-{"run_id":"reports.20260603T120000Z.abcdef12","status":"accepted"}
-```
-
-Check upload status:
-
-```sh
-curl http://127.0.0.1:8080/runs/<run-id>
-```
-
-Check server readiness:
-
-```sh
-curl http://127.0.0.1:8080/healthz
-```
-
-Preview local fan-out publication:
+Use `--format json` when automation needs structured run results. Use `--force` only when the operator has reviewed the destination state conflict and intentionally wants to continue.
+
+### Run HTML And Fan-Out Examples
 
 ```sh
+go run ./cmd/distributor run --config examples/local-html.yml --dry-run
+go run ./cmd/distributor run --config examples/local-index.yml --dry-run
 go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
-```
-
-Preview local archive-plus-latest publication:
-
-```sh
 go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
 ```
 
-Preview a forced replacement before publishing:
+These examples exercise implemented output rendering and destination planning behavior. They still use the same `run` flags and output contract described here.
+
+### Start The HTTP Upload Server
 
 ```sh
-go run ./cmd/distributor run --config <config-path> --dry-run --force
+go run ./cmd/distributor serve --config examples/http-upload-local.yml
 ```
 
-## Output
+The server exposes health, status, and authenticated upload endpoints according to the loaded configuration. Use [Operations](operations.md) for server operation and recovery guidance.
 
-Text output is the default and is intended for humans.
+## Output And Exit Behavior
 
-`run` text output prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Destination action lines include the source bundle path, destination id, destination backend, action, outputs, and reason. Fixed path destinations also print `path_mapping=fixed target=.` to show that the selected bundle targets the destination backend root. Actions include:
-
-- `publish_new`: destination has no managed state and is empty.
-- `replace_older`: destination state is older than the source manifest.
-- `force_replace`: `--force` requested a supported destructive replacement.
-- `skip_same`: destination state already matches the source manifest.
-- `skip_destination_newer`: destination state is newer than the source manifest.
-- `error`: planning or execution failed for that destination.
-
-The command exits non-zero if any destination fails. Independent later destinations are still attempted.
-
-Dry-run output for fixed path destinations prints a warning with the candidate count and selected source bundle. If a fixed path dry run plans a destructive replacement, it prints an additional warning that the destination root would be replaced.
-
-The final status line includes counters for `publish_new`, `replace_older`, `force_replace`, skipped destinations, failures, whether the run was a dry run, and fixed path destinations.
-
-JSON output writes exactly one JSON document to stdout:
+Text output is optimized for direct operator use. JSON output is optimized for automation and uses a command-specific result object with a shared envelope similar to:
 
 ```json
 {
@@ -217,20 +207,16 @@ JSON output writes exactly one JSON document to stdout:
 }
 ```
 
-Warnings are objects in the top-level `warnings` array and are not printed again as text. Fatal setup errors, such as a missing config file or invalid arguments, write no JSON document and return a non-zero exit code with a text error on stderr.
+- Successful JSON commands emit one JSON document on stdout.
+- Usage errors and fatal setup errors exit non-zero and do not emit a JSON result document.
+- `run --format json` emits a JSON result for partial destination failures, sets `ok` to `false`, includes result details and errors, and exits non-zero.
+- Warnings are included in JSON output and are printed in text output when relevant.
 
-`run --format json` returns partial results when destination failures occur after planning or execution begins. In that case stdout contains `ok: false`, a `result` with pipeline summaries, destination actions, final counters, and a top-level `errors` array; the process still exits non-zero.
+## Diagnostics And Recovery
 
-Command-specific JSON results:
-
-- `version`: application name and version.
-- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend.
-- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend.
-- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records.
-- `run`: dry-run status, pipeline summaries, destination action records, destination bundle paths, path mapping markers, optional primary URLs, output records with optional URLs, final counters, warnings, and partial failure records.
-
-## Diagnostics
-
-Use `manifest create` when a local producer has written bundle files but not `manifest.json`. Use `validate` before publication when a producer has written a new bundle; use configured source mode when the bundle is already on an SSH or S3 source. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
-
-For symptom-oriented recovery steps, see [troubleshooting](troubleshooting.md). For destination state and retry behavior, see [operations](operations.md). For config fields and defaults, see [configuration](config.md).
+- Use `validate` before `run` when checking a bundle supplied by another process.
+- Use `inspect --format json` when automation needs manifest metadata, normalized file details, or checksum information.
+- Use `run --dry-run` before publishing to review destination actions.
+- Use [Configuration](config.md) for schema and default details.
+- Use [Troubleshooting](troubleshooting.md) for common errors and corrective action.
+- Use [Operations](operations.md) for HTTP upload operation, state files, and recovery workflows.