eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Refresh README and CLI documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-04 12:04:42 +00:00
parent f3fb51ce7b
commit b19128b77e
2 changed files with 143 additions and 156 deletions
Download Patch File Download Diff File Expand all files Collapse all files

280
docs/cli.md
View File

@@ -1,17 +1,24 @@
# Distributor CLI
## Shortest useful command
Audience: operators, integrators, and developers who run `distributor` from a shell or automation.
This document is the canonical command and flag reference. Configuration schema details live in [Configuration](config.md), operational recovery guidance lives in [Operations](operations.md), and failure diagnosis lives in [Troubleshooting](troubleshooting.md).
## Shortest Useful Command
Run the maintained local publishing example from the repository root:
```sh
go run ./cmd/distributor run --config examples/local-publish.yml
```
This discovers the example source bundle and publishes source files to `workspace/published/source-bundle`.
The example reads `examples/source-bundle/manifest.json`, publishes the configured files into `workspace/published/source-bundle`, and writes destination state metadata beside the published output.
## Command overview
## Command Overview
```sh
```text
distributor [--help]
distributor help
distributor version [--format text|json]
distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
distributor serve [--config <path>]
@@ -19,193 +26,176 @@ distributor validate [--format text|json] <path>
distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
distributor inspect [--format text|json] <path>
distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
distributor manifest
distributor manifest create <bundle-path> --id <bundle-id> [options]
distributor manifest create --id <bundle-id> [options] <bundle-path>
```
- `version`: prints the application name and version. Development builds print `distributor dev`.
- `run`: loads a YAML config, discovers source bundles, plans each configured destination, writes selected outputs unless `--dry-run` is set, and prints a final status summary.
- `serve`: loads a YAML config, resolves HTTP upload bearer tokens, and runs the HTTP upload API.
- `validate`: validates a local source bundle directory, a local source bundle tree, or one configured pipeline source.
- `inspect`: validates source bundles and prints normalized bundle metadata for a local path or one configured pipeline source.
- `manifest create`: creates `manifest.json` for a local source bundle directory.
- `version` prints the application name and version.
- `run` executes configured pipelines against their destinations.
- `serve` starts the authenticated HTTP upload API defined by the configuration file.
- `validate` checks a local bundle path or a configured source bundle.
- `inspect` reports manifest and file metadata for a local bundle path or a configured source bundle.
- `manifest create` writes a `manifest.json` file for an existing bundle directory.
`validate` and `inspect` have two mutually exclusive modes: a local path shortcut, or configured source mode with `--config <path> --pipeline <id>`. Configured source mode opens only the selected pipeline source and supports configured `local`, `ssh`, and `s3` sources. It does not open destinations. `run` executes configured `local`, `ssh`, and `s3` sources and destinations. `serve` executes configured `http_upload` sources through the upload API and normal destination fan-out.
## Flag Reference
## Flag reference
### Help
Root command:
`distributor`, `distributor --help`, `distributor -h`, `distributor help`, and `distributor manifest` print command help. Unknown commands and invalid argument combinations print usage guidance and exit non-zero.
- `--help`, `-h`, or `help`: print root help.
### Common Output Format
All subcommands:
`--format text|json` is supported by `version`, `run`, `validate`, `inspect`, and `manifest create`.
- `--help`, `-h`: print command-specific help.
- `text` is the default human-readable output.
- `json` emits one JSON document for successful command execution.
- Invalid formats are rejected before command execution.
Output-producing subcommands:
### `version`
- `--format text|json`: output format. `text` is the default. Help and usage output are always text.
```sh
distributor version [--format text|json]
```
`run` flags:
`version` accepts no positional arguments. Text output prints the application name and version; JSON output includes `application` and `version` fields.
- `--config <path>`: config file to load. If omitted, `run` uses `/usr/local/etc/distributor/config.yml`.
- `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write output files, destination state, or SSH `known_hosts` entries.
- `--force`: allow explicit destructive replacement for supported conflict cases in this run only.
### `run`
`serve` flags:
```sh
distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
```
- `--config <path>`: config file to load. If omitted, `serve` uses `/usr/local/etc/distributor/config.yml`.
- `--config <path>` loads the pipeline configuration. If omitted, the application uses `/usr/local/etc/distributor/config.yml`.
- `--dry-run` validates inputs and reports destination actions without applying changes.
- `--force` permits a run when destination state indicates a conservative safety check would otherwise block it.
- `--format text|json` selects human-readable or machine-readable output.
`validate` and `inspect` configured source flags:
`run` accepts no positional arguments.
- `--config <path>`: config file to load for source validation or inspection. Required in configured source mode.
- `--pipeline <id>`: pipeline source to validate or inspect. Required in configured source mode.
- `--bundle <path>`: source-root-relative bundle directory to validate or inspect instead of discovering every bundle under the source root.
### `serve`
`manifest create` flags:
```sh
distributor serve [--config <path>]
```
- `--id <bundle-id>`: source bundle id. Required.
- `--file <path>`: bundle-relative file to include. Repeatable. If omitted, files are scanned recursively.
- `--created <time>`: RFC3339 source created timestamp. If omitted, the current UTC time is used.
- `--overwrite`: replace an existing `manifest.json`.
- `--config <path>` loads HTTP, source, destination, and pipeline configuration. If omitted, the application uses `/usr/local/etc/distributor/config.yml`.
`run` does not accept positional arguments. `validate` and `inspect` accept at most one path in local mode. Local paths cannot be combined with `--config`, `--pipeline`, or `--bundle`.
`serve` accepts no positional arguments and runs until interrupted or until the server exits with an error.
## Common workflows
### `validate`
Validate a source bundle:
```sh
distributor validate [--format text|json] <path>
distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
```
`validate` has two source modes:
- Local path mode validates the bundle at `<path>`.
- Configured source mode resolves the source from `--config <path>` and `--pipeline <id>`.
Configured source flags:
- `--config <path>` loads the configuration file.
- `--pipeline <id>` selects the configured pipeline source to validate.
- `--bundle <path>` overrides the configured source bundle path for the selected pipeline.
- `--format text|json` selects output format.
A local positional path cannot be combined with `--config`, `--pipeline`, or `--bundle`. When any configured source flag is used, both `--config` and `--pipeline` are required.
### `inspect`
```sh
distributor inspect [--format text|json] <path>
distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
```
`inspect` uses the same source mode rules as `validate`, then reports bundle metadata instead of only validation status. Local path mode requires exactly one bundle path. Configured source mode requires both `--config` and `--pipeline`; `--bundle` may override the selected pipeline source path.
### `manifest create`
```sh
distributor manifest create <bundle-path> --id <bundle-id> [options]
distributor manifest create --id <bundle-id> [options] <bundle-path>
```
Flags may appear before or after the bundle path. Both `--flag value` and `--flag=value` forms are accepted.
- `--id <bundle-id>` sets the manifest bundle identifier and is required.
- `--created <timestamp>` sets the manifest creation timestamp. If omitted, the current UTC time is used.
- `--file <relative-path>` includes one file in the manifest. The flag may be repeated.
- `--overwrite` allows replacing an existing `manifest.json` file.
- `--format text|json` selects output format.
If no `--file` flags are provided, `manifest create` scans the bundle directory recursively. The command requires exactly one bundle path, refuses unsafe manifest paths, and writes `manifest.json` at the bundle root.
## Common Workflows
### Validate Or Inspect A Local Bundle
```sh
go run ./cmd/distributor validate examples/source-bundle
go run ./cmd/distributor inspect --format json examples/source-bundle
```
Inspect a source bundle:
```sh
go run ./cmd/distributor inspect examples/source-bundle
```
Validate a configured source without opening destinations:
### Validate Or Inspect A Configured Source
```sh
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
go run ./cmd/distributor inspect --config examples/local-publish.yml --pipeline example-source-bundle --format json
```
Inspect one configured source bundle:
Use `--bundle <path>` with configured source mode when automation needs to validate or inspect an alternate bundle path through the selected pipeline configuration.
### Create A Manifest
```sh
go run ./cmd/distributor inspect \
--config <config-path> \
--pipeline <pipeline-id> \
--bundle daily/2026-06-01
go run ./cmd/distributor manifest create examples/source-bundle --id example-source-bundle --overwrite
go run ./cmd/distributor manifest create --id example-source-bundle --overwrite examples/source-bundle
```
Create a manifest for a local producer bundle:
Use repeated `--file` flags when the manifest should include an explicit file list instead of the recursive directory scan:
```sh
go run ./cmd/distributor manifest create <bundle-path> --id <bundle-id>
```
Create a manifest with explicit file order:
```sh
go run ./cmd/distributor manifest create <bundle-path> \
--id <bundle-id> \
--created 2026-06-01T11:00:00Z \
go run ./cmd/distributor manifest create examples/source-bundle \
--id example-source-bundle \
--file report.md \
--file summary.txt
--file summary.txt \
--overwrite
```
Preview local publication without writing:
### Preview Or Publish A Pipeline
```sh
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
```
Publish the local source example:
```sh
go run ./cmd/distributor run --config examples/local-publish.yml
```
Publish the local HTML example:
```sh
go run ./cmd/distributor run --config examples/local-html.yml
```
Start the HTTP upload API:
```sh
DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN=<token> \
go run ./cmd/distributor serve --config examples/http-upload-local.yml
```
Upload an archive to the configured `http_upload` pipeline associated with a bearer token:
```sh
curl -X POST http://127.0.0.1:8080/upload \
-H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \
-H "Content-Type: application/gzip" \
--data-binary @bundle.tar.gz
```
The upload response is returned after the archive is staged and validated; the
destination fan-out continues asynchronously:
```json
{"run_id":"reports.20260603T120000Z.abcdef12","status":"accepted"}
```
Check upload status:
```sh
curl http://127.0.0.1:8080/runs/<run-id>
```
Check server readiness:
```sh
curl http://127.0.0.1:8080/healthz
```
Preview local fan-out publication:
Use `--format json` when automation needs structured run results. Use `--force` only when the operator has reviewed the destination state conflict and intentionally wants to continue.
### Run HTML And Fan-Out Examples
```sh
go run ./cmd/distributor run --config examples/local-html.yml --dry-run
go run ./cmd/distributor run --config examples/local-index.yml --dry-run
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
```
Preview local archive-plus-latest publication:
```sh
go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
```
Preview a forced replacement before publishing:
These examples exercise implemented output rendering and destination planning behavior. They still use the same `run` flags and output contract described here.
### Start The HTTP Upload Server
```sh
go run ./cmd/distributor run --config <config-path> --dry-run --force
go run ./cmd/distributor serve --config examples/http-upload-local.yml
```
## Output
The server exposes health, status, and authenticated upload endpoints according to the loaded configuration. Use [Operations](operations.md) for server operation and recovery guidance.
Text output is the default and is intended for humans.
## Output And Exit Behavior
`run` text output prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Destination action lines include the source bundle path, destination id, destination backend, action, outputs, and reason. Fixed path destinations also print `path_mapping=fixed target=.` to show that the selected bundle targets the destination backend root. Actions include:
- `publish_new`: destination has no managed state and is empty.
- `replace_older`: destination state is older than the source manifest.
- `force_replace`: `--force` requested a supported destructive replacement.
- `skip_same`: destination state already matches the source manifest.
- `skip_destination_newer`: destination state is newer than the source manifest.
- `error`: planning or execution failed for that destination.
The command exits non-zero if any destination fails. Independent later destinations are still attempted.
Dry-run output for fixed path destinations prints a warning with the candidate count and selected source bundle. If a fixed path dry run plans a destructive replacement, it prints an additional warning that the destination root would be replaced.
The final status line includes counters for `publish_new`, `replace_older`, `force_replace`, skipped destinations, failures, whether the run was a dry run, and fixed path destinations.
JSON output writes exactly one JSON document to stdout:
Text output is optimized for direct operator use. JSON output is optimized for automation and uses a command-specific result object with a shared envelope similar to:
```json
{
@@ -217,20 +207,16 @@ JSON output writes exactly one JSON document to stdout:
}
```
Warnings are objects in the top-level `warnings` array and are not printed again as text. Fatal setup errors, such as a missing config file or invalid arguments, write no JSON document and return a non-zero exit code with a text error on stderr.
- Successful JSON commands emit one JSON document on stdout.
- Usage errors and fatal setup errors exit non-zero and do not emit a JSON result document.
- `run --format json` emits a JSON result for partial destination failures, sets `ok` to `false`, includes result details and errors, and exits non-zero.
- Warnings are included in JSON output and are printed in text output when relevant.
`run --format json` returns partial results when destination failures occur after planning or execution begins. In that case stdout contains `ok: false`, a `result` with pipeline summaries, destination actions, final counters, and a top-level `errors` array; the process still exits non-zero.
## Diagnostics And Recovery
Command-specific JSON results:
- `version`: application name and version.
- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend.
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend.
- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records.
- `run`: dry-run status, pipeline summaries, destination action records, destination bundle paths, path mapping markers, optional primary URLs, output records with optional URLs, final counters, warnings, and partial failure records.
## Diagnostics
Use `manifest create` when a local producer has written bundle files but not `manifest.json`. Use `validate` before publication when a producer has written a new bundle; use configured source mode when the bundle is already on an SSH or S3 source. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
For symptom-oriented recovery steps, see [troubleshooting](troubleshooting.md). For destination state and retry behavior, see [operations](operations.md). For config fields and defaults, see [configuration](config.md).
- Use `validate` before `run` when checking a bundle supplied by another process.
- Use `inspect --format json` when automation needs manifest metadata, normalized file details, or checksum information.
- Use `run --dry-run` before publishing to review destination actions.
- Use [Configuration](config.md) for schema and default details.
- Use [Troubleshooting](troubleshooting.md) for common errors and corrective action.
- Use [Operations](operations.md) for HTTP upload operation, state files, and recovery workflows.