eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Refresh README and CLI documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-04 12:04:42 +00:00
parent f3fb51ce7b
commit b19128b77e
2 changed files with 143 additions and 156 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
README.md
View File

@@ -1,18 +1,19 @@
# distributor # distributor
`distributor` validates manifested report bundles and publishes selected source or generated artifacts to configured destinations. `distributor` validates manifested report bundles, plans destination updates, and publishes selected source files or generated HTML outputs to configured destinations.
It is a local-first CLI with SSH/SFTP, S3-compatible storage, and HTTP upload It is a local-first Go CLI for report distribution. A pipeline reads one source bundle tree, validates `manifest.json`, fans out to one or more local, SSH/SFTP, or S3-compatible destinations, records destination state in `.distributor.json`, and can also accept authenticated tar or tar.gz uploads through the HTTP upload server.
support: source bundles can be read from local or remote storage, pushed to the
upload API, published to local directories or remote paths, and rendered from
Markdown to HTML sidecars or `index.html`.
Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate complete local source bundles with the same manifest contract used by `distributor`. Run the maintained local example:
Run the local example pipeline:
```sh ```sh
go run ./cmd/distributor run --config examples/local-publish.yml go run ./cmd/distributor run --config examples/local-publish.yml
``` ```
See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), [docs/operations.md](docs/operations.md), and [docs/troubleshooting.md](docs/troubleshooting.md) for the implemented CLI, configuration, operating notes, and common failure modes. Future and deferred work lives under `docs/roadmap/`. Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate local source bundles with the same manifest contract used by the CLI.
- [CLI reference](docs/cli.md)
- [Configuration reference](docs/config.md)
- [Operations guide](docs/operations.md)
- [Troubleshooting](docs/troubleshooting.md)
- [Development architecture](docs/policy/architecture.md)

280
docs/cli.md
View File

@@ -1,17 +1,24 @@
# Distributor CLI # Distributor CLI
## Shortest useful command Audience: operators, integrators, and developers who run `distributor` from a shell or automation.
This document is the canonical command and flag reference. Configuration schema details live in [Configuration](config.md), operational recovery guidance lives in [Operations](operations.md), and failure diagnosis lives in [Troubleshooting](troubleshooting.md).
## Shortest Useful Command
Run the maintained local publishing example from the repository root:
```sh ```sh
go run ./cmd/distributor run --config examples/local-publish.yml go run ./cmd/distributor run --config examples/local-publish.yml
``` ```
This discovers the example source bundle and publishes source files to `workspace/published/source-bundle`. The example reads `examples/source-bundle/manifest.json`, publishes the configured files into `workspace/published/source-bundle`, and writes destination state metadata beside the published output.
## Command overview ## Command Overview
```sh ```text
distributor [--help] distributor [--help]
distributor help
distributor version [--format text|json] distributor version [--format text|json]
distributor run [--config <path>] [--dry-run] [--force] [--format text|json] distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
distributor serve [--config <path>] distributor serve [--config <path>]
@@ -19,193 +26,176 @@ distributor validate [--format text|json] <path>
distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json] distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
distributor inspect [--format text|json] <path> distributor inspect [--format text|json] <path>
distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json] distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
distributor manifest
distributor manifest create <bundle-path> --id <bundle-id> [options] distributor manifest create <bundle-path> --id <bundle-id> [options]
distributor manifest create --id <bundle-id> [options] <bundle-path>
``` ```
- `version`: prints the application name and version. Development builds print `distributor dev`. - `version` prints the application name and version.
- `run`: loads a YAML config, discovers source bundles, plans each configured destination, writes selected outputs unless `--dry-run` is set, and prints a final status summary. - `run` executes configured pipelines against their destinations.
- `serve`: loads a YAML config, resolves HTTP upload bearer tokens, and runs the HTTP upload API. - `serve` starts the authenticated HTTP upload API defined by the configuration file.
- `validate`: validates a local source bundle directory, a local source bundle tree, or one configured pipeline source. - `validate` checks a local bundle path or a configured source bundle.
- `inspect`: validates source bundles and prints normalized bundle metadata for a local path or one configured pipeline source. - `inspect` reports manifest and file metadata for a local bundle path or a configured source bundle.
- `manifest create`: creates `manifest.json` for a local source bundle directory. - `manifest create` writes a `manifest.json` file for an existing bundle directory.
`validate` and `inspect` have two mutually exclusive modes: a local path shortcut, or configured source mode with `--config <path> --pipeline <id>`. Configured source mode opens only the selected pipeline source and supports configured `local`, `ssh`, and `s3` sources. It does not open destinations. `run` executes configured `local`, `ssh`, and `s3` sources and destinations. `serve` executes configured `http_upload` sources through the upload API and normal destination fan-out. ## Flag Reference
## Flag reference ### Help
Root command: `distributor`, `distributor --help`, `distributor -h`, `distributor help`, and `distributor manifest` print command help. Unknown commands and invalid argument combinations print usage guidance and exit non-zero.
- `--help`, `-h`, or `help`: print root help. ### Common Output Format
All subcommands: `--format text|json` is supported by `version`, `run`, `validate`, `inspect`, and `manifest create`.
- `--help`, `-h`: print command-specific help. - `text` is the default human-readable output.
- `json` emits one JSON document for successful command execution.
- Invalid formats are rejected before command execution.
Output-producing subcommands: ### `version`
- `--format text|json`: output format. `text` is the default. Help and usage output are always text. ```sh
distributor version [--format text|json]
```
`run` flags: `version` accepts no positional arguments. Text output prints the application name and version; JSON output includes `application` and `version` fields.
- `--config <path>`: config file to load. If omitted, `run` uses `/usr/local/etc/distributor/config.yml`. ### `run`
- `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write output files, destination state, or SSH `known_hosts` entries.
- `--force`: allow explicit destructive replacement for supported conflict cases in this run only.
`serve` flags: ```sh
distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
```
- `--config <path>`: config file to load. If omitted, `serve` uses `/usr/local/etc/distributor/config.yml`. - `--config <path>` loads the pipeline configuration. If omitted, the application uses `/usr/local/etc/distributor/config.yml`.
- `--dry-run` validates inputs and reports destination actions without applying changes.
- `--force` permits a run when destination state indicates a conservative safety check would otherwise block it.
- `--format text|json` selects human-readable or machine-readable output.
`validate` and `inspect` configured source flags: `run` accepts no positional arguments.
- `--config <path>`: config file to load for source validation or inspection. Required in configured source mode. ### `serve`
- `--pipeline <id>`: pipeline source to validate or inspect. Required in configured source mode.
- `--bundle <path>`: source-root-relative bundle directory to validate or inspect instead of discovering every bundle under the source root.
`manifest create` flags: ```sh
distributor serve [--config <path>]
```
- `--id <bundle-id>`: source bundle id. Required. - `--config <path>` loads HTTP, source, destination, and pipeline configuration. If omitted, the application uses `/usr/local/etc/distributor/config.yml`.
- `--file <path>`: bundle-relative file to include. Repeatable. If omitted, files are scanned recursively.
- `--created <time>`: RFC3339 source created timestamp. If omitted, the current UTC time is used.
- `--overwrite`: replace an existing `manifest.json`.
`run` does not accept positional arguments. `validate` and `inspect` accept at most one path in local mode. Local paths cannot be combined with `--config`, `--pipeline`, or `--bundle`. `serve` accepts no positional arguments and runs until interrupted or until the server exits with an error.
## Common workflows ### `validate`
Validate a source bundle: ```sh
distributor validate [--format text|json] <path>
distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
```
`validate` has two source modes:
- Local path mode validates the bundle at `<path>`.
- Configured source mode resolves the source from `--config <path>` and `--pipeline <id>`.
Configured source flags:
- `--config <path>` loads the configuration file.
- `--pipeline <id>` selects the configured pipeline source to validate.
- `--bundle <path>` overrides the configured source bundle path for the selected pipeline.
- `--format text|json` selects output format.
A local positional path cannot be combined with `--config`, `--pipeline`, or `--bundle`. When any configured source flag is used, both `--config` and `--pipeline` are required.
### `inspect`
```sh
distributor inspect [--format text|json] <path>
distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
```
`inspect` uses the same source mode rules as `validate`, then reports bundle metadata instead of only validation status. Local path mode requires exactly one bundle path. Configured source mode requires both `--config` and `--pipeline`; `--bundle` may override the selected pipeline source path.
### `manifest create`
```sh
distributor manifest create <bundle-path> --id <bundle-id> [options]
distributor manifest create --id <bundle-id> [options] <bundle-path>
```
Flags may appear before or after the bundle path. Both `--flag value` and `--flag=value` forms are accepted.
- `--id <bundle-id>` sets the manifest bundle identifier and is required.
- `--created <timestamp>` sets the manifest creation timestamp. If omitted, the current UTC time is used.
- `--file <relative-path>` includes one file in the manifest. The flag may be repeated.
- `--overwrite` allows replacing an existing `manifest.json` file.
- `--format text|json` selects output format.
If no `--file` flags are provided, `manifest create` scans the bundle directory recursively. The command requires exactly one bundle path, refuses unsafe manifest paths, and writes `manifest.json` at the bundle root.
## Common Workflows
### Validate Or Inspect A Local Bundle
```sh ```sh
go run ./cmd/distributor validate examples/source-bundle go run ./cmd/distributor validate examples/source-bundle
go run ./cmd/distributor inspect --format json examples/source-bundle
``` ```
Inspect a source bundle: ### Validate Or Inspect A Configured Source
```sh
go run ./cmd/distributor inspect examples/source-bundle
```
Validate a configured source without opening destinations:
```sh ```sh
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
go run ./cmd/distributor inspect --config examples/local-publish.yml --pipeline example-source-bundle --format json
``` ```
Inspect one configured source bundle: Use `--bundle <path>` with configured source mode when automation needs to validate or inspect an alternate bundle path through the selected pipeline configuration.
### Create A Manifest
```sh ```sh
go run ./cmd/distributor inspect \ go run ./cmd/distributor manifest create examples/source-bundle --id example-source-bundle --overwrite
--config <config-path> \ go run ./cmd/distributor manifest create --id example-source-bundle --overwrite examples/source-bundle
--pipeline <pipeline-id> \
--bundle daily/2026-06-01
``` ```
Create a manifest for a local producer bundle: Use repeated `--file` flags when the manifest should include an explicit file list instead of the recursive directory scan:
```sh ```sh
go run ./cmd/distributor manifest create <bundle-path> --id <bundle-id> go run ./cmd/distributor manifest create examples/source-bundle \
``` --id example-source-bundle \
Create a manifest with explicit file order:
```sh
go run ./cmd/distributor manifest create <bundle-path> \
--id <bundle-id> \
--created 2026-06-01T11:00:00Z \
--file report.md \ --file report.md \
--file summary.txt --file summary.txt \
--overwrite
``` ```
Preview local publication without writing: ### Preview Or Publish A Pipeline
```sh ```sh
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
```
Publish the local source example:
```sh
go run ./cmd/distributor run --config examples/local-publish.yml go run ./cmd/distributor run --config examples/local-publish.yml
``` ```
Publish the local HTML example: Use `--format json` when automation needs structured run results. Use `--force` only when the operator has reviewed the destination state conflict and intentionally wants to continue.
```sh ### Run HTML And Fan-Out Examples
go run ./cmd/distributor run --config examples/local-html.yml
```
Start the HTTP upload API:
```sh
DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN=<token> \
go run ./cmd/distributor serve --config examples/http-upload-local.yml
```
Upload an archive to the configured `http_upload` pipeline associated with a bearer token:
```sh
curl -X POST http://127.0.0.1:8080/upload \
-H "Authorization: Bearer $DISTRIBUTOR_EXAMPLE_UPLOAD_TOKEN" \
-H "Content-Type: application/gzip" \
--data-binary @bundle.tar.gz
```
The upload response is returned after the archive is staged and validated; the
destination fan-out continues asynchronously:
```json
{"run_id":"reports.20260603T120000Z.abcdef12","status":"accepted"}
```
Check upload status:
```sh
curl http://127.0.0.1:8080/runs/<run-id>
```
Check server readiness:
```sh
curl http://127.0.0.1:8080/healthz
```
Preview local fan-out publication:
```sh ```sh
go run ./cmd/distributor run --config examples/local-html.yml --dry-run
go run ./cmd/distributor run --config examples/local-index.yml --dry-run
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
```
Preview local archive-plus-latest publication:
```sh
go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
``` ```
Preview a forced replacement before publishing: These examples exercise implemented output rendering and destination planning behavior. They still use the same `run` flags and output contract described here.
### Start The HTTP Upload Server
```sh ```sh
go run ./cmd/distributor run --config <config-path> --dry-run --force go run ./cmd/distributor serve --config examples/http-upload-local.yml
``` ```
## Output The server exposes health, status, and authenticated upload endpoints according to the loaded configuration. Use [Operations](operations.md) for server operation and recovery guidance.
Text output is the default and is intended for humans. ## Output And Exit Behavior
`run` text output prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Destination action lines include the source bundle path, destination id, destination backend, action, outputs, and reason. Fixed path destinations also print `path_mapping=fixed target=.` to show that the selected bundle targets the destination backend root. Actions include: Text output is optimized for direct operator use. JSON output is optimized for automation and uses a command-specific result object with a shared envelope similar to:
- `publish_new`: destination has no managed state and is empty.
- `replace_older`: destination state is older than the source manifest.
- `force_replace`: `--force` requested a supported destructive replacement.
- `skip_same`: destination state already matches the source manifest.
- `skip_destination_newer`: destination state is newer than the source manifest.
- `error`: planning or execution failed for that destination.
The command exits non-zero if any destination fails. Independent later destinations are still attempted.
Dry-run output for fixed path destinations prints a warning with the candidate count and selected source bundle. If a fixed path dry run plans a destructive replacement, it prints an additional warning that the destination root would be replaced.
The final status line includes counters for `publish_new`, `replace_older`, `force_replace`, skipped destinations, failures, whether the run was a dry run, and fixed path destinations.
JSON output writes exactly one JSON document to stdout:
```json ```json
{ {
@@ -217,20 +207,16 @@ JSON output writes exactly one JSON document to stdout:
} }
``` ```
Warnings are objects in the top-level `warnings` array and are not printed again as text. Fatal setup errors, such as a missing config file or invalid arguments, write no JSON document and return a non-zero exit code with a text error on stderr. - Successful JSON commands emit one JSON document on stdout.
- Usage errors and fatal setup errors exit non-zero and do not emit a JSON result document.
- `run --format json` emits a JSON result for partial destination failures, sets `ok` to `false`, includes result details and errors, and exits non-zero.
- Warnings are included in JSON output and are printed in text output when relevant.
`run --format json` returns partial results when destination failures occur after planning or execution begins. In that case stdout contains `ok: false`, a `result` with pipeline summaries, destination actions, final counters, and a top-level `errors` array; the process still exits non-zero. ## Diagnostics And Recovery
Command-specific JSON results: - Use `validate` before `run` when checking a bundle supplied by another process.
- Use `inspect --format json` when automation needs manifest metadata, normalized file details, or checksum information.
- `version`: application name and version. - Use `run --dry-run` before publishing to review destination actions.
- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend. - Use [Configuration](config.md) for schema and default details.
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend. - Use [Troubleshooting](troubleshooting.md) for common errors and corrective action.
- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records. - Use [Operations](operations.md) for HTTP upload operation, state files, and recovery workflows.
- `run`: dry-run status, pipeline summaries, destination action records, destination bundle paths, path mapping markers, optional primary URLs, output records with optional URLs, final counters, warnings, and partial failure records.
## Diagnostics
Use `manifest create` when a local producer has written bundle files but not `manifest.json`. Use `validate` before publication when a producer has written a new bundle; use configured source mode when the bundle is already on an SSH or S3 source. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
For symptom-oriented recovery steps, see [troubleshooting](troubleshooting.md). For destination state and retry behavior, see [operations](operations.md). For config fields and defaults, see [configuration](config.md).