eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Tighten local MVP user documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-31 04:02:09 +00:00
parent c36217d0df
commit b3044c5b7b
5 changed files with 299 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
README.md
View File

@@ -1,6 +1,8 @@
# distributor # distributor
`distributor` validates and publishes manifested report bundles. `distributor` validates manifested report bundles and publishes selected source or generated artifacts to configured destinations.
It is currently a local-first CLI: source bundles are read from local storage, destinations are local directories, and Markdown files can be rendered to HTML sidecars.
Run the local example pipeline: Run the local example pipeline:
@@ -8,4 +10,4 @@ Run the local example pipeline:
go run ./cmd/distributor run --config examples/local-publish.yml go run ./cmd/distributor run --config examples/local-publish.yml
``` ```
See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), and [docs/operations.md](docs/operations.md) for the implemented CLI, configuration, and operating notes. Current design and implementation planning lives under `docs/roadmap/`. See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), [docs/operations.md](docs/operations.md), and [docs/troubleshooting.md](docs/troubleshooting.md) for the implemented CLI, configuration, operating notes, and common failure modes. Future planning lives under `docs/roadmap/`.

59
docs/cli.md
View File

@@ -6,44 +6,41 @@
go run ./cmd/distributor run --config examples/local-publish.yml go run ./cmd/distributor run --config examples/local-publish.yml
``` ```
This validates and publishes the example source bundle to `workspace/published/source-bundle`. This discovers the example source bundle and publishes source files to `workspace/published/source-bundle`.
## Command overview ## Command overview
```sh ```sh
distributor --help distributor [--help]
distributor version distributor version
distributor run distributor run [--config <path>] [--dry-run]
distributor validate distributor validate <path>
distributor inspect distributor inspect <path>
``` ```
`version` prints the application name and version. The default development version is `dev`; release builds may replace it at build time. - `version`: prints the application name and version. Development builds print `distributor dev`.
- `run`: loads a YAML config, discovers local source bundles, plans each configured destination, writes selected outputs unless `--dry-run` is set, and prints a final status summary.
- `validate`: validates a local source bundle directory or a local tree containing source bundles.
- `inspect`: validates local source bundles and prints normalized bundle metadata.
`run --config <path>` executes configured local-to-local pipelines that publish source files, generated HTML files, or both. `validate` and `inspect` accept local paths only. `run` currently executes local backends only. SSH and S3 config can be parsed and validated, but configured SSH or S3 execution fails with a clear unsupported-execution error.
`run --config <path> --dry-run` discovers source bundles, inspects destination state, and prints planned actions plus a final status summary without writing files.
`validate <path>` validates a local source bundle directory or a local tree containing source bundles.
`inspect <path>` validates discovered local source bundles and prints a concise normalized summary.
Remote backends are not implemented yet.
## Flag reference ## Flag reference
The root command supports: Root command:
- `--help`, `-h`: print root help. - `--help`, `-h`, or `help`: print root help.
Each subcommand supports: All subcommands:
- `--help`, `-h`: print command-specific help. - `--help`, `-h`: print command-specific help.
`run` supports: `run` flags:
- `--config <path>`: config file to load. - `--config <path>`: config file to load. If omitted, `run` uses `/usr/local/etc/distributor/config.yml`.
- `--dry-run`: validate config, print planned actions and final status, and do not publish. - `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write files.
`run` does not accept positional arguments. `validate` and `inspect` accept at most one path; omitting the path returns a required-path error.
## Common workflows ## Common workflows
@@ -65,7 +62,7 @@ Preview local publication without writing:
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
``` ```
Publish the local example: Publish the local source example:
```sh ```sh
go run ./cmd/distributor run --config examples/local-publish.yml go run ./cmd/distributor run --config examples/local-publish.yml
@@ -76,3 +73,21 @@ Publish the local HTML example:
```sh ```sh
go run ./cmd/distributor run --config examples/local-html.yml go run ./cmd/distributor run --config examples/local-html.yml
``` ```
## Output
`run` prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Actions include:
- `publish_new`: destination has no managed state and is empty.
- `replace_older`: destination state is older than the source manifest.
- `skip_same`: destination state already matches the source manifest.
- `skip_destination_newer`: destination state is newer than the source manifest.
- `error`: planning or execution failed for that destination.
The command exits non-zero if any destination fails. Independent later destinations are still attempted.
## Diagnostics
Use `validate` before publication when a producer has written a new bundle. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
For symptom-oriented recovery steps, see [troubleshooting](troubleshooting.md). For destination state and retry behavior, see [operations](operations.md). For config fields and defaults, see [configuration](config.md).

102
docs/config.md
View File

@@ -1,18 +1,18 @@
# Distributor Configuration # Distributor Configuration
## Config file location ## Config File Location
`distributor run --config <path>` loads the YAML config at the path provided by `--config`. `distributor run --config <path>` loads the YAML config at the provided path.
If `--config` is omitted during run, the built-in default path is: If `--config` is omitted, `run` uses:
```text ```text
/usr/local/etc/distributor/config.yml /usr/local/etc/distributor/config.yml
``` ```
The current implementation supports local-to-local publication of source files, generated HTML files, or both. Remote backends are not implemented yet. Config parsing rejects unknown YAML fields. The current executable backend support is local only. SSH and S3 config fields are accepted by config validation, but runtime execution for those backends is not implemented.
## Minimal config ## Minimal Local Config
```yaml ```yaml
pipelines: pipelines:
@@ -26,9 +26,9 @@ pipelines:
path: /srv/reports/archive path: /srv/reports/archive
``` ```
This uses the default publish policy of source files only and the default transfer policy. This publishes source files only. It uses the default validation and transfer policies.
## Production-oriented config ## Production-Oriented Local Config
```yaml ```yaml
pipelines: pipelines:
@@ -52,6 +52,22 @@ pipelines:
on_conflict: fail on_conflict: fail
``` ```
## HTML Publication
To publish generated HTML from Markdown files:
```yaml
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
Sidecar generation writes `report.html` for `report.md`. It does not mutate the source bundle.
## Reference ## Reference
Top level: Top level:
@@ -60,37 +76,79 @@ Top level:
Pipeline: Pipeline:
- `id`: required unique identifier. - `id`: required unique slug-like identifier.
- `source`: required backend config. - `source`: required backend config.
- `validation.on_digest_mismatch`: optional; defaults to `fail`; only `fail` is supported.
- `destinations`: required non-empty destination list. - `destinations`: required non-empty destination list.
- `validation.on_digest_mismatch`: optional, defaults to `fail`; only `fail` is supported.
Backends: Source backend:
- `local`: requires `path`. - `backend`: required.
- `ssh`: requires `uri` and `path`. - `path`: required for `local` and `ssh`.
- `s3`: requires `endpoint` and `bucket`; supports optional `prefix`, `region`, `force_path_style`, and `credentials`. - `uri`: required for `ssh`.
- `endpoint`: required for `s3`.
- `bucket`: required for `s3`.
- `prefix`: optional for `s3`.
- `region`: optional for `s3`.
- `force_path_style`: optional for `s3`.
- `credentials.access_key_id_env`: optional S3 credential environment variable name.
- `credentials.secret_access_key_env`: optional S3 credential environment variable name.
Destination policy: Destination:
- `id`: required unique slug-like identifier within the pipeline.
- Backend fields: same accepted shape as source backends, with destination fields at the destination level.
- `publish`: optional; defaults to source-only publication.
- `transform`: required only for generated HTML publication.
- `transfer`: optional; defaults described below.
Accepted backend names:
- `local`: executable; requires `path`.
- `ssh`: config validation only; execution is not implemented.
- `s3`: config validation only; execution is not implemented.
Publish policy:
- `publish.source`: publish source artifacts. - `publish.source`: publish source artifacts.
- `publish.html`: publish generated HTML artifacts from Markdown source files. - `publish.html`: publish generated HTML artifacts from Markdown source files.
- `transfer.on_destination_same`: `skip` or `fail`, defaults to `skip`.
- `transfer.on_destination_older`: `replace` or `fail`, defaults to `replace`.
- `transfer.on_destination_newer`: `skip` or `fail`, defaults to `skip`.
- `transfer.on_conflict`: only `fail`, defaults to `fail`.
When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `transform.markdown_to_html.mode: sidecar` are required. At least one output type must be enabled. When `publish.html` is true, `transform.markdown_to_html.enabled` must be `true` and `transform.markdown_to_html.mode` must be `sidecar`.
Markdown-to-HTML sidecar generation writes `report.html` for `report.md` and does not mutate the source bundle. Transfer policy:
- `transfer.on_destination_same`: `skip` or `fail`; defaults to `skip`.
- `transfer.on_destination_older`: `replace` or `fail`; defaults to `replace`.
- `transfer.on_destination_newer`: `skip` or `fail`; defaults to `skip`.
- `transfer.on_conflict`: only `fail`; defaults to `fail`.
## Defaults
Defaults are applied after YAML decoding and before validation:
- `validation.on_digest_mismatch: fail`
- `publish.source: true`
- `publish.html: false`
- `transfer.on_destination_same: skip`
- `transfer.on_destination_older: replace`
- `transfer.on_destination_newer: skip`
- `transfer.on_conflict: fail`
## Secrets ## Secrets
Do not put literal secrets in config files. S3 credentials may refer to environment variable names with: Do not put literal secrets in config files. S3 credentials may name environment variables:
- `credentials.access_key_id_env` - `credentials.access_key_id_env`
- `credentials.secret_access_key_env` - `credentials.secret_access_key_env`
S3 execution is not implemented yet; these fields are accepted so config shape can be validated ahead of backend implementation.
## Examples ## Examples
Maintained examples live under [examples/](../examples/). Maintained examples live under [examples](../examples/):
- `local-to-local.yml`: minimal local config.
- `local-publish.yml`: runnable local source publication.
- `local-html.yml`: runnable local HTML publication.
`examples/fan-out.yml` currently demonstrates accepted SSH/S3 config shape but is not executable until remote backend support exists.

56
docs/operations.md
View File

@@ -1,6 +1,12 @@
# Distributor Operations # Distributor Operations
## Normal workflow ## Normal Workflow
Validate a source bundle:
```sh
go run ./cmd/distributor validate examples/source-bundle
```
Preview a local publication: Preview a local publication:
@@ -20,30 +26,54 @@ Run the local HTML publication:
go run ./cmd/distributor run --config examples/local-html.yml go run ./cmd/distributor run --config examples/local-html.yml
``` ```
## Filesystem layout ## Filesystem Layout
Source bundles are discovered beneath the configured local source root. Destination bundle paths preserve the source bundle path relative to that source root. Source bundles are discovered beneath the configured local source root. Each bundle is a directory containing `manifest.json`.
The maintained example writes under `workspace/`, which is ignored by Git. Destination bundle paths preserve the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at each destination.
## Destination state The maintained examples write under `workspace/`, which is ignored by Git.
Each published destination bundle contains `.distributor.json`. This state file records the source manifest, copied source outputs, and generated outputs. It is the authoritative marker that a destination path is managed by `distributor`. ## Destination State
Each published destination bundle contains `.distributor.json`. This file is the managed sentinel and destination state record. It stores:
- pipeline and destination identity;
- publication timestamp;
- source manifest used for publication;
- copied source output metadata;
- generated output metadata.
`manifest.json` from the source bundle is not copied as destination state. `manifest.json` from the source bundle is not copied as destination state.
## Retry behavior Do not edit `.distributor.json` by hand during normal operation. If it is missing or invalid while destination files remain, `distributor` treats the destination as unmanaged or conflicted.
If a destination already has matching `.distributor.json`, publication skips it as already published. ## Dry Runs
If destination state is older than the source manifest, publication replaces only managed outputs recorded in `.distributor.json` plus the state file. `--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files or destination state.
If a write fails during local publication, `distributor` removes outputs written during that failed attempt where possible so a retry does not see an unmanaged destination. Dry-run output is useful before publishing to confirm actions such as `publish_new`, `replace_older`, `skip_same`, and `skip_destination_newer`.
If one destination fails, later destinations in the same fan-out are still planned and run where they are independent. The command exits non-zero after printing the final status when any destination fails. ## Retry and Replacement Behavior
After a successful publish or replacement, the internal notifier hook runs as a no-op. Skipped destinations do not invoke it. If a destination has matching `.distributor.json`, publication skips it as already published.
If destination state is older than the source manifest and transfer policy allows replacement, publication deletes only managed outputs recorded in `.distributor.json` plus the state file, then writes the new outputs and state.
If destination state is newer than the source manifest, the default behavior is to skip. If destination state has the same source id and created timestamp but a different digest, publication fails as a conflict.
If a destination path has files but no valid `.distributor.json`, publication fails as unmanaged content. There is no force overwrite option.
## Failure Handling
If one destination fails in a fan-out run, independent later destinations are still planned and executed. The command exits non-zero after printing the final status if any destination failed.
If a write fails during local publication, `distributor` attempts to remove outputs written during that failed attempt so a retry does not see those partial outputs as unmanaged destination content.
After a successful publish or replacement, the internal notifier hook runs. The current default notifier is a no-op. Skipped destinations do not invoke it.
## Caveats ## Caveats
Only local-to-local publication is implemented. SSH, S3, external notification adapters, and force overwrite behavior are not implemented. Only local-to-local execution is implemented. SSH execution, S3 execution, external notification adapters, and force overwrite behavior are not implemented.
For symptom-oriented fixes, see [troubleshooting](troubleshooting.md). For config details, see [configuration](config.md). For command syntax, see [CLI](cli.md).

135
docs/troubleshooting.md Normal file
View File

@@ -0,0 +1,135 @@
# Distributor Troubleshooting
## `load config ... no such file or directory`
Likely cause: `run` could not find the config path. If `--config` is omitted, the default path is `/usr/local/etc/distributor/config.yml`.
Diagnostic:
```sh
ls -l <config-path>
```
Safe fix: pass an existing config path with `--config`, or install a config at the default path. See [configuration](config.md).
## `parse config ... field not found`
Likely cause: the YAML contains an unknown field. Config loading rejects unknown keys.
Diagnostic:
```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: compare the file to the reference in [configuration](config.md) and remove or rename unsupported fields.
## `validate config ... backend ... is unsupported`
Likely cause: a source or destination uses a backend name other than `local`, `ssh`, or `s3`.
Diagnostic:
```sh
rg -n "backend:" <config-path>
```
Safe fix: use `backend: local` for executable workflows. SSH and S3 config shapes are accepted only for validation; runtime execution is not implemented.
## `backend ssh is not implemented for execution` or `backend s3 is not implemented for execution`
Likely cause: the config validates but `run` tried to execute a remote backend.
Diagnostic:
```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: use local destinations for current executable workflows, or keep remote backend configs under roadmap material until those adapters exist. See [configuration](config.md).
## `validate command requires a path` or `inspect command requires a path`
Likely cause: `validate` or `inspect` was run without a path.
Diagnostic:
```sh
go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help
```
Safe fix: pass a local source bundle directory or a local tree containing source bundles.
## `no bundles found under "."`
Likely cause: the selected source root does not contain a `manifest.json` source bundle.
Diagnostic:
```sh
find <source-root> -name manifest.json -print
```
Safe fix: point the command or config at the directory containing the source bundle, or write a valid `manifest.json` and listed files. See [CLI](cli.md).
## `sha256 mismatch`, `size mismatch`, or `digest mismatch`
Likely cause: a listed source file changed after `manifest.json` was created, or the manifest digest does not match its file list.
Diagnostic:
```sh
go run ./cmd/distributor validate <source-root>
```
Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source digest failures.
## `destination has content but no distributor state`
Likely cause: the destination path is not empty and has no `.distributor.json` state file, so `distributor` will not claim it as managed.
Diagnostic:
```sh
find <destination-path> -maxdepth 2 -print
```
Safe fix: choose an empty destination path or move existing files aside after confirming they are not needed. There is no force overwrite option.
## `fail_conflict`
Likely cause: existing `.distributor.json` belongs to a different pipeline, a different destination, a different source id, or a same-created source with a different digest.
Diagnostic:
```sh
cat <destination-path>/.distributor.json
go run ./cmd/distributor inspect <source-root>
```
Safe fix: verify you are publishing the intended source to the intended destination. Use a separate destination path for unrelated content.
## `destination output path collision`
Likely cause: configured publication would write two outputs to the same destination path, such as publishing a source `report.html` while also generating `report.html` from `report.md`.
Diagnostic:
```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```
Safe fix: adjust the source bundle contents or publish policy so source and generated outputs do not collide.
## A run failed after writing some files
Likely cause: a write failed partway through publication. Local execution attempts to clean up outputs written during the failed attempt.
Diagnostic:
```sh
find <destination-path> -maxdepth 2 -print
```
Safe fix: inspect the destination before retrying. If only unrelated unmanaged files remain, move them aside or choose a clean destination. Re-run with `--dry-run` before publishing again. See [operations](operations.md).