Add configured source validation and inspection
This commit is contained in:
37
docs/cli.md
37
docs/cli.md
@@ -15,17 +15,19 @@ distributor [--help]
|
||||
distributor version [--format text|json]
|
||||
distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
|
||||
distributor validate [--format text|json] <path>
|
||||
distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
|
||||
distributor inspect [--format text|json] <path>
|
||||
distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
|
||||
distributor manifest create <bundle-path> --id <bundle-id> [options]
|
||||
```
|
||||
|
||||
- `version`: prints the application name and version. Development builds print `distributor dev`.
|
||||
- `run`: loads a YAML config, discovers source bundles, plans each configured destination, writes selected outputs unless `--dry-run` is set, and prints a final status summary.
|
||||
- `validate`: validates a local source bundle directory or a local tree containing source bundles.
|
||||
- `inspect`: validates local source bundles and prints normalized bundle metadata.
|
||||
- `validate`: validates a local source bundle directory, a local source bundle tree, or one configured pipeline source.
|
||||
- `inspect`: validates source bundles and prints normalized bundle metadata for a local path or one configured pipeline source.
|
||||
- `manifest create`: creates `manifest.json` for a local source bundle directory.
|
||||
|
||||
`validate` and `inspect` accept local paths only. `run` executes `local`, `ssh`, and `s3` backends.
|
||||
`validate` and `inspect` have two mutually exclusive modes: a local path shortcut, or configured source mode with `--config <path> --pipeline <id>`. Configured source mode opens only the selected pipeline source and supports configured `local`, `ssh`, and `s3` sources. It does not open destinations. `run` executes configured sources and destinations.
|
||||
|
||||
## Flag reference
|
||||
|
||||
@@ -47,6 +49,12 @@ Output-producing subcommands:
|
||||
- `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write output files, destination state, or SSH `known_hosts` entries.
|
||||
- `--force`: allow explicit destructive replacement for supported conflict cases in this run only.
|
||||
|
||||
`validate` and `inspect` configured source flags:
|
||||
|
||||
- `--config <path>`: config file to load for source validation or inspection. Required in configured source mode.
|
||||
- `--pipeline <id>`: pipeline source to validate or inspect. Required in configured source mode.
|
||||
- `--bundle <path>`: source-root-relative bundle directory to validate or inspect instead of discovering every bundle under the source root.
|
||||
|
||||
`manifest create` flags:
|
||||
|
||||
- `--id <bundle-id>`: source bundle id. Required.
|
||||
@@ -54,7 +62,7 @@ Output-producing subcommands:
|
||||
- `--created <time>`: RFC3339 source created timestamp. If omitted, the current UTC time is used.
|
||||
- `--overwrite`: replace an existing `manifest.json`.
|
||||
|
||||
`run` does not accept positional arguments. `validate` and `inspect` accept at most one path; omitting the path returns a required-path error.
|
||||
`run` does not accept positional arguments. `validate` and `inspect` accept at most one path in local mode. Local paths cannot be combined with `--config`, `--pipeline`, or `--bundle`.
|
||||
|
||||
## Common workflows
|
||||
|
||||
@@ -70,6 +78,21 @@ Inspect a source bundle:
|
||||
go run ./cmd/distributor inspect examples/source-bundle
|
||||
```
|
||||
|
||||
Validate a configured source without opening destinations:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
|
||||
```
|
||||
|
||||
Inspect one configured source bundle:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor inspect \
|
||||
--config <config-path> \
|
||||
--pipeline <pipeline-id> \
|
||||
--bundle daily/2026-06-01
|
||||
```
|
||||
|
||||
Create a manifest for a local producer bundle:
|
||||
|
||||
```sh
|
||||
@@ -152,13 +175,13 @@ Warnings are objects in the top-level `warnings` array and are not printed again
|
||||
Command-specific JSON results:
|
||||
|
||||
- `version`: application name and version.
|
||||
- `validate`: bundle count and discovered bundle identifiers.
|
||||
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records.
|
||||
- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend.
|
||||
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend.
|
||||
- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records.
|
||||
- `run`: dry-run status, pipeline summaries, destination action records, output records, final counters, warnings, and partial failure records.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
Use `manifest create` when a local producer has written bundle files but not `manifest.json`. Use `validate` before publication when a producer has written a new bundle. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
|
||||
Use `manifest create` when a local producer has written bundle files but not `manifest.json`. Use `validate` before publication when a producer has written a new bundle; use configured source mode when the bundle is already on an SSH or S3 source. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
|
||||
|
||||
For symptom-oriented recovery steps, see [troubleshooting](troubleshooting.md). For destination state and retry behavior, see [operations](operations.md). For config fields and defaults, see [configuration](config.md).
|
||||
|
||||
@@ -50,6 +50,12 @@ Preview an environment-gated S3 destination config after editing it for an S3-co
|
||||
go run ./cmd/distributor run --config examples/s3-destination.yml --dry-run
|
||||
```
|
||||
|
||||
Validate one configured source without opening destinations:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
|
||||
@@ -109,6 +115,28 @@ go run ./cmd/distributor validate <bundle-path>
|
||||
|
||||
Use repeated `--file` flags to preserve a specific file order. If no `--file` flags are provided, the command scans the bundle directory recursively using the same filtering rules as `pkg/bundle.BuildManifest`.
|
||||
|
||||
## Source Validation and Inspection
|
||||
|
||||
`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations.
|
||||
|
||||
Configured source validation is useful when producers write directly to SSH or S3 storage:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
|
||||
go run ./cmd/distributor inspect --config <config-path> --pipeline <pipeline-id>
|
||||
```
|
||||
|
||||
Use `--bundle <path>` to validate or inspect one source-root-relative bundle directory:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate \
|
||||
--config <config-path> \
|
||||
--pipeline <pipeline-id> \
|
||||
--bundle daily/2026-06-01
|
||||
```
|
||||
|
||||
For configured SSH sources, host key and authentication behavior matches `run`. For configured S3 sources, endpoint, bucket, prefix, region, path-style, explicit credential environment variables, and `secrets.directory` handling match `run`.
|
||||
|
||||
## Dry Runs
|
||||
|
||||
`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH `known_hosts` entries.
|
||||
@@ -181,9 +209,9 @@ secrets:
|
||||
directory: /run/secrets/distributor
|
||||
```
|
||||
|
||||
The directory is loaded during `run` before any source or destination backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the run fails before publication work starts.
|
||||
The directory is loaded during `run` and configured-source `validate` or `inspect` before any backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts.
|
||||
|
||||
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` prints a warning naming the ignored secret file variable without printing either value. The process environment is not changed.
|
||||
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` and configured-source diagnostics print a warning naming the ignored secret file variable without printing either value. The process environment is not changed.
|
||||
|
||||
## Caveats
|
||||
|
||||
|
||||
@@ -60,6 +60,31 @@ go run ./cmd/distributor run --config <config-path> --format json
|
||||
|
||||
Safe fix: read the stderr error and fix the setup problem. JSON mode writes a document only after the command has enough information to construct a result.
|
||||
|
||||
## `configured source mode requires --pipeline`
|
||||
|
||||
Likely cause: `validate` or `inspect` was run with `--config` but without an explicit pipeline id.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate --help
|
||||
go run ./cmd/distributor inspect --help
|
||||
```
|
||||
|
||||
Safe fix: add `--pipeline <pipeline-id>`. Configured source diagnostics require an explicit pipeline even when the config contains one pipeline.
|
||||
|
||||
## `does not accept a local path with --config, --pipeline, or --bundle`
|
||||
|
||||
Likely cause: local-path mode and configured source mode were mixed in one `validate` or `inspect` command.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor inspect --help
|
||||
```
|
||||
|
||||
Safe fix: use either `distributor inspect <local-path>` or `distributor inspect --config <path> --pipeline <id>`, not both.
|
||||
|
||||
## `--format json` exited non-zero with `ok: false`
|
||||
|
||||
Likely cause: `run` began planning or executing destinations, and at least one destination failed while other destination results were still available.
|
||||
@@ -234,21 +259,33 @@ ssh-keygen -F <host> -f <known-hosts-path>
|
||||
|
||||
Safe fix: verify the server identity out of band before updating `known_hosts`. Do not switch to `host_key_policy: off` to bypass an unexpected changed key.
|
||||
|
||||
## `stat ssh ... not_found` or `no bundles found`
|
||||
## `pipeline "<id>" not found`
|
||||
|
||||
Likely cause: the configured SSH `path` is wrong, unreadable, or does not contain source bundles.
|
||||
Likely cause: configured source validation or inspection requested a pipeline id that is not present in the config file.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
sftp <user>@<host>
|
||||
rg -n "id:" <config-path>
|
||||
```
|
||||
|
||||
Safe fix: correct the remote root `path`, permissions, or source bundle location.
|
||||
Safe fix: pass an existing pipeline id with `--pipeline`, or update the config.
|
||||
|
||||
## `stat ssh ... not_found`, `stat s3 ... not_found`, or `no bundles found`
|
||||
|
||||
Likely cause: the configured source root is wrong, unreadable, or does not contain source bundles.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
|
||||
```
|
||||
|
||||
Safe fix: correct the configured source root, S3 prefix, permissions, or source bundle location. Use `--bundle <path>` only with a source-root-relative bundle directory that contains `manifest.json`.
|
||||
|
||||
## `validate command requires a path` or `inspect command requires a path`
|
||||
|
||||
Likely cause: `validate` or `inspect` was run without a path.
|
||||
Likely cause: `validate` or `inspect` was run without a local path and without configured source mode.
|
||||
|
||||
Diagnostic:
|
||||
|
||||
@@ -257,7 +294,7 @@ go run ./cmd/distributor validate --help
|
||||
go run ./cmd/distributor inspect --help
|
||||
```
|
||||
|
||||
Safe fix: pass a local source bundle directory or a local tree containing source bundles.
|
||||
Safe fix: pass a local source bundle directory or local tree, or pass both `--config <path>` and `--pipeline <id>`.
|
||||
|
||||
## `no bundles found under "."`
|
||||
|
||||
|
||||
Reference in New Issue
Block a user