Add configured source validation and inspection

This commit is contained in:
2026-06-01 21:11:35 +00:00
parent 8b1e5abf68
commit 29fd0e494c
13 changed files with 728 additions and 50 deletions

View File

@@ -15,17 +15,19 @@ distributor [--help]
distributor version [--format text|json]
distributor run [--config <path>] [--dry-run] [--force] [--format text|json]
distributor validate [--format text|json] <path>
distributor validate --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
distributor inspect [--format text|json] <path>
distributor inspect --config <path> --pipeline <id> [--bundle <path>] [--format text|json]
distributor manifest create <bundle-path> --id <bundle-id> [options]
```
- `version`: prints the application name and version. Development builds print `distributor dev`.
- `run`: loads a YAML config, discovers source bundles, plans each configured destination, writes selected outputs unless `--dry-run` is set, and prints a final status summary.
- `validate`: validates a local source bundle directory or a local tree containing source bundles.
- `inspect`: validates local source bundles and prints normalized bundle metadata.
- `validate`: validates a local source bundle directory, a local source bundle tree, or one configured pipeline source.
- `inspect`: validates source bundles and prints normalized bundle metadata for a local path or one configured pipeline source.
- `manifest create`: creates `manifest.json` for a local source bundle directory.
`validate` and `inspect` accept local paths only. `run` executes `local`, `ssh`, and `s3` backends.
`validate` and `inspect` have two mutually exclusive modes: a local path shortcut, or configured source mode with `--config <path> --pipeline <id>`. Configured source mode opens only the selected pipeline source and supports configured `local`, `ssh`, and `s3` sources. It does not open destinations. `run` executes configured sources and destinations.
## Flag reference
@@ -47,6 +49,12 @@ Output-producing subcommands:
- `--dry-run`: load config, discover bundles, inspect destination state, print planned actions and final status, and do not write output files, destination state, or SSH `known_hosts` entries.
- `--force`: allow explicit destructive replacement for supported conflict cases in this run only.
`validate` and `inspect` configured source flags:
- `--config <path>`: config file to load for source validation or inspection. Required in configured source mode.
- `--pipeline <id>`: pipeline source to validate or inspect. Required in configured source mode.
- `--bundle <path>`: source-root-relative bundle directory to validate or inspect instead of discovering every bundle under the source root.
`manifest create` flags:
- `--id <bundle-id>`: source bundle id. Required.
@@ -54,7 +62,7 @@ Output-producing subcommands:
- `--created <time>`: RFC3339 source created timestamp. If omitted, the current UTC time is used.
- `--overwrite`: replace an existing `manifest.json`.
`run` does not accept positional arguments. `validate` and `inspect` accept at most one path; omitting the path returns a required-path error.
`run` does not accept positional arguments. `validate` and `inspect` accept at most one path in local mode. Local paths cannot be combined with `--config`, `--pipeline`, or `--bundle`.
## Common workflows
@@ -70,6 +78,21 @@ Inspect a source bundle:
go run ./cmd/distributor inspect examples/source-bundle
```
Validate a configured source without opening destinations:
```sh
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
```
Inspect one configured source bundle:
```sh
go run ./cmd/distributor inspect \
--config <config-path> \
--pipeline <pipeline-id> \
--bundle daily/2026-06-01
```
Create a manifest for a local producer bundle:
```sh
@@ -152,13 +175,13 @@ Warnings are objects in the top-level `warnings` array and are not printed again
Command-specific JSON results:
- `version`: application name and version.
- `validate`: bundle count and discovered bundle identifiers.
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records.
- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend.
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend.
- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records.
- `run`: dry-run status, pipeline summaries, destination action records, output records, final counters, warnings, and partial failure records.
## Diagnostics
Use `manifest create` when a local producer has written bundle files but not `manifest.json`. Use `validate` before publication when a producer has written a new bundle. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
Use `manifest create` when a local producer has written bundle files but not `manifest.json`. Use `validate` before publication when a producer has written a new bundle; use configured source mode when the bundle is already on an SSH or S3 source. Use `inspect` to confirm normalized ids, timestamps, digests, file paths, and file sizes.
For symptom-oriented recovery steps, see [troubleshooting](troubleshooting.md). For destination state and retry behavior, see [operations](operations.md). For config fields and defaults, see [configuration](config.md).

View File

@@ -50,6 +50,12 @@ Preview an environment-gated S3 destination config after editing it for an S3-co
go run ./cmd/distributor run --config examples/s3-destination.yml --dry-run
```
Validate one configured source without opening destinations:
```sh
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
```
## Filesystem Layout
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
@@ -109,6 +115,28 @@ go run ./cmd/distributor validate <bundle-path>
Use repeated `--file` flags to preserve a specific file order. If no `--file` flags are provided, the command scans the bundle directory recursively using the same filtering rules as `pkg/bundle.BuildManifest`.
## Source Validation and Inspection
`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations.
Configured source validation is useful when producers write directly to SSH or S3 storage:
```sh
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
go run ./cmd/distributor inspect --config <config-path> --pipeline <pipeline-id>
```
Use `--bundle <path>` to validate or inspect one source-root-relative bundle directory:
```sh
go run ./cmd/distributor validate \
--config <config-path> \
--pipeline <pipeline-id> \
--bundle daily/2026-06-01
```
For configured SSH sources, host key and authentication behavior matches `run`. For configured S3 sources, endpoint, bucket, prefix, region, path-style, explicit credential environment variables, and `secrets.directory` handling match `run`.
## Dry Runs
`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH `known_hosts` entries.
@@ -181,9 +209,9 @@ secrets:
directory: /run/secrets/distributor
```
The directory is loaded during `run` before any source or destination backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the run fails before publication work starts.
The directory is loaded during `run` and configured-source `validate` or `inspect` before any backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts.
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` prints a warning naming the ignored secret file variable without printing either value. The process environment is not changed.
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` and configured-source diagnostics print a warning naming the ignored secret file variable without printing either value. The process environment is not changed.
## Caveats

View File

@@ -60,6 +60,31 @@ go run ./cmd/distributor run --config <config-path> --format json
Safe fix: read the stderr error and fix the setup problem. JSON mode writes a document only after the command has enough information to construct a result.
## `configured source mode requires --pipeline`
Likely cause: `validate` or `inspect` was run with `--config` but without an explicit pipeline id.
Diagnostic:
```sh
go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help
```
Safe fix: add `--pipeline <pipeline-id>`. Configured source diagnostics require an explicit pipeline even when the config contains one pipeline.
## `does not accept a local path with --config, --pipeline, or --bundle`
Likely cause: local-path mode and configured source mode were mixed in one `validate` or `inspect` command.
Diagnostic:
```sh
go run ./cmd/distributor inspect --help
```
Safe fix: use either `distributor inspect <local-path>` or `distributor inspect --config <path> --pipeline <id>`, not both.
## `--format json` exited non-zero with `ok: false`
Likely cause: `run` began planning or executing destinations, and at least one destination failed while other destination results were still available.
@@ -234,21 +259,33 @@ ssh-keygen -F <host> -f <known-hosts-path>
Safe fix: verify the server identity out of band before updating `known_hosts`. Do not switch to `host_key_policy: off` to bypass an unexpected changed key.
## `stat ssh ... not_found` or `no bundles found`
## `pipeline "<id>" not found`
Likely cause: the configured SSH `path` is wrong, unreadable, or does not contain source bundles.
Likely cause: configured source validation or inspection requested a pipeline id that is not present in the config file.
Diagnostic:
```sh
sftp <user>@<host>
rg -n "id:" <config-path>
```
Safe fix: correct the remote root `path`, permissions, or source bundle location.
Safe fix: pass an existing pipeline id with `--pipeline`, or update the config.
## `stat ssh ... not_found`, `stat s3 ... not_found`, or `no bundles found`
Likely cause: the configured source root is wrong, unreadable, or does not contain source bundles.
Diagnostic:
```sh
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
```
Safe fix: correct the configured source root, S3 prefix, permissions, or source bundle location. Use `--bundle <path>` only with a source-root-relative bundle directory that contains `manifest.json`.
## `validate command requires a path` or `inspect command requires a path`
Likely cause: `validate` or `inspect` was run without a path.
Likely cause: `validate` or `inspect` was run without a local path and without configured source mode.
Diagnostic:
@@ -257,7 +294,7 @@ go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help
```
Safe fix: pass a local source bundle directory or a local tree containing source bundles.
Safe fix: pass a local source bundle directory or local tree, or pass both `--config <path>` and `--pipeline <id>`.
## `no bundles found under "."`