eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Add configured source validation and inspection

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-01 21:11:35 +00:00
parent 8b1e5abf68
commit 29fd0e494c
13 changed files with 728 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
docs/operations.md
View File

@@ -50,6 +50,12 @@ Preview an environment-gated S3 destination config after editing it for an S3-co
go run ./cmd/distributor run --config examples/s3-destination.yml --dry-run
```
Validate one configured source without opening destinations:
```sh
go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline example-source-bundle
```
## Filesystem Layout
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
@@ -109,6 +115,28 @@ go run ./cmd/distributor validate <bundle-path>
Use repeated `--file` flags to preserve a specific file order. If no `--file` flags are provided, the command scans the bundle directory recursively using the same filtering rules as `pkg/bundle.BuildManifest`.
## Source Validation and Inspection
`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations.
Configured source validation is useful when producers write directly to SSH or S3 storage:
```sh
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
go run ./cmd/distributor inspect --config <config-path> --pipeline <pipeline-id>
```
Use `--bundle <path>` to validate or inspect one source-root-relative bundle directory:
```sh
go run ./cmd/distributor validate \
--config <config-path> \
--pipeline <pipeline-id> \
--bundle daily/2026-06-01
```
For configured SSH sources, host key and authentication behavior matches `run`. For configured S3 sources, endpoint, bucket, prefix, region, path-style, explicit credential environment variables, and `secrets.directory` handling match `run`.
## Dry Runs
`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH `known_hosts` entries.
@@ -181,9 +209,9 @@ secrets:
directory: /run/secrets/distributor
```
The directory is loaded during `run` before any source or destination backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the run fails before publication work starts.
The directory is loaded during `run` and configured-source `validate` or `inspect` before any backend is opened. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts.
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` prints a warning naming the ignored secret file variable without printing either value. The process environment is not changed.
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, `run` and configured-source diagnostics print a warning naming the ignored secret file variable without printing either value. The process environment is not changed.
## Caveats