distributor/docs/roadmap/remote_validate_inspect.md

Roadmap: Remote validate and inspect

Purpose

Expand distributor validate and distributor inspect so they can operate on configured pipeline sources, including local, ssh, and s3 sources.

This feature improves operator diagnostics after producer-side manifest creation is available.

Current Implementation Grounding

Current validate and inspect accept one local path. App-level code opens a local backend with openLocalPath, then uses bundle.Discover, which walks storage, finds manifest.json, and validates bundles.

run already has the app-level backend factory for configured local, SSH, and S3 sources. run also loads secrets.directory before backend construction so explicit S3 credential environment references can resolve without mutating the process environment.

This feature should reuse that source backend construction path for configured sources and remain read-only.

Goals

• Preserve the existing local path shortcut: distributor validate <path> and distributor inspect <path>.
• Add config-driven source validation and inspection: --config <path> --pipeline <id>.
• Support configured local, ssh, and s3 sources.
• Reuse the same source discovery and validation behavior used by run.
• Reuse secrets loading and backend credential resolution from run.
• Avoid opening or inspecting destinations in the first version.
• Keep output concise and operator-oriented.

Non-Goals

• Do not write to source or destination storage.
• Do not inspect destination state in v1.
• Do not add new backend types.
• Do not change source manifest schema.
• Do not add daemon, API, or producer-package dependencies.

CLI Mode Rules

Use two mutually exclusive modes:

distributor validate <local-path>
distributor inspect <local-path>

and:

distributor validate --config config.yml --pipeline weather-daily
distributor inspect --config config.yml --pipeline weather-daily

Do not allow positional local paths together with --config or --pipeline.

Require --pipeline in config mode, even when the config has exactly one pipeline. This avoids surprising remote access and keeps the initial behavior explicit.

Optional narrowing:

--bundle <source-root-relative-bundle-path>

When provided, --bundle identifies a source-root-relative bundle directory and the command validates or inspects that bundle rather than discovering all bundles.

Behavior

For config mode:

1. load config using the same defaulting and validation path as run;
2. load secrets.directory;
3. find the requested pipeline id;
4. open only the pipeline source backend;
5. discover all bundles or validate the requested --bundle;
6. validate manifest schema, paths, files, per-file digests, and bundle digest;
7. return non-zero if any selected bundle fails validation.

inspect should fully validate selected bundles by default, matching current bundle.Discover behavior. Output can then report reliable normalized metadata:

• pipeline id for config mode;
• backend type;
• source-relative bundle path;
• bundle id;
• created timestamp;
• digest;
• file count and total size;
• file path, size, and SHA-256.

Relationship To Other Roadmaps

Remote validation works well after manifest create and pkg/bundle because operators can validate producer output where it actually lands.

This feature does not depend on HTML index mode, link generation, or latest path destinations. Those features may later make inspection output richer, but v1 should stay source-focused.

Machine-readable output should follow docs/roadmap/cli_output_policy.md and use the shared --format text|json policy rather than command-specific JSON flags.

Testing Expectations

Suggested coverage:

• existing local path validation still works;
• existing local path inspection still works;
• config-mode local source validation works;
• fake configured source validation works through the storage abstraction;
• --bundle validates a specific source-relative bundle when implemented;
• missing pipeline id fails clearly;
• positional path plus --config is rejected;
• source backend open failures include pipeline/backend context;
• digest mismatch and missing manifest failures are clear;
• inspect output includes normalized source metadata.

Avoid live SSH or S3 tests except existing opt-in integration patterns.

Documentation Updates After Implementation

• Update docs/cli.md with both local and config-driven syntax.
• Update docs/operations.md with remote validation examples.
• Update docs/troubleshooting.md for common configured-source failures.
• Update examples only if useful and environment-gated.

Keep this roadmap under docs/roadmap/ until implemented.

Decisions

• Destination-state inspection is not part of v1 remote validate or inspect. These commands remain source-focused and should not open destinations unexpectedly.
• JSON output is deferred to docs/roadmap/cli_output_policy.md or to an implementation pass that selects that policy.
• --pipeline remains required in config mode, even when a config has exactly one pipeline.

Future Work

• Add destination-state inspection behind an explicit flag such as --with-destinations if operators need fan-out status diagnostics from inspect.
• Add JSON output through the CLI-wide --format text|json policy in docs/roadmap/cli_output_policy.md.
• Reconsider optional pipeline selection only if the project later introduces a broader command mode for single-pipeline configs.