159 lines
5.4 KiB
Markdown
159 lines
5.4 KiB
Markdown
# Roadmap: Remote `validate` and `inspect`
|
|
|
|
## Purpose
|
|
|
|
Expand `distributor validate` and `distributor inspect` so they can operate on
|
|
configured pipeline sources, including `local`, `ssh`, and `s3` sources.
|
|
|
|
This feature improves operator diagnostics after producer-side manifest
|
|
creation is available.
|
|
|
|
## Current Implementation Grounding
|
|
|
|
Current `validate` and `inspect` accept one local path. App-level code opens a
|
|
local backend with `openLocalPath`, then uses `bundle.Discover`, which walks
|
|
storage, finds `manifest.json`, and validates bundles.
|
|
|
|
`run` already has the app-level backend factory for configured local, SSH, and
|
|
S3 sources. `run` also loads `secrets.directory` before backend construction so
|
|
explicit S3 credential environment references can resolve without mutating the
|
|
process environment.
|
|
|
|
This feature should reuse that source backend construction path for configured
|
|
sources and remain read-only.
|
|
|
|
## Goals
|
|
|
|
- Preserve the existing local path shortcut:
|
|
`distributor validate <path>` and `distributor inspect <path>`.
|
|
- Add config-driven source validation and inspection:
|
|
`--config <path> --pipeline <id>`.
|
|
- Support configured `local`, `ssh`, and `s3` sources.
|
|
- Reuse the same source discovery and validation behavior used by `run`.
|
|
- Reuse secrets loading and backend credential resolution from `run`.
|
|
- Avoid opening or inspecting destinations in the first version.
|
|
- Keep output concise and operator-oriented.
|
|
|
|
## Non-Goals
|
|
|
|
- Do not write to source or destination storage.
|
|
- Do not inspect destination state in v1.
|
|
- Do not add new backend types.
|
|
- Do not change source manifest schema.
|
|
- Do not add daemon, API, or producer-package dependencies.
|
|
|
|
## CLI Mode Rules
|
|
|
|
Use two mutually exclusive modes:
|
|
|
|
```sh
|
|
distributor validate <local-path>
|
|
distributor inspect <local-path>
|
|
```
|
|
|
|
and:
|
|
|
|
```sh
|
|
distributor validate --config config.yml --pipeline weather-daily
|
|
distributor inspect --config config.yml --pipeline weather-daily
|
|
```
|
|
|
|
Do not allow positional local paths together with `--config` or `--pipeline`.
|
|
|
|
Require `--pipeline` in config mode, even when the config has exactly one
|
|
pipeline. This avoids surprising remote access and keeps the initial behavior
|
|
explicit.
|
|
|
|
Optional narrowing:
|
|
|
|
```sh
|
|
--bundle <source-root-relative-bundle-path>
|
|
```
|
|
|
|
When provided, `--bundle` identifies a source-root-relative bundle directory and
|
|
the command validates or inspects that bundle rather than discovering all
|
|
bundles.
|
|
|
|
## Behavior
|
|
|
|
For config mode:
|
|
|
|
1. load config using the same defaulting and validation path as `run`;
|
|
2. load `secrets.directory`;
|
|
3. find the requested pipeline id;
|
|
4. open only the pipeline source backend;
|
|
5. discover all bundles or validate the requested `--bundle`;
|
|
6. validate manifest schema, paths, files, per-file digests, and bundle digest;
|
|
7. return non-zero if any selected bundle fails validation.
|
|
|
|
`inspect` should fully validate selected bundles by default, matching current
|
|
`bundle.Discover` behavior. Output can then report reliable normalized metadata:
|
|
|
|
- pipeline id for config mode;
|
|
- backend type;
|
|
- source-relative bundle path;
|
|
- bundle id;
|
|
- created timestamp;
|
|
- digest;
|
|
- file count and total size;
|
|
- file path, size, and SHA-256.
|
|
|
|
## Relationship To Other Roadmaps
|
|
|
|
Remote validation works well after `manifest create` and `pkg/bundle` because
|
|
operators can validate producer output where it actually lands.
|
|
|
|
This feature does not depend on HTML index mode, link generation, or latest path
|
|
destinations. Those features may later make inspection output richer, but v1
|
|
should stay source-focused.
|
|
|
|
Machine-readable output should follow `docs/roadmap/cli_output_policy.md` and
|
|
use the shared `--format text|json` policy rather than command-specific JSON
|
|
flags.
|
|
|
|
## Testing Expectations
|
|
|
|
Suggested coverage:
|
|
|
|
- existing local path validation still works;
|
|
- existing local path inspection still works;
|
|
- config-mode local source validation works;
|
|
- fake configured source validation works through the storage abstraction;
|
|
- `--bundle` validates a specific source-relative bundle when implemented;
|
|
- missing pipeline id fails clearly;
|
|
- positional path plus `--config` is rejected;
|
|
- source backend open failures include pipeline/backend context;
|
|
- digest mismatch and missing manifest failures are clear;
|
|
- inspect output includes normalized source metadata.
|
|
|
|
Avoid live SSH or S3 tests except existing opt-in integration patterns.
|
|
|
|
## Documentation Updates After Implementation
|
|
|
|
- Update `docs/cli.md` with both local and config-driven syntax.
|
|
- Update `docs/operations.md` with remote validation examples.
|
|
- Update `docs/troubleshooting.md` for common configured-source failures.
|
|
- Update examples only if useful and environment-gated.
|
|
|
|
Keep this roadmap under `docs/roadmap/` until implemented.
|
|
|
|
## Decisions
|
|
|
|
- Destination-state inspection is not part of v1 remote `validate` or
|
|
`inspect`. These commands remain source-focused and should not open
|
|
destinations unexpectedly.
|
|
- JSON output is deferred to `docs/roadmap/cli_output_policy.md` or to an
|
|
implementation pass that selects that policy.
|
|
- `--pipeline` remains required in config mode, even when a config has exactly
|
|
one pipeline.
|
|
|
|
## Future Work
|
|
|
|
- Add destination-state inspection behind an explicit flag such as
|
|
`--with-destinations` if operators need fan-out status diagnostics from
|
|
`inspect`.
|
|
- Add JSON output through the CLI-wide `--format text|json` policy in
|
|
`docs/roadmap/cli_output_policy.md`.
|
|
- Reconsider optional pipeline selection only if the project later introduces a
|
|
broader command mode for single-pipeline configs.
|