Clean up completed roadmap entries
This commit is contained in:
@@ -1,242 +0,0 @@
|
||||
# Roadmap: CLI Output Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
Define a shared CLI output policy before adding machine-readable JSON output to
|
||||
`distributor` commands.
|
||||
|
||||
Current CLI output is human-readable text. That is appropriate as the default,
|
||||
but upcoming features need a common machine-readable contract so each command
|
||||
does not invent a separate JSON flag, envelope, warning policy, or error shape.
|
||||
|
||||
## Current Implementation Grounding
|
||||
|
||||
The current CLI has these output-producing commands:
|
||||
|
||||
- `version`;
|
||||
- `run`;
|
||||
- `validate`;
|
||||
- `inspect`.
|
||||
|
||||
Current output is text-only. `run` has the most complex output because it can
|
||||
print warnings, pipeline summaries, planned destination actions, partial
|
||||
destination failures, and a final status line. `validate` and `inspect` are
|
||||
currently local-path commands. `version` prints a single text line.
|
||||
|
||||
Planned `distributor manifest create`, remote `validate` and `inspect`, link
|
||||
generation, and latest path destinations increase the need for structured
|
||||
output that scripts can consume consistently.
|
||||
|
||||
## Goals
|
||||
|
||||
- Define one CLI-wide policy for text and JSON output.
|
||||
- Preserve current human-readable text output as the default.
|
||||
- Add JSON output consistently across output-producing commands when this
|
||||
roadmap is implemented.
|
||||
- Keep warning, error, and partial-failure behavior predictable.
|
||||
- Avoid external dependencies; use Go's standard `encoding/json`.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not add YAML, table, NDJSON, streaming JSON, or template output formats.
|
||||
- Do not change source manifest, destination state, config, or backend schemas.
|
||||
- Do not make help or usage output JSON.
|
||||
- Do not turn ordinary fatal setup errors into structured JSON results.
|
||||
- Do not add a root-global output flag in the first implementation.
|
||||
|
||||
## Shared CLI Policy
|
||||
|
||||
Each supported command should accept:
|
||||
|
||||
```sh
|
||||
--format text
|
||||
--format json
|
||||
```
|
||||
|
||||
Policy:
|
||||
|
||||
- `text` is the default and preserves existing output unless a feature
|
||||
explicitly changes text output.
|
||||
- `json` writes exactly one JSON document to stdout.
|
||||
- help and usage output remain text-only.
|
||||
- invalid `--format` values are usage errors.
|
||||
- `--format` is a per-command flag, not a root-global flag.
|
||||
- a command must not accept `--format json` unless it emits the shared JSON
|
||||
envelope for that command.
|
||||
|
||||
The first implementation should add JSON support for all current
|
||||
output-producing commands rather than leaving a mixed CLI where some commands
|
||||
support `--format` and others do not.
|
||||
|
||||
## Stdout, Stderr, Warnings, and Errors
|
||||
|
||||
Text mode keeps the current behavior:
|
||||
|
||||
- normal command output goes to stdout;
|
||||
- warnings may be printed as text;
|
||||
- command errors are printed to stderr by CLI error handling.
|
||||
|
||||
JSON mode:
|
||||
|
||||
- successful commands write one JSON document to stdout;
|
||||
- warnings are included in a top-level `warnings` array and are not duplicated
|
||||
to stderr;
|
||||
- fatal setup errors that prevent construction of a result write text errors to
|
||||
stderr, write no JSON stdout, and exit non-zero;
|
||||
- commands with meaningful partial results may write a JSON document with
|
||||
`ok: false`, structured failure details, and a non-zero exit status.
|
||||
|
||||
`run --format json` should use the partial-result behavior when planning or
|
||||
execution has begun and one or more destinations fail. This lets automation
|
||||
inspect successful actions, failed actions, warnings, and final counters even
|
||||
when the process exits non-zero.
|
||||
|
||||
## JSON Envelope
|
||||
|
||||
All JSON-mode commands should use a common top-level envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"command": "inspect",
|
||||
"ok": true,
|
||||
"warnings": [],
|
||||
"result": {}
|
||||
}
|
||||
```
|
||||
|
||||
Envelope fields:
|
||||
|
||||
- `schema_version`: integer version of the CLI JSON output schema.
|
||||
- `command`: command name, using the public CLI command path where useful, such
|
||||
as `manifest create`.
|
||||
- `ok`: boolean success indicator for the command result.
|
||||
- `warnings`: array of structured warning objects.
|
||||
- `result`: command-specific result object.
|
||||
- `errors`: optional array of structured error objects for commands that can
|
||||
return partial results.
|
||||
|
||||
Field rules:
|
||||
|
||||
- use stable snake_case field names;
|
||||
- use RFC3339 timestamps;
|
||||
- use numeric JSON values for sizes and counts;
|
||||
- use slash-separated logical paths for bundle, source, destination, and output
|
||||
paths;
|
||||
- never include secret values;
|
||||
- keep command-specific data under `result`;
|
||||
- add fields compatibly where practical.
|
||||
|
||||
Warning objects should include at least:
|
||||
|
||||
```json
|
||||
{
|
||||
"message": "secret OBJECT_STORAGE_KEY ignored because the real environment already has that variable"
|
||||
}
|
||||
```
|
||||
|
||||
Error objects for partial results should include enough context for automation
|
||||
and troubleshooting, such as pipeline id, destination id, backend, bundle path,
|
||||
and message when those values are available.
|
||||
|
||||
## Command Adoption
|
||||
|
||||
`version`:
|
||||
|
||||
- text mode keeps the current version string;
|
||||
- JSON mode reports application name and version under `result`.
|
||||
|
||||
`validate`:
|
||||
|
||||
- text mode keeps the current validation summary;
|
||||
- JSON mode reports bundle count and selected bundle identifiers;
|
||||
- future remote validation uses the same envelope and adds pipeline/source
|
||||
context under `result`.
|
||||
|
||||
`inspect`:
|
||||
|
||||
- text mode keeps human-readable bundle metadata;
|
||||
- JSON mode reports normalized bundle metadata, including source-relative path,
|
||||
bundle id, created timestamp, digest, file count, total size, and file
|
||||
records.
|
||||
|
||||
`run`:
|
||||
|
||||
- text mode keeps the current progress and final status style unless a future
|
||||
feature intentionally changes it;
|
||||
- JSON mode reports warnings, pipeline summaries, destination action records,
|
||||
output records, final counters, dry-run status, and partial failure records;
|
||||
- JSON mode may write `ok: false` with partial results and still exit non-zero.
|
||||
|
||||
Future `manifest create`:
|
||||
|
||||
- should use `--format text|json`;
|
||||
- text mode should keep concise human success output;
|
||||
- JSON mode should report the generated manifest summary under `result`;
|
||||
- the command should not introduce a separate `--json` flag.
|
||||
|
||||
## Relationship To Other Roadmaps
|
||||
|
||||
`distributor manifest create` should depend on this roadmap for JSON summary
|
||||
output and should not add a command-specific JSON flag.
|
||||
|
||||
Remote `validate` and `inspect` should use this policy for machine-readable
|
||||
source validation and inspection.
|
||||
|
||||
Link generation should expose primary links through JSON command output only
|
||||
after this policy is implemented or selected for implementation.
|
||||
|
||||
Latest path destinations should report fixed-path selection and destructive
|
||||
replacement summaries through the same `run --format json` result model.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Suggested coverage:
|
||||
|
||||
- CLI parsing accepts `--format text` and `--format json` for supported
|
||||
commands;
|
||||
- CLI parsing rejects invalid `--format` values as usage errors;
|
||||
- help and usage output remain text-only;
|
||||
- each JSON-capable command emits exactly one valid JSON document to stdout on
|
||||
success;
|
||||
- text mode preserves existing output;
|
||||
- JSON-mode warnings appear in `warnings` and are not duplicated to stderr;
|
||||
- fatal setup errors write no JSON stdout and return non-zero;
|
||||
- `run --format json` emits partial-result JSON with `ok: false` and exits
|
||||
non-zero when one or more destination failures occur after planning begins;
|
||||
- JSON output uses RFC3339 timestamps, numeric sizes and counts, and
|
||||
slash-separated logical paths;
|
||||
- no JSON output includes secret values;
|
||||
- documentation consistency checks find no unsupported `--json` references.
|
||||
|
||||
## Documentation Updates After Implementation
|
||||
|
||||
- Update `docs/cli.md` with the shared `--format text|json` policy.
|
||||
- Update command examples only for implemented JSON behavior.
|
||||
- Update `docs/operations.md` where JSON output materially improves automation
|
||||
workflows.
|
||||
- Update `docs/troubleshooting.md` only for implemented JSON-mode recovery
|
||||
behavior.
|
||||
- Update relevant roadmap files when JSON output is no longer deferred.
|
||||
|
||||
Keep this roadmap under `docs/roadmap/` until implemented.
|
||||
|
||||
## Decisions
|
||||
|
||||
- Use per-command `--format text|json`; do not introduce `--json`.
|
||||
- Keep `text` as the default for backward compatibility.
|
||||
- Implement JSON support for all current output-producing commands when this
|
||||
roadmap is selected.
|
||||
- Keep help and usage output text-only.
|
||||
- Put JSON-mode warnings in the top-level `warnings` array.
|
||||
- Allow `run --format json` to emit partial-result JSON with `ok: false` and a
|
||||
non-zero exit status.
|
||||
- Keep fatal setup errors as text stderr with no JSON stdout.
|
||||
|
||||
## Future Work
|
||||
|
||||
- Consider a root-global output flag only if the command parser is later
|
||||
refactored around shared root options.
|
||||
- Consider additional formats only if a concrete consumer requires them.
|
||||
- Consider a versioned JSON schema reference after the first JSON-capable
|
||||
release.
|
||||
@@ -1,160 +0,0 @@
|
||||
# Roadmap: HTML Index Mode
|
||||
|
||||
## Purpose
|
||||
|
||||
Add a Markdown-to-HTML `index` mode that renders one selected Markdown artifact
|
||||
to `index.html` at the destination bundle path.
|
||||
|
||||
This feature improves static-site UX and gives link generation and latest path
|
||||
destinations a clean directory-style output to prefer.
|
||||
|
||||
## Current Implementation Grounding
|
||||
|
||||
Current Markdown transformation lives in `internal/transform/markdown`. It
|
||||
renders every manifest-listed `.md` file to a sidecar `.html` file:
|
||||
|
||||
```text
|
||||
report.md -> report.html
|
||||
```
|
||||
|
||||
Current config validation accepts only `markdown_to_html.mode: sidecar`.
|
||||
`publish.PlanOutputs` asks the configured transformer for generated outputs,
|
||||
then destination state records generated outputs with path, kind, source path,
|
||||
transform name, SHA-256, and size. Current destination state has no field for a
|
||||
transform mode beyond the existing transform string.
|
||||
|
||||
## Goals
|
||||
|
||||
- Preserve existing sidecar behavior as the default and supported explicit mode.
|
||||
- Add `markdown_to_html.mode: index`.
|
||||
- In index mode, render exactly one Markdown input to `index.html`.
|
||||
- Record `index.html` as a normal generated output in `.distributor.json`.
|
||||
- Keep source manifests unchanged.
|
||||
- Keep link generation and latest paths straightforward without making them
|
||||
dependencies.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not add collection index pages.
|
||||
- Do not add multi-page static-site generation.
|
||||
- Do not add feeds, notifications, or link generation in this feature.
|
||||
- Do not require producers to name a file `index.md`.
|
||||
- Do not add custom output filenames in v1.
|
||||
|
||||
## Configuration Shape
|
||||
|
||||
Extend existing destination-level transform config:
|
||||
|
||||
```yaml
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: index
|
||||
input: report.md
|
||||
```
|
||||
|
||||
Mode semantics:
|
||||
|
||||
- `sidecar`: existing behavior; render each manifest-listed Markdown file to a
|
||||
same-directory `.html` file.
|
||||
- `index`: render one selected Markdown file to `index.html` at the destination
|
||||
bundle path.
|
||||
|
||||
`sidecar` remains the current/default mode. `index` is the proposed addition.
|
||||
|
||||
## Input Selection
|
||||
|
||||
Index mode should choose the Markdown input deterministically:
|
||||
|
||||
1. If `transform.markdown_to_html.input` is configured, use that manifest-listed
|
||||
Markdown file.
|
||||
2. If no input is configured and the source manifest lists exactly one Markdown
|
||||
file, use that file.
|
||||
3. If no input is configured and there are zero or multiple Markdown files, fail
|
||||
planning with a clear error.
|
||||
|
||||
The configured input path must be a safe relative source path, must be listed in
|
||||
the source manifest, and must end in `.md`.
|
||||
|
||||
Input selection happens during planning because it depends on the validated
|
||||
source manifest, not only static config.
|
||||
|
||||
## Output Semantics
|
||||
|
||||
Index mode always writes:
|
||||
|
||||
```text
|
||||
index.html
|
||||
```
|
||||
|
||||
relative to the destination bundle path.
|
||||
|
||||
The generated output should use current state model terms:
|
||||
|
||||
- `path`: `index.html`;
|
||||
- `kind`: `generated`;
|
||||
- `source_path`: selected Markdown file;
|
||||
- `transform`: `markdown_to_html`;
|
||||
- `sha256` and `size`: digest and byte size of generated HTML.
|
||||
|
||||
Replacement, skip, cleanup, and force behavior should treat `index.html` like
|
||||
any other distributor-managed generated output.
|
||||
|
||||
Config validation should reject enabled Markdown-to-HTML transform
|
||||
configuration when `publish.html` is false. That keeps publish policy and
|
||||
transform intent aligned and avoids silently accepting unused transform config.
|
||||
|
||||
## Relationship To Other Roadmaps
|
||||
|
||||
Link generation should recognize `index.html` and produce directory-style URLs,
|
||||
but link generation must not be required for index mode.
|
||||
|
||||
Latest path destinations work without index mode, but fixed latest destinations
|
||||
produce nicer stable URLs when they publish `index.html`.
|
||||
|
||||
`manifest create`, `pkg/bundle`, and remote validation remain source-bundle
|
||||
features and should not depend on transform mode.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Suggested coverage:
|
||||
|
||||
- config validation accepts `mode: sidecar` and `mode: index`;
|
||||
- current sidecar behavior remains unchanged;
|
||||
- index mode accepts explicit input;
|
||||
- index mode chooses the only Markdown file when no input is configured;
|
||||
- planning fails when index mode has zero or multiple Markdown candidates
|
||||
without explicit input;
|
||||
- planning fails when explicit input is unsafe, not listed, or not Markdown;
|
||||
- index mode emits `index.html`;
|
||||
- state records `index.html` as a generated output;
|
||||
- source-only publication does not write `index.html`;
|
||||
- dry-run reports `index.html` without writing it;
|
||||
- replacement safely updates prior generated `index.html`.
|
||||
|
||||
## Documentation Updates After Implementation
|
||||
|
||||
- Update `docs/config.md` with Markdown-to-HTML modes and `input`.
|
||||
- Update `docs/integrations/markdown.md` with sidecar and index behavior.
|
||||
- Update `docs/operations.md` with a static-site example.
|
||||
- Add or update examples only for implemented behavior.
|
||||
|
||||
Keep this roadmap under `docs/roadmap/` until implemented.
|
||||
|
||||
## Decisions
|
||||
|
||||
- Enabled Markdown-to-HTML transform config is rejected when `publish.html` is
|
||||
false.
|
||||
- Generated output metadata keeps `transform: markdown_to_html`; the output
|
||||
path and destination config distinguish sidecar from index behavior.
|
||||
- Future multi-page or collection index generation should be a separate
|
||||
transform, not an expansion of this single-input index mode.
|
||||
|
||||
## Future Work
|
||||
|
||||
- Add a separate collection or site-index transform if distributor later needs
|
||||
multi-page aggregation.
|
||||
- Consider richer transform metadata only if future state consumers need more
|
||||
than the transform name and output path.
|
||||
- Consider custom index output names only if fixed `index.html` proves too
|
||||
limiting for real deployments.
|
||||
@@ -1,10 +1,8 @@
|
||||
# Implementation Roadmap
|
||||
# Roadmap
|
||||
|
||||
This roadmap records current implementation status and deferred work for
|
||||
`distributor`. Implemented behavior is documented in the user, operator, and
|
||||
internal documentation listed below.
|
||||
|
||||
Canonical current-behavior docs:
|
||||
This directory contains only future, deferred, or aspirational work for
|
||||
`distributor`. Implemented behavior is documented in the current user,
|
||||
operator, internal, policy, integration, and example documentation:
|
||||
|
||||
- `README.md`
|
||||
- `docs/cli.md`
|
||||
@@ -16,338 +14,89 @@ Canonical current-behavior docs:
|
||||
- `docs/policy/`
|
||||
- `examples/`
|
||||
|
||||
Future, planned, or aspirational behavior belongs under `docs/roadmap/` until
|
||||
it is implemented.
|
||||
|
||||
## Current State
|
||||
|
||||
`distributor` is ready for routine use against producer pipelines using the
|
||||
implemented local, SSH/SFTP, and S3-compatible backends.
|
||||
|
||||
## Active Roadmap
|
||||
|
||||
The active roadmap implements the focused feature roadmaps in an order intended
|
||||
to minimize later rewrites. The sequence establishes shared CLI output before
|
||||
new CLI commands, establishes producer bundle APIs before manifest creation,
|
||||
and implements destination path mapping before link generation so URL
|
||||
construction does not need to be reworked later.
|
||||
|
||||
Each stage is sized for one implementation prompt. Future behavior stays under
|
||||
`docs/roadmap/` until the stage is implemented and current-behavior docs are
|
||||
updated.
|
||||
|
||||
### Stage 1: CLI Output Policy
|
||||
|
||||
Goal: implement the shared CLI output contract in
|
||||
`docs/roadmap/cli_output_policy.md`.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- add per-command `--format text|json` to current output-producing commands:
|
||||
`version`, `validate`, `inspect`, and `run`;
|
||||
- keep `text` as the default and preserve existing text output unless the
|
||||
policy explicitly requires a change;
|
||||
- implement the shared JSON envelope, warnings array, fatal setup error
|
||||
behavior, and `run` partial-result JSON with `ok: false`;
|
||||
- keep help and usage output text-only.
|
||||
|
||||
Documentation updates after implementation:
|
||||
|
||||
- update `docs/cli.md` with the shared `--format text|json` policy;
|
||||
- update `docs/operations.md` where JSON output materially improves automation
|
||||
workflows;
|
||||
- update `docs/troubleshooting.md` only for implemented JSON-mode recovery
|
||||
behavior.
|
||||
|
||||
Tests:
|
||||
|
||||
- CLI parsing accepts `--format text` and `--format json`;
|
||||
- invalid `--format` values are usage errors;
|
||||
- each JSON-capable command emits exactly one valid JSON document on success;
|
||||
- JSON-mode warnings appear in `warnings` and are not duplicated to stderr;
|
||||
- fatal setup errors write no JSON stdout and return non-zero;
|
||||
- `run --format json` emits partial-result JSON and exits non-zero when
|
||||
destination failures occur after planning begins.
|
||||
|
||||
Completion criteria: all current output-producing commands support the shared
|
||||
policy, text output remains backward compatible, and `go test ./...` passes.
|
||||
|
||||
### Stage 2: Public Bundle Package Core
|
||||
|
||||
Goal: implement the core and manifest-building parts of
|
||||
`docs/roadmap/public_bundle_package.md`.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- introduce `pkg/bundle` with the public source manifest model, schema version,
|
||||
digest logic, parsing, validation, explicit file-list building, scan-based
|
||||
building, and zero-`Created` defaulting to current UTC;
|
||||
- implement the locked Stage 2 exported API symbols defined in
|
||||
`docs/roadmap/public_bundle_package.md`;
|
||||
- update internal validation to consume the shared implementation without
|
||||
behavior drift;
|
||||
- keep destination state, publish planning, storage backends, transforms,
|
||||
notifications, and config internal.
|
||||
|
||||
Documentation updates after implementation:
|
||||
|
||||
- add package documentation for `pkg/bundle`;
|
||||
- update `README.md` and `docs/operations.md` only for implemented Go producer
|
||||
support;
|
||||
- update `docs/internal/bundle.md` if internal ownership changes.
|
||||
|
||||
Tests:
|
||||
|
||||
- public package tests cover explicit lists, scan mode, ordering, digest
|
||||
calculation, parsing, validation, unsafe paths, symlinks, and zero `Created`;
|
||||
- internal bundle tests continue to pass against the shared implementation;
|
||||
- public examples compile where practical.
|
||||
|
||||
Completion criteria: public and internal validation use one manifest contract,
|
||||
and tests prove identical semantics.
|
||||
|
||||
### Stage 3: Public Bundle Local Writer
|
||||
|
||||
Goal: implement the local bundle writer portion of
|
||||
`docs/roadmap/public_bundle_package.md`.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- add producer-side local bundle writing with staging, atomic filesystem
|
||||
operations where practical, and best-effort restore on overwrite failure;
|
||||
- implement the locked Stage 3 exported API symbols defined in
|
||||
`docs/roadmap/public_bundle_package.md`;
|
||||
- keep the writer filesystem-local and producer-focused;
|
||||
- do not expose distributor storage backends or publication behavior through
|
||||
the public package.
|
||||
|
||||
Documentation updates after implementation:
|
||||
|
||||
- document the writer API in `pkg/bundle`;
|
||||
- update producer workflow docs only for implemented behavior.
|
||||
|
||||
Tests:
|
||||
|
||||
- writer creates a complete valid local bundle through staged promotion;
|
||||
- writer output validates through distributor's normal validation path;
|
||||
- overwrite and failure behavior avoid leaving a completed bundle path without
|
||||
a valid manifest where practical.
|
||||
|
||||
Completion criteria: producers can create complete valid local bundles through
|
||||
the public API without hand-writing manifest files.
|
||||
|
||||
### Stage 4: Manifest Create CLI
|
||||
|
||||
Goal: implement `docs/roadmap/manifest_create.md`.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- add `distributor manifest create` over `pkg/bundle`;
|
||||
- support explicit-first `--file` selection, recursive scan fallback,
|
||||
`--overwrite`, optional `--created`, and shared `--format text|json`;
|
||||
- use temp-and-rename replacement for `manifest.json` where practical;
|
||||
- do not duplicate bundle contract logic in CLI or app code.
|
||||
|
||||
Documentation updates after implementation:
|
||||
|
||||
- update `docs/cli.md` with command syntax, flags, output, and examples;
|
||||
- update `docs/operations.md` with a producer workflow;
|
||||
- cross-reference `pkg/bundle` for Go producers.
|
||||
|
||||
Tests:
|
||||
|
||||
- command creates valid manifests for explicit and scanned files;
|
||||
- ordering, dotfile inclusion, metadata exclusion, symlink rejection, and
|
||||
overwrite behavior match the roadmap;
|
||||
- text and JSON output follow the shared CLI output policy.
|
||||
|
||||
Completion criteria: the command writes valid `manifest.json`, supports
|
||||
text/JSON output, and generated manifests validate through distributor.
|
||||
|
||||
### Stage 5: Remote Validate and Inspect
|
||||
|
||||
Goal: implement `docs/roadmap/remote_validate_inspect.md`.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- extend `validate` and `inspect` with mutually exclusive local-path and
|
||||
`--config --pipeline` modes;
|
||||
- reuse `run` source backend construction and secrets resolution;
|
||||
- support configured local, SSH, and S3 sources;
|
||||
- preserve `--pipeline` as required in config mode;
|
||||
- keep the commands source-only and do not open destinations.
|
||||
|
||||
Documentation updates after implementation:
|
||||
|
||||
- update `docs/cli.md` with local and config-driven syntax;
|
||||
- update `docs/operations.md` with remote validation examples;
|
||||
- update `docs/troubleshooting.md` for common configured-source failures.
|
||||
|
||||
Tests:
|
||||
|
||||
- existing local behavior remains unchanged;
|
||||
- config-mode validation and inspection work for local and fake/app-level
|
||||
source backends;
|
||||
- SSH and S3 behavior is covered by focused unit tests and existing opt-in
|
||||
integration patterns;
|
||||
- JSON output follows the shared CLI output policy.
|
||||
|
||||
Completion criteria: local behavior is stable, configured local/SSH/S3 sources
|
||||
can be validated and inspected, and no destination backend is opened.
|
||||
|
||||
### Stage 6: HTML Index Mode
|
||||
|
||||
Goal: implement `docs/roadmap/html_index_mode.md`.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- add `markdown_to_html.mode: index` with fixed `index.html` output;
|
||||
- keep `sidecar` as the default and existing explicit mode;
|
||||
- implement deterministic input selection through explicit input or exactly one
|
||||
manifest-listed Markdown file;
|
||||
- reject enabled Markdown-to-HTML transform config when `publish.html` is
|
||||
false;
|
||||
- record generated state metadata with `transform: markdown_to_html`.
|
||||
|
||||
Documentation updates after implementation:
|
||||
|
||||
- update `docs/config.md` with Markdown-to-HTML modes and input selection;
|
||||
- update `docs/integrations/markdown.md`;
|
||||
- update `docs/operations.md` with an implemented static-site example where
|
||||
useful.
|
||||
|
||||
Tests:
|
||||
|
||||
- sidecar behavior remains unchanged;
|
||||
- index mode handles explicit input, single Markdown fallback, ambiguous input,
|
||||
unsafe input, state metadata, dry-run, source-only publication, and output
|
||||
collisions.
|
||||
|
||||
Completion criteria: source publication, sidecar mode, index mode, state
|
||||
metadata, dry-run, and collision behavior are all covered.
|
||||
|
||||
### Stage 7: Latest Path Destinations
|
||||
|
||||
Goal: implement `docs/roadmap/latest_paths.md`.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- add destination-level `path_mapping.mode` with default `preserve_relative`
|
||||
and new `fixed`;
|
||||
- make destination bundle path mapping destination-local;
|
||||
- for fixed destinations, select only the newest discovered bundle per
|
||||
destination before planning writes;
|
||||
- add fixed-path dry-run warnings and summary counts;
|
||||
- allow `--force` for fixed backend roots while keeping deletion bounded to the
|
||||
configured backend root.
|
||||
|
||||
Documentation updates after implementation:
|
||||
|
||||
- update `docs/config.md` with `path_mapping`;
|
||||
- update `docs/operations.md` with archive-plus-latest fan-out examples;
|
||||
- update `docs/cli.md` if dry-run output gains fixed-path indicators.
|
||||
|
||||
Tests:
|
||||
|
||||
- omitted and explicit `preserve_relative` match existing behavior;
|
||||
- fixed destinations publish at local, SSH, and S3 backend roots;
|
||||
- newest-only selection is deterministic;
|
||||
- older discovered bundles are not planned or written to fixed destinations;
|
||||
- dry-run reports candidate count, selected bundle, and destructive replacement
|
||||
warnings;
|
||||
- force remains bounded at the backend root.
|
||||
|
||||
Completion criteria: fixed destination roots work across implemented backends,
|
||||
archive-style destinations remain unchanged, and destructive behavior is clear
|
||||
in dry-run output.
|
||||
|
||||
### Stage 8: Link Generation Support
|
||||
|
||||
Goal: implement `docs/roadmap/link_generation.md`.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- add destination-level `links.base_url` and `links.primary`;
|
||||
- generate per-output and primary URLs using URL semantics, not filesystem or
|
||||
storage joins;
|
||||
- handle `index.html` as a directory-style URL;
|
||||
- store optional URL metadata under destination state schema version 1 while
|
||||
pre-release;
|
||||
- expose links through JSON command output only where supported by the CLI
|
||||
output policy.
|
||||
|
||||
Documentation updates after implementation:
|
||||
|
||||
- update `docs/config.md` with destination `links` fields;
|
||||
- update `docs/operations.md` with static-site URL examples;
|
||||
- update `docs/internal/state.md` for URL metadata;
|
||||
- update `docs/cli.md` only for implemented CLI link output.
|
||||
|
||||
Tests:
|
||||
|
||||
- config validation accepts valid links and rejects invalid schemes, query
|
||||
strings, and fragments;
|
||||
- nested bundle paths and fixed latest paths generate correct URLs;
|
||||
- `index.html` omits the filename;
|
||||
- primary selection is deterministic;
|
||||
- destinations without `links` produce no URL metadata.
|
||||
|
||||
Completion criteria: URL generation respects archive and fixed path mapping,
|
||||
state records optional URL metadata when configured, and unconfigured
|
||||
destinations remain unchanged.
|
||||
|
||||
### Stage 9: Roadmap Closeout
|
||||
|
||||
Goal: remove completed roadmap drift after Stages 1-8 are implemented.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- remove or rewrite completed roadmap files whose behavior is fully documented
|
||||
in current docs;
|
||||
- ensure current docs describe implemented behavior;
|
||||
- keep `docs/roadmap/` focused only on remaining future work.
|
||||
|
||||
Documentation updates after implementation:
|
||||
|
||||
- update `README.md`, `docs/cli.md`, `docs/config.md`,
|
||||
`docs/operations.md`, `docs/troubleshooting.md`, `docs/internal/`,
|
||||
`docs/integrations/markdown.md`, and examples only where implemented
|
||||
behavior requires it.
|
||||
|
||||
Tests:
|
||||
|
||||
- run documentation consistency searches for completed-feature language that
|
||||
still appears only as future work;
|
||||
- run focused tests for any examples or docs backed by tests;
|
||||
- run `go test ./...` if behavior docs and examples changed with code.
|
||||
|
||||
Completion criteria: completed features are documented as current behavior, and
|
||||
`docs/roadmap/` contains only future or deferred work.
|
||||
|
||||
## Deferred Work
|
||||
|
||||
These items are not implemented and should stay out of current-behavior docs
|
||||
until a roadmap entry is selected and implemented:
|
||||
|
||||
- external notification adapters;
|
||||
- warning-only digest mismatch handling;
|
||||
- additional auth mechanisms beyond the implemented SSH and S3 credential
|
||||
paths;
|
||||
- compatibility parsing for legacy SSH URI config;
|
||||
- broad recursive destination deletion outside managed bundle paths;
|
||||
- concurrent fan-out publishing;
|
||||
- streaming, resumable, or multipart S3 uploads;
|
||||
- cloud-provider-specific IAM integration docs;
|
||||
- repository-managed packaging, release, and deployment automation.
|
||||
`distributor` currently supports local, SSH/SFTP, and S3-compatible source and
|
||||
destination backends; producer bundle creation through `pkg/bundle` and
|
||||
`distributor manifest create`; configured source validation and inspection;
|
||||
Markdown sidecar and `index.html` publication; archive and fixed destination
|
||||
path mapping; destination link metadata; shared text/JSON CLI output; and
|
||||
managed destination replacement behavior.
|
||||
|
||||
## Future Work
|
||||
|
||||
These items are not implemented. They should not be documented as current
|
||||
behavior outside `docs/roadmap/` unless a future implementation adds them.
|
||||
|
||||
### CLI And Status Output
|
||||
|
||||
- Add a root-global output flag only if the command parser is later refactored
|
||||
around shared root options.
|
||||
- Add output formats beyond `text` and `json` only if a concrete consumer
|
||||
requires them.
|
||||
- Add a versioned JSON schema reference after the first JSON-capable release.
|
||||
- Add destination-state inspection behind an explicit flag such as
|
||||
`--with-destinations` if operators need fan-out status diagnostics from
|
||||
`inspect`.
|
||||
- Add additional status or inspection presentation for destination primary
|
||||
links beyond the current `run --format json` result model.
|
||||
|
||||
### Producer Workflows
|
||||
|
||||
- Add a no-write manifest creation mode, such as writing manifest JSON to
|
||||
stdout, if producer pipelines need to capture manifests directly.
|
||||
- Add broader producer workflow helpers, such as richer ignore rules or
|
||||
template scaffolding, if real producer use cases require them.
|
||||
- Add remote or storage-backed producer writers only if producer applications
|
||||
need to assemble bundles outside the local filesystem.
|
||||
|
||||
### Publication And Transform Behavior
|
||||
|
||||
- Add a separate collection or site-index transform if distributor needs
|
||||
multi-page aggregation.
|
||||
- Add richer transform metadata only if future state consumers need more than
|
||||
the transform name and output path.
|
||||
- Add custom HTML index output names only if fixed `index.html` is too limiting
|
||||
for real deployments.
|
||||
- Add richer fixed-destination source selection policies if deployments need
|
||||
something other than newest-by-`created`.
|
||||
- Add stricter handling for equal latest timestamps if timestamp ties become
|
||||
common in producer workflows.
|
||||
- Add higher-level status or approval workflows for fixed-root replacements if
|
||||
dry-run output is not enough operational protection.
|
||||
- Add richer link policies only if `auto`, `html`, and `source` prove
|
||||
insufficient.
|
||||
|
||||
### State And Compatibility
|
||||
|
||||
- Define a post-release destination state schema bump policy before introducing
|
||||
materially incompatible state changes.
|
||||
- Add warning-only digest mismatch handling only if an operator workflow needs
|
||||
publication to continue after validation failures.
|
||||
- Add compatibility parsing for legacy SSH URI config only if migration support
|
||||
is required.
|
||||
|
||||
### Backends, Security, And Deployment
|
||||
|
||||
- Add authentication mechanisms beyond the implemented SSH agent/key and S3
|
||||
credential paths only when a concrete backend workflow requires them.
|
||||
- Add broad recursive destination deletion outside managed bundle paths only if
|
||||
a future design can preserve the current safety boundary.
|
||||
- Add concurrent fan-out publishing only if runtime profiling shows it is
|
||||
needed.
|
||||
- Add streaming, resumable, or multipart S3 uploads only if object sizes make
|
||||
the current write path insufficient.
|
||||
- Add cloud-provider-specific IAM integration docs only when the repository
|
||||
includes tested provider-specific behavior.
|
||||
- Add repository-managed packaging, release, and deployment automation when the
|
||||
release process is ready to be standardized.
|
||||
|
||||
## Roadmap Maintenance
|
||||
|
||||
When adding future roadmap work:
|
||||
|
||||
- describe user-visible behavior and safety boundaries;
|
||||
- define which existing docs must change after implementation;
|
||||
- define which current docs must change after implementation;
|
||||
- keep examples secret-free and runnable or clearly environment-gated;
|
||||
- avoid workflow labels in production code, tests, config fields, and user
|
||||
documentation;
|
||||
- keep workflow labels out of production code, tests, config fields, and
|
||||
user-facing documentation;
|
||||
- run focused tests for the changed behavior and `go test ./...` for
|
||||
cross-package changes.
|
||||
|
||||
@@ -1,212 +0,0 @@
|
||||
# Roadmap: Latest Path Destinations
|
||||
|
||||
## Purpose
|
||||
|
||||
Support stable "latest" publication paths as normal fan-out destinations.
|
||||
|
||||
A latest destination republishes the current winning source bundle to a fixed
|
||||
destination path such as `/weather/latest/`, while archive destinations preserve
|
||||
source-relative bundle paths such as `/weather/daily/brentwood/2026-06-01/`.
|
||||
|
||||
This feature should be implemented before link generation so URL construction
|
||||
can use final destination path semantics. It is operationally destructive and
|
||||
benefits from index-mode HTML, although it must not require index mode or link
|
||||
generation.
|
||||
|
||||
## Current Implementation Grounding
|
||||
|
||||
Current run behavior computes the destination bundle path as the source bundle
|
||||
path relative to the source root. Publication then writes selected outputs and
|
||||
`.distributor.json` below that destination bundle path.
|
||||
|
||||
Backend roots differ by backend:
|
||||
|
||||
- local and SSH/SFTP use configured `path` as the backend root;
|
||||
- S3 uses configured bucket plus optional `prefix` as the backend root.
|
||||
|
||||
Destination comparison uses `.distributor.json` at the destination bundle path.
|
||||
Replacement and skip decisions are based on the normalized source manifest
|
||||
recorded in destination state. Unmanaged non-empty destinations fail unless
|
||||
explicit force behavior is requested.
|
||||
|
||||
## Goals
|
||||
|
||||
- Add destination-local path mapping.
|
||||
- Preserve source-relative destination paths as the default.
|
||||
- Add fixed path mapping for latest-style destinations.
|
||||
- For fixed destinations, select only the newest discovered source bundle for
|
||||
publication.
|
||||
- Make fixed mapping work for local, SSH/SFTP, and S3 backends.
|
||||
- Continue using normal destination state comparison and replacement rules.
|
||||
- Keep link generation and index-mode HTML complementary, not required.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not add symlink-based latest behavior.
|
||||
- Do not add feed or index generation.
|
||||
- Do not change source manifest schema.
|
||||
- Do not make producers responsible for latest publication.
|
||||
- Do not add domain-specific latest selection rules.
|
||||
|
||||
## Configuration Shape
|
||||
|
||||
Add destination-level path mapping:
|
||||
|
||||
```yaml
|
||||
path_mapping:
|
||||
mode: preserve_relative
|
||||
```
|
||||
|
||||
Modes:
|
||||
|
||||
- `preserve_relative`: existing/default behavior.
|
||||
- `fixed`: publish selected outputs directly at the destination backend root.
|
||||
|
||||
Example fixed destination:
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: latest-html
|
||||
backend: ssh
|
||||
host: web.example.com
|
||||
user: deploy
|
||||
path: /srv/www/weather/latest
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
```
|
||||
|
||||
For S3, the fixed destination root is the configured bucket plus `prefix`.
|
||||
|
||||
## Path Mapping Semantics
|
||||
|
||||
`preserve_relative`:
|
||||
|
||||
```text
|
||||
destination bundle path = source-root-relative bundle path
|
||||
```
|
||||
|
||||
`fixed`:
|
||||
|
||||
```text
|
||||
destination bundle path = ""
|
||||
```
|
||||
|
||||
That empty logical destination bundle path means the destination backend root.
|
||||
Storage writes still use normal backend-rooted logical paths for outputs and
|
||||
state.
|
||||
|
||||
## Newest Bundle Selection
|
||||
|
||||
`preserve_relative` destinations should continue publishing every selected
|
||||
source bundle independently.
|
||||
|
||||
`fixed` destinations should publish only one bundle per run: the newest
|
||||
discovered source bundle selected for that destination. Newest selection should
|
||||
be deterministic:
|
||||
|
||||
1. choose the bundle with the greatest source manifest `created` timestamp;
|
||||
2. if multiple bundles have the same greatest `created` timestamp, use the
|
||||
source-root-relative bundle path ascending as a tie-breaker.
|
||||
|
||||
The selected bundle then uses the existing destination comparison, skip,
|
||||
replacement, cleanup, and force rules. Older discovered bundles should not be
|
||||
planned or written to that fixed destination in the same run.
|
||||
|
||||
Dry-run output should identify the number of fixed-destination candidates and
|
||||
the selected source bundle so operators can see which bundle would become
|
||||
latest.
|
||||
|
||||
## Safety Rules
|
||||
|
||||
Fixed path mapping is more destructive than archive-style publication because
|
||||
newer bundles replace prior contents at the same destination path.
|
||||
|
||||
Required behavior:
|
||||
|
||||
- dry-run output must identify fixed path mapping and planned replacement;
|
||||
- dry-run output must include an extra fixed-path warning or summary count when
|
||||
destructive replacement is possible;
|
||||
- replacement should delete only managed outputs recorded in valid destination
|
||||
state when possible;
|
||||
- unmanaged non-empty fixed destinations must fail unless force is explicitly
|
||||
requested;
|
||||
- `--force` may be used when fixed mapping targets the backend root, but force
|
||||
replacement must remain bounded to the configured destination backend root and
|
||||
must never delete above it;
|
||||
- fixed mapping to the backend root requires especially clear dry-run reporting.
|
||||
|
||||
## Relationship To Other Roadmaps
|
||||
|
||||
HTML index mode is useful for fixed web destinations because `index.html`
|
||||
supports stable directory-style URLs, but sidecar HTML and source-only outputs
|
||||
should still work.
|
||||
|
||||
Link generation should use fixed path semantics so latest URLs are based on the
|
||||
fixed destination root, not the original source-relative archive path.
|
||||
|
||||
Producer manifest creation and remote validation are independent of destination
|
||||
path mapping.
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
1. Add destination-level `path_mapping` config validation with
|
||||
`preserve_relative` as the default and `fixed` as the new mode.
|
||||
2. Refactor run planning so destination bundle path mapping is destination-local
|
||||
and can be evaluated before writes.
|
||||
3. Add fixed-destination newest selection so each fixed destination receives
|
||||
only the newest discovered source bundle.
|
||||
4. Integrate fixed mapping with destination comparison, managed deletion, force
|
||||
replacement, and dry-run reporting.
|
||||
5. Update local, SSH, and S3 tests to verify fixed root behavior under each
|
||||
backend root model.
|
||||
6. Update current-behavior documentation after implementation.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Suggested coverage:
|
||||
|
||||
- omitted `path_mapping` preserves existing behavior;
|
||||
- `preserve_relative` explicitly matches existing behavior;
|
||||
- `fixed` publishes outputs and `.distributor.json` at the backend root;
|
||||
- fixed destinations select only the newest discovered bundle;
|
||||
- fixed destination tie-break behavior is deterministic;
|
||||
- older discovered bundles are not planned or written to fixed destinations;
|
||||
- newer source replaces older managed fixed destination state;
|
||||
- older source skips newer fixed destination state;
|
||||
- unmanaged non-empty fixed destination fails without force;
|
||||
- dry-run reports fixed path mapping, candidate count, selected bundle, and
|
||||
destructive replacement warnings clearly;
|
||||
- `--force` remains bounded when fixed mapping targets the backend root;
|
||||
- S3 fixed mapping respects bucket plus prefix as backend root.
|
||||
|
||||
## Documentation Updates After Implementation
|
||||
|
||||
- Update `docs/config.md` with `path_mapping`.
|
||||
- Update `docs/operations.md` with archive-plus-latest fan-out examples.
|
||||
- Update `docs/cli.md` if dry-run output gains fixed-path indicators.
|
||||
- Add an environment-safe example only after behavior is implemented.
|
||||
|
||||
Keep this roadmap under `docs/roadmap/` until implemented.
|
||||
|
||||
## Decisions
|
||||
|
||||
- `--force` is allowed when fixed mapping targets the backend root, but dry-run
|
||||
and run output must make the fixed-root replacement explicit and force remains
|
||||
bounded to the configured backend root.
|
||||
- Fixed destinations publish only the newest discovered bundle, rather than
|
||||
processing every discovered bundle and letting later writes replace earlier
|
||||
writes.
|
||||
- Dry-run includes an extra warning or summary count for fixed-path destructive
|
||||
replacements.
|
||||
|
||||
## Future Work
|
||||
|
||||
- Add richer source selection policies if deployments need something other than
|
||||
newest-by-`created` for fixed destinations.
|
||||
- Consider stricter ambiguity handling for equal latest timestamps if real
|
||||
producer workflows make timestamp ties common.
|
||||
- Add higher-level status or approval workflows for fixed-root replacements if
|
||||
dry-run output is not enough operational protection.
|
||||
@@ -1,179 +0,0 @@
|
||||
# Roadmap: Link Generation Support
|
||||
|
||||
## Purpose
|
||||
|
||||
Add destination-aware link generation so distributor can record human-usable URLs
|
||||
for published artifacts.
|
||||
|
||||
This feature should be implemented after destination path mapping so generated
|
||||
URLs use the final archive or fixed destination path semantics. It prepares the
|
||||
project for notification adapters, richer inspect/status output, static-site UX,
|
||||
and future feed generation.
|
||||
|
||||
## Current Implementation Grounding
|
||||
|
||||
Current publish planning produces `publish.Output` values with destination path,
|
||||
source path, kind, transform, SHA-256, size, and optional generated data.
|
||||
|
||||
Current `.distributor.json` output records contain path, kind, source path,
|
||||
transform, SHA-256, and size. They do not contain URL metadata, and destination
|
||||
state has no top-level links block.
|
||||
|
||||
Current destination bundle paths preserve the source-root-relative bundle path.
|
||||
Latest/fixed path mapping is not implemented yet.
|
||||
|
||||
## Goals
|
||||
|
||||
- Add optional destination-level link configuration.
|
||||
- Generate per-output URLs when a destination has `links.base_url`.
|
||||
- Record generated URLs in destination state when configured.
|
||||
- Select a deterministic primary URL when possible.
|
||||
- Keep link generation destination-local and independent of producer manifests.
|
||||
- Do not infer public URLs from backend configuration automatically.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not deliver notifications.
|
||||
- Do not require every destination to expose URLs.
|
||||
- Do not add static-site indexes or feeds.
|
||||
- Do not add latest path mapping.
|
||||
- Do not change source manifest schema.
|
||||
|
||||
## Configuration Shape
|
||||
|
||||
Add optional destination-level config:
|
||||
|
||||
```yaml
|
||||
links:
|
||||
base_url: https://weather.example.com
|
||||
primary: auto
|
||||
```
|
||||
|
||||
Semantics:
|
||||
|
||||
- `links.base_url`: absolute HTTP or HTTPS URL corresponding to the destination
|
||||
backend root.
|
||||
- `links.primary`: optional primary-link selection policy.
|
||||
|
||||
Initial `primary` policies:
|
||||
|
||||
- `auto`: choose the best available output;
|
||||
- `html`: prefer generated HTML outputs;
|
||||
- `source`: prefer copied source outputs.
|
||||
|
||||
If `links` is absent, no URL metadata is generated.
|
||||
|
||||
## URL Construction
|
||||
|
||||
Construct URLs from:
|
||||
|
||||
1. `links.base_url`;
|
||||
2. destination bundle path relative to the destination backend root;
|
||||
3. output path relative to the destination bundle path.
|
||||
|
||||
Use URL path joining and escaping rules, not filesystem or storage path joins.
|
||||
Preserve any path prefix in `base_url`. Reject `base_url` values with query
|
||||
strings or fragments.
|
||||
|
||||
For `index.html`, generate a directory-style URL by omitting the filename:
|
||||
|
||||
```text
|
||||
https://weather.example.com/daily/brentwood/2026-06-01/
|
||||
```
|
||||
|
||||
For other outputs, include the output filename:
|
||||
|
||||
```text
|
||||
https://weather.example.com/daily/brentwood/2026-06-01/report.html
|
||||
```
|
||||
|
||||
`index.html` recognition should apply to any output path ending in
|
||||
`/index.html` or exactly `index.html`.
|
||||
|
||||
## Primary Link Selection
|
||||
|
||||
For `primary: auto`, select in deterministic publish-plan order:
|
||||
|
||||
1. any `index.html` output;
|
||||
2. first generated HTML output;
|
||||
3. first source output;
|
||||
4. no primary URL.
|
||||
|
||||
For `primary: html`, prefer generated HTML outputs. For `primary: source`,
|
||||
prefer source outputs. If no output matches the policy, record no primary link
|
||||
rather than failing publication.
|
||||
|
||||
## Destination State
|
||||
|
||||
When links are configured, destination state should record:
|
||||
|
||||
- optional per-output URL metadata;
|
||||
- optional top-level primary URL metadata.
|
||||
|
||||
Destinations without `links` should continue producing state without URL
|
||||
metadata.
|
||||
|
||||
Because distributor is still pre-release, URL metadata should be added as
|
||||
optional state schema version 1 fields rather than introducing destination state
|
||||
schema version 2 for this feature. Post-release, materially richer state
|
||||
semantics should use an explicit schema versioning policy.
|
||||
|
||||
## Relationship To Other Roadmaps
|
||||
|
||||
HTML index mode is not required, but link generation should treat `index.html`
|
||||
as the preferred URL shape when present.
|
||||
|
||||
Latest path destinations should provide the destination bundle path mapping
|
||||
needed to build fixed/latest URLs correctly.
|
||||
|
||||
Producer-facing manifest creation and remote validation are independent of URL
|
||||
metadata.
|
||||
|
||||
CLI display of generated primary links should follow
|
||||
`docs/roadmap/cli_output_policy.md` so link output is exposed through the shared
|
||||
`--format text|json` model rather than one-off command output.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Suggested coverage:
|
||||
|
||||
- config validation accepts absent `links`;
|
||||
- config validation accepts valid HTTP and HTTPS base URLs;
|
||||
- config validation rejects invalid schemes, query strings, and fragments;
|
||||
- nested bundle paths generate correct URLs;
|
||||
- `index.html` URLs omit the filename;
|
||||
- non-index URLs include filenames;
|
||||
- primary selection is deterministic;
|
||||
- state records URLs when configured;
|
||||
- destinations without `links` produce no URL metadata.
|
||||
|
||||
## Documentation Updates After Implementation
|
||||
|
||||
- Update `docs/config.md` with destination `links` fields.
|
||||
- Update `docs/operations.md` with static-site URL examples.
|
||||
- Update `docs/internal/state.md` if destination state changes.
|
||||
- Update `docs/cli.md` only if run or inspect output displays links.
|
||||
|
||||
Keep this roadmap under `docs/roadmap/` until implemented.
|
||||
|
||||
## Decisions
|
||||
|
||||
- URL metadata is added as optional destination state schema version 1 fields
|
||||
while distributor remains pre-release.
|
||||
- v1 implements `links.primary` so future notification and operator-output
|
||||
features have one canonical primary-link selection policy.
|
||||
- `run` and `inspect` output should not display generated primary links
|
||||
immediately. Link display should wait for
|
||||
`docs/roadmap/cli_output_policy.md` or another explicit inspect/status output
|
||||
mode.
|
||||
|
||||
## Future Work
|
||||
|
||||
- Define a post-release destination state schema bump policy before introducing
|
||||
materially incompatible state changes.
|
||||
- Add CLI display of primary links through an explicit output mode or status
|
||||
command, preferably the shared `--format text|json` policy in
|
||||
`docs/roadmap/cli_output_policy.md`, rather than changing routine `run`
|
||||
output opportunistically.
|
||||
- Add richer link policies only if `auto`, `html`, and `source` prove
|
||||
insufficient.
|
||||
@@ -1,173 +0,0 @@
|
||||
# Roadmap: `distributor manifest create`
|
||||
|
||||
## Purpose
|
||||
|
||||
Add a producer-facing CLI command that creates a valid source `manifest.json` for
|
||||
a local bundle directory.
|
||||
|
||||
This command depends on the public `pkg/bundle` package. It should be useful
|
||||
for shell scripts and non-Go producers while sharing behavior with Go producers
|
||||
through the public package.
|
||||
|
||||
## Current Implementation Grounding
|
||||
|
||||
The current CLI has top-level `version`, `run`, `validate`, and `inspect`
|
||||
commands. `validate` and `inspect` currently accept local paths only.
|
||||
|
||||
The manifest contract is implemented in `internal/bundle`: source manifests have
|
||||
`schema_version`, `id`, `digest`, `created`, and ordered `files[]` entries with
|
||||
`path`, `sha256`, and `size`. Validation already enforces path safety,
|
||||
reserved distributor metadata paths, digest format, duplicate file paths, file
|
||||
sizes, per-file SHA-256, and the canonical bundle digest.
|
||||
|
||||
This command should not add a second implementation of those rules. Once
|
||||
`pkg/bundle` exists, `manifest create` should call it.
|
||||
|
||||
## Goals
|
||||
|
||||
- Add a CLI command that writes `manifest.json` for a local bundle directory.
|
||||
- Reuse public producer-side manifest creation and validation behavior.
|
||||
- Produce deterministic manifests.
|
||||
- Preserve caller-provided file order when explicit files are provided.
|
||||
- Offer convenient recursive scanning when explicit files are omitted.
|
||||
- Refuse to overwrite an existing manifest unless explicitly requested.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not publish or distribute bundles.
|
||||
- Do not write destination `.distributor.json` state.
|
||||
- Do not add config-file dependencies.
|
||||
- Do not add domain-specific metadata.
|
||||
- Do not expose publish, storage, transform, or destination internals.
|
||||
|
||||
## CLI Shape
|
||||
|
||||
Proposed syntax:
|
||||
|
||||
```sh
|
||||
distributor manifest create <bundle-path> --id <bundle-id>
|
||||
```
|
||||
|
||||
Optional flags:
|
||||
|
||||
```sh
|
||||
--file <path> Include a bundle-relative file; repeatable.
|
||||
--created <time> RFC3339 source created timestamp.
|
||||
--overwrite Replace an existing manifest.json.
|
||||
```
|
||||
|
||||
The first implementation should always write `manifest.json` in the selected
|
||||
bundle directory. It should not include a stdout/no-write mode or JSON summary
|
||||
mode until `docs/roadmap/cli_output_policy.md` is implemented or selected for
|
||||
implementation.
|
||||
|
||||
Examples:
|
||||
|
||||
```sh
|
||||
distributor manifest create ./bundle --id weather.daily.2026-06-01
|
||||
|
||||
distributor manifest create ./bundle \
|
||||
--id weather.daily.2026-06-01 \
|
||||
--created 2026-06-01T11:00:00Z \
|
||||
--file report.md \
|
||||
--file summary.txt
|
||||
```
|
||||
|
||||
## File Selection
|
||||
|
||||
Use explicit-first behavior:
|
||||
|
||||
- if one or more `--file` flags are provided, use exactly those files in flag
|
||||
order;
|
||||
- if no `--file` flags are provided, scan the bundle directory recursively and
|
||||
sort files lexically by slash-separated relative path.
|
||||
|
||||
Scanning should include ordinary regular files, including dotfiles, except:
|
||||
|
||||
- `manifest.json`;
|
||||
- `.distributor.json`;
|
||||
- directories;
|
||||
- symlinks and other non-regular entries.
|
||||
|
||||
Explicit file paths must be relative, clean, slash-separated or normalized to
|
||||
slash-separated paths, confined to the bundle root, and listed files must be
|
||||
regular files. Symlinks should be rejected to match current source validation.
|
||||
|
||||
## Manifest Creation Behavior
|
||||
|
||||
The command should:
|
||||
|
||||
1. resolve the local bundle root;
|
||||
2. determine the selected file list;
|
||||
3. build file records with SHA-256 and size;
|
||||
4. compute the canonical bundle digest;
|
||||
5. set `schema_version` to the current source manifest version;
|
||||
6. set `id` from `--id`;
|
||||
7. set `created` from `--created` or the package's selected default behavior;
|
||||
8. refuse to replace `manifest.json` unless `--overwrite` is set;
|
||||
9. write `manifest.json`, using temp-and-rename replacement where practical;
|
||||
10. reload or validate the generated manifest before reporting success.
|
||||
|
||||
Success output should be concise:
|
||||
|
||||
```text
|
||||
created manifest.json
|
||||
bundle: weather.daily.2026-06-01
|
||||
files: 2
|
||||
digest: sha256:...
|
||||
```
|
||||
|
||||
## Relationship To Other Roadmaps
|
||||
|
||||
`pkg/bundle` is the preferred underlying implementation. If `manifest create`
|
||||
is implemented first, its reusable manifest-building logic should be structured
|
||||
so it can move into `pkg/bundle` without changing command behavior.
|
||||
|
||||
Remote `validate` and `inspect` should validate bundles after creation but do
|
||||
not need to participate in manifest writing.
|
||||
|
||||
HTML index mode, link generation, and latest paths are publication features and
|
||||
must not affect source manifest generation.
|
||||
|
||||
JSON summary output should follow `docs/roadmap/cli_output_policy.md` rather
|
||||
than introducing a command-specific `--json` flag or output envelope.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Suggested coverage:
|
||||
|
||||
- creates a manifest for a simple local bundle;
|
||||
- preserves explicit `--file` order;
|
||||
- scan mode sorts files deterministically;
|
||||
- scan mode includes dotfiles and excludes distributor metadata files;
|
||||
- rejects symlinks, unsafe paths, missing explicit files, and non-regular files;
|
||||
- refuses overwrite without `--overwrite`;
|
||||
- overwrites only when `--overwrite` is set;
|
||||
- generated manifests validate with distributor validation;
|
||||
- CLI help and argument validation match existing CLI style.
|
||||
|
||||
## Documentation Updates After Implementation
|
||||
|
||||
- Update `docs/cli.md` with command syntax and examples.
|
||||
- Update `docs/operations.md` with a producer workflow.
|
||||
- Cross-reference `pkg/bundle` for Go producers once available.
|
||||
- Add examples only if they are maintained and load/test friendly.
|
||||
|
||||
Keep this roadmap under `docs/roadmap/` until implemented.
|
||||
|
||||
## Decisions
|
||||
|
||||
- v1 does not include `--stdout` or no-write behavior. The command creates or
|
||||
replaces the bundle's `manifest.json`.
|
||||
- `--overwrite` is required to replace an existing manifest and should use
|
||||
temp-and-rename writes where practical.
|
||||
- v1 does not include JSON summary output. Human-oriented success output remains
|
||||
consistent with the current CLI unless this work is implemented together with
|
||||
`docs/roadmap/cli_output_policy.md`.
|
||||
|
||||
## Future Work
|
||||
|
||||
- Add `--stdout` or another no-write mode if producer pipelines need to capture
|
||||
manifest JSON directly.
|
||||
- Add JSON output through the CLI-wide `--format text|json` policy in
|
||||
`docs/roadmap/cli_output_policy.md`.
|
||||
@@ -1,284 +0,0 @@
|
||||
# Roadmap: Public Bundle Manifest Package
|
||||
|
||||
## Purpose
|
||||
|
||||
Expose a small public Go package that producer applications can import to create
|
||||
valid distributor source bundle manifests.
|
||||
|
||||
The package should encode the producer-side source bundle contract without
|
||||
exposing distributor's publication, storage, transform, destination state,
|
||||
notification, or config internals.
|
||||
|
||||
## Current Implementation Grounding
|
||||
|
||||
The current implementation keeps source bundle behavior in `internal/bundle`:
|
||||
|
||||
- `Manifest` and `ManifestFile` model `manifest.json`;
|
||||
- `ParseManifest` parses JSON and RFC3339 `created` timestamps;
|
||||
- `ValidateManifest` owns schema version, digest format, file path, duplicate,
|
||||
size, and bundle digest validation;
|
||||
- `FileDigest`, `BundleDigest`, and `CanonicalFilePayload` define digest
|
||||
behavior;
|
||||
- source validation rejects unsafe paths, reserved distributor metadata paths,
|
||||
non-file entries, size mismatches, SHA-256 mismatches, and bundle digest
|
||||
mismatches.
|
||||
|
||||
Producers cannot import `internal/bundle`, so Go producers currently need to
|
||||
duplicate this contract or shell out to future CLI tooling.
|
||||
|
||||
## Goals
|
||||
|
||||
- Provide a stable producer-facing Go API for manifest creation and validation.
|
||||
- Reuse the same source manifest, digest, path safety, and RFC3339 behavior used
|
||||
by distributor validation.
|
||||
- Keep the public API intentionally small and producer-only.
|
||||
- Include a safe bundle writer so producers can create complete local bundle
|
||||
directories without hand-rolling manifest-write and staging behavior.
|
||||
- Make the future `distributor manifest create` command a thin wrapper over this
|
||||
package.
|
||||
- Avoid exposing destination state, publish planning, storage backends,
|
||||
transforms, notifications, or config.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not expose the distributor runner or publication workflow as public API.
|
||||
- Do not expose storage backends or destination `.distributor.json` state.
|
||||
- Do not add domain-specific manifest metadata.
|
||||
- Do not require non-Go producers to use Go APIs.
|
||||
- Do not implement latest paths, link generation, transforms, or notification
|
||||
behavior in this package.
|
||||
|
||||
## Package Boundary
|
||||
|
||||
Use `pkg/bundle` as the public package name.
|
||||
|
||||
The package should own only producer-side source bundle concerns:
|
||||
|
||||
- manifest model and schema version constant;
|
||||
- file digest and bundle digest calculation;
|
||||
- manifest building from producer files;
|
||||
- manifest JSON load/write helpers;
|
||||
- manifest and bundle validation;
|
||||
- source path safety matching distributor validation.
|
||||
|
||||
Prefer options structs over long positional functions so future additive
|
||||
behavior can be introduced without avoidable API churn.
|
||||
|
||||
## Initial Exported API
|
||||
|
||||
The initial `pkg/bundle` API is locked to the following exported constants,
|
||||
types, and functions. Implementation should not rename, remove, or reshape
|
||||
these public symbols during the initial implementation pass.
|
||||
|
||||
```go
|
||||
const ManifestName = "manifest.json"
|
||||
const SchemaVersion = 1
|
||||
|
||||
type Manifest struct {
|
||||
SchemaVersion int `json:"schema_version"`
|
||||
ID string `json:"id"`
|
||||
Digest string `json:"digest"`
|
||||
Created time.Time `json:"created"`
|
||||
Files []ManifestFile `json:"files"`
|
||||
}
|
||||
|
||||
type ManifestFile struct {
|
||||
Path string `json:"path"`
|
||||
SHA256 string `json:"sha256"`
|
||||
Size int64 `json:"size"`
|
||||
}
|
||||
|
||||
type BuildOptions struct {
|
||||
Root string
|
||||
ID string
|
||||
Created time.Time
|
||||
Files []string
|
||||
Scan bool
|
||||
}
|
||||
|
||||
type WriteManifestOptions struct {
|
||||
Overwrite bool
|
||||
}
|
||||
|
||||
type BundleFile struct {
|
||||
SourcePath string
|
||||
Path string
|
||||
}
|
||||
|
||||
type WriteBundleOptions struct {
|
||||
Root string
|
||||
ID string
|
||||
Created time.Time
|
||||
Files []BundleFile
|
||||
Overwrite bool
|
||||
}
|
||||
|
||||
func ParseManifest(data []byte) (Manifest, error)
|
||||
func MarshalManifest(manifest Manifest) ([]byte, error)
|
||||
func LoadManifest(root string) (Manifest, error)
|
||||
func WriteManifest(root string, manifest Manifest, opts WriteManifestOptions) error
|
||||
func BuildManifest(opts BuildOptions) (Manifest, error)
|
||||
func ValidateManifest(manifest Manifest) error
|
||||
func ValidateBundle(root string, manifest Manifest) error
|
||||
func WriteBundle(opts WriteBundleOptions) (Manifest, error)
|
||||
func ValidateSourcePath(path string) error
|
||||
func FileDigest(data []byte) string
|
||||
func BundleDigest(files []ManifestFile) string
|
||||
func CanonicalFilePayload(files []ManifestFile) string
|
||||
```
|
||||
|
||||
## API Semantics
|
||||
|
||||
`BuildManifest` requires `Root`, `ID`, and exactly one file-selection mode:
|
||||
explicit `Files` or `Scan: true`.
|
||||
|
||||
Explicit `Files` preserve caller order. `Scan: true` recursively scans `Root`,
|
||||
includes regular files including dotfiles, excludes `manifest.json` and
|
||||
`.distributor.json`, rejects symlinks, and sorts by slash-separated relative
|
||||
path.
|
||||
|
||||
Zero `Created` values default to the current UTC time. All public path fields
|
||||
use slash-separated bundle-relative paths.
|
||||
|
||||
`MarshalManifest` validates before marshaling and emits deterministic JSON with
|
||||
fixed field order and a trailing newline.
|
||||
|
||||
`WriteManifest` writes `manifest.json`; it fails if the file exists unless
|
||||
`WriteManifestOptions.Overwrite` is true, and it uses temp-and-rename
|
||||
replacement where practical.
|
||||
|
||||
`ValidateManifest` checks manifest-only semantics. `ValidateBundle` checks the
|
||||
supplied manifest against local files under `root`, including existence,
|
||||
regular-file type, size, SHA-256, path safety, duplicates, and bundle digest.
|
||||
|
||||
`WriteBundle` copies existing local files from `BundleFile.SourcePath` into a
|
||||
staged bundle at `BundleFile.Path`, writes a compliant manifest, validates the
|
||||
staged bundle, and promotes it to `WriteBundleOptions.Root`.
|
||||
|
||||
`WriteBundleOptions.Overwrite` permits replacing an existing bundle root.
|
||||
Replacement must build the new bundle completely before touching the existing
|
||||
root, then use sibling temp and backup paths for best-effort promotion and
|
||||
restore on failure.
|
||||
|
||||
The writer remains producer-side and filesystem-local. It must not expose
|
||||
distributor storage backends or publication behavior.
|
||||
|
||||
## Manifest Compatibility
|
||||
|
||||
The package should treat the source manifest schema as a compatibility boundary:
|
||||
|
||||
- export the current schema version;
|
||||
- preserve JSON field names exactly;
|
||||
- use lowercase `sha256:<64 hex>` digests;
|
||||
- use slash-separated relative paths in JSON;
|
||||
- use RFC3339 timestamps;
|
||||
- default a zero build or writer `Created` value to the current UTC time;
|
||||
- preserve caller-provided file order for explicit file lists;
|
||||
- produce deterministic ordering when scan-based building is selected;
|
||||
- reject symlinks if distributor validation still rejects source symlinks.
|
||||
|
||||
Scan-based building belongs in v1 of the public package. Explicit file lists
|
||||
should preserve caller order. Scan mode should sort by slash-separated relative
|
||||
path and share the same filtering rules expected by future CLI manifest
|
||||
creation.
|
||||
|
||||
The internal implementation may either move source-bundle core logic into
|
||||
`pkg/bundle` and have internal packages consume it, or keep internal wrappers
|
||||
around public core logic. The important invariant is that public package,
|
||||
future CLI manifest creation, and distributor validation must not drift.
|
||||
|
||||
## Relationship To Other Roadmaps
|
||||
|
||||
`distributor manifest create` should call `pkg/bundle` rather than maintaining a
|
||||
separate manifest builder.
|
||||
|
||||
Remote `validate` and `inspect` should continue using distributor's storage
|
||||
abstraction and internal app wiring; they do not need public producer APIs.
|
||||
|
||||
HTML index mode, link generation, and latest path destinations operate after a
|
||||
bundle has already entered distributor and should not affect this package.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Suggested coverage:
|
||||
|
||||
- build a manifest from explicit files;
|
||||
- build a manifest by scanning a local bundle root;
|
||||
- preserve explicit file order;
|
||||
- sort scan results deterministically by slash-separated relative path;
|
||||
- compute per-file SHA-256 and size;
|
||||
- compute the expected canonical bundle digest;
|
||||
- write and load `manifest.json`;
|
||||
- default zero `Created` to current UTC time while honoring explicit timestamps;
|
||||
- validate generated manifests successfully;
|
||||
- reject unsafe paths, missing files, non-regular files, and symlinks;
|
||||
- emit slash-separated JSON paths;
|
||||
- parse, marshal, load, and write manifests through the exact exported API;
|
||||
- fail `WriteManifest` when `manifest.json` exists unless overwrite is enabled;
|
||||
- emit deterministic manifest JSON with fixed field order and trailing newline;
|
||||
- write a complete local bundle through the public writer;
|
||||
- copy `BundleFile.SourcePath` content to the configured bundle-relative path;
|
||||
- support `WriteBundleOptions.Overwrite` through staged replacement;
|
||||
- avoid leaving a completed bundle path without a valid manifest when writer
|
||||
staging or promotion fails where practical;
|
||||
- compile public examples under `go test` where practical;
|
||||
- prove consistency with distributor validation fixtures.
|
||||
|
||||
## Documentation Updates After Implementation
|
||||
|
||||
- Add Go package documentation under `pkg/bundle`.
|
||||
- Update `README.md` to mention Go producer support.
|
||||
- Update `docs/operations.md` with a producer integration example.
|
||||
- Cross-reference `distributor manifest create` once that CLI command exists.
|
||||
|
||||
Keep this roadmap under `docs/roadmap/` until implemented.
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
`docs/roadmap/implementation.md` intentionally splits this roadmap across two
|
||||
implementation prompts: package core/building first, then the local bundle
|
||||
writer. The split keeps the public API extraction separate from producer-side
|
||||
bundle assembly.
|
||||
|
||||
Core/building stage:
|
||||
|
||||
1. Move or wrap the existing source manifest model, digest logic, path
|
||||
validation, and RFC3339 handling so `pkg/bundle` and internal validation use
|
||||
one contract.
|
||||
2. Implement the locked Stage 2 public API symbols:
|
||||
`ManifestName`, `SchemaVersion`, `Manifest`, `ManifestFile`,
|
||||
`BuildOptions`, `WriteManifestOptions`, `ParseManifest`,
|
||||
`MarshalManifest`, `LoadManifest`, `WriteManifest`, `BuildManifest`,
|
||||
`ValidateManifest`, `ValidateBundle`, `ValidateSourcePath`, `FileDigest`,
|
||||
`BundleDigest`, and `CanonicalFilePayload`.
|
||||
3. Add explicit-list and scan-based manifest building APIs, including zero
|
||||
`Created` defaulting to current UTC time.
|
||||
4. Update internal packages to consume the shared implementation without
|
||||
changing current validation behavior.
|
||||
|
||||
Writer stage:
|
||||
|
||||
1. Implement the locked Stage 3 public API symbols: `BundleFile`,
|
||||
`WriteBundleOptions`, and `WriteBundle`.
|
||||
2. Add the local bundle writer with staged promotion, atomic filesystem
|
||||
operations where practical, and overwrite behavior through sibling temp and
|
||||
backup paths.
|
||||
3. Add package documentation and producer-facing examples.
|
||||
|
||||
## Decisions
|
||||
|
||||
- A zero `Created` value defaults to the current UTC time. Producers may still
|
||||
provide explicit timestamps for reproducible or backfilled bundles.
|
||||
- Scan-based manifest building is included in v1, behind explicit options.
|
||||
Explicit file lists preserve caller order; scan mode sorts deterministically.
|
||||
- A local bundle writer is included in v1. It should be safe and producer-side,
|
||||
but it must not expose distributor publication or storage internals.
|
||||
- The exported API names and signatures in `Initial Exported API` are
|
||||
normative for implementation.
|
||||
|
||||
## Future Work
|
||||
|
||||
- Broader producer workflow helpers, such as richer ignore rules or template
|
||||
scaffolding, can be considered after the first public package exists.
|
||||
- Remote or storage-backed producer writers remain out of scope unless a future
|
||||
producer use case requires them.
|
||||
@@ -1,158 +0,0 @@
|
||||
# Roadmap: Remote `validate` and `inspect`
|
||||
|
||||
## Purpose
|
||||
|
||||
Expand `distributor validate` and `distributor inspect` so they can operate on
|
||||
configured pipeline sources, including `local`, `ssh`, and `s3` sources.
|
||||
|
||||
This feature improves operator diagnostics after producer-side manifest
|
||||
creation is available.
|
||||
|
||||
## Current Implementation Grounding
|
||||
|
||||
Current `validate` and `inspect` accept one local path. App-level code opens a
|
||||
local backend with `openLocalPath`, then uses `bundle.Discover`, which walks
|
||||
storage, finds `manifest.json`, and validates bundles.
|
||||
|
||||
`run` already has the app-level backend factory for configured local, SSH, and
|
||||
S3 sources. `run` also loads `secrets.directory` before backend construction so
|
||||
explicit S3 credential environment references can resolve without mutating the
|
||||
process environment.
|
||||
|
||||
This feature should reuse that source backend construction path for configured
|
||||
sources and remain read-only.
|
||||
|
||||
## Goals
|
||||
|
||||
- Preserve the existing local path shortcut:
|
||||
`distributor validate <path>` and `distributor inspect <path>`.
|
||||
- Add config-driven source validation and inspection:
|
||||
`--config <path> --pipeline <id>`.
|
||||
- Support configured `local`, `ssh`, and `s3` sources.
|
||||
- Reuse the same source discovery and validation behavior used by `run`.
|
||||
- Reuse secrets loading and backend credential resolution from `run`.
|
||||
- Avoid opening or inspecting destinations in the first version.
|
||||
- Keep output concise and operator-oriented.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not write to source or destination storage.
|
||||
- Do not inspect destination state in v1.
|
||||
- Do not add new backend types.
|
||||
- Do not change source manifest schema.
|
||||
- Do not add daemon, API, or producer-package dependencies.
|
||||
|
||||
## CLI Mode Rules
|
||||
|
||||
Use two mutually exclusive modes:
|
||||
|
||||
```sh
|
||||
distributor validate <local-path>
|
||||
distributor inspect <local-path>
|
||||
```
|
||||
|
||||
and:
|
||||
|
||||
```sh
|
||||
distributor validate --config config.yml --pipeline weather-daily
|
||||
distributor inspect --config config.yml --pipeline weather-daily
|
||||
```
|
||||
|
||||
Do not allow positional local paths together with `--config` or `--pipeline`.
|
||||
|
||||
Require `--pipeline` in config mode, even when the config has exactly one
|
||||
pipeline. This avoids surprising remote access and keeps the initial behavior
|
||||
explicit.
|
||||
|
||||
Optional narrowing:
|
||||
|
||||
```sh
|
||||
--bundle <source-root-relative-bundle-path>
|
||||
```
|
||||
|
||||
When provided, `--bundle` identifies a source-root-relative bundle directory and
|
||||
the command validates or inspects that bundle rather than discovering all
|
||||
bundles.
|
||||
|
||||
## Behavior
|
||||
|
||||
For config mode:
|
||||
|
||||
1. load config using the same defaulting and validation path as `run`;
|
||||
2. load `secrets.directory`;
|
||||
3. find the requested pipeline id;
|
||||
4. open only the pipeline source backend;
|
||||
5. discover all bundles or validate the requested `--bundle`;
|
||||
6. validate manifest schema, paths, files, per-file digests, and bundle digest;
|
||||
7. return non-zero if any selected bundle fails validation.
|
||||
|
||||
`inspect` should fully validate selected bundles by default, matching current
|
||||
`bundle.Discover` behavior. Output can then report reliable normalized metadata:
|
||||
|
||||
- pipeline id for config mode;
|
||||
- backend type;
|
||||
- source-relative bundle path;
|
||||
- bundle id;
|
||||
- created timestamp;
|
||||
- digest;
|
||||
- file count and total size;
|
||||
- file path, size, and SHA-256.
|
||||
|
||||
## Relationship To Other Roadmaps
|
||||
|
||||
Remote validation works well after `manifest create` and `pkg/bundle` because
|
||||
operators can validate producer output where it actually lands.
|
||||
|
||||
This feature does not depend on HTML index mode, link generation, or latest path
|
||||
destinations. Those features may later make inspection output richer, but v1
|
||||
should stay source-focused.
|
||||
|
||||
Machine-readable output should follow `docs/roadmap/cli_output_policy.md` and
|
||||
use the shared `--format text|json` policy rather than command-specific JSON
|
||||
flags.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Suggested coverage:
|
||||
|
||||
- existing local path validation still works;
|
||||
- existing local path inspection still works;
|
||||
- config-mode local source validation works;
|
||||
- fake configured source validation works through the storage abstraction;
|
||||
- `--bundle` validates a specific source-relative bundle when implemented;
|
||||
- missing pipeline id fails clearly;
|
||||
- positional path plus `--config` is rejected;
|
||||
- source backend open failures include pipeline/backend context;
|
||||
- digest mismatch and missing manifest failures are clear;
|
||||
- inspect output includes normalized source metadata.
|
||||
|
||||
Avoid live SSH or S3 tests except existing opt-in integration patterns.
|
||||
|
||||
## Documentation Updates After Implementation
|
||||
|
||||
- Update `docs/cli.md` with both local and config-driven syntax.
|
||||
- Update `docs/operations.md` with remote validation examples.
|
||||
- Update `docs/troubleshooting.md` for common configured-source failures.
|
||||
- Update examples only if useful and environment-gated.
|
||||
|
||||
Keep this roadmap under `docs/roadmap/` until implemented.
|
||||
|
||||
## Decisions
|
||||
|
||||
- Destination-state inspection is not part of v1 remote `validate` or
|
||||
`inspect`. These commands remain source-focused and should not open
|
||||
destinations unexpectedly.
|
||||
- JSON output is deferred to `docs/roadmap/cli_output_policy.md` or to an
|
||||
implementation pass that selects that policy.
|
||||
- `--pipeline` remains required in config mode, even when a config has exactly
|
||||
one pipeline.
|
||||
|
||||
## Future Work
|
||||
|
||||
- Add destination-state inspection behind an explicit flag such as
|
||||
`--with-destinations` if operators need fan-out status diagnostics from
|
||||
`inspect`.
|
||||
- Add JSON output through the CLI-wide `--format text|json` policy in
|
||||
`docs/roadmap/cli_output_policy.md`.
|
||||
- Reconsider optional pipeline selection only if the project later introduces a
|
||||
broader command mode for single-pipeline configs.
|
||||
Reference in New Issue
Block a user