From a6c38d3e9610fdc6281f935ac5bac629e5a3097b Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 1 Jun 2026 15:11:00 -0500 Subject: [PATCH] Add implementation roadmap for the next feature set --- docs/roadmap/cli_output_policy.md | 242 +++++++++++++++++++ docs/roadmap/html_index_mode.md | 160 +++++++++++++ docs/roadmap/implementation.md | 297 +++++++++++++++++++++++- docs/roadmap/latest_paths.md | 212 +++++++++++++++++ docs/roadmap/link_generation.md | 179 ++++++++++++++ docs/roadmap/manifest_create.md | 173 ++++++++++++++ docs/roadmap/public_bundle_package.md | 204 ++++++++++++++++ docs/roadmap/remote_validate_inspect.md | 158 +++++++++++++ 8 files changed, 1620 insertions(+), 5 deletions(-) create mode 100644 docs/roadmap/cli_output_policy.md create mode 100644 docs/roadmap/html_index_mode.md create mode 100644 docs/roadmap/latest_paths.md create mode 100644 docs/roadmap/link_generation.md create mode 100644 docs/roadmap/manifest_create.md create mode 100644 docs/roadmap/public_bundle_package.md create mode 100644 docs/roadmap/remote_validate_inspect.md diff --git a/docs/roadmap/cli_output_policy.md b/docs/roadmap/cli_output_policy.md new file mode 100644 index 0000000..e6cb5f8 --- /dev/null +++ b/docs/roadmap/cli_output_policy.md @@ -0,0 +1,242 @@ +# Roadmap: CLI Output Policy + +## Purpose + +Define a shared CLI output policy before adding machine-readable JSON output to +`distributor` commands. + +Current CLI output is human-readable text. That is appropriate as the default, +but upcoming features need a common machine-readable contract so each command +does not invent a separate JSON flag, envelope, warning policy, or error shape. + +## Current Implementation Grounding + +The current CLI has these output-producing commands: + +- `version`; +- `run`; +- `validate`; +- `inspect`. + +Current output is text-only. `run` has the most complex output because it can +print warnings, pipeline summaries, planned destination actions, partial +destination failures, and a final status line. `validate` and `inspect` are +currently local-path commands. `version` prints a single text line. + +Planned `distributor manifest create`, remote `validate` and `inspect`, link +generation, and latest path destinations increase the need for structured +output that scripts can consume consistently. + +## Goals + +- Define one CLI-wide policy for text and JSON output. +- Preserve current human-readable text output as the default. +- Add JSON output consistently across output-producing commands when this + roadmap is implemented. +- Keep warning, error, and partial-failure behavior predictable. +- Avoid external dependencies; use Go's standard `encoding/json`. + +## Non-Goals + +- Do not add YAML, table, NDJSON, streaming JSON, or template output formats. +- Do not change source manifest, destination state, config, or backend schemas. +- Do not make help or usage output JSON. +- Do not turn ordinary fatal setup errors into structured JSON results. +- Do not add a root-global output flag in the first implementation. + +## Shared CLI Policy + +Each supported command should accept: + +```sh +--format text +--format json +``` + +Policy: + +- `text` is the default and preserves existing output unless a feature + explicitly changes text output. +- `json` writes exactly one JSON document to stdout. +- help and usage output remain text-only. +- invalid `--format` values are usage errors. +- `--format` is a per-command flag, not a root-global flag. +- commands that do not yet support JSON must fail validation if `--format json` + is accepted by their parser but not implemented. + +The first implementation should add JSON support for all current +output-producing commands rather than leaving a mixed CLI where some commands +support `--format` and others do not. + +## Stdout, Stderr, Warnings, and Errors + +Text mode keeps the current behavior: + +- normal command output goes to stdout; +- warnings may be printed as text; +- command errors are printed to stderr by CLI error handling. + +JSON mode: + +- successful commands write one JSON document to stdout; +- warnings are included in a top-level `warnings` array and are not duplicated + to stderr; +- fatal setup errors that prevent construction of a result write text errors to + stderr, write no JSON stdout, and exit non-zero; +- commands with meaningful partial results may write a JSON document with + `ok: false`, structured failure details, and a non-zero exit status. + +`run --format json` should use the partial-result behavior when planning or +execution has begun and one or more destinations fail. This lets automation +inspect successful actions, failed actions, warnings, and final counters even +when the process exits non-zero. + +## JSON Envelope + +All JSON-mode commands should use a common top-level envelope: + +```json +{ + "schema_version": 1, + "command": "inspect", + "ok": true, + "warnings": [], + "result": {} +} +``` + +Envelope fields: + +- `schema_version`: integer version of the CLI JSON output schema. +- `command`: command name, using the public CLI command path where useful, such + as `manifest create`. +- `ok`: boolean success indicator for the command result. +- `warnings`: array of structured warning objects. +- `result`: command-specific result object. +- `errors`: optional array of structured error objects for commands that can + return partial results. + +Field rules: + +- use stable snake_case field names; +- use RFC3339 timestamps; +- use numeric JSON values for sizes and counts; +- use slash-separated logical paths for bundle, source, destination, and output + paths; +- never include secret values; +- keep command-specific data under `result`; +- add fields compatibly where practical. + +Warning objects should include at least: + +```json +{ + "message": "secret OBJECT_STORAGE_KEY ignored because the real environment already has that variable" +} +``` + +Error objects for partial results should include enough context for automation +and troubleshooting, such as pipeline id, destination id, backend, bundle path, +and message when those values are available. + +## Command Adoption + +`version`: + +- text mode keeps the current version string; +- JSON mode reports application name and version under `result`. + +`validate`: + +- text mode keeps the current validation summary; +- JSON mode reports bundle count and selected bundle identifiers; +- future remote validation uses the same envelope and adds pipeline/source + context under `result`. + +`inspect`: + +- text mode keeps human-readable bundle metadata; +- JSON mode reports normalized bundle metadata, including source-relative path, + bundle id, created timestamp, digest, file count, total size, and file + records. + +`run`: + +- text mode keeps the current progress and final status style unless a future + feature intentionally changes it; +- JSON mode reports warnings, pipeline summaries, destination action records, + output records, final counters, dry-run status, and partial failure records; +- JSON mode may write `ok: false` with partial results and still exit non-zero. + +Future `manifest create`: + +- should use `--format text|json`; +- text mode should keep concise human success output; +- JSON mode should report the generated manifest summary under `result`; +- the command should not introduce a separate `--json` flag. + +## Relationship To Other Roadmaps + +`distributor manifest create` should depend on this roadmap for JSON summary +output and should not add a command-specific JSON flag. + +Remote `validate` and `inspect` should use this policy for machine-readable +source validation and inspection. + +Link generation should expose primary links through JSON command output only +after this policy is implemented or selected for implementation. + +Latest path destinations should report fixed-path selection and destructive +replacement summaries through the same `run --format json` result model. + +## Testing Expectations + +Suggested coverage: + +- CLI parsing accepts `--format text` and `--format json` for supported + commands; +- CLI parsing rejects invalid `--format` values as usage errors; +- help and usage output remain text-only; +- each JSON-capable command emits exactly one valid JSON document to stdout on + success; +- text mode preserves existing output; +- JSON-mode warnings appear in `warnings` and are not duplicated to stderr; +- fatal setup errors write no JSON stdout and return non-zero; +- `run --format json` emits partial-result JSON with `ok: false` and exits + non-zero when one or more destination failures occur after planning begins; +- JSON output uses RFC3339 timestamps, numeric sizes and counts, and + slash-separated logical paths; +- no JSON output includes secret values; +- documentation consistency checks find no unsupported `--json` references. + +## Documentation Updates After Implementation + +- Update `docs/cli.md` with the shared `--format text|json` policy. +- Update command examples only for implemented JSON behavior. +- Update `docs/operations.md` where JSON output materially improves automation + workflows. +- Update `docs/troubleshooting.md` only for implemented JSON-mode recovery + behavior. +- Update relevant roadmap files when JSON output is no longer deferred. + +Keep this roadmap under `docs/roadmap/` until implemented. + +## Decisions + +- Use per-command `--format text|json`; do not introduce `--json`. +- Keep `text` as the default for backward compatibility. +- Implement JSON support for all current output-producing commands when this + roadmap is selected. +- Keep help and usage output text-only. +- Put JSON-mode warnings in the top-level `warnings` array. +- Allow `run --format json` to emit partial-result JSON with `ok: false` and a + non-zero exit status. +- Keep fatal setup errors as text stderr with no JSON stdout. + +## Future Work + +- Consider a root-global output flag only if the command parser is later + refactored around shared root options. +- Consider additional formats only if a concrete consumer requires them. +- Consider a versioned JSON schema reference after the first JSON-capable + release. diff --git a/docs/roadmap/html_index_mode.md b/docs/roadmap/html_index_mode.md new file mode 100644 index 0000000..f812b6f --- /dev/null +++ b/docs/roadmap/html_index_mode.md @@ -0,0 +1,160 @@ +# Roadmap: HTML Index Mode + +## Purpose + +Add a Markdown-to-HTML `index` mode that renders one selected Markdown artifact +to `index.html` at the destination bundle path. + +This feature improves static-site UX and gives link generation and latest path +destinations a clean directory-style output to prefer. + +## Current Implementation Grounding + +Current Markdown transformation lives in `internal/transform/markdown`. It +renders every manifest-listed `.md` file to a sidecar `.html` file: + +```text +report.md -> report.html +``` + +Current config validation accepts only `markdown_to_html.mode: sidecar`. +`publish.PlanOutputs` asks the configured transformer for generated outputs, +then destination state records generated outputs with path, kind, source path, +transform name, SHA-256, and size. Current destination state has no field for a +transform mode beyond the existing transform string. + +## Goals + +- Preserve existing sidecar behavior as the default and supported explicit mode. +- Add `markdown_to_html.mode: index`. +- In index mode, render exactly one Markdown input to `index.html`. +- Record `index.html` as a normal generated output in `.distributor.json`. +- Keep source manifests unchanged. +- Keep link generation and latest paths straightforward without making them + dependencies. + +## Non-Goals + +- Do not add collection index pages. +- Do not add multi-page static-site generation. +- Do not add feeds, notifications, or link generation in this feature. +- Do not require producers to name a file `index.md`. +- Do not add custom output filenames in v1. + +## Configuration Shape + +Extend existing destination-level transform config: + +```yaml +transform: + markdown_to_html: + enabled: true + mode: index + input: report.md +``` + +Mode semantics: + +- `sidecar`: existing behavior; render each manifest-listed Markdown file to a + same-directory `.html` file. +- `index`: render one selected Markdown file to `index.html` at the destination + bundle path. + +`sidecar` remains the current/default mode. `index` is the proposed addition. + +## Input Selection + +Index mode should choose the Markdown input deterministically: + +1. If `transform.markdown_to_html.input` is configured, use that manifest-listed + Markdown file. +2. If no input is configured and the source manifest lists exactly one Markdown + file, use that file. +3. If no input is configured and there are zero or multiple Markdown files, fail + planning with a clear error. + +The configured input path must be a safe relative source path, must be listed in +the source manifest, and must end in `.md`. + +Input selection happens during planning because it depends on the validated +source manifest, not only static config. + +## Output Semantics + +Index mode always writes: + +```text +index.html +``` + +relative to the destination bundle path. + +The generated output should use current state model terms: + +- `path`: `index.html`; +- `kind`: `generated`; +- `source_path`: selected Markdown file; +- `transform`: `markdown_to_html`; +- `sha256` and `size`: digest and byte size of generated HTML. + +Replacement, skip, cleanup, and force behavior should treat `index.html` like +any other distributor-managed generated output. + +Config validation should reject enabled Markdown-to-HTML transform +configuration when `publish.html` is false. That keeps publish policy and +transform intent aligned and avoids silently accepting unused transform config. + +## Relationship To Other Roadmaps + +Link generation should recognize `index.html` and produce directory-style URLs, +but link generation must not be required for index mode. + +Latest path destinations work without index mode, but fixed latest destinations +produce nicer stable URLs when they publish `index.html`. + +`manifest create`, `pkg/bundle`, and remote validation remain source-bundle +features and should not depend on transform mode. + +## Testing Expectations + +Suggested coverage: + +- config validation accepts `mode: sidecar` and `mode: index`; +- current sidecar behavior remains unchanged; +- index mode accepts explicit input; +- index mode chooses the only Markdown file when no input is configured; +- planning fails when index mode has zero or multiple Markdown candidates + without explicit input; +- planning fails when explicit input is unsafe, not listed, or not Markdown; +- index mode emits `index.html`; +- state records `index.html` as a generated output; +- source-only publication does not write `index.html`; +- dry-run reports `index.html` without writing it; +- replacement safely updates prior generated `index.html`. + +## Documentation Updates After Implementation + +- Update `docs/config.md` with Markdown-to-HTML modes and `input`. +- Update `docs/integrations/markdown.md` with sidecar and index behavior. +- Update `docs/operations.md` with a static-site example. +- Add or update examples only for implemented behavior. + +Keep this roadmap under `docs/roadmap/` until implemented. + +## Decisions + +- Enabled Markdown-to-HTML transform config is rejected when `publish.html` is + false. +- Generated output metadata keeps `transform: markdown_to_html`; the output + path and destination config distinguish sidecar from index behavior. +- Future multi-page or collection index generation should be a separate + transform, not an expansion of this single-input index mode. + +## Future Work + +- Add a separate collection or site-index transform if distributor later needs + multi-page aggregation. +- Consider richer transform metadata only if future state consumers need more + than the transform name and output path. +- Consider custom index output names only if fixed `index.html` proves too + limiting for real deployments. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 4aa509f..f789e7c 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -26,12 +26,299 @@ implemented local, SSH/SFTP, and S3-compatible backends. ## Active Roadmap -There are no active implementation items in this roadmap. +The active roadmap implements the focused feature roadmaps in an order intended +to minimize later rewrites. The sequence establishes shared CLI output before +new CLI commands, establishes producer bundle APIs before manifest creation, +and implements destination path mapping before link generation so URL +construction does not need to be reworked later. -Before implementing new product behavior, add a focused roadmap entry when the -work changes storage semantics, config, CLI behavior, state schema, transform -behavior, notification behavior, operational safety, or user-visible workflows. -Keep those entries out of current-behavior docs until the behavior exists. +Each stage is sized for one implementation prompt. Future behavior stays under +`docs/roadmap/` until the stage is implemented and current-behavior docs are +updated. + +### Stage 1: CLI Output Policy + +Goal: implement the shared CLI output contract in +`docs/roadmap/cli_output_policy.md`. + +Implementation scope: + +- add per-command `--format text|json` to current output-producing commands: + `version`, `validate`, `inspect`, and `run`; +- keep `text` as the default and preserve existing text output unless the + policy explicitly requires a change; +- implement the shared JSON envelope, warnings array, fatal setup error + behavior, and `run` partial-result JSON with `ok: false`; +- keep help and usage output text-only. + +Documentation updates after implementation: + +- update `docs/cli.md` with the shared `--format text|json` policy; +- update `docs/operations.md` where JSON output materially improves automation + workflows; +- update `docs/troubleshooting.md` only for implemented JSON-mode recovery + behavior. + +Tests: + +- CLI parsing accepts `--format text` and `--format json`; +- invalid `--format` values are usage errors; +- each JSON-capable command emits exactly one valid JSON document on success; +- JSON-mode warnings appear in `warnings` and are not duplicated to stderr; +- fatal setup errors write no JSON stdout and return non-zero; +- `run --format json` emits partial-result JSON and exits non-zero when + destination failures occur after planning begins. + +Completion criteria: all current output-producing commands support the shared +policy, text output remains backward compatible, and `go test ./...` passes. + +### Stage 2: Public Bundle Package Core + +Goal: implement the core and manifest-building parts of +`docs/roadmap/public_bundle_package.md`. + +Implementation scope: + +- introduce `pkg/bundle` with the public source manifest model, schema version, + digest logic, parsing, validation, explicit file-list building, scan-based + building, and zero-`Created` defaulting to current UTC; +- update internal validation to consume the shared implementation without + behavior drift; +- keep destination state, publish planning, storage backends, transforms, + notifications, and config internal. + +Documentation updates after implementation: + +- add package documentation for `pkg/bundle`; +- update `README.md` and `docs/operations.md` only for implemented Go producer + support; +- update `docs/internal/bundle.md` if internal ownership changes. + +Tests: + +- public package tests cover explicit lists, scan mode, ordering, digest + calculation, parsing, validation, unsafe paths, symlinks, and zero `Created`; +- internal bundle tests continue to pass against the shared implementation; +- public examples compile where practical. + +Completion criteria: public and internal validation use one manifest contract, +and tests prove identical semantics. + +### Stage 3: Public Bundle Local Writer + +Goal: implement the local bundle writer portion of +`docs/roadmap/public_bundle_package.md`. + +Implementation scope: + +- add producer-side local bundle writing with staging and atomic promotion where + practical; +- keep the writer filesystem-local and producer-focused; +- do not expose distributor storage backends or publication behavior through + the public package. + +Documentation updates after implementation: + +- document the writer API in `pkg/bundle`; +- update producer workflow docs only for implemented behavior. + +Tests: + +- writer creates a complete valid local bundle; +- writer output validates through distributor's normal validation path; +- overwrite and failure behavior avoid leaving a completed bundle path without + a valid manifest where practical. + +Completion criteria: producers can create complete valid local bundles through +the public API without hand-writing manifest files. + +### Stage 4: Manifest Create CLI + +Goal: implement `docs/roadmap/manifest_create.md`. + +Implementation scope: + +- add `distributor manifest create` over `pkg/bundle`; +- support explicit-first `--file` selection, recursive scan fallback, + `--overwrite`, optional `--created`, and shared `--format text|json`; +- use temp-and-rename replacement for `manifest.json` where practical; +- do not duplicate bundle contract logic in CLI or app code. + +Documentation updates after implementation: + +- update `docs/cli.md` with command syntax, flags, output, and examples; +- update `docs/operations.md` with a producer workflow; +- cross-reference `pkg/bundle` for Go producers. + +Tests: + +- command creates valid manifests for explicit and scanned files; +- ordering, dotfile inclusion, metadata exclusion, symlink rejection, and + overwrite behavior match the roadmap; +- text and JSON output follow the shared CLI output policy. + +Completion criteria: the command writes valid `manifest.json`, supports +text/JSON output, and generated manifests validate through distributor. + +### Stage 5: Remote Validate and Inspect + +Goal: implement `docs/roadmap/remote_validate_inspect.md`. + +Implementation scope: + +- extend `validate` and `inspect` with mutually exclusive local-path and + `--config --pipeline` modes; +- reuse `run` source backend construction and secrets resolution; +- support configured local, SSH, and S3 sources; +- preserve `--pipeline` as required in config mode; +- keep the commands source-only and do not open destinations. + +Documentation updates after implementation: + +- update `docs/cli.md` with local and config-driven syntax; +- update `docs/operations.md` with remote validation examples; +- update `docs/troubleshooting.md` for common configured-source failures. + +Tests: + +- existing local behavior remains unchanged; +- config-mode validation and inspection work for local and fake/app-level + source backends; +- SSH and S3 behavior is covered by focused unit tests and existing opt-in + integration patterns; +- JSON output follows the shared CLI output policy. + +Completion criteria: local behavior is stable, configured local/SSH/S3 sources +can be validated and inspected, and no destination backend is opened. + +### Stage 6: HTML Index Mode + +Goal: implement `docs/roadmap/html_index_mode.md`. + +Implementation scope: + +- add `markdown_to_html.mode: index` with fixed `index.html` output; +- keep `sidecar` as the default and existing explicit mode; +- implement deterministic input selection through explicit input or exactly one + manifest-listed Markdown file; +- reject enabled Markdown-to-HTML transform config when `publish.html` is + false; +- record generated state metadata with `transform: markdown_to_html`. + +Documentation updates after implementation: + +- update `docs/config.md` with Markdown-to-HTML modes and input selection; +- update `docs/integrations/markdown.md`; +- update `docs/operations.md` with an implemented static-site example where + useful. + +Tests: + +- sidecar behavior remains unchanged; +- index mode handles explicit input, single Markdown fallback, ambiguous input, + unsafe input, state metadata, dry-run, source-only publication, and output + collisions. + +Completion criteria: source publication, sidecar mode, index mode, state +metadata, dry-run, and collision behavior are all covered. + +### Stage 7: Latest Path Destinations + +Goal: implement `docs/roadmap/latest_paths.md`. + +Implementation scope: + +- add destination-level `path_mapping.mode` with default `preserve_relative` + and new `fixed`; +- make destination bundle path mapping destination-local; +- for fixed destinations, select only the newest discovered bundle per + destination before planning writes; +- add fixed-path dry-run warnings and summary counts; +- allow `--force` for fixed backend roots while keeping deletion bounded to the + configured backend root. + +Documentation updates after implementation: + +- update `docs/config.md` with `path_mapping`; +- update `docs/operations.md` with archive-plus-latest fan-out examples; +- update `docs/cli.md` if dry-run output gains fixed-path indicators. + +Tests: + +- omitted and explicit `preserve_relative` match existing behavior; +- fixed destinations publish at local, SSH, and S3 backend roots; +- newest-only selection is deterministic; +- older discovered bundles are not planned or written to fixed destinations; +- dry-run reports candidate count, selected bundle, and destructive replacement + warnings; +- force remains bounded at the backend root. + +Completion criteria: fixed destination roots work across implemented backends, +archive-style destinations remain unchanged, and destructive behavior is clear +in dry-run output. + +### Stage 8: Link Generation Support + +Goal: implement `docs/roadmap/link_generation.md`. + +Implementation scope: + +- add destination-level `links.base_url` and `links.primary`; +- generate per-output and primary URLs using URL semantics, not filesystem or + storage joins; +- handle `index.html` as a directory-style URL; +- store optional URL metadata under destination state schema version 1 while + pre-release; +- expose links through JSON command output only where supported by the CLI + output policy. + +Documentation updates after implementation: + +- update `docs/config.md` with destination `links` fields; +- update `docs/operations.md` with static-site URL examples; +- update `docs/internal/state.md` for URL metadata; +- update `docs/cli.md` only for implemented CLI link output. + +Tests: + +- config validation accepts valid links and rejects invalid schemes, query + strings, and fragments; +- nested bundle paths and fixed latest paths generate correct URLs; +- `index.html` omits the filename; +- primary selection is deterministic; +- destinations without `links` produce no URL metadata. + +Completion criteria: URL generation respects archive and fixed path mapping, +state records optional URL metadata when configured, and unconfigured +destinations remain unchanged. + +### Stage 9: Roadmap Closeout + +Goal: remove completed roadmap drift after Stages 1-8 are implemented. + +Implementation scope: + +- remove or rewrite completed roadmap files whose behavior is fully documented + in current docs; +- ensure current docs describe implemented behavior; +- keep `docs/roadmap/` focused only on remaining future work. + +Documentation updates after implementation: + +- update `README.md`, `docs/cli.md`, `docs/config.md`, + `docs/operations.md`, `docs/troubleshooting.md`, `docs/internal/`, + `docs/integrations/markdown.md`, and examples only where implemented + behavior requires it. + +Tests: + +- run documentation consistency searches for completed-feature language that + still appears only as future work; +- run focused tests for any examples or docs backed by tests; +- run `go test ./...` if behavior docs and examples changed with code. + +Completion criteria: completed features are documented as current behavior, and +`docs/roadmap/` contains only future or deferred work. ## Deferred Work diff --git a/docs/roadmap/latest_paths.md b/docs/roadmap/latest_paths.md new file mode 100644 index 0000000..eb34428 --- /dev/null +++ b/docs/roadmap/latest_paths.md @@ -0,0 +1,212 @@ +# Roadmap: Latest Path Destinations + +## Purpose + +Support stable "latest" publication paths as normal fan-out destinations. + +A latest destination republishes the current winning source bundle to a fixed +destination path such as `/weather/latest/`, while archive destinations preserve +source-relative bundle paths such as `/weather/daily/brentwood/2026-06-01/`. + +This feature should be implemented before link generation so URL construction +can use final destination path semantics. It is operationally destructive and +benefits from index-mode HTML, although it must not require index mode or link +generation. + +## Current Implementation Grounding + +Current run behavior computes the destination bundle path as the source bundle +path relative to the source root. Publication then writes selected outputs and +`.distributor.json` below that destination bundle path. + +Backend roots differ by backend: + +- local and SSH/SFTP use configured `path` as the backend root; +- S3 uses configured bucket plus optional `prefix` as the backend root. + +Destination comparison uses `.distributor.json` at the destination bundle path. +Replacement and skip decisions are based on the normalized source manifest +recorded in destination state. Unmanaged non-empty destinations fail unless +explicit force behavior is requested. + +## Goals + +- Add destination-local path mapping. +- Preserve source-relative destination paths as the default. +- Add fixed path mapping for latest-style destinations. +- For fixed destinations, select only the newest discovered source bundle for + publication. +- Make fixed mapping work for local, SSH/SFTP, and S3 backends. +- Continue using normal destination state comparison and replacement rules. +- Keep link generation and index-mode HTML complementary, not required. + +## Non-Goals + +- Do not add symlink-based latest behavior. +- Do not add feed or index generation. +- Do not change source manifest schema. +- Do not make producers responsible for latest publication. +- Do not add domain-specific latest selection rules. + +## Configuration Shape + +Add destination-level path mapping: + +```yaml +path_mapping: + mode: preserve_relative +``` + +Modes: + +- `preserve_relative`: existing/default behavior. +- `fixed`: publish selected outputs directly at the destination backend root. + +Example fixed destination: + +```yaml +destinations: + - id: latest-html + backend: ssh + host: web.example.com + user: deploy + path: /srv/www/weather/latest + path_mapping: + mode: fixed + publish: + source: false + html: true +``` + +For S3, the fixed destination root is the configured bucket plus `prefix`. + +## Path Mapping Semantics + +`preserve_relative`: + +```text +destination bundle path = source-root-relative bundle path +``` + +`fixed`: + +```text +destination bundle path = "" +``` + +That empty logical destination bundle path means the destination backend root. +Storage writes still use normal backend-rooted logical paths for outputs and +state. + +## Newest Bundle Selection + +`preserve_relative` destinations should continue publishing every selected +source bundle independently. + +`fixed` destinations should publish only one bundle per run: the newest +discovered source bundle selected for that destination. Newest selection should +be deterministic: + +1. choose the bundle with the greatest source manifest `created` timestamp; +2. if multiple bundles have the same greatest `created` timestamp, use the + source-root-relative bundle path ascending as a tie-breaker. + +The selected bundle then uses the existing destination comparison, skip, +replacement, cleanup, and force rules. Older discovered bundles should not be +planned or written to that fixed destination in the same run. + +Dry-run output should identify the number of fixed-destination candidates and +the selected source bundle so operators can see which bundle would become +latest. + +## Safety Rules + +Fixed path mapping is more destructive than archive-style publication because +newer bundles replace prior contents at the same destination path. + +Required behavior: + +- dry-run output must identify fixed path mapping and planned replacement; +- dry-run output must include an extra fixed-path warning or summary count when + destructive replacement is possible; +- replacement should delete only managed outputs recorded in valid destination + state when possible; +- unmanaged non-empty fixed destinations must fail unless force is explicitly + requested; +- `--force` may be used when fixed mapping targets the backend root, but force + replacement must remain bounded to the configured destination backend root and + must never delete above it; +- fixed mapping to the backend root requires especially clear dry-run reporting. + +## Relationship To Other Roadmaps + +HTML index mode is useful for fixed web destinations because `index.html` +supports stable directory-style URLs, but sidecar HTML and source-only outputs +should still work. + +Link generation should use fixed path semantics so latest URLs are based on the +fixed destination root, not the original source-relative archive path. + +Producer manifest creation and remote validation are independent of destination +path mapping. + +## Implementation Stages + +1. Add destination-level `path_mapping` config validation with + `preserve_relative` as the default and `fixed` as the new mode. +2. Refactor run planning so destination bundle path mapping is destination-local + and can be evaluated before writes. +3. Add fixed-destination newest selection so each fixed destination receives + only the newest discovered source bundle. +4. Integrate fixed mapping with destination comparison, managed deletion, force + replacement, and dry-run reporting. +5. Update local, SSH, and S3 tests to verify fixed root behavior under each + backend root model. +6. Update current-behavior documentation after implementation. + +## Testing Expectations + +Suggested coverage: + +- omitted `path_mapping` preserves existing behavior; +- `preserve_relative` explicitly matches existing behavior; +- `fixed` publishes outputs and `.distributor.json` at the backend root; +- fixed destinations select only the newest discovered bundle; +- fixed destination tie-break behavior is deterministic; +- older discovered bundles are not planned or written to fixed destinations; +- newer source replaces older managed fixed destination state; +- older source skips newer fixed destination state; +- unmanaged non-empty fixed destination fails without force; +- dry-run reports fixed path mapping, candidate count, selected bundle, and + destructive replacement warnings clearly; +- `--force` remains bounded when fixed mapping targets the backend root; +- S3 fixed mapping respects bucket plus prefix as backend root. + +## Documentation Updates After Implementation + +- Update `docs/config.md` with `path_mapping`. +- Update `docs/operations.md` with archive-plus-latest fan-out examples. +- Update `docs/cli.md` if dry-run output gains fixed-path indicators. +- Add an environment-safe example only after behavior is implemented. + +Keep this roadmap under `docs/roadmap/` until implemented. + +## Decisions + +- `--force` is allowed when fixed mapping targets the backend root, but dry-run + and run output must make the fixed-root replacement explicit and force remains + bounded to the configured backend root. +- Fixed destinations publish only the newest discovered bundle, rather than + processing every discovered bundle and letting later writes replace earlier + writes. +- Dry-run includes an extra warning or summary count for fixed-path destructive + replacements. + +## Future Work + +- Add richer source selection policies if deployments need something other than + newest-by-`created` for fixed destinations. +- Consider stricter ambiguity handling for equal latest timestamps if real + producer workflows make timestamp ties common. +- Add higher-level status or approval workflows for fixed-root replacements if + dry-run output is not enough operational protection. diff --git a/docs/roadmap/link_generation.md b/docs/roadmap/link_generation.md new file mode 100644 index 0000000..86875b5 --- /dev/null +++ b/docs/roadmap/link_generation.md @@ -0,0 +1,179 @@ +# Roadmap: Link Generation Support + +## Purpose + +Add destination-aware link generation so distributor can record human-usable URLs +for published artifacts. + +This feature should be implemented after destination path mapping so generated +URLs use the final archive or fixed destination path semantics. It prepares the +project for notification adapters, richer inspect/status output, static-site UX, +and future feed generation. + +## Current Implementation Grounding + +Current publish planning produces `publish.Output` values with destination path, +source path, kind, transform, SHA-256, size, and optional generated data. + +Current `.distributor.json` output records contain path, kind, source path, +transform, SHA-256, and size. They do not contain URL metadata, and destination +state has no top-level links block. + +Current destination bundle paths preserve the source-root-relative bundle path. +Latest/fixed path mapping is not implemented yet. + +## Goals + +- Add optional destination-level link configuration. +- Generate per-output URLs when a destination has `links.base_url`. +- Record generated URLs in destination state when configured. +- Select a deterministic primary URL when possible. +- Keep link generation destination-local and independent of producer manifests. +- Do not infer public URLs from backend configuration automatically. + +## Non-Goals + +- Do not deliver notifications. +- Do not require every destination to expose URLs. +- Do not add static-site indexes or feeds. +- Do not add latest path mapping. +- Do not change source manifest schema. + +## Configuration Shape + +Add optional destination-level config: + +```yaml +links: + base_url: https://weather.example.com + primary: auto +``` + +Semantics: + +- `links.base_url`: absolute HTTP or HTTPS URL corresponding to the destination + backend root. +- `links.primary`: optional primary-link selection policy. + +Initial `primary` policies: + +- `auto`: choose the best available output; +- `html`: prefer generated HTML outputs; +- `source`: prefer copied source outputs. + +If `links` is absent, no URL metadata is generated. + +## URL Construction + +Construct URLs from: + +1. `links.base_url`; +2. destination bundle path relative to the destination backend root; +3. output path relative to the destination bundle path. + +Use URL path joining and escaping rules, not filesystem or storage path joins. +Preserve any path prefix in `base_url`. Reject `base_url` values with query +strings or fragments. + +For `index.html`, generate a directory-style URL by omitting the filename: + +```text +https://weather.example.com/daily/brentwood/2026-06-01/ +``` + +For other outputs, include the output filename: + +```text +https://weather.example.com/daily/brentwood/2026-06-01/report.html +``` + +`index.html` recognition should apply to any output path ending in +`/index.html` or exactly `index.html`. + +## Primary Link Selection + +For `primary: auto`, select in deterministic publish-plan order: + +1. any `index.html` output; +2. first generated HTML output; +3. first source output; +4. no primary URL. + +For `primary: html`, prefer generated HTML outputs. For `primary: source`, +prefer source outputs. If no output matches the policy, record no primary link +rather than failing publication. + +## Destination State + +When links are configured, destination state should record: + +- optional per-output URL metadata; +- optional top-level primary URL metadata. + +Destinations without `links` should continue producing state without URL +metadata. + +Because distributor is still pre-release, URL metadata should be added as +optional state schema version 1 fields rather than introducing destination state +schema version 2 for this feature. Post-release, materially richer state +semantics should use an explicit schema versioning policy. + +## Relationship To Other Roadmaps + +HTML index mode is not required, but link generation should treat `index.html` +as the preferred URL shape when present. + +Latest path destinations should provide the destination bundle path mapping +needed to build fixed/latest URLs correctly. + +Producer-facing manifest creation and remote validation are independent of URL +metadata. + +CLI display of generated primary links should follow +`docs/roadmap/cli_output_policy.md` so link output is exposed through the shared +`--format text|json` model rather than one-off command output. + +## Testing Expectations + +Suggested coverage: + +- config validation accepts absent `links`; +- config validation accepts valid HTTP and HTTPS base URLs; +- config validation rejects invalid schemes, query strings, and fragments; +- nested bundle paths generate correct URLs; +- `index.html` URLs omit the filename; +- non-index URLs include filenames; +- primary selection is deterministic; +- state records URLs when configured; +- destinations without `links` produce no URL metadata. + +## Documentation Updates After Implementation + +- Update `docs/config.md` with destination `links` fields. +- Update `docs/operations.md` with static-site URL examples. +- Update `docs/internal/state.md` if destination state changes. +- Update `docs/cli.md` only if run or inspect output displays links. + +Keep this roadmap under `docs/roadmap/` until implemented. + +## Decisions + +- URL metadata is added as optional destination state schema version 1 fields + while distributor remains pre-release. +- v1 implements `links.primary` so future notification and operator-output + features have one canonical primary-link selection policy. +- `run` and `inspect` output should not display generated primary links + immediately. Link display should wait for + `docs/roadmap/cli_output_policy.md` or another explicit inspect/status output + mode. + +## Future Work + +- Define a post-release destination state schema bump policy before introducing + materially incompatible state changes. +- Add CLI display of primary links through an explicit output mode or status + command, preferably the shared `--format text|json` policy in + `docs/roadmap/cli_output_policy.md`, rather than changing routine `run` + output opportunistically. +- Add richer link policies only if `auto`, `html`, and `source` prove + insufficient. diff --git a/docs/roadmap/manifest_create.md b/docs/roadmap/manifest_create.md new file mode 100644 index 0000000..cad66f7 --- /dev/null +++ b/docs/roadmap/manifest_create.md @@ -0,0 +1,173 @@ +# Roadmap: `distributor manifest create` + +## Purpose + +Add a producer-facing CLI command that creates a valid source `manifest.json` for +a local bundle directory. + +This command depends on the public `pkg/bundle` package. It should be useful +for shell scripts and non-Go producers while sharing behavior with Go producers +through the public package. + +## Current Implementation Grounding + +The current CLI has top-level `version`, `run`, `validate`, and `inspect` +commands. `validate` and `inspect` currently accept local paths only. + +The manifest contract is implemented in `internal/bundle`: source manifests have +`schema_version`, `id`, `digest`, `created`, and ordered `files[]` entries with +`path`, `sha256`, and `size`. Validation already enforces path safety, +reserved distributor metadata paths, digest format, duplicate file paths, file +sizes, per-file SHA-256, and the canonical bundle digest. + +This command should not add a second implementation of those rules. Once +`pkg/bundle` exists, `manifest create` should call it. + +## Goals + +- Add a CLI command that writes `manifest.json` for a local bundle directory. +- Reuse public producer-side manifest creation and validation behavior. +- Produce deterministic manifests. +- Preserve caller-provided file order when explicit files are provided. +- Offer convenient recursive scanning when explicit files are omitted. +- Refuse to overwrite an existing manifest unless explicitly requested. + +## Non-Goals + +- Do not publish or distribute bundles. +- Do not write destination `.distributor.json` state. +- Do not add config-file dependencies. +- Do not add domain-specific metadata. +- Do not expose publish, storage, transform, or destination internals. + +## CLI Shape + +Proposed syntax: + +```sh +distributor manifest create --id +``` + +Optional flags: + +```sh +--file Include a bundle-relative file; repeatable. +--created