diff --git a/README.md b/README.md index 8172117..52c0afd 100644 --- a/README.md +++ b/README.md @@ -12,4 +12,4 @@ Run the local example pipeline: go run ./cmd/distributor run --config examples/local-publish.yml ``` -See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), [docs/operations.md](docs/operations.md), and [docs/troubleshooting.md](docs/troubleshooting.md) for the implemented CLI, configuration, operating notes, and common failure modes. Planning material lives under `docs/roadmap/`. +See [docs/cli.md](docs/cli.md), [docs/config.md](docs/config.md), [docs/operations.md](docs/operations.md), and [docs/troubleshooting.md](docs/troubleshooting.md) for the implemented CLI, configuration, operating notes, and common failure modes. Future and deferred work lives under `docs/roadmap/`. diff --git a/docs/roadmap/cli_output_policy.md b/docs/roadmap/cli_output_policy.md deleted file mode 100644 index 02c05e7..0000000 --- a/docs/roadmap/cli_output_policy.md +++ /dev/null @@ -1,242 +0,0 @@ -# Roadmap: CLI Output Policy - -## Purpose - -Define a shared CLI output policy before adding machine-readable JSON output to -`distributor` commands. - -Current CLI output is human-readable text. That is appropriate as the default, -but upcoming features need a common machine-readable contract so each command -does not invent a separate JSON flag, envelope, warning policy, or error shape. - -## Current Implementation Grounding - -The current CLI has these output-producing commands: - -- `version`; -- `run`; -- `validate`; -- `inspect`. - -Current output is text-only. `run` has the most complex output because it can -print warnings, pipeline summaries, planned destination actions, partial -destination failures, and a final status line. `validate` and `inspect` are -currently local-path commands. `version` prints a single text line. - -Planned `distributor manifest create`, remote `validate` and `inspect`, link -generation, and latest path destinations increase the need for structured -output that scripts can consume consistently. - -## Goals - -- Define one CLI-wide policy for text and JSON output. -- Preserve current human-readable text output as the default. -- Add JSON output consistently across output-producing commands when this - roadmap is implemented. -- Keep warning, error, and partial-failure behavior predictable. -- Avoid external dependencies; use Go's standard `encoding/json`. - -## Non-Goals - -- Do not add YAML, table, NDJSON, streaming JSON, or template output formats. -- Do not change source manifest, destination state, config, or backend schemas. -- Do not make help or usage output JSON. -- Do not turn ordinary fatal setup errors into structured JSON results. -- Do not add a root-global output flag in the first implementation. - -## Shared CLI Policy - -Each supported command should accept: - -```sh ---format text ---format json -``` - -Policy: - -- `text` is the default and preserves existing output unless a feature - explicitly changes text output. -- `json` writes exactly one JSON document to stdout. -- help and usage output remain text-only. -- invalid `--format` values are usage errors. -- `--format` is a per-command flag, not a root-global flag. -- a command must not accept `--format json` unless it emits the shared JSON - envelope for that command. - -The first implementation should add JSON support for all current -output-producing commands rather than leaving a mixed CLI where some commands -support `--format` and others do not. - -## Stdout, Stderr, Warnings, and Errors - -Text mode keeps the current behavior: - -- normal command output goes to stdout; -- warnings may be printed as text; -- command errors are printed to stderr by CLI error handling. - -JSON mode: - -- successful commands write one JSON document to stdout; -- warnings are included in a top-level `warnings` array and are not duplicated - to stderr; -- fatal setup errors that prevent construction of a result write text errors to - stderr, write no JSON stdout, and exit non-zero; -- commands with meaningful partial results may write a JSON document with - `ok: false`, structured failure details, and a non-zero exit status. - -`run --format json` should use the partial-result behavior when planning or -execution has begun and one or more destinations fail. This lets automation -inspect successful actions, failed actions, warnings, and final counters even -when the process exits non-zero. - -## JSON Envelope - -All JSON-mode commands should use a common top-level envelope: - -```json -{ - "schema_version": 1, - "command": "inspect", - "ok": true, - "warnings": [], - "result": {} -} -``` - -Envelope fields: - -- `schema_version`: integer version of the CLI JSON output schema. -- `command`: command name, using the public CLI command path where useful, such - as `manifest create`. -- `ok`: boolean success indicator for the command result. -- `warnings`: array of structured warning objects. -- `result`: command-specific result object. -- `errors`: optional array of structured error objects for commands that can - return partial results. - -Field rules: - -- use stable snake_case field names; -- use RFC3339 timestamps; -- use numeric JSON values for sizes and counts; -- use slash-separated logical paths for bundle, source, destination, and output - paths; -- never include secret values; -- keep command-specific data under `result`; -- add fields compatibly where practical. - -Warning objects should include at least: - -```json -{ - "message": "secret OBJECT_STORAGE_KEY ignored because the real environment already has that variable" -} -``` - -Error objects for partial results should include enough context for automation -and troubleshooting, such as pipeline id, destination id, backend, bundle path, -and message when those values are available. - -## Command Adoption - -`version`: - -- text mode keeps the current version string; -- JSON mode reports application name and version under `result`. - -`validate`: - -- text mode keeps the current validation summary; -- JSON mode reports bundle count and selected bundle identifiers; -- future remote validation uses the same envelope and adds pipeline/source - context under `result`. - -`inspect`: - -- text mode keeps human-readable bundle metadata; -- JSON mode reports normalized bundle metadata, including source-relative path, - bundle id, created timestamp, digest, file count, total size, and file - records. - -`run`: - -- text mode keeps the current progress and final status style unless a future - feature intentionally changes it; -- JSON mode reports warnings, pipeline summaries, destination action records, - output records, final counters, dry-run status, and partial failure records; -- JSON mode may write `ok: false` with partial results and still exit non-zero. - -Future `manifest create`: - -- should use `--format text|json`; -- text mode should keep concise human success output; -- JSON mode should report the generated manifest summary under `result`; -- the command should not introduce a separate `--json` flag. - -## Relationship To Other Roadmaps - -`distributor manifest create` should depend on this roadmap for JSON summary -output and should not add a command-specific JSON flag. - -Remote `validate` and `inspect` should use this policy for machine-readable -source validation and inspection. - -Link generation should expose primary links through JSON command output only -after this policy is implemented or selected for implementation. - -Latest path destinations should report fixed-path selection and destructive -replacement summaries through the same `run --format json` result model. - -## Testing Expectations - -Suggested coverage: - -- CLI parsing accepts `--format text` and `--format json` for supported - commands; -- CLI parsing rejects invalid `--format` values as usage errors; -- help and usage output remain text-only; -- each JSON-capable command emits exactly one valid JSON document to stdout on - success; -- text mode preserves existing output; -- JSON-mode warnings appear in `warnings` and are not duplicated to stderr; -- fatal setup errors write no JSON stdout and return non-zero; -- `run --format json` emits partial-result JSON with `ok: false` and exits - non-zero when one or more destination failures occur after planning begins; -- JSON output uses RFC3339 timestamps, numeric sizes and counts, and - slash-separated logical paths; -- no JSON output includes secret values; -- documentation consistency checks find no unsupported `--json` references. - -## Documentation Updates After Implementation - -- Update `docs/cli.md` with the shared `--format text|json` policy. -- Update command examples only for implemented JSON behavior. -- Update `docs/operations.md` where JSON output materially improves automation - workflows. -- Update `docs/troubleshooting.md` only for implemented JSON-mode recovery - behavior. -- Update relevant roadmap files when JSON output is no longer deferred. - -Keep this roadmap under `docs/roadmap/` until implemented. - -## Decisions - -- Use per-command `--format text|json`; do not introduce `--json`. -- Keep `text` as the default for backward compatibility. -- Implement JSON support for all current output-producing commands when this - roadmap is selected. -- Keep help and usage output text-only. -- Put JSON-mode warnings in the top-level `warnings` array. -- Allow `run --format json` to emit partial-result JSON with `ok: false` and a - non-zero exit status. -- Keep fatal setup errors as text stderr with no JSON stdout. - -## Future Work - -- Consider a root-global output flag only if the command parser is later - refactored around shared root options. -- Consider additional formats only if a concrete consumer requires them. -- Consider a versioned JSON schema reference after the first JSON-capable - release. diff --git a/docs/roadmap/html_index_mode.md b/docs/roadmap/html_index_mode.md deleted file mode 100644 index f812b6f..0000000 --- a/docs/roadmap/html_index_mode.md +++ /dev/null @@ -1,160 +0,0 @@ -# Roadmap: HTML Index Mode - -## Purpose - -Add a Markdown-to-HTML `index` mode that renders one selected Markdown artifact -to `index.html` at the destination bundle path. - -This feature improves static-site UX and gives link generation and latest path -destinations a clean directory-style output to prefer. - -## Current Implementation Grounding - -Current Markdown transformation lives in `internal/transform/markdown`. It -renders every manifest-listed `.md` file to a sidecar `.html` file: - -```text -report.md -> report.html -``` - -Current config validation accepts only `markdown_to_html.mode: sidecar`. -`publish.PlanOutputs` asks the configured transformer for generated outputs, -then destination state records generated outputs with path, kind, source path, -transform name, SHA-256, and size. Current destination state has no field for a -transform mode beyond the existing transform string. - -## Goals - -- Preserve existing sidecar behavior as the default and supported explicit mode. -- Add `markdown_to_html.mode: index`. -- In index mode, render exactly one Markdown input to `index.html`. -- Record `index.html` as a normal generated output in `.distributor.json`. -- Keep source manifests unchanged. -- Keep link generation and latest paths straightforward without making them - dependencies. - -## Non-Goals - -- Do not add collection index pages. -- Do not add multi-page static-site generation. -- Do not add feeds, notifications, or link generation in this feature. -- Do not require producers to name a file `index.md`. -- Do not add custom output filenames in v1. - -## Configuration Shape - -Extend existing destination-level transform config: - -```yaml -transform: - markdown_to_html: - enabled: true - mode: index - input: report.md -``` - -Mode semantics: - -- `sidecar`: existing behavior; render each manifest-listed Markdown file to a - same-directory `.html` file. -- `index`: render one selected Markdown file to `index.html` at the destination - bundle path. - -`sidecar` remains the current/default mode. `index` is the proposed addition. - -## Input Selection - -Index mode should choose the Markdown input deterministically: - -1. If `transform.markdown_to_html.input` is configured, use that manifest-listed - Markdown file. -2. If no input is configured and the source manifest lists exactly one Markdown - file, use that file. -3. If no input is configured and there are zero or multiple Markdown files, fail - planning with a clear error. - -The configured input path must be a safe relative source path, must be listed in -the source manifest, and must end in `.md`. - -Input selection happens during planning because it depends on the validated -source manifest, not only static config. - -## Output Semantics - -Index mode always writes: - -```text -index.html -``` - -relative to the destination bundle path. - -The generated output should use current state model terms: - -- `path`: `index.html`; -- `kind`: `generated`; -- `source_path`: selected Markdown file; -- `transform`: `markdown_to_html`; -- `sha256` and `size`: digest and byte size of generated HTML. - -Replacement, skip, cleanup, and force behavior should treat `index.html` like -any other distributor-managed generated output. - -Config validation should reject enabled Markdown-to-HTML transform -configuration when `publish.html` is false. That keeps publish policy and -transform intent aligned and avoids silently accepting unused transform config. - -## Relationship To Other Roadmaps - -Link generation should recognize `index.html` and produce directory-style URLs, -but link generation must not be required for index mode. - -Latest path destinations work without index mode, but fixed latest destinations -produce nicer stable URLs when they publish `index.html`. - -`manifest create`, `pkg/bundle`, and remote validation remain source-bundle -features and should not depend on transform mode. - -## Testing Expectations - -Suggested coverage: - -- config validation accepts `mode: sidecar` and `mode: index`; -- current sidecar behavior remains unchanged; -- index mode accepts explicit input; -- index mode chooses the only Markdown file when no input is configured; -- planning fails when index mode has zero or multiple Markdown candidates - without explicit input; -- planning fails when explicit input is unsafe, not listed, or not Markdown; -- index mode emits `index.html`; -- state records `index.html` as a generated output; -- source-only publication does not write `index.html`; -- dry-run reports `index.html` without writing it; -- replacement safely updates prior generated `index.html`. - -## Documentation Updates After Implementation - -- Update `docs/config.md` with Markdown-to-HTML modes and `input`. -- Update `docs/integrations/markdown.md` with sidecar and index behavior. -- Update `docs/operations.md` with a static-site example. -- Add or update examples only for implemented behavior. - -Keep this roadmap under `docs/roadmap/` until implemented. - -## Decisions - -- Enabled Markdown-to-HTML transform config is rejected when `publish.html` is - false. -- Generated output metadata keeps `transform: markdown_to_html`; the output - path and destination config distinguish sidecar from index behavior. -- Future multi-page or collection index generation should be a separate - transform, not an expansion of this single-input index mode. - -## Future Work - -- Add a separate collection or site-index transform if distributor later needs - multi-page aggregation. -- Consider richer transform metadata only if future state consumers need more - than the transform name and output path. -- Consider custom index output names only if fixed `index.html` proves too - limiting for real deployments. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 39756da..eeb32bb 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,10 +1,8 @@ -# Implementation Roadmap +# Roadmap -This roadmap records current implementation status and deferred work for -`distributor`. Implemented behavior is documented in the user, operator, and -internal documentation listed below. - -Canonical current-behavior docs: +This directory contains only future, deferred, or aspirational work for +`distributor`. Implemented behavior is documented in the current user, +operator, internal, policy, integration, and example documentation: - `README.md` - `docs/cli.md` @@ -16,338 +14,89 @@ Canonical current-behavior docs: - `docs/policy/` - `examples/` -Future, planned, or aspirational behavior belongs under `docs/roadmap/` until -it is implemented. - -## Current State - -`distributor` is ready for routine use against producer pipelines using the -implemented local, SSH/SFTP, and S3-compatible backends. - -## Active Roadmap - -The active roadmap implements the focused feature roadmaps in an order intended -to minimize later rewrites. The sequence establishes shared CLI output before -new CLI commands, establishes producer bundle APIs before manifest creation, -and implements destination path mapping before link generation so URL -construction does not need to be reworked later. - -Each stage is sized for one implementation prompt. Future behavior stays under -`docs/roadmap/` until the stage is implemented and current-behavior docs are -updated. - -### Stage 1: CLI Output Policy - -Goal: implement the shared CLI output contract in -`docs/roadmap/cli_output_policy.md`. - -Implementation scope: - -- add per-command `--format text|json` to current output-producing commands: - `version`, `validate`, `inspect`, and `run`; -- keep `text` as the default and preserve existing text output unless the - policy explicitly requires a change; -- implement the shared JSON envelope, warnings array, fatal setup error - behavior, and `run` partial-result JSON with `ok: false`; -- keep help and usage output text-only. - -Documentation updates after implementation: - -- update `docs/cli.md` with the shared `--format text|json` policy; -- update `docs/operations.md` where JSON output materially improves automation - workflows; -- update `docs/troubleshooting.md` only for implemented JSON-mode recovery - behavior. - -Tests: - -- CLI parsing accepts `--format text` and `--format json`; -- invalid `--format` values are usage errors; -- each JSON-capable command emits exactly one valid JSON document on success; -- JSON-mode warnings appear in `warnings` and are not duplicated to stderr; -- fatal setup errors write no JSON stdout and return non-zero; -- `run --format json` emits partial-result JSON and exits non-zero when - destination failures occur after planning begins. - -Completion criteria: all current output-producing commands support the shared -policy, text output remains backward compatible, and `go test ./...` passes. - -### Stage 2: Public Bundle Package Core - -Goal: implement the core and manifest-building parts of -`docs/roadmap/public_bundle_package.md`. - -Implementation scope: - -- introduce `pkg/bundle` with the public source manifest model, schema version, - digest logic, parsing, validation, explicit file-list building, scan-based - building, and zero-`Created` defaulting to current UTC; -- implement the locked Stage 2 exported API symbols defined in - `docs/roadmap/public_bundle_package.md`; -- update internal validation to consume the shared implementation without - behavior drift; -- keep destination state, publish planning, storage backends, transforms, - notifications, and config internal. - -Documentation updates after implementation: - -- add package documentation for `pkg/bundle`; -- update `README.md` and `docs/operations.md` only for implemented Go producer - support; -- update `docs/internal/bundle.md` if internal ownership changes. - -Tests: - -- public package tests cover explicit lists, scan mode, ordering, digest - calculation, parsing, validation, unsafe paths, symlinks, and zero `Created`; -- internal bundle tests continue to pass against the shared implementation; -- public examples compile where practical. - -Completion criteria: public and internal validation use one manifest contract, -and tests prove identical semantics. - -### Stage 3: Public Bundle Local Writer - -Goal: implement the local bundle writer portion of -`docs/roadmap/public_bundle_package.md`. - -Implementation scope: - -- add producer-side local bundle writing with staging, atomic filesystem - operations where practical, and best-effort restore on overwrite failure; -- implement the locked Stage 3 exported API symbols defined in - `docs/roadmap/public_bundle_package.md`; -- keep the writer filesystem-local and producer-focused; -- do not expose distributor storage backends or publication behavior through - the public package. - -Documentation updates after implementation: - -- document the writer API in `pkg/bundle`; -- update producer workflow docs only for implemented behavior. - -Tests: - -- writer creates a complete valid local bundle through staged promotion; -- writer output validates through distributor's normal validation path; -- overwrite and failure behavior avoid leaving a completed bundle path without - a valid manifest where practical. - -Completion criteria: producers can create complete valid local bundles through -the public API without hand-writing manifest files. - -### Stage 4: Manifest Create CLI - -Goal: implement `docs/roadmap/manifest_create.md`. - -Implementation scope: - -- add `distributor manifest create` over `pkg/bundle`; -- support explicit-first `--file` selection, recursive scan fallback, - `--overwrite`, optional `--created`, and shared `--format text|json`; -- use temp-and-rename replacement for `manifest.json` where practical; -- do not duplicate bundle contract logic in CLI or app code. - -Documentation updates after implementation: - -- update `docs/cli.md` with command syntax, flags, output, and examples; -- update `docs/operations.md` with a producer workflow; -- cross-reference `pkg/bundle` for Go producers. - -Tests: - -- command creates valid manifests for explicit and scanned files; -- ordering, dotfile inclusion, metadata exclusion, symlink rejection, and - overwrite behavior match the roadmap; -- text and JSON output follow the shared CLI output policy. - -Completion criteria: the command writes valid `manifest.json`, supports -text/JSON output, and generated manifests validate through distributor. - -### Stage 5: Remote Validate and Inspect - -Goal: implement `docs/roadmap/remote_validate_inspect.md`. - -Implementation scope: - -- extend `validate` and `inspect` with mutually exclusive local-path and - `--config --pipeline` modes; -- reuse `run` source backend construction and secrets resolution; -- support configured local, SSH, and S3 sources; -- preserve `--pipeline` as required in config mode; -- keep the commands source-only and do not open destinations. - -Documentation updates after implementation: - -- update `docs/cli.md` with local and config-driven syntax; -- update `docs/operations.md` with remote validation examples; -- update `docs/troubleshooting.md` for common configured-source failures. - -Tests: - -- existing local behavior remains unchanged; -- config-mode validation and inspection work for local and fake/app-level - source backends; -- SSH and S3 behavior is covered by focused unit tests and existing opt-in - integration patterns; -- JSON output follows the shared CLI output policy. - -Completion criteria: local behavior is stable, configured local/SSH/S3 sources -can be validated and inspected, and no destination backend is opened. - -### Stage 6: HTML Index Mode - -Goal: implement `docs/roadmap/html_index_mode.md`. - -Implementation scope: - -- add `markdown_to_html.mode: index` with fixed `index.html` output; -- keep `sidecar` as the default and existing explicit mode; -- implement deterministic input selection through explicit input or exactly one - manifest-listed Markdown file; -- reject enabled Markdown-to-HTML transform config when `publish.html` is - false; -- record generated state metadata with `transform: markdown_to_html`. - -Documentation updates after implementation: - -- update `docs/config.md` with Markdown-to-HTML modes and input selection; -- update `docs/integrations/markdown.md`; -- update `docs/operations.md` with an implemented static-site example where - useful. - -Tests: - -- sidecar behavior remains unchanged; -- index mode handles explicit input, single Markdown fallback, ambiguous input, - unsafe input, state metadata, dry-run, source-only publication, and output - collisions. - -Completion criteria: source publication, sidecar mode, index mode, state -metadata, dry-run, and collision behavior are all covered. - -### Stage 7: Latest Path Destinations - -Goal: implement `docs/roadmap/latest_paths.md`. - -Implementation scope: - -- add destination-level `path_mapping.mode` with default `preserve_relative` - and new `fixed`; -- make destination bundle path mapping destination-local; -- for fixed destinations, select only the newest discovered bundle per - destination before planning writes; -- add fixed-path dry-run warnings and summary counts; -- allow `--force` for fixed backend roots while keeping deletion bounded to the - configured backend root. - -Documentation updates after implementation: - -- update `docs/config.md` with `path_mapping`; -- update `docs/operations.md` with archive-plus-latest fan-out examples; -- update `docs/cli.md` if dry-run output gains fixed-path indicators. - -Tests: - -- omitted and explicit `preserve_relative` match existing behavior; -- fixed destinations publish at local, SSH, and S3 backend roots; -- newest-only selection is deterministic; -- older discovered bundles are not planned or written to fixed destinations; -- dry-run reports candidate count, selected bundle, and destructive replacement - warnings; -- force remains bounded at the backend root. - -Completion criteria: fixed destination roots work across implemented backends, -archive-style destinations remain unchanged, and destructive behavior is clear -in dry-run output. - -### Stage 8: Link Generation Support - -Goal: implement `docs/roadmap/link_generation.md`. - -Implementation scope: - -- add destination-level `links.base_url` and `links.primary`; -- generate per-output and primary URLs using URL semantics, not filesystem or - storage joins; -- handle `index.html` as a directory-style URL; -- store optional URL metadata under destination state schema version 1 while - pre-release; -- expose links through JSON command output only where supported by the CLI - output policy. - -Documentation updates after implementation: - -- update `docs/config.md` with destination `links` fields; -- update `docs/operations.md` with static-site URL examples; -- update `docs/internal/state.md` for URL metadata; -- update `docs/cli.md` only for implemented CLI link output. - -Tests: - -- config validation accepts valid links and rejects invalid schemes, query - strings, and fragments; -- nested bundle paths and fixed latest paths generate correct URLs; -- `index.html` omits the filename; -- primary selection is deterministic; -- destinations without `links` produce no URL metadata. - -Completion criteria: URL generation respects archive and fixed path mapping, -state records optional URL metadata when configured, and unconfigured -destinations remain unchanged. - -### Stage 9: Roadmap Closeout - -Goal: remove completed roadmap drift after Stages 1-8 are implemented. - -Implementation scope: - -- remove or rewrite completed roadmap files whose behavior is fully documented - in current docs; -- ensure current docs describe implemented behavior; -- keep `docs/roadmap/` focused only on remaining future work. - -Documentation updates after implementation: - -- update `README.md`, `docs/cli.md`, `docs/config.md`, - `docs/operations.md`, `docs/troubleshooting.md`, `docs/internal/`, - `docs/integrations/markdown.md`, and examples only where implemented - behavior requires it. - -Tests: - -- run documentation consistency searches for completed-feature language that - still appears only as future work; -- run focused tests for any examples or docs backed by tests; -- run `go test ./...` if behavior docs and examples changed with code. - -Completion criteria: completed features are documented as current behavior, and -`docs/roadmap/` contains only future or deferred work. - -## Deferred Work - -These items are not implemented and should stay out of current-behavior docs -until a roadmap entry is selected and implemented: - -- external notification adapters; -- warning-only digest mismatch handling; -- additional auth mechanisms beyond the implemented SSH and S3 credential - paths; -- compatibility parsing for legacy SSH URI config; -- broad recursive destination deletion outside managed bundle paths; -- concurrent fan-out publishing; -- streaming, resumable, or multipart S3 uploads; -- cloud-provider-specific IAM integration docs; -- repository-managed packaging, release, and deployment automation. +`distributor` currently supports local, SSH/SFTP, and S3-compatible source and +destination backends; producer bundle creation through `pkg/bundle` and +`distributor manifest create`; configured source validation and inspection; +Markdown sidecar and `index.html` publication; archive and fixed destination +path mapping; destination link metadata; shared text/JSON CLI output; and +managed destination replacement behavior. + +## Future Work + +These items are not implemented. They should not be documented as current +behavior outside `docs/roadmap/` unless a future implementation adds them. + +### CLI And Status Output + +- Add a root-global output flag only if the command parser is later refactored + around shared root options. +- Add output formats beyond `text` and `json` only if a concrete consumer + requires them. +- Add a versioned JSON schema reference after the first JSON-capable release. +- Add destination-state inspection behind an explicit flag such as + `--with-destinations` if operators need fan-out status diagnostics from + `inspect`. +- Add additional status or inspection presentation for destination primary + links beyond the current `run --format json` result model. + +### Producer Workflows + +- Add a no-write manifest creation mode, such as writing manifest JSON to + stdout, if producer pipelines need to capture manifests directly. +- Add broader producer workflow helpers, such as richer ignore rules or + template scaffolding, if real producer use cases require them. +- Add remote or storage-backed producer writers only if producer applications + need to assemble bundles outside the local filesystem. + +### Publication And Transform Behavior + +- Add a separate collection or site-index transform if distributor needs + multi-page aggregation. +- Add richer transform metadata only if future state consumers need more than + the transform name and output path. +- Add custom HTML index output names only if fixed `index.html` is too limiting + for real deployments. +- Add richer fixed-destination source selection policies if deployments need + something other than newest-by-`created`. +- Add stricter handling for equal latest timestamps if timestamp ties become + common in producer workflows. +- Add higher-level status or approval workflows for fixed-root replacements if + dry-run output is not enough operational protection. +- Add richer link policies only if `auto`, `html`, and `source` prove + insufficient. + +### State And Compatibility + +- Define a post-release destination state schema bump policy before introducing + materially incompatible state changes. +- Add warning-only digest mismatch handling only if an operator workflow needs + publication to continue after validation failures. +- Add compatibility parsing for legacy SSH URI config only if migration support + is required. + +### Backends, Security, And Deployment + +- Add authentication mechanisms beyond the implemented SSH agent/key and S3 + credential paths only when a concrete backend workflow requires them. +- Add broad recursive destination deletion outside managed bundle paths only if + a future design can preserve the current safety boundary. +- Add concurrent fan-out publishing only if runtime profiling shows it is + needed. +- Add streaming, resumable, or multipart S3 uploads only if object sizes make + the current write path insufficient. +- Add cloud-provider-specific IAM integration docs only when the repository + includes tested provider-specific behavior. +- Add repository-managed packaging, release, and deployment automation when the + release process is ready to be standardized. ## Roadmap Maintenance When adding future roadmap work: - describe user-visible behavior and safety boundaries; -- define which existing docs must change after implementation; +- define which current docs must change after implementation; - keep examples secret-free and runnable or clearly environment-gated; -- avoid workflow labels in production code, tests, config fields, and user - documentation; +- keep workflow labels out of production code, tests, config fields, and + user-facing documentation; - run focused tests for the changed behavior and `go test ./...` for cross-package changes. diff --git a/docs/roadmap/latest_paths.md b/docs/roadmap/latest_paths.md deleted file mode 100644 index eb34428..0000000 --- a/docs/roadmap/latest_paths.md +++ /dev/null @@ -1,212 +0,0 @@ -# Roadmap: Latest Path Destinations - -## Purpose - -Support stable "latest" publication paths as normal fan-out destinations. - -A latest destination republishes the current winning source bundle to a fixed -destination path such as `/weather/latest/`, while archive destinations preserve -source-relative bundle paths such as `/weather/daily/brentwood/2026-06-01/`. - -This feature should be implemented before link generation so URL construction -can use final destination path semantics. It is operationally destructive and -benefits from index-mode HTML, although it must not require index mode or link -generation. - -## Current Implementation Grounding - -Current run behavior computes the destination bundle path as the source bundle -path relative to the source root. Publication then writes selected outputs and -`.distributor.json` below that destination bundle path. - -Backend roots differ by backend: - -- local and SSH/SFTP use configured `path` as the backend root; -- S3 uses configured bucket plus optional `prefix` as the backend root. - -Destination comparison uses `.distributor.json` at the destination bundle path. -Replacement and skip decisions are based on the normalized source manifest -recorded in destination state. Unmanaged non-empty destinations fail unless -explicit force behavior is requested. - -## Goals - -- Add destination-local path mapping. -- Preserve source-relative destination paths as the default. -- Add fixed path mapping for latest-style destinations. -- For fixed destinations, select only the newest discovered source bundle for - publication. -- Make fixed mapping work for local, SSH/SFTP, and S3 backends. -- Continue using normal destination state comparison and replacement rules. -- Keep link generation and index-mode HTML complementary, not required. - -## Non-Goals - -- Do not add symlink-based latest behavior. -- Do not add feed or index generation. -- Do not change source manifest schema. -- Do not make producers responsible for latest publication. -- Do not add domain-specific latest selection rules. - -## Configuration Shape - -Add destination-level path mapping: - -```yaml -path_mapping: - mode: preserve_relative -``` - -Modes: - -- `preserve_relative`: existing/default behavior. -- `fixed`: publish selected outputs directly at the destination backend root. - -Example fixed destination: - -```yaml -destinations: - - id: latest-html - backend: ssh - host: web.example.com - user: deploy - path: /srv/www/weather/latest - path_mapping: - mode: fixed - publish: - source: false - html: true -``` - -For S3, the fixed destination root is the configured bucket plus `prefix`. - -## Path Mapping Semantics - -`preserve_relative`: - -```text -destination bundle path = source-root-relative bundle path -``` - -`fixed`: - -```text -destination bundle path = "" -``` - -That empty logical destination bundle path means the destination backend root. -Storage writes still use normal backend-rooted logical paths for outputs and -state. - -## Newest Bundle Selection - -`preserve_relative` destinations should continue publishing every selected -source bundle independently. - -`fixed` destinations should publish only one bundle per run: the newest -discovered source bundle selected for that destination. Newest selection should -be deterministic: - -1. choose the bundle with the greatest source manifest `created` timestamp; -2. if multiple bundles have the same greatest `created` timestamp, use the - source-root-relative bundle path ascending as a tie-breaker. - -The selected bundle then uses the existing destination comparison, skip, -replacement, cleanup, and force rules. Older discovered bundles should not be -planned or written to that fixed destination in the same run. - -Dry-run output should identify the number of fixed-destination candidates and -the selected source bundle so operators can see which bundle would become -latest. - -## Safety Rules - -Fixed path mapping is more destructive than archive-style publication because -newer bundles replace prior contents at the same destination path. - -Required behavior: - -- dry-run output must identify fixed path mapping and planned replacement; -- dry-run output must include an extra fixed-path warning or summary count when - destructive replacement is possible; -- replacement should delete only managed outputs recorded in valid destination - state when possible; -- unmanaged non-empty fixed destinations must fail unless force is explicitly - requested; -- `--force` may be used when fixed mapping targets the backend root, but force - replacement must remain bounded to the configured destination backend root and - must never delete above it; -- fixed mapping to the backend root requires especially clear dry-run reporting. - -## Relationship To Other Roadmaps - -HTML index mode is useful for fixed web destinations because `index.html` -supports stable directory-style URLs, but sidecar HTML and source-only outputs -should still work. - -Link generation should use fixed path semantics so latest URLs are based on the -fixed destination root, not the original source-relative archive path. - -Producer manifest creation and remote validation are independent of destination -path mapping. - -## Implementation Stages - -1. Add destination-level `path_mapping` config validation with - `preserve_relative` as the default and `fixed` as the new mode. -2. Refactor run planning so destination bundle path mapping is destination-local - and can be evaluated before writes. -3. Add fixed-destination newest selection so each fixed destination receives - only the newest discovered source bundle. -4. Integrate fixed mapping with destination comparison, managed deletion, force - replacement, and dry-run reporting. -5. Update local, SSH, and S3 tests to verify fixed root behavior under each - backend root model. -6. Update current-behavior documentation after implementation. - -## Testing Expectations - -Suggested coverage: - -- omitted `path_mapping` preserves existing behavior; -- `preserve_relative` explicitly matches existing behavior; -- `fixed` publishes outputs and `.distributor.json` at the backend root; -- fixed destinations select only the newest discovered bundle; -- fixed destination tie-break behavior is deterministic; -- older discovered bundles are not planned or written to fixed destinations; -- newer source replaces older managed fixed destination state; -- older source skips newer fixed destination state; -- unmanaged non-empty fixed destination fails without force; -- dry-run reports fixed path mapping, candidate count, selected bundle, and - destructive replacement warnings clearly; -- `--force` remains bounded when fixed mapping targets the backend root; -- S3 fixed mapping respects bucket plus prefix as backend root. - -## Documentation Updates After Implementation - -- Update `docs/config.md` with `path_mapping`. -- Update `docs/operations.md` with archive-plus-latest fan-out examples. -- Update `docs/cli.md` if dry-run output gains fixed-path indicators. -- Add an environment-safe example only after behavior is implemented. - -Keep this roadmap under `docs/roadmap/` until implemented. - -## Decisions - -- `--force` is allowed when fixed mapping targets the backend root, but dry-run - and run output must make the fixed-root replacement explicit and force remains - bounded to the configured backend root. -- Fixed destinations publish only the newest discovered bundle, rather than - processing every discovered bundle and letting later writes replace earlier - writes. -- Dry-run includes an extra warning or summary count for fixed-path destructive - replacements. - -## Future Work - -- Add richer source selection policies if deployments need something other than - newest-by-`created` for fixed destinations. -- Consider stricter ambiguity handling for equal latest timestamps if real - producer workflows make timestamp ties common. -- Add higher-level status or approval workflows for fixed-root replacements if - dry-run output is not enough operational protection. diff --git a/docs/roadmap/link_generation.md b/docs/roadmap/link_generation.md deleted file mode 100644 index 86875b5..0000000 --- a/docs/roadmap/link_generation.md +++ /dev/null @@ -1,179 +0,0 @@ -# Roadmap: Link Generation Support - -## Purpose - -Add destination-aware link generation so distributor can record human-usable URLs -for published artifacts. - -This feature should be implemented after destination path mapping so generated -URLs use the final archive or fixed destination path semantics. It prepares the -project for notification adapters, richer inspect/status output, static-site UX, -and future feed generation. - -## Current Implementation Grounding - -Current publish planning produces `publish.Output` values with destination path, -source path, kind, transform, SHA-256, size, and optional generated data. - -Current `.distributor.json` output records contain path, kind, source path, -transform, SHA-256, and size. They do not contain URL metadata, and destination -state has no top-level links block. - -Current destination bundle paths preserve the source-root-relative bundle path. -Latest/fixed path mapping is not implemented yet. - -## Goals - -- Add optional destination-level link configuration. -- Generate per-output URLs when a destination has `links.base_url`. -- Record generated URLs in destination state when configured. -- Select a deterministic primary URL when possible. -- Keep link generation destination-local and independent of producer manifests. -- Do not infer public URLs from backend configuration automatically. - -## Non-Goals - -- Do not deliver notifications. -- Do not require every destination to expose URLs. -- Do not add static-site indexes or feeds. -- Do not add latest path mapping. -- Do not change source manifest schema. - -## Configuration Shape - -Add optional destination-level config: - -```yaml -links: - base_url: https://weather.example.com - primary: auto -``` - -Semantics: - -- `links.base_url`: absolute HTTP or HTTPS URL corresponding to the destination - backend root. -- `links.primary`: optional primary-link selection policy. - -Initial `primary` policies: - -- `auto`: choose the best available output; -- `html`: prefer generated HTML outputs; -- `source`: prefer copied source outputs. - -If `links` is absent, no URL metadata is generated. - -## URL Construction - -Construct URLs from: - -1. `links.base_url`; -2. destination bundle path relative to the destination backend root; -3. output path relative to the destination bundle path. - -Use URL path joining and escaping rules, not filesystem or storage path joins. -Preserve any path prefix in `base_url`. Reject `base_url` values with query -strings or fragments. - -For `index.html`, generate a directory-style URL by omitting the filename: - -```text -https://weather.example.com/daily/brentwood/2026-06-01/ -``` - -For other outputs, include the output filename: - -```text -https://weather.example.com/daily/brentwood/2026-06-01/report.html -``` - -`index.html` recognition should apply to any output path ending in -`/index.html` or exactly `index.html`. - -## Primary Link Selection - -For `primary: auto`, select in deterministic publish-plan order: - -1. any `index.html` output; -2. first generated HTML output; -3. first source output; -4. no primary URL. - -For `primary: html`, prefer generated HTML outputs. For `primary: source`, -prefer source outputs. If no output matches the policy, record no primary link -rather than failing publication. - -## Destination State - -When links are configured, destination state should record: - -- optional per-output URL metadata; -- optional top-level primary URL metadata. - -Destinations without `links` should continue producing state without URL -metadata. - -Because distributor is still pre-release, URL metadata should be added as -optional state schema version 1 fields rather than introducing destination state -schema version 2 for this feature. Post-release, materially richer state -semantics should use an explicit schema versioning policy. - -## Relationship To Other Roadmaps - -HTML index mode is not required, but link generation should treat `index.html` -as the preferred URL shape when present. - -Latest path destinations should provide the destination bundle path mapping -needed to build fixed/latest URLs correctly. - -Producer-facing manifest creation and remote validation are independent of URL -metadata. - -CLI display of generated primary links should follow -`docs/roadmap/cli_output_policy.md` so link output is exposed through the shared -`--format text|json` model rather than one-off command output. - -## Testing Expectations - -Suggested coverage: - -- config validation accepts absent `links`; -- config validation accepts valid HTTP and HTTPS base URLs; -- config validation rejects invalid schemes, query strings, and fragments; -- nested bundle paths generate correct URLs; -- `index.html` URLs omit the filename; -- non-index URLs include filenames; -- primary selection is deterministic; -- state records URLs when configured; -- destinations without `links` produce no URL metadata. - -## Documentation Updates After Implementation - -- Update `docs/config.md` with destination `links` fields. -- Update `docs/operations.md` with static-site URL examples. -- Update `docs/internal/state.md` if destination state changes. -- Update `docs/cli.md` only if run or inspect output displays links. - -Keep this roadmap under `docs/roadmap/` until implemented. - -## Decisions - -- URL metadata is added as optional destination state schema version 1 fields - while distributor remains pre-release. -- v1 implements `links.primary` so future notification and operator-output - features have one canonical primary-link selection policy. -- `run` and `inspect` output should not display generated primary links - immediately. Link display should wait for - `docs/roadmap/cli_output_policy.md` or another explicit inspect/status output - mode. - -## Future Work - -- Define a post-release destination state schema bump policy before introducing - materially incompatible state changes. -- Add CLI display of primary links through an explicit output mode or status - command, preferably the shared `--format text|json` policy in - `docs/roadmap/cli_output_policy.md`, rather than changing routine `run` - output opportunistically. -- Add richer link policies only if `auto`, `html`, and `source` prove - insufficient. diff --git a/docs/roadmap/manifest_create.md b/docs/roadmap/manifest_create.md deleted file mode 100644 index cad66f7..0000000 --- a/docs/roadmap/manifest_create.md +++ /dev/null @@ -1,173 +0,0 @@ -# Roadmap: `distributor manifest create` - -## Purpose - -Add a producer-facing CLI command that creates a valid source `manifest.json` for -a local bundle directory. - -This command depends on the public `pkg/bundle` package. It should be useful -for shell scripts and non-Go producers while sharing behavior with Go producers -through the public package. - -## Current Implementation Grounding - -The current CLI has top-level `version`, `run`, `validate`, and `inspect` -commands. `validate` and `inspect` currently accept local paths only. - -The manifest contract is implemented in `internal/bundle`: source manifests have -`schema_version`, `id`, `digest`, `created`, and ordered `files[]` entries with -`path`, `sha256`, and `size`. Validation already enforces path safety, -reserved distributor metadata paths, digest format, duplicate file paths, file -sizes, per-file SHA-256, and the canonical bundle digest. - -This command should not add a second implementation of those rules. Once -`pkg/bundle` exists, `manifest create` should call it. - -## Goals - -- Add a CLI command that writes `manifest.json` for a local bundle directory. -- Reuse public producer-side manifest creation and validation behavior. -- Produce deterministic manifests. -- Preserve caller-provided file order when explicit files are provided. -- Offer convenient recursive scanning when explicit files are omitted. -- Refuse to overwrite an existing manifest unless explicitly requested. - -## Non-Goals - -- Do not publish or distribute bundles. -- Do not write destination `.distributor.json` state. -- Do not add config-file dependencies. -- Do not add domain-specific metadata. -- Do not expose publish, storage, transform, or destination internals. - -## CLI Shape - -Proposed syntax: - -```sh -distributor manifest create --id -``` - -Optional flags: - -```sh ---file Include a bundle-relative file; repeatable. ---created