distributor/docs/roadmap/implementation.md

Implementation Roadmap

This roadmap records current implementation status and deferred work for distributor. Implemented behavior is documented in the user, operator, and internal documentation listed below.

Canonical current-behavior docs:

• README.md
• docs/cli.md
• docs/config.md
• docs/operations.md
• docs/troubleshooting.md
• docs/internal/
• docs/integrations/markdown.md
• docs/policy/
• examples/

Future, planned, or aspirational behavior belongs under docs/roadmap/ until it is implemented.

Current State

distributor is ready for routine use against producer pipelines using the implemented local, SSH/SFTP, and S3-compatible backends.

Active Roadmap

The active roadmap implements the focused feature roadmaps in an order intended to minimize later rewrites. The sequence establishes shared CLI output before new CLI commands, establishes producer bundle APIs before manifest creation, and implements destination path mapping before link generation so URL construction does not need to be reworked later.

Each stage is sized for one implementation prompt. Future behavior stays under docs/roadmap/ until the stage is implemented and current-behavior docs are updated.

Stage 1: CLI Output Policy

Goal: implement the shared CLI output contract in docs/roadmap/cli_output_policy.md.

Implementation scope:

• add per-command --format text|json to current output-producing commands: version, validate, inspect, and run;
• keep text as the default and preserve existing text output unless the policy explicitly requires a change;
• implement the shared JSON envelope, warnings array, fatal setup error behavior, and run partial-result JSON with ok: false;
• keep help and usage output text-only.

Documentation updates after implementation:

• update docs/cli.md with the shared --format text|json policy;
• update docs/operations.md where JSON output materially improves automation workflows;
• update docs/troubleshooting.md only for implemented JSON-mode recovery behavior.

Tests:

• CLI parsing accepts --format text and --format json;
• invalid --format values are usage errors;
• each JSON-capable command emits exactly one valid JSON document on success;
• JSON-mode warnings appear in warnings and are not duplicated to stderr;
• fatal setup errors write no JSON stdout and return non-zero;
• run --format json emits partial-result JSON and exits non-zero when destination failures occur after planning begins.

Completion criteria: all current output-producing commands support the shared policy, text output remains backward compatible, and go test ./... passes.

Stage 2: Public Bundle Package Core

Goal: implement the core and manifest-building parts of docs/roadmap/public_bundle_package.md.

Implementation scope:

• introduce pkg/bundle with the public source manifest model, schema version, digest logic, parsing, validation, explicit file-list building, scan-based building, and zero-Created defaulting to current UTC;
• implement the locked Stage 2 exported API symbols defined in docs/roadmap/public_bundle_package.md;
• update internal validation to consume the shared implementation without behavior drift;
• keep destination state, publish planning, storage backends, transforms, notifications, and config internal.

Documentation updates after implementation:

• add package documentation for pkg/bundle;
• update README.md and docs/operations.md only for implemented Go producer support;
• update docs/internal/bundle.md if internal ownership changes.

Tests:

• public package tests cover explicit lists, scan mode, ordering, digest calculation, parsing, validation, unsafe paths, symlinks, and zero Created;
• internal bundle tests continue to pass against the shared implementation;
• public examples compile where practical.

Completion criteria: public and internal validation use one manifest contract, and tests prove identical semantics.

Stage 3: Public Bundle Local Writer

Goal: implement the local bundle writer portion of docs/roadmap/public_bundle_package.md.

Implementation scope:

• add producer-side local bundle writing with staging, atomic filesystem operations where practical, and best-effort restore on overwrite failure;
• implement the locked Stage 3 exported API symbols defined in docs/roadmap/public_bundle_package.md;
• keep the writer filesystem-local and producer-focused;
• do not expose distributor storage backends or publication behavior through the public package.

Documentation updates after implementation:

• document the writer API in pkg/bundle;
• update producer workflow docs only for implemented behavior.

Tests:

• writer creates a complete valid local bundle through staged promotion;
• writer output validates through distributor's normal validation path;
• overwrite and failure behavior avoid leaving a completed bundle path without a valid manifest where practical.

Completion criteria: producers can create complete valid local bundles through the public API without hand-writing manifest files.

Stage 4: Manifest Create CLI

Goal: implement docs/roadmap/manifest_create.md.

Implementation scope:

• add distributor manifest create over pkg/bundle;
• support explicit-first --file selection, recursive scan fallback, --overwrite, optional --created, and shared --format text|json;
• use temp-and-rename replacement for manifest.json where practical;
• do not duplicate bundle contract logic in CLI or app code.

Documentation updates after implementation:

• update docs/cli.md with command syntax, flags, output, and examples;
• update docs/operations.md with a producer workflow;
• cross-reference pkg/bundle for Go producers.

Tests:

• command creates valid manifests for explicit and scanned files;
• ordering, dotfile inclusion, metadata exclusion, symlink rejection, and overwrite behavior match the roadmap;
• text and JSON output follow the shared CLI output policy.

Completion criteria: the command writes valid manifest.json, supports text/JSON output, and generated manifests validate through distributor.

Stage 5: Remote Validate and Inspect

Goal: implement docs/roadmap/remote_validate_inspect.md.

Implementation scope:

• extend validate and inspect with mutually exclusive local-path and --config --pipeline modes;
• reuse run source backend construction and secrets resolution;
• support configured local, SSH, and S3 sources;
• preserve --pipeline as required in config mode;
• keep the commands source-only and do not open destinations.

Documentation updates after implementation:

• update docs/cli.md with local and config-driven syntax;
• update docs/operations.md with remote validation examples;
• update docs/troubleshooting.md for common configured-source failures.

Tests:

• existing local behavior remains unchanged;
• config-mode validation and inspection work for local and fake/app-level source backends;
• SSH and S3 behavior is covered by focused unit tests and existing opt-in integration patterns;
• JSON output follows the shared CLI output policy.

Completion criteria: local behavior is stable, configured local/SSH/S3 sources can be validated and inspected, and no destination backend is opened.

Stage 6: HTML Index Mode

Goal: implement docs/roadmap/html_index_mode.md.

Implementation scope:

• add markdown_to_html.mode: index with fixed index.html output;
• keep sidecar as the default and existing explicit mode;
• implement deterministic input selection through explicit input or exactly one manifest-listed Markdown file;
• reject enabled Markdown-to-HTML transform config when publish.html is false;
• record generated state metadata with transform: markdown_to_html.

Documentation updates after implementation:

• update docs/config.md with Markdown-to-HTML modes and input selection;
• update docs/integrations/markdown.md;
• update docs/operations.md with an implemented static-site example where useful.

Tests:

• sidecar behavior remains unchanged;
• index mode handles explicit input, single Markdown fallback, ambiguous input, unsafe input, state metadata, dry-run, source-only publication, and output collisions.

Completion criteria: source publication, sidecar mode, index mode, state metadata, dry-run, and collision behavior are all covered.

Stage 7: Latest Path Destinations

Goal: implement docs/roadmap/latest_paths.md.

Implementation scope:

• add destination-level path_mapping.mode with default preserve_relative and new fixed;
• make destination bundle path mapping destination-local;
• for fixed destinations, select only the newest discovered bundle per destination before planning writes;
• add fixed-path dry-run warnings and summary counts;
• allow --force for fixed backend roots while keeping deletion bounded to the configured backend root.

Documentation updates after implementation:

• update docs/config.md with path_mapping;
• update docs/operations.md with archive-plus-latest fan-out examples;
• update docs/cli.md if dry-run output gains fixed-path indicators.

Tests:

• omitted and explicit preserve_relative match existing behavior;
• fixed destinations publish at local, SSH, and S3 backend roots;
• newest-only selection is deterministic;
• older discovered bundles are not planned or written to fixed destinations;
• dry-run reports candidate count, selected bundle, and destructive replacement warnings;
• force remains bounded at the backend root.

Completion criteria: fixed destination roots work across implemented backends, archive-style destinations remain unchanged, and destructive behavior is clear in dry-run output.

Stage 8: Link Generation Support

Goal: implement docs/roadmap/link_generation.md.

Implementation scope:

• add destination-level links.base_url and links.primary;
• generate per-output and primary URLs using URL semantics, not filesystem or storage joins;
• handle index.html as a directory-style URL;
• store optional URL metadata under destination state schema version 1 while pre-release;
• expose links through JSON command output only where supported by the CLI output policy.

Documentation updates after implementation:

• update docs/config.md with destination links fields;
• update docs/operations.md with static-site URL examples;
• update docs/internal/state.md for URL metadata;
• update docs/cli.md only for implemented CLI link output.

Tests:

• config validation accepts valid links and rejects invalid schemes, query strings, and fragments;
• nested bundle paths and fixed latest paths generate correct URLs;
• index.html omits the filename;
• primary selection is deterministic;
• destinations without links produce no URL metadata.

Completion criteria: URL generation respects archive and fixed path mapping, state records optional URL metadata when configured, and unconfigured destinations remain unchanged.

Stage 9: Roadmap Closeout

Goal: remove completed roadmap drift after Stages 1-8 are implemented.

Implementation scope:

• remove or rewrite completed roadmap files whose behavior is fully documented in current docs;
• ensure current docs describe implemented behavior;
• keep docs/roadmap/ focused only on remaining future work.

Documentation updates after implementation:

• update README.md, docs/cli.md, docs/config.md, docs/operations.md, docs/troubleshooting.md, docs/internal/, docs/integrations/markdown.md, and examples only where implemented behavior requires it.

Tests:

• run documentation consistency searches for completed-feature language that still appears only as future work;
• run focused tests for any examples or docs backed by tests;
• run go test ./... if behavior docs and examples changed with code.

Completion criteria: completed features are documented as current behavior, and docs/roadmap/ contains only future or deferred work.

Deferred Work

These items are not implemented and should stay out of current-behavior docs until a roadmap entry is selected and implemented:

• external notification adapters;
• warning-only digest mismatch handling;
• additional auth mechanisms beyond the implemented SSH and S3 credential paths;
• compatibility parsing for legacy SSH URI config;
• broad recursive destination deletion outside managed bundle paths;
• concurrent fan-out publishing;
• streaming, resumable, or multipart S3 uploads;
• cloud-provider-specific IAM integration docs;
• repository-managed packaging, release, and deployment automation.

Roadmap Maintenance

When adding future roadmap work:

• describe user-visible behavior and safety boundaries;
• define which existing docs must change after implementation;
• keep examples secret-free and runnable or clearly environment-gated;
• avoid workflow labels in production code, tests, config fields, and user documentation;
• run focused tests for the changed behavior and go test ./... for cross-package changes.