distributor/docs/roadmap/implementation.md

Distributor Implementation Roadmap

This roadmap defines a staged implementation plan for the distributor MVP. Each stage is intended to map cleanly to one Codex implementation prompt.

The roadmap assumes the project includes these planning documents before implementation begins:

• docs/policy/architecture.md
• docs/policy/documentation.md
• docs/roadmap/packages.md
• docs/roadmap/contracts.md
• docs/roadmap/config.md

The MVP goal is a domain-agnostic bundle distributor that discovers source bundles, validates manifest.json, optionally transforms Markdown to HTML, publishes selected outputs to one or more destinations, and records destination state in .distributor.json.

Global Implementation Rules

All stages should preserve these invariants:

• Producer applications own source bundle creation.
• distributor owns validation, transformation, publication, destination state, and future notification hooks.
• Source bundle state is defined by manifest.json.
• Destination publication state is defined by .distributor.json.
• manifest.json is not copied to the destination as destination state.
• Pipelines have exactly one source and one or more destinations.
• Transform and publish policy are destination-specific.
• Destructive replacement is allowed only inside a managed destination bundle path. Unsafe force or unmanaged overwrite behavior is deferred.
• Dry-run behavior should be implemented before broad remote write behavior.
• Config, bundle, state, publish planning, storage adapters, transforms, and CLI wiring should remain separate packages.

Unless a stage explicitly says otherwise, each implementation prompt should:

1. read the project policy and roadmap documents;
2. implement only the current stage;
3. add or update tests for the current stage;
4. run the relevant test suite;
5. update documentation only when the implemented behavior now exists;
6. avoid implementing future roadmap stages early.

Stage 1: Project Skeleton, CLI Shell, and Baseline Tooling

Goal

Create the initial Go application structure and a minimal executable distributor command with no business behavior beyond version/help output and placeholder commands.

Scope

Implement the accepted package skeleton from docs/roadmap/packages.md at the level needed for compilation.

Create:

cmd/distributor/main.go
internal/cli/
internal/app/
internal/config/
internal/logging/

Initial CLI commands:

• distributor --help
• distributor version
• distributor run
• distributor validate
• distributor inspect

At this stage, run, validate, and inspect may return clear “not implemented” errors, but the command structure should be present.

Notes

Prefer a small CLI dependency only if the project already standardizes on one. Otherwise, the standard library is acceptable for the first pass.

Add a version variable that can later be set at build time.

Tests

Add tests for:

• command construction if testable;
• version string behavior if exposed through a package;
• basic package compilation.

Completion Criteria

• go test ./... passes.
• go run ./cmd/distributor --help works.
• go run ./cmd/distributor version works.
• Placeholder operational commands fail clearly and intentionally.

Stage 2: Config Schema, Loading, Defaults, and Validation

Goal

Implement the MVP config.yml schema described in docs/roadmap/config.md.

Scope

Create config structs for:

• top-level config;
• pipelines;
• source backend config;
• destination backend config;
• validation policy;
• publish policy;
• transform policy;
• transfer/replacement policy;
• backend-specific local, SSH, and S3 fields.

Support loading YAML from a file path.

Implement validation for:

• required top-level pipelines;
• unique pipeline ids;
• required pipeline id, source, and non-empty destinations;
• unique destination ids within a pipeline;
• supported backend names: local, ssh, s3;
• required backend fields;
• supported validation action: fail;
• supported transfer actions;
• valid publish policy;
• valid Markdown-to-HTML transform config.

Default behavior should match docs/roadmap/config.md.

CLI Integration

Add --config to run.

For this stage, distributor run --config config.yml --dry-run may only load and validate config, then print a concise summary of configured pipelines and destinations.

Tests

Add unit tests for:

• valid minimal local-to-local config;
• valid fan-out config;
• valid local, SSH, and S3 backend configs;
• duplicate pipeline ids;
• duplicate destination ids;
• missing required fields;
• unsupported backend;
• invalid transfer action;
• invalid validation action, including warn.

Completion Criteria

• Config load/default/validate behavior is implemented and tested.
• distributor run --config <file> --dry-run validates config and prints a summary.
• No bundle discovery or publication occurs yet.

Stage 3: Storage Abstraction, Local Backend, and Fake Backend

Goal

Introduce the storage backend abstraction before bundle validation so source discovery, validation, and publication are backend-agnostic from the start.

Scope

Create:

internal/storage/backend.go
internal/storage/registry.go
internal/storage/path.go
internal/storage/errors.go
internal/adapters/local/backend.go
internal/storage/fake/

Define backend operations needed by the MVP, including:

• read file;
• write file;
• test existence;
• list files or tree entries;
• read destination state file if present;
• create directories/prefixes as needed;
• delete explicit managed files safely;
• write files using temporary/staged writes where practical.

The fake backend should exist for unit tests of config, bundle, state, and publish logic without real local, SSH, or S3 IO.

Safety Requirements

The local backend must:

• clean and join paths safely;
• reject path traversal;
• reject unsafe destructive deletion requests;
• avoid following symlinks for source bundle files unless explicitly supported;
• avoid deleting configured roots;
• classify destination bundle emptiness deterministically.

Tests

Add tests for:

• backend read/write/list/exists behavior;
• safe path joining;
• traversal rejection;
• symlink rejection for source reads;
• explicit-file deletion guard behavior;
• local destination emptiness detection;
• fake backend behavior sufficient for core package tests.

Completion Criteria

• Local backend implements the storage interface.
• Fake backend can support bundle and publish tests without external services.
• go test ./... passes.
• No SSH or S3 implementation exists yet.

Stage 4: Source Bundle Manifest, Digest, Validation, and Discovery

Goal

Implement the source bundle contract from docs/roadmap/contracts.md through the storage abstraction.

Scope

Create:

internal/bundle/manifest.go
internal/bundle/digest.go
internal/bundle/validate.go
internal/bundle/discover.go

Implement:

• parsing manifest.json;
• strict required field validation, including schema_version: 1;
• RFC3339 created parsing;
• lowercase sha256:<64 hex> digest validation;
• source file path safety checks;
• duplicate logical file path rejection;
• per-file SHA256 validation;
• per-file size validation;
• bundle digest validation using the canonical ordered file-record algorithm;
• deterministic storage-backed bundle discovery under a source root;
• nested manifest detection and failure.

Discovery and validation should use internal/storage rather than direct os APIs. The local CLI path should be adapted to the local backend.

CLI Integration

Implement:

distributor validate <path>
distributor inspect <path>

For local paths:

• validate should validate either a single bundle directory or a tree containing bundles.
• inspect should print a concise normalized summary of discovered bundle ids, relative paths, created timestamps, digest values, and files.

Tests

Add fixture bundles under a testdata directory.

Test:

• valid bundle;
• invalid JSON;
• missing required fields;
• invalid schema version;
• invalid timestamp;
• invalid digest format;
• unsafe file paths;
• duplicate normalized file paths;
• missing files;
• size mismatch;
• per-file digest mismatch;
• bundle digest mismatch;
• canonical bundle digest reference fixture;
• multiple discovered bundles in deterministic order;
• nested manifests fail.

Completion Criteria

• Storage-backed bundle validation is deterministic and well-tested.
• distributor validate <path> works for local bundle fixtures.
• distributor inspect <path> works for local bundle fixtures.
• No destination publication occurs yet.

Stage 5: Destination State Contract and Comparison Logic

Goal

Implement .distributor.json parsing, validation, and source-to-destination comparison.

Scope

Create:

internal/state/distributor.go
internal/state/compare.go
internal/state/validate.go

Implement the destination state schema from docs/roadmap/contracts.md, including:

• schema_version;
• optional distributor_version;
• pipeline_id;
• destination_id;
• published_at;
• embedded normalized source manifest;
• outputs array;
• output file metadata.

Implement comparison outcomes:

• destination absent;
• destination unmanaged/non-empty;
• same source manifest;
• same source id, destination older;
• same source id, destination newer;
• same source id and same created but different digest;
• different source id;
• invalid destination state.

Tests

Add unit tests for every comparison outcome.

Test validation for:

• valid state;
• missing fields;
• invalid schema version;
• invalid embedded source manifest;
• invalid output metadata;
• malformed published timestamp.

Completion Criteria

• Destination state can be parsed and validated independently.
• Source manifest to destination state comparison is deterministic and fully tested.
• No publication execution occurs yet.

Stage 6: Publish Planning, Dry-Run, and Local-to-Local Publication Without Transform

Goal

Implement the core publish planner and execute local-to-local publication for source files only.

Scope

Create:

internal/publish/plan.go
internal/publish/reconcile.go
internal/publish/safety.go
internal/publish/output.go
internal/publish/execute.go

Implement planning for one source bundle to one destination based on:

• source manifest;
• destination config;
• publish policy;
• transfer policy;
• existing .distributor.json;
• destination path state.

Actions should include:

• publish new;
• replace older destination;
• skip same;
• skip destination newer;
• fail conflict;
• fail unmanaged destination.

Implement local-to-local execution for publish.source: true and publish.html: false.

Execution should:

• copy listed source files selected by publish policy;
• write .distributor.json with copied source output metadata;
• avoid copying source manifest.json as destination state;
• preserve relative bundle paths from source root beneath destination root;
• detect destination output collisions before writing;
• use staging or equivalent cleanup behavior for local writes;
• support fan-out to multiple local destinations;
• support dry-run without writes.

CLI Integration

distributor run --config <file> should now execute local-to-local pipelines when configured.

--dry-run should print the planned action for each discovered bundle and destination.

Tests

Add integration-style tests using temp directories for:

• new local publication;
• no-op when destination state matches;
• replacement when destination state is older;
• skip when destination state is newer;
• fail on conflict;
• fail on unmanaged non-empty destination;
• fail on output path collision;
• fan-out from one source to two local destinations;
• failed local write does not leave a destination that appears unmanaged on retry;
• dry-run performs no writes;
• .distributor.json is written correctly.

Completion Criteria

• Local-to-local source-file publication works end to end.
• Dry-run produces meaningful planned actions.
• Destination state is authoritative.
• No Markdown-to-HTML transform exists yet.

Stage 7: Markdown-to-HTML Transform and Destination-Specific Publish Policy

Goal

Add MVP Markdown-to-HTML transformation and destination-specific source/html output selection.

Scope

Create:

internal/transform/transform.go
internal/transform/registry.go
internal/transform/plan.go
internal/transform/markdown/markdown.go
internal/transform/markdown/template.go

Implement only:

transform:
  markdown_to_html:
    enabled: true
    mode: sidecar

MVP sidecar behavior:

• for each listed source artifact ending in .md, generate a same-directory .html sidecar;
• preserve the original Markdown file unchanged;
• do not generate HTML for non-Markdown files;
• escape or disable raw HTML embedded in Markdown;
• fail before writing when generated output paths collide with copied source outputs or other generated outputs;
• record generated output metadata in .distributor.json;
• if publish.source: false, do not publish source files;
• if publish.html: true, publish generated HTML files;
• if publish.html: true but transform is disabled or no Markdown files exist, fail with a clear error unless config later defines another behavior.

Use a well-maintained Markdown renderer. Keep HTML templating minimal and deterministic.

Tests

Add tests for:

• Markdown sidecar generation;
• source-only destination;
• HTML-only destination;
• source-plus-HTML destination;
• no mutation of source bundle;
• generated output metadata in .distributor.json;
• failure when HTML publication is requested without transform support;
• failure when generated HTML collides with a source artifact path;
• raw HTML in Markdown is escaped or disabled consistently;
• deterministic output for a fixture Markdown file.

Completion Criteria

• Local-to-local publication supports source-only, HTML-only, and source-plus-HTML destinations.
• Generated outputs are recorded in destination state.
• Dry-run reports transform outputs that would be generated.

Stage 8: No-Op Notification Stage and Pipeline Orchestration Polish

Goal

Add the internal no-op notification stage and polish orchestration around per-destination outcomes.

Scope

Create:

internal/notify/notify.go
internal/notify/noop.go

Integrate a no-op notifier after successful publication/skip handling as appropriate.

Clarify orchestration behavior when one destination fails. For MVP, fan-out should be deterministic and sequential. Continue planning and reporting later destinations where safe, but return non-zero if any destination fails.

Improve run summary output:

• pipeline id;
• source backend;
• discovered bundle count;
• destination ids;
• action per bundle/destination;
• final status.

Tests

Add tests for:

• notifier is invoked at the expected orchestration point where testable;
• pipeline failure when a destination fails;
• run summary contains meaningful status information;
• dry-run does not invoke write-side effects.

Completion Criteria

• The pipeline shape includes notification as an internal no-op stage.
• Run output is useful for unattended operation logs.
• Local MVP behavior remains passing.

Stage 9: Native SSH/SFTP Backend

Goal

Implement SSH/SFTP storage backend support for sources and destinations.

Scope

Create:

internal/adapters/ssh/backend.go
internal/adapters/ssh/config.go

Implement the storage backend interface over native SSH/SFTP.

Required config:

backend: ssh
uri: ssh://user@example.com:22
path: /remote/root

Authentication expectations:

• prefer SSH agent by default;
• use known_hosts validation by default where practical;
• do not require passwords in YAML;
• optional key-file support may be implemented if straightforward, but should not distract from agent-based auth.

Support SSH/SFTP backend as both source and destination:

• local -> ssh;
• ssh -> local;
• ssh -> ssh where feasible through staging or streaming.

Safety Requirements

The SSH backend must enforce the same logical path safety rules as the local backend.

Deletion must remain limited to managed destination bundle paths guarded by valid .distributor.json.

Tests

Unit-test path handling and config validation.

If practical, add integration tests that can be skipped unless an SSH test endpoint is configured through environment variables. Do not require a live SSH server for normal go test ./....

Completion Criteria

• SSH/SFTP backend compiles and satisfies the storage interface.
• Backend config validation is tested.
• Normal tests do not depend on a live SSH server.
• At least local-to-SSH and SSH-to-local flows are documented or manually testable.

Stage 10: S3-Compatible Backend

Goal

Implement S3-compatible backend support for sources and destinations.

Scope

Create:

internal/adapters/s3/backend.go
internal/adapters/s3/config.go

Required config should align with docs/roadmap/config.md:

backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: some/prefix
region: us-east-1
force_path_style: true
credentials:
  access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
  secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY

Implement storage operations over S3 object keys:

• read object;
• write object;
• exists;
• list prefix;
• delete managed prefix or listed managed files;
• read/write .distributor.json.

Set reasonable content types where available:

• .md: text/markdown; charset=utf-8;
• .html: text/html; charset=utf-8;
• .json: application/json;
• .txt: text/plain; charset=utf-8.

Support S3 backend as both source and destination.

Safety Requirements

Treat S3 prefixes as object trees. Do not assume real directories exist.

Deletion must be limited to destination bundle prefixes that are confirmed managed by .distributor.json.

Tests

Add unit tests for:

• config validation;
• key/prefix normalization;
• content type selection;
• path traversal rejection;
• publish planning with S3 destination state fixtures.

If practical, add integration tests gated by environment variables or a local S3-compatible test service. Normal go test ./... must not require live S3 credentials.

Completion Criteria

• S3 backend compiles and satisfies the storage interface.
• S3 source and destination flows are supported through the common pipeline path.
• Normal tests do not require live S3.

Stage 11: Cross-Backend End-to-End Coverage and Hardening

Goal

Harden the MVP across backend combinations, destination policies, and failure cases.

Scope

Add end-to-end coverage for representative scenarios:

• local source -> local archive destination;
• local source -> local HTML destination;
• local source -> two destinations with different publish policies;
• local source -> SSH destination, where integration credentials exist;
• local source -> S3 destination, where integration credentials exist;
• S3 source -> local destination, where integration credentials exist;
• SSH source -> local destination, where integration credentials exist.

Improve logging and error messages for:

• invalid config;
• invalid source manifest;
• digest mismatch;
• destination conflict;
• unmanaged destination path;
• backend read/write/list failures;
• transform failures.

Ensure all destructive paths have tests or explicit safeguards.

Tests

Add or expand tests for:

• dry-run across multiple destinations;
• partial failure behavior;
• repeated run idempotency;
• older/newer destination state behavior;
• destination state output metadata accuracy;
• generated HTML output metadata accuracy.

Completion Criteria

• MVP behavior is reliable across implemented backend types.
• Error messages identify pipeline id, destination id, bundle id, and reason where practical.
• Idempotent repeated runs behave as expected.

Stage 12: User-Facing Documentation Sync

Goal

Update documentation to reflect implemented MVP behavior.

Scope

Following docs/policy/documentation.md, create or update user-facing documentation only for implemented features.

Likely docs:

README.md
docs/config.md
docs/cli.md
docs/policy/architecture.md
docs/internal/bundles.md
docs/internal/backends.md

Document:

• what distributor does;
• bundle contract summary;
• .distributor.json role;
• example source bundle;
• example local-to-local config;
• example local-to-S3 config;
• example local-to-SSH config;
• run, validate, and inspect commands;
• dry-run behavior;
• replacement and safety rules;
• Markdown-to-HTML transform behavior;
• environment-variable credential handling.

Move roadmap material to historical/planning status only if your documentation policy allows it. Do not describe unimplemented notification adapters as available features.

Tests

Run the full test suite.

If docs include command examples, verify that basic examples correspond to actual CLI behavior.

Completion Criteria

• User-facing docs describe the implemented MVP accurately.
• Roadmap docs no longer masquerade as implemented behavior.
• go test ./... passes.

Stage 13: MVP Release Readiness Pass

Goal

Perform a final pre-release quality pass.

Scope

Review:

• package boundaries against docs/policy/architecture.md;
• package layout against docs/roadmap/packages.md;
• implemented contracts against docs/roadmap/contracts.md;
• implemented config behavior against docs/roadmap/config.md;
• docs against docs/policy/documentation.md;
• destructive operation safety;
• logs and errors for unattended operation;
• command UX;
• test coverage for core invariants.

Add any missing small tests or docs discovered during review.

Do not add new product features in this stage.

Completion Criteria

• MVP is ready to deploy against one real producer pipeline.
• A dry-run can be performed safely against a real source and destination.
• Repeated runs are idempotent.
• Destructive replacement cannot occur outside managed destination bundle paths.
• Final docs accurately reflect the application.

Deferred Post-MVP Work

The following items are intentionally outside the MVP unless explicitly pulled into a later roadmap:

• email notifications;
• ntfy/Gotify/Pushover notifications;
• RSS/Atom feed generation;
• static site index pages beyond sidecar HTML output;
• templated HTML themes beyond a minimal deterministic template;
• destination path remapping rules;
• full plugin architecture;
• web UI;
• report editing;
• producer pipeline execution;
• database-backed state;
• complex retry queues;
• concurrent publication workers;
• symlink support;
• warning-only digest mismatch handling;
• force or unmanaged-overwrite destination behavior;
• password-based SSH authentication in YAML.