Added initial MVP roadmap documentation and implementation plan

2026-05-30 19:24:54 -05:00
parent 74709b2e0d
commit 47c9c5c0d1
7 changed files with 2658 additions and 22 deletions
--- a/docs/roadmap/implementation.md
+++ b/docs/roadmap/implementation.md
@@ -0,0 +1,808 @@
+# Distributor Implementation Roadmap
+
+This roadmap defines a staged implementation plan for the `distributor` MVP. Each stage is intended to map cleanly to one Codex implementation prompt.
+
+The roadmap assumes the project includes these planning documents before implementation begins:
+
+- `docs/policy/architecture.md`
+- `docs/policy/documentation.md`
+- `docs/roadmap/packages.md`
+- `docs/roadmap/contracts.md`
+- `docs/roadmap/config.md`
+
+The MVP goal is a domain-agnostic bundle distributor that discovers source bundles, validates `manifest.json`, optionally transforms Markdown to HTML, publishes selected outputs to one or more destinations, and records destination state in `.distributor.json`.
+
+## Global Implementation Rules
+
+All stages should preserve these invariants:
+
+- Producer applications own source bundle creation.
+- `distributor` owns validation, transformation, publication, destination state, and future notification hooks.
+- Source bundle state is defined by `manifest.json`.
+- Destination publication state is defined by `.distributor.json`.
+- `manifest.json` is not copied to the destination as destination state.
+- Pipelines have exactly one source and one or more destinations.
+- Transform and publish policy are destination-specific.
+- Destructive replacement is allowed only inside a managed destination bundle path. Unsafe force or unmanaged overwrite behavior is deferred.
+- Dry-run behavior should be implemented before broad remote write behavior.
+- Config, bundle, state, publish planning, storage adapters, transforms, and CLI wiring should remain separate packages.
+
+Unless a stage explicitly says otherwise, each implementation prompt should:
+
+1. read the project policy and roadmap documents;
+2. implement only the current stage;
+3. add or update tests for the current stage;
+4. run the relevant test suite;
+5. update documentation only when the implemented behavior now exists;
+6. avoid implementing future roadmap stages early.
+
+## Stage 1: Project Skeleton, CLI Shell, and Baseline Tooling
+
+### Goal
+
+Create the initial Go application structure and a minimal executable `distributor` command with no business behavior beyond version/help output and placeholder commands.
+
+### Scope
+
+Implement the accepted package skeleton from `docs/roadmap/packages.md` at the level needed for compilation.
+
+Create:
+
+```text
+cmd/distributor/main.go
+internal/cli/
+internal/app/
+internal/config/
+internal/logging/
+```
+
+Initial CLI commands:
+
+- `distributor --help`
+- `distributor version`
+- `distributor run`
+- `distributor validate`
+- `distributor inspect`
+
+At this stage, `run`, `validate`, and `inspect` may return clear “not implemented” errors, but the command structure should be present.
+
+### Notes
+
+Prefer a small CLI dependency only if the project already standardizes on one. Otherwise, the standard library is acceptable for the first pass.
+
+Add a version variable that can later be set at build time.
+
+### Tests
+
+Add tests for:
+
+- command construction if testable;
+- version string behavior if exposed through a package;
+- basic package compilation.
+
+### Completion Criteria
+
+- `go test ./...` passes.
+- `go run ./cmd/distributor --help` works.
+- `go run ./cmd/distributor version` works.
+- Placeholder operational commands fail clearly and intentionally.
+
+## Stage 2: Config Schema, Loading, Defaults, and Validation
+
+### Goal
+
+Implement the MVP `config.yml` schema described in `docs/roadmap/config.md`.
+
+### Scope
+
+Create config structs for:
+
+- top-level config;
+- pipelines;
+- source backend config;
+- destination backend config;
+- validation policy;
+- publish policy;
+- transform policy;
+- transfer/replacement policy;
+- backend-specific local, SSH, and S3 fields.
+
+Support loading YAML from a file path.
+
+Implement validation for:
+
+- required top-level `pipelines`;
+- unique pipeline ids;
+- required pipeline `id`, `source`, and non-empty `destinations`;
+- unique destination ids within a pipeline;
+- supported backend names: `local`, `ssh`, `s3`;
+- required backend fields;
+- supported validation action: `fail`;
+- supported transfer actions;
+- valid `publish` policy;
+- valid Markdown-to-HTML transform config.
+
+Default behavior should match `docs/roadmap/config.md`.
+
+### CLI Integration
+
+Add `--config` to `run`.
+
+For this stage, `distributor run --config config.yml --dry-run` may only load and validate config, then print a concise summary of configured pipelines and destinations.
+
+### Tests
+
+Add unit tests for:
+
+- valid minimal local-to-local config;
+- valid fan-out config;
+- valid local, SSH, and S3 backend configs;
+- duplicate pipeline ids;
+- duplicate destination ids;
+- missing required fields;
+- unsupported backend;
+- invalid transfer action;
+- invalid validation action, including `warn`.
+
+### Completion Criteria
+
+- Config load/default/validate behavior is implemented and tested.
+- `distributor run --config <file> --dry-run` validates config and prints a summary.
+- No bundle discovery or publication occurs yet.
+
+## Stage 3: Storage Abstraction, Local Backend, and Fake Backend
+
+### Goal
+
+Introduce the storage backend abstraction before bundle validation so source discovery, validation, and publication are backend-agnostic from the start.
+
+### Scope
+
+Create:
+
+```text
+internal/storage/backend.go
+internal/storage/registry.go
+internal/storage/path.go
+internal/storage/errors.go
+internal/adapters/local/backend.go
+internal/storage/fake/
+```
+
+Define backend operations needed by the MVP, including:
+
+- read file;
+- write file;
+- test existence;
+- list files or tree entries;
+- read destination state file if present;
+- create directories/prefixes as needed;
+- delete explicit managed files safely;
+- write files using temporary/staged writes where practical.
+
+The fake backend should exist for unit tests of config, bundle, state, and publish logic without real local, SSH, or S3 IO.
+
+### Safety Requirements
+
+The local backend must:
+
+- clean and join paths safely;
+- reject path traversal;
+- reject unsafe destructive deletion requests;
+- avoid following symlinks for source bundle files unless explicitly supported;
+- avoid deleting configured roots;
+- classify destination bundle emptiness deterministically.
+
+### Tests
+
+Add tests for:
+
+- backend read/write/list/exists behavior;
+- safe path joining;
+- traversal rejection;
+- symlink rejection for source reads;
+- explicit-file deletion guard behavior;
+- local destination emptiness detection;
+- fake backend behavior sufficient for core package tests.
+
+### Completion Criteria
+
+- Local backend implements the storage interface.
+- Fake backend can support bundle and publish tests without external services.
+- `go test ./...` passes.
+- No SSH or S3 implementation exists yet.
+
+## Stage 4: Source Bundle Manifest, Digest, Validation, and Discovery
+
+### Goal
+
+Implement the source bundle contract from `docs/roadmap/contracts.md` through the storage abstraction.
+
+### Scope
+
+Create:
+
+```text
+internal/bundle/manifest.go
+internal/bundle/digest.go
+internal/bundle/validate.go
+internal/bundle/discover.go
+```
+
+Implement:
+
+- parsing `manifest.json`;
+- strict required field validation, including `schema_version: 1`;
+- RFC3339 `created` parsing;
+- lowercase `sha256:<64 hex>` digest validation;
+- source file path safety checks;
+- duplicate logical file path rejection;
+- per-file SHA256 validation;
+- per-file size validation;
+- bundle digest validation using the canonical ordered file-record algorithm;
+- deterministic storage-backed bundle discovery under a source root;
+- nested manifest detection and failure.
+
+Discovery and validation should use `internal/storage` rather than direct `os` APIs. The local CLI path should be adapted to the local backend.
+
+### CLI Integration
+
+Implement:
+
+```text
+distributor validate <path>
+distributor inspect <path>
+```
+
+For local paths:
+
+- `validate` should validate either a single bundle directory or a tree containing bundles.
+- `inspect` should print a concise normalized summary of discovered bundle ids, relative paths, created timestamps, digest values, and files.
+
+### Tests
+
+Add fixture bundles under a testdata directory.
+
+Test:
+
+- valid bundle;
+- invalid JSON;
+- missing required fields;
+- invalid schema version;
+- invalid timestamp;
+- invalid digest format;
+- unsafe file paths;
+- duplicate normalized file paths;
+- missing files;
+- size mismatch;
+- per-file digest mismatch;
+- bundle digest mismatch;
+- canonical bundle digest reference fixture;
+- multiple discovered bundles in deterministic order;
+- nested manifests fail.
+
+### Completion Criteria
+
+- Storage-backed bundle validation is deterministic and well-tested.
+- `distributor validate <path>` works for local bundle fixtures.
+- `distributor inspect <path>` works for local bundle fixtures.
+- No destination publication occurs yet.
+
+## Stage 5: Destination State Contract and Comparison Logic
+
+### Goal
+
+Implement `.distributor.json` parsing, validation, and source-to-destination comparison.
+
+### Scope
+
+Create:
+
+```text
+internal/state/distributor.go
+internal/state/compare.go
+internal/state/validate.go
+```
+
+Implement the destination state schema from `docs/roadmap/contracts.md`, including:
+
+- `schema_version`;
+- optional `distributor_version`;
+- `pipeline_id`;
+- `destination_id`;
+- `published_at`;
+- embedded normalized source manifest;
+- outputs array;
+- output file metadata.
+
+Implement comparison outcomes:
+
+- destination absent;
+- destination unmanaged/non-empty;
+- same source manifest;
+- same source id, destination older;
+- same source id, destination newer;
+- same source id and same created but different digest;
+- different source id;
+- invalid destination state.
+
+### Tests
+
+Add unit tests for every comparison outcome.
+
+Test validation for:
+
+- valid state;
+- missing fields;
+- invalid schema version;
+- invalid embedded source manifest;
+- invalid output metadata;
+- malformed published timestamp.
+
+### Completion Criteria
+
+- Destination state can be parsed and validated independently.
+- Source manifest to destination state comparison is deterministic and fully tested.
+- No publication execution occurs yet.
+
+## Stage 6: Publish Planning, Dry-Run, and Local-to-Local Publication Without Transform
+
+### Goal
+
+Implement the core publish planner and execute local-to-local publication for source files only.
+
+### Scope
+
+Create:
+
+```text
+internal/publish/plan.go
+internal/publish/reconcile.go
+internal/publish/safety.go
+internal/publish/output.go
+internal/publish/execute.go
+```
+
+Implement planning for one source bundle to one destination based on:
+
+- source manifest;
+- destination config;
+- publish policy;
+- transfer policy;
+- existing `.distributor.json`;
+- destination path state.
+
+Actions should include:
+
+- publish new;
+- replace older destination;
+- skip same;
+- skip destination newer;
+- fail conflict;
+- fail unmanaged destination.
+
+Implement local-to-local execution for `publish.source: true` and `publish.html: false`.
+
+Execution should:
+
+- copy listed source files selected by publish policy;
+- write `.distributor.json` with copied source output metadata;
+- avoid copying source `manifest.json` as destination state;
+- preserve relative bundle paths from source root beneath destination root;
+- detect destination output collisions before writing;
+- use staging or equivalent cleanup behavior for local writes;
+- support fan-out to multiple local destinations;
+- support dry-run without writes.
+
+### CLI Integration
+
+`distributor run --config <file>` should now execute local-to-local pipelines when configured.
+
+`--dry-run` should print the planned action for each discovered bundle and destination.
+
+### Tests
+
+Add integration-style tests using temp directories for:
+
+- new local publication;
+- no-op when destination state matches;
+- replacement when destination state is older;
+- skip when destination state is newer;
+- fail on conflict;
+- fail on unmanaged non-empty destination;
+- fail on output path collision;
+- fan-out from one source to two local destinations;
+- failed local write does not leave a destination that appears unmanaged on retry;
+- dry-run performs no writes;
+- `.distributor.json` is written correctly.
+
+### Completion Criteria
+
+- Local-to-local source-file publication works end to end.
+- Dry-run produces meaningful planned actions.
+- Destination state is authoritative.
+- No Markdown-to-HTML transform exists yet.
+
+## Stage 7: Markdown-to-HTML Transform and Destination-Specific Publish Policy
+
+### Goal
+
+Add MVP Markdown-to-HTML transformation and destination-specific source/html output selection.
+
+### Scope
+
+Create:
+
+```text
+internal/transform/transform.go
+internal/transform/registry.go
+internal/transform/plan.go
+internal/transform/markdown/markdown.go
+internal/transform/markdown/template.go
+```
+
+Implement only:
+
+```yaml
+transform:
+  markdown_to_html:
+    enabled: true
+    mode: sidecar
+```
+
+MVP sidecar behavior:
+
+- for each listed source artifact ending in `.md`, generate a same-directory `.html` sidecar;
+- preserve the original Markdown file unchanged;
+- do not generate HTML for non-Markdown files;
+- escape or disable raw HTML embedded in Markdown;
+- fail before writing when generated output paths collide with copied source outputs or other generated outputs;
+- record generated output metadata in `.distributor.json`;
+- if `publish.source: false`, do not publish source files;
+- if `publish.html: true`, publish generated HTML files;
+- if `publish.html: true` but transform is disabled or no Markdown files exist, fail with a clear error unless config later defines another behavior.
+
+Use a well-maintained Markdown renderer. Keep HTML templating minimal and deterministic.
+
+### Tests
+
+Add tests for:
+
+- Markdown sidecar generation;
+- source-only destination;
+- HTML-only destination;
+- source-plus-HTML destination;
+- no mutation of source bundle;
+- generated output metadata in `.distributor.json`;
+- failure when HTML publication is requested without transform support;
+- failure when generated HTML collides with a source artifact path;
+- raw HTML in Markdown is escaped or disabled consistently;
+- deterministic output for a fixture Markdown file.
+
+### Completion Criteria
+
+- Local-to-local publication supports source-only, HTML-only, and source-plus-HTML destinations.
+- Generated outputs are recorded in destination state.
+- Dry-run reports transform outputs that would be generated.
+
+## Stage 8: No-Op Notification Stage and Pipeline Orchestration Polish
+
+### Goal
+
+Add the internal no-op notification stage and polish orchestration around per-destination outcomes.
+
+### Scope
+
+Create:
+
+```text
+internal/notify/notify.go
+internal/notify/noop.go
+```
+
+Integrate a no-op notifier after successful publication/skip handling as appropriate.
+
+Clarify orchestration behavior when one destination fails. For MVP, fan-out should be deterministic and sequential. Continue planning and reporting later destinations where safe, but return non-zero if any destination fails.
+
+Improve run summary output:
+
+- pipeline id;
+- source backend;
+- discovered bundle count;
+- destination ids;
+- action per bundle/destination;
+- final status.
+
+### Tests
+
+Add tests for:
+
+- notifier is invoked at the expected orchestration point where testable;
+- pipeline failure when a destination fails;
+- run summary contains meaningful status information;
+- dry-run does not invoke write-side effects.
+
+### Completion Criteria
+
+- The pipeline shape includes notification as an internal no-op stage.
+- Run output is useful for unattended operation logs.
+- Local MVP behavior remains passing.
+
+## Stage 9: Native SSH/SFTP Backend
+
+### Goal
+
+Implement SSH/SFTP storage backend support for sources and destinations.
+
+### Scope
+
+Create:
+
+```text
+internal/adapters/ssh/backend.go
+internal/adapters/ssh/config.go
+```
+
+Implement the storage backend interface over native SSH/SFTP.
+
+Required config:
+
+```yaml
+backend: ssh
+uri: ssh://user@example.com:22
+path: /remote/root
+```
+
+Authentication expectations:
+
+- prefer SSH agent by default;
+- use known_hosts validation by default where practical;
+- do not require passwords in YAML;
+- optional key-file support may be implemented if straightforward, but should not distract from agent-based auth.
+
+Support SSH/SFTP backend as both source and destination:
+
+- local -> ssh;
+- ssh -> local;
+- ssh -> ssh where feasible through staging or streaming.
+
+### Safety Requirements
+
+The SSH backend must enforce the same logical path safety rules as the local backend.
+
+Deletion must remain limited to managed destination bundle paths guarded by valid `.distributor.json`.
+
+### Tests
+
+Unit-test path handling and config validation.
+
+If practical, add integration tests that can be skipped unless an SSH test endpoint is configured through environment variables. Do not require a live SSH server for normal `go test ./...`.
+
+### Completion Criteria
+
+- SSH/SFTP backend compiles and satisfies the storage interface.
+- Backend config validation is tested.
+- Normal tests do not depend on a live SSH server.
+- At least local-to-SSH and SSH-to-local flows are documented or manually testable.
+
+## Stage 10: S3-Compatible Backend
+
+### Goal
+
+Implement S3-compatible backend support for sources and destinations.
+
+### Scope
+
+Create:
+
+```text
+internal/adapters/s3/backend.go
+internal/adapters/s3/config.go
+```
+
+Required config should align with `docs/roadmap/config.md`:
+
+```yaml
+backend: s3
+endpoint: https://s3.example.com
+bucket: reports
+prefix: some/prefix
+region: us-east-1
+force_path_style: true
+credentials:
+  access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
+  secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
+```
+
+Implement storage operations over S3 object keys:
+
+- read object;
+- write object;
+- exists;
+- list prefix;
+- delete managed prefix or listed managed files;
+- read/write `.distributor.json`.
+
+Set reasonable content types where available:
+
+- `.md`: `text/markdown; charset=utf-8`;
+- `.html`: `text/html; charset=utf-8`;
+- `.json`: `application/json`;
+- `.txt`: `text/plain; charset=utf-8`.
+
+Support S3 backend as both source and destination.
+
+### Safety Requirements
+
+Treat S3 prefixes as object trees. Do not assume real directories exist.
+
+Deletion must be limited to destination bundle prefixes that are confirmed managed by `.distributor.json`.
+
+### Tests
+
+Add unit tests for:
+
+- config validation;
+- key/prefix normalization;
+- content type selection;
+- path traversal rejection;
+- publish planning with S3 destination state fixtures.
+
+If practical, add integration tests gated by environment variables or a local S3-compatible test service. Normal `go test ./...` must not require live S3 credentials.
+
+### Completion Criteria
+
+- S3 backend compiles and satisfies the storage interface.
+- S3 source and destination flows are supported through the common pipeline path.
+- Normal tests do not require live S3.
+
+## Stage 11: Cross-Backend End-to-End Coverage and Hardening
+
+### Goal
+
+Harden the MVP across backend combinations, destination policies, and failure cases.
+
+### Scope
+
+Add end-to-end coverage for representative scenarios:
+
+- local source -> local archive destination;
+- local source -> local HTML destination;
+- local source -> two destinations with different publish policies;
+- local source -> SSH destination, where integration credentials exist;
+- local source -> S3 destination, where integration credentials exist;
+- S3 source -> local destination, where integration credentials exist;
+- SSH source -> local destination, where integration credentials exist.
+
+Improve logging and error messages for:
+
+- invalid config;
+- invalid source manifest;
+- digest mismatch;
+- destination conflict;
+- unmanaged destination path;
+- backend read/write/list failures;
+- transform failures.
+
+Ensure all destructive paths have tests or explicit safeguards.
+
+### Tests
+
+Add or expand tests for:
+
+- dry-run across multiple destinations;
+- partial failure behavior;
+- repeated run idempotency;
+- older/newer destination state behavior;
+- destination state output metadata accuracy;
+- generated HTML output metadata accuracy.
+
+### Completion Criteria
+
+- MVP behavior is reliable across implemented backend types.
+- Error messages identify pipeline id, destination id, bundle id, and reason where practical.
+- Idempotent repeated runs behave as expected.
+
+## Stage 12: User-Facing Documentation Sync
+
+### Goal
+
+Update documentation to reflect implemented MVP behavior.
+
+### Scope
+
+Following `docs/policy/documentation.md`, create or update user-facing documentation only for implemented features.
+
+Likely docs:
+
+```text
+README.md
+docs/config.md
+docs/cli.md
+docs/policy/architecture.md
+docs/internal/bundles.md
+docs/internal/backends.md
+```
+
+Document:
+
+- what `distributor` does;
+- bundle contract summary;
+- `.distributor.json` role;
+- example source bundle;
+- example local-to-local config;
+- example local-to-S3 config;
+- example local-to-SSH config;
+- `run`, `validate`, and `inspect` commands;
+- dry-run behavior;
+- replacement and safety rules;
+- Markdown-to-HTML transform behavior;
+- environment-variable credential handling.
+
+Move roadmap material to historical/planning status only if your documentation policy allows it. Do not describe unimplemented notification adapters as available features.
+
+### Tests
+
+Run the full test suite.
+
+If docs include command examples, verify that basic examples correspond to actual CLI behavior.
+
+### Completion Criteria
+
+- User-facing docs describe the implemented MVP accurately.
+- Roadmap docs no longer masquerade as implemented behavior.
+- `go test ./...` passes.
+
+## Stage 13: MVP Release Readiness Pass
+
+### Goal
+
+Perform a final pre-release quality pass.
+
+### Scope
+
+Review:
+
+- package boundaries against `docs/policy/architecture.md`;
+- package layout against `docs/roadmap/packages.md`;
+- implemented contracts against `docs/roadmap/contracts.md`;
+- implemented config behavior against `docs/roadmap/config.md`;
+- docs against `docs/policy/documentation.md`;
+- destructive operation safety;
+- logs and errors for unattended operation;
+- command UX;
+- test coverage for core invariants.
+
+Add any missing small tests or docs discovered during review.
+
+Do not add new product features in this stage.
+
+### Completion Criteria
+
+- MVP is ready to deploy against one real producer pipeline.
+- A dry-run can be performed safely against a real source and destination.
+- Repeated runs are idempotent.
+- Destructive replacement cannot occur outside managed destination bundle paths.
+- Final docs accurately reflect the application.
+
+## Deferred Post-MVP Work
+
+The following items are intentionally outside the MVP unless explicitly pulled into a later roadmap:
+
+- email notifications;
+- ntfy/Gotify/Pushover notifications;
+- RSS/Atom feed generation;
+- static site index pages beyond sidecar HTML output;
+- templated HTML themes beyond a minimal deterministic template;
+- destination path remapping rules;
+- full plugin architecture;
+- web UI;
+- report editing;
+- producer pipeline execution;
+- database-backed state;
+- complex retry queues;
+- concurrent publication workers;
+- symlink support;
+- warning-only digest mismatch handling;
+- force or unmanaged-overwrite destination behavior;
+- password-based SSH authentication in YAML.