distributor/docs/roadmap/implementation.md

# Distributor Implementation Roadmap

This roadmap defines a staged implementation plan for the `distributor` MVP. Each stage is intended to map cleanly to one Codex implementation prompt.

The roadmap assumes the project includes these planning documents before implementation begins:

- `docs/policy/architecture.md`
- `docs/policy/documentation.md`
- `docs/roadmap/packages.md`
- `docs/roadmap/contracts.md`
- `docs/roadmap/config.md`
- `docs/roadmap/storage.md`

The MVP goal is a domain-agnostic bundle distributor that discovers source bundles, validates `manifest.json`, optionally transforms Markdown to HTML, publishes selected outputs to one or more destinations, and records destination state in `.distributor.json`.

## Global Implementation Rules

All stages should preserve these invariants:

- Producer applications own source bundle creation.
- `distributor` owns validation, transformation, publication, destination state, and future notification hooks.
- Source bundle state is defined by `manifest.json`.
- Destination publication state is defined by `.distributor.json`.
- `manifest.json` is not copied to the destination as destination state.
- Pipelines have exactly one source and one or more destinations.
- Transform and publish policy are destination-specific.
- Destructive replacement is allowed only inside a managed destination bundle path. Unsafe force or unmanaged overwrite behavior is deferred.
- Dry-run behavior should be implemented before broad remote write behavior.
- Config, bundle, state, publish planning, storage adapters, transforms, and CLI wiring should remain separate packages.

Unless a stage explicitly says otherwise, each implementation prompt should:

1. read the project policy and roadmap documents;
2. implement only the current stage;
3. add or update tests for the current stage;
4. run the relevant test suite;
5. update documentation only when the implemented behavior now exists;
6. avoid implementing future roadmap stages early.

## Stage 1: Project Skeleton, CLI Shell, and Baseline Tooling

### Goal

Create the initial Go application structure and a minimal executable `distributor` command with no business behavior beyond version/help output and placeholder commands.

### Scope

Implement the accepted package skeleton from `docs/roadmap/packages.md` at the level needed for compilation.

Create:

```text
cmd/distributor/main.go
internal/cli/
internal/app/
internal/config/
internal/logging/
```

Initial CLI commands:

- `distributor --help`
- `distributor version`
- `distributor run`
- `distributor validate`
- `distributor inspect`

At this stage, `run`, `validate`, and `inspect` may return clear “not implemented” errors, but the command structure should be present.

### Notes

Prefer a small CLI dependency only if the project already standardizes on one. Otherwise, the standard library is acceptable for the first pass.

Add a version variable that can later be set at build time.

### Tests

Add tests for:

- command construction if testable;
- version string behavior if exposed through a package;
- basic package compilation.

### Completion Criteria

- `go test ./...` passes.
- `go run ./cmd/distributor --help` works.
- `go run ./cmd/distributor version` works.
- Placeholder operational commands fail clearly and intentionally.

## Stage 2: Config Schema, Loading, Defaults, and Validation

### Goal

Implement the MVP `config.yml` schema described in `docs/roadmap/config.md`.

### Scope

Create config structs for:

- top-level config;
- pipelines;
- source backend config;
- destination backend config;
- validation policy;
- publish policy;
- transform policy;
- transfer/replacement policy;
- backend-specific local, SSH, and S3 fields.

Support loading YAML from a file path.

Implement validation for:

- required top-level `pipelines`;
- unique pipeline ids;
- required pipeline `id`, `source`, and non-empty `destinations`;
- unique destination ids within a pipeline;
- supported backend names: `local`, `ssh`, `s3`;
- required backend fields;
- supported validation action: `fail`;
- supported transfer actions;
- valid `publish` policy;
- valid Markdown-to-HTML transform config.

Default behavior should match `docs/roadmap/config.md`.

### CLI Integration

Add `--config` to `run`.

For this stage, `distributor run --config config.yml --dry-run` may only load and validate config, then print a concise summary of configured pipelines and destinations.

### Tests

Add unit tests for:

- valid minimal local-to-local config;
- valid fan-out config;
- valid local, SSH, and S3 backend configs;
- duplicate pipeline ids;
- duplicate destination ids;
- missing required fields;
- unsupported backend;
- invalid transfer action;
- invalid validation action, including `warn`.

### Completion Criteria

- Config load/default/validate behavior is implemented and tested.
- `distributor run --config <file> --dry-run` validates config and prints a summary.
- No bundle discovery or publication occurs yet.

## Stage 3: Storage Abstraction, Local Backend, and Fake Backend

### Goal

Introduce the storage backend abstraction before bundle validation so source discovery, validation, and publication are backend-agnostic from the start.

### Scope

Create:

```text
internal/storage/backend.go
internal/storage/registry.go
internal/storage/path.go
internal/storage/errors.go
internal/adapters/local/backend.go
internal/storage/fake/
```

Implement the storage contract in `docs/roadmap/storage.md`, including backend-rooted logical paths, hybrid byte/stream IO, metadata, traversal, typed errors, managed deletion, and efficient destination emptiness helper behavior.

The fake backend should exist for unit tests of config, bundle, state, and publish logic without real local, SSH, or S3 IO.

### Safety Requirements

The local backend must:

- clean and join paths safely;
- reject path traversal;
- reject unsafe destructive deletion requests;
- avoid following symlinks for source bundle files unless explicitly supported;
- avoid deleting configured roots;
- classify destination bundle emptiness deterministically.

### Tests

Add tests for:

- path validation;
- backend byte and stream read/write behavior;
- `Stat`, `Walk`, and materialized deterministic traversal helper behavior;
- traversal rejection;
- symlink entry reporting and source-read rejection;
- staged write behavior where testable;
- typed storage errors and helper predicates;
- managed deletion guard behavior;
- early-stop destination emptiness helper behavior;
- fake backend parity for core package tests.

### Completion Criteria

- Local backend implements the storage interface.
- Fake backend can support bundle and publish tests without external services.
- `go test ./...` passes.
- No SSH or S3 implementation exists yet.

## Stage 4: Source Bundle Manifest, Digest, Validation, and Discovery

### Goal

Implement the source bundle contract from `docs/roadmap/contracts.md` through the storage abstraction.

### Scope

Create:

```text
internal/bundle/manifest.go
internal/bundle/digest.go
internal/bundle/validate.go
internal/bundle/discover.go
```

Implement:

- parsing `manifest.json`;
- strict required field validation, including `schema_version: 1`;
- RFC3339 `created` parsing;
- lowercase `sha256:<64 hex>` digest validation;
- source file path safety checks;
- duplicate logical file path rejection;
- per-file SHA256 validation;
- per-file size validation;
- bundle digest validation using the canonical ordered file-record algorithm;
- deterministic storage-backed bundle discovery under a source root;
- nested manifest detection and failure.

Discovery and validation should use `internal/storage` rather than direct `os` APIs. The local CLI path should be adapted to the local backend.

### CLI Integration

Implement:

```text
distributor validate <path>
distributor inspect <path>
```

For local paths:

- `validate` should validate either a single bundle directory or a tree containing bundles.
- `inspect` should print a concise normalized summary of discovered bundle ids, relative paths, created timestamps, digest values, and files.

### Tests

Add fixture bundles under a testdata directory.

Test:

- valid bundle;
- invalid JSON;
- missing required fields;
- invalid schema version;
- invalid timestamp;
- invalid digest format;
- unsafe file paths;
- duplicate normalized file paths;
- missing files;
- size mismatch;
- per-file digest mismatch;
- bundle digest mismatch;
- canonical bundle digest reference fixture;
- multiple discovered bundles in deterministic order;
- nested manifests fail.

### Completion Criteria

- Storage-backed bundle validation is deterministic and well-tested.
- `distributor validate <path>` works for local bundle fixtures.
- `distributor inspect <path>` works for local bundle fixtures.
- No destination publication occurs yet.

## Stage 5: Destination State Contract and Comparison Logic

### Goal

Implement `.distributor.json` parsing, validation, and source-to-destination comparison.

### Scope

Create:

```text
internal/state/distributor.go
internal/state/compare.go
internal/state/validate.go
```

Implement the destination state schema from `docs/roadmap/contracts.md`, including:

- `schema_version`;
- optional `distributor_version`;
- `pipeline_id`;
- `destination_id`;
- `published_at`;
- embedded normalized source manifest;
- outputs array;
- output file metadata.

Implement comparison outcomes:

- destination absent;
- destination unmanaged/non-empty;
- destination state pipeline or destination id mismatch;
- same source manifest;
- same source id, destination older;
- same source id, destination newer;
- same source id and same created but different digest;
- different source id;
- invalid destination state.

### Tests

Add unit tests for every comparison outcome.

Test validation for:

- valid state;
- missing fields;
- invalid schema version;
- invalid embedded source manifest;
- invalid output metadata;
- malformed published timestamp.

Timestamps should parse RFC3339 input and distributor-written timestamps should normalize to RFC3339 UTC.

### Completion Criteria

- Destination state can be parsed and validated independently.
- Source manifest to destination state comparison is deterministic and fully tested.
- No publication execution occurs yet.

## Stage 6: Publish Planning, Dry-Run, and Local-to-Local Publication Without Transform

### Goal

Implement the core publish planner and execute local-to-local publication for source files only.

### Scope

Create:

```text
internal/publish/plan.go
internal/publish/reconcile.go
internal/publish/safety.go
internal/publish/output.go
internal/publish/execute.go
```

Implement planning for one source bundle to one destination based on:

- source manifest;
- destination config;
- publish policy;
- transfer policy;
- existing `.distributor.json`;
- destination path state.

Actions should include:

- publish new;
- replace older destination;
- skip same;
- skip destination newer;
- fail conflict;
- fail unmanaged destination.

Implement local-to-local execution for `publish.source: true` and `publish.html: false`.

Execution should:

- copy listed source files selected by publish policy;
- write `.distributor.json` with copied source output metadata;
- avoid copying source `manifest.json` as destination state;
- preserve relative bundle paths from source root beneath destination root;
- detect destination output collisions before writing;
- use staging or equivalent cleanup behavior for local writes;
- support fan-out to multiple local destinations;
- support dry-run without writes.

### CLI Integration

`distributor run --config <file>` should now execute local-to-local pipelines when configured.

`--dry-run` should print the planned action for each discovered bundle and destination.

### Tests

Add integration-style tests using temp directories for:

- new local publication;
- no-op when destination state matches;
- replacement when destination state is older;
- skip when destination state is newer;
- fail on conflict;
- fail on unmanaged non-empty destination;
- fail on output path collision;
- fan-out from one source to two local destinations;
- failed local write does not leave a destination that appears unmanaged on retry;
- dry-run performs no writes;
- `.distributor.json` is written correctly.

### Completion Criteria

- Local-to-local source-file publication works end to end.
- Dry-run produces meaningful planned actions.
- Destination state is authoritative.
- No Markdown-to-HTML transform exists yet.

## Stage 7: Markdown-to-HTML Transform and Destination-Specific Publish Policy

### Goal

Add MVP Markdown-to-HTML transformation and destination-specific source/html output selection.

### Scope

Create:

```text
internal/transform/transform.go
internal/transform/registry.go
internal/transform/plan.go
internal/transform/markdown/markdown.go
internal/transform/markdown/template.go
```

Implement only:

```yaml
transform:
  markdown_to_html:
    enabled: true
    mode: sidecar
```

MVP sidecar behavior:

- for each listed source artifact ending in `.md`, generate a same-directory `.html` sidecar;
- preserve the original Markdown file unchanged;
- do not generate HTML for non-Markdown files;
- escape or disable raw HTML embedded in Markdown;
- fail before writing when generated output paths collide with copied source outputs or other generated outputs;
- record generated output metadata in `.distributor.json`;
- if `publish.source: false`, do not publish source files;
- if `publish.html: true`, publish generated HTML files;
- if `publish.html: true` but transform is disabled or no Markdown files exist, fail with a clear error unless config later defines another behavior.

Use a well-maintained Markdown renderer. Keep HTML templating minimal and deterministic.

### Tests

Add tests for:

- Markdown sidecar generation;
- source-only destination;
- HTML-only destination;
- source-plus-HTML destination;
- no mutation of source bundle;
- generated output metadata in `.distributor.json`;
- failure when HTML publication is requested without transform support;
- failure when generated HTML collides with a source artifact path;
- raw HTML in Markdown is escaped or disabled consistently;
- deterministic output for a fixture Markdown file.

### Completion Criteria

- Local-to-local publication supports source-only, HTML-only, and source-plus-HTML destinations.
- Generated outputs are recorded in destination state.
- Dry-run reports transform outputs that would be generated.

## Stage 8: No-Op Notification Stage, Pipeline Polish, and Local MVP Checkpoint

### Goal

Add the internal no-op notification stage and polish orchestration around per-destination outcomes.

### Scope

Create:

```text
internal/notify/notify.go
internal/notify/noop.go
```

Integrate a no-op notifier after actual successful publication or replacement. Skipped destinations should not invoke the notifier in the MVP.

Clarify orchestration behavior when one destination fails. For MVP, fan-out should be deterministic and sequential. Continue planning and reporting later destinations where safe, but return non-zero if any destination fails.

Improve run summary output:

- pipeline id;
- source backend;
- discovered bundle count;
- destination ids;
- action per bundle/destination;
- final status.

### Tests

Add tests for:

- notifier is invoked at the expected orchestration point where testable;
- pipeline failure when a destination fails;
- run summary contains meaningful status information;
- dry-run does not invoke write-side effects.

### Completion Criteria

- The pipeline shape includes notification as an internal no-op stage.
- Run output is useful for unattended operation logs.
- Local MVP behavior remains passing and is ready for one real local producer pipeline.

Stages 1 through 8 define the local MVP checkpoint. Later stages extend the local MVP with remote backends, cross-backend hardening, user-facing documentation sync, and release readiness.

## Stage 9: Native SSH/SFTP Backend Roadmap Extension

### Goal

Implement SSH/SFTP storage backend support for sources and destinations.

### Scope

Create:

```text
internal/adapters/ssh/backend.go
internal/adapters/ssh/config.go
```

Implement the storage backend interface over native SSH/SFTP.

Required config:

```yaml
backend: ssh
uri: ssh://user@example.com:22
path: /remote/root
```

Authentication expectations:

- prefer SSH agent by default;
- use known_hosts validation by default where practical;
- do not require passwords in YAML;
- optional key-file support may be implemented if straightforward, but should not distract from agent-based auth.

Support SSH/SFTP backend as both source and destination:

- local -> ssh;
- ssh -> local;
- ssh -> ssh where feasible through staging or streaming.

### Safety Requirements

The SSH backend must enforce the same logical path safety rules as the local backend.

Deletion must remain limited to managed destination bundle paths guarded by valid `.distributor.json`.

### Tests

Unit-test path handling and config validation.

If practical, add integration tests that can be skipped unless an SSH test endpoint is configured through environment variables. Do not require a live SSH server for normal `go test ./...`.

### Completion Criteria

- SSH/SFTP backend compiles and satisfies the storage interface.
- Backend config validation is tested.
- Normal tests do not depend on a live SSH server.
- At least local-to-SSH and SSH-to-local flows are documented or manually testable.

## Stage 10: S3-Compatible Backend Roadmap Extension

### Goal

Implement S3-compatible backend support for sources and destinations.

### Scope

Create:

```text
internal/adapters/s3/backend.go
internal/adapters/s3/config.go
```

Required config should align with `docs/roadmap/config.md`:

```yaml
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: some/prefix
region: us-east-1
force_path_style: true
credentials:
  access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
  secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
```

Implement storage operations over S3 object keys through the common storage contract:

- `ReadFile` and `OpenReader`;
- `WriteFile` and `WriteFrom`;
- `Stat`;
- `Walk` using object-list pagination where available;
- `HasAny` with early stop for destination emptiness checks;
- `DeleteManagedBundle`;
- read/write `.distributor.json` through normal storage operations.

Set reasonable content types where available:

- `.md`: `text/markdown; charset=utf-8`;
- `.html`: `text/html; charset=utf-8`;
- `.json`: `application/json`;
- `.txt`: `text/plain; charset=utf-8`.

Support S3 backend as both source and destination.

### Safety Requirements

Treat S3 prefixes as object trees. Do not assume real directories exist.

Deletion must be limited to destination bundle prefixes that are confirmed managed by `.distributor.json`.

### Tests

Add unit tests for:

- config validation;
- key/prefix normalization;
- content type selection;
- path traversal rejection;
- publish planning with S3 destination state fixtures.

If practical, add integration tests gated by environment variables or a local S3-compatible test service. Normal `go test ./...` must not require live S3 credentials.

### Completion Criteria

- S3 backend compiles and satisfies the storage interface.
- S3 source and destination flows are supported through the common pipeline path.
- Normal tests do not require live S3.

## Stage 11: Cross-Backend End-to-End Coverage and Hardening Roadmap Extension

### Goal

Harden the MVP across backend combinations, destination policies, and failure cases.

### Scope

Add end-to-end coverage for representative scenarios:

- local source -> local archive destination;
- local source -> local HTML destination;
- local source -> two destinations with different publish policies;
- local source -> SSH destination, where integration credentials exist;
- local source -> S3 destination, where integration credentials exist;
- S3 source -> local destination, where integration credentials exist;
- SSH source -> local destination, where integration credentials exist.

Improve logging and error messages for:

- invalid config;
- invalid source manifest;
- digest mismatch;
- destination conflict;
- unmanaged destination path;
- backend read/write/list failures;
- transform failures.

Ensure all destructive paths have tests or explicit safeguards.

### Tests

Add or expand tests for:

- dry-run across multiple destinations;
- partial failure behavior;
- repeated run idempotency;
- older/newer destination state behavior;
- destination state output metadata accuracy;
- generated HTML output metadata accuracy.

### Completion Criteria

- MVP behavior is reliable across implemented backend types.
- Error messages identify pipeline id, destination id, bundle id, and reason where practical.
- Idempotent repeated runs behave as expected.

## Stage 12: User-Facing Documentation Sync

### Goal

Update documentation to reflect implemented MVP behavior.

### Scope

Following `docs/policy/documentation.md`, create or update user-facing documentation only for implemented features.

Likely docs:

```text
README.md
docs/config.md
docs/cli.md
docs/policy/architecture.md
docs/internal/bundles.md
docs/internal/backends.md
```

Document:

- what `distributor` does;
- bundle contract summary;
- `.distributor.json` role;
- example source bundle;
- example local-to-local config;
- example local-to-S3 config;
- example local-to-SSH config;
- `run`, `validate`, and `inspect` commands;
- dry-run behavior;
- replacement and safety rules;
- Markdown-to-HTML transform behavior;
- environment-variable credential handling.

Move roadmap material to historical/planning status only if your documentation policy allows it. Do not describe unimplemented notification adapters as available features.

### Tests

Run the full test suite.

If docs include command examples, verify that basic examples correspond to actual CLI behavior.

### Completion Criteria

- User-facing docs describe the implemented MVP accurately.
- Roadmap docs no longer masquerade as implemented behavior.
- `go test ./...` passes.

## Stage 13: MVP Release Readiness Pass

### Goal

Perform a final pre-release quality pass.

### Scope

Review:

- package boundaries against `docs/policy/architecture.md`;
- package layout against `docs/roadmap/packages.md`;
- implemented contracts against `docs/roadmap/contracts.md`;
- implemented config behavior against `docs/roadmap/config.md`;
- docs against `docs/policy/documentation.md`;
- destructive operation safety;
- logs and errors for unattended operation;
- command UX;
- test coverage for core invariants.

Add any missing small tests or docs discovered during review.

Do not add new product features in this stage.

### Completion Criteria

- MVP is ready to deploy against one real producer pipeline.
- A dry-run can be performed safely against a real source and destination.
- Repeated runs are idempotent.
- Destructive replacement cannot occur outside managed destination bundle paths.
- Final docs accurately reflect the application.

## Stage 14: Explicit Force Overwrite Roadmap Extension

### Goal

Introduce explicit operator-requested force behavior for controlled overwrite cases that are intentionally outside the local MVP.

### Scope

Add a CLI-only force option such as:

```bash
distributor run --config config.yml --force
```

Define and implement force planning for:

- unmanaged non-empty destination paths;
- destination state with a different source id;
- destination state with matching source id and created timestamp but different digest;
- destination state with mismatched `pipeline_id` or `destination_id`;
- destination newer than source when the transfer policy explicitly allows replacement.

Force behavior must be explicit per run. It should not be a persistent default in config for this stage.

Update transfer policy validation to allow broader values only when force behavior is implemented and documented:

- `on_destination_newer: replace`
- `on_conflict: replace`

### Safety Requirements

- Dry-run must show every file or object that would be written or deleted before a forced run.
- Force must still never delete above the resolved destination bundle path or configured destination prefix.
- Filesystem replacement should remain staged where practical.
- S3 replacement must remain constrained to the destination bundle prefix.
- Logs must clearly mark force decisions and include pipeline id, destination id, bundle id, and reason.

### Tests

Add tests for:

- force rejected when the flag is absent;
- unmanaged non-empty destination overwritten only with force;
- different source id overwritten only with force and allowed policy;
- same id and created but different digest overwritten only with force and allowed policy;
- destination newer replaced only with force and allowed policy;
- pipeline or destination id mismatch overwritten only with force and allowed policy;
- dry-run reports destructive force actions without writing;
- destructive paths remain bounded to the destination bundle path.

### Completion Criteria

- Force overwrite behavior is explicit, logged, dry-runnable, and test-covered.
- Default non-force behavior remains unchanged and conservative.

## Deferred Post-MVP Work

The following items are intentionally outside the MVP unless explicitly pulled into a later roadmap:

- email notifications;
- ntfy/Gotify/Pushover notifications;
- RSS/Atom feed generation;
- static site index pages beyond sidecar HTML output;
- templated HTML themes beyond a minimal deterministic template;
- destination path remapping rules;
- full plugin architecture;
- web UI;
- report editing;
- producer pipeline execution;
- database-backed state;
- complex retry queues;
- concurrent publication workers;
- symlink support;
- warning-only digest mismatch handling;
- password-based SSH authentication in YAML.