# Distributor Implementation Roadmap

This roadmap defines a staged implementation plan for the `distributor` MVP. Each stage is intended to map cleanly to one Codex implementation prompt.

The roadmap assumes the project includes these planning documents before implementation begins:

- `docs/policy/architecture.md`
- `docs/policy/documentation.md`
- `docs/roadmap/packages.md`
- `docs/roadmap/contracts.md`
- `docs/roadmap/config.md`

The MVP goal is a domain-agnostic bundle distributor that discovers source bundles, validates `manifest.json`, optionally transforms Markdown to HTML, publishes selected outputs to one or more destinations, and records destination state in `.distributor.json`.

## Global Implementation Rules

All stages should preserve these invariants:

- Producer applications own source bundle creation.
- `distributor` owns validation, transformation, publication, destination state, and future notification hooks.
- Source bundle state is defined by `manifest.json`.
- Destination publication state is defined by `.distributor.json`.
- `manifest.json` is not copied to the destination as destination state.
- Pipelines have exactly one source and one or more destinations.
- Transform and publish policy are destination-specific.
- Destructive replacement is allowed only inside a managed destination bundle path. Unsafe force or unmanaged overwrite behavior is deferred.
- Dry-run behavior should be implemented before broad remote write behavior.
- Config, bundle, state, publish planning, storage adapters, transforms, and CLI wiring should remain separate packages.

Unless a stage explicitly says otherwise, each implementation prompt should:

1. read the project policy and roadmap documents;
2. implement only the current stage;
3. add or update tests for the current stage;
4. run the relevant test suite;
5. update documentation only when the implemented behavior now exists;
6. avoid implementing future roadmap stages early.

## Stage 1: Project Skeleton, CLI Shell, and Baseline Tooling

### Goal

Create the initial Go application structure and a minimal executable `distributor` command with no business behavior beyond version/help output and placeholder commands.

### Scope

Implement the accepted package skeleton from `docs/roadmap/packages.md` at the level needed for compilation.

Create:

```text
cmd/distributor/main.go
internal/cli/
internal/app/
internal/config/
internal/logging/
```

Initial CLI commands:

- `distributor --help`
- `distributor version`
- `distributor run`
- `distributor validate`
- `distributor inspect`

At this stage, `run`, `validate`, and `inspect` may return clear “not implemented” errors, but the command structure should be present.

### Notes

Prefer a small CLI dependency only if the project already standardizes on one. Otherwise, the standard library is acceptable for the first pass.

Add a version variable that can later be set at build time.

### Tests

Add tests for:

- command construction if testable;
- version string behavior if exposed through a package;
- basic package compilation.

### Completion Criteria

- `go test ./...` passes.
- `go run ./cmd/distributor --help` works.
- `go run ./cmd/distributor version` works.
- Placeholder operational commands fail clearly and intentionally.

## Stage 2: Config Schema, Loading, Defaults, and Validation

### Goal

Implement the MVP `config.yml` schema described in `docs/roadmap/config.md`.

### Scope

Create config structs for:

- top-level config;
- pipelines;
- source backend config;
- destination backend config;
- validation policy;
- publish policy;
- transform policy;
- transfer/replacement policy;
- backend-specific local, SSH, and S3 fields.

Support loading YAML from a file path.

Implement validation for:

- required top-level `pipelines`;
- unique pipeline ids;
- required pipeline `id`, `source`, and non-empty `destinations`;
- unique destination ids within a pipeline;
- supported backend names: `local`, `ssh`, `s3`;
- required backend fields;
- supported validation action: `fail`;
- supported transfer actions;
- valid `publish` policy;
- valid Markdown-to-HTML transform config.

Default behavior should match `docs/roadmap/config.md`.

### CLI Integration

Add `--config` to `run`.

For this stage, `distributor run --config config.yml --dry-run` may only load and validate config, then print a concise summary of configured pipelines and destinations.

### Tests

Add unit tests for:

- valid minimal local-to-local config;
- valid fan-out config;
- valid local, SSH, and S3 backend configs;
- duplicate pipeline ids;
- duplicate destination ids;
- missing required fields;
- unsupported backend;
- invalid transfer action;
- invalid validation action, including `warn`.

### Completion Criteria

- Config load/default/validate behavior is implemented and tested.
- `distributor run --config <file> --dry-run` validates config and prints a summary.
- No bundle discovery or publication occurs yet.

## Stage 3: Storage Abstraction, Local Backend, and Fake Backend

### Goal

Introduce the storage backend abstraction before bundle validation so source discovery, validation, and publication are backend-agnostic from the start.

### Scope

Create:

```text
internal/storage/backend.go
internal/storage/registry.go
internal/storage/path.go
internal/storage/errors.go
internal/adapters/local/backend.go
internal/storage/fake/
```

Define backend operations needed by the MVP, including:

- read file;
- write file;
- test existence;
- list files or tree entries;
- read destination state file if present;
- create directories/prefixes as needed;
- delete explicit managed files safely;
- write files using temporary/staged writes where practical.

The fake backend should exist for unit tests of config, bundle, state, and publish logic without real local, SSH, or S3 IO.

### Safety Requirements

The local backend must:

- clean and join paths safely;
- reject path traversal;
- reject unsafe destructive deletion requests;
- avoid following symlinks for source bundle files unless explicitly supported;
- avoid deleting configured roots;
- classify destination bundle emptiness deterministically.

### Tests

Add tests for:

- backend read/write/list/exists behavior;
- safe path joining;
- traversal rejection;
- symlink rejection for source reads;
- explicit-file deletion guard behavior;
- local destination emptiness detection;
- fake backend behavior sufficient for core package tests.

### Completion Criteria

- Local backend implements the storage interface.
- Fake backend can support bundle and publish tests without external services.
- `go test ./...` passes.
- No SSH or S3 implementation exists yet.

## Stage 4: Source Bundle Manifest, Digest, Validation, and Discovery

### Goal

Implement the source bundle contract from `docs/roadmap/contracts.md` through the storage abstraction.

### Scope

Create:

```text
internal/bundle/manifest.go
internal/bundle/digest.go
internal/bundle/validate.go
internal/bundle/discover.go
```

Implement:

- parsing `manifest.json`;
- strict required field validation, including `schema_version: 1`;
- RFC3339 `created` parsing;
- lowercase `sha256:<64 hex>` digest validation;
- source file path safety checks;
- duplicate logical file path rejection;
- per-file SHA256 validation;
- per-file size validation;
- bundle digest validation using the canonical ordered file-record algorithm;
- deterministic storage-backed bundle discovery under a source root;
- nested manifest detection and failure.

Discovery and validation should use `internal/storage` rather than direct `os` APIs. The local CLI path should be adapted to the local backend.

### CLI Integration

Implement:

```text
distributor validate <path>
distributor inspect <path>
```

For local paths:

- `validate` should validate either a single bundle directory or a tree containing bundles.
- `inspect` should print a concise normalized summary of discovered bundle ids, relative paths, created timestamps, digest values, and files.

### Tests

Add fixture bundles under a testdata directory.

Test:

- valid bundle;
- invalid JSON;
- missing required fields;
- invalid schema version;
- invalid timestamp;
- invalid digest format;
- unsafe file paths;
- duplicate normalized file paths;
- missing files;
- size mismatch;
- per-file digest mismatch;
- bundle digest mismatch;
- canonical bundle digest reference fixture;
- multiple discovered bundles in deterministic order;
- nested manifests fail.

### Completion Criteria

- Storage-backed bundle validation is deterministic and well-tested.
- `distributor validate <path>` works for local bundle fixtures.
- `distributor inspect <path>` works for local bundle fixtures.
- No destination publication occurs yet.

## Stage 5: Destination State Contract and Comparison Logic

### Goal

Implement `.distributor.json` parsing, validation, and source-to-destination comparison.

### Scope

Create:

```text
internal/state/distributor.go
internal/state/compare.go
internal/state/validate.go
```

Implement the destination state schema from `docs/roadmap/contracts.md`, including:

- `schema_version`;
- optional `distributor_version`;
- `pipeline_id`;
- `destination_id`;
- `published_at`;
- embedded normalized source manifest;
- outputs array;
- output file metadata.

Implement comparison outcomes:

- destination absent;
- destination unmanaged/non-empty;
- same source manifest;
- same source id, destination older;
- same source id, destination newer;
- same source id and same created but different digest;
- different source id;
- invalid destination state.

### Tests

Add unit tests for every comparison outcome.

Test validation for:

- valid state;
- missing fields;
- invalid schema version;
- invalid embedded source manifest;
- invalid output metadata;
- malformed published timestamp.

### Completion Criteria

- Destination state can be parsed and validated independently.
- Source manifest to destination state comparison is deterministic and fully tested.
- No publication execution occurs yet.

## Stage 6: Publish Planning, Dry-Run, and Local-to-Local Publication Without Transform

### Goal

Implement the core publish planner and execute local-to-local publication for source files only.

### Scope

Create:

```text
internal/publish/plan.go
internal/publish/reconcile.go
internal/publish/safety.go
internal/publish/output.go
internal/publish/execute.go
```

Implement planning for one source bundle to one destination based on:

- source manifest;
- destination config;
- publish policy;
- transfer policy;
- existing `.distributor.json`;
- destination path state.

Actions should include:

- publish new;
- replace older destination;
- skip same;
- skip destination newer;
- fail conflict;
- fail unmanaged destination.

Implement local-to-local execution for `publish.source: true` and `publish.html: false`.

Execution should:

- copy listed source files selected by publish policy;
- write `.distributor.json` with copied source output metadata;
- avoid copying source `manifest.json` as destination state;
- preserve relative bundle paths from source root beneath destination root;
- detect destination output collisions before writing;
- use staging or equivalent cleanup behavior for local writes;
- support fan-out to multiple local destinations;
- support dry-run without writes.

### CLI Integration

`distributor run --config <file>` should now execute local-to-local pipelines when configured.

`--dry-run` should print the planned action for each discovered bundle and destination.

### Tests

Add integration-style tests using temp directories for:

- new local publication;
- no-op when destination state matches;
- replacement when destination state is older;
- skip when destination state is newer;
- fail on conflict;
- fail on unmanaged non-empty destination;
- fail on output path collision;
- fan-out from one source to two local destinations;
- failed local write does not leave a destination that appears unmanaged on retry;
- dry-run performs no writes;
- `.distributor.json` is written correctly.

### Completion Criteria

- Local-to-local source-file publication works end to end.
- Dry-run produces meaningful planned actions.
- Destination state is authoritative.
- No Markdown-to-HTML transform exists yet.

## Stage 7: Markdown-to-HTML Transform and Destination-Specific Publish Policy

### Goal

Add MVP Markdown-to-HTML transformation and destination-specific source/html output selection.

### Scope

Create:

```text
internal/transform/transform.go
internal/transform/registry.go
internal/transform/plan.go
internal/transform/markdown/markdown.go
internal/transform/markdown/template.go
```

Implement only:

```yaml
transform:
  markdown_to_html:
    enabled: true
    mode: sidecar
```

MVP sidecar behavior:

- for each listed source artifact ending in `.md`, generate a same-directory `.html` sidecar;
- preserve the original Markdown file unchanged;
- do not generate HTML for non-Markdown files;
- escape or disable raw HTML embedded in Markdown;
- fail before writing when generated output paths collide with copied source outputs or other generated outputs;
- record generated output metadata in `.distributor.json`;
- if `publish.source: false`, do not publish source files;
- if `publish.html: true`, publish generated HTML files;
- if `publish.html: true` but transform is disabled or no Markdown files exist, fail with a clear error unless config later defines another behavior.

Use a well-maintained Markdown renderer. Keep HTML templating minimal and deterministic.

### Tests

Add tests for:

- Markdown sidecar generation;
- source-only destination;
- HTML-only destination;
- source-plus-HTML destination;
- no mutation of source bundle;
- generated output metadata in `.distributor.json`;
- failure when HTML publication is requested without transform support;
- failure when generated HTML collides with a source artifact path;
- raw HTML in Markdown is escaped or disabled consistently;
- deterministic output for a fixture Markdown file.

### Completion Criteria

- Local-to-local publication supports source-only, HTML-only, and source-plus-HTML destinations.
- Generated outputs are recorded in destination state.
- Dry-run reports transform outputs that would be generated.

## Stage 8: No-Op Notification Stage and Pipeline Orchestration Polish

### Goal

Add the internal no-op notification stage and polish orchestration around per-destination outcomes.

### Scope

Create:

```text
internal/notify/notify.go
internal/notify/noop.go
```

Integrate a no-op notifier after successful publication/skip handling as appropriate.

Clarify orchestration behavior when one destination fails. For MVP, fan-out should be deterministic and sequential. Continue planning and reporting later destinations where safe, but return non-zero if any destination fails.

Improve run summary output:

- pipeline id;
- source backend;
- discovered bundle count;
- destination ids;
- action per bundle/destination;
- final status.

### Tests

Add tests for:

- notifier is invoked at the expected orchestration point where testable;
- pipeline failure when a destination fails;
- run summary contains meaningful status information;
- dry-run does not invoke write-side effects.

### Completion Criteria

- The pipeline shape includes notification as an internal no-op stage.
- Run output is useful for unattended operation logs.
- Local MVP behavior remains passing.

## Stage 9: Native SSH/SFTP Backend

### Goal

Implement SSH/SFTP storage backend support for sources and destinations.

### Scope

Create:

```text
internal/adapters/ssh/backend.go
internal/adapters/ssh/config.go
```

Implement the storage backend interface over native SSH/SFTP.

Required config:

```yaml
backend: ssh
uri: ssh://user@example.com:22
path: /remote/root
```

Authentication expectations:

- prefer SSH agent by default;
- use known_hosts validation by default where practical;
- do not require passwords in YAML;
- optional key-file support may be implemented if straightforward, but should not distract from agent-based auth.

Support SSH/SFTP backend as both source and destination:

- local -> ssh;
- ssh -> local;
- ssh -> ssh where feasible through staging or streaming.

### Safety Requirements

The SSH backend must enforce the same logical path safety rules as the local backend.

Deletion must remain limited to managed destination bundle paths guarded by valid `.distributor.json`.

### Tests

Unit-test path handling and config validation.

If practical, add integration tests that can be skipped unless an SSH test endpoint is configured through environment variables. Do not require a live SSH server for normal `go test ./...`.

### Completion Criteria

- SSH/SFTP backend compiles and satisfies the storage interface.
- Backend config validation is tested.
- Normal tests do not depend on a live SSH server.
- At least local-to-SSH and SSH-to-local flows are documented or manually testable.

## Stage 10: S3-Compatible Backend

### Goal

Implement S3-compatible backend support for sources and destinations.

### Scope

Create:

```text
internal/adapters/s3/backend.go
internal/adapters/s3/config.go
```

Required config should align with `docs/roadmap/config.md`:

```yaml
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: some/prefix
region: us-east-1
force_path_style: true
credentials:
  access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
  secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
```

Implement storage operations over S3 object keys:

- read object;
- write object;
- exists;
- list prefix;
- delete managed prefix or listed managed files;
- read/write `.distributor.json`.

Set reasonable content types where available:

- `.md`: `text/markdown; charset=utf-8`;
- `.html`: `text/html; charset=utf-8`;
- `.json`: `application/json`;
- `.txt`: `text/plain; charset=utf-8`.

Support S3 backend as both source and destination.

### Safety Requirements

Treat S3 prefixes as object trees. Do not assume real directories exist.

Deletion must be limited to destination bundle prefixes that are confirmed managed by `.distributor.json`.

### Tests

Add unit tests for:

- config validation;
- key/prefix normalization;
- content type selection;
- path traversal rejection;
- publish planning with S3 destination state fixtures.

If practical, add integration tests gated by environment variables or a local S3-compatible test service. Normal `go test ./...` must not require live S3 credentials.

### Completion Criteria

- S3 backend compiles and satisfies the storage interface.
- S3 source and destination flows are supported through the common pipeline path.
- Normal tests do not require live S3.

## Stage 11: Cross-Backend End-to-End Coverage and Hardening

### Goal

Harden the MVP across backend combinations, destination policies, and failure cases.

### Scope

Add end-to-end coverage for representative scenarios:

- local source -> local archive destination;
- local source -> local HTML destination;
- local source -> two destinations with different publish policies;
- local source -> SSH destination, where integration credentials exist;
- local source -> S3 destination, where integration credentials exist;
- S3 source -> local destination, where integration credentials exist;
- SSH source -> local destination, where integration credentials exist.

Improve logging and error messages for:

- invalid config;
- invalid source manifest;
- digest mismatch;
- destination conflict;
- unmanaged destination path;
- backend read/write/list failures;
- transform failures.

Ensure all destructive paths have tests or explicit safeguards.

### Tests

Add or expand tests for:

- dry-run across multiple destinations;
- partial failure behavior;
- repeated run idempotency;
- older/newer destination state behavior;
- destination state output metadata accuracy;
- generated HTML output metadata accuracy.

### Completion Criteria

- MVP behavior is reliable across implemented backend types.
- Error messages identify pipeline id, destination id, bundle id, and reason where practical.
- Idempotent repeated runs behave as expected.

## Stage 12: User-Facing Documentation Sync

### Goal

Update documentation to reflect implemented MVP behavior.

### Scope

Following `docs/policy/documentation.md`, create or update user-facing documentation only for implemented features.

Likely docs:

```text
README.md
docs/config.md
docs/cli.md
docs/policy/architecture.md
docs/internal/bundles.md
docs/internal/backends.md
```

Document:

- what `distributor` does;
- bundle contract summary;
- `.distributor.json` role;
- example source bundle;
- example local-to-local config;
- example local-to-S3 config;
- example local-to-SSH config;
- `run`, `validate`, and `inspect` commands;
- dry-run behavior;
- replacement and safety rules;
- Markdown-to-HTML transform behavior;
- environment-variable credential handling.

Move roadmap material to historical/planning status only if your documentation policy allows it. Do not describe unimplemented notification adapters as available features.

### Tests

Run the full test suite.

If docs include command examples, verify that basic examples correspond to actual CLI behavior.

### Completion Criteria

- User-facing docs describe the implemented MVP accurately.
- Roadmap docs no longer masquerade as implemented behavior.
- `go test ./...` passes.

## Stage 13: MVP Release Readiness Pass

### Goal

Perform a final pre-release quality pass.

### Scope

Review:

- package boundaries against `docs/policy/architecture.md`;
- package layout against `docs/roadmap/packages.md`;
- implemented contracts against `docs/roadmap/contracts.md`;
- implemented config behavior against `docs/roadmap/config.md`;
- docs against `docs/policy/documentation.md`;
- destructive operation safety;
- logs and errors for unattended operation;
- command UX;
- test coverage for core invariants.

Add any missing small tests or docs discovered during review.

Do not add new product features in this stage.

### Completion Criteria

- MVP is ready to deploy against one real producer pipeline.
- A dry-run can be performed safely against a real source and destination.
- Repeated runs are idempotent.
- Destructive replacement cannot occur outside managed destination bundle paths.
- Final docs accurately reflect the application.

## Deferred Post-MVP Work

The following items are intentionally outside the MVP unless explicitly pulled into a later roadmap:

- email notifications;
- ntfy/Gotify/Pushover notifications;
- RSS/Atom feed generation;
- static site index pages beyond sidecar HTML output;
- templated HTML themes beyond a minimal deterministic template;
- destination path remapping rules;
- full plugin architecture;
- web UI;
- report editing;
- producer pipeline execution;
- database-backed state;
- complex retry queues;
- concurrent publication workers;
- symlink support;
- warning-only digest mismatch handling;
- force or unmanaged-overwrite destination behavior;
- password-based SSH authentication in YAML.