diff --git a/docs/roadmap/config.md b/docs/roadmap/config.md deleted file mode 100644 index 7af2da5..0000000 --- a/docs/roadmap/config.md +++ /dev/null @@ -1,580 +0,0 @@ -# Distributor Configuration Roadmap - -This roadmap defines the planned `config.yml` schema for the `distributor` MVP. The goal is to support one-to-many publication pipelines where each pipeline has one source and one or more destinations. Each destination independently controls backend configuration, publication outputs, transform behavior, and replacement policy. - -## Configuration Goals - -The MVP configuration should be: - -- explicit enough to avoid hidden publication behavior; -- compact enough for routine self-hosted use; -- backend-agnostic at the pipeline layer; -- capable of local, SSH/SFTP, and S3-compatible source and destination backends; -- ready for future notification adapters without exposing a fake notification feature in the MVP. - -## Top-Level Shape - -```yaml -pipelines: - - id: weather-daily - source: - backend: local - path: /var/spool/distributor/weather - - validation: - on_digest_mismatch: fail - - destinations: - - id: markdown-archive - backend: s3 - endpoint: https://s3.example.com - bucket: reports - prefix: weather/archive - region: us-east-1 - force_path_style: true - credentials: - access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID - secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY - - publish: - source: true - html: false - - transfer: - on_destination_same: skip - on_destination_older: replace - on_destination_newer: skip - on_conflict: fail - - - id: static-site - backend: ssh - uri: ssh://deploy@example.com:22 - path: /srv/www/weather - - publish: - source: false - html: true - - transform: - markdown_to_html: - enabled: true - mode: sidecar - - transfer: - on_destination_same: skip - on_destination_older: replace - on_destination_newer: skip - on_conflict: fail -``` - -## Pipeline Fields - -Each pipeline must include: - -- `id`: Required stable pipeline identifier. -- `source`: Required source backend configuration. -- `destinations`: Required non-empty list of destination configurations. - -Optional pipeline-level fields: - -- `validation`: Source validation behavior. -- Future: `notifications` or `notify`, when notification adapters are implemented. - -A pipeline has exactly one source and one or more destinations. - -## Pipeline ID Rules - -`pipelines[].id` should: - -- be required; -- be unique across the config file; -- be stable over time; -- use a simple slug-like format, such as `weather-daily` or `dnd-session-recaps`. - -Recommended validation: - -```text -^[a-zA-Z0-9][a-zA-Z0-9._-]*$ -``` - -## Source Configuration - -`source` defines the source root where bundles are discovered. - -The source backend may be: - -- `local`; -- `ssh`; -- `s3`. - -The source is scanned for `manifest.json` files beneath the configured root. - -### Local source - -```yaml -source: - backend: local - path: /var/spool/distributor/weather -``` - -Required fields: - -- `backend: local` -- `path` - -### SSH source - -```yaml -source: - backend: ssh - uri: ssh://reports@example.com:22 - path: /var/spool/distributor/weather -``` - -Required fields: - -- `backend: ssh` -- `uri` -- `path` - -Recommended authentication behavior: - -- use SSH agent by default; -- use local known_hosts validation by default; -- support optional key file configuration later if needed; -- do not require passwords in YAML. - -Optional future fields: - -```yaml -known_hosts: /home/user/.ssh/known_hosts -key_file: /home/user/.ssh/id_ed25519 -``` - -### S3 source - -```yaml -source: - backend: s3 - endpoint: https://s3.example.com - bucket: reports - prefix: incoming/weather - region: us-east-1 - force_path_style: true - credentials: - access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID - secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY -``` - -Required fields: - -- `backend: s3` -- `endpoint` -- `bucket` - -Optional fields: - -- `prefix` -- `region` -- `force_path_style` -- `credentials` - -Credential configuration should prefer environment variables or standard SDK behavior over literal secrets in config. - -## Destination Configuration - -Each destination defines one publication target for a pipeline. - -Required destination fields: - -- `id` -- `backend` -- backend-specific location fields; -- `publish` - -Optional destination fields: - -- `transform` -- `transfer` - -Each destination is independently planned and published. A destination may receive source files, generated HTML, or both. - -## Destination ID Rules - -`destinations[].id` should: - -- be required; -- be unique within the containing pipeline; -- be stable over time; -- use a slug-like format. - -Recommended examples: - -- `markdown-archive` -- `static-site` -- `full-mirror` - -## Local Destination - -```yaml -destinations: - - id: local-static - backend: local - path: /srv/www/reports - publish: - source: false - html: true -``` - -Required fields: - -- `backend: local` -- `path` -- `publish` - -## SSH Destination - -```yaml -destinations: - - id: static-site - backend: ssh - uri: ssh://deploy@example.com:22 - path: /srv/www/weather - publish: - source: false - html: true -``` - -Required fields: - -- `backend: ssh` -- `uri` -- `path` -- `publish` - -The MVP should use a native SFTP implementation rather than shelling out to `ssh`, `scp`, or `rsync`. - -## S3 Destination - -```yaml -destinations: - - id: markdown-archive - backend: s3 - endpoint: https://s3.example.com - bucket: reports - prefix: weather/archive - region: us-east-1 - force_path_style: true - credentials: - access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID - secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY - publish: - source: true - html: false -``` - -Required fields: - -- `backend: s3` -- `endpoint` -- `bucket` -- `publish` - -Optional fields: - -- `prefix` -- `region` -- `force_path_style` -- `credentials` - -## Backend Configuration Normalization - -The config loader should normalize backend configuration into internal source and destination backend specs. Pipeline logic should not branch on backend-specific fields. - -Validation should catch: - -- missing backend names; -- unsupported backend names; -- missing backend-specific required fields; -- duplicate pipeline IDs; -- duplicate destination IDs within a pipeline; -- empty destination lists; -- invalid policy values. - -## Publication Policy - -`publish` controls which categories of files are written to a destination. - -```yaml -publish: - source: true - html: false -``` - -Fields: - -- `source`: Publish source artifacts listed in `manifest.json`. -- `html`: Publish HTML files generated from Markdown source artifacts. - -At least one of `source` or `html` must be true. - -Recommended defaults: - -```yaml -publish: - source: true - html: false -``` - -No implicit HTML transformation should occur. When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `mode: sidecar` are required for the MVP. - -## Transform Configuration - -For MVP, the only supported transform is Markdown to HTML. - -```yaml -transform: - markdown_to_html: - enabled: true - mode: sidecar -``` - -Fields: - -- `enabled`: Whether Markdown-to-HTML transform is enabled. -- `mode`: Output mode. MVP value: `sidecar`. - -MVP `sidecar` behavior: - -- each listed Markdown source file generates an HTML file with the same base path and `.html` extension; -- `report.md` generates `report.html`; -- generated files are destination publication artifacts; -- source bundles are not mutated. - -MVP defaulting: - -- If `publish.html` is false, transform may be omitted. -- If `publish.html` is true and `transform.markdown_to_html` is omitted or disabled, config validation must fail. - -## Validation Policy - -Pipeline-level validation is intentionally narrow in the MVP. - -```yaml -validation: - on_digest_mismatch: fail -``` - -Supported value: - -- `fail` - -Default: - -```yaml -on_digest_mismatch: fail -``` - -Validation should happen before any destination writes. Warning-only digest mismatch handling is deferred and must be rejected if configured. - -## Transfer Policy - -Destination-level transfer policy controls behavior after inspecting `.distributor.json` at the destination bundle path. - -```yaml -transfer: - on_destination_same: skip - on_destination_older: replace - on_destination_newer: skip - on_conflict: fail -``` - -Supported fields: - -- `on_destination_same` -- `on_destination_older` -- `on_destination_newer` -- `on_conflict` - -MVP supported values are intentionally limited by field: - -- `on_destination_same`: `skip` or `fail` -- `on_destination_older`: `replace` or `fail` -- `on_destination_newer`: `skip` or `fail` -- `on_conflict`: `fail` - -Recommended MVP defaults: - -```yaml -transfer: - on_destination_same: skip - on_destination_older: replace - on_destination_newer: skip - on_conflict: fail -``` - -Safety rule: - -- `replace` must never perform broad deletion against a destination root. -- `replace` may only operate within a resolved destination bundle path and should delete only files recorded in existing `.distributor.json.outputs` plus `.distributor.json` where practical. -- Unmanaged non-empty destination paths fail in the MVP. Force or unmanaged overwrite configuration is deferred. -- Broader replacement values, including replacing newer destinations or conflicts, are deferred to a later explicit force-overwrite stage. - -## Path Mapping - -MVP path mapping is fixed: - -```text -destination bundle path = destination root + source relative bundle path -``` - -Example: - -```text -source root: /var/spool/reports -source bundle: /var/spool/reports/weather/daily/brentwood/2026-05-30 -relative bundle path: weather/daily/brentwood/2026-05-30 - -destination root: /srv/www/reports -destination bundle path: /srv/www/reports/weather/daily/brentwood/2026-05-30 -``` - -Future config may support explicit path mapping, but MVP should not. - -## Dry Run Configuration and CLI Behavior - -Dry-run should be a CLI flag rather than a persistent config setting. - -```bash -distributor run --config config.yml --dry-run -``` - -Dry-run should report: - -- pipeline ID; -- source backend; -- destination ID; -- destination backend; -- discovered bundle ID; -- relative bundle path; -- planned action; -- reason; -- transform outputs that would be generated; -- files that would be written or deleted. - -## Example: Weather Pipeline - -```yaml -pipelines: - - id: weather-daily - source: - backend: local - path: /var/spool/distributor/weather - - validation: - on_digest_mismatch: fail - - destinations: - - id: markdown-archive - backend: s3 - endpoint: https://s3.example.com - bucket: reports - prefix: weather/archive - region: us-east-1 - force_path_style: true - credentials: - access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID - secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY - publish: - source: true - html: false - - - id: static-site - backend: ssh - uri: ssh://deploy@web.example.com:22 - path: /srv/www/weather - publish: - source: false - html: true - transform: - markdown_to_html: - enabled: true - mode: sidecar -``` - -## Example: D&D Recap Pipeline - -```yaml -pipelines: - - id: dnd-session-recaps - source: - backend: local - path: /var/spool/distributor/dnd/session-recaps - - destinations: - - id: private-markdown-archive - backend: s3 - endpoint: https://s3.example.com - bucket: reports - prefix: dnd/session-recaps - region: us-east-1 - force_path_style: true - credentials: - access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID - secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY - publish: - source: true - html: false - - - id: private-html-site - backend: local - path: /srv/www/private/dnd/session-recaps - publish: - source: false - html: true - transform: - markdown_to_html: - enabled: true - mode: sidecar -``` - -## Future Notification Configuration - -Notification should not be exposed as a functional MVP feature unless an adapter exists. - -The internal pipeline may include a no-op notification stage. Future config may look like: - -```yaml -notifications: - - id: weather-email - backend: email - after_destinations: - - static-site - subject: "Weather report published" -``` - -Future notification policy should require: - -- notification after successful relevant publication; -- idempotency by source manifest id and digest; -- no duplicate notification unless explicitly forced. - -## Implementation Stages - -1. Define config structs for pipelines, sources, destinations, validation, publish, transform, and transfer policies. -2. Implement config loading and strict validation. -3. Implement backend-specific config validation for local, SSH, and S3. -4. Implement defaulting for validation and transfer policies. -5. Require explicit transform configuration when `publish.html` is true. -6. Add example config fixtures for local-to-local, local-to-SSH, local-to-S3, and fan-out scenarios. -7. Connect config to backend registry and publish planner. -8. Add `--pipeline` filtering for targeted runs. -9. Add `--dry-run` output that reflects the resolved config and planned actions. - -## Deferred Configuration - -The following configuration ideas are intentionally outside the MVP: - -- warning-only digest mismatch handling; -- unmanaged destination overwrite flags until the explicit force-overwrite roadmap stage; -- force replacement of destinations with different source ids until the explicit force-overwrite roadmap stage. diff --git a/docs/roadmap/contracts.md b/docs/roadmap/contracts.md deleted file mode 100644 index 0af8107..0000000 --- a/docs/roadmap/contracts.md +++ /dev/null @@ -1,398 +0,0 @@ -# Distributor Contracts Roadmap - -This roadmap defines the contracts that `distributor` should implement before or alongside the MVP. The goal is to make bundle validation, destination state, digest verification, and safe replacement deterministic and testable before backend-specific publication behavior is layered on top. - -## Purpose - -`distributor` publishes manifested report bundles produced by other applications. Producer applications own domain-specific report generation. `distributor` owns validation, optional transformation, destination publication, and destination state. - -The MVP contract has two durable files: - -- `manifest.json`: source-owned bundle manifest produced by the upstream application. -- `.distributor.json`: destination-owned publication state written by `distributor`. - -`manifest.json` is not copied to the destination as destination state. Instead, `.distributor.json` records the normalized source manifest, generated output metadata, and distributor-owned publication metadata. - -All manifest and state timestamps should be serialized as RFC3339. Internal comparison should use parsed timestamp values, and distributor-written timestamps should be normalized to RFC3339 UTC. - -## Terminology - -- **Source root**: Configured root path for a pipeline source. -- **Bundle root**: Directory beneath the source root that contains `manifest.json`. -- **Relative bundle path**: Bundle root path relative to the source root. -- **Destination root**: Configured root path or prefix for a destination. -- **Destination bundle path**: Destination root plus the relative bundle path, unless a future mapping option overrides that behavior. -- **Source artifact**: File listed in the source `manifest.json`. -- **Generated artifact**: File created by `distributor`, such as an HTML file derived from Markdown. -- **Destination state**: `.distributor.json` at the destination bundle path. - -## Source Bundle Contract - -A source bundle is a directory containing a `manifest.json` file. For MVP, the manifest schema is intentionally minimal. - -### Required `manifest.json` fields - -```json -{ - "schema_version": 1, - "id": "weather.daily.brentwood.2026-05-30", - "digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000", - "created": "2026-05-30T11:10:00Z", - "files": [ - { - "path": "report.md", - "sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111", - "size": 12345 - } - ] -} -``` - -Required top-level fields: - -- `schema_version`: Source manifest schema version. MVP value: `1`. -- `id`: Stable bundle identifier. Required, non-empty string. -- `digest`: Bundle digest. Required, lowercase `sha256:<64 hex>` string. -- `created`: Bundle creation timestamp. Required RFC3339 timestamp. UTC is preferred; explicit offsets are allowed. -- `files`: Non-empty array of file objects. - -Required file fields: - -- `path`: Relative path from bundle root to source artifact. -- `sha256`: Per-file digest as lowercase `sha256:<64 hex>`. -- `size`: File size in bytes. - -No other source manifest fields are required for the MVP. Additional fields may be ignored unless later documented. Destination state records the normalized source manifest model, not raw unknown manifest fields. - -## Source Path Safety Rules - -For every `files[].path`: - -- Path must be relative. -- Path must not be empty. -- Path must not contain `..` segments. -- Path must not resolve outside the bundle root. -- Path must use slash-separated logical paths in the manifest. -- Absolute paths are invalid. -- Symlinks should be rejected for MVP unless a later policy deliberately supports them. -- `manifest.json` itself should not be listed as a source artifact. -- `.distributor.json` should not be listed as a source artifact. -- Duplicate logical file paths are invalid after path normalization. - -The implementation should validate paths before reading file contents. - -## Digest Contract - -The MVP validates both per-file digests and the bundle digest. - -### Per-file digest - -For each file listed in `files`, compute: - -```text -sha256(file bytes) -``` - -The computed digest must match `files[].sha256`. - -The actual file size must match `files[].size`. - -### Bundle digest - -The bundle digest is computed from the listed file records in the listed order. The canonical algorithm is: - -1. For each file listed in `files`, in order: - - validate the file path; - - compute the file SHA256; - - determine the file size. -2. Construct a canonical JSON array containing only: - - `path`; - - `sha256`; - - `size`. -3. Preserve the source manifest's file order. -4. Encode the array deterministically with: - - object fields in exactly this order: `path`, `sha256`, `size`; - - no extra spaces; - - no trailing newline; - - lowercase `sha256:<64 hex>` digest strings. -5. Compute `sha256(canonical JSON bytes)`. -6. Compare the result to top-level `digest`. - -Conceptual canonical payload: - -```json -[ - {"path":"report.md","sha256":"sha256:...","size":12345}, - {"path":"summary.txt","sha256":"sha256:...","size":234} -] -``` - -This avoids ambiguous concatenation of file bytes while keeping the manifest small. - -Implementation fixtures should include at least one reference manifest and canonical payload with known per-file and bundle digests. - -## Digest Mismatch Behavior - -Digest validation is always fatal in the MVP. A digest mismatch must fail the affected bundle, pipeline, and run before any destination writes occur. - -Warning-only digest behavior is intentionally deferred until a later roadmap accepts transitional ingestion semantics. - -## Bundle Discovery Contract - -A source path may contain one bundle or a tree of bundles. Discovery should scan beneath the configured source root for `manifest.json` files. - -For each discovered manifest: - -- Bundle root is the directory containing `manifest.json`. -- Relative bundle path is computed relative to source root. -- Destination bundle path is destination root plus relative bundle path, unless a future mapping option overrides it. - -If nested manifests are found, the MVP should fail with a clear error unless a later policy defines nested-bundle semantics. - -## Destination State Contract - -Destinations are managed by `.distributor.json`, not by copying `manifest.json`. - -A destination bundle path is considered distributor-managed only when it contains a valid `.distributor.json` written by `distributor`. Force or unmanaged-overwrite behavior is not part of the MVP. - -### Required `.distributor.json` shape - -```json -{ - "schema_version": 1, - "distributor_version": "0.1.0", - "pipeline_id": "weather-daily", - "destination_id": "static-site", - "published_at": "2026-05-30T11:12:00Z", - "source": { - "manifest": { - "schema_version": 1, - "id": "weather.daily.brentwood.2026-05-30", - "digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000", - "created": "2026-05-30T11:10:00Z", - "files": [ - { - "path": "report.md", - "sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111", - "size": 12345 - } - ] - } - }, - "outputs": [ - { - "path": "report.html", - "kind": "generated", - "source_path": "report.md", - "transform": "markdown_to_html", - "sha256": "sha256:3333333333333333333333333333333333333333333333333333333333333333", - "size": 23456 - } - ] -} -``` - -Required fields: - -- `schema_version`: Destination state schema version. MVP value: `1`. -- `distributor_version`: Optional diagnostic distributor version. It must not affect source comparison. -- `pipeline_id`: Pipeline that produced the destination publication. -- `destination_id`: Destination within the pipeline. -- `published_at`: RFC3339 timestamp. -- `source.manifest`: Normalized source manifest model used for this publication. -- `outputs`: Array of files written by `distributor` for this destination. - -Required output fields: - -- `path`: Destination-relative output path within the destination bundle path. -- `kind`: `source` or `generated`. -- `source_path`: Source artifact path that produced this output. For copied source files, this should equal `path` unless renamed by a future feature. -- `transform`: Transform identifier for generated files. Empty or omitted may be allowed for copied source files. -- `sha256`: Output file digest as lowercase `sha256:<64 hex>`. -- `size`: Output file size in bytes. - -## Destination Comparison Rules - -Destination comparison uses `.distributor.json`, not destination `manifest.json`. - -For a source bundle and destination bundle path: - -### No `.distributor.json` - -If no `.distributor.json` exists and the destination path is empty: - -- Publish normally. - -If no `.distributor.json` exists and the destination path is non-empty: - -- Fail as unmanaged content. - -Local destination paths are empty when the destination bundle directory does not exist or exists with no entries. S3-compatible destination prefixes are empty when no objects exist below the destination bundle prefix, ignoring objects outside that exact prefix. - -### Pipeline or destination mismatch - -If `.distributor.json` exists but its `pipeline_id` or `destination_id` differs from the current pipeline or destination config: - -- Fail as a conflict. - -### Same source manifest - -If `.distributor.json` exists and `source.manifest` exactly matches the current normalized source manifest: - -- Skip as already published. - -### Same source id, destination older - -If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is older than the source `created`: - -- Replace destination contents, subject to replacement safety rules. - -### Same source id, destination newer - -If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is newer than the source `created`: - -- Skip and log that destination is newer than source. - -### Same source id and created, different digest - -If `.distributor.json` exists, `source.manifest.id` and `created` match but `digest` differs: - -- Fail as a conflict. - -### Different source id - -If `.distributor.json` exists and `source.manifest.id` differs from the source manifest `id`: - -- Fail as a conflict. - -## Replacement Safety Rules - -Replacement is destructive and must be narrow. - -`distributor` must never perform broad deletion against a configured destination root. - -Replacement may occur only at a resolved destination bundle path when: - -- a valid `.distributor.json` exists at that destination bundle path; and -- the state identifies the path as distributor-managed; and -- the replacement decision follows the destination comparison rules. - -For MVP, replacement should delete only known managed outputs where practical: - -- files listed in existing `.distributor.json.outputs`; -- existing `.distributor.json`; -- empty directories created by those files, where applicable for filesystem-like backends. - -For S3, replacement should delete only objects under the destination bundle prefix that are listed in `.distributor.json.outputs` plus `.distributor.json`, unless a later managed-prefix deletion policy is explicitly implemented. - -Before any write, planned destination output paths must be checked for collisions. For example, if `report.md` is copied as a source artifact and Markdown transformation would also generate `report.html`, but `report.html` is already a source artifact or another generated output, planning must fail before writing. - -## Transform Output Contract - -Source files are canonical. Generated files are derived publication artifacts. - -For MVP, the only supported transform is Markdown to HTML. - -Recommended MVP behavior: - -- Transform is configured per destination. -- Markdown source files are files listed in `manifest.json` with `.md` extension. -- Generated HTML files are sidecars by default. -- `report.md` generates `report.html`. -- Source bundles are never mutated. -- Generated outputs are recorded in `.distributor.json.outputs`. -- Raw HTML embedded in Markdown is escaped or disabled by default for the MVP to keep generated output deterministic and conservative. - -Future versions may add templates, `index.html`, CSS assets, email-safe HTML, and per-file transform selection. - -## Destination Output Contract - -Each destination chooses which categories of files it receives. - -MVP categories: - -- `source`: copied source artifacts listed in `manifest.json`. -- `html`: generated HTML derived from Markdown source artifacts. - -Examples: - -- Markdown archive: `source: true`, `html: false`. -- Static HTML site: `source: false`, `html: true`. -- Full mirror: `source: true`, `html: true`. - -Every file written to the destination must be represented in `.distributor.json.outputs`. - -## Atomicity and Partial Failure - -The MVP should prefer staging and promotion where backend semantics permit it. - -Minimum behavior: - -- Validate source before writing destination files. -- Do not write `.distributor.json` until all configured destination outputs are successfully written. -- If publication fails before `.distributor.json` is written, the destination must not be treated as successfully published on a later run. -- Local publication must use staging or equivalent cleanup behavior so a failed write does not leave a confusing unmanaged destination bundle path. -- Later cleanup may remove orphaned files, but MVP correctness should rely on `.distributor.json` as the success marker. - -## Example Source Bundle - -```text -weather/daily/brentwood/2026-05-30/ - manifest.json - report.md - summary.txt -``` - -Example manifest: - -```json -{ - "schema_version": 1, - "id": "weather.daily.brentwood.2026-05-30", - "digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000", - "created": "2026-05-30T11:10:00Z", - "files": [ - { - "path": "report.md", - "sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111", - "size": 12345 - }, - { - "path": "summary.txt", - "sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222", - "size": 234 - } - ] -} -``` - -## Example Destination Bundle: HTML Only - -```text -weather/daily/brentwood/2026-05-30/ - report.html - .distributor.json -``` - -## Example Destination Bundle: Source Archive - -```text -weather/daily/brentwood/2026-05-30/ - report.md - summary.txt - .distributor.json -``` - -## Implementation Stages - -1. Define Go structs for source manifest and destination state. -2. Implement source path validation. -3. Implement per-file SHA256 and size validation. -4. Implement canonical bundle digest validation. -5. Implement source bundle discovery beneath a source root. -6. Implement `.distributor.json` parsing and validation. -7. Implement destination comparison rules. -8. Implement replacement safety checks. -9. Add fixture bundles for valid, invalid, duplicate path, older, newer, same, and conflict scenarios. -10. Add reference canonical digest fixtures. -11. Use the contract layer from the publish pipeline and backend adapters. diff --git a/docs/roadmap/documentation.md b/docs/roadmap/documentation.md deleted file mode 100644 index d98cc81..0000000 --- a/docs/roadmap/documentation.md +++ /dev/null @@ -1,468 +0,0 @@ -# Documentation Roadmap - -## Purpose - -This roadmap defines a documentation-only refresh plan for `distributor` after the local MVP and cleanup roadmap implementation. - -The goal is to make current-behavior documentation concise, accurate, and compliant with `docs/policy/documentation.md` while keeping planned, aspirational, or unimplemented behavior under `docs/roadmap/`. - -This file is written for an LLM coding agent that will implement the documentation refresh in stages. It does not itself rewrite current user, policy, internal, or example documentation. - -## Repository Documentation Inventory - -The current documentation and examples reviewed are: - -| Path | Current status | Notes | -| --- | --- | --- | -| `README.md` | Keep and lightly update | Short, accurate orientation page with a runnable local example command. | -| `docs/cli.md` | Keep and update against CLI tests | Covers `version`, `run`, `validate`, and `inspect`; should be checked against command help and parser tests. | -| `docs/config.md` | Keep and tighten | Current config reference documents local execution and also accepted SSH/S3 config fields. It must clearly distinguish config validation from executable backend support. | -| `docs/operations.md` | Keep and expand slightly | Covers local workflow, destination state, retry behavior, cleanup, fan-out failure handling, and caveats. | -| `docs/troubleshooting.md` | Added | Covers recurring local MVP failure modes. | -| `docs/policy/architecture.md` | Keep and clarify where needed | Development policy is broad and includes future adapter direction. Wording should not imply SSH/S3 adapters currently exist. | -| `docs/policy/development.md` | Updated | Contains contributor and agent workflow guidance. | -| `docs/policy/documentation.md` | Keep | Canonical documentation policy. No change required unless the policy itself changes. | -| `docs/internal/bundle.md` | Keep and verify | Describes implemented bundle parsing, discovery, validation, and digest semantics. | -| `docs/internal/notify.md` | Keep and verify | Accurately states current no-op notification behavior. | -| `docs/internal/publish.md` | Keep and verify | Describes planning, execution, replacement, safety, and current local scope. | -| `docs/internal/state.md` | Keep and verify | Describes implemented `.distributor.json` state and comparison behavior. | -| `docs/internal/storage.md` | Keep and verify | Describes storage interface, typed errors, path rules, traversal, and managed deletion. | -| `docs/internal/transform.md` | Keep and verify | Describes Markdown-to-HTML sidecar behavior and transform boundaries. | -| `docs/internal/app.md` | Added | Covers orchestration, backend factory, transform registry, dry-run, summaries, and notifier handoff. | -| `docs/internal/config.md` | Added | Covers config loading, defaults, validation, accepted-but-not-executable backends, and example tests. | -| `docs/internal/local-backend.md` | Missing, optional | Local adapter behavior may remain in `docs/internal/storage.md`; create this only if local filesystem safety detail outgrows that doc. | -| `docs/integrations/markdown.md` | Added | Documents current Goldmark-backed Markdown rendering behavior. | -| `docs/roadmap/audit.md` | Historical roadmap/report | Keep under roadmap unless replaced by a new audit. | -| `docs/roadmap/cleanup.md` | Historical or completed roadmap | Keep under roadmap; optionally add completion status in the documentation refresh. | -| `docs/roadmap/config.md` | Roadmap | Keep as planning material; avoid linking to it as current config reference. | -| `docs/roadmap/contracts.md` | Roadmap | Keep as planning material; current implemented contracts should be summarized in `docs/internal/` and user docs as needed. | -| `docs/roadmap/implementation.md` | Roadmap | Keep as implementation history plus future stages; status should be clear. | -| `docs/roadmap/packages.md` | Roadmap | Keep as planning material; current package docs belong under `docs/internal/`. | -| `docs/roadmap/storage.md` | Roadmap | Keep as planning material; current storage contract belongs in `docs/internal/storage.md`. | -| `examples/local-to-local.yml` | Keep | Minimal local config; load-tested. | -| `examples/local-publish.yml` | Keep | Runnable local publication example used by README and CLI docs. | -| `examples/local-html.yml` | Keep | Runnable local HTML example. | -| `examples/fan-out.yml` | Updated | Local-only runnable fan-out example. | -| `examples/source-bundle/` | Keep | Copyable valid source bundle fixture for local CLI examples. | - -Implementation source areas inspected for documentation truth: - -- CLI entrypoints: `cmd/distributor`, `internal/cli`. -- Application orchestration: `internal/app`. -- Config loading/defaults/validation: `internal/config`. -- Bundle manifest, discovery, and validation: `internal/bundle`. -- Destination state: `internal/state`. -- Storage abstraction and local/fake backends: `internal/storage`, `internal/storage/fake`, `internal/adapters/local`. -- Publish planning and execution: `internal/publish`. -- Transform registry and Markdown renderer: `internal/transform`, `internal/transform/markdown`. -- Notification hook: `internal/notify`. -- Tests and fixtures: package tests, `internal/testutil`, `examples/`, and `internal/bundle/testdata`. - -Absent areas from earlier planning that should not be documented as implemented: - -- `internal/adapters/ssh` -- `internal/adapters/s3` -- `internal/stage` -- `internal/modules` -- `internal/validators` -- `internal/artifacts` -- `internal/manifest` -- `internal/schema` -- `internal/report` -- `pkg` - -## Policy Compliance Assessment - -Required current-behavior docs now exist for the config-driven, stateful, modular CLI. - -Completed fixes from this refresh: - -- `docs/policy/development.md` contains real workflow guidance. -- Non-roadmap docs are scoped to implemented behavior. SSH/S3 execution, force overwrite, and external notification adapters remain described as unavailable unless the corresponding code exists. -- `examples/fan-out.yml` is local-only and runnable. - -Completed recommended fixes from this refresh: - -- `docs/troubleshooting.md` covers recurring local MVP failure modes. -- `docs/internal/app.md` and `docs/internal/config.md` provide current-behavior internal references. -- `docs/integrations/markdown.md` documents current Goldmark-backed rendering behavior. -- Historical roadmap status notes were added where useful. - -No broad rewrite is needed for `README.md`, `docs/cli.md`, `docs/config.md`, or `docs/operations.md`. They are close to the implemented local MVP and should be tightened against code and tests. - -## Target Documentation Set - -### `README.md` - -- Audience: users, administrators, operators. -- Purpose: short project orientation and fastest useful local command. -- Canonical scope: project purpose, elevator pitch, one local quickstart command, links to current docs. -- Recommended outline: title, one-sentence description, local example command, links. -- Source-of-truth repo areas to inspect: `internal/cli`, `internal/app/run.go`, `examples/local-publish.yml`, `docs/cli.md`. -- Acceptance criteria: command is executable in the current local MVP; README does not describe SSH, S3, force overwrite, notification adapters, or future roadmap behavior as available. - -### `docs/cli.md` - -- Audience: users, administrators, operators. -- Purpose: canonical CLI reference. -- Canonical scope: commands, flags, useful workflows, command output expectations, local-only limits. -- Recommended outline: shortest useful command, command overview, flag reference, common workflows, diagnostics and recovery commands. -- Source-of-truth repo areas to inspect: `internal/cli/*.go`, `internal/cli/*_test.go`, `internal/app/validate.go`, `internal/app/inspect.go`, `internal/app/run.go`. -- Acceptance criteria: every documented command and flag exists; `validate` and `inspect` are documented as local path commands; `run --dry-run` output is described without over-specifying every line; unsupported remote execution is stated clearly. - -### `docs/config.md` - -- Audience: administrators, operators, advanced users. -- Purpose: canonical configuration reference. -- Canonical scope: config file path behavior, minimal local config, production-oriented local config, full schema, defaults, validation rules, secrets handling, links to examples. -- Recommended outline: config file location, minimal local config, production-oriented local config, reference, defaults, secrets, examples. -- Source-of-truth repo areas to inspect: `internal/config/config.go`, `internal/config/defaults.go`, `internal/config/load.go`, `internal/config/validate.go`, `internal/config/load_test.go`, `examples/*.yml`. -- Acceptance criteria: fields and defaults match code; `KnownFields(true)` behavior is noted where useful; SSH/S3 fields are described as accepted by config validation but not implemented for execution; `on_digest_mismatch: warn` and unmanaged overwrite are not documented as active options. - -### `docs/operations.md` - -- Audience: administrators, operators. -- Purpose: operating and recovery notes for the implemented local MVP. -- Canonical scope: local workflow, filesystem layout, destination state, dry-run, retry behavior, replacement safety, failed write cleanup, caveats. -- Recommended outline: normal workflow, filesystem layout, destination state, dry-run and planning, retry and replacement behavior, failure handling, cleanup behavior, caveats. -- Source-of-truth repo areas to inspect: `internal/app/run.go`, `internal/publish/plan.go`, `internal/publish/execute.go`, `internal/publish/reconcile.go`, `internal/publish/safety.go`, `internal/state`, `internal/adapters/local`. -- Acceptance criteria: describes only local-to-local operation; explains `.distributor.json` as the managed sentinel; distinguishes skip, replace, conflict, and unmanaged destination behavior; does not promise resume, remote storage, force overwrite, or external notifications. - -### `docs/troubleshooting.md` - -- Audience: administrators, operators. -- Purpose: symptom-oriented fixes for common local MVP failures. -- Canonical scope: implemented failure modes only. -- Recommended outline: one entry per symptom with symptom, likely cause, diagnostic step, safe fix, and relevant link. -- Source-of-truth repo areas to inspect: `internal/config/validate.go`, `internal/bundle/validate.go`, `internal/state/compare.go`, `internal/publish/plan.go`, `internal/publish/output.go`, CLI tests. -- Acceptance criteria: entries are actionable and do not suggest unsafe deletion; remote backend failures are described only as unsupported execution; all fixes link to `docs/cli.md`, `docs/config.md`, or `docs/operations.md` where useful. - -### `docs/policy/architecture.md` - -- Audience: developers, LLM coding agents. -- Purpose: development principles and architectural invariants. -- Canonical scope: project shape, package boundaries, state/persistence philosophy, external integration philosophy, errors/logging, tests, docs, non-goals. -- Recommended outline: keep the existing outline. -- Source-of-truth repo areas to inspect: full package tree, implemented internal docs, roadmap files for explicitly future work. -- Acceptance criteria: still gives long-term architecture direction, but any unimplemented adapter packages or future capabilities are worded as planned/target architecture rather than implemented behavior. - -### `docs/policy/development.md` - -- Audience: developers, LLM coding agents. -- Purpose: contributor and agent workflow. -- Canonical scope: repo layout, build/test commands, coding conventions, dependency policy, how to add config fields, CLI flags, backends, transforms, examples, and docs. -- Recommended outline: repository layout, common commands, coding conventions, dependency policy, adding config fields, adding CLI flags, adding storage backends, adding transforms, updating examples, documentation expectations. -- Source-of-truth repo areas to inspect: `go.mod`, `cmd/distributor`, `internal/*`, `examples`, tests, `docs/policy/architecture.md`, `docs/policy/documentation.md`. -- Acceptance criteria: no placeholder content remains; commands are real; workflow guidance protects current boundaries; examples and docs update rules match policy. - -### `docs/internal/app.md` - -- Audience: developers, LLM coding agents. -- Purpose: implemented orchestration reference. -- Canonical scope: `Run`, `Validate`, `Inspect`, backend factory, transform registry, dry-run, per-destination fan-out, failure aggregation, notifier invocation. -- Recommended outline: purpose, inputs and outputs, run flow, backend and transform registration, dry-run behavior, failure behavior, notification behavior, tests to inspect, invariants. -- Source-of-truth repo areas to inspect: `internal/app/*.go`, `internal/app/*_test.go`, `internal/cli/root_test.go`. -- Acceptance criteria: documents current local-only backend execution and the no-op default notifier; does not introduce a generic stage framework that does not exist. - -### `docs/internal/config.md` - -- Audience: developers, LLM coding agents. -- Purpose: internal config loading/default/validation reference. -- Canonical scope: YAML decoding, known-field rejection, defaults, validation error model, backend config shape, publish/transform validation helper, example load tests. -- Recommended outline: purpose, inputs and outputs, loading flow, defaults, validation responsibilities, executable support boundary, tests to inspect, invariants. -- Source-of-truth repo areas to inspect: `internal/config/*.go`, `internal/config/*_test.go`, `docs/config.md`, `examples/*.yml`. -- Acceptance criteria: documents that SSH/S3 config validation exists while execution does not; keeps user-facing config reference canonical in `docs/config.md`. - -### `docs/internal/bundle.md` - -- Audience: developers, LLM coding agents. -- Purpose: implemented bundle contract and validation reference. -- Canonical scope: `manifest.json`, discovery, validation, digest semantics, storage interactions, tests. -- Recommended outline: keep current outline and verify against code. -- Source-of-truth repo areas to inspect: `internal/bundle`, `internal/storage`, `internal/bundle/testdata`, `examples/source-bundle`. -- Acceptance criteria: canonical digest, duplicate paths, reserved paths, symlink rejection, RFC3339 parsing, and discovery behavior match implementation. - -### `docs/internal/state.md` - -- Audience: developers, LLM coding agents. -- Purpose: destination state and comparison reference. -- Canonical scope: `.distributor.json` schema, validation, output metadata, comparison outcomes. -- Recommended outline: keep current outline and verify against code. -- Source-of-truth repo areas to inspect: `internal/state`, `internal/publish/reconcile.go`, `internal/publish/execute.go`. -- Acceptance criteria: state schema and comparison outcomes match implemented structs and tests; `distributor_version` is described as optional diagnostic metadata. - -### `docs/internal/storage.md` - -- Audience: developers, LLM coding agents. -- Purpose: storage interface and backend safety reference. -- Canonical scope: logical paths, IO methods, traversal, `HasAny`, typed errors, managed deletion, local and fake backend behavior. -- Recommended outline: keep current outline and add any missing implemented details that matter for callers. -- Source-of-truth repo areas to inspect: `internal/storage`, `internal/storage/fake`, `internal/adapters/local`. -- Acceptance criteria: matches actual `Backend` interface, `WriteOptions`, `DeleteOptions`, `ErrStopWalk`, and `storage.List` helper; does not describe raw recursive delete as available. - -### `docs/internal/publish.md` - -- Audience: developers, LLM coding agents. -- Purpose: publish planning and execution reference. -- Canonical scope: request inputs, output planning, destination inspection, transfer policy, actions, replacement safety, cleanup on failed writes. -- Recommended outline: keep current outline and verify against code. -- Source-of-truth repo areas to inspect: `internal/publish`, `internal/app/run.go`, `internal/config/defaults.go`. -- Acceptance criteria: action names match constants; transfer policy values match code; collision detection and managed deletion behavior are covered. - -### `docs/internal/transform.md` - -- Audience: developers, LLM coding agents. -- Purpose: transform registry and Markdown transform reference. -- Canonical scope: transform interface, registry, Markdown sidecar output, deterministic output metadata, raw HTML behavior. -- Recommended outline: keep current outline; link to integration notes if `docs/integrations/markdown.md` is created. -- Source-of-truth repo areas to inspect: `internal/transform`, `internal/transform/markdown`, markdown tests. -- Acceptance criteria: `.md` to `.html` sidecar naming, skipped non-Markdown files, digest metadata, and source immutability match implementation. - -### `docs/internal/notify.md` - -- Audience: developers, LLM coding agents. -- Purpose: notification hook reference. -- Canonical scope: interface, no-op notifier, invocation points, non-invocation points. -- Recommended outline: keep current outline and add tests to inspect if useful. -- Source-of-truth repo areas to inspect: `internal/notify`, `internal/app/run.go`, `internal/app/run_test.go`. -- Acceptance criteria: says no external notification adapters or user-facing notification config exist. - -### `docs/integrations/markdown.md` - -- Audience: developers, LLM coding agents. -- Purpose: concise external integration note for Markdown rendering. -- Canonical scope: Goldmark dependency, renderer defaults used by `markdown.New`, raw HTML behavior as observed in tests, deterministic wrapper template, supported output mode. -- Recommended outline: purpose, dependency, behavior used, behavior intentionally not customized, tests to inspect, update rules. -- Source-of-truth repo areas to inspect: `go.mod`, `internal/transform/markdown`, markdown tests. -- Acceptance criteria: documents only the Markdown renderer behavior actually used; does not claim full CommonMark compatibility beyond Goldmark defaults. - -### `docs/roadmap/*.md` - -- Audience: maintainers, developers, LLM coding agents. -- Purpose: future work, historical plans, accepted deferred work, and implementation prompts. -- Canonical scope: unimplemented SSH/S3 adapters, force overwrite, notification adapters, future config fields, release readiness, historical cleanup/audit plans. -- Recommended outline: add status notes only where useful; avoid rewriting history unless it causes confusion. -- Source-of-truth repo areas to inspect: current implementation and each roadmap file. -- Acceptance criteria: future work remains under `docs/roadmap/`; completed historical plans are labeled clearly enough that agents do not re-run them blindly. - -## File-by-File Rewrite Guidance - -`README.md`: - -- Cover: purpose, one local quickstart, links. -- Avoid: full config schema, internal package details, remote backend promises. -- Link to: `docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/roadmap/`. -- Inspect: `examples/local-publish.yml`, `internal/cli`. -- Stale claims to remove: any implication that remote publication is implemented. - -`docs/cli.md`: - -- Cover: real command syntax and current local workflows. -- Avoid: roadmap flags such as force overwrite or remote validation. -- Link to: `docs/config.md`, `docs/operations.md`, `docs/troubleshooting.md` if created. -- Inspect: `internal/cli/*_test.go`. -- Stale claims to remove: any command or flag not present in `internal/cli`. - -`docs/config.md`: - -- Cover: current config schema and defaults. -- Avoid: presenting SSH/S3 as executable backend support. -- Link to: examples and operations. -- Inspect: `internal/config/defaults.go`, `internal/config/validate.go`, `internal/config/load_test.go`. -- Stale claims to remove: `on_digest_mismatch: warn`, unmanaged overwrite, or force replacement as active options. - -`docs/operations.md`: - -- Cover: local destination state, managed cleanup, retry behavior, fan-out failure aggregation. -- Avoid: remote storage recovery, force cleanup, external notifier delivery. -- Link to: `docs/troubleshooting.md` for symptom-specific fixes. -- Inspect: `internal/app/run.go`, `internal/publish/execute.go`, `internal/state`. -- Stale claims to remove: any resume or recovery mechanism beyond re-running after safe cleanup. - -`docs/policy/development.md`: - -- Cover: concrete contributor workflow. -- Avoid: placeholder text and invented tools. -- Link to: architecture and documentation policies. -- Inspect: package tree and `go.mod`. -- Stale claims to remove: `# Not yet implemented`. - -`docs/internal/*.md`: - -- Cover: implemented component contracts, boundaries, failure behavior, tests to inspect. -- Avoid: roadmap package names or future stages as if they exist. -- Link to: current user docs only when relevant. -- Inspect: package code and tests. -- Stale claims to remove: broad future backend behavior outside local/fake abstractions. - -`examples/`: - -- Cover: valid, maintained, copyable examples. -- Avoid: examples that look runnable but fail because the backend execution is unsupported. -- Link from: README, CLI docs, config docs. -- Inspect: `internal/config/load_test.go` and optional CLI smoke commands. -- Stale claims to remove: executable remote fan-out examples until remote backends exist. - -## Examples Plan - -Keep these examples: - -- `examples/source-bundle/`: valid source bundle used by local CLI examples. -- `examples/local-to-local.yml`: minimal local config. Use in config docs as the minimal schema example. -- `examples/local-publish.yml`: primary runnable quickstart config. -- `examples/local-html.yml`: runnable Markdown-to-HTML example. - -Revise `examples/fan-out.yml` in the documentation refresh: - -- Preferred option: replace it with a local-only fan-out example using two local destinations, such as one source archive destination and one HTML destination under `workspace/`. -- Alternative option: move the SSH/S3 fan-out material under a roadmap file and remove it from `examples/`. -- Do not keep a non-roadmap example that appears copyable for execution but uses unsupported SSH/S3 execution. - -Future example tests should continue loading every YAML file under `examples/`. If `examples/fan-out.yml` becomes local-only, add or update a CLI/app test that exercises local fan-out behavior or rely on existing fan-out tests if they cover equivalent behavior. - -## Internal Documentation Plan - -Update existing internal docs only after checking package code and tests. - -Create these internal docs: - -- `docs/internal/app.md`: orchestration, backend factory, transform registry, dry-run behavior, fan-out, failure aggregation, notifier invocation. -- `docs/internal/config.md`: YAML loading, defaults, validation, accepted config fields, unsupported execution boundary, example load tests. - -Keep and verify these internal docs: - -- `docs/internal/bundle.md` -- `docs/internal/state.md` -- `docs/internal/storage.md` -- `docs/internal/publish.md` -- `docs/internal/transform.md` -- `docs/internal/notify.md` - -Defer these internal docs unless the corresponding implementation grows: - -- `docs/internal/local-backend.md`: create only if local adapter details become too long for `docs/internal/storage.md`. -- `docs/internal/logging.md`: defer until logging has behavior beyond placeholders. -- `docs/internal/testutil.md`: defer unless test fixture helpers become a stable contributor-facing contract. - -Do not create docs for packages or directories that do not exist. - -## Integration Documentation Plan - -No external service integration docs should be created for SSH, S3, or notification services until those integrations are implemented. - -Recommended current integration doc: - -- `docs/integrations/markdown.md` - -This should document Goldmark usage because Markdown rendering is an implemented external file-format integration with externally visible output. Keep it concise and limited to: - -- dependency and version source: `go.mod`; -- renderer construction: `goldmark.New()`; -- sidecar output path behavior owned by `internal/transform/markdown`; -- raw HTML behavior covered by tests; -- wrapper template behavior; -- tests to inspect before changing renderer behavior. - -Do not create separate YAML integration docs unless configuration parsing behavior outgrows `docs/config.md` and `docs/internal/config.md`. - -## Recommended Implementation Sequence - -### Stage 1: Establish Documentation Status and Contributor Policy - -- Goal: remove the required policy placeholder and classify active versus historical documentation. -- Files to create/update/delete/move: update `docs/policy/development.md`; optionally add short status notes to `docs/roadmap/audit.md` and `docs/roadmap/cleanup.md`. -- Repo areas to inspect: `go.mod`, `cmd/distributor`, `internal/*`, `examples`, `docs/policy/architecture.md`, `docs/policy/documentation.md`. -- Acceptance criteria: `docs/policy/development.md` contains real workflow guidance; no active policy doc says "Not yet implemented"; roadmap status notes do not change current behavior docs. -- Suggested validation commands: `rg -n "Not yet implemented" docs/policy README.md docs/*.md docs/internal`; `git diff -- docs/policy/development.md docs/roadmap/audit.md docs/roadmap/cleanup.md`. -- One prompt: yes. - -### Stage 2: Tighten Current User and Operator Docs - -- Goal: make README, CLI, config, and operations docs exactly match the local MVP. -- Files to create/update/delete/move: update `README.md`, `docs/cli.md`, `docs/config.md`, `docs/operations.md`; create `docs/troubleshooting.md`. -- Repo areas to inspect: `internal/cli`, `internal/app`, `internal/config`, `internal/publish`, `internal/state`, `examples`. -- Acceptance criteria: user docs describe local execution, local validation/inspection, current config schema, current defaults, current state/retry behavior, and clear unsupported remote execution boundaries. -- Suggested validation commands: `rg -n "force|allow_unmanaged|on_digest_mismatch: warn|warn" README.md docs/cli.md docs/config.md docs/operations.md docs/troubleshooting.md`; `rg -n "ssh|s3|remote|notification" README.md docs/cli.md docs/config.md docs/operations.md docs/troubleshooting.md`. -- One prompt: yes. - -### Stage 3: Make Examples Fully Runnable or Clearly Roadmap-Only - -- Goal: ensure `examples/` contains implemented, copyable examples only. -- Files to create/update/delete/move: update or replace `examples/fan-out.yml`; update links in `README.md`, `docs/cli.md`, and `docs/config.md` if needed; move remote fan-out material to a roadmap section if preserving it is useful. -- Repo areas to inspect: `internal/config/load_test.go`, `internal/app/run_test.go`, `examples`. -- Acceptance criteria: every example under `examples/` is valid current config and does not rely on unsupported remote execution; primary examples remain load-tested. -- Suggested validation commands: `go test ./internal/config`; optional `go run ./cmd/distributor run --config examples/local-publish.yml --dry-run`; optional `go run ./cmd/distributor run --config examples/local-html.yml --dry-run`. -- One prompt: yes. - -### Stage 4: Complete Internal Component Docs - -- Goal: give future agents a current-behavior internal map before remote backends are added. -- Files to create/update/delete/move: create `docs/internal/app.md` and `docs/internal/config.md`; update existing `docs/internal/*.md` as needed. -- Repo areas to inspect: `internal/app`, `internal/config`, `internal/bundle`, `internal/state`, `internal/storage`, `internal/storage/fake`, `internal/adapters/local`, `internal/publish`, `internal/transform`, `internal/notify`, package tests. -- Acceptance criteria: every major implemented component has a concise doc with purpose, inputs/outputs, boundaries, failure behavior, tests to inspect, and invariants; no internal doc describes absent SSH/S3 adapters as implemented. -- Suggested validation commands: `rg -n "internal/adapters/ssh|internal/adapters/s3|not implemented|future" docs/internal`; `git diff -- docs/internal`. -- One prompt: yes. - -### Stage 5: Add Markdown Integration Notes - -- Goal: document the one implemented external file-format integration where behavior matters. -- Files to create/update/delete/move: create `docs/integrations/markdown.md`; optionally link from `docs/internal/transform.md`. -- Repo areas to inspect: `go.mod`, `internal/transform/markdown`, markdown tests. -- Acceptance criteria: the doc is concise, version-aware through `go.mod`, and limited to current Goldmark usage and observed renderer behavior. -- Suggested validation commands: `go test ./internal/transform/markdown`; `rg -n "Goldmark|markdown" docs/integrations docs/internal/transform.md`. -- One prompt: yes. - -### Stage 6: Final Documentation Consistency Sweep - -- Goal: catch stale links, stale roadmap references, and unimplemented claims outside roadmap. -- Files to create/update/delete/move: any docs touched in earlier stages. -- Repo areas to inspect: all docs and examples. -- Acceptance criteria: documentation is current, concise, link-consistent, and policy-compliant. -- Suggested validation commands: `go test ./...`; `rg -n "go-application-template|maximumdirect.net|docs/architecture.md|docs/documentation.md" README.md docs examples`; `rg -n "allow_unmanaged_overwrite|on_digest_mismatch: warn" README.md docs examples`; `rg -n "not implemented|future|planned|roadmap|ssh|s3|remote" README.md docs/*.md docs/internal docs/policy examples`. -- One prompt: yes, after the prior stages are complete. - -## Validation Plan - -Run Go tests when examples, CLI docs, config docs, or behavior-linked docs change: - -- `go test ./internal/config`: validates example config loading and config semantics. -- `go test ./internal/cli ./internal/app`: validates documented command behavior and local run workflows. -- `go test ./internal/transform/markdown`: validates documented Markdown rendering behavior. -- `go test ./...`: final broad verification after documentation and example changes. - -Run grep checks: - -- `rg -n "Not yet implemented" docs/policy README.md docs/*.md docs/internal` -- `rg -n "go-application-template|docs/architecture.md|docs/documentation.md" README.md docs examples` -- `rg -n "allow_unmanaged_overwrite|on_digest_mismatch: warn" README.md docs examples` -- `rg -n "force overwrite|--force|remote backends are implemented|notification adapters" README.md docs/*.md docs/internal docs/policy examples` -- `rg -n "ssh|s3|remote" README.md docs/*.md docs/internal docs/policy examples` - -The final SSH/S3/remote grep is not expected to return zero results. Manually review every result and confirm it is either: - -- under `docs/roadmap/`; -- a clearly stated unsupported-execution boundary; -- a config-validation reference that does not imply executable support; or -- an architecture policy statement phrased as future/target direction rather than implemented behavior. - -Manual review checklist: - -- README remains short. -- `docs/config.md` is the only current-behavior config reference. -- `docs/cli.md` is the only current-behavior CLI reference. -- `docs/operations.md` covers state and recovery without unsafe deletion advice. -- `docs/troubleshooting.md` is symptom-oriented and links to canonical docs. -- Internal docs point to tests before changing behavior. -- Examples are copyable and free of secrets. -- Future work remains under `docs/roadmap/`. - -## Open Questions - -No blocking questions remain before implementing this documentation refresh. - -The only non-blocking choice is how to handle `examples/fan-out.yml`: - -- Preferred: convert it to a local-only fan-out example so `examples/` remains fully runnable. -- Acceptable: move the current SSH/S3 fan-out example into roadmap material until remote execution exists. - -Use the preferred option unless a maintainer explicitly wants `examples/` to include config-validated but non-executable examples. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 049490a..6152a69 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,551 +1,75 @@ -# Distributor Implementation Roadmap +# Post-Local-MVP Implementation Roadmap -This roadmap defines a staged implementation plan for the `distributor` MVP. Each stage is intended to map cleanly to one Codex implementation prompt. +This is the canonical active roadmap for `distributor` after the local MVP checkpoint. -The roadmap assumes the project includes these planning documents before implementation begins: +The original MVP stages 1-8 are complete and are no longer listed as pending implementation work. Current behavior is documented outside the roadmap in `README.md`, `docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/troubleshooting.md`, `docs/internal/`, `docs/integrations/markdown.md`, and `docs/policy/development.md`. -- `docs/policy/architecture.md` -- `docs/policy/documentation.md` -- `docs/roadmap/packages.md` -- `docs/roadmap/contracts.md` -- `docs/roadmap/config.md` -- `docs/roadmap/storage.md` +Future, planned, or aspirational behavior should remain under `docs/roadmap/` until implemented. -The MVP goal is a domain-agnostic bundle distributor that discovers source bundles, validates `manifest.json`, optionally transforms Markdown to HTML, publishes selected outputs to one or more destinations, and records destination state in `.distributor.json`. +## Current Baseline -## Global Implementation Rules +The implemented local MVP includes: -All stages should preserve these invariants: +- standard-library CLI commands for `version`, `run`, `validate`, and `inspect`; +- YAML config loading, defaults, known-field rejection, and validation; +- accepted config shapes for `local`, `ssh`, and `s3`, with executable backend support currently limited to `local`; +- backend-rooted storage interface with typed errors, safe logical paths, traversal, `HasAny`, managed deletion, local backend, and fake backend; +- source bundle discovery, manifest parsing, RFC3339 timestamp handling, duplicate path checks, path safety checks, symlink rejection, per-file digest validation, and bundle digest validation; +- destination `.distributor.json` state parsing, validation, output metadata, and source comparison; +- local publication of source files, Markdown sidecar HTML, or both; +- destination output collision detection before writes; +- managed replacement for older destination state; +- unmanaged destination and conflict failures by default; +- deterministic dry-run output and final run summaries; +- deterministic sequential fan-out with aggregated failures; +- cleanup of outputs written during failed local publish attempts where practical; +- no-op notification hook after successful publish or replacement; +- current user, operator, internal, integration, and development documentation for implemented behavior. -- Producer applications own source bundle creation. -- `distributor` owns validation, transformation, publication, destination state, and future notification hooks. -- Source bundle state is defined by `manifest.json`. -- Destination publication state is defined by `.distributor.json`. -- `manifest.json` is not copied to the destination as destination state. -- Pipelines have exactly one source and one or more destinations. -- Transform and publish policy are destination-specific. -- Destructive replacement is allowed only inside a managed destination bundle path. Unsafe force or unmanaged overwrite behavior is deferred. -- Dry-run behavior should be implemented before broad remote write behavior. -- Config, bundle, state, publish planning, storage adapters, transforms, and CLI wiring should remain separate packages. +The local MVP intentionally does not include executable SSH/SFTP backends, executable S3-compatible backends, force overwrite behavior, external notification adapters, warning-only digest mismatch behavior, or broad recursive destination deletion. -Unless a stage explicitly says otherwise, each implementation prompt should: +## Active Roadmap Stages -1. read the project policy and roadmap documents; -2. implement only the current stage; -3. add or update tests for the current stage; -4. run the relevant test suite; -5. update documentation only when the implemented behavior now exists; -6. avoid implementing future roadmap stages early. +Implement each stage independently. Unless a stage explicitly says otherwise: -## Stage 1: Project Skeleton, CLI Shell, and Baseline Tooling +1. read `docs/policy/architecture.md`, `docs/policy/documentation.md`, `docs/policy/development.md`, and this roadmap before editing; +2. preserve current local MVP behavior; +3. keep user-facing docs limited to implemented behavior; +4. add or update focused tests for the behavior changed; +5. run the relevant package tests and `go test ./...` for cross-package changes; +6. avoid implementing later stages early. + +## Stage 1: SSH/SFTP Backend ### Goal -Create the initial Go application structure and a minimal executable `distributor` command with no business behavior beyond version/help output and placeholder commands. +Implement native SSH/SFTP storage backend support for sources and destinations through the existing storage interface and app-level backend factory. -### Scope +### Implementation Scope -Implement the accepted package skeleton from `docs/roadmap/packages.md` at the level needed for compilation. +Add an SSH/SFTP adapter package under `internal/adapters/ssh`. -Create: +The backend must implement the current `internal/storage.Backend` contract: -```text -cmd/distributor/main.go -internal/cli/ -internal/app/ -internal/config/ -internal/logging/ -``` +- `ReadFile` and `OpenReader`; +- `WriteFile` and `WriteFrom`; +- `Stat`; +- `Walk`; +- `HasAny`; +- `DeleteManagedBundle`. -Initial CLI commands: +Use native SFTP operations rather than shelling out to `ssh`, `scp`, or `rsync`. -- `distributor --help` -- `distributor version` -- `distributor run` -- `distributor validate` -- `distributor inspect` +Authentication behavior: -At this stage, `run`, `validate`, and `inspect` may return clear “not implemented” errors, but the command structure should be present. +- prefer SSH agent by default; +- use `known_hosts` validation by default where practical; +- support optional key-file configuration only if it can be added cleanly; +- do not support passwords in YAML in this stage. -### Notes +Config execution behavior: -Prefer a small CLI dependency only if the project already standardizes on one. Otherwise, the standard library is acceptable for the first pass. - -Add a version variable that can later be set at build time. - -### Tests - -Add tests for: - -- command construction if testable; -- version string behavior if exposed through a package; -- basic package compilation. - -### Completion Criteria - -- `go test ./...` passes. -- `go run ./cmd/distributor --help` works. -- `go run ./cmd/distributor version` works. -- Placeholder operational commands fail clearly and intentionally. - -## Stage 2: Config Schema, Loading, Defaults, and Validation - -### Goal - -Implement the MVP `config.yml` schema described in `docs/roadmap/config.md`. - -### Scope - -Create config structs for: - -- top-level config; -- pipelines; -- source backend config; -- destination backend config; -- validation policy; -- publish policy; -- transform policy; -- transfer/replacement policy; -- backend-specific local, SSH, and S3 fields. - -Support loading YAML from a file path. - -Implement validation for: - -- required top-level `pipelines`; -- unique pipeline ids; -- required pipeline `id`, `source`, and non-empty `destinations`; -- unique destination ids within a pipeline; -- supported backend names: `local`, `ssh`, `s3`; -- required backend fields; -- supported validation action: `fail`; -- supported transfer actions; -- valid `publish` policy; -- valid Markdown-to-HTML transform config. - -Default behavior should match `docs/roadmap/config.md`. - -### CLI Integration - -Add `--config` to `run`. - -For this stage, `distributor run --config config.yml --dry-run` may only load and validate config, then print a concise summary of configured pipelines and destinations. - -### Tests - -Add unit tests for: - -- valid minimal local-to-local config; -- valid fan-out config; -- valid local, SSH, and S3 backend configs; -- duplicate pipeline ids; -- duplicate destination ids; -- missing required fields; -- unsupported backend; -- invalid transfer action; -- invalid validation action, including `warn`. - -### Completion Criteria - -- Config load/default/validate behavior is implemented and tested. -- `distributor run --config --dry-run` validates config and prints a summary. -- No bundle discovery or publication occurs yet. - -## Stage 3: Storage Abstraction, Local Backend, and Fake Backend - -### Goal - -Introduce the storage backend abstraction before bundle validation so source discovery, validation, and publication are backend-agnostic from the start. - -### Scope - -Create: - -```text -internal/storage/backend.go -internal/storage/registry.go -internal/storage/path.go -internal/storage/errors.go -internal/adapters/local/backend.go -internal/storage/fake/ -``` - -Implement the storage contract in `docs/roadmap/storage.md`, including backend-rooted logical paths, hybrid byte/stream IO, metadata, traversal, typed errors, managed deletion, and efficient destination emptiness helper behavior. - -The fake backend should exist for unit tests of config, bundle, state, and publish logic without real local, SSH, or S3 IO. - -### Safety Requirements - -The local backend must: - -- clean and join paths safely; -- reject path traversal; -- reject unsafe destructive deletion requests; -- avoid following symlinks for source bundle files unless explicitly supported; -- avoid deleting configured roots; -- classify destination bundle emptiness deterministically. - -### Tests - -Add tests for: - -- path validation; -- backend byte and stream read/write behavior; -- `Stat`, `Walk`, and materialized deterministic traversal helper behavior; -- traversal rejection; -- symlink entry reporting and source-read rejection; -- staged write behavior where testable; -- typed storage errors and helper predicates; -- managed deletion guard behavior; -- early-stop destination emptiness helper behavior; -- fake backend parity for core package tests. - -### Completion Criteria - -- Local backend implements the storage interface. -- Fake backend can support bundle and publish tests without external services. -- `go test ./...` passes. -- No SSH or S3 implementation exists yet. - -## Stage 4: Source Bundle Manifest, Digest, Validation, and Discovery - -### Goal - -Implement the source bundle contract from `docs/roadmap/contracts.md` through the storage abstraction. - -### Scope - -Create: - -```text -internal/bundle/manifest.go -internal/bundle/digest.go -internal/bundle/validate.go -internal/bundle/discover.go -``` - -Implement: - -- parsing `manifest.json`; -- strict required field validation, including `schema_version: 1`; -- RFC3339 `created` parsing; -- lowercase `sha256:<64 hex>` digest validation; -- source file path safety checks; -- duplicate logical file path rejection; -- per-file SHA256 validation; -- per-file size validation; -- bundle digest validation using the canonical ordered file-record algorithm; -- deterministic storage-backed bundle discovery under a source root; -- nested manifest detection and failure. - -Discovery and validation should use `internal/storage` rather than direct `os` APIs. The local CLI path should be adapted to the local backend. - -### CLI Integration - -Implement: - -```text -distributor validate -distributor inspect -``` - -For local paths: - -- `validate` should validate either a single bundle directory or a tree containing bundles. -- `inspect` should print a concise normalized summary of discovered bundle ids, relative paths, created timestamps, digest values, and files. - -### Tests - -Add fixture bundles under a testdata directory. - -Test: - -- valid bundle; -- invalid JSON; -- missing required fields; -- invalid schema version; -- invalid timestamp; -- invalid digest format; -- unsafe file paths; -- duplicate normalized file paths; -- missing files; -- size mismatch; -- per-file digest mismatch; -- bundle digest mismatch; -- canonical bundle digest reference fixture; -- multiple discovered bundles in deterministic order; -- nested manifests fail. - -### Completion Criteria - -- Storage-backed bundle validation is deterministic and well-tested. -- `distributor validate ` works for local bundle fixtures. -- `distributor inspect ` works for local bundle fixtures. -- No destination publication occurs yet. - -## Stage 5: Destination State Contract and Comparison Logic - -### Goal - -Implement `.distributor.json` parsing, validation, and source-to-destination comparison. - -### Scope - -Create: - -```text -internal/state/distributor.go -internal/state/compare.go -internal/state/validate.go -``` - -Implement the destination state schema from `docs/roadmap/contracts.md`, including: - -- `schema_version`; -- optional `distributor_version`; -- `pipeline_id`; -- `destination_id`; -- `published_at`; -- embedded normalized source manifest; -- outputs array; -- output file metadata. - -Implement comparison outcomes: - -- destination absent; -- destination unmanaged/non-empty; -- destination state pipeline or destination id mismatch; -- same source manifest; -- same source id, destination older; -- same source id, destination newer; -- same source id and same created but different digest; -- different source id; -- invalid destination state. - -### Tests - -Add unit tests for every comparison outcome. - -Test validation for: - -- valid state; -- missing fields; -- invalid schema version; -- invalid embedded source manifest; -- invalid output metadata; -- malformed published timestamp. - -Timestamps should parse RFC3339 input and distributor-written timestamps should normalize to RFC3339 UTC. - -### Completion Criteria - -- Destination state can be parsed and validated independently. -- Source manifest to destination state comparison is deterministic and fully tested. -- No publication execution occurs yet. - -## Stage 6: Publish Planning, Dry-Run, and Local-to-Local Publication Without Transform - -### Goal - -Implement the core publish planner and execute local-to-local publication for source files only. - -### Scope - -Create: - -```text -internal/publish/plan.go -internal/publish/reconcile.go -internal/publish/safety.go -internal/publish/output.go -internal/publish/execute.go -``` - -Implement planning for one source bundle to one destination based on: - -- source manifest; -- destination config; -- publish policy; -- transfer policy; -- existing `.distributor.json`; -- destination path state. - -Actions should include: - -- publish new; -- replace older destination; -- skip same; -- skip destination newer; -- fail conflict; -- fail unmanaged destination. - -Implement local-to-local execution for `publish.source: true` and `publish.html: false`. - -Execution should: - -- copy listed source files selected by publish policy; -- write `.distributor.json` with copied source output metadata; -- avoid copying source `manifest.json` as destination state; -- preserve relative bundle paths from source root beneath destination root; -- detect destination output collisions before writing; -- use staging or equivalent cleanup behavior for local writes; -- support fan-out to multiple local destinations; -- support dry-run without writes. - -### CLI Integration - -`distributor run --config ` should now execute local-to-local pipelines when configured. - -`--dry-run` should print the planned action for each discovered bundle and destination. - -### Tests - -Add integration-style tests using temp directories for: - -- new local publication; -- no-op when destination state matches; -- replacement when destination state is older; -- skip when destination state is newer; -- fail on conflict; -- fail on unmanaged non-empty destination; -- fail on output path collision; -- fan-out from one source to two local destinations; -- failed local write does not leave a destination that appears unmanaged on retry; -- dry-run performs no writes; -- `.distributor.json` is written correctly. - -### Completion Criteria - -- Local-to-local source-file publication works end to end. -- Dry-run produces meaningful planned actions. -- Destination state is authoritative. -- No Markdown-to-HTML transform exists yet. - -## Stage 7: Markdown-to-HTML Transform and Destination-Specific Publish Policy - -### Goal - -Add MVP Markdown-to-HTML transformation and destination-specific source/html output selection. - -### Scope - -Create: - -```text -internal/transform/transform.go -internal/transform/registry.go -internal/transform/plan.go -internal/transform/markdown/markdown.go -internal/transform/markdown/template.go -``` - -Implement only: - -```yaml -transform: - markdown_to_html: - enabled: true - mode: sidecar -``` - -MVP sidecar behavior: - -- for each listed source artifact ending in `.md`, generate a same-directory `.html` sidecar; -- preserve the original Markdown file unchanged; -- do not generate HTML for non-Markdown files; -- escape or disable raw HTML embedded in Markdown; -- fail before writing when generated output paths collide with copied source outputs or other generated outputs; -- record generated output metadata in `.distributor.json`; -- if `publish.source: false`, do not publish source files; -- if `publish.html: true`, publish generated HTML files; -- if `publish.html: true` but transform is disabled or no Markdown files exist, fail with a clear error unless config later defines another behavior. - -Use a well-maintained Markdown renderer. Keep HTML templating minimal and deterministic. - -### Tests - -Add tests for: - -- Markdown sidecar generation; -- source-only destination; -- HTML-only destination; -- source-plus-HTML destination; -- no mutation of source bundle; -- generated output metadata in `.distributor.json`; -- failure when HTML publication is requested without transform support; -- failure when generated HTML collides with a source artifact path; -- raw HTML in Markdown is escaped or disabled consistently; -- deterministic output for a fixture Markdown file. - -### Completion Criteria - -- Local-to-local publication supports source-only, HTML-only, and source-plus-HTML destinations. -- Generated outputs are recorded in destination state. -- Dry-run reports transform outputs that would be generated. - -## Stage 8: No-Op Notification Stage, Pipeline Polish, and Local MVP Checkpoint - -### Goal - -Add the internal no-op notification stage and polish orchestration around per-destination outcomes. - -### Scope - -Create: - -```text -internal/notify/notify.go -internal/notify/noop.go -``` - -Integrate a no-op notifier after actual successful publication or replacement. Skipped destinations should not invoke the notifier in the MVP. - -Clarify orchestration behavior when one destination fails. For MVP, fan-out should be deterministic and sequential. Continue planning and reporting later destinations where safe, but return non-zero if any destination fails. - -Improve run summary output: - -- pipeline id; -- source backend; -- discovered bundle count; -- destination ids; -- action per bundle/destination; -- final status. - -### Tests - -Add tests for: - -- notifier is invoked at the expected orchestration point where testable; -- pipeline failure when a destination fails; -- run summary contains meaningful status information; -- dry-run does not invoke write-side effects. - -### Completion Criteria - -- The pipeline shape includes notification as an internal no-op stage. -- Run output is useful for unattended operation logs. -- Local MVP behavior remains passing and is ready for one real local producer pipeline. - -Stages 1 through 8 define the local MVP checkpoint. Later stages extend the local MVP with remote backends, cross-backend hardening, user-facing documentation sync, and release readiness. - -## Stage 9: Native SSH/SFTP Backend Roadmap Extension - -### Goal - -Implement SSH/SFTP storage backend support for sources and destinations. - -### Scope - -Create: - -```text -internal/adapters/ssh/backend.go -internal/adapters/ssh/config.go -``` - -Implement the storage backend interface over native SSH/SFTP. - -Required config: +- use the existing accepted config shape: ```yaml backend: ssh @@ -553,54 +77,80 @@ uri: ssh://user@example.com:22 path: /remote/root ``` -Authentication expectations: +- keep secrets out of config files; +- keep config loading and validation centralized in `internal/config`; +- wire runtime construction through app-level backend factory and storage registry patterns. -- prefer SSH agent by default; -- use known_hosts validation by default where practical; -- do not require passwords in YAML; -- optional key-file support may be implemented if straightforward, but should not distract from agent-based auth. +Supported pipeline combinations: -Support SSH/SFTP backend as both source and destination: +- local source to SSH destination; +- SSH source to local destination; +- SSH source to SSH destination where feasible through streaming or backend-owned staging. -- local -> ssh; -- ssh -> local; -- ssh -> ssh where feasible through staging or streaming. +Safety requirements: -### Safety Requirements +- enforce the same backend-rooted logical path rules as local storage; +- reject path traversal, absolute logical paths, dot segments, and backslashes; +- report or reject symlinks according to storage and bundle validation rules; +- keep deletion limited to managed output paths and `.distributor.json`; +- never delete a configured backend root; +- preserve conservative non-force conflict behavior. -The SSH backend must enforce the same logical path safety rules as the local backend. +### Documentation Updates -Deletion must remain limited to managed destination bundle paths guarded by valid `.distributor.json`. +After implementation, update only current-behavior docs: + +- `docs/config.md`: mark SSH as executable and document any implemented SSH-only fields. +- `docs/operations.md`: add SSH source/destination operating notes and recovery boundaries. +- `docs/troubleshooting.md`: add common SSH authentication, known-hosts, and remote path failures. +- `docs/internal/storage.md`: add implemented SSH adapter behavior and tests. +- `docs/policy/development.md`: update backend addition guidance if implementation changes the pattern. +- `examples/`: add only runnable or clearly environment-gated SSH examples. + +Do not document S3 or force overwrite as implemented in this stage. ### Tests -Unit-test path handling and config validation. +Add unit tests for: -If practical, add integration tests that can be skipped unless an SSH test endpoint is configured through environment variables. Do not require a live SSH server for normal `go test ./...`. +- SSH config execution wiring; +- URI and path handling; +- logical path validation; +- storage error translation where practical; +- `Walk` and `HasAny` behavior through test doubles or controlled fixtures; +- managed deletion boundaries; +- app-level local-to-SSH and SSH-to-local planning or wiring using fakes/mocks where possible. + +Add integration tests only if they are skipped unless explicit SSH test endpoint environment variables are configured. Normal `go test ./...` must not require a live SSH server. ### Completion Criteria -- SSH/SFTP backend compiles and satisfies the storage interface. -- Backend config validation is tested. -- Normal tests do not depend on a live SSH server. -- At least local-to-SSH and SSH-to-local flows are documented or manually testable. +- SSH/SFTP backend compiles and satisfies `storage.Backend`. +- Runtime `run` can execute supported SSH source and destination flows. +- Local MVP tests still pass. +- Normal test runs do not require a live SSH server. +- User docs accurately describe implemented SSH behavior and boundaries. -## Stage 10: S3-Compatible Backend Roadmap Extension +## Stage 2: S3-Compatible Backend ### Goal -Implement S3-compatible backend support for sources and destinations. +Implement S3-compatible object storage backend support for sources and destinations through the existing storage interface and app-level backend factory. -### Scope +### Implementation Scope -Create: +Add an S3-compatible adapter package under `internal/adapters/s3`. -```text -internal/adapters/s3/backend.go -internal/adapters/s3/config.go -``` +The backend must implement the current `internal/storage.Backend` contract: -Required config should align with `docs/roadmap/config.md`: +- `ReadFile` and `OpenReader`; +- `WriteFile` and `WriteFrom`; +- `Stat`; +- `Walk`; +- `HasAny`; +- `DeleteManagedBundle`. + +Use the existing accepted config shape: ```yaml backend: s3 @@ -614,243 +164,274 @@ credentials: secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY ``` -Implement storage operations over S3 object keys through the common storage contract: +Credential behavior: -- `ReadFile` and `OpenReader`; -- `WriteFile` and `WriteFrom`; -- `Stat`; -- `Walk` using object-list pagination where available; -- `HasAny` with early stop for destination emptiness checks; -- `DeleteManagedBundle`; -- read/write `.distributor.json` through normal storage operations. +- read access key and secret key from the named environment variables when configured; +- support standard SDK credential discovery only if it does not weaken explicit config behavior; +- do not put literal secrets in YAML. -Set reasonable content types where available: +Object semantics: + +- treat prefixes as object trees, not real directories; +- normalize configured prefix plus logical path with exact path-boundary matching; +- `Stat` must not synthesize directory metadata only because objects exist below a prefix; +- `Walk` should use object-list pagination and should not load an entire prefix into memory; +- `HasAny` should stop after the first matching object; +- `DeleteManagedBundle` must delete only listed managed output objects plus `.distributor.json`. + +Write behavior: + +- treat successful object PUT as publish-on-success; +- set content type from `storage.WriteOptions` where available; +- spool or buffer `WriteFrom` only when required by the SDK or backend; +- preserve overwrite checks and conservative conflict behavior. + +Content type behavior should cover at least: - `.md`: `text/markdown; charset=utf-8`; - `.html`: `text/html; charset=utf-8`; - `.json`: `application/json`; - `.txt`: `text/plain; charset=utf-8`. -Support S3 backend as both source and destination. +Supported pipeline combinations: -### Safety Requirements +- local source to S3 destination; +- S3 source to local destination; +- S3 source to S3 destination where feasible through streaming or backend-owned staging. -Treat S3 prefixes as object trees. Do not assume real directories exist. +### Documentation Updates -Deletion must be limited to destination bundle prefixes that are confirmed managed by `.distributor.json`. +After implementation, update only current-behavior docs: + +- `docs/config.md`: mark S3 as executable and document implemented credential behavior. +- `docs/operations.md`: add S3 source/destination layout, prefix, and recovery notes. +- `docs/troubleshooting.md`: add common S3 credential, bucket, endpoint, prefix, and permission failures. +- `docs/internal/storage.md`: add implemented S3 adapter behavior and tests. +- `docs/policy/development.md`: update backend addition guidance if implementation changes the pattern. +- `examples/`: add only safe S3 examples that use placeholder endpoints and environment variable names. + +Do not document force overwrite or notification adapters as implemented in this stage. ### Tests Add unit tests for: -- config validation; -- key/prefix normalization; -- content type selection; +- S3 config execution wiring; +- credential environment variable handling; +- key and prefix normalization; +- exact prefix boundary behavior; - path traversal rejection; +- content type selection; +- paginated `Walk` behavior through mocks/fakes; +- early-stop `HasAny`; +- managed deletion boundaries; - publish planning with S3 destination state fixtures. -If practical, add integration tests gated by environment variables or a local S3-compatible test service. Normal `go test ./...` must not require live S3 credentials. +Add integration tests only if they are skipped unless explicit S3-compatible endpoint credentials are configured. Normal `go test ./...` must not require live S3 credentials. ### Completion Criteria -- S3 backend compiles and satisfies the storage interface. -- S3 source and destination flows are supported through the common pipeline path. -- Normal tests do not require live S3. +- S3-compatible backend compiles and satisfies `storage.Backend`. +- Runtime `run` can execute supported S3 source and destination flows. +- Local and SSH behavior, if implemented, remain unchanged. +- Normal test runs do not require live S3. +- User docs accurately describe implemented S3 behavior and boundaries. -## Stage 11: Cross-Backend End-to-End Coverage and Hardening Roadmap Extension +## Stage 3: Cross-Backend Hardening and Documentation ### Goal -Harden the MVP across backend combinations, destination policies, and failure cases. +Harden behavior across implemented backend combinations, improve operator-facing failures, and synchronize current-behavior documentation and examples after remote backend support exists. -### Scope +### Implementation Scope -Add end-to-end coverage for representative scenarios: +Exercise and harden representative flows across all implemented backend types: -- local source -> local archive destination; -- local source -> local HTML destination; -- local source -> two destinations with different publish policies; -- local source -> SSH destination, where integration credentials exist; -- local source -> S3 destination, where integration credentials exist; -- S3 source -> local destination, where integration credentials exist; -- SSH source -> local destination, where integration credentials exist. +- local source to local archive destination; +- local source to local HTML destination; +- local source to multiple destinations with different publish policies; +- local source to SSH destination, when SSH is implemented and test credentials exist; +- SSH source to local destination, when SSH is implemented and test credentials exist; +- local source to S3 destination, when S3 is implemented and test credentials exist; +- S3 source to local destination, when S3 is implemented and test credentials exist. -Improve logging and error messages for: +Improve error context where practical for: - invalid config; - invalid source manifest; - digest mismatch; - destination conflict; - unmanaged destination path; -- backend read/write/list failures; -- transform failures. +- backend read, write, stat, walk, and delete failures; +- transform failures; +- partial fan-out failures. -Ensure all destructive paths have tests or explicit safeguards. +Ensure errors and logs identify pipeline id, destination id, bundle path or id, backend type, and logical path where useful without exposing secrets. + +Do not add force overwrite behavior in this stage. + +### Documentation Updates + +Update current-behavior docs after hardening: + +- `README.md`: keep the quickstart local unless remote examples become safe and concise. +- `docs/cli.md`: document any changed output or diagnostics. +- `docs/config.md`: ensure backend support status and config reference match implementation. +- `docs/operations.md`: document cross-backend state layout, retry behavior, and recovery caveats. +- `docs/troubleshooting.md`: add recurring SSH/S3 failure modes discovered during hardening. +- `docs/internal/`: update storage, publish, app, and config internals where behavior changed. +- `examples/`: keep examples copyable and free of secrets; remote examples must rely on placeholders and environment variables. ### Tests Add or expand tests for: -- dry-run across multiple destinations; +- dry-run across multiple destinations and backend types; - partial failure behavior; - repeated run idempotency; -- older/newer destination state behavior; +- older/newer destination state behavior across backends; - destination state output metadata accuracy; -- generated HTML output metadata accuracy. +- generated HTML output metadata accuracy; +- destructive replacement safety across implemented backends; +- error context for common failures. + +Integration tests for SSH or S3 must remain opt-in through environment variables. ### Completion Criteria -- MVP behavior is reliable across implemented backend types. -- Error messages identify pipeline id, destination id, bundle id, and reason where practical. -- Idempotent repeated runs behave as expected. - -## Stage 12: User-Facing Documentation Sync - -### Goal - -Update documentation to reflect implemented MVP behavior. - -### Scope - -Following `docs/policy/documentation.md`, create or update user-facing documentation only for implemented features. - -Likely docs: - -```text -README.md -docs/config.md -docs/cli.md -docs/policy/architecture.md -docs/internal/bundles.md -docs/internal/backends.md -``` - -Document: - -- what `distributor` does; -- bundle contract summary; -- `.distributor.json` role; -- example source bundle; -- example local-to-local config; -- example local-to-S3 config; -- example local-to-SSH config; -- `run`, `validate`, and `inspect` commands; -- dry-run behavior; -- replacement and safety rules; -- Markdown-to-HTML transform behavior; -- environment-variable credential handling. - -Move roadmap material to historical/planning status only if your documentation policy allows it. Do not describe unimplemented notification adapters as available features. - -### Tests - -Run the full test suite. - -If docs include command examples, verify that basic examples correspond to actual CLI behavior. - -### Completion Criteria - -- User-facing docs describe the implemented MVP accurately. -- Roadmap docs no longer masquerade as implemented behavior. -- `go test ./...` passes. - -## Stage 13: MVP Release Readiness Pass - -### Goal - -Perform a final pre-release quality pass. - -### Scope - -Review: - -- package boundaries against `docs/policy/architecture.md`; -- package layout against `docs/roadmap/packages.md`; -- implemented contracts against `docs/roadmap/contracts.md`; -- implemented config behavior against `docs/roadmap/config.md`; -- docs against `docs/policy/documentation.md`; -- destructive operation safety; -- logs and errors for unattended operation; -- command UX; -- test coverage for core invariants. - -Add any missing small tests or docs discovered during review. - -Do not add new product features in this stage. - -### Completion Criteria - -- MVP is ready to deploy against one real producer pipeline. -- A dry-run can be performed safely against a real source and destination. +- Implemented backend combinations behave consistently through the common pipeline path. - Repeated runs are idempotent. -- Destructive replacement cannot occur outside managed destination bundle paths. -- Final docs accurately reflect the application. +- Destructive paths remain bounded to managed destination bundle paths. +- Operator-facing errors are actionable. +- Current-behavior docs and examples match implemented backend support. -## Stage 14: Explicit Force Overwrite Roadmap Extension +## Stage 4: Explicit Force Overwrite ### Goal -Introduce explicit operator-requested force behavior for controlled overwrite cases that are intentionally outside the local MVP. +Introduce explicit operator-requested force behavior for controlled overwrite cases that remain intentionally unsupported by default. -### Scope +### Implementation Scope -Add a CLI-only force option such as: +Add a CLI-only force option: ```bash distributor run --config config.yml --force ``` +Force must be explicit per run. Do not add a persistent config default for force behavior. + Define and implement force planning for: - unmanaged non-empty destination paths; - destination state with a different source id; -- destination state with matching source id and created timestamp but different digest; +- destination state with matching source id and matching `created` timestamp but different digest; - destination state with mismatched `pipeline_id` or `destination_id`; -- destination newer than source when the transfer policy explicitly allows replacement. +- destination newer than source when transfer policy explicitly allows replacement. -Force behavior must be explicit per run. It should not be a persistent default in config for this stage. +Once force behavior exists, update transfer policy validation only for values supported by implemented force behavior: -Update transfer policy validation to allow broader values only when force behavior is implemented and documented: +- `on_destination_newer: replace`; +- `on_conflict: replace`. -- `on_destination_newer: replace` -- `on_conflict: replace` +Safety requirements: -### Safety Requirements +- non-force behavior remains unchanged and conservative; +- dry-run must show destructive force actions before any forced run; +- force must never delete above the resolved destination bundle path or configured destination prefix; +- local replacement should remain staged where practical; +- S3 replacement must remain constrained to the destination bundle prefix; +- managed state should still be written only after successful output writes; +- logs and output must clearly mark force decisions. -- Dry-run must show every file or object that would be written or deleted before a forced run. -- Force must still never delete above the resolved destination bundle path or configured destination prefix. -- Filesystem replacement should remain staged where practical. -- S3 replacement must remain constrained to the destination bundle prefix. -- Logs must clearly mark force decisions and include pipeline id, destination id, bundle id, and reason. +### Documentation Updates + +After implementation, update: + +- `docs/cli.md`: document `--force` syntax and dry-run workflow. +- `docs/config.md`: document newly accepted transfer policy values and note force is CLI-only. +- `docs/operations.md`: document safe force workflow and recovery boundaries. +- `docs/troubleshooting.md`: describe when force may be appropriate and when it remains unsafe. +- `docs/internal/publish.md` and `docs/internal/state.md`: document force planning and comparison handling. + +Do not document force as a default or config-only behavior. ### Tests Add tests for: -- force rejected when the flag is absent; +- force rejected or unavailable when the flag is absent; - unmanaged non-empty destination overwritten only with force; - different source id overwritten only with force and allowed policy; -- same id and created but different digest overwritten only with force and allowed policy; +- same id and created timestamp with different digest overwritten only with force and allowed policy; - destination newer replaced only with force and allowed policy; - pipeline or destination id mismatch overwritten only with force and allowed policy; - dry-run reports destructive force actions without writing; -- destructive paths remain bounded to the destination bundle path. +- force deletes only bounded destination bundle paths; +- local, SSH, and S3 backends, where implemented, preserve deletion boundaries. ### Completion Criteria - Force overwrite behavior is explicit, logged, dry-runnable, and test-covered. -- Default non-force behavior remains unchanged and conservative. +- Default non-force behavior remains unchanged. +- User docs clearly describe force risks and safe workflow. -## Deferred Post-MVP Work +## Stage 5: Release Readiness -The following items are intentionally outside the MVP unless explicitly pulled into a later roadmap: +### Goal -- email notifications; -- ntfy/Gotify/Pushover notifications; -- RSS/Atom feed generation; +Perform a final quality pass before treating `distributor` as ready for routine use against real producer pipelines and implemented destination backends. + +### Implementation Scope + +Review: + +- package boundaries against `docs/policy/architecture.md`; +- contributor workflow against `docs/policy/development.md`; +- user docs against `docs/policy/documentation.md`; +- CLI UX and command output; +- config validation and examples; +- manifest and state compatibility; +- destructive operation safety; +- backend error handling; +- logging and diagnostics for unattended operation; +- test coverage for core invariants. + +Do not add new product features in this stage. + +### Documentation Updates + +Update current-behavior docs only for issues found during the readiness review. + +If release packaging, version injection, or installation workflow is added, document it in the appropriate current-behavior user or development docs. + +### Tests + +Run: + +```bash +go test ./... +``` + +Also verify representative CLI examples that are documented as runnable. + +### Completion Criteria + +- A dry-run can be performed safely against real configured sources and destinations. +- Repeated runs are idempotent. +- Destructive replacement cannot occur outside managed destination bundle paths. +- Current docs accurately reflect the application. +- The project is ready to deploy against one real producer pipeline. + +## Deferred Work + +The following work remains intentionally deferred unless a future roadmap promotes it: + +- external notification adapters such as email, ntfy, Gotify, or Pushover; +- RSS or Atom feed generation; - static site index pages beyond sidecar HTML output; -- templated HTML themes beyond a minimal deterministic template; - destination path remapping rules; +- HTML themes beyond the minimal deterministic template; - full plugin architecture; - web UI; - report editing; @@ -860,4 +441,23 @@ The following items are intentionally outside the MVP unless explicitly pulled i - concurrent publication workers; - symlink support; - warning-only digest mismatch handling; -- password-based SSH authentication in YAML. +- password-based SSH authentication in YAML; +- broad recursive or prefix deletion outside explicitly bounded force behavior. + +## Validation + +For roadmap-only edits: + +```bash +git status --short +git diff -- docs/roadmap +rg -n "docs/roadmap/(packages|contracts|storage|config|documentation)\\.md" README.md docs examples +rg -n "docs/roadmap/(packages|contracts|storage|config|documentation)\\.md" . +rg -n "SSH|S3|--force|force overwrite|notification adapter|future|planned" README.md docs/*.md docs/internal docs/policy examples +``` + +Also search `docs/roadmap` for old MVP stage headings and titles from deleted roadmap files. That check should return no matches. + +The final SSH/S3/force/future-work search is not expected to return zero results. Review matches and confirm they are either under roadmap material or clearly marked as unsupported current behavior. + +Go tests are not required for documentation-only roadmap rationalization unless examples, behavior docs, or code change. diff --git a/docs/roadmap/packages.md b/docs/roadmap/packages.md deleted file mode 100644 index ac19b5b..0000000 --- a/docs/roadmap/packages.md +++ /dev/null @@ -1,526 +0,0 @@ -# Package Layout Roadmap - -This roadmap defines the proposed package layout, boundaries, and implementation responsibilities for the `distributor` MVP. - -`distributor` is expected to be a domain-agnostic bundle publisher. Producer applications emit source bundles containing `manifest.json`; `distributor` validates those bundles and publishes selected source and generated artifacts to one or more configured destinations. - -This document is roadmap material. It describes the intended package design before implementation and should move into `docs/internal/` only after corresponding behavior exists. - -## Accepted Package Layout - -```text -cmd/distributor/ - main.go - -internal/app/ - app.go - run.go - pipeline.go - -internal/cli/ - root.go - run.go - validate.go - inspect.go - -internal/config/ - config.go - defaults.go - load.go - validate.go - -internal/bundle/ - manifest.go - digest.go - validate.go - discover.go - -internal/state/ - distributor.go - compare.go - validate.go - -internal/storage/ - backend.go - registry.go - path.go - errors.go - -internal/storage/fake/ - backend.go - -internal/adapters/local/ - backend.go - -internal/adapters/ssh/ - backend.go - config.go - -internal/adapters/s3/ - backend.go - config.go - -internal/transform/ - transform.go - registry.go - plan.go - -internal/transform/markdown/ - markdown.go - template.go - -internal/publish/ - plan.go - execute.go - output.go - reconcile.go - safety.go - -internal/notify/ - notify.go - noop.go - -internal/logging/ - logging.go -``` - -## Package Responsibilities - -### `cmd/distributor` - -Application entrypoint only. - -Responsibilities: - -- call CLI execution; -- translate process exit status; -- avoid business logic. - -Non-responsibilities: - -- config loading; -- backend construction; -- bundle validation; -- publish decisions. - -### `internal/cli` - -CLI command definitions, flags, argument parsing, and command wiring. - -Expected MVP commands: - -- `distributor run` — run configured pipelines. -- `distributor run --dry-run` — plan without modifying destinations. -- `distributor run --pipeline ` — run one configured pipeline. -- `distributor validate ` — validate a source bundle or source tree where feasible. -- `distributor inspect ` — inspect a bundle or destination state where feasible. - -Boundaries: - -- CLI should call `internal/app` use cases. -- CLI should not parse manifests directly except through application APIs. -- CLI should not import backend adapter implementation details unless only for registration side effects. - -### `internal/app` - -Application orchestration and top-level use cases. - -Responsibilities: - -- load and validate configuration; -- construct configured pipelines; -- build source and destination backends through registries; -- orchestrate discovery, validation, planning, publishing, and notification; -- coordinate dry-run output; -- run destination fan-out deterministically and sequentially; -- aggregate destination outcomes into run-level failure behavior. - -Core orchestration shape: - -```text -for each selected pipeline: - open source backend - discover source bundles - for each source bundle: - validate source manifest and digest - for each destination: - inspect .distributor.json - build publish plan - transform as required by that destination - execute publish plan unless dry-run - run noop notifier after actual publication or replacement -``` - -Boundaries: - -- `internal/app` composes packages but should not contain backend-specific logic. -- Publish decisions should live in `internal/publish`, not inline in orchestration. -- Destination state comparison should live in `internal/state` or `internal/publish`, not CLI code. - -### `internal/config` - -Configuration structs, defaults, loading, precedence, and validation. - -Responsibilities: - -- load `/usr/local/etc/distributor/config.yml` by default; -- support `--config` override; -- apply defaults; -- validate required fields; -- validate pipeline ids and destination ids; -- validate backend-specific config shapes; -- validate transform and publish policy combinations. - -MVP config model: - -```yaml -pipelines: - - id: weather-daily - source: - backend: local - path: /var/spool/distributor/weather - validation: - on_digest_mismatch: fail - destinations: - - id: markdown-archive - backend: s3 - endpoint: https://s3.example.com - bucket: reports - prefix: weather/archive - region: us-east-1 - force_path_style: true - publish: - source: true - html: false - transfer: - on_destination_same: skip - on_destination_older: replace - on_destination_newer: skip - on_conflict: fail - - id: static-site - backend: ssh - uri: ssh://deploy@example.com:22 - path: /srv/www/weather - publish: - source: false - html: true - transform: - markdown_to_html: - enabled: true - mode: sidecar -``` - -Configuration principles: - -- one source per pipeline; -- one or more destinations per pipeline; -- transforms are destination-specific; -- publish policy is destination-specific; -- secrets should use environment variables, secret files, SSH agent, or standard credential mechanisms rather than raw YAML values. - -### `internal/bundle` - -Source bundle contract and validation. - -Responsibilities: - -- parse source `manifest.json`; -- represent source manifests and files; -- discover bundle roots beneath a configured source root; -- validate required manifest fields; -- validate RFC3339 `created` values; -- validate relative paths; -- validate file existence, size, per-file SHA-256, and bundle digest; -- expose normalized source bundle models to other packages. - -Core types: - -```go -type Manifest struct { - SchemaVersion int - ID string - Digest string - Created time.Time - Files []ManifestFile -} - -type ManifestFile struct { - Path string - SHA256 string - Size int64 -} - -type Bundle struct { - RootRelativePath string - Manifest Manifest -} -``` - -Boundaries: - -- `internal/bundle` does not know about `.distributor.json`. -- `internal/bundle` does not know about destinations, transforms, or notification. -- `internal/bundle` may use the storage abstraction to read source files, but it should not import backend adapter packages. - -### `internal/state` - -Destination state contract for `.distributor.json`. - -Responsibilities: - -- parse `.distributor.json`; -- validate destination state; -- represent copied source outputs and generated outputs; -- embed the source manifest used for publication; -- compare destination state against a current source manifest; -- classify destination state as same, older, newer, conflict, absent, invalid, or unmanaged. - -Core types: - -```go -type DistributorState struct { - SchemaVersion int - DistributorVersion string - PipelineID string - DestinationID string - PublishedAt time.Time - Source SourceState - Outputs []OutputFile -} - -type SourceState struct { - Manifest bundle.Manifest -} - -type OutputFile struct { - Path string - Kind string // source | generated - SourcePath string - Transform string - SHA256 string - Size int64 -} -``` - -Comparison rules: - -- same source manifest: skip; -- same source id, older destination source `created`: replace; -- same source id, newer destination source `created`: skip; -- same source id, same `created`, different digest: conflict; -- different source id: conflict; -- pipeline id or destination id mismatch: conflict; -- absent state: publish only if safe; -- unmanaged non-empty path: fail. - -Boundaries: - -- `internal/state` owns destination state semantics, not publish execution. -- `internal/state` should not know about S3, SSH/SFTP, local filesystem details, or Markdown rendering. - -### `internal/storage` - -Backend abstraction and shared storage types. - -Responsibilities: - -- define storage backend interfaces; -- define object/file metadata types; -- define path/prefix helpers; -- define common storage errors; -- provide backend registry mechanisms; -- provide a fake backend for core package tests. - -The detailed storage contract is defined in `docs/roadmap/storage.md`. Core application code should use that storage interface for backend-rooted logical paths, byte and stream IO, metadata, traversal, typed errors, emptiness checks, and managed deletion. - -Destructive APIs should remain narrow. Prefer managed deletion of files recorded in `.distributor.json` instead of broad recursive deletion. - -Boundaries: - -- `internal/storage` should not contain backend implementation details. -- Adapter dependencies must not leak through storage interfaces. -- The fake backend exists for tests and should not become an application runtime backend. - -### `internal/adapters/local` - -Local filesystem backend. - -Responsibilities: - -- implement `storage.Backend` for local paths; -- clean and constrain paths; -- perform safe reads/writes/listing/deletion; -- use atomic writes where practical; -- reject unsafe path traversal; -- handle symlink policy explicitly. - -Testing expectations: - -- use temporary directories; -- verify path traversal rejection; -- verify write and delete safety. - -### `internal/adapters/ssh` - -SSH/SFTP backend. - -Responsibilities: - -- implement `storage.Backend` over SSH/SFTP; -- support `uri` and `path` config; -- prefer native SFTP implementation; -- use SSH agent, key files, known hosts, or documented auth mechanisms; -- avoid raw passwords in config unless explicitly designed and documented later; -- translate SSH/SFTP errors into storage-level errors. - -Testing expectations: - -- core app tests should use fake backends; -- adapter tests may use local test servers or targeted integration tests if practical; -- do not require a real production SSH host for normal unit tests. - -### `internal/adapters/s3` - -S3-compatible object storage backend. - -Responsibilities: - -- implement `storage.Backend` over S3-compatible object storage; -- support endpoint, bucket, prefix, region, and force-path-style configuration; -- support standard credential mechanisms or explicit environment-variable references; -- treat S3 as an object tree, not a filesystem; -- set reasonable content types where practical; -- guard against prefix/root deletion mistakes. - -Testing expectations: - -- core app tests should use fake backends; -- adapter behavior may be tested through mocks, local S3-compatible services, or narrow integration tests; -- config examples should avoid real secrets. - -### `internal/transform` - -Transform interfaces, registry, and transform planning. - -Responsibilities: - -- define transform interfaces; -- register available transforms; -- represent transform requests and outputs; -- keep transform execution independent of destination backend details. - -Boundaries: - -- transforms operate on source bundle content and destination transform config; -- transforms do not publish outputs; -- transforms do not mutate source bundles; -- transforms should return generated output metadata for `.distributor.json`. - -### `internal/transform/markdown` - -Markdown-to-HTML implementation. - -Responsibilities: - -- render listed Markdown files to HTML; -- support MVP sidecar behavior, such as `report.md` -> `report.html`; -- record generated output path, source path, transform name, SHA-256, and size; -- optionally use embedded templates if needed. - -MVP scope: - -- Markdown to HTML only; -- no PDF generation; -- no email-specific HTML; -- no complex theming unless required for basic output correctness. - -### `internal/publish` - -Destination planning, reconciliation, safety checks, and publish execution. - -Responsibilities: - -- inspect destination state; -- plan destination action; -- enforce destination conflict rules; -- enforce destructive-operation safety rules; -- detect output path collisions before writing; -- combine source files and transform outputs according to destination publish policy; -- write destination outputs; -- write `.distributor.json`; -- use staging or equivalent cleanup behavior where practical; -- support dry-run planning; -- report skipped, replaced, failed, and published actions. - -Action model: - -```text -publish -replace -skip_same -skip_destination_newer -fail_conflict -fail_unmanaged -``` - -Boundaries: - -- publish logic should not parse CLI flags; -- publish logic should not know adapter implementation details; -- publish logic should use `internal/state` for destination state semantics; -- publish logic should use `internal/storage` interfaces for IO. - -### `internal/notify` - -Notification stage abstraction. - -MVP responsibilities: - -- define notifier interface; -- implement no-op notifier; -- preserve future extension point for email, ntfy, Gotify, RSS update hooks, or other notification channels. - -Future notification rules: - -- notify only after successful publication to the relevant destination or destinations; -- notification must be idempotent with respect to source id, digest, pipeline id, and destination id where applicable; -- notification should not run for skipped or failed publications unless explicitly configured. - -### `internal/logging` - -Logging setup and helpers. - -Responsibilities: - -- centralize structured logging setup; -- ensure logs omit secrets; -- provide consistent fields for pipeline id, bundle id, destination id, backend, path, action, and reason. - -## Deferred Ideas - -The following are intentionally out of MVP unless separately accepted in a later roadmap: - -- email, ntfy, Gotify, or other real notification adapters; -- RSS/Atom feed generation; -- PDF generation; -- web UI; -- full-text search; -- dynamic plugin loading; -- arbitrary transform chains; -- workflow DAGs; -- producer execution; -- complex templating/theming; -- bidirectional sync; -- backup semantics. - -## Key Invariants - -- Producer apps own source bundle creation. -- `distributor` owns destination publication state. -- Source `manifest.json` is not copied as destination state. -- Destination `.distributor.json` is the managed sentinel. -- One pipeline has one source and one or more destinations. -- Transform and publish policy are destination-specific. -- Source files are canonical; HTML is derived. -- Destructive replacement is allowed only inside managed destination bundle paths. -- Core logic must be testable without real S3, SSH, or remote services. diff --git a/docs/roadmap/storage.md b/docs/roadmap/storage.md deleted file mode 100644 index 600df9f..0000000 --- a/docs/roadmap/storage.md +++ /dev/null @@ -1,265 +0,0 @@ -# Storage Interface Roadmap - -This roadmap defines the planned `internal/storage` contract for the `distributor` MVP. The goal is to give bundle validation, destination state inspection, publish planning, and backend adapters one consistent IO boundary without leaking local filesystem, SSH/SFTP, or S3-specific behavior into core packages. - -## Purpose and Invariants - -The storage layer is responsible for safe, backend-rooted access to files, objects, prefixes, and destination bundle paths. - -Core invariants: - -- Backends are opened at configured roots. -- Core packages operate on backend-rooted logical paths, not absolute filesystem paths or raw object keys. -- Backend adapters translate native storage behavior into common storage entries and typed errors. -- Destructive operations remain narrow and managed. -- Staging or atomic write behavior belongs behind the storage interface where practical. -- The fake backend exists for tests only and must not be registered as a runtime backend. - -## Logical Path Model - -Storage paths are slash-separated logical paths relative to an already configured backend root. - -File paths: - -- must be non-empty; -- must be relative; -- must be clean; -- must not contain `.` or `..` segments; -- must not start with `/`; -- must not contain backslashes; -- must not resolve outside the backend root. - -Prefix paths use the same slash-separated model. A prefix may be empty to represent the backend root for traversal and destination emptiness checks. - -Prefix matching must preserve logical path boundaries. A prefix of `foo` matches `foo` and entries below `foo/`; it must not match a sibling path such as `foobar`. Backends that map logical paths to object keys must apply the same normalized boundary rule after combining configured backend prefixes with caller-provided logical prefixes. - -Backends own conversion from logical paths to native paths or object keys. Core packages should not construct local filesystem paths, SFTP paths, or S3 object keys directly. - -## Core Interface Shape - -The MVP should use a hybrid byte and stream interface: - -```go -type Backend interface { - ReadFile(ctx context.Context, path string) ([]byte, error) - OpenReader(ctx context.Context, path string) (io.ReadCloser, error) - WriteFile(ctx context.Context, path string, data []byte, opts WriteOptions) (Entry, error) - WriteFrom(ctx context.Context, path string, r io.Reader, opts WriteOptions) (Entry, error) - Stat(ctx context.Context, path string) (Entry, error) - Walk(ctx context.Context, prefix string, opts WalkOptions, fn WalkFunc) error - HasAny(ctx context.Context, prefix string) (bool, error) - DeleteManagedBundle(ctx context.Context, bundlePath string, managedOutputPaths []string, opts DeleteOptions) error -} -``` - -Byte helpers are expected to cover manifests, destination state, small source artifacts, and generated outputs. Stream methods are included from the start for backend flexibility and larger future artifacts. - -Write operations should create required parent directories or prefixes as needed. - -Concrete option and callback types should use this shape: - -```go -type WalkOptions struct { - Recursive bool - Limit int -} - -type WalkFunc func(Entry) error - -var ErrStopWalk = errors.New("stop walk") - -type WriteOptions struct { - ContentType string - Overwrite bool - PreferAtomic bool - Size int64 - SizeKnown bool -} - -type DeleteOptions struct { - IgnoreMissing bool - PruneEmptyDirs bool -} -``` - -`WalkOptions.Limit == 0` means no explicit limit. `SizeKnown` applies primarily to `WriteFrom`; byte writes can infer size from the provided data. - -## Entries and Metadata - -Storage metadata should be represented by an `Entry` model with at least: - -- backend-relative logical path; -- entry type; -- size, where available. - -Entry types: - -- `file`: filesystem file or object-storage object; -- `directory`: filesystem directory or logical prefix; -- `symlink`: local filesystem symlink; -- `other`: unknown or unsupported native entry type. - -`Stat` returns metadata for one exact logical path. It may report a real filesystem directory, symlink, file, or exact object. It must not synthesize S3-like directory metadata solely because objects exist below a prefix; callers that need prefix existence or destination emptiness must use `HasAny` or `Walk`. - -`Walk` traverses entries below a prefix and calls a callback for each entry. `WalkOptions` should include: - -- whether traversal is recursive; -- an optional entry limit for callers that only need to know whether content exists. - -If a callback returns `ErrStopWalk`, traversal stops successfully and `Walk` returns nil. Any other callback error stops traversal and is returned with storage context where practical. If `WalkOptions.Limit` is greater than zero, reaching the limit stops traversal successfully. - -Backends may stream or paginate traversal internally. S3-compatible adapters should not need to load a whole prefix into memory to satisfy traversal. - -Raw traversal is not required to be lexically sorted. A helper that materializes walk results for bundle discovery, tests, or CLI output should sort entries lexically by logical path before returning them. - -`HasAny` reports whether at least one entry exists below a prefix. It should stop as soon as content is found. - -Source validation must reject symlink entries reported by local `Stat` or `Walk`. - -## Read Behavior - -`ReadFile` reads the whole object into memory and is appropriate for MVP manifest, state, and ordinary artifact handling. - -`OpenReader` returns a stream for callers that need to copy or hash content without requiring a second storage-specific API. Callers must close the returned reader. - -Both read methods must: - -- validate logical paths before backend access; -- reject directories, prefixes, symlinks, and unsupported entries; -- return typed not-found and invalid-path errors where applicable. - -## Write Behavior - -`WriteOptions` should include: - -- content type, when the destination backend can use it; -- overwrite permission; -- atomic or staged write preference; -- optional known size for stream writes. - -Backends own staging and atomic behavior where practical: - -- Local backend writes to a temporary file in the destination directory and renames or promotes into place. -- SSH/SFTP backend should use a temporary remote file and rename where available. -- S3-compatible backend treats a successful object PUT as publish-on-success and applies content type metadata. - -Remote adapters may buffer or spool `WriteFrom` input when needed to satisfy backend requirements such as content length, multipart upload, or retry behavior. Callers that know the stream size should set `SizeKnown` and `Size`. - -If overwrite is false and the target exists, writes should fail with an already-exists error. - -`WriteFile` and `WriteFrom` should return the written `Entry`, including final path and size where available. - -`DeleteOptions` should include: - -- whether missing managed output paths are ignored; -- whether empty parent directories may be pruned for filesystem-like backends. - -## Managed Deletion - -The storage interface should expose a guarded managed deletion operation rather than raw recursive delete. - -`DeleteManagedBundle(ctx, bundlePath, managedOutputPaths, opts)` may delete only: - -- files or objects listed in valid `.distributor.json.outputs`; -- `.distributor.json` at the destination bundle path; -- empty directories created by those files, for filesystem-like backends. - -`managedOutputPaths` are relative to the destination bundle path. The backend validates each path and resolves it under `bundlePath`. - -If `bundlePath == ""`, deletion may remove explicit managed files at the destination root, but must never delete the root itself. - -Prefix or recursive deletion is out of MVP scope. A future force-overwrite stage may add broader behavior, but it must remain explicit and separately documented. - -## Destination Emptiness - -Destination emptiness should use `HasAny(prefix)` and typed not-found behavior. Callers that only need emptiness must not materialize a full recursive traversal. - -Rules: - -- A local destination bundle path is empty when the directory does not exist or exists with no entries. -- An S3-compatible prefix is empty when no objects exist below that exact destination bundle prefix. -- Entries outside the exact destination bundle path or prefix do not affect emptiness. - -## Error Model - -Storage should expose typed error categories with wrapping context. Callers should use helper predicates rather than string matching. - -Required categories: - -- not found; -- already exists; -- not empty; -- invalid path; -- conflict; -- permission; -- temporary; -- unsupported; -- unknown. - -Adapters should translate backend-native errors into these categories while preserving useful operation, backend, path, and cause context. - -## Adapter Expectations - -### Local - -The local backend should: - -- constrain all operations beneath the configured root; -- reject traversal and absolute logical paths; -- report symlinks through metadata; -- reject symlink reads for source artifacts; -- use staged writes where practical; -- perform managed deletion only for explicit managed files and `.distributor.json`; -- clean up empty directories created by managed outputs where safe. - -### Fake - -The fake backend should: - -- be in-memory and deterministic; -- implement the same logical path validation rules; -- support `Stat`, `Walk`, `HasAny`, byte reads and writes, stream reads and writes, managed deletion, and destination emptiness helper behavior; -- support configured symlink entries for validation tests; -- be used only by tests. - -### SSH/SFTP - -The SSH/SFTP backend should: - -- use native SFTP operations; -- enforce the same logical path rules as local storage; -- use temporary file plus rename for staged writes where available; -- translate remote errors into storage error categories; -- avoid exposing SSH or SFTP dependency types through `internal/storage`. - -### S3-Compatible - -The S3-compatible backend should: - -- treat prefixes as object trees, not real directories; -- normalize configured prefix plus logical path into object keys; -- use object PUT as publish-on-success; -- set content type from `WriteOptions`; -- implement traversal and emptiness by exact prefix; -- use backend pagination for traversal where available; -- allow `HasAny` to stop after the first matching object; -- constrain managed deletion to listed output objects and `.distributor.json`. - -## Tests and Fixtures - -Storage implementation stages should test: - -- path validation rejects absolute paths, traversal, empty file paths, backslashes, and dot segments; -- `Walk` visits backend-rooted logical paths under a prefix and supports recursive traversal; -- a materializing helper sorts walk results lexically for deterministic tests and CLI output; -- `HasAny` returns quickly for non-empty prefixes without requiring full traversal; -- `ReadFile` and `OpenReader` return equivalent bytes; -- `WriteFile` and `WriteFrom` honor overwrite and content-type options; -- local staged writes do not leave final files on failure where testable; -- managed deletion deletes only state-listed files and `.distributor.json`; -- managed deletion never deletes destination root or unlisted files; -- destination emptiness helper handles missing, empty, and non-empty local paths; -- S3-compatible traversal can use pagination without loading a whole prefix into memory; -- symlink entries are reported and rejected by source validation; -- typed errors are usable through helper predicates; -- fake backend behavior matches local backend semantics relevant to core tests.