eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Added initial MVP roadmap documentation and implementation plan

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-30 19:24:54 -05:00
parent 74709b2e0d
commit 47c9c5c0d1
7 changed files with 2658 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

270
docs/policy/architecture.md
View File

@@ -1,18 +1,184 @@
# Architecture
This document defines the development principles for this Go project. It is inward-facing: developers and LLM coding agents should use it to preserve the project’s shape, boundaries, and invariants as the code evolves.
This document defines the development principles for `distributor`. It is inward-facing: developers and LLM coding agents should use it to preserve the project’s shape, boundaries, and invariants as the code evolves.
## Project Scope
`distributor` is a domain-agnostic report bundle distribution tool.
Producer applications generate manifested bundles. `distributor` discovers those bundles, validates them, optionally derives publication artifacts such as HTML, and publishes selected source and generated artifacts to one or more configured destinations.
`distributor` does not generate domain reports, interpret domain-specific report content, run producer pipelines, edit reports, or act as a CMS. Weather reports, D&D recaps, calendar summaries, email digests, and future report types should all enter `distributor` through the same bundle contract.
## Project Shape
Default to a small, explicit, dependency-light Go application. Keep the design modular enough to test and change safely, but do not add abstraction unless it protects a real boundary or enables a real extension point.
Business/domain logic should live outside CLI, transport, and external-adapter packages.
Business logic should live outside CLI, transport, and external-adapter packages. The core application should reason in terms of pipelines, bundles, destination state, transforms, and publish plans—not S3 SDK calls, SFTP sessions, shell commands, or filesystem details.
The core workflow is:
1. load configured pipelines;
2. open the source backend;
3. discover source bundles beneath the source root;
4. validate each source bundle and its `manifest.json`;
5. for each configured destination, inspect destination state;
6. compare source state to destination state;
7. build a publish plan;
8. optionally transform Markdown to HTML for that destination;
9. publish selected source and generated artifacts;
10. write `.distributor.json` as the destination sentinel/state file;
11. run the notification stage, which is a no-op in the MVP.
## Pipeline Model
A pipeline has exactly one source and one or more destinations.
The source is discovered and validated once. Each destination has independent backend configuration, publication policy, transform policy, replacement behavior, state, and future notification behavior.
The pipeline model is fan-out by design:
```text
source bundle
-> destination A: source files only
-> destination B: HTML only
-> destination C: source files + HTML
```
Destination-specific behavior must not leak back into the source bundle contract. A producer should not need to know whether a bundle will be published to local storage, SSH/SFTP, S3, a static site, email, RSS, or a future notification channel.
## Source Bundle Contract
A source bundle is a directory containing `manifest.json`.
`manifest.json` is the sole producer-to-`distributor` contract. `distributor` must not rely on producer-specific work directory layouts, filenames, metadata, or conventions outside the configured source root and the source manifest.
The MVP source manifest schema is intentionally minimal:
```json
{
"schema_version": 1,
"id": "weather.daily.brentwood.2026-05-30",
"digest": "sha256:...",
"created": "2026-05-30T11:10:00Z",
"files": [
{
"path": "report.md",
"sha256": "sha256:...",
"size": 12345
}
]
}
```
Required fields:
- `schema_version`: source manifest schema version. MVP value: `1`.
- `id`: stable bundle identifier.
- `digest`: SHA-256 digest for the listed files.
- `created`: RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
- `files`: ordered file list.
- `files[].path`: relative path beneath the bundle root.
- `files[].sha256`: SHA-256 digest of the file contents.
- `files[].size`: file size in bytes.
Bundle paths must be relative, clean, and confined to the bundle root. Paths must not be absolute, empty, contain `..` path traversal, or otherwise escape the bundle root. Symlink handling must be explicit; unless documented otherwise, source bundle symlinks should be rejected.
Before processing a bundle, `distributor` must validate file existence, file size, each file SHA-256, and the bundle digest. Digest mismatch must fail in the MVP before any destination writes occur.
The source manifest should remain minimal. Routing, destination selection, publication format, credentials, static-site layout, notification recipients, and destination-specific metadata belong in `distributor` configuration and destination state, not in producer manifests.
## Destination State Contract
`manifest.json` from the source bundle is not copied to destinations as destination state.
Each destination bundle path is managed by `.distributor.json`. This file is both the destination sentinel and the destination state record. It records:
- `distributor` state schema version;
- pipeline id;
- destination id;
- publication timestamp;
- the normalized source manifest used for publication;
- metadata for copied source outputs;
- metadata for generated outputs, such as HTML files;
- any additional metadata required by `distributor`.
A representative destination state file is:
```json
{
"schema_version": 1,
"distributor_version": "0.1.0",
"pipeline_id": "weather-daily",
"destination_id": "static-html",
"published_at": "2026-05-30T11:12:00Z",
"source": {
"manifest": {
"schema_version": 1,
"id": "weather.daily.brentwood.2026-05-30",
"digest": "sha256:...",
"created": "2026-05-30T11:10:00Z",
"files": [
{
"path": "report.md",
"sha256": "sha256:...",
"size": 12345
}
]
}
},
"outputs": [
{
"path": "report.html",
"kind": "generated",
"source_path": "report.md",
"transform": "markdown_to_html",
"sha256": "sha256:...",
"size": 23456
}
]
}
```
Destination comparison rules are based on `.distributor.json`:
- No `.distributor.json`: publish normally only if the destination bundle path is empty.
- Existing state embeds the same normalized source manifest: skip as already published.
- Existing state has the same source id and an older source `created`: replace, subject to destructive-operation safety rules.
- Existing state has the same source id and a newer source `created`: skip because the destination is newer than the source.
- Existing state has the same source id and same `created` but different digest: fail as a conflict.
- Existing state has a different source id: fail as a conflict.
## Publication and Transform Policy
Source files are canonical bundle artifacts. Transform outputs are derived publication artifacts.
Transforms are configured per destination. A destination may receive source files, generated HTML files, or both.
The MVP supports only Markdown-to-HTML transformation. HTML generation must not mutate the source bundle. Generated outputs must be deterministic from the source bundle and destination transform configuration, and must be recorded in `.distributor.json`.
The application should distinguish:
- transform policy: how derived files are generated;
- publish policy: which source and generated files a destination receives.
For example, one destination may publish source files only as a long-term archive, while another publishes HTML only as a static site.
## Backend Abstraction
Sources and destinations use the same storage abstraction. Local filesystem, SSH/SFTP, and S3-compatible object storage are peer backends. Any backend may appear as a source or a destination unless a specific limitation is documented.
Application logic must interact with storage through internal backend interfaces. Backend-specific behavior belongs in adapter packages. Pipeline, bundle, state, publish, and transform packages must not import S3, SSH/SFTP, or filesystem adapter implementation details.
Adapters should be thin. Backend adapters should implement storage operations and translate backend-specific errors, but should not make bundle comparison, transform, routing, or replacement decisions.
SSH support should prefer a native SFTP implementation over shelling out to `ssh`, `scp`, or `rsync`, unless a later design document records a reason to differ.
## Dependency Policy
Prefer the Go standard library where practical.
Use external dependencies only when justified by correctness, security, interoperability, or substantial complexity reduction. Good reasons include complex security-sensitive behavior, such as HTML sanitization, or widely used de facto standards, such as YAML parsing.
Use external dependencies only when justified by correctness, security, interoperability, or substantial complexity reduction. Good reasons include YAML parsing, S3-compatible storage integration, SSH/SFTP integration, and Markdown rendering.
Avoid dependencies for small conveniences. Do not let external dependency types leak across internal package boundaries unless the dependency is itself the explicit public contract of that package.
@@ -20,12 +186,21 @@ Avoid dependencies for small conveniences. Do not let external dependency types
Use this layout unless the project has a documented reason to differ:
- `cmd/distributor`: application entrypoint only.
- `internal/app`: application orchestration and top-level use cases.
- `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
- `internal/config`: configuration structs, defaults, loading, precedence, and validation.
- `internal/adapters/<name>`: adapters for external CLIs, APIs, databases, object stores, or libraries.
- `internal/api`: HTTP API handlers and request/response types, when the application exposes an HTTP API.
- `internal/transport/http`: HTTP client code, when the application calls HTTP services.
- `internal/bundle`: source manifest parsing, source bundle discovery, source digest validation, and source bundle model.
- `internal/state`: `.distributor.json` parsing, validation, comparison, and output metadata.
- `internal/storage`: backend interfaces, shared path/resource types, backend registry, and storage errors.
- `internal/adapters/local`: local filesystem backend.
- `internal/adapters/ssh`: SSH/SFTP backend.
- `internal/adapters/s3`: S3-compatible object storage backend.
- `internal/transform`: transform interfaces, registry, planning, and shared transform models.
- `internal/transform/markdown`: Markdown-to-HTML implementation.
- `internal/publish`: destination planning, reconciliation, safety checks, and publish execution.
- `internal/notify`: notification interface and MVP no-op notifier.
- `internal/logging`: logging setup and shared logging helpers.
Package-private implementation constants may live near the package that owns them, preferably in `constants.go` when useful.
@@ -42,41 +217,54 @@ Unless documented otherwise, precedence is:
3. configuration file
4. built-in defaults
Prefer YAML configuration unless the project has a strong reason to use another format. Config files should be discovered at `/usr/local/etc/<app_name>/config.yml`, with a CLI override via `--config`.
Prefer YAML configuration. Config files should be discovered at `/usr/local/etc/distributor/config.yml`, with a CLI override via `--config`.
Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets.
Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables, secret files, SSH agent usage, standard AWS credential mechanisms, or explicitly named environment variable references for secrets.
## Adapters and External Integrations
Pipeline configuration should express:
Use a hexagonal architecture style for external integrations.
External adapters belong under `internal/adapters/<name>`. If an adapter uses an external dependency, that dependency’s interface must not leak outside the adapter package. Other packages should interact only with the adapter’s API, so the dependency can be swapped, upgraded, or removed without touching unrelated code.
Adapters should be thin. Domain decisions belong in application/domain packages, not inside adapter glue.
- pipeline id;
- one source backend;
- one or more destinations;
- per-destination publish policy;
- per-destination transform policy;
- validation behavior;
- destination conflict/replacement behavior.
## Modules, Stages, and Registries
When the application has stages or modules, each major stage/module should live in its own package and have an explicit input/output contract.
Each major stage should have an explicit input/output contract:
The orchestrator should be able to compose, skip, resume, or run individual stages/modules when their prerequisites are satisfied. Ordering should be explicit: use a default sequence, dependency graph, or documented orchestration rule.
- source discovery;
- source validation;
- destination state inspection;
- destination comparison;
- transform planning/execution;
- publish planning;
- publish execution;
- notification.
If users can select modules, stages, validators, renderers, or adapters, selection should go through a registry or equivalent mechanism rather than scattered conditionals.
If users can select backends, transforms, notifiers, or future renderers, selection should go through a registry or equivalent mechanism rather than scattered conditionals.
The orchestrator should be able to plan, dry-run, and execute configured pipelines. Dry-run behavior should be first-class because the application may delete, overwrite, or publish files to remote storage.
## Embedded Assets
Store embedded JSON schemas, Markdown prompts, templates, and similar assets as separate files, not inline string literals, unless there is a strong reason otherwise.
Store embedded JSON schemas, Markdown templates, HTML templates, CSS, and similar assets as separate files, not inline string literals, unless there is a strong reason otherwise.
## Errors and Logging
Errors should be actionable and preserve context. Wrap errors with operation and path/resource context. CLI code should convert internal errors into concise user-facing messages.
Errors should be actionable and preserve context. Wrap errors with operation, pipeline id, destination id, backend, path, bundle id, and resource context where useful. CLI code should convert internal errors into concise user-facing messages.
Errors and logs must not expose secrets.
Use structured logging where practical. Logs should describe operations, paths, external calls, retries, and failure causes, but should not include large user data by default.
Use structured logging where practical. Logs should describe discovery, validation, planned actions, skipped transfers, conflicts, replacements, external calls, retries, and failure causes, but should not include large report contents by default.
Skip and no-op decisions should be logged at an appropriate level so operators can distinguish successful publication from intentional no-op behavior.
## Context, Timeouts, and Cancellation
Long-running operations should accept `context.Context`. External calls, subprocesses, HTTP requests, storage operations, and multi-stage workflows should respect cancellation and timeouts.
Long-running operations should accept `context.Context`. Storage operations, SSH/SFTP sessions, S3 requests, transforms, and multi-stage workflows should respect cancellation and timeouts.
## State, Files, and Safety
@@ -84,15 +272,53 @@ If the application writes durable state, writes should be atomic where practical
Code that deletes, moves, or overwrites files must use narrow, explicit paths. Avoid broad parent-directory operations. Cleanup that can cause data loss must be opt-in.
`distributor` must never perform broad deletion against a configured source root or destination root. Destructive replacement may occur only inside a resolved destination bundle path when a valid `.distributor.json` confirms that the path is distributor-managed.
Replacement must be narrow, logged, test-covered, and configurable. Prefer deleting files recorded in `.distributor.json` and known generated outputs rather than blindly deleting parent directories. Backend implementations must guard against path traversal, prefix confusion, and accidental root deletion.
Where practical, publish operations should use staging paths or temporary objects and promote them into place only after validation and transform steps succeed.
## Testing
Core logic should be testable without real external services. Use fakes, fixtures, or local test doubles for adapters where practical.
Config examples should be load-tested. Important CLI workflows should have parser or command tests. Stage/module contracts should have focused tests that do not require running the full application unless end-to-end coverage is intentional.
Important tests include:
- source manifest parsing and validation;
- file size and SHA-256 validation;
- bundle digest validation;
- source bundle discovery beneath a source root;
- relative path safety and path traversal rejection;
- destination `.distributor.json` parsing and comparison;
- same/older/newer/conflict publish decisions;
- destructive replacement safety checks;
- transform output planning and metadata recording;
- dry-run output;
- local backend behavior with temporary directories;
- fake backend behavior for S3 and SSH/SFTP-facing core logic.
## Documentation
Documentation should follow the project documentation policy. Keep user docs focused on implemented behavior. Put future, planned, or aspirational work only under `docs/roadmap/`.
When changing architecture, config, CLI behavior, adapters, or stage/module contracts, update the relevant docs and examples in the same change.
When changing architecture, config, CLI behavior, adapters, manifest/state contracts, transform behavior, publish behavior, or stage/module contracts, update the relevant docs and examples in the same change.
The source manifest and destination `.distributor.json` schemas should have canonical documentation once implemented. Example configs should be valid and load-tested where practical.
## Non-Goals
`distributor` is not:
- a report generator;
- a domain-specific weather, D&D, calendar, or email summarization app;
- a workflow DAG engine;
- a CMS;
- a web authoring interface;
- a full-text search service;
- a general-purpose file synchronization tool;
- a backup system;
- a notification platform.
It may later support notification adapters, RSS/feed generation, richer HTML templates, or additional transforms, but those features must preserve the core bundle-distribution boundary.

1
docs/policy/development.md Normal file
View File

@@ -0,0 +1 @@
# Not yet implemented

578
docs/roadmap/config.md Normal file
View File

@@ -0,0 +1,578 @@
# Distributor Configuration Roadmap
This roadmap defines the planned `config.yml` schema for the `distributor` MVP. The goal is to support one-to-many publication pipelines where each pipeline has one source and one or more destinations. Each destination independently controls backend configuration, publication outputs, transform behavior, and replacement policy.
## Configuration Goals
The MVP configuration should be:
- explicit enough to avoid hidden publication behavior;
- compact enough for routine self-hosted use;
- backend-agnostic at the pipeline layer;
- capable of local, SSH/SFTP, and S3-compatible source and destination backends;
- ready for future notification adapters without exposing a fake notification feature in the MVP.
## Top-Level Shape
```yaml
pipelines:
- id: weather-daily
source:
backend: local
path: /var/spool/distributor/weather
validation:
on_digest_mismatch: fail
destinations:
- id: markdown-archive
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: weather/archive
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
publish:
source: true
html: false
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
- id: static-site
backend: ssh
uri: ssh://deploy@example.com:22
path: /srv/www/weather
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
```
## Pipeline Fields
Each pipeline must include:
- `id`: Required stable pipeline identifier.
- `source`: Required source backend configuration.
- `destinations`: Required non-empty list of destination configurations.
Optional pipeline-level fields:
- `validation`: Source validation behavior.
- Future: `notifications` or `notify`, when notification adapters are implemented.
A pipeline has exactly one source and one or more destinations.
## Pipeline ID Rules
`pipelines[].id` should:
- be required;
- be unique across the config file;
- be stable over time;
- use a simple slug-like format, such as `weather-daily` or `dnd-session-recaps`.
Recommended validation:
```text
^[a-zA-Z0-9][a-zA-Z0-9._-]*$
```
## Source Configuration
`source` defines the source root where bundles are discovered.
The source backend may be:
- `local`;
- `ssh`;
- `s3`.
The source is scanned for `manifest.json` files beneath the configured root.
### Local source
```yaml
source:
backend: local
path: /var/spool/distributor/weather
```
Required fields:
- `backend: local`
- `path`
### SSH source
```yaml
source:
backend: ssh
uri: ssh://reports@example.com:22
path: /var/spool/distributor/weather
```
Required fields:
- `backend: ssh`
- `uri`
- `path`
Recommended authentication behavior:
- use SSH agent by default;
- use local known_hosts validation by default;
- support optional key file configuration later if needed;
- do not require passwords in YAML.
Optional future fields:
```yaml
known_hosts: /home/user/.ssh/known_hosts
key_file: /home/user/.ssh/id_ed25519
```
### S3 source
```yaml
source:
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: incoming/weather
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
```
Required fields:
- `backend: s3`
- `endpoint`
- `bucket`
Optional fields:
- `prefix`
- `region`
- `force_path_style`
- `credentials`
Credential configuration should prefer environment variables or standard SDK behavior over literal secrets in config.
## Destination Configuration
Each destination defines one publication target for a pipeline.
Required destination fields:
- `id`
- `backend`
- backend-specific location fields;
- `publish`
Optional destination fields:
- `transform`
- `transfer`
Each destination is independently planned and published. A destination may receive source files, generated HTML, or both.
## Destination ID Rules
`destinations[].id` should:
- be required;
- be unique within the containing pipeline;
- be stable over time;
- use a slug-like format.
Recommended examples:
- `markdown-archive`
- `static-site`
- `full-mirror`
## Local Destination
```yaml
destinations:
- id: local-static
backend: local
path: /srv/www/reports
publish:
source: false
html: true
```
Required fields:
- `backend: local`
- `path`
- `publish`
## SSH Destination
```yaml
destinations:
- id: static-site
backend: ssh
uri: ssh://deploy@example.com:22
path: /srv/www/weather
publish:
source: false
html: true
```
Required fields:
- `backend: ssh`
- `uri`
- `path`
- `publish`
The MVP should use a native SFTP implementation rather than shelling out to `ssh`, `scp`, or `rsync`.
## S3 Destination
```yaml
destinations:
- id: markdown-archive
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: weather/archive
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
publish:
source: true
html: false
```
Required fields:
- `backend: s3`
- `endpoint`
- `bucket`
- `publish`
Optional fields:
- `prefix`
- `region`
- `force_path_style`
- `credentials`
## Backend Configuration Normalization
The config loader should normalize backend configuration into internal source and destination backend specs. Pipeline logic should not branch on backend-specific fields.
Validation should catch:
- missing backend names;
- unsupported backend names;
- missing backend-specific required fields;
- duplicate pipeline IDs;
- duplicate destination IDs within a pipeline;
- empty destination lists;
- invalid policy values.
## Publication Policy
`publish` controls which categories of files are written to a destination.
```yaml
publish:
source: true
html: false
```
Fields:
- `source`: Publish source artifacts listed in `manifest.json`.
- `html`: Publish HTML files generated from Markdown source artifacts.
At least one of `source` or `html` must be true.
Recommended defaults:
```yaml
publish:
source: true
html: false
```
No implicit HTML transformation should occur. When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `mode: sidecar` are required for the MVP.
## Transform Configuration
For MVP, the only supported transform is Markdown to HTML.
```yaml
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
Fields:
- `enabled`: Whether Markdown-to-HTML transform is enabled.
- `mode`: Output mode. MVP value: `sidecar`.
MVP `sidecar` behavior:
- each listed Markdown source file generates an HTML file with the same base path and `.html` extension;
- `report.md` generates `report.html`;
- generated files are destination publication artifacts;
- source bundles are not mutated.
MVP defaulting:
- If `publish.html` is false, transform may be omitted.
- If `publish.html` is true and `transform.markdown_to_html` is omitted or disabled, config validation must fail.
## Validation Policy
Pipeline-level validation is intentionally narrow in the MVP.
```yaml
validation:
on_digest_mismatch: fail
```
Supported value:
- `fail`
Default:
```yaml
on_digest_mismatch: fail
```
Validation should happen before any destination writes. Warning-only digest mismatch handling is deferred and must be rejected if configured.
## Transfer Policy
Destination-level transfer policy controls behavior after inspecting `.distributor.json` at the destination bundle path.
```yaml
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
```
Supported fields:
- `on_destination_same`
- `on_destination_older`
- `on_destination_newer`
- `on_conflict`
Recommended supported values:
- `skip`
- `replace`
- `fail`
Recommended MVP defaults:
```yaml
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
```
Safety rule:
- `replace` must never perform broad deletion against a destination root.
- `replace` may only operate within a resolved destination bundle path and should delete only files recorded in existing `.distributor.json.outputs` plus `.distributor.json` where practical.
- Unmanaged non-empty destination paths fail in the MVP. Force or unmanaged overwrite configuration is deferred.
## Path Mapping
MVP path mapping is fixed:
```text
destination bundle path = destination root + source relative bundle path
```
Example:
```text
source root: /var/spool/reports
source bundle: /var/spool/reports/weather/daily/brentwood/2026-05-30
relative bundle path: weather/daily/brentwood/2026-05-30
destination root: /srv/www/reports
destination bundle path: /srv/www/reports/weather/daily/brentwood/2026-05-30
```
Future config may support explicit path mapping, but MVP should not.
## Dry Run Configuration and CLI Behavior
Dry-run should be a CLI flag rather than a persistent config setting.
```bash
distributor run --config config.yml --dry-run
```
Dry-run should report:
- pipeline ID;
- source backend;
- destination ID;
- destination backend;
- discovered bundle ID;
- relative bundle path;
- planned action;
- reason;
- transform outputs that would be generated;
- files that would be written or deleted.
## Example: Weather Pipeline
```yaml
pipelines:
- id: weather-daily
source:
backend: local
path: /var/spool/distributor/weather
validation:
on_digest_mismatch: fail
destinations:
- id: markdown-archive
backend: s3
endpoint: https://s3.maximumdirect.net
bucket: reports
prefix: weather/archive
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
publish:
source: true
html: false
- id: static-site
backend: ssh
uri: ssh://deploy@web.maximumdirect.net:22
path: /srv/www/weather
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
## Example: D&D Recap Pipeline
```yaml
pipelines:
- id: dnd-session-recaps
source:
backend: local
path: /var/spool/distributor/dnd/session-recaps
destinations:
- id: private-markdown-archive
backend: s3
endpoint: https://s3.maximumdirect.net
bucket: reports
prefix: dnd/session-recaps
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
publish:
source: true
html: false
- id: private-html-site
backend: local
path: /srv/www/private/dnd/session-recaps
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
## Future Notification Configuration
Notification should not be exposed as a functional MVP feature unless an adapter exists.
The internal pipeline may include a no-op notification stage. Future config may look like:
```yaml
notifications:
- id: weather-email
backend: email
after_destinations:
- static-site
subject: "Weather report published"
```
Future notification policy should require:
- notification after successful relevant publication;
- idempotency by source manifest id and digest;
- no duplicate notification unless explicitly forced.
## Implementation Stages
1. Define config structs for pipelines, sources, destinations, validation, publish, transform, and transfer policies.
2. Implement config loading and strict validation.
3. Implement backend-specific config validation for local, SSH, and S3.
4. Implement defaulting for validation and transfer policies.
5. Require explicit transform configuration when `publish.html` is true.
6. Add example config fixtures for local-to-local, local-to-SSH, local-to-S3, and fan-out scenarios.
7. Connect config to backend registry and publish planner.
8. Add `--pipeline` filtering for targeted runs.
9. Add `--dry-run` output that reflects the resolved config and planned actions.
## Deferred Configuration
The following configuration ideas are intentionally outside the MVP:
- warning-only digest mismatch handling;
- unmanaged destination overwrite flags;
- force replacement of destinations with different source ids.

390
docs/roadmap/contracts.md Normal file
View File

@@ -0,0 +1,390 @@
# Distributor Contracts Roadmap
This roadmap defines the contracts that `distributor` should implement before or alongside the MVP. The goal is to make bundle validation, destination state, digest verification, and safe replacement deterministic and testable before backend-specific publication behavior is layered on top.
## Purpose
`distributor` publishes manifested report bundles produced by other applications. Producer applications own domain-specific report generation. `distributor` owns validation, optional transformation, destination publication, and destination state.
The MVP contract has two durable files:
- `manifest.json`: source-owned bundle manifest produced by the upstream application.
- `.distributor.json`: destination-owned publication state written by `distributor`.
`manifest.json` is not copied to the destination as destination state. Instead, `.distributor.json` records the normalized source manifest, generated output metadata, and distributor-owned publication metadata.
## Terminology
- **Source root**: Configured root path for a pipeline source.
- **Bundle root**: Directory beneath the source root that contains `manifest.json`.
- **Relative bundle path**: Bundle root path relative to the source root.
- **Destination root**: Configured root path or prefix for a destination.
- **Destination bundle path**: Destination root plus the relative bundle path, unless a future mapping option overrides that behavior.
- **Source artifact**: File listed in the source `manifest.json`.
- **Generated artifact**: File created by `distributor`, such as an HTML file derived from Markdown.
- **Destination state**: `.distributor.json` at the destination bundle path.
## Source Bundle Contract
A source bundle is a directory containing a `manifest.json` file. For MVP, the manifest schema is intentionally minimal.
### Required `manifest.json` fields
```json
{
"schema_version": 1,
"id": "weather.daily.brentwood.2026-05-30",
"digest": "sha256:...",
"created": "2026-05-30T11:10:00Z",
"files": [
{
"path": "report.md",
"sha256": "sha256:...",
"size": 12345
}
]
}
```
Required top-level fields:
- `schema_version`: Source manifest schema version. MVP value: `1`.
- `id`: Stable bundle identifier. Required, non-empty string.
- `digest`: Bundle digest. Required, lowercase `sha256:<64 hex>` string.
- `created`: Bundle creation timestamp. Required RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
- `files`: Non-empty array of file objects.
Required file fields:
- `path`: Relative path from bundle root to source artifact.
- `sha256`: Per-file digest as lowercase `sha256:<64 hex>`.
- `size`: File size in bytes.
No other source manifest fields are required for the MVP. Additional fields may be ignored unless later documented. Destination state records the normalized source manifest model, not raw unknown manifest fields.
## Source Path Safety Rules
For every `files[].path`:
- Path must be relative.
- Path must not be empty.
- Path must not contain `..` segments.
- Path must not resolve outside the bundle root.
- Path must use slash-separated logical paths in the manifest.
- Absolute paths are invalid.
- Symlinks should be rejected for MVP unless a later policy deliberately supports them.
- `manifest.json` itself should not be listed as a source artifact.
- `.distributor.json` should not be listed as a source artifact.
- Duplicate logical file paths are invalid after path normalization.
The implementation should validate paths before reading file contents.
## Digest Contract
The MVP validates both per-file digests and the bundle digest.
### Per-file digest
For each file listed in `files`, compute:
```text
sha256(file bytes)
```
The computed digest must match `files[].sha256`.
The actual file size must match `files[].size`.
### Bundle digest
The bundle digest is computed from the listed file records in the listed order. The canonical algorithm is:
1. For each file listed in `files`, in order:
- validate the file path;
- compute the file SHA256;
- determine the file size.
2. Construct a canonical JSON array containing only:
- `path`;
- `sha256`;
- `size`.
3. Preserve the source manifest's file order.
4. Encode the array deterministically with:
- object fields in exactly this order: `path`, `sha256`, `size`;
- no extra spaces;
- no trailing newline;
- lowercase `sha256:<64 hex>` digest strings.
5. Compute `sha256(canonical JSON bytes)`.
6. Compare the result to top-level `digest`.
Conceptual canonical payload:
```json
[
{"path":"report.md","sha256":"sha256:...","size":12345},
{"path":"summary.txt","sha256":"sha256:...","size":234}
]
```
This avoids ambiguous concatenation of file bytes while keeping the manifest small.
Implementation fixtures should include at least one reference manifest and canonical payload with known per-file and bundle digests.
## Digest Mismatch Behavior
Digest validation is always fatal in the MVP. A digest mismatch must fail the affected bundle, pipeline, and run before any destination writes occur.
Warning-only digest behavior is intentionally deferred until a later roadmap accepts transitional ingestion semantics.
## Bundle Discovery Contract
A source path may contain one bundle or a tree of bundles. Discovery should scan beneath the configured source root for `manifest.json` files.
For each discovered manifest:
- Bundle root is the directory containing `manifest.json`.
- Relative bundle path is computed relative to source root.
- Destination bundle path is destination root plus relative bundle path, unless a future mapping option overrides it.
If nested manifests are found, the MVP should fail with a clear error unless a later policy defines nested-bundle semantics.
## Destination State Contract
Destinations are managed by `.distributor.json`, not by copying `manifest.json`.
A destination bundle path is considered distributor-managed only when it contains a valid `.distributor.json` written by `distributor`. Force or unmanaged-overwrite behavior is not part of the MVP.
### Required `.distributor.json` shape
```json
{
"schema_version": 1,
"distributor_version": "0.1.0",
"pipeline_id": "weather-daily",
"destination_id": "static-site",
"published_at": "2026-05-30T11:12:00Z",
"source": {
"manifest": {
"schema_version": 1,
"id": "weather.daily.brentwood.2026-05-30",
"digest": "sha256:...",
"created": "2026-05-30T11:10:00Z",
"files": [
{
"path": "report.md",
"sha256": "sha256:...",
"size": 12345
}
]
}
},
"outputs": [
{
"path": "report.html",
"kind": "generated",
"source_path": "report.md",
"transform": "markdown_to_html",
"sha256": "sha256:...",
"size": 23456
}
]
}
```
Required fields:
- `schema_version`: Destination state schema version. MVP value: `1`.
- `distributor_version`: Optional diagnostic distributor version. It must not affect source comparison.
- `pipeline_id`: Pipeline that produced the destination publication.
- `destination_id`: Destination within the pipeline.
- `published_at`: RFC3339 timestamp.
- `source.manifest`: Normalized source manifest model used for this publication.
- `outputs`: Array of files written by `distributor` for this destination.
Required output fields:
- `path`: Destination-relative output path within the destination bundle path.
- `kind`: `source` or `generated`.
- `source_path`: Source artifact path that produced this output. For copied source files, this should equal `path` unless renamed by a future feature.
- `transform`: Transform identifier for generated files. Empty or omitted may be allowed for copied source files.
- `sha256`: Output file digest as lowercase `sha256:<64 hex>`.
- `size`: Output file size in bytes.
## Destination Comparison Rules
Destination comparison uses `.distributor.json`, not destination `manifest.json`.
For a source bundle and destination bundle path:
### No `.distributor.json`
If no `.distributor.json` exists and the destination path is empty:
- Publish normally.
If no `.distributor.json` exists and the destination path is non-empty:
- Fail as unmanaged content.
Local destination paths are empty when the destination bundle directory does not exist or exists with no entries. S3-compatible destination prefixes are empty when no objects exist below the destination bundle prefix, ignoring objects outside that exact prefix.
### Same source manifest
If `.distributor.json` exists and `source.manifest` exactly matches the current normalized source manifest:
- Skip as already published.
### Same source id, destination older
If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is older than the source `created`:
- Replace destination contents, subject to replacement safety rules.
### Same source id, destination newer
If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is newer than the source `created`:
- Skip and log that destination is newer than source.
### Same source id and created, different digest
If `.distributor.json` exists, `source.manifest.id` and `created` match but `digest` differs:
- Fail as a conflict.
### Different source id
If `.distributor.json` exists and `source.manifest.id` differs from the source manifest `id`:
- Fail as a conflict.
## Replacement Safety Rules
Replacement is destructive and must be narrow.
`distributor` must never perform broad deletion against a configured destination root.
Replacement may occur only at a resolved destination bundle path when:
- a valid `.distributor.json` exists at that destination bundle path; and
- the state identifies the path as distributor-managed; and
- the replacement decision follows the destination comparison rules.
For MVP, replacement should delete only known managed outputs where practical:
- files listed in existing `.distributor.json.outputs`;
- existing `.distributor.json`;
- empty directories created by those files, where applicable for filesystem-like backends.
For S3, replacement should delete only objects under the destination bundle prefix that are listed in `.distributor.json.outputs` plus `.distributor.json`, unless a later managed-prefix deletion policy is explicitly implemented.
Before any write, planned destination output paths must be checked for collisions. For example, if `report.md` is copied as a source artifact and Markdown transformation would also generate `report.html`, but `report.html` is already a source artifact or another generated output, planning must fail before writing.
## Transform Output Contract
Source files are canonical. Generated files are derived publication artifacts.
For MVP, the only supported transform is Markdown to HTML.
Recommended MVP behavior:
- Transform is configured per destination.
- Markdown source files are files listed in `manifest.json` with `.md` extension.
- Generated HTML files are sidecars by default.
- `report.md` generates `report.html`.
- Source bundles are never mutated.
- Generated outputs are recorded in `.distributor.json.outputs`.
- Raw HTML embedded in Markdown is escaped or disabled by default for the MVP to keep generated output deterministic and conservative.
Future versions may add templates, `index.html`, CSS assets, email-safe HTML, and per-file transform selection.
## Destination Output Contract
Each destination chooses which categories of files it receives.
MVP categories:
- `source`: copied source artifacts listed in `manifest.json`.
- `html`: generated HTML derived from Markdown source artifacts.
Examples:
- Markdown archive: `source: true`, `html: false`.
- Static HTML site: `source: false`, `html: true`.
- Full mirror: `source: true`, `html: true`.
Every file written to the destination must be represented in `.distributor.json.outputs`.
## Atomicity and Partial Failure
The MVP should prefer staging and promotion where backend semantics permit it.
Minimum behavior:
- Validate source before writing destination files.
- Do not write `.distributor.json` until all configured destination outputs are successfully written.
- If publication fails before `.distributor.json` is written, the destination must not be treated as successfully published on a later run.
- Local publication must use staging or equivalent cleanup behavior so a failed write does not leave a confusing unmanaged destination bundle path.
- Later cleanup may remove orphaned files, but MVP correctness should rely on `.distributor.json` as the success marker.
## Example Source Bundle
```text
weather/daily/brentwood/2026-05-30/
manifest.json
report.md
summary.txt
```
Example manifest:
```json
{
"schema_version": 1,
"id": "weather.daily.brentwood.2026-05-30",
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
"created": "2026-05-30T11:10:00Z",
"files": [
{
"path": "report.md",
"sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
"size": 12345
},
{
"path": "summary.txt",
"sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222",
"size": 234
}
]
}
```
## Example Destination Bundle: HTML Only
```text
weather/daily/brentwood/2026-05-30/
report.html
.distributor.json
```
## Example Destination Bundle: Source Archive
```text
weather/daily/brentwood/2026-05-30/
report.md
summary.txt
.distributor.json
```
## Implementation Stages
1. Define Go structs for source manifest and destination state.
2. Implement source path validation.
3. Implement per-file SHA256 and size validation.
4. Implement canonical bundle digest validation.
5. Implement source bundle discovery beneath a source root.
6. Implement `.distributor.json` parsing and validation.
7. Implement destination comparison rules.
8. Implement replacement safety checks.
9. Add fixture bundles for valid, invalid, duplicate path, older, newer, same, and conflict scenarios.
10. Add reference canonical digest fixtures.
11. Use the contract layer from the publish pipeline and backend adapters.

808
docs/roadmap/implementation.md Normal file
View File

@@ -0,0 +1,808 @@
# Distributor Implementation Roadmap
This roadmap defines a staged implementation plan for the `distributor` MVP. Each stage is intended to map cleanly to one Codex implementation prompt.
The roadmap assumes the project includes these planning documents before implementation begins:
- `docs/policy/architecture.md`
- `docs/policy/documentation.md`
- `docs/roadmap/packages.md`
- `docs/roadmap/contracts.md`
- `docs/roadmap/config.md`
The MVP goal is a domain-agnostic bundle distributor that discovers source bundles, validates `manifest.json`, optionally transforms Markdown to HTML, publishes selected outputs to one or more destinations, and records destination state in `.distributor.json`.
## Global Implementation Rules
All stages should preserve these invariants:
- Producer applications own source bundle creation.
- `distributor` owns validation, transformation, publication, destination state, and future notification hooks.
- Source bundle state is defined by `manifest.json`.
- Destination publication state is defined by `.distributor.json`.
- `manifest.json` is not copied to the destination as destination state.
- Pipelines have exactly one source and one or more destinations.
- Transform and publish policy are destination-specific.
- Destructive replacement is allowed only inside a managed destination bundle path. Unsafe force or unmanaged overwrite behavior is deferred.
- Dry-run behavior should be implemented before broad remote write behavior.
- Config, bundle, state, publish planning, storage adapters, transforms, and CLI wiring should remain separate packages.
Unless a stage explicitly says otherwise, each implementation prompt should:
1. read the project policy and roadmap documents;
2. implement only the current stage;
3. add or update tests for the current stage;
4. run the relevant test suite;
5. update documentation only when the implemented behavior now exists;
6. avoid implementing future roadmap stages early.
## Stage 1: Project Skeleton, CLI Shell, and Baseline Tooling
### Goal
Create the initial Go application structure and a minimal executable `distributor` command with no business behavior beyond version/help output and placeholder commands.
### Scope
Implement the accepted package skeleton from `docs/roadmap/packages.md` at the level needed for compilation.
Create:
```text
cmd/distributor/main.go
internal/cli/
internal/app/
internal/config/
internal/logging/
```
Initial CLI commands:
- `distributor --help`
- `distributor version`
- `distributor run`
- `distributor validate`
- `distributor inspect`
At this stage, `run`, `validate`, and `inspect` may return clear “not implemented” errors, but the command structure should be present.
### Notes
Prefer a small CLI dependency only if the project already standardizes on one. Otherwise, the standard library is acceptable for the first pass.
Add a version variable that can later be set at build time.
### Tests
Add tests for:
- command construction if testable;
- version string behavior if exposed through a package;
- basic package compilation.
### Completion Criteria
- `go test ./...` passes.
- `go run ./cmd/distributor --help` works.
- `go run ./cmd/distributor version` works.
- Placeholder operational commands fail clearly and intentionally.
## Stage 2: Config Schema, Loading, Defaults, and Validation
### Goal
Implement the MVP `config.yml` schema described in `docs/roadmap/config.md`.
### Scope
Create config structs for:
- top-level config;
- pipelines;
- source backend config;
- destination backend config;
- validation policy;
- publish policy;
- transform policy;
- transfer/replacement policy;
- backend-specific local, SSH, and S3 fields.
Support loading YAML from a file path.
Implement validation for:
- required top-level `pipelines`;
- unique pipeline ids;
- required pipeline `id`, `source`, and non-empty `destinations`;
- unique destination ids within a pipeline;
- supported backend names: `local`, `ssh`, `s3`;
- required backend fields;
- supported validation action: `fail`;
- supported transfer actions;
- valid `publish` policy;
- valid Markdown-to-HTML transform config.
Default behavior should match `docs/roadmap/config.md`.
### CLI Integration
Add `--config` to `run`.
For this stage, `distributor run --config config.yml --dry-run` may only load and validate config, then print a concise summary of configured pipelines and destinations.
### Tests
Add unit tests for:
- valid minimal local-to-local config;
- valid fan-out config;
- valid local, SSH, and S3 backend configs;
- duplicate pipeline ids;
- duplicate destination ids;
- missing required fields;
- unsupported backend;
- invalid transfer action;
- invalid validation action, including `warn`.
### Completion Criteria
- Config load/default/validate behavior is implemented and tested.
- `distributor run --config <file> --dry-run` validates config and prints a summary.
- No bundle discovery or publication occurs yet.
## Stage 3: Storage Abstraction, Local Backend, and Fake Backend
### Goal
Introduce the storage backend abstraction before bundle validation so source discovery, validation, and publication are backend-agnostic from the start.
### Scope
Create:
```text
internal/storage/backend.go
internal/storage/registry.go
internal/storage/path.go
internal/storage/errors.go
internal/adapters/local/backend.go
internal/storage/fake/
```
Define backend operations needed by the MVP, including:
- read file;
- write file;
- test existence;
- list files or tree entries;
- read destination state file if present;
- create directories/prefixes as needed;
- delete explicit managed files safely;
- write files using temporary/staged writes where practical.
The fake backend should exist for unit tests of config, bundle, state, and publish logic without real local, SSH, or S3 IO.
### Safety Requirements
The local backend must:
- clean and join paths safely;
- reject path traversal;
- reject unsafe destructive deletion requests;
- avoid following symlinks for source bundle files unless explicitly supported;
- avoid deleting configured roots;
- classify destination bundle emptiness deterministically.
### Tests
Add tests for:
- backend read/write/list/exists behavior;
- safe path joining;
- traversal rejection;
- symlink rejection for source reads;
- explicit-file deletion guard behavior;
- local destination emptiness detection;
- fake backend behavior sufficient for core package tests.
### Completion Criteria
- Local backend implements the storage interface.
- Fake backend can support bundle and publish tests without external services.
- `go test ./...` passes.
- No SSH or S3 implementation exists yet.
## Stage 4: Source Bundle Manifest, Digest, Validation, and Discovery
### Goal
Implement the source bundle contract from `docs/roadmap/contracts.md` through the storage abstraction.
### Scope
Create:
```text
internal/bundle/manifest.go
internal/bundle/digest.go
internal/bundle/validate.go
internal/bundle/discover.go
```
Implement:
- parsing `manifest.json`;
- strict required field validation, including `schema_version: 1`;
- RFC3339 `created` parsing;
- lowercase `sha256:<64 hex>` digest validation;
- source file path safety checks;
- duplicate logical file path rejection;
- per-file SHA256 validation;
- per-file size validation;
- bundle digest validation using the canonical ordered file-record algorithm;
- deterministic storage-backed bundle discovery under a source root;
- nested manifest detection and failure.
Discovery and validation should use `internal/storage` rather than direct `os` APIs. The local CLI path should be adapted to the local backend.
### CLI Integration
Implement:
```text
distributor validate <path>
distributor inspect <path>
```
For local paths:
- `validate` should validate either a single bundle directory or a tree containing bundles.
- `inspect` should print a concise normalized summary of discovered bundle ids, relative paths, created timestamps, digest values, and files.
### Tests
Add fixture bundles under a testdata directory.
Test:
- valid bundle;
- invalid JSON;
- missing required fields;
- invalid schema version;
- invalid timestamp;
- invalid digest format;
- unsafe file paths;
- duplicate normalized file paths;
- missing files;
- size mismatch;
- per-file digest mismatch;
- bundle digest mismatch;
- canonical bundle digest reference fixture;
- multiple discovered bundles in deterministic order;
- nested manifests fail.
### Completion Criteria
- Storage-backed bundle validation is deterministic and well-tested.
- `distributor validate <path>` works for local bundle fixtures.
- `distributor inspect <path>` works for local bundle fixtures.
- No destination publication occurs yet.
## Stage 5: Destination State Contract and Comparison Logic
### Goal
Implement `.distributor.json` parsing, validation, and source-to-destination comparison.
### Scope
Create:
```text
internal/state/distributor.go
internal/state/compare.go
internal/state/validate.go
```
Implement the destination state schema from `docs/roadmap/contracts.md`, including:
- `schema_version`;
- optional `distributor_version`;
- `pipeline_id`;
- `destination_id`;
- `published_at`;
- embedded normalized source manifest;
- outputs array;
- output file metadata.
Implement comparison outcomes:
- destination absent;
- destination unmanaged/non-empty;
- same source manifest;
- same source id, destination older;
- same source id, destination newer;
- same source id and same created but different digest;
- different source id;
- invalid destination state.
### Tests
Add unit tests for every comparison outcome.
Test validation for:
- valid state;
- missing fields;
- invalid schema version;
- invalid embedded source manifest;
- invalid output metadata;
- malformed published timestamp.
### Completion Criteria
- Destination state can be parsed and validated independently.
- Source manifest to destination state comparison is deterministic and fully tested.
- No publication execution occurs yet.
## Stage 6: Publish Planning, Dry-Run, and Local-to-Local Publication Without Transform
### Goal
Implement the core publish planner and execute local-to-local publication for source files only.
### Scope
Create:
```text
internal/publish/plan.go
internal/publish/reconcile.go
internal/publish/safety.go
internal/publish/output.go
internal/publish/execute.go
```
Implement planning for one source bundle to one destination based on:
- source manifest;
- destination config;
- publish policy;
- transfer policy;
- existing `.distributor.json`;
- destination path state.
Actions should include:
- publish new;
- replace older destination;
- skip same;
- skip destination newer;
- fail conflict;
- fail unmanaged destination.
Implement local-to-local execution for `publish.source: true` and `publish.html: false`.
Execution should:
- copy listed source files selected by publish policy;
- write `.distributor.json` with copied source output metadata;
- avoid copying source `manifest.json` as destination state;
- preserve relative bundle paths from source root beneath destination root;
- detect destination output collisions before writing;
- use staging or equivalent cleanup behavior for local writes;
- support fan-out to multiple local destinations;
- support dry-run without writes.
### CLI Integration
`distributor run --config <file>` should now execute local-to-local pipelines when configured.
`--dry-run` should print the planned action for each discovered bundle and destination.
### Tests
Add integration-style tests using temp directories for:
- new local publication;
- no-op when destination state matches;
- replacement when destination state is older;
- skip when destination state is newer;
- fail on conflict;
- fail on unmanaged non-empty destination;
- fail on output path collision;
- fan-out from one source to two local destinations;
- failed local write does not leave a destination that appears unmanaged on retry;
- dry-run performs no writes;
- `.distributor.json` is written correctly.
### Completion Criteria
- Local-to-local source-file publication works end to end.
- Dry-run produces meaningful planned actions.
- Destination state is authoritative.
- No Markdown-to-HTML transform exists yet.
## Stage 7: Markdown-to-HTML Transform and Destination-Specific Publish Policy
### Goal
Add MVP Markdown-to-HTML transformation and destination-specific source/html output selection.
### Scope
Create:
```text
internal/transform/transform.go
internal/transform/registry.go
internal/transform/plan.go
internal/transform/markdown/markdown.go
internal/transform/markdown/template.go
```
Implement only:
```yaml
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
MVP sidecar behavior:
- for each listed source artifact ending in `.md`, generate a same-directory `.html` sidecar;
- preserve the original Markdown file unchanged;
- do not generate HTML for non-Markdown files;
- escape or disable raw HTML embedded in Markdown;
- fail before writing when generated output paths collide with copied source outputs or other generated outputs;
- record generated output metadata in `.distributor.json`;
- if `publish.source: false`, do not publish source files;
- if `publish.html: true`, publish generated HTML files;
- if `publish.html: true` but transform is disabled or no Markdown files exist, fail with a clear error unless config later defines another behavior.
Use a well-maintained Markdown renderer. Keep HTML templating minimal and deterministic.
### Tests
Add tests for:
- Markdown sidecar generation;
- source-only destination;
- HTML-only destination;
- source-plus-HTML destination;
- no mutation of source bundle;
- generated output metadata in `.distributor.json`;
- failure when HTML publication is requested without transform support;
- failure when generated HTML collides with a source artifact path;
- raw HTML in Markdown is escaped or disabled consistently;
- deterministic output for a fixture Markdown file.
### Completion Criteria
- Local-to-local publication supports source-only, HTML-only, and source-plus-HTML destinations.
- Generated outputs are recorded in destination state.
- Dry-run reports transform outputs that would be generated.
## Stage 8: No-Op Notification Stage and Pipeline Orchestration Polish
### Goal
Add the internal no-op notification stage and polish orchestration around per-destination outcomes.
### Scope
Create:
```text
internal/notify/notify.go
internal/notify/noop.go
```
Integrate a no-op notifier after successful publication/skip handling as appropriate.
Clarify orchestration behavior when one destination fails. For MVP, fan-out should be deterministic and sequential. Continue planning and reporting later destinations where safe, but return non-zero if any destination fails.
Improve run summary output:
- pipeline id;
- source backend;
- discovered bundle count;
- destination ids;
- action per bundle/destination;
- final status.
### Tests
Add tests for:
- notifier is invoked at the expected orchestration point where testable;
- pipeline failure when a destination fails;
- run summary contains meaningful status information;
- dry-run does not invoke write-side effects.
### Completion Criteria
- The pipeline shape includes notification as an internal no-op stage.
- Run output is useful for unattended operation logs.
- Local MVP behavior remains passing.
## Stage 9: Native SSH/SFTP Backend
### Goal
Implement SSH/SFTP storage backend support for sources and destinations.
### Scope
Create:
```text
internal/adapters/ssh/backend.go
internal/adapters/ssh/config.go
```
Implement the storage backend interface over native SSH/SFTP.
Required config:
```yaml
backend: ssh
uri: ssh://user@example.com:22
path: /remote/root
```
Authentication expectations:
- prefer SSH agent by default;
- use known_hosts validation by default where practical;
- do not require passwords in YAML;
- optional key-file support may be implemented if straightforward, but should not distract from agent-based auth.
Support SSH/SFTP backend as both source and destination:
- local -> ssh;
- ssh -> local;
- ssh -> ssh where feasible through staging or streaming.
### Safety Requirements
The SSH backend must enforce the same logical path safety rules as the local backend.
Deletion must remain limited to managed destination bundle paths guarded by valid `.distributor.json`.
### Tests
Unit-test path handling and config validation.
If practical, add integration tests that can be skipped unless an SSH test endpoint is configured through environment variables. Do not require a live SSH server for normal `go test ./...`.
### Completion Criteria
- SSH/SFTP backend compiles and satisfies the storage interface.
- Backend config validation is tested.
- Normal tests do not depend on a live SSH server.
- At least local-to-SSH and SSH-to-local flows are documented or manually testable.
## Stage 10: S3-Compatible Backend
### Goal
Implement S3-compatible backend support for sources and destinations.
### Scope
Create:
```text
internal/adapters/s3/backend.go
internal/adapters/s3/config.go
```
Required config should align with `docs/roadmap/config.md`:
```yaml
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: some/prefix
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
```
Implement storage operations over S3 object keys:
- read object;
- write object;
- exists;
- list prefix;
- delete managed prefix or listed managed files;
- read/write `.distributor.json`.
Set reasonable content types where available:
- `.md`: `text/markdown; charset=utf-8`;
- `.html`: `text/html; charset=utf-8`;
- `.json`: `application/json`;
- `.txt`: `text/plain; charset=utf-8`.
Support S3 backend as both source and destination.
### Safety Requirements
Treat S3 prefixes as object trees. Do not assume real directories exist.
Deletion must be limited to destination bundle prefixes that are confirmed managed by `.distributor.json`.
### Tests
Add unit tests for:
- config validation;
- key/prefix normalization;
- content type selection;
- path traversal rejection;
- publish planning with S3 destination state fixtures.
If practical, add integration tests gated by environment variables or a local S3-compatible test service. Normal `go test ./...` must not require live S3 credentials.
### Completion Criteria
- S3 backend compiles and satisfies the storage interface.
- S3 source and destination flows are supported through the common pipeline path.
- Normal tests do not require live S3.
## Stage 11: Cross-Backend End-to-End Coverage and Hardening
### Goal
Harden the MVP across backend combinations, destination policies, and failure cases.
### Scope
Add end-to-end coverage for representative scenarios:
- local source -> local archive destination;
- local source -> local HTML destination;
- local source -> two destinations with different publish policies;
- local source -> SSH destination, where integration credentials exist;
- local source -> S3 destination, where integration credentials exist;
- S3 source -> local destination, where integration credentials exist;
- SSH source -> local destination, where integration credentials exist.
Improve logging and error messages for:
- invalid config;
- invalid source manifest;
- digest mismatch;
- destination conflict;
- unmanaged destination path;
- backend read/write/list failures;
- transform failures.
Ensure all destructive paths have tests or explicit safeguards.
### Tests
Add or expand tests for:
- dry-run across multiple destinations;
- partial failure behavior;
- repeated run idempotency;
- older/newer destination state behavior;
- destination state output metadata accuracy;
- generated HTML output metadata accuracy.
### Completion Criteria
- MVP behavior is reliable across implemented backend types.
- Error messages identify pipeline id, destination id, bundle id, and reason where practical.
- Idempotent repeated runs behave as expected.
## Stage 12: User-Facing Documentation Sync
### Goal
Update documentation to reflect implemented MVP behavior.
### Scope
Following `docs/policy/documentation.md`, create or update user-facing documentation only for implemented features.
Likely docs:
```text
README.md
docs/config.md
docs/cli.md
docs/policy/architecture.md
docs/internal/bundles.md
docs/internal/backends.md
```
Document:
- what `distributor` does;
- bundle contract summary;
- `.distributor.json` role;
- example source bundle;
- example local-to-local config;
- example local-to-S3 config;
- example local-to-SSH config;
- `run`, `validate`, and `inspect` commands;
- dry-run behavior;
- replacement and safety rules;
- Markdown-to-HTML transform behavior;
- environment-variable credential handling.
Move roadmap material to historical/planning status only if your documentation policy allows it. Do not describe unimplemented notification adapters as available features.
### Tests
Run the full test suite.
If docs include command examples, verify that basic examples correspond to actual CLI behavior.
### Completion Criteria
- User-facing docs describe the implemented MVP accurately.
- Roadmap docs no longer masquerade as implemented behavior.
- `go test ./...` passes.
## Stage 13: MVP Release Readiness Pass
### Goal
Perform a final pre-release quality pass.
### Scope
Review:
- package boundaries against `docs/policy/architecture.md`;
- package layout against `docs/roadmap/packages.md`;
- implemented contracts against `docs/roadmap/contracts.md`;
- implemented config behavior against `docs/roadmap/config.md`;
- docs against `docs/policy/documentation.md`;
- destructive operation safety;
- logs and errors for unattended operation;
- command UX;
- test coverage for core invariants.
Add any missing small tests or docs discovered during review.
Do not add new product features in this stage.
### Completion Criteria
- MVP is ready to deploy against one real producer pipeline.
- A dry-run can be performed safely against a real source and destination.
- Repeated runs are idempotent.
- Destructive replacement cannot occur outside managed destination bundle paths.
- Final docs accurately reflect the application.
## Deferred Post-MVP Work
The following items are intentionally outside the MVP unless explicitly pulled into a later roadmap:
- email notifications;
- ntfy/Gotify/Pushover notifications;
- RSS/Atom feed generation;
- static site index pages beyond sidecar HTML output;
- templated HTML themes beyond a minimal deterministic template;
- destination path remapping rules;
- full plugin architecture;
- web UI;
- report editing;
- producer pipeline execution;
- database-backed state;
- complex retry queues;
- concurrent publication workers;
- symlink support;
- warning-only digest mismatch handling;
- force or unmanaged-overwrite destination behavior;
- password-based SSH authentication in YAML.

629
docs/roadmap/packages.md Normal file
View File

@@ -0,0 +1,629 @@
# Package Layout Roadmap
This roadmap defines the proposed package layout, boundaries, and implementation responsibilities for the `distributor` MVP.
`distributor` is expected to be a domain-agnostic bundle publisher. Producer applications emit source bundles containing `manifest.json`; `distributor` validates those bundles and publishes selected source and generated artifacts to one or more configured destinations.
This document is roadmap material. It describes the intended package design before implementation and should move into `docs/internal/` only after corresponding behavior exists.
## Accepted Package Layout
```text
cmd/distributor/
main.go
internal/app/
app.go
run.go
pipeline.go
internal/cli/
root.go
run.go
validate.go
inspect.go
internal/config/
config.go
defaults.go
load.go
validate.go
internal/bundle/
manifest.go
digest.go
validate.go
discover.go
internal/state/
distributor.go
compare.go
validate.go
internal/storage/
backend.go
registry.go
path.go
errors.go
internal/storage/fake/
backend.go
internal/adapters/local/
backend.go
internal/adapters/ssh/
backend.go
config.go
internal/adapters/s3/
backend.go
config.go
internal/transform/
transform.go
registry.go
plan.go
internal/transform/markdown/
markdown.go
template.go
internal/publish/
plan.go
execute.go
output.go
reconcile.go
safety.go
internal/notify/
notify.go
noop.go
internal/logging/
logging.go
```
## Package Responsibilities
### `cmd/distributor`
Application entrypoint only.
Responsibilities:
- call CLI execution;
- translate process exit status;
- avoid business logic.
Non-responsibilities:
- config loading;
- backend construction;
- bundle validation;
- publish decisions.
### `internal/cli`
CLI command definitions, flags, argument parsing, and command wiring.
Expected MVP commands:
- `distributor run` — run configured pipelines.
- `distributor run --dry-run` — plan without modifying destinations.
- `distributor run --pipeline <id>` — run one configured pipeline.
- `distributor validate <path>` — validate a source bundle or source tree where feasible.
- `distributor inspect <path>` — inspect a bundle or destination state where feasible.
Boundaries:
- CLI should call `internal/app` use cases.
- CLI should not parse manifests directly except through application APIs.
- CLI should not import backend adapter implementation details unless only for registration side effects.
### `internal/app`
Application orchestration and top-level use cases.
Responsibilities:
- load and validate configuration;
- construct configured pipelines;
- build source and destination backends through registries;
- orchestrate discovery, validation, planning, publishing, and notification;
- coordinate dry-run output;
- run destination fan-out deterministically and sequentially;
- aggregate destination outcomes into run-level failure behavior.
Core orchestration shape:
```text
for each selected pipeline:
open source backend
discover source bundles
for each source bundle:
validate source manifest and digest
for each destination:
inspect .distributor.json
build publish plan
transform as required by that destination
execute publish plan unless dry-run
run noop notifier
```
Boundaries:
- `internal/app` composes packages but should not contain backend-specific logic.
- Publish decisions should live in `internal/publish`, not inline in orchestration.
- Destination state comparison should live in `internal/state` or `internal/publish`, not CLI code.
### `internal/config`
Configuration structs, defaults, loading, precedence, and validation.
Responsibilities:
- load `/usr/local/etc/distributor/config.yml` by default;
- support `--config` override;
- apply defaults;
- validate required fields;
- validate pipeline ids and destination ids;
- validate backend-specific config shapes;
- validate transform and publish policy combinations.
MVP config model:
```yaml
pipelines:
- id: weather-daily
source:
backend: local
path: /var/spool/distributor/weather
validation:
on_digest_mismatch: fail
destinations:
- id: markdown-archive
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: weather/archive
region: us-east-1
force_path_style: true
publish:
source: true
html: false
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
- id: static-site
backend: ssh
uri: ssh://deploy@example.com:22
path: /srv/www/weather
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
Configuration principles:
- one source per pipeline;
- one or more destinations per pipeline;
- transforms are destination-specific;
- publish policy is destination-specific;
- secrets should use environment variables, secret files, SSH agent, or standard credential mechanisms rather than raw YAML values.
### `internal/bundle`
Source bundle contract and validation.
Responsibilities:
- parse source `manifest.json`;
- represent source manifests and files;
- discover bundle roots beneath a configured source root;
- validate required manifest fields;
- validate RFC3339 `created` values;
- validate relative paths;
- validate file existence, size, per-file SHA-256, and bundle digest;
- expose normalized source bundle models to other packages.
Core types:
```go
type Manifest struct {
SchemaVersion int
ID string
Digest string
Created time.Time
Files []ManifestFile
}
type ManifestFile struct {
Path string
SHA256 string
Size int64
}
type Bundle struct {
RootRelativePath string
Manifest Manifest
}
```
Boundaries:
- `internal/bundle` does not know about `.distributor.json`.
- `internal/bundle` does not know about destinations, transforms, or notification.
- `internal/bundle` may use the storage abstraction to read source files, but it should not import backend adapter packages.
### `internal/state`
Destination state contract for `.distributor.json`.
Responsibilities:
- parse `.distributor.json`;
- validate destination state;
- represent copied source outputs and generated outputs;
- embed the source manifest used for publication;
- compare destination state against a current source manifest;
- classify destination state as same, older, newer, conflict, absent, invalid, or unmanaged.
Core types:
```go
type DistributorState struct {
SchemaVersion int
DistributorVersion string
PipelineID string
DestinationID string
PublishedAt time.Time
Source SourceState
Outputs []OutputFile
}
type SourceState struct {
Manifest bundle.Manifest
}
type OutputFile struct {
Path string
Kind string // source | generated
SourcePath string
Transform string
SHA256 string
Size int64
}
```
Comparison rules:
- same source manifest: skip;
- same source id, older destination source `created`: replace;
- same source id, newer destination source `created`: skip;
- same source id, same `created`, different digest: conflict;
- different source id: conflict;
- absent state: publish only if safe;
- unmanaged non-empty path: fail.
Boundaries:
- `internal/state` owns destination state semantics, not publish execution.
- `internal/state` should not know about S3, SSH/SFTP, local filesystem details, or Markdown rendering.
### `internal/storage`
Backend abstraction and shared storage types.
Responsibilities:
- define storage backend interfaces;
- define object/file metadata types;
- define path/prefix helpers;
- define common storage errors;
- provide backend registry mechanisms.
- provide a fake backend for core package tests.
The core application should use storage interfaces such as:
```go
type Backend interface {
ReadFile(ctx context.Context, path string) ([]byte, error)
WriteFile(ctx context.Context, path string, data []byte, opts WriteOptions) error
Exists(ctx context.Context, path string) (bool, error)
List(ctx context.Context, prefix string) ([]Entry, error)
DeleteFiles(ctx context.Context, paths []string) error
}
```
Destructive APIs should remain narrow. Prefer deleting explicit files recorded in `.distributor.json` instead of broad recursive deletion.
Boundaries:
- `internal/storage` should not contain backend implementation details.
- Adapter dependencies must not leak through storage interfaces.
- The fake backend exists for tests and should not become an application runtime backend.
### `internal/adapters/local`
Local filesystem backend.
Responsibilities:
- implement `storage.Backend` for local paths;
- clean and constrain paths;
- perform safe reads/writes/listing/deletion;
- use atomic writes where practical;
- reject unsafe path traversal;
- handle symlink policy explicitly.
Testing expectations:
- use temporary directories;
- verify path traversal rejection;
- verify write and delete safety.
### `internal/adapters/ssh`
SSH/SFTP backend.
Responsibilities:
- implement `storage.Backend` over SSH/SFTP;
- support `uri` and `path` config;
- prefer native SFTP implementation;
- use SSH agent, key files, known hosts, or documented auth mechanisms;
- avoid raw passwords in config unless explicitly designed and documented later;
- translate SSH/SFTP errors into storage-level errors.
Testing expectations:
- core app tests should use fake backends;
- adapter tests may use local test servers or targeted integration tests if practical;
- do not require a real production SSH host for normal unit tests.
### `internal/adapters/s3`
S3-compatible object storage backend.
Responsibilities:
- implement `storage.Backend` over S3-compatible object storage;
- support endpoint, bucket, prefix, region, and force-path-style configuration;
- support standard credential mechanisms or explicit environment-variable references;
- treat S3 as an object tree, not a filesystem;
- set reasonable content types where practical;
- guard against prefix/root deletion mistakes.
Testing expectations:
- core app tests should use fake backends;
- adapter behavior may be tested through mocks, local S3-compatible services, or narrow integration tests;
- config examples should avoid real secrets.
### `internal/transform`
Transform interfaces, registry, and transform planning.
Responsibilities:
- define transform interfaces;
- register available transforms;
- represent transform requests and outputs;
- keep transform execution independent of destination backend details.
Boundaries:
- transforms operate on source bundle content and destination transform config;
- transforms do not publish outputs;
- transforms do not mutate source bundles;
- transforms should return generated output metadata for `.distributor.json`.
### `internal/transform/markdown`
Markdown-to-HTML implementation.
Responsibilities:
- render listed Markdown files to HTML;
- support MVP sidecar behavior, such as `report.md` -> `report.html`;
- record generated output path, source path, transform name, SHA-256, and size;
- optionally use embedded templates if needed.
MVP scope:
- Markdown to HTML only;
- no PDF generation;
- no email-specific HTML;
- no complex theming unless required for basic output correctness.
### `internal/publish`
Destination planning, reconciliation, safety checks, and publish execution.
Responsibilities:
- inspect destination state;
- plan destination action;
- enforce destination conflict rules;
- enforce destructive-operation safety rules;
- detect output path collisions before writing;
- combine source files and transform outputs according to destination publish policy;
- write destination outputs;
- write `.distributor.json`;
- use staging or equivalent cleanup behavior where practical;
- support dry-run planning;
- report skipped, replaced, failed, and published actions.
Action model:
```text
publish
replace
skip_same
skip_destination_newer
fail_conflict
fail_unmanaged
```
Boundaries:
- publish logic should not parse CLI flags;
- publish logic should not know adapter implementation details;
- publish logic should use `internal/state` for destination state semantics;
- publish logic should use `internal/storage` interfaces for IO.
### `internal/notify`
Notification stage abstraction.
MVP responsibilities:
- define notifier interface;
- implement no-op notifier;
- preserve future extension point for email, ntfy, Gotify, RSS update hooks, or other notification channels.
Future notification rules:
- notify only after successful publication to the relevant destination or destinations;
- notification must be idempotent with respect to source id, digest, pipeline id, and destination id where applicable;
- notification should not run for skipped or failed publications unless explicitly configured.
### `internal/logging`
Logging setup and helpers.
Responsibilities:
- centralize structured logging setup;
- ensure logs omit secrets;
- provide consistent fields for pipeline id, bundle id, destination id, backend, path, action, and reason.
## Implementation Slices
### Slice 1: Skeleton, config, storage, and source bundle validation
Deliver:
- basic CLI skeleton;
- config loading and validation;
- storage interface, local backend, and fake backend;
- source manifest model;
- source bundle discovery;
- file size and SHA-256 validation;
- bundle digest validation;
- local backend sufficient for validation;
- fixtures for valid and invalid bundles.
Useful commands:
```bash
distributor validate ./examples/weather-bundle
```
### Slice 2: Destination state and dry-run planning
Deliver:
- `.distributor.json` model;
- destination state comparison;
- publish plan model;
- dry-run output;
- local source to local destination planning;
- tests for same/older/newer/conflict/unmanaged cases.
Useful command:
```bash
distributor run --config ./examples/local.yml --dry-run
```
### Slice 3: Local publish execution
Deliver:
- local destination writes;
- source-file publication;
- `.distributor.json` writes;
- replacement safety checks;
- skip behavior;
- narrow deletion behavior based on destination state outputs.
Useful command:
```bash
distributor run --config ./examples/local.yml
```
### Slice 4: Markdown-to-HTML transform
Deliver:
- Markdown transform registry;
- Markdown-to-HTML implementation;
- per-destination `publish.source` and `publish.html` behavior;
- generated output metadata in `.distributor.json`;
- tests for source-only, HTML-only, and source-plus-HTML destinations.
### Slice 5: S3 backend
Deliver:
- S3-compatible backend;
- endpoint/bucket/prefix/region/force-path-style config;
- credential handling via environment or standard mechanisms;
- object listing, read, write, and narrow delete operations;
- dry-run and publish coverage using fake or local-compatible test strategy.
### Slice 6: SSH/SFTP backend
Deliver:
- native SFTP backend;
- `uri` and `path` config;
- documented authentication behavior;
- read/write/list/delete operations;
- adapter tests or documented integration test strategy.
### Slice 7: No-op notification stage and future extension seam
Deliver:
- no-op notifier wired into orchestration;
- clear internal contract for future email/ntfy adapters;
- no user-facing notification behavior beyond no-op unless implemented.
## Deferred Ideas
The following are intentionally out of MVP unless separately accepted in a later roadmap:
- email, ntfy, Gotify, or other real notification adapters;
- RSS/Atom feed generation;
- PDF generation;
- web UI;
- full-text search;
- dynamic plugin loading;
- arbitrary transform chains;
- workflow DAGs;
- producer execution;
- complex templating/theming;
- bidirectional sync;
- backup semantics.
## Key Invariants
- Producer apps own source bundle creation.
- `distributor` owns destination publication state.
- Source `manifest.json` is not copied as destination state.
- Destination `.distributor.json` is the managed sentinel.
- One pipeline has one source and one or more destinations.
- Transform and publish policy are destination-specific.
- Source files are canonical; HTML is derived.
- Destructive replacement is allowed only inside managed destination bundle paths.
- Core logic must be testable without real S3, SSH, or remote services.