eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Added initial MVP roadmap documentation and implementation plan

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-30 19:24:54 -05:00
parent 74709b2e0d
commit 47c9c5c0d1
7 changed files with 2658 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

270
docs/policy/architecture.md
View File

@@ -1,18 +1,184 @@
# Architecture
This document defines the development principles for this Go project. It is inward-facing: developers and LLM coding agents should use it to preserve the project’s shape, boundaries, and invariants as the code evolves.
This document defines the development principles for `distributor`. It is inward-facing: developers and LLM coding agents should use it to preserve the project’s shape, boundaries, and invariants as the code evolves.
## Project Scope
`distributor` is a domain-agnostic report bundle distribution tool.
Producer applications generate manifested bundles. `distributor` discovers those bundles, validates them, optionally derives publication artifacts such as HTML, and publishes selected source and generated artifacts to one or more configured destinations.
`distributor` does not generate domain reports, interpret domain-specific report content, run producer pipelines, edit reports, or act as a CMS. Weather reports, D&D recaps, calendar summaries, email digests, and future report types should all enter `distributor` through the same bundle contract.
## Project Shape
Default to a small, explicit, dependency-light Go application. Keep the design modular enough to test and change safely, but do not add abstraction unless it protects a real boundary or enables a real extension point.
Business/domain logic should live outside CLI, transport, and external-adapter packages.
Business logic should live outside CLI, transport, and external-adapter packages. The core application should reason in terms of pipelines, bundles, destination state, transforms, and publish plans—not S3 SDK calls, SFTP sessions, shell commands, or filesystem details.
The core workflow is:
1. load configured pipelines;
2. open the source backend;
3. discover source bundles beneath the source root;
4. validate each source bundle and its `manifest.json`;
5. for each configured destination, inspect destination state;
6. compare source state to destination state;
7. build a publish plan;
8. optionally transform Markdown to HTML for that destination;
9. publish selected source and generated artifacts;
10. write `.distributor.json` as the destination sentinel/state file;
11. run the notification stage, which is a no-op in the MVP.
## Pipeline Model
A pipeline has exactly one source and one or more destinations.
The source is discovered and validated once. Each destination has independent backend configuration, publication policy, transform policy, replacement behavior, state, and future notification behavior.
The pipeline model is fan-out by design:
```text
source bundle
-> destination A: source files only
-> destination B: HTML only
-> destination C: source files + HTML
```
Destination-specific behavior must not leak back into the source bundle contract. A producer should not need to know whether a bundle will be published to local storage, SSH/SFTP, S3, a static site, email, RSS, or a future notification channel.
## Source Bundle Contract
A source bundle is a directory containing `manifest.json`.
`manifest.json` is the sole producer-to-`distributor` contract. `distributor` must not rely on producer-specific work directory layouts, filenames, metadata, or conventions outside the configured source root and the source manifest.
The MVP source manifest schema is intentionally minimal:
```json
{
"schema_version": 1,
"id": "weather.daily.brentwood.2026-05-30",
"digest": "sha256:...",
"created": "2026-05-30T11:10:00Z",
"files": [
{
"path": "report.md",
"sha256": "sha256:...",
"size": 12345
}
]
}
```
Required fields:
- `schema_version`: source manifest schema version. MVP value: `1`.
- `id`: stable bundle identifier.
- `digest`: SHA-256 digest for the listed files.
- `created`: RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
- `files`: ordered file list.
- `files[].path`: relative path beneath the bundle root.
- `files[].sha256`: SHA-256 digest of the file contents.
- `files[].size`: file size in bytes.
Bundle paths must be relative, clean, and confined to the bundle root. Paths must not be absolute, empty, contain `..` path traversal, or otherwise escape the bundle root. Symlink handling must be explicit; unless documented otherwise, source bundle symlinks should be rejected.
Before processing a bundle, `distributor` must validate file existence, file size, each file SHA-256, and the bundle digest. Digest mismatch must fail in the MVP before any destination writes occur.
The source manifest should remain minimal. Routing, destination selection, publication format, credentials, static-site layout, notification recipients, and destination-specific metadata belong in `distributor` configuration and destination state, not in producer manifests.
## Destination State Contract
`manifest.json` from the source bundle is not copied to destinations as destination state.
Each destination bundle path is managed by `.distributor.json`. This file is both the destination sentinel and the destination state record. It records:
- `distributor` state schema version;
- pipeline id;
- destination id;
- publication timestamp;
- the normalized source manifest used for publication;
- metadata for copied source outputs;
- metadata for generated outputs, such as HTML files;
- any additional metadata required by `distributor`.
A representative destination state file is:
```json
{
"schema_version": 1,
"distributor_version": "0.1.0",
"pipeline_id": "weather-daily",
"destination_id": "static-html",
"published_at": "2026-05-30T11:12:00Z",
"source": {
"manifest": {
"schema_version": 1,
"id": "weather.daily.brentwood.2026-05-30",
"digest": "sha256:...",
"created": "2026-05-30T11:10:00Z",
"files": [
{
"path": "report.md",
"sha256": "sha256:...",
"size": 12345
}
]
}
},
"outputs": [
{
"path": "report.html",
"kind": "generated",
"source_path": "report.md",
"transform": "markdown_to_html",
"sha256": "sha256:...",
"size": 23456
}
]
}
```
Destination comparison rules are based on `.distributor.json`:
- No `.distributor.json`: publish normally only if the destination bundle path is empty.
- Existing state embeds the same normalized source manifest: skip as already published.
- Existing state has the same source id and an older source `created`: replace, subject to destructive-operation safety rules.
- Existing state has the same source id and a newer source `created`: skip because the destination is newer than the source.
- Existing state has the same source id and same `created` but different digest: fail as a conflict.
- Existing state has a different source id: fail as a conflict.
## Publication and Transform Policy
Source files are canonical bundle artifacts. Transform outputs are derived publication artifacts.
Transforms are configured per destination. A destination may receive source files, generated HTML files, or both.
The MVP supports only Markdown-to-HTML transformation. HTML generation must not mutate the source bundle. Generated outputs must be deterministic from the source bundle and destination transform configuration, and must be recorded in `.distributor.json`.
The application should distinguish:
- transform policy: how derived files are generated;
- publish policy: which source and generated files a destination receives.
For example, one destination may publish source files only as a long-term archive, while another publishes HTML only as a static site.
## Backend Abstraction
Sources and destinations use the same storage abstraction. Local filesystem, SSH/SFTP, and S3-compatible object storage are peer backends. Any backend may appear as a source or a destination unless a specific limitation is documented.
Application logic must interact with storage through internal backend interfaces. Backend-specific behavior belongs in adapter packages. Pipeline, bundle, state, publish, and transform packages must not import S3, SSH/SFTP, or filesystem adapter implementation details.
Adapters should be thin. Backend adapters should implement storage operations and translate backend-specific errors, but should not make bundle comparison, transform, routing, or replacement decisions.
SSH support should prefer a native SFTP implementation over shelling out to `ssh`, `scp`, or `rsync`, unless a later design document records a reason to differ.
## Dependency Policy
Prefer the Go standard library where practical.
Use external dependencies only when justified by correctness, security, interoperability, or substantial complexity reduction. Good reasons include complex security-sensitive behavior, such as HTML sanitization, or widely used de facto standards, such as YAML parsing.
Use external dependencies only when justified by correctness, security, interoperability, or substantial complexity reduction. Good reasons include YAML parsing, S3-compatible storage integration, SSH/SFTP integration, and Markdown rendering.
Avoid dependencies for small conveniences. Do not let external dependency types leak across internal package boundaries unless the dependency is itself the explicit public contract of that package.
@@ -20,12 +186,21 @@ Avoid dependencies for small conveniences. Do not let external dependency types
Use this layout unless the project has a documented reason to differ:
- `cmd/distributor`: application entrypoint only.
- `internal/app`: application orchestration and top-level use cases.
- `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
- `internal/config`: configuration structs, defaults, loading, precedence, and validation.
- `internal/adapters/<name>`: adapters for external CLIs, APIs, databases, object stores, or libraries.
- `internal/api`: HTTP API handlers and request/response types, when the application exposes an HTTP API.
- `internal/transport/http`: HTTP client code, when the application calls HTTP services.
- `internal/bundle`: source manifest parsing, source bundle discovery, source digest validation, and source bundle model.
- `internal/state`: `.distributor.json` parsing, validation, comparison, and output metadata.
- `internal/storage`: backend interfaces, shared path/resource types, backend registry, and storage errors.
- `internal/adapters/local`: local filesystem backend.
- `internal/adapters/ssh`: SSH/SFTP backend.
- `internal/adapters/s3`: S3-compatible object storage backend.
- `internal/transform`: transform interfaces, registry, planning, and shared transform models.
- `internal/transform/markdown`: Markdown-to-HTML implementation.
- `internal/publish`: destination planning, reconciliation, safety checks, and publish execution.
- `internal/notify`: notification interface and MVP no-op notifier.
- `internal/logging`: logging setup and shared logging helpers.
Package-private implementation constants may live near the package that owns them, preferably in `constants.go` when useful.
@@ -42,41 +217,54 @@ Unless documented otherwise, precedence is:
3. configuration file
4. built-in defaults
Prefer YAML configuration unless the project has a strong reason to use another format. Config files should be discovered at `/usr/local/etc/<app_name>/config.yml`, with a CLI override via `--config`.
Prefer YAML configuration. Config files should be discovered at `/usr/local/etc/distributor/config.yml`, with a CLI override via `--config`.
Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets.
Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables, secret files, SSH agent usage, standard AWS credential mechanisms, or explicitly named environment variable references for secrets.
## Adapters and External Integrations
Pipeline configuration should express:
Use a hexagonal architecture style for external integrations.
External adapters belong under `internal/adapters/<name>`. If an adapter uses an external dependency, that dependency’s interface must not leak outside the adapter package. Other packages should interact only with the adapter’s API, so the dependency can be swapped, upgraded, or removed without touching unrelated code.
Adapters should be thin. Domain decisions belong in application/domain packages, not inside adapter glue.
- pipeline id;
- one source backend;
- one or more destinations;
- per-destination publish policy;
- per-destination transform policy;
- validation behavior;
- destination conflict/replacement behavior.
## Modules, Stages, and Registries
When the application has stages or modules, each major stage/module should live in its own package and have an explicit input/output contract.
Each major stage should have an explicit input/output contract:
The orchestrator should be able to compose, skip, resume, or run individual stages/modules when their prerequisites are satisfied. Ordering should be explicit: use a default sequence, dependency graph, or documented orchestration rule.
- source discovery;
- source validation;
- destination state inspection;
- destination comparison;
- transform planning/execution;
- publish planning;
- publish execution;
- notification.
If users can select modules, stages, validators, renderers, or adapters, selection should go through a registry or equivalent mechanism rather than scattered conditionals.
If users can select backends, transforms, notifiers, or future renderers, selection should go through a registry or equivalent mechanism rather than scattered conditionals.
The orchestrator should be able to plan, dry-run, and execute configured pipelines. Dry-run behavior should be first-class because the application may delete, overwrite, or publish files to remote storage.
## Embedded Assets
Store embedded JSON schemas, Markdown prompts, templates, and similar assets as separate files, not inline string literals, unless there is a strong reason otherwise.
Store embedded JSON schemas, Markdown templates, HTML templates, CSS, and similar assets as separate files, not inline string literals, unless there is a strong reason otherwise.
## Errors and Logging
Errors should be actionable and preserve context. Wrap errors with operation and path/resource context. CLI code should convert internal errors into concise user-facing messages.
Errors should be actionable and preserve context. Wrap errors with operation, pipeline id, destination id, backend, path, bundle id, and resource context where useful. CLI code should convert internal errors into concise user-facing messages.
Errors and logs must not expose secrets.
Use structured logging where practical. Logs should describe operations, paths, external calls, retries, and failure causes, but should not include large user data by default.
Use structured logging where practical. Logs should describe discovery, validation, planned actions, skipped transfers, conflicts, replacements, external calls, retries, and failure causes, but should not include large report contents by default.
Skip and no-op decisions should be logged at an appropriate level so operators can distinguish successful publication from intentional no-op behavior.
## Context, Timeouts, and Cancellation
Long-running operations should accept `context.Context`. External calls, subprocesses, HTTP requests, storage operations, and multi-stage workflows should respect cancellation and timeouts.
Long-running operations should accept `context.Context`. Storage operations, SSH/SFTP sessions, S3 requests, transforms, and multi-stage workflows should respect cancellation and timeouts.
## State, Files, and Safety
@@ -84,15 +272,53 @@ If the application writes durable state, writes should be atomic where practical
Code that deletes, moves, or overwrites files must use narrow, explicit paths. Avoid broad parent-directory operations. Cleanup that can cause data loss must be opt-in.
`distributor` must never perform broad deletion against a configured source root or destination root. Destructive replacement may occur only inside a resolved destination bundle path when a valid `.distributor.json` confirms that the path is distributor-managed.
Replacement must be narrow, logged, test-covered, and configurable. Prefer deleting files recorded in `.distributor.json` and known generated outputs rather than blindly deleting parent directories. Backend implementations must guard against path traversal, prefix confusion, and accidental root deletion.
Where practical, publish operations should use staging paths or temporary objects and promote them into place only after validation and transform steps succeed.
## Testing
Core logic should be testable without real external services. Use fakes, fixtures, or local test doubles for adapters where practical.
Config examples should be load-tested. Important CLI workflows should have parser or command tests. Stage/module contracts should have focused tests that do not require running the full application unless end-to-end coverage is intentional.
Important tests include:
- source manifest parsing and validation;
- file size and SHA-256 validation;
- bundle digest validation;
- source bundle discovery beneath a source root;
- relative path safety and path traversal rejection;
- destination `.distributor.json` parsing and comparison;
- same/older/newer/conflict publish decisions;
- destructive replacement safety checks;
- transform output planning and metadata recording;
- dry-run output;
- local backend behavior with temporary directories;
- fake backend behavior for S3 and SSH/SFTP-facing core logic.
## Documentation
Documentation should follow the project documentation policy. Keep user docs focused on implemented behavior. Put future, planned, or aspirational work only under `docs/roadmap/`.
When changing architecture, config, CLI behavior, adapters, or stage/module contracts, update the relevant docs and examples in the same change.
When changing architecture, config, CLI behavior, adapters, manifest/state contracts, transform behavior, publish behavior, or stage/module contracts, update the relevant docs and examples in the same change.
The source manifest and destination `.distributor.json` schemas should have canonical documentation once implemented. Example configs should be valid and load-tested where practical.
## Non-Goals
`distributor` is not:
- a report generator;
- a domain-specific weather, D&D, calendar, or email summarization app;
- a workflow DAG engine;
- a CMS;
- a web authoring interface;
- a full-text search service;
- a general-purpose file synchronization tool;
- a backup system;
- a notification platform.
It may later support notification adapters, RSS/feed generation, richer HTML templates, or additional transforms, but those features must preserve the core bundle-distribution boundary.