325 lines
16 KiB
Markdown
325 lines
16 KiB
Markdown
# Architecture
|
||
|
||
This document defines the development principles for `distributor`. It is inward-facing: developers and LLM coding agents should use it to preserve the project’s shape, boundaries, and invariants as the code evolves.
|
||
|
||
## Project Scope
|
||
|
||
`distributor` is a domain-agnostic report bundle distribution tool.
|
||
|
||
Producer applications generate manifested bundles. `distributor` discovers those bundles, validates them, optionally derives publication artifacts such as HTML, and publishes selected source and generated artifacts to one or more configured destinations.
|
||
|
||
`distributor` does not generate domain reports, interpret domain-specific report content, run producer pipelines, edit reports, or act as a CMS. Weather reports, D&D recaps, calendar summaries, email digests, and additional report types should all enter `distributor` through the same bundle contract.
|
||
|
||
## Project Shape
|
||
|
||
Default to a small, explicit, dependency-light Go application. Keep the design modular enough to test and change safely, but do not add abstraction unless it protects a real boundary or enables a real extension point.
|
||
|
||
Business logic should live outside CLI, transport, and external-adapter packages. The core application should reason in terms of pipelines, bundles, destination state, transforms, and publish plans—not S3 SDK calls, SFTP sessions, shell commands, or filesystem details.
|
||
|
||
The current core workflow is:
|
||
|
||
1. load configured pipelines;
|
||
2. open the source backend;
|
||
3. discover source bundles beneath the source root;
|
||
4. validate each source bundle and its `manifest.json`;
|
||
5. for each configured destination, inspect destination state;
|
||
6. compare source state to destination state;
|
||
7. build a publish plan;
|
||
8. optionally transform Markdown to HTML for that destination;
|
||
9. publish selected source and generated artifacts;
|
||
10. write `.distributor.json` as the destination sentinel/state file;
|
||
11. run the notification hook, which is a no-op in the MVP.
|
||
|
||
## Pipeline Model
|
||
|
||
A pipeline has exactly one source and one or more destinations.
|
||
|
||
The source is discovered and validated once. Each destination has independent backend configuration, publication policy, transform policy, replacement behavior, state, and notification behavior.
|
||
|
||
The pipeline model is fan-out by design:
|
||
|
||
```text
|
||
source bundle
|
||
-> destination A: source files only
|
||
-> destination B: HTML only
|
||
-> destination C: source files + HTML
|
||
```
|
||
|
||
Destination-specific behavior must not leak back into the source bundle contract. A producer should not need to know whether a bundle will be published to local storage, another storage backend, a static site, email, RSS, or another notification channel.
|
||
|
||
## Source Bundle Contract
|
||
|
||
A source bundle is a directory containing `manifest.json`.
|
||
|
||
`manifest.json` is the sole producer-to-`distributor` contract. `distributor` must not rely on producer-specific work directory layouts, filenames, metadata, or conventions outside the configured source root and the source manifest.
|
||
|
||
The MVP source manifest schema is intentionally minimal:
|
||
|
||
```json
|
||
{
|
||
"schema_version": 1,
|
||
"id": "weather.daily.brentwood.2026-05-30",
|
||
"digest": "sha256:...",
|
||
"created": "2026-05-30T11:10:00Z",
|
||
"files": [
|
||
{
|
||
"path": "report.md",
|
||
"sha256": "sha256:...",
|
||
"size": 12345
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
Required fields:
|
||
|
||
- `schema_version`: source manifest schema version. MVP value: `1`.
|
||
- `id`: stable bundle identifier.
|
||
- `digest`: SHA-256 digest for the listed files.
|
||
- `created`: RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
|
||
- `files`: ordered file list.
|
||
- `files[].path`: relative path beneath the bundle root.
|
||
- `files[].sha256`: SHA-256 digest of the file contents.
|
||
- `files[].size`: file size in bytes.
|
||
|
||
Bundle paths must be relative, clean, and confined to the bundle root. Paths must not be absolute, empty, contain `..` path traversal, or otherwise escape the bundle root. Symlink handling must be explicit; unless documented otherwise, source bundle symlinks should be rejected.
|
||
|
||
Before processing a bundle, `distributor` must validate file existence, file size, each file SHA-256, and the bundle digest. Digest mismatch must fail in the MVP before any destination writes occur.
|
||
|
||
The source manifest should remain minimal. Routing, destination selection, publication format, credentials, static-site layout, notification recipients, and destination-specific metadata belong in `distributor` configuration and destination state, not in producer manifests.
|
||
|
||
## Destination State Contract
|
||
|
||
`manifest.json` from the source bundle is not copied to destinations as destination state.
|
||
|
||
Each destination bundle path is managed by `.distributor.json`. This file is both the destination sentinel and the destination state record. It records:
|
||
|
||
- `distributor` state schema version;
|
||
- pipeline id;
|
||
- destination id;
|
||
- publication timestamp;
|
||
- the normalized source manifest used for publication;
|
||
- metadata for copied source outputs;
|
||
- metadata for generated outputs, such as HTML files;
|
||
- any additional metadata required by `distributor`.
|
||
|
||
A representative destination state file is:
|
||
|
||
```json
|
||
{
|
||
"schema_version": 1,
|
||
"distributor_version": "0.1.0",
|
||
"pipeline_id": "weather-daily",
|
||
"destination_id": "static-html",
|
||
"published_at": "2026-05-30T11:12:00Z",
|
||
"source": {
|
||
"manifest": {
|
||
"schema_version": 1,
|
||
"id": "weather.daily.brentwood.2026-05-30",
|
||
"digest": "sha256:...",
|
||
"created": "2026-05-30T11:10:00Z",
|
||
"files": [
|
||
{
|
||
"path": "report.md",
|
||
"sha256": "sha256:...",
|
||
"size": 12345
|
||
}
|
||
]
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"path": "report.html",
|
||
"kind": "generated",
|
||
"source_path": "report.md",
|
||
"transform": "markdown_to_html",
|
||
"sha256": "sha256:...",
|
||
"size": 23456
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
Destination comparison rules are based on `.distributor.json`:
|
||
|
||
- No `.distributor.json`: publish normally only if the destination bundle path is empty.
|
||
- Existing state embeds the same normalized source manifest: skip as already published.
|
||
- Existing state has the same source id and an older source `created`: replace, subject to destructive-operation safety rules.
|
||
- Existing state has the same source id and a newer source `created`: skip because the destination is newer than the source.
|
||
- Existing state has the same source id and same `created` but different digest: fail as a conflict.
|
||
- Existing state has a different source id: fail as a conflict.
|
||
|
||
## Publication and Transform Policy
|
||
|
||
Source files are canonical bundle artifacts. Transform outputs are derived publication artifacts.
|
||
|
||
Transforms are configured per destination. A destination may receive source files, generated HTML files, or both.
|
||
|
||
The MVP supports only Markdown-to-HTML transformation. HTML generation must not mutate the source bundle. Generated outputs must be deterministic from the source bundle and destination transform configuration, and must be recorded in `.distributor.json`.
|
||
|
||
The application should distinguish:
|
||
|
||
- transform policy: how derived files are generated;
|
||
- publish policy: which source and generated files a destination receives.
|
||
|
||
For example, one destination may publish source files only as a long-term archive, while another publishes HTML only as a static site.
|
||
|
||
## Backend Abstraction
|
||
|
||
Sources and destinations use the same storage abstraction. Current runtime execution uses the local filesystem backend. Additional storage backends should be peer implementations behind the same interface, and any backend-specific execution limitation must be documented.
|
||
|
||
Application logic must interact with storage through internal backend interfaces. Backend-specific behavior belongs in adapter packages. Pipeline, bundle, state, publish, and transform packages must not import service-specific or filesystem adapter implementation details.
|
||
|
||
Adapters should be thin. Backend adapters should implement storage operations and translate backend-specific errors, but should not make bundle comparison, transform, routing, or replacement decisions.
|
||
|
||
Remote file-transfer support should prefer native protocol implementations over shelling out, unless a later design document records a reason to differ.
|
||
|
||
## Dependency Policy
|
||
|
||
Prefer the Go standard library where practical.
|
||
|
||
Use external dependencies only when justified by correctness, security, interoperability, or substantial complexity reduction. Good reasons include YAML parsing, S3-compatible storage integration, SSH/SFTP integration, and Markdown rendering.
|
||
|
||
Avoid dependencies for small conveniences. Do not let external dependency types leak across internal package boundaries unless the dependency is itself the explicit public contract of that package.
|
||
|
||
## Package Layout
|
||
|
||
Use this current layout unless the project has a documented reason to differ:
|
||
|
||
- `cmd/distributor`: application entrypoint only.
|
||
- `internal/app`: application orchestration and top-level use cases.
|
||
- `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
|
||
- `internal/config`: configuration structs, defaults, loading, precedence, and validation.
|
||
- `internal/bundle`: source manifest parsing, source bundle discovery, source digest validation, and source bundle model.
|
||
- `internal/state`: `.distributor.json` parsing, validation, comparison, and output metadata.
|
||
- `internal/storage`: backend interfaces, shared path/resource types, backend registry, and storage errors.
|
||
- `internal/adapters/local`: local filesystem backend.
|
||
- `internal/transform`: transform interfaces, registry, planning, and shared transform models.
|
||
- `internal/transform/markdown`: Markdown-to-HTML implementation.
|
||
- `internal/publish`: destination planning, reconciliation, safety checks, and publish execution.
|
||
- `internal/notify`: notification interface and MVP no-op notifier.
|
||
- `internal/logging`: logging setup and shared logging helpers.
|
||
|
||
New storage adapters should live under `internal/adapters/<name>` and stay thin.
|
||
|
||
Package-private implementation constants may live near the package that owns them, preferably in `constants.go` when useful.
|
||
|
||
## Configuration
|
||
|
||
Centralize configuration loading, processing, precedence, defaults, and validation in `internal/config`.
|
||
|
||
The goal is to make configuration discoverable and avoid implicit or hidden operational values. User-visible defaults and cross-package operational defaults should be defined in `internal/config/defaults.go`.
|
||
|
||
Unless documented otherwise, precedence is:
|
||
|
||
1. CLI flags
|
||
2. environment variables
|
||
3. configuration file
|
||
4. built-in defaults
|
||
|
||
Prefer YAML configuration. Config files should be discovered at `/usr/local/etc/distributor/config.yml`, with a CLI override via `--config`.
|
||
|
||
Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables, secret files, SSH agent usage, standard AWS credential mechanisms, or explicitly named environment variable references for secrets.
|
||
|
||
Pipeline configuration should express:
|
||
|
||
- pipeline id;
|
||
- one source backend;
|
||
- one or more destinations;
|
||
- per-destination publish policy;
|
||
- per-destination transform policy;
|
||
- validation behavior;
|
||
- destination conflict/replacement behavior.
|
||
|
||
## Modules, Stages, and Registries
|
||
|
||
Each major stage should have an explicit input/output contract:
|
||
|
||
- source discovery;
|
||
- source validation;
|
||
- destination state inspection;
|
||
- destination comparison;
|
||
- transform planning/execution;
|
||
- publish planning;
|
||
- publish execution;
|
||
- notification.
|
||
|
||
If users can select backends, transforms, notifiers, or renderers, selection should go through a registry or equivalent mechanism rather than scattered conditionals.
|
||
|
||
The orchestrator should be able to plan, dry-run, and execute configured pipelines. Dry-run behavior should be first-class because the application may delete, overwrite, or publish files.
|
||
|
||
## Embedded Assets
|
||
|
||
Store embedded JSON schemas, Markdown templates, HTML templates, CSS, and similar assets as separate files, not inline string literals, unless there is a strong reason otherwise.
|
||
|
||
## Errors and Logging
|
||
|
||
Errors should be actionable and preserve context. Wrap errors with operation, pipeline id, destination id, backend, path, bundle id, and resource context where useful. CLI code should convert internal errors into concise user-facing messages.
|
||
|
||
Errors and logs must not expose secrets.
|
||
|
||
Use structured logging where practical. Logs should describe discovery, validation, planned actions, skipped transfers, conflicts, replacements, external calls, retries, and failure causes, but should not include large report contents by default.
|
||
|
||
Skip and no-op decisions should be logged at an appropriate level so operators can distinguish successful publication from intentional no-op behavior.
|
||
|
||
## Context, Timeouts, and Cancellation
|
||
|
||
Long-running operations should accept `context.Context`. Storage operations, service requests, transforms, and multi-step workflows should respect cancellation and timeouts.
|
||
|
||
## State, Files, and Safety
|
||
|
||
If the application writes durable state, writes should be atomic where practical. Multi-step workflows should preserve enough state to support inspection, retry, or resume after failure.
|
||
|
||
Code that deletes, moves, or overwrites files must use narrow, explicit paths. Avoid broad parent-directory operations. Cleanup that can cause data loss must be opt-in.
|
||
|
||
`distributor` must never perform broad deletion against a configured source root or destination root. Destructive replacement may occur only inside a resolved destination bundle path when a valid `.distributor.json` confirms that the path is distributor-managed.
|
||
|
||
Replacement must be narrow, logged, test-covered, and configurable. Prefer deleting files recorded in `.distributor.json` and known generated outputs rather than blindly deleting parent directories. Backend implementations must guard against path traversal, prefix confusion, and accidental root deletion.
|
||
|
||
Where practical, publish operations should use staging paths or temporary objects and promote them into place only after validation and transform steps succeed.
|
||
|
||
## Testing
|
||
|
||
Core logic should be testable without real external services. Use fakes, fixtures, or local test doubles for adapters where practical.
|
||
|
||
Config examples should be load-tested. Important CLI workflows should have parser or command tests. Component contracts should have focused tests that do not require running the full application unless end-to-end coverage is intentional.
|
||
|
||
Important tests include:
|
||
|
||
- source manifest parsing and validation;
|
||
- file size and SHA-256 validation;
|
||
- bundle digest validation;
|
||
- source bundle discovery beneath a source root;
|
||
- relative path safety and path traversal rejection;
|
||
- destination `.distributor.json` parsing and comparison;
|
||
- same/older/newer/conflict publish decisions;
|
||
- destructive replacement safety checks;
|
||
- transform output planning and metadata recording;
|
||
- dry-run output;
|
||
- local backend behavior with temporary directories;
|
||
- fake backend behavior for storage-facing core logic.
|
||
|
||
## Documentation
|
||
|
||
Documentation should follow the project documentation policy. Keep user docs focused on implemented behavior. Put future, planned, or aspirational work only under `docs/roadmap/`.
|
||
|
||
When changing architecture, config, CLI behavior, adapters, manifest/state contracts, transform behavior, publish behavior, or component contracts, update the relevant docs and examples in the same change.
|
||
|
||
The source manifest and destination `.distributor.json` schemas should have canonical documentation once implemented. Example configs should be valid and load-tested where practical.
|
||
|
||
## Non-Goals
|
||
|
||
`distributor` is not:
|
||
|
||
- a report generator;
|
||
- a domain-specific weather, D&D, calendar, or email summarization app;
|
||
- a workflow DAG engine;
|
||
- a CMS;
|
||
- a web authoring interface;
|
||
- a full-text search service;
|
||
- a general-purpose file synchronization tool;
|
||
- a backup system;
|
||
- a notification platform.
|
||
|
||
Additional notification, feed, template, or transform behavior must preserve the core bundle-distribution boundary.
|