Added initial MVP roadmap documentation and implementation plan
This commit is contained in:
578
docs/roadmap/config.md
Normal file
578
docs/roadmap/config.md
Normal file
@@ -0,0 +1,578 @@
|
||||
# Distributor Configuration Roadmap
|
||||
|
||||
This roadmap defines the planned `config.yml` schema for the `distributor` MVP. The goal is to support one-to-many publication pipelines where each pipeline has one source and one or more destinations. Each destination independently controls backend configuration, publication outputs, transform behavior, and replacement policy.
|
||||
|
||||
## Configuration Goals
|
||||
|
||||
The MVP configuration should be:
|
||||
|
||||
- explicit enough to avoid hidden publication behavior;
|
||||
- compact enough for routine self-hosted use;
|
||||
- backend-agnostic at the pipeline layer;
|
||||
- capable of local, SSH/SFTP, and S3-compatible source and destination backends;
|
||||
- ready for future notification adapters without exposing a fake notification feature in the MVP.
|
||||
|
||||
## Top-Level Shape
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: weather-daily
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/weather
|
||||
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: weather/archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@example.com:22
|
||||
path: /srv/www/weather
|
||||
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
## Pipeline Fields
|
||||
|
||||
Each pipeline must include:
|
||||
|
||||
- `id`: Required stable pipeline identifier.
|
||||
- `source`: Required source backend configuration.
|
||||
- `destinations`: Required non-empty list of destination configurations.
|
||||
|
||||
Optional pipeline-level fields:
|
||||
|
||||
- `validation`: Source validation behavior.
|
||||
- Future: `notifications` or `notify`, when notification adapters are implemented.
|
||||
|
||||
A pipeline has exactly one source and one or more destinations.
|
||||
|
||||
## Pipeline ID Rules
|
||||
|
||||
`pipelines[].id` should:
|
||||
|
||||
- be required;
|
||||
- be unique across the config file;
|
||||
- be stable over time;
|
||||
- use a simple slug-like format, such as `weather-daily` or `dnd-session-recaps`.
|
||||
|
||||
Recommended validation:
|
||||
|
||||
```text
|
||||
^[a-zA-Z0-9][a-zA-Z0-9._-]*$
|
||||
```
|
||||
|
||||
## Source Configuration
|
||||
|
||||
`source` defines the source root where bundles are discovered.
|
||||
|
||||
The source backend may be:
|
||||
|
||||
- `local`;
|
||||
- `ssh`;
|
||||
- `s3`.
|
||||
|
||||
The source is scanned for `manifest.json` files beneath the configured root.
|
||||
|
||||
### Local source
|
||||
|
||||
```yaml
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/weather
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: local`
|
||||
- `path`
|
||||
|
||||
### SSH source
|
||||
|
||||
```yaml
|
||||
source:
|
||||
backend: ssh
|
||||
uri: ssh://reports@example.com:22
|
||||
path: /var/spool/distributor/weather
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: ssh`
|
||||
- `uri`
|
||||
- `path`
|
||||
|
||||
Recommended authentication behavior:
|
||||
|
||||
- use SSH agent by default;
|
||||
- use local known_hosts validation by default;
|
||||
- support optional key file configuration later if needed;
|
||||
- do not require passwords in YAML.
|
||||
|
||||
Optional future fields:
|
||||
|
||||
```yaml
|
||||
known_hosts: /home/user/.ssh/known_hosts
|
||||
key_file: /home/user/.ssh/id_ed25519
|
||||
```
|
||||
|
||||
### S3 source
|
||||
|
||||
```yaml
|
||||
source:
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: incoming/weather
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: s3`
|
||||
- `endpoint`
|
||||
- `bucket`
|
||||
|
||||
Optional fields:
|
||||
|
||||
- `prefix`
|
||||
- `region`
|
||||
- `force_path_style`
|
||||
- `credentials`
|
||||
|
||||
Credential configuration should prefer environment variables or standard SDK behavior over literal secrets in config.
|
||||
|
||||
## Destination Configuration
|
||||
|
||||
Each destination defines one publication target for a pipeline.
|
||||
|
||||
Required destination fields:
|
||||
|
||||
- `id`
|
||||
- `backend`
|
||||
- backend-specific location fields;
|
||||
- `publish`
|
||||
|
||||
Optional destination fields:
|
||||
|
||||
- `transform`
|
||||
- `transfer`
|
||||
|
||||
Each destination is independently planned and published. A destination may receive source files, generated HTML, or both.
|
||||
|
||||
## Destination ID Rules
|
||||
|
||||
`destinations[].id` should:
|
||||
|
||||
- be required;
|
||||
- be unique within the containing pipeline;
|
||||
- be stable over time;
|
||||
- use a slug-like format.
|
||||
|
||||
Recommended examples:
|
||||
|
||||
- `markdown-archive`
|
||||
- `static-site`
|
||||
- `full-mirror`
|
||||
|
||||
## Local Destination
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: local-static
|
||||
backend: local
|
||||
path: /srv/www/reports
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: local`
|
||||
- `path`
|
||||
- `publish`
|
||||
|
||||
## SSH Destination
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@example.com:22
|
||||
path: /srv/www/weather
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: ssh`
|
||||
- `uri`
|
||||
- `path`
|
||||
- `publish`
|
||||
|
||||
The MVP should use a native SFTP implementation rather than shelling out to `ssh`, `scp`, or `rsync`.
|
||||
|
||||
## S3 Destination
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: weather/archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: s3`
|
||||
- `endpoint`
|
||||
- `bucket`
|
||||
- `publish`
|
||||
|
||||
Optional fields:
|
||||
|
||||
- `prefix`
|
||||
- `region`
|
||||
- `force_path_style`
|
||||
- `credentials`
|
||||
|
||||
## Backend Configuration Normalization
|
||||
|
||||
The config loader should normalize backend configuration into internal source and destination backend specs. Pipeline logic should not branch on backend-specific fields.
|
||||
|
||||
Validation should catch:
|
||||
|
||||
- missing backend names;
|
||||
- unsupported backend names;
|
||||
- missing backend-specific required fields;
|
||||
- duplicate pipeline IDs;
|
||||
- duplicate destination IDs within a pipeline;
|
||||
- empty destination lists;
|
||||
- invalid policy values.
|
||||
|
||||
## Publication Policy
|
||||
|
||||
`publish` controls which categories of files are written to a destination.
|
||||
|
||||
```yaml
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
```
|
||||
|
||||
Fields:
|
||||
|
||||
- `source`: Publish source artifacts listed in `manifest.json`.
|
||||
- `html`: Publish HTML files generated from Markdown source artifacts.
|
||||
|
||||
At least one of `source` or `html` must be true.
|
||||
|
||||
Recommended defaults:
|
||||
|
||||
```yaml
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
```
|
||||
|
||||
No implicit HTML transformation should occur. When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `mode: sidecar` are required for the MVP.
|
||||
|
||||
## Transform Configuration
|
||||
|
||||
For MVP, the only supported transform is Markdown to HTML.
|
||||
|
||||
```yaml
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
Fields:
|
||||
|
||||
- `enabled`: Whether Markdown-to-HTML transform is enabled.
|
||||
- `mode`: Output mode. MVP value: `sidecar`.
|
||||
|
||||
MVP `sidecar` behavior:
|
||||
|
||||
- each listed Markdown source file generates an HTML file with the same base path and `.html` extension;
|
||||
- `report.md` generates `report.html`;
|
||||
- generated files are destination publication artifacts;
|
||||
- source bundles are not mutated.
|
||||
|
||||
MVP defaulting:
|
||||
|
||||
- If `publish.html` is false, transform may be omitted.
|
||||
- If `publish.html` is true and `transform.markdown_to_html` is omitted or disabled, config validation must fail.
|
||||
|
||||
## Validation Policy
|
||||
|
||||
Pipeline-level validation is intentionally narrow in the MVP.
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
```
|
||||
|
||||
Supported value:
|
||||
|
||||
- `fail`
|
||||
|
||||
Default:
|
||||
|
||||
```yaml
|
||||
on_digest_mismatch: fail
|
||||
```
|
||||
|
||||
Validation should happen before any destination writes. Warning-only digest mismatch handling is deferred and must be rejected if configured.
|
||||
|
||||
## Transfer Policy
|
||||
|
||||
Destination-level transfer policy controls behavior after inspecting `.distributor.json` at the destination bundle path.
|
||||
|
||||
```yaml
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
Supported fields:
|
||||
|
||||
- `on_destination_same`
|
||||
- `on_destination_older`
|
||||
- `on_destination_newer`
|
||||
- `on_conflict`
|
||||
|
||||
Recommended supported values:
|
||||
|
||||
- `skip`
|
||||
- `replace`
|
||||
- `fail`
|
||||
|
||||
Recommended MVP defaults:
|
||||
|
||||
```yaml
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
Safety rule:
|
||||
|
||||
- `replace` must never perform broad deletion against a destination root.
|
||||
- `replace` may only operate within a resolved destination bundle path and should delete only files recorded in existing `.distributor.json.outputs` plus `.distributor.json` where practical.
|
||||
- Unmanaged non-empty destination paths fail in the MVP. Force or unmanaged overwrite configuration is deferred.
|
||||
|
||||
## Path Mapping
|
||||
|
||||
MVP path mapping is fixed:
|
||||
|
||||
```text
|
||||
destination bundle path = destination root + source relative bundle path
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
source root: /var/spool/reports
|
||||
source bundle: /var/spool/reports/weather/daily/brentwood/2026-05-30
|
||||
relative bundle path: weather/daily/brentwood/2026-05-30
|
||||
|
||||
destination root: /srv/www/reports
|
||||
destination bundle path: /srv/www/reports/weather/daily/brentwood/2026-05-30
|
||||
```
|
||||
|
||||
Future config may support explicit path mapping, but MVP should not.
|
||||
|
||||
## Dry Run Configuration and CLI Behavior
|
||||
|
||||
Dry-run should be a CLI flag rather than a persistent config setting.
|
||||
|
||||
```bash
|
||||
distributor run --config config.yml --dry-run
|
||||
```
|
||||
|
||||
Dry-run should report:
|
||||
|
||||
- pipeline ID;
|
||||
- source backend;
|
||||
- destination ID;
|
||||
- destination backend;
|
||||
- discovered bundle ID;
|
||||
- relative bundle path;
|
||||
- planned action;
|
||||
- reason;
|
||||
- transform outputs that would be generated;
|
||||
- files that would be written or deleted.
|
||||
|
||||
## Example: Weather Pipeline
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: weather-daily
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/weather
|
||||
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.maximumdirect.net
|
||||
bucket: reports
|
||||
prefix: weather/archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@web.maximumdirect.net:22
|
||||
path: /srv/www/weather
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
## Example: D&D Recap Pipeline
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: dnd-session-recaps
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/dnd/session-recaps
|
||||
|
||||
destinations:
|
||||
- id: private-markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.maximumdirect.net
|
||||
bucket: reports
|
||||
prefix: dnd/session-recaps
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
|
||||
- id: private-html-site
|
||||
backend: local
|
||||
path: /srv/www/private/dnd/session-recaps
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
## Future Notification Configuration
|
||||
|
||||
Notification should not be exposed as a functional MVP feature unless an adapter exists.
|
||||
|
||||
The internal pipeline may include a no-op notification stage. Future config may look like:
|
||||
|
||||
```yaml
|
||||
notifications:
|
||||
- id: weather-email
|
||||
backend: email
|
||||
after_destinations:
|
||||
- static-site
|
||||
subject: "Weather report published"
|
||||
```
|
||||
|
||||
Future notification policy should require:
|
||||
|
||||
- notification after successful relevant publication;
|
||||
- idempotency by source manifest id and digest;
|
||||
- no duplicate notification unless explicitly forced.
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
1. Define config structs for pipelines, sources, destinations, validation, publish, transform, and transfer policies.
|
||||
2. Implement config loading and strict validation.
|
||||
3. Implement backend-specific config validation for local, SSH, and S3.
|
||||
4. Implement defaulting for validation and transfer policies.
|
||||
5. Require explicit transform configuration when `publish.html` is true.
|
||||
6. Add example config fixtures for local-to-local, local-to-SSH, local-to-S3, and fan-out scenarios.
|
||||
7. Connect config to backend registry and publish planner.
|
||||
8. Add `--pipeline` filtering for targeted runs.
|
||||
9. Add `--dry-run` output that reflects the resolved config and planned actions.
|
||||
|
||||
## Deferred Configuration
|
||||
|
||||
The following configuration ideas are intentionally outside the MVP:
|
||||
|
||||
- warning-only digest mismatch handling;
|
||||
- unmanaged destination overwrite flags;
|
||||
- force replacement of destinations with different source ids.
|
||||
390
docs/roadmap/contracts.md
Normal file
390
docs/roadmap/contracts.md
Normal file
@@ -0,0 +1,390 @@
|
||||
# Distributor Contracts Roadmap
|
||||
|
||||
This roadmap defines the contracts that `distributor` should implement before or alongside the MVP. The goal is to make bundle validation, destination state, digest verification, and safe replacement deterministic and testable before backend-specific publication behavior is layered on top.
|
||||
|
||||
## Purpose
|
||||
|
||||
`distributor` publishes manifested report bundles produced by other applications. Producer applications own domain-specific report generation. `distributor` owns validation, optional transformation, destination publication, and destination state.
|
||||
|
||||
The MVP contract has two durable files:
|
||||
|
||||
- `manifest.json`: source-owned bundle manifest produced by the upstream application.
|
||||
- `.distributor.json`: destination-owned publication state written by `distributor`.
|
||||
|
||||
`manifest.json` is not copied to the destination as destination state. Instead, `.distributor.json` records the normalized source manifest, generated output metadata, and distributor-owned publication metadata.
|
||||
|
||||
## Terminology
|
||||
|
||||
- **Source root**: Configured root path for a pipeline source.
|
||||
- **Bundle root**: Directory beneath the source root that contains `manifest.json`.
|
||||
- **Relative bundle path**: Bundle root path relative to the source root.
|
||||
- **Destination root**: Configured root path or prefix for a destination.
|
||||
- **Destination bundle path**: Destination root plus the relative bundle path, unless a future mapping option overrides that behavior.
|
||||
- **Source artifact**: File listed in the source `manifest.json`.
|
||||
- **Generated artifact**: File created by `distributor`, such as an HTML file derived from Markdown.
|
||||
- **Destination state**: `.distributor.json` at the destination bundle path.
|
||||
|
||||
## Source Bundle Contract
|
||||
|
||||
A source bundle is a directory containing a `manifest.json` file. For MVP, the manifest schema is intentionally minimal.
|
||||
|
||||
### Required `manifest.json` fields
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"id": "weather.daily.brentwood.2026-05-30",
|
||||
"digest": "sha256:...",
|
||||
"created": "2026-05-30T11:10:00Z",
|
||||
"files": [
|
||||
{
|
||||
"path": "report.md",
|
||||
"sha256": "sha256:...",
|
||||
"size": 12345
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Required top-level fields:
|
||||
|
||||
- `schema_version`: Source manifest schema version. MVP value: `1`.
|
||||
- `id`: Stable bundle identifier. Required, non-empty string.
|
||||
- `digest`: Bundle digest. Required, lowercase `sha256:<64 hex>` string.
|
||||
- `created`: Bundle creation timestamp. Required RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
|
||||
- `files`: Non-empty array of file objects.
|
||||
|
||||
Required file fields:
|
||||
|
||||
- `path`: Relative path from bundle root to source artifact.
|
||||
- `sha256`: Per-file digest as lowercase `sha256:<64 hex>`.
|
||||
- `size`: File size in bytes.
|
||||
|
||||
No other source manifest fields are required for the MVP. Additional fields may be ignored unless later documented. Destination state records the normalized source manifest model, not raw unknown manifest fields.
|
||||
|
||||
## Source Path Safety Rules
|
||||
|
||||
For every `files[].path`:
|
||||
|
||||
- Path must be relative.
|
||||
- Path must not be empty.
|
||||
- Path must not contain `..` segments.
|
||||
- Path must not resolve outside the bundle root.
|
||||
- Path must use slash-separated logical paths in the manifest.
|
||||
- Absolute paths are invalid.
|
||||
- Symlinks should be rejected for MVP unless a later policy deliberately supports them.
|
||||
- `manifest.json` itself should not be listed as a source artifact.
|
||||
- `.distributor.json` should not be listed as a source artifact.
|
||||
- Duplicate logical file paths are invalid after path normalization.
|
||||
|
||||
The implementation should validate paths before reading file contents.
|
||||
|
||||
## Digest Contract
|
||||
|
||||
The MVP validates both per-file digests and the bundle digest.
|
||||
|
||||
### Per-file digest
|
||||
|
||||
For each file listed in `files`, compute:
|
||||
|
||||
```text
|
||||
sha256(file bytes)
|
||||
```
|
||||
|
||||
The computed digest must match `files[].sha256`.
|
||||
|
||||
The actual file size must match `files[].size`.
|
||||
|
||||
### Bundle digest
|
||||
|
||||
The bundle digest is computed from the listed file records in the listed order. The canonical algorithm is:
|
||||
|
||||
1. For each file listed in `files`, in order:
|
||||
- validate the file path;
|
||||
- compute the file SHA256;
|
||||
- determine the file size.
|
||||
2. Construct a canonical JSON array containing only:
|
||||
- `path`;
|
||||
- `sha256`;
|
||||
- `size`.
|
||||
3. Preserve the source manifest's file order.
|
||||
4. Encode the array deterministically with:
|
||||
- object fields in exactly this order: `path`, `sha256`, `size`;
|
||||
- no extra spaces;
|
||||
- no trailing newline;
|
||||
- lowercase `sha256:<64 hex>` digest strings.
|
||||
5. Compute `sha256(canonical JSON bytes)`.
|
||||
6. Compare the result to top-level `digest`.
|
||||
|
||||
Conceptual canonical payload:
|
||||
|
||||
```json
|
||||
[
|
||||
{"path":"report.md","sha256":"sha256:...","size":12345},
|
||||
{"path":"summary.txt","sha256":"sha256:...","size":234}
|
||||
]
|
||||
```
|
||||
|
||||
This avoids ambiguous concatenation of file bytes while keeping the manifest small.
|
||||
|
||||
Implementation fixtures should include at least one reference manifest and canonical payload with known per-file and bundle digests.
|
||||
|
||||
## Digest Mismatch Behavior
|
||||
|
||||
Digest validation is always fatal in the MVP. A digest mismatch must fail the affected bundle, pipeline, and run before any destination writes occur.
|
||||
|
||||
Warning-only digest behavior is intentionally deferred until a later roadmap accepts transitional ingestion semantics.
|
||||
|
||||
## Bundle Discovery Contract
|
||||
|
||||
A source path may contain one bundle or a tree of bundles. Discovery should scan beneath the configured source root for `manifest.json` files.
|
||||
|
||||
For each discovered manifest:
|
||||
|
||||
- Bundle root is the directory containing `manifest.json`.
|
||||
- Relative bundle path is computed relative to source root.
|
||||
- Destination bundle path is destination root plus relative bundle path, unless a future mapping option overrides it.
|
||||
|
||||
If nested manifests are found, the MVP should fail with a clear error unless a later policy defines nested-bundle semantics.
|
||||
|
||||
## Destination State Contract
|
||||
|
||||
Destinations are managed by `.distributor.json`, not by copying `manifest.json`.
|
||||
|
||||
A destination bundle path is considered distributor-managed only when it contains a valid `.distributor.json` written by `distributor`. Force or unmanaged-overwrite behavior is not part of the MVP.
|
||||
|
||||
### Required `.distributor.json` shape
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"distributor_version": "0.1.0",
|
||||
"pipeline_id": "weather-daily",
|
||||
"destination_id": "static-site",
|
||||
"published_at": "2026-05-30T11:12:00Z",
|
||||
"source": {
|
||||
"manifest": {
|
||||
"schema_version": 1,
|
||||
"id": "weather.daily.brentwood.2026-05-30",
|
||||
"digest": "sha256:...",
|
||||
"created": "2026-05-30T11:10:00Z",
|
||||
"files": [
|
||||
{
|
||||
"path": "report.md",
|
||||
"sha256": "sha256:...",
|
||||
"size": 12345
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"path": "report.html",
|
||||
"kind": "generated",
|
||||
"source_path": "report.md",
|
||||
"transform": "markdown_to_html",
|
||||
"sha256": "sha256:...",
|
||||
"size": 23456
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `schema_version`: Destination state schema version. MVP value: `1`.
|
||||
- `distributor_version`: Optional diagnostic distributor version. It must not affect source comparison.
|
||||
- `pipeline_id`: Pipeline that produced the destination publication.
|
||||
- `destination_id`: Destination within the pipeline.
|
||||
- `published_at`: RFC3339 timestamp.
|
||||
- `source.manifest`: Normalized source manifest model used for this publication.
|
||||
- `outputs`: Array of files written by `distributor` for this destination.
|
||||
|
||||
Required output fields:
|
||||
|
||||
- `path`: Destination-relative output path within the destination bundle path.
|
||||
- `kind`: `source` or `generated`.
|
||||
- `source_path`: Source artifact path that produced this output. For copied source files, this should equal `path` unless renamed by a future feature.
|
||||
- `transform`: Transform identifier for generated files. Empty or omitted may be allowed for copied source files.
|
||||
- `sha256`: Output file digest as lowercase `sha256:<64 hex>`.
|
||||
- `size`: Output file size in bytes.
|
||||
|
||||
## Destination Comparison Rules
|
||||
|
||||
Destination comparison uses `.distributor.json`, not destination `manifest.json`.
|
||||
|
||||
For a source bundle and destination bundle path:
|
||||
|
||||
### No `.distributor.json`
|
||||
|
||||
If no `.distributor.json` exists and the destination path is empty:
|
||||
|
||||
- Publish normally.
|
||||
|
||||
If no `.distributor.json` exists and the destination path is non-empty:
|
||||
|
||||
- Fail as unmanaged content.
|
||||
|
||||
Local destination paths are empty when the destination bundle directory does not exist or exists with no entries. S3-compatible destination prefixes are empty when no objects exist below the destination bundle prefix, ignoring objects outside that exact prefix.
|
||||
|
||||
### Same source manifest
|
||||
|
||||
If `.distributor.json` exists and `source.manifest` exactly matches the current normalized source manifest:
|
||||
|
||||
- Skip as already published.
|
||||
|
||||
### Same source id, destination older
|
||||
|
||||
If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is older than the source `created`:
|
||||
|
||||
- Replace destination contents, subject to replacement safety rules.
|
||||
|
||||
### Same source id, destination newer
|
||||
|
||||
If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is newer than the source `created`:
|
||||
|
||||
- Skip and log that destination is newer than source.
|
||||
|
||||
### Same source id and created, different digest
|
||||
|
||||
If `.distributor.json` exists, `source.manifest.id` and `created` match but `digest` differs:
|
||||
|
||||
- Fail as a conflict.
|
||||
|
||||
### Different source id
|
||||
|
||||
If `.distributor.json` exists and `source.manifest.id` differs from the source manifest `id`:
|
||||
|
||||
- Fail as a conflict.
|
||||
|
||||
## Replacement Safety Rules
|
||||
|
||||
Replacement is destructive and must be narrow.
|
||||
|
||||
`distributor` must never perform broad deletion against a configured destination root.
|
||||
|
||||
Replacement may occur only at a resolved destination bundle path when:
|
||||
|
||||
- a valid `.distributor.json` exists at that destination bundle path; and
|
||||
- the state identifies the path as distributor-managed; and
|
||||
- the replacement decision follows the destination comparison rules.
|
||||
|
||||
For MVP, replacement should delete only known managed outputs where practical:
|
||||
|
||||
- files listed in existing `.distributor.json.outputs`;
|
||||
- existing `.distributor.json`;
|
||||
- empty directories created by those files, where applicable for filesystem-like backends.
|
||||
|
||||
For S3, replacement should delete only objects under the destination bundle prefix that are listed in `.distributor.json.outputs` plus `.distributor.json`, unless a later managed-prefix deletion policy is explicitly implemented.
|
||||
|
||||
Before any write, planned destination output paths must be checked for collisions. For example, if `report.md` is copied as a source artifact and Markdown transformation would also generate `report.html`, but `report.html` is already a source artifact or another generated output, planning must fail before writing.
|
||||
|
||||
## Transform Output Contract
|
||||
|
||||
Source files are canonical. Generated files are derived publication artifacts.
|
||||
|
||||
For MVP, the only supported transform is Markdown to HTML.
|
||||
|
||||
Recommended MVP behavior:
|
||||
|
||||
- Transform is configured per destination.
|
||||
- Markdown source files are files listed in `manifest.json` with `.md` extension.
|
||||
- Generated HTML files are sidecars by default.
|
||||
- `report.md` generates `report.html`.
|
||||
- Source bundles are never mutated.
|
||||
- Generated outputs are recorded in `.distributor.json.outputs`.
|
||||
- Raw HTML embedded in Markdown is escaped or disabled by default for the MVP to keep generated output deterministic and conservative.
|
||||
|
||||
Future versions may add templates, `index.html`, CSS assets, email-safe HTML, and per-file transform selection.
|
||||
|
||||
## Destination Output Contract
|
||||
|
||||
Each destination chooses which categories of files it receives.
|
||||
|
||||
MVP categories:
|
||||
|
||||
- `source`: copied source artifacts listed in `manifest.json`.
|
||||
- `html`: generated HTML derived from Markdown source artifacts.
|
||||
|
||||
Examples:
|
||||
|
||||
- Markdown archive: `source: true`, `html: false`.
|
||||
- Static HTML site: `source: false`, `html: true`.
|
||||
- Full mirror: `source: true`, `html: true`.
|
||||
|
||||
Every file written to the destination must be represented in `.distributor.json.outputs`.
|
||||
|
||||
## Atomicity and Partial Failure
|
||||
|
||||
The MVP should prefer staging and promotion where backend semantics permit it.
|
||||
|
||||
Minimum behavior:
|
||||
|
||||
- Validate source before writing destination files.
|
||||
- Do not write `.distributor.json` until all configured destination outputs are successfully written.
|
||||
- If publication fails before `.distributor.json` is written, the destination must not be treated as successfully published on a later run.
|
||||
- Local publication must use staging or equivalent cleanup behavior so a failed write does not leave a confusing unmanaged destination bundle path.
|
||||
- Later cleanup may remove orphaned files, but MVP correctness should rely on `.distributor.json` as the success marker.
|
||||
|
||||
## Example Source Bundle
|
||||
|
||||
```text
|
||||
weather/daily/brentwood/2026-05-30/
|
||||
manifest.json
|
||||
report.md
|
||||
summary.txt
|
||||
```
|
||||
|
||||
Example manifest:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"id": "weather.daily.brentwood.2026-05-30",
|
||||
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"created": "2026-05-30T11:10:00Z",
|
||||
"files": [
|
||||
{
|
||||
"path": "report.md",
|
||||
"sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
|
||||
"size": 12345
|
||||
},
|
||||
{
|
||||
"path": "summary.txt",
|
||||
"sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222",
|
||||
"size": 234
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Example Destination Bundle: HTML Only
|
||||
|
||||
```text
|
||||
weather/daily/brentwood/2026-05-30/
|
||||
report.html
|
||||
.distributor.json
|
||||
```
|
||||
|
||||
## Example Destination Bundle: Source Archive
|
||||
|
||||
```text
|
||||
weather/daily/brentwood/2026-05-30/
|
||||
report.md
|
||||
summary.txt
|
||||
.distributor.json
|
||||
```
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
1. Define Go structs for source manifest and destination state.
|
||||
2. Implement source path validation.
|
||||
3. Implement per-file SHA256 and size validation.
|
||||
4. Implement canonical bundle digest validation.
|
||||
5. Implement source bundle discovery beneath a source root.
|
||||
6. Implement `.distributor.json` parsing and validation.
|
||||
7. Implement destination comparison rules.
|
||||
8. Implement replacement safety checks.
|
||||
9. Add fixture bundles for valid, invalid, duplicate path, older, newer, same, and conflict scenarios.
|
||||
10. Add reference canonical digest fixtures.
|
||||
11. Use the contract layer from the publish pipeline and backend adapters.
|
||||
808
docs/roadmap/implementation.md
Normal file
808
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,808 @@
|
||||
# Distributor Implementation Roadmap
|
||||
|
||||
This roadmap defines a staged implementation plan for the `distributor` MVP. Each stage is intended to map cleanly to one Codex implementation prompt.
|
||||
|
||||
The roadmap assumes the project includes these planning documents before implementation begins:
|
||||
|
||||
- `docs/policy/architecture.md`
|
||||
- `docs/policy/documentation.md`
|
||||
- `docs/roadmap/packages.md`
|
||||
- `docs/roadmap/contracts.md`
|
||||
- `docs/roadmap/config.md`
|
||||
|
||||
The MVP goal is a domain-agnostic bundle distributor that discovers source bundles, validates `manifest.json`, optionally transforms Markdown to HTML, publishes selected outputs to one or more destinations, and records destination state in `.distributor.json`.
|
||||
|
||||
## Global Implementation Rules
|
||||
|
||||
All stages should preserve these invariants:
|
||||
|
||||
- Producer applications own source bundle creation.
|
||||
- `distributor` owns validation, transformation, publication, destination state, and future notification hooks.
|
||||
- Source bundle state is defined by `manifest.json`.
|
||||
- Destination publication state is defined by `.distributor.json`.
|
||||
- `manifest.json` is not copied to the destination as destination state.
|
||||
- Pipelines have exactly one source and one or more destinations.
|
||||
- Transform and publish policy are destination-specific.
|
||||
- Destructive replacement is allowed only inside a managed destination bundle path. Unsafe force or unmanaged overwrite behavior is deferred.
|
||||
- Dry-run behavior should be implemented before broad remote write behavior.
|
||||
- Config, bundle, state, publish planning, storage adapters, transforms, and CLI wiring should remain separate packages.
|
||||
|
||||
Unless a stage explicitly says otherwise, each implementation prompt should:
|
||||
|
||||
1. read the project policy and roadmap documents;
|
||||
2. implement only the current stage;
|
||||
3. add or update tests for the current stage;
|
||||
4. run the relevant test suite;
|
||||
5. update documentation only when the implemented behavior now exists;
|
||||
6. avoid implementing future roadmap stages early.
|
||||
|
||||
## Stage 1: Project Skeleton, CLI Shell, and Baseline Tooling
|
||||
|
||||
### Goal
|
||||
|
||||
Create the initial Go application structure and a minimal executable `distributor` command with no business behavior beyond version/help output and placeholder commands.
|
||||
|
||||
### Scope
|
||||
|
||||
Implement the accepted package skeleton from `docs/roadmap/packages.md` at the level needed for compilation.
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
cmd/distributor/main.go
|
||||
internal/cli/
|
||||
internal/app/
|
||||
internal/config/
|
||||
internal/logging/
|
||||
```
|
||||
|
||||
Initial CLI commands:
|
||||
|
||||
- `distributor --help`
|
||||
- `distributor version`
|
||||
- `distributor run`
|
||||
- `distributor validate`
|
||||
- `distributor inspect`
|
||||
|
||||
At this stage, `run`, `validate`, and `inspect` may return clear “not implemented” errors, but the command structure should be present.
|
||||
|
||||
### Notes
|
||||
|
||||
Prefer a small CLI dependency only if the project already standardizes on one. Otherwise, the standard library is acceptable for the first pass.
|
||||
|
||||
Add a version variable that can later be set at build time.
|
||||
|
||||
### Tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- command construction if testable;
|
||||
- version string behavior if exposed through a package;
|
||||
- basic package compilation.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- `go test ./...` passes.
|
||||
- `go run ./cmd/distributor --help` works.
|
||||
- `go run ./cmd/distributor version` works.
|
||||
- Placeholder operational commands fail clearly and intentionally.
|
||||
|
||||
## Stage 2: Config Schema, Loading, Defaults, and Validation
|
||||
|
||||
### Goal
|
||||
|
||||
Implement the MVP `config.yml` schema described in `docs/roadmap/config.md`.
|
||||
|
||||
### Scope
|
||||
|
||||
Create config structs for:
|
||||
|
||||
- top-level config;
|
||||
- pipelines;
|
||||
- source backend config;
|
||||
- destination backend config;
|
||||
- validation policy;
|
||||
- publish policy;
|
||||
- transform policy;
|
||||
- transfer/replacement policy;
|
||||
- backend-specific local, SSH, and S3 fields.
|
||||
|
||||
Support loading YAML from a file path.
|
||||
|
||||
Implement validation for:
|
||||
|
||||
- required top-level `pipelines`;
|
||||
- unique pipeline ids;
|
||||
- required pipeline `id`, `source`, and non-empty `destinations`;
|
||||
- unique destination ids within a pipeline;
|
||||
- supported backend names: `local`, `ssh`, `s3`;
|
||||
- required backend fields;
|
||||
- supported validation action: `fail`;
|
||||
- supported transfer actions;
|
||||
- valid `publish` policy;
|
||||
- valid Markdown-to-HTML transform config.
|
||||
|
||||
Default behavior should match `docs/roadmap/config.md`.
|
||||
|
||||
### CLI Integration
|
||||
|
||||
Add `--config` to `run`.
|
||||
|
||||
For this stage, `distributor run --config config.yml --dry-run` may only load and validate config, then print a concise summary of configured pipelines and destinations.
|
||||
|
||||
### Tests
|
||||
|
||||
Add unit tests for:
|
||||
|
||||
- valid minimal local-to-local config;
|
||||
- valid fan-out config;
|
||||
- valid local, SSH, and S3 backend configs;
|
||||
- duplicate pipeline ids;
|
||||
- duplicate destination ids;
|
||||
- missing required fields;
|
||||
- unsupported backend;
|
||||
- invalid transfer action;
|
||||
- invalid validation action, including `warn`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Config load/default/validate behavior is implemented and tested.
|
||||
- `distributor run --config <file> --dry-run` validates config and prints a summary.
|
||||
- No bundle discovery or publication occurs yet.
|
||||
|
||||
## Stage 3: Storage Abstraction, Local Backend, and Fake Backend
|
||||
|
||||
### Goal
|
||||
|
||||
Introduce the storage backend abstraction before bundle validation so source discovery, validation, and publication are backend-agnostic from the start.
|
||||
|
||||
### Scope
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
internal/storage/backend.go
|
||||
internal/storage/registry.go
|
||||
internal/storage/path.go
|
||||
internal/storage/errors.go
|
||||
internal/adapters/local/backend.go
|
||||
internal/storage/fake/
|
||||
```
|
||||
|
||||
Define backend operations needed by the MVP, including:
|
||||
|
||||
- read file;
|
||||
- write file;
|
||||
- test existence;
|
||||
- list files or tree entries;
|
||||
- read destination state file if present;
|
||||
- create directories/prefixes as needed;
|
||||
- delete explicit managed files safely;
|
||||
- write files using temporary/staged writes where practical.
|
||||
|
||||
The fake backend should exist for unit tests of config, bundle, state, and publish logic without real local, SSH, or S3 IO.
|
||||
|
||||
### Safety Requirements
|
||||
|
||||
The local backend must:
|
||||
|
||||
- clean and join paths safely;
|
||||
- reject path traversal;
|
||||
- reject unsafe destructive deletion requests;
|
||||
- avoid following symlinks for source bundle files unless explicitly supported;
|
||||
- avoid deleting configured roots;
|
||||
- classify destination bundle emptiness deterministically.
|
||||
|
||||
### Tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- backend read/write/list/exists behavior;
|
||||
- safe path joining;
|
||||
- traversal rejection;
|
||||
- symlink rejection for source reads;
|
||||
- explicit-file deletion guard behavior;
|
||||
- local destination emptiness detection;
|
||||
- fake backend behavior sufficient for core package tests.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Local backend implements the storage interface.
|
||||
- Fake backend can support bundle and publish tests without external services.
|
||||
- `go test ./...` passes.
|
||||
- No SSH or S3 implementation exists yet.
|
||||
|
||||
## Stage 4: Source Bundle Manifest, Digest, Validation, and Discovery
|
||||
|
||||
### Goal
|
||||
|
||||
Implement the source bundle contract from `docs/roadmap/contracts.md` through the storage abstraction.
|
||||
|
||||
### Scope
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
internal/bundle/manifest.go
|
||||
internal/bundle/digest.go
|
||||
internal/bundle/validate.go
|
||||
internal/bundle/discover.go
|
||||
```
|
||||
|
||||
Implement:
|
||||
|
||||
- parsing `manifest.json`;
|
||||
- strict required field validation, including `schema_version: 1`;
|
||||
- RFC3339 `created` parsing;
|
||||
- lowercase `sha256:<64 hex>` digest validation;
|
||||
- source file path safety checks;
|
||||
- duplicate logical file path rejection;
|
||||
- per-file SHA256 validation;
|
||||
- per-file size validation;
|
||||
- bundle digest validation using the canonical ordered file-record algorithm;
|
||||
- deterministic storage-backed bundle discovery under a source root;
|
||||
- nested manifest detection and failure.
|
||||
|
||||
Discovery and validation should use `internal/storage` rather than direct `os` APIs. The local CLI path should be adapted to the local backend.
|
||||
|
||||
### CLI Integration
|
||||
|
||||
Implement:
|
||||
|
||||
```text
|
||||
distributor validate <path>
|
||||
distributor inspect <path>
|
||||
```
|
||||
|
||||
For local paths:
|
||||
|
||||
- `validate` should validate either a single bundle directory or a tree containing bundles.
|
||||
- `inspect` should print a concise normalized summary of discovered bundle ids, relative paths, created timestamps, digest values, and files.
|
||||
|
||||
### Tests
|
||||
|
||||
Add fixture bundles under a testdata directory.
|
||||
|
||||
Test:
|
||||
|
||||
- valid bundle;
|
||||
- invalid JSON;
|
||||
- missing required fields;
|
||||
- invalid schema version;
|
||||
- invalid timestamp;
|
||||
- invalid digest format;
|
||||
- unsafe file paths;
|
||||
- duplicate normalized file paths;
|
||||
- missing files;
|
||||
- size mismatch;
|
||||
- per-file digest mismatch;
|
||||
- bundle digest mismatch;
|
||||
- canonical bundle digest reference fixture;
|
||||
- multiple discovered bundles in deterministic order;
|
||||
- nested manifests fail.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Storage-backed bundle validation is deterministic and well-tested.
|
||||
- `distributor validate <path>` works for local bundle fixtures.
|
||||
- `distributor inspect <path>` works for local bundle fixtures.
|
||||
- No destination publication occurs yet.
|
||||
|
||||
## Stage 5: Destination State Contract and Comparison Logic
|
||||
|
||||
### Goal
|
||||
|
||||
Implement `.distributor.json` parsing, validation, and source-to-destination comparison.
|
||||
|
||||
### Scope
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
internal/state/distributor.go
|
||||
internal/state/compare.go
|
||||
internal/state/validate.go
|
||||
```
|
||||
|
||||
Implement the destination state schema from `docs/roadmap/contracts.md`, including:
|
||||
|
||||
- `schema_version`;
|
||||
- optional `distributor_version`;
|
||||
- `pipeline_id`;
|
||||
- `destination_id`;
|
||||
- `published_at`;
|
||||
- embedded normalized source manifest;
|
||||
- outputs array;
|
||||
- output file metadata.
|
||||
|
||||
Implement comparison outcomes:
|
||||
|
||||
- destination absent;
|
||||
- destination unmanaged/non-empty;
|
||||
- same source manifest;
|
||||
- same source id, destination older;
|
||||
- same source id, destination newer;
|
||||
- same source id and same created but different digest;
|
||||
- different source id;
|
||||
- invalid destination state.
|
||||
|
||||
### Tests
|
||||
|
||||
Add unit tests for every comparison outcome.
|
||||
|
||||
Test validation for:
|
||||
|
||||
- valid state;
|
||||
- missing fields;
|
||||
- invalid schema version;
|
||||
- invalid embedded source manifest;
|
||||
- invalid output metadata;
|
||||
- malformed published timestamp.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Destination state can be parsed and validated independently.
|
||||
- Source manifest to destination state comparison is deterministic and fully tested.
|
||||
- No publication execution occurs yet.
|
||||
|
||||
## Stage 6: Publish Planning, Dry-Run, and Local-to-Local Publication Without Transform
|
||||
|
||||
### Goal
|
||||
|
||||
Implement the core publish planner and execute local-to-local publication for source files only.
|
||||
|
||||
### Scope
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
internal/publish/plan.go
|
||||
internal/publish/reconcile.go
|
||||
internal/publish/safety.go
|
||||
internal/publish/output.go
|
||||
internal/publish/execute.go
|
||||
```
|
||||
|
||||
Implement planning for one source bundle to one destination based on:
|
||||
|
||||
- source manifest;
|
||||
- destination config;
|
||||
- publish policy;
|
||||
- transfer policy;
|
||||
- existing `.distributor.json`;
|
||||
- destination path state.
|
||||
|
||||
Actions should include:
|
||||
|
||||
- publish new;
|
||||
- replace older destination;
|
||||
- skip same;
|
||||
- skip destination newer;
|
||||
- fail conflict;
|
||||
- fail unmanaged destination.
|
||||
|
||||
Implement local-to-local execution for `publish.source: true` and `publish.html: false`.
|
||||
|
||||
Execution should:
|
||||
|
||||
- copy listed source files selected by publish policy;
|
||||
- write `.distributor.json` with copied source output metadata;
|
||||
- avoid copying source `manifest.json` as destination state;
|
||||
- preserve relative bundle paths from source root beneath destination root;
|
||||
- detect destination output collisions before writing;
|
||||
- use staging or equivalent cleanup behavior for local writes;
|
||||
- support fan-out to multiple local destinations;
|
||||
- support dry-run without writes.
|
||||
|
||||
### CLI Integration
|
||||
|
||||
`distributor run --config <file>` should now execute local-to-local pipelines when configured.
|
||||
|
||||
`--dry-run` should print the planned action for each discovered bundle and destination.
|
||||
|
||||
### Tests
|
||||
|
||||
Add integration-style tests using temp directories for:
|
||||
|
||||
- new local publication;
|
||||
- no-op when destination state matches;
|
||||
- replacement when destination state is older;
|
||||
- skip when destination state is newer;
|
||||
- fail on conflict;
|
||||
- fail on unmanaged non-empty destination;
|
||||
- fail on output path collision;
|
||||
- fan-out from one source to two local destinations;
|
||||
- failed local write does not leave a destination that appears unmanaged on retry;
|
||||
- dry-run performs no writes;
|
||||
- `.distributor.json` is written correctly.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Local-to-local source-file publication works end to end.
|
||||
- Dry-run produces meaningful planned actions.
|
||||
- Destination state is authoritative.
|
||||
- No Markdown-to-HTML transform exists yet.
|
||||
|
||||
## Stage 7: Markdown-to-HTML Transform and Destination-Specific Publish Policy
|
||||
|
||||
### Goal
|
||||
|
||||
Add MVP Markdown-to-HTML transformation and destination-specific source/html output selection.
|
||||
|
||||
### Scope
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
internal/transform/transform.go
|
||||
internal/transform/registry.go
|
||||
internal/transform/plan.go
|
||||
internal/transform/markdown/markdown.go
|
||||
internal/transform/markdown/template.go
|
||||
```
|
||||
|
||||
Implement only:
|
||||
|
||||
```yaml
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
MVP sidecar behavior:
|
||||
|
||||
- for each listed source artifact ending in `.md`, generate a same-directory `.html` sidecar;
|
||||
- preserve the original Markdown file unchanged;
|
||||
- do not generate HTML for non-Markdown files;
|
||||
- escape or disable raw HTML embedded in Markdown;
|
||||
- fail before writing when generated output paths collide with copied source outputs or other generated outputs;
|
||||
- record generated output metadata in `.distributor.json`;
|
||||
- if `publish.source: false`, do not publish source files;
|
||||
- if `publish.html: true`, publish generated HTML files;
|
||||
- if `publish.html: true` but transform is disabled or no Markdown files exist, fail with a clear error unless config later defines another behavior.
|
||||
|
||||
Use a well-maintained Markdown renderer. Keep HTML templating minimal and deterministic.
|
||||
|
||||
### Tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- Markdown sidecar generation;
|
||||
- source-only destination;
|
||||
- HTML-only destination;
|
||||
- source-plus-HTML destination;
|
||||
- no mutation of source bundle;
|
||||
- generated output metadata in `.distributor.json`;
|
||||
- failure when HTML publication is requested without transform support;
|
||||
- failure when generated HTML collides with a source artifact path;
|
||||
- raw HTML in Markdown is escaped or disabled consistently;
|
||||
- deterministic output for a fixture Markdown file.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Local-to-local publication supports source-only, HTML-only, and source-plus-HTML destinations.
|
||||
- Generated outputs are recorded in destination state.
|
||||
- Dry-run reports transform outputs that would be generated.
|
||||
|
||||
## Stage 8: No-Op Notification Stage and Pipeline Orchestration Polish
|
||||
|
||||
### Goal
|
||||
|
||||
Add the internal no-op notification stage and polish orchestration around per-destination outcomes.
|
||||
|
||||
### Scope
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
internal/notify/notify.go
|
||||
internal/notify/noop.go
|
||||
```
|
||||
|
||||
Integrate a no-op notifier after successful publication/skip handling as appropriate.
|
||||
|
||||
Clarify orchestration behavior when one destination fails. For MVP, fan-out should be deterministic and sequential. Continue planning and reporting later destinations where safe, but return non-zero if any destination fails.
|
||||
|
||||
Improve run summary output:
|
||||
|
||||
- pipeline id;
|
||||
- source backend;
|
||||
- discovered bundle count;
|
||||
- destination ids;
|
||||
- action per bundle/destination;
|
||||
- final status.
|
||||
|
||||
### Tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- notifier is invoked at the expected orchestration point where testable;
|
||||
- pipeline failure when a destination fails;
|
||||
- run summary contains meaningful status information;
|
||||
- dry-run does not invoke write-side effects.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- The pipeline shape includes notification as an internal no-op stage.
|
||||
- Run output is useful for unattended operation logs.
|
||||
- Local MVP behavior remains passing.
|
||||
|
||||
## Stage 9: Native SSH/SFTP Backend
|
||||
|
||||
### Goal
|
||||
|
||||
Implement SSH/SFTP storage backend support for sources and destinations.
|
||||
|
||||
### Scope
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
internal/adapters/ssh/backend.go
|
||||
internal/adapters/ssh/config.go
|
||||
```
|
||||
|
||||
Implement the storage backend interface over native SSH/SFTP.
|
||||
|
||||
Required config:
|
||||
|
||||
```yaml
|
||||
backend: ssh
|
||||
uri: ssh://user@example.com:22
|
||||
path: /remote/root
|
||||
```
|
||||
|
||||
Authentication expectations:
|
||||
|
||||
- prefer SSH agent by default;
|
||||
- use known_hosts validation by default where practical;
|
||||
- do not require passwords in YAML;
|
||||
- optional key-file support may be implemented if straightforward, but should not distract from agent-based auth.
|
||||
|
||||
Support SSH/SFTP backend as both source and destination:
|
||||
|
||||
- local -> ssh;
|
||||
- ssh -> local;
|
||||
- ssh -> ssh where feasible through staging or streaming.
|
||||
|
||||
### Safety Requirements
|
||||
|
||||
The SSH backend must enforce the same logical path safety rules as the local backend.
|
||||
|
||||
Deletion must remain limited to managed destination bundle paths guarded by valid `.distributor.json`.
|
||||
|
||||
### Tests
|
||||
|
||||
Unit-test path handling and config validation.
|
||||
|
||||
If practical, add integration tests that can be skipped unless an SSH test endpoint is configured through environment variables. Do not require a live SSH server for normal `go test ./...`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- SSH/SFTP backend compiles and satisfies the storage interface.
|
||||
- Backend config validation is tested.
|
||||
- Normal tests do not depend on a live SSH server.
|
||||
- At least local-to-SSH and SSH-to-local flows are documented or manually testable.
|
||||
|
||||
## Stage 10: S3-Compatible Backend
|
||||
|
||||
### Goal
|
||||
|
||||
Implement S3-compatible backend support for sources and destinations.
|
||||
|
||||
### Scope
|
||||
|
||||
Create:
|
||||
|
||||
```text
|
||||
internal/adapters/s3/backend.go
|
||||
internal/adapters/s3/config.go
|
||||
```
|
||||
|
||||
Required config should align with `docs/roadmap/config.md`:
|
||||
|
||||
```yaml
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: some/prefix
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
Implement storage operations over S3 object keys:
|
||||
|
||||
- read object;
|
||||
- write object;
|
||||
- exists;
|
||||
- list prefix;
|
||||
- delete managed prefix or listed managed files;
|
||||
- read/write `.distributor.json`.
|
||||
|
||||
Set reasonable content types where available:
|
||||
|
||||
- `.md`: `text/markdown; charset=utf-8`;
|
||||
- `.html`: `text/html; charset=utf-8`;
|
||||
- `.json`: `application/json`;
|
||||
- `.txt`: `text/plain; charset=utf-8`.
|
||||
|
||||
Support S3 backend as both source and destination.
|
||||
|
||||
### Safety Requirements
|
||||
|
||||
Treat S3 prefixes as object trees. Do not assume real directories exist.
|
||||
|
||||
Deletion must be limited to destination bundle prefixes that are confirmed managed by `.distributor.json`.
|
||||
|
||||
### Tests
|
||||
|
||||
Add unit tests for:
|
||||
|
||||
- config validation;
|
||||
- key/prefix normalization;
|
||||
- content type selection;
|
||||
- path traversal rejection;
|
||||
- publish planning with S3 destination state fixtures.
|
||||
|
||||
If practical, add integration tests gated by environment variables or a local S3-compatible test service. Normal `go test ./...` must not require live S3 credentials.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- S3 backend compiles and satisfies the storage interface.
|
||||
- S3 source and destination flows are supported through the common pipeline path.
|
||||
- Normal tests do not require live S3.
|
||||
|
||||
## Stage 11: Cross-Backend End-to-End Coverage and Hardening
|
||||
|
||||
### Goal
|
||||
|
||||
Harden the MVP across backend combinations, destination policies, and failure cases.
|
||||
|
||||
### Scope
|
||||
|
||||
Add end-to-end coverage for representative scenarios:
|
||||
|
||||
- local source -> local archive destination;
|
||||
- local source -> local HTML destination;
|
||||
- local source -> two destinations with different publish policies;
|
||||
- local source -> SSH destination, where integration credentials exist;
|
||||
- local source -> S3 destination, where integration credentials exist;
|
||||
- S3 source -> local destination, where integration credentials exist;
|
||||
- SSH source -> local destination, where integration credentials exist.
|
||||
|
||||
Improve logging and error messages for:
|
||||
|
||||
- invalid config;
|
||||
- invalid source manifest;
|
||||
- digest mismatch;
|
||||
- destination conflict;
|
||||
- unmanaged destination path;
|
||||
- backend read/write/list failures;
|
||||
- transform failures.
|
||||
|
||||
Ensure all destructive paths have tests or explicit safeguards.
|
||||
|
||||
### Tests
|
||||
|
||||
Add or expand tests for:
|
||||
|
||||
- dry-run across multiple destinations;
|
||||
- partial failure behavior;
|
||||
- repeated run idempotency;
|
||||
- older/newer destination state behavior;
|
||||
- destination state output metadata accuracy;
|
||||
- generated HTML output metadata accuracy.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- MVP behavior is reliable across implemented backend types.
|
||||
- Error messages identify pipeline id, destination id, bundle id, and reason where practical.
|
||||
- Idempotent repeated runs behave as expected.
|
||||
|
||||
## Stage 12: User-Facing Documentation Sync
|
||||
|
||||
### Goal
|
||||
|
||||
Update documentation to reflect implemented MVP behavior.
|
||||
|
||||
### Scope
|
||||
|
||||
Following `docs/policy/documentation.md`, create or update user-facing documentation only for implemented features.
|
||||
|
||||
Likely docs:
|
||||
|
||||
```text
|
||||
README.md
|
||||
docs/config.md
|
||||
docs/cli.md
|
||||
docs/policy/architecture.md
|
||||
docs/internal/bundles.md
|
||||
docs/internal/backends.md
|
||||
```
|
||||
|
||||
Document:
|
||||
|
||||
- what `distributor` does;
|
||||
- bundle contract summary;
|
||||
- `.distributor.json` role;
|
||||
- example source bundle;
|
||||
- example local-to-local config;
|
||||
- example local-to-S3 config;
|
||||
- example local-to-SSH config;
|
||||
- `run`, `validate`, and `inspect` commands;
|
||||
- dry-run behavior;
|
||||
- replacement and safety rules;
|
||||
- Markdown-to-HTML transform behavior;
|
||||
- environment-variable credential handling.
|
||||
|
||||
Move roadmap material to historical/planning status only if your documentation policy allows it. Do not describe unimplemented notification adapters as available features.
|
||||
|
||||
### Tests
|
||||
|
||||
Run the full test suite.
|
||||
|
||||
If docs include command examples, verify that basic examples correspond to actual CLI behavior.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- User-facing docs describe the implemented MVP accurately.
|
||||
- Roadmap docs no longer masquerade as implemented behavior.
|
||||
- `go test ./...` passes.
|
||||
|
||||
## Stage 13: MVP Release Readiness Pass
|
||||
|
||||
### Goal
|
||||
|
||||
Perform a final pre-release quality pass.
|
||||
|
||||
### Scope
|
||||
|
||||
Review:
|
||||
|
||||
- package boundaries against `docs/policy/architecture.md`;
|
||||
- package layout against `docs/roadmap/packages.md`;
|
||||
- implemented contracts against `docs/roadmap/contracts.md`;
|
||||
- implemented config behavior against `docs/roadmap/config.md`;
|
||||
- docs against `docs/policy/documentation.md`;
|
||||
- destructive operation safety;
|
||||
- logs and errors for unattended operation;
|
||||
- command UX;
|
||||
- test coverage for core invariants.
|
||||
|
||||
Add any missing small tests or docs discovered during review.
|
||||
|
||||
Do not add new product features in this stage.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- MVP is ready to deploy against one real producer pipeline.
|
||||
- A dry-run can be performed safely against a real source and destination.
|
||||
- Repeated runs are idempotent.
|
||||
- Destructive replacement cannot occur outside managed destination bundle paths.
|
||||
- Final docs accurately reflect the application.
|
||||
|
||||
## Deferred Post-MVP Work
|
||||
|
||||
The following items are intentionally outside the MVP unless explicitly pulled into a later roadmap:
|
||||
|
||||
- email notifications;
|
||||
- ntfy/Gotify/Pushover notifications;
|
||||
- RSS/Atom feed generation;
|
||||
- static site index pages beyond sidecar HTML output;
|
||||
- templated HTML themes beyond a minimal deterministic template;
|
||||
- destination path remapping rules;
|
||||
- full plugin architecture;
|
||||
- web UI;
|
||||
- report editing;
|
||||
- producer pipeline execution;
|
||||
- database-backed state;
|
||||
- complex retry queues;
|
||||
- concurrent publication workers;
|
||||
- symlink support;
|
||||
- warning-only digest mismatch handling;
|
||||
- force or unmanaged-overwrite destination behavior;
|
||||
- password-based SSH authentication in YAML.
|
||||
629
docs/roadmap/packages.md
Normal file
629
docs/roadmap/packages.md
Normal file
@@ -0,0 +1,629 @@
|
||||
# Package Layout Roadmap
|
||||
|
||||
This roadmap defines the proposed package layout, boundaries, and implementation responsibilities for the `distributor` MVP.
|
||||
|
||||
`distributor` is expected to be a domain-agnostic bundle publisher. Producer applications emit source bundles containing `manifest.json`; `distributor` validates those bundles and publishes selected source and generated artifacts to one or more configured destinations.
|
||||
|
||||
This document is roadmap material. It describes the intended package design before implementation and should move into `docs/internal/` only after corresponding behavior exists.
|
||||
|
||||
## Accepted Package Layout
|
||||
|
||||
```text
|
||||
cmd/distributor/
|
||||
main.go
|
||||
|
||||
internal/app/
|
||||
app.go
|
||||
run.go
|
||||
pipeline.go
|
||||
|
||||
internal/cli/
|
||||
root.go
|
||||
run.go
|
||||
validate.go
|
||||
inspect.go
|
||||
|
||||
internal/config/
|
||||
config.go
|
||||
defaults.go
|
||||
load.go
|
||||
validate.go
|
||||
|
||||
internal/bundle/
|
||||
manifest.go
|
||||
digest.go
|
||||
validate.go
|
||||
discover.go
|
||||
|
||||
internal/state/
|
||||
distributor.go
|
||||
compare.go
|
||||
validate.go
|
||||
|
||||
internal/storage/
|
||||
backend.go
|
||||
registry.go
|
||||
path.go
|
||||
errors.go
|
||||
|
||||
internal/storage/fake/
|
||||
backend.go
|
||||
|
||||
internal/adapters/local/
|
||||
backend.go
|
||||
|
||||
internal/adapters/ssh/
|
||||
backend.go
|
||||
config.go
|
||||
|
||||
internal/adapters/s3/
|
||||
backend.go
|
||||
config.go
|
||||
|
||||
internal/transform/
|
||||
transform.go
|
||||
registry.go
|
||||
plan.go
|
||||
|
||||
internal/transform/markdown/
|
||||
markdown.go
|
||||
template.go
|
||||
|
||||
internal/publish/
|
||||
plan.go
|
||||
execute.go
|
||||
output.go
|
||||
reconcile.go
|
||||
safety.go
|
||||
|
||||
internal/notify/
|
||||
notify.go
|
||||
noop.go
|
||||
|
||||
internal/logging/
|
||||
logging.go
|
||||
```
|
||||
|
||||
## Package Responsibilities
|
||||
|
||||
### `cmd/distributor`
|
||||
|
||||
Application entrypoint only.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- call CLI execution;
|
||||
- translate process exit status;
|
||||
- avoid business logic.
|
||||
|
||||
Non-responsibilities:
|
||||
|
||||
- config loading;
|
||||
- backend construction;
|
||||
- bundle validation;
|
||||
- publish decisions.
|
||||
|
||||
### `internal/cli`
|
||||
|
||||
CLI command definitions, flags, argument parsing, and command wiring.
|
||||
|
||||
Expected MVP commands:
|
||||
|
||||
- `distributor run` — run configured pipelines.
|
||||
- `distributor run --dry-run` — plan without modifying destinations.
|
||||
- `distributor run --pipeline <id>` — run one configured pipeline.
|
||||
- `distributor validate <path>` — validate a source bundle or source tree where feasible.
|
||||
- `distributor inspect <path>` — inspect a bundle or destination state where feasible.
|
||||
|
||||
Boundaries:
|
||||
|
||||
- CLI should call `internal/app` use cases.
|
||||
- CLI should not parse manifests directly except through application APIs.
|
||||
- CLI should not import backend adapter implementation details unless only for registration side effects.
|
||||
|
||||
### `internal/app`
|
||||
|
||||
Application orchestration and top-level use cases.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- load and validate configuration;
|
||||
- construct configured pipelines;
|
||||
- build source and destination backends through registries;
|
||||
- orchestrate discovery, validation, planning, publishing, and notification;
|
||||
- coordinate dry-run output;
|
||||
- run destination fan-out deterministically and sequentially;
|
||||
- aggregate destination outcomes into run-level failure behavior.
|
||||
|
||||
Core orchestration shape:
|
||||
|
||||
```text
|
||||
for each selected pipeline:
|
||||
open source backend
|
||||
discover source bundles
|
||||
for each source bundle:
|
||||
validate source manifest and digest
|
||||
for each destination:
|
||||
inspect .distributor.json
|
||||
build publish plan
|
||||
transform as required by that destination
|
||||
execute publish plan unless dry-run
|
||||
run noop notifier
|
||||
```
|
||||
|
||||
Boundaries:
|
||||
|
||||
- `internal/app` composes packages but should not contain backend-specific logic.
|
||||
- Publish decisions should live in `internal/publish`, not inline in orchestration.
|
||||
- Destination state comparison should live in `internal/state` or `internal/publish`, not CLI code.
|
||||
|
||||
### `internal/config`
|
||||
|
||||
Configuration structs, defaults, loading, precedence, and validation.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- load `/usr/local/etc/distributor/config.yml` by default;
|
||||
- support `--config` override;
|
||||
- apply defaults;
|
||||
- validate required fields;
|
||||
- validate pipeline ids and destination ids;
|
||||
- validate backend-specific config shapes;
|
||||
- validate transform and publish policy combinations.
|
||||
|
||||
MVP config model:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: weather-daily
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/weather
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: weather/archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@example.com:22
|
||||
path: /srv/www/weather
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
Configuration principles:
|
||||
|
||||
- one source per pipeline;
|
||||
- one or more destinations per pipeline;
|
||||
- transforms are destination-specific;
|
||||
- publish policy is destination-specific;
|
||||
- secrets should use environment variables, secret files, SSH agent, or standard credential mechanisms rather than raw YAML values.
|
||||
|
||||
### `internal/bundle`
|
||||
|
||||
Source bundle contract and validation.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- parse source `manifest.json`;
|
||||
- represent source manifests and files;
|
||||
- discover bundle roots beneath a configured source root;
|
||||
- validate required manifest fields;
|
||||
- validate RFC3339 `created` values;
|
||||
- validate relative paths;
|
||||
- validate file existence, size, per-file SHA-256, and bundle digest;
|
||||
- expose normalized source bundle models to other packages.
|
||||
|
||||
Core types:
|
||||
|
||||
```go
|
||||
type Manifest struct {
|
||||
SchemaVersion int
|
||||
ID string
|
||||
Digest string
|
||||
Created time.Time
|
||||
Files []ManifestFile
|
||||
}
|
||||
|
||||
type ManifestFile struct {
|
||||
Path string
|
||||
SHA256 string
|
||||
Size int64
|
||||
}
|
||||
|
||||
type Bundle struct {
|
||||
RootRelativePath string
|
||||
Manifest Manifest
|
||||
}
|
||||
```
|
||||
|
||||
Boundaries:
|
||||
|
||||
- `internal/bundle` does not know about `.distributor.json`.
|
||||
- `internal/bundle` does not know about destinations, transforms, or notification.
|
||||
- `internal/bundle` may use the storage abstraction to read source files, but it should not import backend adapter packages.
|
||||
|
||||
### `internal/state`
|
||||
|
||||
Destination state contract for `.distributor.json`.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- parse `.distributor.json`;
|
||||
- validate destination state;
|
||||
- represent copied source outputs and generated outputs;
|
||||
- embed the source manifest used for publication;
|
||||
- compare destination state against a current source manifest;
|
||||
- classify destination state as same, older, newer, conflict, absent, invalid, or unmanaged.
|
||||
|
||||
Core types:
|
||||
|
||||
```go
|
||||
type DistributorState struct {
|
||||
SchemaVersion int
|
||||
DistributorVersion string
|
||||
PipelineID string
|
||||
DestinationID string
|
||||
PublishedAt time.Time
|
||||
Source SourceState
|
||||
Outputs []OutputFile
|
||||
}
|
||||
|
||||
type SourceState struct {
|
||||
Manifest bundle.Manifest
|
||||
}
|
||||
|
||||
type OutputFile struct {
|
||||
Path string
|
||||
Kind string // source | generated
|
||||
SourcePath string
|
||||
Transform string
|
||||
SHA256 string
|
||||
Size int64
|
||||
}
|
||||
```
|
||||
|
||||
Comparison rules:
|
||||
|
||||
- same source manifest: skip;
|
||||
- same source id, older destination source `created`: replace;
|
||||
- same source id, newer destination source `created`: skip;
|
||||
- same source id, same `created`, different digest: conflict;
|
||||
- different source id: conflict;
|
||||
- absent state: publish only if safe;
|
||||
- unmanaged non-empty path: fail.
|
||||
|
||||
Boundaries:
|
||||
|
||||
- `internal/state` owns destination state semantics, not publish execution.
|
||||
- `internal/state` should not know about S3, SSH/SFTP, local filesystem details, or Markdown rendering.
|
||||
|
||||
### `internal/storage`
|
||||
|
||||
Backend abstraction and shared storage types.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- define storage backend interfaces;
|
||||
- define object/file metadata types;
|
||||
- define path/prefix helpers;
|
||||
- define common storage errors;
|
||||
- provide backend registry mechanisms.
|
||||
- provide a fake backend for core package tests.
|
||||
|
||||
The core application should use storage interfaces such as:
|
||||
|
||||
```go
|
||||
type Backend interface {
|
||||
ReadFile(ctx context.Context, path string) ([]byte, error)
|
||||
WriteFile(ctx context.Context, path string, data []byte, opts WriteOptions) error
|
||||
Exists(ctx context.Context, path string) (bool, error)
|
||||
List(ctx context.Context, prefix string) ([]Entry, error)
|
||||
DeleteFiles(ctx context.Context, paths []string) error
|
||||
}
|
||||
```
|
||||
|
||||
Destructive APIs should remain narrow. Prefer deleting explicit files recorded in `.distributor.json` instead of broad recursive deletion.
|
||||
|
||||
Boundaries:
|
||||
|
||||
- `internal/storage` should not contain backend implementation details.
|
||||
- Adapter dependencies must not leak through storage interfaces.
|
||||
- The fake backend exists for tests and should not become an application runtime backend.
|
||||
|
||||
### `internal/adapters/local`
|
||||
|
||||
Local filesystem backend.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- implement `storage.Backend` for local paths;
|
||||
- clean and constrain paths;
|
||||
- perform safe reads/writes/listing/deletion;
|
||||
- use atomic writes where practical;
|
||||
- reject unsafe path traversal;
|
||||
- handle symlink policy explicitly.
|
||||
|
||||
Testing expectations:
|
||||
|
||||
- use temporary directories;
|
||||
- verify path traversal rejection;
|
||||
- verify write and delete safety.
|
||||
|
||||
### `internal/adapters/ssh`
|
||||
|
||||
SSH/SFTP backend.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- implement `storage.Backend` over SSH/SFTP;
|
||||
- support `uri` and `path` config;
|
||||
- prefer native SFTP implementation;
|
||||
- use SSH agent, key files, known hosts, or documented auth mechanisms;
|
||||
- avoid raw passwords in config unless explicitly designed and documented later;
|
||||
- translate SSH/SFTP errors into storage-level errors.
|
||||
|
||||
Testing expectations:
|
||||
|
||||
- core app tests should use fake backends;
|
||||
- adapter tests may use local test servers or targeted integration tests if practical;
|
||||
- do not require a real production SSH host for normal unit tests.
|
||||
|
||||
### `internal/adapters/s3`
|
||||
|
||||
S3-compatible object storage backend.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- implement `storage.Backend` over S3-compatible object storage;
|
||||
- support endpoint, bucket, prefix, region, and force-path-style configuration;
|
||||
- support standard credential mechanisms or explicit environment-variable references;
|
||||
- treat S3 as an object tree, not a filesystem;
|
||||
- set reasonable content types where practical;
|
||||
- guard against prefix/root deletion mistakes.
|
||||
|
||||
Testing expectations:
|
||||
|
||||
- core app tests should use fake backends;
|
||||
- adapter behavior may be tested through mocks, local S3-compatible services, or narrow integration tests;
|
||||
- config examples should avoid real secrets.
|
||||
|
||||
### `internal/transform`
|
||||
|
||||
Transform interfaces, registry, and transform planning.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- define transform interfaces;
|
||||
- register available transforms;
|
||||
- represent transform requests and outputs;
|
||||
- keep transform execution independent of destination backend details.
|
||||
|
||||
Boundaries:
|
||||
|
||||
- transforms operate on source bundle content and destination transform config;
|
||||
- transforms do not publish outputs;
|
||||
- transforms do not mutate source bundles;
|
||||
- transforms should return generated output metadata for `.distributor.json`.
|
||||
|
||||
### `internal/transform/markdown`
|
||||
|
||||
Markdown-to-HTML implementation.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- render listed Markdown files to HTML;
|
||||
- support MVP sidecar behavior, such as `report.md` -> `report.html`;
|
||||
- record generated output path, source path, transform name, SHA-256, and size;
|
||||
- optionally use embedded templates if needed.
|
||||
|
||||
MVP scope:
|
||||
|
||||
- Markdown to HTML only;
|
||||
- no PDF generation;
|
||||
- no email-specific HTML;
|
||||
- no complex theming unless required for basic output correctness.
|
||||
|
||||
### `internal/publish`
|
||||
|
||||
Destination planning, reconciliation, safety checks, and publish execution.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- inspect destination state;
|
||||
- plan destination action;
|
||||
- enforce destination conflict rules;
|
||||
- enforce destructive-operation safety rules;
|
||||
- detect output path collisions before writing;
|
||||
- combine source files and transform outputs according to destination publish policy;
|
||||
- write destination outputs;
|
||||
- write `.distributor.json`;
|
||||
- use staging or equivalent cleanup behavior where practical;
|
||||
- support dry-run planning;
|
||||
- report skipped, replaced, failed, and published actions.
|
||||
|
||||
Action model:
|
||||
|
||||
```text
|
||||
publish
|
||||
replace
|
||||
skip_same
|
||||
skip_destination_newer
|
||||
fail_conflict
|
||||
fail_unmanaged
|
||||
```
|
||||
|
||||
Boundaries:
|
||||
|
||||
- publish logic should not parse CLI flags;
|
||||
- publish logic should not know adapter implementation details;
|
||||
- publish logic should use `internal/state` for destination state semantics;
|
||||
- publish logic should use `internal/storage` interfaces for IO.
|
||||
|
||||
### `internal/notify`
|
||||
|
||||
Notification stage abstraction.
|
||||
|
||||
MVP responsibilities:
|
||||
|
||||
- define notifier interface;
|
||||
- implement no-op notifier;
|
||||
- preserve future extension point for email, ntfy, Gotify, RSS update hooks, or other notification channels.
|
||||
|
||||
Future notification rules:
|
||||
|
||||
- notify only after successful publication to the relevant destination or destinations;
|
||||
- notification must be idempotent with respect to source id, digest, pipeline id, and destination id where applicable;
|
||||
- notification should not run for skipped or failed publications unless explicitly configured.
|
||||
|
||||
### `internal/logging`
|
||||
|
||||
Logging setup and helpers.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- centralize structured logging setup;
|
||||
- ensure logs omit secrets;
|
||||
- provide consistent fields for pipeline id, bundle id, destination id, backend, path, action, and reason.
|
||||
|
||||
## Implementation Slices
|
||||
|
||||
### Slice 1: Skeleton, config, storage, and source bundle validation
|
||||
|
||||
Deliver:
|
||||
|
||||
- basic CLI skeleton;
|
||||
- config loading and validation;
|
||||
- storage interface, local backend, and fake backend;
|
||||
- source manifest model;
|
||||
- source bundle discovery;
|
||||
- file size and SHA-256 validation;
|
||||
- bundle digest validation;
|
||||
- local backend sufficient for validation;
|
||||
- fixtures for valid and invalid bundles.
|
||||
|
||||
Useful commands:
|
||||
|
||||
```bash
|
||||
distributor validate ./examples/weather-bundle
|
||||
```
|
||||
|
||||
### Slice 2: Destination state and dry-run planning
|
||||
|
||||
Deliver:
|
||||
|
||||
- `.distributor.json` model;
|
||||
- destination state comparison;
|
||||
- publish plan model;
|
||||
- dry-run output;
|
||||
- local source to local destination planning;
|
||||
- tests for same/older/newer/conflict/unmanaged cases.
|
||||
|
||||
Useful command:
|
||||
|
||||
```bash
|
||||
distributor run --config ./examples/local.yml --dry-run
|
||||
```
|
||||
|
||||
### Slice 3: Local publish execution
|
||||
|
||||
Deliver:
|
||||
|
||||
- local destination writes;
|
||||
- source-file publication;
|
||||
- `.distributor.json` writes;
|
||||
- replacement safety checks;
|
||||
- skip behavior;
|
||||
- narrow deletion behavior based on destination state outputs.
|
||||
|
||||
Useful command:
|
||||
|
||||
```bash
|
||||
distributor run --config ./examples/local.yml
|
||||
```
|
||||
|
||||
### Slice 4: Markdown-to-HTML transform
|
||||
|
||||
Deliver:
|
||||
|
||||
- Markdown transform registry;
|
||||
- Markdown-to-HTML implementation;
|
||||
- per-destination `publish.source` and `publish.html` behavior;
|
||||
- generated output metadata in `.distributor.json`;
|
||||
- tests for source-only, HTML-only, and source-plus-HTML destinations.
|
||||
|
||||
### Slice 5: S3 backend
|
||||
|
||||
Deliver:
|
||||
|
||||
- S3-compatible backend;
|
||||
- endpoint/bucket/prefix/region/force-path-style config;
|
||||
- credential handling via environment or standard mechanisms;
|
||||
- object listing, read, write, and narrow delete operations;
|
||||
- dry-run and publish coverage using fake or local-compatible test strategy.
|
||||
|
||||
### Slice 6: SSH/SFTP backend
|
||||
|
||||
Deliver:
|
||||
|
||||
- native SFTP backend;
|
||||
- `uri` and `path` config;
|
||||
- documented authentication behavior;
|
||||
- read/write/list/delete operations;
|
||||
- adapter tests or documented integration test strategy.
|
||||
|
||||
### Slice 7: No-op notification stage and future extension seam
|
||||
|
||||
Deliver:
|
||||
|
||||
- no-op notifier wired into orchestration;
|
||||
- clear internal contract for future email/ntfy adapters;
|
||||
- no user-facing notification behavior beyond no-op unless implemented.
|
||||
|
||||
## Deferred Ideas
|
||||
|
||||
The following are intentionally out of MVP unless separately accepted in a later roadmap:
|
||||
|
||||
- email, ntfy, Gotify, or other real notification adapters;
|
||||
- RSS/Atom feed generation;
|
||||
- PDF generation;
|
||||
- web UI;
|
||||
- full-text search;
|
||||
- dynamic plugin loading;
|
||||
- arbitrary transform chains;
|
||||
- workflow DAGs;
|
||||
- producer execution;
|
||||
- complex templating/theming;
|
||||
- bidirectional sync;
|
||||
- backup semantics.
|
||||
|
||||
## Key Invariants
|
||||
|
||||
- Producer apps own source bundle creation.
|
||||
- `distributor` owns destination publication state.
|
||||
- Source `manifest.json` is not copied as destination state.
|
||||
- Destination `.distributor.json` is the managed sentinel.
|
||||
- One pipeline has one source and one or more destinations.
|
||||
- Transform and publish policy are destination-specific.
|
||||
- Source files are canonical; HTML is derived.
|
||||
- Destructive replacement is allowed only inside managed destination bundle paths.
|
||||
- Core logic must be testable without real S3, SSH, or remote services.
|
||||
Reference in New Issue
Block a user