Added initial MVP roadmap documentation and implementation plan
This commit is contained in:
578
docs/roadmap/config.md
Normal file
578
docs/roadmap/config.md
Normal file
@@ -0,0 +1,578 @@
|
||||
# Distributor Configuration Roadmap
|
||||
|
||||
This roadmap defines the planned `config.yml` schema for the `distributor` MVP. The goal is to support one-to-many publication pipelines where each pipeline has one source and one or more destinations. Each destination independently controls backend configuration, publication outputs, transform behavior, and replacement policy.
|
||||
|
||||
## Configuration Goals
|
||||
|
||||
The MVP configuration should be:
|
||||
|
||||
- explicit enough to avoid hidden publication behavior;
|
||||
- compact enough for routine self-hosted use;
|
||||
- backend-agnostic at the pipeline layer;
|
||||
- capable of local, SSH/SFTP, and S3-compatible source and destination backends;
|
||||
- ready for future notification adapters without exposing a fake notification feature in the MVP.
|
||||
|
||||
## Top-Level Shape
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: weather-daily
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/weather
|
||||
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: weather/archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@example.com:22
|
||||
path: /srv/www/weather
|
||||
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
## Pipeline Fields
|
||||
|
||||
Each pipeline must include:
|
||||
|
||||
- `id`: Required stable pipeline identifier.
|
||||
- `source`: Required source backend configuration.
|
||||
- `destinations`: Required non-empty list of destination configurations.
|
||||
|
||||
Optional pipeline-level fields:
|
||||
|
||||
- `validation`: Source validation behavior.
|
||||
- Future: `notifications` or `notify`, when notification adapters are implemented.
|
||||
|
||||
A pipeline has exactly one source and one or more destinations.
|
||||
|
||||
## Pipeline ID Rules
|
||||
|
||||
`pipelines[].id` should:
|
||||
|
||||
- be required;
|
||||
- be unique across the config file;
|
||||
- be stable over time;
|
||||
- use a simple slug-like format, such as `weather-daily` or `dnd-session-recaps`.
|
||||
|
||||
Recommended validation:
|
||||
|
||||
```text
|
||||
^[a-zA-Z0-9][a-zA-Z0-9._-]*$
|
||||
```
|
||||
|
||||
## Source Configuration
|
||||
|
||||
`source` defines the source root where bundles are discovered.
|
||||
|
||||
The source backend may be:
|
||||
|
||||
- `local`;
|
||||
- `ssh`;
|
||||
- `s3`.
|
||||
|
||||
The source is scanned for `manifest.json` files beneath the configured root.
|
||||
|
||||
### Local source
|
||||
|
||||
```yaml
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/weather
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: local`
|
||||
- `path`
|
||||
|
||||
### SSH source
|
||||
|
||||
```yaml
|
||||
source:
|
||||
backend: ssh
|
||||
uri: ssh://reports@example.com:22
|
||||
path: /var/spool/distributor/weather
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: ssh`
|
||||
- `uri`
|
||||
- `path`
|
||||
|
||||
Recommended authentication behavior:
|
||||
|
||||
- use SSH agent by default;
|
||||
- use local known_hosts validation by default;
|
||||
- support optional key file configuration later if needed;
|
||||
- do not require passwords in YAML.
|
||||
|
||||
Optional future fields:
|
||||
|
||||
```yaml
|
||||
known_hosts: /home/user/.ssh/known_hosts
|
||||
key_file: /home/user/.ssh/id_ed25519
|
||||
```
|
||||
|
||||
### S3 source
|
||||
|
||||
```yaml
|
||||
source:
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: incoming/weather
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: s3`
|
||||
- `endpoint`
|
||||
- `bucket`
|
||||
|
||||
Optional fields:
|
||||
|
||||
- `prefix`
|
||||
- `region`
|
||||
- `force_path_style`
|
||||
- `credentials`
|
||||
|
||||
Credential configuration should prefer environment variables or standard SDK behavior over literal secrets in config.
|
||||
|
||||
## Destination Configuration
|
||||
|
||||
Each destination defines one publication target for a pipeline.
|
||||
|
||||
Required destination fields:
|
||||
|
||||
- `id`
|
||||
- `backend`
|
||||
- backend-specific location fields;
|
||||
- `publish`
|
||||
|
||||
Optional destination fields:
|
||||
|
||||
- `transform`
|
||||
- `transfer`
|
||||
|
||||
Each destination is independently planned and published. A destination may receive source files, generated HTML, or both.
|
||||
|
||||
## Destination ID Rules
|
||||
|
||||
`destinations[].id` should:
|
||||
|
||||
- be required;
|
||||
- be unique within the containing pipeline;
|
||||
- be stable over time;
|
||||
- use a slug-like format.
|
||||
|
||||
Recommended examples:
|
||||
|
||||
- `markdown-archive`
|
||||
- `static-site`
|
||||
- `full-mirror`
|
||||
|
||||
## Local Destination
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: local-static
|
||||
backend: local
|
||||
path: /srv/www/reports
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: local`
|
||||
- `path`
|
||||
- `publish`
|
||||
|
||||
## SSH Destination
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@example.com:22
|
||||
path: /srv/www/weather
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: ssh`
|
||||
- `uri`
|
||||
- `path`
|
||||
- `publish`
|
||||
|
||||
The MVP should use a native SFTP implementation rather than shelling out to `ssh`, `scp`, or `rsync`.
|
||||
|
||||
## S3 Destination
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.example.com
|
||||
bucket: reports
|
||||
prefix: weather/archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `backend: s3`
|
||||
- `endpoint`
|
||||
- `bucket`
|
||||
- `publish`
|
||||
|
||||
Optional fields:
|
||||
|
||||
- `prefix`
|
||||
- `region`
|
||||
- `force_path_style`
|
||||
- `credentials`
|
||||
|
||||
## Backend Configuration Normalization
|
||||
|
||||
The config loader should normalize backend configuration into internal source and destination backend specs. Pipeline logic should not branch on backend-specific fields.
|
||||
|
||||
Validation should catch:
|
||||
|
||||
- missing backend names;
|
||||
- unsupported backend names;
|
||||
- missing backend-specific required fields;
|
||||
- duplicate pipeline IDs;
|
||||
- duplicate destination IDs within a pipeline;
|
||||
- empty destination lists;
|
||||
- invalid policy values.
|
||||
|
||||
## Publication Policy
|
||||
|
||||
`publish` controls which categories of files are written to a destination.
|
||||
|
||||
```yaml
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
```
|
||||
|
||||
Fields:
|
||||
|
||||
- `source`: Publish source artifacts listed in `manifest.json`.
|
||||
- `html`: Publish HTML files generated from Markdown source artifacts.
|
||||
|
||||
At least one of `source` or `html` must be true.
|
||||
|
||||
Recommended defaults:
|
||||
|
||||
```yaml
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
```
|
||||
|
||||
No implicit HTML transformation should occur. When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `mode: sidecar` are required for the MVP.
|
||||
|
||||
## Transform Configuration
|
||||
|
||||
For MVP, the only supported transform is Markdown to HTML.
|
||||
|
||||
```yaml
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
Fields:
|
||||
|
||||
- `enabled`: Whether Markdown-to-HTML transform is enabled.
|
||||
- `mode`: Output mode. MVP value: `sidecar`.
|
||||
|
||||
MVP `sidecar` behavior:
|
||||
|
||||
- each listed Markdown source file generates an HTML file with the same base path and `.html` extension;
|
||||
- `report.md` generates `report.html`;
|
||||
- generated files are destination publication artifacts;
|
||||
- source bundles are not mutated.
|
||||
|
||||
MVP defaulting:
|
||||
|
||||
- If `publish.html` is false, transform may be omitted.
|
||||
- If `publish.html` is true and `transform.markdown_to_html` is omitted or disabled, config validation must fail.
|
||||
|
||||
## Validation Policy
|
||||
|
||||
Pipeline-level validation is intentionally narrow in the MVP.
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
```
|
||||
|
||||
Supported value:
|
||||
|
||||
- `fail`
|
||||
|
||||
Default:
|
||||
|
||||
```yaml
|
||||
on_digest_mismatch: fail
|
||||
```
|
||||
|
||||
Validation should happen before any destination writes. Warning-only digest mismatch handling is deferred and must be rejected if configured.
|
||||
|
||||
## Transfer Policy
|
||||
|
||||
Destination-level transfer policy controls behavior after inspecting `.distributor.json` at the destination bundle path.
|
||||
|
||||
```yaml
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
Supported fields:
|
||||
|
||||
- `on_destination_same`
|
||||
- `on_destination_older`
|
||||
- `on_destination_newer`
|
||||
- `on_conflict`
|
||||
|
||||
Recommended supported values:
|
||||
|
||||
- `skip`
|
||||
- `replace`
|
||||
- `fail`
|
||||
|
||||
Recommended MVP defaults:
|
||||
|
||||
```yaml
|
||||
transfer:
|
||||
on_destination_same: skip
|
||||
on_destination_older: replace
|
||||
on_destination_newer: skip
|
||||
on_conflict: fail
|
||||
```
|
||||
|
||||
Safety rule:
|
||||
|
||||
- `replace` must never perform broad deletion against a destination root.
|
||||
- `replace` may only operate within a resolved destination bundle path and should delete only files recorded in existing `.distributor.json.outputs` plus `.distributor.json` where practical.
|
||||
- Unmanaged non-empty destination paths fail in the MVP. Force or unmanaged overwrite configuration is deferred.
|
||||
|
||||
## Path Mapping
|
||||
|
||||
MVP path mapping is fixed:
|
||||
|
||||
```text
|
||||
destination bundle path = destination root + source relative bundle path
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
source root: /var/spool/reports
|
||||
source bundle: /var/spool/reports/weather/daily/brentwood/2026-05-30
|
||||
relative bundle path: weather/daily/brentwood/2026-05-30
|
||||
|
||||
destination root: /srv/www/reports
|
||||
destination bundle path: /srv/www/reports/weather/daily/brentwood/2026-05-30
|
||||
```
|
||||
|
||||
Future config may support explicit path mapping, but MVP should not.
|
||||
|
||||
## Dry Run Configuration and CLI Behavior
|
||||
|
||||
Dry-run should be a CLI flag rather than a persistent config setting.
|
||||
|
||||
```bash
|
||||
distributor run --config config.yml --dry-run
|
||||
```
|
||||
|
||||
Dry-run should report:
|
||||
|
||||
- pipeline ID;
|
||||
- source backend;
|
||||
- destination ID;
|
||||
- destination backend;
|
||||
- discovered bundle ID;
|
||||
- relative bundle path;
|
||||
- planned action;
|
||||
- reason;
|
||||
- transform outputs that would be generated;
|
||||
- files that would be written or deleted.
|
||||
|
||||
## Example: Weather Pipeline
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: weather-daily
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/weather
|
||||
|
||||
validation:
|
||||
on_digest_mismatch: fail
|
||||
|
||||
destinations:
|
||||
- id: markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.maximumdirect.net
|
||||
bucket: reports
|
||||
prefix: weather/archive
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
|
||||
- id: static-site
|
||||
backend: ssh
|
||||
uri: ssh://deploy@web.maximumdirect.net:22
|
||||
path: /srv/www/weather
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
## Example: D&D Recap Pipeline
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: dnd-session-recaps
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/dnd/session-recaps
|
||||
|
||||
destinations:
|
||||
- id: private-markdown-archive
|
||||
backend: s3
|
||||
endpoint: https://s3.maximumdirect.net
|
||||
bucket: reports
|
||||
prefix: dnd/session-recaps
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
credentials:
|
||||
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
|
||||
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
|
||||
- id: private-html-site
|
||||
backend: local
|
||||
path: /srv/www/private/dnd/session-recaps
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: sidecar
|
||||
```
|
||||
|
||||
## Future Notification Configuration
|
||||
|
||||
Notification should not be exposed as a functional MVP feature unless an adapter exists.
|
||||
|
||||
The internal pipeline may include a no-op notification stage. Future config may look like:
|
||||
|
||||
```yaml
|
||||
notifications:
|
||||
- id: weather-email
|
||||
backend: email
|
||||
after_destinations:
|
||||
- static-site
|
||||
subject: "Weather report published"
|
||||
```
|
||||
|
||||
Future notification policy should require:
|
||||
|
||||
- notification after successful relevant publication;
|
||||
- idempotency by source manifest id and digest;
|
||||
- no duplicate notification unless explicitly forced.
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
1. Define config structs for pipelines, sources, destinations, validation, publish, transform, and transfer policies.
|
||||
2. Implement config loading and strict validation.
|
||||
3. Implement backend-specific config validation for local, SSH, and S3.
|
||||
4. Implement defaulting for validation and transfer policies.
|
||||
5. Require explicit transform configuration when `publish.html` is true.
|
||||
6. Add example config fixtures for local-to-local, local-to-SSH, local-to-S3, and fan-out scenarios.
|
||||
7. Connect config to backend registry and publish planner.
|
||||
8. Add `--pipeline` filtering for targeted runs.
|
||||
9. Add `--dry-run` output that reflects the resolved config and planned actions.
|
||||
|
||||
## Deferred Configuration
|
||||
|
||||
The following configuration ideas are intentionally outside the MVP:
|
||||
|
||||
- warning-only digest mismatch handling;
|
||||
- unmanaged destination overwrite flags;
|
||||
- force replacement of destinations with different source ids.
|
||||
Reference in New Issue
Block a user