eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity
Files
0818733b194266e4afcf9d37cb0c2a93d4421c73
distributor/docs/roadmap/config.md

581 lines
13 KiB
Markdown
Raw Blame History

# Distributor Configuration Roadmap
This roadmap defines the planned `config.yml` schema for the `distributor` MVP. The goal is to support one-to-many publication pipelines where each pipeline has one source and one or more destinations. Each destination independently controls backend configuration, publication outputs, transform behavior, and replacement policy.
## Configuration Goals
The MVP configuration should be:
- explicit enough to avoid hidden publication behavior;
- compact enough for routine self-hosted use;
- backend-agnostic at the pipeline layer;
- capable of local, SSH/SFTP, and S3-compatible source and destination backends;
- ready for future notification adapters without exposing a fake notification feature in the MVP.
## Top-Level Shape
```yaml
pipelines:
- id: weather-daily
source:
backend: local
path: /var/spool/distributor/weather
validation:
on_digest_mismatch: fail
destinations:
- id: markdown-archive
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: weather/archive
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
publish:
source: true
html: false
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
- id: static-site
backend: ssh
uri: ssh://deploy@example.com:22
path: /srv/www/weather
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
```
## Pipeline Fields
Each pipeline must include:
- `id`: Required stable pipeline identifier.
- `source`: Required source backend configuration.
- `destinations`: Required non-empty list of destination configurations.
Optional pipeline-level fields:
- `validation`: Source validation behavior.
- Future: `notifications` or `notify`, when notification adapters are implemented.
A pipeline has exactly one source and one or more destinations.
## Pipeline ID Rules
`pipelines[].id` should:
- be required;
- be unique across the config file;
- be stable over time;
- use a simple slug-like format, such as `weather-daily` or `dnd-session-recaps`.
Recommended validation:
```text
^[a-zA-Z0-9][a-zA-Z0-9._-]*$
```
## Source Configuration
`source` defines the source root where bundles are discovered.
The source backend may be:
- `local`;
- `ssh`;
- `s3`.
The source is scanned for `manifest.json` files beneath the configured root.
### Local source
```yaml
source:
backend: local
path: /var/spool/distributor/weather
```
Required fields:
- `backend: local`
- `path`
### SSH source
```yaml
source:
backend: ssh
uri: ssh://reports@example.com:22
path: /var/spool/distributor/weather
```
Required fields:
- `backend: ssh`
- `uri`
- `path`
Recommended authentication behavior:
- use SSH agent by default;
- use local known_hosts validation by default;
- support optional key file configuration later if needed;
- do not require passwords in YAML.
Optional future fields:
```yaml
known_hosts: /home/user/.ssh/known_hosts
key_file: /home/user/.ssh/id_ed25519
```
### S3 source
```yaml
source:
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: incoming/weather
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
```
Required fields:
- `backend: s3`
- `endpoint`
- `bucket`
Optional fields:
- `prefix`
- `region`
- `force_path_style`
- `credentials`
Credential configuration should prefer environment variables or standard SDK behavior over literal secrets in config.
## Destination Configuration
Each destination defines one publication target for a pipeline.
Required destination fields:
- `id`
- `backend`
- backend-specific location fields;
- `publish`
Optional destination fields:
- `transform`
- `transfer`
Each destination is independently planned and published. A destination may receive source files, generated HTML, or both.
## Destination ID Rules
`destinations[].id` should:
- be required;
- be unique within the containing pipeline;
- be stable over time;
- use a slug-like format.
Recommended examples:
- `markdown-archive`
- `static-site`
- `full-mirror`
## Local Destination
```yaml
destinations:
- id: local-static
backend: local
path: /srv/www/reports
publish:
source: false
html: true
```
Required fields:
- `backend: local`
- `path`
- `publish`
## SSH Destination
```yaml
destinations:
- id: static-site
backend: ssh
uri: ssh://deploy@example.com:22
path: /srv/www/weather
publish:
source: false
html: true
```
Required fields:
- `backend: ssh`
- `uri`
- `path`
- `publish`
The MVP should use a native SFTP implementation rather than shelling out to `ssh`, `scp`, or `rsync`.
## S3 Destination
```yaml
destinations:
- id: markdown-archive
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: weather/archive
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
publish:
source: true
html: false
```
Required fields:
- `backend: s3`
- `endpoint`
- `bucket`
- `publish`
Optional fields:
- `prefix`
- `region`
- `force_path_style`
- `credentials`
## Backend Configuration Normalization
The config loader should normalize backend configuration into internal source and destination backend specs. Pipeline logic should not branch on backend-specific fields.
Validation should catch:
- missing backend names;
- unsupported backend names;
- missing backend-specific required fields;
- duplicate pipeline IDs;
- duplicate destination IDs within a pipeline;
- empty destination lists;
- invalid policy values.
## Publication Policy
`publish` controls which categories of files are written to a destination.
```yaml
publish:
source: true
html: false
```
Fields:
- `source`: Publish source artifacts listed in `manifest.json`.
- `html`: Publish HTML files generated from Markdown source artifacts.
At least one of `source` or `html` must be true.
Recommended defaults:
```yaml
publish:
source: true
html: false
```
No implicit HTML transformation should occur. When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `mode: sidecar` are required for the MVP.
## Transform Configuration
For MVP, the only supported transform is Markdown to HTML.
```yaml
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
Fields:
- `enabled`: Whether Markdown-to-HTML transform is enabled.
- `mode`: Output mode. MVP value: `sidecar`.
MVP `sidecar` behavior:
- each listed Markdown source file generates an HTML file with the same base path and `.html` extension;
- `report.md` generates `report.html`;
- generated files are destination publication artifacts;
- source bundles are not mutated.
MVP defaulting:
- If `publish.html` is false, transform may be omitted.
- If `publish.html` is true and `transform.markdown_to_html` is omitted or disabled, config validation must fail.
## Validation Policy
Pipeline-level validation is intentionally narrow in the MVP.
```yaml
validation:
on_digest_mismatch: fail
```
Supported value:
- `fail`
Default:
```yaml
on_digest_mismatch: fail
```
Validation should happen before any destination writes. Warning-only digest mismatch handling is deferred and must be rejected if configured.
## Transfer Policy
Destination-level transfer policy controls behavior after inspecting `.distributor.json` at the destination bundle path.
```yaml
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
```
Supported fields:
- `on_destination_same`
- `on_destination_older`
- `on_destination_newer`
- `on_conflict`
MVP supported values are intentionally limited by field:
- `on_destination_same`: `skip` or `fail`
- `on_destination_older`: `replace` or `fail`
- `on_destination_newer`: `skip` or `fail`
- `on_conflict`: `fail`
Recommended MVP defaults:
```yaml
transfer:
on_destination_same: skip
on_destination_older: replace
on_destination_newer: skip
on_conflict: fail
```
Safety rule:
- `replace` must never perform broad deletion against a destination root.
- `replace` may only operate within a resolved destination bundle path and should delete only files recorded in existing `.distributor.json.outputs` plus `.distributor.json` where practical.
- Unmanaged non-empty destination paths fail in the MVP. Force or unmanaged overwrite configuration is deferred.
- Broader replacement values, including replacing newer destinations or conflicts, are deferred to a later explicit force-overwrite stage.
## Path Mapping
MVP path mapping is fixed:
```text
destination bundle path = destination root + source relative bundle path
```
Example:
```text
source root: /var/spool/reports
source bundle: /var/spool/reports/weather/daily/brentwood/2026-05-30
relative bundle path: weather/daily/brentwood/2026-05-30
destination root: /srv/www/reports
destination bundle path: /srv/www/reports/weather/daily/brentwood/2026-05-30
```
Future config may support explicit path mapping, but MVP should not.
## Dry Run Configuration and CLI Behavior
Dry-run should be a CLI flag rather than a persistent config setting.
```bash
distributor run --config config.yml --dry-run
```
Dry-run should report:
- pipeline ID;
- source backend;
- destination ID;
- destination backend;
- discovered bundle ID;
- relative bundle path;
- planned action;
- reason;
- transform outputs that would be generated;
- files that would be written or deleted.
## Example: Weather Pipeline
```yaml
pipelines:
- id: weather-daily
source:
backend: local
path: /var/spool/distributor/weather
validation:
on_digest_mismatch: fail
destinations:
- id: markdown-archive
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: weather/archive
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
publish:
source: true
html: false
- id: static-site
backend: ssh
uri: ssh://deploy@web.example.com:22
path: /srv/www/weather
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
## Example: D&D Recap Pipeline
```yaml
pipelines:
- id: dnd-session-recaps
source:
backend: local
path: /var/spool/distributor/dnd/session-recaps
destinations:
- id: private-markdown-archive
backend: s3
endpoint: https://s3.example.com
bucket: reports
prefix: dnd/session-recaps
region: us-east-1
force_path_style: true
credentials:
access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
publish:
source: true
html: false
- id: private-html-site
backend: local
path: /srv/www/private/dnd/session-recaps
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
```
## Future Notification Configuration
Notification should not be exposed as a functional MVP feature unless an adapter exists.
The internal pipeline may include a no-op notification stage. Future config may look like:
```yaml
notifications:
- id: weather-email
backend: email
after_destinations:
- static-site
subject: "Weather report published"
```
Future notification policy should require:
- notification after successful relevant publication;
- idempotency by source manifest id and digest;
- no duplicate notification unless explicitly forced.
## Implementation Stages
1. Define config structs for pipelines, sources, destinations, validation, publish, transform, and transfer policies.
2. Implement config loading and strict validation.
3. Implement backend-specific config validation for local, SSH, and S3.
4. Implement defaulting for validation and transfer policies.
5. Require explicit transform configuration when `publish.html` is true.
6. Add example config fixtures for local-to-local, local-to-SSH, local-to-S3, and fan-out scenarios.
7. Connect config to backend registry and publish planner.
8. Add `--pipeline` filtering for targeted runs.
9. Add `--dry-run` output that reflects the resolved config and planned actions.
## Deferred Configuration
The following configuration ideas are intentionally outside the MVP:
- warning-only digest mismatch handling;
- unmanaged destination overwrite flags until the explicit force-overwrite roadmap stage;
- force replacement of destinations with different source ids until the explicit force-overwrite roadmap stage.
Reference in New Issue View Git Blame Copy Permalink