distributor/docs/roadmap/config.md

# Distributor Configuration Roadmap

This roadmap defines the planned `config.yml` schema for the `distributor` MVP. The goal is to support one-to-many publication pipelines where each pipeline has one source and one or more destinations. Each destination independently controls backend configuration, publication outputs, transform behavior, and replacement policy.

## Configuration Goals

The MVP configuration should be:

- explicit enough to avoid hidden publication behavior;
- compact enough for routine self-hosted use;
- backend-agnostic at the pipeline layer;
- capable of local, SSH/SFTP, and S3-compatible source and destination backends;
- ready for future notification adapters without exposing a fake notification feature in the MVP.

## Top-Level Shape

```yaml
pipelines:
  - id: weather-daily
    source:
      backend: local
      path: /var/spool/distributor/weather

    validation:
      on_digest_mismatch: fail

    destinations:
      - id: markdown-archive
        backend: s3
        endpoint: https://s3.example.com
        bucket: reports
        prefix: weather/archive
        region: us-east-1
        force_path_style: true
        credentials:
          access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
          secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY

        publish:
          source: true
          html: false

        transfer:
          on_destination_same: skip
          on_destination_older: replace
          on_destination_newer: skip
          on_conflict: fail

      - id: static-site
        backend: ssh
        uri: ssh://deploy@example.com:22
        path: /srv/www/weather

        publish:
          source: false
          html: true

        transform:
          markdown_to_html:
            enabled: true
            mode: sidecar

        transfer:
          on_destination_same: skip
          on_destination_older: replace
          on_destination_newer: skip
          on_conflict: fail
```

## Pipeline Fields

Each pipeline must include:

- `id`: Required stable pipeline identifier.
- `source`: Required source backend configuration.
- `destinations`: Required non-empty list of destination configurations.

Optional pipeline-level fields:

- `validation`: Source validation behavior.
- Future: `notifications` or `notify`, when notification adapters are implemented.

A pipeline has exactly one source and one or more destinations.

## Pipeline ID Rules

`pipelines[].id` should:

- be required;
- be unique across the config file;
- be stable over time;
- use a simple slug-like format, such as `weather-daily` or `dnd-session-recaps`.

Recommended validation:

```text
^[a-zA-Z0-9][a-zA-Z0-9._-]*$
```

## Source Configuration

`source` defines the source root where bundles are discovered.

The source backend may be:

- `local`;
- `ssh`;
- `s3`.

The source is scanned for `manifest.json` files beneath the configured root.

### Local source

```yaml
source:
  backend: local
  path: /var/spool/distributor/weather
```

Required fields:

- `backend: local`
- `path`

### SSH source

```yaml
source:
  backend: ssh
  uri: ssh://reports@example.com:22
  path: /var/spool/distributor/weather
```

Required fields:

- `backend: ssh`
- `uri`
- `path`

Recommended authentication behavior:

- use SSH agent by default;
- use local known_hosts validation by default;
- support optional key file configuration later if needed;
- do not require passwords in YAML.

Optional future fields:

```yaml
known_hosts: /home/user/.ssh/known_hosts
key_file: /home/user/.ssh/id_ed25519
```

### S3 source

```yaml
source:
  backend: s3
  endpoint: https://s3.example.com
  bucket: reports
  prefix: incoming/weather
  region: us-east-1
  force_path_style: true
  credentials:
    access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
    secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
```

Required fields:

- `backend: s3`
- `endpoint`
- `bucket`

Optional fields:

- `prefix`
- `region`
- `force_path_style`
- `credentials`

Credential configuration should prefer environment variables or standard SDK behavior over literal secrets in config.

## Destination Configuration

Each destination defines one publication target for a pipeline.

Required destination fields:

- `id`
- `backend`
- backend-specific location fields;
- `publish`

Optional destination fields:

- `transform`
- `transfer`

Each destination is independently planned and published. A destination may receive source files, generated HTML, or both.

## Destination ID Rules

`destinations[].id` should:

- be required;
- be unique within the containing pipeline;
- be stable over time;
- use a slug-like format.

Recommended examples:

- `markdown-archive`
- `static-site`
- `full-mirror`

## Local Destination

```yaml
destinations:
  - id: local-static
    backend: local
    path: /srv/www/reports
    publish:
      source: false
      html: true
```

Required fields:

- `backend: local`
- `path`
- `publish`

## SSH Destination

```yaml
destinations:
  - id: static-site
    backend: ssh
    uri: ssh://deploy@example.com:22
    path: /srv/www/weather
    publish:
      source: false
      html: true
```

Required fields:

- `backend: ssh`
- `uri`
- `path`
- `publish`

The MVP should use a native SFTP implementation rather than shelling out to `ssh`, `scp`, or `rsync`.

## S3 Destination

```yaml
destinations:
  - id: markdown-archive
    backend: s3
    endpoint: https://s3.example.com
    bucket: reports
    prefix: weather/archive
    region: us-east-1
    force_path_style: true
    credentials:
      access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
      secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
    publish:
      source: true
      html: false
```

Required fields:

- `backend: s3`
- `endpoint`
- `bucket`
- `publish`

Optional fields:

- `prefix`
- `region`
- `force_path_style`
- `credentials`

## Backend Configuration Normalization

The config loader should normalize backend configuration into internal source and destination backend specs. Pipeline logic should not branch on backend-specific fields.

Validation should catch:

- missing backend names;
- unsupported backend names;
- missing backend-specific required fields;
- duplicate pipeline IDs;
- duplicate destination IDs within a pipeline;
- empty destination lists;
- invalid policy values.

## Publication Policy

`publish` controls which categories of files are written to a destination.

```yaml
publish:
  source: true
  html: false
```

Fields:

- `source`: Publish source artifacts listed in `manifest.json`.
- `html`: Publish HTML files generated from Markdown source artifacts.

At least one of `source` or `html` must be true.

Recommended defaults:

```yaml
publish:
  source: true
  html: false
```

No implicit HTML transformation should occur. When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `mode: sidecar` are required for the MVP.

## Transform Configuration

For MVP, the only supported transform is Markdown to HTML.

```yaml
transform:
  markdown_to_html:
    enabled: true
    mode: sidecar
```

Fields:

- `enabled`: Whether Markdown-to-HTML transform is enabled.
- `mode`: Output mode. MVP value: `sidecar`.

MVP `sidecar` behavior:

- each listed Markdown source file generates an HTML file with the same base path and `.html` extension;
- `report.md` generates `report.html`;
- generated files are destination publication artifacts;
- source bundles are not mutated.

MVP defaulting:

- If `publish.html` is false, transform may be omitted.
- If `publish.html` is true and `transform.markdown_to_html` is omitted or disabled, config validation must fail.

## Validation Policy

Pipeline-level validation is intentionally narrow in the MVP.

```yaml
validation:
  on_digest_mismatch: fail
```

Supported value:

- `fail`

Default:

```yaml
on_digest_mismatch: fail
```

Validation should happen before any destination writes. Warning-only digest mismatch handling is deferred and must be rejected if configured.

## Transfer Policy

Destination-level transfer policy controls behavior after inspecting `.distributor.json` at the destination bundle path.

```yaml
transfer:
  on_destination_same: skip
  on_destination_older: replace
  on_destination_newer: skip
  on_conflict: fail
```

Supported fields:

- `on_destination_same`
- `on_destination_older`
- `on_destination_newer`
- `on_conflict`

Recommended supported values:

- `skip`
- `replace`
- `fail`

Recommended MVP defaults:

```yaml
transfer:
  on_destination_same: skip
  on_destination_older: replace
  on_destination_newer: skip
  on_conflict: fail
```

Safety rule:

- `replace` must never perform broad deletion against a destination root.
- `replace` may only operate within a resolved destination bundle path and should delete only files recorded in existing `.distributor.json.outputs` plus `.distributor.json` where practical.
- Unmanaged non-empty destination paths fail in the MVP. Force or unmanaged overwrite configuration is deferred.

## Path Mapping

MVP path mapping is fixed:

```text
destination bundle path = destination root + source relative bundle path
```

Example:

```text
source root: /var/spool/reports
source bundle: /var/spool/reports/weather/daily/brentwood/2026-05-30
relative bundle path: weather/daily/brentwood/2026-05-30

destination root: /srv/www/reports
destination bundle path: /srv/www/reports/weather/daily/brentwood/2026-05-30
```

Future config may support explicit path mapping, but MVP should not.

## Dry Run Configuration and CLI Behavior

Dry-run should be a CLI flag rather than a persistent config setting.

```bash
distributor run --config config.yml --dry-run
```

Dry-run should report:

- pipeline ID;
- source backend;
- destination ID;
- destination backend;
- discovered bundle ID;
- relative bundle path;
- planned action;
- reason;
- transform outputs that would be generated;
- files that would be written or deleted.

## Example: Weather Pipeline

```yaml
pipelines:
  - id: weather-daily
    source:
      backend: local
      path: /var/spool/distributor/weather

    validation:
      on_digest_mismatch: fail

    destinations:
      - id: markdown-archive
        backend: s3
        endpoint: https://s3.maximumdirect.net
        bucket: reports
        prefix: weather/archive
        region: us-east-1
        force_path_style: true
        credentials:
          access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
          secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
        publish:
          source: true
          html: false

      - id: static-site
        backend: ssh
        uri: ssh://deploy@web.maximumdirect.net:22
        path: /srv/www/weather
        publish:
          source: false
          html: true
        transform:
          markdown_to_html:
            enabled: true
            mode: sidecar
```

## Example: D&D Recap Pipeline

```yaml
pipelines:
  - id: dnd-session-recaps
    source:
      backend: local
      path: /var/spool/distributor/dnd/session-recaps

    destinations:
      - id: private-markdown-archive
        backend: s3
        endpoint: https://s3.maximumdirect.net
        bucket: reports
        prefix: dnd/session-recaps
        region: us-east-1
        force_path_style: true
        credentials:
          access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
          secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
        publish:
          source: true
          html: false

      - id: private-html-site
        backend: local
        path: /srv/www/private/dnd/session-recaps
        publish:
          source: false
          html: true
        transform:
          markdown_to_html:
            enabled: true
            mode: sidecar
```

## Future Notification Configuration

Notification should not be exposed as a functional MVP feature unless an adapter exists.

The internal pipeline may include a no-op notification stage. Future config may look like:

```yaml
notifications:
  - id: weather-email
    backend: email
    after_destinations:
      - static-site
    subject: "Weather report published"
```

Future notification policy should require:

- notification after successful relevant publication;
- idempotency by source manifest id and digest;
- no duplicate notification unless explicitly forced.

## Implementation Stages

1. Define config structs for pipelines, sources, destinations, validation, publish, transform, and transfer policies.
2. Implement config loading and strict validation.
3. Implement backend-specific config validation for local, SSH, and S3.
4. Implement defaulting for validation and transfer policies.
5. Require explicit transform configuration when `publish.html` is true.
6. Add example config fixtures for local-to-local, local-to-SSH, local-to-S3, and fan-out scenarios.
7. Connect config to backend registry and publish planner.
8. Add `--pipeline` filtering for targeted runs.
9. Add `--dry-run` output that reflects the resolved config and planned actions.

## Deferred Configuration

The following configuration ideas are intentionally outside the MVP:

- warning-only digest mismatch handling;
- unmanaged destination overwrite flags;
- force replacement of destinations with different source ids.