distributor/docs/roadmap/config.md

Distributor Configuration Roadmap

This roadmap defines the planned config.yml schema for the distributor MVP. The goal is to support one-to-many publication pipelines where each pipeline has one source and one or more destinations. Each destination independently controls backend configuration, publication outputs, transform behavior, and replacement policy.

Configuration Goals

The MVP configuration should be:

• explicit enough to avoid hidden publication behavior;
• compact enough for routine self-hosted use;
• backend-agnostic at the pipeline layer;
• capable of local, SSH/SFTP, and S3-compatible source and destination backends;
• ready for future notification adapters without exposing a fake notification feature in the MVP.

Top-Level Shape

pipelines:
  - id: weather-daily
    source:
      backend: local
      path: /var/spool/distributor/weather

    validation:
      on_digest_mismatch: fail

    destinations:
      - id: markdown-archive
        backend: s3
        endpoint: https://s3.example.com
        bucket: reports
        prefix: weather/archive
        region: us-east-1
        force_path_style: true
        credentials:
          access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
          secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY

        publish:
          source: true
          html: false

        transfer:
          on_destination_same: skip
          on_destination_older: replace
          on_destination_newer: skip
          on_conflict: fail

      - id: static-site
        backend: ssh
        uri: ssh://deploy@example.com:22
        path: /srv/www/weather

        publish:
          source: false
          html: true

        transform:
          markdown_to_html:
            enabled: true
            mode: sidecar

        transfer:
          on_destination_same: skip
          on_destination_older: replace
          on_destination_newer: skip
          on_conflict: fail

Pipeline Fields

Each pipeline must include:

• id: Required stable pipeline identifier.
• source: Required source backend configuration.
• destinations: Required non-empty list of destination configurations.

Optional pipeline-level fields:

• validation: Source validation behavior.
• Future: notifications or notify, when notification adapters are implemented.

A pipeline has exactly one source and one or more destinations.

Pipeline ID Rules

pipelines[].id should:

• be required;
• be unique across the config file;
• be stable over time;
• use a simple slug-like format, such as weather-daily or dnd-session-recaps.

Recommended validation:

^[a-zA-Z0-9][a-zA-Z0-9._-]*$

Source Configuration

source defines the source root where bundles are discovered.

The source backend may be:

• local;
• ssh;
• s3.

The source is scanned for manifest.json files beneath the configured root.

Local source

source:
  backend: local
  path: /var/spool/distributor/weather

Required fields:

• backend: local
• path

SSH source

source:
  backend: ssh
  uri: ssh://reports@example.com:22
  path: /var/spool/distributor/weather

Required fields:

• backend: ssh
• uri
• path

Recommended authentication behavior:

• use SSH agent by default;
• use local known_hosts validation by default;
• support optional key file configuration later if needed;
• do not require passwords in YAML.

Optional future fields:

known_hosts: /home/user/.ssh/known_hosts
key_file: /home/user/.ssh/id_ed25519

S3 source

source:
  backend: s3
  endpoint: https://s3.example.com
  bucket: reports
  prefix: incoming/weather
  region: us-east-1
  force_path_style: true
  credentials:
    access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
    secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY

Required fields:

• backend: s3
• endpoint
• bucket

Optional fields:

• prefix
• region
• force_path_style
• credentials

Credential configuration should prefer environment variables or standard SDK behavior over literal secrets in config.

Destination Configuration

Each destination defines one publication target for a pipeline.

Required destination fields:

• id
• backend
• backend-specific location fields;
• publish

Optional destination fields:

• transform
• transfer

Each destination is independently planned and published. A destination may receive source files, generated HTML, or both.

Destination ID Rules

destinations[].id should:

• be required;
• be unique within the containing pipeline;
• be stable over time;
• use a slug-like format.

Recommended examples:

• markdown-archive
• static-site
• full-mirror

Local Destination

destinations:
  - id: local-static
    backend: local
    path: /srv/www/reports
    publish:
      source: false
      html: true

Required fields:

• backend: local
• path
• publish

SSH Destination

destinations:
  - id: static-site
    backend: ssh
    uri: ssh://deploy@example.com:22
    path: /srv/www/weather
    publish:
      source: false
      html: true

Required fields:

• backend: ssh
• uri
• path
• publish

The MVP should use a native SFTP implementation rather than shelling out to ssh, scp, or rsync.

S3 Destination

destinations:
  - id: markdown-archive
    backend: s3
    endpoint: https://s3.example.com
    bucket: reports
    prefix: weather/archive
    region: us-east-1
    force_path_style: true
    credentials:
      access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
      secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
    publish:
      source: true
      html: false

Required fields:

• backend: s3
• endpoint
• bucket
• publish

Optional fields:

• prefix
• region
• force_path_style
• credentials

Backend Configuration Normalization

The config loader should normalize backend configuration into internal source and destination backend specs. Pipeline logic should not branch on backend-specific fields.

Validation should catch:

• missing backend names;
• unsupported backend names;
• missing backend-specific required fields;
• duplicate pipeline IDs;
• duplicate destination IDs within a pipeline;
• empty destination lists;
• invalid policy values.

Publication Policy

publish controls which categories of files are written to a destination.

publish:
  source: true
  html: false

Fields:

• source: Publish source artifacts listed in manifest.json.
• html: Publish HTML files generated from Markdown source artifacts.

At least one of source or html must be true.

Recommended defaults:

publish:
  source: true
  html: false

No implicit HTML transformation should occur. When publish.html is true, transform.markdown_to_html.enabled: true and mode: sidecar are required for the MVP.

Transform Configuration

For MVP, the only supported transform is Markdown to HTML.

transform:
  markdown_to_html:
    enabled: true
    mode: sidecar

Fields:

• enabled: Whether Markdown-to-HTML transform is enabled.
• mode: Output mode. MVP value: sidecar.

MVP sidecar behavior:

• each listed Markdown source file generates an HTML file with the same base path and .html extension;
• report.md generates report.html;
• generated files are destination publication artifacts;
• source bundles are not mutated.

MVP defaulting:

• If publish.html is false, transform may be omitted.
• If publish.html is true and transform.markdown_to_html is omitted or disabled, config validation must fail.

Validation Policy

Pipeline-level validation is intentionally narrow in the MVP.

validation:
  on_digest_mismatch: fail

Supported value:

• fail

Default:

on_digest_mismatch: fail

Validation should happen before any destination writes. Warning-only digest mismatch handling is deferred and must be rejected if configured.

Transfer Policy

Destination-level transfer policy controls behavior after inspecting .distributor.json at the destination bundle path.

transfer:
  on_destination_same: skip
  on_destination_older: replace
  on_destination_newer: skip
  on_conflict: fail

Supported fields:

• on_destination_same
• on_destination_older
• on_destination_newer
• on_conflict

MVP supported values are intentionally limited by field:

• on_destination_same: skip or fail
• on_destination_older: replace or fail
• on_destination_newer: skip or fail
• on_conflict: fail

Recommended MVP defaults:

transfer:
  on_destination_same: skip
  on_destination_older: replace
  on_destination_newer: skip
  on_conflict: fail

Safety rule:

• replace must never perform broad deletion against a destination root.
• replace may only operate within a resolved destination bundle path and should delete only files recorded in existing .distributor.json.outputs plus .distributor.json where practical.
• Unmanaged non-empty destination paths fail in the MVP. Force or unmanaged overwrite configuration is deferred.
• Broader replacement values, including replacing newer destinations or conflicts, are deferred to a later explicit force-overwrite stage.

Path Mapping

MVP path mapping is fixed:

destination bundle path = destination root + source relative bundle path

Example:

source root: /var/spool/reports
source bundle: /var/spool/reports/weather/daily/brentwood/2026-05-30
relative bundle path: weather/daily/brentwood/2026-05-30

destination root: /srv/www/reports
destination bundle path: /srv/www/reports/weather/daily/brentwood/2026-05-30

Future config may support explicit path mapping, but MVP should not.

Dry Run Configuration and CLI Behavior

Dry-run should be a CLI flag rather than a persistent config setting.

distributor run --config config.yml --dry-run

Dry-run should report:

• pipeline ID;
• source backend;
• destination ID;
• destination backend;
• discovered bundle ID;
• relative bundle path;
• planned action;
• reason;
• transform outputs that would be generated;
• files that would be written or deleted.

Example: Weather Pipeline

pipelines:
  - id: weather-daily
    source:
      backend: local
      path: /var/spool/distributor/weather

    validation:
      on_digest_mismatch: fail

    destinations:
      - id: markdown-archive
        backend: s3
        endpoint: https://s3.maximumdirect.net
        bucket: reports
        prefix: weather/archive
        region: us-east-1
        force_path_style: true
        credentials:
          access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
          secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
        publish:
          source: true
          html: false

      - id: static-site
        backend: ssh
        uri: ssh://deploy@web.maximumdirect.net:22
        path: /srv/www/weather
        publish:
          source: false
          html: true
        transform:
          markdown_to_html:
            enabled: true
            mode: sidecar

Example: D&D Recap Pipeline

pipelines:
  - id: dnd-session-recaps
    source:
      backend: local
      path: /var/spool/distributor/dnd/session-recaps

    destinations:
      - id: private-markdown-archive
        backend: s3
        endpoint: https://s3.maximumdirect.net
        bucket: reports
        prefix: dnd/session-recaps
        region: us-east-1
        force_path_style: true
        credentials:
          access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
          secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
        publish:
          source: true
          html: false

      - id: private-html-site
        backend: local
        path: /srv/www/private/dnd/session-recaps
        publish:
          source: false
          html: true
        transform:
          markdown_to_html:
            enabled: true
            mode: sidecar

Future Notification Configuration

Notification should not be exposed as a functional MVP feature unless an adapter exists.

The internal pipeline may include a no-op notification stage. Future config may look like:

notifications:
  - id: weather-email
    backend: email
    after_destinations:
      - static-site
    subject: "Weather report published"

Future notification policy should require:

• notification after successful relevant publication;
• idempotency by source manifest id and digest;
• no duplicate notification unless explicitly forced.

Implementation Stages

1. Define config structs for pipelines, sources, destinations, validation, publish, transform, and transfer policies.
2. Implement config loading and strict validation.
3. Implement backend-specific config validation for local, SSH, and S3.
4. Implement defaulting for validation and transfer policies.
5. Require explicit transform configuration when publish.html is true.
6. Add example config fixtures for local-to-local, local-to-SSH, local-to-S3, and fan-out scenarios.
7. Connect config to backend registry and publish planner.
8. Add --pipeline filtering for targeted runs.
9. Add --dry-run output that reflects the resolved config and planned actions.

Deferred Configuration

The following configuration ideas are intentionally outside the MVP:

• warning-only digest mismatch handling;
• unmanaged destination overwrite flags until the explicit force-overwrite roadmap stage;
• force replacement of destinations with different source ids until the explicit force-overwrite roadmap stage.