distributor/docs/roadmap/packages.md

Package Layout Roadmap

This roadmap defines the proposed package layout, boundaries, and implementation responsibilities for the distributor MVP.

distributor is expected to be a domain-agnostic bundle publisher. Producer applications emit source bundles containing manifest.json; distributor validates those bundles and publishes selected source and generated artifacts to one or more configured destinations.

This document is roadmap material. It describes the intended package design before implementation and should move into docs/internal/ only after corresponding behavior exists.

Accepted Package Layout

cmd/distributor/
  main.go

internal/app/
  app.go
  run.go
  pipeline.go

internal/cli/
  root.go
  run.go
  validate.go
  inspect.go

internal/config/
  config.go
  defaults.go
  load.go
  validate.go

internal/bundle/
  manifest.go
  digest.go
  validate.go
  discover.go

internal/state/
  distributor.go
  compare.go
  validate.go

internal/storage/
  backend.go
  registry.go
  path.go
  errors.go

internal/storage/fake/
  backend.go

internal/adapters/local/
  backend.go

internal/adapters/ssh/
  backend.go
  config.go

internal/adapters/s3/
  backend.go
  config.go

internal/transform/
  transform.go
  registry.go
  plan.go

internal/transform/markdown/
  markdown.go
  template.go

internal/publish/
  plan.go
  execute.go
  output.go
  reconcile.go
  safety.go

internal/notify/
  notify.go
  noop.go

internal/logging/
  logging.go

Package Responsibilities

cmd/distributor

Application entrypoint only.

Responsibilities:

• call CLI execution;
• translate process exit status;
• avoid business logic.

Non-responsibilities:

• config loading;
• backend construction;
• bundle validation;
• publish decisions.

internal/cli

CLI command definitions, flags, argument parsing, and command wiring.

Expected MVP commands:

• distributor run — run configured pipelines.
• distributor run --dry-run — plan without modifying destinations.
• distributor run --pipeline <id> — run one configured pipeline.
• distributor validate <path> — validate a source bundle or source tree where feasible.
• distributor inspect <path> — inspect a bundle or destination state where feasible.

Boundaries:

• CLI should call internal/app use cases.
• CLI should not parse manifests directly except through application APIs.
• CLI should not import backend adapter implementation details unless only for registration side effects.

internal/app

Application orchestration and top-level use cases.

Responsibilities:

• load and validate configuration;
• construct configured pipelines;
• build source and destination backends through registries;
• orchestrate discovery, validation, planning, publishing, and notification;
• coordinate dry-run output;
• run destination fan-out deterministically and sequentially;
• aggregate destination outcomes into run-level failure behavior.

Core orchestration shape:

for each selected pipeline:
  open source backend
  discover source bundles
  for each source bundle:
    validate source manifest and digest
    for each destination:
      inspect .distributor.json
      build publish plan
      transform as required by that destination
      execute publish plan unless dry-run
      run noop notifier after actual publication or replacement

Boundaries:

• internal/app composes packages but should not contain backend-specific logic.
• Publish decisions should live in internal/publish, not inline in orchestration.
• Destination state comparison should live in internal/state or internal/publish, not CLI code.

internal/config

Configuration structs, defaults, loading, precedence, and validation.

Responsibilities:

• load /usr/local/etc/distributor/config.yml by default;
• support --config override;
• apply defaults;
• validate required fields;
• validate pipeline ids and destination ids;
• validate backend-specific config shapes;
• validate transform and publish policy combinations.

MVP config model:

pipelines:
  - id: weather-daily
    source:
      backend: local
      path: /var/spool/distributor/weather
    validation:
      on_digest_mismatch: fail
    destinations:
      - id: markdown-archive
        backend: s3
        endpoint: https://s3.example.com
        bucket: reports
        prefix: weather/archive
        region: us-east-1
        force_path_style: true
        publish:
          source: true
          html: false
        transfer:
          on_destination_same: skip
          on_destination_older: replace
          on_destination_newer: skip
          on_conflict: fail
      - id: static-site
        backend: ssh
        uri: ssh://deploy@example.com:22
        path: /srv/www/weather
        publish:
          source: false
          html: true
        transform:
          markdown_to_html:
            enabled: true
            mode: sidecar

Configuration principles:

• one source per pipeline;
• one or more destinations per pipeline;
• transforms are destination-specific;
• publish policy is destination-specific;
• secrets should use environment variables, secret files, SSH agent, or standard credential mechanisms rather than raw YAML values.

internal/bundle

Source bundle contract and validation.

Responsibilities:

• parse source manifest.json;
• represent source manifests and files;
• discover bundle roots beneath a configured source root;
• validate required manifest fields;
• validate RFC3339 created values;
• validate relative paths;
• validate file existence, size, per-file SHA-256, and bundle digest;
• expose normalized source bundle models to other packages.

Core types:

type Manifest struct {
    SchemaVersion int
    ID      string
    Digest  string
    Created time.Time
    Files   []ManifestFile
}

type ManifestFile struct {
    Path   string
    SHA256 string
    Size   int64
}

type Bundle struct {
    RootRelativePath string
    Manifest         Manifest
}

Boundaries:

• internal/bundle does not know about .distributor.json.
• internal/bundle does not know about destinations, transforms, or notification.
• internal/bundle may use the storage abstraction to read source files, but it should not import backend adapter packages.

internal/state

Destination state contract for .distributor.json.

Responsibilities:

• parse .distributor.json;
• validate destination state;
• represent copied source outputs and generated outputs;
• embed the source manifest used for publication;
• compare destination state against a current source manifest;
• classify destination state as same, older, newer, conflict, absent, invalid, or unmanaged.

Core types:

type DistributorState struct {
    SchemaVersion int
    DistributorVersion string
    PipelineID    string
    DestinationID string
    PublishedAt   time.Time
    Source        SourceState
    Outputs       []OutputFile
}

type SourceState struct {
    Manifest bundle.Manifest
}

type OutputFile struct {
    Path       string
    Kind       string // source | generated
    SourcePath string
    Transform  string
    SHA256     string
    Size       int64
}

Comparison rules:

• same source manifest: skip;
• same source id, older destination source created: replace;
• same source id, newer destination source created: skip;
• same source id, same created, different digest: conflict;
• different source id: conflict;
• pipeline id or destination id mismatch: conflict;
• absent state: publish only if safe;
• unmanaged non-empty path: fail.

Boundaries:

• internal/state owns destination state semantics, not publish execution.
• internal/state should not know about S3, SSH/SFTP, local filesystem details, or Markdown rendering.

internal/storage

Backend abstraction and shared storage types.

Responsibilities:

• define storage backend interfaces;
• define object/file metadata types;
• define path/prefix helpers;
• define common storage errors;
• provide backend registry mechanisms;
• provide a fake backend for core package tests.

The detailed storage contract is defined in docs/roadmap/storage.md. Core application code should use that storage interface for backend-rooted logical paths, byte and stream IO, metadata, traversal, typed errors, emptiness checks, and managed deletion.

Destructive APIs should remain narrow. Prefer managed deletion of files recorded in .distributor.json instead of broad recursive deletion.

Boundaries:

• internal/storage should not contain backend implementation details.
• Adapter dependencies must not leak through storage interfaces.
• The fake backend exists for tests and should not become an application runtime backend.

internal/adapters/local

Local filesystem backend.

Responsibilities:

• implement storage.Backend for local paths;
• clean and constrain paths;
• perform safe reads/writes/listing/deletion;
• use atomic writes where practical;
• reject unsafe path traversal;
• handle symlink policy explicitly.

Testing expectations:

• use temporary directories;
• verify path traversal rejection;
• verify write and delete safety.

internal/adapters/ssh

SSH/SFTP backend.

Responsibilities:

• implement storage.Backend over SSH/SFTP;
• support uri and path config;
• prefer native SFTP implementation;
• use SSH agent, key files, known hosts, or documented auth mechanisms;
• avoid raw passwords in config unless explicitly designed and documented later;
• translate SSH/SFTP errors into storage-level errors.

Testing expectations:

• core app tests should use fake backends;
• adapter tests may use local test servers or targeted integration tests if practical;
• do not require a real production SSH host for normal unit tests.

internal/adapters/s3

S3-compatible object storage backend.

Responsibilities:

• implement storage.Backend over S3-compatible object storage;
• support endpoint, bucket, prefix, region, and force-path-style configuration;
• support standard credential mechanisms or explicit environment-variable references;
• treat S3 as an object tree, not a filesystem;
• set reasonable content types where practical;
• guard against prefix/root deletion mistakes.

Testing expectations:

• core app tests should use fake backends;
• adapter behavior may be tested through mocks, local S3-compatible services, or narrow integration tests;
• config examples should avoid real secrets.

internal/transform

Transform interfaces, registry, and transform planning.

Responsibilities:

• define transform interfaces;
• register available transforms;
• represent transform requests and outputs;
• keep transform execution independent of destination backend details.

Boundaries:

• transforms operate on source bundle content and destination transform config;
• transforms do not publish outputs;
• transforms do not mutate source bundles;
• transforms should return generated output metadata for .distributor.json.

internal/transform/markdown

Markdown-to-HTML implementation.

Responsibilities:

• render listed Markdown files to HTML;
• support MVP sidecar behavior, such as report.md -> report.html;
• record generated output path, source path, transform name, SHA-256, and size;
• optionally use embedded templates if needed.

MVP scope:

• Markdown to HTML only;
• no PDF generation;
• no email-specific HTML;
• no complex theming unless required for basic output correctness.

internal/publish

Destination planning, reconciliation, safety checks, and publish execution.

Responsibilities:

• inspect destination state;
• plan destination action;
• enforce destination conflict rules;
• enforce destructive-operation safety rules;
• detect output path collisions before writing;
• combine source files and transform outputs according to destination publish policy;
• write destination outputs;
• write .distributor.json;
• use staging or equivalent cleanup behavior where practical;
• support dry-run planning;
• report skipped, replaced, failed, and published actions.

Action model:

publish
replace
skip_same
skip_destination_newer
fail_conflict
fail_unmanaged

Boundaries:

• publish logic should not parse CLI flags;
• publish logic should not know adapter implementation details;
• publish logic should use internal/state for destination state semantics;
• publish logic should use internal/storage interfaces for IO.

internal/notify

Notification stage abstraction.

MVP responsibilities:

• define notifier interface;
• implement no-op notifier;
• preserve future extension point for email, ntfy, Gotify, RSS update hooks, or other notification channels.

Future notification rules:

• notify only after successful publication to the relevant destination or destinations;
• notification must be idempotent with respect to source id, digest, pipeline id, and destination id where applicable;
• notification should not run for skipped or failed publications unless explicitly configured.

internal/logging

Logging setup and helpers.

Responsibilities:

• centralize structured logging setup;
• ensure logs omit secrets;
• provide consistent fields for pipeline id, bundle id, destination id, backend, path, action, and reason.

Deferred Ideas

The following are intentionally out of MVP unless separately accepted in a later roadmap:

• email, ntfy, Gotify, or other real notification adapters;
• RSS/Atom feed generation;
• PDF generation;
• web UI;
• full-text search;
• dynamic plugin loading;
• arbitrary transform chains;
• workflow DAGs;
• producer execution;
• complex templating/theming;
• bidirectional sync;
• backup semantics.

Key Invariants

• Producer apps own source bundle creation.
• distributor owns destination publication state.
• Source manifest.json is not copied as destination state.
• Destination .distributor.json is the managed sentinel.
• One pipeline has one source and one or more destinations.
• Transform and publish policy are destination-specific.
• Source files are canonical; HTML is derived.
• Destructive replacement is allowed only inside managed destination bundle paths.
• Core logic must be testable without real S3, SSH, or remote services.