distributor/docs/roadmap/packages.md

# Package Layout Roadmap

This roadmap defines the proposed package layout, boundaries, and implementation responsibilities for the `distributor` MVP.

`distributor` is expected to be a domain-agnostic bundle publisher. Producer applications emit source bundles containing `manifest.json`; `distributor` validates those bundles and publishes selected source and generated artifacts to one or more configured destinations.

This document is roadmap material. It describes the intended package design before implementation and should move into `docs/internal/` only after corresponding behavior exists.

## Accepted Package Layout

```text
cmd/distributor/
  main.go

internal/app/
  app.go
  run.go
  pipeline.go

internal/cli/
  root.go
  run.go
  validate.go
  inspect.go

internal/config/
  config.go
  defaults.go
  load.go
  validate.go

internal/bundle/
  manifest.go
  digest.go
  validate.go
  discover.go

internal/state/
  distributor.go
  compare.go
  validate.go

internal/storage/
  backend.go
  registry.go
  path.go
  errors.go

internal/storage/fake/
  backend.go

internal/adapters/local/
  backend.go

internal/adapters/ssh/
  backend.go
  config.go

internal/adapters/s3/
  backend.go
  config.go

internal/transform/
  transform.go
  registry.go
  plan.go

internal/transform/markdown/
  markdown.go
  template.go

internal/publish/
  plan.go
  execute.go
  output.go
  reconcile.go
  safety.go

internal/notify/
  notify.go
  noop.go

internal/logging/
  logging.go
```

## Package Responsibilities

### `cmd/distributor`

Application entrypoint only.

Responsibilities:

- call CLI execution;
- translate process exit status;
- avoid business logic.

Non-responsibilities:

- config loading;
- backend construction;
- bundle validation;
- publish decisions.

### `internal/cli`

CLI command definitions, flags, argument parsing, and command wiring.

Expected MVP commands:

- `distributor run` — run configured pipelines.
- `distributor run --dry-run` — plan without modifying destinations.
- `distributor run --pipeline <id>` — run one configured pipeline.
- `distributor validate <path>` — validate a source bundle or source tree where feasible.
- `distributor inspect <path>` — inspect a bundle or destination state where feasible.

Boundaries:

- CLI should call `internal/app` use cases.
- CLI should not parse manifests directly except through application APIs.
- CLI should not import backend adapter implementation details unless only for registration side effects.

### `internal/app`

Application orchestration and top-level use cases.

Responsibilities:

- load and validate configuration;
- construct configured pipelines;
- build source and destination backends through registries;
- orchestrate discovery, validation, planning, publishing, and notification;
- coordinate dry-run output;
- run destination fan-out deterministically and sequentially;
- aggregate destination outcomes into run-level failure behavior.

Core orchestration shape:

```text
for each selected pipeline:
  open source backend
  discover source bundles
  for each source bundle:
    validate source manifest and digest
    for each destination:
      inspect .distributor.json
      build publish plan
      transform as required by that destination
      execute publish plan unless dry-run
      run noop notifier
```

Boundaries:

- `internal/app` composes packages but should not contain backend-specific logic.
- Publish decisions should live in `internal/publish`, not inline in orchestration.
- Destination state comparison should live in `internal/state` or `internal/publish`, not CLI code.

### `internal/config`

Configuration structs, defaults, loading, precedence, and validation.

Responsibilities:

- load `/usr/local/etc/distributor/config.yml` by default;
- support `--config` override;
- apply defaults;
- validate required fields;
- validate pipeline ids and destination ids;
- validate backend-specific config shapes;
- validate transform and publish policy combinations.

MVP config model:

```yaml
pipelines:
  - id: weather-daily
    source:
      backend: local
      path: /var/spool/distributor/weather
    validation:
      on_digest_mismatch: fail
    destinations:
      - id: markdown-archive
        backend: s3
        endpoint: https://s3.example.com
        bucket: reports
        prefix: weather/archive
        region: us-east-1
        force_path_style: true
        publish:
          source: true
          html: false
        transfer:
          on_destination_same: skip
          on_destination_older: replace
          on_destination_newer: skip
          on_conflict: fail
      - id: static-site
        backend: ssh
        uri: ssh://deploy@example.com:22
        path: /srv/www/weather
        publish:
          source: false
          html: true
        transform:
          markdown_to_html:
            enabled: true
            mode: sidecar
```

Configuration principles:

- one source per pipeline;
- one or more destinations per pipeline;
- transforms are destination-specific;
- publish policy is destination-specific;
- secrets should use environment variables, secret files, SSH agent, or standard credential mechanisms rather than raw YAML values.

### `internal/bundle`

Source bundle contract and validation.

Responsibilities:

- parse source `manifest.json`;
- represent source manifests and files;
- discover bundle roots beneath a configured source root;
- validate required manifest fields;
- validate RFC3339 `created` values;
- validate relative paths;
- validate file existence, size, per-file SHA-256, and bundle digest;
- expose normalized source bundle models to other packages.

Core types:

```go
type Manifest struct {
    SchemaVersion int
    ID      string
    Digest  string
    Created time.Time
    Files   []ManifestFile
}

type ManifestFile struct {
    Path   string
    SHA256 string
    Size   int64
}

type Bundle struct {
    RootRelativePath string
    Manifest         Manifest
}
```

Boundaries:

- `internal/bundle` does not know about `.distributor.json`.
- `internal/bundle` does not know about destinations, transforms, or notification.
- `internal/bundle` may use the storage abstraction to read source files, but it should not import backend adapter packages.

### `internal/state`

Destination state contract for `.distributor.json`.

Responsibilities:

- parse `.distributor.json`;
- validate destination state;
- represent copied source outputs and generated outputs;
- embed the source manifest used for publication;
- compare destination state against a current source manifest;
- classify destination state as same, older, newer, conflict, absent, invalid, or unmanaged.

Core types:

```go
type DistributorState struct {
    SchemaVersion int
    DistributorVersion string
    PipelineID    string
    DestinationID string
    PublishedAt   time.Time
    Source        SourceState
    Outputs       []OutputFile
}

type SourceState struct {
    Manifest bundle.Manifest
}

type OutputFile struct {
    Path       string
    Kind       string // source | generated
    SourcePath string
    Transform  string
    SHA256     string
    Size       int64
}
```

Comparison rules:

- same source manifest: skip;
- same source id, older destination source `created`: replace;
- same source id, newer destination source `created`: skip;
- same source id, same `created`, different digest: conflict;
- different source id: conflict;
- absent state: publish only if safe;
- unmanaged non-empty path: fail.

Boundaries:

- `internal/state` owns destination state semantics, not publish execution.
- `internal/state` should not know about S3, SSH/SFTP, local filesystem details, or Markdown rendering.

### `internal/storage`

Backend abstraction and shared storage types.

Responsibilities:

- define storage backend interfaces;
- define object/file metadata types;
- define path/prefix helpers;
- define common storage errors;
- provide backend registry mechanisms.
- provide a fake backend for core package tests.

The core application should use storage interfaces such as:

```go
type Backend interface {
    ReadFile(ctx context.Context, path string) ([]byte, error)
    WriteFile(ctx context.Context, path string, data []byte, opts WriteOptions) error
    Exists(ctx context.Context, path string) (bool, error)
    List(ctx context.Context, prefix string) ([]Entry, error)
    DeleteFiles(ctx context.Context, paths []string) error
}
```

Destructive APIs should remain narrow. Prefer deleting explicit files recorded in `.distributor.json` instead of broad recursive deletion.

Boundaries:

- `internal/storage` should not contain backend implementation details.
- Adapter dependencies must not leak through storage interfaces.
- The fake backend exists for tests and should not become an application runtime backend.

### `internal/adapters/local`

Local filesystem backend.

Responsibilities:

- implement `storage.Backend` for local paths;
- clean and constrain paths;
- perform safe reads/writes/listing/deletion;
- use atomic writes where practical;
- reject unsafe path traversal;
- handle symlink policy explicitly.

Testing expectations:

- use temporary directories;
- verify path traversal rejection;
- verify write and delete safety.

### `internal/adapters/ssh`

SSH/SFTP backend.

Responsibilities:

- implement `storage.Backend` over SSH/SFTP;
- support `uri` and `path` config;
- prefer native SFTP implementation;
- use SSH agent, key files, known hosts, or documented auth mechanisms;
- avoid raw passwords in config unless explicitly designed and documented later;
- translate SSH/SFTP errors into storage-level errors.

Testing expectations:

- core app tests should use fake backends;
- adapter tests may use local test servers or targeted integration tests if practical;
- do not require a real production SSH host for normal unit tests.

### `internal/adapters/s3`

S3-compatible object storage backend.

Responsibilities:

- implement `storage.Backend` over S3-compatible object storage;
- support endpoint, bucket, prefix, region, and force-path-style configuration;
- support standard credential mechanisms or explicit environment-variable references;
- treat S3 as an object tree, not a filesystem;
- set reasonable content types where practical;
- guard against prefix/root deletion mistakes.

Testing expectations:

- core app tests should use fake backends;
- adapter behavior may be tested through mocks, local S3-compatible services, or narrow integration tests;
- config examples should avoid real secrets.

### `internal/transform`

Transform interfaces, registry, and transform planning.

Responsibilities:

- define transform interfaces;
- register available transforms;
- represent transform requests and outputs;
- keep transform execution independent of destination backend details.

Boundaries:

- transforms operate on source bundle content and destination transform config;
- transforms do not publish outputs;
- transforms do not mutate source bundles;
- transforms should return generated output metadata for `.distributor.json`.

### `internal/transform/markdown`

Markdown-to-HTML implementation.

Responsibilities:

- render listed Markdown files to HTML;
- support MVP sidecar behavior, such as `report.md` -> `report.html`;
- record generated output path, source path, transform name, SHA-256, and size;
- optionally use embedded templates if needed.

MVP scope:

- Markdown to HTML only;
- no PDF generation;
- no email-specific HTML;
- no complex theming unless required for basic output correctness.

### `internal/publish`

Destination planning, reconciliation, safety checks, and publish execution.

Responsibilities:

- inspect destination state;
- plan destination action;
- enforce destination conflict rules;
- enforce destructive-operation safety rules;
- detect output path collisions before writing;
- combine source files and transform outputs according to destination publish policy;
- write destination outputs;
- write `.distributor.json`;
- use staging or equivalent cleanup behavior where practical;
- support dry-run planning;
- report skipped, replaced, failed, and published actions.

Action model:

```text
publish
replace
skip_same
skip_destination_newer
fail_conflict
fail_unmanaged
```

Boundaries:

- publish logic should not parse CLI flags;
- publish logic should not know adapter implementation details;
- publish logic should use `internal/state` for destination state semantics;
- publish logic should use `internal/storage` interfaces for IO.

### `internal/notify`

Notification stage abstraction.

MVP responsibilities:

- define notifier interface;
- implement no-op notifier;
- preserve future extension point for email, ntfy, Gotify, RSS update hooks, or other notification channels.

Future notification rules:

- notify only after successful publication to the relevant destination or destinations;
- notification must be idempotent with respect to source id, digest, pipeline id, and destination id where applicable;
- notification should not run for skipped or failed publications unless explicitly configured.

### `internal/logging`

Logging setup and helpers.

Responsibilities:

- centralize structured logging setup;
- ensure logs omit secrets;
- provide consistent fields for pipeline id, bundle id, destination id, backend, path, action, and reason.

## Implementation Slices

### Slice 1: Skeleton, config, storage, and source bundle validation

Deliver:

- basic CLI skeleton;
- config loading and validation;
- storage interface, local backend, and fake backend;
- source manifest model;
- source bundle discovery;
- file size and SHA-256 validation;
- bundle digest validation;
- local backend sufficient for validation;
- fixtures for valid and invalid bundles.

Useful commands:

```bash
distributor validate ./examples/weather-bundle
```

### Slice 2: Destination state and dry-run planning

Deliver:

- `.distributor.json` model;
- destination state comparison;
- publish plan model;
- dry-run output;
- local source to local destination planning;
- tests for same/older/newer/conflict/unmanaged cases.

Useful command:

```bash
distributor run --config ./examples/local.yml --dry-run
```

### Slice 3: Local publish execution

Deliver:

- local destination writes;
- source-file publication;
- `.distributor.json` writes;
- replacement safety checks;
- skip behavior;
- narrow deletion behavior based on destination state outputs.

Useful command:

```bash
distributor run --config ./examples/local.yml
```

### Slice 4: Markdown-to-HTML transform

Deliver:

- Markdown transform registry;
- Markdown-to-HTML implementation;
- per-destination `publish.source` and `publish.html` behavior;
- generated output metadata in `.distributor.json`;
- tests for source-only, HTML-only, and source-plus-HTML destinations.

### Slice 5: S3 backend

Deliver:

- S3-compatible backend;
- endpoint/bucket/prefix/region/force-path-style config;
- credential handling via environment or standard mechanisms;
- object listing, read, write, and narrow delete operations;
- dry-run and publish coverage using fake or local-compatible test strategy.

### Slice 6: SSH/SFTP backend

Deliver:

- native SFTP backend;
- `uri` and `path` config;
- documented authentication behavior;
- read/write/list/delete operations;
- adapter tests or documented integration test strategy.

### Slice 7: No-op notification stage and future extension seam

Deliver:

- no-op notifier wired into orchestration;
- clear internal contract for future email/ntfy adapters;
- no user-facing notification behavior beyond no-op unless implemented.

## Deferred Ideas

The following are intentionally out of MVP unless separately accepted in a later roadmap:

- email, ntfy, Gotify, or other real notification adapters;
- RSS/Atom feed generation;
- PDF generation;
- web UI;
- full-text search;
- dynamic plugin loading;
- arbitrary transform chains;
- workflow DAGs;
- producer execution;
- complex templating/theming;
- bidirectional sync;
- backup semantics.

## Key Invariants

- Producer apps own source bundle creation.
- `distributor` owns destination publication state.
- Source `manifest.json` is not copied as destination state.
- Destination `.distributor.json` is the managed sentinel.
- One pipeline has one source and one or more destinations.
- Transform and publish policy are destination-specific.
- Source files are canonical; HTML is derived.
- Destructive replacement is allowed only inside managed destination bundle paths.
- Core logic must be testable without real S3, SSH, or remote services.