distributor/docs/policy/architecture.md

Architecture

This document defines the development principles for distributor. It is inward-facing: developers and LLM coding agents should use it to preserve the project’s shape, boundaries, and invariants as the code evolves.

Project Scope

distributor is a domain-agnostic report bundle distribution tool.

Producer applications generate manifested bundles. distributor discovers those bundles, validates them, optionally derives publication artifacts such as HTML, and publishes selected source and generated artifacts to one or more configured destinations.

distributor does not generate domain reports, interpret domain-specific report content, run producer pipelines, edit reports, or act as a CMS. Weather reports, D&D recaps, calendar summaries, email digests, and additional report types should all enter distributor through the same bundle contract.

Project Shape

Default to a small, explicit, dependency-light Go application. Keep the design modular enough to test and change safely, but do not add abstraction unless it protects a real boundary or enables a real extension point.

Business logic should live outside CLI, transport, and external-adapter packages. The core application should reason in terms of pipelines, bundles, destination state, transforms, and publish plans—not S3 SDK calls, SFTP sessions, shell commands, or filesystem details.

The current core workflow is:

1. load configured pipelines;
2. open the source backend;
3. discover source bundles beneath the source root;
4. validate each source bundle and its manifest.json;
5. select the source bundle or bundles for each destination according to that destination's path mapping policy;
6. open each destination backend independently;
7. inspect destination state at the resolved destination bundle path;
8. compare source state to destination state;
9. build a publish plan that selects source files, generated files, destination paths, and optional public URLs;
10. optionally transform Markdown to HTML for that destination;
11. publish selected source and generated artifacts;
12. write .distributor.json as the destination sentinel/state file;
13. run the notification hook, whose default implementation is currently a no-op.

Pipeline Model

A pipeline has exactly one source and one or more destinations.

The source is discovered and validated once. Each destination has independent backend configuration, path mapping, publication policy, transform policy, public link policy, replacement behavior, state, and notification behavior.

The pipeline model is fan-out by design:

source bundle
  -> destination A: source files only
  -> destination B: HTML only
  -> destination C: source files + HTML

Destination-specific behavior must not leak back into the source bundle contract. A producer should not need to know whether a bundle will be published to local storage, another storage backend, a static site, email, RSS, or another notification channel.

Source Bundle Contract

A source bundle is a directory containing manifest.json.

manifest.json is the sole producer-to-distributor contract. distributor must not rely on producer-specific work directory layouts, filenames, metadata, or conventions outside the configured source root and the source manifest.

The source manifest schema is intentionally minimal:

{
  "schema_version": 1,
  "id": "weather.daily.brentwood.2026-05-30",
  "digest": "sha256:...",
  "created": "2026-05-30T11:10:00Z",
  "files": [
    {
      "path": "report.md",
      "sha256": "sha256:...",
      "size": 12345
    }
  ]
}

Required fields:

• schema_version: source manifest schema version. Current value: 1.
• id: stable bundle identifier.
• digest: SHA-256 digest for the listed files.
• created: RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
• files: ordered file list.
• files[].path: relative path beneath the bundle root.
• files[].sha256: SHA-256 digest of the file contents.
• files[].size: file size in bytes.

Bundle paths must be relative, clean, and confined to the bundle root. Paths must not be absolute, empty, contain .. path traversal, or otherwise escape the bundle root. Symlink handling must be explicit; unless documented otherwise, source bundle symlinks should be rejected.

Before processing a bundle, distributor must validate file existence, file size, each file SHA-256, and the bundle digest. Digest mismatch must fail in the MVP before any destination writes occur.

The source manifest should remain minimal. Routing, destination selection, publication format, credentials, static-site layout, notification recipients, and destination-specific metadata belong in distributor configuration and destination state, not in producer manifests.

Destination State Contract

manifest.json from the source bundle is not copied to destinations as destination state.

Each destination bundle path is managed by .distributor.json. This file is both the destination sentinel and the destination state record.

Publish execution writes single-owner or shared-root destination state according to destination state.mode. In shared-root state, one .distributor.json records multiple pipeline/destination owners and every managed output carries its owner identity.

Single-owner state records:

• distributor state schema version;
• pipeline id;
• destination id;
• publication timestamp;
• state creation and update timestamps;
• single-owner state mode;
• reconciliation mode;
• the normalized source manifest used for publication;
• metadata for copied source outputs;
• metadata for generated outputs, such as HTML files;
• optional URL metadata for published outputs;
• any additional metadata required by distributor.

A representative destination state file is:

{
  "schema_version": 2,
  "distributor_version": "0.1.0",
  "pipeline_id": "weather-daily",
  "destination_id": "static-html",
  "published_at": "2026-05-30T11:12:00Z",
  "created_at": "2026-05-30T11:12:00Z",
  "updated_at": "2026-05-30T11:12:00Z",
  "state": {
    "mode": "single_owner"
  },
  "reconciliation": {
    "mode": "replace"
  },
  "source": {
    "manifest": {
      "schema_version": 1,
      "id": "weather.daily.brentwood.2026-05-30",
      "digest": "sha256:...",
      "created": "2026-05-30T11:10:00Z",
      "files": [
        {
          "path": "report.md",
          "sha256": "sha256:...",
          "size": 12345
        }
      ]
    }
  },
  "links": {
    "primary_url": "https://reports.example.com/weather-daily/"
  },
  "outputs": [
    {
      "path": "index.html",
      "kind": "generated",
      "source_path": "report.md",
      "transform": "markdown_to_html",
      "sha256": "sha256:...",
      "size": 23456,
      "url": "https://reports.example.com/weather-daily/",
      "created_at": "2026-05-30T11:12:00Z",
      "updated_at": "2026-05-30T11:12:00Z"
    }
  ]
}

Destination comparison rules are based on .distributor.json:

• No .distributor.json: publish normally only if the destination bundle path is empty.
• Existing state embeds the same normalized source manifest: skip as already published.
• Existing state has the same source id and an older source created: replace, subject to destructive-operation safety rules.
• Existing state has the same source id and a newer source created: skip because the destination is newer than the source.
• Existing state has the same source id and same created but different digest: fail as a conflict.
• Existing valid managed state has a different source id, pipeline id, destination id, or shared-root output owner: replace only when destination takeover.mode permits that ownership transfer; otherwise fail as a conflict.

For older destination state, destination reconciliation.mode controls output updates. replace rewrites the managed output set to match the new plan. merge retains omitted managed outputs, overwrites only existing managed paths, and rejects planned paths that collide with unmanaged storage content.

Publication and Transform Policy

Source files are canonical bundle artifacts. Transform outputs are derived publication artifacts.

Transforms are configured per destination. A destination may receive source files, generated HTML files, or both.

The MVP supports only Markdown-to-HTML transformation. HTML generation must not mutate the source bundle. Generated outputs must be deterministic from the source bundle and destination transform configuration, and must be recorded in .distributor.json.

Destination path mapping and public link generation are destination behavior. Source manifests do not declare where a bundle is published or which public URLs are recorded.

The application should distinguish:

• transform policy: how derived files are generated;
• publish policy: which source and generated files a destination receives.

For example, one destination may publish source files only as a long-term archive, while another publishes HTML only as a static site.

Backend Abstraction

Sources and destinations use the same storage abstraction. Current runtime execution uses the local filesystem, SSH/SFTP, and S3-compatible backends. Additional storage backends should be peer implementations behind the same interface, and any backend-specific execution limitation must be documented.

Application logic must interact with storage through internal backend interfaces. Backend-specific behavior belongs in adapter packages. Pipeline, bundle, state, publish, and transform packages must not import service-specific or filesystem adapter implementation details.

Adapters should be thin. Backend adapters should implement storage operations and translate backend-specific errors, but should not make bundle comparison, transform, routing, or replacement decisions.

Remote file-transfer support should prefer native protocol implementations over shelling out, unless a later design document records a reason to differ.

Dependency Policy

Prefer the Go standard library where practical.

Use external dependencies only when justified by correctness, security, interoperability, or substantial complexity reduction. Good reasons include YAML parsing, S3-compatible storage integration, SSH/SFTP integration, and Markdown rendering.

Avoid dependencies for small conveniences. Do not let external dependency types leak across internal package boundaries unless the dependency is itself the explicit public contract of that package.

Package Layout

Use this current layout unless the project has a documented reason to differ:

• cmd/distributor: application entrypoint only.
• pkg/bundle: public producer-facing source manifest model, digest logic, parsing, manifest building, complete local bundle writing, and local validation helpers.
• pkg/upload: public producer-facing HTTP upload client built on pkg/bundle.
• internal/app: application orchestration and top-level use cases.
• internal/cli: CLI command definitions, flags, argument parsing, and command wiring.
• internal/config: configuration structs, defaults, loading, precedence, and validation.
• internal/bundle: storage-backed source bundle discovery and validation over the public manifest contract.
• internal/state: .distributor.json parsing, validation, comparison, and output metadata.
• internal/link: shared HTTP URL validation for configured and persisted link metadata.
• internal/storage: backend interfaces, shared path/resource types, backend registry, and storage errors.
• internal/adapters/local: local filesystem backend.
• internal/adapters/ssh: SSH/SFTP backend.
• internal/adapters/s3: S3-compatible object storage backend.
• internal/transform: transform interfaces, registry, planning, and shared transform models.
• internal/transform/markdown: Markdown-to-HTML implementation.
• internal/publish: destination planning, reconciliation, safety checks, and publish execution.
• internal/notify: notification interface and MVP no-op notifier.
• internal/logging: logging setup and shared logging helpers.

New storage adapters should live under internal/adapters/<name> and stay thin.

Package-private implementation constants may live near the package that owns them, preferably in constants.go when useful.

Configuration

Centralize configuration loading, processing, precedence, defaults, and validation in internal/config.

The goal is to make configuration discoverable and avoid implicit or hidden operational values. User-visible defaults and cross-package operational defaults should be defined in internal/config/defaults.go.

Unless documented otherwise, precedence is:

1. CLI flags
2. environment variables
3. configuration file
4. built-in defaults

Prefer YAML configuration. Config files should be discovered at /usr/local/etc/distributor/config.yml, with a CLI override via --config.

Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables, secret files, SSH agent usage, standard AWS credential mechanisms, or explicitly named environment variable references for secrets.

Pipeline configuration should express:

• pipeline id;
• one source backend;
• one or more destinations;
• per-destination path mapping;
• per-destination publish policy;
• per-destination transform policy;
• per-destination public link policy;
• validation behavior;
• per-destination takeover behavior;
• destination conflict/replacement behavior.

Modules and Registries

Each major workflow step should have an explicit input/output contract:

• source discovery;
• source validation;
• destination bundle selection;
• destination state inspection;
• destination comparison;
• transform planning/execution;
• publish planning;
• publish execution;
• notification.

If users can select backends, transforms, notifiers, or renderers, selection should go through a registry or equivalent mechanism rather than scattered conditionals.

The orchestrator should be able to plan, dry-run, and execute configured pipelines. Dry-run behavior should be first-class because the application may delete, overwrite, or publish files.

Embedded Assets

Store embedded JSON schemas, Markdown templates, HTML templates, CSS, and similar assets as separate files, not inline string literals, unless there is a strong reason otherwise.

Errors and Logging

Errors should be actionable and preserve context. Wrap errors with operation, pipeline id, destination id, backend, path, bundle id, and resource context where useful. CLI code should convert internal errors into concise user-facing messages.

Errors and logs must not expose secrets.

Use structured logging where practical. Logs should describe discovery, validation, planned actions, skipped transfers, conflicts, replacements, external calls, retries, and failure causes, but should not include large report contents by default.

Skip and no-op decisions should be logged at an appropriate level so operators can distinguish successful publication from intentional no-op behavior.

Context, Timeouts, and Cancellation

Long-running operations should accept context.Context. Storage operations, service requests, transforms, and multi-step workflows should respect cancellation and timeouts.

State, Files, and Safety

If the application writes durable state, writes should be atomic where practical. Multi-step workflows should preserve enough state to support inspection, retry, or resume after failure.

Code that deletes, moves, or overwrites files must use narrow, explicit paths. Avoid broad parent-directory operations. Cleanup that can cause data loss must be opt-in.

distributor must never perform broad deletion against a configured source root. Destination deletion must be bounded to the resolved destination bundle path for the current source bundle and backend root.

Normal destructive replacement may occur only when a valid .distributor.json confirms that the destination bundle path is distributor-managed. Explicit forced replacement is a per-run CLI workflow for supported conflict and unmanaged-content cases; it must be dry-runnable, clearly reported, and constrained to the destination bundle path.

Replacement must be narrow, reported, test-covered, and configurable. Prefer normal replacement that deletes files recorded in .distributor.json and known generated outputs. Forced replacement may delete a bounded destination bundle prefix only when the operator explicitly requests it. Backend implementations must guard against path traversal, prefix confusion, and accidental deletion above the configured backend root.

Where practical, publish operations should use staging paths or temporary objects and promote them into place only after validation and transform steps succeed.

Testing

Core logic should be testable without real external services. Use fakes, fixtures, or local test doubles for adapters where practical.

Config examples should be load-tested. Important CLI workflows should have parser or command tests. Component contracts should have focused tests that do not require running the full application unless end-to-end coverage is intentional.

Important tests include:

• source manifest parsing and validation;
• file size and SHA-256 validation;
• bundle digest validation;
• source bundle discovery beneath a source root;
• relative path safety and path traversal rejection;
• destination .distributor.json parsing and comparison;
• same/older/newer/conflict publish decisions;
• destination bundle path mapping;
• destructive replacement safety checks;
• transform output planning and metadata recording;
• public URL planning and state metadata;
• dry-run output;
• local backend behavior with temporary directories;
• fake backend behavior for storage-facing core logic.

Documentation

Documentation should follow the project documentation policy. Keep user docs focused on implemented behavior. Put future, planned, or aspirational work only under docs/roadmap/.

When changing architecture, config, CLI behavior, adapters, manifest/state contracts, transform behavior, publish behavior, public package/API behavior, or component contracts, update the relevant docs and examples in the same change.

The source manifest and destination .distributor.json schemas should have canonical documentation once implemented. Producer-facing package and API workflows belong under docs/consumers/. Example configs should be valid and load-tested where practical.

Non-Goals

distributor is not:

• a report generator;
• a domain-specific weather, D&D, calendar, or email summarization app;
• a workflow DAG engine;
• a CMS;
• a web authoring interface;
• a full-text search service;
• a general-purpose file synchronization tool;
• a backup system;
• a notification platform.

Additional notification, feed, template, or transform behavior must preserve the core bundle-distribution boundary.