distributor/docs/roadmap/contracts.md

Distributor Contracts Roadmap

This roadmap defines the contracts that distributor should implement before or alongside the MVP. The goal is to make bundle validation, destination state, digest verification, and safe replacement deterministic and testable before backend-specific publication behavior is layered on top.

Purpose

distributor publishes manifested report bundles produced by other applications. Producer applications own domain-specific report generation. distributor owns validation, optional transformation, destination publication, and destination state.

The MVP contract has two durable files:

• manifest.json: source-owned bundle manifest produced by the upstream application.
• .distributor.json: destination-owned publication state written by distributor.

manifest.json is not copied to the destination as destination state. Instead, .distributor.json records the normalized source manifest, generated output metadata, and distributor-owned publication metadata.

Terminology

• Source root: Configured root path for a pipeline source.
• Bundle root: Directory beneath the source root that contains manifest.json.
• Relative bundle path: Bundle root path relative to the source root.
• Destination root: Configured root path or prefix for a destination.
• Destination bundle path: Destination root plus the relative bundle path, unless a future mapping option overrides that behavior.
• Source artifact: File listed in the source manifest.json.
• Generated artifact: File created by distributor, such as an HTML file derived from Markdown.
• Destination state: .distributor.json at the destination bundle path.

Source Bundle Contract

A source bundle is a directory containing a manifest.json file. For MVP, the manifest schema is intentionally minimal.

Required manifest.json fields

{
  "schema_version": 1,
  "id": "weather.daily.brentwood.2026-05-30",
  "digest": "sha256:...",
  "created": "2026-05-30T11:10:00Z",
  "files": [
    {
      "path": "report.md",
      "sha256": "sha256:...",
      "size": 12345
    }
  ]
}

Required top-level fields:

• schema_version: Source manifest schema version. MVP value: 1.
• id: Stable bundle identifier. Required, non-empty string.
• digest: Bundle digest. Required, lowercase sha256:<64 hex> string.
• created: Bundle creation timestamp. Required RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
• files: Non-empty array of file objects.

Required file fields:

• path: Relative path from bundle root to source artifact.
• sha256: Per-file digest as lowercase sha256:<64 hex>.
• size: File size in bytes.

No other source manifest fields are required for the MVP. Additional fields may be ignored unless later documented. Destination state records the normalized source manifest model, not raw unknown manifest fields.

Source Path Safety Rules

For every files[].path:

• Path must be relative.
• Path must not be empty.
• Path must not contain .. segments.
• Path must not resolve outside the bundle root.
• Path must use slash-separated logical paths in the manifest.
• Absolute paths are invalid.
• Symlinks should be rejected for MVP unless a later policy deliberately supports them.
• manifest.json itself should not be listed as a source artifact.
• .distributor.json should not be listed as a source artifact.
• Duplicate logical file paths are invalid after path normalization.

The implementation should validate paths before reading file contents.

Digest Contract

The MVP validates both per-file digests and the bundle digest.

Per-file digest

For each file listed in files, compute:

sha256(file bytes)

The computed digest must match files[].sha256.

The actual file size must match files[].size.

Bundle digest

The bundle digest is computed from the listed file records in the listed order. The canonical algorithm is:

1. For each file listed in files, in order:
   • validate the file path;
   • compute the file SHA256;
   • determine the file size.
2. Construct a canonical JSON array containing only:
   • path;
   • sha256;
   • size.
3. Preserve the source manifest's file order.
4. Encode the array deterministically with:
   • object fields in exactly this order: path, sha256, size;
   • no extra spaces;
   • no trailing newline;
   • lowercase sha256:<64 hex> digest strings.
5. Compute sha256(canonical JSON bytes).
6. Compare the result to top-level digest.

Conceptual canonical payload:

[
  {"path":"report.md","sha256":"sha256:...","size":12345},
  {"path":"summary.txt","sha256":"sha256:...","size":234}
]

This avoids ambiguous concatenation of file bytes while keeping the manifest small.

Implementation fixtures should include at least one reference manifest and canonical payload with known per-file and bundle digests.

Digest Mismatch Behavior

Digest validation is always fatal in the MVP. A digest mismatch must fail the affected bundle, pipeline, and run before any destination writes occur.

Warning-only digest behavior is intentionally deferred until a later roadmap accepts transitional ingestion semantics.

Bundle Discovery Contract

A source path may contain one bundle or a tree of bundles. Discovery should scan beneath the configured source root for manifest.json files.

For each discovered manifest:

• Bundle root is the directory containing manifest.json.
• Relative bundle path is computed relative to source root.
• Destination bundle path is destination root plus relative bundle path, unless a future mapping option overrides it.

If nested manifests are found, the MVP should fail with a clear error unless a later policy defines nested-bundle semantics.

Destination State Contract

Destinations are managed by .distributor.json, not by copying manifest.json.

A destination bundle path is considered distributor-managed only when it contains a valid .distributor.json written by distributor. Force or unmanaged-overwrite behavior is not part of the MVP.

Required .distributor.json shape

{
  "schema_version": 1,
  "distributor_version": "0.1.0",
  "pipeline_id": "weather-daily",
  "destination_id": "static-site",
  "published_at": "2026-05-30T11:12:00Z",
  "source": {
    "manifest": {
      "schema_version": 1,
      "id": "weather.daily.brentwood.2026-05-30",
      "digest": "sha256:...",
      "created": "2026-05-30T11:10:00Z",
      "files": [
        {
          "path": "report.md",
          "sha256": "sha256:...",
          "size": 12345
        }
      ]
    }
  },
  "outputs": [
    {
      "path": "report.html",
      "kind": "generated",
      "source_path": "report.md",
      "transform": "markdown_to_html",
      "sha256": "sha256:...",
      "size": 23456
    }
  ]
}

Required fields:

• schema_version: Destination state schema version. MVP value: 1.
• distributor_version: Optional diagnostic distributor version. It must not affect source comparison.
• pipeline_id: Pipeline that produced the destination publication.
• destination_id: Destination within the pipeline.
• published_at: RFC3339 timestamp.
• source.manifest: Normalized source manifest model used for this publication.
• outputs: Array of files written by distributor for this destination.

Required output fields:

• path: Destination-relative output path within the destination bundle path.
• kind: source or generated.
• source_path: Source artifact path that produced this output. For copied source files, this should equal path unless renamed by a future feature.
• transform: Transform identifier for generated files. Empty or omitted may be allowed for copied source files.
• sha256: Output file digest as lowercase sha256:<64 hex>.
• size: Output file size in bytes.

Destination Comparison Rules

Destination comparison uses .distributor.json, not destination manifest.json.

For a source bundle and destination bundle path:

No .distributor.json

If no .distributor.json exists and the destination path is empty:

• Publish normally.

If no .distributor.json exists and the destination path is non-empty:

• Fail as unmanaged content.

Local destination paths are empty when the destination bundle directory does not exist or exists with no entries. S3-compatible destination prefixes are empty when no objects exist below the destination bundle prefix, ignoring objects outside that exact prefix.

Same source manifest

If .distributor.json exists and source.manifest exactly matches the current normalized source manifest:

• Skip as already published.

Same source id, destination older

If .distributor.json exists, source.manifest.id matches the source manifest id, and destination source.manifest.created is older than the source created:

• Replace destination contents, subject to replacement safety rules.

Same source id, destination newer

If .distributor.json exists, source.manifest.id matches the source manifest id, and destination source.manifest.created is newer than the source created:

• Skip and log that destination is newer than source.

Same source id and created, different digest

If .distributor.json exists, source.manifest.id and created match but digest differs:

• Fail as a conflict.

Different source id

If .distributor.json exists and source.manifest.id differs from the source manifest id:

• Fail as a conflict.

Replacement Safety Rules

Replacement is destructive and must be narrow.

distributor must never perform broad deletion against a configured destination root.

Replacement may occur only at a resolved destination bundle path when:

• a valid .distributor.json exists at that destination bundle path; and
• the state identifies the path as distributor-managed; and
• the replacement decision follows the destination comparison rules.

For MVP, replacement should delete only known managed outputs where practical:

• files listed in existing .distributor.json.outputs;
• existing .distributor.json;
• empty directories created by those files, where applicable for filesystem-like backends.

For S3, replacement should delete only objects under the destination bundle prefix that are listed in .distributor.json.outputs plus .distributor.json, unless a later managed-prefix deletion policy is explicitly implemented.

Before any write, planned destination output paths must be checked for collisions. For example, if report.md is copied as a source artifact and Markdown transformation would also generate report.html, but report.html is already a source artifact or another generated output, planning must fail before writing.

Transform Output Contract

Source files are canonical. Generated files are derived publication artifacts.

For MVP, the only supported transform is Markdown to HTML.

Recommended MVP behavior:

• Transform is configured per destination.
• Markdown source files are files listed in manifest.json with .md extension.
• Generated HTML files are sidecars by default.
• report.md generates report.html.
• Source bundles are never mutated.
• Generated outputs are recorded in .distributor.json.outputs.
• Raw HTML embedded in Markdown is escaped or disabled by default for the MVP to keep generated output deterministic and conservative.

Future versions may add templates, index.html, CSS assets, email-safe HTML, and per-file transform selection.

Destination Output Contract

Each destination chooses which categories of files it receives.

MVP categories:

• source: copied source artifacts listed in manifest.json.
• html: generated HTML derived from Markdown source artifacts.

Examples:

• Markdown archive: source: true, html: false.
• Static HTML site: source: false, html: true.
• Full mirror: source: true, html: true.

Every file written to the destination must be represented in .distributor.json.outputs.

Atomicity and Partial Failure

The MVP should prefer staging and promotion where backend semantics permit it.

Minimum behavior:

• Validate source before writing destination files.
• Do not write .distributor.json until all configured destination outputs are successfully written.
• If publication fails before .distributor.json is written, the destination must not be treated as successfully published on a later run.
• Local publication must use staging or equivalent cleanup behavior so a failed write does not leave a confusing unmanaged destination bundle path.
• Later cleanup may remove orphaned files, but MVP correctness should rely on .distributor.json as the success marker.

Example Source Bundle

weather/daily/brentwood/2026-05-30/
  manifest.json
  report.md
  summary.txt

Example manifest:

{
  "schema_version": 1,
  "id": "weather.daily.brentwood.2026-05-30",
  "digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
  "created": "2026-05-30T11:10:00Z",
  "files": [
    {
      "path": "report.md",
      "sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
      "size": 12345
    },
    {
      "path": "summary.txt",
      "sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222",
      "size": 234
    }
  ]
}

Example Destination Bundle: HTML Only

weather/daily/brentwood/2026-05-30/
  report.html
  .distributor.json

Example Destination Bundle: Source Archive

weather/daily/brentwood/2026-05-30/
  report.md
  summary.txt
  .distributor.json

Implementation Stages

1. Define Go structs for source manifest and destination state.
2. Implement source path validation.
3. Implement per-file SHA256 and size validation.
4. Implement canonical bundle digest validation.
5. Implement source bundle discovery beneath a source root.
6. Implement .distributor.json parsing and validation.
7. Implement destination comparison rules.
8. Implement replacement safety checks.
9. Add fixture bundles for valid, invalid, duplicate path, older, newer, same, and conflict scenarios.
10. Add reference canonical digest fixtures.
11. Use the contract layer from the publish pipeline and backend adapters.