391 lines
14 KiB
Markdown
391 lines
14 KiB
Markdown
# Distributor Contracts Roadmap
|
|
|
|
This roadmap defines the contracts that `distributor` should implement before or alongside the MVP. The goal is to make bundle validation, destination state, digest verification, and safe replacement deterministic and testable before backend-specific publication behavior is layered on top.
|
|
|
|
## Purpose
|
|
|
|
`distributor` publishes manifested report bundles produced by other applications. Producer applications own domain-specific report generation. `distributor` owns validation, optional transformation, destination publication, and destination state.
|
|
|
|
The MVP contract has two durable files:
|
|
|
|
- `manifest.json`: source-owned bundle manifest produced by the upstream application.
|
|
- `.distributor.json`: destination-owned publication state written by `distributor`.
|
|
|
|
`manifest.json` is not copied to the destination as destination state. Instead, `.distributor.json` records the normalized source manifest, generated output metadata, and distributor-owned publication metadata.
|
|
|
|
## Terminology
|
|
|
|
- **Source root**: Configured root path for a pipeline source.
|
|
- **Bundle root**: Directory beneath the source root that contains `manifest.json`.
|
|
- **Relative bundle path**: Bundle root path relative to the source root.
|
|
- **Destination root**: Configured root path or prefix for a destination.
|
|
- **Destination bundle path**: Destination root plus the relative bundle path, unless a future mapping option overrides that behavior.
|
|
- **Source artifact**: File listed in the source `manifest.json`.
|
|
- **Generated artifact**: File created by `distributor`, such as an HTML file derived from Markdown.
|
|
- **Destination state**: `.distributor.json` at the destination bundle path.
|
|
|
|
## Source Bundle Contract
|
|
|
|
A source bundle is a directory containing a `manifest.json` file. For MVP, the manifest schema is intentionally minimal.
|
|
|
|
### Required `manifest.json` fields
|
|
|
|
```json
|
|
{
|
|
"schema_version": 1,
|
|
"id": "weather.daily.brentwood.2026-05-30",
|
|
"digest": "sha256:...",
|
|
"created": "2026-05-30T11:10:00Z",
|
|
"files": [
|
|
{
|
|
"path": "report.md",
|
|
"sha256": "sha256:...",
|
|
"size": 12345
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Required top-level fields:
|
|
|
|
- `schema_version`: Source manifest schema version. MVP value: `1`.
|
|
- `id`: Stable bundle identifier. Required, non-empty string.
|
|
- `digest`: Bundle digest. Required, lowercase `sha256:<64 hex>` string.
|
|
- `created`: Bundle creation timestamp. Required RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
|
|
- `files`: Non-empty array of file objects.
|
|
|
|
Required file fields:
|
|
|
|
- `path`: Relative path from bundle root to source artifact.
|
|
- `sha256`: Per-file digest as lowercase `sha256:<64 hex>`.
|
|
- `size`: File size in bytes.
|
|
|
|
No other source manifest fields are required for the MVP. Additional fields may be ignored unless later documented. Destination state records the normalized source manifest model, not raw unknown manifest fields.
|
|
|
|
## Source Path Safety Rules
|
|
|
|
For every `files[].path`:
|
|
|
|
- Path must be relative.
|
|
- Path must not be empty.
|
|
- Path must not contain `..` segments.
|
|
- Path must not resolve outside the bundle root.
|
|
- Path must use slash-separated logical paths in the manifest.
|
|
- Absolute paths are invalid.
|
|
- Symlinks should be rejected for MVP unless a later policy deliberately supports them.
|
|
- `manifest.json` itself should not be listed as a source artifact.
|
|
- `.distributor.json` should not be listed as a source artifact.
|
|
- Duplicate logical file paths are invalid after path normalization.
|
|
|
|
The implementation should validate paths before reading file contents.
|
|
|
|
## Digest Contract
|
|
|
|
The MVP validates both per-file digests and the bundle digest.
|
|
|
|
### Per-file digest
|
|
|
|
For each file listed in `files`, compute:
|
|
|
|
```text
|
|
sha256(file bytes)
|
|
```
|
|
|
|
The computed digest must match `files[].sha256`.
|
|
|
|
The actual file size must match `files[].size`.
|
|
|
|
### Bundle digest
|
|
|
|
The bundle digest is computed from the listed file records in the listed order. The canonical algorithm is:
|
|
|
|
1. For each file listed in `files`, in order:
|
|
- validate the file path;
|
|
- compute the file SHA256;
|
|
- determine the file size.
|
|
2. Construct a canonical JSON array containing only:
|
|
- `path`;
|
|
- `sha256`;
|
|
- `size`.
|
|
3. Preserve the source manifest's file order.
|
|
4. Encode the array deterministically with:
|
|
- object fields in exactly this order: `path`, `sha256`, `size`;
|
|
- no extra spaces;
|
|
- no trailing newline;
|
|
- lowercase `sha256:<64 hex>` digest strings.
|
|
5. Compute `sha256(canonical JSON bytes)`.
|
|
6. Compare the result to top-level `digest`.
|
|
|
|
Conceptual canonical payload:
|
|
|
|
```json
|
|
[
|
|
{"path":"report.md","sha256":"sha256:...","size":12345},
|
|
{"path":"summary.txt","sha256":"sha256:...","size":234}
|
|
]
|
|
```
|
|
|
|
This avoids ambiguous concatenation of file bytes while keeping the manifest small.
|
|
|
|
Implementation fixtures should include at least one reference manifest and canonical payload with known per-file and bundle digests.
|
|
|
|
## Digest Mismatch Behavior
|
|
|
|
Digest validation is always fatal in the MVP. A digest mismatch must fail the affected bundle, pipeline, and run before any destination writes occur.
|
|
|
|
Warning-only digest behavior is intentionally deferred until a later roadmap accepts transitional ingestion semantics.
|
|
|
|
## Bundle Discovery Contract
|
|
|
|
A source path may contain one bundle or a tree of bundles. Discovery should scan beneath the configured source root for `manifest.json` files.
|
|
|
|
For each discovered manifest:
|
|
|
|
- Bundle root is the directory containing `manifest.json`.
|
|
- Relative bundle path is computed relative to source root.
|
|
- Destination bundle path is destination root plus relative bundle path, unless a future mapping option overrides it.
|
|
|
|
If nested manifests are found, the MVP should fail with a clear error unless a later policy defines nested-bundle semantics.
|
|
|
|
## Destination State Contract
|
|
|
|
Destinations are managed by `.distributor.json`, not by copying `manifest.json`.
|
|
|
|
A destination bundle path is considered distributor-managed only when it contains a valid `.distributor.json` written by `distributor`. Force or unmanaged-overwrite behavior is not part of the MVP.
|
|
|
|
### Required `.distributor.json` shape
|
|
|
|
```json
|
|
{
|
|
"schema_version": 1,
|
|
"distributor_version": "0.1.0",
|
|
"pipeline_id": "weather-daily",
|
|
"destination_id": "static-site",
|
|
"published_at": "2026-05-30T11:12:00Z",
|
|
"source": {
|
|
"manifest": {
|
|
"schema_version": 1,
|
|
"id": "weather.daily.brentwood.2026-05-30",
|
|
"digest": "sha256:...",
|
|
"created": "2026-05-30T11:10:00Z",
|
|
"files": [
|
|
{
|
|
"path": "report.md",
|
|
"sha256": "sha256:...",
|
|
"size": 12345
|
|
}
|
|
]
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"path": "report.html",
|
|
"kind": "generated",
|
|
"source_path": "report.md",
|
|
"transform": "markdown_to_html",
|
|
"sha256": "sha256:...",
|
|
"size": 23456
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Required fields:
|
|
|
|
- `schema_version`: Destination state schema version. MVP value: `1`.
|
|
- `distributor_version`: Optional diagnostic distributor version. It must not affect source comparison.
|
|
- `pipeline_id`: Pipeline that produced the destination publication.
|
|
- `destination_id`: Destination within the pipeline.
|
|
- `published_at`: RFC3339 timestamp.
|
|
- `source.manifest`: Normalized source manifest model used for this publication.
|
|
- `outputs`: Array of files written by `distributor` for this destination.
|
|
|
|
Required output fields:
|
|
|
|
- `path`: Destination-relative output path within the destination bundle path.
|
|
- `kind`: `source` or `generated`.
|
|
- `source_path`: Source artifact path that produced this output. For copied source files, this should equal `path` unless renamed by a future feature.
|
|
- `transform`: Transform identifier for generated files. Empty or omitted may be allowed for copied source files.
|
|
- `sha256`: Output file digest as lowercase `sha256:<64 hex>`.
|
|
- `size`: Output file size in bytes.
|
|
|
|
## Destination Comparison Rules
|
|
|
|
Destination comparison uses `.distributor.json`, not destination `manifest.json`.
|
|
|
|
For a source bundle and destination bundle path:
|
|
|
|
### No `.distributor.json`
|
|
|
|
If no `.distributor.json` exists and the destination path is empty:
|
|
|
|
- Publish normally.
|
|
|
|
If no `.distributor.json` exists and the destination path is non-empty:
|
|
|
|
- Fail as unmanaged content.
|
|
|
|
Local destination paths are empty when the destination bundle directory does not exist or exists with no entries. S3-compatible destination prefixes are empty when no objects exist below the destination bundle prefix, ignoring objects outside that exact prefix.
|
|
|
|
### Same source manifest
|
|
|
|
If `.distributor.json` exists and `source.manifest` exactly matches the current normalized source manifest:
|
|
|
|
- Skip as already published.
|
|
|
|
### Same source id, destination older
|
|
|
|
If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is older than the source `created`:
|
|
|
|
- Replace destination contents, subject to replacement safety rules.
|
|
|
|
### Same source id, destination newer
|
|
|
|
If `.distributor.json` exists, `source.manifest.id` matches the source manifest `id`, and destination `source.manifest.created` is newer than the source `created`:
|
|
|
|
- Skip and log that destination is newer than source.
|
|
|
|
### Same source id and created, different digest
|
|
|
|
If `.distributor.json` exists, `source.manifest.id` and `created` match but `digest` differs:
|
|
|
|
- Fail as a conflict.
|
|
|
|
### Different source id
|
|
|
|
If `.distributor.json` exists and `source.manifest.id` differs from the source manifest `id`:
|
|
|
|
- Fail as a conflict.
|
|
|
|
## Replacement Safety Rules
|
|
|
|
Replacement is destructive and must be narrow.
|
|
|
|
`distributor` must never perform broad deletion against a configured destination root.
|
|
|
|
Replacement may occur only at a resolved destination bundle path when:
|
|
|
|
- a valid `.distributor.json` exists at that destination bundle path; and
|
|
- the state identifies the path as distributor-managed; and
|
|
- the replacement decision follows the destination comparison rules.
|
|
|
|
For MVP, replacement should delete only known managed outputs where practical:
|
|
|
|
- files listed in existing `.distributor.json.outputs`;
|
|
- existing `.distributor.json`;
|
|
- empty directories created by those files, where applicable for filesystem-like backends.
|
|
|
|
For S3, replacement should delete only objects under the destination bundle prefix that are listed in `.distributor.json.outputs` plus `.distributor.json`, unless a later managed-prefix deletion policy is explicitly implemented.
|
|
|
|
Before any write, planned destination output paths must be checked for collisions. For example, if `report.md` is copied as a source artifact and Markdown transformation would also generate `report.html`, but `report.html` is already a source artifact or another generated output, planning must fail before writing.
|
|
|
|
## Transform Output Contract
|
|
|
|
Source files are canonical. Generated files are derived publication artifacts.
|
|
|
|
For MVP, the only supported transform is Markdown to HTML.
|
|
|
|
Recommended MVP behavior:
|
|
|
|
- Transform is configured per destination.
|
|
- Markdown source files are files listed in `manifest.json` with `.md` extension.
|
|
- Generated HTML files are sidecars by default.
|
|
- `report.md` generates `report.html`.
|
|
- Source bundles are never mutated.
|
|
- Generated outputs are recorded in `.distributor.json.outputs`.
|
|
- Raw HTML embedded in Markdown is escaped or disabled by default for the MVP to keep generated output deterministic and conservative.
|
|
|
|
Future versions may add templates, `index.html`, CSS assets, email-safe HTML, and per-file transform selection.
|
|
|
|
## Destination Output Contract
|
|
|
|
Each destination chooses which categories of files it receives.
|
|
|
|
MVP categories:
|
|
|
|
- `source`: copied source artifacts listed in `manifest.json`.
|
|
- `html`: generated HTML derived from Markdown source artifacts.
|
|
|
|
Examples:
|
|
|
|
- Markdown archive: `source: true`, `html: false`.
|
|
- Static HTML site: `source: false`, `html: true`.
|
|
- Full mirror: `source: true`, `html: true`.
|
|
|
|
Every file written to the destination must be represented in `.distributor.json.outputs`.
|
|
|
|
## Atomicity and Partial Failure
|
|
|
|
The MVP should prefer staging and promotion where backend semantics permit it.
|
|
|
|
Minimum behavior:
|
|
|
|
- Validate source before writing destination files.
|
|
- Do not write `.distributor.json` until all configured destination outputs are successfully written.
|
|
- If publication fails before `.distributor.json` is written, the destination must not be treated as successfully published on a later run.
|
|
- Local publication must use staging or equivalent cleanup behavior so a failed write does not leave a confusing unmanaged destination bundle path.
|
|
- Later cleanup may remove orphaned files, but MVP correctness should rely on `.distributor.json` as the success marker.
|
|
|
|
## Example Source Bundle
|
|
|
|
```text
|
|
weather/daily/brentwood/2026-05-30/
|
|
manifest.json
|
|
report.md
|
|
summary.txt
|
|
```
|
|
|
|
Example manifest:
|
|
|
|
```json
|
|
{
|
|
"schema_version": 1,
|
|
"id": "weather.daily.brentwood.2026-05-30",
|
|
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
|
"created": "2026-05-30T11:10:00Z",
|
|
"files": [
|
|
{
|
|
"path": "report.md",
|
|
"sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
|
|
"size": 12345
|
|
},
|
|
{
|
|
"path": "summary.txt",
|
|
"sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222",
|
|
"size": 234
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## Example Destination Bundle: HTML Only
|
|
|
|
```text
|
|
weather/daily/brentwood/2026-05-30/
|
|
report.html
|
|
.distributor.json
|
|
```
|
|
|
|
## Example Destination Bundle: Source Archive
|
|
|
|
```text
|
|
weather/daily/brentwood/2026-05-30/
|
|
report.md
|
|
summary.txt
|
|
.distributor.json
|
|
```
|
|
|
|
## Implementation Stages
|
|
|
|
1. Define Go structs for source manifest and destination state.
|
|
2. Implement source path validation.
|
|
3. Implement per-file SHA256 and size validation.
|
|
4. Implement canonical bundle digest validation.
|
|
5. Implement source bundle discovery beneath a source root.
|
|
6. Implement `.distributor.json` parsing and validation.
|
|
7. Implement destination comparison rules.
|
|
8. Implement replacement safety checks.
|
|
9. Add fixture bundles for valid, invalid, duplicate path, older, newer, same, and conflict scenarios.
|
|
10. Add reference canonical digest fixtures.
|
|
11. Use the contract layer from the publish pipeline and backend adapters.
|