distributor/docs/roadmap/public_bundle_package.md

Roadmap: Public Bundle Manifest Package

Purpose

Expose a small public Go package that producer applications can import to create valid distributor source bundle manifests.

The package should encode the producer-side source bundle contract without exposing distributor's publication, storage, transform, destination state, notification, or config internals.

Current Implementation Grounding

The current implementation keeps source bundle behavior in internal/bundle:

• Manifest and ManifestFile model manifest.json;
• ParseManifest parses JSON and RFC3339 created timestamps;
• ValidateManifest owns schema version, digest format, file path, duplicate, size, and bundle digest validation;
• FileDigest, BundleDigest, and CanonicalFilePayload define digest behavior;
• source validation rejects unsafe paths, reserved distributor metadata paths, non-file entries, size mismatches, SHA-256 mismatches, and bundle digest mismatches.

Producers cannot import internal/bundle, so Go producers currently need to duplicate this contract or shell out to future CLI tooling.

Goals

• Provide a stable producer-facing Go API for manifest creation and validation.
• Reuse the same source manifest, digest, path safety, and RFC3339 behavior used by distributor validation.
• Keep the public API intentionally small and producer-only.
• Include a safe bundle writer so producers can create complete local bundle directories without hand-rolling manifest-write and staging behavior.
• Make the future distributor manifest create command a thin wrapper over this package.
• Avoid exposing destination state, publish planning, storage backends, transforms, notifications, or config.

Non-Goals

• Do not expose the distributor runner or publication workflow as public API.
• Do not expose storage backends or destination .distributor.json state.
• Do not add domain-specific manifest metadata.
• Do not require non-Go producers to use Go APIs.
• Do not implement latest paths, link generation, transforms, or notification behavior in this package.

Package Boundary

Use pkg/bundle as the public package name.

The package should own only producer-side source bundle concerns:

• manifest model and schema version constant;
• file digest and bundle digest calculation;
• manifest building from producer files;
• manifest JSON load/write helpers;
• manifest and bundle validation;
• source path safety matching distributor validation.

Prefer options structs over long positional functions so future additive behavior can be introduced without avoidable API churn.

Initial Exported API

The initial pkg/bundle API is locked to the following exported constants, types, and functions. Implementation should not rename, remove, or reshape these public symbols during the initial implementation pass.

const ManifestName = "manifest.json"
const SchemaVersion = 1

type Manifest struct {
    SchemaVersion int            `json:"schema_version"`
    ID            string         `json:"id"`
    Digest        string         `json:"digest"`
    Created       time.Time      `json:"created"`
    Files         []ManifestFile `json:"files"`
}

type ManifestFile struct {
    Path   string `json:"path"`
    SHA256 string `json:"sha256"`
    Size   int64  `json:"size"`
}

type BuildOptions struct {
    Root    string
    ID      string
    Created time.Time
    Files   []string
    Scan    bool
}

type WriteManifestOptions struct {
    Overwrite bool
}

type BundleFile struct {
    SourcePath string
    Path       string
}

type WriteBundleOptions struct {
    Root      string
    ID        string
    Created   time.Time
    Files     []BundleFile
    Overwrite bool
}

func ParseManifest(data []byte) (Manifest, error)
func MarshalManifest(manifest Manifest) ([]byte, error)
func LoadManifest(root string) (Manifest, error)
func WriteManifest(root string, manifest Manifest, opts WriteManifestOptions) error
func BuildManifest(opts BuildOptions) (Manifest, error)
func ValidateManifest(manifest Manifest) error
func ValidateBundle(root string, manifest Manifest) error
func WriteBundle(opts WriteBundleOptions) (Manifest, error)
func ValidateSourcePath(path string) error
func FileDigest(data []byte) string
func BundleDigest(files []ManifestFile) string
func CanonicalFilePayload(files []ManifestFile) string

API Semantics

BuildManifest requires Root, ID, and exactly one file-selection mode: explicit Files or Scan: true.

Explicit Files preserve caller order. Scan: true recursively scans Root, includes regular files including dotfiles, excludes manifest.json and .distributor.json, rejects symlinks, and sorts by slash-separated relative path.

Zero Created values default to the current UTC time. All public path fields use slash-separated bundle-relative paths.

MarshalManifest validates before marshaling and emits deterministic JSON with fixed field order and a trailing newline.

WriteManifest writes manifest.json; it fails if the file exists unless WriteManifestOptions.Overwrite is true, and it uses temp-and-rename replacement where practical.

ValidateManifest checks manifest-only semantics. ValidateBundle checks the supplied manifest against local files under root, including existence, regular-file type, size, SHA-256, path safety, duplicates, and bundle digest.

WriteBundle copies existing local files from BundleFile.SourcePath into a staged bundle at BundleFile.Path, writes a compliant manifest, validates the staged bundle, and promotes it to WriteBundleOptions.Root.

WriteBundleOptions.Overwrite permits replacing an existing bundle root. Replacement must build the new bundle completely before touching the existing root, then use sibling temp and backup paths for best-effort promotion and restore on failure.

The writer remains producer-side and filesystem-local. It must not expose distributor storage backends or publication behavior.

Manifest Compatibility

The package should treat the source manifest schema as a compatibility boundary:

• export the current schema version;
• preserve JSON field names exactly;
• use lowercase sha256:<64 hex> digests;
• use slash-separated relative paths in JSON;
• use RFC3339 timestamps;
• default a zero build or writer Created value to the current UTC time;
• preserve caller-provided file order for explicit file lists;
• produce deterministic ordering when scan-based building is selected;
• reject symlinks if distributor validation still rejects source symlinks.

Scan-based building belongs in v1 of the public package. Explicit file lists should preserve caller order. Scan mode should sort by slash-separated relative path and share the same filtering rules expected by future CLI manifest creation.

The internal implementation may either move source-bundle core logic into pkg/bundle and have internal packages consume it, or keep internal wrappers around public core logic. The important invariant is that public package, future CLI manifest creation, and distributor validation must not drift.

Relationship To Other Roadmaps

distributor manifest create should call pkg/bundle rather than maintaining a separate manifest builder.

Remote validate and inspect should continue using distributor's storage abstraction and internal app wiring; they do not need public producer APIs.

HTML index mode, link generation, and latest path destinations operate after a bundle has already entered distributor and should not affect this package.

Testing Expectations

Suggested coverage:

• build a manifest from explicit files;
• build a manifest by scanning a local bundle root;
• preserve explicit file order;
• sort scan results deterministically by slash-separated relative path;
• compute per-file SHA-256 and size;
• compute the expected canonical bundle digest;
• write and load manifest.json;
• default zero Created to current UTC time while honoring explicit timestamps;
• validate generated manifests successfully;
• reject unsafe paths, missing files, non-regular files, and symlinks;
• emit slash-separated JSON paths;
• parse, marshal, load, and write manifests through the exact exported API;
• fail WriteManifest when manifest.json exists unless overwrite is enabled;
• emit deterministic manifest JSON with fixed field order and trailing newline;
• write a complete local bundle through the public writer;
• copy BundleFile.SourcePath content to the configured bundle-relative path;
• support WriteBundleOptions.Overwrite through staged replacement;
• avoid leaving a completed bundle path without a valid manifest when writer staging or promotion fails where practical;
• compile public examples under go test where practical;
• prove consistency with distributor validation fixtures.

Documentation Updates After Implementation

• Add Go package documentation under pkg/bundle.
• Update README.md to mention Go producer support.
• Update docs/operations.md with a producer integration example.
• Cross-reference distributor manifest create once that CLI command exists.

Keep this roadmap under docs/roadmap/ until implemented.

Implementation Stages

docs/roadmap/implementation.md intentionally splits this roadmap across two implementation prompts: package core/building first, then the local bundle writer. The split keeps the public API extraction separate from producer-side bundle assembly.

Core/building stage:

1. Move or wrap the existing source manifest model, digest logic, path validation, and RFC3339 handling so pkg/bundle and internal validation use one contract.
2. Implement the locked Stage 2 public API symbols: ManifestName, SchemaVersion, Manifest, ManifestFile, BuildOptions, WriteManifestOptions, ParseManifest, MarshalManifest, LoadManifest, WriteManifest, BuildManifest, ValidateManifest, ValidateBundle, ValidateSourcePath, FileDigest, BundleDigest, and CanonicalFilePayload.
3. Add explicit-list and scan-based manifest building APIs, including zero Created defaulting to current UTC time.
4. Update internal packages to consume the shared implementation without changing current validation behavior.

Writer stage:

1. Implement the locked Stage 3 public API symbols: BundleFile, WriteBundleOptions, and WriteBundle.
2. Add the local bundle writer with staged promotion, atomic filesystem operations where practical, and overwrite behavior through sibling temp and backup paths.
3. Add package documentation and producer-facing examples.

Decisions

• A zero Created value defaults to the current UTC time. Producers may still provide explicit timestamps for reproducible or backfilled bundles.
• Scan-based manifest building is included in v1, behind explicit options. Explicit file lists preserve caller order; scan mode sorts deterministically.
• A local bundle writer is included in v1. It should be safe and producer-side, but it must not expose distributor publication or storage internals.
• The exported API names and signatures in Initial Exported API are normative for implementation.

Future Work

• Broader producer workflow helpers, such as richer ignore rules or template scaffolding, can be considered after the first public package exists.
• Remote or storage-backed producer writers remain out of scope unless a future producer use case requires them.