Add public bundle manifest package

This commit is contained in:
2026-06-01 20:52:23 +00:00
parent e51bc28b05
commit bb68cb6602
17 changed files with 941 additions and 181 deletions

View File

@@ -4,6 +4,8 @@
It is a local-first CLI with SSH/SFTP and S3-compatible storage support: source bundles can be read from local or remote storage, destinations can be local directories or remote paths, and Markdown files can be rendered to HTML sidecars. It is a local-first CLI with SSH/SFTP and S3-compatible storage support: source bundles can be read from local or remote storage, destinations can be local directories or remote paths, and Markdown files can be rendered to HTML sidecars.
Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, parse, write, and validate source bundle manifests with the same contract used by `distributor`.
Run the local example pipeline: Run the local example pipeline:
```sh ```sh

View File

@@ -2,7 +2,7 @@
## Purpose ## Purpose
`internal/bundle` parses, discovers, and validates source bundles through the storage interface. `internal/bundle` discovers and validates source bundles through the storage interface. The source manifest model, manifest parsing, manifest validation, path rules, and digest calculation come from `pkg/bundle` so producer-facing APIs and distributor validation share one manifest contract.
## Inputs and outputs ## Inputs and outputs
@@ -22,9 +22,9 @@ Each file requires `path`, `sha256`, and `size`. Digests must use lowercase `sha
## Validation ## Validation
`ValidateManifest` owns normalized source manifest semantics: schema version, id, digest format, timestamp presence, file list presence, source path safety, duplicate file paths, reserved paths, file digest format, non-negative file sizes, and the top-level bundle digest. `pkg/bundle.ValidateManifest` owns normalized source manifest semantics: schema version, id, digest format, timestamp presence, file list presence, source path safety, duplicate file paths, reserved paths, file digest format, non-negative file sizes, and the top-level bundle digest.
Storage-backed bundle validation additionally checks file existence, regular-file type, file size, and per-file SHA-256. Storage-backed bundle validation in `internal/bundle` additionally checks file existence, regular-file type, file size, and per-file SHA-256 for configured storage backends.
The bundle digest is SHA-256 of a deterministic JSON array of file records in manifest order with fields `path`, `sha256`, and `size`. The bundle digest is SHA-256 of a deterministic JSON array of file records in manifest order with fields `path`, `sha256`, and `size`.
@@ -38,11 +38,11 @@ Manifest parsing and validation fail before destination planning. Storage-backed
## Boundaries ## Boundaries
Bundle code uses `internal/storage` and does not import concrete adapters. CLI local path support is wired in `internal/app`. Internal bundle discovery uses `internal/storage` and does not import concrete adapters. Producer-side local filesystem manifest building and validation belong to `pkg/bundle`. CLI local path support is wired in `internal/app`.
## Tests ## Tests
Before changing bundle behavior, inspect tests under `internal/bundle`. Before changing bundle behavior, inspect tests under `pkg/bundle` and `internal/bundle`.
## Invariants ## Invariants

View File

@@ -76,6 +76,28 @@ Each published destination bundle contains `.distributor.json`. This file is the
Do not edit `.distributor.json` by hand during normal operation. If it is missing or invalid while destination files remain, `distributor` treats the destination as unmanaged or conflicted. Do not edit `.distributor.json` by hand during normal operation. If it is missing or invalid while destination files remain, `distributor` treats the destination as unmanaged or conflicted.
## Go Producer Manifests
Go producer applications can import `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to create source manifests with the same path, digest, timestamp, and validation rules used by `distributor`.
Minimal producer-side manifest creation:
```go
manifest, err := bundle.BuildManifest(bundle.BuildOptions{
Root: outputDir,
ID: "reports.example.2026-05-30",
Files: []string{"report.md", "summary.txt"},
})
if err != nil {
return err
}
if err := bundle.WriteManifest(outputDir, manifest, bundle.WriteManifestOptions{}); err != nil {
return err
}
```
Use explicit `Files` to preserve caller order, or `Scan: true` to recursively include regular files under `Root` in deterministic slash-path order. Scan mode includes dotfiles, excludes files named `manifest.json` or `.distributor.json`, and rejects symlinks.
## Dry Runs ## Dry Runs
`--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH `known_hosts` entries. `--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH `known_hosts` entries.

View File

@@ -28,7 +28,7 @@ The current core workflow is:
8. optionally transform Markdown to HTML for that destination; 8. optionally transform Markdown to HTML for that destination;
9. publish selected source and generated artifacts; 9. publish selected source and generated artifacts;
10. write `.distributor.json` as the destination sentinel/state file; 10. write `.distributor.json` as the destination sentinel/state file;
11. run the notification hook, which is a no-op in the MVP. 11. run the notification hook, whose default implementation is currently a no-op.
## Pipeline Model ## Pipeline Model
@@ -53,7 +53,7 @@ A source bundle is a directory containing `manifest.json`.
`manifest.json` is the sole producer-to-`distributor` contract. `distributor` must not rely on producer-specific work directory layouts, filenames, metadata, or conventions outside the configured source root and the source manifest. `manifest.json` is the sole producer-to-`distributor` contract. `distributor` must not rely on producer-specific work directory layouts, filenames, metadata, or conventions outside the configured source root and the source manifest.
The MVP source manifest schema is intentionally minimal: The source manifest schema is intentionally minimal:
```json ```json
{ {
@@ -73,7 +73,7 @@ The MVP source manifest schema is intentionally minimal:
Required fields: Required fields:
- `schema_version`: source manifest schema version. MVP value: `1`. - `schema_version`: source manifest schema version. Current value: `1`.
- `id`: stable bundle identifier. - `id`: stable bundle identifier.
- `digest`: SHA-256 digest for the listed files. - `digest`: SHA-256 digest for the listed files.
- `created`: RFC3339 timestamp. UTC is preferred; explicit offsets are allowed. - `created`: RFC3339 timestamp. UTC is preferred; explicit offsets are allowed.
@@ -187,10 +187,11 @@ Avoid dependencies for small conveniences. Do not let external dependency types
Use this current layout unless the project has a documented reason to differ: Use this current layout unless the project has a documented reason to differ:
- `cmd/distributor`: application entrypoint only. - `cmd/distributor`: application entrypoint only.
- `pkg/bundle`: public producer-facing source manifest model, digest logic, parsing, building, manifest writing, and local validation helpers.
- `internal/app`: application orchestration and top-level use cases. - `internal/app`: application orchestration and top-level use cases.
- `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring. - `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
- `internal/config`: configuration structs, defaults, loading, precedence, and validation. - `internal/config`: configuration structs, defaults, loading, precedence, and validation.
- `internal/bundle`: source manifest parsing, source bundle discovery, source digest validation, and source bundle model. - `internal/bundle`: storage-backed source bundle discovery and validation over the public manifest contract.
- `internal/state`: `.distributor.json` parsing, validation, comparison, and output metadata. - `internal/state`: `.distributor.json` parsing, validation, comparison, and output metadata.
- `internal/storage`: backend interfaces, shared path/resource types, backend registry, and storage errors. - `internal/storage`: backend interfaces, shared path/resource types, backend registry, and storage errors.
- `internal/adapters/local`: local filesystem backend. - `internal/adapters/local`: local filesystem backend.

View File

@@ -6,10 +6,11 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
## Repository Layout ## Repository Layout
- `cmd/distributor`: executable entrypoint only. - `cmd/distributor`: executable entrypoint only.
- `pkg/bundle`: public producer-facing source manifest helpers.
- `internal/app`: top-level use cases for `run`, `validate`, and `inspect`. - `internal/app`: top-level use cases for `run`, `validate`, and `inspect`.
- `internal/cli`: standard-library command parsing, flags, help text, and command wiring. - `internal/cli`: standard-library command parsing, flags, help text, and command wiring.
- `internal/config`: YAML configuration structs, loading, defaults, and validation. - `internal/config`: YAML configuration structs, loading, defaults, and validation.
- `internal/bundle`: source bundle discovery, manifest parsing, digest calculation, and validation. - `internal/bundle`: storage-backed source bundle discovery and validation using the public manifest contract.
- `internal/state`: destination `.distributor.json` parsing, validation, and comparison. - `internal/state`: destination `.distributor.json` parsing, validation, and comparison.
- `internal/storage`: backend interface, registry, logical path rules, typed errors, and shared storage helpers. - `internal/storage`: backend interface, registry, logical path rules, typed errors, and shared storage helpers.
- `internal/adapters/local`: local filesystem backend. - `internal/adapters/local`: local filesystem backend.
@@ -24,7 +25,7 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.
- `docs`: current user, operator, policy, internal, and roadmap documentation. - `docs`: current user, operator, policy, internal, and roadmap documentation.
- `examples`: copyable example configs and source bundles. - `examples`: copyable example configs and source bundles.
Do not create new top-level package families such as `pkg`, `internal/stage`, Do not create new top-level package families such as public `pkg/...` packages beyond `pkg/bundle`, `internal/stage`,
`internal/modules`, or service-specific adapter directories unless the `internal/modules`, or service-specific adapter directories unless the
architecture policy or a current roadmap explicitly calls for them. architecture policy or a current roadmap explicitly calls for them.
@@ -73,7 +74,7 @@ GOCACHE=/private/tmp/distributor-gocache GOMODCACHE=/private/tmp/distributor-gom
- Keep adapter packages thin. Backend-specific filesystem or service behavior belongs in adapters; bundle, state, transform, and publish policy belongs outside adapters. - Keep adapter packages thin. Backend-specific filesystem or service behavior belongs in adapters; bundle, state, transform, and publish policy belongs outside adapters.
- Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and implemented backend behavior unless the current task explicitly changes them. - Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and implemented backend behavior unless the current task explicitly changes them.
- Use `storage.DisplayPath`, `storage.StateFileName`, `storage.StatePath`, and `storage.ManagedBundleTargets` instead of duplicating those conventions. - Use `storage.DisplayPath`, `storage.StateFileName`, `storage.StatePath`, and `storage.ManagedBundleTargets` instead of duplicating those conventions.
- Use `bundle.ValidateManifest` for normalized source manifest semantics, including embedded source manifests in destination state. - Use `pkg/bundle` for normalized source manifest semantics. Internal packages should reach those rules through `internal/bundle` when they also need storage-backed bundle discovery or validation.
- Use `config.ValidatePublishTransformPolicy` for publish and transform policy combinations. - Use `config.ValidatePublishTransformPolicy` for publish and transform policy combinations.
- Do not import concrete transform implementations from `internal/publish`; app-level wiring owns transform registration. - Do not import concrete transform implementations from `internal/publish`; app-level wiring owns transform registration.
- Do not import `internal/testutil` from production code. - Do not import `internal/testutil` from production code.

View File

@@ -1,12 +1,10 @@
package bundle package bundle
import ( import (
"crypto/sha256"
"encoding/hex"
"fmt" "fmt"
"regexp" "regexp"
"strconv"
"strings" publicbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
) )
var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
@@ -19,31 +17,13 @@ func ValidateDigest(value string) error {
} }
func FileDigest(data []byte) string { func FileDigest(data []byte) string {
sum := sha256.Sum256(data) return publicbundle.FileDigest(data)
return "sha256:" + hex.EncodeToString(sum[:])
} }
func BundleDigest(files []ManifestFile) string { func BundleDigest(files []ManifestFile) string {
canonical := CanonicalFilePayload(files) return publicbundle.BundleDigest(files)
sum := sha256.Sum256([]byte(canonical))
return "sha256:" + hex.EncodeToString(sum[:])
} }
func CanonicalFilePayload(files []ManifestFile) string { func CanonicalFilePayload(files []ManifestFile) string {
var builder strings.Builder return publicbundle.CanonicalFilePayload(files)
builder.WriteByte('[')
for index, file := range files {
if index > 0 {
builder.WriteByte(',')
}
builder.WriteString(`{"path":`)
builder.WriteString(strconv.Quote(file.Path))
builder.WriteString(`,"sha256":`)
builder.WriteString(strconv.Quote(file.SHA256))
builder.WriteString(`,"size":`)
builder.WriteString(strconv.FormatInt(file.Size, 10))
builder.WriteByte('}')
}
builder.WriteByte(']')
return builder.String()
} }

View File

@@ -1,110 +1,32 @@
package bundle package bundle
import ( import publicbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
"bytes"
"encoding/json"
"fmt"
"io"
"time"
)
const ManifestName = "manifest.json" const ManifestName = publicbundle.ManifestName
type Manifest struct { const SchemaVersion = publicbundle.SchemaVersion
SchemaVersion int `json:"schema_version"`
ID string `json:"id"`
Digest string `json:"digest"`
Created time.Time `json:"created"`
Files []ManifestFile `json:"files"`
}
type ManifestFile struct { type Manifest = publicbundle.Manifest
Path string `json:"path"`
SHA256 string `json:"sha256"` type ManifestFile = publicbundle.ManifestFile
Size int64 `json:"size"`
}
type Bundle struct { type Bundle struct {
RootRelativePath string RootRelativePath string
Manifest Manifest Manifest Manifest
} }
type rawManifest struct {
SchemaVersion *int `json:"schema_version"`
ID *string `json:"id"`
Digest *string `json:"digest"`
Created *string `json:"created"`
Files []rawManifestFile `json:"files"`
}
type rawManifestFile struct {
Path *string `json:"path"`
SHA256 *string `json:"sha256"`
Size *int64 `json:"size"`
}
func ParseManifest(data []byte) (Manifest, error) { func ParseManifest(data []byte) (Manifest, error) {
decoder := json.NewDecoder(bytes.NewReader(data)) return publicbundle.ParseManifest(data)
var raw rawManifest
if err := decoder.Decode(&raw); err != nil {
return Manifest{}, fmt.Errorf("parse manifest: %w", err)
}
var extra any
if err := decoder.Decode(&extra); err != io.EOF {
return Manifest{}, fmt.Errorf("parse manifest: trailing data")
}
var manifest Manifest
if raw.SchemaVersion == nil {
return Manifest{}, fmt.Errorf("manifest schema_version is required")
}
manifest.SchemaVersion = *raw.SchemaVersion
if raw.ID == nil || *raw.ID == "" {
return Manifest{}, fmt.Errorf("manifest id is required")
}
manifest.ID = *raw.ID
if raw.Digest == nil || *raw.Digest == "" {
return Manifest{}, fmt.Errorf("manifest digest is required")
}
manifest.Digest = *raw.Digest
if raw.Created == nil || *raw.Created == "" {
return Manifest{}, fmt.Errorf("manifest created is required")
}
created, err := time.Parse(time.RFC3339, *raw.Created)
if err != nil {
return Manifest{}, fmt.Errorf("manifest created must be RFC3339: %w", err)
}
manifest.Created = created
if len(raw.Files) == 0 {
return Manifest{}, fmt.Errorf("manifest files is required")
}
for index, rawFile := range raw.Files {
file, err := parseManifestFile(index, rawFile)
if err != nil {
return Manifest{}, err
}
manifest.Files = append(manifest.Files, file)
}
if err := ValidateManifest(manifest); err != nil {
return Manifest{}, fmt.Errorf("manifest %w", err)
}
return manifest, nil
} }
func parseManifestFile(index int, raw rawManifestFile) (ManifestFile, error) { func MarshalManifest(manifest Manifest) ([]byte, error) {
if raw.Path == nil || *raw.Path == "" { return publicbundle.MarshalManifest(manifest)
return ManifestFile{}, fmt.Errorf("manifest files[%d].path is required", index) }
}
if raw.SHA256 == nil || *raw.SHA256 == "" { func ValidateManifest(manifest Manifest) error {
return ManifestFile{}, fmt.Errorf("manifest files[%d].sha256 is required", index) return publicbundle.ValidateManifest(manifest)
} }
if raw.Size == nil {
return ManifestFile{}, fmt.Errorf("manifest files[%d].size is required", index) func ValidateSourcePath(path string) error {
} return publicbundle.ValidateSourcePath(path)
return ManifestFile{
Path: *raw.Path,
SHA256: *raw.SHA256,
Size: *raw.Size,
}, nil
} }

View File

@@ -7,55 +7,6 @@ import (
"gitea.maximumdirect.net/eric/distributor/internal/storage" "gitea.maximumdirect.net/eric/distributor/internal/storage"
) )
func ValidateSourcePath(path string) error {
if err := storage.ValidatePath(path); err != nil {
return err
}
switch path {
case ManifestName, storage.StateFileName:
return fmt.Errorf("%q is reserved", path)
}
return nil
}
func ValidateManifest(manifest Manifest) error {
if manifest.SchemaVersion != 1 {
return fmt.Errorf("schema_version must be 1")
}
if manifest.ID == "" {
return fmt.Errorf("id is required")
}
if err := ValidateDigest(manifest.Digest); err != nil {
return fmt.Errorf("digest: %w", err)
}
if manifest.Created.IsZero() {
return fmt.Errorf("created is required")
}
if len(manifest.Files) == 0 {
return fmt.Errorf("files is required")
}
seen := make(map[string]struct{}, len(manifest.Files))
for index, file := range manifest.Files {
if err := ValidateSourcePath(file.Path); err != nil {
return fmt.Errorf("files[%d].path: %w", index, err)
}
if err := ValidateDigest(file.SHA256); err != nil {
return fmt.Errorf("files[%d].sha256: %w", index, err)
}
if file.Size < 0 {
return fmt.Errorf("files[%d].size must be non-negative", index)
}
if _, exists := seen[file.Path]; exists {
return fmt.Errorf("files[%d].path duplicates %q", index, file.Path)
}
seen[file.Path] = struct{}{}
}
if actual := BundleDigest(manifest.Files); actual != manifest.Digest {
return fmt.Errorf("digest mismatch: got %s want %s", actual, manifest.Digest)
}
return nil
}
func Validate(ctx context.Context, backend storage.Backend, bundleRoot string) (Bundle, error) { func Validate(ctx context.Context, backend storage.Backend, bundleRoot string) (Bundle, error) {
return validateAt(ctx, backend, bundleRoot, bundleRoot) return validateAt(ctx, backend, bundleRoot, bundleRoot)
} }

118
pkg/bundle/build.go Normal file
View File

@@ -0,0 +1,118 @@
package bundle
import (
"fmt"
"os"
"path"
"path/filepath"
"sort"
"time"
)
func BuildManifest(opts BuildOptions) (Manifest, error) {
if opts.Root == "" {
return Manifest{}, fmt.Errorf("root is required")
}
if opts.ID == "" {
return Manifest{}, fmt.Errorf("id is required")
}
explicit := len(opts.Files) > 0
if explicit == opts.Scan {
return Manifest{}, fmt.Errorf("select exactly one file mode")
}
created := opts.Created
if created.IsZero() {
created = time.Now().UTC()
}
paths := append([]string(nil), opts.Files...)
var err error
if opts.Scan {
paths, err = scanSourcePaths(opts.Root)
if err != nil {
return Manifest{}, err
}
}
files := make([]ManifestFile, 0, len(paths))
for _, sourcePath := range paths {
file, err := buildManifestFile(opts.Root, sourcePath)
if err != nil {
return Manifest{}, err
}
files = append(files, file)
}
manifest := Manifest{
SchemaVersion: SchemaVersion,
ID: opts.ID,
Created: created,
Files: files,
}
manifest.Digest = BundleDigest(manifest.Files)
if err := ValidateManifest(manifest); err != nil {
return Manifest{}, err
}
return manifest, nil
}
func scanSourcePaths(root string) ([]string, error) {
var paths []string
err := filepath.WalkDir(root, func(filePath string, entry os.DirEntry, err error) error {
if err != nil {
return err
}
if filePath == root {
return nil
}
relative, err := filepath.Rel(root, filePath)
if err != nil {
return err
}
sourcePath := filepath.ToSlash(relative)
if entry.Type()&os.ModeSymlink != 0 {
return fmt.Errorf("source file %q must be a regular file", sourcePath)
}
if entry.IsDir() {
return nil
}
if path.Base(sourcePath) == ManifestName || path.Base(sourcePath) == distributorStateName {
return nil
}
info, err := entry.Info()
if err != nil {
return err
}
if !info.Mode().IsRegular() {
return fmt.Errorf("source file %q must be a regular file", sourcePath)
}
paths = append(paths, sourcePath)
return nil
})
if err != nil {
return nil, err
}
sort.Strings(paths)
return paths, nil
}
func buildManifestFile(root, sourcePath string) (ManifestFile, error) {
if err := ValidateSourcePath(sourcePath); err != nil {
return ManifestFile{}, fmt.Errorf("source file %q: %w", sourcePath, err)
}
fullPath := filepath.Join(root, filepath.FromSlash(sourcePath))
info, err := os.Lstat(fullPath)
if err != nil {
return ManifestFile{}, fmt.Errorf("source file %q stat: %w", sourcePath, err)
}
if !info.Mode().IsRegular() {
return ManifestFile{}, fmt.Errorf("source file %q must be a regular file", sourcePath)
}
data, err := os.ReadFile(fullPath)
if err != nil {
return ManifestFile{}, fmt.Errorf("source file %q read: %w", sourcePath, err)
}
return ManifestFile{
Path: sourcePath,
SHA256: FileDigest(data),
Size: int64(len(data)),
}, nil
}

363
pkg/bundle/bundle_test.go Normal file
View File

@@ -0,0 +1,363 @@
package bundle
import (
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
)
func TestBuildManifestExplicitFilesPreservesOrder(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "b.txt", "bravo")
writeFile(t, root, "a.txt", "alpha")
before := time.Now().UTC()
manifest, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.example",
Files: []string{"b.txt", "a.txt"},
})
if err != nil {
t.Fatalf("BuildManifest() error = %v", err)
}
after := time.Now().UTC()
paths := manifestPaths(manifest)
if want := []string{"b.txt", "a.txt"}; !reflect.DeepEqual(paths, want) {
t.Fatalf("paths = %v, want %v", paths, want)
}
if manifest.SchemaVersion != SchemaVersion {
t.Fatalf("schema version = %d, want %d", manifest.SchemaVersion, SchemaVersion)
}
if manifest.ID != "reports.example" {
t.Fatalf("id = %q", manifest.ID)
}
if manifest.Created.Before(before) || manifest.Created.After(after) {
t.Fatalf("created = %s, want between %s and %s", manifest.Created, before, after)
}
if err := ValidateManifest(manifest); err != nil {
t.Fatalf("ValidateManifest() error = %v", err)
}
if err := ValidateBundle(root, manifest); err != nil {
t.Fatalf("ValidateBundle() error = %v", err)
}
}
func TestBuildManifestScanSortsAndFiltersMetadata(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "z.txt", "zulu")
writeFile(t, root, "nested/.hidden", "hidden")
writeFile(t, root, ManifestName, "old manifest")
writeFile(t, root, distributorStateName, "state")
writeFile(t, root, "nested/manifest.json", "nested manifest")
writeFile(t, root, "nested/.distributor.json", "nested state")
created := time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC)
manifest, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.scan",
Created: created,
Scan: true,
})
if err != nil {
t.Fatalf("BuildManifest() error = %v", err)
}
if !manifest.Created.Equal(created) {
t.Fatalf("created = %s, want %s", manifest.Created, created)
}
paths := manifestPaths(manifest)
if want := []string{"nested/.hidden", "z.txt"}; !reflect.DeepEqual(paths, want) {
t.Fatalf("paths = %v, want %v", paths, want)
}
}
func TestBuildManifestRequiresOneFileMode(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "report.txt", "report")
tests := []BuildOptions{
{Root: root, ID: "reports.none"},
{Root: root, ID: "reports.both", Files: []string{"report.txt"}, Scan: true},
}
for _, opts := range tests {
if _, err := BuildManifest(opts); err == nil {
t.Fatalf("BuildManifest(%+v) error = nil, want error", opts)
}
}
}
func TestBuildManifestRejectsUnsafePath(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "report.txt", "report")
_, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.unsafe",
Files: []string{"../report.txt"},
})
if err == nil {
t.Fatal("BuildManifest() error = nil, want unsafe path error")
}
}
func TestBuildManifestRejectsSymlink(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "target.txt", "target")
if err := os.Symlink("target.txt", filepath.Join(root, "link.txt")); err != nil {
t.Skipf("symlink unavailable: %v", err)
}
_, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.symlink",
Files: []string{"link.txt"},
})
if err == nil || !strings.Contains(err.Error(), "regular file") {
t.Fatalf("BuildManifest() error = %v, want regular file error", err)
}
}
func TestScanRejectsSymlink(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "target.txt", "target")
if err := os.Symlink("target.txt", filepath.Join(root, "link.txt")); err != nil {
t.Skipf("symlink unavailable: %v", err)
}
_, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.scan.symlink",
Scan: true,
})
if err == nil || !strings.Contains(err.Error(), "regular file") {
t.Fatalf("BuildManifest() error = %v, want regular file error", err)
}
}
func TestDigestFunctionsUseCanonicalFilePayload(t *testing.T) {
files := []ManifestFile{
{Path: "report.md", SHA256: FileDigest([]byte("report")), Size: 6},
{Path: "summary.txt", SHA256: FileDigest([]byte("summary")), Size: 7},
}
payload := CanonicalFilePayload(files)
if want := `[{"path":"report.md","sha256":"sha256:845e91831319e89c4d656bdb80c278ac09a7230d61e5dfd2e1b1fbb436ac8917","size":6},{"path":"summary.txt","sha256":"sha256:761b7ad8ad439b2855fcbb611331c646ef0870b0631247bba3f3025cb6df5a53","size":7}]`; payload != want {
t.Fatalf("payload = %q, want %q", payload, want)
}
if digest := BundleDigest(files); !strings.HasPrefix(digest, "sha256:") || len(digest) != len("sha256:")+64 {
t.Fatalf("BundleDigest() = %q, want sha256 digest", digest)
}
}
func TestParseMarshalLoadAndWriteManifest(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "report.txt", "report")
manifest, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.json",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Files: []string{"report.txt"},
})
if err != nil {
t.Fatalf("BuildManifest() error = %v", err)
}
data, err := MarshalManifest(manifest)
if err != nil {
t.Fatalf("MarshalManifest() error = %v", err)
}
if !strings.HasSuffix(string(data), "\n") {
t.Fatalf("manifest JSON = %q, want trailing newline", string(data))
}
if !strings.Contains(string(data), `"schema_version": 1`) || !strings.Contains(string(data), `"sha256": "`) {
t.Fatalf("manifest JSON = %q, want fixed manifest fields", string(data))
}
parsed, err := ParseManifest(data)
if err != nil {
t.Fatalf("ParseManifest() error = %v", err)
}
if !reflect.DeepEqual(parsed, manifest) {
t.Fatalf("parsed manifest = %#v, want %#v", parsed, manifest)
}
if err := WriteManifest(root, manifest, WriteManifestOptions{}); err != nil {
t.Fatalf("WriteManifest() error = %v", err)
}
loaded, err := LoadManifest(root)
if err != nil {
t.Fatalf("LoadManifest() error = %v", err)
}
if !reflect.DeepEqual(loaded, manifest) {
t.Fatalf("loaded manifest = %#v, want %#v", loaded, manifest)
}
if err := WriteManifest(root, manifest, WriteManifestOptions{}); err == nil {
t.Fatal("WriteManifest() error = nil, want exists error")
}
manifest.ID = "reports.updated"
manifest.Digest = BundleDigest(manifest.Files)
if err := WriteManifest(root, manifest, WriteManifestOptions{Overwrite: true}); err != nil {
t.Fatalf("WriteManifest(overwrite) error = %v", err)
}
loaded, err = LoadManifest(root)
if err != nil {
t.Fatalf("LoadManifest() after overwrite error = %v", err)
}
if loaded.ID != "reports.updated" {
t.Fatalf("loaded id = %q, want updated", loaded.ID)
}
}
func TestValidateManifestRejectsInvalidManifest(t *testing.T) {
created := time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC)
fileDigest := FileDigest([]byte("report"))
valid := Manifest{
SchemaVersion: SchemaVersion,
ID: "reports.valid",
Created: created,
Files: []ManifestFile{{Path: "report.txt", SHA256: fileDigest, Size: 6}},
}
valid.Digest = BundleDigest(valid.Files)
tests := map[string]func(Manifest) Manifest{
"schema version": func(manifest Manifest) Manifest {
manifest.SchemaVersion = 2
return manifest
},
"id": func(manifest Manifest) Manifest {
manifest.ID = ""
return manifest
},
"created": func(manifest Manifest) Manifest {
manifest.Created = time.Time{}
return manifest
},
"files": func(manifest Manifest) Manifest {
manifest.Files = nil
manifest.Digest = BundleDigest(manifest.Files)
return manifest
},
"path": func(manifest Manifest) Manifest {
manifest.Files[0].Path = "manifest.json"
manifest.Digest = BundleDigest(manifest.Files)
return manifest
},
"duplicate": func(manifest Manifest) Manifest {
manifest.Files = append(manifest.Files, manifest.Files[0])
manifest.Digest = BundleDigest(manifest.Files)
return manifest
},
"size": func(manifest Manifest) Manifest {
manifest.Files[0].Size = -1
manifest.Digest = BundleDigest(manifest.Files)
return manifest
},
"digest": func(manifest Manifest) Manifest {
manifest.Digest = "sha256:0000000000000000000000000000000000000000000000000000000000000000"
return manifest
},
}
for name, mutate := range tests {
t.Run(name, func(t *testing.T) {
if err := ValidateManifest(mutate(valid)); err == nil {
t.Fatal("ValidateManifest() error = nil, want error")
}
})
}
}
func TestValidateBundleRejectsLocalFileProblems(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "report.txt", "report")
manifest, err := BuildManifest(BuildOptions{
Root: root,
ID: "reports.bundle",
Files: []string{"report.txt"},
})
if err != nil {
t.Fatalf("BuildManifest() error = %v", err)
}
if err := ValidateBundle(root, manifest); err != nil {
t.Fatalf("ValidateBundle() error = %v", err)
}
missing := cloneManifest(manifest)
missing.Files[0].Path = "missing.txt"
missing.Digest = BundleDigest(missing.Files)
if err := ValidateBundle(root, missing); err == nil {
t.Fatal("ValidateBundle() missing error = nil, want error")
}
sizeMismatch := cloneManifest(manifest)
sizeMismatch.Files[0].Size++
sizeMismatch.Digest = BundleDigest(sizeMismatch.Files)
if err := ValidateBundle(root, sizeMismatch); err == nil || !strings.Contains(err.Error(), "size mismatch") {
t.Fatalf("ValidateBundle() size error = %v, want size mismatch", err)
}
digestMismatch := cloneManifest(manifest)
writeFile(t, root, "report.txt", "change")
if err := ValidateBundle(root, digestMismatch); err == nil || !strings.Contains(err.Error(), "sha256 mismatch") {
t.Fatalf("ValidateBundle() digest error = %v, want digest mismatch", err)
}
}
func TestValidateBundleRejectsSymlink(t *testing.T) {
root := t.TempDir()
writeFile(t, root, "target.txt", "target")
if err := os.Symlink("target.txt", filepath.Join(root, "link.txt")); err != nil {
t.Skipf("symlink unavailable: %v", err)
}
file := ManifestFile{Path: "link.txt", SHA256: FileDigest([]byte("target")), Size: 6}
manifest := Manifest{
SchemaVersion: SchemaVersion,
ID: "reports.link",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Files: []ManifestFile{file},
}
manifest.Digest = BundleDigest(manifest.Files)
err := ValidateBundle(root, manifest)
if err == nil || !strings.Contains(err.Error(), "regular file") {
t.Fatalf("ValidateBundle() error = %v, want regular file error", err)
}
}
func TestValidateSourcePath(t *testing.T) {
valid := []string{"report.md", "nested/report.md", ".well-known/report.txt"}
for _, path := range valid {
if err := ValidateSourcePath(path); err != nil {
t.Fatalf("ValidateSourcePath(%q) error = %v", path, err)
}
}
invalid := []string{"", "../report.md", "/report.md", "nested/../report.md", `nested\report.md`, ManifestName, distributorStateName}
for _, path := range invalid {
if err := ValidateSourcePath(path); err == nil {
t.Fatalf("ValidateSourcePath(%q) error = nil, want error", path)
}
}
}
func manifestPaths(manifest Manifest) []string {
paths := make([]string, 0, len(manifest.Files))
for _, file := range manifest.Files {
paths = append(paths, file.Path)
}
return paths
}
func cloneManifest(manifest Manifest) Manifest {
manifest.Files = append([]ManifestFile(nil), manifest.Files...)
return manifest
}
func writeFile(t *testing.T, root, relative, body string) {
t.Helper()
path := filepath.Join(root, filepath.FromSlash(relative))
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatalf("mkdir %s: %v", filepath.Dir(path), err)
}
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatalf("write %s: %v", path, err)
}
}

49
pkg/bundle/digest.go Normal file
View File

@@ -0,0 +1,49 @@
package bundle
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"regexp"
"strconv"
"strings"
)
var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
func validateDigest(value string) error {
if !digestPattern.MatchString(value) {
return fmt.Errorf("must be lowercase sha256:<64 hex>")
}
return nil
}
func FileDigest(data []byte) string {
sum := sha256.Sum256(data)
return "sha256:" + hex.EncodeToString(sum[:])
}
func BundleDigest(files []ManifestFile) string {
canonical := CanonicalFilePayload(files)
sum := sha256.Sum256([]byte(canonical))
return "sha256:" + hex.EncodeToString(sum[:])
}
func CanonicalFilePayload(files []ManifestFile) string {
var builder strings.Builder
builder.WriteByte('[')
for index, file := range files {
if index > 0 {
builder.WriteByte(',')
}
builder.WriteString(`{"path":`)
builder.WriteString(strconv.Quote(file.Path))
builder.WriteString(`,"sha256":`)
builder.WriteString(strconv.Quote(file.SHA256))
builder.WriteString(`,"size":`)
builder.WriteString(strconv.FormatInt(file.Size, 10))
builder.WriteByte('}')
}
builder.WriteByte(']')
return builder.String()
}

8
pkg/bundle/doc.go Normal file
View File

@@ -0,0 +1,8 @@
// Package bundle provides producer-facing helpers for distributor source
// bundle manifests.
//
// A source bundle is a local directory containing a manifest.json file and the
// files listed by that manifest. This package owns the public manifest model,
// digest calculation, path validation, manifest parsing, manifest building, and
// local bundle validation used by Go producer applications.
package bundle

View File

@@ -0,0 +1,40 @@
package bundle_test
import (
"fmt"
"os"
"path/filepath"
"time"
"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
)
func ExampleBuildManifest() {
root, err := os.MkdirTemp("", "distributor-bundle-*")
if err != nil {
panic(err)
}
defer os.RemoveAll(root)
if err := os.WriteFile(filepath.Join(root, "report.txt"), []byte("report\n"), 0o600); err != nil {
panic(err)
}
manifest, err := bundle.BuildManifest(bundle.BuildOptions{
Root: root,
ID: "reports.example",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Files: []string{"report.txt"},
})
if err != nil {
panic(err)
}
fmt.Println(manifest.ID)
fmt.Println(manifest.Files[0].Path)
fmt.Println(manifest.Files[0].Size)
// Output:
// reports.example
// report.txt
// 7
}

160
pkg/bundle/manifest.go Normal file
View File

@@ -0,0 +1,160 @@
package bundle
import (
"bytes"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"time"
)
type rawManifest struct {
SchemaVersion *int `json:"schema_version"`
ID *string `json:"id"`
Digest *string `json:"digest"`
Created *string `json:"created"`
Files []rawManifestFile `json:"files"`
}
type rawManifestFile struct {
Path *string `json:"path"`
SHA256 *string `json:"sha256"`
Size *int64 `json:"size"`
}
func ParseManifest(data []byte) (Manifest, error) {
decoder := json.NewDecoder(bytes.NewReader(data))
var raw rawManifest
if err := decoder.Decode(&raw); err != nil {
return Manifest{}, fmt.Errorf("parse manifest: %w", err)
}
var extra any
if err := decoder.Decode(&extra); err != io.EOF {
return Manifest{}, fmt.Errorf("parse manifest: trailing data")
}
var manifest Manifest
if raw.SchemaVersion == nil {
return Manifest{}, fmt.Errorf("manifest schema_version is required")
}
manifest.SchemaVersion = *raw.SchemaVersion
if raw.ID == nil || *raw.ID == "" {
return Manifest{}, fmt.Errorf("manifest id is required")
}
manifest.ID = *raw.ID
if raw.Digest == nil || *raw.Digest == "" {
return Manifest{}, fmt.Errorf("manifest digest is required")
}
manifest.Digest = *raw.Digest
if raw.Created == nil || *raw.Created == "" {
return Manifest{}, fmt.Errorf("manifest created is required")
}
created, err := time.Parse(time.RFC3339, *raw.Created)
if err != nil {
return Manifest{}, fmt.Errorf("manifest created must be RFC3339: %w", err)
}
manifest.Created = created
if len(raw.Files) == 0 {
return Manifest{}, fmt.Errorf("manifest files is required")
}
for index, rawFile := range raw.Files {
file, err := parseManifestFile(index, rawFile)
if err != nil {
return Manifest{}, err
}
manifest.Files = append(manifest.Files, file)
}
if err := ValidateManifest(manifest); err != nil {
return Manifest{}, fmt.Errorf("manifest %w", err)
}
return manifest, nil
}
func parseManifestFile(index int, raw rawManifestFile) (ManifestFile, error) {
if raw.Path == nil || *raw.Path == "" {
return ManifestFile{}, fmt.Errorf("manifest files[%d].path is required", index)
}
if raw.SHA256 == nil || *raw.SHA256 == "" {
return ManifestFile{}, fmt.Errorf("manifest files[%d].sha256 is required", index)
}
if raw.Size == nil {
return ManifestFile{}, fmt.Errorf("manifest files[%d].size is required", index)
}
return ManifestFile{
Path: *raw.Path,
SHA256: *raw.SHA256,
Size: *raw.Size,
}, nil
}
func MarshalManifest(manifest Manifest) ([]byte, error) {
if err := ValidateManifest(manifest); err != nil {
return nil, err
}
data, err := json.MarshalIndent(manifest, "", " ")
if err != nil {
return nil, err
}
return append(data, '\n'), nil
}
func LoadManifest(root string) (Manifest, error) {
data, err := os.ReadFile(filepath.Join(root, ManifestName))
if err != nil {
return Manifest{}, fmt.Errorf("read manifest: %w", err)
}
return ParseManifest(data)
}
func WriteManifest(root string, manifest Manifest, opts WriteManifestOptions) error {
data, err := MarshalManifest(manifest)
if err != nil {
return err
}
manifestPath := filepath.Join(root, ManifestName)
if !opts.Overwrite {
file, err := os.OpenFile(manifestPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o666)
if err != nil {
return fmt.Errorf("write manifest: %w", err)
}
if _, err := file.Write(data); err != nil {
_ = file.Close()
return fmt.Errorf("write manifest: %w", err)
}
if err := file.Close(); err != nil {
return fmt.Errorf("write manifest: %w", err)
}
return nil
}
dir := root
if dir == "" {
dir = "."
}
tmp, err := os.CreateTemp(dir, ".manifest-*.tmp")
if err != nil {
return fmt.Errorf("write manifest temp: %w", err)
}
tmpPath := tmp.Name()
cleanup := true
defer func() {
if cleanup {
_ = os.Remove(tmpPath)
}
}()
if _, err := tmp.Write(data); err != nil {
_ = tmp.Close()
return fmt.Errorf("write manifest temp: %w", err)
}
if err := tmp.Close(); err != nil {
return fmt.Errorf("write manifest temp: %w", err)
}
if err := os.Rename(tmpPath, manifestPath); err != nil {
return fmt.Errorf("replace manifest: %w", err)
}
cleanup = false
return nil
}

31
pkg/bundle/path.go Normal file
View File

@@ -0,0 +1,31 @@
package bundle
import (
"fmt"
"path"
"strings"
)
const distributorStateName = ".distributor.json"
func ValidateSourcePath(value string) error {
if value == "" {
return fmt.Errorf("source path is required")
}
if strings.Contains(value, "\\") || strings.HasPrefix(value, "/") {
return fmt.Errorf("source path %q must be a clean relative slash-separated path", value)
}
if path.Clean(value) != value {
return fmt.Errorf("source path %q must be a clean relative slash-separated path", value)
}
for _, segment := range strings.Split(value, "/") {
if segment == "" || segment == "." || segment == ".." {
return fmt.Errorf("source path %q must be a clean relative slash-separated path", value)
}
}
switch value {
case ManifestName, distributorStateName:
return fmt.Errorf("%q is reserved", value)
}
return nil
}

33
pkg/bundle/types.go Normal file
View File

@@ -0,0 +1,33 @@
package bundle
import "time"
const ManifestName = "manifest.json"
const SchemaVersion = 1
type Manifest struct {
SchemaVersion int `json:"schema_version"`
ID string `json:"id"`
Digest string `json:"digest"`
Created time.Time `json:"created"`
Files []ManifestFile `json:"files"`
}
type ManifestFile struct {
Path string `json:"path"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
type BuildOptions struct {
Root string
ID string
Created time.Time
Files []string
Scan bool
}
type WriteManifestOptions struct {
Overwrite bool
}

79
pkg/bundle/validate.go Normal file
View File

@@ -0,0 +1,79 @@
package bundle
import (
"fmt"
"os"
"path/filepath"
)
func ValidateManifest(manifest Manifest) error {
if manifest.SchemaVersion != SchemaVersion {
return fmt.Errorf("schema_version must be %d", SchemaVersion)
}
if manifest.ID == "" {
return fmt.Errorf("id is required")
}
if err := validateDigest(manifest.Digest); err != nil {
return fmt.Errorf("digest: %w", err)
}
if manifest.Created.IsZero() {
return fmt.Errorf("created is required")
}
if len(manifest.Files) == 0 {
return fmt.Errorf("files is required")
}
seen := make(map[string]struct{}, len(manifest.Files))
for index, file := range manifest.Files {
if err := ValidateSourcePath(file.Path); err != nil {
return fmt.Errorf("files[%d].path: %w", index, err)
}
if err := validateDigest(file.SHA256); err != nil {
return fmt.Errorf("files[%d].sha256: %w", index, err)
}
if file.Size < 0 {
return fmt.Errorf("files[%d].size must be non-negative", index)
}
if _, exists := seen[file.Path]; exists {
return fmt.Errorf("files[%d].path duplicates %q", index, file.Path)
}
seen[file.Path] = struct{}{}
}
if actual := BundleDigest(manifest.Files); actual != manifest.Digest {
return fmt.Errorf("digest mismatch: got %s want %s", actual, manifest.Digest)
}
return nil
}
func ValidateBundle(root string, manifest Manifest) error {
if err := ValidateManifest(manifest); err != nil {
return err
}
files := append([]ManifestFile(nil), manifest.Files...)
for index, manifestFile := range files {
fullPath := filepath.Join(root, filepath.FromSlash(manifestFile.Path))
info, err := os.Lstat(fullPath)
if err != nil {
return fmt.Errorf("file %q stat: %w", manifestFile.Path, err)
}
if !info.Mode().IsRegular() {
return fmt.Errorf("file %q must be a regular file", manifestFile.Path)
}
if info.Size() != manifestFile.Size {
return fmt.Errorf("file %q size mismatch: got %d want %d", manifestFile.Path, info.Size(), manifestFile.Size)
}
data, err := os.ReadFile(fullPath)
if err != nil {
return fmt.Errorf("file %q read: %w", manifestFile.Path, err)
}
actualDigest := FileDigest(data)
if actualDigest != manifestFile.SHA256 {
return fmt.Errorf("file %q sha256 mismatch: got %s want %s", manifestFile.Path, actualDigest, manifestFile.SHA256)
}
files[index].SHA256 = actualDigest
files[index].Size = int64(len(data))
}
if actualDigest := BundleDigest(files); actualDigest != manifest.Digest {
return fmt.Errorf("digest mismatch: got %s want %s", actualDigest, manifest.Digest)
}
return nil
}