diff --git a/README.md b/README.md index 292ce40..18ba95e 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,8 @@ It is a local-first CLI with SSH/SFTP and S3-compatible storage support: source bundles can be read from local or remote storage, destinations can be local directories or remote paths, and Markdown files can be rendered to HTML sidecars. +Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, parse, write, and validate source bundle manifests with the same contract used by `distributor`. + Run the local example pipeline: ```sh diff --git a/docs/internal/bundle.md b/docs/internal/bundle.md index 4a330ef..026abca 100644 --- a/docs/internal/bundle.md +++ b/docs/internal/bundle.md @@ -2,7 +2,7 @@ ## Purpose -`internal/bundle` parses, discovers, and validates source bundles through the storage interface. +`internal/bundle` discovers and validates source bundles through the storage interface. The source manifest model, manifest parsing, manifest validation, path rules, and digest calculation come from `pkg/bundle` so producer-facing APIs and distributor validation share one manifest contract. ## Inputs and outputs @@ -22,9 +22,9 @@ Each file requires `path`, `sha256`, and `size`. Digests must use lowercase `sha ## Validation -`ValidateManifest` owns normalized source manifest semantics: schema version, id, digest format, timestamp presence, file list presence, source path safety, duplicate file paths, reserved paths, file digest format, non-negative file sizes, and the top-level bundle digest. +`pkg/bundle.ValidateManifest` owns normalized source manifest semantics: schema version, id, digest format, timestamp presence, file list presence, source path safety, duplicate file paths, reserved paths, file digest format, non-negative file sizes, and the top-level bundle digest. -Storage-backed bundle validation additionally checks file existence, regular-file type, file size, and per-file SHA-256. +Storage-backed bundle validation in `internal/bundle` additionally checks file existence, regular-file type, file size, and per-file SHA-256 for configured storage backends. The bundle digest is SHA-256 of a deterministic JSON array of file records in manifest order with fields `path`, `sha256`, and `size`. @@ -38,11 +38,11 @@ Manifest parsing and validation fail before destination planning. Storage-backed ## Boundaries -Bundle code uses `internal/storage` and does not import concrete adapters. CLI local path support is wired in `internal/app`. +Internal bundle discovery uses `internal/storage` and does not import concrete adapters. Producer-side local filesystem manifest building and validation belong to `pkg/bundle`. CLI local path support is wired in `internal/app`. ## Tests -Before changing bundle behavior, inspect tests under `internal/bundle`. +Before changing bundle behavior, inspect tests under `pkg/bundle` and `internal/bundle`. ## Invariants diff --git a/docs/operations.md b/docs/operations.md index bf76f04..886b347 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -76,6 +76,28 @@ Each published destination bundle contains `.distributor.json`. This file is the Do not edit `.distributor.json` by hand during normal operation. If it is missing or invalid while destination files remain, `distributor` treats the destination as unmanaged or conflicted. +## Go Producer Manifests + +Go producer applications can import `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to create source manifests with the same path, digest, timestamp, and validation rules used by `distributor`. + +Minimal producer-side manifest creation: + +```go +manifest, err := bundle.BuildManifest(bundle.BuildOptions{ + Root: outputDir, + ID: "reports.example.2026-05-30", + Files: []string{"report.md", "summary.txt"}, +}) +if err != nil { + return err +} +if err := bundle.WriteManifest(outputDir, manifest, bundle.WriteManifestOptions{}); err != nil { + return err +} +``` + +Use explicit `Files` to preserve caller order, or `Scan: true` to recursively include regular files under `Root` in deterministic slash-path order. Scan mode includes dotfiles, excludes files named `manifest.json` or `.distributor.json`, and rejects symlinks. + ## Dry Runs `--dry-run` loads and validates config, discovers source bundles, inspects destination state, plans outputs, and prints summary lines. It does not write output files, destination state, or SSH `known_hosts` entries. diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index b3a65b2..a631b73 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -28,7 +28,7 @@ The current core workflow is: 8. optionally transform Markdown to HTML for that destination; 9. publish selected source and generated artifacts; 10. write `.distributor.json` as the destination sentinel/state file; -11. run the notification hook, which is a no-op in the MVP. +11. run the notification hook, whose default implementation is currently a no-op. ## Pipeline Model @@ -53,7 +53,7 @@ A source bundle is a directory containing `manifest.json`. `manifest.json` is the sole producer-to-`distributor` contract. `distributor` must not rely on producer-specific work directory layouts, filenames, metadata, or conventions outside the configured source root and the source manifest. -The MVP source manifest schema is intentionally minimal: +The source manifest schema is intentionally minimal: ```json { @@ -73,7 +73,7 @@ The MVP source manifest schema is intentionally minimal: Required fields: -- `schema_version`: source manifest schema version. MVP value: `1`. +- `schema_version`: source manifest schema version. Current value: `1`. - `id`: stable bundle identifier. - `digest`: SHA-256 digest for the listed files. - `created`: RFC3339 timestamp. UTC is preferred; explicit offsets are allowed. @@ -187,10 +187,11 @@ Avoid dependencies for small conveniences. Do not let external dependency types Use this current layout unless the project has a documented reason to differ: - `cmd/distributor`: application entrypoint only. +- `pkg/bundle`: public producer-facing source manifest model, digest logic, parsing, building, manifest writing, and local validation helpers. - `internal/app`: application orchestration and top-level use cases. - `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring. - `internal/config`: configuration structs, defaults, loading, precedence, and validation. -- `internal/bundle`: source manifest parsing, source bundle discovery, source digest validation, and source bundle model. +- `internal/bundle`: storage-backed source bundle discovery and validation over the public manifest contract. - `internal/state`: `.distributor.json` parsing, validation, comparison, and output metadata. - `internal/storage`: backend interfaces, shared path/resource types, backend registry, and storage errors. - `internal/adapters/local`: local filesystem backend. diff --git a/docs/policy/development.md b/docs/policy/development.md index 83db6b0..b07a9cf 100644 --- a/docs/policy/development.md +++ b/docs/policy/development.md @@ -6,10 +6,11 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`. ## Repository Layout - `cmd/distributor`: executable entrypoint only. +- `pkg/bundle`: public producer-facing source manifest helpers. - `internal/app`: top-level use cases for `run`, `validate`, and `inspect`. - `internal/cli`: standard-library command parsing, flags, help text, and command wiring. - `internal/config`: YAML configuration structs, loading, defaults, and validation. -- `internal/bundle`: source bundle discovery, manifest parsing, digest calculation, and validation. +- `internal/bundle`: storage-backed source bundle discovery and validation using the public manifest contract. - `internal/state`: destination `.distributor.json` parsing, validation, and comparison. - `internal/storage`: backend interface, registry, logical path rules, typed errors, and shared storage helpers. - `internal/adapters/local`: local filesystem backend. @@ -24,7 +25,7 @@ Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`. - `docs`: current user, operator, policy, internal, and roadmap documentation. - `examples`: copyable example configs and source bundles. -Do not create new top-level package families such as `pkg`, `internal/stage`, +Do not create new top-level package families such as public `pkg/...` packages beyond `pkg/bundle`, `internal/stage`, `internal/modules`, or service-specific adapter directories unless the architecture policy or a current roadmap explicitly calls for them. @@ -73,7 +74,7 @@ GOCACHE=/private/tmp/distributor-gocache GOMODCACHE=/private/tmp/distributor-gom - Keep adapter packages thin. Backend-specific filesystem or service behavior belongs in adapters; bundle, state, transform, and publish policy belongs outside adapters. - Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and implemented backend behavior unless the current task explicitly changes them. - Use `storage.DisplayPath`, `storage.StateFileName`, `storage.StatePath`, and `storage.ManagedBundleTargets` instead of duplicating those conventions. -- Use `bundle.ValidateManifest` for normalized source manifest semantics, including embedded source manifests in destination state. +- Use `pkg/bundle` for normalized source manifest semantics. Internal packages should reach those rules through `internal/bundle` when they also need storage-backed bundle discovery or validation. - Use `config.ValidatePublishTransformPolicy` for publish and transform policy combinations. - Do not import concrete transform implementations from `internal/publish`; app-level wiring owns transform registration. - Do not import `internal/testutil` from production code. diff --git a/internal/bundle/digest.go b/internal/bundle/digest.go index 65f3d0e..01f7ee5 100644 --- a/internal/bundle/digest.go +++ b/internal/bundle/digest.go @@ -1,12 +1,10 @@ package bundle import ( - "crypto/sha256" - "encoding/hex" "fmt" "regexp" - "strconv" - "strings" + + publicbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle" ) var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) @@ -19,31 +17,13 @@ func ValidateDigest(value string) error { } func FileDigest(data []byte) string { - sum := sha256.Sum256(data) - return "sha256:" + hex.EncodeToString(sum[:]) + return publicbundle.FileDigest(data) } func BundleDigest(files []ManifestFile) string { - canonical := CanonicalFilePayload(files) - sum := sha256.Sum256([]byte(canonical)) - return "sha256:" + hex.EncodeToString(sum[:]) + return publicbundle.BundleDigest(files) } func CanonicalFilePayload(files []ManifestFile) string { - var builder strings.Builder - builder.WriteByte('[') - for index, file := range files { - if index > 0 { - builder.WriteByte(',') - } - builder.WriteString(`{"path":`) - builder.WriteString(strconv.Quote(file.Path)) - builder.WriteString(`,"sha256":`) - builder.WriteString(strconv.Quote(file.SHA256)) - builder.WriteString(`,"size":`) - builder.WriteString(strconv.FormatInt(file.Size, 10)) - builder.WriteByte('}') - } - builder.WriteByte(']') - return builder.String() + return publicbundle.CanonicalFilePayload(files) } diff --git a/internal/bundle/manifest.go b/internal/bundle/manifest.go index b0a70df..ade0028 100644 --- a/internal/bundle/manifest.go +++ b/internal/bundle/manifest.go @@ -1,110 +1,32 @@ package bundle -import ( - "bytes" - "encoding/json" - "fmt" - "io" - "time" -) +import publicbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle" -const ManifestName = "manifest.json" +const ManifestName = publicbundle.ManifestName -type Manifest struct { - SchemaVersion int `json:"schema_version"` - ID string `json:"id"` - Digest string `json:"digest"` - Created time.Time `json:"created"` - Files []ManifestFile `json:"files"` -} +const SchemaVersion = publicbundle.SchemaVersion -type ManifestFile struct { - Path string `json:"path"` - SHA256 string `json:"sha256"` - Size int64 `json:"size"` -} +type Manifest = publicbundle.Manifest + +type ManifestFile = publicbundle.ManifestFile type Bundle struct { RootRelativePath string Manifest Manifest } -type rawManifest struct { - SchemaVersion *int `json:"schema_version"` - ID *string `json:"id"` - Digest *string `json:"digest"` - Created *string `json:"created"` - Files []rawManifestFile `json:"files"` -} - -type rawManifestFile struct { - Path *string `json:"path"` - SHA256 *string `json:"sha256"` - Size *int64 `json:"size"` -} - func ParseManifest(data []byte) (Manifest, error) { - decoder := json.NewDecoder(bytes.NewReader(data)) - var raw rawManifest - if err := decoder.Decode(&raw); err != nil { - return Manifest{}, fmt.Errorf("parse manifest: %w", err) - } - var extra any - if err := decoder.Decode(&extra); err != io.EOF { - return Manifest{}, fmt.Errorf("parse manifest: trailing data") - } - - var manifest Manifest - if raw.SchemaVersion == nil { - return Manifest{}, fmt.Errorf("manifest schema_version is required") - } - manifest.SchemaVersion = *raw.SchemaVersion - if raw.ID == nil || *raw.ID == "" { - return Manifest{}, fmt.Errorf("manifest id is required") - } - manifest.ID = *raw.ID - if raw.Digest == nil || *raw.Digest == "" { - return Manifest{}, fmt.Errorf("manifest digest is required") - } - manifest.Digest = *raw.Digest - if raw.Created == nil || *raw.Created == "" { - return Manifest{}, fmt.Errorf("manifest created is required") - } - created, err := time.Parse(time.RFC3339, *raw.Created) - if err != nil { - return Manifest{}, fmt.Errorf("manifest created must be RFC3339: %w", err) - } - manifest.Created = created - if len(raw.Files) == 0 { - return Manifest{}, fmt.Errorf("manifest files is required") - } - - for index, rawFile := range raw.Files { - file, err := parseManifestFile(index, rawFile) - if err != nil { - return Manifest{}, err - } - manifest.Files = append(manifest.Files, file) - } - if err := ValidateManifest(manifest); err != nil { - return Manifest{}, fmt.Errorf("manifest %w", err) - } - return manifest, nil + return publicbundle.ParseManifest(data) } -func parseManifestFile(index int, raw rawManifestFile) (ManifestFile, error) { - if raw.Path == nil || *raw.Path == "" { - return ManifestFile{}, fmt.Errorf("manifest files[%d].path is required", index) - } - if raw.SHA256 == nil || *raw.SHA256 == "" { - return ManifestFile{}, fmt.Errorf("manifest files[%d].sha256 is required", index) - } - if raw.Size == nil { - return ManifestFile{}, fmt.Errorf("manifest files[%d].size is required", index) - } - return ManifestFile{ - Path: *raw.Path, - SHA256: *raw.SHA256, - Size: *raw.Size, - }, nil +func MarshalManifest(manifest Manifest) ([]byte, error) { + return publicbundle.MarshalManifest(manifest) +} + +func ValidateManifest(manifest Manifest) error { + return publicbundle.ValidateManifest(manifest) +} + +func ValidateSourcePath(path string) error { + return publicbundle.ValidateSourcePath(path) } diff --git a/internal/bundle/validate.go b/internal/bundle/validate.go index 2e1bc16..aac3832 100644 --- a/internal/bundle/validate.go +++ b/internal/bundle/validate.go @@ -7,55 +7,6 @@ import ( "gitea.maximumdirect.net/eric/distributor/internal/storage" ) -func ValidateSourcePath(path string) error { - if err := storage.ValidatePath(path); err != nil { - return err - } - switch path { - case ManifestName, storage.StateFileName: - return fmt.Errorf("%q is reserved", path) - } - return nil -} - -func ValidateManifest(manifest Manifest) error { - if manifest.SchemaVersion != 1 { - return fmt.Errorf("schema_version must be 1") - } - if manifest.ID == "" { - return fmt.Errorf("id is required") - } - if err := ValidateDigest(manifest.Digest); err != nil { - return fmt.Errorf("digest: %w", err) - } - if manifest.Created.IsZero() { - return fmt.Errorf("created is required") - } - if len(manifest.Files) == 0 { - return fmt.Errorf("files is required") - } - seen := make(map[string]struct{}, len(manifest.Files)) - for index, file := range manifest.Files { - if err := ValidateSourcePath(file.Path); err != nil { - return fmt.Errorf("files[%d].path: %w", index, err) - } - if err := ValidateDigest(file.SHA256); err != nil { - return fmt.Errorf("files[%d].sha256: %w", index, err) - } - if file.Size < 0 { - return fmt.Errorf("files[%d].size must be non-negative", index) - } - if _, exists := seen[file.Path]; exists { - return fmt.Errorf("files[%d].path duplicates %q", index, file.Path) - } - seen[file.Path] = struct{}{} - } - if actual := BundleDigest(manifest.Files); actual != manifest.Digest { - return fmt.Errorf("digest mismatch: got %s want %s", actual, manifest.Digest) - } - return nil -} - func Validate(ctx context.Context, backend storage.Backend, bundleRoot string) (Bundle, error) { return validateAt(ctx, backend, bundleRoot, bundleRoot) } diff --git a/pkg/bundle/build.go b/pkg/bundle/build.go new file mode 100644 index 0000000..36c04f9 --- /dev/null +++ b/pkg/bundle/build.go @@ -0,0 +1,118 @@ +package bundle + +import ( + "fmt" + "os" + "path" + "path/filepath" + "sort" + "time" +) + +func BuildManifest(opts BuildOptions) (Manifest, error) { + if opts.Root == "" { + return Manifest{}, fmt.Errorf("root is required") + } + if opts.ID == "" { + return Manifest{}, fmt.Errorf("id is required") + } + explicit := len(opts.Files) > 0 + if explicit == opts.Scan { + return Manifest{}, fmt.Errorf("select exactly one file mode") + } + created := opts.Created + if created.IsZero() { + created = time.Now().UTC() + } + + paths := append([]string(nil), opts.Files...) + var err error + if opts.Scan { + paths, err = scanSourcePaths(opts.Root) + if err != nil { + return Manifest{}, err + } + } + files := make([]ManifestFile, 0, len(paths)) + for _, sourcePath := range paths { + file, err := buildManifestFile(opts.Root, sourcePath) + if err != nil { + return Manifest{}, err + } + files = append(files, file) + } + manifest := Manifest{ + SchemaVersion: SchemaVersion, + ID: opts.ID, + Created: created, + Files: files, + } + manifest.Digest = BundleDigest(manifest.Files) + if err := ValidateManifest(manifest); err != nil { + return Manifest{}, err + } + return manifest, nil +} + +func scanSourcePaths(root string) ([]string, error) { + var paths []string + err := filepath.WalkDir(root, func(filePath string, entry os.DirEntry, err error) error { + if err != nil { + return err + } + if filePath == root { + return nil + } + relative, err := filepath.Rel(root, filePath) + if err != nil { + return err + } + sourcePath := filepath.ToSlash(relative) + if entry.Type()&os.ModeSymlink != 0 { + return fmt.Errorf("source file %q must be a regular file", sourcePath) + } + if entry.IsDir() { + return nil + } + if path.Base(sourcePath) == ManifestName || path.Base(sourcePath) == distributorStateName { + return nil + } + info, err := entry.Info() + if err != nil { + return err + } + if !info.Mode().IsRegular() { + return fmt.Errorf("source file %q must be a regular file", sourcePath) + } + paths = append(paths, sourcePath) + return nil + }) + if err != nil { + return nil, err + } + sort.Strings(paths) + return paths, nil +} + +func buildManifestFile(root, sourcePath string) (ManifestFile, error) { + if err := ValidateSourcePath(sourcePath); err != nil { + return ManifestFile{}, fmt.Errorf("source file %q: %w", sourcePath, err) + } + fullPath := filepath.Join(root, filepath.FromSlash(sourcePath)) + info, err := os.Lstat(fullPath) + if err != nil { + return ManifestFile{}, fmt.Errorf("source file %q stat: %w", sourcePath, err) + } + if !info.Mode().IsRegular() { + return ManifestFile{}, fmt.Errorf("source file %q must be a regular file", sourcePath) + } + data, err := os.ReadFile(fullPath) + if err != nil { + return ManifestFile{}, fmt.Errorf("source file %q read: %w", sourcePath, err) + } + return ManifestFile{ + Path: sourcePath, + SHA256: FileDigest(data), + Size: int64(len(data)), + }, nil +} diff --git a/pkg/bundle/bundle_test.go b/pkg/bundle/bundle_test.go new file mode 100644 index 0000000..15b1628 --- /dev/null +++ b/pkg/bundle/bundle_test.go @@ -0,0 +1,363 @@ +package bundle + +import ( + "os" + "path/filepath" + "reflect" + "strings" + "testing" + "time" +) + +func TestBuildManifestExplicitFilesPreservesOrder(t *testing.T) { + root := t.TempDir() + writeFile(t, root, "b.txt", "bravo") + writeFile(t, root, "a.txt", "alpha") + before := time.Now().UTC() + + manifest, err := BuildManifest(BuildOptions{ + Root: root, + ID: "reports.example", + Files: []string{"b.txt", "a.txt"}, + }) + if err != nil { + t.Fatalf("BuildManifest() error = %v", err) + } + after := time.Now().UTC() + + paths := manifestPaths(manifest) + if want := []string{"b.txt", "a.txt"}; !reflect.DeepEqual(paths, want) { + t.Fatalf("paths = %v, want %v", paths, want) + } + if manifest.SchemaVersion != SchemaVersion { + t.Fatalf("schema version = %d, want %d", manifest.SchemaVersion, SchemaVersion) + } + if manifest.ID != "reports.example" { + t.Fatalf("id = %q", manifest.ID) + } + if manifest.Created.Before(before) || manifest.Created.After(after) { + t.Fatalf("created = %s, want between %s and %s", manifest.Created, before, after) + } + if err := ValidateManifest(manifest); err != nil { + t.Fatalf("ValidateManifest() error = %v", err) + } + if err := ValidateBundle(root, manifest); err != nil { + t.Fatalf("ValidateBundle() error = %v", err) + } +} + +func TestBuildManifestScanSortsAndFiltersMetadata(t *testing.T) { + root := t.TempDir() + writeFile(t, root, "z.txt", "zulu") + writeFile(t, root, "nested/.hidden", "hidden") + writeFile(t, root, ManifestName, "old manifest") + writeFile(t, root, distributorStateName, "state") + writeFile(t, root, "nested/manifest.json", "nested manifest") + writeFile(t, root, "nested/.distributor.json", "nested state") + created := time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC) + + manifest, err := BuildManifest(BuildOptions{ + Root: root, + ID: "reports.scan", + Created: created, + Scan: true, + }) + if err != nil { + t.Fatalf("BuildManifest() error = %v", err) + } + + if !manifest.Created.Equal(created) { + t.Fatalf("created = %s, want %s", manifest.Created, created) + } + paths := manifestPaths(manifest) + if want := []string{"nested/.hidden", "z.txt"}; !reflect.DeepEqual(paths, want) { + t.Fatalf("paths = %v, want %v", paths, want) + } +} + +func TestBuildManifestRequiresOneFileMode(t *testing.T) { + root := t.TempDir() + writeFile(t, root, "report.txt", "report") + + tests := []BuildOptions{ + {Root: root, ID: "reports.none"}, + {Root: root, ID: "reports.both", Files: []string{"report.txt"}, Scan: true}, + } + for _, opts := range tests { + if _, err := BuildManifest(opts); err == nil { + t.Fatalf("BuildManifest(%+v) error = nil, want error", opts) + } + } +} + +func TestBuildManifestRejectsUnsafePath(t *testing.T) { + root := t.TempDir() + writeFile(t, root, "report.txt", "report") + + _, err := BuildManifest(BuildOptions{ + Root: root, + ID: "reports.unsafe", + Files: []string{"../report.txt"}, + }) + if err == nil { + t.Fatal("BuildManifest() error = nil, want unsafe path error") + } +} + +func TestBuildManifestRejectsSymlink(t *testing.T) { + root := t.TempDir() + writeFile(t, root, "target.txt", "target") + if err := os.Symlink("target.txt", filepath.Join(root, "link.txt")); err != nil { + t.Skipf("symlink unavailable: %v", err) + } + + _, err := BuildManifest(BuildOptions{ + Root: root, + ID: "reports.symlink", + Files: []string{"link.txt"}, + }) + if err == nil || !strings.Contains(err.Error(), "regular file") { + t.Fatalf("BuildManifest() error = %v, want regular file error", err) + } +} + +func TestScanRejectsSymlink(t *testing.T) { + root := t.TempDir() + writeFile(t, root, "target.txt", "target") + if err := os.Symlink("target.txt", filepath.Join(root, "link.txt")); err != nil { + t.Skipf("symlink unavailable: %v", err) + } + + _, err := BuildManifest(BuildOptions{ + Root: root, + ID: "reports.scan.symlink", + Scan: true, + }) + if err == nil || !strings.Contains(err.Error(), "regular file") { + t.Fatalf("BuildManifest() error = %v, want regular file error", err) + } +} + +func TestDigestFunctionsUseCanonicalFilePayload(t *testing.T) { + files := []ManifestFile{ + {Path: "report.md", SHA256: FileDigest([]byte("report")), Size: 6}, + {Path: "summary.txt", SHA256: FileDigest([]byte("summary")), Size: 7}, + } + payload := CanonicalFilePayload(files) + if want := `[{"path":"report.md","sha256":"sha256:845e91831319e89c4d656bdb80c278ac09a7230d61e5dfd2e1b1fbb436ac8917","size":6},{"path":"summary.txt","sha256":"sha256:761b7ad8ad439b2855fcbb611331c646ef0870b0631247bba3f3025cb6df5a53","size":7}]`; payload != want { + t.Fatalf("payload = %q, want %q", payload, want) + } + if digest := BundleDigest(files); !strings.HasPrefix(digest, "sha256:") || len(digest) != len("sha256:")+64 { + t.Fatalf("BundleDigest() = %q, want sha256 digest", digest) + } +} + +func TestParseMarshalLoadAndWriteManifest(t *testing.T) { + root := t.TempDir() + writeFile(t, root, "report.txt", "report") + manifest, err := BuildManifest(BuildOptions{ + Root: root, + ID: "reports.json", + Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC), + Files: []string{"report.txt"}, + }) + if err != nil { + t.Fatalf("BuildManifest() error = %v", err) + } + data, err := MarshalManifest(manifest) + if err != nil { + t.Fatalf("MarshalManifest() error = %v", err) + } + if !strings.HasSuffix(string(data), "\n") { + t.Fatalf("manifest JSON = %q, want trailing newline", string(data)) + } + if !strings.Contains(string(data), `"schema_version": 1`) || !strings.Contains(string(data), `"sha256": "`) { + t.Fatalf("manifest JSON = %q, want fixed manifest fields", string(data)) + } + parsed, err := ParseManifest(data) + if err != nil { + t.Fatalf("ParseManifest() error = %v", err) + } + if !reflect.DeepEqual(parsed, manifest) { + t.Fatalf("parsed manifest = %#v, want %#v", parsed, manifest) + } + if err := WriteManifest(root, manifest, WriteManifestOptions{}); err != nil { + t.Fatalf("WriteManifest() error = %v", err) + } + loaded, err := LoadManifest(root) + if err != nil { + t.Fatalf("LoadManifest() error = %v", err) + } + if !reflect.DeepEqual(loaded, manifest) { + t.Fatalf("loaded manifest = %#v, want %#v", loaded, manifest) + } + if err := WriteManifest(root, manifest, WriteManifestOptions{}); err == nil { + t.Fatal("WriteManifest() error = nil, want exists error") + } + manifest.ID = "reports.updated" + manifest.Digest = BundleDigest(manifest.Files) + if err := WriteManifest(root, manifest, WriteManifestOptions{Overwrite: true}); err != nil { + t.Fatalf("WriteManifest(overwrite) error = %v", err) + } + loaded, err = LoadManifest(root) + if err != nil { + t.Fatalf("LoadManifest() after overwrite error = %v", err) + } + if loaded.ID != "reports.updated" { + t.Fatalf("loaded id = %q, want updated", loaded.ID) + } +} + +func TestValidateManifestRejectsInvalidManifest(t *testing.T) { + created := time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC) + fileDigest := FileDigest([]byte("report")) + valid := Manifest{ + SchemaVersion: SchemaVersion, + ID: "reports.valid", + Created: created, + Files: []ManifestFile{{Path: "report.txt", SHA256: fileDigest, Size: 6}}, + } + valid.Digest = BundleDigest(valid.Files) + + tests := map[string]func(Manifest) Manifest{ + "schema version": func(manifest Manifest) Manifest { + manifest.SchemaVersion = 2 + return manifest + }, + "id": func(manifest Manifest) Manifest { + manifest.ID = "" + return manifest + }, + "created": func(manifest Manifest) Manifest { + manifest.Created = time.Time{} + return manifest + }, + "files": func(manifest Manifest) Manifest { + manifest.Files = nil + manifest.Digest = BundleDigest(manifest.Files) + return manifest + }, + "path": func(manifest Manifest) Manifest { + manifest.Files[0].Path = "manifest.json" + manifest.Digest = BundleDigest(manifest.Files) + return manifest + }, + "duplicate": func(manifest Manifest) Manifest { + manifest.Files = append(manifest.Files, manifest.Files[0]) + manifest.Digest = BundleDigest(manifest.Files) + return manifest + }, + "size": func(manifest Manifest) Manifest { + manifest.Files[0].Size = -1 + manifest.Digest = BundleDigest(manifest.Files) + return manifest + }, + "digest": func(manifest Manifest) Manifest { + manifest.Digest = "sha256:0000000000000000000000000000000000000000000000000000000000000000" + return manifest + }, + } + for name, mutate := range tests { + t.Run(name, func(t *testing.T) { + if err := ValidateManifest(mutate(valid)); err == nil { + t.Fatal("ValidateManifest() error = nil, want error") + } + }) + } +} + +func TestValidateBundleRejectsLocalFileProblems(t *testing.T) { + root := t.TempDir() + writeFile(t, root, "report.txt", "report") + manifest, err := BuildManifest(BuildOptions{ + Root: root, + ID: "reports.bundle", + Files: []string{"report.txt"}, + }) + if err != nil { + t.Fatalf("BuildManifest() error = %v", err) + } + if err := ValidateBundle(root, manifest); err != nil { + t.Fatalf("ValidateBundle() error = %v", err) + } + + missing := cloneManifest(manifest) + missing.Files[0].Path = "missing.txt" + missing.Digest = BundleDigest(missing.Files) + if err := ValidateBundle(root, missing); err == nil { + t.Fatal("ValidateBundle() missing error = nil, want error") + } + + sizeMismatch := cloneManifest(manifest) + sizeMismatch.Files[0].Size++ + sizeMismatch.Digest = BundleDigest(sizeMismatch.Files) + if err := ValidateBundle(root, sizeMismatch); err == nil || !strings.Contains(err.Error(), "size mismatch") { + t.Fatalf("ValidateBundle() size error = %v, want size mismatch", err) + } + + digestMismatch := cloneManifest(manifest) + writeFile(t, root, "report.txt", "change") + if err := ValidateBundle(root, digestMismatch); err == nil || !strings.Contains(err.Error(), "sha256 mismatch") { + t.Fatalf("ValidateBundle() digest error = %v, want digest mismatch", err) + } +} + +func TestValidateBundleRejectsSymlink(t *testing.T) { + root := t.TempDir() + writeFile(t, root, "target.txt", "target") + if err := os.Symlink("target.txt", filepath.Join(root, "link.txt")); err != nil { + t.Skipf("symlink unavailable: %v", err) + } + file := ManifestFile{Path: "link.txt", SHA256: FileDigest([]byte("target")), Size: 6} + manifest := Manifest{ + SchemaVersion: SchemaVersion, + ID: "reports.link", + Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC), + Files: []ManifestFile{file}, + } + manifest.Digest = BundleDigest(manifest.Files) + + err := ValidateBundle(root, manifest) + if err == nil || !strings.Contains(err.Error(), "regular file") { + t.Fatalf("ValidateBundle() error = %v, want regular file error", err) + } +} + +func TestValidateSourcePath(t *testing.T) { + valid := []string{"report.md", "nested/report.md", ".well-known/report.txt"} + for _, path := range valid { + if err := ValidateSourcePath(path); err != nil { + t.Fatalf("ValidateSourcePath(%q) error = %v", path, err) + } + } + invalid := []string{"", "../report.md", "/report.md", "nested/../report.md", `nested\report.md`, ManifestName, distributorStateName} + for _, path := range invalid { + if err := ValidateSourcePath(path); err == nil { + t.Fatalf("ValidateSourcePath(%q) error = nil, want error", path) + } + } +} + +func manifestPaths(manifest Manifest) []string { + paths := make([]string, 0, len(manifest.Files)) + for _, file := range manifest.Files { + paths = append(paths, file.Path) + } + return paths +} + +func cloneManifest(manifest Manifest) Manifest { + manifest.Files = append([]ManifestFile(nil), manifest.Files...) + return manifest +} + +func writeFile(t *testing.T, root, relative, body string) { + t.Helper() + path := filepath.Join(root, filepath.FromSlash(relative)) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir %s: %v", filepath.Dir(path), err) + } + if err := os.WriteFile(path, []byte(body), 0o600); err != nil { + t.Fatalf("write %s: %v", path, err) + } +} diff --git a/pkg/bundle/digest.go b/pkg/bundle/digest.go new file mode 100644 index 0000000..a675ee6 --- /dev/null +++ b/pkg/bundle/digest.go @@ -0,0 +1,49 @@ +package bundle + +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "regexp" + "strconv" + "strings" +) + +var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) + +func validateDigest(value string) error { + if !digestPattern.MatchString(value) { + return fmt.Errorf("must be lowercase sha256:<64 hex>") + } + return nil +} + +func FileDigest(data []byte) string { + sum := sha256.Sum256(data) + return "sha256:" + hex.EncodeToString(sum[:]) +} + +func BundleDigest(files []ManifestFile) string { + canonical := CanonicalFilePayload(files) + sum := sha256.Sum256([]byte(canonical)) + return "sha256:" + hex.EncodeToString(sum[:]) +} + +func CanonicalFilePayload(files []ManifestFile) string { + var builder strings.Builder + builder.WriteByte('[') + for index, file := range files { + if index > 0 { + builder.WriteByte(',') + } + builder.WriteString(`{"path":`) + builder.WriteString(strconv.Quote(file.Path)) + builder.WriteString(`,"sha256":`) + builder.WriteString(strconv.Quote(file.SHA256)) + builder.WriteString(`,"size":`) + builder.WriteString(strconv.FormatInt(file.Size, 10)) + builder.WriteByte('}') + } + builder.WriteByte(']') + return builder.String() +} diff --git a/pkg/bundle/doc.go b/pkg/bundle/doc.go new file mode 100644 index 0000000..85a6854 --- /dev/null +++ b/pkg/bundle/doc.go @@ -0,0 +1,8 @@ +// Package bundle provides producer-facing helpers for distributor source +// bundle manifests. +// +// A source bundle is a local directory containing a manifest.json file and the +// files listed by that manifest. This package owns the public manifest model, +// digest calculation, path validation, manifest parsing, manifest building, and +// local bundle validation used by Go producer applications. +package bundle diff --git a/pkg/bundle/example_test.go b/pkg/bundle/example_test.go new file mode 100644 index 0000000..213c3e6 --- /dev/null +++ b/pkg/bundle/example_test.go @@ -0,0 +1,40 @@ +package bundle_test + +import ( + "fmt" + "os" + "path/filepath" + "time" + + "gitea.maximumdirect.net/eric/distributor/pkg/bundle" +) + +func ExampleBuildManifest() { + root, err := os.MkdirTemp("", "distributor-bundle-*") + if err != nil { + panic(err) + } + defer os.RemoveAll(root) + + if err := os.WriteFile(filepath.Join(root, "report.txt"), []byte("report\n"), 0o600); err != nil { + panic(err) + } + + manifest, err := bundle.BuildManifest(bundle.BuildOptions{ + Root: root, + ID: "reports.example", + Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC), + Files: []string{"report.txt"}, + }) + if err != nil { + panic(err) + } + + fmt.Println(manifest.ID) + fmt.Println(manifest.Files[0].Path) + fmt.Println(manifest.Files[0].Size) + // Output: + // reports.example + // report.txt + // 7 +} diff --git a/pkg/bundle/manifest.go b/pkg/bundle/manifest.go new file mode 100644 index 0000000..55883de --- /dev/null +++ b/pkg/bundle/manifest.go @@ -0,0 +1,160 @@ +package bundle + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "os" + "path/filepath" + "time" +) + +type rawManifest struct { + SchemaVersion *int `json:"schema_version"` + ID *string `json:"id"` + Digest *string `json:"digest"` + Created *string `json:"created"` + Files []rawManifestFile `json:"files"` +} + +type rawManifestFile struct { + Path *string `json:"path"` + SHA256 *string `json:"sha256"` + Size *int64 `json:"size"` +} + +func ParseManifest(data []byte) (Manifest, error) { + decoder := json.NewDecoder(bytes.NewReader(data)) + var raw rawManifest + if err := decoder.Decode(&raw); err != nil { + return Manifest{}, fmt.Errorf("parse manifest: %w", err) + } + var extra any + if err := decoder.Decode(&extra); err != io.EOF { + return Manifest{}, fmt.Errorf("parse manifest: trailing data") + } + + var manifest Manifest + if raw.SchemaVersion == nil { + return Manifest{}, fmt.Errorf("manifest schema_version is required") + } + manifest.SchemaVersion = *raw.SchemaVersion + if raw.ID == nil || *raw.ID == "" { + return Manifest{}, fmt.Errorf("manifest id is required") + } + manifest.ID = *raw.ID + if raw.Digest == nil || *raw.Digest == "" { + return Manifest{}, fmt.Errorf("manifest digest is required") + } + manifest.Digest = *raw.Digest + if raw.Created == nil || *raw.Created == "" { + return Manifest{}, fmt.Errorf("manifest created is required") + } + created, err := time.Parse(time.RFC3339, *raw.Created) + if err != nil { + return Manifest{}, fmt.Errorf("manifest created must be RFC3339: %w", err) + } + manifest.Created = created + if len(raw.Files) == 0 { + return Manifest{}, fmt.Errorf("manifest files is required") + } + + for index, rawFile := range raw.Files { + file, err := parseManifestFile(index, rawFile) + if err != nil { + return Manifest{}, err + } + manifest.Files = append(manifest.Files, file) + } + if err := ValidateManifest(manifest); err != nil { + return Manifest{}, fmt.Errorf("manifest %w", err) + } + return manifest, nil +} + +func parseManifestFile(index int, raw rawManifestFile) (ManifestFile, error) { + if raw.Path == nil || *raw.Path == "" { + return ManifestFile{}, fmt.Errorf("manifest files[%d].path is required", index) + } + if raw.SHA256 == nil || *raw.SHA256 == "" { + return ManifestFile{}, fmt.Errorf("manifest files[%d].sha256 is required", index) + } + if raw.Size == nil { + return ManifestFile{}, fmt.Errorf("manifest files[%d].size is required", index) + } + return ManifestFile{ + Path: *raw.Path, + SHA256: *raw.SHA256, + Size: *raw.Size, + }, nil +} + +func MarshalManifest(manifest Manifest) ([]byte, error) { + if err := ValidateManifest(manifest); err != nil { + return nil, err + } + data, err := json.MarshalIndent(manifest, "", " ") + if err != nil { + return nil, err + } + return append(data, '\n'), nil +} + +func LoadManifest(root string) (Manifest, error) { + data, err := os.ReadFile(filepath.Join(root, ManifestName)) + if err != nil { + return Manifest{}, fmt.Errorf("read manifest: %w", err) + } + return ParseManifest(data) +} + +func WriteManifest(root string, manifest Manifest, opts WriteManifestOptions) error { + data, err := MarshalManifest(manifest) + if err != nil { + return err + } + manifestPath := filepath.Join(root, ManifestName) + if !opts.Overwrite { + file, err := os.OpenFile(manifestPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o666) + if err != nil { + return fmt.Errorf("write manifest: %w", err) + } + if _, err := file.Write(data); err != nil { + _ = file.Close() + return fmt.Errorf("write manifest: %w", err) + } + if err := file.Close(); err != nil { + return fmt.Errorf("write manifest: %w", err) + } + return nil + } + + dir := root + if dir == "" { + dir = "." + } + tmp, err := os.CreateTemp(dir, ".manifest-*.tmp") + if err != nil { + return fmt.Errorf("write manifest temp: %w", err) + } + tmpPath := tmp.Name() + cleanup := true + defer func() { + if cleanup { + _ = os.Remove(tmpPath) + } + }() + if _, err := tmp.Write(data); err != nil { + _ = tmp.Close() + return fmt.Errorf("write manifest temp: %w", err) + } + if err := tmp.Close(); err != nil { + return fmt.Errorf("write manifest temp: %w", err) + } + if err := os.Rename(tmpPath, manifestPath); err != nil { + return fmt.Errorf("replace manifest: %w", err) + } + cleanup = false + return nil +} diff --git a/pkg/bundle/path.go b/pkg/bundle/path.go new file mode 100644 index 0000000..cabe829 --- /dev/null +++ b/pkg/bundle/path.go @@ -0,0 +1,31 @@ +package bundle + +import ( + "fmt" + "path" + "strings" +) + +const distributorStateName = ".distributor.json" + +func ValidateSourcePath(value string) error { + if value == "" { + return fmt.Errorf("source path is required") + } + if strings.Contains(value, "\\") || strings.HasPrefix(value, "/") { + return fmt.Errorf("source path %q must be a clean relative slash-separated path", value) + } + if path.Clean(value) != value { + return fmt.Errorf("source path %q must be a clean relative slash-separated path", value) + } + for _, segment := range strings.Split(value, "/") { + if segment == "" || segment == "." || segment == ".." { + return fmt.Errorf("source path %q must be a clean relative slash-separated path", value) + } + } + switch value { + case ManifestName, distributorStateName: + return fmt.Errorf("%q is reserved", value) + } + return nil +} diff --git a/pkg/bundle/types.go b/pkg/bundle/types.go new file mode 100644 index 0000000..faf2253 --- /dev/null +++ b/pkg/bundle/types.go @@ -0,0 +1,33 @@ +package bundle + +import "time" + +const ManifestName = "manifest.json" + +const SchemaVersion = 1 + +type Manifest struct { + SchemaVersion int `json:"schema_version"` + ID string `json:"id"` + Digest string `json:"digest"` + Created time.Time `json:"created"` + Files []ManifestFile `json:"files"` +} + +type ManifestFile struct { + Path string `json:"path"` + SHA256 string `json:"sha256"` + Size int64 `json:"size"` +} + +type BuildOptions struct { + Root string + ID string + Created time.Time + Files []string + Scan bool +} + +type WriteManifestOptions struct { + Overwrite bool +} diff --git a/pkg/bundle/validate.go b/pkg/bundle/validate.go new file mode 100644 index 0000000..2cdfb8c --- /dev/null +++ b/pkg/bundle/validate.go @@ -0,0 +1,79 @@ +package bundle + +import ( + "fmt" + "os" + "path/filepath" +) + +func ValidateManifest(manifest Manifest) error { + if manifest.SchemaVersion != SchemaVersion { + return fmt.Errorf("schema_version must be %d", SchemaVersion) + } + if manifest.ID == "" { + return fmt.Errorf("id is required") + } + if err := validateDigest(manifest.Digest); err != nil { + return fmt.Errorf("digest: %w", err) + } + if manifest.Created.IsZero() { + return fmt.Errorf("created is required") + } + if len(manifest.Files) == 0 { + return fmt.Errorf("files is required") + } + seen := make(map[string]struct{}, len(manifest.Files)) + for index, file := range manifest.Files { + if err := ValidateSourcePath(file.Path); err != nil { + return fmt.Errorf("files[%d].path: %w", index, err) + } + if err := validateDigest(file.SHA256); err != nil { + return fmt.Errorf("files[%d].sha256: %w", index, err) + } + if file.Size < 0 { + return fmt.Errorf("files[%d].size must be non-negative", index) + } + if _, exists := seen[file.Path]; exists { + return fmt.Errorf("files[%d].path duplicates %q", index, file.Path) + } + seen[file.Path] = struct{}{} + } + if actual := BundleDigest(manifest.Files); actual != manifest.Digest { + return fmt.Errorf("digest mismatch: got %s want %s", actual, manifest.Digest) + } + return nil +} + +func ValidateBundle(root string, manifest Manifest) error { + if err := ValidateManifest(manifest); err != nil { + return err + } + files := append([]ManifestFile(nil), manifest.Files...) + for index, manifestFile := range files { + fullPath := filepath.Join(root, filepath.FromSlash(manifestFile.Path)) + info, err := os.Lstat(fullPath) + if err != nil { + return fmt.Errorf("file %q stat: %w", manifestFile.Path, err) + } + if !info.Mode().IsRegular() { + return fmt.Errorf("file %q must be a regular file", manifestFile.Path) + } + if info.Size() != manifestFile.Size { + return fmt.Errorf("file %q size mismatch: got %d want %d", manifestFile.Path, info.Size(), manifestFile.Size) + } + data, err := os.ReadFile(fullPath) + if err != nil { + return fmt.Errorf("file %q read: %w", manifestFile.Path, err) + } + actualDigest := FileDigest(data) + if actualDigest != manifestFile.SHA256 { + return fmt.Errorf("file %q sha256 mismatch: got %s want %s", manifestFile.Path, actualDigest, manifestFile.SHA256) + } + files[index].SHA256 = actualDigest + files[index].Size = int64(len(data)) + } + if actualDigest := BundleDigest(files); actualDigest != manifest.Digest { + return fmt.Errorf("digest mismatch: got %s want %s", actualDigest, manifest.Digest) + } + return nil +}