Update documentation relating to the public packages and http_upload API

2026-06-07 13:33:27 -05:00
parent c12ec64066
commit 25fbfc4677
12 changed files with 615 additions and 29 deletions
--- a/docs/consumers/api.md
+++ b/docs/consumers/api.md
@@ -0,0 +1,127 @@
+# Upstream Producer Integration
+
+Audience: developers and LLM coding agents adding `distributor` support to an upstream Go producer application.
+
+This document is the copyable implementation guide for submitting producer outputs to a `distributor` pipeline whose source backend is `http_upload`.
+
+## Required Inputs
+
+The upstream application needs these values from deployment or operator configuration:
+
+- distributor endpoint: the HTTP server base URL, such as `https://distributor.example.com`;
+- upload token: bearer token for exactly one configured `http_upload` pipeline;
+- generated files: regular local files to include in the source bundle;
+- bundle id: stable identifier for this producer output;
+- idempotency key: stable key for retrying the same producer operation.
+
+Do not put destination routing, public URLs, transform settings, or credentials in the source manifest. Those belong in the `distributor` pipeline configuration.
+
+## Recommended Workflow
+
+Use `gitea.maximumdirect.net/eric/distributor/pkg/upload`.
+
+For most producers, use `UploadFiles`. It accepts producer-generated files, builds a temporary valid source bundle with `pkg/bundle`, uploads a gzip-compressed tar archive, and removes temporary files when the call returns.
+
+Use `UploadBundle` only when the producer already assembled a complete bundle directory containing `manifest.json`.
+
+Add the dependency from the upstream application:
+
+```sh
+go get gitea.maximumdirect.net/eric/distributor
+```
+
+## Minimal Go Example
+
+```go
+package reports
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"time"
+
+	"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+	"gitea.maximumdirect.net/eric/distributor/pkg/upload"
+)
+
+func SubmitReport(reportPath, summaryPath string) error {
+	endpoint := os.Getenv("DISTRIBUTOR_UPLOAD_ENDPOINT")
+	token := os.Getenv("DISTRIBUTOR_UPLOAD_TOKEN")
+	if endpoint == "" || token == "" {
+		return fmt.Errorf("distributor endpoint and token are required")
+	}
+
+	reportID := "weather.hourly.brentwood.2026-06-07T15"
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+
+	client, err := upload.NewClient(upload.ClientOptions{
+		Endpoint: endpoint,
+		Token:    token,
+	})
+	if err != nil {
+		return err
+	}
+
+	result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
+		ID:             reportID,
+		IdempotencyKey: reportID,
+		Files: []bundle.BundleFile{
+			{SourcePath: reportPath, Path: "report.md"},
+			{SourcePath: summaryPath, Path: "summary.txt"},
+		},
+	})
+	if err != nil {
+		var conflict *upload.IdempotencyConflictError
+		if errors.As(err, &conflict) {
+			return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
+		}
+		return err
+	}
+
+	fmt.Printf("distributor accepted run %s\n", result.RunID)
+	return nil
+}
+```
+
+## Producer Responsibilities
+
+- Use a stable bundle id for the producer output, such as a report type plus logical timestamp.
+- Use a stable idempotency key for cross-process retries of the same producer operation.
+- Map each generated file to a clean slash-separated bundle path, such as `report.md` or `assets/chart.png`.
+- Include only regular files. Symlinks, directories as files, devices, FIFOs, and sockets are rejected.
+- Keep file contents stable after upload inputs are selected. Bundle digests are calculated from file bytes.
+- Treat upload success as admission only. `UploadFiles` and `UploadBundle` return after the server accepts and validates the upload, not after all destinations publish.
+
+Valid bundle paths are relative slash paths. They must not be empty, absolute, contain backslashes, contain `.` or `..` path segments, contain empty path segments, or use reserved basenames `manifest.json` or `.distributor.json`.
+
+## Idempotency And Status
+
+`pkg/upload` sends `Idempotency-Key` on every upload. If the caller omits one, the package generates a random key for that call and reuses it for in-process retries. That is enough for transient network retry within one process.
+
+For producer jobs that may retry after process restart, supply a stable key derived from the producer operation, such as the report id or job id. Reusing the same key with the same normalized source manifest returns the original accepted run. Reusing the same key with different source content returns a conflict.
+
+`Status` polls `/runs/<run-id>` while the distributor server retains the in-memory status record. Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to the server's `server.http.retention` setting, and server restart clears status and idempotency records.
+
+Optional status check:
+
+```go
+status, err := client.Status(ctx, result.RunID)
+if err != nil {
+	return err
+}
+if status.Status == "failed" {
+	return fmt.Errorf("distributor run failed: %s", status.Error)
+}
+```
+
+## References
+
+In the `distributor` source tree:
+
+- `docs/consumers/pkg-upload.md`: Go upload package workflow.
+- `docs/consumers/pkg-bundle.md`: Go bundle package workflow.
+- `docs/integrations/http-upload.md`: canonical HTTP upload wire contract.
+- `docs/integrations/source-bundle.md`: canonical source bundle file-format contract.
--- a/docs/consumers/pkg-bundle.md
+++ b/docs/consumers/pkg-bundle.md
@@ -0,0 +1,88 @@
+# `pkg/bundle`
+
+Audience: upstream Go producer developers and LLM coding agents using `distributor` source bundle helpers.
+
+Import path:
+
+```go
+import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+```
+
+`pkg/bundle` builds, writes, parses, and validates local source bundles. Use it directly when a producer writes bundles for `distributor` to discover, or when a producer wants to assemble and validate a bundle before using another transport.
+
+The canonical source bundle file-format contract is [Source Bundle Contract](../integrations/source-bundle.md).
+
+## Preferred Complete-Bundle Workflow
+
+Use `WriteBundle` when producer-generated files live outside the final bundle root.
+
+```go
+manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{
+	Root: "/var/spool/distributor/weather/hourly-2026-06-07T15",
+	ID:   "weather.hourly.brentwood.2026-06-07T15",
+	Files: []bundle.BundleFile{
+		{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
+		{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
+	},
+})
+if err != nil {
+	return err
+}
+_ = manifest
+```
+
+`WriteBundle` copies each source file into a staged bundle root, writes `manifest.json`, validates the staged bundle, and promotes it into place. Set `Overwrite: true` only when the producer intentionally replaces an existing bundle root.
+
+## Existing Bundle Root Workflow
+
+Use `BuildManifest` and `WriteManifest` when files are already staged under the final bundle root.
+
+```go
+root := "/var/spool/distributor/weather/hourly-2026-06-07T15"
+manifest, err := bundle.BuildManifest(bundle.BuildOptions{
+	Root:  root,
+	ID:    "weather.hourly.brentwood.2026-06-07T15",
+	Files: []string{"report.md", "summary.txt"},
+})
+if err != nil {
+	return err
+}
+if err := bundle.WriteManifest(root, manifest, bundle.WriteManifestOptions{}); err != nil {
+	return err
+}
+if err := bundle.ValidateBundle(root, manifest); err != nil {
+	return err
+}
+```
+
+Use `Scan: true` instead of `Files` only when every valid regular file under the root should be included. Scan mode includes dotfiles, skips reserved metadata files, rejects symlinks, and sorts paths lexically.
+
+## Paths And Ordering
+
+Bundle paths are slash-separated paths relative to the bundle root.
+
+Invalid paths include:
+
+- empty paths;
+- absolute paths;
+- paths containing backslashes;
+- `.` or `..` path segments;
+- empty path segments;
+- any basename of `manifest.json` or `.distributor.json`.
+
+Explicit file lists preserve caller order. File order is part of the bundle digest, so producers should choose it deliberately and keep it stable.
+
+## Validation And Digest Helpers
+
+Use `ValidateBundle` before handing an existing local bundle to another process. It verifies manifest semantics, file existence, regular-file type, file size, per-file SHA-256 digests, and bundle digest.
+
+Useful helpers:
+
+- `LoadManifest`: read `manifest.json` from a bundle root.
+- `ParseManifest` and `MarshalManifest`: parse or write manifest bytes.
+- `ValidateManifest`: validate manifest-only semantics.
+- `FileDigest`, `BundleDigest`, and `ValidateDigest`: digest helpers for diagnostics and tests.
+
+## Boundaries
+
+`pkg/bundle` does not upload bundles, publish destinations, transform Markdown, select pipelines, configure credentials, or write destination state. Those concerns belong to `pkg/upload` or the `distributor` application.
--- a/docs/consumers/pkg-upload.md
+++ b/docs/consumers/pkg-upload.md
@@ -0,0 +1,118 @@
+# `pkg/upload`
+
+Audience: upstream Go producer developers and LLM coding agents submitting bundles to `distributor serve`.
+
+Import path:
+
+```go
+import "gitea.maximumdirect.net/eric/distributor/pkg/upload"
+```
+
+`pkg/upload` is the producer-facing HTTP upload client. It builds on `pkg/bundle`, packages valid source bundles as gzip-compressed tar archives, sends bearer authentication, includes idempotency keys, and exposes a status polling helper.
+
+`UploadFiles` examples also use:
+
+```go
+import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+```
+
+The canonical HTTP wire contract is [HTTP Upload API Contract](../integrations/http-upload.md).
+
+## Client Construction
+
+```go
+client, err := upload.NewClient(upload.ClientOptions{
+	Endpoint: "https://distributor.example.com",
+	Token:    token,
+})
+if err != nil {
+	return err
+}
+```
+
+`Endpoint` is the distributor server base URL. The client derives `/upload` and `/runs/<run-id>`. `Token` is required and is sent as `Authorization: Bearer <token>`. Token values are redacted from client errors.
+
+`HTTPClient` and `Retry` are optional. Defaults use a 30 second HTTP timeout and safe retry settings.
+
+## Upload Producer Files
+
+Use `UploadFiles` when the producer has generated output files but has not assembled a bundle directory.
+
+```go
+result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
+	ID:             "weather.hourly.brentwood.2026-06-07T15",
+	IdempotencyKey: "weather.hourly.brentwood.2026-06-07T15",
+	Files: []bundle.BundleFile{
+		{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
+		{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
+	},
+})
+if err != nil {
+	return err
+}
+_ = result.RunID
+```
+
+`UploadFiles` creates a temporary bundle, writes and validates a manifest, uploads the archive, and removes temporary files when the call returns. It does not write into producer source directories.
+
+## Upload An Existing Bundle
+
+Use `UploadBundle` when the producer already has a complete local bundle root containing `manifest.json`.
+
+```go
+result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
+	Root:           "/var/spool/weather/hourly-2026-06-07T15",
+	IdempotencyKey: "weather.hourly.brentwood.2026-06-07T15",
+})
+if err != nil {
+	return err
+}
+_ = result.RunID
+```
+
+`UploadBundle` validates the local bundle by default and uploads only `manifest.json` plus manifest-listed files. Unlisted files are not uploaded.
+
+## Result And Status
+
+Upload success means the server returned `202 Accepted` after staging and validating the upload. It does not mean all configured destinations have published.
+
+Poll status while the server retains the in-memory run record:
+
+```go
+status, err := client.Status(ctx, result.RunID)
+if err != nil {
+	return err
+}
+if status.Status == "failed" {
+	return fmt.Errorf("distributor run failed: %s", status.Error)
+}
+```
+
+Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to `server.http.retention`; server restart clears run status and idempotency records.
+
+## Idempotency And Retry
+
+Every upload request includes `Idempotency-Key`.
+
+If `IdempotencyKey` is omitted, the client generates a random 128-bit lowercase hexadecimal key for that upload operation and reuses it for retries within the same call. For cross-process retry safety, producers should pass a stable key derived from the producer job or report id.
+
+The client retries only safe cases:
+
+- `503 Service Unavailable`;
+- temporary network errors;
+- ambiguous mid-upload failures.
+
+It does not retry after `202 Accepted` and does not retry `400`, `401`, `409`, `413`, or `415`.
+
+Detect conflicting key reuse with `errors.As`:
+
+```go
+var conflict *upload.IdempotencyConflictError
+if errors.As(err, &conflict) {
+	return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
+}
+```
+
+## Boundaries
+
+`pkg/upload` does not configure server pipelines, choose destinations, wait for publication completion automatically, persist client queues, provide durable idempotency across server restarts, or expose destination state. It submits complete source bundles to the configured HTTP upload API.