Add an implementation plan to support distributor uploads for produced artifacts

2026-06-07 18:04:40 -05:00
parent 63749a9572
commit 9677835d84
5 changed files with 966 additions and 0 deletions
--- a/docs/integrations/distributor/api.md
+++ b/docs/integrations/distributor/api.md
@@ -0,0 +1,127 @@
+# Upstream Producer Integration
+
+Audience: developers and LLM coding agents adding `distributor` support to an upstream Go producer application.
+
+This document is the copyable implementation guide for submitting producer outputs to a `distributor` pipeline whose source backend is `http_upload`.
+
+## Required Inputs
+
+The upstream application needs these values from deployment or operator configuration:
+
+- distributor endpoint: the HTTP server base URL, such as `https://distributor.example.com`;
+- upload token: bearer token for exactly one configured `http_upload` pipeline;
+- generated files: regular local files to include in the source bundle;
+- bundle id: stable identifier for this producer output;
+- idempotency key: stable key for retrying the same producer operation.
+
+Do not put destination routing, public URLs, transform settings, or credentials in the source manifest. Those belong in the `distributor` pipeline configuration.
+
+## Recommended Workflow
+
+Use `gitea.maximumdirect.net/eric/distributor/pkg/upload`.
+
+For most producers, use `UploadFiles`. It accepts producer-generated files, builds a temporary valid source bundle with `pkg/bundle`, uploads a gzip-compressed tar archive, and removes temporary files when the call returns.
+
+Use `UploadBundle` only when the producer already assembled a complete bundle directory containing `manifest.json`.
+
+Add the dependency from the upstream application:
+
+```sh
+go get gitea.maximumdirect.net/eric/distributor
+```
+
+## Minimal Go Example
+
+```go
+package reports
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"time"
+
+	"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+	"gitea.maximumdirect.net/eric/distributor/pkg/upload"
+)
+
+func SubmitReport(reportPath, summaryPath string) error {
+	endpoint := os.Getenv("DISTRIBUTOR_UPLOAD_ENDPOINT")
+	token := os.Getenv("DISTRIBUTOR_UPLOAD_TOKEN")
+	if endpoint == "" || token == "" {
+		return fmt.Errorf("distributor endpoint and token are required")
+	}
+
+	reportID := "weather.hourly.brentwood.2026-06-07T15"
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+
+	client, err := upload.NewClient(upload.ClientOptions{
+		Endpoint: endpoint,
+		Token:    token,
+	})
+	if err != nil {
+		return err
+	}
+
+	result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
+		ID:             reportID,
+		IdempotencyKey: reportID,
+		Files: []bundle.BundleFile{
+			{SourcePath: reportPath, Path: "report.md"},
+			{SourcePath: summaryPath, Path: "summary.txt"},
+		},
+	})
+	if err != nil {
+		var conflict *upload.IdempotencyConflictError
+		if errors.As(err, &conflict) {
+			return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
+		}
+		return err
+	}
+
+	fmt.Printf("distributor accepted run %s\n", result.RunID)
+	return nil
+}
+```
+
+## Producer Responsibilities
+
+- Use a stable bundle id for the producer output, such as a report type plus logical timestamp.
+- Use a stable idempotency key for cross-process retries of the same producer operation.
+- Map each generated file to a clean slash-separated bundle path, such as `report.md` or `assets/chart.png`.
+- Include only regular files. Symlinks, directories as files, devices, FIFOs, and sockets are rejected.
+- Keep file contents stable after upload inputs are selected. Bundle digests are calculated from file bytes.
+- Treat upload success as admission only. `UploadFiles` and `UploadBundle` return after the server accepts and validates the upload, not after all destinations publish.
+
+Valid bundle paths are relative slash paths. They must not be empty, absolute, contain backslashes, contain `.` or `..` path segments, contain empty path segments, or use reserved basenames `manifest.json` or `.distributor.json`.
+
+## Idempotency And Status
+
+`pkg/upload` sends `Idempotency-Key` on every upload. If the caller omits one, the package generates a random key for that call and reuses it for in-process retries. That is enough for transient network retry within one process.
+
+For producer jobs that may retry after process restart, supply a stable key derived from the producer operation, such as the report id or job id. Reusing the same key with the same normalized source manifest returns the original accepted run. Reusing the same key with different source content returns a conflict.
+
+`Status` polls `/runs/<run-id>` while the distributor server retains the in-memory status record. Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to the server's `server.http.retention` setting, and server restart clears status and idempotency records.
+
+Optional status check:
+
+```go
+status, err := client.Status(ctx, result.RunID)
+if err != nil {
+	return err
+}
+if status.Status == "failed" {
+	return fmt.Errorf("distributor run failed: %s", status.Error)
+}
+```
+
+## References
+
+In the `distributor` source tree:
+
+- `docs/consumers/pkg-upload.md`: Go upload package workflow.
+- `docs/consumers/pkg-bundle.md`: Go bundle package workflow.
+- `docs/integrations/http-upload.md`: canonical HTTP upload wire contract.
+- `docs/integrations/source-bundle.md`: canonical source bundle file-format contract.
--- a/docs/integrations/distributor/pkg-bundle.md
+++ b/docs/integrations/distributor/pkg-bundle.md
@@ -0,0 +1,88 @@
+# `pkg/bundle`
+
+Audience: upstream Go producer developers and LLM coding agents using `distributor` source bundle helpers.
+
+Import path:
+
+```go
+import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+```
+
+`pkg/bundle` builds, writes, parses, and validates local source bundles. Use it directly when a producer writes bundles for `distributor` to discover, or when a producer wants to assemble and validate a bundle before using another transport.
+
+The canonical source bundle file-format contract is [Source Bundle Contract](../integrations/source-bundle.md).
+
+## Preferred Complete-Bundle Workflow
+
+Use `WriteBundle` when producer-generated files live outside the final bundle root.
+
+```go
+manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{
+	Root: "/var/spool/distributor/weather/hourly-2026-06-07T15",
+	ID:   "weather.hourly.brentwood.2026-06-07T15",
+	Files: []bundle.BundleFile{
+		{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
+		{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
+	},
+})
+if err != nil {
+	return err
+}
+_ = manifest
+```
+
+`WriteBundle` copies each source file into a staged bundle root, writes `manifest.json`, validates the staged bundle, and promotes it into place. Set `Overwrite: true` only when the producer intentionally replaces an existing bundle root.
+
+## Existing Bundle Root Workflow
+
+Use `BuildManifest` and `WriteManifest` when files are already staged under the final bundle root.
+
+```go
+root := "/var/spool/distributor/weather/hourly-2026-06-07T15"
+manifest, err := bundle.BuildManifest(bundle.BuildOptions{
+	Root:  root,
+	ID:    "weather.hourly.brentwood.2026-06-07T15",
+	Files: []string{"report.md", "summary.txt"},
+})
+if err != nil {
+	return err
+}
+if err := bundle.WriteManifest(root, manifest, bundle.WriteManifestOptions{}); err != nil {
+	return err
+}
+if err := bundle.ValidateBundle(root, manifest); err != nil {
+	return err
+}
+```
+
+Use `Scan: true` instead of `Files` only when every valid regular file under the root should be included. Scan mode includes dotfiles, skips reserved metadata files, rejects symlinks, and sorts paths lexically.
+
+## Paths And Ordering
+
+Bundle paths are slash-separated paths relative to the bundle root.
+
+Invalid paths include:
+
+- empty paths;
+- absolute paths;
+- paths containing backslashes;
+- `.` or `..` path segments;
+- empty path segments;
+- any basename of `manifest.json` or `.distributor.json`.
+
+Explicit file lists preserve caller order. File order is part of the bundle digest, so producers should choose it deliberately and keep it stable.
+
+## Validation And Digest Helpers
+
+Use `ValidateBundle` before handing an existing local bundle to another process. It verifies manifest semantics, file existence, regular-file type, file size, per-file SHA-256 digests, and bundle digest.
+
+Useful helpers:
+
+- `LoadManifest`: read `manifest.json` from a bundle root.
+- `ParseManifest` and `MarshalManifest`: parse or write manifest bytes.
+- `ValidateManifest`: validate manifest-only semantics.
+- `FileDigest`, `BundleDigest`, and `ValidateDigest`: digest helpers for diagnostics and tests.
+
+## Boundaries
+
+`pkg/bundle` does not upload bundles, publish destinations, transform Markdown, select pipelines, configure credentials, or write destination state. Those concerns belong to `pkg/upload` or the `distributor` application.
--- a/docs/integrations/distributor/pkg-upload.md
+++ b/docs/integrations/distributor/pkg-upload.md
@@ -0,0 +1,118 @@
+# `pkg/upload`
+
+Audience: upstream Go producer developers and LLM coding agents submitting bundles to `distributor serve`.
+
+Import path:
+
+```go
+import "gitea.maximumdirect.net/eric/distributor/pkg/upload"
+```
+
+`pkg/upload` is the producer-facing HTTP upload client. It builds on `pkg/bundle`, packages valid source bundles as gzip-compressed tar archives, sends bearer authentication, includes idempotency keys, and exposes a status polling helper.
+
+`UploadFiles` examples also use:
+
+```go
+import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
+```
+
+The canonical HTTP wire contract is [HTTP Upload API Contract](../integrations/http-upload.md).
+
+## Client Construction
+
+```go
+client, err := upload.NewClient(upload.ClientOptions{
+	Endpoint: "https://distributor.example.com",
+	Token:    token,
+})
+if err != nil {
+	return err
+}
+```
+
+`Endpoint` is the distributor server base URL. The client derives `/upload` and `/runs/<run-id>`. `Token` is required and is sent as `Authorization: Bearer <token>`. Token values are redacted from client errors.
+
+`HTTPClient` and `Retry` are optional. Defaults use a 30 second HTTP timeout and safe retry settings.
+
+## Upload Producer Files
+
+Use `UploadFiles` when the producer has generated output files but has not assembled a bundle directory.
+
+```go
+result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
+	ID:             "weather.hourly.brentwood.2026-06-07T15",
+	IdempotencyKey: "weather.hourly.brentwood.2026-06-07T15",
+	Files: []bundle.BundleFile{
+		{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
+		{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
+	},
+})
+if err != nil {
+	return err
+}
+_ = result.RunID
+```
+
+`UploadFiles` creates a temporary bundle, writes and validates a manifest, uploads the archive, and removes temporary files when the call returns. It does not write into producer source directories.
+
+## Upload An Existing Bundle
+
+Use `UploadBundle` when the producer already has a complete local bundle root containing `manifest.json`.
+
+```go
+result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
+	Root:           "/var/spool/weather/hourly-2026-06-07T15",
+	IdempotencyKey: "weather.hourly.brentwood.2026-06-07T15",
+})
+if err != nil {
+	return err
+}
+_ = result.RunID
+```
+
+`UploadBundle` validates the local bundle by default and uploads only `manifest.json` plus manifest-listed files. Unlisted files are not uploaded.
+
+## Result And Status
+
+Upload success means the server returned `202 Accepted` after staging and validating the upload. It does not mean all configured destinations have published.
+
+Poll status while the server retains the in-memory run record:
+
+```go
+status, err := client.Status(ctx, result.RunID)
+if err != nil {
+	return err
+}
+if status.Status == "failed" {
+	return fmt.Errorf("distributor run failed: %s", status.Error)
+}
+```
+
+Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to `server.http.retention`; server restart clears run status and idempotency records.
+
+## Idempotency And Retry
+
+Every upload request includes `Idempotency-Key`.
+
+If `IdempotencyKey` is omitted, the client generates a random 128-bit lowercase hexadecimal key for that upload operation and reuses it for retries within the same call. For cross-process retry safety, producers should pass a stable key derived from the producer job or report id.
+
+The client retries only safe cases:
+
+- `503 Service Unavailable`;
+- temporary network errors;
+- ambiguous mid-upload failures.
+
+It does not retry after `202 Accepted` and does not retry `400`, `401`, `409`, `413`, or `415`.
+
+Detect conflicting key reuse with `errors.As`:
+
+```go
+var conflict *upload.IdempotencyConflictError
+if errors.As(err, &conflict) {
+	return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
+}
+```
+
+## Boundaries
+
+`pkg/upload` does not configure server pipelines, choose destinations, wait for publication completion automatically, persist client queues, provide durable idempotency across server restarts, or expose destination state. It submits complete source bundles to the configured HTTP upload API.
--- a/docs/roadmap/distributor.md
+++ b/docs/roadmap/distributor.md
@@ -0,0 +1,336 @@
+# Distributor Integration Roadmap
+
+This roadmap describes planned work for adding `distributor` notification
+support to `weatherreporter`. The feature is not implemented yet, so this plan
+lives under `docs/roadmap/`; non-roadmap documentation should not describe
+`notify.distributor` as available until implementation is complete.
+
+## Purpose
+
+`distributor` collects outputs from producer applications and publishes them
+through configured downstream pipelines. For `weatherreporter`, the planned
+integration should upload generated Markdown weather reports after successful
+report generation, without requiring `distributor` to scan or understand the
+managed `workspace/` layout.
+
+## Current Repository Facts
+
+- `weatherreporter` does not currently implement a notify hook.
+- Generated Markdown reports are written to the managed report path returned as
+  `ReportResult.ReportPath`.
+- Optional `--out` and `--out-dir` copies are user-requested extra copies and
+  are not canonical integration inputs.
+- The managed workspace also contains briefing snapshots, metadata, prompt data
+  packages, and preflight artifacts; these should not be treated as a
+  distributor source tree.
+- The provided distributor docs recommend `pkg/upload.UploadFiles` for
+  producers that already have generated files, so the initial integration can
+  upload selected report files directly.
+
+## Decisions Locked
+
+- Add generic configurable secrets-directory support before the distributor
+  adapter work. This is project infrastructure, not a distributor-only token
+  helper.
+- When a secrets directory is configured, each regular file directly under that
+  directory maps filename to environment variable name and file contents to
+  environment variable value.
+- Secret-file values overwrite pre-existing environment variables.
+- Secrets loading must never log, print, persist, or include secret values in
+  errors, metadata, batch output, examples, or docs.
+- Add a narrow `internal/adapters/distributor` adapter for the external
+  integration.
+- Keep `gitea.maximumdirect.net/eric/distributor/pkg/upload` and
+  `gitea.maximumdirect.net/eric/distributor/pkg/bundle` types inside that
+  adapter.
+- Upload one distributor bundle per successfully generated report.
+- Include only the generated Markdown report in the initial bundle.
+- Use the report definition's batch output filename as the default
+  bundle-relative Markdown path, such as `daily.md`, `tomorrow.md`,
+  `three-day.md`, `weekend.md`, or `storm.md`.
+- Derive the default bundle ID and idempotency key from producer name, location
+  ID, report ID, and RunID.
+- Treat distributor upload failure as a report failure when notification is
+  enabled; single-report and batch commands should exit nonzero for enabled
+  upload failures.
+- Keep destination routing, Markdown-to-HTML transformation, public URLs, and
+  nginx directory layout in distributor configuration, not in
+  `weatherreporter`.
+
+## Planned Configuration
+
+Add future generic secrets configuration:
+
+```yaml
+secrets:
+  directory: ""
+```
+
+Add future distributor configuration under `notify.distributor`:
+
+```yaml
+notify:
+  distributor:
+    enabled: false
+    endpoint: https://distributor.example.com
+    token_env: DISTRIBUTOR_UPLOAD_TOKEN
+    timeout: 30s
+    failure_policy: error
+    bundle_id_template: "weatherreporter.{location_id}.{report_id}.{run_id}"
+    idempotency_key_template: "{bundle_id}"
+    report_path_template: "{batch_output_name}"
+```
+
+Planned behavior:
+
+- `secrets.directory` defaults to an empty string, which disables secrets
+  loading.
+- When `secrets.directory` is configured, load only regular files directly
+  under that directory.
+- Each secret filename must be a valid environment variable name matching
+  `[A-Za-z_][A-Za-z0-9_]*`.
+- Reject missing configured directory, invalid filenames, subdirectories,
+  symlinks, unreadable files, and empty filenames.
+- Strip one trailing `\n` or `\r\n` from secret file contents for compatibility
+  with common mounted-secret formats. Preserve all other bytes.
+- `enabled` defaults to `false`.
+- `endpoint` is required only when distributor notification is enabled.
+- `token_env` is required only when enabled. Raw bearer tokens should not be
+  stored in config files. The token value is read from normal environment state
+  after secrets-directory loading has been applied.
+- `timeout` controls the upload call timeout and should default to `30s`.
+- `failure_policy` should initially support `error`. Reserve `warn` as a
+  future option only if it is implemented and tested.
+- Template variables should be explicit and validated:
+  `location_id`, `report_id`, `run_id`, `artifact_group`, and
+  `batch_output_name`.
+- `idempotency_key_template` may reference `{bundle_id}` after the bundle ID is
+  rendered.
+- `report_path_template` should be user-visible from the first implementation
+  because bundle-relative paths can affect downstream distributor routing.
+- Rendered template values must produce valid distributor bundle paths: relative
+  slash-separated paths with no empty path, absolute path, backslash, `.`, `..`,
+  empty segment, `manifest.json`, or `.distributor.json`.
+
+## Recommended Implementation Stages
+
+### Stage 1: Secrets Directory Support
+
+Goal: add generic file-backed environment secret loading before distributor
+notification depends on environment tokens.
+
+Implementation guidance:
+
+- Add a generic `secrets.directory` config field under `internal/config`.
+- Default the directory to an empty string so secrets loading is disabled unless
+  explicitly configured.
+- Load secrets after config file parsing and before validation that needs
+  environment-backed values.
+- Map each regular file directly under the configured directory to an
+  environment variable: filename becomes the variable name and contents become
+  the value.
+- Use secret-file values to overwrite pre-existing environment variables.
+- Strip one trailing `\n` or `\r\n`; preserve all other bytes.
+- Reject missing directories, invalid environment-variable filenames,
+  subdirectories, symlinks, unreadable files, and empty filenames.
+- Keep secret values out of errors, logs, CLI output, metadata, and examples.
+
+Acceptance criteria:
+
+- Empty `secrets.directory` performs no environment changes.
+- Configured secrets directory values are available through normal environment
+  lookup after config loading.
+- Secret files overwrite existing environment variables with the same name.
+- Invalid directory entries fail with path/name context but without secret
+  values.
+- Existing config loading behavior remains unchanged when secrets are disabled.
+
+Suggested tests:
+
+- Config defaults leave secrets loading disabled.
+- A configured missing secrets directory fails.
+- Regular files set environment variables and overwrite existing values.
+- Invalid filenames, subdirectories, symlinks, and unreadable files fail.
+- One trailing newline or CRLF is stripped; other content is preserved.
+- No secret values appear in returned errors.
+
+### Stage 2: Config And Validation
+
+Goal: add disabled-by-default distributor notification configuration.
+
+Implementation guidance:
+
+- Add notify/distributor config structs under `internal/config`.
+- Add defaults for disabled notification, timeout, failure policy, bundle ID
+  template, idempotency key template, and report path template.
+- Validate enabled config: endpoint must be an absolute URL, `token_env` must be
+  non-empty, timeout must be positive, failure policy must be supported, and
+  templates must use only known variables.
+- Keep `token_env` as the configured token selector. Do not add raw token config.
+- Update maintained examples and config documentation only after code support is
+  implemented.
+
+Acceptance criteria:
+
+- Disabled distributor notification requires no endpoint or token env.
+- Enabled distributor notification rejects invalid endpoint, missing token env,
+  nonpositive timeout, unsupported failure policy, unknown template variables,
+  and invalid rendered report bundle paths.
+- Existing config loading behavior and precedence remain unchanged.
+
+Suggested tests:
+
+- Config defaults include disabled distributor notification.
+- Example configs load successfully.
+- Enabled config validation covers required fields and invalid template cases.
+- A distributor token supplied through `secrets.directory` is visible through
+  the configured `token_env` name after config loading.
+
+### Stage 3: Distributor Adapter
+
+Goal: isolate distributor package usage behind an internal adapter.
+
+Implementation guidance:
+
+- Add `internal/adapters/distributor` with package-owned request and result
+  types.
+- Read the bearer token from the configured environment variable after
+  secrets-directory loading has populated environment state.
+- Use distributor `UploadFiles` with one file:
+  the managed Markdown report as the source path and the rendered
+  `report_path_template` as the bundle path.
+- Return upload result information that app orchestration can record or expose
+  without leaking distributor dependency types.
+- Wrap upload errors with endpoint, bundle ID, and report path context while
+  avoiding token exposure.
+
+Acceptance criteria:
+
+- No distributor dependency types leak outside `internal/adapters/distributor`.
+- Missing source report path, missing token, invalid bundle path, and upload
+  failure return actionable errors.
+- Idempotency conflict errors are preserved or wrapped clearly enough for
+  troubleshooting.
+
+Suggested tests:
+
+- Adapter request construction uses the configured endpoint, token, bundle ID,
+  idempotency key, source report path, and bundle-relative path.
+- Token values do not appear in returned errors.
+- Upload conflict and generic upload failure produce useful wrapped errors.
+- Tokens supplied through the secrets directory are accepted via `token_env`.
+
+### Stage 4: App Notify Hook
+
+Goal: upload successful generated reports through the configured notifier.
+
+Implementation guidance:
+
+- Add an app-owned notifier interface so app tests can use fakes.
+- Construct distributor notification requests from the successful
+  `ReportResult`, resolved report definition, configured location, and
+  notification config.
+- Invoke notification only after Scriptorium report generation succeeds and
+  final metadata has been saved.
+- Do not notify after Weather API, briefing, prompt input, render preflight, or
+  Scriptorium run failure.
+- In batch mode, notify each successful report independently. If notification
+  fails and distributor notification is enabled, mark that report failed and
+  make the aggregate batch result nonzero.
+
+Acceptance criteria:
+
+- Disabled notification is a no-op.
+- Single-report generation returns an error when enabled upload fails.
+- Batch generation continues independent reports, records upload failures per
+  report, and exits nonzero when any enabled notification fails.
+- Existing report artifact paths, optional output copies, and Scriptorium
+  behavior remain unchanged.
+
+Suggested tests:
+
+- No notifier call occurs before successful report generation.
+- Successful notifier call receives the managed report path, not `--out` or
+  `--out-dir` copy paths.
+- Enabled notifier failure affects single-report and batch command outcomes.
+- Batch still continues later reports after one notification failure.
+
+### Stage 5: CLI And Output Behavior
+
+Goal: preserve CLI syntax while surfacing notification outcome where useful.
+
+Implementation guidance:
+
+- Do not add distributor-specific CLI flags in the first implementation; use
+  configuration only.
+- Preserve existing command names, flags, help text shape, workspace paths, and
+  generated Markdown outputs.
+- Extend batch JSON report items only if the implementation records useful
+  notification status, accepted distributor run ID, or notification error.
+- Keep stderr status lines concise and avoid exposing secrets.
+
+Acceptance criteria:
+
+- `go run ./cmd/weatherreporter --help` remains accurate.
+- Existing generate and run command syntax remains stable.
+- Batch summaries clearly indicate notification-caused report failures if
+  notification status is added.
+
+Suggested tests:
+
+- Existing CLI parser tests continue to pass.
+- Batch JSON includes notification fields only when implemented and documented.
+- Secret-like token values never appear in CLI output.
+
+### Stage 6: Documentation
+
+Goal: document implemented distributor behavior after the feature exists.
+
+Implementation guidance:
+
+- Update `docs/config.md` with `secrets.directory`,
+  `notify.distributor` fields, and defaults.
+- Update `docs/operations.md` with notification timing, failure behavior, and
+  the fact that managed report paths are uploaded.
+- Update `docs/troubleshooting.md` with invalid secrets directory, missing
+  token, upload conflict, upload rejection, and distributor unavailable cases.
+- Update `docs/internal/app-orchestration.md` to include notify ordering.
+- Add `docs/internal/distributor-adapter.md` for the adapter contract.
+- Keep distributor API details in `docs/integrations/distributor/`; link there
+  instead of duplicating the full upstream contract.
+
+Acceptance criteria:
+
+- Non-roadmap docs describe only implemented notification behavior.
+- Config examples include no raw tokens.
+- Secret handling documentation describes mechanisms, not secret values.
+- Documentation clearly distinguishes distributor destination routing from
+  weatherreporter upload responsibilities.
+
+### Stage 7: Final Validation
+
+Goal: verify the feature without requiring live distributor service access.
+
+Validation commands:
+
+```bash
+go test ./...
+go run ./cmd/weatherreporter --help
+git diff --check
+```
+
+Additional validation:
+
+- Run focused config, adapter, app, and CLI tests.
+- Confirm maintained example configs load.
+- Search for accidental raw-token config examples.
+- Confirm `go.mod` includes distributor only after implementation requires it.
+- Confirm no test failures leak secret values in error output.
+
+## Open Questions
+
+No questions block this roadmap. The initial implementation should use the
+locked defaults above and expose configurable templates for bundle identity and
+bundle-relative report path. Secrets-directory support is generic project
+infrastructure and should remain useful for future integrations beyond
+distributor.
--- a/docs/roadmap/implementation.md
+++ b/docs/roadmap/implementation.md
@@ -0,0 +1,297 @@
+# Distributor Feature Implementation Roadmap
+
+This roadmap turns `docs/roadmap/distributor.md` into staged implementation
+work for a future coding agent. It is a planning document only. The feature is
+not implemented until the stages below are completed and the implemented docs
+are updated.
+
+## Purpose
+
+Implement generic file-backed environment secrets and distributor notification
+support while preserving current CLI syntax, managed artifact paths, RunIDs,
+and report output behavior. Distributor package types must stay behind a narrow
+adapter boundary, and distributor should receive explicitly selected generated
+Markdown files rather than scanning `workspace/`.
+
+## Locked Decisions
+
+- Implement generic `secrets.directory` support before distributor notification.
+- Secret files overwrite pre-existing environment variables.
+- Never expose secret values in errors, logs, CLI output, metadata, docs, or
+  examples.
+- Add `notify.distributor` config; do not add raw token config.
+- Keep public CLI commands and flags unchanged.
+- Upload one distributor bundle per successfully generated report.
+- Initial distributor bundles include only the managed Markdown report at
+  `ReportResult.ReportPath`.
+- The default bundle-relative Markdown path uses the report definition's
+  `BatchOutputName`.
+- Default bundle ID and idempotency key derive from producer, location ID,
+  report ID, and RunID.
+- Enabled distributor upload failure is a report failure and should produce a
+  nonzero command exit.
+
+## Stage 1: Secrets Directory Support
+
+Goal: implement generic file-backed environment secret loading in
+`internal/config`.
+
+Implementation guidance:
+
+- Add `secrets.directory` config with a built-in default of `""`, meaning
+  disabled.
+- Load secrets after config file parsing and CLI overrides, before validation
+  completes.
+- Read only regular files directly under the configured directory.
+- Map each file basename to an environment variable name and file contents to
+  the value.
+- Validate names with `[A-Za-z_][A-Za-z0-9_]*`.
+- Reject missing configured directory, invalid filenames, subdirectories,
+  symlinks, unreadable files, and empty filenames.
+- Strip one trailing `\n` or `\r\n`; preserve all other bytes.
+- Keep error messages actionable but do not include secret values.
+
+Acceptance criteria:
+
+- Empty `secrets.directory` performs no environment changes.
+- Valid secret files set environment variables after config loading.
+- Secret file values overwrite existing environment variables.
+- Invalid entries fail before application workflows run.
+- Existing config loading behavior remains unchanged when secrets are disabled.
+
+Tests:
+
+- Defaults leave secrets loading disabled.
+- Valid file sets an environment variable.
+- File value overwrites an existing environment variable.
+- Missing directory, invalid filename, subdirectory, symlink, and unreadable
+  file cases fail.
+- Newline and CRLF trimming works exactly once.
+- Secret values are absent from error strings.
+
+## Stage 2: Notify Configuration And Templates
+
+Goal: add disabled-by-default `notify.distributor` config and template
+validation.
+
+Planned config:
+
+```yaml
+notify:
+  distributor:
+    enabled: false
+    endpoint: https://distributor.example.com
+    token_env: DISTRIBUTOR_UPLOAD_TOKEN
+    timeout: 30s
+    failure_policy: error
+    bundle_id_template: "weatherreporter.{location_id}.{report_id}.{run_id}"
+    idempotency_key_template: "{bundle_id}"
+    report_path_template: "{batch_output_name}"
+```
+
+Implementation guidance:
+
+- Add notify/distributor config structs, defaults, YAML loading, and validation
+  under `internal/config`.
+- When disabled, omit endpoint and token requirements.
+- When enabled, validate absolute endpoint URL, non-empty `token_env`, positive
+  timeout, `failure_policy=error`, known template variables, and valid rendered
+  report bundle paths.
+- Supported template variables are `location_id`, `report_id`, `run_id`,
+  `artifact_group`, and `batch_output_name`; `idempotency_key_template` may
+  also reference `{bundle_id}`.
+- Do not add distributor-specific CLI flags.
+
+Acceptance criteria:
+
+- Defaults include disabled distributor notification.
+- Enabled config rejects invalid endpoint, token env, timeout, failure policy,
+  templates, and rendered bundle paths.
+- Rendered bundle paths reject absolute paths, backslashes, `.`, `..`, empty
+  segments, `manifest.json`, and `.distributor.json`.
+- Tokens supplied by `secrets.directory` are visible through `token_env` after
+  config loading.
+
+Tests:
+
+- Config defaults and examples load.
+- Disabled distributor config accepts omitted distributor fields.
+- Enabled validation covers each invalid field.
+- Template rendering and validation are covered with table tests.
+
+## Stage 3: Distributor Adapter
+
+Goal: isolate distributor upload behavior in `internal/adapters/distributor`.
+
+Implementation guidance:
+
+- Add weatherreporter-owned request and result types.
+- Keep `gitea.maximumdirect.net/eric/distributor/pkg/upload` and
+  `gitea.maximumdirect.net/eric/distributor/pkg/bundle` types inside the
+  adapter.
+- Read the bearer token from `token_env` after config/secrets loading.
+- Use `UploadFiles` with exactly one file: source path is the managed Markdown
+  report path and bundle path is the rendered `report_path_template`.
+- Return the accepted distributor run ID when available.
+- Wrap endpoint, bundle ID, idempotency key, source path, and bundle path
+  context into errors without exposing token values.
+- Preserve idempotency conflict diagnosis.
+
+Acceptance criteria:
+
+- No distributor dependency types leak outside the adapter.
+- Missing token, source path, bundle path, or upload client inputs fail with
+  actionable errors.
+- Upload success returns a weatherreporter-owned result.
+- Upload failure errors do not include token values.
+
+Tests:
+
+- Fake upload client receives expected endpoint, token, bundle ID, idempotency
+  key, source path, and bundle path.
+- Missing token/source path/bundle path errors are actionable.
+- Token values are not present in errors.
+- Upload conflict and generic failure are wrapped clearly.
+
+## Stage 4: App Notification Hook
+
+Goal: add app-level notification orchestration after successful report
+generation.
+
+Implementation guidance:
+
+- Add an app-owned notifier interface for tests.
+- Default to no-op notification when distributor notification is disabled.
+- Notify only after successful `scriptorium run` and final metadata save.
+- Use `ReportResult.ReportPath` as the distributor source file.
+- Never use `--out` or `--out-dir` copies as distributor source files.
+- Do not notify after Weather API, briefing, prompt input, render preflight, or
+  Scriptorium run failure.
+- Single-report generation should return an error when enabled notification
+  fails.
+- Batch generation should continue independent reports; notification failure
+  marks that report failed and makes the aggregate batch nonzero.
+
+Acceptance criteria:
+
+- Disabled notification leaves current generation behavior unchanged.
+- Successful notification is recorded in the app result when useful.
+- Enabled notification failure fails a single-report command.
+- Batch results distinguish generation failure from notification failure enough
+  for operators to diagnose the failed report.
+- Existing artifact paths, output copies, and Scriptorium behavior remain
+  unchanged.
+
+Tests:
+
+- No notifier call occurs on pre-generation failures.
+- Notifier receives the managed report path.
+- Enabled notifier success and failure are covered.
+- Batch continues after one notification failure and reports aggregate failure.
+
+## Stage 5: CLI Output And Batch Reporting
+
+Goal: keep public CLI syntax stable while surfacing notification outcome if the
+app records it.
+
+Implementation guidance:
+
+- Do not add distributor-specific CLI flags.
+- Preserve current command names, help shape, generated paths, `--out`, and
+  `--out-dir`.
+- If batch JSON is extended, use explicit fields such as
+  `notificationStatus`, `notificationRunId`, and `notificationError`.
+- Keep stderr status lines concise and secret-free.
+
+Acceptance criteria:
+
+- `weatherreporter --help` remains accurate.
+- Existing CLI parser behavior remains stable.
+- Batch JSON includes notification details only when implemented and
+  documented.
+- CLI stdout and stderr do not include token values.
+
+Tests:
+
+- Existing CLI parser tests pass, except intentional JSON additions.
+- Batch output covers notification success and failure.
+- Secret-like token values are absent from CLI output.
+
+## Stage 6: Documentation
+
+Goal: update implemented documentation after the feature exists.
+
+Implementation guidance:
+
+- Update `docs/config.md` with `secrets.directory`, `notify.distributor`,
+  defaults, and secret handling.
+- Update `docs/operations.md` with notification timing, failure behavior, and
+  managed report upload source.
+- Update `docs/troubleshooting.md` with invalid secrets directory, missing
+  token, upload conflict, upload rejection, and distributor unavailable cases.
+- Update `docs/internal/app-orchestration.md` to include notify ordering.
+- Add `docs/internal/distributor-adapter.md` for adapter inputs, outputs,
+  boundaries, config fields, failure behavior, tests, and invariants.
+- Keep full upstream distributor contract details in
+  `docs/integrations/distributor/`.
+
+Acceptance criteria:
+
+- Non-roadmap docs describe only implemented behavior.
+- Config examples load and contain no raw secrets.
+- Secret handling docs describe mechanisms, not secret values.
+- Docs distinguish weatherreporter upload responsibility from distributor
+  destination routing and Markdown-to-HTML transformation.
+
+Tests and checks:
+
+- Run config example loading tests.
+- Search docs and examples for accidental raw tokens.
+- Run `git diff --check`.
+
+## Stage 7: Final Validation
+
+Goal: verify behavior without a live distributor service.
+
+Validation commands:
+
+```bash
+go test ./...
+go run ./cmd/weatherreporter --help
+git diff --check
+go test ./internal/config ./internal/app ./internal/cli ./internal/adapters/distributor
+```
+
+Manual validation:
+
+- No token values appear in errors, CLI output, docs, examples, metadata, or
+  batch JSON.
+- `go.mod` includes distributor only after adapter implementation requires it.
+- Existing report paths, RunIDs, and `--out` / `--out-dir` behavior are
+  unchanged.
+- Distributor upload tests use fakes or local test doubles, not a live service.
+
+## Deferred Work
+
+These are out of scope for the initial implementation:
+
+- `failure_policy: warn`.
+- Uploading metadata, briefing, data package, or preflight artifacts.
+- Distributor status polling after `202 Accepted`.
+- Durable upload retry queues.
+- Distributor-specific CLI flags.
+- Making distributor scan `workspace/`.
+- Destination routing, Markdown-to-HTML transformation, public URLs, or nginx
+  layout in `weatherreporter`.
+
+## Global Validation Checklist
+
+- Preserve current public CLI syntax.
+- Preserve managed workspace artifact paths.
+- Keep distributor dependency types behind `internal/adapters/distributor`.
+- Keep secrets out of user-facing and persisted output.
+- Keep examples valid and secret-free.
+- Update implemented docs in the same change as implementation.
+- Run `go test ./...`, `go run ./cmd/weatherreporter --help`, and
+  `git diff --check` before finishing implementation.
+