weatherreporter/docs/roadmap/distributor.md

Distributor Integration Roadmap

This roadmap describes planned work for adding distributor notification support to weatherreporter. The feature is not implemented yet, so this plan lives under docs/roadmap/; non-roadmap documentation should not describe notify.distributor as available until implementation is complete.

Purpose

distributor collects outputs from producer applications and publishes them through configured downstream pipelines. For weatherreporter, the planned integration should upload generated Markdown weather reports after successful report generation, without requiring distributor to scan or understand the managed workspace/ layout.

Current Repository Facts

• weatherreporter does not currently implement a notify hook.
• Generated Markdown reports are written to the managed report path returned as ReportResult.ReportPath.
• Optional --out and --out-dir copies are user-requested extra copies and are not canonical integration inputs.
• The managed workspace also contains briefing snapshots, metadata, prompt data packages, and preflight artifacts; these should not be treated as a distributor source tree.
• The provided distributor docs recommend pkg/upload.UploadFiles for producers that already have generated files, so the initial integration can upload selected report files directly.

Decisions Locked

• Add generic configurable secrets-directory support before the distributor adapter work. This is project infrastructure, not a distributor-only token helper.
• When a secrets directory is configured, each regular file directly under that directory maps filename to environment variable name and file contents to environment variable value.
• Secret-file values overwrite pre-existing environment variables.
• Secrets loading must never log, print, persist, or include secret values in errors, metadata, batch output, examples, or docs.
• Add a narrow internal/adapters/distributor adapter for the external integration.
• Keep gitea.maximumdirect.net/eric/distributor/pkg/upload and gitea.maximumdirect.net/eric/distributor/pkg/bundle types inside that adapter.
• Upload one distributor bundle per successfully generated report.
• Include only the generated Markdown report in the initial bundle.
• Use the report definition's batch output filename as the default bundle-relative Markdown path, such as daily.md, tomorrow.md, three-day.md, weekend.md, or storm.md.
• Derive the default bundle ID and idempotency key from producer name, location ID, report ID, and RunID.
• Treat distributor upload failure as a report failure when notification is enabled; single-report and batch commands should exit nonzero for enabled upload failures.
• Keep destination routing, Markdown-to-HTML transformation, public URLs, and nginx directory layout in distributor configuration, not in weatherreporter.

Planned Configuration

Add future generic secrets configuration:

secrets:
  directory: ""

Add future distributor configuration under notify.distributor:

notify:
  distributor:
    enabled: false
    endpoint: https://distributor.example.com
    token_env: DISTRIBUTOR_UPLOAD_TOKEN
    timeout: 30s
    failure_policy: error
    bundle_id_template: "weatherreporter.{location_id}.{report_id}.{run_id}"
    idempotency_key_template: "{bundle_id}"
    report_path_template: "{batch_output_name}"

Planned behavior:

• secrets.directory defaults to an empty string, which disables secrets loading.
• When secrets.directory is configured, load only regular files directly under that directory.
• Each secret filename must be a valid environment variable name matching [A-Za-z_][A-Za-z0-9_]*.
• Reject missing configured directory, invalid filenames, subdirectories, symlinks, unreadable files, and empty filenames.
• Strip one trailing \n or \r\n from secret file contents for compatibility with common mounted-secret formats. Preserve all other bytes.
• enabled defaults to false.
• endpoint is required only when distributor notification is enabled.
• token_env is required only when enabled. Raw bearer tokens should not be stored in config files. The token value is read from normal environment state after secrets-directory loading has been applied.
• timeout controls the upload call timeout and should default to 30s.
• failure_policy should initially support error. Reserve warn as a future option only if it is implemented and tested.
• Template variables should be explicit and validated: location_id, report_id, run_id, artifact_group, and batch_output_name.
• idempotency_key_template may reference {bundle_id} after the bundle ID is rendered.
• report_path_template should be user-visible from the first implementation because bundle-relative paths can affect downstream distributor routing.
• Rendered template values must produce valid distributor bundle paths: relative slash-separated paths with no empty path, absolute path, backslash, ., .., empty segment, manifest.json, or .distributor.json.

Recommended Implementation Stages

Stage 1: Secrets Directory Support

Goal: add generic file-backed environment secret loading before distributor notification depends on environment tokens.

Implementation guidance:

• Add a generic secrets.directory config field under internal/config.
• Default the directory to an empty string so secrets loading is disabled unless explicitly configured.
• Load secrets after config file parsing and before validation that needs environment-backed values.
• Map each regular file directly under the configured directory to an environment variable: filename becomes the variable name and contents become the value.
• Use secret-file values to overwrite pre-existing environment variables.
• Strip one trailing \n or \r\n; preserve all other bytes.
• Reject missing directories, invalid environment-variable filenames, subdirectories, symlinks, unreadable files, and empty filenames.
• Keep secret values out of errors, logs, CLI output, metadata, and examples.

Acceptance criteria:

• Empty secrets.directory performs no environment changes.
• Configured secrets directory values are available through normal environment lookup after config loading.
• Secret files overwrite existing environment variables with the same name.
• Invalid directory entries fail with path/name context but without secret values.
• Existing config loading behavior remains unchanged when secrets are disabled.

Suggested tests:

• Config defaults leave secrets loading disabled.
• A configured missing secrets directory fails.
• Regular files set environment variables and overwrite existing values.
• Invalid filenames, subdirectories, symlinks, and unreadable files fail.
• One trailing newline or CRLF is stripped; other content is preserved.
• No secret values appear in returned errors.

Stage 2: Config And Validation

Goal: add disabled-by-default distributor notification configuration.

Implementation guidance:

• Add notify/distributor config structs under internal/config.
• Add defaults for disabled notification, timeout, failure policy, bundle ID template, idempotency key template, and report path template.
• Validate enabled config: endpoint must be an absolute URL, token_env must be non-empty, timeout must be positive, failure policy must be supported, and templates must use only known variables.
• Keep token_env as the configured token selector. Do not add raw token config.
• Update maintained examples and config documentation only after code support is implemented.

Acceptance criteria:

• Disabled distributor notification requires no endpoint or token env.
• Enabled distributor notification rejects invalid endpoint, missing token env, nonpositive timeout, unsupported failure policy, unknown template variables, and invalid rendered report bundle paths.
• Existing config loading behavior and precedence remain unchanged.

Suggested tests:

• Config defaults include disabled distributor notification.
• Example configs load successfully.
• Enabled config validation covers required fields and invalid template cases.
• A distributor token supplied through secrets.directory is visible through the configured token_env name after config loading.

Stage 3: Distributor Adapter

Goal: isolate distributor package usage behind an internal adapter.

Implementation guidance:

• Add internal/adapters/distributor with package-owned request and result types.
• Read the bearer token from the configured environment variable after secrets-directory loading has populated environment state.
• Use distributor UploadFiles with one file: the managed Markdown report as the source path and the rendered report_path_template as the bundle path.
• Return upload result information that app orchestration can record or expose without leaking distributor dependency types.
• Wrap upload errors with endpoint, bundle ID, and report path context while avoiding token exposure.

Acceptance criteria:

• No distributor dependency types leak outside internal/adapters/distributor.
• Missing source report path, missing token, invalid bundle path, and upload failure return actionable errors.
• Idempotency conflict errors are preserved or wrapped clearly enough for troubleshooting.

Suggested tests:

• Adapter request construction uses the configured endpoint, token, bundle ID, idempotency key, source report path, and bundle-relative path.
• Token values do not appear in returned errors.
• Upload conflict and generic upload failure produce useful wrapped errors.
• Tokens supplied through the secrets directory are accepted via token_env.

Stage 4: App Notify Hook

Goal: upload successful generated reports through the configured notifier.

Implementation guidance:

• Add an app-owned notifier interface so app tests can use fakes.
• Construct distributor notification requests from the successful ReportResult, resolved report definition, configured location, and notification config.
• Invoke notification only after Scriptorium report generation succeeds and final metadata has been saved.
• Do not notify after Weather API, briefing, prompt input, render preflight, or Scriptorium run failure.
• In batch mode, notify each successful report independently. If notification fails and distributor notification is enabled, mark that report failed and make the aggregate batch result nonzero.

Acceptance criteria:

• Disabled notification is a no-op.
• Single-report generation returns an error when enabled upload fails.
• Batch generation continues independent reports, records upload failures per report, and exits nonzero when any enabled notification fails.
• Existing report artifact paths, optional output copies, and Scriptorium behavior remain unchanged.

Suggested tests:

• No notifier call occurs before successful report generation.
• Successful notifier call receives the managed report path, not --out or --out-dir copy paths.
• Enabled notifier failure affects single-report and batch command outcomes.
• Batch still continues later reports after one notification failure.

Stage 5: CLI And Output Behavior

Goal: preserve CLI syntax while surfacing notification outcome where useful.

Implementation guidance:

• Do not add distributor-specific CLI flags in the first implementation; use configuration only.
• Preserve existing command names, flags, help text shape, workspace paths, and generated Markdown outputs.
• Extend batch JSON report items only if the implementation records useful notification status, accepted distributor run ID, or notification error.
• Keep stderr status lines concise and avoid exposing secrets.

Acceptance criteria:

• go run ./cmd/weatherreporter --help remains accurate.
• Existing generate and run command syntax remains stable.
• Batch summaries clearly indicate notification-caused report failures if notification status is added.

Suggested tests:

• Existing CLI parser tests continue to pass.
• Batch JSON includes notification fields only when implemented and documented.
• Secret-like token values never appear in CLI output.

Stage 6: Documentation

Goal: document implemented distributor behavior after the feature exists.

Implementation guidance:

• Update docs/config.md with secrets.directory, notify.distributor fields, and defaults.
• Update docs/operations.md with notification timing, failure behavior, and the fact that managed report paths are uploaded.
• Update docs/troubleshooting.md with invalid secrets directory, missing token, upload conflict, upload rejection, and distributor unavailable cases.
• Update docs/internal/app-orchestration.md to include notify ordering.
• Add docs/internal/distributor-adapter.md for the adapter contract.
• Keep distributor API details in docs/integrations/distributor/; link there instead of duplicating the full upstream contract.

Acceptance criteria:

• Non-roadmap docs describe only implemented notification behavior.
• Config examples include no raw tokens.
• Secret handling documentation describes mechanisms, not secret values.
• Documentation clearly distinguishes distributor destination routing from weatherreporter upload responsibilities.

Stage 7: Final Validation

Goal: verify the feature without requiring live distributor service access.

Validation commands:

go test ./...
go run ./cmd/weatherreporter --help
git diff --check

Additional validation:

• Run focused config, adapter, app, and CLI tests.
• Confirm maintained example configs load.
• Search for accidental raw-token config examples.
• Confirm go.mod includes distributor only after implementation requires it.
• Confirm no test failures leak secret values in error output.

Open Questions

No questions block this roadmap. The initial implementation should use the locked defaults above and expose configurable templates for bundle identity and bundle-relative report path. Secrets-directory support is generic project infrastructure and should remain useful for future integrations beyond distributor.