weatherreporter/docs/roadmap/distributor.md

# Distributor Integration Roadmap

This roadmap describes planned work for adding `distributor` notification
support to `weatherreporter`. The feature is not implemented yet, so this plan
lives under `docs/roadmap/`; non-roadmap documentation should not describe
`notify.distributor` as available until implementation is complete.

## Purpose

`distributor` collects outputs from producer applications and publishes them
through configured downstream pipelines. For `weatherreporter`, the planned
integration should upload generated Markdown weather reports after successful
report generation, without requiring `distributor` to scan or understand the
managed `workspace/` layout.

## Current Repository Facts

- `weatherreporter` does not currently implement a notify hook.
- Generated Markdown reports are written to the managed report path returned as
  `ReportResult.ReportPath`.
- Optional `--out` and `--out-dir` copies are user-requested extra copies and
  are not canonical integration inputs.
- The managed workspace also contains briefing snapshots, metadata, prompt data
  packages, and preflight artifacts; these should not be treated as a
  distributor source tree.
- The provided distributor docs recommend `pkg/upload.UploadFiles` for
  producers that already have generated files, so the initial integration can
  upload selected report files directly.

## Decisions Locked

- Add generic configurable secrets-directory support before the distributor
  adapter work. This is project infrastructure, not a distributor-only token
  helper.
- When a secrets directory is configured, each regular file directly under that
  directory maps filename to environment variable name and file contents to
  environment variable value.
- Secret-file values overwrite pre-existing environment variables.
- Secrets loading must never log, print, persist, or include secret values in
  errors, metadata, batch output, examples, or docs.
- Add a narrow `internal/adapters/distributor` adapter for the external
  integration.
- Keep `gitea.maximumdirect.net/eric/distributor/pkg/upload` and
  `gitea.maximumdirect.net/eric/distributor/pkg/bundle` types inside that
  adapter.
- Upload one distributor bundle per successfully generated report.
- Include only the generated Markdown report in the initial bundle.
- Use the report definition's batch output filename as the default
  bundle-relative Markdown path, such as `daily.md`, `tomorrow.md`,
  `three-day.md`, `weekend.md`, or `storm.md`.
- Derive the default bundle ID and idempotency key from producer name, location
  ID, report ID, and RunID.
- Treat distributor upload failure as a report failure when notification is
  enabled; single-report and batch commands should exit nonzero for enabled
  upload failures.
- Keep destination routing, Markdown-to-HTML transformation, public URLs, and
  nginx directory layout in distributor configuration, not in
  `weatherreporter`.

## Planned Configuration

Add future generic secrets configuration:

```yaml
secrets:
  directory: ""
```

Add future distributor configuration under `notify.distributor`:

```yaml
notify:
  distributor:
    enabled: false
    endpoint: https://distributor.example.com
    token_env: DISTRIBUTOR_UPLOAD_TOKEN
    timeout: 30s
    failure_policy: error
    bundle_id_template: "weatherreporter.{location_id}.{report_id}.{run_id}"
    idempotency_key_template: "{bundle_id}"
    report_path_template: "{batch_output_name}"
```

Planned behavior:

- `secrets.directory` defaults to an empty string, which disables secrets
  loading.
- When `secrets.directory` is configured, load only regular files directly
  under that directory.
- Each secret filename must be a valid environment variable name matching
  `[A-Za-z_][A-Za-z0-9_]*`.
- Reject missing configured directory, invalid filenames, subdirectories,
  symlinks, unreadable files, and empty filenames.
- Strip one trailing `\n` or `\r\n` from secret file contents for compatibility
  with common mounted-secret formats. Preserve all other bytes.
- `enabled` defaults to `false`.
- `endpoint` is required only when distributor notification is enabled.
- `token_env` is required only when enabled. Raw bearer tokens should not be
  stored in config files. The token value is read from normal environment state
  after secrets-directory loading has been applied.
- `timeout` controls the upload call timeout and should default to `30s`.
- `failure_policy` should initially support `error`. Reserve `warn` as a
  future option only if it is implemented and tested.
- Template variables should be explicit and validated:
  `location_id`, `report_id`, `run_id`, `artifact_group`, and
  `batch_output_name`.
- `idempotency_key_template` may reference `{bundle_id}` after the bundle ID is
  rendered.
- `report_path_template` should be user-visible from the first implementation
  because bundle-relative paths can affect downstream distributor routing.
- Rendered template values must produce valid distributor bundle paths: relative
  slash-separated paths with no empty path, absolute path, backslash, `.`, `..`,
  empty segment, `manifest.json`, or `.distributor.json`.

## Recommended Implementation Stages

### Stage 1: Secrets Directory Support

Goal: add generic file-backed environment secret loading before distributor
notification depends on environment tokens.

Implementation guidance:

- Add a generic `secrets.directory` config field under `internal/config`.
- Default the directory to an empty string so secrets loading is disabled unless
  explicitly configured.
- Load secrets after config file parsing and before validation that needs
  environment-backed values.
- Map each regular file directly under the configured directory to an
  environment variable: filename becomes the variable name and contents become
  the value.
- Use secret-file values to overwrite pre-existing environment variables.
- Strip one trailing `\n` or `\r\n`; preserve all other bytes.
- Reject missing directories, invalid environment-variable filenames,
  subdirectories, symlinks, unreadable files, and empty filenames.
- Keep secret values out of errors, logs, CLI output, metadata, and examples.

Acceptance criteria:

- Empty `secrets.directory` performs no environment changes.
- Configured secrets directory values are available through normal environment
  lookup after config loading.
- Secret files overwrite existing environment variables with the same name.
- Invalid directory entries fail with path/name context but without secret
  values.
- Existing config loading behavior remains unchanged when secrets are disabled.

Suggested tests:

- Config defaults leave secrets loading disabled.
- A configured missing secrets directory fails.
- Regular files set environment variables and overwrite existing values.
- Invalid filenames, subdirectories, symlinks, and unreadable files fail.
- One trailing newline or CRLF is stripped; other content is preserved.
- No secret values appear in returned errors.

### Stage 2: Config And Validation

Goal: add disabled-by-default distributor notification configuration.

Implementation guidance:

- Add notify/distributor config structs under `internal/config`.
- Add defaults for disabled notification, timeout, failure policy, bundle ID
  template, idempotency key template, and report path template.
- Validate enabled config: endpoint must be an absolute URL, `token_env` must be
  non-empty, timeout must be positive, failure policy must be supported, and
  templates must use only known variables.
- Keep `token_env` as the configured token selector. Do not add raw token config.
- Update maintained examples and config documentation only after code support is
  implemented.

Acceptance criteria:

- Disabled distributor notification requires no endpoint or token env.
- Enabled distributor notification rejects invalid endpoint, missing token env,
  nonpositive timeout, unsupported failure policy, unknown template variables,
  and invalid rendered report bundle paths.
- Existing config loading behavior and precedence remain unchanged.

Suggested tests:

- Config defaults include disabled distributor notification.
- Example configs load successfully.
- Enabled config validation covers required fields and invalid template cases.
- A distributor token supplied through `secrets.directory` is visible through
  the configured `token_env` name after config loading.

### Stage 3: Distributor Adapter

Goal: isolate distributor package usage behind an internal adapter.

Implementation guidance:

- Add `internal/adapters/distributor` with package-owned request and result
  types.
- Read the bearer token from the configured environment variable after
  secrets-directory loading has populated environment state.
- Use distributor `UploadFiles` with one file:
  the managed Markdown report as the source path and the rendered
  `report_path_template` as the bundle path.
- Return upload result information that app orchestration can record or expose
  without leaking distributor dependency types.
- Wrap upload errors with endpoint, bundle ID, and report path context while
  avoiding token exposure.

Acceptance criteria:

- No distributor dependency types leak outside `internal/adapters/distributor`.
- Missing source report path, missing token, invalid bundle path, and upload
  failure return actionable errors.
- Idempotency conflict errors are preserved or wrapped clearly enough for
  troubleshooting.

Suggested tests:

- Adapter request construction uses the configured endpoint, token, bundle ID,
  idempotency key, source report path, and bundle-relative path.
- Token values do not appear in returned errors.
- Upload conflict and generic upload failure produce useful wrapped errors.
- Tokens supplied through the secrets directory are accepted via `token_env`.

### Stage 4: App Notify Hook

Goal: upload successful generated reports through the configured notifier.

Implementation guidance:

- Add an app-owned notifier interface so app tests can use fakes.
- Construct distributor notification requests from the successful
  `ReportResult`, resolved report definition, configured location, and
  notification config.
- Invoke notification only after Scriptorium report generation succeeds and
  final metadata has been saved.
- Do not notify after Weather API, briefing, prompt input, render preflight, or
  Scriptorium run failure.
- In batch mode, notify each successful report independently. If notification
  fails and distributor notification is enabled, mark that report failed and
  make the aggregate batch result nonzero.

Acceptance criteria:

- Disabled notification is a no-op.
- Single-report generation returns an error when enabled upload fails.
- Batch generation continues independent reports, records upload failures per
  report, and exits nonzero when any enabled notification fails.
- Existing report artifact paths, optional output copies, and Scriptorium
  behavior remain unchanged.

Suggested tests:

- No notifier call occurs before successful report generation.
- Successful notifier call receives the managed report path, not `--out` or
  `--out-dir` copy paths.
- Enabled notifier failure affects single-report and batch command outcomes.
- Batch still continues later reports after one notification failure.

### Stage 5: CLI And Output Behavior

Goal: preserve CLI syntax while surfacing notification outcome where useful.

Implementation guidance:

- Do not add distributor-specific CLI flags in the first implementation; use
  configuration only.
- Preserve existing command names, flags, help text shape, workspace paths, and
  generated Markdown outputs.
- Extend batch JSON report items only if the implementation records useful
  notification status, accepted distributor run ID, or notification error.
- Keep stderr status lines concise and avoid exposing secrets.

Acceptance criteria:

- `go run ./cmd/weatherreporter --help` remains accurate.
- Existing generate and run command syntax remains stable.
- Batch summaries clearly indicate notification-caused report failures if
  notification status is added.

Suggested tests:

- Existing CLI parser tests continue to pass.
- Batch JSON includes notification fields only when implemented and documented.
- Secret-like token values never appear in CLI output.

### Stage 6: Documentation

Goal: document implemented distributor behavior after the feature exists.

Implementation guidance:

- Update `docs/config.md` with `secrets.directory`,
  `notify.distributor` fields, and defaults.
- Update `docs/operations.md` with notification timing, failure behavior, and
  the fact that managed report paths are uploaded.
- Update `docs/troubleshooting.md` with invalid secrets directory, missing
  token, upload conflict, upload rejection, and distributor unavailable cases.
- Update `docs/internal/app-orchestration.md` to include notify ordering.
- Add `docs/internal/distributor-adapter.md` for the adapter contract.
- Keep distributor API details in `docs/integrations/distributor/`; link there
  instead of duplicating the full upstream contract.

Acceptance criteria:

- Non-roadmap docs describe only implemented notification behavior.
- Config examples include no raw tokens.
- Secret handling documentation describes mechanisms, not secret values.
- Documentation clearly distinguishes distributor destination routing from
  weatherreporter upload responsibilities.

### Stage 7: Final Validation

Goal: verify the feature without requiring live distributor service access.

Validation commands:

```bash
go test ./...
go run ./cmd/weatherreporter --help
git diff --check
```

Additional validation:

- Run focused config, adapter, app, and CLI tests.
- Confirm maintained example configs load.
- Search for accidental raw-token config examples.
- Confirm `go.mod` includes distributor only after implementation requires it.
- Confirm no test failures leak secret values in error output.

## Open Questions

No questions block this roadmap. The initial implementation should use the
locked defaults above and expose configurable templates for bundle identity and
bundle-relative report path. Secrets-directory support is generic project
infrastructure and should remain useful for future integrations beyond
distributor.