weatherreporter/docs/roadmap/implementation.md

Distributor Feature Implementation Roadmap

This roadmap turns docs/roadmap/distributor.md into staged implementation work for a future coding agent. It is a planning document only. The feature is not implemented until the stages below are completed and the implemented docs are updated.

Purpose

Implement generic file-backed environment secrets and distributor notification support while preserving current CLI syntax, managed artifact paths, RunIDs, and report output behavior. Distributor package types must stay behind a narrow adapter boundary, and distributor should receive explicitly selected generated Markdown files rather than scanning workspace/.

Locked Decisions

• Implement generic secrets.directory support before distributor notification.
• Secret files overwrite pre-existing environment variables.
• Never expose secret values in errors, logs, CLI output, metadata, docs, or examples.
• Add notify.distributor config; do not add raw token config.
• Keep public CLI commands and flags unchanged.
• Upload one distributor bundle per successfully generated report.
• Initial distributor bundles include only the managed Markdown report at ReportResult.ReportPath.
• The default bundle-relative Markdown path uses the report definition's BatchOutputName.
• Default bundle ID and idempotency key derive from producer, location ID, report ID, and RunID.
• Enabled distributor upload failure is a report failure and should produce a nonzero command exit.

Stage 1: Secrets Directory Support

Goal: implement generic file-backed environment secret loading in internal/config.

Implementation guidance:

• Add secrets.directory config with a built-in default of "", meaning disabled.
• Load secrets after config file parsing and CLI overrides, before validation completes.
• Read only regular files directly under the configured directory.
• Map each file basename to an environment variable name and file contents to the value.
• Validate names with [A-Za-z_][A-Za-z0-9_]*.
• Reject missing configured directory, invalid filenames, subdirectories, symlinks, unreadable files, and empty filenames.
• Strip one trailing \n or \r\n; preserve all other bytes.
• Keep error messages actionable but do not include secret values.

Acceptance criteria:

• Empty secrets.directory performs no environment changes.
• Valid secret files set environment variables after config loading.
• Secret file values overwrite existing environment variables.
• Invalid entries fail before application workflows run.
• Existing config loading behavior remains unchanged when secrets are disabled.

Tests:

• Defaults leave secrets loading disabled.
• Valid file sets an environment variable.
• File value overwrites an existing environment variable.
• Missing directory, invalid filename, subdirectory, symlink, and unreadable file cases fail.
• Newline and CRLF trimming works exactly once.
• Secret values are absent from error strings.

Stage 2: Notify Configuration And Templates

Goal: add disabled-by-default notify.distributor config and template validation.

Planned config:

notify:
  distributor:
    enabled: false
    endpoint: https://distributor.example.com
    token_env: DISTRIBUTOR_UPLOAD_TOKEN
    timeout: 30s
    failure_policy: error
    bundle_id_template: "weatherreporter.{location_id}.{report_id}.{run_id}"
    idempotency_key_template: "{bundle_id}"
    report_path_template: "{batch_output_name}"

Implementation guidance:

• Add notify/distributor config structs, defaults, YAML loading, and validation under internal/config.
• When disabled, omit endpoint and token requirements.
• When enabled, validate absolute endpoint URL, non-empty token_env, positive timeout, failure_policy=error, known template variables, and valid rendered report bundle paths.
• Supported template variables are location_id, report_id, run_id, artifact_group, and batch_output_name; idempotency_key_template may also reference {bundle_id}.
• Do not add distributor-specific CLI flags.

Acceptance criteria:

• Defaults include disabled distributor notification.
• Enabled config rejects invalid endpoint, token env, timeout, failure policy, templates, and rendered bundle paths.
• Rendered bundle paths reject absolute paths, backslashes, ., .., empty segments, manifest.json, and .distributor.json.
• Tokens supplied by secrets.directory are visible through token_env after config loading.

Tests:

• Config defaults and examples load.
• Disabled distributor config accepts omitted distributor fields.
• Enabled validation covers each invalid field.
• Template rendering and validation are covered with table tests.

Stage 3: Distributor Adapter

Goal: isolate distributor upload behavior in internal/adapters/distributor.

Implementation guidance:

• Add weatherreporter-owned request and result types.
• Keep gitea.maximumdirect.net/eric/distributor/pkg/upload and gitea.maximumdirect.net/eric/distributor/pkg/bundle types inside the adapter.
• Read the bearer token from token_env after config/secrets loading.
• Use UploadFiles with exactly one file: source path is the managed Markdown report path and bundle path is the rendered report_path_template.
• Return the accepted distributor run ID when available.
• Wrap endpoint, bundle ID, idempotency key, source path, and bundle path context into errors without exposing token values.
• Preserve idempotency conflict diagnosis.

Acceptance criteria:

• No distributor dependency types leak outside the adapter.
• Missing token, source path, bundle path, or upload client inputs fail with actionable errors.
• Upload success returns a weatherreporter-owned result.
• Upload failure errors do not include token values.

Tests:

• Fake upload client receives expected endpoint, token, bundle ID, idempotency key, source path, and bundle path.
• Missing token/source path/bundle path errors are actionable.
• Token values are not present in errors.
• Upload conflict and generic failure are wrapped clearly.

Stage 4: App Notification Hook

Goal: add app-level notification orchestration after successful report generation.

Implementation guidance:

• Add an app-owned notifier interface for tests.
• Default to no-op notification when distributor notification is disabled.
• Notify only after successful scriptorium run and final metadata save.
• Use ReportResult.ReportPath as the distributor source file.
• Never use --out or --out-dir copies as distributor source files.
• Do not notify after Weather API, briefing, prompt input, render preflight, or Scriptorium run failure.
• Single-report generation should return an error when enabled notification fails.
• Batch generation should continue independent reports; notification failure marks that report failed and makes the aggregate batch nonzero.

Acceptance criteria:

• Disabled notification leaves current generation behavior unchanged.
• Successful notification is recorded in the app result when useful.
• Enabled notification failure fails a single-report command.
• Batch results distinguish generation failure from notification failure enough for operators to diagnose the failed report.
• Existing artifact paths, output copies, and Scriptorium behavior remain unchanged.

Tests:

• No notifier call occurs on pre-generation failures.
• Notifier receives the managed report path.
• Enabled notifier success and failure are covered.
• Batch continues after one notification failure and reports aggregate failure.

Stage 5: CLI Output And Batch Reporting

Goal: keep public CLI syntax stable while surfacing notification outcome if the app records it.

Implementation guidance:

• Do not add distributor-specific CLI flags.
• Preserve current command names, help shape, generated paths, --out, and --out-dir.
• If batch JSON is extended, use explicit fields such as notificationStatus, notificationRunId, and notificationError.
• Keep stderr status lines concise and secret-free.

Acceptance criteria:

• weatherreporter --help remains accurate.
• Existing CLI parser behavior remains stable.
• Batch JSON includes notification details only when implemented and documented.
• CLI stdout and stderr do not include token values.

Tests:

• Existing CLI parser tests pass, except intentional JSON additions.
• Batch output covers notification success and failure.
• Secret-like token values are absent from CLI output.

Stage 6: Documentation

Goal: update implemented documentation after the feature exists.

Implementation guidance:

• Update docs/config.md with secrets.directory, notify.distributor, defaults, and secret handling.
• Update docs/operations.md with notification timing, failure behavior, and managed report upload source.
• Update docs/troubleshooting.md with invalid secrets directory, missing token, upload conflict, upload rejection, and distributor unavailable cases.
• Update docs/internal/app-orchestration.md to include notify ordering.
• Add docs/internal/distributor-adapter.md for adapter inputs, outputs, boundaries, config fields, failure behavior, tests, and invariants.
• Keep full upstream distributor contract details in docs/integrations/distributor/.

Acceptance criteria:

• Non-roadmap docs describe only implemented behavior.
• Config examples load and contain no raw secrets.
• Secret handling docs describe mechanisms, not secret values.
• Docs distinguish weatherreporter upload responsibility from distributor destination routing and Markdown-to-HTML transformation.

Tests and checks:

• Run config example loading tests.
• Search docs and examples for accidental raw tokens.
• Run git diff --check.

Stage 7: Final Validation

Goal: verify behavior without a live distributor service.

Validation commands:

go test ./...
go run ./cmd/weatherreporter --help
git diff --check
go test ./internal/config ./internal/app ./internal/cli ./internal/adapters/distributor

Manual validation:

• No token values appear in errors, CLI output, docs, examples, metadata, or batch JSON.
• go.mod includes distributor only after adapter implementation requires it.
• Existing report paths, RunIDs, and --out / --out-dir behavior are unchanged.
• Distributor upload tests use fakes or local test doubles, not a live service.

Deferred Work

These are out of scope for the initial implementation:

• failure_policy: warn.
• Uploading metadata, briefing, data package, or preflight artifacts.
• Distributor status polling after 202 Accepted.
• Durable upload retry queues.
• Distributor-specific CLI flags.
• Making distributor scan workspace/.
• Destination routing, Markdown-to-HTML transformation, public URLs, or nginx layout in weatherreporter.

Global Validation Checklist

• Preserve current public CLI syntax.
• Preserve managed workspace artifact paths.
• Keep distributor dependency types behind internal/adapters/distributor.
• Keep secrets out of user-facing and persisted output.
• Keep examples valid and secret-free.
• Update implemented docs in the same change as implementation.
• Run go test ./..., go run ./cmd/weatherreporter --help, and git diff --check before finishing implementation.