Compare commits
8 Commits
9677835d84
...
138bc4e7e4
| Author | SHA1 | Date | |
|---|---|---|---|
| 138bc4e7e4 | |||
| e2529cfdf2 | |||
| b982b27f84 | |||
| c573cd5b4d | |||
| c2758d7a91 | |||
| 64cae8c4d9 | |||
| a2ba6f5382 | |||
| 7c8d9191c1 |
@@ -3,7 +3,8 @@
|
||||
`weatherreporter` is a Go application for preparing human-facing weather
|
||||
reports from normalized forecast data. It builds structured briefing packages,
|
||||
runs them through `scriptorium`, and keeps inspectable artifacts under a local
|
||||
workspace.
|
||||
workspace. It can also upload successfully generated managed Markdown reports
|
||||
to a configured `distributor` HTTP upload endpoint.
|
||||
|
||||
## Quickstart
|
||||
|
||||
|
||||
19
docs/cli.md
19
docs/cli.md
@@ -11,7 +11,9 @@ weatherreporter generate daily --date 2026-05-29 --out ./daily.md
|
||||
|
||||
This loads configuration, fetches weather data, writes managed workspace
|
||||
artifacts, runs `scriptorium render` as a preflight check, runs
|
||||
`scriptorium run`, and writes an extra Markdown copy to `./daily.md`.
|
||||
`scriptorium run`, and writes an extra Markdown copy to `./daily.md`. If
|
||||
distributor notification is enabled in configuration, the command also uploads
|
||||
the managed Markdown report after final metadata is saved.
|
||||
|
||||
## Commands
|
||||
|
||||
@@ -33,14 +35,20 @@ weatherreporter inspect sources [--config PATH] RUN_ID
|
||||
```
|
||||
|
||||
`generate` commands write briefing, data package, preflight, report, and
|
||||
metadata artifacts under the configured workspace. `generate storm` requires
|
||||
explicit event-window bounds with `--start` and `--end`.
|
||||
metadata artifacts under the configured workspace. `--out` writes an extra
|
||||
Markdown copy for the operator; distributor notification uses the managed
|
||||
report path, not the extra copy. `generate storm` requires explicit
|
||||
event-window bounds with `--start` and `--end`.
|
||||
|
||||
`run morning` generates Daily Today and the 3-Day Outlook, plus Weekend Outlook
|
||||
except on Sunday. `run evening` generates the Tomorrow Planning Brief. Batch
|
||||
runs continue independent reports after a failure, print a JSON summary to
|
||||
stdout, write compact status lines to stderr, and return nonzero when any report
|
||||
failed.
|
||||
failed. `--out-dir` writes extra Markdown copies for the operator; distributor
|
||||
notification uses each managed report path, not the extra copies. When
|
||||
notification is enabled, batch summaries and status lines include notification
|
||||
status, accepted distributor run ID, or notification error fields for each
|
||||
attempted report.
|
||||
|
||||
`inspect` commands read existing workspace artifacts and emit JSON to stdout.
|
||||
They do not fetch weather data or invoke `scriptorium`.
|
||||
@@ -61,6 +69,9 @@ They do not fetch weather data or invoke `scriptorium`.
|
||||
Storm times accept `YYYY-MM-DDTHH:MM` in the configured timezone or RFC3339
|
||||
timestamps with explicit offsets.
|
||||
|
||||
Distributor notification is configured only through `notify.distributor`; there
|
||||
are no distributor-specific CLI flags.
|
||||
|
||||
## Common Workflows
|
||||
|
||||
```sh
|
||||
|
||||
@@ -18,7 +18,7 @@ Precedence is:
|
||||
|
||||
The implemented configuration overrides are `--units` and `--tz`. Output flags
|
||||
control report copies for the current command but do not change configuration
|
||||
files. Environment-variable configuration is not implemented.
|
||||
files. Environment variables do not override configuration fields.
|
||||
|
||||
## Minimal Config
|
||||
|
||||
@@ -64,6 +64,53 @@ multiple configured forecast locations.
|
||||
The prompt-facing location object also includes `timezone`, derived from the
|
||||
effective `weather_api.timezone` after CLI overrides such as `--tz`.
|
||||
|
||||
### `secrets`
|
||||
|
||||
- `directory`: optional directory of file-backed environment secrets. Default:
|
||||
empty, which disables secret loading.
|
||||
|
||||
When configured, each regular file directly under `secrets.directory` is loaded
|
||||
after config file parsing and CLI overrides. The file basename must be a valid
|
||||
environment variable name matching `[A-Za-z_][A-Za-z0-9_]*`; the file contents
|
||||
become the environment variable value and overwrite any existing value. One
|
||||
trailing LF or CRLF is stripped. Subdirectories, symlinks, invalid filenames,
|
||||
missing directories, and unreadable files fail config loading.
|
||||
|
||||
### `notify`
|
||||
|
||||
`notify.distributor` controls distributor notification after successful report
|
||||
generation. It is disabled by default and does not add CLI flags. When enabled,
|
||||
weatherreporter uploads one distributor bundle per generated report after
|
||||
`scriptorium run` succeeds and final metadata is saved.
|
||||
|
||||
- `enabled`: whether distributor notification config is active. Default:
|
||||
`false`.
|
||||
- `endpoint`: absolute distributor endpoint URL. Required when enabled.
|
||||
Default: `https://distributor.example.com`.
|
||||
- `token_env`: environment variable name that will contain the distributor
|
||||
upload token. Required when enabled. Default: `DISTRIBUTOR_UPLOAD_TOKEN`.
|
||||
- `timeout`: distributor operation timeout. Must be greater than zero when
|
||||
enabled. Default: `30s`.
|
||||
- `failure_policy`: must be `error` when enabled. Default: `error`.
|
||||
- `bundle_id_template`: template for distributor bundle IDs. Default:
|
||||
`weatherreporter.{location_id}.{report_id}.{run_id}`.
|
||||
- `idempotency_key_template`: template for distributor idempotency keys.
|
||||
Default: `{bundle_id}`.
|
||||
- `report_path_template`: template for the Markdown report path inside the
|
||||
distributor bundle. Default: `{batch_output_name}`.
|
||||
|
||||
Supported template variables are `location_id`, `report_id`, `run_id`,
|
||||
`artifact_group`, and `batch_output_name`. `idempotency_key_template` may also
|
||||
use `bundle_id`.
|
||||
|
||||
Rendered report paths must be relative paths with `/` separators. They must not
|
||||
contain backslashes, empty path segments, `.`, `..`, `manifest.json`, or
|
||||
`.distributor.json`.
|
||||
|
||||
The upload token is read from the environment variable named by `token_env`
|
||||
after config loading and `secrets.directory` processing. Config files should
|
||||
name the variable only; they should not contain the token value.
|
||||
|
||||
### `missing_source`
|
||||
|
||||
- `default`: missing-source behavior for optional sources. One of `error`, `warn`, or `none`. Default: `warn`.
|
||||
@@ -115,8 +162,13 @@ snapshot exists and a threshold is crossed.
|
||||
|
||||
## Secrets
|
||||
|
||||
Configuration files should not contain secrets. The current Weather API and
|
||||
Scriptorium integration settings do not require secret fields.
|
||||
Configuration files should not contain raw secrets. Use `secrets.directory` to
|
||||
load secret values from files into environment variables for integrations that
|
||||
read credentials from the environment. Secret file names become environment
|
||||
variable names, and secret file contents become values. For distributor
|
||||
notification, this allows a file such as
|
||||
`<secrets.directory>/DISTRIBUTOR_UPLOAD_TOKEN` to supply the token referenced by
|
||||
`notify.distributor.token_env`.
|
||||
|
||||
## Maintained Examples
|
||||
|
||||
|
||||
@@ -7,8 +7,9 @@ This document describes the implemented workflow coordinator in `internal/app`.
|
||||
`internal/app` coordinates the top-level use cases after CLI parsing and config
|
||||
loading are complete. It resolves report definitions, fetches weather data,
|
||||
builds briefing and prompt-input artifacts, invokes Scriptorium through the
|
||||
adapter boundary, persists managed state, runs batches, and reads existing
|
||||
artifacts for inspection.
|
||||
adapter boundary, optionally notifies distributor through an app-owned notifier
|
||||
boundary, persists managed state, runs batches, and reads existing artifacts for
|
||||
inspection.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
@@ -22,13 +23,15 @@ Inputs:
|
||||
- resolved report definitions from `internal/report`
|
||||
- forecast bundles from `internal/adapters/weatherapi`
|
||||
- prior snapshots loaded from `internal/state`
|
||||
- optional renderer and state-store fakes for tests
|
||||
- optional renderer, notifier, and state-store fakes for tests
|
||||
|
||||
Outputs:
|
||||
|
||||
- generated report results with briefing, data package, preflight, report,
|
||||
metadata, prior snapshot, Recent Changes, and Scriptorium result details
|
||||
- batch summaries with per-report status, artifact paths, and error text
|
||||
metadata, prior snapshot, Recent Changes, Scriptorium result details, and
|
||||
notification result when attempted
|
||||
- batch summaries with per-report status, artifact paths, error text, and
|
||||
notification outcome when attempted
|
||||
- saved Weather API bundle JSON for fetch workflows
|
||||
- inspection JSON values for reports, metadata, briefings, data packages, prior
|
||||
snapshots, and source provenance
|
||||
@@ -42,8 +45,9 @@ Scriptorium argv.
|
||||
|
||||
Report selection and report identity policy come from `internal/report`.
|
||||
Weather API transport stays in `internal/adapters/weatherapi`. Scriptorium
|
||||
subprocess behavior stays in `internal/adapters/scriptorium`. Filesystem layout
|
||||
and persisted metadata stay in `internal/state`.
|
||||
subprocess behavior stays in `internal/adapters/scriptorium`. Distributor
|
||||
upload behavior stays in `internal/adapters/distributor`. Filesystem layout and
|
||||
persisted metadata stay in `internal/state`.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
@@ -52,6 +56,7 @@ and persisted metadata stay in `internal/state`.
|
||||
- `workspace.*` for filesystem state
|
||||
- `dayparts` for daily and outlook summarization
|
||||
- `recent_change.*` for structured Recent Changes thresholds
|
||||
- `notify.distributor.*` for optional notification after report generation
|
||||
|
||||
Output copy flags are command request fields. They are not configuration
|
||||
defaults.
|
||||
@@ -74,11 +79,15 @@ Single-report generation follows this order:
|
||||
12. Run Scriptorium report generation to the managed report path.
|
||||
13. Copy the managed report to the requested `--out` path when provided.
|
||||
14. Save metadata with the managed report path.
|
||||
15. If distributor notification is enabled, notify using the managed report
|
||||
path as the source file.
|
||||
|
||||
If render preflight returns both a result and an error, preflight JSON and
|
||||
metadata are persisted before the error is returned. If Scriptorium report
|
||||
generation returns an error after writing output, the managed report and
|
||||
metadata remain inspectable.
|
||||
metadata remain inspectable. Notification is not attempted after Weather API,
|
||||
briefing, prompt input, render, Scriptorium run, or metadata-save failures.
|
||||
`--out` copies are never used as notification source files.
|
||||
|
||||
## Batch Workflow
|
||||
|
||||
@@ -87,6 +96,10 @@ on Sunday. `run evening` resolves Daily Tomorrow. Batch output copy names come
|
||||
from report definitions. Batch generation continues independent reports after a
|
||||
failure, records each result, writes compact status lines to stderr, emits a
|
||||
JSON summary to stdout, and returns an aggregate error when any report failed.
|
||||
When notification is enabled, each successfully generated report is notified
|
||||
independently. Notification failure marks that report failed, records
|
||||
notification fields in the batch result, and does not stop later reports.
|
||||
`--out-dir` copies are never used as notification source files.
|
||||
|
||||
## Inspection Workflow
|
||||
|
||||
@@ -101,6 +114,8 @@ inspection view.
|
||||
- Weather API and briefing errors stop that report before Scriptorium runs.
|
||||
- Prompt input validation fails before render preflight.
|
||||
- Render and run errors preserve Scriptorium stderr and exit-code context.
|
||||
- Notification errors are wrapped with report ID, RunID, and managed report path
|
||||
context and are recorded separately in batch results.
|
||||
- Metadata and artifact path errors include filesystem context.
|
||||
- Batch failures are recorded per report and surfaced through an aggregate
|
||||
batch error.
|
||||
@@ -121,3 +136,5 @@ Inspect:
|
||||
- Render preflight precedes Scriptorium report generation.
|
||||
- Recent Changes are computed from structured briefing snapshots.
|
||||
- Metadata links artifacts produced for a run.
|
||||
- Distributor notification uses the managed Markdown report path, not extra
|
||||
output copies.
|
||||
|
||||
106
docs/internal/distributor-adapter.md
Normal file
106
docs/internal/distributor-adapter.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# Distributor Adapter Internals
|
||||
|
||||
This document describes the distributor upload adapter in
|
||||
`internal/adapters/distributor`.
|
||||
|
||||
## Purpose
|
||||
|
||||
The adapter submits generated weatherreporter Markdown reports to a configured
|
||||
distributor HTTP upload endpoint. It isolates distributor package types,
|
||||
token-env lookup, upload client construction, source-bundle file mapping,
|
||||
timeout handling, and upload error wrapping from app orchestration.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs:
|
||||
|
||||
- distributor endpoint URL
|
||||
- token environment variable name
|
||||
- upload timeout
|
||||
- bundle ID
|
||||
- idempotency key
|
||||
- source Markdown report path
|
||||
- bundle-relative Markdown path
|
||||
- context for cancellation
|
||||
|
||||
Outputs:
|
||||
|
||||
- accepted distributor run ID
|
||||
- accepted distributor status
|
||||
- weatherreporter-owned idempotency conflict error when applicable
|
||||
|
||||
## Boundaries
|
||||
|
||||
`internal/adapters/distributor` is the only weatherreporter package that imports
|
||||
`gitea.maximumdirect.net/eric/distributor/pkg/upload` or
|
||||
`gitea.maximumdirect.net/eric/distributor/pkg/bundle`.
|
||||
|
||||
The app layer passes weatherreporter-owned request values to the adapter. The
|
||||
adapter does not choose report types, render templates, select output copies,
|
||||
configure destinations, wait for downstream publication, transform Markdown, or
|
||||
persist notification state.
|
||||
|
||||
Full upstream distributor package and HTTP contract details stay under
|
||||
`docs/integrations/distributor/`.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
The adapter is built from `notify.distributor` config:
|
||||
|
||||
- `endpoint`
|
||||
- `token_env`
|
||||
- `timeout`
|
||||
|
||||
The app layer renders bundle ID, idempotency key, and bundle path from:
|
||||
|
||||
- `bundle_id_template`
|
||||
- `idempotency_key_template`
|
||||
- `report_path_template`
|
||||
|
||||
The token value is read from the environment variable named by `token_env`
|
||||
after config loading and `secrets.directory` processing.
|
||||
|
||||
## Upload Behavior
|
||||
|
||||
The adapter calls distributor `UploadFiles` with exactly one file:
|
||||
|
||||
- source path: the managed Markdown report path selected by app orchestration
|
||||
- bundle path: the rendered bundle-relative report path
|
||||
|
||||
The adapter creates a distributor upload client with the configured endpoint,
|
||||
bearer token, and timeout-backed HTTP client. It also wraps the upload context
|
||||
with the configured timeout when the timeout is greater than zero.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
The adapter validates required endpoint, token env name, token value, bundle ID,
|
||||
idempotency key, source path, bundle path, and upload client inputs before
|
||||
uploading.
|
||||
|
||||
Upload failures include endpoint, bundle ID, idempotency key, source path, and
|
||||
bundle path context. Token values are redacted from adapter errors.
|
||||
|
||||
Distributor idempotency conflicts are exposed as a weatherreporter-owned
|
||||
`IdempotencyConflictError`, so callers do not depend on upstream distributor
|
||||
types.
|
||||
|
||||
## Tests
|
||||
|
||||
Inspect:
|
||||
|
||||
- `internal/adapters/distributor/client_test.go`
|
||||
- `internal/app/app_test.go`
|
||||
- `internal/cli/root_test.go`
|
||||
|
||||
Adapter tests use a fake upload client factory and do not require a live
|
||||
distributor service.
|
||||
|
||||
## Invariants
|
||||
|
||||
- Distributor package types do not leak outside the adapter.
|
||||
- Only the managed Markdown report is uploaded.
|
||||
- The adapter never scans the workspace.
|
||||
- Token values are not included in errors, CLI output, metadata, docs, or
|
||||
examples.
|
||||
- Destination routing and Markdown-to-HTML transformation belong to
|
||||
distributor, not weatherreporter.
|
||||
@@ -19,7 +19,10 @@ weatherreporter generate storm --start 2026-05-29T18:00 --end 2026-05-30T06:00
|
||||
Each command resolves a report period, fetches a Weather API bundle, builds a
|
||||
briefing, builds a prompt input data package, runs `scriptorium render`, runs
|
||||
`scriptorium run`, and writes managed artifacts under the configured workspace.
|
||||
`--out PATH` writes an extra Markdown copy for the current generated report.
|
||||
When distributor notification is enabled, weatherreporter uploads the managed
|
||||
Markdown report after `scriptorium run` succeeds and final metadata is saved.
|
||||
`--out PATH` writes an extra Markdown copy for the current generated report; it
|
||||
is not used as the distributor upload source.
|
||||
|
||||
Implemented batch commands:
|
||||
|
||||
@@ -32,9 +35,12 @@ weatherreporter run evening
|
||||
except on Sunday. `run evening` generates the Tomorrow Planning Brief. Batch
|
||||
commands print a JSON summary to stdout, write compact per-report status lines
|
||||
to stderr, continue independent reports after one report fails, and return
|
||||
nonzero when any report failed. `--out-dir PATH` writes extra Markdown copies
|
||||
using report default filenames such as `daily.md`, `three-day.md`,
|
||||
`weekend.md`, and `tomorrow.md`.
|
||||
nonzero when any report failed. When notification is configured, the summary and
|
||||
status lines include notification status, accepted distributor run ID, or
|
||||
notification error fields for each attempted report. `--out-dir PATH` writes
|
||||
extra Markdown copies using report default filenames such as `daily.md`,
|
||||
`three-day.md`, `weekend.md`, and `tomorrow.md`; these copies are not used as
|
||||
distributor upload sources.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
@@ -117,8 +123,40 @@ Each generated report writes metadata that links:
|
||||
- preflight output path
|
||||
- managed Markdown report path
|
||||
|
||||
Batch summaries include report status, error text when applicable, valid
|
||||
period, and known artifact paths for each attempted report.
|
||||
Batch summaries include report status, error text when applicable, notification
|
||||
outcome when attempted, valid period, and known artifact paths for each
|
||||
attempted report. Notification fields are `notificationStatus`,
|
||||
`notificationRunId`, and `notificationError`.
|
||||
|
||||
## Distributor Notification
|
||||
|
||||
Distributor notification is configured with `notify.distributor` and is disabled
|
||||
by default. When enabled, weatherreporter uploads exactly one file per
|
||||
successfully generated report: the managed Markdown report path recorded in the
|
||||
report result and metadata. Extra copies written by `--out` or `--out-dir` are
|
||||
operator conveniences only.
|
||||
|
||||
The default bundle ID is derived from producer name, location ID, report ID, and
|
||||
RunID:
|
||||
|
||||
```text
|
||||
weatherreporter.{location_id}.{report_id}.{run_id}
|
||||
```
|
||||
|
||||
The default idempotency key is the rendered bundle ID. The default bundle path
|
||||
for the Markdown file is the report definition's batch output name, such as
|
||||
`daily.md`, `tomorrow.md`, `three-day.md`, or `weekend.md`.
|
||||
|
||||
Notification happens after final metadata save. Weather API, briefing,
|
||||
data-package, render preflight, Scriptorium run, and metadata-save failures do
|
||||
not trigger notification. A notification failure fails that report. In a batch,
|
||||
other reports continue, the failed report includes notification fields in the
|
||||
JSON summary, and the batch returns nonzero.
|
||||
|
||||
Weatherreporter is responsible for selecting the managed Markdown report,
|
||||
constructing a source bundle, and submitting it to the configured distributor
|
||||
HTTP endpoint. Distributor remains responsible for destination routing,
|
||||
publication, and any downstream Markdown-to-HTML transformation.
|
||||
|
||||
## Inspection
|
||||
|
||||
@@ -164,6 +202,8 @@ A failed generation run may still leave useful artifacts:
|
||||
preflight JSON and metadata are written for inspection.
|
||||
- If `scriptorium run` exits nonzero after writing a report, the managed report
|
||||
and metadata remain available.
|
||||
- If distributor notification fails, report artifacts and final metadata remain
|
||||
available, but the report or batch command returns nonzero.
|
||||
- For batch commands, inspect the stdout JSON summary first, then inspect the
|
||||
artifact paths for each failed report.
|
||||
|
||||
|
||||
@@ -13,6 +13,12 @@ Generated reports must be associated with explicit metadata, including report ty
|
||||
|
||||
`scriptorium` is an external adapter, not domain logic. Subprocess execution must be isolated under `internal/adapters/scriptorium`, use context-aware execution, avoid shell interpolation, capture actionable stderr, and keep scriptorium-specific flags from leaking into domain packages.
|
||||
|
||||
`distributor` is also an external adapter. Upload behavior must be isolated
|
||||
under `internal/adapters/distributor`, dependency types from the distributor
|
||||
module must not leak outside that adapter, and the selected upload source must
|
||||
be the managed Markdown report rather than optional output copies or broad
|
||||
workspace scans.
|
||||
|
||||
## Project Shape
|
||||
|
||||
Default to a small, explicit, dependency-light Go application. Keep the design modular enough to test and change safely, but do not add abstraction unless it protects a real boundary or enables a real extension point.
|
||||
@@ -54,7 +60,7 @@ Configuration precedence is:
|
||||
|
||||
Prefer YAML configuration unless the project has a strong reason to use another format. Config files should be discovered at `/usr/local/etc/<app_name>/config.yml`, with a CLI override via `--config`.
|
||||
|
||||
Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets.
|
||||
Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets. File-backed secrets are loaded through `secrets.directory`; secret values must not be logged, persisted, or included in user-facing output.
|
||||
|
||||
## Adapters and External Integrations
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ Developers and LLM coding agents should use it with
|
||||
- `internal/config`: configuration structs, defaults, loading, overrides, and
|
||||
validation.
|
||||
- `internal/fileutil`: shared atomic filesystem write and copy helpers.
|
||||
- `internal/adapters/distributor`: Distributor upload adapter.
|
||||
- `internal/adapters/weatherapi`: Weather API HTTP adapter.
|
||||
- `internal/adapters/scriptorium`: Scriptorium subprocess adapter.
|
||||
- `internal/forecast`: normalized bundle types and deterministic forecast
|
||||
@@ -45,7 +46,7 @@ Useful focused checks:
|
||||
```bash
|
||||
go test ./internal/cli ./internal/config
|
||||
go test ./internal/app ./internal/state
|
||||
go test ./internal/adapters/weatherapi ./internal/adapters/scriptorium
|
||||
go test ./internal/adapters/distributor ./internal/adapters/weatherapi ./internal/adapters/scriptorium
|
||||
go test ./internal/forecast ./internal/report ./internal/briefing ./internal/changes ./internal/promptinput
|
||||
```
|
||||
|
||||
@@ -64,6 +65,8 @@ Run `gofmt -w` on changed Go files before committing.
|
||||
- Use atomic writes for durable JSON artifacts where practical.
|
||||
- Keep report selection and prompt IDs centralized in `internal/report`.
|
||||
- Keep Scriptorium argv construction inside `internal/adapters/scriptorium`.
|
||||
- Keep distributor package types and upload-client construction inside
|
||||
`internal/adapters/distributor`.
|
||||
- Keep Weather API transport and envelope handling inside
|
||||
`internal/adapters/weatherapi`.
|
||||
|
||||
@@ -72,8 +75,10 @@ Run `gofmt -w` on changed Go files before committing.
|
||||
Prefer the Go standard library. Add dependencies only when they materially
|
||||
improve correctness, interoperability, security, or maintainability.
|
||||
|
||||
Current external dependency:
|
||||
Current external dependencies:
|
||||
|
||||
- `gitea.maximumdirect.net/eric/distributor` for distributor source bundle
|
||||
construction and HTTP upload client behavior.
|
||||
- `gopkg.in/yaml.v3` for YAML configuration parsing.
|
||||
|
||||
When adding a dependency:
|
||||
@@ -145,12 +150,14 @@ When an external contract changes, update the matching file under
|
||||
|
||||
## Tests
|
||||
|
||||
Core tests must not require live Weather API or Scriptorium services.
|
||||
Core tests must not require live Weather API, Scriptorium, or distributor
|
||||
services.
|
||||
|
||||
Preferred test patterns:
|
||||
|
||||
- fake command runners for subprocess behavior;
|
||||
- `httptest.Server` for Weather API behavior;
|
||||
- fake distributor upload clients for notification behavior;
|
||||
- filesystem temp directories for state behavior;
|
||||
- deterministic clocks for report periods and RunIDs;
|
||||
- table tests for config validation, CLI parsing, period resolution, and
|
||||
@@ -187,7 +194,8 @@ Update:
|
||||
behavior;
|
||||
- `docs/troubleshooting.md` for recurring operator-facing failure modes;
|
||||
- `docs/internal/` for component contracts and invariants;
|
||||
- `docs/integrations/` for external Weather API or Scriptorium contract changes;
|
||||
- `docs/integrations/` for external Weather API, Scriptorium, or distributor
|
||||
contract changes;
|
||||
- `docs/roadmap/` only for unimplemented or deferred work.
|
||||
|
||||
Non-roadmap docs must describe implemented behavior only.
|
||||
|
||||
@@ -1,336 +1,34 @@
|
||||
# Distributor Integration Roadmap
|
||||
# Distributor Roadmap
|
||||
|
||||
This roadmap describes planned work for adding `distributor` notification
|
||||
support to `weatherreporter`. The feature is not implemented yet, so this plan
|
||||
lives under `docs/roadmap/`; non-roadmap documentation should not describe
|
||||
`notify.distributor` as available until implementation is complete.
|
||||
Current distributor notification behavior is documented outside the roadmap:
|
||||
|
||||
## Purpose
|
||||
- [Configuration reference](../config.md)
|
||||
- [Operations guide](../operations.md)
|
||||
- [Troubleshooting](../troubleshooting.md)
|
||||
- [Distributor adapter internals](../internal/distributor-adapter.md)
|
||||
|
||||
`distributor` collects outputs from producer applications and publishes them
|
||||
through configured downstream pipelines. For `weatherreporter`, the planned
|
||||
integration should upload generated Markdown weather reports after successful
|
||||
report generation, without requiring `distributor` to scan or understand the
|
||||
managed `workspace/` layout.
|
||||
This file tracks future distributor-related work only.
|
||||
|
||||
## Current Repository Facts
|
||||
## Deferred Enhancements
|
||||
|
||||
- `weatherreporter` does not currently implement a notify hook.
|
||||
- Generated Markdown reports are written to the managed report path returned as
|
||||
`ReportResult.ReportPath`.
|
||||
- Optional `--out` and `--out-dir` copies are user-requested extra copies and
|
||||
are not canonical integration inputs.
|
||||
- The managed workspace also contains briefing snapshots, metadata, prompt data
|
||||
packages, and preflight artifacts; these should not be treated as a
|
||||
distributor source tree.
|
||||
- The provided distributor docs recommend `pkg/upload.UploadFiles` for
|
||||
producers that already have generated files, so the initial integration can
|
||||
upload selected report files directly.
|
||||
- Add a supported warning-only notification policy.
|
||||
- Include selected non-report artifacts in uploaded bundles.
|
||||
- Poll distributor run status after upload acceptance.
|
||||
- Persist upload retry state across process restarts.
|
||||
- Add explicit CLI controls for distributor behavior.
|
||||
|
||||
## Decisions Locked
|
||||
## Non-Goals Without A Separate Design
|
||||
|
||||
- Add generic configurable secrets-directory support before the distributor
|
||||
adapter work. This is project infrastructure, not a distributor-only token
|
||||
helper.
|
||||
- When a secrets directory is configured, each regular file directly under that
|
||||
directory maps filename to environment variable name and file contents to
|
||||
environment variable value.
|
||||
- Secret-file values overwrite pre-existing environment variables.
|
||||
- Secrets loading must never log, print, persist, or include secret values in
|
||||
errors, metadata, batch output, examples, or docs.
|
||||
- Add a narrow `internal/adapters/distributor` adapter for the external
|
||||
integration.
|
||||
- Keep `gitea.maximumdirect.net/eric/distributor/pkg/upload` and
|
||||
`gitea.maximumdirect.net/eric/distributor/pkg/bundle` types inside that
|
||||
adapter.
|
||||
- Upload one distributor bundle per successfully generated report.
|
||||
- Include only the generated Markdown report in the initial bundle.
|
||||
- Use the report definition's batch output filename as the default
|
||||
bundle-relative Markdown path, such as `daily.md`, `tomorrow.md`,
|
||||
`three-day.md`, `weekend.md`, or `storm.md`.
|
||||
- Derive the default bundle ID and idempotency key from producer name, location
|
||||
ID, report ID, and RunID.
|
||||
- Treat distributor upload failure as a report failure when notification is
|
||||
enabled; single-report and batch commands should exit nonzero for enabled
|
||||
upload failures.
|
||||
- Keep destination routing, Markdown-to-HTML transformation, public URLs, and
|
||||
nginx directory layout in distributor configuration, not in
|
||||
`weatherreporter`.
|
||||
- Do not make distributor scan the weatherreporter workspace.
|
||||
- Do not move destination routing into weatherreporter.
|
||||
- Do not move Markdown-to-HTML transformation into weatherreporter.
|
||||
- Do not store raw bearer tokens in configuration files.
|
||||
|
||||
## Planned Configuration
|
||||
## Required Constraints For Future Work
|
||||
|
||||
Add future generic secrets configuration:
|
||||
|
||||
```yaml
|
||||
secrets:
|
||||
directory: ""
|
||||
```
|
||||
|
||||
Add future distributor configuration under `notify.distributor`:
|
||||
|
||||
```yaml
|
||||
notify:
|
||||
distributor:
|
||||
enabled: false
|
||||
endpoint: https://distributor.example.com
|
||||
token_env: DISTRIBUTOR_UPLOAD_TOKEN
|
||||
timeout: 30s
|
||||
failure_policy: error
|
||||
bundle_id_template: "weatherreporter.{location_id}.{report_id}.{run_id}"
|
||||
idempotency_key_template: "{bundle_id}"
|
||||
report_path_template: "{batch_output_name}"
|
||||
```
|
||||
|
||||
Planned behavior:
|
||||
|
||||
- `secrets.directory` defaults to an empty string, which disables secrets
|
||||
loading.
|
||||
- When `secrets.directory` is configured, load only regular files directly
|
||||
under that directory.
|
||||
- Each secret filename must be a valid environment variable name matching
|
||||
`[A-Za-z_][A-Za-z0-9_]*`.
|
||||
- Reject missing configured directory, invalid filenames, subdirectories,
|
||||
symlinks, unreadable files, and empty filenames.
|
||||
- Strip one trailing `\n` or `\r\n` from secret file contents for compatibility
|
||||
with common mounted-secret formats. Preserve all other bytes.
|
||||
- `enabled` defaults to `false`.
|
||||
- `endpoint` is required only when distributor notification is enabled.
|
||||
- `token_env` is required only when enabled. Raw bearer tokens should not be
|
||||
stored in config files. The token value is read from normal environment state
|
||||
after secrets-directory loading has been applied.
|
||||
- `timeout` controls the upload call timeout and should default to `30s`.
|
||||
- `failure_policy` should initially support `error`. Reserve `warn` as a
|
||||
future option only if it is implemented and tested.
|
||||
- Template variables should be explicit and validated:
|
||||
`location_id`, `report_id`, `run_id`, `artifact_group`, and
|
||||
`batch_output_name`.
|
||||
- `idempotency_key_template` may reference `{bundle_id}` after the bundle ID is
|
||||
rendered.
|
||||
- `report_path_template` should be user-visible from the first implementation
|
||||
because bundle-relative paths can affect downstream distributor routing.
|
||||
- Rendered template values must produce valid distributor bundle paths: relative
|
||||
slash-separated paths with no empty path, absolute path, backslash, `.`, `..`,
|
||||
empty segment, `manifest.json`, or `.distributor.json`.
|
||||
|
||||
## Recommended Implementation Stages
|
||||
|
||||
### Stage 1: Secrets Directory Support
|
||||
|
||||
Goal: add generic file-backed environment secret loading before distributor
|
||||
notification depends on environment tokens.
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Add a generic `secrets.directory` config field under `internal/config`.
|
||||
- Default the directory to an empty string so secrets loading is disabled unless
|
||||
explicitly configured.
|
||||
- Load secrets after config file parsing and before validation that needs
|
||||
environment-backed values.
|
||||
- Map each regular file directly under the configured directory to an
|
||||
environment variable: filename becomes the variable name and contents become
|
||||
the value.
|
||||
- Use secret-file values to overwrite pre-existing environment variables.
|
||||
- Strip one trailing `\n` or `\r\n`; preserve all other bytes.
|
||||
- Reject missing directories, invalid environment-variable filenames,
|
||||
subdirectories, symlinks, unreadable files, and empty filenames.
|
||||
- Keep secret values out of errors, logs, CLI output, metadata, and examples.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Empty `secrets.directory` performs no environment changes.
|
||||
- Configured secrets directory values are available through normal environment
|
||||
lookup after config loading.
|
||||
- Secret files overwrite existing environment variables with the same name.
|
||||
- Invalid directory entries fail with path/name context but without secret
|
||||
values.
|
||||
- Existing config loading behavior remains unchanged when secrets are disabled.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Config defaults leave secrets loading disabled.
|
||||
- A configured missing secrets directory fails.
|
||||
- Regular files set environment variables and overwrite existing values.
|
||||
- Invalid filenames, subdirectories, symlinks, and unreadable files fail.
|
||||
- One trailing newline or CRLF is stripped; other content is preserved.
|
||||
- No secret values appear in returned errors.
|
||||
|
||||
### Stage 2: Config And Validation
|
||||
|
||||
Goal: add disabled-by-default distributor notification configuration.
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Add notify/distributor config structs under `internal/config`.
|
||||
- Add defaults for disabled notification, timeout, failure policy, bundle ID
|
||||
template, idempotency key template, and report path template.
|
||||
- Validate enabled config: endpoint must be an absolute URL, `token_env` must be
|
||||
non-empty, timeout must be positive, failure policy must be supported, and
|
||||
templates must use only known variables.
|
||||
- Keep `token_env` as the configured token selector. Do not add raw token config.
|
||||
- Update maintained examples and config documentation only after code support is
|
||||
implemented.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Disabled distributor notification requires no endpoint or token env.
|
||||
- Enabled distributor notification rejects invalid endpoint, missing token env,
|
||||
nonpositive timeout, unsupported failure policy, unknown template variables,
|
||||
and invalid rendered report bundle paths.
|
||||
- Existing config loading behavior and precedence remain unchanged.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Config defaults include disabled distributor notification.
|
||||
- Example configs load successfully.
|
||||
- Enabled config validation covers required fields and invalid template cases.
|
||||
- A distributor token supplied through `secrets.directory` is visible through
|
||||
the configured `token_env` name after config loading.
|
||||
|
||||
### Stage 3: Distributor Adapter
|
||||
|
||||
Goal: isolate distributor package usage behind an internal adapter.
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Add `internal/adapters/distributor` with package-owned request and result
|
||||
types.
|
||||
- Read the bearer token from the configured environment variable after
|
||||
secrets-directory loading has populated environment state.
|
||||
- Use distributor `UploadFiles` with one file:
|
||||
the managed Markdown report as the source path and the rendered
|
||||
`report_path_template` as the bundle path.
|
||||
- Return upload result information that app orchestration can record or expose
|
||||
without leaking distributor dependency types.
|
||||
- Wrap upload errors with endpoint, bundle ID, and report path context while
|
||||
avoiding token exposure.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- No distributor dependency types leak outside `internal/adapters/distributor`.
|
||||
- Missing source report path, missing token, invalid bundle path, and upload
|
||||
failure return actionable errors.
|
||||
- Idempotency conflict errors are preserved or wrapped clearly enough for
|
||||
troubleshooting.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Adapter request construction uses the configured endpoint, token, bundle ID,
|
||||
idempotency key, source report path, and bundle-relative path.
|
||||
- Token values do not appear in returned errors.
|
||||
- Upload conflict and generic upload failure produce useful wrapped errors.
|
||||
- Tokens supplied through the secrets directory are accepted via `token_env`.
|
||||
|
||||
### Stage 4: App Notify Hook
|
||||
|
||||
Goal: upload successful generated reports through the configured notifier.
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Add an app-owned notifier interface so app tests can use fakes.
|
||||
- Construct distributor notification requests from the successful
|
||||
`ReportResult`, resolved report definition, configured location, and
|
||||
notification config.
|
||||
- Invoke notification only after Scriptorium report generation succeeds and
|
||||
final metadata has been saved.
|
||||
- Do not notify after Weather API, briefing, prompt input, render preflight, or
|
||||
Scriptorium run failure.
|
||||
- In batch mode, notify each successful report independently. If notification
|
||||
fails and distributor notification is enabled, mark that report failed and
|
||||
make the aggregate batch result nonzero.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Disabled notification is a no-op.
|
||||
- Single-report generation returns an error when enabled upload fails.
|
||||
- Batch generation continues independent reports, records upload failures per
|
||||
report, and exits nonzero when any enabled notification fails.
|
||||
- Existing report artifact paths, optional output copies, and Scriptorium
|
||||
behavior remain unchanged.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- No notifier call occurs before successful report generation.
|
||||
- Successful notifier call receives the managed report path, not `--out` or
|
||||
`--out-dir` copy paths.
|
||||
- Enabled notifier failure affects single-report and batch command outcomes.
|
||||
- Batch still continues later reports after one notification failure.
|
||||
|
||||
### Stage 5: CLI And Output Behavior
|
||||
|
||||
Goal: preserve CLI syntax while surfacing notification outcome where useful.
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Do not add distributor-specific CLI flags in the first implementation; use
|
||||
configuration only.
|
||||
- Preserve existing command names, flags, help text shape, workspace paths, and
|
||||
generated Markdown outputs.
|
||||
- Extend batch JSON report items only if the implementation records useful
|
||||
notification status, accepted distributor run ID, or notification error.
|
||||
- Keep stderr status lines concise and avoid exposing secrets.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- `go run ./cmd/weatherreporter --help` remains accurate.
|
||||
- Existing generate and run command syntax remains stable.
|
||||
- Batch summaries clearly indicate notification-caused report failures if
|
||||
notification status is added.
|
||||
|
||||
Suggested tests:
|
||||
|
||||
- Existing CLI parser tests continue to pass.
|
||||
- Batch JSON includes notification fields only when implemented and documented.
|
||||
- Secret-like token values never appear in CLI output.
|
||||
|
||||
### Stage 6: Documentation
|
||||
|
||||
Goal: document implemented distributor behavior after the feature exists.
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Update `docs/config.md` with `secrets.directory`,
|
||||
`notify.distributor` fields, and defaults.
|
||||
- Update `docs/operations.md` with notification timing, failure behavior, and
|
||||
the fact that managed report paths are uploaded.
|
||||
- Update `docs/troubleshooting.md` with invalid secrets directory, missing
|
||||
token, upload conflict, upload rejection, and distributor unavailable cases.
|
||||
- Update `docs/internal/app-orchestration.md` to include notify ordering.
|
||||
- Add `docs/internal/distributor-adapter.md` for the adapter contract.
|
||||
- Keep distributor API details in `docs/integrations/distributor/`; link there
|
||||
instead of duplicating the full upstream contract.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Non-roadmap docs describe only implemented notification behavior.
|
||||
- Config examples include no raw tokens.
|
||||
- Secret handling documentation describes mechanisms, not secret values.
|
||||
- Documentation clearly distinguishes distributor destination routing from
|
||||
weatherreporter upload responsibilities.
|
||||
|
||||
### Stage 7: Final Validation
|
||||
|
||||
Goal: verify the feature without requiring live distributor service access.
|
||||
|
||||
Validation commands:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Additional validation:
|
||||
|
||||
- Run focused config, adapter, app, and CLI tests.
|
||||
- Confirm maintained example configs load.
|
||||
- Search for accidental raw-token config examples.
|
||||
- Confirm `go.mod` includes distributor only after implementation requires it.
|
||||
- Confirm no test failures leak secret values in error output.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No questions block this roadmap. The initial implementation should use the
|
||||
locked defaults above and expose configurable templates for bundle identity and
|
||||
bundle-relative report path. Secrets-directory support is generic project
|
||||
infrastructure and should remain useful for future integrations beyond
|
||||
distributor.
|
||||
- Distributor package types stay inside `internal/adapters/distributor`.
|
||||
- Weatherreporter submits explicit source bundles built from generated files.
|
||||
- Optional `--out` and `--out-dir` copies remain operator conveniences, not
|
||||
canonical upload sources.
|
||||
- Secret values stay out of errors, logs, CLI output, metadata, examples, and
|
||||
documentation.
|
||||
|
||||
@@ -49,6 +49,26 @@ These ideas are not current behavior:
|
||||
Each item needs its own design note before implementation. Non-roadmap docs
|
||||
must not describe these as available behavior.
|
||||
|
||||
## Deferred: Distributor Notification Enhancements
|
||||
|
||||
Distributor notification currently uploads one managed Markdown report per
|
||||
successful generated report through the configured HTTP upload endpoint.
|
||||
|
||||
These enhancements are not current behavior:
|
||||
|
||||
- `failure_policy: warn`;
|
||||
- uploading metadata, briefing snapshots, data packages, or preflight artifacts;
|
||||
- polling distributor status after upload acceptance;
|
||||
- durable upload retry queues;
|
||||
- distributor-specific CLI flags;
|
||||
- making distributor scan the weatherreporter workspace;
|
||||
- handling destination routing, Markdown-to-HTML transformation, public URLs,
|
||||
or nginx layout inside weatherreporter.
|
||||
|
||||
Any distributor enhancement should preserve the existing adapter boundary:
|
||||
weatherreporter selects explicit generated files and submits source bundles,
|
||||
while distributor owns destination routing and publication behavior.
|
||||
|
||||
## Deferred: Cleanup Refactors
|
||||
|
||||
The initial cleanup pass intentionally left these refactors out because the
|
||||
|
||||
@@ -1,297 +1,36 @@
|
||||
# Distributor Feature Implementation Roadmap
|
||||
# Distributor Follow-Up Roadmap
|
||||
|
||||
This roadmap turns `docs/roadmap/distributor.md` into staged implementation
|
||||
work for a future coding agent. It is a planning document only. The feature is
|
||||
not implemented until the stages below are completed and the implemented docs
|
||||
are updated.
|
||||
Current distributor notification behavior is documented in:
|
||||
|
||||
## Purpose
|
||||
- [Configuration reference](../config.md)
|
||||
- [Operations guide](../operations.md)
|
||||
- [Troubleshooting](../troubleshooting.md)
|
||||
- [App orchestration internals](../internal/app-orchestration.md)
|
||||
- [Distributor adapter internals](../internal/distributor-adapter.md)
|
||||
- [Distributor integration contracts](../integrations/distributor/api.md)
|
||||
|
||||
Implement generic file-backed environment secrets and distributor notification
|
||||
support while preserving current CLI syntax, managed artifact paths, RunIDs,
|
||||
and report output behavior. Distributor package types must stay behind a narrow
|
||||
adapter boundary, and distributor should receive explicitly selected generated
|
||||
Markdown files rather than scanning `workspace/`.
|
||||
This roadmap tracks only future work that is not implemented.
|
||||
|
||||
## Locked Decisions
|
||||
## Deferred Enhancements
|
||||
|
||||
- Implement generic `secrets.directory` support before distributor notification.
|
||||
- Secret files overwrite pre-existing environment variables.
|
||||
- Never expose secret values in errors, logs, CLI output, metadata, docs, or
|
||||
examples.
|
||||
- Add `notify.distributor` config; do not add raw token config.
|
||||
- Keep public CLI commands and flags unchanged.
|
||||
- Upload one distributor bundle per successfully generated report.
|
||||
- Initial distributor bundles include only the managed Markdown report at
|
||||
`ReportResult.ReportPath`.
|
||||
- The default bundle-relative Markdown path uses the report definition's
|
||||
`BatchOutputName`.
|
||||
- Default bundle ID and idempotency key derive from producer, location ID,
|
||||
report ID, and RunID.
|
||||
- Enabled distributor upload failure is a report failure and should produce a
|
||||
nonzero command exit.
|
||||
- Support a non-failing distributor notification policy such as
|
||||
`failure_policy: warn`.
|
||||
- Upload additional generated artifacts, such as metadata, briefing snapshots,
|
||||
data packages, or preflight output.
|
||||
- Poll distributor status after upload acceptance and expose downstream
|
||||
publication failures.
|
||||
- Add durable retry queues for upload failures.
|
||||
- Add distributor-specific CLI controls if operators need per-run behavior that
|
||||
configuration cannot cover cleanly.
|
||||
|
||||
## Stage 1: Secrets Directory Support
|
||||
## Boundaries To Preserve
|
||||
|
||||
Goal: implement generic file-backed environment secret loading in
|
||||
`internal/config`.
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Add `secrets.directory` config with a built-in default of `""`, meaning
|
||||
disabled.
|
||||
- Load secrets after config file parsing and CLI overrides, before validation
|
||||
completes.
|
||||
- Read only regular files directly under the configured directory.
|
||||
- Map each file basename to an environment variable name and file contents to
|
||||
the value.
|
||||
- Validate names with `[A-Za-z_][A-Za-z0-9_]*`.
|
||||
- Reject missing configured directory, invalid filenames, subdirectories,
|
||||
symlinks, unreadable files, and empty filenames.
|
||||
- Strip one trailing `\n` or `\r\n`; preserve all other bytes.
|
||||
- Keep error messages actionable but do not include secret values.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Empty `secrets.directory` performs no environment changes.
|
||||
- Valid secret files set environment variables after config loading.
|
||||
- Secret file values overwrite existing environment variables.
|
||||
- Invalid entries fail before application workflows run.
|
||||
- Existing config loading behavior remains unchanged when secrets are disabled.
|
||||
|
||||
Tests:
|
||||
|
||||
- Defaults leave secrets loading disabled.
|
||||
- Valid file sets an environment variable.
|
||||
- File value overwrites an existing environment variable.
|
||||
- Missing directory, invalid filename, subdirectory, symlink, and unreadable
|
||||
file cases fail.
|
||||
- Newline and CRLF trimming works exactly once.
|
||||
- Secret values are absent from error strings.
|
||||
|
||||
## Stage 2: Notify Configuration And Templates
|
||||
|
||||
Goal: add disabled-by-default `notify.distributor` config and template
|
||||
validation.
|
||||
|
||||
Planned config:
|
||||
|
||||
```yaml
|
||||
notify:
|
||||
distributor:
|
||||
enabled: false
|
||||
endpoint: https://distributor.example.com
|
||||
token_env: DISTRIBUTOR_UPLOAD_TOKEN
|
||||
timeout: 30s
|
||||
failure_policy: error
|
||||
bundle_id_template: "weatherreporter.{location_id}.{report_id}.{run_id}"
|
||||
idempotency_key_template: "{bundle_id}"
|
||||
report_path_template: "{batch_output_name}"
|
||||
```
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Add notify/distributor config structs, defaults, YAML loading, and validation
|
||||
under `internal/config`.
|
||||
- When disabled, omit endpoint and token requirements.
|
||||
- When enabled, validate absolute endpoint URL, non-empty `token_env`, positive
|
||||
timeout, `failure_policy=error`, known template variables, and valid rendered
|
||||
report bundle paths.
|
||||
- Supported template variables are `location_id`, `report_id`, `run_id`,
|
||||
`artifact_group`, and `batch_output_name`; `idempotency_key_template` may
|
||||
also reference `{bundle_id}`.
|
||||
- Do not add distributor-specific CLI flags.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Defaults include disabled distributor notification.
|
||||
- Enabled config rejects invalid endpoint, token env, timeout, failure policy,
|
||||
templates, and rendered bundle paths.
|
||||
- Rendered bundle paths reject absolute paths, backslashes, `.`, `..`, empty
|
||||
segments, `manifest.json`, and `.distributor.json`.
|
||||
- Tokens supplied by `secrets.directory` are visible through `token_env` after
|
||||
config loading.
|
||||
|
||||
Tests:
|
||||
|
||||
- Config defaults and examples load.
|
||||
- Disabled distributor config accepts omitted distributor fields.
|
||||
- Enabled validation covers each invalid field.
|
||||
- Template rendering and validation are covered with table tests.
|
||||
|
||||
## Stage 3: Distributor Adapter
|
||||
|
||||
Goal: isolate distributor upload behavior in `internal/adapters/distributor`.
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Add weatherreporter-owned request and result types.
|
||||
- Keep `gitea.maximumdirect.net/eric/distributor/pkg/upload` and
|
||||
`gitea.maximumdirect.net/eric/distributor/pkg/bundle` types inside the
|
||||
adapter.
|
||||
- Read the bearer token from `token_env` after config/secrets loading.
|
||||
- Use `UploadFiles` with exactly one file: source path is the managed Markdown
|
||||
report path and bundle path is the rendered `report_path_template`.
|
||||
- Return the accepted distributor run ID when available.
|
||||
- Wrap endpoint, bundle ID, idempotency key, source path, and bundle path
|
||||
context into errors without exposing token values.
|
||||
- Preserve idempotency conflict diagnosis.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- No distributor dependency types leak outside the adapter.
|
||||
- Missing token, source path, bundle path, or upload client inputs fail with
|
||||
actionable errors.
|
||||
- Upload success returns a weatherreporter-owned result.
|
||||
- Upload failure errors do not include token values.
|
||||
|
||||
Tests:
|
||||
|
||||
- Fake upload client receives expected endpoint, token, bundle ID, idempotency
|
||||
key, source path, and bundle path.
|
||||
- Missing token/source path/bundle path errors are actionable.
|
||||
- Token values are not present in errors.
|
||||
- Upload conflict and generic failure are wrapped clearly.
|
||||
|
||||
## Stage 4: App Notification Hook
|
||||
|
||||
Goal: add app-level notification orchestration after successful report
|
||||
generation.
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Add an app-owned notifier interface for tests.
|
||||
- Default to no-op notification when distributor notification is disabled.
|
||||
- Notify only after successful `scriptorium run` and final metadata save.
|
||||
- Use `ReportResult.ReportPath` as the distributor source file.
|
||||
- Never use `--out` or `--out-dir` copies as distributor source files.
|
||||
- Do not notify after Weather API, briefing, prompt input, render preflight, or
|
||||
Scriptorium run failure.
|
||||
- Single-report generation should return an error when enabled notification
|
||||
fails.
|
||||
- Batch generation should continue independent reports; notification failure
|
||||
marks that report failed and makes the aggregate batch nonzero.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Disabled notification leaves current generation behavior unchanged.
|
||||
- Successful notification is recorded in the app result when useful.
|
||||
- Enabled notification failure fails a single-report command.
|
||||
- Batch results distinguish generation failure from notification failure enough
|
||||
for operators to diagnose the failed report.
|
||||
- Existing artifact paths, output copies, and Scriptorium behavior remain
|
||||
unchanged.
|
||||
|
||||
Tests:
|
||||
|
||||
- No notifier call occurs on pre-generation failures.
|
||||
- Notifier receives the managed report path.
|
||||
- Enabled notifier success and failure are covered.
|
||||
- Batch continues after one notification failure and reports aggregate failure.
|
||||
|
||||
## Stage 5: CLI Output And Batch Reporting
|
||||
|
||||
Goal: keep public CLI syntax stable while surfacing notification outcome if the
|
||||
app records it.
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Do not add distributor-specific CLI flags.
|
||||
- Preserve current command names, help shape, generated paths, `--out`, and
|
||||
`--out-dir`.
|
||||
- If batch JSON is extended, use explicit fields such as
|
||||
`notificationStatus`, `notificationRunId`, and `notificationError`.
|
||||
- Keep stderr status lines concise and secret-free.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- `weatherreporter --help` remains accurate.
|
||||
- Existing CLI parser behavior remains stable.
|
||||
- Batch JSON includes notification details only when implemented and
|
||||
documented.
|
||||
- CLI stdout and stderr do not include token values.
|
||||
|
||||
Tests:
|
||||
|
||||
- Existing CLI parser tests pass, except intentional JSON additions.
|
||||
- Batch output covers notification success and failure.
|
||||
- Secret-like token values are absent from CLI output.
|
||||
|
||||
## Stage 6: Documentation
|
||||
|
||||
Goal: update implemented documentation after the feature exists.
|
||||
|
||||
Implementation guidance:
|
||||
|
||||
- Update `docs/config.md` with `secrets.directory`, `notify.distributor`,
|
||||
defaults, and secret handling.
|
||||
- Update `docs/operations.md` with notification timing, failure behavior, and
|
||||
managed report upload source.
|
||||
- Update `docs/troubleshooting.md` with invalid secrets directory, missing
|
||||
token, upload conflict, upload rejection, and distributor unavailable cases.
|
||||
- Update `docs/internal/app-orchestration.md` to include notify ordering.
|
||||
- Add `docs/internal/distributor-adapter.md` for adapter inputs, outputs,
|
||||
boundaries, config fields, failure behavior, tests, and invariants.
|
||||
- Keep full upstream distributor contract details in
|
||||
`docs/integrations/distributor/`.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Non-roadmap docs describe only implemented behavior.
|
||||
- Config examples load and contain no raw secrets.
|
||||
- Secret handling docs describe mechanisms, not secret values.
|
||||
- Docs distinguish weatherreporter upload responsibility from distributor
|
||||
destination routing and Markdown-to-HTML transformation.
|
||||
|
||||
Tests and checks:
|
||||
|
||||
- Run config example loading tests.
|
||||
- Search docs and examples for accidental raw tokens.
|
||||
- Run `git diff --check`.
|
||||
|
||||
## Stage 7: Final Validation
|
||||
|
||||
Goal: verify behavior without a live distributor service.
|
||||
|
||||
Validation commands:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
go test ./internal/config ./internal/app ./internal/cli ./internal/adapters/distributor
|
||||
```
|
||||
|
||||
Manual validation:
|
||||
|
||||
- No token values appear in errors, CLI output, docs, examples, metadata, or
|
||||
batch JSON.
|
||||
- `go.mod` includes distributor only after adapter implementation requires it.
|
||||
- Existing report paths, RunIDs, and `--out` / `--out-dir` behavior are
|
||||
unchanged.
|
||||
- Distributor upload tests use fakes or local test doubles, not a live service.
|
||||
|
||||
## Deferred Work
|
||||
|
||||
These are out of scope for the initial implementation:
|
||||
|
||||
- `failure_policy: warn`.
|
||||
- Uploading metadata, briefing, data package, or preflight artifacts.
|
||||
- Distributor status polling after `202 Accepted`.
|
||||
- Durable upload retry queues.
|
||||
- Distributor-specific CLI flags.
|
||||
- Making distributor scan `workspace/`.
|
||||
- Destination routing, Markdown-to-HTML transformation, public URLs, or nginx
|
||||
layout in `weatherreporter`.
|
||||
|
||||
## Global Validation Checklist
|
||||
|
||||
- Preserve current public CLI syntax.
|
||||
- Preserve managed workspace artifact paths.
|
||||
- Keep public CLI syntax stable unless a separate CLI design changes it.
|
||||
- Keep managed workspace artifact paths and RunIDs stable.
|
||||
- Keep distributor dependency types behind `internal/adapters/distributor`.
|
||||
- Keep secrets out of user-facing and persisted output.
|
||||
- Keep examples valid and secret-free.
|
||||
- Update implemented docs in the same change as implementation.
|
||||
- Run `go test ./...`, `go run ./cmd/weatherreporter --help`, and
|
||||
`git diff --check` before finishing implementation.
|
||||
|
||||
- Keep raw token values out of errors, CLI output, metadata, docs, examples,
|
||||
and persisted artifacts.
|
||||
- Upload explicitly selected generated files; do not make distributor scan the
|
||||
weatherreporter workspace.
|
||||
- Leave destination routing, Markdown-to-HTML transformation, public URLs, and
|
||||
nginx layout to distributor.
|
||||
|
||||
@@ -199,6 +199,101 @@ metadata, sources, briefing, and data package for that RunID.
|
||||
|
||||
Relevant docs: [CLI reference](cli.md), [Operations guide](operations.md).
|
||||
|
||||
## Invalid Secrets Directory
|
||||
|
||||
Symptom: config loading fails with `read secrets directory`, `secret file`, or
|
||||
environment variable name context.
|
||||
|
||||
Likely cause: `secrets.directory` points to a missing directory or contains an
|
||||
invalid entry. Secret entries must be regular files directly under the
|
||||
configured directory, and file basenames must match
|
||||
`[A-Za-z_][A-Za-z0-9_]*`.
|
||||
|
||||
Diagnostic: list the configured directory and inspect entry names and file
|
||||
types. Do not print secret file contents.
|
||||
|
||||
Safe fix: create the directory, remove subdirectories or symlinks, fix invalid
|
||||
filenames, and ensure the weatherreporter process can read each secret file.
|
||||
|
||||
Relevant docs: [Configuration reference](config.md).
|
||||
|
||||
## Distributor Token Is Missing
|
||||
|
||||
Symptom: notification fails with a message that the distributor token
|
||||
environment variable is not set.
|
||||
|
||||
Likely cause: `notify.distributor.enabled` is true, but the environment
|
||||
variable named by `notify.distributor.token_env` was not populated directly or
|
||||
through `secrets.directory`.
|
||||
|
||||
Diagnostic: check `notify.distributor.token_env`, then verify a matching secret
|
||||
file exists under `secrets.directory` or that the process environment includes
|
||||
the variable. Do not print the token value.
|
||||
|
||||
Safe fix: create a readable secret file whose basename matches `token_env`, or
|
||||
set the environment variable through the service manager.
|
||||
|
||||
Relevant docs: [Configuration reference](config.md),
|
||||
[Operations guide](operations.md).
|
||||
|
||||
## Distributor Upload Conflict
|
||||
|
||||
Symptom: notification fails with idempotency conflict context.
|
||||
|
||||
Likely cause: the same idempotency key was reused for different bundle content.
|
||||
By default the key derives from bundle ID, which includes location ID, report
|
||||
ID, and RunID.
|
||||
|
||||
Diagnostic: inspect the failed batch JSON or stderr line for bundle and
|
||||
idempotency context. Compare the configured templates with the report RunID and
|
||||
report path.
|
||||
|
||||
Safe fix: keep idempotency templates stable for retries of the same generated
|
||||
report, but do not reuse the same rendered key for different generated report
|
||||
content.
|
||||
|
||||
Relevant docs: [Operations guide](operations.md),
|
||||
[Distributor adapter internals](internal/distributor-adapter.md).
|
||||
|
||||
## Distributor Upload Rejected
|
||||
|
||||
Symptom: notification fails with distributor upload rejection, HTTP status, or
|
||||
bundle validation context.
|
||||
|
||||
Likely cause: the distributor endpoint rejected the token, bundle ID,
|
||||
idempotency key, source file, or bundle path.
|
||||
|
||||
Diagnostic: inspect stdout JSON or stderr status lines for
|
||||
`notificationError`. Confirm `notify.distributor.endpoint`,
|
||||
`notify.distributor.report_path_template`, and token configuration. Token values
|
||||
are redacted from weatherreporter errors.
|
||||
|
||||
Safe fix: fix the endpoint, token, templates, or distributor-side upload
|
||||
configuration. The weatherreporter upload source is the managed Markdown report,
|
||||
not `--out` or `--out-dir` copies.
|
||||
|
||||
Relevant docs: [Configuration reference](config.md),
|
||||
[Operations guide](operations.md),
|
||||
[Distributor adapter internals](internal/distributor-adapter.md).
|
||||
|
||||
## Distributor Unavailable
|
||||
|
||||
Symptom: notification fails with network, timeout, or service unavailable
|
||||
context.
|
||||
|
||||
Likely cause: the configured distributor endpoint is unreachable, slow, or
|
||||
temporarily unavailable.
|
||||
|
||||
Diagnostic: check network access from the weatherreporter host to
|
||||
`notify.distributor.endpoint`. For batch runs, inspect which reports have
|
||||
`notificationStatus: "failed"`.
|
||||
|
||||
Safe fix: restore distributor service availability and rerun the affected
|
||||
report or batch. Stable idempotency keys make retrying the same generated report
|
||||
safe unless the distributor reports a conflict.
|
||||
|
||||
Relevant docs: [Operations guide](operations.md).
|
||||
|
||||
## Unknown RunID
|
||||
|
||||
Symptom: an inspect command fails with `metadata for run id ... was not found`.
|
||||
|
||||
@@ -11,6 +11,20 @@ location:
|
||||
name: Brentwood
|
||||
region: St. Louis Metro
|
||||
|
||||
secrets:
|
||||
directory: ""
|
||||
|
||||
notify:
|
||||
distributor:
|
||||
enabled: false
|
||||
endpoint: https://distributor.example.com
|
||||
token_env: DISTRIBUTOR_UPLOAD_TOKEN
|
||||
timeout: 30s
|
||||
failure_policy: error
|
||||
bundle_id_template: "weatherreporter.{location_id}.{report_id}.{run_id}"
|
||||
idempotency_key_template: "{bundle_id}"
|
||||
report_path_template: "{batch_output_name}"
|
||||
|
||||
missing_source:
|
||||
default: warn
|
||||
sources:
|
||||
|
||||
2
go.mod
2
go.mod
@@ -3,3 +3,5 @@ module gitea.maximumdirect.net/eric/weatherreporter
|
||||
go 1.26
|
||||
|
||||
require gopkg.in/yaml.v3 v3.0.1
|
||||
|
||||
require gitea.maximumdirect.net/eric/distributor v0.4.0
|
||||
|
||||
48
go.sum
48
go.sum
@@ -1,3 +1,51 @@
|
||||
gitea.maximumdirect.net/eric/distributor v0.4.0 h1:SRrTFjVLMv4wFZmMLwnaYtJaQlQ/wsB4OzcaSEEb524=
|
||||
gitea.maximumdirect.net/eric/distributor v0.4.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11/go.mod h1:dnakxebH6UwFvcvujL0LVggYQ8nEvBGjU4G/V79Nv94=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.20 h1:8VMDnWc/kEzxsI/1ngGM9mG81a8IGmIHD8KLcYGwagc=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.20/go.mod h1:PuwEpciweIXGULWeOeSTXtSbH4CW9mWdWrhdCKQI1sM=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.19 h1:yuFzSV1U0aRNYCQGVaTY2zW2M/L93pYHnXnrJUphYhU=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.19/go.mod h1:7y63L1kGzeoDlJaQ3Z578KrnmfBut96JjvJUzGwR+YE=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.25 h1:0w6dCiO8iez+YKwRhRBlL1CH/E3GTfdkuzrwj1by8vo=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.25/go.mod h1:9FDWUothyr5RCRAHc45XOiVCzUR8n/IhCYX+uVqw6vk=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25 h1:Uii3frf9ztec/ABM2/FSH9/z7PLzxfpG8h4RpkUFflQ=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25/go.mod h1:G6kntsA2GorAxDPbap6xgB2F+amSLUF8GJTi7PUoX44=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25 h1:r1+/l6m+WaUJF9HISEsNOLHSNj5EXYQxK8VX6Cz9NlA=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25/go.mod h1:cKf+D+NMDK1LndD7BowHbBZPgR9V0/5HubH0PFWvA+c=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26 h1:A1PmWU2zfkIm9EyFlJncFXL4W4phML+h8KjltUsCvNQ=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26/go.mod h1:dY4MRzXEizrD4hqtpKvWVGPX7QleSGGVY+EBolo1RmM=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10 h1:d5/908OJ4bXg8lyjeMPvXetEKqoDoLi5Owy1zNue3yg=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10/go.mod h1:a57l7Hwh+FWI+we50g5NPJHYUKeJKfXbc4w8SyXu8Ig=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18 h1:W/EyPFl9A5rXrtoilfwHYEvzHER+K4SpBPtMXi24Mos=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18/go.mod h1:UG50K+pvd/uy6xExbobg0rjqFBFZe6I3l75EPDZw4tg=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25 h1:dD3dhHNglpd98gs72my22Ndqi1hqQGllFFg1F+twfxg=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25/go.mod h1:0yAbjPfd64gG7mj85RW+fMEYdfBgCRZw8g/oWcL1pjc=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25 h1:2pQEbwf+/6EDbiit/GcBE2K4IUpMZymaA0kOz3xK978=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25/go.mod h1:KvT6NCcQ0EZ+ZkVRrlBMt04Po3ok23YELEp7WimhLhM=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2 h1:ie4ElCmUKS26pzrZcIk/lmt4yWjAqLLcawstyQCh298=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2/go.mod h1:zjsomFeX5duj+4PlMB+o4JoWTIx+G0XMyzjYrUbQkN0=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.1.1 h1:1VwbP3qMNfxUDEXWki4rCE5iA+44VA1lokTz9HasGzw=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.1.1/go.mod h1:vUtyoSj0OPji3kjIVSc/GlKuWEiL33f/WFxl6dmpy/A=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.19 h1:N6pIsdFOW1Kd9S4KyFKXdGRBojPPxkP32+uHFWLv4Hc=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.19/go.mod h1:3gt5WJArFooNmyLONS+h/R4J+o86II8du38IgCwj9dE=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.2 h1:hc+lBYiiTr8Zk4MTzIsQ92MeDWCIDvWGmzKUWOaBcOg=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.2/go.mod h1:hU6fqB3OJA6/ePheD47LQnxvjYk6br6PtQxs+Q9ojvk=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.42.3 h1:ErklX/7uhSbkAAeyQD/Y1OoQ9hO3SJXQNEgksORW3Js=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.42.3/go.mod h1:ULe4HCzfKPiR6R3HEurE3b1upEkuk8AkMrOKtaOxKO8=
|
||||
github.com/aws/smithy-go v1.26.0 h1:9ouqbi+NyKP7fV3Te7UElCwdAb6Y8uk7LGwPE5tVe/s=
|
||||
github.com/aws/smithy-go v1.26.0/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
|
||||
github.com/kr/fs v0.1.0 h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8=
|
||||
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
|
||||
github.com/pkg/sftp v1.13.10 h1:+5FbKNTe5Z9aspU88DPIKJ9z2KZoaGCu6Sr6kKR/5mU=
|
||||
github.com/pkg/sftp v1.13.10/go.mod h1:bJ1a7uDhrX/4OII+agvy28lzRvQrmIQuaHrcI1HbeGA=
|
||||
github.com/yuin/goldmark v1.8.2 h1:kEGpgqJXdgbkhcOgBxkC0X0PmoPG1ZyoZ117rDVp4zE=
|
||||
github.com/yuin/goldmark v1.8.2/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
|
||||
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
|
||||
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
|
||||
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
|
||||
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
|
||||
229
internal/adapters/distributor/client.go
Normal file
229
internal/adapters/distributor/client.go
Normal file
@@ -0,0 +1,229 @@
|
||||
// Package distributor adapts weatherreporter report artifacts to distributor uploads.
|
||||
package distributor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
distributorbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
|
||||
distributorupload "gitea.maximumdirect.net/eric/distributor/pkg/upload"
|
||||
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/config"
|
||||
)
|
||||
|
||||
type Client struct {
|
||||
Endpoint string
|
||||
TokenEnv string
|
||||
Timeout time.Duration
|
||||
newUploadClient uploadClientFactory
|
||||
}
|
||||
|
||||
type UploadRequest struct {
|
||||
BundleID string
|
||||
IdempotencyKey string
|
||||
SourcePath string
|
||||
BundlePath string
|
||||
}
|
||||
|
||||
type UploadResult struct {
|
||||
RunID string
|
||||
Status string
|
||||
}
|
||||
|
||||
type IdempotencyConflictError struct {
|
||||
Err error
|
||||
}
|
||||
|
||||
func (e *IdempotencyConflictError) Error() string {
|
||||
if e == nil || e.Err == nil {
|
||||
return "distributor idempotency conflict"
|
||||
}
|
||||
return e.Err.Error()
|
||||
}
|
||||
|
||||
func (e *IdempotencyConflictError) Unwrap() error {
|
||||
if e == nil {
|
||||
return nil
|
||||
}
|
||||
return e.Err
|
||||
}
|
||||
|
||||
type uploadClientFactory func(endpoint, token string, timeout time.Duration) (uploadClient, error)
|
||||
|
||||
type uploadClient interface {
|
||||
UploadFiles(ctx context.Context, opts uploadFilesOptions) (uploadFilesResult, error)
|
||||
}
|
||||
|
||||
type uploadFilesOptions struct {
|
||||
BundleID string
|
||||
IdempotencyKey string
|
||||
SourcePath string
|
||||
BundlePath string
|
||||
}
|
||||
|
||||
type uploadFilesResult struct {
|
||||
RunID string
|
||||
Status string
|
||||
}
|
||||
|
||||
func New(cfg config.DistributorNotifyConfig) *Client {
|
||||
return newClient(cfg, newDistributorUploadClient)
|
||||
}
|
||||
|
||||
func newClient(cfg config.DistributorNotifyConfig, factory uploadClientFactory) *Client {
|
||||
if factory == nil {
|
||||
factory = newDistributorUploadClient
|
||||
}
|
||||
return &Client{
|
||||
Endpoint: cfg.Endpoint,
|
||||
TokenEnv: cfg.TokenEnv,
|
||||
Timeout: cfg.Timeout,
|
||||
newUploadClient: factory,
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Client) Upload(ctx context.Context, req UploadRequest) (UploadResult, error) {
|
||||
if c == nil {
|
||||
return UploadResult{}, fmt.Errorf("distributor client is nil")
|
||||
}
|
||||
if c.Endpoint == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor endpoint is required")
|
||||
}
|
||||
if c.TokenEnv == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor token environment variable is required")
|
||||
}
|
||||
if req.BundleID == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor bundle id is required")
|
||||
}
|
||||
if req.IdempotencyKey == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor idempotency key is required for bundle %q", req.BundleID)
|
||||
}
|
||||
if req.SourcePath == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor source path is required for bundle %q", req.BundleID)
|
||||
}
|
||||
if req.BundlePath == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor bundle path is required for bundle %q", req.BundleID)
|
||||
}
|
||||
if c.newUploadClient == nil {
|
||||
return UploadResult{}, fmt.Errorf("distributor upload client factory is required for endpoint %q", c.Endpoint)
|
||||
}
|
||||
|
||||
token := os.Getenv(c.TokenEnv)
|
||||
if token == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor token environment variable %q is not set", c.TokenEnv)
|
||||
}
|
||||
|
||||
uploadClient, err := c.newUploadClient(c.Endpoint, token, c.Timeout)
|
||||
if err != nil {
|
||||
return UploadResult{}, fmt.Errorf("create distributor upload client for endpoint %q: %w", c.Endpoint, redactToken(err, token))
|
||||
}
|
||||
|
||||
runCtx := ctx
|
||||
if runCtx == nil {
|
||||
runCtx = context.Background()
|
||||
}
|
||||
cancel := func() {}
|
||||
if c.Timeout > 0 {
|
||||
runCtx, cancel = context.WithTimeout(runCtx, c.Timeout)
|
||||
}
|
||||
defer cancel()
|
||||
|
||||
result, err := uploadClient.UploadFiles(runCtx, uploadFilesOptions{
|
||||
BundleID: req.BundleID,
|
||||
IdempotencyKey: req.IdempotencyKey,
|
||||
SourcePath: req.SourcePath,
|
||||
BundlePath: req.BundlePath,
|
||||
})
|
||||
if err != nil {
|
||||
return UploadResult{}, wrapUploadError(err, uploadErrorContext{
|
||||
Endpoint: c.Endpoint,
|
||||
BundleID: req.BundleID,
|
||||
IdempotencyKey: req.IdempotencyKey,
|
||||
SourcePath: req.SourcePath,
|
||||
BundlePath: req.BundlePath,
|
||||
Token: token,
|
||||
})
|
||||
}
|
||||
|
||||
return UploadResult{
|
||||
RunID: result.RunID,
|
||||
Status: result.Status,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type distributorUploadClient struct {
|
||||
client *distributorupload.Client
|
||||
}
|
||||
|
||||
func newDistributorUploadClient(endpoint, token string, timeout time.Duration) (uploadClient, error) {
|
||||
httpClient := (*http.Client)(nil)
|
||||
if timeout > 0 {
|
||||
httpClient = &http.Client{Timeout: timeout}
|
||||
}
|
||||
client, err := distributorupload.NewClient(distributorupload.ClientOptions{
|
||||
Endpoint: endpoint,
|
||||
Token: token,
|
||||
HTTPClient: httpClient,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return distributorUploadClient{client: client}, nil
|
||||
}
|
||||
|
||||
func (c distributorUploadClient) UploadFiles(ctx context.Context, opts uploadFilesOptions) (uploadFilesResult, error) {
|
||||
result, err := c.client.UploadFiles(ctx, distributorupload.UploadFilesOptions{
|
||||
ID: opts.BundleID,
|
||||
IdempotencyKey: opts.IdempotencyKey,
|
||||
Files: []distributorbundle.BundleFile{
|
||||
{SourcePath: opts.SourcePath, Path: opts.BundlePath},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return uploadFilesResult{}, err
|
||||
}
|
||||
return uploadFilesResult{
|
||||
RunID: result.RunID,
|
||||
Status: result.Status,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type uploadErrorContext struct {
|
||||
Endpoint string
|
||||
BundleID string
|
||||
IdempotencyKey string
|
||||
SourcePath string
|
||||
BundlePath string
|
||||
Token string
|
||||
}
|
||||
|
||||
func wrapUploadError(err error, ctx uploadErrorContext) error {
|
||||
var conflict *distributorupload.IdempotencyConflictError
|
||||
isConflict := errors.As(err, &conflict)
|
||||
err = redactToken(err, ctx.Token)
|
||||
if isConflict {
|
||||
return &IdempotencyConflictError{
|
||||
Err: fmt.Errorf("upload distributor bundle %q to endpoint %q with idempotency key %q from source %q as bundle path %q: idempotency conflict: %w", ctx.BundleID, ctx.Endpoint, ctx.IdempotencyKey, ctx.SourcePath, ctx.BundlePath, err),
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("upload distributor bundle %q to endpoint %q with idempotency key %q from source %q as bundle path %q: %w", ctx.BundleID, ctx.Endpoint, ctx.IdempotencyKey, ctx.SourcePath, ctx.BundlePath, err)
|
||||
}
|
||||
|
||||
func redactToken(err error, token string) error {
|
||||
if err == nil || token == "" {
|
||||
return err
|
||||
}
|
||||
return errors.New(redactTokenString(err.Error(), token))
|
||||
}
|
||||
|
||||
func redactTokenString(value, token string) string {
|
||||
if token == "" {
|
||||
return value
|
||||
}
|
||||
return strings.ReplaceAll(value, token, "[redacted]")
|
||||
}
|
||||
243
internal/adapters/distributor/client_test.go
Normal file
243
internal/adapters/distributor/client_test.go
Normal file
@@ -0,0 +1,243 @@
|
||||
package distributor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
distributorupload "gitea.maximumdirect.net/eric/distributor/pkg/upload"
|
||||
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/config"
|
||||
)
|
||||
|
||||
func TestUploadUsesConfiguredClientAndSingleFile(t *testing.T) {
|
||||
cfg := config.Defaults().Notify.Distributor
|
||||
cfg.Endpoint = "https://distributor.example.test"
|
||||
cfg.TokenEnv = "DISTRIBUTOR_UPLOAD_TOKEN"
|
||||
cfg.Timeout = 15 * time.Second
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
|
||||
factory := &fakeUploadFactory{
|
||||
client: &fakeUploadClient{
|
||||
result: uploadFilesResult{RunID: "run-123", Status: "accepted"},
|
||||
},
|
||||
}
|
||||
client := newClient(cfg, factory.newClient)
|
||||
|
||||
result, err := client.Upload(context.Background(), UploadRequest{
|
||||
BundleID: "weatherreporter.home.daily.run",
|
||||
IdempotencyKey: "weatherreporter.home.daily.run",
|
||||
SourcePath: "/tmp/report.md",
|
||||
BundlePath: "daily.md",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Upload() error = %v", err)
|
||||
}
|
||||
if result.RunID != "run-123" || result.Status != "accepted" {
|
||||
t.Fatalf("result = %#v, want accepted run", result)
|
||||
}
|
||||
if factory.endpoint != cfg.Endpoint {
|
||||
t.Fatalf("factory endpoint = %q, want %q", factory.endpoint, cfg.Endpoint)
|
||||
}
|
||||
if factory.token != "secret-token" {
|
||||
t.Fatalf("factory token = %q, want secret-token", factory.token)
|
||||
}
|
||||
if factory.timeout != 15*time.Second {
|
||||
t.Fatalf("factory timeout = %s, want 15s", factory.timeout)
|
||||
}
|
||||
got := factory.client.opts
|
||||
if got.BundleID != "weatherreporter.home.daily.run" {
|
||||
t.Fatalf("BundleID = %q, want weatherreporter.home.daily.run", got.BundleID)
|
||||
}
|
||||
if got.IdempotencyKey != "weatherreporter.home.daily.run" {
|
||||
t.Fatalf("IdempotencyKey = %q, want weatherreporter.home.daily.run", got.IdempotencyKey)
|
||||
}
|
||||
if got.SourcePath != "/tmp/report.md" {
|
||||
t.Fatalf("SourcePath = %q, want /tmp/report.md", got.SourcePath)
|
||||
}
|
||||
if got.BundlePath != "daily.md" {
|
||||
t.Fatalf("BundlePath = %q, want daily.md", got.BundlePath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadRejectsMissingInputs(t *testing.T) {
|
||||
cfg := config.Defaults().Notify.Distributor
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*Client, *UploadRequest)
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "Token",
|
||||
mutate: func(c *Client, req *UploadRequest) {
|
||||
t.Setenv(c.TokenEnv, "")
|
||||
},
|
||||
wantErr: "token environment variable",
|
||||
},
|
||||
{
|
||||
name: "SourcePath",
|
||||
mutate: func(c *Client, req *UploadRequest) {
|
||||
req.SourcePath = ""
|
||||
},
|
||||
wantErr: "source path is required",
|
||||
},
|
||||
{
|
||||
name: "BundlePath",
|
||||
mutate: func(c *Client, req *UploadRequest) {
|
||||
req.BundlePath = ""
|
||||
},
|
||||
wantErr: "bundle path is required",
|
||||
},
|
||||
{
|
||||
name: "UploadClientFactory",
|
||||
mutate: func(c *Client, req *UploadRequest) {
|
||||
c.newUploadClient = nil
|
||||
},
|
||||
wantErr: "upload client factory is required",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
client := newClient(cfg, (&fakeUploadFactory{client: &fakeUploadClient{}}).newClient)
|
||||
req := validUploadRequest()
|
||||
tt.mutate(client, &req)
|
||||
|
||||
_, err := client.Upload(context.Background(), req)
|
||||
if err == nil {
|
||||
t.Fatal("Upload() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
if strings.Contains(err.Error(), "secret-token") {
|
||||
t.Fatalf("error = %q, want no token value", err.Error())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadWrapsFactoryErrorWithoutToken(t *testing.T) {
|
||||
cfg := config.Defaults().Notify.Distributor
|
||||
cfg.Endpoint = "https://distributor.example.test"
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
factory := &fakeUploadFactory{
|
||||
err: fmt.Errorf("factory failed with secret-token"),
|
||||
}
|
||||
client := newClient(cfg, factory.newClient)
|
||||
|
||||
_, err := client.Upload(context.Background(), validUploadRequest())
|
||||
if err == nil {
|
||||
t.Fatal("Upload() error = nil, want error")
|
||||
}
|
||||
if strings.Contains(err.Error(), "secret-token") {
|
||||
t.Fatalf("error = %q, want no token value", err.Error())
|
||||
}
|
||||
if !strings.Contains(err.Error(), cfg.Endpoint) {
|
||||
t.Fatalf("error = %q, want endpoint context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadWrapsUploadFailureWithContextWithoutToken(t *testing.T) {
|
||||
cfg := config.Defaults().Notify.Distributor
|
||||
cfg.Endpoint = "https://distributor.example.test"
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
factory := &fakeUploadFactory{
|
||||
client: &fakeUploadClient{err: fmt.Errorf("server rejected secret-token")},
|
||||
}
|
||||
client := newClient(cfg, factory.newClient)
|
||||
req := validUploadRequest()
|
||||
|
||||
_, err := client.Upload(context.Background(), req)
|
||||
if err == nil {
|
||||
t.Fatal("Upload() error = nil, want error")
|
||||
}
|
||||
for _, want := range []string{cfg.Endpoint, req.BundleID, req.IdempotencyKey, req.SourcePath, req.BundlePath} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("error = %q, want context %q", err.Error(), want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(err.Error(), "secret-token") {
|
||||
t.Fatalf("error = %q, want no token value", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadPreservesIdempotencyConflictDiagnosis(t *testing.T) {
|
||||
cfg := config.Defaults().Notify.Distributor
|
||||
cfg.Endpoint = "https://distributor.example.test"
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
factory := &fakeUploadFactory{
|
||||
client: &fakeUploadClient{
|
||||
err: &distributorupload.IdempotencyConflictError{
|
||||
HTTPError: distributorupload.HTTPError{
|
||||
StatusCode: 409,
|
||||
Status: "409 Conflict",
|
||||
Message: "conflicting upload for secret-token",
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
client := newClient(cfg, factory.newClient)
|
||||
|
||||
_, err := client.Upload(context.Background(), validUploadRequest())
|
||||
if err == nil {
|
||||
t.Fatal("Upload() error = nil, want error")
|
||||
}
|
||||
var conflict *IdempotencyConflictError
|
||||
if !errors.As(err, &conflict) {
|
||||
t.Fatalf("Upload() error = %T %v, want IdempotencyConflictError", err, err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "idempotency conflict") {
|
||||
t.Fatalf("error = %q, want idempotency conflict diagnosis", err.Error())
|
||||
}
|
||||
if strings.Contains(err.Error(), "secret-token") {
|
||||
t.Fatalf("error = %q, want no token value", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func validUploadRequest() UploadRequest {
|
||||
return UploadRequest{
|
||||
BundleID: "weatherreporter.home.daily.run",
|
||||
IdempotencyKey: "weatherreporter.home.daily.run",
|
||||
SourcePath: "/tmp/report.md",
|
||||
BundlePath: "daily.md",
|
||||
}
|
||||
}
|
||||
|
||||
type fakeUploadFactory struct {
|
||||
endpoint string
|
||||
token string
|
||||
timeout time.Duration
|
||||
client *fakeUploadClient
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeUploadFactory) newClient(endpoint, token string, timeout time.Duration) (uploadClient, error) {
|
||||
f.endpoint = endpoint
|
||||
f.token = token
|
||||
f.timeout = timeout
|
||||
if f.err != nil {
|
||||
return nil, f.err
|
||||
}
|
||||
return f.client, nil
|
||||
}
|
||||
|
||||
type fakeUploadClient struct {
|
||||
opts uploadFilesOptions
|
||||
result uploadFilesResult
|
||||
err error
|
||||
}
|
||||
|
||||
func (c *fakeUploadClient) UploadFiles(ctx context.Context, opts uploadFilesOptions) (uploadFilesResult, error) {
|
||||
c.opts = opts
|
||||
if c.err != nil {
|
||||
return uploadFilesResult{}, c.err
|
||||
}
|
||||
return c.result, nil
|
||||
}
|
||||
@@ -3,10 +3,12 @@ package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
distributoradapter "gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/distributor"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/scriptorium"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/weatherapi"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/briefing"
|
||||
@@ -45,6 +47,7 @@ type GenerateRequest struct {
|
||||
Date time.Time
|
||||
StormStart time.Time
|
||||
StormEnd time.Time
|
||||
Notifier Notifier
|
||||
}
|
||||
|
||||
type BatchRequest struct {
|
||||
@@ -54,6 +57,7 @@ type BatchRequest struct {
|
||||
OutputDir string
|
||||
Renderer Renderer
|
||||
Store state.Store
|
||||
Notifier Notifier
|
||||
}
|
||||
|
||||
type FetchBundleRequest struct {
|
||||
@@ -73,6 +77,7 @@ type ReportRequest struct {
|
||||
OutputPath string
|
||||
Renderer Renderer
|
||||
Store state.Store
|
||||
Notifier Notifier
|
||||
}
|
||||
|
||||
type BriefingResult struct {
|
||||
@@ -94,6 +99,7 @@ type ReportResult struct {
|
||||
RecentChanges []changes.Change
|
||||
RenderResult *scriptorium.RenderResult
|
||||
RunResult *scriptorium.RunResult
|
||||
Notification *NotificationResult
|
||||
}
|
||||
|
||||
type BatchResult struct {
|
||||
@@ -107,20 +113,23 @@ type BatchResult struct {
|
||||
}
|
||||
|
||||
type BatchReportResult struct {
|
||||
ReportID report.ID `json:"reportId"`
|
||||
ReportName string `json:"reportName"`
|
||||
PromptID string `json:"promptId"`
|
||||
RunID string `json:"runId"`
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
GeneratedAt time.Time `json:"generatedAt"`
|
||||
ValidPeriod timeutil.Period `json:"validPeriod"`
|
||||
BriefingPath string `json:"briefingPath,omitempty"`
|
||||
DataPackagePath string `json:"dataPackagePath,omitempty"`
|
||||
PreflightPath string `json:"preflightPath,omitempty"`
|
||||
ReportPath string `json:"reportPath,omitempty"`
|
||||
OutputPath string `json:"outputPath,omitempty"`
|
||||
MetadataPath string `json:"metadataPath,omitempty"`
|
||||
ReportID report.ID `json:"reportId"`
|
||||
ReportName string `json:"reportName"`
|
||||
PromptID string `json:"promptId"`
|
||||
RunID string `json:"runId"`
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
NotificationStatus string `json:"notificationStatus,omitempty"`
|
||||
NotificationRunID string `json:"notificationRunId,omitempty"`
|
||||
NotificationError string `json:"notificationError,omitempty"`
|
||||
GeneratedAt time.Time `json:"generatedAt"`
|
||||
ValidPeriod timeutil.Period `json:"validPeriod"`
|
||||
BriefingPath string `json:"briefingPath,omitempty"`
|
||||
DataPackagePath string `json:"dataPackagePath,omitempty"`
|
||||
PreflightPath string `json:"preflightPath,omitempty"`
|
||||
ReportPath string `json:"reportPath,omitempty"`
|
||||
OutputPath string `json:"outputPath,omitempty"`
|
||||
MetadataPath string `json:"metadataPath,omitempty"`
|
||||
}
|
||||
|
||||
type BatchError struct {
|
||||
@@ -139,6 +148,45 @@ type Renderer interface {
|
||||
Run(context.Context, scriptorium.RunRequest) (*scriptorium.RunResult, error)
|
||||
}
|
||||
|
||||
type Notifier interface {
|
||||
Notify(context.Context, NotificationRequest) (*NotificationResult, error)
|
||||
}
|
||||
|
||||
type NotificationRequest struct {
|
||||
ReportID report.ID
|
||||
RunID string
|
||||
BundleID string
|
||||
IdempotencyKey string
|
||||
ReportPath string
|
||||
BundlePath string
|
||||
}
|
||||
|
||||
type NotificationResult struct {
|
||||
BundleID string
|
||||
IdempotencyKey string
|
||||
RunID string
|
||||
Status string
|
||||
}
|
||||
|
||||
type NotificationError struct {
|
||||
Request NotificationRequest
|
||||
Err error
|
||||
}
|
||||
|
||||
func (e *NotificationError) Error() string {
|
||||
if e == nil || e.Err == nil {
|
||||
return "notification failed"
|
||||
}
|
||||
return e.Err.Error()
|
||||
}
|
||||
|
||||
func (e *NotificationError) Unwrap() error {
|
||||
if e == nil {
|
||||
return nil
|
||||
}
|
||||
return e.Err
|
||||
}
|
||||
|
||||
func Generate(ctx context.Context, req GenerateRequest) error {
|
||||
now := req.Now
|
||||
if now.IsZero() {
|
||||
@@ -153,6 +201,7 @@ func Generate(ctx context.Context, req GenerateRequest) error {
|
||||
Config: req.Config,
|
||||
Resolved: resolved,
|
||||
OutputPath: req.OutputPath,
|
||||
Notifier: req.Notifier,
|
||||
})
|
||||
return err
|
||||
}
|
||||
@@ -211,10 +260,16 @@ func RunBatchDetailed(ctx context.Context, req BatchRequest) (*BatchResult, erro
|
||||
OutputPath: outputPath,
|
||||
Renderer: req.Renderer,
|
||||
Store: store,
|
||||
Notifier: req.Notifier,
|
||||
})
|
||||
if err != nil {
|
||||
item.Status = "failed"
|
||||
item.Error = err.Error()
|
||||
var notificationErr *NotificationError
|
||||
if errors.As(err, ¬ificationErr) {
|
||||
item.NotificationStatus = "failed"
|
||||
item.NotificationError = notificationErr.Error()
|
||||
}
|
||||
result.Failed++
|
||||
} else {
|
||||
item.Status = "succeeded"
|
||||
@@ -224,6 +279,10 @@ func RunBatchDetailed(ctx context.Context, req BatchRequest) (*BatchResult, erro
|
||||
item.ReportPath = reportResult.ReportPath
|
||||
item.OutputPath = reportResult.OutputPath
|
||||
item.MetadataPath = reportResult.MetadataPath
|
||||
if reportResult.Notification != nil {
|
||||
item.NotificationStatus = reportResult.Notification.Status
|
||||
item.NotificationRunID = reportResult.Notification.RunID
|
||||
}
|
||||
result.Succeeded++
|
||||
}
|
||||
result.Reports = append(result.Reports, item)
|
||||
@@ -481,6 +540,11 @@ func GenerateReport(ctx context.Context, req ReportRequest) (*ReportResult, erro
|
||||
return nil, runErr
|
||||
}
|
||||
|
||||
notification, err := notifyReport(ctx, req.Config, req.Resolved, reportPath, metadata, req.Notifier)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &ReportResult{
|
||||
Briefing: briefingPackage,
|
||||
BriefingPath: briefingPath,
|
||||
@@ -495,6 +559,97 @@ func GenerateReport(ctx context.Context, req ReportRequest) (*ReportResult, erro
|
||||
RecentChanges: recentChanges,
|
||||
RenderResult: renderResult,
|
||||
RunResult: runResult,
|
||||
Notification: notification,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func notifyReport(ctx context.Context, cfg config.Config, resolved report.Resolved, reportPath string, metadata state.Metadata, notifier Notifier) (*NotificationResult, error) {
|
||||
notifier, enabled := reportNotifier(cfg, notifier)
|
||||
if !enabled {
|
||||
return nil, nil
|
||||
}
|
||||
notificationRequest, err := buildNotificationRequest(cfg, resolved, reportPath, metadata)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result, err := notifier.Notify(ctx, notificationRequest)
|
||||
if err != nil {
|
||||
return nil, &NotificationError{
|
||||
Request: notificationRequest,
|
||||
Err: fmt.Errorf("notify report %q run %q from managed report %q: %w", resolved.Definition.ID, metadata.RunID, reportPath, err),
|
||||
}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func reportNotifier(cfg config.Config, notifier Notifier) (Notifier, bool) {
|
||||
if !cfg.Notify.Distributor.Enabled {
|
||||
return noopNotifier{}, false
|
||||
}
|
||||
if notifier != nil {
|
||||
return notifier, true
|
||||
}
|
||||
return distributorNotifier{
|
||||
client: distributoradapter.New(cfg.Notify.Distributor),
|
||||
}, true
|
||||
}
|
||||
|
||||
func buildNotificationRequest(cfg config.Config, resolved report.Resolved, reportPath string, metadata state.Metadata) (NotificationRequest, error) {
|
||||
values := config.DistributorTemplateValues{
|
||||
LocationID: cfg.Location.ID,
|
||||
ReportID: string(resolved.Definition.ID),
|
||||
RunID: metadata.RunID,
|
||||
ArtifactGroup: resolved.Definition.ArtifactGroup,
|
||||
BatchOutputName: resolved.Definition.BatchOutputName,
|
||||
}
|
||||
bundleID, err := config.RenderDistributorBundleID(cfg.Notify.Distributor.BundleIDTemplate, values)
|
||||
if err != nil {
|
||||
return NotificationRequest{}, err
|
||||
}
|
||||
values.BundleID = bundleID
|
||||
idempotencyKey, err := config.RenderDistributorIdempotencyKey(cfg.Notify.Distributor.IdempotencyKeyTemplate, values)
|
||||
if err != nil {
|
||||
return NotificationRequest{}, err
|
||||
}
|
||||
bundlePath, err := config.RenderDistributorReportPath(cfg.Notify.Distributor.ReportPathTemplate, values)
|
||||
if err != nil {
|
||||
return NotificationRequest{}, err
|
||||
}
|
||||
return NotificationRequest{
|
||||
ReportID: resolved.Definition.ID,
|
||||
RunID: metadata.RunID,
|
||||
BundleID: bundleID,
|
||||
IdempotencyKey: idempotencyKey,
|
||||
ReportPath: reportPath,
|
||||
BundlePath: bundlePath,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type noopNotifier struct{}
|
||||
|
||||
func (noopNotifier) Notify(context.Context, NotificationRequest) (*NotificationResult, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
type distributorNotifier struct {
|
||||
client *distributoradapter.Client
|
||||
}
|
||||
|
||||
func (n distributorNotifier) Notify(ctx context.Context, req NotificationRequest) (*NotificationResult, error) {
|
||||
result, err := n.client.Upload(ctx, distributoradapter.UploadRequest{
|
||||
BundleID: req.BundleID,
|
||||
IdempotencyKey: req.IdempotencyKey,
|
||||
SourcePath: req.ReportPath,
|
||||
BundlePath: req.BundlePath,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &NotificationResult{
|
||||
BundleID: req.BundleID,
|
||||
IdempotencyKey: req.IdempotencyKey,
|
||||
RunID: result.RunID,
|
||||
Status: result.Status,
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -255,6 +255,209 @@ func TestGenerateReportWritesReportAndPreflight(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportDisabledNotificationDoesNotCallNotifier(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := dailyTestConfig(t, server)
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
resolved, err := ResolveGenerate(GenerateRequest{
|
||||
Config: cfg,
|
||||
Report: ReportDaily,
|
||||
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||
}, mustParse("2026-05-29T05:00:00-05:00"))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveGenerate() error = %v", err)
|
||||
}
|
||||
notifier := &recordingNotifier{}
|
||||
|
||||
_, err = GenerateReport(context.Background(), ReportRequest{
|
||||
Config: cfg,
|
||||
Resolved: resolved,
|
||||
Renderer: successfulRenderer("# Daily Report\n"),
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateReport() error = %v", err)
|
||||
}
|
||||
if len(notifier.requests) != 0 {
|
||||
t.Fatalf("notification requests = %#v, want none when disabled", notifier.requests)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportNotifiesManagedReportPath(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := dailyTestConfig(t, server)
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
resolved, err := ResolveGenerate(GenerateRequest{
|
||||
Config: cfg,
|
||||
Report: ReportDaily,
|
||||
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||
}, mustParse("2026-05-29T05:00:00-05:00"))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveGenerate() error = %v", err)
|
||||
}
|
||||
notifier := &recordingNotifier{
|
||||
result: &NotificationResult{RunID: "distributor-run", Status: "accepted"},
|
||||
}
|
||||
outputPath := filepath.Join(t.TempDir(), "daily-copy.md")
|
||||
|
||||
result, err := GenerateReport(context.Background(), ReportRequest{
|
||||
Config: cfg,
|
||||
Resolved: resolved,
|
||||
OutputPath: outputPath,
|
||||
Renderer: successfulRenderer("# Daily Report\n"),
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateReport() error = %v", err)
|
||||
}
|
||||
if result.Notification == nil {
|
||||
t.Fatal("Notification = nil, want notification result")
|
||||
}
|
||||
if result.Notification.RunID != "distributor-run" || result.Notification.Status != "accepted" {
|
||||
t.Fatalf("Notification = %#v, want accepted distributor run", result.Notification)
|
||||
}
|
||||
if len(notifier.requests) != 1 {
|
||||
t.Fatalf("notification requests = %d, want 1", len(notifier.requests))
|
||||
}
|
||||
req := notifier.requests[0]
|
||||
if req.ReportPath != result.ReportPath {
|
||||
t.Fatalf("notification ReportPath = %q, want managed path %q", req.ReportPath, result.ReportPath)
|
||||
}
|
||||
if req.ReportPath == outputPath {
|
||||
t.Fatalf("notification used output copy %q, want managed report path", outputPath)
|
||||
}
|
||||
if req.BundlePath != "daily.md" {
|
||||
t.Fatalf("notification BundlePath = %q, want daily.md", req.BundlePath)
|
||||
}
|
||||
if req.BundleID != "weatherreporter.home.daily_today."+result.Metadata.RunID {
|
||||
t.Fatalf("notification BundleID = %q, want default template", req.BundleID)
|
||||
}
|
||||
if req.IdempotencyKey != req.BundleID {
|
||||
t.Fatalf("IdempotencyKey = %q, want bundle id %q", req.IdempotencyKey, req.BundleID)
|
||||
}
|
||||
if req.RunID != result.Metadata.RunID {
|
||||
t.Fatalf("notification RunID = %q, want report run id %q", req.RunID, result.Metadata.RunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportNotificationFailureFailsReport(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := dailyTestConfig(t, server)
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
resolved, err := ResolveGenerate(GenerateRequest{
|
||||
Config: cfg,
|
||||
Report: ReportDaily,
|
||||
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||
}, mustParse("2026-05-29T05:00:00-05:00"))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveGenerate() error = %v", err)
|
||||
}
|
||||
notifier := &recordingNotifier{err: errors.New("upload rejected")}
|
||||
|
||||
_, err = GenerateReport(context.Background(), ReportRequest{
|
||||
Config: cfg,
|
||||
Resolved: resolved,
|
||||
Renderer: successfulRenderer("# Daily Report\n"),
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("GenerateReport() error = nil, want notification error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "notify report") || !strings.Contains(err.Error(), "upload rejected") {
|
||||
t.Fatalf("error = %q, want notification context", err.Error())
|
||||
}
|
||||
if len(notifier.requests) != 1 {
|
||||
t.Fatalf("notification requests = %d, want one attempted notification", len(notifier.requests))
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportDoesNotNotifyAfterRenderOrRunFailure(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
tests := []struct {
|
||||
name string
|
||||
renderer Renderer
|
||||
}{
|
||||
{
|
||||
name: "Render",
|
||||
renderer: &recordingRenderer{
|
||||
renderResult: &scriptorium.RenderResult{ExitCode: 1, Stderr: "render failed"},
|
||||
err: errors.New("render failed"),
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "Run",
|
||||
renderer: &recordingRenderer{
|
||||
renderResult: &scriptorium.RenderResult{ExitCode: 0},
|
||||
runResult: &scriptorium.RunResult{ExitCode: 2, Stderr: "run failed"},
|
||||
runErr: errors.New("run failed"),
|
||||
runBody: "# Daily Report\n",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := dailyTestConfig(t, server)
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
resolved, err := ResolveGenerate(GenerateRequest{
|
||||
Config: cfg,
|
||||
Report: ReportDaily,
|
||||
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||
}, mustParse("2026-05-29T05:00:00-05:00"))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveGenerate() error = %v", err)
|
||||
}
|
||||
notifier := &recordingNotifier{}
|
||||
|
||||
_, err = GenerateReport(context.Background(), ReportRequest{
|
||||
Config: cfg,
|
||||
Resolved: resolved,
|
||||
Renderer: tt.renderer,
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("GenerateReport() error = nil, want generation error")
|
||||
}
|
||||
if len(notifier.requests) != 0 {
|
||||
t.Fatalf("notification requests = %#v, want none after generation failure", notifier.requests)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportDoesNotNotifyAfterFetchFailure(t *testing.T) {
|
||||
cfg := config.Defaults()
|
||||
cfg.WeatherAPI.BaseURL = ""
|
||||
cfg.WeatherAPI.Timezone = "America/Chicago"
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
resolved, err := ResolveGenerate(GenerateRequest{
|
||||
Config: cfg,
|
||||
Report: ReportDaily,
|
||||
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||
}, mustParse("2026-05-29T05:00:00-05:00"))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveGenerate() error = %v", err)
|
||||
}
|
||||
notifier := &recordingNotifier{}
|
||||
|
||||
_, err = GenerateReport(context.Background(), ReportRequest{
|
||||
Config: cfg,
|
||||
Resolved: resolved,
|
||||
Renderer: successfulRenderer("# Daily Report\n"),
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("GenerateReport() error = nil, want fetch setup error")
|
||||
}
|
||||
if len(notifier.requests) != 0 {
|
||||
t.Fatalf("notification requests = %#v, want none after fetch failure", notifier.requests)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportPersistsFailedPreflight(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := config.Defaults()
|
||||
@@ -1008,6 +1211,62 @@ func TestRunBatchContinuesAfterReportFailure(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunBatchContinuesAfterNotificationFailure(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := config.Defaults()
|
||||
cfg.WeatherAPI.BaseURL = server.URL + "/"
|
||||
cfg.WeatherAPI.Timezone = "America/Chicago"
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
notifier := &recordingNotifier{
|
||||
errByReport: map[report.ID]error{
|
||||
report.ThreeDay: errors.New("distributor unavailable"),
|
||||
},
|
||||
}
|
||||
|
||||
result, err := RunBatchDetailed(context.Background(), BatchRequest{
|
||||
Config: cfg,
|
||||
Batch: BatchMorning,
|
||||
Now: mustParse("2026-05-29T05:00:00-05:00"),
|
||||
Renderer: &selectiveRenderer{runBody: "# Batch Report\n"},
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("RunBatchDetailed() error = %v", err)
|
||||
}
|
||||
|
||||
if result.Total != 3 || result.Succeeded != 2 || result.Failed != 1 {
|
||||
t.Fatalf("summary total/succeeded/failed = %d/%d/%d, want 3/2/1", result.Total, result.Succeeded, result.Failed)
|
||||
}
|
||||
if len(notifier.requests) != 3 {
|
||||
t.Fatalf("notification requests = %d, want one per generated report", len(notifier.requests))
|
||||
}
|
||||
var failedThreeDay bool
|
||||
for _, item := range result.Reports {
|
||||
if item.ReportID == report.ThreeDay {
|
||||
if item.Status == "failed" && strings.Contains(item.Error, "notify report") && strings.Contains(item.Error, "distributor unavailable") {
|
||||
failedThreeDay = true
|
||||
}
|
||||
if item.NotificationStatus != "failed" {
|
||||
t.Fatalf("3-day notification status = %q, want failed", item.NotificationStatus)
|
||||
}
|
||||
if !strings.Contains(item.NotificationError, "distributor unavailable") {
|
||||
t.Fatalf("3-day notification error = %q, want distributor unavailable", item.NotificationError)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if item.Status != "succeeded" {
|
||||
t.Fatalf("report %s status = %s, want succeeded", item.ReportID, item.Status)
|
||||
}
|
||||
if item.NotificationStatus != "accepted" {
|
||||
t.Fatalf("report %s notification status = %q, want accepted", item.ReportID, item.NotificationStatus)
|
||||
}
|
||||
}
|
||||
if !failedThreeDay {
|
||||
t.Fatalf("reports = %#v, want notification failure on 3-day item", result.Reports)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunBatchUsesOutputDirectory(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := config.Defaults()
|
||||
@@ -1164,6 +1423,14 @@ type recordingRenderer struct {
|
||||
runBody string
|
||||
}
|
||||
|
||||
func successfulRenderer(body string) *recordingRenderer {
|
||||
return &recordingRenderer{
|
||||
renderResult: &scriptorium.RenderResult{ExitCode: 0},
|
||||
runResult: &scriptorium.RunResult{ExitCode: 0},
|
||||
runBody: body,
|
||||
}
|
||||
}
|
||||
|
||||
type selectiveRenderer struct {
|
||||
renderCalls int
|
||||
runCalls int
|
||||
@@ -1171,6 +1438,38 @@ type selectiveRenderer struct {
|
||||
runBody string
|
||||
}
|
||||
|
||||
type recordingNotifier struct {
|
||||
requests []NotificationRequest
|
||||
result *NotificationResult
|
||||
err error
|
||||
errByReport map[report.ID]error
|
||||
}
|
||||
|
||||
func (n *recordingNotifier) Notify(_ context.Context, req NotificationRequest) (*NotificationResult, error) {
|
||||
n.requests = append(n.requests, req)
|
||||
if err := n.errByReport[req.ReportID]; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if n.err != nil {
|
||||
return nil, n.err
|
||||
}
|
||||
if n.result != nil {
|
||||
result := *n.result
|
||||
if result.BundleID == "" {
|
||||
result.BundleID = req.BundleID
|
||||
}
|
||||
if result.IdempotencyKey == "" {
|
||||
result.IdempotencyKey = req.IdempotencyKey
|
||||
}
|
||||
return &result, nil
|
||||
}
|
||||
return &NotificationResult{
|
||||
BundleID: req.BundleID,
|
||||
IdempotencyKey: req.IdempotencyKey,
|
||||
Status: "accepted",
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (r *selectiveRenderer) Render(_ context.Context, req scriptorium.RenderRequest) (*scriptorium.RenderResult, error) {
|
||||
r.renderCalls++
|
||||
if req.PromptID == r.failRenderPrompt {
|
||||
|
||||
@@ -352,11 +352,21 @@ func writeRunLogs(stderr io.Writer, result *app.BatchResult) {
|
||||
return
|
||||
}
|
||||
for _, item := range result.Reports {
|
||||
notificationFields := ""
|
||||
if item.NotificationStatus != "" {
|
||||
notificationFields += fmt.Sprintf(" notificationStatus=%q", item.NotificationStatus)
|
||||
}
|
||||
if item.NotificationRunID != "" {
|
||||
notificationFields += fmt.Sprintf(" notificationRunId=%q", item.NotificationRunID)
|
||||
}
|
||||
if item.NotificationError != "" {
|
||||
notificationFields += fmt.Sprintf(" notificationError=%q", item.NotificationError)
|
||||
}
|
||||
if item.Status == "failed" {
|
||||
_, _ = fmt.Fprintf(stderr, "report=%s status=failed error=%q\n", item.ReportID, item.Error)
|
||||
_, _ = fmt.Fprintf(stderr, "report=%s status=failed error=%q%s\n", item.ReportID, item.Error, notificationFields)
|
||||
continue
|
||||
}
|
||||
_, _ = fmt.Fprintf(stderr, "report=%s status=succeeded output=%q\n", item.ReportID, item.OutputPath)
|
||||
_, _ = fmt.Fprintf(stderr, "report=%s status=succeeded output=%q%s\n", item.ReportID, item.OutputPath, notificationFields)
|
||||
}
|
||||
_, _ = fmt.Fprintf(stderr, "batch=%s total=%d succeeded=%d failed=%d\n", result.Batch, result.Total, result.Succeeded, result.Failed)
|
||||
}
|
||||
|
||||
@@ -346,6 +346,88 @@ func TestRunMorningReportsPartialFailureAndContinues(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestBatchOutputIncludesNotificationDetails(t *testing.T) {
|
||||
result := &app.BatchResult{
|
||||
Batch: app.BatchMorning,
|
||||
Total: 2,
|
||||
Succeeded: 1,
|
||||
Failed: 1,
|
||||
Reports: []app.BatchReportResult{
|
||||
{
|
||||
ReportID: "daily_today",
|
||||
Status: "succeeded",
|
||||
OutputPath: "/tmp/daily.md",
|
||||
NotificationStatus: "accepted",
|
||||
NotificationRunID: "distributor-run-1",
|
||||
},
|
||||
{
|
||||
ReportID: "three_day",
|
||||
Status: "failed",
|
||||
Error: "notify report three_day: upload failed",
|
||||
NotificationStatus: "failed",
|
||||
NotificationError: "notify report three_day: upload failed",
|
||||
},
|
||||
},
|
||||
}
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
if err := writeJSON(&stdout, result); err != nil {
|
||||
t.Fatalf("writeJSON() error = %v", err)
|
||||
}
|
||||
writeRunLogs(&stderr, result)
|
||||
|
||||
var decoded app.BatchResult
|
||||
if err := json.Unmarshal(stdout.Bytes(), &decoded); err != nil {
|
||||
t.Fatalf("decode batch JSON: %v\n%s", err, stdout.String())
|
||||
}
|
||||
if decoded.Reports[0].NotificationStatus != "accepted" || decoded.Reports[0].NotificationRunID != "distributor-run-1" {
|
||||
t.Fatalf("success notification fields = %#v", decoded.Reports[0])
|
||||
}
|
||||
if decoded.Reports[1].NotificationStatus != "failed" || !strings.Contains(decoded.Reports[1].NotificationError, "upload failed") {
|
||||
t.Fatalf("failure notification fields = %#v", decoded.Reports[1])
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `notificationStatus="accepted"`) || !strings.Contains(stderr.String(), `notificationRunId="distributor-run-1"`) {
|
||||
t.Fatalf("stderr missing success notification fields:\n%s", stderr.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `notificationStatus="failed"`) || !strings.Contains(stderr.String(), `notificationError="notify report three_day: upload failed"`) {
|
||||
t.Fatalf("stderr missing failure notification fields:\n%s", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestBatchOutputDoesNotExposeSecretLikeNotificationErrors(t *testing.T) {
|
||||
result := &app.BatchResult{
|
||||
Batch: app.BatchMorning,
|
||||
Total: 1,
|
||||
Failed: 1,
|
||||
Reports: []app.BatchReportResult{
|
||||
{
|
||||
ReportID: "daily_today",
|
||||
Status: "failed",
|
||||
Error: "notify report daily_today: upload failed: [redacted]",
|
||||
NotificationStatus: "failed",
|
||||
NotificationError: "notify report daily_today: upload failed: [redacted]",
|
||||
},
|
||||
},
|
||||
}
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
if err := writeJSON(&stdout, result); err != nil {
|
||||
t.Fatalf("writeJSON() error = %v", err)
|
||||
}
|
||||
writeRunLogs(&stderr, result)
|
||||
|
||||
for _, output := range []string{stdout.String(), stderr.String()} {
|
||||
if strings.Contains(output, "DISTRIBUTOR_SECRET_TOKEN") {
|
||||
t.Fatalf("output contains token value:\n%s", output)
|
||||
}
|
||||
if !strings.Contains(output, "[redacted]") {
|
||||
t.Fatalf("output missing redacted marker:\n%s", output)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunEveningUsesOutputDirectoryAndSummary(t *testing.T) {
|
||||
server := dailyServer(t)
|
||||
tempDir := t.TempDir()
|
||||
@@ -384,6 +466,103 @@ func TestRunEveningUsesOutputDirectoryAndSummary(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunEveningReportsNotificationSuccess(t *testing.T) {
|
||||
server := dailyServer(t)
|
||||
distributorServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/upload" {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
_, _ = w.Write([]byte(`{"run_id":"distributor-run-1","status":"accepted"}`))
|
||||
}))
|
||||
t.Cleanup(distributorServer.Close)
|
||||
tempDir := t.TempDir()
|
||||
scriptoriumPath := writeFakeScriptorium(t, tempDir)
|
||||
configPath := filepath.Join(tempDir, "config.yml")
|
||||
workspaceRoot := filepath.Join(tempDir, "workspace")
|
||||
configBody := "weather_api:\n base_url: " + server.URL + "/\n timezone: America/Chicago\nscriptorium:\n binary: " + scriptoriumPath + "\nworkspace:\n root: " + workspaceRoot + "\nnotify:\n distributor:\n enabled: true\n endpoint: " + distributorServer.URL + "\n token_env: CLI_DISTRIBUTOR_TOKEN\n"
|
||||
if err := os.WriteFile(configPath, []byte(configBody), 0o600); err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
t.Setenv("CLI_DISTRIBUTOR_TOKEN", "cli-secret-token")
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
runner := Runner{Clock: fixedClock()}
|
||||
|
||||
err := runner.Run(context.Background(), []string{
|
||||
"run", "evening",
|
||||
"--config", configPath,
|
||||
}, &stdout, &stderr)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
|
||||
var summary app.BatchResult
|
||||
if decodeErr := json.Unmarshal(stdout.Bytes(), &summary); decodeErr != nil {
|
||||
t.Fatalf("decode summary: %v\n%s", decodeErr, stdout.String())
|
||||
}
|
||||
if len(summary.Reports) != 1 {
|
||||
t.Fatalf("reports = %#v, want one report", summary.Reports)
|
||||
}
|
||||
if summary.Reports[0].NotificationStatus != "accepted" || summary.Reports[0].NotificationRunID != "distributor-run-1" {
|
||||
t.Fatalf("notification fields = %#v", summary.Reports[0])
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `notificationStatus="accepted"`) || !strings.Contains(stderr.String(), `notificationRunId="distributor-run-1"`) {
|
||||
t.Fatalf("stderr missing notification fields:\n%s", stderr.String())
|
||||
}
|
||||
if strings.Contains(stdout.String(), "cli-secret-token") || strings.Contains(stderr.String(), "cli-secret-token") {
|
||||
t.Fatalf("output contains token value\nstdout=%s\nstderr=%s", stdout.String(), stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunEveningReportsNotificationFailureWithoutToken(t *testing.T) {
|
||||
server := dailyServer(t)
|
||||
distributorServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
_, _ = w.Write([]byte(`{"error":"rejected cli-secret-token","retryable":false}`))
|
||||
}))
|
||||
t.Cleanup(distributorServer.Close)
|
||||
tempDir := t.TempDir()
|
||||
scriptoriumPath := writeFakeScriptorium(t, tempDir)
|
||||
configPath := filepath.Join(tempDir, "config.yml")
|
||||
workspaceRoot := filepath.Join(tempDir, "workspace")
|
||||
configBody := "weather_api:\n base_url: " + server.URL + "/\n timezone: America/Chicago\nscriptorium:\n binary: " + scriptoriumPath + "\nworkspace:\n root: " + workspaceRoot + "\nnotify:\n distributor:\n enabled: true\n endpoint: " + distributorServer.URL + "\n token_env: CLI_DISTRIBUTOR_TOKEN\n"
|
||||
if err := os.WriteFile(configPath, []byte(configBody), 0o600); err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
t.Setenv("CLI_DISTRIBUTOR_TOKEN", "cli-secret-token")
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
runner := Runner{Clock: fixedClock()}
|
||||
|
||||
err := runner.Run(context.Background(), []string{
|
||||
"run", "evening",
|
||||
"--config", configPath,
|
||||
}, &stdout, &stderr)
|
||||
if err == nil {
|
||||
t.Fatal("Run() error = nil, want notification failure")
|
||||
}
|
||||
|
||||
var summary app.BatchResult
|
||||
if decodeErr := json.Unmarshal(stdout.Bytes(), &summary); decodeErr != nil {
|
||||
t.Fatalf("decode summary: %v\n%s", decodeErr, stdout.String())
|
||||
}
|
||||
if len(summary.Reports) != 1 || summary.Reports[0].NotificationStatus != "failed" {
|
||||
t.Fatalf("summary reports = %#v, want failed notification", summary.Reports)
|
||||
}
|
||||
for _, output := range []string{stdout.String(), stderr.String(), err.Error()} {
|
||||
if strings.Contains(output, "cli-secret-token") {
|
||||
t.Fatalf("output contains token value:\n%s", output)
|
||||
}
|
||||
}
|
||||
for _, output := range []string{stdout.String(), stderr.String()} {
|
||||
if !strings.Contains(output, "[redacted]") {
|
||||
t.Fatalf("output missing redaction marker:\n%s", output)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunMorningGeneratesDailyAndThreeDayOnSunday(t *testing.T) {
|
||||
server := dailyServer(t)
|
||||
tempDir := t.TempDir()
|
||||
|
||||
@@ -5,16 +5,21 @@ package config
|
||||
import "time"
|
||||
|
||||
type MissingSourcePolicy string
|
||||
type NotifyFailurePolicy string
|
||||
|
||||
const (
|
||||
MissingSourceError MissingSourcePolicy = "error"
|
||||
MissingSourceWarn MissingSourcePolicy = "warn"
|
||||
MissingSourceNone MissingSourcePolicy = "none"
|
||||
|
||||
NotifyFailureError NotifyFailurePolicy = "error"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
WeatherAPI WeatherAPIConfig `yaml:"weather_api"`
|
||||
Location LocationConfig `yaml:"location"`
|
||||
Secrets SecretsConfig `yaml:"secrets"`
|
||||
Notify NotifyConfig `yaml:"notify"`
|
||||
MissingSource MissingSourceConfig `yaml:"missing_source"`
|
||||
Scriptorium ScriptoriumConfig `yaml:"scriptorium"`
|
||||
Workspace WorkspaceConfig `yaml:"workspace"`
|
||||
@@ -37,6 +42,25 @@ type LocationConfig struct {
|
||||
Region string `yaml:"region"`
|
||||
}
|
||||
|
||||
type SecretsConfig struct {
|
||||
Directory string `yaml:"directory"`
|
||||
}
|
||||
|
||||
type NotifyConfig struct {
|
||||
Distributor DistributorNotifyConfig `yaml:"distributor"`
|
||||
}
|
||||
|
||||
type DistributorNotifyConfig struct {
|
||||
Enabled bool `yaml:"enabled"`
|
||||
Endpoint string `yaml:"endpoint"`
|
||||
TokenEnv string `yaml:"token_env"`
|
||||
Timeout time.Duration `yaml:"timeout"`
|
||||
FailurePolicy NotifyFailurePolicy `yaml:"failure_policy"`
|
||||
BundleIDTemplate string `yaml:"bundle_id_template"`
|
||||
IdempotencyKeyTemplate string `yaml:"idempotency_key_template"`
|
||||
ReportPathTemplate string `yaml:"report_path_template"`
|
||||
}
|
||||
|
||||
type MissingSourceConfig struct {
|
||||
Default MissingSourcePolicy `yaml:"default"`
|
||||
Sources map[string]MissingSourcePolicy `yaml:"sources"`
|
||||
|
||||
@@ -26,6 +26,33 @@ func TestDefaults(t *testing.T) {
|
||||
if cfg.Location.ID != "home" || cfg.Location.Name != "Brentwood" || cfg.Location.Region != "St. Louis Metro" {
|
||||
t.Fatalf("Location = %#v, want home/Brentwood/St. Louis Metro", cfg.Location)
|
||||
}
|
||||
if cfg.Secrets.Directory != "" {
|
||||
t.Fatalf("Secrets.Directory = %q, want empty", cfg.Secrets.Directory)
|
||||
}
|
||||
if cfg.Notify.Distributor.Enabled {
|
||||
t.Fatalf("Notify.Distributor.Enabled = true, want false")
|
||||
}
|
||||
if cfg.Notify.Distributor.Endpoint != "https://distributor.example.com" {
|
||||
t.Fatalf("Notify.Distributor.Endpoint = %q, want default endpoint", cfg.Notify.Distributor.Endpoint)
|
||||
}
|
||||
if cfg.Notify.Distributor.TokenEnv != "DISTRIBUTOR_UPLOAD_TOKEN" {
|
||||
t.Fatalf("Notify.Distributor.TokenEnv = %q, want DISTRIBUTOR_UPLOAD_TOKEN", cfg.Notify.Distributor.TokenEnv)
|
||||
}
|
||||
if cfg.Notify.Distributor.Timeout != 30*time.Second {
|
||||
t.Fatalf("Notify.Distributor.Timeout = %s, want 30s", cfg.Notify.Distributor.Timeout)
|
||||
}
|
||||
if cfg.Notify.Distributor.FailurePolicy != NotifyFailureError {
|
||||
t.Fatalf("Notify.Distributor.FailurePolicy = %q, want error", cfg.Notify.Distributor.FailurePolicy)
|
||||
}
|
||||
if cfg.Notify.Distributor.BundleIDTemplate != "weatherreporter.{location_id}.{report_id}.{run_id}" {
|
||||
t.Fatalf("Notify.Distributor.BundleIDTemplate = %q, want default", cfg.Notify.Distributor.BundleIDTemplate)
|
||||
}
|
||||
if cfg.Notify.Distributor.IdempotencyKeyTemplate != "{bundle_id}" {
|
||||
t.Fatalf("Notify.Distributor.IdempotencyKeyTemplate = %q, want default", cfg.Notify.Distributor.IdempotencyKeyTemplate)
|
||||
}
|
||||
if cfg.Notify.Distributor.ReportPathTemplate != "{batch_output_name}" {
|
||||
t.Fatalf("Notify.Distributor.ReportPathTemplate = %q, want default", cfg.Notify.Distributor.ReportPathTemplate)
|
||||
}
|
||||
if cfg.MissingSource.Default != MissingSourceWarn {
|
||||
t.Fatalf("MissingSource.Default = %q, want warn", cfg.MissingSource.Default)
|
||||
}
|
||||
@@ -112,3 +139,417 @@ func TestLoadAppliesOverrides(t *testing.T) {
|
||||
t.Fatalf("Timezone = %q, want +09:30", cfg.WeatherAPI.Timezone)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDisabledDistributorNotifyAcceptsOmittedFields(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "config.yml")
|
||||
if err := os.WriteFile(path, []byte("notify:\n distributor:\n enabled: false\n"), 0o600); err != nil {
|
||||
t.Fatalf("write config fixture: %v", err)
|
||||
}
|
||||
|
||||
cfg, err := LoadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadFile() error = %v", err)
|
||||
}
|
||||
if cfg.Notify.Distributor.Enabled {
|
||||
t.Fatalf("Notify.Distributor.Enabled = true, want false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnabledDistributorNotifyValidation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*Config)
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "Endpoint",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.Endpoint = "distributor.example.com"
|
||||
},
|
||||
wantErr: "notify.distributor.endpoint",
|
||||
},
|
||||
{
|
||||
name: "TokenEnvEmpty",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.TokenEnv = ""
|
||||
},
|
||||
wantErr: "notify.distributor.token_env",
|
||||
},
|
||||
{
|
||||
name: "TokenEnvInvalid",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.TokenEnv = "1TOKEN"
|
||||
},
|
||||
wantErr: "notify.distributor.token_env",
|
||||
},
|
||||
{
|
||||
name: "Timeout",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.Timeout = 0
|
||||
},
|
||||
wantErr: "notify.distributor.timeout",
|
||||
},
|
||||
{
|
||||
name: "FailurePolicy",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.FailurePolicy = "warn"
|
||||
},
|
||||
wantErr: "notify.distributor.failure_policy",
|
||||
},
|
||||
{
|
||||
name: "BundleTemplate",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.BundleIDTemplate = "{unknown}"
|
||||
},
|
||||
wantErr: "notify.distributor.bundle_id_template",
|
||||
},
|
||||
{
|
||||
name: "IdempotencyTemplate",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.IdempotencyKeyTemplate = "{unknown}"
|
||||
},
|
||||
wantErr: "notify.distributor.idempotency_key_template",
|
||||
},
|
||||
{
|
||||
name: "ReportPathTemplateUnknown",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.ReportPathTemplate = "{bundle_id}"
|
||||
},
|
||||
wantErr: "notify.distributor.report_path_template",
|
||||
},
|
||||
{
|
||||
name: "ReportPathTemplateInvalidPath",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.ReportPathTemplate = "/{batch_output_name}"
|
||||
},
|
||||
wantErr: "notify.distributor.report_path_template",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := Defaults()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
tt.mutate(&cfg)
|
||||
|
||||
err := Validate(cfg)
|
||||
if err == nil {
|
||||
t.Fatal("Validate() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDistributorTemplateRendering(t *testing.T) {
|
||||
values := DistributorTemplateValues{
|
||||
LocationID: "home",
|
||||
ReportID: "daily",
|
||||
RunID: "20260607T120000Z",
|
||||
ArtifactGroup: "daily",
|
||||
BatchOutputName: "daily.md",
|
||||
BundleID: "weatherreporter.home.daily.20260607T120000Z",
|
||||
}
|
||||
|
||||
bundleID, err := RenderDistributorBundleID("weatherreporter.{location_id}.{report_id}.{run_id}", values)
|
||||
if err != nil {
|
||||
t.Fatalf("RenderDistributorBundleID() error = %v", err)
|
||||
}
|
||||
if bundleID != "weatherreporter.home.daily.20260607T120000Z" {
|
||||
t.Fatalf("bundleID = %q, want rendered value", bundleID)
|
||||
}
|
||||
|
||||
idempotencyKey, err := RenderDistributorIdempotencyKey("{bundle_id}", values)
|
||||
if err != nil {
|
||||
t.Fatalf("RenderDistributorIdempotencyKey() error = %v", err)
|
||||
}
|
||||
if idempotencyKey != "weatherreporter.home.daily.20260607T120000Z" {
|
||||
t.Fatalf("idempotencyKey = %q, want rendered bundle ID", idempotencyKey)
|
||||
}
|
||||
|
||||
reportPath, err := RenderDistributorReportPath("reports/{batch_output_name}", values)
|
||||
if err != nil {
|
||||
t.Fatalf("RenderDistributorReportPath() error = %v", err)
|
||||
}
|
||||
if reportPath != "reports/daily.md" {
|
||||
t.Fatalf("reportPath = %q, want reports/daily.md", reportPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDistributorTemplateRejectsUnknownAndMalformedVariables(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
template string
|
||||
}{
|
||||
{name: "Unknown", template: "{unknown}"},
|
||||
{name: "Unclosed", template: "{location_id"},
|
||||
{name: "Unopened", template: "location_id}"},
|
||||
{name: "Empty", template: "{}"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
_, err := RenderDistributorBundleID(tt.template, DistributorTemplateValues{})
|
||||
if err == nil {
|
||||
t.Fatal("RenderDistributorBundleID() error = nil, want error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDistributorReportPathValidation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
path string
|
||||
ok bool
|
||||
}{
|
||||
{name: "Simple", path: "daily.md", ok: true},
|
||||
{name: "Nested", path: "reports/daily.md", ok: true},
|
||||
{name: "Empty", path: "", ok: false},
|
||||
{name: "Absolute", path: "/reports/daily.md", ok: false},
|
||||
{name: "WindowsAbsolute", path: "C:/reports/daily.md", ok: false},
|
||||
{name: "Backslash", path: `reports\daily.md`, ok: false},
|
||||
{name: "CurrentSegment", path: "reports/./daily.md", ok: false},
|
||||
{name: "ParentSegment", path: "reports/../daily.md", ok: false},
|
||||
{name: "EmptySegment", path: "reports//daily.md", ok: false},
|
||||
{name: "Manifest", path: "reports/manifest.json", ok: false},
|
||||
{name: "DistributorMetadata", path: "reports/.distributor.json", ok: false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := ValidateDistributorReportPath(tt.path)
|
||||
if tt.ok && err != nil {
|
||||
t.Fatalf("ValidateDistributorReportPath() error = %v", err)
|
||||
}
|
||||
if !tt.ok && err == nil {
|
||||
t.Fatal("ValidateDistributorReportPath() error = nil, want error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDistributorReportPathRenderingRejectsInvalidValues(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
batchOutputName string
|
||||
}{
|
||||
{name: "Absolute", batchOutputName: "/daily.md"},
|
||||
{name: "Backslash", batchOutputName: `reports\daily.md`},
|
||||
{name: "CurrentSegment", batchOutputName: "./daily.md"},
|
||||
{name: "ParentSegment", batchOutputName: "../daily.md"},
|
||||
{name: "EmptySegment", batchOutputName: "reports//daily.md"},
|
||||
{name: "Manifest", batchOutputName: "manifest.json"},
|
||||
{name: "DistributorMetadata", batchOutputName: ".distributor.json"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
_, err := RenderDistributorReportPath("{batch_output_name}", DistributorTemplateValues{
|
||||
BatchOutputName: tt.batchOutputName,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("RenderDistributorReportPath() error = nil, want error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFileLoadsSecretsBeforeReturningNotifyConfig(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
secretsDir := filepath.Join(dir, "secrets")
|
||||
if err := os.Mkdir(secretsDir, 0o700); err != nil {
|
||||
t.Fatalf("create secrets directory: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(secretsDir, "DISTRIBUTOR_UPLOAD_TOKEN"), []byte("loaded-token"), 0o600); err != nil {
|
||||
t.Fatalf("write secret: %v", err)
|
||||
}
|
||||
path := filepath.Join(dir, "config.yml")
|
||||
configYAML := "secrets:\n" +
|
||||
" directory: " + secretsDir + "\n" +
|
||||
"notify:\n" +
|
||||
" distributor:\n" +
|
||||
" enabled: true\n"
|
||||
if err := os.WriteFile(path, []byte(configYAML), 0o600); err != nil {
|
||||
t.Fatalf("write config fixture: %v", err)
|
||||
}
|
||||
|
||||
t.Setenv("DISTRIBUTOR_UPLOAD_TOKEN", "")
|
||||
cfg, err := LoadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadFile() error = %v", err)
|
||||
}
|
||||
if cfg.Notify.Distributor.TokenEnv != "DISTRIBUTOR_UPLOAD_TOKEN" {
|
||||
t.Fatalf("TokenEnv = %q, want DISTRIBUTOR_UPLOAD_TOKEN", cfg.Notify.Distributor.TokenEnv)
|
||||
}
|
||||
if got := os.Getenv(cfg.Notify.Distributor.TokenEnv); got != "loaded-token" {
|
||||
t.Fatalf("environment value = %q, want loaded-token", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSecretsDisabledLeavesEnvironmentUnchanged(t *testing.T) {
|
||||
t.Setenv("WEATHERREPORTER_DISABLED_SECRET", "original")
|
||||
|
||||
if err := loadSecrets(SecretsConfig{}); err != nil {
|
||||
t.Fatalf("loadSecrets() error = %v", err)
|
||||
}
|
||||
if got := os.Getenv("WEATHERREPORTER_DISABLED_SECRET"); got != "original" {
|
||||
t.Fatalf("environment value = %q, want original", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFileLoadsSecretsDirectory(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
secretsDir := filepath.Join(dir, "secrets")
|
||||
if err := os.Mkdir(secretsDir, 0o700); err != nil {
|
||||
t.Fatalf("create secrets directory: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(secretsDir, "WEATHERREPORTER_SECRET"), []byte("from-file"), 0o600); err != nil {
|
||||
t.Fatalf("write secret: %v", err)
|
||||
}
|
||||
path := filepath.Join(dir, "config.yml")
|
||||
if err := os.WriteFile(path, []byte("secrets:\n directory: "+secretsDir+"\n"), 0o600); err != nil {
|
||||
t.Fatalf("write config fixture: %v", err)
|
||||
}
|
||||
|
||||
t.Setenv("WEATHERREPORTER_SECRET", "")
|
||||
if _, err := LoadFile(path); err != nil {
|
||||
t.Fatalf("LoadFile() error = %v", err)
|
||||
}
|
||||
if got := os.Getenv("WEATHERREPORTER_SECRET"); got != "from-file" {
|
||||
t.Fatalf("environment value = %q, want from-file", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSecretsOverwritesExistingEnvironment(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, "WEATHERREPORTER_SECRET"), []byte("from-file"), 0o600); err != nil {
|
||||
t.Fatalf("write secret: %v", err)
|
||||
}
|
||||
t.Setenv("WEATHERREPORTER_SECRET", "existing")
|
||||
|
||||
if err := loadSecrets(SecretsConfig{Directory: dir}); err != nil {
|
||||
t.Fatalf("loadSecrets() error = %v", err)
|
||||
}
|
||||
if got := os.Getenv("WEATHERREPORTER_SECRET"); got != "from-file" {
|
||||
t.Fatalf("environment value = %q, want from-file", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSecretsTrimsOneTrailingLineEnding(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
want string
|
||||
}{
|
||||
{name: "LF", input: "value\n", want: "value"},
|
||||
{name: "CRLF", input: "value\r\n", want: "value"},
|
||||
{name: "TwoLF", input: "value\n\n", want: "value\n"},
|
||||
{name: "LoneCR", input: "value\r", want: "value\r"},
|
||||
{name: "NoNewline", input: "value", want: "value"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, "WEATHERREPORTER_SECRET"), []byte(tt.input), 0o600); err != nil {
|
||||
t.Fatalf("write secret: %v", err)
|
||||
}
|
||||
t.Setenv("WEATHERREPORTER_SECRET", "")
|
||||
|
||||
if err := loadSecrets(SecretsConfig{Directory: dir}); err != nil {
|
||||
t.Fatalf("loadSecrets() error = %v", err)
|
||||
}
|
||||
if got := os.Getenv("WEATHERREPORTER_SECRET"); got != tt.want {
|
||||
t.Fatalf("environment value = %q, want %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSecretsRejectsInvalidDirectoryEntries(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
setup func(t *testing.T, dir string)
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "InvalidFilename",
|
||||
setup: func(t *testing.T, dir string) {
|
||||
if err := os.WriteFile(filepath.Join(dir, "1INVALID"), []byte("secret-value"), 0o600); err != nil {
|
||||
t.Fatalf("write invalid secret: %v", err)
|
||||
}
|
||||
},
|
||||
wantErr: "invalid environment variable name",
|
||||
},
|
||||
{
|
||||
name: "Subdirectory",
|
||||
setup: func(t *testing.T, dir string) {
|
||||
if err := os.Mkdir(filepath.Join(dir, "SUBDIR"), 0o700); err != nil {
|
||||
t.Fatalf("create subdirectory: %v", err)
|
||||
}
|
||||
},
|
||||
wantErr: "not a directory",
|
||||
},
|
||||
{
|
||||
name: "Symlink",
|
||||
setup: func(t *testing.T, dir string) {
|
||||
target := filepath.Join(dir, "TARGET")
|
||||
if err := os.WriteFile(target, []byte("secret-value"), 0o600); err != nil {
|
||||
t.Fatalf("write target: %v", err)
|
||||
}
|
||||
if err := os.Symlink(target, filepath.Join(dir, "SYMLINK")); err != nil {
|
||||
t.Fatalf("create symlink: %v", err)
|
||||
}
|
||||
},
|
||||
wantErr: "not a symlink",
|
||||
},
|
||||
{
|
||||
name: "Unreadable",
|
||||
setup: func(t *testing.T, dir string) {
|
||||
path := filepath.Join(dir, "UNREADABLE")
|
||||
if err := os.WriteFile(path, []byte("secret-value"), 0o600); err != nil {
|
||||
t.Fatalf("write unreadable secret: %v", err)
|
||||
}
|
||||
if err := os.Chmod(path, 0o000); err != nil {
|
||||
t.Fatalf("chmod unreadable secret: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
_ = os.Chmod(path, 0o600)
|
||||
})
|
||||
},
|
||||
wantErr: "read secret file",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
tt.setup(t, dir)
|
||||
|
||||
err := loadSecrets(SecretsConfig{Directory: dir})
|
||||
if err == nil {
|
||||
t.Fatal("loadSecrets() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
if strings.Contains(err.Error(), "secret-value") {
|
||||
t.Fatalf("error = %q, want no secret value", err.Error())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSecretsRejectsMissingDirectory(t *testing.T) {
|
||||
err := loadSecrets(SecretsConfig{Directory: filepath.Join(t.TempDir(), "missing")})
|
||||
if err == nil {
|
||||
t.Fatal("loadSecrets() error = nil, want missing directory error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "read secrets directory") {
|
||||
t.Fatalf("error = %q, want read secrets directory context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18,6 +18,21 @@ func Defaults() Config {
|
||||
Name: "Brentwood",
|
||||
Region: "St. Louis Metro",
|
||||
},
|
||||
Secrets: SecretsConfig{
|
||||
Directory: "",
|
||||
},
|
||||
Notify: NotifyConfig{
|
||||
Distributor: DistributorNotifyConfig{
|
||||
Enabled: false,
|
||||
Endpoint: "https://distributor.example.com",
|
||||
TokenEnv: "DISTRIBUTOR_UPLOAD_TOKEN",
|
||||
Timeout: 30 * time.Second,
|
||||
FailurePolicy: NotifyFailureError,
|
||||
BundleIDTemplate: "weatherreporter.{location_id}.{report_id}.{run_id}",
|
||||
IdempotencyKeyTemplate: "{bundle_id}",
|
||||
ReportPathTemplate: "{batch_output_name}",
|
||||
},
|
||||
},
|
||||
MissingSource: MissingSourceConfig{
|
||||
Default: MissingSourceWarn,
|
||||
Sources: map[string]MissingSourcePolicy{},
|
||||
|
||||
@@ -35,6 +35,10 @@ func Load(opts LoadOptions) (Config, error) {
|
||||
cfg.WeatherAPI.Timezone = opts.Timezone
|
||||
}
|
||||
|
||||
if err := loadSecrets(cfg.Secrets); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
|
||||
if err := Validate(cfg); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
|
||||
145
internal/config/notify_templates.go
Normal file
145
internal/config/notify_templates.go
Normal file
@@ -0,0 +1,145 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type DistributorTemplateValues struct {
|
||||
LocationID string
|
||||
ReportID string
|
||||
RunID string
|
||||
ArtifactGroup string
|
||||
BatchOutputName string
|
||||
BundleID string
|
||||
}
|
||||
|
||||
var distributorTemplateVariables = map[string]struct{}{
|
||||
"location_id": {},
|
||||
"report_id": {},
|
||||
"run_id": {},
|
||||
"artifact_group": {},
|
||||
"batch_output_name": {},
|
||||
}
|
||||
|
||||
var distributorIdempotencyTemplateVariables = map[string]struct{}{
|
||||
"location_id": {},
|
||||
"report_id": {},
|
||||
"run_id": {},
|
||||
"artifact_group": {},
|
||||
"batch_output_name": {},
|
||||
"bundle_id": {},
|
||||
}
|
||||
|
||||
func RenderDistributorBundleID(template string, values DistributorTemplateValues) (string, error) {
|
||||
return renderDistributorTemplate("notify.distributor.bundle_id_template", template, values, distributorTemplateVariables)
|
||||
}
|
||||
|
||||
func RenderDistributorIdempotencyKey(template string, values DistributorTemplateValues) (string, error) {
|
||||
return renderDistributorTemplate("notify.distributor.idempotency_key_template", template, values, distributorIdempotencyTemplateVariables)
|
||||
}
|
||||
|
||||
func RenderDistributorReportPath(template string, values DistributorTemplateValues) (string, error) {
|
||||
rendered, err := renderDistributorTemplate("notify.distributor.report_path_template", template, values, distributorTemplateVariables)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := ValidateDistributorReportPath(rendered); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return rendered, nil
|
||||
}
|
||||
|
||||
func validateDistributorTemplate(name, template string, allowed map[string]struct{}) error {
|
||||
_, err := renderDistributorTemplate(name, template, DistributorTemplateValues{}, allowed)
|
||||
return err
|
||||
}
|
||||
|
||||
func renderDistributorTemplate(name, template string, values DistributorTemplateValues, allowed map[string]struct{}) (string, error) {
|
||||
var rendered strings.Builder
|
||||
for i := 0; i < len(template); {
|
||||
switch template[i] {
|
||||
case '{':
|
||||
end := strings.IndexByte(template[i+1:], '}')
|
||||
if end < 0 {
|
||||
return "", fmt.Errorf("%s contains an unclosed template variable", name)
|
||||
}
|
||||
variable := template[i+1 : i+1+end]
|
||||
if variable == "" {
|
||||
return "", fmt.Errorf("%s contains an empty template variable", name)
|
||||
}
|
||||
if _, ok := allowed[variable]; !ok {
|
||||
return "", fmt.Errorf("%s contains unknown template variable %q", name, variable)
|
||||
}
|
||||
rendered.WriteString(distributorTemplateValue(variable, values))
|
||||
i += end + 2
|
||||
case '}':
|
||||
return "", fmt.Errorf("%s contains an unopened template variable", name)
|
||||
default:
|
||||
rendered.WriteByte(template[i])
|
||||
i++
|
||||
}
|
||||
}
|
||||
return rendered.String(), nil
|
||||
}
|
||||
|
||||
func distributorTemplateValue(variable string, values DistributorTemplateValues) string {
|
||||
switch variable {
|
||||
case "location_id":
|
||||
return values.LocationID
|
||||
case "report_id":
|
||||
return values.ReportID
|
||||
case "run_id":
|
||||
return values.RunID
|
||||
case "artifact_group":
|
||||
return values.ArtifactGroup
|
||||
case "batch_output_name":
|
||||
return values.BatchOutputName
|
||||
case "bundle_id":
|
||||
return values.BundleID
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
func ValidateDistributorReportPath(path string) error {
|
||||
if path == "" {
|
||||
return fmt.Errorf("notify.distributor.report_path_template renders an empty path")
|
||||
}
|
||||
if isDistributorAbsolutePath(path) {
|
||||
return fmt.Errorf("notify.distributor.report_path_template must render a relative path")
|
||||
}
|
||||
if strings.Contains(path, "\\") {
|
||||
return fmt.Errorf("notify.distributor.report_path_template must not render backslashes")
|
||||
}
|
||||
|
||||
segments := strings.Split(path, "/")
|
||||
for _, segment := range segments {
|
||||
if segment == "" {
|
||||
return fmt.Errorf("notify.distributor.report_path_template must not render empty path segments")
|
||||
}
|
||||
if segment == "." || segment == ".." {
|
||||
return fmt.Errorf("notify.distributor.report_path_template must not render . or .. path segments")
|
||||
}
|
||||
if segment == "manifest.json" || segment == ".distributor.json" {
|
||||
return fmt.Errorf("notify.distributor.report_path_template must not render reserved path segment %q", segment)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func isDistributorAbsolutePath(path string) bool {
|
||||
if filepath.IsAbs(path) || strings.HasPrefix(path, "/") {
|
||||
return true
|
||||
}
|
||||
if len(path) >= 3 && isASCIIAlpha(path[0]) && path[1] == ':' && (path[2] == '/' || path[2] == '\\') {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func isASCIIAlpha(ch byte) bool {
|
||||
return (ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z')
|
||||
}
|
||||
63
internal/config/secrets.go
Normal file
63
internal/config/secrets.go
Normal file
@@ -0,0 +1,63 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var secretNamePattern = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
|
||||
|
||||
func loadSecrets(cfg SecretsConfig) error {
|
||||
if cfg.Directory == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
entries, err := os.ReadDir(cfg.Directory)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read secrets directory %q: %w", cfg.Directory, err)
|
||||
}
|
||||
|
||||
for _, entry := range entries {
|
||||
name := entry.Name()
|
||||
if name == "" {
|
||||
return fmt.Errorf("secrets directory %q contains an empty filename", cfg.Directory)
|
||||
}
|
||||
if !secretNamePattern.MatchString(name) {
|
||||
return fmt.Errorf("secret file %q has invalid environment variable name", name)
|
||||
}
|
||||
if entry.Type()&os.ModeSymlink != 0 {
|
||||
return fmt.Errorf("secret file %q must be a regular file, not a symlink", name)
|
||||
}
|
||||
if entry.IsDir() {
|
||||
return fmt.Errorf("secret file %q must be a regular file, not a directory", name)
|
||||
}
|
||||
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
return fmt.Errorf("inspect secret file %q: %w", name, err)
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return fmt.Errorf("secret file %q must be a regular file", name)
|
||||
}
|
||||
|
||||
path := filepath.Join(cfg.Directory, name)
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read secret file %q: %w", name, err)
|
||||
}
|
||||
value := string(data)
|
||||
if strings.HasSuffix(value, "\r\n") {
|
||||
value = strings.TrimSuffix(value, "\r\n")
|
||||
} else {
|
||||
value = strings.TrimSuffix(value, "\n")
|
||||
}
|
||||
if err := os.Setenv(name, value); err != nil {
|
||||
return fmt.Errorf("set environment variable from secret file %q: %w", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -49,6 +49,10 @@ func Validate(cfg Config) error {
|
||||
}
|
||||
}
|
||||
|
||||
if err := validateDistributorNotify(cfg.Notify.Distributor); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if cfg.Scriptorium.Binary == "" {
|
||||
return fmt.Errorf("scriptorium.binary is required")
|
||||
}
|
||||
@@ -75,6 +79,56 @@ func Validate(cfg Config) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateDistributorNotify(cfg DistributorNotifyConfig) error {
|
||||
if !cfg.Enabled {
|
||||
return nil
|
||||
}
|
||||
|
||||
parsed, err := url.Parse(cfg.Endpoint)
|
||||
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
|
||||
return fmt.Errorf("notify.distributor.endpoint must be an absolute URL when enabled")
|
||||
}
|
||||
if cfg.TokenEnv == "" {
|
||||
return fmt.Errorf("notify.distributor.token_env is required when enabled")
|
||||
}
|
||||
if !secretNamePattern.MatchString(cfg.TokenEnv) {
|
||||
return fmt.Errorf("notify.distributor.token_env must be a valid environment variable name")
|
||||
}
|
||||
if cfg.Timeout <= 0 {
|
||||
return fmt.Errorf("notify.distributor.timeout must be greater than zero when enabled")
|
||||
}
|
||||
if cfg.FailurePolicy != NotifyFailureError {
|
||||
return fmt.Errorf("notify.distributor.failure_policy must be error when enabled")
|
||||
}
|
||||
if cfg.BundleIDTemplate == "" {
|
||||
return fmt.Errorf("notify.distributor.bundle_id_template is required when enabled")
|
||||
}
|
||||
if err := validateDistributorTemplate("notify.distributor.bundle_id_template", cfg.BundleIDTemplate, distributorTemplateVariables); err != nil {
|
||||
return err
|
||||
}
|
||||
if cfg.IdempotencyKeyTemplate == "" {
|
||||
return fmt.Errorf("notify.distributor.idempotency_key_template is required when enabled")
|
||||
}
|
||||
if err := validateDistributorTemplate("notify.distributor.idempotency_key_template", cfg.IdempotencyKeyTemplate, distributorIdempotencyTemplateVariables); err != nil {
|
||||
return err
|
||||
}
|
||||
if cfg.ReportPathTemplate == "" {
|
||||
return fmt.Errorf("notify.distributor.report_path_template is required when enabled")
|
||||
}
|
||||
values := DistributorTemplateValues{
|
||||
LocationID: "location",
|
||||
ReportID: "report",
|
||||
RunID: "run",
|
||||
ArtifactGroup: "artifact",
|
||||
BatchOutputName: "report.md",
|
||||
}
|
||||
if _, err := RenderDistributorReportPath(cfg.ReportPathTemplate, values); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func validatePolicy(name string, policy MissingSourcePolicy) error {
|
||||
switch policy {
|
||||
case MissingSourceError, MissingSourceWarn, MissingSourceNone:
|
||||
|
||||
Reference in New Issue
Block a user