17 Commits

Author SHA1 Message Date
d8b417458b Add a roadmap and a staged implementation plan to move weatherreporter toward deterministic, reusable briefing modules that can
be composed per report type
2026-06-09 15:09:31 -05:00
195a130124 Updated defaults to upload only the current generated report path 2026-06-09 11:26:50 -05:00
8577fc29e4 Updated the distributor bundle path template 2026-06-08 10:29:42 -05:00
d71c7e4d28 Implement the distributor v0.5 PipelineID update 2026-06-08 07:04:31 -05:00
9bc8156615 Update default config example to use a stable distributor bundle_id and a unique per-run idempotency_key 2026-06-07 20:46:19 -05:00
183b23cf5a Implemented debug artifacts for the distributor notification adapter 2026-06-07 20:21:26 -05:00
138bc4e7e4 Refresh distributor documentation 2026-06-07 23:50:55 +00:00
e2529cfdf2 Validate distributor notification implementation 2026-06-07 23:47:47 +00:00
b982b27f84 Document distributor notification behavior 2026-06-07 23:46:39 +00:00
c573cd5b4d Report distributor notification outcomes in batches 2026-06-07 23:42:27 +00:00
c2758d7a91 Notify distributor after report generation 2026-06-07 23:38:03 +00:00
64cae8c4d9 Add distributor upload adapter 2026-06-07 23:34:21 +00:00
a2ba6f5382 Add distributor notification config validation 2026-06-07 23:18:39 +00:00
7c8d9191c1 Add file-backed environment secrets 2026-06-07 23:13:54 +00:00
9677835d84 Add an implementation plan to support distributor uploads for produced artifacts 2026-06-07 18:04:40 -05:00
63749a9572 Implement support for NWS weather stories 2026-05-30 07:47:49 -05:00
9ff90d33fc Add Woodpecker CI support 2026-05-30 07:47:13 -05:00
49 changed files with 5327 additions and 129 deletions

50
.woodpecker/release.yml Normal file
View File

@@ -0,0 +1,50 @@
when:
- event: tag
steps:
- name: build-release-assets
image: golang:1.25
commands:
- |
set -eu
version="$CI_COMMIT_TAG"
dist="dist"
pkg="gitea.maximumdirect.net/eric/weatherreporter/cmd/weatherreporter"
rm -rf "$dist"
mkdir -p "$dist"
build_binary() {
goos="$1"
goarch="$2"
suffix="$3"
output="$dist/weatherreporter-$version-$goos-$goarch$suffix"
CGO_ENABLED=0 GOOS="$goos" GOARCH="$goarch" \
go build -trimpath -ldflags "-s -w -X gitea.maximumdirect.net/eric/weatherreporter/internal/buildinfo.Version=$version" \
-o "$output" "$pkg"
}
build_binary linux amd64 ""
build_binary linux arm64 ""
build_binary darwin amd64 ""
build_binary darwin arm64 ""
build_binary windows amd64 ".exe"
build_binary windows arm64 ".exe"
- name: publish-release
image: woodpeckerci/plugin-release
depends_on:
- build-release-assets
settings:
api_key:
from_secret: GITEA_RELEASE_TOKEN
files:
- dist/weatherreporter-*
checksum: sha256
checksum-file: SHA256SUMS
checksum-flatten: true
file-exists: skip
overwrite: false
prerelease: false

View File

@@ -3,7 +3,8 @@
`weatherreporter` is a Go application for preparing human-facing weather `weatherreporter` is a Go application for preparing human-facing weather
reports from normalized forecast data. It builds structured briefing packages, reports from normalized forecast data. It builds structured briefing packages,
runs them through `scriptorium`, and keeps inspectable artifacts under a local runs them through `scriptorium`, and keeps inspectable artifacts under a local
workspace. workspace. It can also upload successfully generated managed Markdown reports
to a configured `distributor` HTTP upload endpoint.
## Quickstart ## Quickstart

View File

@@ -11,7 +11,9 @@ weatherreporter generate daily --date 2026-05-29 --out ./daily.md
This loads configuration, fetches weather data, writes managed workspace This loads configuration, fetches weather data, writes managed workspace
artifacts, runs `scriptorium render` as a preflight check, runs artifacts, runs `scriptorium render` as a preflight check, runs
`scriptorium run`, and writes an extra Markdown copy to `./daily.md`. `scriptorium run`, and writes an extra Markdown copy to `./daily.md`. If
distributor notification is enabled in configuration, the command also uploads
the managed Markdown report after final metadata is saved.
## Commands ## Commands
@@ -33,14 +35,20 @@ weatherreporter inspect sources [--config PATH] RUN_ID
``` ```
`generate` commands write briefing, data package, preflight, report, and `generate` commands write briefing, data package, preflight, report, and
metadata artifacts under the configured workspace. `generate storm` requires metadata artifacts under the configured workspace. `--out` writes an extra
explicit event-window bounds with `--start` and `--end`. Markdown copy for the operator; distributor notification uses the managed
report path, not the extra copy. `generate storm` requires explicit
event-window bounds with `--start` and `--end`.
`run morning` generates Daily Today and the 3-Day Outlook, plus Weekend Outlook `run morning` generates Daily Today and the 3-Day Outlook, plus Weekend Outlook
except on Sunday. `run evening` generates the Tomorrow Planning Brief. Batch except on Sunday. `run evening` generates the Tomorrow Planning Brief. Batch
runs continue independent reports after a failure, print a JSON summary to runs continue independent reports after a failure, print a JSON summary to
stdout, write compact status lines to stderr, and return nonzero when any report stdout, write compact status lines to stderr, and return nonzero when any report
failed. failed. `--out-dir` writes extra Markdown copies for the operator; distributor
notification uses each managed report path, not the extra copies. When
notification is enabled, batch summaries and status lines include notification
status, accepted distributor run ID, or notification error fields for each
attempted report.
`inspect` commands read existing workspace artifacts and emit JSON to stdout. `inspect` commands read existing workspace artifacts and emit JSON to stdout.
They do not fetch weather data or invoke `scriptorium`. They do not fetch weather data or invoke `scriptorium`.
@@ -61,6 +69,9 @@ They do not fetch weather data or invoke `scriptorium`.
Storm times accept `YYYY-MM-DDTHH:MM` in the configured timezone or RFC3339 Storm times accept `YYYY-MM-DDTHH:MM` in the configured timezone or RFC3339
timestamps with explicit offsets. timestamps with explicit offsets.
Distributor notification is configured only through `notify.distributor`; there
are no distributor-specific CLI flags.
## Common Workflows ## Common Workflows
```sh ```sh

View File

@@ -18,7 +18,7 @@ Precedence is:
The implemented configuration overrides are `--units` and `--tz`. Output flags The implemented configuration overrides are `--units` and `--tz`. Output flags
control report copies for the current command but do not change configuration control report copies for the current command but do not change configuration
files. Environment-variable configuration is not implemented. files. Environment variables do not override configuration fields.
## Minimal Config ## Minimal Config
@@ -64,6 +64,66 @@ multiple configured forecast locations.
The prompt-facing location object also includes `timezone`, derived from the The prompt-facing location object also includes `timezone`, derived from the
effective `weather_api.timezone` after CLI overrides such as `--tz`. effective `weather_api.timezone` after CLI overrides such as `--tz`.
### `secrets`
- `directory`: optional directory of file-backed environment secrets. Default:
empty, which disables secret loading.
When configured, each regular file directly under `secrets.directory` is loaded
after config file parsing and CLI overrides. The file basename must be a valid
environment variable name matching `[A-Za-z_][A-Za-z0-9_]*`; the file contents
become the environment variable value and overwrite any existing value. One
trailing LF or CRLF is stripped. Subdirectories, symlinks, invalid filenames,
missing directories, and unreadable files fail config loading.
### `notify`
`notify.distributor` controls distributor notification after successful report
generation. It is disabled by default and does not add CLI flags. When enabled,
weatherreporter uploads one distributor bundle per generated report after
`scriptorium run` succeeds and final metadata is saved.
- `enabled`: whether distributor notification config is active. Default:
`false`.
- `endpoint`: absolute distributor endpoint URL. Required when enabled.
Default: `https://distributor.example.com`.
- `token_env`: environment variable name that will contain the distributor
upload token. Required when enabled. Default: `DISTRIBUTOR_UPLOAD_TOKEN`.
- `timeout`: distributor operation timeout. Must be greater than zero when
enabled. Default: `30s`.
- `failure_policy`: must be `error` when enabled. Default: `error`.
- `pipeline_id_template`: template for the distributor pipeline ID. Required
when enabled. Default: empty.
- `bundle_id_template`: template for distributor bundle IDs. Default:
`weatherreporter.{location_id}.{report_id}`.
- `idempotency_key_template`: template for distributor idempotency keys.
Default: `{bundle_id}.{run_id}`.
- `report_path_templates`: ordered list of templates for Markdown report paths
inside the distributor bundle. Each rendered path maps to the same managed
Markdown report source. Default:
```yaml
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
```
Supported template variables are `location_id`, `report_id`, `run_id`,
`artifact_group`, `batch_output_name`, `valid_start_date`, `valid_end_date`,
`valid_start_time`, `valid_end_time`, `valid_start_stamp`, and
`valid_end_stamp`. Date values use `YYYY-MM-DD`, time values use `HHMM`, and
stamp values use `YYYY-MM-DDTHHMM` in the effective report timezone.
`pipeline_id_template` and `idempotency_key_template` may also use `bundle_id`.
The rendered pipeline ID selects the configured distributor `http_upload`
workflow. The rendered bundle ID is the stable logical source identity for the
report stream. The rendered idempotency key is the per-run retry identity.
Rendered report paths must be unique relative paths with `/` separators. They
must not contain backslashes, empty path segments, `.`, `..`, `manifest.json`,
or `.distributor.json`.
The upload token is read from the environment variable named by `token_env`
after config loading and `secrets.directory` processing. Config files should
name the variable only; they should not contain the token value.
### `missing_source` ### `missing_source`
- `default`: missing-source behavior for optional sources. One of `error`, `warn`, or `none`. Default: `warn`. - `default`: missing-source behavior for optional sources. One of `error`, `warn`, or `none`. Default: `warn`.
@@ -87,6 +147,7 @@ stub source slots use the missing-source policy.
- `reports_dir`: managed Markdown report directory under `workspace.root`. Default: `reports`. - `reports_dir`: managed Markdown report directory under `workspace.root`. Default: `reports`.
- `data_packages_dir`: prompt input package directory under `workspace.root`. Default: `data-packages`. - `data_packages_dir`: prompt input package directory under `workspace.root`. Default: `data-packages`.
- `preflight_dir`: Scriptorium render output directory under `workspace.root`. Default: `preflight`. - `preflight_dir`: Scriptorium render output directory under `workspace.root`. Default: `preflight`.
- `notifications_dir`: distributor notification debug artifact directory under `workspace.root`. Default: `notifications`.
Workspace subdirectories must be relative paths that stay inside Workspace subdirectories must be relative paths that stay inside
`workspace.root`. `workspace.root`.
@@ -115,8 +176,13 @@ snapshot exists and a threshold is crossed.
## Secrets ## Secrets
Configuration files should not contain secrets. The current Weather API and Configuration files should not contain raw secrets. Use `secrets.directory` to
Scriptorium integration settings do not require secret fields. load secret values from files into environment variables for integrations that
read credentials from the environment. Secret file names become environment
variable names, and secret file contents become values. For distributor
notification, this allows a file such as
`<secrets.directory>/DISTRIBUTOR_UPLOAD_TOKEN` to supply the token referenced by
`notify.distributor.token_env`.
## Maintained Examples ## Maintained Examples

View File

@@ -0,0 +1,136 @@
# Upstream Producer Integration
Audience: developers and LLM coding agents adding `distributor` support to an upstream Go producer application.
This document is the copyable implementation guide for submitting producer outputs to a `distributor` pipeline whose source backend is `http_upload`.
## Required Inputs
The upstream application needs these values from deployment or operator configuration:
- distributor endpoint: the HTTP server base URL, such as `https://distributor.example.com`;
- upload token: bearer token that authenticates the producer;
- pipeline id: configured `http_upload` pipeline that should process this upload;
- generated files: regular local files to include in the source bundle;
- bundle id: stable identifier for the logical report stream or artifact;
- idempotency key: unique key for one producer run, reused only when retrying that same run.
Do not put destination routing, public URLs, transform settings, or credentials in the source manifest. Those belong in the `distributor` pipeline configuration.
The token, pipeline id, bundle id, and idempotency key have different jobs. The token authenticates the producer. The pipeline id selects the configured distributor workflow, including destinations and publishing policy. The bundle id tells `distributor` whether a new upload is a newer version of the same source; keep it stable across runs that should replace the same managed destination artifact. The idempotency key tells `distributor` whether an upload request is a retry; change it for each distinct producer run so new content is enqueued.
## Recommended Workflow
Use `gitea.maximumdirect.net/eric/distributor/pkg/upload`.
For most producers, use `UploadFiles`. It accepts producer-generated files, builds a temporary valid source bundle with `pkg/bundle`, uploads a gzip-compressed tar archive, and removes temporary files when the call returns.
Use `UploadBundle` only when the producer already assembled a complete bundle directory containing `manifest.json`.
Add the dependency from the upstream application:
```sh
go get gitea.maximumdirect.net/eric/distributor
```
## Minimal Go Example
```go
package reports
import (
"context"
"errors"
"fmt"
"os"
"time"
"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
"gitea.maximumdirect.net/eric/distributor/pkg/upload"
)
func SubmitReport(reportPath, summaryPath string) error {
endpoint := os.Getenv("DISTRIBUTOR_UPLOAD_ENDPOINT")
token := os.Getenv("DISTRIBUTOR_UPLOAD_TOKEN")
if endpoint == "" || token == "" {
return fmt.Errorf("distributor endpoint and token are required")
}
pipelineID := "weather-hourly"
reportID := "weather.hourly.brentwood"
runID := time.Now().UTC().Format("20060102T150405.000000000Z")
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
client, err := upload.NewClient(upload.ClientOptions{
Endpoint: endpoint,
Token: token,
})
if err != nil {
return err
}
result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
PipelineID: pipelineID,
ID: reportID,
IdempotencyKey: reportID + "." + runID,
Files: []bundle.BundleFile{
{SourcePath: reportPath, Path: "report.md"},
{SourcePath: summaryPath, Path: "summary.txt"},
},
})
if err != nil {
var conflict *upload.IdempotencyConflictError
if errors.As(err, &conflict) {
return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
}
return err
}
fmt.Printf("distributor accepted run %s\n", result.RunID)
return nil
}
```
## Producer Responsibilities
- Use a stable bundle id for the logical producer output that should replace the same destination artifact, such as `weather.hourly.brentwood`.
- Set `PipelineID` to the configured upload pipeline that should process the bundle.
- Do not include per-run timestamps, random values, or job ids in the bundle id unless each run should be treated as a different source.
- Use an idempotency key that changes for every distinct producer run, such as `<bundle-id>.<run-id>`.
- Reuse the same idempotency key only when retrying the exact same producer run with the same source manifest.
- Map each generated file to a clean slash-separated bundle path, such as `report.md` or `assets/chart.png`.
- Include only regular files. Symlinks, directories as files, devices, FIFOs, and sockets are rejected.
- Keep file contents stable after upload inputs are selected. Bundle digests are calculated from file bytes.
- Treat upload success as admission only. `UploadFiles` and `UploadBundle` return after the server accepts and validates the upload, not after all destinations publish.
Valid bundle paths are relative slash paths. They must not be empty, absolute, contain backslashes, contain `.` or `..` path segments, contain empty path segments, or use reserved basenames `manifest.json` or `.distributor.json`.
## Idempotency And Status
`pkg/upload` sends `Idempotency-Key` on every upload. If the caller omits one, the package generates a random key for that call and reuses it for in-process retries. That is enough for transient network retry within one process, but it does not give cross-process retry identity.
For producer jobs that may retry after process restart, supply a key derived from the producer run, such as `<bundle-id>.<run-id>`. Reusing the same key with the same token, pipeline id, and normalized source manifest returns the original accepted run. Reusing the same key with different source content in that scope returns a conflict. Reusing one key across multiple distinct report generations prevents those generations from being treated as new uploads.
`Status` polls `/runs/<run-id>` while the distributor server retains the in-memory status record. Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to the server's `server.http.retention` setting, and server restart clears status and idempotency records.
Optional status check:
```go
status, err := client.Status(ctx, result.RunID)
if err != nil {
return err
}
if status.Status == "failed" {
return fmt.Errorf("distributor run failed: %s", status.Error)
}
```
## References
In the `distributor` source tree:
- `docs/consumers/pkg-upload.md`: Go upload package workflow.
- `docs/consumers/pkg-bundle.md`: Go bundle package workflow.
- `docs/integrations/http-upload.md`: canonical HTTP upload wire contract.
- `docs/integrations/source-bundle.md`: canonical source bundle file-format contract.

View File

@@ -0,0 +1,90 @@
# `pkg/bundle`
Audience: upstream Go producer developers and LLM coding agents using `distributor` source bundle helpers.
Import path:
```go
import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
```
`pkg/bundle` builds, writes, parses, and validates local source bundles. Use it directly when a producer writes bundles for `distributor` to discover, or when a producer wants to assemble and validate a bundle before using another transport.
The canonical source bundle file-format contract is [Source Bundle Contract](../integrations/source-bundle.md).
## Preferred Complete-Bundle Workflow
Use `WriteBundle` when producer-generated files live outside the final bundle root.
```go
manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{
Root: "/var/spool/distributor/weather/hourly-2026-06-07T15",
ID: "weather.hourly.brentwood",
Files: []bundle.BundleFile{
{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
},
})
if err != nil {
return err
}
_ = manifest
```
`WriteBundle` copies each source file into a staged bundle root, writes `manifest.json`, validates the staged bundle, and promotes it into place. Set `Overwrite: true` only when the producer intentionally replaces an existing bundle root.
## Existing Bundle Root Workflow
Use `BuildManifest` and `WriteManifest` when files are already staged under the final bundle root.
```go
root := "/var/spool/distributor/weather/hourly-2026-06-07T15"
manifest, err := bundle.BuildManifest(bundle.BuildOptions{
Root: root,
ID: "weather.hourly.brentwood",
Files: []string{"report.md", "summary.txt"},
})
if err != nil {
return err
}
if err := bundle.WriteManifest(root, manifest, bundle.WriteManifestOptions{}); err != nil {
return err
}
if err := bundle.ValidateBundle(root, manifest); err != nil {
return err
}
```
Use `Scan: true` instead of `Files` only when every valid regular file under the root should be included. Scan mode includes dotfiles, skips reserved metadata files, rejects symlinks, and sorts paths lexically.
## Paths And Ordering
Bundle paths are slash-separated paths relative to the bundle root.
Invalid paths include:
- empty paths;
- absolute paths;
- paths containing backslashes;
- `.` or `..` path segments;
- empty path segments;
- any basename of `manifest.json` or `.distributor.json`.
Explicit file lists preserve caller order. File order is part of the bundle digest, so producers should choose it deliberately and keep it stable.
The manifest `ID` is the logical source identity used by `distributor` destination comparison. Keep it stable for runs that should replace the same managed destination artifact. If every run uses a different manifest `ID`, `distributor` treats those runs as different sources and may report a destination conflict instead of replacing older output.
## Validation And Digest Helpers
Use `ValidateBundle` before handing an existing local bundle to another process. It verifies manifest semantics, file existence, regular-file type, file size, per-file SHA-256 digests, and bundle digest.
Useful helpers:
- `LoadManifest`: read `manifest.json` from a bundle root.
- `ParseManifest` and `MarshalManifest`: parse or write manifest bytes.
- `ValidateManifest`: validate manifest-only semantics.
- `FileDigest`, `BundleDigest`, and `ValidateDigest`: digest helpers for diagnostics and tests.
## Boundaries
`pkg/bundle` does not upload bundles, publish destinations, transform Markdown, select pipelines, configure credentials, or write destination state. Those concerns belong to `pkg/upload` or the `distributor` application.

View File

@@ -0,0 +1,122 @@
# `pkg/upload`
Audience: upstream Go producer developers and LLM coding agents submitting bundles to `distributor serve`.
Import path:
```go
import "gitea.maximumdirect.net/eric/distributor/pkg/upload"
```
`pkg/upload` is the producer-facing HTTP upload client. It builds on `pkg/bundle`, packages valid source bundles as gzip-compressed tar archives, sends bearer authentication, routes uploads to a configured pipeline, includes idempotency keys, and exposes a status polling helper.
`UploadFiles` examples also use:
```go
import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
```
The canonical HTTP wire contract is [HTTP Upload API Contract](../integrations/http-upload.md).
## Client Construction
```go
client, err := upload.NewClient(upload.ClientOptions{
Endpoint: "https://distributor.example.com",
Token: token,
})
if err != nil {
return err
}
```
`Endpoint` is the distributor server base URL. The client derives `/v1/pipelines/<pipeline-id>/upload` and `/runs/<run-id>`. `Token` is required and is sent as `Authorization: Bearer <token>`. Token values are redacted from client errors.
`HTTPClient` and `Retry` are optional. Defaults use a 30 second HTTP timeout and safe retry settings.
## Upload Producer Files
Use `UploadFiles` when the producer has generated output files but has not assembled a bundle directory.
```go
result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
PipelineID: "weather-hourly",
ID: "weather.hourly.brentwood",
IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
Files: []bundle.BundleFile{
{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
},
})
if err != nil {
return err
}
_ = result.RunID
```
`PipelineID` is required and selects the configured distributor workflow for this upload. `ID` is the source manifest id and identifies the logical artifact inside that workflow. `UploadFiles` creates a temporary bundle, writes and validates a manifest, uploads the archive, and removes temporary files when the call returns. It does not write into producer source directories.
## Upload An Existing Bundle
Use `UploadBundle` when the producer already has a complete local bundle root containing `manifest.json`.
```go
result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
PipelineID: "weather-hourly",
Root: "/var/spool/weather/hourly-2026-06-07T15",
IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
})
if err != nil {
return err
}
_ = result.RunID
```
`PipelineID` is required for existing bundles too. `UploadBundle` validates the local bundle by default and uploads only `manifest.json` plus manifest-listed files. Unlisted files are not uploaded.
## Result And Status
Upload success means the server returned `202 Accepted` after staging and validating the upload. It does not mean all configured destinations have published.
Poll status while the server retains the in-memory run record:
```go
status, err := client.Status(ctx, result.RunID)
if err != nil {
return err
}
if status.Status == "failed" {
return fmt.Errorf("distributor run failed: %s", status.Error)
}
```
Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to `server.http.retention`; server restart clears run status and idempotency records.
## Idempotency And Retry
Every upload request includes `Idempotency-Key`.
If `IdempotencyKey` is omitted, the client generates a random 128-bit lowercase hexadecimal key for that upload operation and reuses it for retries within the same call. For cross-process retry safety, producers should pass a key derived from the producer run, such as `<bundle-id>.<run-id>`.
Do not reuse the same idempotency key for multiple distinct report generations. Reuse it only when retrying the exact same run with the same token, pipeline id, and source manifest. A repeated key with the same manifest in that scope returns the original accepted run instead of enqueueing another run; a repeated key with different content returns an idempotency conflict.
The client retries only safe cases:
- `503 Service Unavailable`;
- temporary network errors;
- ambiguous mid-upload failures.
It does not retry after `202 Accepted` and does not retry `400`, `401`, `403`, `404`, `409`, `413`, or `415`.
Detect conflicting key reuse with `errors.As`:
```go
var conflict *upload.IdempotencyConflictError
if errors.As(err, &conflict) {
return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
}
```
## Boundaries
`pkg/upload` does not configure server pipelines, choose destinations, wait for publication completion automatically, persist client queues, provide durable idempotency across server restarts, or expose destination state. It submits complete source bundles to the configured HTTP upload API.

View File

@@ -53,7 +53,8 @@ The adapter sends these query parameters:
- `tz`: from `weather_api.timezone` on hourly forecast, narrative forecast, and - `tz`: from `weather_api.timezone` on hourly forecast, narrative forecast, and
discussion requests discussion requests
Alerts do not receive `precision` or `tz`. Alerts do not receive `precision` or `tz`. Weather story requests receive only
`format=json`.
## Endpoints Used ## Endpoints Used
@@ -65,6 +66,7 @@ The adapter fetches these endpoints once per bundle:
- `/forecast/narrative` - `/forecast/narrative`
- `/alerts/active` - `/alerts/active`
- `/discussion` - `/discussion`
- `/weatherstories/latest`
`weatherreporter` does not call day-slice forecast endpoints or discussion `weatherreporter` does not call day-slice forecast endpoints or discussion
subsection endpoints. Report-period selection and daypart summarization happen subsection endpoints. Report-period selection and daypart summarization happen
@@ -86,10 +88,11 @@ source-specific `missing_source.sources` policy:
- `narrative` for `/forecast/narrative` - `narrative` for `/forecast/narrative`
- `alerts` for `/alerts/active` - `alerts` for `/alerts/active`
- `discussion` for `/discussion` - `discussion` for `/discussion`
- `weather_story` for `/weatherstories/latest`
The adapter also creates missing stub source records for `daily` and The adapter also creates a missing stub source record for `daily` because that
`weather_story` because those source slots exist in the internal bundle but are source slot exists in the internal bundle but is not fetched from the Weather
not fetched from the Weather API. API.
Policy behavior: Policy behavior:
@@ -126,6 +129,8 @@ types in `internal/forecast/bundle.go`, including:
- forecast run metadata and `periods` - forecast run metadata and `periods`
- active alert run data - active alert run data
- discussion metadata, key messages, and short/long-term section text - discussion metadata, key messages, and short/long-term section text
- latest weather story title, description, timing, priority, order, alt text,
and download URL
The adapter intentionally keeps upstream transport and envelope details inside The adapter intentionally keeps upstream transport and envelope details inside
`internal/adapters/weatherapi`; downstream packages consume the normalized `internal/adapters/weatherapi`; downstream packages consume the normalized

View File

@@ -7,8 +7,9 @@ This document describes the implemented workflow coordinator in `internal/app`.
`internal/app` coordinates the top-level use cases after CLI parsing and config `internal/app` coordinates the top-level use cases after CLI parsing and config
loading are complete. It resolves report definitions, fetches weather data, loading are complete. It resolves report definitions, fetches weather data,
builds briefing and prompt-input artifacts, invokes Scriptorium through the builds briefing and prompt-input artifacts, invokes Scriptorium through the
adapter boundary, persists managed state, runs batches, and reads existing adapter boundary, optionally notifies distributor through an app-owned notifier
artifacts for inspection. boundary, persists managed state, runs batches, and reads existing artifacts for
inspection.
## Inputs And Outputs ## Inputs And Outputs
@@ -22,13 +23,15 @@ Inputs:
- resolved report definitions from `internal/report` - resolved report definitions from `internal/report`
- forecast bundles from `internal/adapters/weatherapi` - forecast bundles from `internal/adapters/weatherapi`
- prior snapshots loaded from `internal/state` - prior snapshots loaded from `internal/state`
- optional renderer and state-store fakes for tests - optional renderer, notifier, and state-store fakes for tests
Outputs: Outputs:
- generated report results with briefing, data package, preflight, report, - generated report results with briefing, data package, preflight, report,
metadata, prior snapshot, Recent Changes, and Scriptorium result details metadata, prior snapshot, Recent Changes, Scriptorium result details, and
- batch summaries with per-report status, artifact paths, and error text notification result when attempted
- batch summaries with per-report status, artifact paths, error text, and
notification outcome when attempted
- saved Weather API bundle JSON for fetch workflows - saved Weather API bundle JSON for fetch workflows
- inspection JSON values for reports, metadata, briefings, data packages, prior - inspection JSON values for reports, metadata, briefings, data packages, prior
snapshots, and source provenance snapshots, and source provenance
@@ -42,8 +45,9 @@ Scriptorium argv.
Report selection and report identity policy come from `internal/report`. Report selection and report identity policy come from `internal/report`.
Weather API transport stays in `internal/adapters/weatherapi`. Scriptorium Weather API transport stays in `internal/adapters/weatherapi`. Scriptorium
subprocess behavior stays in `internal/adapters/scriptorium`. Filesystem layout subprocess behavior stays in `internal/adapters/scriptorium`. Distributor
and persisted metadata stay in `internal/state`. upload behavior stays in `internal/adapters/distributor`. Filesystem layout and
persisted metadata stay in `internal/state`.
## Config Fields Used ## Config Fields Used
@@ -52,6 +56,7 @@ and persisted metadata stay in `internal/state`.
- `workspace.*` for filesystem state - `workspace.*` for filesystem state
- `dayparts` for daily and outlook summarization - `dayparts` for daily and outlook summarization
- `recent_change.*` for structured Recent Changes thresholds - `recent_change.*` for structured Recent Changes thresholds
- `notify.distributor.*` for optional notification after report generation
Output copy flags are command request fields. They are not configuration Output copy flags are command request fields. They are not configuration
defaults. defaults.
@@ -74,11 +79,21 @@ Single-report generation follows this order:
12. Run Scriptorium report generation to the managed report path. 12. Run Scriptorium report generation to the managed report path.
13. Copy the managed report to the requested `--out` path when provided. 13. Copy the managed report to the requested `--out` path when provided.
14. Save metadata with the managed report path. 14. Save metadata with the managed report path.
15. If distributor notification is enabled, notify using the managed report
path as the source file.
16. Save a distributor notification debug artifact and update metadata with its
path.
If render preflight returns both a result and an error, preflight JSON and If render preflight returns both a result and an error, preflight JSON and
metadata are persisted before the error is returned. If Scriptorium report metadata are persisted before the error is returned. If Scriptorium report
generation returns an error after writing output, the managed report and generation returns an error after writing output, the managed report and
metadata remain inspectable. metadata remain inspectable. Notification is not attempted after Weather API,
briefing, prompt input, render, Scriptorium run, or metadata-save failures.
When notification is attempted, the debug artifact records request identity,
including rendered pipeline ID, bundle paths, accepted upload fields,
distributor status fields, raw status report JSON when available, and redacted
failure context.
`--out` copies are never used as notification source files.
## Batch Workflow ## Batch Workflow
@@ -87,6 +102,10 @@ on Sunday. `run evening` resolves Daily Tomorrow. Batch output copy names come
from report definitions. Batch generation continues independent reports after a from report definitions. Batch generation continues independent reports after a
failure, records each result, writes compact status lines to stderr, emits a failure, records each result, writes compact status lines to stderr, emits a
JSON summary to stdout, and returns an aggregate error when any report failed. JSON summary to stdout, and returns an aggregate error when any report failed.
When notification is enabled, each successfully generated report is notified
independently. Notification failure marks that report failed, records
notification fields in the batch result, and does not stop later reports.
`--out-dir` copies are never used as notification source files.
## Inspection Workflow ## Inspection Workflow
@@ -101,6 +120,8 @@ inspection view.
- Weather API and briefing errors stop that report before Scriptorium runs. - Weather API and briefing errors stop that report before Scriptorium runs.
- Prompt input validation fails before render preflight. - Prompt input validation fails before render preflight.
- Render and run errors preserve Scriptorium stderr and exit-code context. - Render and run errors preserve Scriptorium stderr and exit-code context.
- Notification errors are wrapped with report ID, RunID, and managed report path
context and are recorded separately in batch results.
- Metadata and artifact path errors include filesystem context. - Metadata and artifact path errors include filesystem context.
- Batch failures are recorded per report and surfaced through an aggregate - Batch failures are recorded per report and surfaced through an aggregate
batch error. batch error.
@@ -121,3 +142,5 @@ Inspect:
- Render preflight precedes Scriptorium report generation. - Render preflight precedes Scriptorium report generation.
- Recent Changes are computed from structured briefing snapshots. - Recent Changes are computed from structured briefing snapshots.
- Metadata links artifacts produced for a run. - Metadata links artifacts produced for a run.
- Distributor notification maps the managed Markdown report path to configured
bundle paths; extra output copies are not upload sources.

View File

@@ -24,6 +24,8 @@ Outputs:
object for Daily, 3-Day, Weekend, or Storm Report object for Daily, 3-Day, Weekend, or Storm Report
- optional `currentConditions` prompt context from normalized - optional `currentConditions` prompt context from normalized
`/conditions/current` data when available `/conditions/current` data when available
- optional structured `weatherStory` context on report-specific briefing
objects when `/weatherstories/latest` is available
- optional JSON file written by `briefing.Save` - optional JSON file written by `briefing.Save`
## Boundaries ## Boundaries

View File

@@ -0,0 +1,123 @@
# Distributor Adapter Internals
This document describes the distributor upload adapter in
`internal/adapters/distributor`.
## Purpose
The adapter submits generated weatherreporter Markdown reports to a configured
distributor HTTP upload endpoint. It isolates distributor package types,
token-env lookup, upload client construction, source-bundle file mapping,
timeout handling, status polling, and upload error wrapping from app
orchestration.
## Inputs And Outputs
Inputs:
- distributor endpoint URL
- token environment variable name
- upload timeout
- pipeline ID
- bundle ID
- idempotency key
- source Markdown report path and bundle-relative path mappings
- bundle created timestamp
- context for cancellation
Outputs:
- accepted distributor run ID
- accepted distributor upload status
- distributor run status, status polling error, and raw run report JSON when available
- weatherreporter-owned idempotency conflict error when applicable
## Boundaries
`internal/adapters/distributor` is the only weatherreporter package that imports
`gitea.maximumdirect.net/eric/distributor/pkg/upload` or
`gitea.maximumdirect.net/eric/distributor/pkg/bundle`.
The app layer passes weatherreporter-owned request values to the adapter. The
adapter does not choose report types, render templates, select output copies,
configure destinations, wait for downstream publication, transform Markdown, or
persist notification state.
Full upstream distributor package and HTTP contract details stay under
`docs/integrations/distributor/`.
## Config Fields Used
The adapter is built from `notify.distributor` config:
- `endpoint`
- `token_env`
- `timeout`
The app layer renders pipeline ID, bundle ID, idempotency key, and bundle paths
from:
- `pipeline_id_template`
- `bundle_id_template`
- `idempotency_key_template`
- `report_path_templates`
The token value is read from the environment variable named by `token_env`
after config loading and `secrets.directory` processing.
## Upload Behavior
The adapter calls distributor `UploadFiles` with one or more file mappings:
- pipeline ID: the rendered distributor workflow selector
- source path: the managed Markdown report path selected by app orchestration
- bundle paths: rendered bundle-relative report paths
- created: the report generation timestamp
The adapter creates a distributor upload client with the configured endpoint,
bearer token, and timeout-backed HTTP client. It also wraps the upload context
with the configured timeout when the timeout is greater than zero.
After upload acceptance, the adapter polls distributor `Status` for the accepted
run ID until the run reaches `succeeded` or `failed`, or until the configured
timeout expires. It returns the latest status, error text, and raw report JSON in
weatherreporter-owned types so app orchestration can persist them in the
notification debug artifact. Status lookup failures or timeout before a terminal
state are kept as debug status errors on an otherwise accepted upload. A
terminal distributor run status of `failed` is returned as a notification failure
with the status report preserved.
## Failure Behavior
The adapter validates required endpoint, token env name, token value, pipeline
ID, bundle ID, idempotency key, upload files, source paths, bundle paths, and
upload client inputs before uploading.
Upload failures include endpoint, pipeline ID, bundle ID, idempotency key,
source paths, and bundle paths context. Token values are redacted from adapter
errors.
Distributor idempotency conflicts are exposed as a weatherreporter-owned
`IdempotencyConflictError`, so callers do not depend on upstream distributor
types.
## Tests
Inspect:
- `internal/adapters/distributor/client_test.go`
- `internal/app/app_test.go`
- `internal/cli/root_test.go`
Adapter tests use a fake upload client factory and do not require a live
distributor service.
## Invariants
- Distributor package types do not leak outside the adapter.
- Only the managed Markdown report is uploaded.
- The adapter never scans the workspace.
- Token values are not included in errors, CLI output, metadata, docs, or
examples.
- Destination routing and Markdown-to-HTML transformation belong to
distributor, not weatherreporter.

View File

@@ -19,8 +19,8 @@ Outputs:
- `promptinput.Package` containing schema version, RunID, report metadata, - `promptinput.Package` containing schema version, RunID, report metadata,
briefing content, Recent Changes, and source warnings. Briefing content briefing content, Recent Changes, and source warnings. Briefing content
includes configured location context, current conditions when available, includes configured location context, current conditions when available,
discussion key messages, and short/long-term AFD narratives when the Weather structured weather story context when available, discussion key messages, and
API provides them. short/long-term AFD narratives when the Weather API provides them.
- report metadata includes `currentLocalDate`, the generation date formatted as - report metadata includes `currentLocalDate`, the generation date formatted as
`YYYY-MM-DD` in the effective report timezone. `YYYY-MM-DD` in the effective report timezone.
- optional JSON file written by `promptinput.Save` - optional JSON file written by `promptinput.Save`

View File

@@ -47,6 +47,7 @@ converts adapter render results into that shape before saving.
- `workspace.reports_dir` - `workspace.reports_dir`
- `workspace.data_packages_dir` - `workspace.data_packages_dir`
- `workspace.preflight_dir` - `workspace.preflight_dir`
- `workspace.notifications_dir`
Workspace subdirectories must be relative paths that stay under Workspace subdirectories must be relative paths that stay under
`workspace.root`. `workspace.root`.
@@ -62,12 +63,14 @@ valid-period start date for JSON artifacts, and the RunID.
snapshots/<artifact_group>/<YYYY-MM-DD>/<run_id>.metadata.json snapshots/<artifact_group>/<YYYY-MM-DD>/<run_id>.metadata.json
data-packages/<artifact_group>/<YYYY-MM-DD>/<run_id>.data_package.json data-packages/<artifact_group>/<YYYY-MM-DD>/<run_id>.data_package.json
preflight/<artifact_group>/<YYYY-MM-DD>/<run_id>.render.json preflight/<artifact_group>/<YYYY-MM-DD>/<run_id>.render.json
notifications/<artifact_group>/<YYYY-MM-DD>/<run_id>.distributor.json
reports/<artifact_group>/<run_id>.md reports/<artifact_group>/<run_id>.md
``` ```
Metadata is stored beside briefing snapshots and links the briefing, data Metadata is stored beside briefing snapshots and links the briefing, data
package, preflight, report paths, and configured prompt location. Report package, preflight, report paths, notification path when attempted, and
listing walks metadata files under the snapshots directory. configured prompt location. Report listing walks metadata files under the
snapshots directory.
## Prior Lookup ## Prior Lookup
@@ -89,6 +92,10 @@ current report definition.
Durable JSON writes use shared atomic file helpers. Managed Markdown reports are Durable JSON writes use shared atomic file helpers. Managed Markdown reports are
prepared by creating their parent directory; Scriptorium writes the report body prepared by creating their parent directory; Scriptorium writes the report body
to the prepared path. Extra Markdown copies are handled by app orchestration. to the prepared path. Extra Markdown copies are handled by app orchestration.
Distributor notification debug artifacts are written atomically when
notification is attempted and include rendered distributor pipeline ID, bundle
ID, idempotency key, bundle paths, upload status, latest run status, and
redacted errors.
Inspection helpers read existing metadata, briefing, and data package files. Inspection helpers read existing metadata, briefing, and data package files.
Missing metadata directories return no inspection records or no prior snapshot Missing metadata directories return no inspection records or no prior snapshot

View File

@@ -7,7 +7,7 @@ This document describes Weather API ingestion into `forecast.Bundle`.
`internal/adapters/weatherapi` fetches normalized weather data from one `internal/adapters/weatherapi` fetches normalized weather data from one
configured Weather API endpoint and assembles the bundle consumed by forecast configured Weather API endpoint and assembles the bundle consumed by forecast
derivation and briefing builders. Briefing builders expose normalized current derivation and briefing builders. Briefing builders expose normalized current
conditions as prompt context when `/conditions/current` is available. conditions and weather story context when those sources are available.
## Inputs And Outputs ## Inputs And Outputs
@@ -20,9 +20,9 @@ Inputs:
Outputs: Outputs:
- `forecast.Bundle` with observation, current conditions, hourly forecast, - `forecast.Bundle` with observation, current conditions, hourly forecast,
narrative forecast, active alerts, discussion, source records, and source narrative forecast, active alerts, discussion, latest weather story, source
warnings records, and source warnings
- stub source records for daily forecast and weather story source slots - stub source record for the daily forecast source slot
- optional saved bundle JSON through app fetch helpers - optional saved bundle JSON through app fetch helpers
## Boundaries ## Boundaries

View File

@@ -19,7 +19,10 @@ weatherreporter generate storm --start 2026-05-29T18:00 --end 2026-05-30T06:00
Each command resolves a report period, fetches a Weather API bundle, builds a Each command resolves a report period, fetches a Weather API bundle, builds a
briefing, builds a prompt input data package, runs `scriptorium render`, runs briefing, builds a prompt input data package, runs `scriptorium render`, runs
`scriptorium run`, and writes managed artifacts under the configured workspace. `scriptorium run`, and writes managed artifacts under the configured workspace.
`--out PATH` writes an extra Markdown copy for the current generated report. When distributor notification is enabled, weatherreporter uploads the managed
Markdown report after `scriptorium run` succeeds and final metadata is saved.
`--out PATH` writes an extra Markdown copy for the current generated report; it
is not used as the distributor upload source.
Implemented batch commands: Implemented batch commands:
@@ -32,9 +35,12 @@ weatherreporter run evening
except on Sunday. `run evening` generates the Tomorrow Planning Brief. Batch except on Sunday. `run evening` generates the Tomorrow Planning Brief. Batch
commands print a JSON summary to stdout, write compact per-report status lines commands print a JSON summary to stdout, write compact per-report status lines
to stderr, continue independent reports after one report fails, and return to stderr, continue independent reports after one report fails, and return
nonzero when any report failed. `--out-dir PATH` writes extra Markdown copies nonzero when any report failed. When notification is configured, the summary and
using report default filenames such as `daily.md`, `three-day.md`, status lines include notification status, accepted distributor run ID, or
`weekend.md`, and `tomorrow.md`. notification error fields for each attempted report. `--out-dir PATH` writes
extra Markdown copies using report default filenames such as `daily.md`,
`three-day.md`, `weekend.md`, and `tomorrow.md`; these copies are not used as
distributor upload sources.
## Filesystem Layout ## Filesystem Layout
@@ -85,6 +91,19 @@ workspace/
storm/ storm/
YYYY-MM-DD/ YYYY-MM-DD/
<run_id>.render.json <run_id>.render.json
notifications/
daily/
YYYY-MM-DD/
<run_id>.distributor.json
three-day/
YYYY-MM-DD/
<run_id>.distributor.json
weekend/
YYYY-MM-DD/
<run_id>.distributor.json
storm/
YYYY-MM-DD/
<run_id>.distributor.json
reports/ reports/
daily/ daily/
<run_id>.md <run_id>.md
@@ -116,9 +135,57 @@ Each generated report writes metadata that links:
- prompt input data package path - prompt input data package path
- preflight output path - preflight output path
- managed Markdown report path - managed Markdown report path
- distributor notification debug artifact path, when notification is attempted
Batch summaries include report status, error text when applicable, valid Batch summaries include report status, error text when applicable, notification
period, and known artifact paths for each attempted report. outcome when attempted, valid period, and known artifact paths for each
attempted report. Notification fields are `notificationStatus`,
`notificationRunId`, and `notificationError`.
## Distributor Notification
Distributor notification is configured with `notify.distributor` and is
disabled by default. When enabled, weatherreporter uploads the managed Markdown
report path recorded in the report result and metadata. That single source file
can be mapped to one or more configured bundle paths. By default, it is mapped
to one dated report path. Extra copies written by `--out` or `--out-dir` are
operator conveniences only.
The rendered pipeline ID selects the configured distributor `http_upload`
workflow. The default bundle ID is a stable logical source identity derived from
producer name, location ID, and report ID:
```text
weatherreporter.{location_id}.{report_id}
```
The default idempotency key appends RunID to the rendered bundle ID so each
report generation has a distinct retry identity. The default bundle path uses
the valid-period start date, artifact group, and RunID. Distributor owns
destination merge, retention, and derived snapshot behavior such as `latest`.
Notification happens after final metadata save. Weather API, briefing,
data-package, render preflight, Scriptorium run, and metadata-save failures do
not trigger notification. A notification failure fails that report. In a batch,
other reports continue, the failed report includes notification fields in the
JSON summary, and the batch returns nonzero.
Each notification attempt writes a debug artifact under `notifications/`. The
artifact records the rendered pipeline ID, bundle ID, idempotency key, managed
source path, bundle-relative paths, bundle created timestamp, accepted upload
response, and the latest distributor run status response when available.
Weatherreporter polls status until distributor reports `succeeded` or `failed`,
or until the configured notification timeout expires. The run status includes
the distributor status, error text, and raw run report JSON, which can show
actions such as `replace_older`, `skip_same`, `skip_destination_newer`, or
`failed`. Token values are not written.
Weatherreporter is responsible for selecting the managed Markdown report,
constructing a source bundle, and submitting it to the configured distributor
HTTP endpoint. Distributor remains responsible for destination routing,
publication, and any downstream Markdown-to-HTML transformation. Distributor
leaves destination files alone when they are not tracked by a newly uploaded
bundle, so previously uploaded dated report paths can remain available.
## Inspection ## Inspection
@@ -164,6 +231,8 @@ A failed generation run may still leave useful artifacts:
preflight JSON and metadata are written for inspection. preflight JSON and metadata are written for inspection.
- If `scriptorium run` exits nonzero after writing a report, the managed report - If `scriptorium run` exits nonzero after writing a report, the managed report
and metadata remain available. and metadata remain available.
- If distributor notification fails, report artifacts and final metadata remain
available, but the report or batch command returns nonzero.
- For batch commands, inspect the stdout JSON summary first, then inspect the - For batch commands, inspect the stdout JSON summary first, then inspect the
artifact paths for each failed report. artifact paths for each failed report.

View File

@@ -13,6 +13,12 @@ Generated reports must be associated with explicit metadata, including report ty
`scriptorium` is an external adapter, not domain logic. Subprocess execution must be isolated under `internal/adapters/scriptorium`, use context-aware execution, avoid shell interpolation, capture actionable stderr, and keep scriptorium-specific flags from leaking into domain packages. `scriptorium` is an external adapter, not domain logic. Subprocess execution must be isolated under `internal/adapters/scriptorium`, use context-aware execution, avoid shell interpolation, capture actionable stderr, and keep scriptorium-specific flags from leaking into domain packages.
`distributor` is also an external adapter. Upload behavior must be isolated
under `internal/adapters/distributor`, dependency types from the distributor
module must not leak outside that adapter, and the selected upload source must
be the managed Markdown report rather than optional output copies or broad
workspace scans.
## Project Shape ## Project Shape
Default to a small, explicit, dependency-light Go application. Keep the design modular enough to test and change safely, but do not add abstraction unless it protects a real boundary or enables a real extension point. Default to a small, explicit, dependency-light Go application. Keep the design modular enough to test and change safely, but do not add abstraction unless it protects a real boundary or enables a real extension point.
@@ -54,7 +60,7 @@ Configuration precedence is:
Prefer YAML configuration unless the project has a strong reason to use another format. Config files should be discovered at `/usr/local/etc/<app_name>/config.yml`, with a CLI override via `--config`. Prefer YAML configuration unless the project has a strong reason to use another format. Config files should be discovered at `/usr/local/etc/<app_name>/config.yml`, with a CLI override via `--config`.
Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets. Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets. File-backed secrets are loaded through `secrets.directory`; secret values must not be logged, persisted, or included in user-facing output.
## Adapters and External Integrations ## Adapters and External Integrations

View File

@@ -13,6 +13,7 @@ Developers and LLM coding agents should use it with
- `internal/config`: configuration structs, defaults, loading, overrides, and - `internal/config`: configuration structs, defaults, loading, overrides, and
validation. validation.
- `internal/fileutil`: shared atomic filesystem write and copy helpers. - `internal/fileutil`: shared atomic filesystem write and copy helpers.
- `internal/adapters/distributor`: Distributor upload adapter.
- `internal/adapters/weatherapi`: Weather API HTTP adapter. - `internal/adapters/weatherapi`: Weather API HTTP adapter.
- `internal/adapters/scriptorium`: Scriptorium subprocess adapter. - `internal/adapters/scriptorium`: Scriptorium subprocess adapter.
- `internal/forecast`: normalized bundle types and deterministic forecast - `internal/forecast`: normalized bundle types and deterministic forecast
@@ -45,7 +46,7 @@ Useful focused checks:
```bash ```bash
go test ./internal/cli ./internal/config go test ./internal/cli ./internal/config
go test ./internal/app ./internal/state go test ./internal/app ./internal/state
go test ./internal/adapters/weatherapi ./internal/adapters/scriptorium go test ./internal/adapters/distributor ./internal/adapters/weatherapi ./internal/adapters/scriptorium
go test ./internal/forecast ./internal/report ./internal/briefing ./internal/changes ./internal/promptinput go test ./internal/forecast ./internal/report ./internal/briefing ./internal/changes ./internal/promptinput
``` ```
@@ -64,6 +65,8 @@ Run `gofmt -w` on changed Go files before committing.
- Use atomic writes for durable JSON artifacts where practical. - Use atomic writes for durable JSON artifacts where practical.
- Keep report selection and prompt IDs centralized in `internal/report`. - Keep report selection and prompt IDs centralized in `internal/report`.
- Keep Scriptorium argv construction inside `internal/adapters/scriptorium`. - Keep Scriptorium argv construction inside `internal/adapters/scriptorium`.
- Keep distributor package types and upload-client construction inside
`internal/adapters/distributor`.
- Keep Weather API transport and envelope handling inside - Keep Weather API transport and envelope handling inside
`internal/adapters/weatherapi`. `internal/adapters/weatherapi`.
@@ -72,8 +75,10 @@ Run `gofmt -w` on changed Go files before committing.
Prefer the Go standard library. Add dependencies only when they materially Prefer the Go standard library. Add dependencies only when they materially
improve correctness, interoperability, security, or maintainability. improve correctness, interoperability, security, or maintainability.
Current external dependency: Current external dependencies:
- `gitea.maximumdirect.net/eric/distributor` for distributor source bundle
construction and HTTP upload client behavior.
- `gopkg.in/yaml.v3` for YAML configuration parsing. - `gopkg.in/yaml.v3` for YAML configuration parsing.
When adding a dependency: When adding a dependency:
@@ -145,12 +150,14 @@ When an external contract changes, update the matching file under
## Tests ## Tests
Core tests must not require live Weather API or Scriptorium services. Core tests must not require live Weather API, Scriptorium, or distributor
services.
Preferred test patterns: Preferred test patterns:
- fake command runners for subprocess behavior; - fake command runners for subprocess behavior;
- `httptest.Server` for Weather API behavior; - `httptest.Server` for Weather API behavior;
- fake distributor upload clients for notification behavior;
- filesystem temp directories for state behavior; - filesystem temp directories for state behavior;
- deterministic clocks for report periods and RunIDs; - deterministic clocks for report periods and RunIDs;
- table tests for config validation, CLI parsing, period resolution, and - table tests for config validation, CLI parsing, period resolution, and
@@ -187,7 +194,8 @@ Update:
behavior; behavior;
- `docs/troubleshooting.md` for recurring operator-facing failure modes; - `docs/troubleshooting.md` for recurring operator-facing failure modes;
- `docs/internal/` for component contracts and invariants; - `docs/internal/` for component contracts and invariants;
- `docs/integrations/` for external Weather API or Scriptorium contract changes; - `docs/integrations/` for external Weather API, Scriptorium, or distributor
contract changes;
- `docs/roadmap/` only for unimplemented or deferred work. - `docs/roadmap/` only for unimplemented or deferred work.
Non-roadmap docs must describe implemented behavior only. Non-roadmap docs must describe implemented behavior only.

View File

@@ -0,0 +1,34 @@
# Distributor Roadmap
Current distributor notification behavior is documented outside the roadmap:
- [Configuration reference](../config.md)
- [Operations guide](../operations.md)
- [Troubleshooting](../troubleshooting.md)
- [Distributor adapter internals](../internal/distributor-adapter.md)
This file tracks future distributor-related work only.
## Deferred Enhancements
- Add a supported warning-only notification policy.
- Include selected non-report artifacts in uploaded bundles.
- Poll distributor run status after upload acceptance.
- Persist upload retry state across process restarts.
- Add explicit CLI controls for distributor behavior.
## Non-Goals Without A Separate Design
- Do not make distributor scan the weatherreporter workspace.
- Do not move destination routing into weatherreporter.
- Do not move Markdown-to-HTML transformation into weatherreporter.
- Do not store raw bearer tokens in configuration files.
## Required Constraints For Future Work
- Distributor package types stay inside `internal/adapters/distributor`.
- Weatherreporter submits explicit source bundles built from generated files.
- Optional `--out` and `--out-dir` copies remain operator conveniences, not
canonical upload sources.
- Secret values stay out of errors, logs, CLI output, metadata, examples, and
documentation.

View File

@@ -12,8 +12,7 @@ evaluation remains deferred.
Proposed direction: Proposed direction:
1. detect candidate events deterministically from alerts, forecast discussion, 1. detect candidate events deterministically from alerts, forecast discussion,
weather story context when available, hourly thresholds, and material weather story context, hourly thresholds, and material forecast changes;
forecast changes;
2. evaluate candidates through Scriptorium or another narrow evaluator adapter; 2. evaluate candidates through Scriptorium or another narrow evaluator adapter;
3. persist storm lifecycle state; 3. persist storm lifecycle state;
4. generate or update Storm Reports only when a meaningful event is present; 4. generate or update Storm Reports only when a meaningful event is present;
@@ -50,6 +49,26 @@ These ideas are not current behavior:
Each item needs its own design note before implementation. Non-roadmap docs Each item needs its own design note before implementation. Non-roadmap docs
must not describe these as available behavior. must not describe these as available behavior.
## Deferred: Distributor Notification Enhancements
Distributor notification currently uploads one managed Markdown report per
successful generated report through the configured HTTP upload endpoint.
These enhancements are not current behavior:
- `failure_policy: warn`;
- uploading metadata, briefing snapshots, data packages, or preflight artifacts;
- polling distributor status after upload acceptance;
- durable upload retry queues;
- distributor-specific CLI flags;
- making distributor scan the weatherreporter workspace;
- handling destination routing, Markdown-to-HTML transformation, public URLs,
or nginx layout inside weatherreporter.
Any distributor enhancement should preserve the existing adapter boundary:
weatherreporter selects explicit generated files and submits source bundles,
while distributor owns destination routing and publication behavior.
## Deferred: Cleanup Refactors ## Deferred: Cleanup Refactors
The initial cleanup pass intentionally left these refactors out because the The initial cleanup pass intentionally left these refactors out because the

View File

@@ -0,0 +1,791 @@
# Modular Data Package Implementation Roadmap
This roadmap is a staged implementation plan for
[`docs/roadmap/modules.md`](modules.md). It is future-work planning only. The
target audience is an LLM coding agent implementing each stage in order.
## Purpose
Implement a pre-release hard cutover from report-shaped briefing packages to a
module-oriented prompt package architecture:
```text
CollectedFacts -> DerivedFacts -> ModuleOutput
```
The implementation should produce YAML `data_package` artifacts with named
stanzas for Scriptorium prompts, persist JSON module snapshots for inspection
and Recent Changes, and make report composition configurable through ordered
module IDs plus typed module options.
## Source Roadmap
[`docs/roadmap/modules.md`](modules.md) is authoritative for the conceptual
policy, user intent, target prompt shape, boundaries, and acceptance criteria
for this refactor. This document is authoritative for implementation order,
stage scope, file/package guidance, and validation commands.
If this implementation plan appears to conflict with `modules.md`, stop and
reconcile the roadmap before changing code. Do not infer a different policy
from stage sequencing.
## Locked Decisions
- Do a clean break. Do not preserve old report-shaped briefing JSON as a
compatibility layer.
- Introduce `internal/weatherdata` for normalized collected source types.
- Keep `internal/forecast` for forecast-specific derivation algorithms such as
period selection, daily summaries, daypart grouping, and precipitation
timing.
- Introduce an internal fact contract for `CollectedFacts` and `DerivedFacts`.
- Introduce a narrow module contract for module IDs, typed options, outputs,
and snapshots.
- Persist both artifacts:
- JSON module snapshots for state, inspection, and Recent Changes;
- YAML prompt data packages passed to Scriptorium as `data_package`.
- Replace `weatherreporter inspect briefing` with
`weatherreporter inspect modules`.
- Support typed module options from the first configurable composition pass.
- Use named YAML stanzas, not a generic array of module objects.
- Omit QPF fields until a real upstream QPF source is represented in
`CollectedFacts`.
- Keep public generate/run command names, report IDs, prompt IDs, RunID format,
managed Markdown report paths, and distributor upload source stable.
- Do not introduce plugins, dynamic loading, generic workflow engines, or
module-owned upstream fetching.
## Target Packages
Implementation should converge on this package ownership:
- `internal/weatherdata`: normalized collected source facts, source metadata,
source warnings, and broad weather data types.
- `internal/forecast`: deterministic forecast-specific algorithms over
`weatherdata` types.
- `internal/facts`: `CollectedFacts`, `DerivedFacts`, and their builders.
- `internal/module`: stable module IDs, module output envelope, module snapshot
shape, module config item shape, and shared option/output contracts that must
be imported by both `internal/report` and `internal/briefing`.
- `internal/briefing`: module registry and module builders.
- `internal/report`: report definitions, valid periods, output identity,
comparison strategy, and default module composition.
- `internal/config`: YAML config structs, defaults, loading, and validation for
module composition overrides.
- `internal/promptinput`: YAML prompt package assembly, validation, and save
behavior.
- `internal/changes`: structured comparison over module snapshots.
- `internal/state`: module snapshot paths, YAML data package paths, metadata
links, and inspection loads.
- `internal/app`: orchestration only.
Avoid import cycles. In particular, `internal/report` may import
`internal/module` for module IDs, but `internal/module` must not import
`internal/report`.
## Target Artifacts
Use explicit schema versions:
- Module snapshot JSON: `weatherreporter.modules.v1`
- YAML prompt data package: `weatherreporter.data_package.v2`
Target workspace paths:
```text
workspace/
snapshots/<artifact_group>/<valid_date>/<run_id>.modules.json
snapshots/<artifact_group>/<valid_date>/<run_id>.metadata.json
data-packages/<artifact_group>/<valid_date>/<run_id>.data_package.yaml
```
Metadata should link the module snapshot and YAML data package paths. Existing
metadata links for preflight, rendered report, source warnings, source hashes,
and distributor notification artifacts should remain.
## Target Module Defaults
Initial implemented default module IDs should cover current behavior without
QPF-specific fields:
- `metadata`
- `current_conditions`
- `derived_daily_summary`
- `derived_daypart_summaries`
- `precip_timing`
- `alert_digest`
- `area_forecast_discussion`
- `weather_story`
- `forecast_delta`
- `outdoor_windows`
- `weekend_planning`
- `storm_window_summary`
Default report composition should be declared in `internal/report`:
- Daily Today:
`metadata`, `current_conditions`, `derived_daily_summary`,
`derived_daypart_summaries`, `precip_timing`, `alert_digest`,
`forecast_delta`, `area_forecast_discussion`, `weather_story`,
`outdoor_windows`
- Daily Tomorrow:
same as Daily Today, plus any tomorrow-planning module needed to preserve
current tomorrow behavior.
- 3-Day:
`metadata`, `current_conditions`, `derived_daypart_summaries`,
`precip_timing`, `alert_digest`, `forecast_delta`,
`area_forecast_discussion`, `weather_story`, `outdoor_windows`
- Weekend:
`metadata`, `current_conditions`, `derived_daypart_summaries`,
`precip_timing`, `alert_digest`, `area_forecast_discussion`,
`weather_story`, `outdoor_windows`, `weekend_planning`
- Storm:
`metadata`, `current_conditions`, `hourly_table`, `precip_timing`,
`alert_digest`, `area_forecast_discussion`, `weather_story`,
`storm_window_summary`
If preserving a current report behavior requires a narrower module, add a
specific module rather than keeping old report-shaped containers.
## Stage 1: Weatherdata Package Split
Goal: separate broad normalized weather data from forecast-specific derivation.
Files/packages to change:
- create `internal/weatherdata`;
- update `internal/forecast`;
- update `internal/adapters/weatherapi`;
- update packages that currently import normalized source types from
`internal/forecast`.
Implementation guidance:
- Move normalized source/domain types out of `internal/forecast` when they are
not forecast algorithms:
- bundle/source metadata/warnings;
- current conditions;
- observation run types if present;
- alert run and alert overlap source types;
- forecast run and forecast period source types;
- discussion and discussion section types;
- weather story types.
- Keep deterministic derivation functions in `internal/forecast`.
- Update Weather API adapter return types to use `weatherdata.Bundle`.
- Keep JSON field names and Weather API fixture behavior unchanged.
- Do not change CLI behavior, artifact paths, or prompt input yet.
Acceptance criteria:
- Weather API adapter tests pass with `weatherdata` types.
- Forecast derivation tests pass using `weatherdata` inputs.
- No external adapter dependency types leak into `weatherdata`.
- Existing generated report behavior is unchanged at this stage.
Validation:
```bash
go test ./internal/weatherdata ./internal/forecast ./internal/adapters/weatherapi
go test ./internal/app ./internal/briefing ./internal/promptinput
```
This stage is suitable for one implementation prompt if kept mechanical.
## Stage 2: Fact Contracts
Goal: add explicit `CollectedFacts` and `DerivedFacts` contracts.
Files/packages to change:
- create `internal/facts`;
- update `internal/app`;
- update `internal/forecast` tests as needed.
Implementation guidance:
- Define `CollectedFacts` as normalized upstream facts collected once per
report run.
- Define `DerivedFacts` as conservative, reusable, report-scoped
transformations.
- Add builders:
- `BuildCollected(bundle *weatherdata.Bundle) CollectedFacts`
- `BuildDerived(req BuildDerivedRequest) (DerivedFacts, error)`
- `BuildDerivedRequest` should include the resolved report, timezone,
configured dayparts, and `CollectedFacts`.
- `DerivedFacts` may include:
- valid-period hourly periods;
- valid-period narrative periods;
- alert overlaps;
- daily summaries;
- daypart summaries where reusable;
- precipitation timing if reused by multiple modules.
- Do not put prompt wording, prose strings, module-specific ranking, or
one-off presentation decisions in `DerivedFacts`.
- Keep source provenance and warnings separate from ordinary fact access.
Acceptance criteria:
- `CollectedFacts` can be built once from a fetched bundle.
- `DerivedFacts` can be built for Daily, Tomorrow, 3-Day, Weekend, and Storm.
- Derived fact builders have tests for valid-period slicing, daypart grouping,
alert overlaps, and missing optional sources.
- No module or prompt code exists yet that fetches upstream data.
Validation:
```bash
go test ./internal/facts ./internal/forecast ./internal/app
go test ./internal/...
```
This stage is suitable for one implementation prompt.
## Stage 3: Module Core Contracts
Goal: define module IDs, options, outputs, snapshots, and registry mechanics.
Files/packages to change:
- create `internal/module`;
- update `internal/report`;
- update `internal/briefing`.
Implementation guidance:
- Define:
- `module.ID`;
- module ID constants;
- `module.ConfigItem`;
- `module.Output`;
- `module.Snapshot`;
- shared schema version constants.
- `module.Output` should contain module ID, stanza name, and typed value.
- `module.Snapshot` should preserve ordered outputs and support typed stanza
lookup for comparison code.
- Add duplicate module and duplicate stanza-name validation.
- Add typed option structs for initial modules. Empty option structs are fine
for modules without options.
- Add a module registry in `internal/briefing` that maps module IDs to builder
definitions.
- Module definitions should declare:
- ID;
- stanza name;
- option type;
- default options;
- required collected facts;
- required derived facts;
- supported report IDs or report categories;
- missing-data behavior.
- Do not execute modules from app orchestration yet unless needed for tests.
Acceptance criteria:
- Report definitions can refer to `module.ID` without import cycles.
- Module registry tests reject unknown modules, duplicate module IDs, duplicate
stanza names, incompatible reports, and invalid option shapes.
- Module output and snapshot JSON marshal deterministically enough for tests.
Validation:
```bash
go test ./internal/module ./internal/briefing ./internal/report
```
This stage is suitable for one implementation prompt.
## Stage 4: Base Modules
Goal: implement source-oriented modules that mostly pass through normalized or
lightly selected facts.
Files/packages to change:
- `internal/briefing`;
- `internal/module`;
- tests under `internal/briefing`.
Implementation guidance:
- Implement these modules:
- `metadata`;
- `current_conditions`;
- `alert_digest`;
- `area_forecast_discussion`;
- `weather_story`.
- The `metadata` module should expose report metadata, configured location,
units, timezone, valid period, source warnings summary, and alert checked
status where appropriate.
- `area_forecast_discussion` should expose key messages, short-term text, and
long-term text when present.
- `weather_story` should expose structured story fields when present and omit
the stanza when missing/suppressed by missing-source policy.
- `alert_digest` should distinguish checked/no-active-alerts from missing alert
source data.
- Ordinary modules should not expose endpoint, hash, or transport provenance;
provenance should remain metadata/source-warning oriented.
Acceptance criteria:
- Each module has focused tests for available data, missing optional data, and
empty output omission.
- No module fetches upstream data or reads/writes durable state.
- Output field names use YAML-friendly snake_case and unit suffixes where
needed.
Validation:
```bash
go test ./internal/briefing ./internal/module ./internal/facts
```
This stage is suitable for one implementation prompt.
## Stage 5: Derived Fact Modules
Goal: implement deterministic modules that package reusable forecast
derivations for the LLM.
Files/packages to change:
- `internal/forecast`;
- `internal/facts`;
- `internal/briefing`;
- `internal/module`.
Implementation guidance:
- Implement:
- `derived_daily_summary`;
- `derived_daypart_summaries`;
- `precip_timing`;
- `outdoor_windows`;
- any tomorrow-planning module needed to preserve Tomorrow output quality.
- `derived_daily_summary` should include current implementable fields:
- `high_temp_f`;
- `low_temp_f`;
- `max_pop_percent`;
- `max_pop_window`;
- `first_precip_hour`;
- `last_precip_hour`;
- `thunder_mentioned`;
- `max_wind_gust_mph`;
- `heat_index_max_f` when source data supports it.
- Do not implement `measurable_qpf_total_in` or `max_hourly_qpf_in` until QPF
exists in `CollectedFacts`.
- `derived_daypart_summaries` should expose daypart keyed values using the
configured daypart definitions.
- Keep broad reusable calculations in `DerivedFacts`; keep prompt-shape
packaging inside modules.
Acceptance criteria:
- Derived modules have fixture coverage across ordinary, dry, rainy, windy,
cold/heat, and missing-data scenarios.
- QPF fields are absent unless an upstream QPF source exists.
- Daily and Tomorrow module outputs contain enough data to replace current
report-shaped daily briefing content.
Validation:
```bash
go test ./internal/forecast ./internal/facts ./internal/briefing ./internal/module
```
This stage may be too large for one prompt if all modules are implemented at
once. Split into Daily-derived modules first, then outlook/storm derived
modules if needed.
## Stage 6: Report Composition And Config Overrides
Goal: make report definitions and config the source of module composition.
Files/packages to change:
- `internal/report`;
- `internal/config`;
- `examples/config.yml`;
- config tests.
Implementation guidance:
- Extend `report.Definition` with default ordered module IDs.
- Keep valid-period resolution, prompt IDs, output naming, generated flag, and
comparison strategy in `internal/report`.
- Add config support:
```yaml
reports:
daily:
deterministic_modules:
- current_conditions
- id: area_forecast_discussion
options:
sections:
- short_term
```
- Support both string shorthand and object form for module entries.
- Normalize config into typed `module.ConfigItem` values.
- Decode module options into typed option structs during validation or before
module execution.
- Reject:
- unknown report IDs;
- unknown module IDs;
- duplicate modules unless explicitly allowed by that module;
- duplicate stanza names;
- incompatible report/module combinations;
- invalid options.
- Built-in defaults should work when no report module config is present.
- Example config may omit module overrides unless an example is needed.
Acceptance criteria:
- Defaults reproduce intended module composition for all implemented reports.
- A config edit can add/remove an implemented module for a report.
- Invalid module config errors are actionable and do not mention raw internal
panic/details.
- Config examples load.
Validation:
```bash
go test ./internal/report ./internal/config ./internal/briefing
go run ./cmd/weatherreporter --help
```
This stage is suitable for one implementation prompt.
## Stage 7: Module Snapshot State
Goal: persist and inspect JSON module snapshots without changing Scriptorium
input yet.
Files/packages to change:
- `internal/state`;
- `internal/app`;
- `internal/cli`;
- app/state/CLI tests.
Implementation guidance:
- Add state paths for `<run_id>.modules.json`.
- Add save/load methods for module snapshots.
- Update metadata to include `ModuleSnapshotPath`.
- Add `weatherreporter inspect modules [--config PATH] RUN_ID`.
- Remove `inspect briefing` from parser support and help text in this stage.
- Keep old data package generation in place only until Stage 8, but do not
leave generation without a module snapshot.
Acceptance criteria:
- Generated runs persist module snapshots before prompt package construction.
- `inspect modules` returns the module snapshot.
- `inspect briefing` is gone from help text and parser tests.
- Metadata links the module snapshot path.
- Existing report generation still succeeds with fake Scriptorium.
Validation:
```bash
go test ./internal/state ./internal/cli ./internal/app
go run ./cmd/weatherreporter --help
```
This stage is suitable for one implementation prompt.
## Stage 8: YAML Prompt Package Cutover
Goal: replace JSON prompt data packages with YAML named-stanza data packages.
Files/packages to change:
- `internal/promptinput`;
- `internal/state`;
- `internal/adapters/scriptorium` tests;
- `internal/app`.
Implementation guidance:
- Set prompt package schema version to `weatherreporter.data_package.v2`.
- Build prompt package content from module snapshots, report metadata, recent
changes, and source warnings.
- Save prompt packages as `.data_package.yaml`.
- Continue passing Scriptorium input as `--input data_package=<path>`.
- Update render/run tests to avoid assuming `.json` filenames.
- Ensure YAML uses named stanzas under `briefing`.
- Omit empty optional fields.
- Keep module snapshot JSON as the comparison/inspection source.
- Update metadata `DataPackagePath` to point to YAML.
Acceptance criteria:
- Scriptorium render and run receive a YAML `data_package` path.
- YAML output is deterministic enough for tests.
- `inspect data-package` returns YAML content or a parsed representation
chosen consistently in CLI tests.
- No code assumes data package paths end in `.json`.
Validation:
```bash
go test ./internal/promptinput ./internal/adapters/scriptorium ./internal/state ./internal/app ./internal/cli
go test ./...
```
This stage is suitable for one implementation prompt.
## Stage 9: App Orchestration Cutover
Goal: make module execution the only generation path for all implemented
reports.
Files/packages to change:
- `internal/app`;
- `internal/briefing`;
- `internal/facts`;
- app workflow tests.
Implementation guidance:
- In `GenerateReport`, fetch Weather API data once, build `CollectedFacts`,
build `DerivedFacts`, execute configured modules, save module snapshot, build
YAML prompt package, then continue preflight/run/metadata/distributor flow.
- Preserve ordering:
1. resolve prior comparable metadata;
2. fetch bundle;
3. build facts;
4. execute modules;
5. save module snapshot;
6. compute Recent Changes;
7. save YAML data package;
8. run render preflight;
9. save metadata;
10. run Scriptorium;
11. copy optional output;
12. save final metadata;
13. notify distributor if enabled.
- Do not use `--out` or `--out-dir` copies for distributor notification.
- Do not invoke modules after Scriptorium failures.
- Keep batch behavior unchanged: continue independent reports, return nonzero
aggregate status if any report fails.
Acceptance criteria:
- Daily, Tomorrow, 3-Day, Weekend, and Storm generation all use module
snapshots and YAML data packages.
- Existing public CLI syntax remains stable except `inspect modules` replacing
`inspect briefing`.
- Managed Markdown report paths and distributor upload source remain stable.
- App tests assert generated module snapshots and YAML data packages.
Validation:
```bash
go test ./internal/app ./internal/cli ./internal/state ./internal/briefing ./internal/promptinput
go test ./...
```
This stage may be large. Split by report family if needed: Daily/Tomorrow,
Outlooks, then Storm.
## Stage 10: Recent Changes Migration
Goal: compare structured module snapshots instead of report-shaped briefing
packages.
Files/packages to change:
- `internal/changes`;
- `internal/state`;
- `internal/app`;
- changes tests.
Implementation guidance:
- Define which module stanzas each comparison strategy consumes.
- Daily comparison should use `derived_daily_summary`,
`derived_daypart_summaries`, `alert_digest`, and `precip_timing` where
present.
- 3-Day and Weekend comparisons should use module snapshot outputs that replace
current outlook day comparisons.
- Storm comparison should remain explicit-window based and consume storm
module outputs when implemented.
- Do not compare rendered Markdown or rendered YAML text.
- If a comparison-required module is missing, return an actionable error or an
inspectable warning according to the report policy chosen in code. Prefer an
error for required comparison modules and no-op only for optional comparison
stanzas.
Acceptance criteria:
- Prior snapshot lookup still uses report compatibility and valid-period rules.
- Recent Changes output remains deterministic.
- Tests cover unchanged forecasts, threshold-crossing changes, alert changes,
precip timing changes, and missing comparison stanzas.
- Old `briefing.Package` comparison code is removed.
Validation:
```bash
go test ./internal/changes ./internal/state ./internal/app
go test ./...
```
This stage is suitable for one implementation prompt if module snapshots are
already available.
## Stage 11: Remove Old Briefing Shapes
Goal: remove obsolete report-shaped briefing containers and stale JSON package
assumptions.
Files/packages to change:
- `internal/briefing`;
- `internal/promptinput`;
- `internal/state`;
- `internal/app`;
- tests throughout `internal`.
Implementation guidance:
- Remove old `Daily`, `ThreeDay`, `Weekend`, and `Storm` briefing container
structs when no longer used.
- Remove old `briefing.Package` if it no longer represents the module
snapshot. If the package keeps a `Package` type, it must be module-oriented.
- Remove tests that construct old report-shaped briefing fixtures.
- Remove stale `.data_package.json` assumptions.
- Remove dead helper functions that only supported old report-shaped output.
- Keep generated report Markdown behavior stable.
Acceptance criteria:
- `rg -n "data_package\\.json|inspect briefing|briefing\\.Package" internal docs -g '!docs/roadmap/**'`
has no production-code matches, except deliberate roadmap/history references
where appropriate.
- No old report-shaped content structs remain on the generation path.
- All tests pass.
Validation:
```bash
rg -n "data_package\\.json|inspect briefing|briefing\\.Package" internal docs -g '!docs/roadmap/**'
go test ./...
go run ./cmd/weatherreporter --help
git diff --check
```
This stage is suitable for one implementation prompt.
## Stage 12: Documentation And Example Alignment
Goal: align non-roadmap docs with implemented module behavior.
Files to inspect/update:
- `README.md`, only if the orientation or quickstart changed;
- `docs/cli.md`;
- `docs/config.md`;
- `docs/operations.md`;
- `docs/troubleshooting.md`;
- `docs/internal/app-orchestration.md`;
- `docs/internal/briefing.md` or replacement module internals doc;
- `docs/internal/changes.md`;
- `docs/internal/forecast-derivation.md`;
- `docs/internal/prompt-input.md`;
- `docs/internal/state.md`;
- `docs/internal/weather-data.md`;
- `docs/integrations/scriptorium.md`;
- `examples/config.yml`.
Implementation guidance:
- Document only implemented behavior outside `docs/roadmap/`.
- Add or update an internal module contract document if module behavior is now
implemented.
- Document `inspect modules` and remove `inspect briefing`.
- Document YAML data packages and JSON module snapshots.
- Document report module overrides and typed options only if implemented.
- Keep QPF as future-only unless upstream support was added.
- Keep Scriptorium contract focused on `--input data_package=<path>` and the
actual file format now passed.
Acceptance criteria:
- Non-roadmap docs no longer describe old report-shaped briefing packages.
- Config examples load.
- CLI examples match `weatherreporter --help`.
- Docs clearly distinguish module snapshots from prompt data packages.
Validation:
```bash
go test ./...
go run ./cmd/weatherreporter --help
git diff --check
rg -n "inspect briefing|data_package\\.json|report-shaped|vars-file|promptvars" README.md docs examples internal -g '!docs/roadmap/**'
```
This stage is suitable for one implementation prompt.
## Stage 13: Final Validation
Goal: run full validation and catch stale assumptions after the cutover.
Required commands:
```bash
go test ./...
go run ./cmd/weatherreporter --help
git diff --check
```
Required grep checks:
```bash
rg -n "inspect briefing|data_package\\.json|briefing\\.Package|Daily struct|ThreeDay struct|Weekend struct|Storm struct" internal docs examples -g '!docs/roadmap/**'
rg -n "measurable_qpf_total_in|max_hourly_qpf_in" internal docs examples -g '!docs/roadmap/**'
```
Expected grep results:
- no production-code references to `inspect briefing`;
- no production-code assumption that prompt packages are JSON;
- no production-code dependence on old report-shaped briefing containers;
- QPF references appear only as future-target docs or omitted-field tests until
upstream QPF exists.
Manual review:
- Generate command output still writes managed Markdown reports.
- Batch behavior still continues independent reports and returns nonzero on
aggregate failure.
- Distributor notification still uploads the managed Markdown report, not
module snapshots or YAML prompt packages.
- Secrets are not printed or persisted.
- YAML prompt package is readable and contains named stanzas.
## Deferred Work
Do not include these in the initial module cutover:
- plugin architecture;
- dynamic module loading;
- YAML-defined module schemas;
- user-authored module code;
- module-owned Weather API fetching;
- QPF fields before upstream QPF exists;
- SPC modules before upstream SPC data exists;
- radar modules before upstream radar data exists;
- event/storm-review reports unless a separate roadmap implements them.
## Open Questions
No blocking open questions remain for this implementation plan. The previously
identified choices are locked above:
- persist JSON module snapshots and YAML prompt data packages;
- split broad normalized source types into `internal/weatherdata`;
- replace `inspect briefing` with `inspect modules`;
- support typed module options from the first config implementation.

677
docs/roadmap/modules.md Normal file
View File

@@ -0,0 +1,677 @@
# Modular Data Package Roadmap
This roadmap describes planned refactoring work that is not implemented.
Current behavior is documented outside `docs/roadmap/`.
## Purpose
Move weatherreporter toward deterministic, reusable briefing modules that can
be composed per report type. The goal is to make prompt input easier for the
LLM to understand, easier for operators to inspect, and easier for developers
to change without touching a cross-cutting set of report-builder files.
The target outcome is a prompt-facing YAML data package with named stanzas. Each
stanza should be built by a self-contained module that derives clear,
deterministic facts from normalized forecast inputs. Reports should choose
modules by ordered module IDs, so experimenting with a report can be as small
as changing one configuration line, plus any matching prompt change outside
weatherreporter.
The internal target shape is:
```text
CollectedFacts -> DerivedFacts -> ModuleOutput
```
Each arrow should be a stable internal contract. `DerivedFacts` should not need
to know how `CollectedFacts` were collected. Modules should not need to know
the provenance of any collected or derived fact they consume. Report building
should not need to know how a module sourced its underlying facts or calculated
its output.
## Intent And Context
The current application already curates source data before passing it to the
LLM. This refactor should strengthen that design. Modules should not expose raw
source complexity merely because it is available. They should compute and
package the facts the LLM should not have to infer from raw hourly periods,
alerts, narrative periods, forecast discussions, or weather stories.
The desired module behavior is deterministic. A module should answer a narrow
question such as:
- what are the current conditions;
- what are the key daily forecast facts;
- what are the daypart summaries;
- when is precipitation most likely;
- which alerts overlap the report period;
- what short-term AFD text is relevant;
- what weather story text is relevant.
The primary maintainability goal is local reasoning. For example, updating the
derived daily summary should mostly involve one module implementation and its
tests. Adding AFD short-term text to a future `next_six_hours` report should be
a report composition change, not a copy/paste change across multiple builders.
## Target Prompt Shape
The target prompt-facing data package should be YAML with named stanzas under
`briefing`. Named stanzas are preferred over an array of generic module objects
because they are easier to read, inspect, and reference in prompts.
Example target shape:
```yaml
report:
id: daily_today
prompt_id: weather.daily_report
generated_at: 2026-06-09T07:15:00-05:00
timezone: America/Chicago
current_local_date: 2026-06-09
valid_period:
start: 2026-06-09T00:00:00-05:00
end: 2026-06-10T00:00:00-05:00
briefing:
metadata:
location:
id: home
name: Brentwood
region: St. Louis Metro
timezone: America/Chicago
current_conditions:
condition_text: Partly cloudy
temperature_f: 74
apparent_temperature_f: 76
dewpoint_f: 66
relative_humidity_percent: 71
wind_speed_mph: 8
wind_direction_degrees: 190
derived_daily_summary:
high_temp_f: 86
low_temp_f: 68
max_pop_percent: 70
max_pop_window: "2 PM-6 PM"
measurable_qpf_total_in: 0.35
max_hourly_qpf_in: 0.12
first_precip_hour: "1 PM"
last_precip_hour: "8 PM"
thunder_mentioned: true
max_wind_gust_mph: 28
heat_index_max_f: 91
derived_daypart_summaries:
morning:
temp_range_f: "70-78"
max_pop_percent: 20
dominant_condition: Partly sunny
afternoon:
temp_range_f: "82-86"
max_pop_percent: 70
dominant_condition: Showers and thunderstorms likely
alert_digest:
checked: true
active_count: 0
relevant_count: 0
area_forecast_discussion:
key_messages:
- Scattered storms are possible this afternoon.
short_term: Showers and storms increase during the afternoon.
long_term: Periodic rain chances continue into the weekend.
weather_story:
available: true
title: Several Chances for Rain Through Monday
description: Scattered showers and thunderstorms remain possible.
recent_changes:
items: []
```
Field names should include units where the unit is not obvious:
`high_temp_f`, `max_pop_percent`, `measurable_qpf_total_in`,
`max_wind_gust_mph`, and similar names are preferred over ambiguous generic
names. Time and range strings should be formatted for prompt readability, while
machine-oriented timestamps should remain available in report metadata.
The QPF fields in the example are target output fields for a future upstream
source. They should not be treated as immediately implementable from the
current weatherfeeder-backed `CollectedFacts` sources. Until an upstream QPF
source exists, QPF fields should be omitted rather than fabricated from
precipitation probability or narrative text.
## Architecture Target
Keep the current package boundaries:
- `internal/forecast` owns normalized source data, deterministic forecast
derivation, period slicing, daypart grouping, and weather-signal calculations.
- `internal/briefing` owns prompt-facing module builders and module output
schemas.
- `internal/report` owns report identity, prompt ID, valid-period resolution,
output naming, comparison strategy, and default module composition.
- `internal/config` owns optional report module composition overrides.
- `internal/promptinput` owns final data-package assembly, validation, and
prompt-facing serialization.
- `internal/app` remains orchestration: resolve report, fetch sources, build
module context, execute configured modules, persist artifacts, run
Scriptorium, and notify distributor.
Do not move source fetching, subprocess execution, distributor upload behavior,
or raw external dependency types into module code.
## Layered Fact Contracts
Introduce explicit internal contracts for three layers:
1. `CollectedFacts`
2. `DerivedFacts`
3. `ModuleOutput`
`CollectedFacts` are normalized upstream inputs collected once per report run.
They should be broad and source-oriented, but not tied to Weather API transport
details. Examples include current conditions, observations, hourly forecast
runs, narrative forecast runs, active alerts, AFD discussion, weather story,
future radar inputs, and future historical observation totals.
`DerivedFacts` are reusable, report-scoped deterministic products calculated
from `CollectedFacts`. They may slice, combine, group, or summarize collected
facts when the result is broadly useful to more than one module or needed for
consistent behavior across modules. Examples include valid-period hourly
periods, valid-period narrative periods, alert overlaps, daily summaries,
configured daypart summaries, and reusable precipitation timing windows.
`ModuleOutput` is the prompt-facing output contract produced by one module. A
module may pass through raw-ish facts, such as AFD text, or expose derived
facts, such as daily summary fields. In both cases, the module owns the named
stanza shape and should produce stable, readable, unit-explicit prompt fields.
Unless implementation discovers a strong reason otherwise, the internal
contract for accessing `CollectedFacts` and `DerivedFacts` should have the same
shape:
- typed Go structs with named fields;
- nil pointers, empty slices, or zero values to represent absent facts;
- no `map[string]any` or string-keyed fact lookup as the primary API;
- immutable-by-convention values once passed to modules;
- helper methods only for repeated access patterns that would otherwise be
error-prone;
- source provenance and warnings stored separately from the primary fact
values, available to metadata/source-warning modules but not required by
ordinary modules.
Illustrative shape:
```go
type CollectedFacts struct {
Current *weatherdata.CurrentConditions
Observations *weatherdata.ObservationRun
Alerts *weatherdata.AlertRun
Hourly *weatherdata.ForecastRun
Narrative *weatherdata.ForecastRun
Discussion *weatherdata.Discussion
WeatherStory *weatherdata.WeatherStory
// Future: radar, historical observations, snow/rain totals, etc.
}
type DerivedFacts struct {
HourlyPeriods []weatherdata.ForecastPeriod
NarrativePeriods []weatherdata.ForecastPeriod
AlertOverlaps []weatherdata.AlertOverlap
DailySummaries []forecast.DailySummary
DaypartSummaries []forecast.DaypartSummary
PrecipTiming *forecast.PrecipTiming
}
type ModuleContext struct {
Report report.Resolved
Collected CollectedFacts
Derived DerivedFacts
Units string
Timezone string
Location *LocationContext
}
```
The exact package names may differ during implementation. The important
boundary is semantic: `CollectedFacts` represent upstream facts after
normalization; `DerivedFacts` represent reusable report-scoped transformations;
modules represent prompt-facing stanza construction.
### DerivedFacts Boundary
Be conservative about what belongs in `DerivedFacts`. Add a value to this layer
only when it is:
- deterministic;
- report-scoped;
- reusable by multiple modules or needed to keep modules consistent;
- independent of prompt wording and presentation decisions.
`DerivedFacts` may:
- slice source periods to the report valid period;
- group hourly data into configured dayparts;
- compute reusable summaries;
- compute alert overlaps;
- normalize repeated time-window selections.
`DerivedFacts` should not:
- decide prompt-facing wording;
- decide which facts are important for one module only;
- format prose-like strings for the LLM;
- fetch upstream data;
- write artifacts;
- depend on Scriptorium or distributor.
Module-specific calculations should remain inside the module when they are
presentation-specific, used by only one module, or likely to change while
tuning prompt behavior.
## Module Model
Introduce a typed module model rather than generic maps. A module should have:
- stable module ID;
- self-contained output struct;
- one focused builder function;
- fixture or unit tests near the module;
- declared input requirements, such as hourly forecast, alerts, discussion, or
weather story;
- deterministic handling for missing optional source data;
- prompt-facing field names that are stable and unit-explicit.
The implementation may use a simple function registry rather than a broad
interface if that is enough:
```go
type ModuleID string
type ModuleBuilder func(ModuleContext) (ModuleOutput, error)
```
`ModuleOutput` should include the stable module ID, the YAML stanza name, and a
typed value owned by the module:
```go
type ModuleOutput struct {
ID ModuleID
StanzaName string
Value any
}
```
The module registry should preserve output order from report composition, but
the serialized YAML should use named stanzas for clarity.
Each module should be able to produce exactly one named stanza. If one source
can usefully feed multiple stanzas, split that into multiple modules rather than
making one module produce unrelated output.
## Report Composition Target
Report definitions should declare default ordered module IDs. Configuration may
override the ordered module list for implemented reports.
Illustrative future config shape:
```yaml
reports:
next_6_hours:
deterministic_modules:
- hourly_table
- precip_timing
- alert_digest
- afd_short_term_text
- weather_story_text
- spc_products
daily:
deterministic_modules:
- current_conditions
- derived_daily_summary
- derived_daypart_summaries
- precip_timing
- alert_digest
- forecast_delta
- afd_short_term_text
- weather_story_text
- spc_products
```
Configuration should validate unknown module IDs, duplicate module IDs when
duplicates are not meaningful, and modules that are incompatible with the
selected report period. Defaults should remain in Go so the application works
without report composition config.
## Clean-Break Cutover Policy
This project is still pre-release. Prefer a direct cutover to the new internal
shape instead of preserving transitional report-shaped briefing structures.
Implementation should:
- replace report-shaped briefing containers with module-oriented snapshots;
- replace JSON prompt package output with YAML prompt package output;
- update inspect commands, Recent Changes, tests, and docs in the same cutover;
- remove obsolete `Daily`, `ThreeDay`, `Weekend`, and `Storm` briefing
container shapes when no longer needed;
- avoid compatibility aliases unless they materially reduce implementation
risk inside one stage.
The public CLI command names, report IDs, prompt IDs, RunID format, managed
Markdown report paths, and distributor upload source should remain stable unless
a separate roadmap explicitly changes them.
## Artifact And State Target
The durable artifacts should reflect the new module-oriented model.
Recommended target:
- module snapshot artifact: structured JSON for stable inspection, state
lookup, and Recent Changes comparisons;
- prompt data package artifact: YAML with named stanzas, passed to Scriptorium
as `data_package`;
- metadata artifact: JSON linking the module snapshot, YAML data package,
preflight output, rendered report, source warnings, source hashes, and
distributor notification artifact when present.
The workspace path names should make the artifact type clear. A future
implementation may keep the existing `data-packages/` directory, but file
extensions and metadata fields should reflect the real format, such as:
```text
workspace/
snapshots/<artifact_group>/<valid_date>/<run_id>.modules.json
data-packages/<artifact_group>/<valid_date>/<run_id>.data_package.yaml
```
Replace `inspect briefing` with `inspect modules` during the cutover.
`inspect modules` should return the module snapshot. `inspect data-package`
should return the YAML artifact or a parsed representation of the YAML artifact.
Do not leave inspect commands pointed at obsolete report-shaped data.
## Module Options And Compatibility
Each module should have a typed options struct, even when initially empty.
Configuration may decode module options from YAML, but internal module builders
should receive typed options rather than `map[string]any`.
Illustrative config shape:
```yaml
reports:
next_6_hours:
deterministic_modules:
- id: hourly_table
options:
range: valid_period
fields:
- time
- temperature_f
- pop_percent
- wind_gust_mph
- id: afd_short_term_text
```
Module definitions should declare:
- module ID;
- stanza name;
- typed options schema;
- supported report IDs or report categories;
- required collected facts;
- required derived facts;
- whether missing optional facts omit the stanza, emit an empty stanza, or
produce a warning;
- whether duplicate use of the module is allowed.
Configuration validation should reject:
- unknown report IDs;
- unknown module IDs;
- duplicate module IDs unless explicitly allowed;
- two modules that render the same stanza name;
- module options that do not match the module's typed option schema;
- modules that are incompatible with the report's valid-period strategy or
available facts.
## Recent Changes Target
Recent Changes must remain structured and deterministic. During the clean-break
cutover, move comparison inputs away from report-shaped `briefing.Package`
values and toward module-oriented snapshots.
Recommended target:
- compare `ModuleOutput` values or typed module snapshot stanzas, not rendered
YAML and not rendered Markdown;
- keep report-compatible matching policy in `internal/report`;
- keep threshold configuration in `internal/config`;
- keep comparison algorithms in `internal/changes`;
- make each comparison explicit about which module stanzas it needs.
For example, Daily comparison should primarily consume
`derived_daily_summary`, `derived_daypart_summaries`, `alert_digest`, and
`precip_timing` if present. If a required stanza is missing, the comparison
should return no change with an inspectable warning or an actionable error,
depending on the report's configured missing-data policy.
## Package Naming Target
The current `internal/forecast` package owns both forecast-specific derivation
and broader normalized weather data. Because planned sources include current
observations, radar, and historical review inputs, implementation should
consider splitting names during the clean-break refactor:
- `internal/weatherdata`: normalized collected source facts, source metadata,
warnings, and broad weather data types;
- `internal/forecast`: forecast-specific algorithms such as period slicing,
daily summaries, daypart grouping, and precipitation timing.
If this split is too large for the first cutover, introduce `CollectedFacts` in
the package that minimizes churn, but avoid expanding the meaning of
`internal/forecast` further in new module contracts.
## Initial Module Candidates
The first module catalog should start with the modules needed to replace current
report-shaped briefing output and should clearly distinguish implemented
modules from future-only candidates.
Initial candidates:
- `metadata`
- `current_conditions`
- `derived_daily_summary`
- `derived_daypart_summaries`
- `hourly_table`
- `precip_timing`
- `alert_digest`
- `area_forecast_discussion`
- `afd_key_messages`
- `afd_short_term_text`
- `afd_long_term_text`
- `weather_story`
- `forecast_delta`
- `outdoor_windows`
- `weekend_planning`
- `storm_window_summary`
Each module should declare inputs, outputs, report applicability,
missing-data behavior, compatibility behavior, and options.
## Target Derived Daily Summary
The intended `derived_daily_summary` shape is:
```yaml
derived_daily_summary:
high_temp_f: 86
low_temp_f: 68
max_pop_percent: 70
max_pop_window: "2 PM-6 PM"
measurable_qpf_total_in: 0.35
max_hourly_qpf_in: 0.12
first_precip_hour: "1 PM"
last_precip_hour: "8 PM"
thunder_mentioned: true
max_wind_gust_mph: 28
heat_index_max_f: 91
```
`measurable_qpf_total_in` and `max_hourly_qpf_in` are future target fields.
They require a real upstream quantitative precipitation source and should be
omitted until such a source is represented in `CollectedFacts`.
## Target Derived Daypart Summaries
The intended `derived_daypart_summaries` shape is:
```yaml
derived_daypart_summaries:
morning:
temp_range_f: "70-78"
max_pop_percent: 20
dominant_condition: Partly sunny
afternoon:
temp_range_f: "82-86"
max_pop_percent: 70
dominant_condition: Showers and thunderstorms likely
```
## Configurable Composition Target
The target configuration model should allow report module composition to be
changed without editing cross-cutting report-builder code. Built-in defaults
should remain in Go so the application works with no module override config.
Illustrative config:
```yaml
reports:
daily:
deterministic_modules:
- current_conditions
- derived_daily_summary
- derived_daypart_summaries
- precip_timing
- alert_digest
- afd_short_term_text
- weather_story_text
```
Adding or removing an implemented module from an implemented report should be a
single config edit. Unknown modules, invalid options, duplicate stanzas, and
incompatible report/module combinations should fail with actionable errors.
## Acceptance Criteria
The refactor is complete when:
- current implemented reports generate successfully from named-stanza YAML
prompt packages;
- module snapshots are persisted as structured JSON and linked from metadata;
- `inspect modules` returns module snapshots;
- `inspect briefing` is removed from CLI help, parser support, and
non-roadmap docs;
- Recent Changes compares structured module snapshots, not rendered Markdown or
YAML text;
- report definitions declare default module order in one place;
- implemented report module composition can be overridden by config;
- implemented modules have typed options and compatibility contracts;
- `CollectedFacts` are built once per report run and reused by all modules;
- `DerivedFacts` are built from `CollectedFacts` and do not depend on adapter
transport details;
- module builders do not call Weather API, Scriptorium, distributor, or
filesystem state directly;
- stale report-shaped briefing containers are removed from the generation path;
- QPF output fields remain omitted until upstream QPF exists.
## Design Rules
- Keep modules deterministic.
- Keep modules self-contained where practical.
- Preserve the `CollectedFacts -> DerivedFacts -> ModuleOutput` boundary.
- Build `CollectedFacts` once per report run.
- Build `DerivedFacts` from `CollectedFacts`, not from adapter-specific
transport details.
- Keep `DerivedFacts` conservative and reusable.
- Keep raw external source details behind adapters and forecast normalization.
- Keep report composition centralized and ordered.
- Prefer typed outputs over generic maps.
- Prefer typed fact contracts over string-keyed fact registries.
- Prefer named YAML stanzas over generic module arrays.
- Use unit-explicit field names.
- Do not require the LLM to calculate obvious derived facts.
- Do not let module builders call external services or write durable state.
- Do not introduce plugins, dynamic loading, or a generic workflow engine.
## Risks And Mitigations
- Prompt contract churn: stage YAML introduction after module outputs are
tested and inspectable.
- Recent Changes drift: compare stable module outputs and keep snapshot tests.
- Over-abstraction: start with simple builders and a registry, not a framework.
- Config complexity: expose ordered module selection and typed options only for
implemented modules; defer broad parameterization.
- Loss of useful context: preserve focused source excerpts and source warnings,
but avoid reintroducing raw unbounded payloads.
## Deferred Work
These are out of scope for the initial module refactor:
- dynamic plugin loading;
- user-defined module code;
- YAML-defined module schemas;
- module parameterization beyond implemented typed options;
- replacing Weather API source fetching with module-owned fetches;
- moving prompt authoring or Scriptorium prompt changes into weatherreporter;
- adding future source modules, such as SPC products, before upstream data and
report requirements exist.
## Open Questions
### Should module snapshots and prompt packages both be persisted?
Recommended approach: persist module snapshots as JSON and prompt packages as
YAML. JSON module snapshots are better for structured Recent Changes, state
lookup, and tests. YAML prompt packages are better for prompt readability and
LLM consumption. Keeping both artifacts gives each use case the right format
without asking comparison code to parse prompt-oriented YAML.
Viable alternative: persist only the YAML prompt package and parse it for
inspection and Recent Changes. This reduces artifact count, but it couples
machine comparison to prompt formatting and makes future prompt-oriented
formatting changes riskier.
### Should `internal/forecast` be split during the first cutover?
Recommended approach: split broad normalized source types into
`internal/weatherdata` during the clean-break cutover if the implementation
scope remains manageable. This name fits current conditions, alerts,
discussion, weather story, future radar, and future historical data better than
`forecast`.
Viable alternative: keep existing `internal/forecast` types for the first
module implementation and introduce `CollectedFacts` as a wrapper. This reduces
short-term churn, but it leaves a package name that will become increasingly
misleading as non-forecast sources grow.
### Should module config support typed options immediately?
Recommended approach: support typed options for implemented modules from the
start, even if most modules use empty options. This establishes the extension
point needed for hourly ranges, field selection, and AFD section choices
without adding dynamic maps to module builders.
Viable alternative: initially support only ordered module IDs and add options
later. This is simpler, but it may force another config shape change as soon as
hourly range or section-selection experiments begin.

View File

@@ -199,6 +199,111 @@ metadata, sources, briefing, and data package for that RunID.
Relevant docs: [CLI reference](cli.md), [Operations guide](operations.md). Relevant docs: [CLI reference](cli.md), [Operations guide](operations.md).
## Invalid Secrets Directory
Symptom: config loading fails with `read secrets directory`, `secret file`, or
environment variable name context.
Likely cause: `secrets.directory` points to a missing directory or contains an
invalid entry. Secret entries must be regular files directly under the
configured directory, and file basenames must match
`[A-Za-z_][A-Za-z0-9_]*`.
Diagnostic: list the configured directory and inspect entry names and file
types. Do not print secret file contents.
Safe fix: create the directory, remove subdirectories or symlinks, fix invalid
filenames, and ensure the weatherreporter process can read each secret file.
Relevant docs: [Configuration reference](config.md).
## Distributor Token Is Missing
Symptom: notification fails with a message that the distributor token
environment variable is not set.
Likely cause: `notify.distributor.enabled` is true, but the environment
variable named by `notify.distributor.token_env` was not populated directly or
through `secrets.directory`.
Diagnostic: check `notify.distributor.token_env`, then verify a matching secret
file exists under `secrets.directory` or that the process environment includes
the variable. Do not print the token value.
Safe fix: create a readable secret file whose basename matches `token_env`, or
set the environment variable through the service manager.
Relevant docs: [Configuration reference](config.md),
[Operations guide](operations.md).
## Distributor Upload Conflict
Symptom: notification fails with idempotency conflict context.
Likely cause: the same idempotency key was reused for different bundle content
within the same distributor token and pipeline. By default the bundle ID is a
stable report-stream identity and the idempotency key appends RunID.
Diagnostic: inspect the failed batch JSON or stderr line for pipeline, bundle,
and idempotency context. Compare the configured templates with the report RunID
and report path.
Also inspect the notification artifact linked from metadata. It records the
rendered pipeline ID, bundle ID, idempotency key, upload result, distributor run
status, status error, and raw run report JSON when available.
Safe fix: keep idempotency templates stable for retries of the same generated
report, but do not reuse the same rendered key for different generated report
content.
Relevant docs: [Operations guide](operations.md),
[Distributor adapter internals](internal/distributor-adapter.md).
## Distributor Upload Rejected
Symptom: notification fails with distributor upload rejection, HTTP status, or
bundle validation context.
Likely cause: the distributor endpoint rejected the token, pipeline ID, bundle
ID, idempotency key, source file, or one of the rendered bundle paths.
Diagnostic: inspect stdout JSON or stderr status lines for
`notificationError`. Confirm `notify.distributor.endpoint`,
`notify.distributor.pipeline_id_template`,
`notify.distributor.report_path_templates`, and token configuration. Token
values are redacted from weatherreporter errors.
If the upload was accepted but destination output did not change, inspect the
notification artifact's `runStatus.report`. Distributor actions such as
`replace_older`, `skip_same`, `skip_destination_newer`, or `failed` explain how
the destination handled the uploaded bundle.
Safe fix: fix the endpoint, token, templates, or distributor-side upload
configuration. The weatherreporter upload source is the managed Markdown report,
not `--out` or `--out-dir` copies.
Relevant docs: [Configuration reference](config.md),
[Operations guide](operations.md),
[Distributor adapter internals](internal/distributor-adapter.md).
## Distributor Unavailable
Symptom: notification fails with network, timeout, or service unavailable
context.
Likely cause: the configured distributor endpoint is unreachable, slow, or
temporarily unavailable.
Diagnostic: check network access from the weatherreporter host to
`notify.distributor.endpoint`. For batch runs, inspect which reports have
`notificationStatus: "failed"`.
Safe fix: restore distributor service availability and rerun the affected
report or batch. Stable idempotency keys make retrying the same generated report
safe unless the distributor reports a conflict.
Relevant docs: [Operations guide](operations.md).
## Unknown RunID ## Unknown RunID
Symptom: an inspect command fails with `metadata for run id ... was not found`. Symptom: an inspect command fails with `metadata for run id ... was not found`.

View File

@@ -11,6 +11,22 @@ location:
name: Brentwood name: Brentwood
region: St. Louis Metro region: St. Louis Metro
secrets:
directory: ""
notify:
distributor:
enabled: false
endpoint: https://distributor.example.com
token_env: DISTRIBUTOR_UPLOAD_TOKEN
timeout: 30s
failure_policy: error
pipeline_id_template: "weatherreporter.{artifact_group}"
bundle_id_template: "weatherreporter.{location_id}.{report_id}"
idempotency_key_template: "{bundle_id}.{run_id}"
report_path_templates:
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
missing_source: missing_source:
default: warn default: warn
sources: sources:
@@ -26,6 +42,7 @@ workspace:
reports_dir: reports reports_dir: reports
data_packages_dir: data-packages data_packages_dir: data-packages
preflight_dir: preflight preflight_dir: preflight
notifications_dir: notifications
dayparts: dayparts:
- name: overnight - name: overnight

2
go.mod
View File

@@ -3,3 +3,5 @@ module gitea.maximumdirect.net/eric/weatherreporter
go 1.26 go 1.26
require gopkg.in/yaml.v3 v3.0.1 require gopkg.in/yaml.v3 v3.0.1
require gitea.maximumdirect.net/eric/distributor v0.5.0

48
go.sum
View File

@@ -1,3 +1,51 @@
gitea.maximumdirect.net/eric/distributor v0.5.0 h1:+al7Bw+kMv6V35a3Sm5rUtCTQhwOn5b9x3RsclPMKJk=
gitea.maximumdirect.net/eric/distributor v0.5.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8=
github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11/go.mod h1:dnakxebH6UwFvcvujL0LVggYQ8nEvBGjU4G/V79Nv94=
github.com/aws/aws-sdk-go-v2/config v1.32.20 h1:8VMDnWc/kEzxsI/1ngGM9mG81a8IGmIHD8KLcYGwagc=
github.com/aws/aws-sdk-go-v2/config v1.32.20/go.mod h1:PuwEpciweIXGULWeOeSTXtSbH4CW9mWdWrhdCKQI1sM=
github.com/aws/aws-sdk-go-v2/credentials v1.19.19 h1:yuFzSV1U0aRNYCQGVaTY2zW2M/L93pYHnXnrJUphYhU=
github.com/aws/aws-sdk-go-v2/credentials v1.19.19/go.mod h1:7y63L1kGzeoDlJaQ3Z578KrnmfBut96JjvJUzGwR+YE=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.25 h1:0w6dCiO8iez+YKwRhRBlL1CH/E3GTfdkuzrwj1by8vo=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.25/go.mod h1:9FDWUothyr5RCRAHc45XOiVCzUR8n/IhCYX+uVqw6vk=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25 h1:Uii3frf9ztec/ABM2/FSH9/z7PLzxfpG8h4RpkUFflQ=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25/go.mod h1:G6kntsA2GorAxDPbap6xgB2F+amSLUF8GJTi7PUoX44=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25 h1:r1+/l6m+WaUJF9HISEsNOLHSNj5EXYQxK8VX6Cz9NlA=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25/go.mod h1:cKf+D+NMDK1LndD7BowHbBZPgR9V0/5HubH0PFWvA+c=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26 h1:A1PmWU2zfkIm9EyFlJncFXL4W4phML+h8KjltUsCvNQ=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26/go.mod h1:dY4MRzXEizrD4hqtpKvWVGPX7QleSGGVY+EBolo1RmM=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10 h1:d5/908OJ4bXg8lyjeMPvXetEKqoDoLi5Owy1zNue3yg=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10/go.mod h1:a57l7Hwh+FWI+we50g5NPJHYUKeJKfXbc4w8SyXu8Ig=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18 h1:W/EyPFl9A5rXrtoilfwHYEvzHER+K4SpBPtMXi24Mos=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18/go.mod h1:UG50K+pvd/uy6xExbobg0rjqFBFZe6I3l75EPDZw4tg=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25 h1:dD3dhHNglpd98gs72my22Ndqi1hqQGllFFg1F+twfxg=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25/go.mod h1:0yAbjPfd64gG7mj85RW+fMEYdfBgCRZw8g/oWcL1pjc=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25 h1:2pQEbwf+/6EDbiit/GcBE2K4IUpMZymaA0kOz3xK978=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25/go.mod h1:KvT6NCcQ0EZ+ZkVRrlBMt04Po3ok23YELEp7WimhLhM=
github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2 h1:ie4ElCmUKS26pzrZcIk/lmt4yWjAqLLcawstyQCh298=
github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2/go.mod h1:zjsomFeX5duj+4PlMB+o4JoWTIx+G0XMyzjYrUbQkN0=
github.com/aws/aws-sdk-go-v2/service/signin v1.1.1 h1:1VwbP3qMNfxUDEXWki4rCE5iA+44VA1lokTz9HasGzw=
github.com/aws/aws-sdk-go-v2/service/signin v1.1.1/go.mod h1:vUtyoSj0OPji3kjIVSc/GlKuWEiL33f/WFxl6dmpy/A=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.19 h1:N6pIsdFOW1Kd9S4KyFKXdGRBojPPxkP32+uHFWLv4Hc=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.19/go.mod h1:3gt5WJArFooNmyLONS+h/R4J+o86II8du38IgCwj9dE=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.2 h1:hc+lBYiiTr8Zk4MTzIsQ92MeDWCIDvWGmzKUWOaBcOg=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.2/go.mod h1:hU6fqB3OJA6/ePheD47LQnxvjYk6br6PtQxs+Q9ojvk=
github.com/aws/aws-sdk-go-v2/service/sts v1.42.3 h1:ErklX/7uhSbkAAeyQD/Y1OoQ9hO3SJXQNEgksORW3Js=
github.com/aws/aws-sdk-go-v2/service/sts v1.42.3/go.mod h1:ULe4HCzfKPiR6R3HEurE3b1upEkuk8AkMrOKtaOxKO8=
github.com/aws/smithy-go v1.26.0 h1:9ouqbi+NyKP7fV3Te7UElCwdAb6Y8uk7LGwPE5tVe/s=
github.com/aws/smithy-go v1.26.0/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/kr/fs v0.1.0 h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8=
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
github.com/pkg/sftp v1.13.10 h1:+5FbKNTe5Z9aspU88DPIKJ9z2KZoaGCu6Sr6kKR/5mU=
github.com/pkg/sftp v1.13.10/go.mod h1:bJ1a7uDhrX/4OII+agvy28lzRvQrmIQuaHrcI1HbeGA=
github.com/yuin/goldmark v1.8.2 h1:kEGpgqJXdgbkhcOgBxkC0X0PmoPG1ZyoZ117rDVp4zE=
github.com/yuin/goldmark v1.8.2/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=

View File

@@ -0,0 +1,371 @@
// Package distributor adapts weatherreporter report artifacts to distributor uploads.
package distributor
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"os"
"strings"
"time"
distributorbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
distributorupload "gitea.maximumdirect.net/eric/distributor/pkg/upload"
"gitea.maximumdirect.net/eric/weatherreporter/internal/config"
)
type Client struct {
Endpoint string
TokenEnv string
Timeout time.Duration
newUploadClient uploadClientFactory
}
type UploadRequest struct {
PipelineID string
BundleID string
IdempotencyKey string
Files []UploadFile
CreatedAt time.Time
}
type UploadFile struct {
SourcePath string
BundlePath string
}
type UploadResult struct {
RunID string
Status string
UploadStatus string
StatusError string
RunStatus *RunStatus
}
type RunStatus struct {
RunID string
PipelineID string
Status string
AcceptedAt time.Time
StartedAt *time.Time
FinishedAt *time.Time
Report json.RawMessage
Error string
}
type IdempotencyConflictError struct {
Err error
}
func (e *IdempotencyConflictError) Error() string {
if e == nil || e.Err == nil {
return "distributor idempotency conflict"
}
return e.Err.Error()
}
func (e *IdempotencyConflictError) Unwrap() error {
if e == nil {
return nil
}
return e.Err
}
type uploadClientFactory func(endpoint, token string, timeout time.Duration) (uploadClient, error)
type uploadClient interface {
UploadFiles(ctx context.Context, opts uploadFilesOptions) (uploadFilesResult, error)
Status(ctx context.Context, runID string) (runStatus, error)
}
type uploadFilesOptions struct {
PipelineID string
BundleID string
IdempotencyKey string
Files []UploadFile
CreatedAt time.Time
}
type uploadFilesResult struct {
RunID string
Status string
}
type runStatus struct {
RunID string
PipelineID string
Status string
AcceptedAt time.Time
StartedAt *time.Time
FinishedAt *time.Time
Report json.RawMessage
Error string
}
const statusPollInterval = 250 * time.Millisecond
func New(cfg config.DistributorNotifyConfig) *Client {
return newClient(cfg, newDistributorUploadClient)
}
func newClient(cfg config.DistributorNotifyConfig, factory uploadClientFactory) *Client {
if factory == nil {
factory = newDistributorUploadClient
}
return &Client{
Endpoint: cfg.Endpoint,
TokenEnv: cfg.TokenEnv,
Timeout: cfg.Timeout,
newUploadClient: factory,
}
}
func (c *Client) Upload(ctx context.Context, req UploadRequest) (UploadResult, error) {
if c == nil {
return UploadResult{}, fmt.Errorf("distributor client is nil")
}
if c.Endpoint == "" {
return UploadResult{}, fmt.Errorf("distributor endpoint is required")
}
if c.TokenEnv == "" {
return UploadResult{}, fmt.Errorf("distributor token environment variable is required")
}
if req.PipelineID == "" {
return UploadResult{}, fmt.Errorf("distributor pipeline id is required")
}
if req.BundleID == "" {
return UploadResult{}, fmt.Errorf("distributor bundle id is required")
}
if req.IdempotencyKey == "" {
return UploadResult{}, fmt.Errorf("distributor idempotency key is required for bundle %q", req.BundleID)
}
if len(req.Files) == 0 {
return UploadResult{}, fmt.Errorf("distributor upload files are required for bundle %q", req.BundleID)
}
for i, file := range req.Files {
if file.SourcePath == "" {
return UploadResult{}, fmt.Errorf("distributor source path is required for bundle %q file %d", req.BundleID, i)
}
if file.BundlePath == "" {
return UploadResult{}, fmt.Errorf("distributor bundle path is required for bundle %q file %d", req.BundleID, i)
}
}
if c.newUploadClient == nil {
return UploadResult{}, fmt.Errorf("distributor upload client factory is required for endpoint %q", c.Endpoint)
}
token := os.Getenv(c.TokenEnv)
if token == "" {
return UploadResult{}, fmt.Errorf("distributor token environment variable %q is not set", c.TokenEnv)
}
uploadClient, err := c.newUploadClient(c.Endpoint, token, c.Timeout)
if err != nil {
return UploadResult{}, fmt.Errorf("create distributor upload client for endpoint %q: %w", c.Endpoint, redactToken(err, token))
}
runCtx := ctx
if runCtx == nil {
runCtx = context.Background()
}
cancel := func() {}
if c.Timeout > 0 {
runCtx, cancel = context.WithTimeout(runCtx, c.Timeout)
}
defer cancel()
result, err := uploadClient.UploadFiles(runCtx, uploadFilesOptions{
PipelineID: req.PipelineID,
BundleID: req.BundleID,
IdempotencyKey: req.IdempotencyKey,
Files: append([]UploadFile(nil), req.Files...),
CreatedAt: req.CreatedAt,
})
if err != nil {
return UploadResult{}, wrapUploadError(err, uploadErrorContext{
Endpoint: c.Endpoint,
PipelineID: req.PipelineID,
BundleID: req.BundleID,
IdempotencyKey: req.IdempotencyKey,
SourcePaths: uploadSourcePaths(req.Files),
BundlePaths: uploadBundlePaths(req.Files),
Token: token,
})
}
uploadResult := UploadResult{
RunID: result.RunID,
Status: result.Status,
UploadStatus: result.Status,
}
status, statusErr := waitForRunStatus(runCtx, uploadClient, result.RunID, c.Timeout > 0)
if status.RunID != "" || status.Status != "" {
uploadResult.RunStatus = &RunStatus{
RunID: status.RunID,
PipelineID: status.PipelineID,
Status: status.Status,
AcceptedAt: status.AcceptedAt,
StartedAt: status.StartedAt,
FinishedAt: status.FinishedAt,
Report: append(json.RawMessage(nil), status.Report...),
Error: redactTokenString(status.Error, token),
}
if status.Status != "" {
uploadResult.Status = status.Status
}
}
if statusErr != nil {
uploadResult.StatusError = redactTokenString(statusErr.Error(), token)
return uploadResult, nil
}
if status.Status == "failed" {
return uploadResult, fmt.Errorf("distributor run %q failed: %s", status.RunID, uploadResult.RunStatus.Error)
}
return uploadResult, nil
}
func waitForRunStatus(ctx context.Context, client uploadClient, runID string, poll bool) (runStatus, error) {
status, err := client.Status(ctx, runID)
if err != nil || terminalRunStatus(status.Status) || !poll {
return status, err
}
for {
timer := time.NewTimer(statusPollInterval)
select {
case <-ctx.Done():
timer.Stop()
return status, fmt.Errorf("distributor run %q did not reach terminal status before timeout: %w", runID, ctx.Err())
case <-timer.C:
}
next, err := client.Status(ctx, runID)
if err != nil {
return status, err
}
status = next
if terminalRunStatus(status.Status) {
return status, nil
}
}
}
func terminalRunStatus(status string) bool {
return status == "succeeded" || status == "failed"
}
type distributorUploadClient struct {
client *distributorupload.Client
}
func newDistributorUploadClient(endpoint, token string, timeout time.Duration) (uploadClient, error) {
httpClient := (*http.Client)(nil)
if timeout > 0 {
httpClient = &http.Client{Timeout: timeout}
}
client, err := distributorupload.NewClient(distributorupload.ClientOptions{
Endpoint: endpoint,
Token: token,
HTTPClient: httpClient,
})
if err != nil {
return nil, err
}
return distributorUploadClient{client: client}, nil
}
func (c distributorUploadClient) UploadFiles(ctx context.Context, opts uploadFilesOptions) (uploadFilesResult, error) {
files := make([]distributorbundle.BundleFile, 0, len(opts.Files))
for _, file := range opts.Files {
files = append(files, distributorbundle.BundleFile{
SourcePath: file.SourcePath,
Path: file.BundlePath,
})
}
result, err := c.client.UploadFiles(ctx, distributorupload.UploadFilesOptions{
PipelineID: opts.PipelineID,
ID: opts.BundleID,
Created: opts.CreatedAt,
IdempotencyKey: opts.IdempotencyKey,
Files: files,
})
if err != nil {
return uploadFilesResult{}, err
}
return uploadFilesResult{
RunID: result.RunID,
Status: result.Status,
}, nil
}
func (c distributorUploadClient) Status(ctx context.Context, runID string) (runStatus, error) {
status, err := c.client.Status(ctx, runID)
if err != nil {
return runStatus{}, err
}
return runStatus{
RunID: status.RunID,
PipelineID: status.PipelineID,
Status: status.Status,
AcceptedAt: status.AcceptedAt,
StartedAt: status.StartedAt,
FinishedAt: status.FinishedAt,
Report: append(json.RawMessage(nil), status.Report...),
Error: status.Error,
}, nil
}
type uploadErrorContext struct {
Endpoint string
PipelineID string
BundleID string
IdempotencyKey string
SourcePaths []string
BundlePaths []string
Token string
}
func wrapUploadError(err error, ctx uploadErrorContext) error {
var conflict *distributorupload.IdempotencyConflictError
isConflict := errors.As(err, &conflict)
err = redactToken(err, ctx.Token)
if isConflict {
return &IdempotencyConflictError{
Err: fmt.Errorf("upload distributor bundle %q to pipeline %q at endpoint %q with idempotency key %q from sources %q as bundle paths %q: idempotency conflict: %w", ctx.BundleID, ctx.PipelineID, ctx.Endpoint, ctx.IdempotencyKey, ctx.SourcePaths, ctx.BundlePaths, err),
}
}
return fmt.Errorf("upload distributor bundle %q to pipeline %q at endpoint %q with idempotency key %q from sources %q as bundle paths %q: %w", ctx.BundleID, ctx.PipelineID, ctx.Endpoint, ctx.IdempotencyKey, ctx.SourcePaths, ctx.BundlePaths, err)
}
func uploadSourcePaths(files []UploadFile) []string {
paths := make([]string, 0, len(files))
for _, file := range files {
paths = append(paths, file.SourcePath)
}
return paths
}
func uploadBundlePaths(files []UploadFile) []string {
paths := make([]string, 0, len(files))
for _, file := range files {
paths = append(paths, file.BundlePath)
}
return paths
}
func redactToken(err error, token string) error {
if err == nil || token == "" {
return err
}
return errors.New(redactTokenString(err.Error(), token))
}
func redactTokenString(value, token string) string {
if token == "" {
return value
}
return strings.ReplaceAll(value, token, "[redacted]")
}

View File

@@ -0,0 +1,406 @@
package distributor
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
distributorupload "gitea.maximumdirect.net/eric/distributor/pkg/upload"
"gitea.maximumdirect.net/eric/weatherreporter/internal/config"
)
func TestUploadUsesConfiguredClientAndFiles(t *testing.T) {
cfg := config.Defaults().Notify.Distributor
cfg.Endpoint = "https://distributor.example.test"
cfg.TokenEnv = "DISTRIBUTOR_UPLOAD_TOKEN"
cfg.Timeout = 15 * time.Second
t.Setenv(cfg.TokenEnv, "secret-token")
factory := &fakeUploadFactory{
client: &fakeUploadClient{
result: uploadFilesResult{RunID: "run-123", Status: "accepted"},
status: runStatus{RunID: "run-123", PipelineID: "reports", Status: "succeeded", Report: json.RawMessage(`{"actions":[{"action":"replace_older"}]}`)},
},
}
client := newClient(cfg, factory.newClient)
result, err := client.Upload(context.Background(), UploadRequest{
PipelineID: "weatherreporter.daily",
BundleID: "weatherreporter.home.daily.run",
IdempotencyKey: "weatherreporter.home.daily.run",
Files: []UploadFile{
{SourcePath: "/tmp/report.md", BundlePath: "2026-06-07/daily/report.md"},
{SourcePath: "/tmp/report.md", BundlePath: "2026-06-07/daily/latest.md"},
},
CreatedAt: time.Date(2026, 6, 7, 12, 0, 0, 123, time.UTC),
})
if err != nil {
t.Fatalf("Upload() error = %v", err)
}
if result.RunID != "run-123" || result.Status != "succeeded" || result.UploadStatus != "accepted" {
t.Fatalf("result = %#v, want accepted run", result)
}
if result.RunStatus == nil || result.RunStatus.PipelineID != "reports" || !strings.Contains(string(result.RunStatus.Report), "replace_older") {
t.Fatalf("RunStatus = %#v, want parsed run report", result.RunStatus)
}
if factory.endpoint != cfg.Endpoint {
t.Fatalf("factory endpoint = %q, want %q", factory.endpoint, cfg.Endpoint)
}
if factory.token != "secret-token" {
t.Fatalf("factory token = %q, want secret-token", factory.token)
}
if factory.timeout != 15*time.Second {
t.Fatalf("factory timeout = %s, want 15s", factory.timeout)
}
got := factory.client.opts
if got.PipelineID != "weatherreporter.daily" {
t.Fatalf("PipelineID = %q, want weatherreporter.daily", got.PipelineID)
}
if got.BundleID != "weatherreporter.home.daily.run" {
t.Fatalf("BundleID = %q, want weatherreporter.home.daily.run", got.BundleID)
}
if got.IdempotencyKey != "weatherreporter.home.daily.run" {
t.Fatalf("IdempotencyKey = %q, want weatherreporter.home.daily.run", got.IdempotencyKey)
}
if len(got.Files) != 2 {
t.Fatalf("files = %#v, want two mappings", got.Files)
}
if got.Files[0].SourcePath != "/tmp/report.md" || got.Files[0].BundlePath != "2026-06-07/daily/report.md" {
t.Fatalf("first file = %#v, want archive mapping", got.Files[0])
}
if got.Files[1].SourcePath != "/tmp/report.md" || got.Files[1].BundlePath != "2026-06-07/daily/latest.md" {
t.Fatalf("second file = %#v, want latest mapping", got.Files[1])
}
if got.CreatedAt.IsZero() {
t.Fatal("CreatedAt is zero, want generated report timestamp")
}
if factory.client.statusRunID != "run-123" {
t.Fatalf("Status runID = %q, want run-123", factory.client.statusRunID)
}
}
func TestUploadRejectsMissingInputs(t *testing.T) {
cfg := config.Defaults().Notify.Distributor
t.Setenv(cfg.TokenEnv, "secret-token")
tests := []struct {
name string
mutate func(*Client, *UploadRequest)
wantErr string
}{
{
name: "Token",
mutate: func(c *Client, req *UploadRequest) {
t.Setenv(c.TokenEnv, "")
},
wantErr: "token environment variable",
},
{
name: "PipelineID",
mutate: func(c *Client, req *UploadRequest) {
req.PipelineID = ""
},
wantErr: "pipeline id is required",
},
{
name: "Files",
mutate: func(c *Client, req *UploadRequest) {
req.Files = nil
},
wantErr: "upload files are required",
},
{
name: "SourcePath",
mutate: func(c *Client, req *UploadRequest) {
req.Files[0].SourcePath = ""
},
wantErr: "source path is required",
},
{
name: "BundlePath",
mutate: func(c *Client, req *UploadRequest) {
req.Files[0].BundlePath = ""
},
wantErr: "bundle path is required",
},
{
name: "UploadClientFactory",
mutate: func(c *Client, req *UploadRequest) {
c.newUploadClient = nil
},
wantErr: "upload client factory is required",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Setenv(cfg.TokenEnv, "secret-token")
client := newClient(cfg, (&fakeUploadFactory{client: &fakeUploadClient{}}).newClient)
req := validUploadRequest()
tt.mutate(client, &req)
_, err := client.Upload(context.Background(), req)
if err == nil {
t.Fatal("Upload() error = nil, want error")
}
if !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
}
if strings.Contains(err.Error(), "secret-token") {
t.Fatalf("error = %q, want no token value", err.Error())
}
})
}
}
func TestUploadWrapsFactoryErrorWithoutToken(t *testing.T) {
cfg := config.Defaults().Notify.Distributor
cfg.Endpoint = "https://distributor.example.test"
t.Setenv(cfg.TokenEnv, "secret-token")
factory := &fakeUploadFactory{
err: fmt.Errorf("factory failed with secret-token"),
}
client := newClient(cfg, factory.newClient)
_, err := client.Upload(context.Background(), validUploadRequest())
if err == nil {
t.Fatal("Upload() error = nil, want error")
}
if strings.Contains(err.Error(), "secret-token") {
t.Fatalf("error = %q, want no token value", err.Error())
}
if !strings.Contains(err.Error(), cfg.Endpoint) {
t.Fatalf("error = %q, want endpoint context", err.Error())
}
}
func TestUploadWrapsUploadFailureWithContextWithoutToken(t *testing.T) {
cfg := config.Defaults().Notify.Distributor
cfg.Endpoint = "https://distributor.example.test"
t.Setenv(cfg.TokenEnv, "secret-token")
factory := &fakeUploadFactory{
client: &fakeUploadClient{err: fmt.Errorf("server rejected secret-token")},
}
client := newClient(cfg, factory.newClient)
req := validUploadRequest()
_, err := client.Upload(context.Background(), req)
if err == nil {
t.Fatal("Upload() error = nil, want error")
}
for _, want := range []string{cfg.Endpoint, req.PipelineID, req.BundleID, req.IdempotencyKey, req.Files[0].SourcePath, req.Files[0].BundlePath, req.Files[1].BundlePath} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("error = %q, want context %q", err.Error(), want)
}
}
if strings.Contains(err.Error(), "secret-token") {
t.Fatalf("error = %q, want no token value", err.Error())
}
}
func TestUploadReturnsAcceptedWhenStatusLookupFails(t *testing.T) {
cfg := config.Defaults().Notify.Distributor
cfg.Endpoint = "https://distributor.example.test"
t.Setenv(cfg.TokenEnv, "secret-token")
factory := &fakeUploadFactory{
client: &fakeUploadClient{
result: uploadFilesResult{RunID: "run-123", Status: "accepted"},
statusErr: fmt.Errorf("status rejected secret-token"),
},
}
client := newClient(cfg, factory.newClient)
result, err := client.Upload(context.Background(), validUploadRequest())
if err != nil {
t.Fatalf("Upload() error = %v, want accepted upload despite status lookup failure", err)
}
if result.Status != "accepted" || result.StatusError == "" {
t.Fatalf("result = %#v, want accepted status with status error", result)
}
if strings.Contains(result.StatusError, "secret-token") {
t.Fatalf("StatusError = %q, want token redacted", result.StatusError)
}
}
func TestUploadPollsUntilTerminalStatus(t *testing.T) {
cfg := config.Defaults().Notify.Distributor
cfg.Endpoint = "https://distributor.example.test"
cfg.Timeout = 2 * time.Second
t.Setenv(cfg.TokenEnv, "secret-token")
factory := &fakeUploadFactory{
client: &fakeUploadClient{
result: uploadFilesResult{RunID: "run-123", Status: "accepted"},
statuses: []runStatus{
{RunID: "run-123", Status: "accepted"},
{RunID: "run-123", Status: "succeeded", Report: json.RawMessage(`{"actions":[{"action":"replace_older"}]}`)},
},
},
}
client := newClient(cfg, factory.newClient)
result, err := client.Upload(context.Background(), validUploadRequest())
if err != nil {
t.Fatalf("Upload() error = %v", err)
}
if result.Status != "succeeded" || result.RunStatus == nil || !strings.Contains(string(result.RunStatus.Report), "replace_older") {
t.Fatalf("result = %#v, want terminal succeeded status with run report", result)
}
if factory.client.statusCalls != 2 {
t.Fatalf("status calls = %d, want 2", factory.client.statusCalls)
}
}
func TestUploadReturnsLatestStatusWhenPollingTimesOut(t *testing.T) {
cfg := config.Defaults().Notify.Distributor
cfg.Endpoint = "https://distributor.example.test"
cfg.Timeout = time.Millisecond
t.Setenv(cfg.TokenEnv, "secret-token")
factory := &fakeUploadFactory{
client: &fakeUploadClient{
result: uploadFilesResult{RunID: "run-123", Status: "accepted"},
status: runStatus{RunID: "run-123", Status: "running"},
},
}
client := newClient(cfg, factory.newClient)
result, err := client.Upload(context.Background(), validUploadRequest())
if err != nil {
t.Fatalf("Upload() error = %v, want accepted upload with status timeout recorded", err)
}
if result.Status != "running" || result.StatusError == "" {
t.Fatalf("result = %#v, want latest status and status timeout", result)
}
}
func TestUploadFailsWhenDistributorRunFailed(t *testing.T) {
cfg := config.Defaults().Notify.Distributor
cfg.Endpoint = "https://distributor.example.test"
t.Setenv(cfg.TokenEnv, "secret-token")
factory := &fakeUploadFactory{
client: &fakeUploadClient{
result: uploadFilesResult{RunID: "run-123", Status: "accepted"},
status: runStatus{
RunID: "run-123",
Status: "failed",
Error: "destination rejected secret-token",
Report: json.RawMessage(`{"actions":[{"action":"failed"}]}`),
},
},
}
client := newClient(cfg, factory.newClient)
result, err := client.Upload(context.Background(), validUploadRequest())
if err == nil {
t.Fatal("Upload() error = nil, want failed distributor run error")
}
if result.RunStatus == nil || result.RunStatus.Status != "failed" || !strings.Contains(string(result.RunStatus.Report), "failed") {
t.Fatalf("result = %#v, want failed run status report", result)
}
if strings.Contains(err.Error(), "secret-token") || strings.Contains(result.RunStatus.Error, "secret-token") {
t.Fatalf("error/result leaked token: err=%q result=%#v", err.Error(), result)
}
}
func TestUploadPreservesIdempotencyConflictDiagnosis(t *testing.T) {
cfg := config.Defaults().Notify.Distributor
cfg.Endpoint = "https://distributor.example.test"
t.Setenv(cfg.TokenEnv, "secret-token")
factory := &fakeUploadFactory{
client: &fakeUploadClient{
err: &distributorupload.IdempotencyConflictError{
HTTPError: distributorupload.HTTPError{
StatusCode: 409,
Status: "409 Conflict",
Message: "conflicting upload for secret-token",
},
},
},
}
client := newClient(cfg, factory.newClient)
_, err := client.Upload(context.Background(), validUploadRequest())
if err == nil {
t.Fatal("Upload() error = nil, want error")
}
var conflict *IdempotencyConflictError
if !errors.As(err, &conflict) {
t.Fatalf("Upload() error = %T %v, want IdempotencyConflictError", err, err)
}
if !strings.Contains(err.Error(), "idempotency conflict") {
t.Fatalf("error = %q, want idempotency conflict diagnosis", err.Error())
}
if strings.Contains(err.Error(), "secret-token") {
t.Fatalf("error = %q, want no token value", err.Error())
}
}
func validUploadRequest() UploadRequest {
return UploadRequest{
PipelineID: "weatherreporter.daily",
BundleID: "weatherreporter.home.daily.run",
IdempotencyKey: "weatherreporter.home.daily.run",
Files: []UploadFile{
{SourcePath: "/tmp/report.md", BundlePath: "2026-06-07/daily/report.md"},
{SourcePath: "/tmp/report.md", BundlePath: "2026-06-07/daily/latest.md"},
},
CreatedAt: time.Date(2026, 6, 7, 12, 0, 0, 123, time.UTC),
}
}
type fakeUploadFactory struct {
endpoint string
token string
timeout time.Duration
client *fakeUploadClient
err error
}
func (f *fakeUploadFactory) newClient(endpoint, token string, timeout time.Duration) (uploadClient, error) {
f.endpoint = endpoint
f.token = token
f.timeout = timeout
if f.err != nil {
return nil, f.err
}
return f.client, nil
}
type fakeUploadClient struct {
opts uploadFilesOptions
statusRunID string
statusCalls int
result uploadFilesResult
status runStatus
statuses []runStatus
err error
statusErr error
}
func (c *fakeUploadClient) UploadFiles(ctx context.Context, opts uploadFilesOptions) (uploadFilesResult, error) {
c.opts = opts
if c.err != nil {
return uploadFilesResult{}, c.err
}
return c.result, nil
}
func (c *fakeUploadClient) Status(ctx context.Context, runID string) (runStatus, error) {
c.statusRunID = runID
c.statusCalls++
if c.statusErr != nil {
return runStatus{}, c.statusErr
}
if len(c.statuses) > 0 {
index := c.statusCalls - 1
if index >= len(c.statuses) {
index = len(c.statuses) - 1
}
return c.statuses[index], nil
}
return c.status, nil
}

View File

@@ -109,10 +109,10 @@ func (c *Client) FetchBundle(ctx context.Context) (*forecast.Bundle, error) {
if err := builder.fetchDiscussion(ctx); err != nil { if err := builder.fetchDiscussion(ctx); err != nil {
return nil, err return nil, err
} }
if err := builder.addStub("daily", "daily forecast data is not available from the weather API yet"); err != nil { if err := builder.fetchWeatherStory(ctx); err != nil {
return nil, err return nil, err
} }
if err := builder.addStub("weather_story", "NWS weather story is not available from the weather API yet"); err != nil { if err := builder.addStub("daily", "daily forecast data is not available from the weather API yet"); err != nil {
return nil, err return nil, err
} }
@@ -246,6 +246,27 @@ func (b *bundleBuilder) fetchDiscussion(ctx context.Context) error {
return nil return nil
} }
func (b *bundleBuilder) fetchWeatherStory(ctx context.Context) error {
raw, source, err := b.client.fetch(ctx, "weather_story", "/weatherstories/latest", queryOptions{omitUnits: true})
if err != nil {
return err
}
if raw == nil {
return b.handleMissing(&source, "NWS weather story data is missing", false)
}
var story forecast.WeatherStory
if err := decodeSource(raw, &story); err != nil {
return b.handleMalformed(&source, err, false)
}
if !story.StartTime.IsZero() {
source.IssuedAt = &story.StartTime
}
source.UpdatedAt = story.UpdatedAt
b.bundle.WeatherStory = &story
b.addSource(source)
return nil
}
func (b *bundleBuilder) addStub(sourceName string, message string) error { func (b *bundleBuilder) addStub(sourceName string, message string) error {
source := forecast.Source{ source := forecast.Source{
Name: sourceName, Name: sourceName,
@@ -307,6 +328,7 @@ type queryOptions struct {
precision bool precision bool
timezone bool timezone bool
allowNull bool allowNull bool
omitUnits bool
} }
type envelope struct { type envelope struct {
@@ -366,7 +388,9 @@ func (c *Client) endpointURL(endpoint string, opts queryOptions) *url.URL {
reqURL.Path = path.Join(c.baseURL.Path, endpoint) reqURL.Path = path.Join(c.baseURL.Path, endpoint)
query := reqURL.Query() query := reqURL.Query()
query.Set("format", c.format) query.Set("format", c.format)
query.Set("units", c.units) if !opts.omitUnits {
query.Set("units", c.units)
}
if opts.precision { if opts.precision {
query.Set("precision", strconv.Itoa(c.precision)) query.Set("precision", strconv.Itoa(c.precision))
} }

View File

@@ -49,11 +49,17 @@ func TestFetchBundleFromFixtures(t *testing.T) {
if bundle.Discussion.LongTerm == nil || bundle.Discussion.LongTerm.Text != "Warmer temperatures and periodic rain chances continue into the weekend." { if bundle.Discussion.LongTerm == nil || bundle.Discussion.LongTerm.Text != "Warmer temperatures and periodic rain chances continue into the weekend." {
t.Fatalf("Discussion.LongTerm = %#v, want long-term AFD text", bundle.Discussion.LongTerm) t.Fatalf("Discussion.LongTerm = %#v, want long-term AFD text", bundle.Discussion.LongTerm)
} }
if bundle.WeatherStory == nil || bundle.WeatherStory.Title != "Several Chances for Rain Through Monday" {
t.Fatalf("WeatherStory = %#v, want latest weather story", bundle.WeatherStory)
}
if bundle.WeatherStory.UpdatedAt == nil {
t.Fatalf("WeatherStory.UpdatedAt = nil, want update timestamp")
}
if len(bundle.Sources) != 8 { if len(bundle.Sources) != 8 {
t.Fatalf("Sources length = %d, want 8", len(bundle.Sources)) t.Fatalf("Sources length = %d, want 8", len(bundle.Sources))
} }
if len(bundle.Warnings) != 2 { if len(bundle.Warnings) != 1 {
t.Fatalf("Warnings length = %d, want daily and weather story warnings", len(bundle.Warnings)) t.Fatalf("Warnings length = %d, want daily warning", len(bundle.Warnings))
} }
if !containsPath(requested, "/forecast/hourly") || containsPath(requested, "/forecast/hourly/today") { if !containsPath(requested, "/forecast/hourly") || containsPath(requested, "/forecast/hourly/today") {
t.Fatalf("requested paths = %v, want full hourly endpoint only", requested) t.Fatalf("requested paths = %v, want full hourly endpoint only", requested)
@@ -61,6 +67,9 @@ func TestFetchBundleFromFixtures(t *testing.T) {
if !containsPath(requested, "/forecast/narrative") || containsPath(requested, "/forecast/narrative/today") { if !containsPath(requested, "/forecast/narrative") || containsPath(requested, "/forecast/narrative/today") {
t.Fatalf("requested paths = %v, want full narrative endpoint only", requested) t.Fatalf("requested paths = %v, want full narrative endpoint only", requested)
} }
if !containsPath(requested, "/weatherstories/latest") {
t.Fatalf("requested paths = %v, want weather story endpoint", requested)
}
} }
func TestFetchBundleBuildsExpectedQueries(t *testing.T) { func TestFetchBundleBuildsExpectedQueries(t *testing.T) {
@@ -74,8 +83,17 @@ func TestFetchBundleBuildsExpectedQueries(t *testing.T) {
} }
for _, rawURL := range requested { for _, rawURL := range requested {
if !strings.Contains(rawURL, "format=json") || !strings.Contains(rawURL, "units=us") { if !strings.Contains(rawURL, "format=json") {
t.Fatalf("request %q missing format=json or units=us", rawURL) t.Fatalf("request %q missing format=json", rawURL)
}
if strings.HasPrefix(rawURL, "/weatherstories/") {
if strings.Contains(rawURL, "units=") || strings.Contains(rawURL, "precision=") || strings.Contains(rawURL, "tz=") {
t.Fatalf("weather story request %q should use format only", rawURL)
}
continue
}
if !strings.Contains(rawURL, "units=us") {
t.Fatalf("request %q missing units=us", rawURL)
} }
if strings.HasPrefix(rawURL, "/forecast/") { if strings.HasPrefix(rawURL, "/forecast/") {
if !strings.Contains(rawURL, "precision=1") || !strings.Contains(rawURL, "tz=America%2FChicago") { if !strings.Contains(rawURL, "precision=1") || !strings.Contains(rawURL, "tz=America%2FChicago") {
@@ -99,6 +117,16 @@ func TestFetchBundleRecordsSourceHash(t *testing.T) {
if observation.DataSHA256 != want { if observation.DataSHA256 != want {
t.Fatalf("DataSHA256 = %q, want %q", observation.DataSHA256, want) t.Fatalf("DataSHA256 = %q, want %q", observation.DataSHA256, want)
} }
story := sourceByName(t, bundle.Sources, "weather_story")
if story.Endpoint != "/weatherstories/latest" {
t.Fatalf("weather story endpoint = %q, want /weatherstories/latest", story.Endpoint)
}
if story.DataSHA256 != hashFixtureData(t, "weather_story.json") {
t.Fatalf("weather story DataSHA256 = %q, want fixture hash", story.DataSHA256)
}
if story.IssuedAt == nil || story.UpdatedAt == nil {
t.Fatalf("weather story source timestamps = issued %#v updated %#v, want both", story.IssuedAt, story.UpdatedAt)
}
} }
func TestHTTPErrorIsActionable(t *testing.T) { func TestHTTPErrorIsActionable(t *testing.T) {
@@ -169,7 +197,7 @@ func TestMissingSourcePolicyWarnNoneError(t *testing.T) {
wantWarns int wantWarns int
wantSource bool wantSource bool
}{ }{
{name: "warn", policy: config.MissingSourceWarn, wantWarns: 3, wantSource: true}, {name: "warn", policy: config.MissingSourceWarn, wantWarns: 2, wantSource: true},
{name: "none", policy: config.MissingSourceNone, wantWarns: 0, wantSource: true}, {name: "none", policy: config.MissingSourceNone, wantWarns: 0, wantSource: true},
{name: "error", policy: config.MissingSourceError, wantErr: true}, {name: "error", policy: config.MissingSourceError, wantErr: true},
} }
@@ -230,6 +258,45 @@ func TestMalformedNonRequiredSourceUsesPolicy(t *testing.T) {
} }
} }
func TestMissingWeatherStoryUsesPolicy(t *testing.T) {
server := fixtureServer(t, map[string]handlerOverride{
"/weatherstories/latest": {status: http.StatusOK, body: `{"data": null}`},
}, nil)
client := newTestClient(t, server.URL+"/", map[string]config.MissingSourcePolicy{
"weather_story": config.MissingSourceWarn,
})
bundle, err := client.FetchBundle(context.Background())
if err != nil {
t.Fatalf("FetchBundle() error = %v", err)
}
if bundle.WeatherStory != nil {
t.Fatalf("WeatherStory = %#v, want nil for missing source", bundle.WeatherStory)
}
source := sourceByName(t, bundle.Sources, "weather_story")
if !source.Missing || len(source.Warnings) != 1 {
t.Fatalf("weather_story source = %#v, want missing source warning", source)
}
}
func TestMalformedWeatherStoryUsesPolicy(t *testing.T) {
server := fixtureServer(t, map[string]handlerOverride{
"/weatherstories/latest": {status: http.StatusOK, body: `{"data": {"startTime": 123}}`},
}, nil)
client := newTestClient(t, server.URL+"/", map[string]config.MissingSourcePolicy{
"weather_story": config.MissingSourceWarn,
})
bundle, err := client.FetchBundle(context.Background())
if err != nil {
t.Fatalf("FetchBundle() error = %v", err)
}
source := sourceByName(t, bundle.Sources, "weather_story")
if !source.Missing || len(source.Warnings) != 1 || source.Warnings[0].Code != "malformed_source" {
t.Fatalf("weather_story source = %#v, want malformed source warning", source)
}
}
func TestContextCancellation(t *testing.T) { func TestContextCancellation(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
<-r.Context().Done() <-r.Context().Done()
@@ -296,12 +363,13 @@ type handlerOverride struct {
func fixtureServer(t *testing.T, overrides map[string]handlerOverride, requested *[]string) *httptest.Server { func fixtureServer(t *testing.T, overrides map[string]handlerOverride, requested *[]string) *httptest.Server {
t.Helper() t.Helper()
fixtures := map[string]string{ fixtures := map[string]string{
"/observations": "observations.json", "/observations": "observations.json",
"/conditions/current": "current.json", "/conditions/current": "current.json",
"/forecast/hourly": "hourly.json", "/forecast/hourly": "hourly.json",
"/forecast/narrative": "narrative.json", "/forecast/narrative": "narrative.json",
"/alerts/active": "alerts.json", "/alerts/active": "alerts.json",
"/discussion": "discussion.json", "/discussion": "discussion.json",
"/weatherstories/latest": "weather_story.json",
} }
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if requested != nil { if requested != nil {

View File

@@ -0,0 +1,14 @@
{
"data": {
"officeId": "LSX",
"startTime": "2026-05-30T08:46:00Z",
"endTime": "2026-05-31T11:00:00Z",
"updatedAt": "2026-05-30T09:00:34Z",
"title": "Several Chances for Rain Through Monday",
"description": "A stagnant weather pattern with low pressure over the Great Plains and high pressure over the Great Lakes will continue to produce scattered showers and thunderstorms, for areas mainly along and west of the Mississippi River today and Sunday.",
"altText": "This slide shows the forecast for today through Tuesday with icons for showers and thunderstorms and a picture of a cumulonimbus cloud on the right side.",
"priority": false,
"order": 1,
"downloadUrl": "https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f"
}
}

View File

@@ -3,10 +3,12 @@ package app
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"path/filepath" "path/filepath"
"time" "time"
distributoradapter "gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/distributor"
"gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/scriptorium" "gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/scriptorium"
"gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/weatherapi" "gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/weatherapi"
"gitea.maximumdirect.net/eric/weatherreporter/internal/briefing" "gitea.maximumdirect.net/eric/weatherreporter/internal/briefing"
@@ -45,6 +47,7 @@ type GenerateRequest struct {
Date time.Time Date time.Time
StormStart time.Time StormStart time.Time
StormEnd time.Time StormEnd time.Time
Notifier Notifier
} }
type BatchRequest struct { type BatchRequest struct {
@@ -54,6 +57,7 @@ type BatchRequest struct {
OutputDir string OutputDir string
Renderer Renderer Renderer Renderer
Store state.Store Store state.Store
Notifier Notifier
} }
type FetchBundleRequest struct { type FetchBundleRequest struct {
@@ -73,6 +77,7 @@ type ReportRequest struct {
OutputPath string OutputPath string
Renderer Renderer Renderer Renderer
Store state.Store Store state.Store
Notifier Notifier
} }
type BriefingResult struct { type BriefingResult struct {
@@ -81,19 +86,21 @@ type BriefingResult struct {
} }
type ReportResult struct { type ReportResult struct {
Briefing briefing.Package Briefing briefing.Package
BriefingPath string BriefingPath string
DataPackage promptinput.Package DataPackage promptinput.Package
DataPackagePath string DataPackagePath string
PreflightPath string PreflightPath string
ReportPath string ReportPath string
OutputPath string OutputPath string
Metadata state.Metadata NotificationPath string
MetadataPath string Metadata state.Metadata
PriorSnapshot *state.PriorSnapshot MetadataPath string
RecentChanges []changes.Change PriorSnapshot *state.PriorSnapshot
RenderResult *scriptorium.RenderResult RecentChanges []changes.Change
RunResult *scriptorium.RunResult RenderResult *scriptorium.RenderResult
RunResult *scriptorium.RunResult
Notification *NotificationResult
} }
type BatchResult struct { type BatchResult struct {
@@ -107,20 +114,25 @@ type BatchResult struct {
} }
type BatchReportResult struct { type BatchReportResult struct {
ReportID report.ID `json:"reportId"` ReportID report.ID `json:"reportId"`
ReportName string `json:"reportName"` ReportName string `json:"reportName"`
PromptID string `json:"promptId"` PromptID string `json:"promptId"`
RunID string `json:"runId"` RunID string `json:"runId"`
Status string `json:"status"` Status string `json:"status"`
Error string `json:"error,omitempty"` Error string `json:"error,omitempty"`
GeneratedAt time.Time `json:"generatedAt"` NotificationStatus string `json:"notificationStatus,omitempty"`
ValidPeriod timeutil.Period `json:"validPeriod"` NotificationRunID string `json:"notificationRunId,omitempty"`
BriefingPath string `json:"briefingPath,omitempty"` NotificationPipelineID string `json:"notificationPipelineId,omitempty"`
DataPackagePath string `json:"dataPackagePath,omitempty"` NotificationError string `json:"notificationError,omitempty"`
PreflightPath string `json:"preflightPath,omitempty"` NotificationPath string `json:"notificationPath,omitempty"`
ReportPath string `json:"reportPath,omitempty"` GeneratedAt time.Time `json:"generatedAt"`
OutputPath string `json:"outputPath,omitempty"` ValidPeriod timeutil.Period `json:"validPeriod"`
MetadataPath string `json:"metadataPath,omitempty"` BriefingPath string `json:"briefingPath,omitempty"`
DataPackagePath string `json:"dataPackagePath,omitempty"`
PreflightPath string `json:"preflightPath,omitempty"`
ReportPath string `json:"reportPath,omitempty"`
OutputPath string `json:"outputPath,omitempty"`
MetadataPath string `json:"metadataPath,omitempty"`
} }
type BatchError struct { type BatchError struct {
@@ -139,6 +151,55 @@ type Renderer interface {
Run(context.Context, scriptorium.RunRequest) (*scriptorium.RunResult, error) Run(context.Context, scriptorium.RunRequest) (*scriptorium.RunResult, error)
} }
type Notifier interface {
Notify(context.Context, NotificationRequest) (*NotificationResult, error)
}
type NotificationRequest struct {
ReportID report.ID
RunID string
PipelineID string
BundleID string
IdempotencyKey string
ReportPath string
BundlePaths []string
CreatedAt time.Time
}
type NotificationResult struct {
BundleID string
IdempotencyKey string
RunID string
Status string
UploadStatus string
StatusError string
PipelineID string
AcceptedAt time.Time
StartedAt *time.Time
FinishedAt *time.Time
Report []byte
Error string
}
type NotificationError struct {
Request NotificationRequest
Err error
}
func (e *NotificationError) Error() string {
if e == nil || e.Err == nil {
return "notification failed"
}
return e.Err.Error()
}
func (e *NotificationError) Unwrap() error {
if e == nil {
return nil
}
return e.Err
}
func Generate(ctx context.Context, req GenerateRequest) error { func Generate(ctx context.Context, req GenerateRequest) error {
now := req.Now now := req.Now
if now.IsZero() { if now.IsZero() {
@@ -153,6 +214,7 @@ func Generate(ctx context.Context, req GenerateRequest) error {
Config: req.Config, Config: req.Config,
Resolved: resolved, Resolved: resolved,
OutputPath: req.OutputPath, OutputPath: req.OutputPath,
Notifier: req.Notifier,
}) })
return err return err
} }
@@ -211,10 +273,20 @@ func RunBatchDetailed(ctx context.Context, req BatchRequest) (*BatchResult, erro
OutputPath: outputPath, OutputPath: outputPath,
Renderer: req.Renderer, Renderer: req.Renderer,
Store: store, Store: store,
Notifier: req.Notifier,
}) })
if err != nil { if err != nil {
item.Status = "failed" item.Status = "failed"
item.Error = err.Error() item.Error = err.Error()
var notificationErr *NotificationError
if errors.As(err, &notificationErr) {
item.NotificationStatus = "failed"
item.NotificationError = notificationErr.Error()
item.NotificationPipelineID = notificationErr.Request.PipelineID
if paths, pathErr := store.Paths(resolved); pathErr == nil {
item.NotificationPath = paths.Notification
}
}
result.Failed++ result.Failed++
} else { } else {
item.Status = "succeeded" item.Status = "succeeded"
@@ -224,6 +296,12 @@ func RunBatchDetailed(ctx context.Context, req BatchRequest) (*BatchResult, erro
item.ReportPath = reportResult.ReportPath item.ReportPath = reportResult.ReportPath
item.OutputPath = reportResult.OutputPath item.OutputPath = reportResult.OutputPath
item.MetadataPath = reportResult.MetadataPath item.MetadataPath = reportResult.MetadataPath
item.NotificationPath = reportResult.NotificationPath
if reportResult.Notification != nil {
item.NotificationStatus = reportResult.Notification.Status
item.NotificationRunID = reportResult.Notification.RunID
item.NotificationPipelineID = reportResult.Notification.PipelineID
}
result.Succeeded++ result.Succeeded++
} }
result.Reports = append(result.Reports, item) result.Reports = append(result.Reports, item)
@@ -481,23 +559,234 @@ func GenerateReport(ctx context.Context, req ReportRequest) (*ReportResult, erro
return nil, runErr return nil, runErr
} }
notification, notificationPath, err := notifyReport(ctx, req.Config, req.Resolved, reportPath, metadata, req.Notifier, store)
if notificationPath != "" {
metadata.NotificationPath = notificationPath
metadataPath, metadataErr = store.SaveMetadata(ctx, metadata)
if metadataErr != nil {
return nil, metadataErr
}
}
if err != nil {
return nil, err
}
return &ReportResult{ return &ReportResult{
Briefing: briefingPackage, Briefing: briefingPackage,
BriefingPath: briefingPath, BriefingPath: briefingPath,
DataPackage: dataPackage, DataPackage: dataPackage,
DataPackagePath: dataPackagePath, DataPackagePath: dataPackagePath,
PreflightPath: preflightPath, PreflightPath: preflightPath,
ReportPath: reportPath, ReportPath: reportPath,
OutputPath: outputPath, OutputPath: outputPath,
Metadata: metadata, NotificationPath: notificationPath,
MetadataPath: metadataPath, Metadata: metadata,
PriorSnapshot: priorSnapshot, MetadataPath: metadataPath,
RecentChanges: recentChanges, PriorSnapshot: priorSnapshot,
RenderResult: renderResult, RecentChanges: recentChanges,
RunResult: runResult, RenderResult: renderResult,
RunResult: runResult,
Notification: notification,
}, nil }, nil
} }
func notifyReport(ctx context.Context, cfg config.Config, resolved report.Resolved, reportPath string, metadata state.Metadata, notifier Notifier, store state.Store) (*NotificationResult, string, error) {
notifier, enabled := reportNotifier(cfg, notifier)
if !enabled {
return nil, "", nil
}
notificationRequest, err := buildNotificationRequest(cfg, resolved, reportPath, metadata)
if err != nil {
notificationPath, saveErr := saveNotificationArtifact(ctx, store, resolved, cfg, metadata, NotificationRequest{}, nil, err)
if saveErr != nil {
return nil, "", saveErr
}
return nil, notificationPath, err
}
result, err := notifier.Notify(ctx, notificationRequest)
notificationPath, saveErr := saveNotificationArtifact(ctx, store, resolved, cfg, metadata, notificationRequest, result, err)
if saveErr != nil {
return nil, "", saveErr
}
if err != nil {
return result, notificationPath, &NotificationError{
Request: notificationRequest,
Err: fmt.Errorf("notify report %q run %q from managed report %q: %w", resolved.Definition.ID, metadata.RunID, reportPath, err),
}
}
return result, notificationPath, nil
}
func reportNotifier(cfg config.Config, notifier Notifier) (Notifier, bool) {
if !cfg.Notify.Distributor.Enabled {
return noopNotifier{}, false
}
if notifier != nil {
return notifier, true
}
return distributorNotifier{
client: distributoradapter.New(cfg.Notify.Distributor),
}, true
}
func buildNotificationRequest(cfg config.Config, resolved report.Resolved, reportPath string, metadata state.Metadata) (NotificationRequest, error) {
values := config.DistributorTemplateValues{
LocationID: cfg.Location.ID,
ReportID: string(resolved.Definition.ID),
RunID: metadata.RunID,
ArtifactGroup: resolved.Definition.ArtifactGroup,
BatchOutputName: resolved.Definition.BatchOutputName,
}
if err := addDistributorValidPeriodValues(&values, resolved.ValidPeriod, cfg.WeatherAPI.Timezone); err != nil {
return NotificationRequest{}, err
}
bundleID, err := config.RenderDistributorBundleID(cfg.Notify.Distributor.BundleIDTemplate, values)
if err != nil {
return NotificationRequest{}, err
}
values.BundleID = bundleID
pipelineID, err := config.RenderDistributorPipelineID(cfg.Notify.Distributor.PipelineIDTemplate, values)
if err != nil {
return NotificationRequest{}, err
}
idempotencyKey, err := config.RenderDistributorIdempotencyKey(cfg.Notify.Distributor.IdempotencyKeyTemplate, values)
if err != nil {
return NotificationRequest{}, err
}
bundlePaths, err := config.RenderDistributorReportPaths(cfg.Notify.Distributor.ReportPathTemplates, values)
if err != nil {
return NotificationRequest{}, err
}
return NotificationRequest{
ReportID: resolved.Definition.ID,
RunID: metadata.RunID,
PipelineID: pipelineID,
BundleID: bundleID,
IdempotencyKey: idempotencyKey,
ReportPath: reportPath,
BundlePaths: bundlePaths,
CreatedAt: metadata.GeneratedAt,
}, nil
}
func addDistributorValidPeriodValues(values *config.DistributorTemplateValues, period timeutil.Period, timezone string) error {
location, err := timeutil.LoadLocation(timezone)
if err != nil {
return err
}
start := period.Start.In(location)
end := period.End.In(location)
values.ValidStartDate = start.Format(timeutil.DateLayout)
values.ValidEndDate = end.Format(timeutil.DateLayout)
values.ValidStartTime = start.Format("1504")
values.ValidEndTime = end.Format("1504")
values.ValidStartStamp = start.Format("2006-01-02T1504")
values.ValidEndStamp = end.Format("2006-01-02T1504")
return nil
}
func saveNotificationArtifact(ctx context.Context, store state.Store, resolved report.Resolved, cfg config.Config, metadata state.Metadata, req NotificationRequest, result *NotificationResult, notifyErr error) (string, error) {
if store == nil {
return "", fmt.Errorf("state store is required")
}
artifact := state.DistributorNotificationArtifact{
SchemaVersion: state.DistributorNotificationSchemaVersion,
RunID: metadata.RunID,
ReportID: resolved.Definition.ID,
AttemptedAt: time.Now(),
Endpoint: cfg.Notify.Distributor.Endpoint,
PipelineID: req.PipelineID,
BundleID: req.BundleID,
IdempotencyKey: req.IdempotencyKey,
SourcePath: req.ReportPath,
BundlePaths: append([]string(nil), req.BundlePaths...),
BundleCreated: req.CreatedAt,
Status: "attempted",
}
if result != nil {
artifact.Status = result.Status
artifact.Upload = &state.DistributorUploadResult{
RunID: result.RunID,
Status: result.UploadStatus,
}
if result.PipelineID != "" || !result.AcceptedAt.IsZero() || result.StartedAt != nil || result.FinishedAt != nil || len(result.Report) > 0 || result.Error != "" {
artifact.RunStatus = &state.DistributorRunStatus{
RunID: result.RunID,
PipelineID: result.PipelineID,
Status: result.Status,
AcceptedAt: result.AcceptedAt,
StartedAt: result.StartedAt,
FinishedAt: result.FinishedAt,
Report: append([]byte(nil), result.Report...),
Error: result.Error,
}
}
artifact.StatusError = result.StatusError
}
if notifyErr != nil {
artifact.Status = "failed"
artifact.Error = notifyErr.Error()
}
if artifact.Status == "" {
artifact.Status = "unknown"
}
return store.SaveDistributorNotification(ctx, resolved, artifact)
}
type noopNotifier struct{}
func (noopNotifier) Notify(context.Context, NotificationRequest) (*NotificationResult, error) {
return nil, nil
}
type distributorNotifier struct {
client *distributoradapter.Client
}
func (n distributorNotifier) Notify(ctx context.Context, req NotificationRequest) (*NotificationResult, error) {
result, err := n.client.Upload(ctx, distributoradapter.UploadRequest{
PipelineID: req.PipelineID,
BundleID: req.BundleID,
IdempotencyKey: req.IdempotencyKey,
Files: distributorUploadFiles(req.ReportPath, req.BundlePaths),
CreatedAt: req.CreatedAt,
})
notification := &NotificationResult{
PipelineID: req.PipelineID,
BundleID: req.BundleID,
IdempotencyKey: req.IdempotencyKey,
RunID: result.RunID,
Status: result.Status,
UploadStatus: result.UploadStatus,
StatusError: result.StatusError,
}
if result.RunStatus != nil {
if result.RunStatus.PipelineID != "" {
notification.PipelineID = result.RunStatus.PipelineID
}
notification.AcceptedAt = result.RunStatus.AcceptedAt
notification.StartedAt = result.RunStatus.StartedAt
notification.FinishedAt = result.RunStatus.FinishedAt
notification.Report = append([]byte(nil), result.RunStatus.Report...)
notification.Error = result.RunStatus.Error
}
if err != nil {
return notification, err
}
return notification, nil
}
func distributorUploadFiles(sourcePath string, bundlePaths []string) []distributoradapter.UploadFile {
files := make([]distributoradapter.UploadFile, 0, len(bundlePaths))
for _, bundlePath := range bundlePaths {
files = append(files, distributoradapter.UploadFile{
SourcePath: sourcePath,
BundlePath: bundlePath,
})
}
return files
}
func BuildBriefing(req BriefingRequest, bundle *forecast.Bundle) (briefing.Package, error) { func BuildBriefing(req BriefingRequest, bundle *forecast.Bundle) (briefing.Package, error) {
location, err := timeutil.LoadLocation(req.Config.WeatherAPI.Timezone) location, err := timeutil.LoadLocation(req.Config.WeatherAPI.Timezone)
if err != nil { if err != nil {

View File

@@ -35,6 +35,8 @@ func TestFetchAndSaveBundle(t *testing.T) {
_, _ = w.Write([]byte(`{"data":{"alerts":[]}}`)) _, _ = w.Write([]byte(`{"data":{"alerts":[]}}`))
case "/discussion": case "/discussion":
_, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":[],"shortTerm":{"qualifier":"(Short Term)","text":"Short-term AFD narrative for saved bundle."},"longTerm":{"qualifier":"(Long Term)","text":"Long-term AFD narrative for saved bundle."}}}`)) _, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":[],"shortTerm":{"qualifier":"(Short Term)","text":"Short-term AFD narrative for saved bundle."},"longTerm":{"qualifier":"(Long Term)","text":"Long-term AFD narrative for saved bundle."}}}`))
case "/weatherstories/latest":
_, _ = w.Write([]byte(`{"data":{"officeId":"LSX","startTime":"2026-05-30T08:46:00Z","endTime":"2026-05-31T11:00:00Z","updatedAt":"2026-05-30T09:00:34Z","title":"Several Chances for Rain Through Monday","description":"Scattered showers and thunderstorms remain possible.","altText":"Forecast weather story graphic.","priority":false,"order":1,"downloadUrl":"https://api.weather.gov/offices/LSX/weatherstories/download/test"}}`))
default: default:
http.NotFound(w, r) http.NotFound(w, r)
} }
@@ -59,6 +61,9 @@ func TestFetchAndSaveBundle(t *testing.T) {
if !strings.Contains(string(data), `"product": "hourly"`) { if !strings.Contains(string(data), `"product": "hourly"`) {
t.Fatalf("saved bundle missing hourly product:\n%s", string(data)) t.Fatalf("saved bundle missing hourly product:\n%s", string(data))
} }
if !strings.Contains(string(data), `"title": "Several Chances for Rain Through Monday"`) {
t.Fatalf("saved bundle missing weather story title:\n%s", string(data))
}
} }
func TestFetchAndSaveBundleRequiresOutputPath(t *testing.T) { func TestFetchAndSaveBundleRequiresOutputPath(t *testing.T) {
@@ -216,6 +221,9 @@ func TestGenerateReportWritesReportAndPreflight(t *testing.T) {
if current == nil || current.ConditionText != "Clear" || current.TemperatureF == nil || *current.TemperatureF != 75 { if current == nil || current.ConditionText != "Clear" || current.TemperatureF == nil || *current.TemperatureF != 75 {
t.Fatalf("data package current conditions = %#v, want current conditions", current) t.Fatalf("data package current conditions = %#v, want current conditions", current)
} }
if savedDataPackage.Briefing.Daily == nil || savedDataPackage.Briefing.Daily.WeatherStory == nil || savedDataPackage.Briefing.Daily.WeatherStory.Title != "Several Chances for Rain Through Monday" {
t.Fatalf("data package weather story = %#v, want weather story title", savedDataPackage.Briefing.Daily)
}
if !strings.Contains(string(data), "Short-term AFD narrative for generated report.") || !strings.Contains(string(data), "Long-term AFD narrative for generated report.") { if !strings.Contains(string(data), "Short-term AFD narrative for generated report.") || !strings.Contains(string(data), "Long-term AFD narrative for generated report.") {
t.Fatalf("data package missing AFD short/long-term discussion:\n%s", string(data)) t.Fatalf("data package missing AFD short/long-term discussion:\n%s", string(data))
} }
@@ -247,6 +255,262 @@ func TestGenerateReportWritesReportAndPreflight(t *testing.T) {
} }
} }
func TestGenerateReportDisabledNotificationDoesNotCallNotifier(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyTestConfig(t, server)
cfg.Workspace.Root = t.TempDir()
resolved, err := ResolveGenerate(GenerateRequest{
Config: cfg,
Report: ReportDaily,
Date: mustParse("2026-05-29T12:00:00-05:00"),
}, mustParse("2026-05-29T05:00:00-05:00"))
if err != nil {
t.Fatalf("ResolveGenerate() error = %v", err)
}
notifier := &recordingNotifier{}
_, err = GenerateReport(context.Background(), ReportRequest{
Config: cfg,
Resolved: resolved,
Renderer: successfulRenderer("# Daily Report\n"),
Notifier: notifier,
})
if err != nil {
t.Fatalf("GenerateReport() error = %v", err)
}
if len(notifier.requests) != 0 {
t.Fatalf("notification requests = %#v, want none when disabled", notifier.requests)
}
}
func TestGenerateReportNotifiesManagedReportPath(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyTestConfig(t, server)
cfg.Workspace.Root = t.TempDir()
cfg.Notify.Distributor.Enabled = true
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{artifact_group}"
resolved, err := ResolveGenerate(GenerateRequest{
Config: cfg,
Report: ReportDaily,
Date: mustParse("2026-05-29T12:00:00-05:00"),
}, mustParse("2026-05-29T05:00:00-05:00"))
if err != nil {
t.Fatalf("ResolveGenerate() error = %v", err)
}
notifier := &recordingNotifier{
result: &NotificationResult{
RunID: "distributor-run",
Status: "succeeded",
UploadStatus: "accepted",
PipelineID: "reports",
Report: []byte(`{"actions":[{"action":"replace_older"}]}`),
},
}
outputPath := filepath.Join(t.TempDir(), "daily-copy.md")
result, err := GenerateReport(context.Background(), ReportRequest{
Config: cfg,
Resolved: resolved,
OutputPath: outputPath,
Renderer: successfulRenderer("# Daily Report\n"),
Notifier: notifier,
})
if err != nil {
t.Fatalf("GenerateReport() error = %v", err)
}
if result.Notification == nil {
t.Fatal("Notification = nil, want notification result")
}
if result.Notification.RunID != "distributor-run" || result.Notification.Status != "succeeded" {
t.Fatalf("Notification = %#v, want succeeded distributor run", result.Notification)
}
if result.NotificationPath == "" || result.Metadata.NotificationPath != result.NotificationPath {
t.Fatalf("NotificationPath result=%q metadata=%q, want linked artifact", result.NotificationPath, result.Metadata.NotificationPath)
}
notificationData, err := os.ReadFile(result.NotificationPath)
if err != nil {
t.Fatalf("read notification artifact: %v", err)
}
var notificationArtifact state.DistributorNotificationArtifact
if err := json.Unmarshal(notificationData, &notificationArtifact); err != nil {
t.Fatalf("decode notification artifact: %v", err)
}
wantBundlePaths := []string{
"2026-05-29/daily/2026-05-29-daily-" + result.Metadata.RunID + ".md",
}
if notificationArtifact.PipelineID != "weatherreporter.daily" || strings.Join(notificationArtifact.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") || notificationArtifact.BundleCreated.IsZero() || notificationArtifact.RunStatus == nil || !strings.Contains(string(notificationArtifact.RunStatus.Report), "replace_older") {
t.Fatalf("notification artifact = %#v, want requested pipeline, status report, and created timestamp", notificationArtifact)
}
if len(notifier.requests) != 1 {
t.Fatalf("notification requests = %d, want 1", len(notifier.requests))
}
req := notifier.requests[0]
if req.ReportPath != result.ReportPath {
t.Fatalf("notification ReportPath = %q, want managed path %q", req.ReportPath, result.ReportPath)
}
if req.ReportPath == outputPath {
t.Fatalf("notification used output copy %q, want managed report path", outputPath)
}
if strings.Join(req.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") {
t.Fatalf("notification BundlePaths = %#v, want %#v", req.BundlePaths, wantBundlePaths)
}
if req.PipelineID != "weatherreporter.daily" {
t.Fatalf("notification PipelineID = %q, want rendered pipeline", req.PipelineID)
}
if req.BundleID != "weatherreporter.home.daily_today" {
t.Fatalf("notification BundleID = %q, want default template", req.BundleID)
}
if req.IdempotencyKey != req.BundleID+"."+result.Metadata.RunID {
t.Fatalf("IdempotencyKey = %q, want per-run key", req.IdempotencyKey)
}
if req.RunID != result.Metadata.RunID {
t.Fatalf("notification RunID = %q, want report run id %q", req.RunID, result.Metadata.RunID)
}
if !req.CreatedAt.Equal(result.Metadata.GeneratedAt) {
t.Fatalf("notification CreatedAt = %s, want generated at %s", req.CreatedAt, result.Metadata.GeneratedAt)
}
}
func TestGenerateReportNotificationFailureFailsReport(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyTestConfig(t, server)
cfg.Workspace.Root = t.TempDir()
cfg.Notify.Distributor.Enabled = true
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{artifact_group}"
resolved, err := ResolveGenerate(GenerateRequest{
Config: cfg,
Report: ReportDaily,
Date: mustParse("2026-05-29T12:00:00-05:00"),
}, mustParse("2026-05-29T05:00:00-05:00"))
if err != nil {
t.Fatalf("ResolveGenerate() error = %v", err)
}
notifier := &recordingNotifier{err: errors.New("upload rejected")}
store, err := state.NewFilesystemStore(cfg.Workspace)
if err != nil {
t.Fatalf("NewFilesystemStore() error = %v", err)
}
_, err = GenerateReport(context.Background(), ReportRequest{
Config: cfg,
Resolved: resolved,
Renderer: successfulRenderer("# Daily Report\n"),
Store: store,
Notifier: notifier,
})
if err == nil {
t.Fatal("GenerateReport() error = nil, want notification error")
}
if !strings.Contains(err.Error(), "notify report") || !strings.Contains(err.Error(), "upload rejected") {
t.Fatalf("error = %q, want notification context", err.Error())
}
if len(notifier.requests) != 1 {
t.Fatalf("notification requests = %d, want one attempted notification", len(notifier.requests))
}
paths, pathErr := store.Paths(resolved)
if pathErr != nil {
t.Fatalf("Paths() error = %v", pathErr)
}
notificationData, readErr := os.ReadFile(paths.Notification)
if readErr != nil {
t.Fatalf("read notification artifact after failure: %v", readErr)
}
var notification state.DistributorNotificationArtifact
if err := json.Unmarshal(notificationData, &notification); err != nil {
t.Fatalf("decode notification artifact: %v", err)
}
if notification.Status != "failed" || !strings.Contains(notification.Error, "upload rejected") {
t.Fatalf("notification failure artifact = %+v, want failed upload context", notification)
}
}
func TestGenerateReportDoesNotNotifyAfterRenderOrRunFailure(t *testing.T) {
server := dailyBundleServer(t)
tests := []struct {
name string
renderer Renderer
}{
{
name: "Render",
renderer: &recordingRenderer{
renderResult: &scriptorium.RenderResult{ExitCode: 1, Stderr: "render failed"},
err: errors.New("render failed"),
},
},
{
name: "Run",
renderer: &recordingRenderer{
renderResult: &scriptorium.RenderResult{ExitCode: 0},
runResult: &scriptorium.RunResult{ExitCode: 2, Stderr: "run failed"},
runErr: errors.New("run failed"),
runBody: "# Daily Report\n",
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := dailyTestConfig(t, server)
cfg.Workspace.Root = t.TempDir()
cfg.Notify.Distributor.Enabled = true
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{artifact_group}"
resolved, err := ResolveGenerate(GenerateRequest{
Config: cfg,
Report: ReportDaily,
Date: mustParse("2026-05-29T12:00:00-05:00"),
}, mustParse("2026-05-29T05:00:00-05:00"))
if err != nil {
t.Fatalf("ResolveGenerate() error = %v", err)
}
notifier := &recordingNotifier{}
_, err = GenerateReport(context.Background(), ReportRequest{
Config: cfg,
Resolved: resolved,
Renderer: tt.renderer,
Notifier: notifier,
})
if err == nil {
t.Fatal("GenerateReport() error = nil, want generation error")
}
if len(notifier.requests) != 0 {
t.Fatalf("notification requests = %#v, want none after generation failure", notifier.requests)
}
})
}
}
func TestGenerateReportDoesNotNotifyAfterFetchFailure(t *testing.T) {
cfg := config.Defaults()
cfg.WeatherAPI.BaseURL = ""
cfg.WeatherAPI.Timezone = "America/Chicago"
cfg.Workspace.Root = t.TempDir()
cfg.Notify.Distributor.Enabled = true
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{artifact_group}"
resolved, err := ResolveGenerate(GenerateRequest{
Config: cfg,
Report: ReportDaily,
Date: mustParse("2026-05-29T12:00:00-05:00"),
}, mustParse("2026-05-29T05:00:00-05:00"))
if err != nil {
t.Fatalf("ResolveGenerate() error = %v", err)
}
notifier := &recordingNotifier{}
_, err = GenerateReport(context.Background(), ReportRequest{
Config: cfg,
Resolved: resolved,
Renderer: successfulRenderer("# Daily Report\n"),
Notifier: notifier,
})
if err == nil {
t.Fatal("GenerateReport() error = nil, want fetch setup error")
}
if len(notifier.requests) != 0 {
t.Fatalf("notification requests = %#v, want none after fetch failure", notifier.requests)
}
}
func TestGenerateReportPersistsFailedPreflight(t *testing.T) { func TestGenerateReportPersistsFailedPreflight(t *testing.T) {
server := dailyBundleServer(t) server := dailyBundleServer(t)
cfg := config.Defaults() cfg := config.Defaults()
@@ -906,6 +1170,8 @@ func dailyBundleServer(t *testing.T) *httptest.Server {
_, _ = w.Write([]byte(`{"data":{"alerts":[{"event":"Flood Watch","effective":"2026-05-29T05:00:00-05:00","expires":"2026-05-29T09:00:00-05:00"}]}}`)) _, _ = w.Write([]byte(`{"data":{"alerts":[{"event":"Flood Watch","effective":"2026-05-29T05:00:00-05:00","expires":"2026-05-29T09:00:00-05:00"}]}}`))
case "/discussion": case "/discussion":
_, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":["Storms are most likely during the morning."],"shortTerm":{"qualifier":"(Short Term)","text":"Short-term AFD narrative for generated report."},"longTerm":{"qualifier":"(Long Term)","text":"Long-term AFD narrative for generated report."}}}`)) _, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":["Storms are most likely during the morning."],"shortTerm":{"qualifier":"(Short Term)","text":"Short-term AFD narrative for generated report."},"longTerm":{"qualifier":"(Long Term)","text":"Long-term AFD narrative for generated report."}}}`))
case "/weatherstories/latest":
_, _ = w.Write([]byte(`{"data":{"officeId":"LSX","startTime":"2026-05-30T08:46:00Z","endTime":"2026-05-31T11:00:00Z","updatedAt":"2026-05-30T09:00:34Z","title":"Several Chances for Rain Through Monday","description":"Scattered showers and thunderstorms remain possible.","altText":"Forecast weather story graphic.","priority":false,"order":1,"downloadUrl":"https://api.weather.gov/offices/LSX/weatherstories/download/test"}}`))
default: default:
http.NotFound(w, r) http.NotFound(w, r)
} }
@@ -998,6 +1264,69 @@ func TestRunBatchContinuesAfterReportFailure(t *testing.T) {
} }
} }
func TestRunBatchContinuesAfterNotificationFailure(t *testing.T) {
server := dailyBundleServer(t)
cfg := config.Defaults()
cfg.WeatherAPI.BaseURL = server.URL + "/"
cfg.WeatherAPI.Timezone = "America/Chicago"
cfg.Workspace.Root = t.TempDir()
cfg.Notify.Distributor.Enabled = true
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{artifact_group}"
notifier := &recordingNotifier{
errByReport: map[report.ID]error{
report.ThreeDay: errors.New("distributor unavailable"),
},
}
result, err := RunBatchDetailed(context.Background(), BatchRequest{
Config: cfg,
Batch: BatchMorning,
Now: mustParse("2026-05-29T05:00:00-05:00"),
Renderer: &selectiveRenderer{runBody: "# Batch Report\n"},
Notifier: notifier,
})
if err != nil {
t.Fatalf("RunBatchDetailed() error = %v", err)
}
if result.Total != 3 || result.Succeeded != 2 || result.Failed != 1 {
t.Fatalf("summary total/succeeded/failed = %d/%d/%d, want 3/2/1", result.Total, result.Succeeded, result.Failed)
}
if len(notifier.requests) != 3 {
t.Fatalf("notification requests = %d, want one per generated report", len(notifier.requests))
}
var failedThreeDay bool
for _, item := range result.Reports {
if item.ReportID == report.ThreeDay {
if item.Status == "failed" && strings.Contains(item.Error, "notify report") && strings.Contains(item.Error, "distributor unavailable") {
failedThreeDay = true
}
if item.NotificationStatus != "failed" {
t.Fatalf("3-day notification status = %q, want failed", item.NotificationStatus)
}
if item.NotificationPipelineID != "weatherreporter.three-day" {
t.Fatalf("3-day notification pipeline = %q, want weatherreporter.three-day", item.NotificationPipelineID)
}
if !strings.Contains(item.NotificationError, "distributor unavailable") {
t.Fatalf("3-day notification error = %q, want distributor unavailable", item.NotificationError)
}
continue
}
if item.Status != "succeeded" {
t.Fatalf("report %s status = %s, want succeeded", item.ReportID, item.Status)
}
if item.NotificationStatus != "accepted" {
t.Fatalf("report %s notification status = %q, want accepted", item.ReportID, item.NotificationStatus)
}
if item.NotificationPipelineID == "" {
t.Fatalf("report %s notification pipeline is empty", item.ReportID)
}
}
if !failedThreeDay {
t.Fatalf("reports = %#v, want notification failure on 3-day item", result.Reports)
}
}
func TestRunBatchUsesOutputDirectory(t *testing.T) { func TestRunBatchUsesOutputDirectory(t *testing.T) {
server := dailyBundleServer(t) server := dailyBundleServer(t)
cfg := config.Defaults() cfg := config.Defaults()
@@ -1154,6 +1483,14 @@ type recordingRenderer struct {
runBody string runBody string
} }
func successfulRenderer(body string) *recordingRenderer {
return &recordingRenderer{
renderResult: &scriptorium.RenderResult{ExitCode: 0},
runResult: &scriptorium.RunResult{ExitCode: 0},
runBody: body,
}
}
type selectiveRenderer struct { type selectiveRenderer struct {
renderCalls int renderCalls int
runCalls int runCalls int
@@ -1161,6 +1498,43 @@ type selectiveRenderer struct {
runBody string runBody string
} }
type recordingNotifier struct {
requests []NotificationRequest
result *NotificationResult
err error
errByReport map[report.ID]error
}
func (n *recordingNotifier) Notify(_ context.Context, req NotificationRequest) (*NotificationResult, error) {
n.requests = append(n.requests, req)
if err := n.errByReport[req.ReportID]; err != nil {
return nil, err
}
if n.err != nil {
return nil, n.err
}
if n.result != nil {
result := *n.result
if result.BundleID == "" {
result.BundleID = req.BundleID
}
if result.IdempotencyKey == "" {
result.IdempotencyKey = req.IdempotencyKey
}
if result.PipelineID == "" {
result.PipelineID = req.PipelineID
}
return &result, nil
}
return &NotificationResult{
PipelineID: req.PipelineID,
BundleID: req.BundleID,
IdempotencyKey: req.IdempotencyKey,
Status: "accepted",
UploadStatus: "accepted",
}, nil
}
func (r *selectiveRenderer) Render(_ context.Context, req scriptorium.RenderRequest) (*scriptorium.RenderResult, error) { func (r *selectiveRenderer) Render(_ context.Context, req scriptorium.RenderRequest) (*scriptorium.RenderResult, error) {
r.renderCalls++ r.renderCalls++
if req.PromptID == r.failRenderPrompt { if req.PromptID == r.failRenderPrompt {

View File

@@ -5,6 +5,7 @@ import (
"math" "math"
"sort" "sort"
"strings" "strings"
"time"
"gitea.maximumdirect.net/eric/weatherreporter/internal/forecast" "gitea.maximumdirect.net/eric/weatherreporter/internal/forecast"
"gitea.maximumdirect.net/eric/weatherreporter/internal/report" "gitea.maximumdirect.net/eric/weatherreporter/internal/report"
@@ -57,8 +58,17 @@ type DiscussionContext struct {
} }
type WeatherStoryContext struct { type WeatherStoryContext struct {
Available bool `json:"available"` Available bool `json:"available"`
Summary string `json:"summary,omitempty"` OfficeID string `json:"officeId,omitempty"`
StartTime time.Time `json:"startTime"`
EndTime time.Time `json:"endTime"`
UpdatedAt *time.Time `json:"updatedAt,omitempty"`
Title string `json:"title,omitempty"`
Description string `json:"description,omitempty"`
AltText string `json:"altText,omitempty"`
Priority bool `json:"priority"`
Order int `json:"order"`
DownloadURL string `json:"downloadUrl,omitempty"`
} }
func BuildDaily(ctx BuildContext, summary *forecast.DailySummary) (Package, error) { func BuildDaily(ctx BuildContext, summary *forecast.DailySummary) (Package, error) {
@@ -258,10 +268,31 @@ func buildDiscussion(discussion *forecast.Discussion) DiscussionContext {
} }
func buildWeatherStory(bundle *forecast.Bundle) *WeatherStoryContext { func buildWeatherStory(bundle *forecast.Bundle) *WeatherStoryContext {
if bundle == nil || bundle.WeatherStory == nil || len(bundle.WeatherStory.Raw) == 0 { if bundle == nil || bundle.WeatherStory == nil {
return nil return nil
} }
return &WeatherStoryContext{Available: true, Summary: string(bundle.WeatherStory.Raw)} story := bundle.WeatherStory
return &WeatherStoryContext{
Available: true,
OfficeID: story.OfficeID,
StartTime: story.StartTime,
EndTime: story.EndTime,
UpdatedAt: copyTime(story.UpdatedAt),
Title: story.Title,
Description: story.Description,
AltText: story.AltText,
Priority: story.Priority,
Order: story.Order,
DownloadURL: story.DownloadURL,
}
}
func copyTime(value *time.Time) *time.Time {
if value == nil {
return nil
}
copied := *value
return &copied
} }
func scoreOutdoorWindow(daypart forecast.DaypartSummary) OutdoorWindow { func scoreOutdoorWindow(daypart forecast.DaypartSummary) OutdoorWindow {

View File

@@ -162,8 +162,12 @@ func stormConfidenceInputs(bundle *forecast.Bundle) []string {
items = appendUnique(items, "Short-term discussion is available for confidence context.") items = appendUnique(items, "Short-term discussion is available for confidence context.")
} }
} }
if bundle.WeatherStory != nil && len(bundle.WeatherStory.Raw) > 0 { if bundle.WeatherStory != nil {
items = appendUnique(items, "Weather story source is available.") if bundle.WeatherStory.Title != "" {
items = appendUnique(items, "Weather story: "+bundle.WeatherStory.Title+".")
} else {
items = appendUnique(items, "Weather story source is available.")
}
} }
for _, warning := range bundle.Warnings { for _, warning := range bundle.Warnings {
if warning.Code != "" { if warning.Code != "" {

View File

@@ -50,8 +50,16 @@ func TestStormBriefingWithActiveAlert(t *testing.T) {
ShortTerm: &forecast.DiscussionSection{Text: "Short-term storm coverage peaks this morning."}, ShortTerm: &forecast.DiscussionSection{Text: "Short-term storm coverage peaks this morning."},
LongTerm: &forecast.DiscussionSection{Text: "Long-term pattern stays unsettled after the event."}, LongTerm: &forecast.DiscussionSection{Text: "Long-term pattern stays unsettled after the event."},
}, },
WeatherStory: &forecast.WeatherStory{Raw: json.RawMessage(`{"headline":"Storm risk"}`)}, WeatherStory: &forecast.WeatherStory{
Sources: []forecast.Source{{Name: "hourly", FetchedAt: time.Now()}}, OfficeID: "LSX",
StartTime: mustParse("2026-05-29T06:00:00Z"),
EndTime: mustParse("2026-05-29T18:00:00Z"),
Title: "Storm Risk",
Description: "Strong storms are possible.",
AltText: "Weather story graphic showing storm risk.",
Order: 1,
},
Sources: []forecast.Source{{Name: "hourly", FetchedAt: time.Now()}},
} }
pkg, err := BuildStorm(BuildContext{Resolved: resolved, Bundle: bundle, Units: "us", Timezone: "America/Chicago"}) pkg, err := BuildStorm(BuildContext{Resolved: resolved, Bundle: bundle, Units: "us", Timezone: "America/Chicago"})
@@ -80,6 +88,9 @@ func TestStormBriefingWithActiveAlert(t *testing.T) {
if pkg.Storm.WeatherStory == nil { if pkg.Storm.WeatherStory == nil {
t.Fatal("WeatherStory = nil, want available story context") t.Fatal("WeatherStory = nil, want available story context")
} }
if pkg.Storm.WeatherStory.Title != "Storm Risk" || pkg.Storm.WeatherStory.Description != "Strong storms are possible." {
t.Fatalf("WeatherStory = %#v, want structured story context", pkg.Storm.WeatherStory)
}
if pkg.Storm.Discussion.ShortTerm != "Short-term storm coverage peaks this morning." { if pkg.Storm.Discussion.ShortTerm != "Short-term storm coverage peaks this morning." {
t.Fatalf("Discussion.ShortTerm = %q, want short-term AFD narrative", pkg.Storm.Discussion.ShortTerm) t.Fatalf("Discussion.ShortTerm = %q, want short-term AFD narrative", pkg.Storm.Discussion.ShortTerm)
} }

View File

@@ -352,11 +352,21 @@ func writeRunLogs(stderr io.Writer, result *app.BatchResult) {
return return
} }
for _, item := range result.Reports { for _, item := range result.Reports {
notificationFields := ""
if item.NotificationStatus != "" {
notificationFields += fmt.Sprintf(" notificationStatus=%q", item.NotificationStatus)
}
if item.NotificationRunID != "" {
notificationFields += fmt.Sprintf(" notificationRunId=%q", item.NotificationRunID)
}
if item.NotificationError != "" {
notificationFields += fmt.Sprintf(" notificationError=%q", item.NotificationError)
}
if item.Status == "failed" { if item.Status == "failed" {
_, _ = fmt.Fprintf(stderr, "report=%s status=failed error=%q\n", item.ReportID, item.Error) _, _ = fmt.Fprintf(stderr, "report=%s status=failed error=%q%s\n", item.ReportID, item.Error, notificationFields)
continue continue
} }
_, _ = fmt.Fprintf(stderr, "report=%s status=succeeded output=%q\n", item.ReportID, item.OutputPath) _, _ = fmt.Fprintf(stderr, "report=%s status=succeeded output=%q%s\n", item.ReportID, item.OutputPath, notificationFields)
} }
_, _ = fmt.Fprintf(stderr, "batch=%s total=%d succeeded=%d failed=%d\n", result.Batch, result.Total, result.Succeeded, result.Failed) _, _ = fmt.Fprintf(stderr, "batch=%s total=%d succeeded=%d failed=%d\n", result.Batch, result.Total, result.Succeeded, result.Failed)
} }

View File

@@ -346,6 +346,88 @@ func TestRunMorningReportsPartialFailureAndContinues(t *testing.T) {
} }
} }
func TestBatchOutputIncludesNotificationDetails(t *testing.T) {
result := &app.BatchResult{
Batch: app.BatchMorning,
Total: 2,
Succeeded: 1,
Failed: 1,
Reports: []app.BatchReportResult{
{
ReportID: "daily_today",
Status: "succeeded",
OutputPath: "/tmp/daily.md",
NotificationStatus: "accepted",
NotificationRunID: "distributor-run-1",
},
{
ReportID: "three_day",
Status: "failed",
Error: "notify report three_day: upload failed",
NotificationStatus: "failed",
NotificationError: "notify report three_day: upload failed",
},
},
}
var stdout bytes.Buffer
var stderr bytes.Buffer
if err := writeJSON(&stdout, result); err != nil {
t.Fatalf("writeJSON() error = %v", err)
}
writeRunLogs(&stderr, result)
var decoded app.BatchResult
if err := json.Unmarshal(stdout.Bytes(), &decoded); err != nil {
t.Fatalf("decode batch JSON: %v\n%s", err, stdout.String())
}
if decoded.Reports[0].NotificationStatus != "accepted" || decoded.Reports[0].NotificationRunID != "distributor-run-1" {
t.Fatalf("success notification fields = %#v", decoded.Reports[0])
}
if decoded.Reports[1].NotificationStatus != "failed" || !strings.Contains(decoded.Reports[1].NotificationError, "upload failed") {
t.Fatalf("failure notification fields = %#v", decoded.Reports[1])
}
if !strings.Contains(stderr.String(), `notificationStatus="accepted"`) || !strings.Contains(stderr.String(), `notificationRunId="distributor-run-1"`) {
t.Fatalf("stderr missing success notification fields:\n%s", stderr.String())
}
if !strings.Contains(stderr.String(), `notificationStatus="failed"`) || !strings.Contains(stderr.String(), `notificationError="notify report three_day: upload failed"`) {
t.Fatalf("stderr missing failure notification fields:\n%s", stderr.String())
}
}
func TestBatchOutputDoesNotExposeSecretLikeNotificationErrors(t *testing.T) {
result := &app.BatchResult{
Batch: app.BatchMorning,
Total: 1,
Failed: 1,
Reports: []app.BatchReportResult{
{
ReportID: "daily_today",
Status: "failed",
Error: "notify report daily_today: upload failed: [redacted]",
NotificationStatus: "failed",
NotificationError: "notify report daily_today: upload failed: [redacted]",
},
},
}
var stdout bytes.Buffer
var stderr bytes.Buffer
if err := writeJSON(&stdout, result); err != nil {
t.Fatalf("writeJSON() error = %v", err)
}
writeRunLogs(&stderr, result)
for _, output := range []string{stdout.String(), stderr.String()} {
if strings.Contains(output, "DISTRIBUTOR_SECRET_TOKEN") {
t.Fatalf("output contains token value:\n%s", output)
}
if !strings.Contains(output, "[redacted]") {
t.Fatalf("output missing redacted marker:\n%s", output)
}
}
}
func TestRunEveningUsesOutputDirectoryAndSummary(t *testing.T) { func TestRunEveningUsesOutputDirectoryAndSummary(t *testing.T) {
server := dailyServer(t) server := dailyServer(t)
tempDir := t.TempDir() tempDir := t.TempDir()
@@ -384,6 +466,108 @@ func TestRunEveningUsesOutputDirectoryAndSummary(t *testing.T) {
} }
} }
func TestRunEveningReportsNotificationSuccess(t *testing.T) {
server := dailyServer(t)
distributorServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/runs/distributor-run-1" {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"run_id":"distributor-run-1","pipeline_id":"weatherreporter.daily","status":"succeeded","report":{"actions":[{"action":"replace_older"}]}}`))
return
}
if r.URL.Path != "/v1/pipelines/weatherreporter.daily/upload" {
http.NotFound(w, r)
return
}
w.WriteHeader(http.StatusAccepted)
_, _ = w.Write([]byte(`{"run_id":"distributor-run-1","status":"accepted"}`))
}))
t.Cleanup(distributorServer.Close)
tempDir := t.TempDir()
scriptoriumPath := writeFakeScriptorium(t, tempDir)
configPath := filepath.Join(tempDir, "config.yml")
workspaceRoot := filepath.Join(tempDir, "workspace")
configBody := "weather_api:\n base_url: " + server.URL + "/\n timezone: America/Chicago\nscriptorium:\n binary: " + scriptoriumPath + "\nworkspace:\n root: " + workspaceRoot + "\nnotify:\n distributor:\n enabled: true\n endpoint: " + distributorServer.URL + "\n token_env: CLI_DISTRIBUTOR_TOKEN\n pipeline_id_template: weatherreporter.{artifact_group}\n"
if err := os.WriteFile(configPath, []byte(configBody), 0o600); err != nil {
t.Fatalf("write config: %v", err)
}
t.Setenv("CLI_DISTRIBUTOR_TOKEN", "cli-secret-token")
var stdout bytes.Buffer
var stderr bytes.Buffer
runner := Runner{Clock: fixedClock()}
err := runner.Run(context.Background(), []string{
"run", "evening",
"--config", configPath,
}, &stdout, &stderr)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
var summary app.BatchResult
if decodeErr := json.Unmarshal(stdout.Bytes(), &summary); decodeErr != nil {
t.Fatalf("decode summary: %v\n%s", decodeErr, stdout.String())
}
if len(summary.Reports) != 1 {
t.Fatalf("reports = %#v, want one report", summary.Reports)
}
if summary.Reports[0].NotificationStatus != "succeeded" || summary.Reports[0].NotificationRunID != "distributor-run-1" || summary.Reports[0].NotificationPipelineID != "weatherreporter.daily" {
t.Fatalf("notification fields = %#v", summary.Reports[0])
}
if !strings.Contains(stderr.String(), `notificationStatus="succeeded"`) || !strings.Contains(stderr.String(), `notificationRunId="distributor-run-1"`) {
t.Fatalf("stderr missing notification fields:\n%s", stderr.String())
}
if strings.Contains(stdout.String(), "cli-secret-token") || strings.Contains(stderr.String(), "cli-secret-token") {
t.Fatalf("output contains token value\nstdout=%s\nstderr=%s", stdout.String(), stderr.String())
}
}
func TestRunEveningReportsNotificationFailureWithoutToken(t *testing.T) {
server := dailyServer(t)
distributorServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusBadRequest)
_, _ = w.Write([]byte(`{"error":"rejected cli-secret-token","retryable":false}`))
}))
t.Cleanup(distributorServer.Close)
tempDir := t.TempDir()
scriptoriumPath := writeFakeScriptorium(t, tempDir)
configPath := filepath.Join(tempDir, "config.yml")
workspaceRoot := filepath.Join(tempDir, "workspace")
configBody := "weather_api:\n base_url: " + server.URL + "/\n timezone: America/Chicago\nscriptorium:\n binary: " + scriptoriumPath + "\nworkspace:\n root: " + workspaceRoot + "\nnotify:\n distributor:\n enabled: true\n endpoint: " + distributorServer.URL + "\n token_env: CLI_DISTRIBUTOR_TOKEN\n pipeline_id_template: weatherreporter.{artifact_group}\n"
if err := os.WriteFile(configPath, []byte(configBody), 0o600); err != nil {
t.Fatalf("write config: %v", err)
}
t.Setenv("CLI_DISTRIBUTOR_TOKEN", "cli-secret-token")
var stdout bytes.Buffer
var stderr bytes.Buffer
runner := Runner{Clock: fixedClock()}
err := runner.Run(context.Background(), []string{
"run", "evening",
"--config", configPath,
}, &stdout, &stderr)
if err == nil {
t.Fatal("Run() error = nil, want notification failure")
}
var summary app.BatchResult
if decodeErr := json.Unmarshal(stdout.Bytes(), &summary); decodeErr != nil {
t.Fatalf("decode summary: %v\n%s", decodeErr, stdout.String())
}
if len(summary.Reports) != 1 || summary.Reports[0].NotificationStatus != "failed" {
t.Fatalf("summary reports = %#v, want failed notification", summary.Reports)
}
for _, output := range []string{stdout.String(), stderr.String(), err.Error()} {
if strings.Contains(output, "cli-secret-token") {
t.Fatalf("output contains token value:\n%s", output)
}
}
for _, output := range []string{stdout.String(), stderr.String()} {
if !strings.Contains(output, "[redacted]") {
t.Fatalf("output missing redaction marker:\n%s", output)
}
}
}
func TestRunMorningGeneratesDailyAndThreeDayOnSunday(t *testing.T) { func TestRunMorningGeneratesDailyAndThreeDayOnSunday(t *testing.T) {
server := dailyServer(t) server := dailyServer(t)
tempDir := t.TempDir() tempDir := t.TempDir()
@@ -813,6 +997,8 @@ func dailyServer(t *testing.T) *httptest.Server {
_, _ = w.Write([]byte(`{"data":{"alerts":[]}}`)) _, _ = w.Write([]byte(`{"data":{"alerts":[]}}`))
case "/discussion": case "/discussion":
_, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":["Storms are most likely during the morning."]}}`)) _, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":["Storms are most likely during the morning."]}}`))
case "/weatherstories/latest":
_, _ = w.Write([]byte(`{"data":{"officeId":"LSX","startTime":"2026-05-30T08:46:00Z","endTime":"2026-05-31T11:00:00Z","updatedAt":"2026-05-30T09:00:34Z","title":"Several Chances for Rain Through Monday","description":"Scattered showers and thunderstorms remain possible.","altText":"Forecast weather story graphic.","priority":false,"order":1,"downloadUrl":"https://api.weather.gov/offices/LSX/weatherstories/download/test"}}`))
default: default:
http.NotFound(w, r) http.NotFound(w, r)
} }

View File

@@ -5,16 +5,21 @@ package config
import "time" import "time"
type MissingSourcePolicy string type MissingSourcePolicy string
type NotifyFailurePolicy string
const ( const (
MissingSourceError MissingSourcePolicy = "error" MissingSourceError MissingSourcePolicy = "error"
MissingSourceWarn MissingSourcePolicy = "warn" MissingSourceWarn MissingSourcePolicy = "warn"
MissingSourceNone MissingSourcePolicy = "none" MissingSourceNone MissingSourcePolicy = "none"
NotifyFailureError NotifyFailurePolicy = "error"
) )
type Config struct { type Config struct {
WeatherAPI WeatherAPIConfig `yaml:"weather_api"` WeatherAPI WeatherAPIConfig `yaml:"weather_api"`
Location LocationConfig `yaml:"location"` Location LocationConfig `yaml:"location"`
Secrets SecretsConfig `yaml:"secrets"`
Notify NotifyConfig `yaml:"notify"`
MissingSource MissingSourceConfig `yaml:"missing_source"` MissingSource MissingSourceConfig `yaml:"missing_source"`
Scriptorium ScriptoriumConfig `yaml:"scriptorium"` Scriptorium ScriptoriumConfig `yaml:"scriptorium"`
Workspace WorkspaceConfig `yaml:"workspace"` Workspace WorkspaceConfig `yaml:"workspace"`
@@ -37,6 +42,26 @@ type LocationConfig struct {
Region string `yaml:"region"` Region string `yaml:"region"`
} }
type SecretsConfig struct {
Directory string `yaml:"directory"`
}
type NotifyConfig struct {
Distributor DistributorNotifyConfig `yaml:"distributor"`
}
type DistributorNotifyConfig struct {
Enabled bool `yaml:"enabled"`
Endpoint string `yaml:"endpoint"`
TokenEnv string `yaml:"token_env"`
Timeout time.Duration `yaml:"timeout"`
FailurePolicy NotifyFailurePolicy `yaml:"failure_policy"`
PipelineIDTemplate string `yaml:"pipeline_id_template"`
BundleIDTemplate string `yaml:"bundle_id_template"`
IdempotencyKeyTemplate string `yaml:"idempotency_key_template"`
ReportPathTemplates []string `yaml:"report_path_templates"`
}
type MissingSourceConfig struct { type MissingSourceConfig struct {
Default MissingSourcePolicy `yaml:"default"` Default MissingSourcePolicy `yaml:"default"`
Sources map[string]MissingSourcePolicy `yaml:"sources"` Sources map[string]MissingSourcePolicy `yaml:"sources"`
@@ -51,11 +76,12 @@ type ScriptoriumConfig struct {
} }
type WorkspaceConfig struct { type WorkspaceConfig struct {
Root string `yaml:"root"` Root string `yaml:"root"`
SnapshotsDir string `yaml:"snapshots_dir"` SnapshotsDir string `yaml:"snapshots_dir"`
ReportsDir string `yaml:"reports_dir"` ReportsDir string `yaml:"reports_dir"`
DataPackagesDir string `yaml:"data_packages_dir"` DataPackagesDir string `yaml:"data_packages_dir"`
PreflightDir string `yaml:"preflight_dir"` PreflightDir string `yaml:"preflight_dir"`
NotificationsDir string `yaml:"notifications_dir"`
} }
type DaypartConfig struct { type DaypartConfig struct {

View File

@@ -26,6 +26,39 @@ func TestDefaults(t *testing.T) {
if cfg.Location.ID != "home" || cfg.Location.Name != "Brentwood" || cfg.Location.Region != "St. Louis Metro" { if cfg.Location.ID != "home" || cfg.Location.Name != "Brentwood" || cfg.Location.Region != "St. Louis Metro" {
t.Fatalf("Location = %#v, want home/Brentwood/St. Louis Metro", cfg.Location) t.Fatalf("Location = %#v, want home/Brentwood/St. Louis Metro", cfg.Location)
} }
if cfg.Secrets.Directory != "" {
t.Fatalf("Secrets.Directory = %q, want empty", cfg.Secrets.Directory)
}
if cfg.Notify.Distributor.Enabled {
t.Fatalf("Notify.Distributor.Enabled = true, want false")
}
if cfg.Notify.Distributor.Endpoint != "https://distributor.example.com" {
t.Fatalf("Notify.Distributor.Endpoint = %q, want default endpoint", cfg.Notify.Distributor.Endpoint)
}
if cfg.Notify.Distributor.TokenEnv != "DISTRIBUTOR_UPLOAD_TOKEN" {
t.Fatalf("Notify.Distributor.TokenEnv = %q, want DISTRIBUTOR_UPLOAD_TOKEN", cfg.Notify.Distributor.TokenEnv)
}
if cfg.Notify.Distributor.Timeout != 30*time.Second {
t.Fatalf("Notify.Distributor.Timeout = %s, want 30s", cfg.Notify.Distributor.Timeout)
}
if cfg.Notify.Distributor.FailurePolicy != NotifyFailureError {
t.Fatalf("Notify.Distributor.FailurePolicy = %q, want error", cfg.Notify.Distributor.FailurePolicy)
}
if cfg.Notify.Distributor.PipelineIDTemplate != "" {
t.Fatalf("Notify.Distributor.PipelineIDTemplate = %q, want empty", cfg.Notify.Distributor.PipelineIDTemplate)
}
if cfg.Notify.Distributor.BundleIDTemplate != "weatherreporter.{location_id}.{report_id}" {
t.Fatalf("Notify.Distributor.BundleIDTemplate = %q, want default", cfg.Notify.Distributor.BundleIDTemplate)
}
if cfg.Notify.Distributor.IdempotencyKeyTemplate != "{bundle_id}.{run_id}" {
t.Fatalf("Notify.Distributor.IdempotencyKeyTemplate = %q, want default", cfg.Notify.Distributor.IdempotencyKeyTemplate)
}
wantReportPaths := []string{
"{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md",
}
if strings.Join(cfg.Notify.Distributor.ReportPathTemplates, "\n") != strings.Join(wantReportPaths, "\n") {
t.Fatalf("Notify.Distributor.ReportPathTemplates = %#v, want %#v", cfg.Notify.Distributor.ReportPathTemplates, wantReportPaths)
}
if cfg.MissingSource.Default != MissingSourceWarn { if cfg.MissingSource.Default != MissingSourceWarn {
t.Fatalf("MissingSource.Default = %q, want warn", cfg.MissingSource.Default) t.Fatalf("MissingSource.Default = %q, want warn", cfg.MissingSource.Default)
} }
@@ -49,6 +82,12 @@ func TestLoadExampleConfig(t *testing.T) {
if cfg.Location.ID != "home" || cfg.Location.Name != "Brentwood" || cfg.Location.Region != "St. Louis Metro" { if cfg.Location.ID != "home" || cfg.Location.Name != "Brentwood" || cfg.Location.Region != "St. Louis Metro" {
t.Fatalf("Location = %#v, want example location", cfg.Location) t.Fatalf("Location = %#v, want example location", cfg.Location)
} }
if cfg.Notify.Distributor.PipelineIDTemplate != "weatherreporter.{artifact_group}" {
t.Fatalf("PipelineIDTemplate = %q, want example pipeline template", cfg.Notify.Distributor.PipelineIDTemplate)
}
if len(cfg.Notify.Distributor.ReportPathTemplates) != 1 {
t.Fatalf("ReportPathTemplates = %#v, want example archive path", cfg.Notify.Distributor.ReportPathTemplates)
}
} }
func TestLoadMinimalExampleConfig(t *testing.T) { func TestLoadMinimalExampleConfig(t *testing.T) {
@@ -69,6 +108,9 @@ func TestLoadMinimalExampleConfig(t *testing.T) {
if cfg.Workspace.Root != "workspace" { if cfg.Workspace.Root != "workspace" {
t.Fatalf("Workspace.Root = %q, want default workspace", cfg.Workspace.Root) t.Fatalf("Workspace.Root = %q, want default workspace", cfg.Workspace.Root)
} }
if cfg.Workspace.NotificationsDir != "notifications" {
t.Fatalf("Workspace.NotificationsDir = %q, want notifications", cfg.Workspace.NotificationsDir)
}
if cfg.Location.Name != "Brentwood" { if cfg.Location.Name != "Brentwood" {
t.Fatalf("Location.Name = %q, want default Brentwood", cfg.Location.Name) t.Fatalf("Location.Name = %q, want default Brentwood", cfg.Location.Name)
} }
@@ -112,3 +154,475 @@ func TestLoadAppliesOverrides(t *testing.T) {
t.Fatalf("Timezone = %q, want +09:30", cfg.WeatherAPI.Timezone) t.Fatalf("Timezone = %q, want +09:30", cfg.WeatherAPI.Timezone)
} }
} }
func TestDisabledDistributorNotifyAcceptsOmittedFields(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "config.yml")
if err := os.WriteFile(path, []byte("notify:\n distributor:\n enabled: false\n"), 0o600); err != nil {
t.Fatalf("write config fixture: %v", err)
}
cfg, err := LoadFile(path)
if err != nil {
t.Fatalf("LoadFile() error = %v", err)
}
if cfg.Notify.Distributor.Enabled {
t.Fatalf("Notify.Distributor.Enabled = true, want false")
}
}
func TestEnabledDistributorNotifyValidation(t *testing.T) {
tests := []struct {
name string
mutate func(*Config)
wantErr string
}{
{
name: "Endpoint",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.Endpoint = "distributor.example.com"
},
wantErr: "notify.distributor.endpoint",
},
{
name: "TokenEnvEmpty",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.TokenEnv = ""
},
wantErr: "notify.distributor.token_env",
},
{
name: "TokenEnvInvalid",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.TokenEnv = "1TOKEN"
},
wantErr: "notify.distributor.token_env",
},
{
name: "Timeout",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.Timeout = 0
},
wantErr: "notify.distributor.timeout",
},
{
name: "FailurePolicy",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.FailurePolicy = "warn"
},
wantErr: "notify.distributor.failure_policy",
},
{
name: "PipelineTemplateEmpty",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.PipelineIDTemplate = ""
},
wantErr: "notify.distributor.pipeline_id_template",
},
{
name: "PipelineTemplateUnknown",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.PipelineIDTemplate = "{unknown}"
},
wantErr: "notify.distributor.pipeline_id_template",
},
{
name: "PipelineTemplateRenderedEmpty",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.PipelineIDTemplate = " "
},
wantErr: "notify.distributor.pipeline_id_template",
},
{
name: "BundleTemplate",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.BundleIDTemplate = "{unknown}"
},
wantErr: "notify.distributor.bundle_id_template",
},
{
name: "IdempotencyTemplate",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.IdempotencyKeyTemplate = "{unknown}"
},
wantErr: "notify.distributor.idempotency_key_template",
},
{
name: "ReportPathTemplatesEmpty",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = nil
},
wantErr: "notify.distributor.report_path_templates",
},
{
name: "ReportPathTemplateUnknown",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = []string{"{unknown}"}
},
wantErr: "notify.distributor.report_path_templates",
},
{
name: "ReportPathTemplateInvalidPath",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = []string{"/{batch_output_name}"}
},
wantErr: "notify.distributor.report_path_templates",
},
{
name: "ReportPathTemplateDuplicatePath",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = []string{"latest.md", "latest.md"}
},
wantErr: "notify.distributor.report_path_templates",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Defaults()
cfg.Notify.Distributor.Enabled = true
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{artifact_group}"
tt.mutate(&cfg)
err := Validate(cfg)
if err == nil {
t.Fatal("Validate() error = nil, want error")
}
if !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
}
})
}
}
func TestDistributorTemplateRendering(t *testing.T) {
values := DistributorTemplateValues{
LocationID: "home",
ReportID: "daily",
RunID: "20260607T120000Z",
ArtifactGroup: "daily",
BatchOutputName: "daily.md",
ValidStartDate: "2026-06-07",
ValidEndDate: "2026-06-08",
ValidStartTime: "1800",
ValidEndTime: "0600",
ValidStartStamp: "2026-06-07T1800",
ValidEndStamp: "2026-06-08T0600",
BundleID: "weatherreporter.home.daily",
}
bundleID, err := RenderDistributorBundleID("weatherreporter.{location_id}.{report_id}", values)
if err != nil {
t.Fatalf("RenderDistributorBundleID() error = %v", err)
}
if bundleID != "weatherreporter.home.daily" {
t.Fatalf("bundleID = %q, want rendered value", bundleID)
}
pipelineID, err := RenderDistributorPipelineID("weatherreporter.{artifact_group}.{bundle_id}", values)
if err != nil {
t.Fatalf("RenderDistributorPipelineID() error = %v", err)
}
if pipelineID != "weatherreporter.daily.weatherreporter.home.daily" {
t.Fatalf("pipelineID = %q, want rendered pipeline ID", pipelineID)
}
idempotencyKey, err := RenderDistributorIdempotencyKey("{bundle_id}.{run_id}", values)
if err != nil {
t.Fatalf("RenderDistributorIdempotencyKey() error = %v", err)
}
if idempotencyKey != "weatherreporter.home.daily.20260607T120000Z" {
t.Fatalf("idempotencyKey = %q, want rendered run key", idempotencyKey)
}
reportPaths, err := RenderDistributorReportPaths([]string{
"{valid_start_date}/{artifact_group}/{valid_start_stamp}-{valid_end_stamp}-{run_id}.md",
"{valid_start_date}/{artifact_group}/latest.md",
}, values)
if err != nil {
t.Fatalf("RenderDistributorReportPaths() error = %v", err)
}
wantPaths := []string{
"2026-06-07/daily/2026-06-07T1800-2026-06-08T0600-20260607T120000Z.md",
"2026-06-07/daily/latest.md",
}
if strings.Join(reportPaths, "\n") != strings.Join(wantPaths, "\n") {
t.Fatalf("reportPaths = %#v, want %#v", reportPaths, wantPaths)
}
}
func TestDistributorTemplateRejectsUnknownAndMalformedVariables(t *testing.T) {
tests := []struct {
name string
template string
}{
{name: "Unknown", template: "{unknown}"},
{name: "Unclosed", template: "{location_id"},
{name: "Unopened", template: "location_id}"},
{name: "Empty", template: "{}"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := RenderDistributorBundleID(tt.template, DistributorTemplateValues{})
if err == nil {
t.Fatal("RenderDistributorBundleID() error = nil, want error")
}
})
}
}
func TestDistributorReportPathValidation(t *testing.T) {
tests := []struct {
name string
path string
ok bool
}{
{name: "Simple", path: "daily.md", ok: true},
{name: "Nested", path: "reports/daily.md", ok: true},
{name: "Empty", path: "", ok: false},
{name: "Absolute", path: "/reports/daily.md", ok: false},
{name: "WindowsAbsolute", path: "C:/reports/daily.md", ok: false},
{name: "Backslash", path: `reports\daily.md`, ok: false},
{name: "CurrentSegment", path: "reports/./daily.md", ok: false},
{name: "ParentSegment", path: "reports/../daily.md", ok: false},
{name: "EmptySegment", path: "reports//daily.md", ok: false},
{name: "Manifest", path: "reports/manifest.json", ok: false},
{name: "DistributorMetadata", path: "reports/.distributor.json", ok: false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateDistributorReportPath("test.path", tt.path)
if tt.ok && err != nil {
t.Fatalf("ValidateDistributorReportPath() error = %v", err)
}
if !tt.ok && err == nil {
t.Fatal("ValidateDistributorReportPath() error = nil, want error")
}
})
}
}
func TestDistributorReportPathRenderingRejectsInvalidValues(t *testing.T) {
tests := []struct {
name string
batchOutputName string
}{
{name: "Absolute", batchOutputName: "/daily.md"},
{name: "Backslash", batchOutputName: `reports\daily.md`},
{name: "CurrentSegment", batchOutputName: "./daily.md"},
{name: "ParentSegment", batchOutputName: "../daily.md"},
{name: "EmptySegment", batchOutputName: "reports//daily.md"},
{name: "Manifest", batchOutputName: "manifest.json"},
{name: "DistributorMetadata", batchOutputName: ".distributor.json"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := RenderDistributorReportPaths([]string{"{batch_output_name}"}, DistributorTemplateValues{
BatchOutputName: tt.batchOutputName,
})
if err == nil {
t.Fatal("RenderDistributorReportPaths() error = nil, want error")
}
})
}
}
func TestLoadFileLoadsSecretsBeforeReturningNotifyConfig(t *testing.T) {
dir := t.TempDir()
secretsDir := filepath.Join(dir, "secrets")
if err := os.Mkdir(secretsDir, 0o700); err != nil {
t.Fatalf("create secrets directory: %v", err)
}
if err := os.WriteFile(filepath.Join(secretsDir, "DISTRIBUTOR_UPLOAD_TOKEN"), []byte("loaded-token"), 0o600); err != nil {
t.Fatalf("write secret: %v", err)
}
path := filepath.Join(dir, "config.yml")
configYAML := "secrets:\n" +
" directory: " + secretsDir + "\n" +
"notify:\n" +
" distributor:\n" +
" enabled: true\n" +
" pipeline_id_template: weatherreporter.{artifact_group}\n"
if err := os.WriteFile(path, []byte(configYAML), 0o600); err != nil {
t.Fatalf("write config fixture: %v", err)
}
t.Setenv("DISTRIBUTOR_UPLOAD_TOKEN", "")
cfg, err := LoadFile(path)
if err != nil {
t.Fatalf("LoadFile() error = %v", err)
}
if cfg.Notify.Distributor.TokenEnv != "DISTRIBUTOR_UPLOAD_TOKEN" {
t.Fatalf("TokenEnv = %q, want DISTRIBUTOR_UPLOAD_TOKEN", cfg.Notify.Distributor.TokenEnv)
}
if got := os.Getenv(cfg.Notify.Distributor.TokenEnv); got != "loaded-token" {
t.Fatalf("environment value = %q, want loaded-token", got)
}
}
func TestLoadSecretsDisabledLeavesEnvironmentUnchanged(t *testing.T) {
t.Setenv("WEATHERREPORTER_DISABLED_SECRET", "original")
if err := loadSecrets(SecretsConfig{}); err != nil {
t.Fatalf("loadSecrets() error = %v", err)
}
if got := os.Getenv("WEATHERREPORTER_DISABLED_SECRET"); got != "original" {
t.Fatalf("environment value = %q, want original", got)
}
}
func TestLoadFileLoadsSecretsDirectory(t *testing.T) {
dir := t.TempDir()
secretsDir := filepath.Join(dir, "secrets")
if err := os.Mkdir(secretsDir, 0o700); err != nil {
t.Fatalf("create secrets directory: %v", err)
}
if err := os.WriteFile(filepath.Join(secretsDir, "WEATHERREPORTER_SECRET"), []byte("from-file"), 0o600); err != nil {
t.Fatalf("write secret: %v", err)
}
path := filepath.Join(dir, "config.yml")
if err := os.WriteFile(path, []byte("secrets:\n directory: "+secretsDir+"\n"), 0o600); err != nil {
t.Fatalf("write config fixture: %v", err)
}
t.Setenv("WEATHERREPORTER_SECRET", "")
if _, err := LoadFile(path); err != nil {
t.Fatalf("LoadFile() error = %v", err)
}
if got := os.Getenv("WEATHERREPORTER_SECRET"); got != "from-file" {
t.Fatalf("environment value = %q, want from-file", got)
}
}
func TestLoadSecretsOverwritesExistingEnvironment(t *testing.T) {
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "WEATHERREPORTER_SECRET"), []byte("from-file"), 0o600); err != nil {
t.Fatalf("write secret: %v", err)
}
t.Setenv("WEATHERREPORTER_SECRET", "existing")
if err := loadSecrets(SecretsConfig{Directory: dir}); err != nil {
t.Fatalf("loadSecrets() error = %v", err)
}
if got := os.Getenv("WEATHERREPORTER_SECRET"); got != "from-file" {
t.Fatalf("environment value = %q, want from-file", got)
}
}
func TestLoadSecretsTrimsOneTrailingLineEnding(t *testing.T) {
tests := []struct {
name string
input string
want string
}{
{name: "LF", input: "value\n", want: "value"},
{name: "CRLF", input: "value\r\n", want: "value"},
{name: "TwoLF", input: "value\n\n", want: "value\n"},
{name: "LoneCR", input: "value\r", want: "value\r"},
{name: "NoNewline", input: "value", want: "value"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "WEATHERREPORTER_SECRET"), []byte(tt.input), 0o600); err != nil {
t.Fatalf("write secret: %v", err)
}
t.Setenv("WEATHERREPORTER_SECRET", "")
if err := loadSecrets(SecretsConfig{Directory: dir}); err != nil {
t.Fatalf("loadSecrets() error = %v", err)
}
if got := os.Getenv("WEATHERREPORTER_SECRET"); got != tt.want {
t.Fatalf("environment value = %q, want %q", got, tt.want)
}
})
}
}
func TestLoadSecretsRejectsInvalidDirectoryEntries(t *testing.T) {
tests := []struct {
name string
setup func(t *testing.T, dir string)
wantErr string
}{
{
name: "InvalidFilename",
setup: func(t *testing.T, dir string) {
if err := os.WriteFile(filepath.Join(dir, "1INVALID"), []byte("secret-value"), 0o600); err != nil {
t.Fatalf("write invalid secret: %v", err)
}
},
wantErr: "invalid environment variable name",
},
{
name: "Subdirectory",
setup: func(t *testing.T, dir string) {
if err := os.Mkdir(filepath.Join(dir, "SUBDIR"), 0o700); err != nil {
t.Fatalf("create subdirectory: %v", err)
}
},
wantErr: "not a directory",
},
{
name: "Symlink",
setup: func(t *testing.T, dir string) {
target := filepath.Join(dir, "TARGET")
if err := os.WriteFile(target, []byte("secret-value"), 0o600); err != nil {
t.Fatalf("write target: %v", err)
}
if err := os.Symlink(target, filepath.Join(dir, "SYMLINK")); err != nil {
t.Fatalf("create symlink: %v", err)
}
},
wantErr: "not a symlink",
},
{
name: "Unreadable",
setup: func(t *testing.T, dir string) {
path := filepath.Join(dir, "UNREADABLE")
if err := os.WriteFile(path, []byte("secret-value"), 0o600); err != nil {
t.Fatalf("write unreadable secret: %v", err)
}
if err := os.Chmod(path, 0o000); err != nil {
t.Fatalf("chmod unreadable secret: %v", err)
}
t.Cleanup(func() {
_ = os.Chmod(path, 0o600)
})
},
wantErr: "read secret file",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
dir := t.TempDir()
tt.setup(t, dir)
err := loadSecrets(SecretsConfig{Directory: dir})
if err == nil {
t.Fatal("loadSecrets() error = nil, want error")
}
if !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
}
if strings.Contains(err.Error(), "secret-value") {
t.Fatalf("error = %q, want no secret value", err.Error())
}
})
}
}
func TestLoadSecretsRejectsMissingDirectory(t *testing.T) {
err := loadSecrets(SecretsConfig{Directory: filepath.Join(t.TempDir(), "missing")})
if err == nil {
t.Fatal("loadSecrets() error = nil, want missing directory error")
}
if !strings.Contains(err.Error(), "read secrets directory") {
t.Fatalf("error = %q, want read secrets directory context", err.Error())
}
}

View File

@@ -18,6 +18,24 @@ func Defaults() Config {
Name: "Brentwood", Name: "Brentwood",
Region: "St. Louis Metro", Region: "St. Louis Metro",
}, },
Secrets: SecretsConfig{
Directory: "",
},
Notify: NotifyConfig{
Distributor: DistributorNotifyConfig{
Enabled: false,
Endpoint: "https://distributor.example.com",
TokenEnv: "DISTRIBUTOR_UPLOAD_TOKEN",
Timeout: 30 * time.Second,
FailurePolicy: NotifyFailureError,
PipelineIDTemplate: "",
BundleIDTemplate: "weatherreporter.{location_id}.{report_id}",
IdempotencyKeyTemplate: "{bundle_id}.{run_id}",
ReportPathTemplates: []string{
"{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md",
},
},
},
MissingSource: MissingSourceConfig{ MissingSource: MissingSourceConfig{
Default: MissingSourceWarn, Default: MissingSourceWarn,
Sources: map[string]MissingSourcePolicy{}, Sources: map[string]MissingSourcePolicy{},
@@ -27,11 +45,12 @@ func Defaults() Config {
Timeout: 2 * time.Minute, Timeout: 2 * time.Minute,
}, },
Workspace: WorkspaceConfig{ Workspace: WorkspaceConfig{
Root: "workspace", Root: "workspace",
SnapshotsDir: "snapshots", SnapshotsDir: "snapshots",
ReportsDir: "reports", ReportsDir: "reports",
DataPackagesDir: "data-packages", DataPackagesDir: "data-packages",
PreflightDir: "preflight", PreflightDir: "preflight",
NotificationsDir: "notifications",
}, },
Dayparts: []DaypartConfig{ Dayparts: []DaypartConfig{
{Name: "overnight", Start: "00:00", End: "06:00"}, {Name: "overnight", Start: "00:00", End: "06:00"},

View File

@@ -35,6 +35,10 @@ func Load(opts LoadOptions) (Config, error) {
cfg.WeatherAPI.Timezone = opts.Timezone cfg.WeatherAPI.Timezone = opts.Timezone
} }
if err := loadSecrets(cfg.Secrets); err != nil {
return Config{}, err
}
if err := Validate(cfg); err != nil { if err := Validate(cfg); err != nil {
return Config{}, err return Config{}, err
} }

View File

@@ -0,0 +1,201 @@
package config
import (
"fmt"
"path/filepath"
"strings"
)
type DistributorTemplateValues struct {
LocationID string
ReportID string
RunID string
ArtifactGroup string
BatchOutputName string
ValidStartDate string
ValidEndDate string
ValidStartTime string
ValidEndTime string
ValidStartStamp string
ValidEndStamp string
BundleID string
}
var distributorTemplateVariables = map[string]struct{}{
"location_id": {},
"report_id": {},
"run_id": {},
"artifact_group": {},
"batch_output_name": {},
"valid_start_date": {},
"valid_end_date": {},
"valid_start_time": {},
"valid_end_time": {},
"valid_start_stamp": {},
"valid_end_stamp": {},
}
var distributorIdempotencyTemplateVariables = map[string]struct{}{
"location_id": {},
"report_id": {},
"run_id": {},
"artifact_group": {},
"batch_output_name": {},
"valid_start_date": {},
"valid_end_date": {},
"valid_start_time": {},
"valid_end_time": {},
"valid_start_stamp": {},
"valid_end_stamp": {},
"bundle_id": {},
}
var distributorPipelineTemplateVariables = distributorIdempotencyTemplateVariables
func RenderDistributorBundleID(template string, values DistributorTemplateValues) (string, error) {
return renderDistributorTemplate("notify.distributor.bundle_id_template", template, values, distributorTemplateVariables)
}
func RenderDistributorPipelineID(template string, values DistributorTemplateValues) (string, error) {
rendered, err := renderDistributorTemplate("notify.distributor.pipeline_id_template", template, values, distributorPipelineTemplateVariables)
if err != nil {
return "", err
}
if strings.TrimSpace(rendered) == "" {
return "", fmt.Errorf("notify.distributor.pipeline_id_template renders an empty pipeline id")
}
return rendered, nil
}
func RenderDistributorIdempotencyKey(template string, values DistributorTemplateValues) (string, error) {
return renderDistributorTemplate("notify.distributor.idempotency_key_template", template, values, distributorIdempotencyTemplateVariables)
}
func RenderDistributorReportPaths(templates []string, values DistributorTemplateValues) ([]string, error) {
if len(templates) == 0 {
return nil, fmt.Errorf("notify.distributor.report_path_templates must contain at least one entry")
}
paths := make([]string, 0, len(templates))
seen := make(map[string]struct{}, len(templates))
for i, template := range templates {
name := fmt.Sprintf("notify.distributor.report_path_templates[%d]", i)
rendered, err := renderDistributorTemplate(name, template, values, distributorTemplateVariables)
if err != nil {
return nil, err
}
if err := ValidateDistributorReportPath(name, rendered); err != nil {
return nil, err
}
if _, ok := seen[rendered]; ok {
return nil, fmt.Errorf("notify.distributor.report_path_templates renders duplicate path %q", rendered)
}
seen[rendered] = struct{}{}
paths = append(paths, rendered)
}
return paths, nil
}
func validateDistributorTemplate(name, template string, allowed map[string]struct{}) error {
_, err := renderDistributorTemplate(name, template, DistributorTemplateValues{}, allowed)
return err
}
func renderDistributorTemplate(name, template string, values DistributorTemplateValues, allowed map[string]struct{}) (string, error) {
var rendered strings.Builder
for i := 0; i < len(template); {
switch template[i] {
case '{':
end := strings.IndexByte(template[i+1:], '}')
if end < 0 {
return "", fmt.Errorf("%s contains an unclosed template variable", name)
}
variable := template[i+1 : i+1+end]
if variable == "" {
return "", fmt.Errorf("%s contains an empty template variable", name)
}
if _, ok := allowed[variable]; !ok {
return "", fmt.Errorf("%s contains unknown template variable %q", name, variable)
}
rendered.WriteString(distributorTemplateValue(variable, values))
i += end + 2
case '}':
return "", fmt.Errorf("%s contains an unopened template variable", name)
default:
rendered.WriteByte(template[i])
i++
}
}
return rendered.String(), nil
}
func distributorTemplateValue(variable string, values DistributorTemplateValues) string {
switch variable {
case "location_id":
return values.LocationID
case "report_id":
return values.ReportID
case "run_id":
return values.RunID
case "artifact_group":
return values.ArtifactGroup
case "batch_output_name":
return values.BatchOutputName
case "valid_start_date":
return values.ValidStartDate
case "valid_end_date":
return values.ValidEndDate
case "valid_start_time":
return values.ValidStartTime
case "valid_end_time":
return values.ValidEndTime
case "valid_start_stamp":
return values.ValidStartStamp
case "valid_end_stamp":
return values.ValidEndStamp
case "bundle_id":
return values.BundleID
default:
return ""
}
}
func ValidateDistributorReportPath(name, path string) error {
if path == "" {
return fmt.Errorf("%s renders an empty path", name)
}
if isDistributorAbsolutePath(path) {
return fmt.Errorf("%s must render a relative path", name)
}
if strings.Contains(path, "\\") {
return fmt.Errorf("%s must not render backslashes", name)
}
segments := strings.Split(path, "/")
for _, segment := range segments {
if segment == "" {
return fmt.Errorf("%s must not render empty path segments", name)
}
if segment == "." || segment == ".." {
return fmt.Errorf("%s must not render . or .. path segments", name)
}
if segment == "manifest.json" || segment == ".distributor.json" {
return fmt.Errorf("%s must not render reserved path segment %q", name, segment)
}
}
return nil
}
func isDistributorAbsolutePath(path string) bool {
if filepath.IsAbs(path) || strings.HasPrefix(path, "/") {
return true
}
if len(path) >= 3 && isASCIIAlpha(path[0]) && path[1] == ':' && (path[2] == '/' || path[2] == '\\') {
return true
}
return false
}
func isASCIIAlpha(ch byte) bool {
return (ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z')
}

View File

@@ -0,0 +1,63 @@
package config
import (
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
)
var secretNamePattern = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
func loadSecrets(cfg SecretsConfig) error {
if cfg.Directory == "" {
return nil
}
entries, err := os.ReadDir(cfg.Directory)
if err != nil {
return fmt.Errorf("read secrets directory %q: %w", cfg.Directory, err)
}
for _, entry := range entries {
name := entry.Name()
if name == "" {
return fmt.Errorf("secrets directory %q contains an empty filename", cfg.Directory)
}
if !secretNamePattern.MatchString(name) {
return fmt.Errorf("secret file %q has invalid environment variable name", name)
}
if entry.Type()&os.ModeSymlink != 0 {
return fmt.Errorf("secret file %q must be a regular file, not a symlink", name)
}
if entry.IsDir() {
return fmt.Errorf("secret file %q must be a regular file, not a directory", name)
}
info, err := entry.Info()
if err != nil {
return fmt.Errorf("inspect secret file %q: %w", name, err)
}
if !info.Mode().IsRegular() {
return fmt.Errorf("secret file %q must be a regular file", name)
}
path := filepath.Join(cfg.Directory, name)
data, err := os.ReadFile(path)
if err != nil {
return fmt.Errorf("read secret file %q: %w", name, err)
}
value := string(data)
if strings.HasSuffix(value, "\r\n") {
value = strings.TrimSuffix(value, "\r\n")
} else {
value = strings.TrimSuffix(value, "\n")
}
if err := os.Setenv(name, value); err != nil {
return fmt.Errorf("set environment variable from secret file %q: %w", name, err)
}
}
return nil
}

View File

@@ -49,6 +49,10 @@ func Validate(cfg Config) error {
} }
} }
if err := validateDistributorNotify(cfg.Notify.Distributor); err != nil {
return err
}
if cfg.Scriptorium.Binary == "" { if cfg.Scriptorium.Binary == "" {
return fmt.Errorf("scriptorium.binary is required") return fmt.Errorf("scriptorium.binary is required")
} }
@@ -75,6 +79,76 @@ func Validate(cfg Config) error {
return nil return nil
} }
func validateDistributorNotify(cfg DistributorNotifyConfig) error {
if !cfg.Enabled {
return nil
}
parsed, err := url.Parse(cfg.Endpoint)
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
return fmt.Errorf("notify.distributor.endpoint must be an absolute URL when enabled")
}
if cfg.TokenEnv == "" {
return fmt.Errorf("notify.distributor.token_env is required when enabled")
}
if !secretNamePattern.MatchString(cfg.TokenEnv) {
return fmt.Errorf("notify.distributor.token_env must be a valid environment variable name")
}
if cfg.Timeout <= 0 {
return fmt.Errorf("notify.distributor.timeout must be greater than zero when enabled")
}
if cfg.FailurePolicy != NotifyFailureError {
return fmt.Errorf("notify.distributor.failure_policy must be error when enabled")
}
if cfg.PipelineIDTemplate == "" {
return fmt.Errorf("notify.distributor.pipeline_id_template is required when enabled")
}
if err := validateDistributorTemplate("notify.distributor.pipeline_id_template", cfg.PipelineIDTemplate, distributorPipelineTemplateVariables); err != nil {
return err
}
if cfg.BundleIDTemplate == "" {
return fmt.Errorf("notify.distributor.bundle_id_template is required when enabled")
}
if err := validateDistributorTemplate("notify.distributor.bundle_id_template", cfg.BundleIDTemplate, distributorTemplateVariables); err != nil {
return err
}
if cfg.IdempotencyKeyTemplate == "" {
return fmt.Errorf("notify.distributor.idempotency_key_template is required when enabled")
}
if err := validateDistributorTemplate("notify.distributor.idempotency_key_template", cfg.IdempotencyKeyTemplate, distributorIdempotencyTemplateVariables); err != nil {
return err
}
if len(cfg.ReportPathTemplates) == 0 {
return fmt.Errorf("notify.distributor.report_path_templates must contain at least one entry when enabled")
}
values := DistributorTemplateValues{
LocationID: "location",
ReportID: "report",
RunID: "run",
ArtifactGroup: "artifact",
BatchOutputName: "report.md",
ValidStartDate: "2026-05-29",
ValidEndDate: "2026-05-30",
ValidStartTime: "0000",
ValidEndTime: "0000",
ValidStartStamp: "2026-05-29T0000",
ValidEndStamp: "2026-05-30T0000",
}
bundleID, err := RenderDistributorBundleID(cfg.BundleIDTemplate, values)
if err != nil {
return err
}
values.BundleID = bundleID
if _, err := RenderDistributorPipelineID(cfg.PipelineIDTemplate, values); err != nil {
return err
}
if _, err := RenderDistributorReportPaths(cfg.ReportPathTemplates, values); err != nil {
return err
}
return nil
}
func validatePolicy(name string, policy MissingSourcePolicy) error { func validatePolicy(name string, policy MissingSourcePolicy) error {
switch policy { switch policy {
case MissingSourceError, MissingSourceWarn, MissingSourceNone: case MissingSourceError, MissingSourceWarn, MissingSourceNone:

View File

@@ -155,7 +155,14 @@ type DiscussionSection struct {
} }
type WeatherStory struct { type WeatherStory struct {
IssuedAt *time.Time `json:"issuedAt,omitempty"` OfficeID string `json:"officeId,omitempty"`
UpdatedAt *time.Time `json:"updatedAt,omitempty"` StartTime time.Time `json:"startTime"`
Raw json.RawMessage `json:"raw,omitempty"` EndTime time.Time `json:"endTime"`
UpdatedAt *time.Time `json:"updatedAt,omitempty"`
Title string `json:"title,omitempty"`
Description string `json:"description,omitempty"`
AltText string `json:"altText,omitempty"`
Priority bool `json:"priority"`
Order int `json:"order"`
DownloadURL string `json:"downloadUrl,omitempty"`
} }

View File

@@ -18,11 +18,12 @@ import (
) )
type FilesystemStore struct { type FilesystemStore struct {
root string root string
snapshotsDir string snapshotsDir string
reportsDir string reportsDir string
dataPackagesDir string dataPackagesDir string
preflightDir string preflightDir string
notificationsDir string
} }
type ArtifactPaths struct { type ArtifactPaths struct {
@@ -30,6 +31,7 @@ type ArtifactPaths struct {
Metadata string `json:"metadata"` Metadata string `json:"metadata"`
DataPackage string `json:"dataPackage"` DataPackage string `json:"dataPackage"`
Preflight string `json:"preflight"` Preflight string `json:"preflight"`
Notification string `json:"notification,omitempty"`
RenderedReport string `json:"renderedReport,omitempty"` RenderedReport string `json:"renderedReport,omitempty"`
} }
@@ -57,17 +59,19 @@ func NewFilesystemStore(cfg config.WorkspaceConfig) (*FilesystemStore, error) {
"reports_dir": cfg.ReportsDir, "reports_dir": cfg.ReportsDir,
"data_packages_dir": cfg.DataPackagesDir, "data_packages_dir": cfg.DataPackagesDir,
"preflight_dir": cfg.PreflightDir, "preflight_dir": cfg.PreflightDir,
"notifications_dir": cfg.NotificationsDir,
} { } {
if err := validateRelativeDir(name, value); err != nil { if err := validateRelativeDir(name, value); err != nil {
return nil, err return nil, err
} }
} }
return &FilesystemStore{ return &FilesystemStore{
root: filepath.Clean(cfg.Root), root: filepath.Clean(cfg.Root),
snapshotsDir: filepath.Clean(cfg.SnapshotsDir), snapshotsDir: filepath.Clean(cfg.SnapshotsDir),
reportsDir: filepath.Clean(cfg.ReportsDir), reportsDir: filepath.Clean(cfg.ReportsDir),
dataPackagesDir: filepath.Clean(cfg.DataPackagesDir), dataPackagesDir: filepath.Clean(cfg.DataPackagesDir),
preflightDir: filepath.Clean(cfg.PreflightDir), preflightDir: filepath.Clean(cfg.PreflightDir),
notificationsDir: filepath.Clean(cfg.NotificationsDir),
}, nil }, nil
} }
@@ -90,6 +94,7 @@ func (s *FilesystemStore) Paths(resolved report.Resolved) (ArtifactPaths, error)
Metadata: s.join(s.snapshotsDir, group, validDate, filenameBase+".metadata.json"), Metadata: s.join(s.snapshotsDir, group, validDate, filenameBase+".metadata.json"),
DataPackage: s.join(s.dataPackagesDir, group, validDate, filenameBase+".data_package.json"), DataPackage: s.join(s.dataPackagesDir, group, validDate, filenameBase+".data_package.json"),
Preflight: s.join(s.preflightDir, group, validDate, filenameBase+".render.json"), Preflight: s.join(s.preflightDir, group, validDate, filenameBase+".render.json"),
Notification: s.join(s.notificationsDir, group, validDate, filenameBase+".distributor.json"),
RenderedReport: s.join(s.reportsDir, group, filenameBase+".md"), RenderedReport: s.join(s.reportsDir, group, filenameBase+".md"),
}, nil }, nil
} }
@@ -130,6 +135,20 @@ func (s *FilesystemStore) SavePreflight(_ context.Context, resolved report.Resol
return paths.Preflight, nil return paths.Preflight, nil
} }
func (s *FilesystemStore) SaveDistributorNotification(_ context.Context, resolved report.Resolved, artifact DistributorNotificationArtifact) (string, error) {
paths, err := s.Paths(resolved)
if err != nil {
return "", err
}
if artifact.SchemaVersion == "" {
artifact.SchemaVersion = DistributorNotificationSchemaVersion
}
if err := fileutil.WriteJSONAtomic(paths.Notification, artifact); err != nil {
return "", err
}
return paths.Notification, nil
}
func (s *FilesystemStore) PrepareRenderedReport(_ context.Context, resolved report.Resolved) (string, error) { func (s *FilesystemStore) PrepareRenderedReport(_ context.Context, resolved report.Resolved) (string, error) {
paths, err := s.Paths(resolved) paths, err := s.Paths(resolved)
if err != nil { if err != nil {

View File

@@ -30,6 +30,7 @@ func TestPathsUseRunIDAndWorkspace(t *testing.T) {
filepath.Join("snapshots", "daily", "2026-05-29", "20260529T100000.000000000Z_daily_today.metadata.json"), filepath.Join("snapshots", "daily", "2026-05-29", "20260529T100000.000000000Z_daily_today.metadata.json"),
filepath.Join("data-packages", "daily", "2026-05-29", "20260529T100000.000000000Z_daily_today.data_package.json"), filepath.Join("data-packages", "daily", "2026-05-29", "20260529T100000.000000000Z_daily_today.data_package.json"),
filepath.Join("preflight", "daily", "2026-05-29", "20260529T100000.000000000Z_daily_today.render.json"), filepath.Join("preflight", "daily", "2026-05-29", "20260529T100000.000000000Z_daily_today.render.json"),
filepath.Join("notifications", "daily", "2026-05-29", "20260529T100000.000000000Z_daily_today.distributor.json"),
filepath.Join("reports", "daily", "20260529T100000.000000000Z_daily_today.md"), filepath.Join("reports", "daily", "20260529T100000.000000000Z_daily_today.md"),
} { } {
if !strings.Contains(pathsString(paths), want) { if !strings.Contains(pathsString(paths), want) {
@@ -59,6 +60,23 @@ func TestSaveArtifactsAndMetadataRoundTrip(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("SavePreflight() error = %v", err) t.Fatalf("SavePreflight() error = %v", err)
} }
notificationPath, err := store.SaveDistributorNotification(context.Background(), resolved, DistributorNotificationArtifact{
RunID: resolved.Metadata().RunID,
ReportID: resolved.Definition.ID,
AttemptedAt: resolved.GeneratedAt,
Endpoint: "https://distributor.example.test",
PipelineID: "weatherreporter.daily",
BundleID: "weatherreporter.home.daily.run",
IdempotencyKey: "weatherreporter.home.daily.run",
SourcePath: "/tmp/report.md",
BundlePaths: []string{"2026-05-29/daily/report.md"},
BundleCreated: resolved.GeneratedAt,
Status: "succeeded",
RunStatus: &DistributorRunStatus{RunID: "distributor-run", Status: "succeeded"},
})
if err != nil {
t.Fatalf("SaveDistributorNotification() error = %v", err)
}
renderedReportPath, err := store.PrepareRenderedReport(context.Background(), resolved) renderedReportPath, err := store.PrepareRenderedReport(context.Background(), resolved)
if err != nil { if err != nil {
t.Fatalf("PrepareRenderedReport() error = %v", err) t.Fatalf("PrepareRenderedReport() error = %v", err)
@@ -77,6 +95,17 @@ func TestSaveArtifactsAndMetadataRoundTrip(t *testing.T) {
if preflight.Stdout != `{"ok":true}` { if preflight.Stdout != `{"ok":true}` {
t.Fatalf("preflight stdout = %q, want render stdout", preflight.Stdout) t.Fatalf("preflight stdout = %q, want render stdout", preflight.Stdout)
} }
var notification DistributorNotificationArtifact
notificationData, err := os.ReadFile(notificationPath)
if err != nil {
t.Fatalf("read notification: %v", err)
}
if err := json.Unmarshal(notificationData, &notification); err != nil {
t.Fatalf("decode notification: %v", err)
}
if notification.SchemaVersion != DistributorNotificationSchemaVersion || notification.PipelineID != "weatherreporter.daily" || len(notification.BundlePaths) != 1 || notification.RunStatus == nil || notification.RunStatus.Status != "succeeded" {
t.Fatalf("notification = %#v, want persisted distributor status", notification)
}
paths, err := store.Paths(resolved) paths, err := store.Paths(resolved)
if err != nil { if err != nil {
t.Fatalf("Paths() error = %v", err) t.Fatalf("Paths() error = %v", err)
@@ -93,7 +122,7 @@ func TestSaveArtifactsAndMetadataRoundTrip(t *testing.T) {
t.Fatalf("SaveMetadata() error = %v", err) t.Fatalf("SaveMetadata() error = %v", err)
} }
for _, path := range []string{briefingPath, dataPackagePath, preflightPath, renderedReportPath, metadataPath} { for _, path := range []string{briefingPath, dataPackagePath, preflightPath, notificationPath, renderedReportPath, metadataPath} {
if _, err := os.Stat(path); err != nil { if _, err := os.Stat(path); err != nil {
t.Fatalf("expected artifact %q: %v", path, err) t.Fatalf("expected artifact %q: %v", path, err)
} }
@@ -494,6 +523,7 @@ func pathsString(paths ArtifactPaths) string {
paths.Metadata, paths.Metadata,
paths.DataPackage, paths.DataPackage,
paths.Preflight, paths.Preflight,
paths.Notification,
paths.RenderedReport, paths.RenderedReport,
}, "\n") }, "\n")
} }

View File

@@ -29,6 +29,7 @@ type Metadata struct {
BriefingPath string `json:"briefingPath"` BriefingPath string `json:"briefingPath"`
DataPackagePath string `json:"dataPackagePath"` DataPackagePath string `json:"dataPackagePath"`
PreflightPath string `json:"preflightPath"` PreflightPath string `json:"preflightPath"`
NotificationPath string `json:"notificationPath,omitempty"`
RenderedReportPath string `json:"renderedReportPath,omitempty"` RenderedReportPath string `json:"renderedReportPath,omitempty"`
} }

View File

@@ -3,6 +3,8 @@ package state
import ( import (
"context" "context"
"encoding/json"
"time"
"gitea.maximumdirect.net/eric/weatherreporter/internal/briefing" "gitea.maximumdirect.net/eric/weatherreporter/internal/briefing"
"gitea.maximumdirect.net/eric/weatherreporter/internal/promptinput" "gitea.maximumdirect.net/eric/weatherreporter/internal/promptinput"
@@ -14,6 +16,7 @@ type Store interface {
SaveBriefing(context.Context, report.Resolved, briefing.Package) (string, error) SaveBriefing(context.Context, report.Resolved, briefing.Package) (string, error)
SaveDataPackage(context.Context, report.Resolved, promptinput.Package) (string, error) SaveDataPackage(context.Context, report.Resolved, promptinput.Package) (string, error)
SavePreflight(context.Context, report.Resolved, PreflightArtifact) (string, error) SavePreflight(context.Context, report.Resolved, PreflightArtifact) (string, error)
SaveDistributorNotification(context.Context, report.Resolved, DistributorNotificationArtifact) (string, error)
PrepareRenderedReport(context.Context, report.Resolved) (string, error) PrepareRenderedReport(context.Context, report.Resolved) (string, error)
SaveMetadata(context.Context, Metadata) (string, error) SaveMetadata(context.Context, Metadata) (string, error)
FindPriorSnapshot(context.Context, report.Resolved) (*PriorSnapshot, error) FindPriorSnapshot(context.Context, report.Resolved) (*PriorSnapshot, error)
@@ -33,3 +36,40 @@ type PreflightArtifact struct {
StderrTruncated bool `json:"stderrTruncated,omitempty"` StderrTruncated bool `json:"stderrTruncated,omitempty"`
ExitCode int `json:"exitCode"` ExitCode int `json:"exitCode"`
} }
const DistributorNotificationSchemaVersion = "weatherreporter.distributor_notification.v1"
type DistributorNotificationArtifact struct {
SchemaVersion string `json:"schemaVersion"`
RunID string `json:"runId"`
ReportID report.ID `json:"reportId"`
AttemptedAt time.Time `json:"attemptedAt"`
Endpoint string `json:"endpoint"`
PipelineID string `json:"pipelineId,omitempty"`
BundleID string `json:"bundleId,omitempty"`
IdempotencyKey string `json:"idempotencyKey,omitempty"`
SourcePath string `json:"sourcePath,omitempty"`
BundlePaths []string `json:"bundlePaths,omitempty"`
BundleCreated time.Time `json:"bundleCreated,omitempty"`
Status string `json:"status"`
Upload *DistributorUploadResult `json:"upload,omitempty"`
RunStatus *DistributorRunStatus `json:"runStatus,omitempty"`
StatusError string `json:"statusError,omitempty"`
Error string `json:"error,omitempty"`
}
type DistributorUploadResult struct {
RunID string `json:"runId,omitempty"`
Status string `json:"status,omitempty"`
}
type DistributorRunStatus struct {
RunID string `json:"runId,omitempty"`
PipelineID string `json:"pipelineId,omitempty"`
Status string `json:"status,omitempty"`
AcceptedAt time.Time `json:"acceptedAt,omitempty"`
StartedAt *time.Time `json:"startedAt,omitempty"`
FinishedAt *time.Time `json:"finishedAt,omitempty"`
Report json.RawMessage `json:"report,omitempty"`
Error string `json:"error,omitempty"`
}