From fba519cab0198d6cc1482f1a5ed8683a06b5f0c5 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 11 Jun 2026 00:30:49 +0000 Subject: [PATCH] Update documentation for SPC outlook support --- README.md | 1 + docs/config.md | 12 ++++ docs/integrations/spc.md | 134 +++++++++++++++++++++++++++++++++++ docs/internal/runtime.md | 2 +- docs/operations.md | 13 ++-- docs/troubleshooting.md | 17 ++++- examples/config.nats.yml | 14 +++- examples/config.postgres.yml | 14 +++- 8 files changed, 199 insertions(+), 8 deletions(-) create mode 100644 docs/integrations/spc.md diff --git a/README.md b/README.md index a517f95..f3f4df6 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,7 @@ current working directory. - [Event wire contract](docs/integrations/events.md) - [Postgres table contract](docs/integrations/postgres.md) - [NWS integration notes](docs/integrations/nws.md) +- [SPC integration notes](docs/integrations/spc.md) - [Open-Meteo integration notes](docs/integrations/openmeteo.md) - [OpenWeather integration notes](docs/integrations/openweather.md) - [Architecture policy](docs/policy/architecture.md) diff --git a/docs/config.md b/docs/config.md index 1d7baff..edb7fa8 100644 --- a/docs/config.md +++ b/docs/config.md @@ -52,6 +52,18 @@ sources: url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20" user_agent: "Example weatherfeeder operator (ops@example.com)" + - name: SPCConvectiveOutlookLocal + mode: poll + driver: spc_convective_outlook + every: 30m + kinds: ["outlook"] + params: + latitude: 38.6239 + longitude: -90.3571 + location_id: "local" + location_name: "Configured point" + user_agent: "Example weatherfeeder operator (ops@example.com)" + sinks: - name: nats_weather driver: nats diff --git a/docs/integrations/spc.md b/docs/integrations/spc.md new file mode 100644 index 0000000..253639e --- /dev/null +++ b/docs/integrations/spc.md @@ -0,0 +1,134 @@ +# SPC Integration Notes + +## Purpose + +This document describes the Storm Prediction Center convective outlook usage +implemented by `weatherfeeder`. It is for developers and operators maintaining +SPC source configuration, provider helpers, fixtures, and tests. + +General config syntax belongs in [configuration](../config.md). Emitted JSON +events are documented in [event wire contract](events.md). + +## Implemented Driver + +| Driver | Kind | Raw schema | Canonical schema | +| --- | --- | --- | --- | +| `spc_convective_outlook` | `outlook` | `raw.spc.convective_outlook.v1` | `weather.outlook.v1` | + +## Config Requirements + +The driver requires: + +- `latitude` +- `longitude` +- `user_agent` + +Optional params are: + +- `location_id` +- `location_name` +- `geojson_urls` +- `discussion_urls` +- `rss_url` +- `http_timeout` +- `http_response_body_limit_bytes` + +RSS is not fetched unless `rss_url` is configured. URL override maps are intended +for tests and upstream URL changes; the default driver configuration owns the +current Day 1-3 SPC product URLs. + +## Upstream Products Used + +The source fetches twelve required GeoJSON products every poll: + +- Day 1 categorical, tornado, hail, and wind +- Day 2 categorical, tornado, hail, and wind +- Day 3 categorical, tornado, hail, and wind + +It also fetches three required print pages: + +- Day 1 convective outlook print page +- Day 2 convective outlook print page +- Day 3 convective outlook print page + +GeoJSON products are authoritative for outlook polygons, valid windows, issue +times, labels, and severity rank. Print pages are authoritative for headline, +summary, and discussion text. + +## Accept Headers + +GeoJSON requests use: + +```text +application/geo+json, application/json +``` + +Print-page requests use: + +```text +text/html, application/xhtml+xml +``` + +RSS requests, when configured, use: + +```text +application/rss+xml, application/xml, text/xml +``` + +## Polling And Raw Events + +The source polls all required GeoJSON and print-page products as one bundle. If +any required request fails or returns a non-2xx response, the poll returns an +error and emits no partial event. + +The raw payload contains fetched bodies plus configured location metadata and +per-product metadata. The source parses only the timestamp metadata needed for +event effective time selection; canonical mapping belongs to the normalizer. + +The source emits no event when a complete fetched bundle is unchanged from the +previous successful poll. It does this with a source-local hash of the fetched +document bodies. + +## Time Handling + +Raw source `effective_at` prefers: + +1. the latest valid GeoJSON `ISSUE_ISO`; +2. the latest print-page `Updated:` timestamp; +3. RSS `lastBuildDate` when RSS was fetched and parseable; +4. fetch time. + +The normalizer sets canonical `asOf` and normalized event `effective_at` from +the latest valid outlook feature `issuedAt`, with fallback to print-page update +time and then the incoming event envelope. + +## Mapping Notes + +Each GeoJSON feature becomes one canonical outlook. Products are ordered by day, +then categorical, tornado, hail, and wind. Feature order is preserved within +each product. + +The normalizer computes `containsLocation` with the configured latitude and +longitude against compact GeoJSON `Polygon` or `MultiPolygon` geometry. +Coordinates use GeoJSON order, `[longitude, latitude]`, and boundary points +count as contained. + +All outlook polygons are preserved, including polygons that do not contain the +configured point. Matching print-page headline, summary, and discussion text is +attached to every outlook for the same day. + +## Failure Behavior + +Constructor validation failures stop daemon startup. Polling failures are +returned to the scheduler and emit no event for that poll. + +Normalization fails when required GeoJSON timestamps, labels, geometry, or +configured coordinates are missing or invalid. Print-page extraction failures +also fail normalization because print pages are required inputs. + +## Tests To Inspect + +- `internal/providers/spc/*_test.go` +- `internal/sources/spc/*_test.go` +- `internal/normalizers/spc/*_test.go` +- fixtures under `internal/providers/spc/testdata` diff --git a/docs/internal/runtime.md b/docs/internal/runtime.md index 0fb42ee..f631fe9 100644 --- a/docs/internal/runtime.md +++ b/docs/internal/runtime.md @@ -74,7 +74,7 @@ Weatherfeeder-owned runtime state is in process: - event channel contents; - the bounded dedupe key set; -- source instances and their HTTP conditional validators; +- source instances and their in-memory unchanged-content state; - scheduler and dispatcher goroutines. There is no weatherfeeder-owned durable scheduler state, checkpoint, replay log, diff --git a/docs/operations.md b/docs/operations.md index 301f18a..08080e3 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -64,16 +64,20 @@ and before each interval tick. If no jitter is configured in code, feedkit uses Poll failures are logged and do not stop the daemon. A failed poll emits no events for that source until a subsequent poll succeeds. -## Conditional HTTP Fetches +## Unchanged Upstream Content -All current sources use feedkit's HTTP polling helper. By default, -`params.conditional` is `true`, so the helper keeps ETag and Last-Modified -validators in memory for each source instance. +Most sources use feedkit's single-document HTTP polling helper. By default, +`params.conditional` is `true` for those sources, so the helper keeps ETag and +Last-Modified validators in memory for each source instance. If the upstream returns `304 Not Modified`, the source emits no events for that poll. Validator state is in memory only; restarting the process starts with no cached validators. +The SPC convective outlook source polls multiple required documents as one +bundle. It emits no event when a later complete bundle has the same fetched +document bodies as the previous successful poll. + ## Processing And Dedupe Every event passes through normalization first and dedupe second. @@ -128,6 +132,7 @@ table in that same transaction. See the - scheduler goroutines and timers; - HTTP conditional request validators; +- source-local unchanged-content state; - event channel buffers; - per-sink fanout queues; - the dedupe ID set. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index b80b807..0764957 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -58,7 +58,8 @@ Diagnostic: compare the configured driver with the source driver table in Safe fix: correct the driver name. Current drivers include `nws_observation`, `nws_alerts`, `nws_forecast_hourly`, `nws_forecast_narrative`, `nws_forecast_discussion`, `nws_weatherstories`, `openmeteo_observation`, -`openmeteo_forecast`, and `openweather_observation`. +`openmeteo_forecast`, `openweather_observation`, and +`spc_convective_outlook`. ## `unknown sink driver` @@ -94,6 +95,20 @@ Diagnostic: inspect the named source in the error and check its `params`. Safe fix: add non-empty `url` and `user_agent` values. See [HTTP source params](config.md#http-source-params). +SPC convective outlook sources do not use `params.url`; they require +`latitude`, `longitude`, and `user_agent`. See +[SPC convective outlook params](config.md#spc-convective-outlook-params). + +## `params.latitude is required` Or `params.longitude is required` + +Symptom: startup exits for an `spc_convective_outlook` source. + +Likely cause: the source is missing one of the configured point coordinates. + +Diagnostic: inspect the named SPC source in the error and check its `params`. + +Safe fix: add numeric `latitude` and `longitude` values in decimal degrees. + ## `url must include units=metric` Symptom: startup exits for an `openweather_observation` source. diff --git a/examples/config.nats.yml b/examples/config.nats.yml index 0927a12..8d41f50 100644 --- a/examples/config.nats.yml +++ b/examples/config.nats.yml @@ -18,6 +18,18 @@ sources: url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20" user_agent: "weatherfeeder example (operator@example.com)" + - name: SPCConvectiveOutlookSTL + mode: poll + kinds: ["outlook"] + driver: spc_convective_outlook + every: 30m + params: + latitude: 38.6239 + longitude: -90.3571 + location_id: "stl" + location_name: "St. Louis, MO" + user_agent: "weatherfeeder example (operator@example.com)" + sinks: - name: nats_weather driver: nats @@ -27,4 +39,4 @@ sinks: routes: - sink: nats_weather - kinds: ["observation", "alert"] + kinds: ["observation", "alert", "outlook"] diff --git a/examples/config.postgres.yml b/examples/config.postgres.yml index 60f9d94..f857bf7 100644 --- a/examples/config.postgres.yml +++ b/examples/config.postgres.yml @@ -18,6 +18,18 @@ sources: url: "https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&hourly=temperature_2m,relative_humidity_2m,dew_point_2m,apparent_temperature,precipitation_probability,precipitation,snowfall,weather_code,surface_pressure,wind_speed_10m,wind_direction_10m&forecast_days=3" user_agent: "weatherfeeder example (operator@example.com)" + - name: SPCConvectiveOutlookSTL + mode: poll + kinds: ["outlook"] + driver: spc_convective_outlook + every: 30m + params: + latitude: 38.6239 + longitude: -90.3571 + location_id: "stl" + location_name: "St. Louis, MO" + user_agent: "weatherfeeder example (operator@example.com)" + sinks: - name: pg_weather driver: postgres @@ -29,4 +41,4 @@ sinks: routes: - sink: pg_weather - kinds: ["observation", "forecast"] + kinds: ["observation", "forecast", "outlook"]