diff --git a/README.md b/README.md index 4db5010..a517f95 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,9 @@ provider-specific responses into canonical weather events, and routes those events to configured sinks. It currently supports NWS observations, alerts, hourly forecasts, narrative -forecasts, forecast discussions, and weather stories; Open-Meteo observations -and hourly forecasts; and OpenWeather observations. Implemented sinks are -stdout, NATS, and Postgres. +forecasts, forecast discussions, and weather stories; SPC Day 1-3 convective +outlooks; Open-Meteo observations and hourly forecasts; and OpenWeather +observations. Implemented sinks are stdout, NATS, and Postgres. ## Quickstart diff --git a/cmd/weatherfeeder/config.yml b/cmd/weatherfeeder/config.yml index 0c16594..7371f9e 100644 --- a/cmd/weatherfeeder/config.yml +++ b/cmd/weatherfeeder/config.yml @@ -99,6 +99,18 @@ sources: url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20" user_agent: "HomeOps (eric@maximumdirect.net)" + - name: SPCConvectiveOutlookSTL + mode: poll + kinds: ["outlook"] + driver: spc_convective_outlook + every: 30m + params: + latitude: 38.6239 + longitude: -90.3571 + location_id: "stl" + location_name: "St. Louis, MO" + user_agent: "HomeOps (eric@maximumdirect.net)" + sinks: - name: stdout driver: stdout @@ -126,13 +138,13 @@ sinks: routes: - sink: stdout - kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"] + kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"] - sink: nats_weatherfeeder - kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"] + kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"] # - sink: pg_weatherfeeder -# kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"] +# kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"] # - sink: logfile -# kinds: ["observation", "alert", "forecast", "forecast_discussion", "weather_story"] +# kinds: ["observation", "alert", "forecast", "forecast_discussion", "weather_story", "outlook"] diff --git a/docs/config.md b/docs/config.md index a23758d..1d7baff 100644 --- a/docs/config.md +++ b/docs/config.md @@ -69,10 +69,10 @@ sinks: routes: - sink: nats_weather - kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"] + kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"] - sink: pg_weather - kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"] + kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"] ``` Do not commit real API keys, database passwords, or personal contact addresses in @@ -95,10 +95,10 @@ copyable configs. | `mode` | no | `poll`, `stream`, or omitted for auto. Current weatherfeeder drivers are polling drivers. | | `every` | yes | Poll interval for current weatherfeeder source drivers. | | `kinds` | no | Expected event kinds. If present, startup verifies they match the source driver. | -| `params` | driver-specific | Driver parameters. Current source drivers require HTTP params. | +| `params` | driver-specific | Driver parameters. See the source-specific sections below. | Current event kinds are `observation`, `forecast`, `forecast_discussion`, -`weather_story`, and `alert`. +`weather_story`, `alert`, and `outlook`. ## Source Drivers @@ -113,10 +113,11 @@ Current event kinds are `observation`, `forecast`, `forecast_discussion`, | `openmeteo_observation` | `observation` | Open-Meteo current conditions. | | `openmeteo_forecast` | `forecast` | Open-Meteo hourly forecast. | | `openweather_observation` | `observation` | OpenWeather current weather. | +| `spc_convective_outlook` | `outlook` | SPC Day 1-3 convective outlooks. | ## HTTP Source Params -All current source drivers use the shared HTTP polling helper. +Most source drivers use the shared HTTP polling helper. | Param | Required | Description | |---|:---:|---| @@ -132,6 +133,44 @@ source emits no events for that poll. OpenWeather observation URLs must include `units=metric`. Startup fails if the URL omits it or sets another unit system. +## SPC Convective Outlook Params + +`spc_convective_outlook` fetches the twelve required Day 1-3 GeoJSON outlook +products and the three required Day 1-3 print pages as one atomic bundle. + +| Param | Required | Description | +|---|:---:|---| +| `latitude` | yes | Location latitude in decimal degrees. | +| `longitude` | yes | Location longitude in decimal degrees. | +| `user_agent` | yes | User-Agent sent to SPC. `userAgent` is also accepted. | +| `location_id` | no | Operator-defined location identifier copied into canonical outlook runs. | +| `location_name` | no | Human location label copied into canonical outlook runs. | +| `geojson_urls` | no | Map of product key to override URL. Used for tests and upstream URL changes. | +| `discussion_urls` | no | Map of day key to override print-page URL. Used for tests and upstream URL changes. | +| `rss_url` | no | Optional RSS URL. RSS is not fetched unless this is configured. | +| `http_timeout` | no | Positive duration for the HTTP client timeout. | +| `http_response_body_limit_bytes` | no | Positive integer response body limit in bytes. | + +GeoJSON product keys are `day1_categorical`, `day1_tornado`, `day1_hail`, +`day1_wind`, `day2_categorical`, `day2_tornado`, `day2_hail`, `day2_wind`, +`day3_categorical`, `day3_tornado`, `day3_hail`, and `day3_wind`. +Discussion keys are `day1`, `day2`, and `day3`. + +```yaml +sources: + - name: SPCConvectiveOutlookSTL + mode: poll + kinds: ["outlook"] + driver: spc_convective_outlook + every: 30m + params: + latitude: 38.6239 + longitude: -90.3571 + location_id: "stl" + location_name: "St. Louis, MO" + user_agent: "Example weatherfeeder operator (ops@example.com)" +``` + ## Sink Fields | Field | Required | Description | diff --git a/docs/integrations/events.md b/docs/integrations/events.md index 4a08a6e..5d2dc79 100644 --- a/docs/integrations/events.md +++ b/docs/integrations/events.md @@ -37,6 +37,7 @@ Canonical schemas emitted after normalization: | `forecast_discussion` | `weather.forecast_discussion.v1` | `WeatherForecastDiscussion` | | `weather_story` | `weather.weather_story.v1` | `WeatherStoryRun` | | `alert` | `weather.alert.v1` | `WeatherAlertRun` | +| `outlook` | `weather.outlook.v1` | `WeatherOutlookRun` | Raw upstream schemas emitted by current sources: @@ -51,6 +52,7 @@ Raw upstream schemas emitted by current sources: | `weather_story` | `raw.nws.weatherstories.v1` | NWS weather stories JSON | | `forecast` | `raw.openmeteo.hourly.forecast.v1` | Open-Meteo hourly forecast JSON | | `alert` | `raw.nws.alerts.v1` | NWS alerts JSON | +| `outlook` | `raw.spc.convective_outlook.v1` | SPC convective outlook raw bundle | `standards.SchemaRawOpenWeatherHourlyForecastV1` exists in code, but no current registered source emits it. @@ -218,6 +220,55 @@ Payload type: `WeatherAlertRun`. `references[]` entries contain optional `id`, `identifier`, `sender`, and `sent` fields. +## `weather.outlook.v1` + +Payload type: `WeatherOutlookRun`. + +The current producer is the SPC convective outlook normalizer. It emits Day 1-3 +convective outlook polygons for categorical, tornado, hail, and wind products. +All timestamps are UTC. + +| Field | Type | Required | Notes | +|---|---|:---:|---| +| `locationId` | string | no | Operator-configured location identifier. | +| `locationName` | string | no | Operator-configured location label. | +| `latitude` | number | no | Configured point latitude in decimal degrees. | +| `longitude` | number | no | Configured point longitude in decimal degrees. | +| `asOf` | timestamp | yes | Snapshot time. For SPC, this is the latest outlook issue time when available. | +| `issuedAt` | timestamp | no | Latest issue time across outlook features when any feature exists. | +| `outlooks` | array | yes | Ordered outlook polygons. | + +`outlooks[]` entries: + +| Field | Type | Required | Notes | +|---|---|:---:|---| +| `id` | string | yes | Deterministic weatherfeeder outlook identifier. | +| `provider` | string | yes | Current value is `spc`. | +| `product` | string | yes | Current value is `convective`. | +| `day` | integer | yes | SPC outlook day, currently `1`, `2`, or `3`. | +| `outlookType` | string | yes | `categorical`, `tornado`, `hail`, or `wind`. | +| `label` | string | yes | SPC outlook label such as `SLGT` or `15`. | +| `labelText` | string | no | Human label text from SPC, when present. | +| `severityRank` | integer | no | SPC `DN` value, when present. | +| `validFrom` | timestamp | yes | Valid period start. | +| `validTo` | timestamp | yes | Valid period end. | +| `issuedAt` | timestamp | yes | Feature issue time. | +| `expiresAt` | timestamp | yes | Expiration time; currently equal to `validTo`. | +| `forecaster` | string | no | SPC forecaster text, when present. | +| `headline` | string | no | Matching Day 1-3 print-page product title. | +| `summary` | string | no | Text from the print-page `...SUMMARY...` section. | +| `discussion` | string | no | Cleaned full print-page product text. | +| `sourceUrl` | string | no | GeoJSON product URL for this outlook feature. | +| `imageUrl` | string | no | Reserved for provider image URLs; currently empty. | +| `containsLocation` | boolean | yes | Whether the configured point is inside or on the boundary of the polygon. | +| `geometry` | object | yes | Compact GeoJSON `Polygon` or `MultiPolygon` geometry. | + +`geometry` preserves the SPC feature geometry as compact GeoJSON using +`[longitude, latitude]` coordinate order. `containsLocation` is computed with +that geometry and the configured source `latitude`/`longitude`; boundary points +count as contained. All outlook polygons are emitted, including polygons that do +not contain the configured point. + ## Compact Example ```json diff --git a/docs/integrations/postgres.md b/docs/integrations/postgres.md index 2ff6731..767510c 100644 --- a/docs/integrations/postgres.md +++ b/docs/integrations/postgres.md @@ -23,6 +23,7 @@ Events are mapped only for canonical weather schemas: - `weather.forecast_discussion.v1` - `weather.weather_story.v1` - `weather.alert.v1` +- `weather.outlook.v1` Unsupported schemas produce no writes for this sink. Mapped events are inserted transactionally. Inserts use ordinary `INSERT`; duplicate primary keys fail the @@ -56,6 +57,8 @@ Parent tables store the feed event envelope: | `alert_runs` | `event_id` | `as_of` | | `alerts` | `run_event_id`, `alert_index` | `as_of` | | `alert_references` | `run_event_id`, `alert_index`, `reference_index` | `as_of` | +| `outlook_runs` | `event_id` | `as_of` | +| `outlooks` | `run_event_id`, `outlook_index` | `as_of` | ## Table Contract @@ -379,6 +382,72 @@ Indexes: | `sender` | `TEXT` | yes | `payload.alerts[].references[].sender` | | `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].references[].sent` | +### `outlook_runs` + +Primary key: `event_id` + +Prune column: `as_of` + +Indexes: + +- `idx_wf_outlook_run_location_as_of` on `location_id`, `as_of` +- `idx_wf_outlook_run_as_of` on `as_of` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `event_id` | `TEXT` | no | `event.id` | +| `event_kind` | `TEXT` | no | `event.kind` | +| `event_source` | `TEXT` | no | `event.source` | +| `event_schema` | `TEXT` | no | `event.schema` | +| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` | +| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` | +| `location_id` | `TEXT` | yes | `payload.locationId` | +| `location_name` | `TEXT` | yes | `payload.locationName` | +| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` | +| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` | +| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` | +| `issued_at` | `TIMESTAMPTZ` | yes | `payload.issuedAt` | +| `outlook_count` | `INTEGER` | no | `len(payload.outlooks)` | + +### `outlooks` + +Primary key: `run_event_id`, `outlook_index` + +Prune column: `as_of` + +Foreign key: `run_event_id` references `outlook_runs(event_id)` with cascade +delete. + +Indexes: + +- `idx_wf_outlooks_contains_valid` on `contains_location`, `valid_from`, `valid_to` +- `idx_wf_outlooks_day_type_label` on `day`, `outlook_type`, `label` +- `idx_wf_outlooks_valid` on `valid_from`, `valid_to` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `run_event_id` | `TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. | +| `outlook_index` | `INTEGER` | no | `payload.outlooks[]` index. | +| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` | +| `product` | `TEXT` | no | `payload.outlooks[].product` | +| `day` | `INTEGER` | no | `payload.outlooks[].day` | +| `outlook_type` | `TEXT` | no | `payload.outlooks[].outlookType` | +| `label` | `TEXT` | no | `payload.outlooks[].label` | +| `label_text` | `TEXT` | yes | `payload.outlooks[].labelText` | +| `severity_rank` | `INTEGER` | yes | `payload.outlooks[].severityRank` | +| `valid_from` | `TIMESTAMPTZ` | no | `payload.outlooks[].validFrom` | +| `valid_to` | `TIMESTAMPTZ` | no | `payload.outlooks[].validTo` | +| `issued_at` | `TIMESTAMPTZ` | no | `payload.outlooks[].issuedAt` | +| `expires_at` | `TIMESTAMPTZ` | no | `payload.outlooks[].expiresAt` | +| `forecaster` | `TEXT` | yes | `payload.outlooks[].forecaster` | +| `headline` | `TEXT` | yes | `payload.outlooks[].headline` | +| `summary` | `TEXT` | yes | `payload.outlooks[].summary` | +| `discussion` | `TEXT` | yes | `payload.outlooks[].discussion` | +| `source_url` | `TEXT` | yes | `payload.outlooks[].sourceUrl` | +| `image_url` | `TEXT` | yes | `payload.outlooks[].imageUrl` | +| `contains_location` | `BOOLEAN` | no | `payload.outlooks[].containsLocation` | +| `geometry_json` | `TEXT` | no | Compact JSON from `payload.outlooks[].geometry` | + ## Retention When sink param `prune` is set, every successful write transaction deletes rows @@ -401,3 +470,5 @@ binary does not provide CLI commands for them. - `WeatherAlertRun`: read `alert_runs`, join `alerts` by `run_event_id` ordered by `alert_index`, then join `alert_references` by `run_event_id` and `alert_index` ordered by `reference_index`. +- `WeatherOutlookRun`: read `outlook_runs`, then join `outlooks` by + `run_event_id` ordered by `outlook_index`. diff --git a/docs/internal/normalizers.md b/docs/internal/normalizers.md index 2ee8280..9b149e4 100644 --- a/docs/internal/normalizers.md +++ b/docs/internal/normalizers.md @@ -27,6 +27,7 @@ Current mappings: | `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` | | `raw.nws.weatherstories.v1` | `weather.weather_story.v1` | | `raw.nws.alerts.v1` | `weather.alert.v1` | +| `raw.spc.convective_outlook.v1` | `weather.outlook.v1` | ## Boundaries @@ -54,9 +55,18 @@ Weatherfeeder registers normalizers in a stable order: 1. NWS 2. Open-Meteo 3. OpenWeather +4. SPC The current normalizers avoid ambiguous matches by using schema equality. +The SPC outlook normalizer decodes the raw multi-document bundle, maps each +GeoJSON feature to a canonical outlook, and enriches all outlooks for a day with +the matching print-page headline, summary, and discussion. It preserves compact +GeoJSON feature geometry and computes `containsLocation` with +`internal/geo.ContainsPoint` using the source-configured point. Boundary points +count as contained, and all polygons are preserved whether or not they contain +the point. + ## State Normalizers should be stateless. Shared helpers should be deterministic and free @@ -78,6 +88,7 @@ to four digits after the decimal point. - provider normalizer tests under `internal/normalizers/nws` - provider normalizer tests under `internal/normalizers/openmeteo` - provider normalizer tests under `internal/normalizers/openweather` +- provider normalizer tests under `internal/normalizers/spc` - common helper tests under `internal/normalizers/common` ## Invariants diff --git a/docs/internal/postgres-sink.md b/docs/internal/postgres-sink.md index 4b9d77d..f15612c 100644 --- a/docs/internal/postgres-sink.md +++ b/docs/internal/postgres-sink.md @@ -19,6 +19,7 @@ Inputs are canonical feed events. The mapper currently handles these schemas: - `weather.forecast_discussion.v1` - `weather.weather_story.v1` - `weather.alert.v1` +- `weather.outlook.v1` Outputs are feedkit `PostgresWrite` values for weatherfeeder-owned tables. Unsupported schemas produce no writes and no error. @@ -80,6 +81,7 @@ Child rows use positional indexes to preserve canonical array order: - `story_index` - `alert_index` - `reference_index` +- `outlook_index` Required canonical fields are validated before writes are returned: @@ -89,10 +91,14 @@ Required canonical fields are validated before writes are returned: - forecast discussions require `issuedAt` and `product`; - weather story runs require `asOf`, and each story requires `startTime`, `endTime`, and `updatedAt`; -- alert runs require `asOf`, and each alert requires `id`. +- alert runs require `asOf`, and each alert requires `id`; +- outlook runs require `asOf`, and each outlook requires `id`, `provider`, + `product`, `day`, `outlookType`, `label`, `validFrom`, `validTo`, `issuedAt`, + `expiresAt`, and `geometry`. Nullable canonical values are converted to SQL nulls by mapper helpers. -Observation present-weather raw values are stored as compact JSON text. +Observation present-weather raw values and outlook geometry values are stored as +compact JSON text. ## Failure Behavior diff --git a/docs/internal/sources.md b/docs/internal/sources.md index 71613bc..0e2e9aa 100644 --- a/docs/internal/sources.md +++ b/docs/internal/sources.md @@ -28,6 +28,7 @@ Current drivers: | `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` | | `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` | | `openweather_observation` | `observation` | `raw.openweather.current.v1` | +| `spc_convective_outlook` | `outlook` | `raw.spc.convective_outlook.v1` | ## Boundaries @@ -40,7 +41,7 @@ Current drivers: ## Config Fields Used -All current source drivers use feedkit `HTTPSource`. +Most source drivers use feedkit `HTTPSource`. Required params: @@ -56,12 +57,18 @@ Optional params: OpenWeather observation sources additionally require the configured URL to use metric units. This is enforced by `internal/providers/openweather`. +The SPC convective outlook source is a multi-document poll source rather than a +single-URL `HTTPSource`. It requires `latitude`, `longitude`, and `user_agent`; +accepts optional `location_id`, `location_name`, `geojson_urls`, +`discussion_urls`, and `rss_url`; and supports `http_timeout` and +`http_response_body_limit_bytes`. + Source-level `kinds`, when configured, are validated against the source's advertised `Kinds()`. ## External Adapters Used -Sources use feedkit's HTTP helper for: +Most sources use feedkit's HTTP helper for: - request construction; - `User-Agent` and `Accept` headers; @@ -71,11 +78,15 @@ Sources use feedkit's HTTP helper for: NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs. +SPC helpers define Day 1-3 product metadata, parse GeoJSON timestamps, extract +cleaned print-page discussion text, and parse optional RSS metadata. ## State -HTTP conditional validators are held in each source instance. They are not -persisted across process restarts. +HTTP conditional validators are held in each single-document HTTP source +instance. They are not persisted across process restarts. The SPC outlook source +keeps only a source-local hash of the most recent complete required product +bundle and emits no event when a later complete bundle is unchanged. ## Failure Behavior @@ -88,12 +99,18 @@ to default ID/effective-time behavior from feedkit source helpers. Unchanged conditional responses return no events and no error. +The SPC outlook source fetches all configured Day 1-3 GeoJSON products and print +pages atomically. If any required GeoJSON or print-page request fails, the poll +returns an error and emits no partial bundle. RSS is fetched only when `rss_url` +is configured. + ## Tests To Inspect - `internal/sources/builtins_test.go` - provider source tests under `internal/sources/nws` - provider source tests under `internal/sources/openmeteo` - provider source tests under `internal/sources/openweather` +- provider source tests under `internal/sources/spc` - provider helper tests under `internal/providers` ## Invariants