weatherfeeder/docs/roadmap/spc.md

SPC Convective Outlook Support Roadmap

Summary

Add weatherfeeder support for Storm Prediction Center convective outlooks as a new canonical outlook domain. The feature should poll SPC GeoJSON outlook products, optionally enrich them with RSS discussion metadata, compute whether the configured forecast point is inside each active outlook polygon, normalize the result into a provider-independent schema, and persist it through existing sinks.

This is a new domain, not an extension of weather.alert.v1. SPC outlooks describe probabilistic/categorical risk areas over a validity window; alerts describe active actionable hazard messages.

Public Contract

Add schema constants:

• Raw schema: raw.spc.convective_outlook.v1
• Canonical schema: weather.outlook.v1

Add source driver:

• spc_convective_outlook

Add event kind:

• outlook

Add canonical model types:

• model.WeatherOutlookRun
• model.WeatherOutlook

Recommended canonical run fields:

• locationId, locationName
• latitude, longitude
• asOf
• issuedAt
• outlooks

Recommended canonical outlook fields:

• id
• provider
• product
• day
• outlookType
• label
• labelText
• severityRank
• validFrom
• validTo
• issuedAt
• expiresAt
• forecaster
• headline
• summary
• discussion
• sourceUrl
• imageUrl
• containsLocation
• geometry

Contract defaults:

• product should be convective.
• outlookType should be one of categorical, tornado, hail, wind.
• day should be 1, 2, or 3 for this first implementation.
• containsLocation is computed against configured forecast coordinates.
• geometry should preserve compact GeoJSON geometry for auditability and future API use.

Source Scope

The source should fetch a bundle of SPC products in one poll cycle and emit one raw event containing the fetched RSS metadata, GeoJSON products, configured point, and per-product fetch metadata.

Poll these GeoJSON URLs:

• https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson

Note: the initial candidate list duplicated Day 3 wind and omitted Day 2 wind. Use the corrected 12-product matrix above.

Also poll:

• https://www.spc.noaa.gov/products/spcacrss.xml

RSS usage:

• Use RSS as discussion/discovery metadata, not as the geometry source.
• Preserve item title, link, description text, pubDate, and guid where useful.
• Extract the narrative outlook text from the RSS item description when practical.
• Do not attempt to parse polygons from RSS HTML.

Polling cadence:

• Use a default cadence of 30m, not daily/twice-daily. SPC current outlook files can update several times per day, and a 30-minute poll with HTTP caching is low cost and less likely to miss updates.
• Rely on ETag/Last-Modified handling from the HTTP source layer where available.
• Keep the cadence configurable via every.

Recommended config shape:

- name: SPCConvectiveOutlookSTL
  mode: poll
  kinds: ["outlook"]
  driver: spc_convective_outlook
  every: 30m
  params:
    latitude: 38.6239
    longitude: -90.3571
    location_id: "stl"
    location_name: "St. Louis, MO"
    user_agent: "HomeOps (eric@maximumdirect.net)"

The source should own default SPC URLs, with optional params to override the RSS URL and product URLs for tests or future provider changes.

Implementation Stages

Stage 1: Raw Source and Schema

• Add schema constants and model placeholders.
• Add internal/sources/spc package.
• Implement spc_convective_outlook as a poll source.
• Fetch all configured GeoJSON products and RSS during a poll cycle.
• Emit one raw event with kind outlook and schema raw.spc.convective_outlook.v1.
• Use the latest valid ISSUE_ISO, RSS lastBuildDate, or fetch time for effectiveAt, in that order.
• Preserve partial fetch errors as source errors for the first implementation; do not emit incomplete outlook bundles unless a later explicit degraded-mode policy is added.

Tests:

• source driver builds as a PollSource
• source advertises kind outlook
• source emits one raw bundled event
• source chooses expected effective time
• source uses configured location metadata
• source fails clearly on missing latitude/longitude

Stage 2: GeoJSON and Spatial Matching

• Add an internal geometry helper package, for example internal/geo.
• Support GeoJSON Polygon and MultiPolygon.
• Implement point-in-polygon with ring support:
  • first ring is the exterior
  • subsequent rings are holes
  • boundary counts as inside
  • GeoJSON coordinate order is [longitude, latitude]
• Use planar ray casting for this operational point-in-polygon check.

Tests:

• point inside polygon
• point outside polygon
• point on polygon boundary
• point inside a hole is outside
• point inside one multipolygon member is inside
• longitude/latitude order regression test

Stage 3: Normalizer and Canonical Mapping

• Add SPCConvectiveOutlookNormalizer.
• Match only raw.spc.convective_outlook.v1.
• Decode the raw source bundle.
• Map each GeoJSON feature to one WeatherOutlook.
• Preserve SPC feature order within each product, then order products by day and type.
• Map properties:
  • VALID_ISO -> validFrom
  • EXPIRE_ISO -> validTo and expiresAt
  • ISSUE_ISO -> issuedAt
  • FORECASTER -> forecaster
  • LABEL -> label
  • LABEL2 -> labelText
  • DN -> severityRank
• Derive:
  • day from product key or URL
  • outlookType from product key or URL
  • id from day, type, label, issuedAt, validFrom, and feature index
  • containsLocation from configured point and GeoJSON geometry
• Enrich narrative fields from RSS where a matching day/product item can be determined.
• Set run asOf to the latest valid issuedAt across features, falling back to RSS lastBuildDate, then input event time.
• Set run issuedAt to the latest valid issuedAt across features.
• Set output event effectiveAt to run asOf.

Tests:

• normalizer routes only raw SPC schema
• categorical and probabilistic products map expected fields
• containsLocation is true for a known point inside a fixture polygon
• containsLocation is false outside
• missing optional RSS still permits GeoJSON normalization
• malformed required GeoJSON timestamps fail with useful context
• canonical JSON shape does not expose raw-provider-only bundle internals

Stage 4: Postgres Sink

Add tables:

• outlook_runs
• outlooks

Suggested outlook_runs columns:

• event envelope columns
• location_id
• location_name
• latitude
• longitude
• as_of
• issued_at
• outlook_count

Suggested outlooks columns:

• run_event_id
• outlook_index
• as_of
• product
• day
• outlook_type
• label
• label_text
• severity_rank
• valid_from
• valid_to
• issued_at
• expires_at
• forecaster
• headline
• summary
• discussion
• source_url
• image_url
• contains_location
• geometry_json

Suggested indexes:

• outlook_runs(location_id, as_of)
• outlooks(contains_location, valid_from, valid_to)
• outlooks(day, outlook_type, label)
• outlooks(valid_from, valid_to)

Mapping rules:

• Store geometry as compact GeoJSON text in geometry_json.
• Require run asOf.
• Require outlook validFrom, validTo, issuedAt, day, outlookType, and label.
• Preserve all outlook polygons, not only polygons containing the configured point.

Tests:

• schema includes outlook tables and indexes
• mapper writes one run row plus one row per outlook
• mapper stores compact geometry JSON
• mapper rejects missing required run/outlook fields

Stage 5: Config and Documentation

• Update sample config with SPCConvectiveOutlookSTL.
• Add outlook to route examples.
• Update README provider capabilities.
• Update API.md with weather.outlook.v1.
• Update Postgres sink docs with outlook table contract.

Docs should clearly state:

• RSS is used for narrative/discussion metadata.
• GeoJSON is used for polygons and point matching.
• containsLocation is computed by weatherfeeder at ingestion time.
• Geometry is stored for downstream audit/display.

Stage 6: Weatherapi Follow-Up

Do not include weatherapi changes in the first weatherfeeder implementation unless explicitly requested.

Likely future weatherapi endpoints:

• GET /outlooks/convective
• GET /outlooks/convective/active
• GET /outlooks/convective/location

Recommended API behavior:

• latest run by default
• active outlooks filtered by current time and containsLocation=true
• optional filter query params for day, outlookType, and containsLocation

Open Questions for Implementation

• Whether to include Day 4-8 probabilistic outlooks in a later version. Exclude them from v1.
• Whether to make partial source bundles acceptable if one product URL fails. Default v1 behavior should fail the poll and retry.
• Whether to parse detailed discussion text from RSS only, HTML pages, or both. Default v1 should use RSS only.
• Whether to keep geometry in canonical JSON permanently. Default v1 should include it because it preserves source context and enables downstream display.

Verification Commands

Run focused tests:

go test ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./model

Run full weatherfeeder tests:

go test ./...

Acceptance Criteria

• A configured SPC source emits raw outlook bundles.
• Normalization produces weather.outlook.v1 events.
• Each outlook indicates whether the configured forecast point is inside its polygon.
• All current Day 1-3 categorical/tornado/hail/wind products are represented.
• RSS discussion metadata is preserved where available.
• Postgres sink persists outlook runs and outlook rows.
• Sample config and public docs describe the new kind, driver, schema, and storage contract.