weatherfeeder/docs/roadmap/spc.md

SPC Convective Outlook Support Roadmap

Summary

Add weatherfeeder support for Storm Prediction Center convective outlooks as a new canonical outlook domain. The feature should poll SPC GeoJSON outlook products, enrich them with Day 1-3 print-page discussion text, compute whether the configured forecast point is inside each active outlook polygon, normalize the result into a provider-independent schema, and persist it through existing sinks.

This is a new domain, not an extension of weather.alert.v1. SPC outlooks describe probabilistic/categorical risk areas over a validity window; alerts describe active actionable hazard messages.

Public Contract

Add schema constants:

• Raw schema: raw.spc.convective_outlook.v1
• Canonical schema: weather.outlook.v1

Add source driver:

• spc_convective_outlook

Add event kind:

• outlook

Add canonical model types:

• model.WeatherOutlookRun
• model.WeatherOutlook

Recommended canonical run fields:

• locationId, locationName
• latitude, longitude
• asOf
• issuedAt
• outlooks

Recommended canonical outlook fields:

• id
• provider
• product
• day
• outlookType
• label
• labelText
• severityRank
• validFrom
• validTo
• issuedAt
• expiresAt
• forecaster
• headline
• summary
• discussion
• sourceUrl
• imageUrl
• containsLocation
• geometry

Contract defaults:

• product should be convective.
• outlookType should be one of categorical, tornado, hail, wind.
• day should be 1, 2, or 3 for this first implementation.
• containsLocation is computed against configured forecast coordinates.
• geometry should preserve compact GeoJSON geometry for auditability and future API use.
• Day 4-8 outlooks are out of scope for this first implementation. They have different product semantics and should be added in a later schema-compatible extension only after Day 1-3 support is proven.

Source Scope

The source should fetch a bundle of SPC products in one poll cycle and emit one raw event containing fetched GeoJSON products, fetched Day 1-3 print pages, optional RSS metadata, configured point metadata, and per-product fetch metadata. The source may decode minimal timestamps for event identity/effective time, but provider-to-canonical mapping belongs in the normalizer.

Poll these GeoJSON URLs:

• https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson
• https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson

Implementation note: the initial candidate list duplicated Day 3 wind and omitted Day 2 wind. Use the corrected 12-product matrix above.

Poll these print-page discussion URLs:

• https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html
• https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html
• https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html

Print-page usage:

• Use print pages as the authoritative source for Day 1-3 outlook discussion text.
• Preserve source URL, page title/header where available, page-level Updated: timestamp where parseable, and the raw HTML or extracted <pre> product block in the raw bundle.
• Extract canonical headline, summary, discussion, and discussion metadata in the normalizer or shared SPC provider helper.
• Treat the GeoJSON files, not print pages, as authoritative for polygons and validity windows.
• Ignore page risk tables and image assets for v1 unless they are needed to support discussion parsing.

Optionally poll:

• https://www.spc.noaa.gov/products/spcacrss.xml

RSS usage:

• Use RSS as supplemental discovery or health metadata, not as the primary discussion or geometry source.
• Preserve item title, link, description text, pubDate, and guid where useful.
• Do not depend on RSS completeness for Day 1-3 discussion text. The RSS feed can omit current outlook products, including a current Day 2 discussion.
• Do not attempt to parse polygons from RSS HTML.

Polling cadence:

• Use a default cadence of 30m, not daily/twice-daily. SPC current outlook files can update several times per day, and a 30-minute poll with HTTP caching is low cost and less likely to miss updates.
• Rely on ETag/Last-Modified handling from the HTTP source layer where available.
• Keep the cadence configurable via every.

Recommended config shape:

- name: SPCConvectiveOutlookSTL
  mode: poll
  kinds: ["outlook"]
  driver: spc_convective_outlook
  every: 30m
  params:
    latitude: 38.6239
    longitude: -90.3571
    location_id: "stl"
    location_name: "St. Louis, MO"
    user_agent: "HomeOps (eric@maximumdirect.net)"

The source should own default SPC URLs, with optional params to override GeoJSON URLs, print-page URLs, and the RSS URL for tests or future provider changes.

Implementation Stages

Stage 1: Raw Source and Schema

• Add schema constants.
• Add internal/sources/spc package.
• Implement spc_convective_outlook as a poll source.
• Fetch all configured GeoJSON products and Day 1-3 print pages during a poll cycle.
• Fetch RSS only when configured or retained as an optional supplemental source.
• Emit one raw event with kind outlook and schema raw.spc.convective_outlook.v1.
• Use the latest valid ISSUE_ISO, print-page Updated: timestamp, RSS lastBuildDate, or fetch time for effectiveAt, in that order.
• Preserve partial fetch errors as source errors for the first implementation; do not emit incomplete outlook bundles unless a later explicit degraded-mode policy is added.
• Keep source tests deterministic with local test servers and fixtures; do not call live SPC services from unit tests.

Tests:

• source driver builds as a PollSource
• source advertises kind outlook
• source emits one raw bundled event
• source chooses expected effective time
• source uses configured location metadata
• source fails clearly on missing latitude/longitude
• source fails clearly when a configured GeoJSON or print-page fetch fails

Stage 2: GeoJSON and Spatial Matching

• Add an internal geometry helper package, for example internal/geo.
• Support GeoJSON Polygon and MultiPolygon.
• Implement point-in-polygon with ring support:
  • first ring is the exterior
  • subsequent rings are holes
  • boundary counts as inside
  • GeoJSON coordinate order is [longitude, latitude]
• Use planar ray casting for this operational point-in-polygon check.

Tests:

• point inside polygon
• point outside polygon
• point on polygon boundary
• point inside a hole is outside
• point inside one multipolygon member is inside
• longitude/latitude order regression test

Stage 3: Normalizer and Canonical Mapping

• Add model.WeatherOutlookRun and model.WeatherOutlook.
• Add SPCConvectiveOutlookNormalizer.
• Match only raw.spc.convective_outlook.v1.
• Decode the raw source bundle.
• Map each GeoJSON feature to one WeatherOutlook.
• Preserve SPC feature order within each product, then order products by day and type.
• Map properties:
  • VALID_ISO -> validFrom
  • EXPIRE_ISO -> validTo and expiresAt
  • ISSUE_ISO -> issuedAt
  • FORECASTER -> forecaster
  • LABEL -> label
  • LABEL2 -> labelText
  • DN -> severityRank
• Derive:
  • day from product key or URL
  • outlookType from product key or URL
  • id from day, type, label, issuedAt, validFrom, and feature index
  • containsLocation from configured point and GeoJSON geometry
• Enrich narrative fields from the matching Day 1-3 print page:
  • parse the main product text from the <pre> block
  • strip embedded tags and scripts before text parsing
  • preserve line breaks in discussion text
  • parse page/title metadata, product title, issued line, valid line, correction markers such as CORR 1, forecaster, summary, and full discussion where practical
  • use print-page text for canonical headline, summary, and discussion
• Use RSS only as optional supplemental metadata when present; missing or incomplete RSS must not prevent GeoJSON plus print-page normalization.
• Set run asOf to the latest valid issuedAt across features, falling back to print-page Updated: timestamps, RSS lastBuildDate, then input event time.
• Set run issuedAt to the latest valid issuedAt across features.
• Set output event effectiveAt to run asOf.

Tests:

• normalizer routes only raw SPC schema
• categorical and probabilistic products map expected fields
• containsLocation is true for a known point inside a fixture polygon
• containsLocation is false outside
• print-page discussion text maps to canonical headline, summary, and discussion
• correction markers such as CORR 1 are preserved where present
• missing optional RSS still permits GeoJSON and print-page normalization
• malformed required GeoJSON timestamps fail with useful context
• canonical JSON shape does not expose raw-provider-only bundle internals

Stage 4: Postgres Sink

Add tables:

• outlook_runs
• outlooks

Suggested outlook_runs columns:

• event envelope columns
• location_id
• location_name
• latitude
• longitude
• as_of
• issued_at
• outlook_count

Suggested outlooks columns:

• run_event_id
• outlook_index
• as_of
• product
• day
• outlook_type
• label
• label_text
• severity_rank
• valid_from
• valid_to
• issued_at
• expires_at
• forecaster
• headline
• summary
• discussion
• source_url
• image_url
• contains_location
• geometry_json

Suggested indexes:

• outlook_runs(location_id, as_of)
• outlooks(contains_location, valid_from, valid_to)
• outlooks(day, outlook_type, label)
• outlooks(valid_from, valid_to)

Mapping rules:

• Store geometry as compact GeoJSON text in geometry_json.
• Require run asOf.
• Require outlook validFrom, validTo, issuedAt, day, outlookType, and label.
• Preserve all outlook polygons, not only polygons containing the configured point.

Tests:

• schema includes outlook tables and indexes
• mapper writes one run row plus one row per outlook
• mapper stores compact geometry JSON
• mapper rejects missing required run/outlook fields

Stage 5: Config and Documentation

• Update sample config with SPCConvectiveOutlookSTL.
• Add outlook to route examples.
• Update current-behavior docs in the implementation change, not before behavior exists.
• Update docs/config.md with source params.
• Update docs/integrations/events.md with weather.outlook.v1.
• Update docs/integrations/postgres.md with outlook table contract.
• Update relevant internal docs under docs/internal/ if source, normalizer, or sink behavior needs maintainer context.
• Update README.md only with brief implemented-provider capability and links to canonical docs.

Docs should clearly state:

• Print pages are used for Day 1-3 narrative/discussion text.
• RSS, if retained, is supplemental and not required for complete discussion coverage.
• GeoJSON is used for polygons, validity windows, and point matching.
• containsLocation is computed by weatherfeeder at ingestion time.
• Geometry is stored for downstream audit/display.
• Day 4-8 outlooks are not included in the initial implementation.

Stage 6: Weatherapi Follow-Up

Do not include weatherapi changes in the first weatherfeeder implementation unless explicitly requested.

Likely future weatherapi endpoints:

• GET /outlooks/convective
• GET /outlooks/convective/active
• GET /outlooks/convective/location

Recommended API behavior:

• latest run by default
• active outlooks filtered by current time and containsLocation=true
• optional filter query params for day, outlookType, and containsLocation

Verification Commands

Run focused tests:

go test ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./model

Run full weatherfeeder tests:

go test ./...

Acceptance Criteria

• A configured SPC source emits raw outlook bundles.
• Normalization produces weather.outlook.v1 events.
• Each outlook indicates whether the configured forecast point is inside its polygon.
• All current Day 1-3 categorical/tornado/hail/wind products are represented.
• Day 1-3 print-page discussion text is preserved where available.
• RSS metadata is optional and supplemental if retained.
• Postgres sink persists outlook runs and outlook rows.
• Sample config and public docs describe the new kind, driver, schema, and storage contract.