ejr/weatherfeeder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
abb8f218ec1986785fa7c8ea1e2093eb99b3e782
weatherfeeder/docs/roadmap/spc.md

324 lines
10 KiB
Markdown
Raw Blame History

# SPC Convective Outlook Support Roadmap
## Summary
Add `weatherfeeder` support for Storm Prediction Center convective outlooks as a new canonical outlook domain. The feature should poll SPC GeoJSON outlook products, optionally enrich them with RSS discussion metadata, compute whether the configured forecast point is inside each active outlook polygon, normalize the result into a provider-independent schema, and persist it through existing sinks.
This is a new domain, not an extension of `weather.alert.v1`. SPC outlooks describe probabilistic/categorical risk areas over a validity window; alerts describe active actionable hazard messages.
## Public Contract
Add schema constants:
- Raw schema: `raw.spc.convective_outlook.v1`
- Canonical schema: `weather.outlook.v1`
Add source driver:
- `spc_convective_outlook`
Add event kind:
- `outlook`
Add canonical model types:
- `model.WeatherOutlookRun`
- `model.WeatherOutlook`
Recommended canonical run fields:
- `locationId`, `locationName`
- `latitude`, `longitude`
- `asOf`
- `issuedAt`
- `outlooks`
Recommended canonical outlook fields:
- `id`
- `provider`
- `product`
- `day`
- `outlookType`
- `label`
- `labelText`
- `severityRank`
- `validFrom`
- `validTo`
- `issuedAt`
- `expiresAt`
- `forecaster`
- `headline`
- `summary`
- `discussion`
- `sourceUrl`
- `imageUrl`
- `containsLocation`
- `geometry`
Contract defaults:
- `product` should be `convective`.
- `outlookType` should be one of `categorical`, `tornado`, `hail`, `wind`.
- `day` should be `1`, `2`, or `3` for this first implementation.
- `containsLocation` is computed against configured forecast coordinates.
- `geometry` should preserve compact GeoJSON geometry for auditability and future API use.
## Source Scope
The source should fetch a bundle of SPC products in one poll cycle and emit one raw event containing the fetched RSS metadata, GeoJSON products, configured point, and per-product fetch metadata.
Poll these GeoJSON URLs:
- `https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson`
Note: the initial candidate list duplicated Day 3 wind and omitted Day 2 wind. Use the corrected 12-product matrix above.
Also poll:
- `https://www.spc.noaa.gov/products/spcacrss.xml`
RSS usage:
- Use RSS as discussion/discovery metadata, not as the geometry source.
- Preserve item title, link, description text, pubDate, and guid where useful.
- Extract the narrative outlook text from the RSS item description when practical.
- Do not attempt to parse polygons from RSS HTML.
Polling cadence:
- Use a default cadence of `30m`, not daily/twice-daily. SPC current outlook files can update several times per day, and a 30-minute poll with HTTP caching is low cost and less likely to miss updates.
- Rely on ETag/Last-Modified handling from the HTTP source layer where available.
- Keep the cadence configurable via `every`.
Recommended config shape:
```yaml
- name: SPCConvectiveOutlookSTL
mode: poll
kinds: ["outlook"]
driver: spc_convective_outlook
every: 30m
params:
latitude: 38.6239
longitude: -90.3571
location_id: "stl"
location_name: "St. Louis, MO"
user_agent: "HomeOps (eric@maximumdirect.net)"
```
The source should own default SPC URLs, with optional params to override the RSS URL and product URLs for tests or future provider changes.
## Implementation Stages
### Stage 1: Raw Source and Schema
- Add schema constants and model placeholders.
- Add `internal/sources/spc` package.
- Implement `spc_convective_outlook` as a poll source.
- Fetch all configured GeoJSON products and RSS during a poll cycle.
- Emit one raw event with kind `outlook` and schema `raw.spc.convective_outlook.v1`.
- Use the latest valid `ISSUE_ISO`, RSS `lastBuildDate`, or fetch time for `effectiveAt`, in that order.
- Preserve partial fetch errors as source errors for the first implementation; do not emit incomplete outlook bundles unless a later explicit degraded-mode policy is added.
Tests:
- source driver builds as a `PollSource`
- source advertises kind `outlook`
- source emits one raw bundled event
- source chooses expected effective time
- source uses configured location metadata
- source fails clearly on missing latitude/longitude
### Stage 2: GeoJSON and Spatial Matching
- Add an internal geometry helper package, for example `internal/geo`.
- Support GeoJSON `Polygon` and `MultiPolygon`.
- Implement point-in-polygon with ring support:
- first ring is the exterior
- subsequent rings are holes
- boundary counts as inside
- GeoJSON coordinate order is `[longitude, latitude]`
- Use planar ray casting for this operational point-in-polygon check.
Tests:
- point inside polygon
- point outside polygon
- point on polygon boundary
- point inside a hole is outside
- point inside one multipolygon member is inside
- longitude/latitude order regression test
### Stage 3: Normalizer and Canonical Mapping
- Add `SPCConvectiveOutlookNormalizer`.
- Match only `raw.spc.convective_outlook.v1`.
- Decode the raw source bundle.
- Map each GeoJSON feature to one `WeatherOutlook`.
- Preserve SPC feature order within each product, then order products by day and type.
- Map properties:
- `VALID_ISO` -> `validFrom`
- `EXPIRE_ISO` -> `validTo` and `expiresAt`
- `ISSUE_ISO` -> `issuedAt`
- `FORECASTER` -> `forecaster`
- `LABEL` -> `label`
- `LABEL2` -> `labelText`
- `DN` -> `severityRank`
- Derive:
- `day` from product key or URL
- `outlookType` from product key or URL
- `id` from day, type, label, issuedAt, validFrom, and feature index
- `containsLocation` from configured point and GeoJSON geometry
- Enrich narrative fields from RSS where a matching day/product item can be determined.
- Set run `asOf` to the latest valid `issuedAt` across features, falling back to RSS `lastBuildDate`, then input event time.
- Set run `issuedAt` to the latest valid `issuedAt` across features.
- Set output event `effectiveAt` to run `asOf`.
Tests:
- normalizer routes only raw SPC schema
- categorical and probabilistic products map expected fields
- `containsLocation` is true for a known point inside a fixture polygon
- `containsLocation` is false outside
- missing optional RSS still permits GeoJSON normalization
- malformed required GeoJSON timestamps fail with useful context
- canonical JSON shape does not expose raw-provider-only bundle internals
### Stage 4: Postgres Sink
Add tables:
- `outlook_runs`
- `outlooks`
Suggested `outlook_runs` columns:
- event envelope columns
- `location_id`
- `location_name`
- `latitude`
- `longitude`
- `as_of`
- `issued_at`
- `outlook_count`
Suggested `outlooks` columns:
- `run_event_id`
- `outlook_index`
- `as_of`
- `product`
- `day`
- `outlook_type`
- `label`
- `label_text`
- `severity_rank`
- `valid_from`
- `valid_to`
- `issued_at`
- `expires_at`
- `forecaster`
- `headline`
- `summary`
- `discussion`
- `source_url`
- `image_url`
- `contains_location`
- `geometry_json`
Suggested indexes:
- `outlook_runs(location_id, as_of)`
- `outlooks(contains_location, valid_from, valid_to)`
- `outlooks(day, outlook_type, label)`
- `outlooks(valid_from, valid_to)`
Mapping rules:
- Store `geometry` as compact GeoJSON text in `geometry_json`.
- Require run `asOf`.
- Require outlook `validFrom`, `validTo`, `issuedAt`, `day`, `outlookType`, and `label`.
- Preserve all outlook polygons, not only polygons containing the configured point.
Tests:
- schema includes outlook tables and indexes
- mapper writes one run row plus one row per outlook
- mapper stores compact geometry JSON
- mapper rejects missing required run/outlook fields
### Stage 5: Config and Documentation
- Update sample config with `SPCConvectiveOutlookSTL`.
- Add `outlook` to route examples.
- Update README provider capabilities.
- Update `API.md` with `weather.outlook.v1`.
- Update Postgres sink docs with outlook table contract.
Docs should clearly state:
- RSS is used for narrative/discussion metadata.
- GeoJSON is used for polygons and point matching.
- `containsLocation` is computed by weatherfeeder at ingestion time.
- Geometry is stored for downstream audit/display.
### Stage 6: Weatherapi Follow-Up
Do not include weatherapi changes in the first weatherfeeder implementation unless explicitly requested.
Likely future weatherapi endpoints:
- `GET /outlooks/convective`
- `GET /outlooks/convective/active`
- `GET /outlooks/convective/location`
Recommended API behavior:
- latest run by default
- active outlooks filtered by current time and `containsLocation=true`
- optional filter query params for `day`, `outlookType`, and `containsLocation`
## Open Questions for Implementation
- Whether to include Day 4-8 probabilistic outlooks in a later version. Exclude them from v1.
- Whether to make partial source bundles acceptable if one product URL fails. Default v1 behavior should fail the poll and retry.
- Whether to parse detailed discussion text from RSS only, HTML pages, or both. Default v1 should use RSS only.
- Whether to keep `geometry` in canonical JSON permanently. Default v1 should include it because it preserves source context and enables downstream display.
## Verification Commands
Run focused tests:
```sh
go test ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./model
```
Run full weatherfeeder tests:
```sh
go test ./...
```
## Acceptance Criteria
- A configured SPC source emits raw outlook bundles.
- Normalization produces `weather.outlook.v1` events.
- Each outlook indicates whether the configured forecast point is inside its polygon.
- All current Day 1-3 categorical/tornado/hail/wind products are represented.
- RSS discussion metadata is preserved where available.
- Postgres sink persists outlook runs and outlook rows.
- Sample config and public docs describe the new kind, driver, schema, and storage contract.
Reference in New Issue View Git Blame Copy Permalink