# SPC Convective Outlook Implementation Plan ## Purpose Implement `weatherfeeder` support for Storm Prediction Center Day 1-3 convective outlooks described in [`docs/roadmap/spc.md`](spc.md). This plan is written for an LLM coding agent and should be followed stage by stage. This is a planning document only. The implementation must preserve the existing weatherfeeder architecture: sources emit raw provider events, normalizers map raw payloads into canonical model types, and sinks persist canonical schemas. ## Decisions The following choices are fixed for this implementation: - Scope is Day 1-3 SPC convective outlooks only. - Day 4-8 outlooks are out of scope. - GeoJSON files are authoritative for polygons, validity windows, issue times, outlook labels, and severity ranking. - Day 1-3 print pages are authoritative for discussion text. - RSS is optional supplemental metadata only and must not be required for correctness. - Do not fetch RSS by default. Include RSS only when an optional `rss_url` source param is configured. - A poll is atomic for required products. If any configured GeoJSON or print-page URL fails or returns a non-2xx response, return an error and emit no event. - Use compact GeoJSON geometry in the canonical payload for auditability and downstream display. - Use standard-library-first parsing. Do not add an HTML parsing dependency unless string-based extraction proves unmaintainable during implementation. - Keep all new planned behavior inside `weatherfeeder`; do not make `weatherapi` changes in this pass. ## Public Contract Add schema constants in `standards/schema.go`: - `SchemaRawSPCConvectiveOutlookV1 = "raw.spc.convective_outlook.v1"` - `SchemaWeatherOutlookV1 = "weather.outlook.v1"` Add source driver: - `spc_convective_outlook` Add event kind: - `outlook` Add canonical model types: - `model.WeatherOutlookRun` - `model.WeatherOutlook` Canonical run fields: ```go type WeatherOutlookRun struct { LocationID string `json:"locationId,omitempty"` LocationName string `json:"locationName,omitempty"` Latitude *float64 `json:"latitude,omitempty"` Longitude *float64 `json:"longitude,omitempty"` AsOf time.Time `json:"asOf"` IssuedAt *time.Time `json:"issuedAt,omitempty"` Outlooks []WeatherOutlook `json:"outlooks"` } ``` Canonical outlook fields: ```go type WeatherOutlook struct { ID string `json:"id"` Provider string `json:"provider"` Product string `json:"product"` Day int `json:"day"` OutlookType string `json:"outlookType"` Label string `json:"label"` LabelText string `json:"labelText,omitempty"` SeverityRank *int `json:"severityRank,omitempty"` ValidFrom time.Time `json:"validFrom"` ValidTo time.Time `json:"validTo"` IssuedAt time.Time `json:"issuedAt"` ExpiresAt time.Time `json:"expiresAt"` Forecaster string `json:"forecaster,omitempty"` Headline string `json:"headline,omitempty"` Summary string `json:"summary,omitempty"` Discussion string `json:"discussion,omitempty"` SourceURL string `json:"sourceUrl,omitempty"` ImageURL string `json:"imageUrl,omitempty"` ContainsLocation bool `json:"containsLocation"` Geometry json.RawMessage `json:"geometry"` } ``` Required canonical fields: - Run: `asOf`, `outlooks`. - Outlook: `id`, `provider`, `product`, `day`, `outlookType`, `label`, `validFrom`, `validTo`, `issuedAt`, `expiresAt`, `containsLocation`, `geometry`. Canonical values: - `provider` is `spc`. - `product` is `convective`. - `outlookType` is one of `categorical`, `tornado`, `hail`, `wind`. - `day` is one of `1`, `2`, `3`. ## Source Inputs Default required GeoJSON products: - `https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson` - `https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson` - `https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson` - `https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson` - `https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson` - `https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson` - `https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson` - `https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson` - `https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson` - `https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson` - `https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson` - `https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson` Default required print-page products: - `https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html` - `https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html` - `https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html` Optional RSS product: - `https://www.spc.noaa.gov/products/spcacrss.xml` Recommended config shape: ```yaml - name: SPCConvectiveOutlookSTL mode: poll kinds: ["outlook"] driver: spc_convective_outlook every: 30m params: latitude: 38.6239 longitude: -90.3571 location_id: "stl" location_name: "St. Louis, MO" user_agent: "HomeOps (eric@maximumdirect.net)" ``` Optional source params: - `geojson_urls`: map from product key to URL, used by tests and future upstream changes. - `discussion_urls`: map from day key to URL, used by tests and future upstream changes. - `rss_url`: string; when non-empty, fetch RSS as supplemental metadata. Product keys for `geojson_urls`: - `day1_categorical`, `day1_tornado`, `day1_hail`, `day1_wind` - `day2_categorical`, `day2_tornado`, `day2_hail`, `day2_wind` - `day3_categorical`, `day3_tornado`, `day3_hail`, `day3_wind` Discussion keys for `discussion_urls`: - `day1`, `day2`, `day3` ## Raw Bundle Shape Create a provider raw bundle type under `internal/providers/spc` or `internal/normalizers/spc` and use it consistently between source tests and normalizer tests. Prefer `internal/providers/spc` if source metadata extraction and normalizer parsing share helpers. Suggested raw payload shape: ```go type RawConvectiveOutlookBundle struct { LocationID string `json:"locationId,omitempty"` LocationName string `json:"locationName,omitempty"` Latitude float64 `json:"latitude"` Longitude float64 `json:"longitude"` FetchedAt time.Time `json:"fetchedAt"` Products []RawOutlookProduct `json:"products"` Discussions []RawDiscussionPage `json:"discussions"` RSS *RawRSSFeed `json:"rss,omitempty"` } ``` ```go type RawOutlookProduct struct { Key string `json:"key"` Day int `json:"day"` OutlookType string `json:"outlookType"` URL string `json:"url"` FetchedAt time.Time `json:"fetchedAt"` Body json.RawMessage `json:"body"` } ``` ```go type RawDiscussionPage struct { Key string `json:"key"` Day int `json:"day"` URL string `json:"url"` FetchedAt time.Time `json:"fetchedAt"` Body string `json:"body"` } ``` ```go type RawRSSFeed struct { URL string `json:"url"` FetchedAt time.Time `json:"fetchedAt"` Body string `json:"body"` } ``` Do not put parsed canonical fields into the raw bundle except configured metadata and product keys needed to identify each fetched upstream document. Source-level timestamp parsing is allowed only for event `effectiveAt` selection. ## Stage 1: Provider Helpers And Fixtures Goal: add deterministic SPC parsing primitives and test fixtures before wiring the source or normalizer. Files to add: - `internal/providers/spc/doc.go` - `internal/providers/spc/time.go` - `internal/providers/spc/product.go` - `internal/providers/spc/geojson.go` - `internal/providers/spc/discussion.go` - `internal/providers/spc/rss.go`, only if optional RSS parsing is implemented - `internal/providers/spc/testdata/day1_cat.geojson` - `internal/providers/spc/testdata/day2_torn.geojson` - `internal/providers/spc/testdata/day3_wind.geojson` - `internal/providers/spc/testdata/day1_prt.html` - `internal/providers/spc/testdata/day2_prt_corr.html` - `internal/providers/spc/testdata/day3_prt.html` Provider helper behavior: - Define stable product metadata for the 12 required GeoJSON products. - Define stable discussion metadata for the 3 required print-page products. - Parse SPC ISO timestamps from GeoJSON properties using `time.Parse(time.RFC3339, value)` after trimming whitespace. - Decode enough GeoJSON to expose feature properties and raw geometry without owning canonical mapping. - Preserve raw geometry as compact JSON bytes for later canonical use. - Extract print-page product text from the first useful `
` block. - Strip embedded `