From a4cd63ca4ebd3462869eec5353612ee8df38f5e1 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 10 Jun 2026 19:40:46 -0500 Subject: [PATCH] Update the SPC outlook implementation plan to identify remaining gaps and corrections --- docs/roadmap/implementation.md | 732 ++++----------------------------- 1 file changed, 69 insertions(+), 663 deletions(-) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index cb60fa4..1490b4e 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,697 +1,103 @@ -# SPC Convective Outlook Implementation Plan +# SPC Convective Outlook Remaining Work Plan -## Purpose +## Summary -Implement `weatherfeeder` support for Storm Prediction Center Day 1-3 convective outlooks described in [`docs/roadmap/spc.md`](spc.md). This plan is written for an LLM coding agent and should be followed stage by stage. +The SPC convective outlook feature is mostly implemented. This roadmap now tracks only the remaining remediation work needed to align the implementation with the intended contract and live SPC print-page behavior. -This is a planning document only. The implementation must preserve the existing weatherfeeder architecture: sources emit raw provider events, normalizers map raw payloads into canonical model types, and sinks persist canonical schemas. +Do not use this document as the original full feature implementation plan. It is a focused completion plan for the outstanding gaps below. -## Decisions +## Remaining Work -The following choices are fixed for this implementation: +### Postgres Persistence Gap -- Scope is Day 1-3 SPC convective outlooks only. -- Day 4-8 outlooks are out of scope. -- GeoJSON files are authoritative for polygons, validity windows, issue times, outlook labels, and severity ranking. -- Day 1-3 print pages are authoritative for discussion text. -- RSS is optional supplemental metadata only and must not be required for correctness. -- Do not fetch RSS by default. Include RSS only when an optional `rss_url` source param is configured. -- A poll is atomic for required products. If any configured GeoJSON or print-page URL fails or returns a non-2xx response, return an error and emit no event. -- Use compact GeoJSON geometry in the canonical payload for auditability and downstream display. -- Use standard-library-first parsing. Do not add an HTML parsing dependency unless string-based extraction proves unmaintainable during implementation. -- Keep all new planned behavior inside `weatherfeeder`; do not make `weatherapi` changes in this pass. +Canonical `model.WeatherOutlook` includes required `id` and `provider` fields, but the current Postgres `outlooks` table and mapper do not persist them. This makes Postgres persistence lossy for `weather.outlook.v1`. -## Public Contract +Required changes: -Add schema constants in `standards/schema.go`: +- Add `outlook_id TEXT NOT NULL` to the `outlooks` table contract. +- Add `provider TEXT NOT NULL` to the `outlooks` table contract. +- Map `model.WeatherOutlook.ID` to `outlook_id`. +- Map `model.WeatherOutlook.Provider` to `provider`. +- Keep existing required-field validation for empty `id` and `provider`. +- Update Postgres schema, mapper, schema tests, mapper tests, internal Postgres docs, and Postgres integration docs. +- Do not include manual database migration commands in this roadmap; keep this plan focused on schema/code/docs expectations. -- `SchemaRawSPCConvectiveOutlookV1 = "raw.spc.convective_outlook.v1"` -- `SchemaWeatherOutlookV1 = "weather.outlook.v1"` +### SPC Print-Page Parsing Gap -Add source driver: +The current parser fixtures do not match the live SPC print-page structure closely enough. Live SPC print pages currently place the page-level `Updated:` row outside the `
` product text, and the product text inside `
` begins with an SPC product code line such as `SPC AC 101959` before the human title line.
 
-- `spc_convective_outlook`
+Required changes:
 
-Add event kind:
+- Update SPC print-page fixtures to match live shape:
+  - page-level `Updated:` appears outside the `
` block
+  - `
` content begins with an `SPC AC ...` product code line
+  - the human outlook title appears on the following line, such as `Day 1 Convective Outlook`
+- Parse page-level `Updated:` timestamps from the full HTML document, not only from extracted product text.
+- Keep cleaned product text extraction focused on the `
` block.
+- Preserve the leading `SPC AC ...` line in canonical `discussion` text.
+- Update product-title/headline parsing to skip product code lines like `SPC AC 101959` and choose the human outlook title line.
+- Preserve correction markers such as `CORR 1` in the selected headline/title when present.
+- Continue stripping scripts and HTML tags from extracted product text.
 
-- `outlook`
+### Source Effective-Time Correction
 
-Add canonical model types:
+The source should continue to prefer GeoJSON issue timestamps, but the print-page fallback should use the page-level `Updated:` parser described above.
 
-- `model.WeatherOutlookRun`
-- `model.WeatherOutlook`
+Effective-time order should remain:
 
-Canonical run fields:
+1. latest valid GeoJSON `ISSUE_ISO`
+2. latest valid print-page `Updated:` timestamp parsed from full HTML
+3. RSS `lastBuildDate`, when optional RSS is configured and parseable
+4. fetch time
 
-```go
-type WeatherOutlookRun struct {
-    LocationID   string           `json:"locationId,omitempty"`
-    LocationName string           `json:"locationName,omitempty"`
-    Latitude     *float64         `json:"latitude,omitempty"`
-    Longitude    *float64         `json:"longitude,omitempty"`
-    AsOf         time.Time        `json:"asOf"`
-    IssuedAt     *time.Time       `json:"issuedAt,omitempty"`
-    Outlooks     []WeatherOutlook `json:"outlooks"`
-}
-```
+Required changes:
 
-Canonical outlook fields:
+- Update source discussion timestamp extraction to parse from full HTML.
+- Keep RSS optional and supplemental only.
+- Keep unchanged-response hashing, atomic required fetch behavior, and fetch-time fallback unchanged.
 
-```go
-type WeatherOutlook struct {
-    ID               string          `json:"id"`
-    Provider         string          `json:"provider"`
-    Product          string          `json:"product"`
-    Day              int             `json:"day"`
-    OutlookType      string          `json:"outlookType"`
-    Label            string          `json:"label"`
-    LabelText        string          `json:"labelText,omitempty"`
-    SeverityRank     *int            `json:"severityRank,omitempty"`
-    ValidFrom        time.Time       `json:"validFrom"`
-    ValidTo          time.Time       `json:"validTo"`
-    IssuedAt         time.Time       `json:"issuedAt"`
-    ExpiresAt        time.Time       `json:"expiresAt"`
-    Forecaster       string          `json:"forecaster,omitempty"`
-    Headline         string          `json:"headline,omitempty"`
-    Summary          string          `json:"summary,omitempty"`
-    Discussion       string          `json:"discussion,omitempty"`
-    SourceURL        string          `json:"sourceUrl,omitempty"`
-    ImageURL         string          `json:"imageUrl,omitempty"`
-    ContainsLocation bool            `json:"containsLocation"`
-    Geometry         json.RawMessage `json:"geometry"`
-}
-```
+## Test Plan
 
-Required canonical fields:
+Provider parser tests:
 
-- Run: `asOf`, `outlooks`.
-- Outlook: `id`, `provider`, `product`, `day`, `outlookType`, `label`, `validFrom`, `validTo`, `issuedAt`, `expiresAt`, `containsLocation`, `geometry`.
+- Fixture covers live print-page shape with `Updated:` outside `
`.
+- Fixture product text begins with `SPC AC ...`.
+- Headline resolves to `Day X Convective Outlook`, not `SPC AC ...`.
+- Page-level `Updated:` parses to UTC.
+- Full cleaned discussion text preserves the `SPC AC ...` line.
+- Script and tag cleanup remains covered.
+- `CORR 1` remains preserved in headline/title and discussion text.
 
-Canonical values:
+Source tests:
 
-- `provider` is `spc`.
-- `product` is `convective`.
-- `outlookType` is one of `categorical`, `tornado`, `hail`, `wind`.
-- `day` is one of `1`, `2`, `3`.
+- `effectiveAt` falls back to page-level `Updated:` when GeoJSON issue times are unavailable.
+- Existing atomic fetch, unchanged-response, required URL failure, and optional RSS tests remain passing.
 
-## Source Inputs
+Normalizer tests:
 
-Default required GeoJSON products:
+- Canonical `headline` uses the human outlook title.
+- Canonical `discussion` preserves full cleaned product text.
+- Existing `CORR 1`, summary, and day-to-discussion mapping tests remain passing.
 
-- `https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson`
+Postgres tests:
 
-Default required print-page products:
+- Schema includes `outlook_id` and `provider` on `outlooks`.
+- Mapper writes `outlook_id` from `WeatherOutlook.ID`.
+- Mapper writes `provider` from `WeatherOutlook.Provider`.
+- Required-field validation still rejects empty `id` and `provider`.
+- Existing compact geometry and `contains_location=false` tests remain passing.
 
-- `https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html`
-- `https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html`
-- `https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html`
-
-Optional RSS product:
-
-- `https://www.spc.noaa.gov/products/spcacrss.xml`
-
-Recommended config shape:
-
-```yaml
-- name: SPCConvectiveOutlookSTL
-  mode: poll
-  kinds: ["outlook"]
-  driver: spc_convective_outlook
-  every: 30m
-  params:
-    latitude: 38.6239
-    longitude: -90.3571
-    location_id: "stl"
-    location_name: "St. Louis, MO"
-    user_agent: "HomeOps (eric@maximumdirect.net)"
-```
-
-Optional source params:
-
-- `geojson_urls`: map from product key to URL, used by tests and future upstream changes.
-- `discussion_urls`: map from day key to URL, used by tests and future upstream changes.
-- `rss_url`: string; when non-empty, fetch RSS as supplemental metadata.
-
-Product keys for `geojson_urls`:
-
-- `day1_categorical`, `day1_tornado`, `day1_hail`, `day1_wind`
-- `day2_categorical`, `day2_tornado`, `day2_hail`, `day2_wind`
-- `day3_categorical`, `day3_tornado`, `day3_hail`, `day3_wind`
-
-Discussion keys for `discussion_urls`:
-
-- `day1`, `day2`, `day3`
-
-## Raw Bundle Shape
-
-Create a provider raw bundle type under `internal/providers/spc` or `internal/normalizers/spc` and use it consistently between source tests and normalizer tests. Prefer `internal/providers/spc` if source metadata extraction and normalizer parsing share helpers.
-
-Suggested raw payload shape:
-
-```go
-type RawConvectiveOutlookBundle struct {
-    LocationID   string              `json:"locationId,omitempty"`
-    LocationName string              `json:"locationName,omitempty"`
-    Latitude     float64             `json:"latitude"`
-    Longitude    float64             `json:"longitude"`
-    FetchedAt    time.Time           `json:"fetchedAt"`
-    Products     []RawOutlookProduct `json:"products"`
-    Discussions  []RawDiscussionPage `json:"discussions"`
-    RSS          *RawRSSFeed         `json:"rss,omitempty"`
-}
-```
-
-```go
-type RawOutlookProduct struct {
-    Key         string          `json:"key"`
-    Day         int             `json:"day"`
-    OutlookType string          `json:"outlookType"`
-    URL         string          `json:"url"`
-    FetchedAt   time.Time       `json:"fetchedAt"`
-    Body        json.RawMessage `json:"body"`
-}
-```
-
-```go
-type RawDiscussionPage struct {
-    Key       string    `json:"key"`
-    Day       int       `json:"day"`
-    URL       string    `json:"url"`
-    FetchedAt time.Time `json:"fetchedAt"`
-    Body      string    `json:"body"`
-}
-```
-
-```go
-type RawRSSFeed struct {
-    URL       string    `json:"url"`
-    FetchedAt time.Time `json:"fetchedAt"`
-    Body      string    `json:"body"`
-}
-```
-
-Do not put parsed canonical fields into the raw bundle except configured metadata and product keys needed to identify each fetched upstream document. Source-level timestamp parsing is allowed only for event `effectiveAt` selection.
-
-## Stage 1: Provider Helpers And Fixtures
-
-Goal: add deterministic SPC parsing primitives and test fixtures before wiring the source or normalizer.
-
-Files to add:
-
-- `internal/providers/spc/doc.go`
-- `internal/providers/spc/time.go`
-- `internal/providers/spc/product.go`
-- `internal/providers/spc/geojson.go`
-- `internal/providers/spc/discussion.go`
-- `internal/providers/spc/rss.go`, only if optional RSS parsing is implemented
-- `internal/providers/spc/testdata/day1_cat.geojson`
-- `internal/providers/spc/testdata/day2_torn.geojson`
-- `internal/providers/spc/testdata/day3_wind.geojson`
-- `internal/providers/spc/testdata/day1_prt.html`
-- `internal/providers/spc/testdata/day2_prt_corr.html`
-- `internal/providers/spc/testdata/day3_prt.html`
-
-Provider helper behavior:
-
-- Define stable product metadata for the 12 required GeoJSON products.
-- Define stable discussion metadata for the 3 required print-page products.
-- Parse SPC ISO timestamps from GeoJSON properties using `time.Parse(time.RFC3339, value)` after trimming whitespace.
-- Decode enough GeoJSON to expose feature properties and raw geometry without owning canonical mapping.
-- Preserve raw geometry as compact JSON bytes for later canonical use.
-- Extract print-page product text from the first useful `
` block.
-- Strip embedded `