diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index cb60fa4..1490b4e 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,697 +1,103 @@ -# SPC Convective Outlook Implementation Plan +# SPC Convective Outlook Remaining Work Plan -## Purpose +## Summary -Implement `weatherfeeder` support for Storm Prediction Center Day 1-3 convective outlooks described in [`docs/roadmap/spc.md`](spc.md). This plan is written for an LLM coding agent and should be followed stage by stage. +The SPC convective outlook feature is mostly implemented. This roadmap now tracks only the remaining remediation work needed to align the implementation with the intended contract and live SPC print-page behavior. -This is a planning document only. The implementation must preserve the existing weatherfeeder architecture: sources emit raw provider events, normalizers map raw payloads into canonical model types, and sinks persist canonical schemas. +Do not use this document as the original full feature implementation plan. It is a focused completion plan for the outstanding gaps below. -## Decisions +## Remaining Work -The following choices are fixed for this implementation: +### Postgres Persistence Gap -- Scope is Day 1-3 SPC convective outlooks only. -- Day 4-8 outlooks are out of scope. -- GeoJSON files are authoritative for polygons, validity windows, issue times, outlook labels, and severity ranking. -- Day 1-3 print pages are authoritative for discussion text. -- RSS is optional supplemental metadata only and must not be required for correctness. -- Do not fetch RSS by default. Include RSS only when an optional `rss_url` source param is configured. -- A poll is atomic for required products. If any configured GeoJSON or print-page URL fails or returns a non-2xx response, return an error and emit no event. -- Use compact GeoJSON geometry in the canonical payload for auditability and downstream display. -- Use standard-library-first parsing. Do not add an HTML parsing dependency unless string-based extraction proves unmaintainable during implementation. -- Keep all new planned behavior inside `weatherfeeder`; do not make `weatherapi` changes in this pass. +Canonical `model.WeatherOutlook` includes required `id` and `provider` fields, but the current Postgres `outlooks` table and mapper do not persist them. This makes Postgres persistence lossy for `weather.outlook.v1`. -## Public Contract +Required changes: -Add schema constants in `standards/schema.go`: +- Add `outlook_id TEXT NOT NULL` to the `outlooks` table contract. +- Add `provider TEXT NOT NULL` to the `outlooks` table contract. +- Map `model.WeatherOutlook.ID` to `outlook_id`. +- Map `model.WeatherOutlook.Provider` to `provider`. +- Keep existing required-field validation for empty `id` and `provider`. +- Update Postgres schema, mapper, schema tests, mapper tests, internal Postgres docs, and Postgres integration docs. +- Do not include manual database migration commands in this roadmap; keep this plan focused on schema/code/docs expectations. -- `SchemaRawSPCConvectiveOutlookV1 = "raw.spc.convective_outlook.v1"` -- `SchemaWeatherOutlookV1 = "weather.outlook.v1"` +### SPC Print-Page Parsing Gap -Add source driver: +The current parser fixtures do not match the live SPC print-page structure closely enough. Live SPC print pages currently place the page-level `Updated:` row outside the `
` product text, and the product text inside `` begins with an SPC product code line such as `SPC AC 101959` before the human title line. -- `spc_convective_outlook` +Required changes: -Add event kind: +- Update SPC print-page fixtures to match live shape: + - page-level `Updated:` appears outside the `` block + - `` content begins with an `SPC AC ...` product code line + - the human outlook title appears on the following line, such as `Day 1 Convective Outlook` +- Parse page-level `Updated:` timestamps from the full HTML document, not only from extracted product text. +- Keep cleaned product text extraction focused on the `` block. +- Preserve the leading `SPC AC ...` line in canonical `discussion` text. +- Update product-title/headline parsing to skip product code lines like `SPC AC 101959` and choose the human outlook title line. +- Preserve correction markers such as `CORR 1` in the selected headline/title when present. +- Continue stripping scripts and HTML tags from extracted product text. -- `outlook` +### Source Effective-Time Correction -Add canonical model types: +The source should continue to prefer GeoJSON issue timestamps, but the print-page fallback should use the page-level `Updated:` parser described above. -- `model.WeatherOutlookRun` -- `model.WeatherOutlook` +Effective-time order should remain: -Canonical run fields: +1. latest valid GeoJSON `ISSUE_ISO` +2. latest valid print-page `Updated:` timestamp parsed from full HTML +3. RSS `lastBuildDate`, when optional RSS is configured and parseable +4. fetch time -```go -type WeatherOutlookRun struct { - LocationID string `json:"locationId,omitempty"` - LocationName string `json:"locationName,omitempty"` - Latitude *float64 `json:"latitude,omitempty"` - Longitude *float64 `json:"longitude,omitempty"` - AsOf time.Time `json:"asOf"` - IssuedAt *time.Time `json:"issuedAt,omitempty"` - Outlooks []WeatherOutlook `json:"outlooks"` -} -``` +Required changes: -Canonical outlook fields: +- Update source discussion timestamp extraction to parse from full HTML. +- Keep RSS optional and supplemental only. +- Keep unchanged-response hashing, atomic required fetch behavior, and fetch-time fallback unchanged. -```go -type WeatherOutlook struct { - ID string `json:"id"` - Provider string `json:"provider"` - Product string `json:"product"` - Day int `json:"day"` - OutlookType string `json:"outlookType"` - Label string `json:"label"` - LabelText string `json:"labelText,omitempty"` - SeverityRank *int `json:"severityRank,omitempty"` - ValidFrom time.Time `json:"validFrom"` - ValidTo time.Time `json:"validTo"` - IssuedAt time.Time `json:"issuedAt"` - ExpiresAt time.Time `json:"expiresAt"` - Forecaster string `json:"forecaster,omitempty"` - Headline string `json:"headline,omitempty"` - Summary string `json:"summary,omitempty"` - Discussion string `json:"discussion,omitempty"` - SourceURL string `json:"sourceUrl,omitempty"` - ImageURL string `json:"imageUrl,omitempty"` - ContainsLocation bool `json:"containsLocation"` - Geometry json.RawMessage `json:"geometry"` -} -``` +## Test Plan -Required canonical fields: +Provider parser tests: -- Run: `asOf`, `outlooks`. -- Outlook: `id`, `provider`, `product`, `day`, `outlookType`, `label`, `validFrom`, `validTo`, `issuedAt`, `expiresAt`, `containsLocation`, `geometry`. +- Fixture covers live print-page shape with `Updated:` outside ``. +- Fixture product text begins with `SPC AC ...`. +- Headline resolves to `Day X Convective Outlook`, not `SPC AC ...`. +- Page-level `Updated:` parses to UTC. +- Full cleaned discussion text preserves the `SPC AC ...` line. +- Script and tag cleanup remains covered. +- `CORR 1` remains preserved in headline/title and discussion text. -Canonical values: +Source tests: -- `provider` is `spc`. -- `product` is `convective`. -- `outlookType` is one of `categorical`, `tornado`, `hail`, `wind`. -- `day` is one of `1`, `2`, `3`. +- `effectiveAt` falls back to page-level `Updated:` when GeoJSON issue times are unavailable. +- Existing atomic fetch, unchanged-response, required URL failure, and optional RSS tests remain passing. -## Source Inputs +Normalizer tests: -Default required GeoJSON products: +- Canonical `headline` uses the human outlook title. +- Canonical `discussion` preserves full cleaned product text. +- Existing `CORR 1`, summary, and day-to-discussion mapping tests remain passing. -- `https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson` -- `https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson` -- `https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson` -- `https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson` -- `https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson` -- `https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson` -- `https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson` -- `https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson` -- `https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson` -- `https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson` -- `https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson` -- `https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson` +Postgres tests: -Default required print-page products: +- Schema includes `outlook_id` and `provider` on `outlooks`. +- Mapper writes `outlook_id` from `WeatherOutlook.ID`. +- Mapper writes `provider` from `WeatherOutlook.Provider`. +- Required-field validation still rejects empty `id` and `provider`. +- Existing compact geometry and `contains_location=false` tests remain passing. -- `https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html` -- `https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html` -- `https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html` - -Optional RSS product: - -- `https://www.spc.noaa.gov/products/spcacrss.xml` - -Recommended config shape: - -```yaml -- name: SPCConvectiveOutlookSTL - mode: poll - kinds: ["outlook"] - driver: spc_convective_outlook - every: 30m - params: - latitude: 38.6239 - longitude: -90.3571 - location_id: "stl" - location_name: "St. Louis, MO" - user_agent: "HomeOps (eric@maximumdirect.net)" -``` - -Optional source params: - -- `geojson_urls`: map from product key to URL, used by tests and future upstream changes. -- `discussion_urls`: map from day key to URL, used by tests and future upstream changes. -- `rss_url`: string; when non-empty, fetch RSS as supplemental metadata. - -Product keys for `geojson_urls`: - -- `day1_categorical`, `day1_tornado`, `day1_hail`, `day1_wind` -- `day2_categorical`, `day2_tornado`, `day2_hail`, `day2_wind` -- `day3_categorical`, `day3_tornado`, `day3_hail`, `day3_wind` - -Discussion keys for `discussion_urls`: - -- `day1`, `day2`, `day3` - -## Raw Bundle Shape - -Create a provider raw bundle type under `internal/providers/spc` or `internal/normalizers/spc` and use it consistently between source tests and normalizer tests. Prefer `internal/providers/spc` if source metadata extraction and normalizer parsing share helpers. - -Suggested raw payload shape: - -```go -type RawConvectiveOutlookBundle struct { - LocationID string `json:"locationId,omitempty"` - LocationName string `json:"locationName,omitempty"` - Latitude float64 `json:"latitude"` - Longitude float64 `json:"longitude"` - FetchedAt time.Time `json:"fetchedAt"` - Products []RawOutlookProduct `json:"products"` - Discussions []RawDiscussionPage `json:"discussions"` - RSS *RawRSSFeed `json:"rss,omitempty"` -} -``` - -```go -type RawOutlookProduct struct { - Key string `json:"key"` - Day int `json:"day"` - OutlookType string `json:"outlookType"` - URL string `json:"url"` - FetchedAt time.Time `json:"fetchedAt"` - Body json.RawMessage `json:"body"` -} -``` - -```go -type RawDiscussionPage struct { - Key string `json:"key"` - Day int `json:"day"` - URL string `json:"url"` - FetchedAt time.Time `json:"fetchedAt"` - Body string `json:"body"` -} -``` - -```go -type RawRSSFeed struct { - URL string `json:"url"` - FetchedAt time.Time `json:"fetchedAt"` - Body string `json:"body"` -} -``` - -Do not put parsed canonical fields into the raw bundle except configured metadata and product keys needed to identify each fetched upstream document. Source-level timestamp parsing is allowed only for event `effectiveAt` selection. - -## Stage 1: Provider Helpers And Fixtures - -Goal: add deterministic SPC parsing primitives and test fixtures before wiring the source or normalizer. - -Files to add: - -- `internal/providers/spc/doc.go` -- `internal/providers/spc/time.go` -- `internal/providers/spc/product.go` -- `internal/providers/spc/geojson.go` -- `internal/providers/spc/discussion.go` -- `internal/providers/spc/rss.go`, only if optional RSS parsing is implemented -- `internal/providers/spc/testdata/day1_cat.geojson` -- `internal/providers/spc/testdata/day2_torn.geojson` -- `internal/providers/spc/testdata/day3_wind.geojson` -- `internal/providers/spc/testdata/day1_prt.html` -- `internal/providers/spc/testdata/day2_prt_corr.html` -- `internal/providers/spc/testdata/day3_prt.html` - -Provider helper behavior: - -- Define stable product metadata for the 12 required GeoJSON products. -- Define stable discussion metadata for the 3 required print-page products. -- Parse SPC ISO timestamps from GeoJSON properties using `time.Parse(time.RFC3339, value)` after trimming whitespace. -- Decode enough GeoJSON to expose feature properties and raw geometry without owning canonical mapping. -- Preserve raw geometry as compact JSON bytes for later canonical use. -- Extract print-page product text from the first useful `` block. -- Strip embedded `