diff --git a/docs/integrations/postgres.md b/docs/integrations/postgres.md
index 767510c..0a1d894 100644
--- a/docs/integrations/postgres.md
+++ b/docs/integrations/postgres.md
@@ -429,6 +429,8 @@ Indexes:
| `run_event_id` | `TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `outlook_index` | `INTEGER` | no | `payload.outlooks[]` index. |
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
+| `outlook_id` | `TEXT` | no | `payload.outlooks[].id` |
+| `provider` | `TEXT` | no | `payload.outlooks[].provider` |
| `product` | `TEXT` | no | `payload.outlooks[].product` |
| `day` | `INTEGER` | no | `payload.outlooks[].day` |
| `outlook_type` | `TEXT` | no | `payload.outlooks[].outlookType` |
diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md
new file mode 100644
index 0000000..d5426a1
--- /dev/null
+++ b/docs/roadmap/future.md
@@ -0,0 +1,60 @@
+# Future Work
+
+## Purpose
+
+This document is the catch-all roadmap for planned, deferred, aspirational, experimental, or unimplemented weatherfeeder work. Current behavior belongs in the canonical docs outside `docs/roadmap/`.
+
+## SPC Convective Outlook Follow-Ups
+
+### Weatherapi Outlook Endpoints
+
+Expose persisted SPC convective outlooks through `weatherapi` after the weatherfeeder storage contract is stable.
+
+Likely endpoints:
+
+- `GET /outlooks/convective`
+- `GET /outlooks/convective/active`
+- `GET /outlooks/convective/location`
+
+Recommended behavior:
+
+- Return the latest outlook run by default.
+- Support active outlook filtering by current time and `containsLocation=true`.
+- Consider optional query filters for `day`, `outlookType`, and `containsLocation`.
+- Preserve canonical outlook geometry for downstream display and audit use.
+
+### SPC Day 4-8 Outlooks
+
+Add SPC Day 4-8 convective outlook support as a schema-compatible extension only after Day 1-3 operation is proven.
+
+Notes:
+
+- Day 4-8 products have different semantics from Day 1-3 categorical/tornado/hail/wind products.
+- Avoid forcing Day 4-8 assumptions into the current Day 1-3 model until the source shapes and consumer needs are reviewed.
+- Prefer reusing `weather.outlook.v1` if the fields remain accurate; otherwise write a separate roadmap before changing the canonical contract.
+
+### Degraded SPC Bundle Mode
+
+Evaluate whether the SPC source should support degraded partial bundles when one required upstream product fails.
+
+Current behavior should remain atomic: if a required GeoJSON or print-page fetch fails, emit no event for that poll.
+
+Future degraded mode would need a clear contract for:
+
+- distinguishing "no risk polygon" from "product missing";
+- exposing per-product fetch errors without leaking raw provider internals into canonical events;
+- deciding whether downstream sinks and APIs should store or serve partial snapshots.
+
+### Richer SPC Page Assets And Tables
+
+Evaluate whether to parse additional SPC print-page metadata beyond the current discussion text.
+
+Possible additions:
+
+- archive GeoJSON/shapefile/KML links;
+- image URLs;
+- page risk tables;
+- city tables;
+- richer discussion section metadata.
+
+Keep GeoJSON products authoritative for polygons, validity windows, and point matching unless a future roadmap explicitly changes that contract.
diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md
deleted file mode 100644
index 1490b4e..0000000
--- a/docs/roadmap/implementation.md
+++ /dev/null
@@ -1,103 +0,0 @@
-# SPC Convective Outlook Remaining Work Plan
-
-## Summary
-
-The SPC convective outlook feature is mostly implemented. This roadmap now tracks only the remaining remediation work needed to align the implementation with the intended contract and live SPC print-page behavior.
-
-Do not use this document as the original full feature implementation plan. It is a focused completion plan for the outstanding gaps below.
-
-## Remaining Work
-
-### Postgres Persistence Gap
-
-Canonical `model.WeatherOutlook` includes required `id` and `provider` fields, but the current Postgres `outlooks` table and mapper do not persist them. This makes Postgres persistence lossy for `weather.outlook.v1`.
-
-Required changes:
-
-- Add `outlook_id TEXT NOT NULL` to the `outlooks` table contract.
-- Add `provider TEXT NOT NULL` to the `outlooks` table contract.
-- Map `model.WeatherOutlook.ID` to `outlook_id`.
-- Map `model.WeatherOutlook.Provider` to `provider`.
-- Keep existing required-field validation for empty `id` and `provider`.
-- Update Postgres schema, mapper, schema tests, mapper tests, internal Postgres docs, and Postgres integration docs.
-- Do not include manual database migration commands in this roadmap; keep this plan focused on schema/code/docs expectations.
-
-### SPC Print-Page Parsing Gap
-
-The current parser fixtures do not match the live SPC print-page structure closely enough. Live SPC print pages currently place the page-level `Updated:` row outside the `` product text, and the product text inside `` begins with an SPC product code line such as `SPC AC 101959` before the human title line.
-
-Required changes:
-
-- Update SPC print-page fixtures to match live shape:
- - page-level `Updated:` appears outside the `` block
- - `` content begins with an `SPC AC ...` product code line
- - the human outlook title appears on the following line, such as `Day 1 Convective Outlook`
-- Parse page-level `Updated:` timestamps from the full HTML document, not only from extracted product text.
-- Keep cleaned product text extraction focused on the `` block.
-- Preserve the leading `SPC AC ...` line in canonical `discussion` text.
-- Update product-title/headline parsing to skip product code lines like `SPC AC 101959` and choose the human outlook title line.
-- Preserve correction markers such as `CORR 1` in the selected headline/title when present.
-- Continue stripping scripts and HTML tags from extracted product text.
-
-### Source Effective-Time Correction
-
-The source should continue to prefer GeoJSON issue timestamps, but the print-page fallback should use the page-level `Updated:` parser described above.
-
-Effective-time order should remain:
-
-1. latest valid GeoJSON `ISSUE_ISO`
-2. latest valid print-page `Updated:` timestamp parsed from full HTML
-3. RSS `lastBuildDate`, when optional RSS is configured and parseable
-4. fetch time
-
-Required changes:
-
-- Update source discussion timestamp extraction to parse from full HTML.
-- Keep RSS optional and supplemental only.
-- Keep unchanged-response hashing, atomic required fetch behavior, and fetch-time fallback unchanged.
-
-## Test Plan
-
-Provider parser tests:
-
-- Fixture covers live print-page shape with `Updated:` outside ``.
-- Fixture product text begins with `SPC AC ...`.
-- Headline resolves to `Day X Convective Outlook`, not `SPC AC ...`.
-- Page-level `Updated:` parses to UTC.
-- Full cleaned discussion text preserves the `SPC AC ...` line.
-- Script and tag cleanup remains covered.
-- `CORR 1` remains preserved in headline/title and discussion text.
-
-Source tests:
-
-- `effectiveAt` falls back to page-level `Updated:` when GeoJSON issue times are unavailable.
-- Existing atomic fetch, unchanged-response, required URL failure, and optional RSS tests remain passing.
-
-Normalizer tests:
-
-- Canonical `headline` uses the human outlook title.
-- Canonical `discussion` preserves full cleaned product text.
-- Existing `CORR 1`, summary, and day-to-discussion mapping tests remain passing.
-
-Postgres tests:
-
-- Schema includes `outlook_id` and `provider` on `outlooks`.
-- Mapper writes `outlook_id` from `WeatherOutlook.ID`.
-- Mapper writes `provider` from `WeatherOutlook.Provider`.
-- Required-field validation still rejects empty `id` and `provider`.
-- Existing compact geometry and `contains_location=false` tests remain passing.
-
-Verification commands:
-
-```sh
-go test ./internal/providers/spc ./internal/sources/spc ./internal/normalizers/spc ./internal/sinks/postgres
-go test ./...
-```
-
-## Assumptions
-
-- The canonical API shape remains unchanged: `WeatherOutlook.id` and `WeatherOutlook.provider` are already required fields.
-- Postgres persistence should be lossless for required canonical outlook fields.
-- Existing database migration commands are intentionally omitted from this roadmap.
-- RSS remains optional supplemental metadata only.
-- This remediation plan does not include weatherapi changes.
diff --git a/docs/roadmap/spc.md b/docs/roadmap/spc.md
deleted file mode 100644
index a288c5b..0000000
--- a/docs/roadmap/spc.md
+++ /dev/null
@@ -1,349 +0,0 @@
-# SPC Convective Outlook Support Roadmap
-
-## Summary
-
-Add `weatherfeeder` support for Storm Prediction Center convective outlooks as a new canonical outlook domain. The feature should poll SPC GeoJSON outlook products, enrich them with Day 1-3 print-page discussion text, compute whether the configured forecast point is inside each active outlook polygon, normalize the result into a provider-independent schema, and persist it through existing sinks.
-
-This is a new domain, not an extension of `weather.alert.v1`. SPC outlooks describe probabilistic/categorical risk areas over a validity window; alerts describe active actionable hazard messages.
-
-## Public Contract
-
-Add schema constants:
-
-- Raw schema: `raw.spc.convective_outlook.v1`
-- Canonical schema: `weather.outlook.v1`
-
-Add source driver:
-
-- `spc_convective_outlook`
-
-Add event kind:
-
-- `outlook`
-
-Add canonical model types:
-
-- `model.WeatherOutlookRun`
-- `model.WeatherOutlook`
-
-Recommended canonical run fields:
-
-- `locationId`, `locationName`
-- `latitude`, `longitude`
-- `asOf`
-- `issuedAt`
-- `outlooks`
-
-Recommended canonical outlook fields:
-
-- `id`
-- `provider`
-- `product`
-- `day`
-- `outlookType`
-- `label`
-- `labelText`
-- `severityRank`
-- `validFrom`
-- `validTo`
-- `issuedAt`
-- `expiresAt`
-- `forecaster`
-- `headline`
-- `summary`
-- `discussion`
-- `sourceUrl`
-- `imageUrl`
-- `containsLocation`
-- `geometry`
-
-Contract defaults:
-
-- `product` should be `convective`.
-- `outlookType` should be one of `categorical`, `tornado`, `hail`, `wind`.
-- `day` should be `1`, `2`, or `3` for this first implementation.
-- `containsLocation` is computed against configured forecast coordinates.
-- `geometry` should preserve compact GeoJSON geometry for auditability and future API use.
-- Day 4-8 outlooks are out of scope for this first implementation. They have different product semantics and should be added in a later schema-compatible extension only after Day 1-3 support is proven.
-
-## Source Scope
-
-The source should fetch a bundle of SPC products in one poll cycle and emit one raw event containing fetched GeoJSON products, fetched Day 1-3 print pages, optional RSS metadata, configured point metadata, and per-product fetch metadata. The source may decode minimal timestamps for event identity/effective time, but provider-to-canonical mapping belongs in the normalizer.
-
-Poll these GeoJSON URLs:
-
-- `https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson`
-- `https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson`
-
-Implementation note: the initial candidate list duplicated Day 3 wind and omitted Day 2 wind. Use the corrected 12-product matrix above.
-
-Poll these print-page discussion URLs:
-
-- `https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html`
-- `https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html`
-- `https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html`
-
-Print-page usage:
-
-- Use print pages as the authoritative source for Day 1-3 outlook discussion text.
-- Preserve source URL, page title/header where available, page-level `Updated:` timestamp where parseable, and the raw HTML or extracted `` product block in the raw bundle.
-- Extract canonical `headline`, `summary`, `discussion`, and discussion metadata in the normalizer or shared SPC provider helper.
-- Treat the GeoJSON files, not print pages, as authoritative for polygons and validity windows.
-- Ignore page risk tables and image assets for v1 unless they are needed to support discussion parsing.
-
-Optionally poll:
-
-- `https://www.spc.noaa.gov/products/spcacrss.xml`
-
-RSS usage:
-
-- Use RSS as supplemental discovery or health metadata, not as the primary discussion or geometry source.
-- Preserve item title, link, description text, pubDate, and guid where useful.
-- Do not depend on RSS completeness for Day 1-3 discussion text. The RSS feed can omit current outlook products, including a current Day 2 discussion.
-- Do not attempt to parse polygons from RSS HTML.
-
-Polling cadence:
-
-- Use a default cadence of `30m`, not daily/twice-daily. SPC current outlook files can update several times per day, and a 30-minute poll with HTTP caching is low cost and less likely to miss updates.
-- Rely on ETag/Last-Modified handling from the HTTP source layer where available.
-- Keep the cadence configurable via `every`.
-
-Recommended config shape:
-
-```yaml
-- name: SPCConvectiveOutlookSTL
- mode: poll
- kinds: ["outlook"]
- driver: spc_convective_outlook
- every: 30m
- params:
- latitude: 38.6239
- longitude: -90.3571
- location_id: "stl"
- location_name: "St. Louis, MO"
- user_agent: "HomeOps (eric@maximumdirect.net)"
-```
-
-The source should own default SPC URLs, with optional params to override GeoJSON URLs, print-page URLs, and the RSS URL for tests or future provider changes.
-
-## Implementation Stages
-
-### Stage 1: Raw Source and Schema
-
-- Add schema constants.
-- Add `internal/sources/spc` package.
-- Implement `spc_convective_outlook` as a poll source.
-- Fetch all configured GeoJSON products and Day 1-3 print pages during a poll cycle.
-- Fetch RSS only when configured or retained as an optional supplemental source.
-- Emit one raw event with kind `outlook` and schema `raw.spc.convective_outlook.v1`.
-- Use the latest valid `ISSUE_ISO`, print-page `Updated:` timestamp, RSS `lastBuildDate`, or fetch time for `effectiveAt`, in that order.
-- Preserve partial fetch errors as source errors for the first implementation; do not emit incomplete outlook bundles unless a later explicit degraded-mode policy is added.
-- Keep source tests deterministic with local test servers and fixtures; do not call live SPC services from unit tests.
-
-Tests:
-
-- source driver builds as a `PollSource`
-- source advertises kind `outlook`
-- source emits one raw bundled event
-- source chooses expected effective time
-- source uses configured location metadata
-- source fails clearly on missing latitude/longitude
-- source fails clearly when a configured GeoJSON or print-page fetch fails
-
-### Stage 2: GeoJSON and Spatial Matching
-
-- Add an internal geometry helper package, for example `internal/geo`.
-- Support GeoJSON `Polygon` and `MultiPolygon`.
-- Implement point-in-polygon with ring support:
- - first ring is the exterior
- - subsequent rings are holes
- - boundary counts as inside
- - GeoJSON coordinate order is `[longitude, latitude]`
-- Use planar ray casting for this operational point-in-polygon check.
-
-Tests:
-
-- point inside polygon
-- point outside polygon
-- point on polygon boundary
-- point inside a hole is outside
-- point inside one multipolygon member is inside
-- longitude/latitude order regression test
-
-### Stage 3: Normalizer and Canonical Mapping
-
-- Add `model.WeatherOutlookRun` and `model.WeatherOutlook`.
-- Add `SPCConvectiveOutlookNormalizer`.
-- Match only `raw.spc.convective_outlook.v1`.
-- Decode the raw source bundle.
-- Map each GeoJSON feature to one `WeatherOutlook`.
-- Preserve SPC feature order within each product, then order products by day and type.
-- Map properties:
- - `VALID_ISO` -> `validFrom`
- - `EXPIRE_ISO` -> `validTo` and `expiresAt`
- - `ISSUE_ISO` -> `issuedAt`
- - `FORECASTER` -> `forecaster`
- - `LABEL` -> `label`
- - `LABEL2` -> `labelText`
- - `DN` -> `severityRank`
-- Derive:
- - `day` from product key or URL
- - `outlookType` from product key or URL
- - `id` from day, type, label, issuedAt, validFrom, and feature index
- - `containsLocation` from configured point and GeoJSON geometry
-- Enrich narrative fields from the matching Day 1-3 print page:
- - parse the main product text from the `` block
- - strip embedded tags and scripts before text parsing
- - preserve line breaks in discussion text
- - parse page/title metadata, product title, issued line, valid line, correction markers such as `CORR 1`, forecaster, summary, and full discussion where practical
- - use print-page text for canonical `headline`, `summary`, and `discussion`
-- Use RSS only as optional supplemental metadata when present; missing or incomplete RSS must not prevent GeoJSON plus print-page normalization.
-- Set run `asOf` to the latest valid `issuedAt` across features, falling back to print-page `Updated:` timestamps, RSS `lastBuildDate`, then input event time.
-- Set run `issuedAt` to the latest valid `issuedAt` across features.
-- Set output event `effectiveAt` to run `asOf`.
-
-Tests:
-
-- normalizer routes only raw SPC schema
-- categorical and probabilistic products map expected fields
-- `containsLocation` is true for a known point inside a fixture polygon
-- `containsLocation` is false outside
-- print-page discussion text maps to canonical headline, summary, and discussion
-- correction markers such as `CORR 1` are preserved where present
-- missing optional RSS still permits GeoJSON and print-page normalization
-- malformed required GeoJSON timestamps fail with useful context
-- canonical JSON shape does not expose raw-provider-only bundle internals
-
-### Stage 4: Postgres Sink
-
-Add tables:
-
-- `outlook_runs`
-- `outlooks`
-
-Suggested `outlook_runs` columns:
-
-- event envelope columns
-- `location_id`
-- `location_name`
-- `latitude`
-- `longitude`
-- `as_of`
-- `issued_at`
-- `outlook_count`
-
-Suggested `outlooks` columns:
-
-- `run_event_id`
-- `outlook_index`
-- `as_of`
-- `product`
-- `day`
-- `outlook_type`
-- `label`
-- `label_text`
-- `severity_rank`
-- `valid_from`
-- `valid_to`
-- `issued_at`
-- `expires_at`
-- `forecaster`
-- `headline`
-- `summary`
-- `discussion`
-- `source_url`
-- `image_url`
-- `contains_location`
-- `geometry_json`
-
-Suggested indexes:
-
-- `outlook_runs(location_id, as_of)`
-- `outlooks(contains_location, valid_from, valid_to)`
-- `outlooks(day, outlook_type, label)`
-- `outlooks(valid_from, valid_to)`
-
-Mapping rules:
-
-- Store `geometry` as compact GeoJSON text in `geometry_json`.
-- Require run `asOf`.
-- Require outlook `validFrom`, `validTo`, `issuedAt`, `day`, `outlookType`, and `label`.
-- Preserve all outlook polygons, not only polygons containing the configured point.
-
-Tests:
-
-- schema includes outlook tables and indexes
-- mapper writes one run row plus one row per outlook
-- mapper stores compact geometry JSON
-- mapper rejects missing required run/outlook fields
-
-### Stage 5: Config and Documentation
-
-- Update sample config with `SPCConvectiveOutlookSTL`.
-- Add `outlook` to route examples.
-- Update current-behavior docs in the implementation change, not before behavior exists.
-- Update `docs/config.md` with source params.
-- Update `docs/integrations/events.md` with `weather.outlook.v1`.
-- Update `docs/integrations/postgres.md` with outlook table contract.
-- Update relevant internal docs under `docs/internal/` if source, normalizer, or sink behavior needs maintainer context.
-- Update `README.md` only with brief implemented-provider capability and links to canonical docs.
-
-Docs should clearly state:
-
-- Print pages are used for Day 1-3 narrative/discussion text.
-- RSS, if retained, is supplemental and not required for complete discussion coverage.
-- GeoJSON is used for polygons, validity windows, and point matching.
-- `containsLocation` is computed by weatherfeeder at ingestion time.
-- Geometry is stored for downstream audit/display.
-- Day 4-8 outlooks are not included in the initial implementation.
-
-### Stage 6: Weatherapi Follow-Up
-
-Do not include weatherapi changes in the first weatherfeeder implementation unless explicitly requested.
-
-Likely future weatherapi endpoints:
-
-- `GET /outlooks/convective`
-- `GET /outlooks/convective/active`
-- `GET /outlooks/convective/location`
-
-Recommended API behavior:
-
-- latest run by default
-- active outlooks filtered by current time and `containsLocation=true`
-- optional filter query params for `day`, `outlookType`, and `containsLocation`
-
-## Verification Commands
-
-Run focused tests:
-
-```sh
-go test ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./model
-```
-
-Run full weatherfeeder tests:
-
-```sh
-go test ./...
-```
-
-## Acceptance Criteria
-
-- A configured SPC source emits raw outlook bundles.
-- Normalization produces `weather.outlook.v1` events.
-- Each outlook indicates whether the configured forecast point is inside its polygon.
-- All current Day 1-3 categorical/tornado/hail/wind products are represented.
-- Day 1-3 print-page discussion text is preserved where available.
-- RSS metadata is optional and supplemental if retained.
-- Postgres sink persists outlook runs and outlook rows.
-- Sample config and public docs describe the new kind, driver, schema, and storage contract.
diff --git a/internal/normalizers/spc/convective_outlook.go b/internal/normalizers/spc/convective_outlook.go
index de5481c..d6681c3 100644
--- a/internal/normalizers/spc/convective_outlook.go
+++ b/internal/normalizers/spc/convective_outlook.go
@@ -138,11 +138,10 @@ func parseDiscussions(pages []spcprovider.RawDiscussionPage) (map[int]parsedDisc
out := map[int]parsedDiscussion{}
var latestUpdated time.Time
for _, page := range pages {
- text, err := spcprovider.ExtractProductText(page.Body)
+ parsed, err := spcprovider.ParseDiscussionHTML(page.Body)
if err != nil {
return nil, time.Time{}, fmt.Errorf("discussion %s: %w", page.Key, err)
}
- parsed := spcprovider.ParseDiscussionText(text)
day := page.Day
if day == 0 {
if meta, ok := spcprovider.DiscussionProductByKey(page.Key); ok {
diff --git a/internal/normalizers/spc/convective_outlook_test.go b/internal/normalizers/spc/convective_outlook_test.go
index 0ba86e7..aa59322 100644
--- a/internal/normalizers/spc/convective_outlook_test.go
+++ b/internal/normalizers/spc/convective_outlook_test.go
@@ -99,6 +99,9 @@ func TestConvectiveOutlookNormalizerProducesCanonicalSchemaAndMapsSample(t *test
if !strings.Contains(got.Discussion, "...DISCUSSION...") {
t.Fatalf("Discussion missing product text: %q", got.Discussion)
}
+ if !strings.HasPrefix(got.Discussion, "SPC AC 111234") {
+ t.Fatalf("Discussion = %q, want SPC product code prefix", got.Discussion)
+ }
if got.ID != "spc-convective-day1-categorical-slgt-2026-06-11T12:34:56Z-2026-06-11T13:00:00Z-0" {
t.Fatalf("ID = %q", got.ID)
}
diff --git a/internal/providers/spc/discussion.go b/internal/providers/spc/discussion.go
index ae79230..9227c7b 100644
--- a/internal/providers/spc/discussion.go
+++ b/internal/providers/spc/discussion.go
@@ -13,6 +13,8 @@ var (
preBlockRE = regexp.MustCompile(`(?is)]*>(.*?)`)
tagRE = regexp.MustCompile(`(?is)<[^>]+>`)
updatedRE = regexp.MustCompile(`(?im)^\s*Updated:\s*(.+?)\s*$`)
+ pageUpdatedRE = regexp.MustCompile(`(?i)\bUpdated:\s*((?:\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z)|(?:[A-Z][a-z]{2}\s+[A-Z][a-z]{2}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}\s+UTC\s+\d{4})|(?:\d{4}\s+UTC\s+[A-Z][a-z]{2}\s+[A-Z][a-z]{2}\s+\d{1,2}\s+\d{4})|(?:\d{4}Z\s+[A-Z][a-z]{2}\s+[A-Z][a-z]{2}\s+\d{1,2}\s+\d{4}))`)
+ productCodeRE = regexp.MustCompile(`(?i)^SPC\s+AC\s+\d+\s*$`)
sectionRE = regexp.MustCompile(`^\s*\.\.\.[A-Z0-9 /-]+\.{3}\s*$`)
)
@@ -41,6 +43,20 @@ func ExtractProductText(rawHTML string) (string, error) {
return "", fmt.Errorf("no useful pre block found")
}
+// ParseDiscussionHTML extracts SPC product text and page-level metadata from a
+// print-page HTML document.
+func ParseDiscussionHTML(rawHTML string) (DiscussionText, error) {
+ text, err := ExtractProductText(rawHTML)
+ if err != nil {
+ return DiscussionText{}, err
+ }
+ parsed := ParseDiscussionText(text)
+ if updatedAt := ParsePageUpdatedTimestamp(rawHTML); updatedAt != nil {
+ parsed.UpdatedAt = updatedAt
+ }
+ return parsed, nil
+}
+
// ParseDiscussionText extracts common SPC narrative metadata from cleaned
// product text.
func ParseDiscussionText(text string) DiscussionText {
@@ -55,6 +71,19 @@ func ParseDiscussionText(text string) DiscussionText {
}
}
+// ParsePageUpdatedTimestamp parses the page-level Updated row from an SPC print
+// page. SPC currently places this outside the product block.
+func ParsePageUpdatedTimestamp(rawHTML string) *time.Time {
+ text := cleanHTMLText(rawHTML)
+ text = strings.ReplaceAll(text, "\u00a0", " ")
+ text = strings.Join(strings.Fields(text), " ")
+ match := pageUpdatedRE.FindStringSubmatch(text)
+ if len(match) != 2 {
+ return nil
+ }
+ return parseUpdatedValue(match[1])
+}
+
// ParseUpdatedTimestamp parses an SPC print-page Updated line when present.
func ParseUpdatedTimestamp(text string) *time.Time {
match := updatedRE.FindStringSubmatch(normalizeNewlines(text))
@@ -68,7 +97,7 @@ func ParseUpdatedTimestamp(text string) *time.Time {
func ParseProductTitle(text string) string {
for _, line := range strings.Split(normalizeNewlines(text), "\n") {
line = strings.TrimSpace(line)
- if line == "" || strings.HasPrefix(line, "Updated:") {
+ if line == "" || strings.HasPrefix(line, "Updated:") || productCodeRE.MatchString(line) {
continue
}
return line
@@ -142,6 +171,7 @@ func parseUpdatedValue(value string) *time.Time {
return t
}
for _, layout := range []string{
+ "Mon Jan 2 15:04:05 UTC 2006",
"1504 UTC Mon Jan 2 2006",
"1504Z Mon Jan 2 2006",
"3:04 PM UTC Mon Jan 2 2006",
diff --git a/internal/providers/spc/discussion_test.go b/internal/providers/spc/discussion_test.go
index 1640da8..b8c9ae9 100644
--- a/internal/providers/spc/discussion_test.go
+++ b/internal/providers/spc/discussion_test.go
@@ -22,18 +22,19 @@ func TestExtractProductTextCleansPreBlock(t *testing.T) {
if !strings.Contains(got, "Day 1 Convective Outlook") {
t.Fatalf("ExtractProductText() missing headline: %q", got)
}
+ if !strings.HasPrefix(got, "SPC AC 111234") {
+ t.Fatalf("ExtractProductText() = %q, want product code prefix", got)
+ }
if strings.HasPrefix(got, "\n") || strings.HasSuffix(got, "\n") {
t.Fatalf("ExtractProductText() retained surrounding blank lines: %q", got)
}
}
-func TestParseDiscussionTextExtractsHeadlineSummaryAndUpdated(t *testing.T) {
- text, err := ExtractProductText(string(readTestFile(t, "day1_prt.html")))
+func TestParseDiscussionHTMLExtractsHeadlineSummaryAndUpdated(t *testing.T) {
+ got, err := ParseDiscussionHTML(string(readTestFile(t, "day1_prt.html")))
if err != nil {
- t.Fatalf("ExtractProductText() error = %v", err)
+ t.Fatalf("ParseDiscussionHTML() error = %v", err)
}
-
- got := ParseDiscussionText(text)
if got.ProductTitle != "Day 1 Convective Outlook" {
t.Fatalf("ProductTitle = %q", got.ProductTitle)
}
@@ -47,27 +48,37 @@ func TestParseDiscussionTextExtractsHeadlineSummaryAndUpdated(t *testing.T) {
if !strings.Contains(got.Discussion, "...DISCUSSION...") {
t.Fatalf("Discussion missing full text: %q", got.Discussion)
}
+ if !strings.HasPrefix(got.Discussion, "SPC AC 111234") {
+ t.Fatalf("Discussion = %q, want product code prefix", got.Discussion)
+ }
wantUpdated := time.Date(2026, 6, 11, 12, 45, 0, 0, time.UTC)
if got.UpdatedAt == nil || !got.UpdatedAt.Equal(wantUpdated) {
t.Fatalf("UpdatedAt = %v, want %s", got.UpdatedAt, wantUpdated)
}
}
+func TestParseProductTitleSkipsSPCProductCode(t *testing.T) {
+ got := ParseProductTitle("SPC AC 101959\nDay 1 Convective Outlook\nNWS Storm Prediction Center Norman OK")
+ if got != "Day 1 Convective Outlook" {
+ t.Fatalf("ParseProductTitle() = %q, want Day 1 Convective Outlook", got)
+ }
+}
+
func TestParseDiscussionTextPreservesCorrectionMarker(t *testing.T) {
- text, err := ExtractProductText(string(readTestFile(t, "day2_prt_corr.html")))
+ got, err := ParseDiscussionHTML(string(readTestFile(t, "day2_prt_corr.html")))
if err != nil {
- t.Fatalf("ExtractProductText() error = %v", err)
+ t.Fatalf("ParseDiscussionHTML() error = %v", err)
}
- got := ParseDiscussionText(text)
if !strings.Contains(got.Headline, "CORR 1") {
t.Fatalf("Headline = %q, want correction marker", got.Headline)
}
if !strings.Contains(got.Discussion, "CORR 1") {
t.Fatalf("Discussion = %q, want correction marker", got.Discussion)
}
- if got.UpdatedAt != nil {
- t.Fatalf("UpdatedAt = %v, want nil", got.UpdatedAt)
+ wantUpdated := time.Date(2026, 6, 11, 17, 30, 0, 0, time.UTC)
+ if got.UpdatedAt == nil || !got.UpdatedAt.Equal(wantUpdated) {
+ t.Fatalf("UpdatedAt = %v, want %s", got.UpdatedAt, wantUpdated)
}
}
@@ -88,3 +99,11 @@ func TestParseUpdatedTimestampAcceptsSPCUTCFormat(t *testing.T) {
t.Fatalf("ParseUpdatedTimestamp() = %v, want %s", got, want)
}
}
+
+func TestParsePageUpdatedTimestampAcceptsLiveSPCShape(t *testing.T) {
+ got := ParsePageUpdatedTimestamp(string(readTestFile(t, "day3_prt.html")))
+ want := time.Date(2026, 6, 11, 20, 0, 0, 0, time.UTC)
+ if got == nil || !got.Equal(want) {
+ t.Fatalf("ParsePageUpdatedTimestamp() = %v, want %s", got, want)
+ }
+}
diff --git a/internal/providers/spc/testdata/day1_prt.html b/internal/providers/spc/testdata/day1_prt.html
index 954a232..f811388 100644
--- a/internal/providers/spc/testdata/day1_prt.html
+++ b/internal/providers/spc/testdata/day1_prt.html
@@ -2,11 +2,14 @@
Day 1 Convective Outlook
+
+| Updated: Thu Jun 11 12:45:00 UTC 2026 (geojson) |
+
+SPC AC 111234
Day 1 Convective Outlook
NWS Storm Prediction Center Norman OK
-Updated: 2026-06-11T12:45:00Z
...SUMMARY...
Severe thunderstorms are possible across parts of the central Plains
diff --git a/internal/providers/spc/testdata/day2_prt_corr.html b/internal/providers/spc/testdata/day2_prt_corr.html
index fe9ec52..030a7cf 100644
--- a/internal/providers/spc/testdata/day2_prt_corr.html
+++ b/internal/providers/spc/testdata/day2_prt_corr.html
@@ -1,7 +1,11 @@
+
+| Updated: Thu Jun 11 17:30:00 UTC 2026 |
+
+SPC AC 111730
Day 2 Convective Outlook CORR 1
NWS Storm Prediction Center Norman OK
diff --git a/internal/providers/spc/testdata/day3_prt.html b/internal/providers/spc/testdata/day3_prt.html
index 20b2051..a8544aa 100644
--- a/internal/providers/spc/testdata/day3_prt.html
+++ b/internal/providers/spc/testdata/day3_prt.html
@@ -1,10 +1,13 @@
+
+| Updated: Thu Jun 11 20:00:00 UTC 2026 |
+
+SPC AC 112000
Day 3 Convective Outlook
NWS Storm Prediction Center Norman OK
-Updated: 2026-06-11T20:00:00Z
...SUMMARY...
A corridor of strong to severe storms may develop near a frontal zone.
diff --git a/internal/sinks/postgres/doc.go b/internal/sinks/postgres/doc.go
index c0a5998..8c0d094 100644
--- a/internal/sinks/postgres/doc.go
+++ b/internal/sinks/postgres/doc.go
@@ -233,6 +233,8 @@
// - run_event_id TEXT -> outlook_runs.event_id / payload.outlooks[i]
// - outlook_index INTEGER -> i (array position in payload.outlooks)
// - as_of TIMESTAMPTZ -> payload.asOf (copied from parent)
+// - outlook_id TEXT -> payload.outlooks[i].id
+// - provider TEXT -> payload.outlooks[i].provider
// - product TEXT -> payload.outlooks[i].product
// - day INTEGER -> payload.outlooks[i].day
// - outlook_type TEXT -> payload.outlooks[i].outlookType
diff --git a/internal/sinks/postgres/map.go b/internal/sinks/postgres/map.go
index 02afddd..e6adebd 100644
--- a/internal/sinks/postgres/map.go
+++ b/internal/sinks/postgres/map.go
@@ -404,6 +404,8 @@ func mapOutlookEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
"run_event_id": e.ID,
"outlook_index": i,
"as_of": asOf,
+ "outlook_id": outlook.ID,
+ "provider": outlook.Provider,
"product": outlook.Product,
"day": outlook.Day,
"outlook_type": outlook.OutlookType,
diff --git a/internal/sinks/postgres/map_test.go b/internal/sinks/postgres/map_test.go
index 15b550a..570d08b 100644
--- a/internal/sinks/postgres/map_test.go
+++ b/internal/sinks/postgres/map_test.go
@@ -312,6 +312,12 @@ func TestMapPostgresEventOutlookStructPayload(t *testing.T) {
if got := writes[1].Values["outlook_index"]; got != 0 {
t.Fatalf("first outlook index = %#v, want 0", got)
}
+ if got := writes[1].Values["outlook_id"]; got != "outlook-1" {
+ t.Fatalf("first outlook_id = %#v, want outlook-1", got)
+ }
+ if got := writes[1].Values["provider"]; got != "spc" {
+ t.Fatalf("first provider = %#v, want spc", got)
+ }
if got := writes[1].Values["valid_from"]; got != run.Outlooks[0].ValidFrom.UTC() {
t.Fatalf("first valid_from = %#v, want UTC %s", got, run.Outlooks[0].ValidFrom.UTC())
}
@@ -335,6 +341,57 @@ func TestMapPostgresEventOutlookRejectsMissingAsOf(t *testing.T) {
}
}
+func TestMapPostgresEventOutlookRejectsMissingIDAndProvider(t *testing.T) {
+ base := model.WeatherOutlook{
+ ID: "outlook-1",
+ Provider: "spc",
+ Product: "convective",
+ Day: 1,
+ OutlookType: "categorical",
+ Label: "SLGT",
+ ValidFrom: time.Date(2026, 6, 11, 13, 0, 0, 0, time.UTC),
+ ValidTo: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
+ IssuedAt: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
+ ExpiresAt: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
+ Geometry: json.RawMessage(`{"type":"Polygon","coordinates":[[[-91,38],[-90,38],[-90,39],[-91,39],[-91,38]]]}`),
+ }
+
+ tests := []struct {
+ name string
+ mutate func(*model.WeatherOutlook)
+ wantErr string
+ }{
+ {
+ name: "missing id",
+ mutate: func(outlook *model.WeatherOutlook) { outlook.ID = "" },
+ wantErr: "outlooks[0].id is required",
+ },
+ {
+ name: "missing provider",
+ mutate: func(outlook *model.WeatherOutlook) { outlook.Provider = "" },
+ wantErr: "outlooks[0].provider is required",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ outlook := base
+ tt.mutate(&outlook)
+ run := model.WeatherOutlookRun{
+ AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
+ Outlooks: []model.WeatherOutlook{outlook},
+ }
+ _, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV1, "outlook", run))
+ if err == nil {
+ t.Fatalf("mapPostgresEvent() error = nil, want %q", tt.wantErr)
+ }
+ if !strings.Contains(err.Error(), tt.wantErr) {
+ t.Fatalf("error = %q, want %q", err, tt.wantErr)
+ }
+ })
+ }
+}
+
func TestMapPostgresEventOutlookRejectsMissingRequiredTimes(t *testing.T) {
run := model.WeatherOutlookRun{
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
diff --git a/internal/sinks/postgres/schema.go b/internal/sinks/postgres/schema.go
index d6c5560..8c1a9a2 100644
--- a/internal/sinks/postgres/schema.go
+++ b/internal/sinks/postgres/schema.go
@@ -329,6 +329,8 @@ func PostgresSchema() fksinks.PostgresSchema {
{Name: "run_event_id", Type: "TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE", Nullable: false},
{Name: "outlook_index", Type: "INTEGER", Nullable: false},
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
+ {Name: "outlook_id", Type: "TEXT", Nullable: false},
+ {Name: "provider", Type: "TEXT", Nullable: false},
{Name: "product", Type: "TEXT", Nullable: false},
{Name: "day", Type: "INTEGER", Nullable: false},
{Name: "outlook_type", Type: "TEXT", Nullable: false},
diff --git a/internal/sinks/postgres/schema_test.go b/internal/sinks/postgres/schema_test.go
index 32f775d..0e92c88 100644
--- a/internal/sinks/postgres/schema_test.go
+++ b/internal/sinks/postgres/schema_test.go
@@ -60,7 +60,7 @@ func TestWeatherPostgresSchemaIncludesOutlookTables(t *testing.T) {
assertTableIndex(t, tableOutlookRuns, "idx_wf_outlook_run_as_of", []string{"as_of"})
outlookColumns := columnsForTable(t, tableOutlooks)
- for _, col := range []string{"run_event_id", "outlook_index", "as_of", "product", "day", "outlook_type", "label", "label_text", "severity_rank", "valid_from", "valid_to", "issued_at", "expires_at", "forecaster", "headline", "summary", "discussion", "source_url", "image_url", "contains_location", "geometry_json"} {
+ for _, col := range []string{"run_event_id", "outlook_index", "as_of", "outlook_id", "provider", "product", "day", "outlook_type", "label", "label_text", "severity_rank", "valid_from", "valid_to", "issued_at", "expires_at", "forecaster", "headline", "summary", "discussion", "source_url", "image_url", "contains_location", "geometry_json"} {
if !outlookColumns[col] {
t.Fatalf("%s missing %s column", tableOutlooks, col)
}
diff --git a/internal/sources/spc/convective_outlook.go b/internal/sources/spc/convective_outlook.go
index 89c2c44..c3a1833 100644
--- a/internal/sources/spc/convective_outlook.go
+++ b/internal/sources/spc/convective_outlook.go
@@ -377,11 +377,7 @@ func latestIssueTime(raw []byte) time.Time {
}
func discussionUpdatedTime(rawHTML string) time.Time {
- text, err := spcprovider.ExtractProductText(rawHTML)
- if err != nil {
- return time.Time{}
- }
- t := spcprovider.ParseUpdatedTimestamp(text)
+ t := spcprovider.ParsePageUpdatedTimestamp(rawHTML)
if t == nil {
return time.Time{}
}