Added an implementation plan for SPC convective outlook support, and completed the documentation cleanup

docs/roadmap/spc.md

@@ -2,7 +2,7 @@
 
 ## Summary
 
-Add `weatherfeeder` support for Storm Prediction Center convective outlooks as a new canonical outlook domain. The feature should poll SPC GeoJSON outlook products, optionally enrich them with RSS discussion metadata, compute whether the configured forecast point is inside each active outlook polygon, normalize the result into a provider-independent schema, and persist it through existing sinks.
+Add `weatherfeeder` support for Storm Prediction Center convective outlooks as a new canonical outlook domain. The feature should poll SPC GeoJSON outlook products, enrich them with Day 1-3 print-page discussion text, compute whether the configured forecast point is inside each active outlook polygon, normalize the result into a provider-independent schema, and persist it through existing sinks.
 
 This is a new domain, not an extension of `weather.alert.v1`. SPC outlooks describe probabilistic/categorical risk areas over a validity window; alerts describe active actionable hazard messages.
 
@@ -64,10 +64,11 @@ Contract defaults:
 - `day` should be `1`, `2`, or `3` for this first implementation.
 - `containsLocation` is computed against configured forecast coordinates.
 - `geometry` should preserve compact GeoJSON geometry for auditability and future API use.
+- Day 4-8 outlooks are out of scope for this first implementation. They have different product semantics and should be added in a later schema-compatible extension only after Day 1-3 support is proven.
 
 ## Source Scope
 
-The source should fetch a bundle of SPC products in one poll cycle and emit one raw event containing the fetched RSS metadata, GeoJSON products, configured point, and per-product fetch metadata.
+The source should fetch a bundle of SPC products in one poll cycle and emit one raw event containing fetched GeoJSON products, fetched Day 1-3 print pages, optional RSS metadata, configured point metadata, and per-product fetch metadata. The source may decode minimal timestamps for event identity/effective time, but provider-to-canonical mapping belongs in the normalizer.
 
 Poll these GeoJSON URLs:
 
@@ -84,17 +85,31 @@ Poll these GeoJSON URLs:
 - `https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson`
 - `https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson`
 
-Note: the initial candidate list duplicated Day 3 wind and omitted Day 2 wind. Use the corrected 12-product matrix above.
+Implementation note: the initial candidate list duplicated Day 3 wind and omitted Day 2 wind. Use the corrected 12-product matrix above.
 
-Also poll:
+Poll these print-page discussion URLs:
+
+- `https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html`
+- `https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html`
+- `https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html`
+
+Print-page usage:
+
+- Use print pages as the authoritative source for Day 1-3 outlook discussion text.
+- Preserve source URL, page title/header where available, page-level `Updated:` timestamp where parseable, and the raw HTML or extracted `<pre>` product block in the raw bundle.
+- Extract canonical `headline`, `summary`, `discussion`, and discussion metadata in the normalizer or shared SPC provider helper.
+- Treat the GeoJSON files, not print pages, as authoritative for polygons and validity windows.
+- Ignore page risk tables and image assets for v1 unless they are needed to support discussion parsing.
+
+Optionally poll:
 
 - `https://www.spc.noaa.gov/products/spcacrss.xml`
 
 RSS usage:
 
-- Use RSS as discussion/discovery metadata, not as the geometry source.
+- Use RSS as supplemental discovery or health metadata, not as the primary discussion or geometry source.
 - Preserve item title, link, description text, pubDate, and guid where useful.
-- Extract the narrative outlook text from the RSS item description when practical.
+- Do not depend on RSS completeness for Day 1-3 discussion text. The RSS feed can omit current outlook products, including a current Day 2 discussion.
 - Do not attempt to parse polygons from RSS HTML.
 
 Polling cadence:
@@ -119,19 +134,21 @@ Recommended config shape:
     user_agent: "HomeOps (eric@maximumdirect.net)"
 ```
 
-The source should own default SPC URLs, with optional params to override the RSS URL and product URLs for tests or future provider changes.
+The source should own default SPC URLs, with optional params to override GeoJSON URLs, print-page URLs, and the RSS URL for tests or future provider changes.
 
 ## Implementation Stages
 
 ### Stage 1: Raw Source and Schema
 
-- Add schema constants and model placeholders.
+- Add schema constants.
 - Add `internal/sources/spc` package.
 - Implement `spc_convective_outlook` as a poll source.
-- Fetch all configured GeoJSON products and RSS during a poll cycle.
+- Fetch all configured GeoJSON products and Day 1-3 print pages during a poll cycle.
+- Fetch RSS only when configured or retained as an optional supplemental source.
 - Emit one raw event with kind `outlook` and schema `raw.spc.convective_outlook.v1`.
-- Use the latest valid `ISSUE_ISO`, RSS `lastBuildDate`, or fetch time for `effectiveAt`, in that order.
+- Use the latest valid `ISSUE_ISO`, print-page `Updated:` timestamp, RSS `lastBuildDate`, or fetch time for `effectiveAt`, in that order.
 - Preserve partial fetch errors as source errors for the first implementation; do not emit incomplete outlook bundles unless a later explicit degraded-mode policy is added.
+- Keep source tests deterministic with local test servers and fixtures; do not call live SPC services from unit tests.
 
 Tests:
 
@@ -141,6 +158,7 @@ Tests:
 - source chooses expected effective time
 - source uses configured location metadata
 - source fails clearly on missing latitude/longitude
+- source fails clearly when a configured GeoJSON or print-page fetch fails
 
 ### Stage 2: GeoJSON and Spatial Matching
 
@@ -164,6 +182,7 @@ Tests:
 
 ### Stage 3: Normalizer and Canonical Mapping
 
+- Add `model.WeatherOutlookRun` and `model.WeatherOutlook`.
 - Add `SPCConvectiveOutlookNormalizer`.
 - Match only `raw.spc.convective_outlook.v1`.
 - Decode the raw source bundle.
@@ -182,8 +201,14 @@ Tests:
   - `outlookType` from product key or URL
   - `id` from day, type, label, issuedAt, validFrom, and feature index
   - `containsLocation` from configured point and GeoJSON geometry
-- Enrich narrative fields from RSS where a matching day/product item can be determined.
-- Set run `asOf` to the latest valid `issuedAt` across features, falling back to RSS `lastBuildDate`, then input event time.
+- Enrich narrative fields from the matching Day 1-3 print page:
+  - parse the main product text from the `<pre>` block
+  - strip embedded tags and scripts before text parsing
+  - preserve line breaks in discussion text
+  - parse page/title metadata, product title, issued line, valid line, correction markers such as `CORR 1`, forecaster, summary, and full discussion where practical
+  - use print-page text for canonical `headline`, `summary`, and `discussion`
+- Use RSS only as optional supplemental metadata when present; missing or incomplete RSS must not prevent GeoJSON plus print-page normalization.
+- Set run `asOf` to the latest valid `issuedAt` across features, falling back to print-page `Updated:` timestamps, RSS `lastBuildDate`, then input event time.
 - Set run `issuedAt` to the latest valid `issuedAt` across features.
 - Set output event `effectiveAt` to run `asOf`.
 
@@ -193,7 +218,9 @@ Tests:
 - categorical and probabilistic products map expected fields
 - `containsLocation` is true for a known point inside a fixture polygon
 - `containsLocation` is false outside
-- missing optional RSS still permits GeoJSON normalization
+- print-page discussion text maps to canonical headline, summary, and discussion
+- correction markers such as `CORR 1` are preserved where present
+- missing optional RSS still permits GeoJSON and print-page normalization
 - malformed required GeoJSON timestamps fail with useful context
 - canonical JSON shape does not expose raw-provider-only bundle internals
 
@@ -264,16 +291,21 @@ Tests:
 
 - Update sample config with `SPCConvectiveOutlookSTL`.
 - Add `outlook` to route examples.
-- Update README provider capabilities.
-- Update `API.md` with `weather.outlook.v1`.
-- Update Postgres sink docs with outlook table contract.
+- Update current-behavior docs in the implementation change, not before behavior exists.
+- Update `docs/config.md` with source params.
+- Update `docs/integrations/events.md` with `weather.outlook.v1`.
+- Update `docs/integrations/postgres.md` with outlook table contract.
+- Update relevant internal docs under `docs/internal/` if source, normalizer, or sink behavior needs maintainer context.
+- Update `README.md` only with brief implemented-provider capability and links to canonical docs.
 
 Docs should clearly state:
 
-- RSS is used for narrative/discussion metadata.
-- GeoJSON is used for polygons and point matching.
+- Print pages are used for Day 1-3 narrative/discussion text.
+- RSS, if retained, is supplemental and not required for complete discussion coverage.
+- GeoJSON is used for polygons, validity windows, and point matching.
 - `containsLocation` is computed by weatherfeeder at ingestion time.
 - Geometry is stored for downstream audit/display.
+- Day 4-8 outlooks are not included in the initial implementation.
 
 ### Stage 6: Weatherapi Follow-Up
 
@@ -291,13 +323,6 @@ Recommended API behavior:
 - active outlooks filtered by current time and `containsLocation=true`
 - optional filter query params for `day`, `outlookType`, and `containsLocation`
 
-## Open Questions for Implementation
-
-- Whether to include Day 4-8 probabilistic outlooks in a later version. Exclude them from v1.
-- Whether to make partial source bundles acceptable if one product URL fails. Default v1 behavior should fail the poll and retry.
-- Whether to parse detailed discussion text from RSS only, HTML pages, or both. Default v1 should use RSS only.
-- Whether to keep `geometry` in canonical JSON permanently. Default v1 should include it because it preserves source context and enables downstream display.
-
 ## Verification Commands
 
 Run focused tests:
@@ -318,6 +343,7 @@ go test ./...
 - Normalization produces `weather.outlook.v1` events.
 - Each outlook indicates whether the configured forecast point is inside its polygon.
 - All current Day 1-3 categorical/tornado/hail/wind products are represented.
-- RSS discussion metadata is preserved where available.
+- Day 1-3 print-page discussion text is preserved where available.
+- RSS metadata is optional and supplemental if retained.
 - Postgres sink persists outlook runs and outlook rows.
 - Sample config and public docs describe the new kind, driver, schema, and storage contract.