weatherfeeder/docs/roadmap/implementation.md

SPC Convective Outlook Remaining Work Plan

Summary

The SPC convective outlook feature is mostly implemented. This roadmap now tracks only the remaining remediation work needed to align the implementation with the intended contract and live SPC print-page behavior.

Do not use this document as the original full feature implementation plan. It is a focused completion plan for the outstanding gaps below.

Remaining Work

Postgres Persistence Gap

Canonical model.WeatherOutlook includes required id and provider fields, but the current Postgres outlooks table and mapper do not persist them. This makes Postgres persistence lossy for weather.outlook.v1.

Required changes:

• Add outlook_id TEXT NOT NULL to the outlooks table contract.
• Add provider TEXT NOT NULL to the outlooks table contract.
• Map model.WeatherOutlook.ID to outlook_id.
• Map model.WeatherOutlook.Provider to provider.
• Keep existing required-field validation for empty id and provider.
• Update Postgres schema, mapper, schema tests, mapper tests, internal Postgres docs, and Postgres integration docs.
• Do not include manual database migration commands in this roadmap; keep this plan focused on schema/code/docs expectations.

SPC Print-Page Parsing Gap

The current parser fixtures do not match the live SPC print-page structure closely enough. Live SPC print pages currently place the page-level Updated: row outside the <pre> product text, and the product text inside <pre> begins with an SPC product code line such as SPC AC 101959 before the human title line.

Required changes:

• Update SPC print-page fixtures to match live shape:
  • page-level Updated: appears outside the <pre> block
  • <pre> content begins with an SPC AC ... product code line
  • the human outlook title appears on the following line, such as Day 1 Convective Outlook
• Parse page-level Updated: timestamps from the full HTML document, not only from extracted product text.
• Keep cleaned product text extraction focused on the <pre> block.
• Preserve the leading SPC AC ... line in canonical discussion text.
• Update product-title/headline parsing to skip product code lines like SPC AC 101959 and choose the human outlook title line.
• Preserve correction markers such as CORR 1 in the selected headline/title when present.
• Continue stripping scripts and HTML tags from extracted product text.

Source Effective-Time Correction

The source should continue to prefer GeoJSON issue timestamps, but the print-page fallback should use the page-level Updated: parser described above.

Effective-time order should remain:

1. latest valid GeoJSON ISSUE_ISO
2. latest valid print-page Updated: timestamp parsed from full HTML
3. RSS lastBuildDate, when optional RSS is configured and parseable
4. fetch time

Required changes:

• Update source discussion timestamp extraction to parse from full HTML.
• Keep RSS optional and supplemental only.
• Keep unchanged-response hashing, atomic required fetch behavior, and fetch-time fallback unchanged.

Test Plan

Provider parser tests:

• Fixture covers live print-page shape with Updated: outside <pre>.
• Fixture product text begins with SPC AC ....
• Headline resolves to Day X Convective Outlook, not SPC AC ....
• Page-level Updated: parses to UTC.
• Full cleaned discussion text preserves the SPC AC ... line.
• Script and tag cleanup remains covered.
• CORR 1 remains preserved in headline/title and discussion text.

Source tests:

• effectiveAt falls back to page-level Updated: when GeoJSON issue times are unavailable.
• Existing atomic fetch, unchanged-response, required URL failure, and optional RSS tests remain passing.

Normalizer tests:

• Canonical headline uses the human outlook title.
• Canonical discussion preserves full cleaned product text.
• Existing CORR 1, summary, and day-to-discussion mapping tests remain passing.

Postgres tests:

• Schema includes outlook_id and provider on outlooks.
• Mapper writes outlook_id from WeatherOutlook.ID.
• Mapper writes provider from WeatherOutlook.Provider.
• Required-field validation still rejects empty id and provider.
• Existing compact geometry and contains_location=false tests remain passing.

Verification commands:

go test ./internal/providers/spc ./internal/sources/spc ./internal/normalizers/spc ./internal/sinks/postgres
go test ./...

Assumptions

• The canonical API shape remains unchanged: WeatherOutlook.id and WeatherOutlook.provider are already required fields.
• Postgres persistence should be lossless for required canonical outlook fields.
• Existing database migration commands are intentionally omitted from this roadmap.
• RSS remains optional supplemental metadata only.
• This remediation plan does not include weatherapi changes.