Eric Rakestraw
a4cd63ca4e
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
104 lines
4.8 KiB
Markdown
104 lines
4.8 KiB
Markdown
# SPC Convective Outlook Remaining Work Plan
|
|
|
|
## Summary
|
|
|
|
The SPC convective outlook feature is mostly implemented. This roadmap now tracks only the remaining remediation work needed to align the implementation with the intended contract and live SPC print-page behavior.
|
|
|
|
Do not use this document as the original full feature implementation plan. It is a focused completion plan for the outstanding gaps below.
|
|
|
|
## Remaining Work
|
|
|
|
### Postgres Persistence Gap
|
|
|
|
Canonical `model.WeatherOutlook` includes required `id` and `provider` fields, but the current Postgres `outlooks` table and mapper do not persist them. This makes Postgres persistence lossy for `weather.outlook.v1`.
|
|
|
|
Required changes:
|
|
|
|
- Add `outlook_id TEXT NOT NULL` to the `outlooks` table contract.
|
|
- Add `provider TEXT NOT NULL` to the `outlooks` table contract.
|
|
- Map `model.WeatherOutlook.ID` to `outlook_id`.
|
|
- Map `model.WeatherOutlook.Provider` to `provider`.
|
|
- Keep existing required-field validation for empty `id` and `provider`.
|
|
- Update Postgres schema, mapper, schema tests, mapper tests, internal Postgres docs, and Postgres integration docs.
|
|
- Do not include manual database migration commands in this roadmap; keep this plan focused on schema/code/docs expectations.
|
|
|
|
### SPC Print-Page Parsing Gap
|
|
|
|
The current parser fixtures do not match the live SPC print-page structure closely enough. Live SPC print pages currently place the page-level `Updated:` row outside the `<pre>` product text, and the product text inside `<pre>` begins with an SPC product code line such as `SPC AC 101959` before the human title line.
|
|
|
|
Required changes:
|
|
|
|
- Update SPC print-page fixtures to match live shape:
|
|
- page-level `Updated:` appears outside the `<pre>` block
|
|
- `<pre>` content begins with an `SPC AC ...` product code line
|
|
- the human outlook title appears on the following line, such as `Day 1 Convective Outlook`
|
|
- Parse page-level `Updated:` timestamps from the full HTML document, not only from extracted product text.
|
|
- Keep cleaned product text extraction focused on the `<pre>` block.
|
|
- Preserve the leading `SPC AC ...` line in canonical `discussion` text.
|
|
- Update product-title/headline parsing to skip product code lines like `SPC AC 101959` and choose the human outlook title line.
|
|
- Preserve correction markers such as `CORR 1` in the selected headline/title when present.
|
|
- Continue stripping scripts and HTML tags from extracted product text.
|
|
|
|
### Source Effective-Time Correction
|
|
|
|
The source should continue to prefer GeoJSON issue timestamps, but the print-page fallback should use the page-level `Updated:` parser described above.
|
|
|
|
Effective-time order should remain:
|
|
|
|
1. latest valid GeoJSON `ISSUE_ISO`
|
|
2. latest valid print-page `Updated:` timestamp parsed from full HTML
|
|
3. RSS `lastBuildDate`, when optional RSS is configured and parseable
|
|
4. fetch time
|
|
|
|
Required changes:
|
|
|
|
- Update source discussion timestamp extraction to parse from full HTML.
|
|
- Keep RSS optional and supplemental only.
|
|
- Keep unchanged-response hashing, atomic required fetch behavior, and fetch-time fallback unchanged.
|
|
|
|
## Test Plan
|
|
|
|
Provider parser tests:
|
|
|
|
- Fixture covers live print-page shape with `Updated:` outside `<pre>`.
|
|
- Fixture product text begins with `SPC AC ...`.
|
|
- Headline resolves to `Day X Convective Outlook`, not `SPC AC ...`.
|
|
- Page-level `Updated:` parses to UTC.
|
|
- Full cleaned discussion text preserves the `SPC AC ...` line.
|
|
- Script and tag cleanup remains covered.
|
|
- `CORR 1` remains preserved in headline/title and discussion text.
|
|
|
|
Source tests:
|
|
|
|
- `effectiveAt` falls back to page-level `Updated:` when GeoJSON issue times are unavailable.
|
|
- Existing atomic fetch, unchanged-response, required URL failure, and optional RSS tests remain passing.
|
|
|
|
Normalizer tests:
|
|
|
|
- Canonical `headline` uses the human outlook title.
|
|
- Canonical `discussion` preserves full cleaned product text.
|
|
- Existing `CORR 1`, summary, and day-to-discussion mapping tests remain passing.
|
|
|
|
Postgres tests:
|
|
|
|
- Schema includes `outlook_id` and `provider` on `outlooks`.
|
|
- Mapper writes `outlook_id` from `WeatherOutlook.ID`.
|
|
- Mapper writes `provider` from `WeatherOutlook.Provider`.
|
|
- Required-field validation still rejects empty `id` and `provider`.
|
|
- Existing compact geometry and `contains_location=false` tests remain passing.
|
|
|
|
Verification commands:
|
|
|
|
```sh
|
|
go test ./internal/providers/spc ./internal/sources/spc ./internal/normalizers/spc ./internal/sinks/postgres
|
|
go test ./...
|
|
```
|
|
|
|
## Assumptions
|
|
|
|
- The canonical API shape remains unchanged: `WeatherOutlook.id` and `WeatherOutlook.provider` are already required fields.
|
|
- Postgres persistence should be lossless for required canonical outlook fields.
|
|
- Existing database migration commands are intentionally omitted from this roadmap.
|
|
- RSS remains optional supplemental metadata only.
|
|
- This remediation plan does not include weatherapi changes.
|