Finalize SPC outlook feature addition and clean up implemented roadmap documentation
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
This commit is contained in:
@@ -1,103 +0,0 @@
|
||||
# SPC Convective Outlook Remaining Work Plan
|
||||
|
||||
## Summary
|
||||
|
||||
The SPC convective outlook feature is mostly implemented. This roadmap now tracks only the remaining remediation work needed to align the implementation with the intended contract and live SPC print-page behavior.
|
||||
|
||||
Do not use this document as the original full feature implementation plan. It is a focused completion plan for the outstanding gaps below.
|
||||
|
||||
## Remaining Work
|
||||
|
||||
### Postgres Persistence Gap
|
||||
|
||||
Canonical `model.WeatherOutlook` includes required `id` and `provider` fields, but the current Postgres `outlooks` table and mapper do not persist them. This makes Postgres persistence lossy for `weather.outlook.v1`.
|
||||
|
||||
Required changes:
|
||||
|
||||
- Add `outlook_id TEXT NOT NULL` to the `outlooks` table contract.
|
||||
- Add `provider TEXT NOT NULL` to the `outlooks` table contract.
|
||||
- Map `model.WeatherOutlook.ID` to `outlook_id`.
|
||||
- Map `model.WeatherOutlook.Provider` to `provider`.
|
||||
- Keep existing required-field validation for empty `id` and `provider`.
|
||||
- Update Postgres schema, mapper, schema tests, mapper tests, internal Postgres docs, and Postgres integration docs.
|
||||
- Do not include manual database migration commands in this roadmap; keep this plan focused on schema/code/docs expectations.
|
||||
|
||||
### SPC Print-Page Parsing Gap
|
||||
|
||||
The current parser fixtures do not match the live SPC print-page structure closely enough. Live SPC print pages currently place the page-level `Updated:` row outside the `<pre>` product text, and the product text inside `<pre>` begins with an SPC product code line such as `SPC AC 101959` before the human title line.
|
||||
|
||||
Required changes:
|
||||
|
||||
- Update SPC print-page fixtures to match live shape:
|
||||
- page-level `Updated:` appears outside the `<pre>` block
|
||||
- `<pre>` content begins with an `SPC AC ...` product code line
|
||||
- the human outlook title appears on the following line, such as `Day 1 Convective Outlook`
|
||||
- Parse page-level `Updated:` timestamps from the full HTML document, not only from extracted product text.
|
||||
- Keep cleaned product text extraction focused on the `<pre>` block.
|
||||
- Preserve the leading `SPC AC ...` line in canonical `discussion` text.
|
||||
- Update product-title/headline parsing to skip product code lines like `SPC AC 101959` and choose the human outlook title line.
|
||||
- Preserve correction markers such as `CORR 1` in the selected headline/title when present.
|
||||
- Continue stripping scripts and HTML tags from extracted product text.
|
||||
|
||||
### Source Effective-Time Correction
|
||||
|
||||
The source should continue to prefer GeoJSON issue timestamps, but the print-page fallback should use the page-level `Updated:` parser described above.
|
||||
|
||||
Effective-time order should remain:
|
||||
|
||||
1. latest valid GeoJSON `ISSUE_ISO`
|
||||
2. latest valid print-page `Updated:` timestamp parsed from full HTML
|
||||
3. RSS `lastBuildDate`, when optional RSS is configured and parseable
|
||||
4. fetch time
|
||||
|
||||
Required changes:
|
||||
|
||||
- Update source discussion timestamp extraction to parse from full HTML.
|
||||
- Keep RSS optional and supplemental only.
|
||||
- Keep unchanged-response hashing, atomic required fetch behavior, and fetch-time fallback unchanged.
|
||||
|
||||
## Test Plan
|
||||
|
||||
Provider parser tests:
|
||||
|
||||
- Fixture covers live print-page shape with `Updated:` outside `<pre>`.
|
||||
- Fixture product text begins with `SPC AC ...`.
|
||||
- Headline resolves to `Day X Convective Outlook`, not `SPC AC ...`.
|
||||
- Page-level `Updated:` parses to UTC.
|
||||
- Full cleaned discussion text preserves the `SPC AC ...` line.
|
||||
- Script and tag cleanup remains covered.
|
||||
- `CORR 1` remains preserved in headline/title and discussion text.
|
||||
|
||||
Source tests:
|
||||
|
||||
- `effectiveAt` falls back to page-level `Updated:` when GeoJSON issue times are unavailable.
|
||||
- Existing atomic fetch, unchanged-response, required URL failure, and optional RSS tests remain passing.
|
||||
|
||||
Normalizer tests:
|
||||
|
||||
- Canonical `headline` uses the human outlook title.
|
||||
- Canonical `discussion` preserves full cleaned product text.
|
||||
- Existing `CORR 1`, summary, and day-to-discussion mapping tests remain passing.
|
||||
|
||||
Postgres tests:
|
||||
|
||||
- Schema includes `outlook_id` and `provider` on `outlooks`.
|
||||
- Mapper writes `outlook_id` from `WeatherOutlook.ID`.
|
||||
- Mapper writes `provider` from `WeatherOutlook.Provider`.
|
||||
- Required-field validation still rejects empty `id` and `provider`.
|
||||
- Existing compact geometry and `contains_location=false` tests remain passing.
|
||||
|
||||
Verification commands:
|
||||
|
||||
```sh
|
||||
go test ./internal/providers/spc ./internal/sources/spc ./internal/normalizers/spc ./internal/sinks/postgres
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Assumptions
|
||||
|
||||
- The canonical API shape remains unchanged: `WeatherOutlook.id` and `WeatherOutlook.provider` are already required fields.
|
||||
- Postgres persistence should be lossless for required canonical outlook fields.
|
||||
- Existing database migration commands are intentionally omitted from this roadmap.
|
||||
- RSS remains optional supplemental metadata only.
|
||||
- This remediation plan does not include weatherapi changes.
|
||||
Reference in New Issue
Block a user