ejr/weatherfeeder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update the SPC outlook implementation plan to identify remaining gaps and corrections
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
Details

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-10 19:40:46 -05:00
parent fba519cab0
commit a4cd63ca4e
1 changed files with 69 additions and 663 deletions
Download Patch File Download Diff File Expand all files Collapse all files

732
docs/roadmap/implementation.md
View File

@@ -1,697 +1,103 @@
# SPC Convective Outlook Implementation Plan
# SPC Convective Outlook Remaining Work Plan
## Purpose
## Summary
Implement `weatherfeeder` support for Storm Prediction Center Day 1-3 convective outlooks described in [`docs/roadmap/spc.md`](spc.md). This plan is written for an LLM coding agent and should be followed stage by stage.
The SPC convective outlook feature is mostly implemented. This roadmap now tracks only the remaining remediation work needed to align the implementation with the intended contract and live SPC print-page behavior.
This is a planning document only. The implementation must preserve the existing weatherfeeder architecture: sources emit raw provider events, normalizers map raw payloads into canonical model types, and sinks persist canonical schemas.
Do not use this document as the original full feature implementation plan. It is a focused completion plan for the outstanding gaps below.
## Decisions
## Remaining Work
The following choices are fixed for this implementation:
### Postgres Persistence Gap
- Scope is Day 1-3 SPC convective outlooks only.
- Day 4-8 outlooks are out of scope.
- GeoJSON files are authoritative for polygons, validity windows, issue times, outlook labels, and severity ranking.
- Day 1-3 print pages are authoritative for discussion text.
- RSS is optional supplemental metadata only and must not be required for correctness.
- Do not fetch RSS by default. Include RSS only when an optional `rss_url` source param is configured.
- A poll is atomic for required products. If any configured GeoJSON or print-page URL fails or returns a non-2xx response, return an error and emit no event.
- Use compact GeoJSON geometry in the canonical payload for auditability and downstream display.
- Use standard-library-first parsing. Do not add an HTML parsing dependency unless string-based extraction proves unmaintainable during implementation.
- Keep all new planned behavior inside `weatherfeeder`; do not make `weatherapi` changes in this pass.
Canonical `model.WeatherOutlook` includes required `id` and `provider` fields, but the current Postgres `outlooks` table and mapper do not persist them. This makes Postgres persistence lossy for `weather.outlook.v1`.
## Public Contract
Required changes:
Add schema constants in `standards/schema.go`:
- Add `outlook_id TEXT NOT NULL` to the `outlooks` table contract.
- Add `provider TEXT NOT NULL` to the `outlooks` table contract.
- Map `model.WeatherOutlook.ID` to `outlook_id`.
- Map `model.WeatherOutlook.Provider` to `provider`.
- Keep existing required-field validation for empty `id` and `provider`.
- Update Postgres schema, mapper, schema tests, mapper tests, internal Postgres docs, and Postgres integration docs.
- Do not include manual database migration commands in this roadmap; keep this plan focused on schema/code/docs expectations.
- `SchemaRawSPCConvectiveOutlookV1 = "raw.spc.convective_outlook.v1"`
- `SchemaWeatherOutlookV1 = "weather.outlook.v1"`
### SPC Print-Page Parsing Gap
Add source driver:
The current parser fixtures do not match the live SPC print-page structure closely enough. Live SPC print pages currently place the page-level `Updated:` row outside the `<pre>` product text, and the product text inside `<pre>` begins with an SPC product code line such as `SPC AC 101959` before the human title line.
- `spc_convective_outlook`
Required changes:
Add event kind:
- Update SPC print-page fixtures to match live shape:
- page-level `Updated:` appears outside the `<pre>` block
- `<pre>` content begins with an `SPC AC ...` product code line
- the human outlook title appears on the following line, such as `Day 1 Convective Outlook`
- Parse page-level `Updated:` timestamps from the full HTML document, not only from extracted product text.
- Keep cleaned product text extraction focused on the `<pre>` block.
- Preserve the leading `SPC AC ...` line in canonical `discussion` text.
- Update product-title/headline parsing to skip product code lines like `SPC AC 101959` and choose the human outlook title line.
- Preserve correction markers such as `CORR 1` in the selected headline/title when present.
- Continue stripping scripts and HTML tags from extracted product text.
- `outlook`
### Source Effective-Time Correction
Add canonical model types:
The source should continue to prefer GeoJSON issue timestamps, but the print-page fallback should use the page-level `Updated:` parser described above.
- `model.WeatherOutlookRun`
- `model.WeatherOutlook`
Effective-time order should remain:
Canonical run fields:
1. latest valid GeoJSON `ISSUE_ISO`
2. latest valid print-page `Updated:` timestamp parsed from full HTML
3. RSS `lastBuildDate`, when optional RSS is configured and parseable
4. fetch time
```go
type WeatherOutlookRun struct {
LocationID string `json:"locationId,omitempty"`
LocationName string `json:"locationName,omitempty"`
Latitude *float64 `json:"latitude,omitempty"`
Longitude *float64 `json:"longitude,omitempty"`
AsOf time.Time `json:"asOf"`
IssuedAt *time.Time `json:"issuedAt,omitempty"`
Outlooks []WeatherOutlook `json:"outlooks"`
}
```
Required changes:
Canonical outlook fields:
- Update source discussion timestamp extraction to parse from full HTML.
- Keep RSS optional and supplemental only.
- Keep unchanged-response hashing, atomic required fetch behavior, and fetch-time fallback unchanged.
```go
type WeatherOutlook struct {
ID string `json:"id"`
Provider string `json:"provider"`
Product string `json:"product"`
Day int `json:"day"`
OutlookType string `json:"outlookType"`
Label string `json:"label"`
LabelText string `json:"labelText,omitempty"`
SeverityRank *int `json:"severityRank,omitempty"`
ValidFrom time.Time `json:"validFrom"`
ValidTo time.Time `json:"validTo"`
IssuedAt time.Time `json:"issuedAt"`
ExpiresAt time.Time `json:"expiresAt"`
Forecaster string `json:"forecaster,omitempty"`
Headline string `json:"headline,omitempty"`
Summary string `json:"summary,omitempty"`
Discussion string `json:"discussion,omitempty"`
SourceURL string `json:"sourceUrl,omitempty"`
ImageURL string `json:"imageUrl,omitempty"`
ContainsLocation bool `json:"containsLocation"`
Geometry json.RawMessage `json:"geometry"`
}
```
## Test Plan
Required canonical fields:
Provider parser tests:
- Run: `asOf`, `outlooks`.
- Outlook: `id`, `provider`, `product`, `day`, `outlookType`, `label`, `validFrom`, `validTo`, `issuedAt`, `expiresAt`, `containsLocation`, `geometry`.
- Fixture covers live print-page shape with `Updated:` outside `<pre>`.
- Fixture product text begins with `SPC AC ...`.
- Headline resolves to `Day X Convective Outlook`, not `SPC AC ...`.
- Page-level `Updated:` parses to UTC.
- Full cleaned discussion text preserves the `SPC AC ...` line.
- Script and tag cleanup remains covered.
- `CORR 1` remains preserved in headline/title and discussion text.
Canonical values:
Source tests:
- `provider` is `spc`.
- `product` is `convective`.
- `outlookType` is one of `categorical`, `tornado`, `hail`, `wind`.
- `day` is one of `1`, `2`, `3`.
- `effectiveAt` falls back to page-level `Updated:` when GeoJSON issue times are unavailable.
- Existing atomic fetch, unchanged-response, required URL failure, and optional RSS tests remain passing.
## Source Inputs
Normalizer tests:
Default required GeoJSON products:
- Canonical `headline` uses the human outlook title.
- Canonical `discussion` preserves full cleaned product text.
- Existing `CORR 1`, summary, and day-to-discussion mapping tests remain passing.
- `https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson`
Postgres tests:
Default required print-page products:
- Schema includes `outlook_id` and `provider` on `outlooks`.
- Mapper writes `outlook_id` from `WeatherOutlook.ID`.
- Mapper writes `provider` from `WeatherOutlook.Provider`.
- Required-field validation still rejects empty `id` and `provider`.
- Existing compact geometry and `contains_location=false` tests remain passing.
- `https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html`
Optional RSS product:
- `https://www.spc.noaa.gov/products/spcacrss.xml`
Recommended config shape:
```yaml
- name: SPCConvectiveOutlookSTL
mode: poll
kinds: ["outlook"]
driver: spc_convective_outlook
every: 30m
params:
latitude: 38.6239
longitude: -90.3571
location_id: "stl"
location_name: "St. Louis, MO"
user_agent: "HomeOps (eric@maximumdirect.net)"
```
Optional source params:
- `geojson_urls`: map from product key to URL, used by tests and future upstream changes.
- `discussion_urls`: map from day key to URL, used by tests and future upstream changes.
- `rss_url`: string; when non-empty, fetch RSS as supplemental metadata.
Product keys for `geojson_urls`:
- `day1_categorical`, `day1_tornado`, `day1_hail`, `day1_wind`
- `day2_categorical`, `day2_tornado`, `day2_hail`, `day2_wind`
- `day3_categorical`, `day3_tornado`, `day3_hail`, `day3_wind`
Discussion keys for `discussion_urls`:
- `day1`, `day2`, `day3`
## Raw Bundle Shape
Create a provider raw bundle type under `internal/providers/spc` or `internal/normalizers/spc` and use it consistently between source tests and normalizer tests. Prefer `internal/providers/spc` if source metadata extraction and normalizer parsing share helpers.
Suggested raw payload shape:
```go
type RawConvectiveOutlookBundle struct {
LocationID string `json:"locationId,omitempty"`
LocationName string `json:"locationName,omitempty"`
Latitude float64 `json:"latitude"`
Longitude float64 `json:"longitude"`
FetchedAt time.Time `json:"fetchedAt"`
Products []RawOutlookProduct `json:"products"`
Discussions []RawDiscussionPage `json:"discussions"`
RSS *RawRSSFeed `json:"rss,omitempty"`
}
```
```go
type RawOutlookProduct struct {
Key string `json:"key"`
Day int `json:"day"`
OutlookType string `json:"outlookType"`
URL string `json:"url"`
FetchedAt time.Time `json:"fetchedAt"`
Body json.RawMessage `json:"body"`
}
```
```go
type RawDiscussionPage struct {
Key string `json:"key"`
Day int `json:"day"`
URL string `json:"url"`
FetchedAt time.Time `json:"fetchedAt"`
Body string `json:"body"`
}
```
```go
type RawRSSFeed struct {
URL string `json:"url"`
FetchedAt time.Time `json:"fetchedAt"`
Body string `json:"body"`
}
```
Do not put parsed canonical fields into the raw bundle except configured metadata and product keys needed to identify each fetched upstream document. Source-level timestamp parsing is allowed only for event `effectiveAt` selection.
## Stage 1: Provider Helpers And Fixtures
Goal: add deterministic SPC parsing primitives and test fixtures before wiring the source or normalizer.
Files to add:
- `internal/providers/spc/doc.go`
- `internal/providers/spc/time.go`
- `internal/providers/spc/product.go`
- `internal/providers/spc/geojson.go`
- `internal/providers/spc/discussion.go`
- `internal/providers/spc/rss.go`, only if optional RSS parsing is implemented
- `internal/providers/spc/testdata/day1_cat.geojson`
- `internal/providers/spc/testdata/day2_torn.geojson`
- `internal/providers/spc/testdata/day3_wind.geojson`
- `internal/providers/spc/testdata/day1_prt.html`
- `internal/providers/spc/testdata/day2_prt_corr.html`
- `internal/providers/spc/testdata/day3_prt.html`
Provider helper behavior:
- Define stable product metadata for the 12 required GeoJSON products.
- Define stable discussion metadata for the 3 required print-page products.
- Parse SPC ISO timestamps from GeoJSON properties using `time.Parse(time.RFC3339, value)` after trimming whitespace.
- Decode enough GeoJSON to expose feature properties and raw geometry without owning canonical mapping.
- Preserve raw geometry as compact JSON bytes for later canonical use.
- Extract print-page product text from the first useful `<pre>` block.
- Strip embedded `<script>` blocks and remaining tags from extracted `<pre>` content.
- Use `html.UnescapeString`, normalize CRLF to LF, and trim surrounding blank lines.
- Parse print-page `Updated:` timestamps when present.
- Parse discussion headline/product title from product text.
- Parse `...SUMMARY...` content through the next section heading.
- Preserve full product text as discussion text after cleanup.
- Preserve correction markers such as `CORR 1` in headline or metadata when present.
Dependency rule:
- Start with `strings`, `regexp`, `encoding/json`, `encoding/xml`, and `html` from the standard library.
- Do not add `golang.org/x/net/html` or another HTML parser unless tests show the string extraction is too brittle.
Tests:
- `go test ./internal/providers/spc`
- Product metadata contains exactly 12 GeoJSON products in day/type order.
- Discussion metadata contains exactly 3 print pages in day order.
- GeoJSON fixture decode exposes `VALID_ISO`, `EXPIRE_ISO`, `ISSUE_ISO`, `FORECASTER`, `LABEL`, `LABEL2`, `DN`, and geometry.
- Print-page fixture extraction returns product text without scripts or tags.
- Day 2 correction fixture preserves `CORR 1`.
- Summary extraction returns only the summary paragraph content.
- Updated timestamp parser returns UTC time when present and nil when absent.
Stage completion criteria:
- Provider helper tests pass.
- No source, normalizer, model, standards, sink, config, or docs current-behavior files are changed in this stage unless needed for package compilation.
## Stage 2: Geometry Helper
Goal: implement point-in-polygon support independent of SPC parsing.
Files to add:
- `internal/geo/geojson.go`
- `internal/geo/point.go`
- `internal/geo/point_test.go`
Behavior:
- Support GeoJSON `Polygon` and `MultiPolygon` only.
- Accept geometry as `json.RawMessage` or `[]byte`.
- Interpret GeoJSON coordinate order as `[longitude, latitude]`.
- Treat the first ring as the exterior ring.
- Treat subsequent rings as holes.
- Count boundary points as inside.
- Return a clear error for unsupported geometry types, malformed coordinates, empty rings, or invalid JSON.
- Use planar ray casting. This is sufficient for operational point-in-polygon checks at SPC polygon scale.
Tests:
- Point inside simple polygon returns true.
- Point outside simple polygon returns false.
- Point on polygon boundary returns true.
- Point in a hole returns false.
- Point inside one `MultiPolygon` member returns true.
- Longitude/latitude ordering regression test fails if coordinates are reversed.
- Unsupported geometry returns a useful error.
Verification:
```sh
go test ./internal/geo
```
Stage completion criteria:
- Geometry helper has no dependency on SPC, source, normalizer, or sink packages.
## Stage 3: Source Driver And Raw Schema
Goal: emit raw SPC outlook bundles from a new poll source.
Files to update:
- `standards/schema.go`
- `internal/sources/builtins.go`
- `internal/sources/builtins_test.go`
Files to add:
- `internal/sources/spc/convective_outlook.go`
- `internal/sources/spc/convective_outlook_test.go`
Source constructor:
- Export `NewConvectiveOutlookSource(cfg config.SourceConfig) (*ConvectiveOutlookSource, error)`.
- Register driver `spc_convective_outlook` in `internal/sources/builtins.go`.
- Validate required params `latitude` and `longitude`.
- Accept optional params `location_id`, `location_name`, `geojson_urls`, `discussion_urls`, and `rss_url`.
- Require `user_agent` when the existing HTTP source conventions require it.
- Advertise `Kinds() []event.Kind{event.Kind("outlook")}`.
Fetch behavior:
- Fetch all required GeoJSON URLs and print-page URLs every poll cycle.
- Fetch optional RSS only when `rss_url` is configured and non-empty.
- Use Accept `application/geo+json, application/json` for GeoJSON requests.
- Use Accept `text/html, application/xhtml+xml` for print pages.
- Use Accept `application/rss+xml, application/xml, text/xml` for RSS.
- Respect `context.Context` on all requests.
- Do not emit partial bundles.
- If the complete raw bundle is unchanged from the previous successful poll, return no events. Implement source-local change detection by hashing the combined fetched bodies if feedkit HTTP conditional helpers do not directly support multi-document polling.
- Build the raw bundle with fetched bodies and metadata.
- Emit one raw event with kind `outlook` and schema `standards.SchemaRawSPCConvectiveOutlookV1`.
- Use `fksources.DefaultEventID("", sourceName, effectiveAt, emittedAt)`.
Effective time policy:
- Prefer latest valid `ISSUE_ISO` across all GeoJSON features.
- Fallback to latest valid print-page `Updated:` timestamp.
- Fallback to RSS `lastBuildDate` if RSS was fetched and parseable.
- Fallback to fetch time if no provider timestamp is parseable.
- Keep fetch-time fallback as the last source effective-time option so downstream consumers still receive a stable effective time when provider metadata is absent.
Tests:
- Builtin registry builds `spc_convective_outlook` as a `PollSource`.
- `Kinds()` returns `outlook`.
- Constructor rejects missing `latitude` or `longitude`.
- Poll emits one raw event with schema `raw.spc.convective_outlook.v1`.
- Raw bundle includes 12 products and 3 discussions when defaults are overridden to test-server URLs.
- Poll effectiveAt prefers latest GeoJSON `ISSUE_ISO`.
- Poll effectiveAt falls back to print-page `Updated:` when GeoJSON issue times are absent.
- Poll includes RSS only when `rss_url` is configured.
- Unchanged response emits no events on a second poll.
- A configured GeoJSON fetch failure returns error and emits no event.
- A configured print-page fetch failure returns error and emits no event.
- Tests use `httptest.Server`; do not call live SPC services.
Verification:
```sh
go test ./internal/providers/spc ./internal/sources
```
Stage completion criteria:
- Source emits raw bundles only.
- No canonical model, normalizer, or sink mapping is added in this stage except schema constants required for compilation.
## Stage 4: Canonical Model And Normalizer
Goal: convert raw SPC bundles into `weather.outlook.v1` events.
Files to update:
- `standards/schema.go`
- `internal/normalizers/builtins_test.go`
Files to add:
- `model/outlook.go`
- `internal/normalizers/spc/convective_outlook.go`
- `internal/normalizers/spc/register.go`
- `internal/normalizers/spc/convective_outlook_test.go`
Files to update for registration:
- `internal/normalizers/builtins.go`
Normalizer behavior:
- Add `SPCConvectiveOutlookNormalizer`.
- Match only `standards.SchemaRawSPCConvectiveOutlookV1`.
- Decode `RawConvectiveOutlookBundle`.
- Build one `model.WeatherOutlookRun` per raw bundle.
- Map each GeoJSON feature to one `model.WeatherOutlook`.
- Preserve feature order within each product.
- Order products by day then outlook type in this order: `categorical`, `tornado`, `hail`, `wind`.
- Compute `containsLocation` with `internal/geo` using the configured bundle latitude and longitude.
- Store compact feature geometry JSON in `WeatherOutlook.Geometry`.
- Set output schema to `standards.SchemaWeatherOutlookV1`.
- Set output effectiveAt to run `AsOf`.
- Use `internal/normalizers/common.Finalize` or existing normalizer helper patterns so envelope handling remains consistent.
GeoJSON property mapping:
- `VALID_ISO` maps to `validFrom` and is required.
- `EXPIRE_ISO` maps to `validTo` and `expiresAt` and is required.
- `ISSUE_ISO` maps to `issuedAt` and is required.
- `FORECASTER` maps to `forecaster` and is optional.
- `LABEL` maps to `label` and is required.
- `LABEL2` maps to `labelText` and is optional.
- `DN` maps to `severityRank` and is optional.
Derived mapping:
- `provider` is `spc`.
- `product` is `convective`.
- `day` and `outlookType` come from raw product metadata, not from brittle URL parsing when product metadata is available.
- `id` is deterministic: join day, outlook type, normalized label, issuedAt UTC in RFC3339 format, validFrom UTC in RFC3339 format, and product-local feature index. Use a stable ASCII-safe format.
- `sourceUrl` is the GeoJSON product URL for geometry/probability features.
- `imageUrl` is empty for v1.
Discussion enrichment:
- Parse each print-page discussion with `internal/providers/spc` helpers.
- Attach Day 1 print-page headline, summary, and discussion to Day 1 outlooks, Day 2 to Day 2 outlooks, and Day 3 to Day 3 outlooks.
- Use the same discussion text for categorical, tornado, hail, and wind outlooks for the same day.
- If a print page is fetched but discussion extraction fails, normalization should fail because print pages are required v1 inputs.
- RSS metadata must not be required for discussion enrichment.
Run timestamp policy:
- `AsOf` is latest valid `issuedAt` across all outlook features.
- Fallback to latest print-page `Updated:` timestamp.
- Fallback to input event `EffectiveAt`.
- Fallback to input event `EmittedAt`.
- `IssuedAt` is latest valid `issuedAt` across all outlook features when any feature exists; otherwise nil.
Error behavior:
- Fail normalization if required GeoJSON timestamps are missing or unparseable.
- Fail normalization if required `LABEL` is empty.
- Fail normalization if required geometry is missing or invalid.
- Fail normalization if latitude or longitude is missing or invalid in the raw bundle.
- Include product key and feature index in errors.
Tests:
- Normalizer matches only `raw.spc.convective_outlook.v1`.
- Builtin normalizer order includes the SPC normalizer after existing provider normalizers unless a specific order is needed.
- Canonical schema is `weather.outlook.v1`.
- Categorical fixture maps expected day, type, label, label text, severity rank, valid times, issue time, forecaster, source URL, and geometry.
- Probabilistic fixtures for tornado, hail, and wind map expected `outlookType` values.
- `containsLocation` is true for a fixture point inside a polygon.
- `containsLocation` is false for a fixture point outside a polygon.
- Print-page text maps to headline, summary, and discussion.
- Day 2 correction marker `CORR 1` is preserved in headline or discussion metadata.
- Missing optional RSS still normalizes successfully.
- Invalid required timestamp fails with product key and feature index context.
- Invalid geometry fails with product key and feature index context.
- Output JSON uses the intended field names and does not expose raw bundle internals.
Verification:
```sh
go test ./model ./internal/geo ./internal/providers/spc ./internal/normalizers ./internal/normalizers/spc
```
Stage completion criteria:
- A raw SPC bundle normalizes into canonical `model.WeatherOutlookRun`.
- No Postgres schema or current-behavior docs are updated in this stage unless the canonical schema documentation is intentionally updated with implementation.
## Stage 5: Postgres Sink
Goal: persist canonical outlook events through the weatherfeeder Postgres sink.
Files to update:
- `internal/sinks/postgres/schema.go`
- `internal/sinks/postgres/map.go`
- `internal/sinks/postgres/schema_test.go`
- `internal/sinks/postgres/map_test.go`
- `internal/sinks/postgres/doc.go`
Tables:
- `outlook_runs`
- `outlooks`
Add table constants:
- `tableOutlookRuns = "outlook_runs"`
- `tableOutlooks = "outlooks"`
`outlook_runs` columns:
- `event_id TEXT NOT NULL`
- `event_kind TEXT NOT NULL`
- `event_source TEXT NOT NULL`
- `event_schema TEXT NOT NULL`
- `event_emitted_at TIMESTAMPTZ NOT NULL`
- `event_effective_at TIMESTAMPTZ NULL`
- `location_id TEXT NULL`
- `location_name TEXT NULL`
- `latitude DOUBLE PRECISION NULL`
- `longitude DOUBLE PRECISION NULL`
- `as_of TIMESTAMPTZ NOT NULL`
- `issued_at TIMESTAMPTZ NULL`
- `outlook_count INTEGER NOT NULL`
`outlook_runs` keys and indexes:
- Primary key: `event_id`
- Prune column: `as_of`
- Index `idx_wf_outlook_run_location_as_of` on `location_id, as_of`
- Index `idx_wf_outlook_run_as_of` on `as_of`
`outlooks` columns:
- `run_event_id TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE NOT NULL`
- `outlook_index INTEGER NOT NULL`
- `as_of TIMESTAMPTZ NOT NULL`
- `product TEXT NOT NULL`
- `day INTEGER NOT NULL`
- `outlook_type TEXT NOT NULL`
- `label TEXT NOT NULL`
- `label_text TEXT NULL`
- `severity_rank INTEGER NULL`
- `valid_from TIMESTAMPTZ NOT NULL`
- `valid_to TIMESTAMPTZ NOT NULL`
- `issued_at TIMESTAMPTZ NOT NULL`
- `expires_at TIMESTAMPTZ NOT NULL`
- `forecaster TEXT NULL`
- `headline TEXT NULL`
- `summary TEXT NULL`
- `discussion TEXT NULL`
- `source_url TEXT NULL`
- `image_url TEXT NULL`
- `contains_location BOOLEAN NOT NULL`
- `geometry_json TEXT NOT NULL`
`outlooks` keys and indexes:
- Primary key: `run_event_id, outlook_index`
- Prune column: `as_of`
- Index `idx_wf_outlooks_contains_valid` on `contains_location, valid_from, valid_to`
- Index `idx_wf_outlooks_day_type_label` on `day, outlook_type, label`
- Index `idx_wf_outlooks_valid` on `valid_from, valid_to`
Mapper behavior:
- Extend `mapPostgresEvent` for `standards.SchemaWeatherOutlookV1`.
- Decode `model.WeatherOutlookRun`.
- Require run `AsOf`.
- Map envelope columns exactly like existing parent run tables.
- Store all times as UTC.
- Write one `outlook_runs` row and one `outlooks` row per outlook.
- Use `outlook_index` as the zero-based slice index.
- Require outlook `ID`, `Provider`, `Product`, `Day`, `OutlookType`, `Label`, `ValidFrom`, `ValidTo`, `IssuedAt`, `ExpiresAt`, and non-empty `Geometry`.
- Store compact geometry JSON text in `geometry_json` using the existing compact JSON helper or a similar helper.
- Preserve all outlook polygons, not only those containing the configured point.
Tests:
- Schema includes `outlook_runs` and `outlooks`.
- Schema includes required columns and indexes.
- Mapper writes one run row plus one row per outlook.
- Mapper stores UTC times.
- Mapper stores compact geometry JSON.
- Mapper preserves `contains_location` false as false, not nil.
- Mapper rejects missing run `asOf`.
- Mapper rejects missing required outlook times.
- Mapper rejects empty geometry.
- Mapper ignores unrelated schemas unchanged.
Verification:
```sh
go test ./internal/sinks/postgres
```
Stage completion criteria:
- Postgres schema and mapper tests pass.
- No weatherapi read path is added in this stage.
## Stage 6: Config, Runtime Wiring, And Current-Behavior Docs
Goal: expose the completed source/normalizer/sink behavior in maintained configuration and documentation after implementation exists.
Files to update:
- `cmd/weatherfeeder/config.yml`
- `cmd/weatherfeeder/main_test.go`, if config load expectations need updates
- `docs/config.md`
- `docs/integrations/events.md`
- `docs/integrations/postgres.md`
- `docs/internal/sources.md`
- `docs/internal/normalizers.md`
- `docs/internal/postgres-sink.md`
- `README.md`
Config changes:
- Add sample source `SPCConvectiveOutlookSTL` with driver `spc_convective_outlook`, kind `outlook`, `every: 30m`, latitude, longitude, location metadata, and `user_agent`.
- Add `outlook` route examples for stdout, NATS, and Postgres where the existing config style includes kind lists.
- Keep optional RSS disabled in sample config unless there is a specific operator reason to include it.
Docs changes:
- Update current-behavior docs only after the code for that behavior exists.
- `docs/config.md` should document required and optional source params.
- `docs/integrations/events.md` should document `weather.outlook.v1`, including field definitions, required fields, optional fields, geometry semantics, and `containsLocation` semantics.
- `docs/integrations/postgres.md` should document `outlook_runs` and `outlooks`.
- Internal docs should explain provider boundaries, print-page discussion parsing, and point-in-polygon behavior where useful for maintainers.
- `README.md` should only briefly list SPC convective outlook support and link to canonical docs.
- Do not document weatherapi endpoints in weatherfeeder current-behavior docs.
Tests:
- Existing config load tests pass.
- Add config test coverage if the sample config is expected to build scheduler jobs for the new source.
- Documentation examples should use the real driver name and event kind.
Verification:
```sh
go test ./cmd/weatherfeeder ./internal/sources ./internal/normalizers ./internal/sinks/postgres
```
Stage completion criteria:
- Sample config remains loadable.
- Current-behavior docs match implemented code.
- No roadmap-only claims leak into non-roadmap docs beyond the implemented behavior.
## Stage 7: Full Verification And Cleanup
Goal: validate the complete feature and remove implementation-only rough edges.
Run focused tests:
```sh
go test ./internal/providers/spc ./internal/geo ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./model ./cmd/weatherfeeder
```
Run full tests:
Verification commands:
```sh
go test ./internal/providers/spc ./internal/sources/spc ./internal/normalizers/spc ./internal/sinks/postgres
go test ./...
```
Manual review checklist:
## Assumptions
- `standards/schema.go` contains raw and canonical SPC schema constants.
- `model/outlook.go` uses stable JSON tags and no provider-specific names except canonical strings.
- Source registry includes `spc_convective_outlook`.
- Normalizer registry includes the SPC normalizer.
- Source tests do not use live SPC services.
- Normalizer tests use fixtures and cover discussion parsing, geometry, and timestamp failures.
- Postgres tests cover schema shape and mapper validation.
- Docs follow `docs/policy/documentation.md` and use canonical homes.
- No weatherapi files are changed.
- No broad dependency was added without clear justification.
Acceptance criteria:
- A configured SPC source emits one raw bundled `outlook` event per changed complete product snapshot.
- Normalization produces `weather.outlook.v1` events.
- All Day 1-3 categorical, tornado, hail, and wind products are represented.
- Each outlook includes `containsLocation` for the configured latitude and longitude.
- Each outlook preserves compact GeoJSON geometry.
- Day 1-3 print-page discussion text is preserved in canonical headline, summary, and discussion fields where parseable.
- RSS metadata is optional and supplemental only.
- Postgres sink persists outlook runs and outlook rows.
- Sample config and current-behavior docs describe the implemented kind, driver, schema, and storage contract.
## Suggested Prompt Boundaries
This feature is too broad for a single safe implementation prompt. Use these implementation prompts in order:
1. Implement Stage 1 provider helpers and fixtures only.
2. Implement Stage 2 geometry helper only.
3. Implement Stage 3 source driver and raw schema only.
4. Implement Stage 4 canonical model and normalizer only.
5. Implement Stage 5 Postgres sink mapping only.
6. Implement Stage 6 config and current-behavior documentation only.
7. Run Stage 7 verification and perform targeted fixes only.
Each prompt should run its stage-specific tests before moving on. Do not proceed to the next stage with failing tests unless the failure is unrelated and explicitly documented.
- The canonical API shape remains unchanged: `WeatherOutlook.id` and `WeatherOutlook.provider` are already required fields.
- Postgres persistence should be lossless for required canonical outlook fields.
- Existing database migration commands are intentionally omitted from this roadmap.
- RSS remains optional supplemental metadata only.
- This remediation plan does not include weatherapi changes.