Files
weatherfeeder/docs/roadmap/implementation.md
2026-06-10 18:59:16 -05:00

28 KiB

SPC Convective Outlook Implementation Plan

Purpose

Implement weatherfeeder support for Storm Prediction Center Day 1-3 convective outlooks described in docs/roadmap/spc.md. This plan is written for an LLM coding agent and should be followed stage by stage.

This is a planning document only. The implementation must preserve the existing weatherfeeder architecture: sources emit raw provider events, normalizers map raw payloads into canonical model types, and sinks persist canonical schemas.

Decisions

The following choices are fixed for this implementation:

  • Scope is Day 1-3 SPC convective outlooks only.
  • Day 4-8 outlooks are out of scope.
  • GeoJSON files are authoritative for polygons, validity windows, issue times, outlook labels, and severity ranking.
  • Day 1-3 print pages are authoritative for discussion text.
  • RSS is optional supplemental metadata only and must not be required for correctness.
  • Do not fetch RSS by default. Include RSS only when an optional rss_url source param is configured.
  • A poll is atomic for required products. If any configured GeoJSON or print-page URL fails or returns a non-2xx response, return an error and emit no event.
  • Use compact GeoJSON geometry in the canonical payload for auditability and downstream display.
  • Use standard-library-first parsing. Do not add an HTML parsing dependency unless string-based extraction proves unmaintainable during implementation.
  • Keep all new planned behavior inside weatherfeeder; do not make weatherapi changes in this pass.

Public Contract

Add schema constants in standards/schema.go:

  • SchemaRawSPCConvectiveOutlookV1 = "raw.spc.convective_outlook.v1"
  • SchemaWeatherOutlookV1 = "weather.outlook.v1"

Add source driver:

  • spc_convective_outlook

Add event kind:

  • outlook

Add canonical model types:

  • model.WeatherOutlookRun
  • model.WeatherOutlook

Canonical run fields:

type WeatherOutlookRun struct {
    LocationID   string           `json:"locationId,omitempty"`
    LocationName string           `json:"locationName,omitempty"`
    Latitude     *float64         `json:"latitude,omitempty"`
    Longitude    *float64         `json:"longitude,omitempty"`
    AsOf         time.Time        `json:"asOf"`
    IssuedAt     *time.Time       `json:"issuedAt,omitempty"`
    Outlooks     []WeatherOutlook `json:"outlooks"`
}

Canonical outlook fields:

type WeatherOutlook struct {
    ID               string          `json:"id"`
    Provider         string          `json:"provider"`
    Product          string          `json:"product"`
    Day              int             `json:"day"`
    OutlookType      string          `json:"outlookType"`
    Label            string          `json:"label"`
    LabelText        string          `json:"labelText,omitempty"`
    SeverityRank     *int            `json:"severityRank,omitempty"`
    ValidFrom        time.Time       `json:"validFrom"`
    ValidTo          time.Time       `json:"validTo"`
    IssuedAt         time.Time       `json:"issuedAt"`
    ExpiresAt        time.Time       `json:"expiresAt"`
    Forecaster       string          `json:"forecaster,omitempty"`
    Headline         string          `json:"headline,omitempty"`
    Summary          string          `json:"summary,omitempty"`
    Discussion       string          `json:"discussion,omitempty"`
    SourceURL        string          `json:"sourceUrl,omitempty"`
    ImageURL         string          `json:"imageUrl,omitempty"`
    ContainsLocation bool            `json:"containsLocation"`
    Geometry         json.RawMessage `json:"geometry"`
}

Required canonical fields:

  • Run: asOf, outlooks.
  • Outlook: id, provider, product, day, outlookType, label, validFrom, validTo, issuedAt, expiresAt, containsLocation, geometry.

Canonical values:

  • provider is spc.
  • product is convective.
  • outlookType is one of categorical, tornado, hail, wind.
  • day is one of 1, 2, 3.

Source Inputs

Default required GeoJSON products:

  • https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson
  • https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson
  • https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson
  • https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson
  • https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson
  • https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson
  • https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson
  • https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson
  • https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson
  • https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson
  • https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson
  • https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson

Default required print-page products:

  • https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html
  • https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html
  • https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html

Optional RSS product:

  • https://www.spc.noaa.gov/products/spcacrss.xml

Recommended config shape:

- name: SPCConvectiveOutlookSTL
  mode: poll
  kinds: ["outlook"]
  driver: spc_convective_outlook
  every: 30m
  params:
    latitude: 38.6239
    longitude: -90.3571
    location_id: "stl"
    location_name: "St. Louis, MO"
    user_agent: "HomeOps (eric@maximumdirect.net)"

Optional source params:

  • geojson_urls: map from product key to URL, used by tests and future upstream changes.
  • discussion_urls: map from day key to URL, used by tests and future upstream changes.
  • rss_url: string; when non-empty, fetch RSS as supplemental metadata.

Product keys for geojson_urls:

  • day1_categorical, day1_tornado, day1_hail, day1_wind
  • day2_categorical, day2_tornado, day2_hail, day2_wind
  • day3_categorical, day3_tornado, day3_hail, day3_wind

Discussion keys for discussion_urls:

  • day1, day2, day3

Raw Bundle Shape

Create a provider raw bundle type under internal/providers/spc or internal/normalizers/spc and use it consistently between source tests and normalizer tests. Prefer internal/providers/spc if source metadata extraction and normalizer parsing share helpers.

Suggested raw payload shape:

type RawConvectiveOutlookBundle struct {
    LocationID   string              `json:"locationId,omitempty"`
    LocationName string              `json:"locationName,omitempty"`
    Latitude     float64             `json:"latitude"`
    Longitude    float64             `json:"longitude"`
    FetchedAt    time.Time           `json:"fetchedAt"`
    Products     []RawOutlookProduct `json:"products"`
    Discussions  []RawDiscussionPage `json:"discussions"`
    RSS          *RawRSSFeed         `json:"rss,omitempty"`
}
type RawOutlookProduct struct {
    Key         string          `json:"key"`
    Day         int             `json:"day"`
    OutlookType string          `json:"outlookType"`
    URL         string          `json:"url"`
    FetchedAt   time.Time       `json:"fetchedAt"`
    Body        json.RawMessage `json:"body"`
}
type RawDiscussionPage struct {
    Key       string    `json:"key"`
    Day       int       `json:"day"`
    URL       string    `json:"url"`
    FetchedAt time.Time `json:"fetchedAt"`
    Body      string    `json:"body"`
}
type RawRSSFeed struct {
    URL       string    `json:"url"`
    FetchedAt time.Time `json:"fetchedAt"`
    Body      string    `json:"body"`
}

Do not put parsed canonical fields into the raw bundle except configured metadata and product keys needed to identify each fetched upstream document. Source-level timestamp parsing is allowed only for event effectiveAt selection.

Stage 1: Provider Helpers And Fixtures

Goal: add deterministic SPC parsing primitives and test fixtures before wiring the source or normalizer.

Files to add:

  • internal/providers/spc/doc.go
  • internal/providers/spc/time.go
  • internal/providers/spc/product.go
  • internal/providers/spc/geojson.go
  • internal/providers/spc/discussion.go
  • internal/providers/spc/rss.go, only if optional RSS parsing is implemented
  • internal/providers/spc/testdata/day1_cat.geojson
  • internal/providers/spc/testdata/day2_torn.geojson
  • internal/providers/spc/testdata/day3_wind.geojson
  • internal/providers/spc/testdata/day1_prt.html
  • internal/providers/spc/testdata/day2_prt_corr.html
  • internal/providers/spc/testdata/day3_prt.html

Provider helper behavior:

  • Define stable product metadata for the 12 required GeoJSON products.
  • Define stable discussion metadata for the 3 required print-page products.
  • Parse SPC ISO timestamps from GeoJSON properties using time.Parse(time.RFC3339, value) after trimming whitespace.
  • Decode enough GeoJSON to expose feature properties and raw geometry without owning canonical mapping.
  • Preserve raw geometry as compact JSON bytes for later canonical use.
  • Extract print-page product text from the first useful <pre> block.
  • Strip embedded <script> blocks and remaining tags from extracted <pre> content.
  • Use html.UnescapeString, normalize CRLF to LF, and trim surrounding blank lines.
  • Parse print-page Updated: timestamps when present.
  • Parse discussion headline/product title from product text.
  • Parse ...SUMMARY... content through the next section heading.
  • Preserve full product text as discussion text after cleanup.
  • Preserve correction markers such as CORR 1 in headline or metadata when present.

Dependency rule:

  • Start with strings, regexp, encoding/json, encoding/xml, and html from the standard library.
  • Do not add golang.org/x/net/html or another HTML parser unless tests show the string extraction is too brittle.

Tests:

  • go test ./internal/providers/spc
  • Product metadata contains exactly 12 GeoJSON products in day/type order.
  • Discussion metadata contains exactly 3 print pages in day order.
  • GeoJSON fixture decode exposes VALID_ISO, EXPIRE_ISO, ISSUE_ISO, FORECASTER, LABEL, LABEL2, DN, and geometry.
  • Print-page fixture extraction returns product text without scripts or tags.
  • Day 2 correction fixture preserves CORR 1.
  • Summary extraction returns only the summary paragraph content.
  • Updated timestamp parser returns UTC time when present and nil when absent.

Stage completion criteria:

  • Provider helper tests pass.
  • No source, normalizer, model, standards, sink, config, or docs current-behavior files are changed in this stage unless needed for package compilation.

Stage 2: Geometry Helper

Goal: implement point-in-polygon support independent of SPC parsing.

Files to add:

  • internal/geo/geojson.go
  • internal/geo/point.go
  • internal/geo/point_test.go

Behavior:

  • Support GeoJSON Polygon and MultiPolygon only.
  • Accept geometry as json.RawMessage or []byte.
  • Interpret GeoJSON coordinate order as [longitude, latitude].
  • Treat the first ring as the exterior ring.
  • Treat subsequent rings as holes.
  • Count boundary points as inside.
  • Return a clear error for unsupported geometry types, malformed coordinates, empty rings, or invalid JSON.
  • Use planar ray casting. This is sufficient for operational point-in-polygon checks at SPC polygon scale.

Tests:

  • Point inside simple polygon returns true.
  • Point outside simple polygon returns false.
  • Point on polygon boundary returns true.
  • Point in a hole returns false.
  • Point inside one MultiPolygon member returns true.
  • Longitude/latitude ordering regression test fails if coordinates are reversed.
  • Unsupported geometry returns a useful error.

Verification:

go test ./internal/geo

Stage completion criteria:

  • Geometry helper has no dependency on SPC, source, normalizer, or sink packages.

Stage 3: Source Driver And Raw Schema

Goal: emit raw SPC outlook bundles from a new poll source.

Files to update:

  • standards/schema.go
  • internal/sources/builtins.go
  • internal/sources/builtins_test.go

Files to add:

  • internal/sources/spc/convective_outlook.go
  • internal/sources/spc/convective_outlook_test.go

Source constructor:

  • Export NewConvectiveOutlookSource(cfg config.SourceConfig) (*ConvectiveOutlookSource, error).
  • Register driver spc_convective_outlook in internal/sources/builtins.go.
  • Validate required params latitude and longitude.
  • Accept optional params location_id, location_name, geojson_urls, discussion_urls, and rss_url.
  • Require user_agent when the existing HTTP source conventions require it.
  • Advertise Kinds() []event.Kind{event.Kind("outlook")}.

Fetch behavior:

  • Fetch all required GeoJSON URLs and print-page URLs every poll cycle.
  • Fetch optional RSS only when rss_url is configured and non-empty.
  • Use Accept application/geo+json, application/json for GeoJSON requests.
  • Use Accept text/html, application/xhtml+xml for print pages.
  • Use Accept application/rss+xml, application/xml, text/xml for RSS.
  • Respect context.Context on all requests.
  • Do not emit partial bundles.
  • If the complete raw bundle is unchanged from the previous successful poll, return no events. Implement source-local change detection by hashing the combined fetched bodies if feedkit HTTP conditional helpers do not directly support multi-document polling.
  • Build the raw bundle with fetched bodies and metadata.
  • Emit one raw event with kind outlook and schema standards.SchemaRawSPCConvectiveOutlookV1.
  • Use fksources.DefaultEventID("", sourceName, effectiveAt, emittedAt).

Effective time policy:

  • Prefer latest valid ISSUE_ISO across all GeoJSON features.
  • Fallback to latest valid print-page Updated: timestamp.
  • Fallback to RSS lastBuildDate if RSS was fetched and parseable.
  • Fallback to fetch time if no provider timestamp is parseable.
  • Keep fetch-time fallback as the last source effective-time option so downstream consumers still receive a stable effective time when provider metadata is absent.

Tests:

  • Builtin registry builds spc_convective_outlook as a PollSource.
  • Kinds() returns outlook.
  • Constructor rejects missing latitude or longitude.
  • Poll emits one raw event with schema raw.spc.convective_outlook.v1.
  • Raw bundle includes 12 products and 3 discussions when defaults are overridden to test-server URLs.
  • Poll effectiveAt prefers latest GeoJSON ISSUE_ISO.
  • Poll effectiveAt falls back to print-page Updated: when GeoJSON issue times are absent.
  • Poll includes RSS only when rss_url is configured.
  • Unchanged response emits no events on a second poll.
  • A configured GeoJSON fetch failure returns error and emits no event.
  • A configured print-page fetch failure returns error and emits no event.
  • Tests use httptest.Server; do not call live SPC services.

Verification:

go test ./internal/providers/spc ./internal/sources

Stage completion criteria:

  • Source emits raw bundles only.
  • No canonical model, normalizer, or sink mapping is added in this stage except schema constants required for compilation.

Stage 4: Canonical Model And Normalizer

Goal: convert raw SPC bundles into weather.outlook.v1 events.

Files to update:

  • standards/schema.go
  • internal/normalizers/builtins_test.go

Files to add:

  • model/outlook.go
  • internal/normalizers/spc/convective_outlook.go
  • internal/normalizers/spc/register.go
  • internal/normalizers/spc/convective_outlook_test.go

Files to update for registration:

  • internal/normalizers/builtins.go

Normalizer behavior:

  • Add SPCConvectiveOutlookNormalizer.
  • Match only standards.SchemaRawSPCConvectiveOutlookV1.
  • Decode RawConvectiveOutlookBundle.
  • Build one model.WeatherOutlookRun per raw bundle.
  • Map each GeoJSON feature to one model.WeatherOutlook.
  • Preserve feature order within each product.
  • Order products by day then outlook type in this order: categorical, tornado, hail, wind.
  • Compute containsLocation with internal/geo using the configured bundle latitude and longitude.
  • Store compact feature geometry JSON in WeatherOutlook.Geometry.
  • Set output schema to standards.SchemaWeatherOutlookV1.
  • Set output effectiveAt to run AsOf.
  • Use internal/normalizers/common.Finalize or existing normalizer helper patterns so envelope handling remains consistent.

GeoJSON property mapping:

  • VALID_ISO maps to validFrom and is required.
  • EXPIRE_ISO maps to validTo and expiresAt and is required.
  • ISSUE_ISO maps to issuedAt and is required.
  • FORECASTER maps to forecaster and is optional.
  • LABEL maps to label and is required.
  • LABEL2 maps to labelText and is optional.
  • DN maps to severityRank and is optional.

Derived mapping:

  • provider is spc.
  • product is convective.
  • day and outlookType come from raw product metadata, not from brittle URL parsing when product metadata is available.
  • id is deterministic: join day, outlook type, normalized label, issuedAt UTC in RFC3339 format, validFrom UTC in RFC3339 format, and product-local feature index. Use a stable ASCII-safe format.
  • sourceUrl is the GeoJSON product URL for geometry/probability features.
  • imageUrl is empty for v1.

Discussion enrichment:

  • Parse each print-page discussion with internal/providers/spc helpers.
  • Attach Day 1 print-page headline, summary, and discussion to Day 1 outlooks, Day 2 to Day 2 outlooks, and Day 3 to Day 3 outlooks.
  • Use the same discussion text for categorical, tornado, hail, and wind outlooks for the same day.
  • If a print page is fetched but discussion extraction fails, normalization should fail because print pages are required v1 inputs.
  • RSS metadata must not be required for discussion enrichment.

Run timestamp policy:

  • AsOf is latest valid issuedAt across all outlook features.
  • Fallback to latest print-page Updated: timestamp.
  • Fallback to input event EffectiveAt.
  • Fallback to input event EmittedAt.
  • IssuedAt is latest valid issuedAt across all outlook features when any feature exists; otherwise nil.

Error behavior:

  • Fail normalization if required GeoJSON timestamps are missing or unparseable.
  • Fail normalization if required LABEL is empty.
  • Fail normalization if required geometry is missing or invalid.
  • Fail normalization if latitude or longitude is missing or invalid in the raw bundle.
  • Include product key and feature index in errors.

Tests:

  • Normalizer matches only raw.spc.convective_outlook.v1.
  • Builtin normalizer order includes the SPC normalizer after existing provider normalizers unless a specific order is needed.
  • Canonical schema is weather.outlook.v1.
  • Categorical fixture maps expected day, type, label, label text, severity rank, valid times, issue time, forecaster, source URL, and geometry.
  • Probabilistic fixtures for tornado, hail, and wind map expected outlookType values.
  • containsLocation is true for a fixture point inside a polygon.
  • containsLocation is false for a fixture point outside a polygon.
  • Print-page text maps to headline, summary, and discussion.
  • Day 2 correction marker CORR 1 is preserved in headline or discussion metadata.
  • Missing optional RSS still normalizes successfully.
  • Invalid required timestamp fails with product key and feature index context.
  • Invalid geometry fails with product key and feature index context.
  • Output JSON uses the intended field names and does not expose raw bundle internals.

Verification:

go test ./model ./internal/geo ./internal/providers/spc ./internal/normalizers ./internal/normalizers/spc

Stage completion criteria:

  • A raw SPC bundle normalizes into canonical model.WeatherOutlookRun.
  • No Postgres schema or current-behavior docs are updated in this stage unless the canonical schema documentation is intentionally updated with implementation.

Stage 5: Postgres Sink

Goal: persist canonical outlook events through the weatherfeeder Postgres sink.

Files to update:

  • internal/sinks/postgres/schema.go
  • internal/sinks/postgres/map.go
  • internal/sinks/postgres/schema_test.go
  • internal/sinks/postgres/map_test.go
  • internal/sinks/postgres/doc.go

Tables:

  • outlook_runs
  • outlooks

Add table constants:

  • tableOutlookRuns = "outlook_runs"
  • tableOutlooks = "outlooks"

outlook_runs columns:

  • event_id TEXT NOT NULL
  • event_kind TEXT NOT NULL
  • event_source TEXT NOT NULL
  • event_schema TEXT NOT NULL
  • event_emitted_at TIMESTAMPTZ NOT NULL
  • event_effective_at TIMESTAMPTZ NULL
  • location_id TEXT NULL
  • location_name TEXT NULL
  • latitude DOUBLE PRECISION NULL
  • longitude DOUBLE PRECISION NULL
  • as_of TIMESTAMPTZ NOT NULL
  • issued_at TIMESTAMPTZ NULL
  • outlook_count INTEGER NOT NULL

outlook_runs keys and indexes:

  • Primary key: event_id
  • Prune column: as_of
  • Index idx_wf_outlook_run_location_as_of on location_id, as_of
  • Index idx_wf_outlook_run_as_of on as_of

outlooks columns:

  • run_event_id TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE NOT NULL
  • outlook_index INTEGER NOT NULL
  • as_of TIMESTAMPTZ NOT NULL
  • product TEXT NOT NULL
  • day INTEGER NOT NULL
  • outlook_type TEXT NOT NULL
  • label TEXT NOT NULL
  • label_text TEXT NULL
  • severity_rank INTEGER NULL
  • valid_from TIMESTAMPTZ NOT NULL
  • valid_to TIMESTAMPTZ NOT NULL
  • issued_at TIMESTAMPTZ NOT NULL
  • expires_at TIMESTAMPTZ NOT NULL
  • forecaster TEXT NULL
  • headline TEXT NULL
  • summary TEXT NULL
  • discussion TEXT NULL
  • source_url TEXT NULL
  • image_url TEXT NULL
  • contains_location BOOLEAN NOT NULL
  • geometry_json TEXT NOT NULL

outlooks keys and indexes:

  • Primary key: run_event_id, outlook_index
  • Prune column: as_of
  • Index idx_wf_outlooks_contains_valid on contains_location, valid_from, valid_to
  • Index idx_wf_outlooks_day_type_label on day, outlook_type, label
  • Index idx_wf_outlooks_valid on valid_from, valid_to

Mapper behavior:

  • Extend mapPostgresEvent for standards.SchemaWeatherOutlookV1.
  • Decode model.WeatherOutlookRun.
  • Require run AsOf.
  • Map envelope columns exactly like existing parent run tables.
  • Store all times as UTC.
  • Write one outlook_runs row and one outlooks row per outlook.
  • Use outlook_index as the zero-based slice index.
  • Require outlook ID, Provider, Product, Day, OutlookType, Label, ValidFrom, ValidTo, IssuedAt, ExpiresAt, and non-empty Geometry.
  • Store compact geometry JSON text in geometry_json using the existing compact JSON helper or a similar helper.
  • Preserve all outlook polygons, not only those containing the configured point.

Tests:

  • Schema includes outlook_runs and outlooks.
  • Schema includes required columns and indexes.
  • Mapper writes one run row plus one row per outlook.
  • Mapper stores UTC times.
  • Mapper stores compact geometry JSON.
  • Mapper preserves contains_location false as false, not nil.
  • Mapper rejects missing run asOf.
  • Mapper rejects missing required outlook times.
  • Mapper rejects empty geometry.
  • Mapper ignores unrelated schemas unchanged.

Verification:

go test ./internal/sinks/postgres

Stage completion criteria:

  • Postgres schema and mapper tests pass.
  • No weatherapi read path is added in this stage.

Stage 6: Config, Runtime Wiring, And Current-Behavior Docs

Goal: expose the completed source/normalizer/sink behavior in maintained configuration and documentation after implementation exists.

Files to update:

  • cmd/weatherfeeder/config.yml
  • cmd/weatherfeeder/main_test.go, if config load expectations need updates
  • docs/config.md
  • docs/integrations/events.md
  • docs/integrations/postgres.md
  • docs/internal/sources.md
  • docs/internal/normalizers.md
  • docs/internal/postgres-sink.md
  • README.md

Config changes:

  • Add sample source SPCConvectiveOutlookSTL with driver spc_convective_outlook, kind outlook, every: 30m, latitude, longitude, location metadata, and user_agent.
  • Add outlook route examples for stdout, NATS, and Postgres where the existing config style includes kind lists.
  • Keep optional RSS disabled in sample config unless there is a specific operator reason to include it.

Docs changes:

  • Update current-behavior docs only after the code for that behavior exists.
  • docs/config.md should document required and optional source params.
  • docs/integrations/events.md should document weather.outlook.v1, including field definitions, required fields, optional fields, geometry semantics, and containsLocation semantics.
  • docs/integrations/postgres.md should document outlook_runs and outlooks.
  • Internal docs should explain provider boundaries, print-page discussion parsing, and point-in-polygon behavior where useful for maintainers.
  • README.md should only briefly list SPC convective outlook support and link to canonical docs.
  • Do not document weatherapi endpoints in weatherfeeder current-behavior docs.

Tests:

  • Existing config load tests pass.
  • Add config test coverage if the sample config is expected to build scheduler jobs for the new source.
  • Documentation examples should use the real driver name and event kind.

Verification:

go test ./cmd/weatherfeeder ./internal/sources ./internal/normalizers ./internal/sinks/postgres

Stage completion criteria:

  • Sample config remains loadable.
  • Current-behavior docs match implemented code.
  • No roadmap-only claims leak into non-roadmap docs beyond the implemented behavior.

Stage 7: Full Verification And Cleanup

Goal: validate the complete feature and remove implementation-only rough edges.

Run focused tests:

go test ./internal/providers/spc ./internal/geo ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./model ./cmd/weatherfeeder

Run full tests:

go test ./...

Manual review checklist:

  • standards/schema.go contains raw and canonical SPC schema constants.
  • model/outlook.go uses stable JSON tags and no provider-specific names except canonical strings.
  • Source registry includes spc_convective_outlook.
  • Normalizer registry includes the SPC normalizer.
  • Source tests do not use live SPC services.
  • Normalizer tests use fixtures and cover discussion parsing, geometry, and timestamp failures.
  • Postgres tests cover schema shape and mapper validation.
  • Docs follow docs/policy/documentation.md and use canonical homes.
  • No weatherapi files are changed.
  • No broad dependency was added without clear justification.

Acceptance criteria:

  • A configured SPC source emits one raw bundled outlook event per changed complete product snapshot.
  • Normalization produces weather.outlook.v1 events.
  • All Day 1-3 categorical, tornado, hail, and wind products are represented.
  • Each outlook includes containsLocation for the configured latitude and longitude.
  • Each outlook preserves compact GeoJSON geometry.
  • Day 1-3 print-page discussion text is preserved in canonical headline, summary, and discussion fields where parseable.
  • RSS metadata is optional and supplemental only.
  • Postgres sink persists outlook runs and outlook rows.
  • Sample config and current-behavior docs describe the implemented kind, driver, schema, and storage contract.

Suggested Prompt Boundaries

This feature is too broad for a single safe implementation prompt. Use these implementation prompts in order:

  1. Implement Stage 1 provider helpers and fixtures only.
  2. Implement Stage 2 geometry helper only.
  3. Implement Stage 3 source driver and raw schema only.
  4. Implement Stage 4 canonical model and normalizer only.
  5. Implement Stage 5 Postgres sink mapping only.
  6. Implement Stage 6 config and current-behavior documentation only.
  7. Run Stage 7 verification and perform targeted fixes only.

Each prompt should run its stage-specific tests before moving on. Do not proceed to the next stage with failing tests unless the failure is unrelated and explicitly documented.