From 0041845935b5f80d0f293dd8017cd8b46dae0261 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 12 Jun 2026 14:51:54 +0000 Subject: [PATCH] Fetch SPC convective outlook data --- docs/config.md | 4 +- docs/integrations/weatherapi.md | 18 ++- docs/internal/weather-data.md | 18 +-- internal/adapters/weatherapi/client.go | 31 +++++ internal/adapters/weatherapi/client_test.go | 109 ++++++++++++++++-- .../testdata/convective_outlooks.json | 50 ++++++++ internal/app/app_test.go | 4 + internal/cli/root_test.go | 2 + 8 files changed, 216 insertions(+), 20 deletions(-) create mode 100644 internal/adapters/weatherapi/testdata/convective_outlooks.json diff --git a/docs/config.md b/docs/config.md index 94e7506..b05a17e 100644 --- a/docs/config.md +++ b/docs/config.md @@ -130,7 +130,9 @@ name the variable only; they should not contain the token value. - `sources`: optional map of source-specific overrides, using the same policy values. Hourly forecast data is required for generated reports. Optional sources use -the missing-source policy. +the missing-source policy. Source override keys include `observations`, +`current`, `narrative`, `alerts`, `discussion`, `weather_story`, and +`spc_convective_outlooks`. ### `scriptorium` diff --git a/docs/integrations/weatherapi.md b/docs/integrations/weatherapi.md index 0b4aea8..4b9a409 100644 --- a/docs/integrations/weatherapi.md +++ b/docs/integrations/weatherapi.md @@ -37,6 +37,10 @@ generation. means the endpoint was checked and there are no current active alerts. The adapter records a non-missing alerts source and an empty alert run. +For `/outlooks/convective`, `data: null` means no latest run is available and +follows missing-source policy. A non-null run with empty `outlooks` and +`discussions` arrays is checked empty data, not a missing source. + Malformed JSON envelopes, non-2xx statuses, and response read failures include endpoint context in returned errors. Decode errors include source context when they fail the fetch; optional malformed sources follow the missing-source policy. @@ -49,11 +53,12 @@ The adapter sends these query parameters: - `units`: from `weather_api.units` - `precision`: from `weather_api.precision` on observations, current conditions, hourly forecast, and narrative forecast requests -- `tz`: from `weather_api.timezone` on hourly forecast, narrative forecast, and - discussion requests +- `tz`: from `weather_api.timezone` on hourly forecast, narrative forecast, + discussion, and SPC convective outlook requests Alerts do not receive `precision` or `tz`. Weather story requests receive only -`format=json`. +`format=json`. SPC convective outlook requests receive only `format=json` and +`tz`; they do not receive `units` or `precision`. ## Endpoints Used @@ -66,6 +71,7 @@ The adapter fetches these endpoints once per bundle: - `/alerts/active` - `/discussion` - `/weatherstories/latest` +- `/outlooks/convective` `weatherreporter` does not call day-slice forecast endpoints or discussion subsection endpoints. Report-period selection and daypart summarization happen @@ -88,6 +94,7 @@ source-specific `missing_source.sources` policy: - `alerts` for `/alerts/active` - `discussion` for `/discussion` - `weather_story` for `/weatherstories/latest` +- `spc_convective_outlooks` for `/outlooks/convective` Policy behavior: @@ -99,6 +106,9 @@ For `/alerts/active`, an HTTP error or missing `data` field still fails or follows the relevant error path, but explicit `data: null` is not a missing-source condition. +For `/outlooks/convective`, a non-null data object with empty outlook and +discussion arrays is accepted as checked empty data. + ## Source Identity For source payloads accepted into the bundle, including the explicit `null` @@ -126,6 +136,8 @@ types in `internal/forecast/bundle.go`, including: - discussion metadata, key messages, and short/long-term section text - latest weather story title, description, timing, priority, order, alt text, and download URL +- SPC convective outlook run metadata, outlooks, discussions, and GeoJSON + geometry The adapter intentionally keeps upstream transport and envelope details inside `internal/adapters/weatherapi`; downstream packages consume the normalized diff --git a/docs/internal/weather-data.md b/docs/internal/weather-data.md index 594ac95..5857837 100644 --- a/docs/internal/weather-data.md +++ b/docs/internal/weather-data.md @@ -4,10 +4,10 @@ This document describes Weather API ingestion into `weatherdata.Bundle`. ## Purpose -`internal/adapters/weatherapi` fetches normalized weather data from one -configured Weather API endpoint and assembles the bundle consumed by forecast -derivation and module builders. Module builders expose normalized current -conditions and weather story context when those sources are available. +`internal/adapters/weatherapi` fetches normalized weather data from the +configured Weather API and assembles the bundle consumed by forecast derivation +and module builders. Module builders expose normalized current conditions and +weather story context when those sources are available. ## Inputs And Outputs @@ -21,8 +21,8 @@ Outputs: - `weatherdata.Bundle` with observation, current conditions, hourly forecast, narrative forecast, active alerts, discussion, latest weather story, source - records, source warnings, and an optional typed SPC convective outlook field - when that source has been populated + records, source warnings, and typed SPC convective outlook data when that + optional source is available - optional saved bundle JSON through app fetch helpers ## Boundaries @@ -55,7 +55,9 @@ contract used by this project. The adapter records source name, endpoint, query, fetch time, source timestamps when available, SHA-256 hash over compact raw `data` JSON, missing status, and source warnings. Successful `data: null` responses from `/alerts/active` -represent a checked empty active-alert list, not a missing source. +represent a checked empty active-alert list, not a missing source. Successful +non-null `/outlooks/convective` responses with empty outlook and discussion +arrays represent checked empty outlook data. `app.FetchAndSaveBundle` can write bundle JSON atomically for inspection. ## Skip And Resume Behavior @@ -73,6 +75,8 @@ data is required and cannot be skipped. - Optional sources follow missing-source policy. - Explicit `data: null` from `/alerts/active` produces an empty, non-missing alert run. +- Explicit `data: null` from `/outlooks/convective` follows optional + missing-source policy. ## Tests diff --git a/internal/adapters/weatherapi/client.go b/internal/adapters/weatherapi/client.go index 614e618..d59f630 100644 --- a/internal/adapters/weatherapi/client.go +++ b/internal/adapters/weatherapi/client.go @@ -21,6 +21,11 @@ import ( "gitea.maximumdirect.net/eric/weatherreporter/internal/weatherdata" ) +const ( + convectiveOutlooksEndpoint = "/outlooks/convective" + sourceSPCConvectiveOutlooks = "spc_convective_outlooks" +) + type Client struct { baseURL *url.URL httpClient *http.Client @@ -112,6 +117,9 @@ func (c *Client) FetchBundle(ctx context.Context) (*weatherdata.Bundle, error) { if err := builder.fetchWeatherStory(ctx); err != nil { return nil, err } + if err := builder.fetchSPCConvectiveOutlooks(ctx); err != nil { + return nil, err + } return builder.bundle, nil } @@ -264,6 +272,29 @@ func (b *bundleBuilder) fetchWeatherStory(ctx context.Context) error { return nil } +func (b *bundleBuilder) fetchSPCConvectiveOutlooks(ctx context.Context) error { + raw, source, err := b.client.fetch(ctx, sourceSPCConvectiveOutlooks, convectiveOutlooksEndpoint, queryOptions{timezone: true, omitUnits: true}) + if err != nil { + return err + } + if raw == nil { + return b.handleMissing(&source, "SPC convective outlook data is missing", false) + } + var run weatherdata.ConvectiveOutlookRun + if err := decodeSource(raw, &run); err != nil { + return b.handleMalformed(&source, err, false) + } + if run.IssuedAt != nil { + source.IssuedAt = run.IssuedAt + } else { + source.IssuedAt = run.AsOf + } + source.UpdatedAt = run.UpdatedAt + b.bundle.SPCConvectiveOutlooks = &run + b.addSource(source) + return nil +} + func (b *bundleBuilder) handleMissing(source *weatherdata.Source, message string, required bool) error { source.Missing = true if required { diff --git a/internal/adapters/weatherapi/client_test.go b/internal/adapters/weatherapi/client_test.go index 98e990a..bd8a28a 100644 --- a/internal/adapters/weatherapi/client_test.go +++ b/internal/adapters/weatherapi/client_test.go @@ -55,8 +55,17 @@ func TestFetchBundleFromFixtures(t *testing.T) { if bundle.WeatherStory.UpdatedAt == nil { t.Fatalf("WeatherStory.UpdatedAt = nil, want update timestamp") } - if len(bundle.Sources) != 7 { - t.Fatalf("Sources length = %d, want 7", len(bundle.Sources)) + if bundle.SPCConvectiveOutlooks == nil || len(bundle.SPCConvectiveOutlooks.Outlooks) != 1 { + t.Fatalf("SPCConvectiveOutlooks = %#v, want one outlook", bundle.SPCConvectiveOutlooks) + } + if len(bundle.SPCConvectiveOutlooks.Outlooks[0].Geometry) == 0 { + t.Fatalf("SPCConvectiveOutlooks.Outlooks[0].Geometry is empty, want GeoJSON") + } + if len(bundle.SPCConvectiveOutlooks.Discussions) != 1 || bundle.SPCConvectiveOutlooks.Discussions[0].Headline != "Severe storms possible" { + t.Fatalf("SPCConvectiveOutlooks.Discussions = %#v, want one discussion", bundle.SPCConvectiveOutlooks.Discussions) + } + if len(bundle.Sources) != 8 { + t.Fatalf("Sources length = %d, want 8", len(bundle.Sources)) } if len(bundle.Warnings) != 0 { t.Fatalf("Warnings length = %d, want no warnings", len(bundle.Warnings)) @@ -70,6 +79,9 @@ func TestFetchBundleFromFixtures(t *testing.T) { if !containsPath(requested, "/weatherstories/latest") { t.Fatalf("requested paths = %v, want weather story endpoint", requested) } + if !containsPath(requested, convectiveOutlooksEndpoint) { + t.Fatalf("requested paths = %v, want convective outlook endpoint", requested) + } } func TestFetchBundleBuildsExpectedQueries(t *testing.T) { @@ -92,6 +104,12 @@ func TestFetchBundleBuildsExpectedQueries(t *testing.T) { } continue } + if strings.HasPrefix(rawURL, convectiveOutlooksEndpoint) { + if strings.Contains(rawURL, "units=") || strings.Contains(rawURL, "precision=") || !strings.Contains(rawURL, "tz=America%2FChicago") { + t.Fatalf("convective outlook request %q should use format and tz only", rawURL) + } + continue + } if !strings.Contains(rawURL, "units=us") { t.Fatalf("request %q missing units=us", rawURL) } @@ -127,6 +145,25 @@ func TestFetchBundleRecordsSourceHash(t *testing.T) { if story.IssuedAt == nil || story.UpdatedAt == nil { t.Fatalf("weather story source timestamps = issued %#v updated %#v, want both", story.IssuedAt, story.UpdatedAt) } + outlooks := sourceByName(t, bundle.Sources, sourceSPCConvectiveOutlooks) + if outlooks.Endpoint != convectiveOutlooksEndpoint { + t.Fatalf("convective outlook endpoint = %q, want %s", outlooks.Endpoint, convectiveOutlooksEndpoint) + } + if outlooks.Query["format"] != "json" || outlooks.Query["tz"] != "America/Chicago" || outlooks.Query["units"] != "" || outlooks.Query["precision"] != "" { + t.Fatalf("convective outlook query = %#v, want format and tz only", outlooks.Query) + } + if outlooks.DataSHA256 != hashFixtureData(t, "convective_outlooks.json") { + t.Fatalf("convective outlook DataSHA256 = %q, want fixture hash", outlooks.DataSHA256) + } + if outlooks.Missing { + t.Fatal("convective outlook source Missing = true, want false") + } + if outlooks.IssuedAt == nil || outlooks.IssuedAt.Format(time.RFC3339) != "2026-05-29T15:45:00Z" { + t.Fatalf("convective outlook IssuedAt = %#v, want run issuedAt", outlooks.IssuedAt) + } + if outlooks.UpdatedAt == nil || outlooks.UpdatedAt.Format(time.RFC3339) != "2026-05-29T16:05:00Z" { + t.Fatalf("convective outlook UpdatedAt = %#v, want run updatedAt", outlooks.UpdatedAt) + } } func TestHTTPErrorIsActionable(t *testing.T) { @@ -189,6 +226,59 @@ func TestNullAlertsMeansNoActiveAlerts(t *testing.T) { } } +func TestMissingSPCConvectiveOutlooksUsesPolicy(t *testing.T) { + server := fixtureServer(t, map[string]handlerOverride{ + convectiveOutlooksEndpoint: {status: http.StatusOK, body: `{"data": null}`}, + }, nil) + client := newTestClient(t, server.URL+"/", map[string]config.MissingSourcePolicy{ + sourceSPCConvectiveOutlooks: config.MissingSourceWarn, + }) + + bundle, err := client.FetchBundle(context.Background()) + if err != nil { + t.Fatalf("FetchBundle() error = %v", err) + } + if bundle.SPCConvectiveOutlooks != nil { + t.Fatalf("SPCConvectiveOutlooks = %#v, want nil for missing source", bundle.SPCConvectiveOutlooks) + } + source := sourceByName(t, bundle.Sources, sourceSPCConvectiveOutlooks) + if !source.Missing || len(source.Warnings) != 1 { + t.Fatalf("convective outlook source = %#v, want missing source warning", source) + } +} + +func TestEmptySPCConvectiveOutlooksAreCheckedData(t *testing.T) { + server := fixtureServer(t, map[string]handlerOverride{ + convectiveOutlooksEndpoint: {status: http.StatusOK, body: `{"data":{"asOf":"2026-05-29T16:00:00Z","outlooks":[],"discussions":[]}}`}, + }, nil) + client := newTestClient(t, server.URL+"/", map[string]config.MissingSourcePolicy{ + sourceSPCConvectiveOutlooks: config.MissingSourceWarn, + }) + + bundle, err := client.FetchBundle(context.Background()) + if err != nil { + t.Fatalf("FetchBundle() error = %v", err) + } + if bundle.SPCConvectiveOutlooks == nil { + t.Fatal("SPCConvectiveOutlooks = nil, want checked empty run") + } + if len(bundle.SPCConvectiveOutlooks.Outlooks) != 0 || len(bundle.SPCConvectiveOutlooks.Discussions) != 0 { + t.Fatalf("SPCConvectiveOutlooks = %#v, want empty arrays", bundle.SPCConvectiveOutlooks) + } + source := sourceByName(t, bundle.Sources, sourceSPCConvectiveOutlooks) + if source.Missing || len(source.Warnings) != 0 { + t.Fatalf("convective outlook source = %#v, want non-missing source without warnings", source) + } + if source.IssuedAt == nil || source.IssuedAt.Format(time.RFC3339) != "2026-05-29T16:00:00Z" { + t.Fatalf("convective outlook IssuedAt = %#v, want fallback to asOf", source.IssuedAt) + } + for _, warning := range bundle.Warnings { + if warning.Source == sourceSPCConvectiveOutlooks { + t.Fatalf("warnings = %#v, want no convective outlook warning", bundle.Warnings) + } + } +} + func TestMissingSourcePolicyWarnNoneError(t *testing.T) { tests := []struct { name string @@ -363,13 +453,14 @@ type handlerOverride struct { func fixtureServer(t *testing.T, overrides map[string]handlerOverride, requested *[]string) *httptest.Server { t.Helper() fixtures := map[string]string{ - "/observations": "observations.json", - "/conditions/current": "current.json", - "/forecast/hourly": "hourly.json", - "/forecast/narrative": "narrative.json", - "/alerts/active": "alerts.json", - "/discussion": "discussion.json", - "/weatherstories/latest": "weather_story.json", + "/observations": "observations.json", + "/conditions/current": "current.json", + "/forecast/hourly": "hourly.json", + "/forecast/narrative": "narrative.json", + "/alerts/active": "alerts.json", + "/discussion": "discussion.json", + "/weatherstories/latest": "weather_story.json", + convectiveOutlooksEndpoint: "convective_outlooks.json", } server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if requested != nil { diff --git a/internal/adapters/weatherapi/testdata/convective_outlooks.json b/internal/adapters/weatherapi/testdata/convective_outlooks.json new file mode 100644 index 0000000..852ea24 --- /dev/null +++ b/internal/adapters/weatherapi/testdata/convective_outlooks.json @@ -0,0 +1,50 @@ +{ + "data": { + "locationId": "nws-lsx-grid-90-74", + "locationName": "St. Louis, MO", + "asOf": "2026-05-29T16:00:00Z", + "issuedAt": "2026-05-29T15:45:00Z", + "updatedAt": "2026-05-29T16:05:00Z", + "product": "convective_outlook", + "outlooks": [ + { + "id": "day1-categorical-slight", + "provider": "spc", + "product": "convective_outlook", + "day": 1, + "outlookType": "categorical", + "label": "SLGT", + "labelText": "Slight Risk", + "forecaster": "Smith", + "severityRank": 3, + "validFrom": "2026-05-29T13:00:00-05:00", + "validTo": "2026-05-30T07:00:00-05:00", + "issuedAt": "2026-05-29T15:45:00Z", + "expiresAt": "2026-05-30T07:00:00-05:00", + "sourceUrl": "https://www.spc.noaa.gov/products/outlook/day1otlk.html", + "imageUrl": "https://www.spc.noaa.gov/products/outlook/day1probotlk_2000_torn.gif", + "containsLocation": true, + "geometry": { + "type": "Polygon", + "coordinates": [ + [ + [-91.0, 38.0], + [-90.0, 38.5], + [-89.5, 37.8], + [-91.0, 38.0] + ] + ] + } + } + ], + "discussions": [ + { + "day": 1, + "headline": "Severe storms possible", + "summary": "Scattered severe storms are possible.", + "discussion": "A few storms may become severe during the afternoon.", + "updatedAt": "2026-05-29T16:05:00Z" + } + ] + } +} diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 5701b0f..c636ebc 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -39,6 +39,8 @@ func TestFetchAndSaveBundle(t *testing.T) { _, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":[],"shortTerm":{"qualifier":"(Short Term)","text":"Short-term AFD narrative for saved bundle."},"longTerm":{"qualifier":"(Long Term)","text":"Long-term AFD narrative for saved bundle."}}}`)) case "/weatherstories/latest": _, _ = w.Write([]byte(`{"data":{"officeId":"LSX","startTime":"2026-05-30T08:46:00Z","endTime":"2026-05-31T11:00:00Z","updatedAt":"2026-05-30T09:00:34Z","title":"Several Chances for Rain Through Monday","description":"Scattered showers and thunderstorms remain possible.","altText":"Forecast weather story graphic.","priority":false,"order":1,"downloadUrl":"https://api.weather.gov/offices/LSX/weatherstories/download/test"}}`)) + case "/outlooks/convective": + _, _ = w.Write([]byte(`{"data":{"asOf":"2026-05-29T16:00:00Z","outlooks":[],"discussions":[]}}`)) default: http.NotFound(w, r) } @@ -1134,6 +1136,8 @@ func dailyBundleServer(t *testing.T) *httptest.Server { _, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":["Storms are most likely during the morning."],"shortTerm":{"qualifier":"(Short Term)","text":"Short-term AFD narrative for generated report."},"longTerm":{"qualifier":"(Long Term)","text":"Long-term AFD narrative for generated report."}}}`)) case "/weatherstories/latest": _, _ = w.Write([]byte(`{"data":{"officeId":"LSX","startTime":"2026-05-30T08:46:00Z","endTime":"2026-05-31T11:00:00Z","updatedAt":"2026-05-30T09:00:34Z","title":"Several Chances for Rain Through Monday","description":"Scattered showers and thunderstorms remain possible.","altText":"Forecast weather story graphic.","priority":false,"order":1,"downloadUrl":"https://api.weather.gov/offices/LSX/weatherstories/download/test"}}`)) + case "/outlooks/convective": + _, _ = w.Write([]byte(`{"data":{"asOf":"2026-05-29T16:00:00Z","outlooks":[],"discussions":[]}}`)) default: http.NotFound(w, r) } diff --git a/internal/cli/root_test.go b/internal/cli/root_test.go index dc9fd44..ac8f4e8 100644 --- a/internal/cli/root_test.go +++ b/internal/cli/root_test.go @@ -996,6 +996,8 @@ func dailyServer(t *testing.T) *httptest.Server { _, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":["Storms are most likely during the morning."]}}`)) case "/weatherstories/latest": _, _ = w.Write([]byte(`{"data":{"officeId":"LSX","startTime":"2026-05-30T08:46:00Z","endTime":"2026-05-31T11:00:00Z","updatedAt":"2026-05-30T09:00:34Z","title":"Several Chances for Rain Through Monday","description":"Scattered showers and thunderstorms remain possible.","altText":"Forecast weather story graphic.","priority":false,"order":1,"downloadUrl":"https://api.weather.gov/offices/LSX/weatherstories/download/test"}}`)) + case "/outlooks/convective": + _, _ = w.Write([]byte(`{"data":{"asOf":"2026-05-29T16:00:00Z","outlooks":[],"discussions":[]}}`)) default: http.NotFound(w, r) }