16 Commits

Author SHA1 Message Date
9b88be4dd2 Handle additional NWS forecast discussion formats
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
2026-08-03 01:14:47 +00:00
456a46e01b Mark NWS forecast discussion resilience implemented 2026-08-03 00:24:26 +00:00
3740c779eb Add NWS forecast discussion fixture coverage 2026-08-03 00:22:51 +00:00
6943a5ebc9 Harden NWS key message parsing 2026-08-03 00:19:48 +00:00
9a09454621 Recognize NWS key points sections 2026-08-03 00:15:52 +00:00
b8c6708439 Support parenthesized NWS discussion headings 2026-08-03 00:13:58 +00:00
a62cb87b78 Mark NWS forecast discussion resilience implemented 2026-08-02 23:02:43 +00:00
0b5eaf46f4 Add cross-office NWS forecast discussion coverage 2026-08-02 23:01:11 +00:00
882059014c Normalize NWS forecast discussion preambles 2026-08-02 22:58:48 +00:00
0a3e52d0e5 Scan NWS forecast discussion blocks once 2026-08-02 22:56:03 +00:00
2b19a121fa Generalize NWS forecast discussion heading parsing 2026-08-02 22:53:00 +00:00
29c65971eb Mark NWS forecast discussion heading variants implemented 2026-08-02 20:54:12 +00:00
3ecf4c5b7f Add NWS forecast discussion heading regressions 2026-08-02 20:53:30 +00:00
f402e27542 Centralize NWS forecast discussion section headings 2026-08-02 20:50:51 +00:00
f720b6cdc0 Plan support for NWS AFD section heading variants 2026-08-02 20:47:24 +00:00
50215d2105 Add a new field to the alert schema to fix a mismatch between the prior schema and the upstream NWS API
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-06-16 19:57:48 -05:00
22 changed files with 2431 additions and 72 deletions

View File

@@ -41,9 +41,6 @@ Related child types include:
- `WeatherOutlookDiscussion`
- `WMOCode`
`WeatherOutlookRun` includes `WeatherOutlookDiscussion` entries as run-level
SPC day discussions.
## Wire And Compatibility Rules
- JSON tags define canonical payload field names.

View File

@@ -211,8 +211,9 @@ Payload type: `WeatherAlertRun`.
| `instruction` | string | no | Alert instruction. |
| `sent` | timestamp | no | Provider sent time. |
| `effective` | timestamp | no | Effective time. |
| `onset` | timestamp | no | Onset time. |
| `expires` | timestamp | no | Expiration time. |
| `onset` | timestamp | no | Alert period start. |
| `ends` | timestamp | no | Alert period end. |
| `expires` | timestamp | no | Provider expiration metadata; not necessarily the alert period end. |
| `areaDescription` | string | no | Affected area description. |
| `senderName` | string | no | Provider sender name. |
| `references` | array | no | Related alerts. |

View File

@@ -44,6 +44,9 @@ normalizer uses fields under `properties` such as `stationId`, `stationName`,
`nws_alerts` expects an alerts FeatureCollection. The normalizer uses the
collection `updated` timestamp, `title`, each feature ID, alert classification
fields, narrative fields, timing fields, sender fields, and references.
`properties.onset` and `properties.ends` map to the canonical alert period
start and end. `properties.expires` maps only to canonical `expires` provider
metadata and is not treated as the alert period end.
`nws_forecast_hourly` and `nws_forecast_narrative` expect gridpoint forecast
GeoJSON with `properties.generatedAt`, `properties.updateTime`, elevation,
@@ -98,8 +101,9 @@ unset. Forecast temperatures are converted to Celsius when NWS supplies
Fahrenheit, and wind speed strings are converted to kilometers per hour.
Alert timing fields are parsed best-effort. Invalid per-alert timestamps are
left unset rather than failing the whole alert run. Missing alert IDs are
synthesized from the run snapshot time and array position.
left unset rather than failing the whole alert run. NWS `ends` is preserved
separately from `expires`; `expires` does not fall back to `ends`. Missing alert
IDs are synthesized from the run snapshot time and array position.
Forecast discussion parsing requires an issue time. Weather story entries require
start time, end time, and update time.

View File

@@ -353,6 +353,7 @@ Indexes:
| `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].sent` |
| `effective` | `TIMESTAMPTZ` | yes | `payload.alerts[].effective` |
| `onset` | `TIMESTAMPTZ` | yes | `payload.alerts[].onset` |
| `ends` | `TIMESTAMPTZ` | yes | `payload.alerts[].ends` |
| `expires` | `TIMESTAMPTZ` | yes | `payload.alerts[].expires` |
| `area_description` | `TEXT` | yes | `payload.alerts[].areaDescription` |
| `sender_name` | `TEXT` | yes | `payload.alerts[].senderName` |

View File

@@ -4,6 +4,45 @@
This document is the catch-all roadmap for planned, deferred, aspirational, experimental, or unimplemented weatherfeeder work. Current behavior belongs in the canonical docs outside `docs/roadmap/`.
## NWS AFD Parsing Resilience
The current parser handles the concrete RAH, LWX, and MFR variants that
motivated these ideas. Future work should keep those extension points
maintainable as additional evidence appears.
### Generalize Wrapper-Scoped Embedded Sections
The scanner currently permits undotted nested headings only inside `PREV
DISCUSSION`. If other wrapper identities are observed, replace the single
wrapper check with a small explicit provider-local registry and add a fixture
for each wrapper family. Do not make the leading dot globally optional: wrapper
scope is the safeguard against classifying uppercase prose as a section.
### Extend Conservative Preamble Classification
Leading key-message metadata currently supports validated `Issued at`, `Updated
at`, and `As of <clock> <weekday>...` forms. Add future wording variants as
small, ordered classifiers with strict label boundaries and value grammars.
Every addition should include collision tests proving that similar message prose
and malformed metadata remain canonical content.
### Keep List-Marker Recognition Extensible
The marker parser currently supports hyphens, asterisks, `N)`, `N.`, `(N)`, and
composite forms such as `- (N)`. If new decorators appear, evolve the helper
toward an explicit marker grammar or typed classification result rather than a
broad punctuation heuristic. Preserve positive-number and whitespace-boundary
checks so ordinary prose is not stripped.
### Maintain a Cross-Office Fixture Corpus
The compact RAH, LWX, and current MFR fixtures seed regression coverage for the
observed layouts. Future parser changes should add concise, deterministic HTML
fixtures for materially distinct office formats and exercise them through both
the provider parser and normalizer. Fixture comments should identify the format
family and state that edited prose is not an archived product; tests must remain
offline and assert both intended extraction and adjacent-section isolation.
## SPC Convective Outlook Follow-Ups
### Weatherapi Outlook Endpoints

View File

@@ -93,12 +93,8 @@ func buildAlerts(parsed nwsAlertsResponse, fallbackAsOf time.Time) (model.Weathe
sent := nwscommon.ParseTimePtr(p.Sent)
effective := nwscommon.ParseTimePtr(p.Effective)
onset := nwscommon.ParseTimePtr(p.Onset)
// Expires: prefer "expires"; fall back to "ends" if present.
ends := nwscommon.ParseTimePtr(p.Ends)
expires := nwscommon.ParseTimePtr(p.Expires)
if expires == nil {
expires = nwscommon.ParseTimePtr(p.Ends)
}
refs := parseNWSAlertReferences(p.References)
@@ -123,6 +119,7 @@ func buildAlerts(parsed nwsAlertsResponse, fallbackAsOf time.Time) (model.Weathe
Sent: sent,
Effective: effective,
Onset: onset,
Ends: ends,
Expires: expires,
AreaDescription: strings.TrimSpace(p.AreaDesc),

View File

@@ -0,0 +1,136 @@
package nws
import (
"context"
"encoding/json"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
func TestAlertsNormalizerMapsEndsSeparatelyFromExpires(t *testing.T) {
raw := []byte(`{
"updated":"2026-06-16T10:00:00+00:00",
"title":"Current watches, warnings, and advisories for St. Louis",
"features":[{
"id":"https://api.weather.gov/alerts/alert-1",
"properties":{
"event":"Flood Warning",
"headline":"Flood Warning issued",
"sent":"2026-06-16T09:55:00+00:00",
"effective":"2026-06-16T10:00:00+00:00",
"onset":"2026-06-16T10:15:00+00:00",
"ends":"2026-06-16T14:00:00+00:00",
"expires":"2026-06-16T11:00:00+00:00"
}
}]
}`)
out, err := AlertsNormalizer{}.Normalize(context.Background(), alertRawEvent(raw))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
run := decodeAlertRun(t, out)
if len(run.Alerts) != 1 {
t.Fatalf("expected 1 alert, got %d", len(run.Alerts))
}
alert := run.Alerts[0]
wantEnds := time.Date(2026, 6, 16, 14, 0, 0, 0, time.UTC)
wantExpires := time.Date(2026, 6, 16, 11, 0, 0, 0, time.UTC)
if alert.Ends == nil || !alert.Ends.Equal(wantEnds) {
t.Fatalf("ends = %v, want %s", alert.Ends, wantEnds)
}
if alert.Expires == nil || !alert.Expires.Equal(wantExpires) {
t.Fatalf("expires = %v, want %s", alert.Expires, wantExpires)
}
}
func TestAlertsNormalizerDoesNotFallbackExpiresToEnds(t *testing.T) {
raw := []byte(`{
"updated":"2026-06-16T10:00:00+00:00",
"features":[{
"id":"alert-ends-only",
"properties":{
"event":"Heat Advisory",
"ends":"2026-06-16T22:00:00+00:00"
}
}]
}`)
out, err := AlertsNormalizer{}.Normalize(context.Background(), alertRawEvent(raw))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
run := decodeAlertRun(t, out)
alert := run.Alerts[0]
if alert.Ends == nil {
t.Fatal("expected ends to be populated")
}
if alert.Expires != nil {
t.Fatalf("expected expires nil when upstream expires is absent, got %v", alert.Expires)
}
}
func TestAlertsNormalizerIgnoresInvalidEnds(t *testing.T) {
raw := []byte(`{
"updated":"2026-06-16T10:00:00+00:00",
"features":[{
"id":"alert-invalid-ends",
"properties":{
"event":"Special Weather Statement",
"ends":"not-a-time",
"expires":"2026-06-16T11:00:00+00:00"
}
}]
}`)
out, err := AlertsNormalizer{}.Normalize(context.Background(), alertRawEvent(raw))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
run := decodeAlertRun(t, out)
alert := run.Alerts[0]
if alert.Ends != nil {
t.Fatalf("expected invalid ends to map nil, got %v", alert.Ends)
}
if alert.Expires == nil {
t.Fatal("expected expires to remain populated")
}
}
func alertRawEvent(raw []byte) event.Event {
emittedAt := time.Date(2026, 6, 16, 10, 5, 0, 0, time.UTC)
return event.Event{
ID: "raw-alerts",
Kind: event.Kind(standards.KindAlert),
Source: "NWSAlerts",
Schema: standards.SchemaRawNWSAlertsV1,
EmittedAt: emittedAt,
Payload: json.RawMessage(raw),
}
}
func decodeAlertRun(t *testing.T, e *event.Event) model.WeatherAlertRun {
t.Helper()
if e == nil {
t.Fatal("expected normalized event")
}
if e.Schema != standards.SchemaWeatherAlertV1 {
t.Fatalf("schema = %q, want %q", e.Schema, standards.SchemaWeatherAlertV1)
}
var run model.WeatherAlertRun
raw, err := json.Marshal(e.Payload)
if err != nil {
t.Fatalf("marshal alert payload: %v", err)
}
if err := json.Unmarshal(raw, &run); err != nil {
t.Fatalf("decode alert payload: %v", err)
}
return run
}

View File

@@ -4,6 +4,7 @@ import (
"encoding/json"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
@@ -71,6 +72,277 @@ func TestForecastDiscussionNormalizerProducesCanonicalSchema(t *testing.T) {
}
}
func TestForecastDiscussionNormalizerSupportsMixedHeadingFormats(t *testing.T) {
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
ID: "evt-discussion-mixed-format",
Kind: event.Kind(standards.KindForecastDiscussion),
Source: "nws-discussion-test",
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
Schema: standards.SchemaRawNWSForecastDiscussionV1,
Payload: loadMixedFormatForecastDiscussionSampleHTML(t),
})
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
if out == nil {
t.Fatalf("Normalize() returned nil output")
}
if out.Kind != event.Kind(standards.KindForecastDiscussion) {
t.Fatalf("Kind = %q, want forecast_discussion", out.Kind)
}
if out.Schema != standards.SchemaWeatherForecastDiscussionV1 {
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastDiscussionV1)
}
wantEffectiveAt := time.Date(2026, 3, 28, 19, 24, 0, 0, time.UTC)
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
}
payload, ok := out.Payload.(model.WeatherForecastDiscussion)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherForecastDiscussion", out.Payload)
}
if payload.ShortTerm == nil || payload.LongTerm == nil {
t.Fatalf("ShortTerm=%v LongTerm=%v, want both populated", payload.ShortTerm, payload.LongTerm)
}
if payload.ShortTerm.Qualifier != "Through Late Sunday Night" {
t.Fatalf("ShortTerm.Qualifier = %q", payload.ShortTerm.Qualifier)
}
if !strings.Contains(payload.ShortTerm.Text, "After a chilly morning") {
t.Fatalf("ShortTerm.Text missing expected prose: %q", payload.ShortTerm.Text)
}
if payload.LongTerm.Qualifier != "Monday through Next Saturday" {
t.Fatalf("LongTerm.Qualifier = %q", payload.LongTerm.Qualifier)
}
if !strings.Contains(payload.LongTerm.Text, "The peak of the warmth arrives Monday and Tuesday") {
t.Fatalf("LongTerm.Text missing expected prose: %q", payload.LongTerm.Text)
}
if strings.Contains(payload.LongTerm.Text, "AVIATION") || strings.Contains(payload.LongTerm.Text, "VFR conditions are expected") {
t.Fatalf("LongTerm.Text includes aviation content: %q", payload.LongTerm.Text)
}
b, err := json.Marshal(out.Payload)
if err != nil {
t.Fatalf("json.Marshal(payload) error = %v", err)
}
var fields map[string]any
if err := json.Unmarshal(b, &fields); err != nil {
t.Fatalf("json.Unmarshal(payload) error = %v", err)
}
for _, key := range []string{"aviation", "sections"} {
if _, ok := fields[key]; ok {
t.Fatalf("unexpected key %q in canonical payload", key)
}
}
}
func TestForecastDiscussionNormalizerSupportsCrossOfficeLayout(t *testing.T) {
in := event.Event{
ID: "evt-discussion-bou",
Kind: event.Kind(standards.KindForecastDiscussion),
Source: "nws-discussion-bou-test",
EmittedAt: time.Date(2026, 4, 7, 19, 1, 0, 0, time.UTC),
Schema: standards.SchemaRawNWSForecastDiscussionV1,
Payload: loadForecastDiscussionBOUSampleHTML(t),
}
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, in)
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
if out == nil {
t.Fatalf("Normalize() returned nil output")
}
if out.ID != in.ID || out.Source != in.Source || !out.EmittedAt.Equal(in.EmittedAt) {
t.Fatalf("envelope = %#v, want ID/source/emittedAt from input", out)
}
if out.Kind != event.Kind(standards.KindForecastDiscussion) {
t.Fatalf("Kind = %q, want forecast_discussion", out.Kind)
}
if out.Schema != standards.SchemaWeatherForecastDiscussionV1 {
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastDiscussionV1)
}
wantEffectiveAt := time.Date(2026, 4, 7, 19, 0, 0, 0, time.UTC)
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
}
payload, ok := out.Payload.(model.WeatherForecastDiscussion)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherForecastDiscussion", out.Payload)
}
if payload.OfficeID != "BOU" || payload.OfficeName != "National Weather Service Denver CO" {
t.Fatalf("OfficeID=%q OfficeName=%q", payload.OfficeID, payload.OfficeName)
}
wantMessages := []string{
"Strong winds are expected along the Front Range this evening.",
"Cooler temperatures arrive on Wednesday.",
}
if len(payload.KeyMessages) != len(wantMessages) {
t.Fatalf("KeyMessages = %#v, want %#v", payload.KeyMessages, wantMessages)
}
for i := range wantMessages {
if payload.KeyMessages[i] != wantMessages[i] {
t.Fatalf("KeyMessages[%d] = %q, want %q", i, payload.KeyMessages[i], wantMessages[i])
}
}
if payload.ShortTerm == nil || payload.LongTerm == nil {
t.Fatalf("ShortTerm=%v LongTerm=%v, want both populated", payload.ShortTerm, payload.LongTerm)
}
if payload.ShortTerm.Qualifier != "(Tonight through Wednesday)" || payload.ShortTerm.Text != "Gusty west winds will continue through the evening before decreasing overnight." {
t.Fatalf("ShortTerm = %#v", payload.ShortTerm)
}
if payload.LongTerm.Qualifier != "(Thursday through Saturday)" || payload.LongTerm.Text != "Warmer and drier conditions return Thursday, followed by a chance of showers Friday." {
t.Fatalf("LongTerm = %#v", payload.LongTerm)
}
if payload.ShortTerm.IssuedAt == nil || payload.LongTerm.IssuedAt == nil ||
!payload.ShortTerm.IssuedAt.Equal(wantEffectiveAt) || !payload.LongTerm.IssuedAt.Equal(wantEffectiveAt) {
t.Fatalf("section issue times = short %v long %v, want %s", payload.ShortTerm.IssuedAt, payload.LongTerm.IssuedAt, wantEffectiveAt.Format(time.RFC3339))
}
b, err := json.Marshal(out.Payload)
if err != nil {
t.Fatalf("json.Marshal(payload) error = %v", err)
}
var fields map[string]any
if err := json.Unmarshal(b, &fields); err != nil {
t.Fatalf("json.Unmarshal(payload) error = %v", err)
}
for _, key := range []string{"aviation", "discussion", "sections"} {
if _, ok := fields[key]; ok {
t.Fatalf("unexpected key %q in canonical payload", key)
}
}
}
func TestForecastDiscussionNormalizerSupportsCrossOfficeKeyMessageFixtures(t *testing.T) {
tests := []struct {
name string
filename string
id string
source string
emittedAt time.Time
effectiveAt time.Time
messages []string
}{
{
name: "numbered key messages",
filename: "forecast_discussion_bgm_numbered_sample.html",
id: "evt-discussion-bgm",
source: "nws-discussion-bgm-test",
emittedAt: time.Date(2026, 4, 10, 17, 31, 0, 0, time.UTC),
effectiveAt: time.Date(2026, 4, 10, 17, 30, 0, 0, time.UTC),
messages: []string{
"Periods of rain are expected through Saturday, with locally heavier amounts possible.",
"Cooler temperatures return late this weekend.",
},
},
{
name: "key points alias",
filename: "forecast_discussion_mfr_key_points_sample.html",
id: "evt-discussion-mfr",
source: "nws-discussion-mfr-test",
emittedAt: time.Date(2026, 4, 10, 19, 46, 0, 0, time.UTC),
effectiveAt: time.Date(2026, 4, 10, 19, 45, 0, 0, time.UTC),
messages: []string{
"Gusty winds will develop over exposed ridges, especially during the afternoon.",
"Inland valleys remain dry through Saturday.",
},
},
{
name: "as of preamble",
filename: "forecast_discussion_rah_as_of_sample.html",
id: "evt-discussion-rah",
source: "nws-discussion-rah-test",
emittedAt: time.Date(2026, 8, 2, 16, 36, 0, 0, time.UTC),
effectiveAt: time.Date(2026, 8, 2, 16, 35, 0, 0, time.UTC),
messages: []string{
"Scattered storms may produce locally heavy rain this afternoon.",
"Drier weather arrives Monday.",
},
},
{
name: "parenthesized numeric markers",
filename: "forecast_discussion_lwx_parenthesized_number_sample.html",
id: "evt-discussion-lwx",
source: "nws-discussion-lwx-test",
emittedAt: time.Date(2026, 8, 2, 18, 1, 0, 0, time.UTC),
effectiveAt: time.Date(2026, 8, 2, 18, 0, 0, 0, time.UTC),
messages: []string{
"Thunderstorms remain possible near the Blue Ridge this evening.",
"Seasonably warm conditions continue Monday.",
},
},
{
name: "embedded key messages in previous discussion",
filename: "forecast_discussion_mfr_prev_discussion_sample.html",
id: "evt-discussion-mfr-previous",
source: "nws-discussion-mfr-previous-test",
emittedAt: time.Date(2026, 8, 2, 22, 20, 0, 0, time.UTC),
effectiveAt: time.Date(2026, 8, 2, 22, 19, 0, 0, time.UTC),
messages: []string{
"Heat returns to inland valleys Monday.",
"Gusty afternoon winds develop east of the Cascades.",
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
in := event.Event{
ID: tt.id,
Kind: event.Kind(standards.KindForecastDiscussion),
Source: tt.source,
EmittedAt: tt.emittedAt,
Schema: standards.SchemaRawNWSForecastDiscussionV1,
Payload: loadForecastDiscussionFixtureHTML(t, tt.filename),
}
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, in)
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
if out == nil {
t.Fatalf("Normalize() returned nil output")
}
if out.ID != in.ID || out.Source != in.Source || !out.EmittedAt.Equal(in.EmittedAt) {
t.Fatalf("envelope = %#v, want ID/source/emittedAt from input", out)
}
if out.Kind != event.Kind(standards.KindForecastDiscussion) {
t.Fatalf("Kind = %q, want forecast_discussion", out.Kind)
}
if out.Schema != standards.SchemaWeatherForecastDiscussionV1 {
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastDiscussionV1)
}
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(tt.effectiveAt) {
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, tt.effectiveAt.Format(time.RFC3339))
}
payload, ok := out.Payload.(model.WeatherForecastDiscussion)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherForecastDiscussion", out.Payload)
}
if !reflect.DeepEqual(payload.KeyMessages, tt.messages) {
t.Fatalf("KeyMessages = %#v, want %#v", payload.KeyMessages, tt.messages)
}
b, err := json.Marshal(payload)
if err != nil {
t.Fatalf("json.Marshal(payload) error = %v", err)
}
var fields map[string]any
if err := json.Unmarshal(b, &fields); err != nil {
t.Fatalf("json.Unmarshal(payload) error = %v", err)
}
for _, key := range []string{"nearTerm", "discussion", "aviation", "sections"} {
if _, ok := fields[key]; ok {
t.Fatalf("unexpected key %q in canonical payload", key)
}
}
})
}
}
func TestForecastDiscussionNormalizerRejectsMissingIssueTime(t *testing.T) {
_, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
ID: "evt-discussion-bad",
@@ -128,3 +400,56 @@ func loadForecastDiscussionSampleHTML(t *testing.T) string {
}
return string(b)
}
func loadForecastDiscussionBOUSampleHTML(t *testing.T) string {
t.Helper()
path := filepath.Join("..", "..", "providers", "nws", "testdata", "forecast_discussion_bou_sample.html")
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("os.ReadFile(%q) error = %v", path, err)
}
return string(b)
}
func loadForecastDiscussionFixtureHTML(t *testing.T, filename string) string {
t.Helper()
path := filepath.Join("..", "..", "providers", "nws", "testdata", filename)
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("os.ReadFile(%q) error = %v", path, err)
}
return string(b)
}
func loadMixedFormatForecastDiscussionSampleHTML(t *testing.T) string {
t.Helper()
raw := loadForecastDiscussionSampleHTML(t)
replacements := []struct {
original string
replacement string
}{
{
original: ".SHORT TERM... (Through Late Sunday Night)",
replacement: ".SHORT TERM /Through Late Sunday Night/...",
},
{
original: ".LONG TERM... (Monday through Next Saturday)",
replacement: ".LONG TERM /Monday through Next Saturday/...",
},
{
original: ".AVIATION... (For the 18z TAFs through 18z Sunday Afternoon)",
replacement: ".AVIATION /For the 18z TAFs through 18z Sunday Afternoon/...",
},
}
for _, replacement := range replacements {
if !strings.Contains(raw, replacement.original) {
t.Fatalf("fixture missing heading %q", replacement.original)
}
raw = strings.Replace(raw, replacement.original, replacement.replacement, 1)
}
return raw
}

View File

@@ -27,8 +27,31 @@ type ForecastDiscussionSection struct {
Text string
}
type forecastDiscussionSectionRole uint8
const (
forecastDiscussionSectionRoleKeyMessages forecastDiscussionSectionRole = iota
forecastDiscussionSectionRoleShortTerm
forecastDiscussionSectionRoleLongTerm
)
type forecastDiscussionSectionHeading struct {
section string
qualifier string
}
type forecastDiscussionSectionBlock struct {
heading forecastDiscussionSectionHeading
body []string
}
var (
forecastDiscussionHeaderRE = regexp.MustCompile(`^\.(KEY MESSAGES|SHORT TERM|LONG TERM|AVIATION)\.\.\.(.*)$`)
forecastDiscussionSectionRoles = map[string]forecastDiscussionSectionRole{
"KEY MESSAGES": forecastDiscussionSectionRoleKeyMessages,
"KEY POINTS": forecastDiscussionSectionRoleKeyMessages,
"SHORT TERM": forecastDiscussionSectionRoleShortTerm,
"LONG TERM": forecastDiscussionSectionRoleLongTerm,
}
forecastDiscussionAFDRE = regexp.MustCompile(`^AFD([A-Z]{3})$`)
forecastDiscussionWMORE = regexp.MustCompile(`\bK([A-Z]{3})\b`)
forecastDiscussionSigRE = regexp.MustCompile(`^[A-Z]{2,6}$`)
@@ -99,23 +122,31 @@ func ParseForecastDiscussionText(text string) (ForecastDiscussion, error) {
IssuedAt: issuedAt.UTC(),
}
if block, ok := extractForecastDiscussionSection(lines, "KEY MESSAGES"); ok {
out.KeyMessages = parseForecastDiscussionKeyMessages(block)
seenRoles := make(map[forecastDiscussionSectionRole]bool, len(forecastDiscussionSectionRoles))
for _, block := range parseForecastDiscussionSectionBlocks(lines) {
role, ok := forecastDiscussionSectionRoles[block.heading.section]
if !ok || seenRoles[role] {
continue
}
if block, ok := extractForecastDiscussionSection(lines, "SHORT TERM"); ok {
seenRoles[role] = true
switch role {
case forecastDiscussionSectionRoleKeyMessages:
out.KeyMessages = parseForecastDiscussionKeyMessages(block.body)
case forecastDiscussionSectionRoleShortTerm:
section, err := parseForecastDiscussionTextSection(block)
if err != nil {
return ForecastDiscussion{}, fmt.Errorf("parse SHORT TERM: %w", err)
return ForecastDiscussion{}, fmt.Errorf("parse %s: %w", block.heading.section, err)
}
out.ShortTerm = &section
}
if block, ok := extractForecastDiscussionSection(lines, "LONG TERM"); ok {
case forecastDiscussionSectionRoleLongTerm:
section, err := parseForecastDiscussionTextSection(block)
if err != nil {
return ForecastDiscussion{}, fmt.Errorf("parse LONG TERM: %w", err)
return ForecastDiscussion{}, fmt.Errorf("parse %s: %w", block.heading.section, err)
}
out.LongTerm = &section
}
}
return out, nil
}
@@ -285,8 +316,9 @@ func parseForecastDiscussionHeader(lines []string) (string, time.Time, error) {
func parseForecastDiscussionIssueTime(line string) (time.Time, error) {
line = strings.TrimSpace(line)
line = strings.TrimPrefix(line, "Issued at ")
line = strings.TrimSpace(line)
if isForecastDiscussionIssuedAtLine(line) {
line = strings.TrimSpace(line[len("Issued at"):])
}
parts := strings.Fields(line)
if len(parts) != 7 {
@@ -386,40 +418,217 @@ func forecastDiscussionLocation(abbrev string) (*time.Location, error) {
return time.FixedZone(abbr, offset), nil
}
func extractForecastDiscussionSection(lines []string, section string) ([]string, bool) {
target := "." + section + "..."
for i, raw := range lines {
func parseForecastDiscussionSectionHeading(line string) (forecastDiscussionSectionHeading, bool) {
line = strings.TrimSpace(line)
if len(line) < 2 || line[0] != '.' {
return forecastDiscussionSectionHeading{}, false
}
if strings.HasSuffix(line, "/...") {
return parseForecastDiscussionSlashQualifiedHeading(line)
}
if strings.HasSuffix(line, "...") {
if heading, ok := parseForecastDiscussionParenthesizedTerminalHeading(line); ok {
return heading, true
}
}
return parseForecastDiscussionEllipsisHeading(line)
}
func parseForecastDiscussionSlashQualifiedHeading(line string) (forecastDiscussionSectionHeading, bool) {
content := strings.TrimSuffix(line[1:], "/...")
separator := -1
for i := 1; i < len(content); i++ {
if content[i] == '/' && isForecastDiscussionHorizontalWhitespace(content[i-1]) {
separator = i
break
}
}
if separator < 0 {
return forecastDiscussionSectionHeading{}, false
}
section, ok := normalizeForecastDiscussionSectionIdentity(content[:separator])
if !ok {
return forecastDiscussionSectionHeading{}, false
}
qualifier := strings.TrimSpace(content[separator+1:])
if qualifier == "" {
return forecastDiscussionSectionHeading{}, false
}
return forecastDiscussionSectionHeading{section: section, qualifier: qualifier}, true
}
func parseForecastDiscussionParenthesizedTerminalHeading(line string) (forecastDiscussionSectionHeading, bool) {
if len(line) < 4 || line[0] != '.' || !strings.HasSuffix(line, "...") {
return forecastDiscussionSectionHeading{}, false
}
content := strings.TrimRight(line[1:len(line)-3], " \t")
if !strings.HasSuffix(content, ")") {
return forecastDiscussionSectionHeading{}, false
}
separator := -1
for i := 1; i < len(content); i++ {
if content[i] == '(' && isForecastDiscussionHorizontalWhitespace(content[i-1]) {
separator = i
break
}
}
if separator < 0 {
return forecastDiscussionSectionHeading{}, false
}
section, ok := normalizeForecastDiscussionSectionIdentity(content[:separator])
if !ok {
return forecastDiscussionSectionHeading{}, false
}
qualifier := content[separator:]
if len(qualifier) <= 2 || strings.TrimSpace(qualifier[1:len(qualifier)-1]) == "" {
return forecastDiscussionSectionHeading{}, false
}
return forecastDiscussionSectionHeading{section: section, qualifier: qualifier}, true
}
func parseForecastDiscussionEllipsisHeading(line string) (forecastDiscussionSectionHeading, bool) {
content := line[1:]
delimiter := strings.Index(content, "...")
if delimiter < 0 {
return forecastDiscussionSectionHeading{}, false
}
section, ok := normalizeForecastDiscussionSectionIdentity(content[:delimiter])
if !ok {
return forecastDiscussionSectionHeading{}, false
}
return forecastDiscussionSectionHeading{
section: section,
qualifier: strings.TrimSpace(content[delimiter+3:]),
}, true
}
func normalizeForecastDiscussionSectionIdentity(raw string) (string, bool) {
var normalized strings.Builder
pendingSpace := false
hasLetterOrDigit := false
for i := 0; i < len(raw); i++ {
b := raw[i]
switch {
case isForecastDiscussionIdentityLetterOrDigit(b):
hasLetterOrDigit = true
case b == ' ' || b == '\t':
pendingSpace = normalized.Len() > 0
continue
case b == '/' && i > 0 && isForecastDiscussionHorizontalWhitespace(raw[i-1]):
return "", false
case b != '/' && b != '&' && b != '\'' && b != '-':
return "", false
}
if pendingSpace {
normalized.WriteByte(' ')
pendingSpace = false
}
normalized.WriteByte(b)
}
if !hasLetterOrDigit {
return "", false
}
return normalized.String(), true
}
func isForecastDiscussionIdentityLetterOrDigit(b byte) bool {
return b >= 'A' && b <= 'Z' || b >= '0' && b <= '9'
}
func isForecastDiscussionHorizontalWhitespace(b byte) bool {
return b == ' ' || b == '\t'
}
func parseForecastDiscussionSectionBlocks(lines []string) []forecastDiscussionSectionBlock {
var blocks []forecastDiscussionSectionBlock
var active *forecastDiscussionSectionBlock
embeddedHeadings := false
finish := func() {
if active == nil {
return
}
blocks = append(blocks, *active)
active = nil
}
for _, raw := range lines {
line := strings.TrimSpace(raw)
if !strings.HasPrefix(line, target) {
if line == "$$" {
finish()
break
}
if line == "&&" || strings.Contains(line, "WATCHES/WARNINGS/ADVISORIES") {
finish()
embeddedHeadings = false
continue
}
out := []string{line}
for j := i + 1; j < len(lines); j++ {
next := strings.TrimSpace(lines[j])
if next == "&&" || next == "$$" || strings.Contains(next, "WATCHES/WARNINGS/ADVISORIES") {
break
heading, ok := parseForecastDiscussionSectionHeading(raw)
if ok {
finish()
active = &forecastDiscussionSectionBlock{heading: heading}
embeddedHeadings = isForecastDiscussionEmbeddedSectionWrapper(heading.section)
continue
}
if j > i+1 && isForecastDiscussionSectionHeader(next) {
break
if embeddedHeadings {
heading, ok = parseForecastDiscussionEmbeddedSectionHeading(raw)
if ok {
finish()
active = &forecastDiscussionSectionBlock{heading: heading}
continue
}
out = append(out, lines[j])
}
return out, true
if active != nil {
active.body = append(active.body, raw)
}
return nil, false
}
finish()
return blocks
}
func isForecastDiscussionSectionHeader(line string) bool {
return forecastDiscussionHeaderRE.MatchString(strings.TrimSpace(line))
func isForecastDiscussionEmbeddedSectionWrapper(section string) bool {
return section == "PREV DISCUSSION"
}
func parseForecastDiscussionKeyMessages(block []string) []string {
if len(block) <= 1 {
func parseForecastDiscussionEmbeddedSectionHeading(line string) (forecastDiscussionSectionHeading, bool) {
line = strings.TrimSpace(line)
if line == "" || line[0] == '.' {
return forecastDiscussionSectionHeading{}, false
}
return parseForecastDiscussionSectionHeading("." + line)
}
func parseForecastDiscussionKeyMessages(body []string) []string {
body = removeForecastDiscussionPresentationMarkers(body)
body = trimBlankLines(body)
if len(body) > 0 && isForecastDiscussionKeyMessageMetadataLine(body[0]) {
body = trimBlankLines(body[1:])
}
if len(body) == 0 {
return nil
}
hasMarkers := false
for _, raw := range body {
line := strings.TrimSpace(raw)
if line == "" {
continue
}
if _, ok := stripForecastDiscussionKeyMessageMarker(line); ok {
hasMarkers = true
break
}
}
body := trimBlankLines(block[1:])
var messages []string
var current strings.Builder
@@ -431,15 +640,21 @@ func parseForecastDiscussionKeyMessages(block []string) []string {
current.Reset()
}
seenMarker := false
for _, raw := range body {
line := strings.TrimSpace(raw)
if line == "" {
if !hasMarkers || !seenMarker {
flush()
}
continue
}
if strings.HasPrefix(line, "-") {
if stripped, ok := stripForecastDiscussionKeyMessageMarker(line); ok {
flush()
line = strings.TrimSpace(strings.TrimPrefix(line, "-"))
current.WriteString(line)
seenMarker = true
line = stripped
}
if line == "" {
continue
}
if current.Len() > 0 {
@@ -452,25 +667,24 @@ func parseForecastDiscussionKeyMessages(block []string) []string {
return messages
}
func parseForecastDiscussionTextSection(block []string) (ForecastDiscussionSection, error) {
if len(block) == 0 {
return ForecastDiscussionSection{}, fmt.Errorf("empty section")
}
func parseForecastDiscussionTextSection(block forecastDiscussionSectionBlock) (ForecastDiscussionSection, error) {
section := ForecastDiscussionSection{
Qualifier: parseForecastDiscussionQualifier(strings.TrimSpace(block[0])),
Qualifier: block.heading.qualifier,
}
body := trimBlankLines(block[1:])
body := trimBlankLines(removeForecastDiscussionPresentationMarkers(block.body))
if section.Qualifier == "" && len(body) > 0 && isForecastDiscussionStandaloneParenthetical(body[0]) {
section.Qualifier = strings.TrimSpace(body[0])
body = trimBlankLines(body[1:])
}
if len(body) == 0 {
return section, nil
}
first := strings.TrimSpace(body[0])
if strings.HasPrefix(first, "Issued at ") {
issuedAt, err := parseForecastDiscussionIssueTime(first)
if isForecastDiscussionIssuedAtLine(body[0]) {
issuedAt, err := parseForecastDiscussionIssueTime(body[0])
if err != nil {
return ForecastDiscussionSection{}, fmt.Errorf("parse section issuedAt %q: %w", first, err)
return ForecastDiscussionSection{}, fmt.Errorf("parse section issuedAt %q: %w", strings.TrimSpace(body[0]), err)
}
tt := issuedAt.UTC()
section.IssuedAt = &tt
@@ -482,12 +696,147 @@ func parseForecastDiscussionTextSection(block []string) (ForecastDiscussionSecti
return section, nil
}
func parseForecastDiscussionQualifier(header string) string {
m := forecastDiscussionHeaderRE.FindStringSubmatch(header)
if len(m) != 3 {
return ""
func isForecastDiscussionPresentationMarker(line string) bool {
switch {
case strings.EqualFold(strings.TrimSpace(line), "-- Changed Discussion --"):
return true
case strings.EqualFold(strings.TrimSpace(line), "-- End Changed Discussion --"):
return true
default:
return false
}
return strings.TrimSpace(m[2])
}
func removeForecastDiscussionPresentationMarkers(lines []string) []string {
body := make([]string, 0, len(lines))
for _, line := range lines {
if !isForecastDiscussionPresentationMarker(line) {
body = append(body, line)
}
}
return body
}
func isForecastDiscussionStandaloneParenthetical(line string) bool {
line = strings.TrimSpace(line)
return len(line) > 2 && line[0] == '(' && line[len(line)-1] == ')' && strings.TrimSpace(line[1:len(line)-1]) != ""
}
func isForecastDiscussionIssuedAtLine(line string) bool {
line = strings.TrimSpace(line)
return len(line) > len("Issued at") &&
strings.EqualFold(line[:len("Issued at")], "Issued at") &&
isForecastDiscussionHorizontalWhitespace(line[len("Issued at")])
}
func isForecastDiscussionKeyMessageMetadataLine(line string) bool {
line = strings.TrimSpace(line)
for _, label := range []string{"Issued at", "Updated at"} {
if !hasForecastDiscussionASCIIPrefix(line, label) || len(line) == len(label) || !isForecastDiscussionHorizontalWhitespace(line[len(label)]) {
continue
}
if _, err := parseForecastDiscussionIssueTime(strings.TrimSpace(line[len(label):])); err == nil {
return true
}
}
return isForecastDiscussionKeyMessageAsOfLine(line)
}
func isForecastDiscussionKeyMessageAsOfLine(line string) bool {
const label = "As of"
if !hasForecastDiscussionASCIIPrefix(line, label) || len(line) == len(label) || !isForecastDiscussionHorizontalWhitespace(line[len(label)]) {
return false
}
remainder := strings.TrimSpace(line[len(label):])
if !strings.HasSuffix(remainder, "...") {
return false
}
fields := strings.Fields(strings.TrimSpace(strings.TrimSuffix(remainder, "...")))
if len(fields) != 3 {
return false
}
if _, _, err := parseForecastDiscussionClock(fields[0], fields[1]); err != nil {
return false
}
switch strings.ToLower(fields[2]) {
case "monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday":
return true
default:
return false
}
}
func stripForecastDiscussionKeyMessageMarker(line string) (string, bool) {
if line == "" {
return "", false
}
if line[0] == '-' || line[0] == '*' {
content := line[1:]
hadWhitespace := len(content) > 0 && isForecastDiscussionHorizontalWhitespace(content[0])
content = strings.TrimLeft(content, " \t")
if hadWhitespace {
if stripped, ok := stripForecastDiscussionKeyMessageNumericMarker(content); ok {
content = stripped
}
}
return content, true
}
return stripForecastDiscussionKeyMessageNumericMarker(line)
}
func stripForecastDiscussionKeyMessageNumericMarker(line string) (string, bool) {
digitStart := 0
digitEnd := 0
parenthesized := len(line) > 0 && line[0] == '('
if parenthesized {
digitStart = 1
digitEnd = 1
}
for digitEnd < len(line) && line[digitEnd] >= '0' && line[digitEnd] <= '9' {
digitEnd++
}
if digitEnd == digitStart || digitEnd == len(line) {
return "", false
}
if parenthesized && line[digitEnd] != ')' {
return "", false
}
if !parenthesized && line[digitEnd] != ')' && line[digitEnd] != '.' {
return "", false
}
markerEnd := digitEnd + 1
if markerEnd < len(line) && !isForecastDiscussionHorizontalWhitespace(line[markerEnd]) {
return "", false
}
value, err := strconv.ParseUint(line[digitStart:digitEnd], 10, 0)
if err != nil || value == 0 {
return "", false
}
return strings.TrimLeft(line[markerEnd:], " \t"), true
}
func hasForecastDiscussionASCIIPrefix(line, prefix string) bool {
if len(line) < len(prefix) {
return false
}
for i := range prefix {
actual := line[i]
if actual >= 'A' && actual <= 'Z' {
actual += 'a' - 'A'
}
expected := prefix[i]
if expected >= 'A' && expected <= 'Z' {
expected += 'a' - 'A'
}
if actual != expected {
return false
}
}
return true
}
func trimBlankLines(lines []string) []string {

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,26 @@
<!DOCTYPE html>
<!-- Representative BGM/CTP-style layout; prose is concise edited test data, not an archived product. -->
<html>
<body>
<pre class="glossaryProduct">
FXUS61 KBGM 101730
AFDBGM
Area Forecast Discussion
National Weather Service Binghamton NY
130 PM EDT Fri Apr 10 2026
.KEY MESSAGES...
1) Periods of rain are expected through Saturday,
with locally heavier amounts possible.
2. Cooler temperatures return late this weekend.
.DISCUSSION...
Discussion details remain boundary-only content.
$$
WFO BGM
</pre>
</body>
</html>

View File

@@ -0,0 +1,48 @@
<!DOCTYPE html>
<html>
<body>
<pre class="glossaryProduct">
FXUS65 KBOU 071900
AFDBOU
Area Forecast Discussion
National Weather Service Denver CO
100 PM MDT Tue Apr 7 2026
.KEY MESSAGES...
-- Changed Discussion --
Updated at 100 PM MDT Tue Apr 7 2026
- Strong winds are expected along the Front Range this evening.
- Cooler temperatures arrive on Wednesday.
-- End Changed Discussion --
&&
.SHORT TERM...
(Tonight through Wednesday)
Issued at 100 PM MDT Tue Apr 7 2026
Gusty west winds will continue through the evening before decreasing overnight.
&&
.LONG TERM...
(Thursday through Saturday)
ISSUED AT 100 PM MDT Tue Apr 7 2026
Warmer and drier conditions return Thursday, followed by a chance of showers Friday.
&&
.AVIATION...
VFR conditions are expected at Denver-area terminals through Wednesday morning.
&&
$$
WFO BOU
</pre>
</body>
</html>

View File

@@ -0,0 +1,27 @@
<!DOCTYPE html>
<!-- Representative LWX-style layout; prose is concise edited test data, not an archived product. -->
<html>
<body>
<pre class="glossaryProduct">
FXUS61 KLWX 021800
AFDLWX
Area Forecast Discussion
National Weather Service Baltimore MD/Washington DC
200 PM EDT Sun Aug 2 2026
.KEY MESSAGES...
- (1) Thunderstorms remain possible near the Blue Ridge this evening.
- (2) Seasonably warm conditions continue Monday.
&&
.AVIATION...
Aviation details remain boundary-only content.
$$
WFO LWX
</pre>
</body>
</html>

View File

@@ -0,0 +1,25 @@
<!DOCTYPE html>
<!-- Representative MFR-style layout; prose is concise edited test data, not an archived product. -->
<html>
<body>
<pre class="glossaryProduct">
FXUS66 KMFR 101945
AFDMFR
Area Forecast Discussion
National Weather Service Medford OR
1245 PM PDT Fri Apr 10 2026
.KEY POINTS...
* Gusty winds will develop over exposed ridges,
especially during the afternoon.
* Inland valleys remain dry through Saturday.
.DISCUSSION (Today through Thursday)...
Discussion details must not be included with key points.
$$
WFO MFR
</pre>
</body>
</html>

View File

@@ -0,0 +1,33 @@
<!DOCTYPE html>
<!-- Representative current MFR previous-discussion wrapper layout; prose is concise edited test data, not an archived product. -->
<html>
<body>
<pre class="glossaryProduct">
FXUS66 KMFR 022219
AFDMFR
Area Forecast Discussion
National Weather Service Medford OR
319 PM PDT Sun Aug 2 2026
.PREV DISCUSSION... /Issued 319 PM PDT Sun Aug 2 2026/
KEY MESSAGES...
* Heat returns to inland valleys Monday.
* Gusty afternoon winds develop east of the Cascades.
DISCUSSION...
Discussion details must not be included with key messages.
&&
.MFR WATCHES/WARNINGS/ADVISORIES...
None.
$$
WFO MFR
</pre>
</body>
</html>

View File

@@ -0,0 +1,29 @@
<!DOCTYPE html>
<!-- Representative RAH-style layout; prose is concise edited test data, not an archived product. -->
<html>
<body>
<pre class="glossaryProduct">
FXUS62 KRAH 021635
AFDRAH
Area Forecast Discussion
National Weather Service Raleigh NC
1235 PM EDT Sun Aug 2 2026
.KEY MESSAGES...
As of 1235 PM Sunday...
1) Scattered storms may produce locally heavy rain this afternoon.
2) Drier weather arrives Monday.
&&
.DISCUSSION...
Discussion details remain boundary-only content.
$$
WFO RAH
</pre>
</body>
</html>

View File

@@ -198,6 +198,7 @@
// - sent TIMESTAMPTZ NULL -> payload.alerts[i].sent
// - effective TIMESTAMPTZ NULL -> payload.alerts[i].effective
// - onset TIMESTAMPTZ NULL -> payload.alerts[i].onset
// - ends TIMESTAMPTZ NULL -> payload.alerts[i].ends
// - expires TIMESTAMPTZ NULL -> payload.alerts[i].expires
// - area_description TEXT NULL -> payload.alerts[i].areaDescription
// - sender_name TEXT NULL -> payload.alerts[i].senderName

View File

@@ -302,6 +302,7 @@ func mapAlertEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
"sent": nullableTime(a.Sent),
"effective": nullableTime(a.Effective),
"onset": nullableTime(a.Onset),
"ends": nullableTime(a.Ends),
"expires": nullableTime(a.Expires),
"area_description": nullableString(a.AreaDescription),
"sender_name": nullableString(a.SenderName),

View File

@@ -103,6 +103,8 @@ func TestMapPostgresEventForecastStructPayload(t *testing.T) {
func TestMapPostgresEventAlertStructPayload(t *testing.T) {
sent := time.Date(2026, 3, 16, 17, 0, 0, 0, time.UTC)
ends := time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC)
expires := time.Date(2026, 3, 16, 18, 30, 0, 0, time.UTC)
run := model.WeatherAlertRun{
AsOf: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
Alerts: []model.WeatherAlert{
@@ -110,6 +112,8 @@ func TestMapPostgresEventAlertStructPayload(t *testing.T) {
ID: "urn:alert:1",
Headline: "Winter Weather Advisory",
Severity: "Moderate",
Ends: &ends,
Expires: &expires,
References: []model.AlertReference{
{ID: "urn:ref:1", Sent: &sent},
{Identifier: "ref-two"},
@@ -145,6 +149,20 @@ func TestMapPostgresEventAlertStructPayload(t *testing.T) {
if got := firstAlert.Values["reference_count"]; got != 2 {
t.Fatalf("alerts reference_count = %#v, want 2", got)
}
if got := firstAlert.Values["ends"]; got != ends {
t.Fatalf("alerts ends = %#v, want %#v", got, ends)
}
if got := firstAlert.Values["expires"]; got != expires {
t.Fatalf("alerts expires = %#v, want %#v", got, expires)
}
alertWrites := writesForTable(writes, tableAlerts)
if len(alertWrites) != 2 {
t.Fatalf("alert writes len = %d, want 2", len(alertWrites))
}
if got := alertWrites[1].Values["ends"]; got != nil {
t.Fatalf("second alert ends = %#v, want nil", got)
}
assertAllWritesIncludeAllColumns(t, writes)
}
@@ -758,6 +776,16 @@ func firstWriteForTable(writes []fksinks.PostgresWrite, table string) (fksinks.P
return fksinks.PostgresWrite{}, false
}
func writesForTable(writes []fksinks.PostgresWrite, table string) []fksinks.PostgresWrite {
out := make([]fksinks.PostgresWrite, 0)
for _, w := range writes {
if w.Table == table {
out = append(out, w)
}
}
return out
}
func assertAllWritesIncludeAllColumns(t *testing.T, writes []fksinks.PostgresWrite) {
t.Helper()
colCounts := tableColumnCounts()

View File

@@ -238,6 +238,7 @@ func PostgresSchema() fksinks.PostgresSchema {
{Name: "sent", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "effective", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "onset", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "ends", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "expires", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "area_description", Type: "TEXT", Nullable: true},
{Name: "sender_name", Type: "TEXT", Nullable: true},

View File

@@ -91,6 +91,15 @@ func TestWeatherPostgresSchemaIncludesOutlookTables(t *testing.T) {
assertTableUniqueIndex(t, tableOutlookDiscussions, "idx_wf_outlook_discussions_run_day", []string{"run_event_id", "day"})
}
func TestWeatherPostgresSchemaIncludesAlertEndsColumn(t *testing.T) {
alertColumns := columnsForTable(t, tableAlerts)
for _, col := range []string{"run_event_id", "alert_index", "as_of", "alert_id", "onset", "ends", "expires"} {
if !alertColumns[col] {
t.Fatalf("%s missing %s column", tableAlerts, col)
}
}
}
func TestWeatherPostgresSchemaIncludesWeatherStoryColumns(t *testing.T) {
runColumns := columnsForTable(t, tableWeatherStoryRuns)
if !runColumns["as_of"] {

View File

@@ -55,9 +55,11 @@ type WeatherAlert struct {
Instruction string `json:"instruction,omitempty"`
// Timing (all optional; provider-dependent).
// Onset and Ends describe the alert period. Expires is provider expiration metadata.
Sent *time.Time `json:"sent,omitempty"`
Effective *time.Time `json:"effective,omitempty"`
Onset *time.Time `json:"onset,omitempty"`
Ends *time.Time `json:"ends,omitempty"`
Expires *time.Time `json:"expires,omitempty"`
// Scope / affected area.