7 Commits

Author SHA1 Message Date
fd820fd964 Implemented NWS weather stories support
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-05-30 06:47:18 -05:00
cca873cafb Made forecast-period conditionCode optional
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
2026-05-28 07:47:07 -05:00
f457bab039 Updated dependencies to feedkit v0.9.1
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-03-29 10:54:36 -05:00
6712c16167 Updated to feedkit v0.9.0
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-03-29 08:35:56 -05:00
a0389ebce8 Added support for Area Forecast Discussions issued by the NWS
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-03-28 16:17:03 -05:00
40f17c9d86 Updates to track upstream feedkit v0.8.2
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-03-28 13:53:54 -05:00
c76088c38c Code cleanup and deduplication pass through weatherfeeder
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
ci/woodpecker/manual/build-image Pipeline was successful
2026-03-28 12:01:07 -05:00
51 changed files with 3098 additions and 462 deletions

109
API.md
View File

@@ -37,14 +37,28 @@ Examples:
## Canonical schemas ## Canonical schemas
weatherfeeder emits three canonical domain schemas: weatherfeeder emits five canonical domain schemas:
- `weather.observation.v1` - `weather.observation.v1`
- `weather.forecast.v1` - `weather.forecast.v1`
- `weather.forecast_discussion.v1`
- `weather.weather_story.v1`
- `weather.alert.v1` - `weather.alert.v1`
Each payload is described below using the JSON field names as the contract. Each payload is described below using the JSON field names as the contract.
### Raw upstream schemas
weatherfeeder sources also emit provider-specific raw schemas before normalization.
Relevant raw source schemas include:
- `raw.nws.forecast_discussion.v1`
- payload type: string
- payload contents: exact fetched HTML response body
- `raw.nws.weatherstories.v1`
- payload type: object
- payload contents: exact fetched JSON response body
--- ---
## Shared Conventions ## Shared Conventions
@@ -142,7 +156,7 @@ A `WeatherForecastPeriod` is valid for `[startTime, endTime)`.
| `endTime` | string (timestamp) | yes | Period end | | `endTime` | string (timestamp) | yes | Period end |
| `name` | string | no | Human label (often empty for hourly) | | `name` | string | no | Human label (often empty for hourly) |
| `isDay` | bool | no | Day/night hint | | `isDay` | bool | no | Day/night hint |
| `conditionCode` | int | yes | WMO code (`-1` for unknown) | | `conditionCode` | int | no | WMO code when applicable (`-1` for unknown) |
| `textDescription` | string | no | Human-facing short phrase | | `textDescription` | string | no | Human-facing short phrase |
| `temperatureC` | number | no | °C | | `temperatureC` | number | no | °C |
| `temperatureCMin` | number | no | °C (aggregated products) | | `temperatureCMin` | number | no | °C (aggregated products) |
@@ -217,6 +231,68 @@ A run may contain zero, one, or many alerts.
--- ---
## Schema: `weather.weather_story.v1`
Payload type: `WeatherStoryRun`
A `WeatherStoryRun` is a snapshot of NWS weather stories for an office as-of a point in time.
The run may contain zero, one, or many stories.
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `officeId` | string | no | NWS office identifier, e.g. `LSX` |
| `asOf` | string (timestamp) | yes | Latest story update time or source fallback |
| `stories` | array | yes | Weather stories (order provider-dependent) |
### Nested: `stories[]` (`WeatherStory`)
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `officeId` | string | no | NWS office identifier |
| `startTime` | string (timestamp) | yes | Story validity start |
| `endTime` | string (timestamp) | yes | Story validity end |
| `updatedAt` | string (timestamp) | yes | Story update time |
| `title` | string | no | Human story title |
| `description` | string | no | Story narrative text |
| `altText` | string | no | Accessibility text for the provider graphic |
| `priority` | bool | yes | Provider priority flag |
| `order` | int | yes | Provider display order |
| `downloadUrl` | string | no | Provider download URL; weatherfeeder does not fetch the asset |
---
## Schema: `weather.forecast_discussion.v1`
Payload type: `WeatherForecastDiscussion`
A `WeatherForecastDiscussion` is an issued narrative bulletin for an NWS office.
It is distinct from `weather.forecast.v1`, which is period-based.
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `officeId` | string | no | NWS office identifier, e.g. `LSX` |
| `officeName` | string | no | Human office name |
| `product` | string | yes | Currently `afd` |
| `issuedAt` | string (timestamp) | yes | Bulletin issue time |
| `updatedAt` | string (timestamp) | no | Optional page/update timestamp |
| `keyMessages` | array | no | Ordered key-message bullet list |
| `shortTerm` | object | no | Short-term discussion section |
| `longTerm` | object | no | Long-term discussion section |
### Nested: `shortTerm` / `longTerm`
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `qualifier` | string | no | Header qualifier such as `(Through Late Sunday Night)` |
| `issuedAt` | string (timestamp) | no | Optional section-local issue time |
| `text` | string | no | Paragraph-preserved prose text |
---
## Compatibility rules ## Compatibility rules
- Consumers **must** ignore unknown fields. - Consumers **must** ignore unknown fields.
@@ -297,3 +373,32 @@ A run may contain zero, one, or many alerts.
} }
} }
``` ```
### Weather story event (`weather.weather_story.v1`)
```json
{
"id": "nws:weatherstories:2026-05-30T09:00:34Z",
"schema": "weather.weather_story.v1",
"source": "nws_weatherstories",
"effectiveAt": "2026-05-30T09:00:34Z",
"payload": {
"officeId": "LSX",
"asOf": "2026-05-30T09:00:34Z",
"stories": [
{
"officeId": "LSX",
"startTime": "2026-05-30T08:46:00Z",
"endTime": "2026-05-31T11:00:00Z",
"updatedAt": "2026-05-30T09:00:34Z",
"title": "Several Chances for Rain Through Monday",
"description": "Scattered showers and thunderstorms remain possible.",
"altText": "This slide shows the forecast for today through Tuesday.",
"priority": false,
"order": 1,
"downloadUrl": "https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f"
}
]
}
}
```

View File

@@ -14,6 +14,8 @@ Canonical domain schemas emitted after normalization:
- `weather.observation.v1``WeatherObservation` - `weather.observation.v1``WeatherObservation`
- `weather.forecast.v1``WeatherForecastRun` - `weather.forecast.v1``WeatherForecastRun`
- `weather.forecast_discussion.v1``WeatherForecastDiscussion`
- `weather.weather_story.v1``WeatherStoryRun`
- `weather.alert.v1``WeatherAlertRun` - `weather.alert.v1``WeatherAlertRun`
For the complete wire contract (event envelope + payload schemas, fields, units, and compatibility rules), see: For the complete wire contract (event envelope + payload schemas, fields, units, and compatibility rules), see:
@@ -22,7 +24,7 @@ For the complete wire contract (event envelope + payload schemas, fields, units,
## Upstream providers (current MVP) ## Upstream providers (current MVP)
- NWS: observations, hourly forecasts, narrative forecasts, alerts - NWS: observations, hourly forecasts, narrative forecasts, forecast discussions, weather stories, alerts
- Open-Meteo: observations, hourly forecasts - Open-Meteo: observations, hourly forecasts
- OpenWeather: observations - OpenWeather: observations

View File

@@ -15,7 +15,7 @@ sources:
# driver: openmeteo_observation # driver: openmeteo_observation
# every: 10m # every: 10m
# params: # params:
# url: "https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&current=temperature_2m,relative_humidity_2m,weather_code,wind_speed_10m,wind_direction_10m,precipitation,surface_pressure,rain,showers,snowfall,cloud_cover,apparent_temperature,is_day,wind_gusts_10m,pressure_msl&forecast_days=1" # url: "https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&current=temperature_2m,relative_humidity_2m,weather_code,wind_speed_10m,wind_direction_10m,precipitation,surface_pressure,rain,showers,snowfall,cloud_cover,apparent_temperature,is_day,wind_gusts_10m,pressure_msl"
# user_agent: "HomeOps (eric@maximumdirect.net)" # user_agent: "HomeOps (eric@maximumdirect.net)"
# - name: OpenWeatherObservation # - name: OpenWeatherObservation
@@ -63,6 +63,24 @@ sources:
url: "https://api.weather.gov/gridpoints/LSX/90,74/forecast?units=us" url: "https://api.weather.gov/gridpoints/LSX/90,74/forecast?units=us"
user_agent: "HomeOps (eric@maximumdirect.net)" user_agent: "HomeOps (eric@maximumdirect.net)"
- name: NWSForecastDiscussionSTL
mode: poll
kinds: ["forecast_discussion"]
driver: nws_forecast_discussion
every: 30m
params:
url: "https://forecast.weather.gov/product.php?site=LSX&issuedby=LSX&product=AFD&format=TXT&version=1&glossary=0"
user_agent: "HomeOps (eric@maximumdirect.net)"
- name: NWSWeatherStoriesSTL
mode: poll
kinds: ["weather_story"]
driver: nws_weatherstories
every: 30m
params:
url: "https://api.weather.gov/offices/LSX/weatherstories"
user_agent: "HomeOps (eric@maximumdirect.net)"
- name: OpenMeteoHourlyForecastSTL - name: OpenMeteoHourlyForecastSTL
mode: poll mode: poll
kinds: ["forecast"] kinds: ["forecast"]
@@ -90,7 +108,7 @@ sinks:
driver: nats driver: nats
params: params:
url: nats://nats:4222 url: nats://nats:4222
exchange: weatherfeeder subject: weatherfeeder
# - name: pg_weatherfeeder # - name: pg_weatherfeeder
# driver: postgres # driver: postgres
@@ -108,13 +126,13 @@ sinks:
routes: routes:
- sink: stdout - sink: stdout
kinds: ["observation", "forecast", "alert"] kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"]
- sink: nats_weatherfeeder - sink: nats_weatherfeeder
kinds: ["observation", "forecast", "alert"] kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"]
# - sink: pg_weatherfeeder # - sink: pg_weatherfeeder
# kinds: ["observation", "forecast", "alert"] # kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"]
# - sink: logfile # - sink: logfile
# kinds: ["observation", "alert", "forecast"] # kinds: ["observation", "alert", "forecast", "forecast_discussion", "weather_story"]

View File

@@ -37,9 +37,6 @@ func main() {
if err != nil { if err != nil {
log.Fatalf("config load failed: %v", err) log.Fatalf("config load failed: %v", err)
} }
if err := fksinks.RegisterPostgresSchemaForConfiguredSinks(cfg, wfpgsink.PostgresSchema()); err != nil {
log.Fatalf("postgres schema registration failed: %v", err)
}
// --- Registries --- // --- Registries ---
srcReg := fksources.NewRegistry() srcReg := fksources.NewRegistry()
@@ -48,6 +45,7 @@ func main() {
// Compile stdout, Postgres, and NATS sinks for weatherfeeder. The former is useful for debugging and the latter are the main intended outputs. // Compile stdout, Postgres, and NATS sinks for weatherfeeder. The former is useful for debugging and the latter are the main intended outputs.
sinkReg := fksinks.NewRegistry() sinkReg := fksinks.NewRegistry()
fksinks.RegisterBuiltins(sinkReg) fksinks.RegisterBuiltins(sinkReg)
sinkReg.Register("postgres", fksinks.PostgresFactory(wfpgsink.PostgresSchema()))
// --- Build sources into scheduler jobs --- // --- Build sources into scheduler jobs ---
var jobs []fkscheduler.Job var jobs []fkscheduler.Job
@@ -61,19 +59,11 @@ func main() {
log.Fatalf("source expected kinds validation failed (sources[%d] name=%q driver=%q): %v", i, sc.Name, sc.Driver, err) log.Fatalf("source expected kinds validation failed (sources[%d] name=%q driver=%q): %v", i, sc.Name, sc.Driver, err)
} }
// If this is a polling source, every is required. job, err := fkscheduler.JobFromSourceConfig(in, sc)
if _, ok := in.(fksources.PollSource); ok && sc.Every.Duration <= 0 { if err != nil {
log.Fatalf( log.Fatalf("build scheduler job failed (sources[%d] name=%q driver=%q): %v", i, sc.Name, sc.Driver, err)
"polling source missing/invalid interval (sources[%d] name=%q driver=%q): sources[].every must be > 0",
i, sc.Name, sc.Driver,
)
} }
jobs = append(jobs, job)
// For stream sources, Every is ignored; it is fine if omitted/zero.
jobs = append(jobs, fkscheduler.Job{
Source: in,
Every: sc.Every.Duration,
})
} }
// --- Build sinks --- // --- Build sinks ---

View File

@@ -13,9 +13,11 @@ import (
fkprocessors "gitea.maximumdirect.net/ejr/feedkit/processors" fkprocessors "gitea.maximumdirect.net/ejr/feedkit/processors"
fkdedupe "gitea.maximumdirect.net/ejr/feedkit/processors/dedupe" fkdedupe "gitea.maximumdirect.net/ejr/feedkit/processors/dedupe"
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize" fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
fkscheduler "gitea.maximumdirect.net/ejr/feedkit/scheduler"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources" fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
wfnormalizers "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers" wfnormalizers "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers"
wfsources "gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources"
) )
type testInput struct { type testInput struct {
@@ -24,13 +26,6 @@ type testInput struct {
func (s testInput) Name() string { return s.name } func (s testInput) Name() string { return s.name }
type testKindSource struct {
testInput
kind fkevent.Kind
}
func (s testKindSource) Kind() fkevent.Kind { return s.kind }
type testKindsSource struct { type testKindsSource struct {
testInput testInput
kinds []fkevent.Kind kinds []fkevent.Kind
@@ -38,18 +33,6 @@ type testKindsSource struct {
func (s testKindsSource) Kinds() []fkevent.Kind { return s.kinds } func (s testKindsSource) Kinds() []fkevent.Kind { return s.kinds }
func TestValidateSourceExpectedKindsLegacyKindFallback(t *testing.T) {
sc := config.SourceConfig{Kind: "observation"}
in := testKindSource{
testInput: testInput{name: "test"},
kind: fkevent.Kind("observation"),
}
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
t.Fatalf("ValidateExpectedKinds() unexpected error: %v", err)
}
}
func TestValidateSourceExpectedKindsSubsetAllowed(t *testing.T) { func TestValidateSourceExpectedKindsSubsetAllowed(t *testing.T) {
sc := config.SourceConfig{Kinds: []string{"observation"}} sc := config.SourceConfig{Kinds: []string{"observation"}}
in := testKindsSource{ in := testKindsSource{
@@ -93,6 +76,31 @@ func TestExampleConfigLoads(t *testing.T) {
} }
} }
func TestExampleConfigSourcesBuildSchedulerJobs(t *testing.T) {
cfg, err := config.Load("config.yml")
if err != nil {
t.Fatalf("config.Load(config.yml) unexpected error: %v", err)
}
reg := fksources.NewRegistry()
wfsources.RegisterBuiltins(reg)
for i, sc := range cfg.Sources {
in, err := reg.BuildInput(sc)
if err != nil {
t.Fatalf("BuildInput(sources[%d]) error = %v", i, err)
}
job, err := fkscheduler.JobFromSourceConfig(in, sc)
if err != nil {
t.Fatalf("JobFromSourceConfig(sources[%d]) error = %v", i, err)
}
if job.Source == nil {
t.Fatalf("JobFromSourceConfig(sources[%d]) returned nil source", i)
}
}
}
func TestProcessorRegistryBuildsNormalizeThenDedupeChain(t *testing.T) { func TestProcessorRegistryBuildsNormalizeThenDedupeChain(t *testing.T) {
chain, err := buildProcessorChainForTests() chain, err := buildProcessorChainForTests()
if err != nil { if err != nil {

2
go.mod
View File

@@ -2,7 +2,7 @@ module gitea.maximumdirect.net/ejr/weatherfeeder
go 1.25 go 1.25
require gitea.maximumdirect.net/ejr/feedkit v0.8.1 require gitea.maximumdirect.net/ejr/feedkit v0.9.1
require ( require (
github.com/klauspost/compress v1.17.2 // indirect github.com/klauspost/compress v1.17.2 // indirect

4
go.sum
View File

@@ -1,5 +1,5 @@
gitea.maximumdirect.net/ejr/feedkit v0.8.1 h1:gSZ5J/mYHfNZ6dp27TS0V9RQ0JuP5ZAHXhSeCJaBu60= gitea.maximumdirect.net/ejr/feedkit v0.9.1 h1:YghBQT1podqc+FJuPGuIZImV4A9dMr56Hikd5xuniig=
gitea.maximumdirect.net/ejr/feedkit v0.8.1/go.mod h1:U6xC9xZLN3cL4yi7YBVyzGoHYRLJXusFCAKlj2kdYYQ= gitea.maximumdirect.net/ejr/feedkit v0.9.1/go.mod h1:U6xC9xZLN3cL4yi7YBVyzGoHYRLJXusFCAKlj2kdYYQ=
github.com/klauspost/compress v1.17.2 h1:RlWWUY/Dr4fL8qk9YG7DTZ7PDgME2V4csBXA8L/ixi4= github.com/klauspost/compress v1.17.2 h1:RlWWUY/Dr4fL8qk9YG7DTZ7PDgME2V4csBXA8L/ixi4=
github.com/klauspost/compress v1.17.2/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/klauspost/compress v1.17.2/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw= github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=

View File

@@ -9,6 +9,12 @@ import (
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openweather" "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openweather"
) )
var builtinRegistrations = []func([]fknormalize.Normalizer) []fknormalize.Normalizer{
nws.Register,
openmeteo.Register,
openweather.Register,
}
// RegisterBuiltins registers all normalizers shipped with this binary. // RegisterBuiltins registers all normalizers shipped with this binary.
// //
// This mirrors internal/sources.RegisterBuiltins, but note the selection model: // This mirrors internal/sources.RegisterBuiltins, but note the selection model:
@@ -27,9 +33,9 @@ func RegisterBuiltins(in []fknormalize.Normalizer) []fknormalize.Normalizer {
// //
// Order here should be stable across releases to reduce surprises when adding // Order here should be stable across releases to reduce surprises when adding
// new normalizers. // new normalizers.
out = nws.Register(out) for _, register := range builtinRegistrations {
out = openmeteo.Register(out) out = register(out)
out = openweather.Register(out) }
return out return out
} }

View File

@@ -19,6 +19,8 @@ func TestRegisterBuiltinsOrder(t *testing.T) {
want := []fknormalize.Normalizer{ want := []fknormalize.Normalizer{
nws.ObservationNormalizer{}, nws.ObservationNormalizer{},
nws.ForecastNormalizer{}, nws.ForecastNormalizer{},
nws.ForecastDiscussionNormalizer{},
nws.WeatherStoriesNormalizer{},
nws.AlertsNormalizer{}, nws.AlertsNormalizer{},
openmeteo.ObservationNormalizer{}, openmeteo.ObservationNormalizer{},
openmeteo.ForecastNormalizer{}, openmeteo.ForecastNormalizer{},

View File

@@ -23,10 +23,11 @@ import (
// builders by raw schema. // builders by raw schema.
// //
// Caveats / policy: // Caveats / policy:
// 1. NWS forecast periods do not include METAR presentWeather phenomena, so ConditionCode // 1. Hourly NWS forecast periods do not include METAR presentWeather phenomena, so
// is inferred from period.shortForecast (with a conservative icon-based fallback). // ConditionCode is inferred from period.shortForecast (with a conservative icon fallback).
// 2. Temperature is converted to °C when NWS supplies °F. // 2. Narrative NWS periods intentionally leave ConditionCode unset.
// 3. WindSpeed is parsed from strings like "9 mph" / "10 to 15 mph" and converted to km/h. // 3. Temperature is converted to °C when NWS supplies °F.
// 4. WindSpeed is parsed from strings like "9 mph" / "10 to 15 mph" and converted to km/h.
type ForecastNormalizer struct{} type ForecastNormalizer struct{}
func (ForecastNormalizer) Match(e event.Event) bool { func (ForecastNormalizer) Match(e event.Event) bool {
@@ -75,62 +76,63 @@ func normalizeNarrativeForecastEvent(in event.Event) (*event.Event, error) {
) )
} }
type forecastPeriodMapper[T any] func(idx int, period T) (model.WeatherForecastPeriod, error)
// buildHourlyForecast contains hourly forecast mapping logic (provider -> canonical model). // buildHourlyForecast contains hourly forecast mapping logic (provider -> canonical model).
func buildHourlyForecast(parsed nwsHourlyForecastResponse) (model.WeatherForecastRun, time.Time, error) { func buildHourlyForecast(parsed nwsHourlyForecastResponse) (model.WeatherForecastRun, time.Time, error) {
issuedAt, updatedAt, err := parseForecastRunTimes(parsed.Properties.GeneratedAt, parsed.Properties.UpdateTime) return buildForecastRun(
if err != nil { parsed.Properties.GeneratedAt,
return model.WeatherForecastRun{}, time.Time{}, err parsed.Properties.UpdateTime,
} parsed.Geometry.Coordinates,
// Best-effort location centroid from the GeoJSON polygon (optional).
lat, lon := centroidLatLon(parsed.Geometry.Coordinates)
run := newForecastRunBase(
issuedAt,
updatedAt,
model.ForecastProductHourly,
lat,
lon,
parsed.Properties.Elevation.Value, parsed.Properties.Elevation.Value,
model.ForecastProductHourly,
parsed.Properties.Periods,
mapHourlyForecastPeriod,
) )
periods := make([]model.WeatherForecastPeriod, 0, len(parsed.Properties.Periods))
for i, p := range parsed.Properties.Periods {
period, err := mapHourlyForecastPeriod(i, p)
if err != nil {
return model.WeatherForecastRun{}, time.Time{}, err
}
periods = append(periods, period)
}
run.Periods = periods
// EffectiveAt policy for forecasts: treat IssuedAt as the effective time (dedupe-friendly).
return run, issuedAt, nil
} }
// buildNarrativeForecast contains narrative forecast mapping logic (provider -> canonical model). // buildNarrativeForecast contains narrative forecast mapping logic (provider -> canonical model).
func buildNarrativeForecast(parsed nwsNarrativeForecastResponse) (model.WeatherForecastRun, time.Time, error) { func buildNarrativeForecast(parsed nwsNarrativeForecastResponse) (model.WeatherForecastRun, time.Time, error) {
issuedAt, updatedAt, err := parseForecastRunTimes(parsed.Properties.GeneratedAt, parsed.Properties.UpdateTime) return buildForecastRun(
parsed.Properties.GeneratedAt,
parsed.Properties.UpdateTime,
parsed.Geometry.Coordinates,
parsed.Properties.Elevation.Value,
model.ForecastProductNarrative,
parsed.Properties.Periods,
mapNarrativeForecastPeriod,
)
}
func buildForecastRun[T any](
generatedAt string,
updateTime string,
coordinates [][][]float64,
elevation *float64,
product model.ForecastProduct,
srcPeriods []T,
mapPeriod forecastPeriodMapper[T],
) (model.WeatherForecastRun, time.Time, error) {
issuedAt, updatedAt, err := parseForecastRunTimes(generatedAt, updateTime)
if err != nil { if err != nil {
return model.WeatherForecastRun{}, time.Time{}, err return model.WeatherForecastRun{}, time.Time{}, err
} }
// Best-effort location centroid from the GeoJSON polygon (optional). // Best-effort location centroid from the GeoJSON polygon (optional).
lat, lon := centroidLatLon(parsed.Geometry.Coordinates) lat, lon := centroidLatLon(coordinates)
run := newForecastRunBase( run := newForecastRunBase(
issuedAt, issuedAt,
updatedAt, updatedAt,
model.ForecastProductNarrative, product,
lat, lat,
lon, lon,
parsed.Properties.Elevation.Value, elevation,
) )
periods := make([]model.WeatherForecastPeriod, 0, len(parsed.Properties.Periods)) periods := make([]model.WeatherForecastPeriod, 0, len(srcPeriods))
for i, p := range parsed.Properties.Periods { for i, p := range srcPeriods {
period, err := mapNarrativeForecastPeriod(i, p) period, err := mapPeriod(i, p)
if err != nil { if err != nil {
return model.WeatherForecastRun{}, time.Time{}, err return model.WeatherForecastRun{}, time.Time{}, err
} }
@@ -223,6 +225,7 @@ func mapHourlyForecastPeriod(idx int, p nwsHourlyForecastPeriod) (model.WeatherF
// Infer WMO from shortForecast (and fall back to icon token). // Infer WMO from shortForecast (and fall back to icon token).
providerDesc := strings.TrimSpace(p.ShortForecast) providerDesc := strings.TrimSpace(p.ShortForecast)
wmo := wmoFromNWSForecast(providerDesc, p.Icon, tempC) wmo := wmoFromNWSForecast(providerDesc, p.Icon, tempC)
wmoPtr := wmoCodePtr(wmo)
return model.WeatherForecastPeriod{ return model.WeatherForecastPeriod{
StartTime: start, StartTime: start,
@@ -231,7 +234,7 @@ func mapHourlyForecastPeriod(idx int, p nwsHourlyForecastPeriod) (model.WeatherF
Name: strings.TrimSpace(p.Name), Name: strings.TrimSpace(p.Name),
IsDay: isDay, IsDay: isDay,
ConditionCode: wmo, ConditionCode: wmoPtr,
// For forecasts, keep provider short forecast text as the human-facing description. // For forecasts, keep provider short forecast text as the human-facing description.
TextDescription: providerDesc, TextDescription: providerDesc,
@@ -263,9 +266,7 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
tempC := tempCFromNWS(p.Temperature, p.TemperatureUnit) tempC := tempCFromNWS(p.Temperature, p.TemperatureUnit)
// Infer WMO from shortForecast (and fall back to icon token).
shortForecast := strings.TrimSpace(p.ShortForecast) shortForecast := strings.TrimSpace(p.ShortForecast)
wmo := wmoFromNWSForecast(shortForecast, p.Icon, tempC)
textDescription := strings.TrimSpace(p.DetailedForecast) textDescription := strings.TrimSpace(p.DetailedForecast)
if textDescription == "" { if textDescription == "" {
@@ -279,7 +280,7 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
Name: strings.TrimSpace(p.Name), Name: strings.TrimSpace(p.Name),
IsDay: isDay, IsDay: isDay,
ConditionCode: wmo, ConditionCode: nil,
TextDescription: textDescription, TextDescription: textDescription,
@@ -291,3 +292,8 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
ProbabilityOfPrecipitationPercent: p.ProbabilityOfPrecipitation.Value, ProbabilityOfPrecipitationPercent: p.ProbabilityOfPrecipitation.Value,
}, nil }, nil
} }
func wmoCodePtr(code model.WMOCode) *model.WMOCode {
out := code
return &out
}

View File

@@ -0,0 +1,72 @@
package nws
import (
"context"
"fmt"
"strings"
"gitea.maximumdirect.net/ejr/feedkit/event"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
type ForecastDiscussionNormalizer struct{}
func (ForecastDiscussionNormalizer) Match(e event.Event) bool {
return strings.TrimSpace(e.Schema) == standards.SchemaRawNWSForecastDiscussionV1
}
func (ForecastDiscussionNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx
rawHTML, err := decodeStringPayload(in.Payload)
if err != nil {
return nil, fmt.Errorf("nws forecast discussion normalize: %w", err)
}
parsed, err := nwscommon.ParseForecastDiscussionHTML(rawHTML)
if err != nil {
return nil, fmt.Errorf("nws forecast discussion normalize: build: %w", err)
}
payload := model.WeatherForecastDiscussion{
OfficeID: strings.TrimSpace(parsed.OfficeID),
OfficeName: strings.TrimSpace(parsed.OfficeName),
Product: model.ForecastDiscussionProduct(strings.TrimSpace(parsed.Product)),
IssuedAt: parsed.IssuedAt.UTC(),
UpdatedAt: parsed.UpdatedAt,
KeyMessages: append([]string(nil), parsed.KeyMessages...),
ShortTerm: mapForecastDiscussionSection(parsed.ShortTerm),
LongTerm: mapForecastDiscussionSection(parsed.LongTerm),
}
out, err := normcommon.Finalize(in, standards.SchemaWeatherForecastDiscussionV1, payload, payload.IssuedAt)
if err != nil {
return nil, fmt.Errorf("nws forecast discussion normalize: %w", err)
}
return out, nil
}
func mapForecastDiscussionSection(in *nwscommon.ForecastDiscussionSection) *model.WeatherForecastDiscussionSection {
if in == nil {
return nil
}
return &model.WeatherForecastDiscussionSection{
Qualifier: strings.TrimSpace(in.Qualifier),
IssuedAt: in.IssuedAt,
Text: strings.TrimSpace(in.Text),
}
}
func decodeStringPayload(payload any) (string, error) {
switch v := payload.(type) {
case string:
return v, nil
case []byte:
return string(v), nil
default:
return "", fmt.Errorf("extract payload: expected string payload, got %T", payload)
}
}

View File

@@ -0,0 +1,130 @@
package nws
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
func TestForecastDiscussionNormalizerProducesCanonicalSchema(t *testing.T) {
rawHTML := loadForecastDiscussionSampleHTML(t)
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
ID: "evt-discussion-1",
Kind: event.Kind("forecast_discussion"),
Source: "nws-discussion-test",
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
Schema: standards.SchemaRawNWSForecastDiscussionV1,
Payload: rawHTML,
})
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
if out == nil {
t.Fatalf("Normalize() returned nil output")
}
if out.Schema != standards.SchemaWeatherForecastDiscussionV1 {
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastDiscussionV1)
}
if out.Kind != event.Kind("forecast_discussion") {
t.Fatalf("Kind = %q, want forecast_discussion", out.Kind)
}
payload, ok := out.Payload.(model.WeatherForecastDiscussion)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherForecastDiscussion", out.Payload)
}
if payload.OfficeID != "LSX" {
t.Fatalf("OfficeID = %q, want LSX", payload.OfficeID)
}
if payload.Product != model.ForecastDiscussionProductAFD {
t.Fatalf("Product = %q, want %q", payload.Product, model.ForecastDiscussionProductAFD)
}
if len(payload.KeyMessages) != 3 {
t.Fatalf("KeyMessages len = %d, want 3", len(payload.KeyMessages))
}
if payload.ShortTerm == nil || payload.LongTerm == nil {
t.Fatalf("ShortTerm=%v LongTerm=%v, want both populated", payload.ShortTerm, payload.LongTerm)
}
if payload.ShortTerm.Qualifier != "(Through Late Sunday Night)" {
t.Fatalf("ShortTerm.Qualifier = %q", payload.ShortTerm.Qualifier)
}
if !strings.Contains(payload.ShortTerm.Text, "After a chilly morning") {
t.Fatalf("ShortTerm.Text = %q, want normalized prose", payload.ShortTerm.Text)
}
if strings.Contains(payload.ShortTerm.Text, "BRC") {
t.Fatalf("ShortTerm.Text contains signature: %q", payload.ShortTerm.Text)
}
if strings.Contains(payload.LongTerm.Text, "AVIATION") || strings.Contains(payload.LongTerm.Text, "WATCHES/WARNINGS/ADVISORIES") {
t.Fatalf("LongTerm.Text includes downstream sections: %q", payload.LongTerm.Text)
}
wantEffectiveAt := time.Date(2026, 3, 28, 19, 24, 0, 0, time.UTC)
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
}
}
func TestForecastDiscussionNormalizerRejectsMissingIssueTime(t *testing.T) {
_, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
ID: "evt-discussion-bad",
Kind: event.Kind("forecast_discussion"),
Source: "nws-discussion-test",
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
Schema: standards.SchemaRawNWSForecastDiscussionV1,
Payload: "<html><body><pre class=\"glossaryProduct\">National Weather Service Saint Louis MO</pre></body></html>",
})
if err == nil {
t.Fatalf("Normalize() error = nil, want error")
}
if !strings.Contains(err.Error(), "issue time") {
t.Fatalf("error = %q, want issue time context", err)
}
}
func TestForecastDiscussionNormalizerWireShapeHasNoUnexpectedKeys(t *testing.T) {
rawHTML := loadForecastDiscussionSampleHTML(t)
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
ID: "evt-discussion-2",
Kind: event.Kind("forecast_discussion"),
Source: "nws-discussion-test",
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
Schema: standards.SchemaRawNWSForecastDiscussionV1,
Payload: rawHTML,
})
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
b, err := json.Marshal(out.Payload)
if err != nil {
t.Fatalf("json.Marshal(payload) error = %v", err)
}
var got map[string]any
if err := json.Unmarshal(b, &got); err != nil {
t.Fatalf("json.Unmarshal(payload) error = %v", err)
}
for _, key := range []string{"sections", "aviation"} {
if _, ok := got[key]; ok {
t.Fatalf("unexpected key %q in canonical payload", key)
}
}
}
func loadForecastDiscussionSampleHTML(t *testing.T) string {
t.Helper()
path := filepath.Join("..", "..", "providers", "nws", "testdata", "forecast_discussion_sample.html")
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("os.ReadFile(%q) error = %v", path, err)
}
return string(b)
}

View File

@@ -35,6 +35,9 @@ func TestBuildHourlyForecastUsesShortForecastAsTextDescription(t *testing.T) {
if got, want := run.Periods[0].TextDescription, "Mostly Cloudy"; got != want { if got, want := run.Periods[0].TextDescription, "Mostly Cloudy"; got != want {
t.Fatalf("TextDescription = %q, want %q", got, want) t.Fatalf("TextDescription = %q, want %q", got, want)
} }
if run.Periods[0].ConditionCode == nil {
t.Fatalf("ConditionCode is nil, want inferred hourly WMO code")
}
wantIssued := time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC) wantIssued := time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC)
if !run.IssuedAt.Equal(wantIssued) { if !run.IssuedAt.Equal(wantIssued) {
@@ -47,6 +50,55 @@ func TestBuildHourlyForecastUsesShortForecastAsTextDescription(t *testing.T) {
assertNoLegacyForecastDescriptionKeys(t, run.Periods[0]) assertNoLegacyForecastDescriptionKeys(t, run.Periods[0])
} }
func TestBuildHourlyForecastPreservesUpdatedAtCentroidAndElevation(t *testing.T) {
parsed := nwsHourlyForecastResponse{}
parsed.Properties.GeneratedAt = "2026-03-16T18:00:00Z"
parsed.Properties.UpdateTime = "2026-03-16T18:30:00Z"
elevation := 123.4
parsed.Properties.Elevation.Value = &elevation
parsed.Geometry.Coordinates = [][][]float64{
{
{-90.0, 38.0},
{-89.0, 38.0},
{-89.0, 39.0},
{-90.0, 39.0},
},
}
parsed.Properties.Periods = []nwsHourlyForecastPeriod{
{
StartTime: "2026-03-16T19:00:00Z",
EndTime: "2026-03-16T20:00:00Z",
ShortForecast: "Cloudy",
},
}
run, effectiveAt, err := buildHourlyForecast(parsed)
if err != nil {
t.Fatalf("buildHourlyForecast() error = %v", err)
}
wantIssued := time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC)
wantUpdated := time.Date(2026, 3, 16, 18, 30, 0, 0, time.UTC)
if !run.IssuedAt.Equal(wantIssued) {
t.Fatalf("IssuedAt = %s, want %s", run.IssuedAt.Format(time.RFC3339), wantIssued.Format(time.RFC3339))
}
if run.UpdatedAt == nil || !run.UpdatedAt.Equal(wantUpdated) {
t.Fatalf("UpdatedAt = %v, want %s", run.UpdatedAt, wantUpdated.Format(time.RFC3339))
}
if run.Latitude == nil || math.Abs(*run.Latitude-38.5) > 0.0001 {
t.Fatalf("Latitude = %v, want 38.5", run.Latitude)
}
if run.Longitude == nil || math.Abs(*run.Longitude+89.5) > 0.0001 {
t.Fatalf("Longitude = %v, want -89.5", run.Longitude)
}
if run.ElevationMeters == nil || math.Abs(*run.ElevationMeters-elevation) > 0.0001 {
t.Fatalf("ElevationMeters = %v, want %.1f", run.ElevationMeters, elevation)
}
if !effectiveAt.Equal(wantIssued) {
t.Fatalf("effectiveAt = %s, want %s", effectiveAt.Format(time.RFC3339), wantIssued.Format(time.RFC3339))
}
}
func TestNormalizeForecastEventBySchemaRejectsUnsupportedSchema(t *testing.T) { func TestNormalizeForecastEventBySchemaRejectsUnsupportedSchema(t *testing.T) {
_, err := normalizeForecastEventBySchema(event.Event{ _, err := normalizeForecastEventBySchema(event.Event{
Schema: "raw.nws.daily.forecast.v1", Schema: "raw.nws.daily.forecast.v1",
@@ -85,6 +137,70 @@ func TestNormalizeForecastEventBySchemaRoutesNarrative(t *testing.T) {
} }
} }
func TestNormalizeForecastEventBySchemaProducesCanonicalWeatherForecastSchema(t *testing.T) {
tests := []struct {
name string
schema string
payload map[string]any
}{
{
name: "hourly",
schema: standards.SchemaRawNWSHourlyForecastV1,
payload: map[string]any{
"properties": map[string]any{
"generatedAt": "2026-03-16T18:00:00Z",
"periods": []map[string]any{
{
"startTime": "2026-03-16T19:00:00Z",
"endTime": "2026-03-16T20:00:00Z",
"shortForecast": "Cloudy",
},
},
},
},
},
{
name: "narrative",
schema: standards.SchemaRawNWSNarrativeForecastV1,
payload: map[string]any{
"properties": map[string]any{
"generatedAt": "2026-03-16T18:00:00Z",
"periods": []map[string]any{
{
"startTime": "2026-03-16T19:00:00Z",
"endTime": "2026-03-16T20:00:00Z",
"shortForecast": "Cloudy",
"detailedForecast": "Cloudy",
},
},
},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
out, err := normalizeForecastEventBySchema(event.Event{
ID: "evt-1",
Kind: event.Kind("forecast"),
Source: "nws-test",
EmittedAt: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
Schema: tt.schema,
Payload: tt.payload,
})
if err != nil {
t.Fatalf("normalizeForecastEventBySchema() error = %v", err)
}
if out == nil {
t.Fatalf("normalizeForecastEventBySchema() returned nil output")
}
if out.Schema != standards.SchemaWeatherForecastV1 {
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastV1)
}
})
}
}
func TestBuildNarrativeForecastMapsExpectedFields(t *testing.T) { func TestBuildNarrativeForecastMapsExpectedFields(t *testing.T) {
parsed := nwsNarrativeForecastResponse{} parsed := nwsNarrativeForecastResponse{}
parsed.Properties.GeneratedAt = "2026-03-27T15:17:01Z" parsed.Properties.GeneratedAt = "2026-03-27T15:17:01Z"
@@ -148,6 +264,9 @@ func TestBuildNarrativeForecastMapsExpectedFields(t *testing.T) {
if p.ProbabilityOfPrecipitationPercent == nil || *p.ProbabilityOfPrecipitationPercent != 20 { if p.ProbabilityOfPrecipitationPercent == nil || *p.ProbabilityOfPrecipitationPercent != 20 {
t.Fatalf("ProbabilityOfPrecipitationPercent = %v, want 20", p.ProbabilityOfPrecipitationPercent) t.Fatalf("ProbabilityOfPrecipitationPercent = %v, want 20", p.ProbabilityOfPrecipitationPercent)
} }
if p.ConditionCode != nil {
t.Fatalf("ConditionCode = %v, want nil for narrative period", p.ConditionCode)
}
wantIssued := time.Date(2026, 3, 27, 15, 17, 1, 0, time.UTC) wantIssued := time.Date(2026, 3, 27, 15, 17, 1, 0, time.UTC)
if !run.IssuedAt.Equal(wantIssued) { if !run.IssuedAt.Equal(wantIssued) {

View File

@@ -5,18 +5,15 @@ import (
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize" fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
) )
var builtins = []fknormalize.Normalizer{
ObservationNormalizer{},
ForecastNormalizer{},
ForecastDiscussionNormalizer{},
WeatherStoriesNormalizer{},
AlertsNormalizer{},
}
// Register appends NWS normalizers in stable order. // Register appends NWS normalizers in stable order.
func Register(in []fknormalize.Normalizer) []fknormalize.Normalizer { func Register(in []fknormalize.Normalizer) []fknormalize.Normalizer {
out := in return append(in, builtins...)
// Observations
out = append(out, ObservationNormalizer{})
// Forecasts
out = append(out, ForecastNormalizer{})
// Alerts
out = append(out, AlertsNormalizer{})
return out
} }

View File

@@ -262,6 +262,25 @@ type nwsAlertProperties struct {
References json.RawMessage `json:"references"` References json.RawMessage `json:"references"`
} }
// nwsWeatherStoriesResponse is a minimal representation of the NWS /weatherstories
// payload needed for mapping into model.WeatherStoryRun.
type nwsWeatherStoriesResponse struct {
Stories []nwsWeatherStory `json:"stories"`
}
type nwsWeatherStory struct {
OfficeID string `json:"officeId"`
StartTime string `json:"startTime"`
EndTime string `json:"endTime"`
UpdateTime string `json:"updateTime"`
Title string `json:"title"`
Description string `json:"description"`
AltText string `json:"altText"`
Priority bool `json:"priority"`
Order int `json:"order"`
Download string `json:"download"`
}
type nwsAlertReference struct { type nwsAlertReference struct {
ID string `json:"id"` ID string `json:"id"`
Identifier string `json:"identifier"` Identifier string `json:"identifier"`

View File

@@ -0,0 +1,107 @@
package nws
import (
"context"
"fmt"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
// WeatherStoriesNormalizer converts:
//
// standards.SchemaRawNWSWeatherStoriesV1 -> standards.SchemaWeatherStoryV1
//
// It maps the NWS /weatherstories JSON response into a canonical story snapshot.
type WeatherStoriesNormalizer struct{}
func (WeatherStoriesNormalizer) Match(e event.Event) bool {
return strings.TrimSpace(e.Schema) == standards.SchemaRawNWSWeatherStoriesV1
}
func (WeatherStoriesNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx
fallbackAsOf := in.EmittedAt.UTC()
if in.EffectiveAt != nil && !in.EffectiveAt.IsZero() {
fallbackAsOf = in.EffectiveAt.UTC()
}
return normcommon.NormalizeJSON(
in,
"nws weatherstories",
standards.SchemaWeatherStoryV1,
func(parsed nwsWeatherStoriesResponse) (model.WeatherStoryRun, time.Time, error) {
return buildWeatherStories(parsed, fallbackAsOf)
},
)
}
func buildWeatherStories(parsed nwsWeatherStoriesResponse, fallbackAsOf time.Time) (model.WeatherStoryRun, time.Time, error) {
stories := make([]model.WeatherStory, 0, len(parsed.Stories))
var officeID string
var asOf time.Time
for i, raw := range parsed.Stories {
startTime, err := parseRequiredNWSTime(raw.StartTime, fmt.Sprintf("stories[%d].startTime", i))
if err != nil {
return model.WeatherStoryRun{}, time.Time{}, err
}
endTime, err := parseRequiredNWSTime(raw.EndTime, fmt.Sprintf("stories[%d].endTime", i))
if err != nil {
return model.WeatherStoryRun{}, time.Time{}, err
}
updatedAt, err := parseRequiredNWSTime(raw.UpdateTime, fmt.Sprintf("stories[%d].updateTime", i))
if err != nil {
return model.WeatherStoryRun{}, time.Time{}, err
}
storyOfficeID := strings.TrimSpace(raw.OfficeID)
if officeID == "" && storyOfficeID != "" {
officeID = storyOfficeID
}
if asOf.IsZero() || updatedAt.After(asOf) {
asOf = updatedAt
}
stories = append(stories, model.WeatherStory{
OfficeID: storyOfficeID,
StartTime: startTime,
EndTime: endTime,
UpdatedAt: updatedAt,
Title: strings.TrimSpace(raw.Title),
Description: strings.TrimSpace(raw.Description),
AltText: strings.TrimSpace(raw.AltText),
Priority: raw.Priority,
Order: raw.Order,
DownloadURL: strings.TrimSpace(raw.Download),
})
}
if asOf.IsZero() {
asOf = fallbackAsOf.UTC()
}
run := model.WeatherStoryRun{
OfficeID: officeID,
AsOf: asOf,
Stories: stories,
}
return run, asOf, nil
}
func parseRequiredNWSTime(raw, field string) (time.Time, error) {
if strings.TrimSpace(raw) == "" {
return time.Time{}, fmt.Errorf("%s is required", field)
}
t, err := nwscommon.ParseTime(raw)
if err != nil {
return time.Time{}, fmt.Errorf("%s: %w", field, err)
}
return t.UTC(), nil
}

View File

@@ -0,0 +1,146 @@
package nws
import (
"encoding/json"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
func TestWeatherStoriesNormalizerProducesCanonicalSchemaAndMapsSample(t *testing.T) {
out, err := (WeatherStoriesNormalizer{}).Normalize(nil, weatherStoriesRawEvent(weatherStoriesSamplePayload()))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
if out == nil {
t.Fatalf("Normalize() returned nil output")
}
if out.Schema != standards.SchemaWeatherStoryV1 {
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherStoryV1)
}
if out.Kind != event.Kind("weather_story") {
t.Fatalf("Kind = %q, want weather_story", out.Kind)
}
payload, ok := out.Payload.(model.WeatherStoryRun)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherStoryRun", out.Payload)
}
if payload.OfficeID != "LSX" {
t.Fatalf("OfficeID = %q, want LSX", payload.OfficeID)
}
wantAsOf := time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC)
if !payload.AsOf.Equal(wantAsOf) {
t.Fatalf("AsOf = %s, want %s", payload.AsOf, wantAsOf)
}
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantAsOf) {
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantAsOf)
}
if len(payload.Stories) != 1 {
t.Fatalf("Stories len = %d, want 1", len(payload.Stories))
}
story := payload.Stories[0]
if story.Title != "Several Chances for Rain Through Monday" {
t.Fatalf("Title = %q", story.Title)
}
if story.Description != "A stagnant weather pattern." {
t.Fatalf("Description = %q", story.Description)
}
if story.AltText != "This slide shows the forecast." {
t.Fatalf("AltText = %q", story.AltText)
}
if story.Priority {
t.Fatalf("Priority = true, want false")
}
if story.Order != 1 {
t.Fatalf("Order = %d, want 1", story.Order)
}
if story.DownloadURL != "https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f" {
t.Fatalf("DownloadURL = %q", story.DownloadURL)
}
}
func TestWeatherStoriesNormalizerEmptyStoriesUsesFallbackAsOf(t *testing.T) {
effectiveAt := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC)
in := weatherStoriesRawEvent(`{"stories":[]}`)
in.EffectiveAt = &effectiveAt
out, err := (WeatherStoriesNormalizer{}).Normalize(nil, in)
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
payload, ok := out.Payload.(model.WeatherStoryRun)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherStoryRun", out.Payload)
}
if !payload.AsOf.Equal(effectiveAt) {
t.Fatalf("AsOf = %s, want fallback %s", payload.AsOf, effectiveAt)
}
if payload.Stories == nil {
t.Fatalf("Stories = nil, want empty slice")
}
if len(payload.Stories) != 0 {
t.Fatalf("Stories len = %d, want 0", len(payload.Stories))
}
}
func TestWeatherStoriesNormalizerRejectsInvalidRequiredStoryTime(t *testing.T) {
_, err := (WeatherStoriesNormalizer{}).Normalize(nil, weatherStoriesRawEvent(`{
"stories": [{
"officeId": "LSX",
"startTime": "bad",
"endTime": "2026-05-31T11:00:00+00:00",
"updateTime": "2026-05-30T09:00:34+00:00"
}]
}`))
if err == nil {
t.Fatalf("Normalize() error = nil, want error")
}
if !strings.Contains(err.Error(), "stories[0].startTime") {
t.Fatalf("error = %q, want field context", err)
}
}
func TestWeatherStoriesNormalizerMatch(t *testing.T) {
n := WeatherStoriesNormalizer{}
if !n.Match(event.Event{Schema: standards.SchemaRawNWSWeatherStoriesV1}) {
t.Fatalf("Match(raw weatherstories) = false, want true")
}
if n.Match(event.Event{Schema: standards.SchemaRawNWSAlertsV1}) {
t.Fatalf("Match(raw alerts) = true, want false")
}
}
func weatherStoriesRawEvent(payload string) event.Event {
return event.Event{
ID: "evt-weatherstories-1",
Kind: event.Kind("weather_story"),
Source: "nws-weatherstories-test",
EmittedAt: time.Date(2026, 5, 30, 9, 5, 0, 0, time.UTC),
Schema: standards.SchemaRawNWSWeatherStoriesV1,
Payload: json.RawMessage(payload),
}
}
func weatherStoriesSamplePayload() string {
return `{
"stories": [
{
"officeId": " LSX ",
"startTime": "2026-05-30T08:46:00+00:00",
"endTime": "2026-05-31T11:00:00+00:00",
"updateTime": "2026-05-30T09:00:34+00:00",
"title": " Several Chances for Rain Through Monday ",
"description": " A stagnant weather pattern. ",
"altText": " This slide shows the forecast. ",
"priority": false,
"order": 1,
"download": " https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f "
}
]
}`
}

View File

@@ -98,6 +98,7 @@ func buildForecast(parsed omForecastResponse, fallbackIssued time.Time) (model.W
} }
wmo := wmoAt(parsed.Hourly.WeatherCode, i) wmo := wmoAt(parsed.Hourly.WeatherCode, i)
wmoPtr := wmoCodePtr(wmo)
canonicalText := standards.WMOText(wmo, isDay) canonicalText := standards.WMOText(wmo, isDay)
period := model.WeatherForecastPeriod{ period := model.WeatherForecastPeriod{
@@ -107,7 +108,7 @@ func buildForecast(parsed omForecastResponse, fallbackIssued time.Time) (model.W
Name: "", Name: "",
IsDay: isDay, IsDay: isDay,
ConditionCode: wmo, ConditionCode: wmoPtr,
TextDescription: canonicalText, TextDescription: canonicalText,
} }
@@ -237,3 +238,8 @@ func wmoAt(vals []*int, idx int) model.WMOCode {
} }
return model.WMOUnknown return model.WMOUnknown
} }
func wmoCodePtr(code model.WMOCode) *model.WMOCode {
out := code
return &out
}

View File

@@ -35,6 +35,9 @@ func TestBuildForecastUsesCanonicalTextDescription(t *testing.T) {
if got := run.Periods[0].TextDescription; got != expectedText { if got := run.Periods[0].TextDescription; got != expectedText {
t.Fatalf("TextDescription = %q, want %q", got, expectedText) t.Fatalf("TextDescription = %q, want %q", got, expectedText)
} }
if run.Periods[0].ConditionCode == nil {
t.Fatalf("ConditionCode is nil, want mapped WMO code")
}
wantIssued := time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC) wantIssued := time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC)
if !run.IssuedAt.Equal(wantIssued) { if !run.IssuedAt.Equal(wantIssued) {

View File

@@ -5,14 +5,12 @@ import (
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize" fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
) )
var builtins = []fknormalize.Normalizer{
ObservationNormalizer{},
ForecastNormalizer{},
}
// Register appends Open-Meteo normalizers in stable order. // Register appends Open-Meteo normalizers in stable order.
func Register(in []fknormalize.Normalizer) []fknormalize.Normalizer { func Register(in []fknormalize.Normalizer) []fknormalize.Normalizer {
out := in return append(in, builtins...)
// Observations
out = append(out, ObservationNormalizer{})
// Forecasts
out = append(out, ForecastNormalizer{})
return out
} }

View File

@@ -5,12 +5,11 @@ import (
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize" fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
) )
var builtins = []fknormalize.Normalizer{
ObservationNormalizer{},
}
// Register appends OpenWeather normalizers in stable order. // Register appends OpenWeather normalizers in stable order.
func Register(in []fknormalize.Normalizer) []fknormalize.Normalizer { func Register(in []fknormalize.Normalizer) []fknormalize.Normalizer {
out := in return append(in, builtins...)
// Observations
out = append(out, ObservationNormalizer{})
return out
} }

View File

@@ -0,0 +1,552 @@
package nws
import (
"fmt"
"html"
"regexp"
"strconv"
"strings"
"time"
)
type ForecastDiscussion struct {
OfficeID string
OfficeName string
Product string
IssuedAt time.Time
UpdatedAt *time.Time
KeyMessages []string
ShortTerm *ForecastDiscussionSection
LongTerm *ForecastDiscussionSection
}
type ForecastDiscussionSection struct {
Qualifier string
IssuedAt *time.Time
Text string
}
var (
forecastDiscussionHeaderRE = regexp.MustCompile(`^\.(KEY MESSAGES|SHORT TERM|LONG TERM|AVIATION)\.\.\.(.*)$`)
forecastDiscussionAFDRE = regexp.MustCompile(`^AFD([A-Z]{3})$`)
forecastDiscussionWMORE = regexp.MustCompile(`\bK([A-Z]{3})\b`)
forecastDiscussionSigRE = regexp.MustCompile(`^[A-Z]{2,6}$`)
)
func ParseForecastDiscussionHTML(raw string) (ForecastDiscussion, error) {
text, err := ExtractForecastDiscussionText(raw)
if err != nil {
return ForecastDiscussion{}, err
}
parsed, err := ParseForecastDiscussionText(text)
if err != nil {
return ForecastDiscussion{}, err
}
parsed.UpdatedAt = parseForecastDiscussionUpdatedAt(raw)
return parsed, nil
}
func ExtractForecastDiscussionText(raw string) (string, error) {
lower := strings.ToLower(raw)
searchFrom := 0
for {
openStart := strings.Index(lower[searchFrom:], "<pre")
if openStart < 0 {
return "", fmt.Errorf("missing <pre class=\"glossaryProduct\"> block")
}
openStart += searchFrom
openEnd := strings.Index(lower[openStart:], ">")
if openEnd < 0 {
return "", fmt.Errorf("unterminated <pre> tag")
}
openEnd += openStart
tag := lower[openStart : openEnd+1]
if isGlossaryProductTag(tag) {
closeStart := strings.Index(lower[openEnd+1:], "</pre>")
if closeStart < 0 {
return "", fmt.Errorf("missing closing </pre> for glossaryProduct block")
}
closeStart += openEnd + 1
text := html.UnescapeString(raw[openEnd+1 : closeStart])
text = strings.ReplaceAll(text, "\r\n", "\n")
text = strings.ReplaceAll(text, "\r", "\n")
return text, nil
}
searchFrom = openEnd + 1
}
}
func ParseForecastDiscussionText(text string) (ForecastDiscussion, error) {
lines := splitLines(text)
officeID := parseForecastDiscussionOfficeID(lines)
officeName, issuedAt, err := parseForecastDiscussionHeader(lines)
if err != nil {
return ForecastDiscussion{}, err
}
out := ForecastDiscussion{
OfficeID: officeID,
OfficeName: officeName,
Product: "afd",
IssuedAt: issuedAt.UTC(),
}
if block, ok := extractForecastDiscussionSection(lines, "KEY MESSAGES"); ok {
out.KeyMessages = parseForecastDiscussionKeyMessages(block)
}
if block, ok := extractForecastDiscussionSection(lines, "SHORT TERM"); ok {
section, err := parseForecastDiscussionTextSection(block)
if err != nil {
return ForecastDiscussion{}, fmt.Errorf("parse SHORT TERM: %w", err)
}
out.ShortTerm = &section
}
if block, ok := extractForecastDiscussionSection(lines, "LONG TERM"); ok {
section, err := parseForecastDiscussionTextSection(block)
if err != nil {
return ForecastDiscussion{}, fmt.Errorf("parse LONG TERM: %w", err)
}
out.LongTerm = &section
}
return out, nil
}
func isGlossaryProductTag(tag string) bool {
tag = strings.ToLower(tag)
return strings.Contains(tag, `class="glossaryproduct"`) ||
strings.Contains(tag, `class='glossaryproduct'`) ||
strings.Contains(tag, `class="glossaryproduct `) ||
strings.Contains(tag, `class='glossaryproduct `)
}
func parseForecastDiscussionUpdatedAt(raw string) *time.Time {
lower := strings.ToLower(raw)
searchFrom := 0
for {
metaStart := strings.Index(lower[searchFrom:], "<meta")
if metaStart < 0 {
return nil
}
metaStart += searchFrom
metaEnd := strings.Index(lower[metaStart:], ">")
if metaEnd < 0 {
return nil
}
metaEnd += metaStart
tag := raw[metaStart : metaEnd+1]
if !strings.EqualFold(strings.TrimSpace(extractHTMLAttr(tag, "name")), "DC.date.created") {
searchFrom = metaEnd + 1
continue
}
content := strings.TrimSpace(extractHTMLAttr(tag, "content"))
if content == "" {
return nil
}
t, err := ParseTime(content)
if err != nil {
return nil
}
tt := t.UTC()
return &tt
}
}
func extractHTMLAttr(tag, attr string) string {
lower := strings.ToLower(tag)
attrLower := strings.ToLower(attr)
for i := 0; i < len(lower); i++ {
idx := strings.Index(lower[i:], attrLower)
if idx < 0 {
return ""
}
idx += i
if idx > 0 {
prev := lower[idx-1]
if isAttrNameChar(prev) {
i = idx + len(attrLower)
continue
}
}
j := idx + len(attrLower)
for j < len(lower) && isHTMLSpace(lower[j]) {
j++
}
if j >= len(lower) || lower[j] != '=' {
i = idx + len(attrLower)
continue
}
j++
for j < len(lower) && isHTMLSpace(lower[j]) {
j++
}
if j >= len(tag) {
return ""
}
quote := tag[j]
if quote != '"' && quote != '\'' {
return ""
}
j++
k := j
for k < len(tag) && tag[k] != quote {
k++
}
if k >= len(tag) {
return ""
}
return html.UnescapeString(tag[j:k])
}
return ""
}
func isHTMLSpace(b byte) bool {
switch b {
case ' ', '\n', '\r', '\t', '\f':
return true
default:
return false
}
}
func isAttrNameChar(b byte) bool {
switch {
case b >= 'a' && b <= 'z':
return true
case b >= 'A' && b <= 'Z':
return true
case b >= '0' && b <= '9':
return true
case b == '-' || b == '_' || b == ':':
return true
default:
return false
}
}
func splitLines(text string) []string {
text = strings.ReplaceAll(text, "\r\n", "\n")
text = strings.ReplaceAll(text, "\r", "\n")
return strings.Split(text, "\n")
}
func parseForecastDiscussionOfficeID(lines []string) string {
for _, raw := range lines {
line := strings.TrimSpace(raw)
if m := forecastDiscussionAFDRE.FindStringSubmatch(line); len(m) == 2 {
return m[1]
}
}
for _, raw := range lines {
line := strings.TrimSpace(raw)
if m := forecastDiscussionWMORE.FindStringSubmatch(line); len(m) == 2 {
return m[1]
}
}
return ""
}
func parseForecastDiscussionHeader(lines []string) (string, time.Time, error) {
for i, raw := range lines {
line := strings.TrimSpace(raw)
if !strings.HasPrefix(line, "National Weather Service ") {
continue
}
officeName := line
for j := i + 1; j < len(lines); j++ {
tsLine := strings.TrimSpace(lines[j])
if tsLine == "" {
continue
}
issuedAt, err := parseForecastDiscussionIssueTime(tsLine)
if err != nil {
return "", time.Time{}, fmt.Errorf("parse bulletin issuedAt %q: %w", tsLine, err)
}
return officeName, issuedAt.UTC(), nil
}
return "", time.Time{}, fmt.Errorf("missing bulletin issue time after office line")
}
return "", time.Time{}, fmt.Errorf("missing office header")
}
func parseForecastDiscussionIssueTime(line string) (time.Time, error) {
line = strings.TrimSpace(line)
line = strings.TrimPrefix(line, "Issued at ")
line = strings.TrimSpace(line)
parts := strings.Fields(line)
if len(parts) != 7 {
return time.Time{}, fmt.Errorf("unexpected issue time format")
}
loc, err := forecastDiscussionLocation(parts[2])
if err != nil {
return time.Time{}, err
}
datePart, err := time.Parse("Mon Jan 2 2006", strings.Join(parts[3:], " "))
if err != nil {
return time.Time{}, err
}
hour, minute, err := parseForecastDiscussionClock(parts[0], parts[1])
if err != nil {
return time.Time{}, err
}
return time.Date(
datePart.Year(),
datePart.Month(),
datePart.Day(),
hour,
minute,
0,
0,
loc,
), nil
}
func parseForecastDiscussionClock(rawClock, rawAMPM string) (int, int, error) {
clock := strings.TrimSpace(rawClock)
ampm := strings.ToUpper(strings.TrimSpace(rawAMPM))
if ampm != "AM" && ampm != "PM" {
return 0, 0, fmt.Errorf("unexpected meridiem %q", rawAMPM)
}
n, err := strconv.Atoi(clock)
if err != nil {
return 0, 0, fmt.Errorf("invalid clock %q", rawClock)
}
hour := n
minute := 0
if len(clock) >= 3 {
hour = n / 100
minute = n % 100
}
if hour < 1 || hour > 12 {
return 0, 0, fmt.Errorf("invalid hour %q", rawClock)
}
if minute < 0 || minute > 59 {
return 0, 0, fmt.Errorf("invalid minute %q", rawClock)
}
if ampm == "AM" {
if hour == 12 {
hour = 0
}
return hour, minute, nil
}
if hour != 12 {
hour += 12
}
return hour, minute, nil
}
func forecastDiscussionLocation(abbrev string) (*time.Location, error) {
offsets := map[string]int{
"AST": -4 * 3600,
"ADT": -3 * 3600,
"EST": -5 * 3600,
"EDT": -4 * 3600,
"CST": -6 * 3600,
"CDT": -5 * 3600,
"MST": -7 * 3600,
"MDT": -6 * 3600,
"PST": -8 * 3600,
"PDT": -7 * 3600,
"AKST": -9 * 3600,
"AKDT": -8 * 3600,
"HST": -10 * 3600,
"UTC": 0,
"GMT": 0,
}
abbr := strings.ToUpper(strings.TrimSpace(abbrev))
offset, ok := offsets[abbr]
if !ok {
return nil, fmt.Errorf("unsupported time zone %q", abbrev)
}
return time.FixedZone(abbr, offset), nil
}
func extractForecastDiscussionSection(lines []string, section string) ([]string, bool) {
target := "." + section + "..."
for i, raw := range lines {
line := strings.TrimSpace(raw)
if !strings.HasPrefix(line, target) {
continue
}
out := []string{line}
for j := i + 1; j < len(lines); j++ {
next := strings.TrimSpace(lines[j])
if next == "&&" || next == "$$" || strings.Contains(next, "WATCHES/WARNINGS/ADVISORIES") {
break
}
if j > i+1 && isForecastDiscussionSectionHeader(next) {
break
}
out = append(out, lines[j])
}
return out, true
}
return nil, false
}
func isForecastDiscussionSectionHeader(line string) bool {
return forecastDiscussionHeaderRE.MatchString(strings.TrimSpace(line))
}
func parseForecastDiscussionKeyMessages(block []string) []string {
if len(block) <= 1 {
return nil
}
body := trimBlankLines(block[1:])
var messages []string
var current strings.Builder
flush := func() {
msg := strings.TrimSpace(current.String())
if msg != "" {
messages = append(messages, msg)
}
current.Reset()
}
for _, raw := range body {
line := strings.TrimSpace(raw)
if line == "" {
continue
}
if strings.HasPrefix(line, "-") {
flush()
line = strings.TrimSpace(strings.TrimPrefix(line, "-"))
current.WriteString(line)
continue
}
if current.Len() > 0 {
current.WriteByte(' ')
}
current.WriteString(line)
}
flush()
return messages
}
func parseForecastDiscussionTextSection(block []string) (ForecastDiscussionSection, error) {
if len(block) == 0 {
return ForecastDiscussionSection{}, fmt.Errorf("empty section")
}
section := ForecastDiscussionSection{
Qualifier: parseForecastDiscussionQualifier(strings.TrimSpace(block[0])),
}
body := trimBlankLines(block[1:])
if len(body) == 0 {
return section, nil
}
first := strings.TrimSpace(body[0])
if strings.HasPrefix(first, "Issued at ") {
issuedAt, err := parseForecastDiscussionIssueTime(first)
if err != nil {
return ForecastDiscussionSection{}, fmt.Errorf("parse section issuedAt %q: %w", first, err)
}
tt := issuedAt.UTC()
section.IssuedAt = &tt
body = trimBlankLines(body[1:])
}
body = trimForecastDiscussionSignatureLines(body)
section.Text = joinForecastDiscussionParagraphs(body)
return section, nil
}
func parseForecastDiscussionQualifier(header string) string {
m := forecastDiscussionHeaderRE.FindStringSubmatch(header)
if len(m) != 3 {
return ""
}
return strings.TrimSpace(m[2])
}
func trimBlankLines(lines []string) []string {
start := 0
for start < len(lines) && strings.TrimSpace(lines[start]) == "" {
start++
}
end := len(lines)
for end > start && strings.TrimSpace(lines[end-1]) == "" {
end--
}
return lines[start:end]
}
func trimForecastDiscussionSignatureLines(lines []string) []string {
lines = trimBlankLines(lines)
for len(lines) > 0 {
last := strings.TrimSpace(lines[len(lines)-1])
if last == "" {
lines = lines[:len(lines)-1]
continue
}
if forecastDiscussionSigRE.MatchString(last) {
lines = trimBlankLines(lines[:len(lines)-1])
continue
}
break
}
return lines
}
func joinForecastDiscussionParagraphs(lines []string) string {
lines = trimBlankLines(lines)
if len(lines) == 0 {
return ""
}
var paragraphs []string
current := make([]string, 0, len(lines))
flush := func() {
if len(current) == 0 {
return
}
paragraphs = append(paragraphs, strings.Join(current, " "))
current = current[:0]
}
for _, raw := range lines {
line := strings.TrimSpace(raw)
if line == "" {
flush()
continue
}
current = append(current, line)
}
flush()
return strings.Join(paragraphs, "\n\n")
}

View File

@@ -0,0 +1,118 @@
package nws
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestParseForecastDiscussionHTMLParsesExpectedFields(t *testing.T) {
raw := loadForecastDiscussionSampleHTML(t)
got, err := ParseForecastDiscussionHTML(raw)
if err != nil {
t.Fatalf("ParseForecastDiscussionHTML() error = %v", err)
}
if got.OfficeID != "LSX" {
t.Fatalf("OfficeID = %q, want LSX", got.OfficeID)
}
if got.OfficeName != "National Weather Service Saint Louis MO" {
t.Fatalf("OfficeName = %q", got.OfficeName)
}
if got.Product != "afd" {
t.Fatalf("Product = %q, want afd", got.Product)
}
wantIssuedAt := time.Date(2026, 3, 28, 19, 24, 0, 0, time.UTC)
if !got.IssuedAt.Equal(wantIssuedAt) {
t.Fatalf("IssuedAt = %s, want %s", got.IssuedAt.Format(time.RFC3339), wantIssuedAt.Format(time.RFC3339))
}
wantUpdatedAt := time.Date(2026, 3, 28, 20, 29, 47, 0, time.UTC)
if got.UpdatedAt == nil || !got.UpdatedAt.Equal(wantUpdatedAt) {
t.Fatalf("UpdatedAt = %v, want %s", got.UpdatedAt, wantUpdatedAt.Format(time.RFC3339))
}
wantMessages := []string{
"Elevated fire danger conditions are expected across a broad area tomorrow afternoon due to breezy southwest winds and low humidity.",
"Very warm temperatures are expected once again Monday and Tuesday, with highs well into the 80s.",
"A cold front late Tuesday or early Wednesday brings our next chance of thunderstorms, followed by a cooldown and possibly more chances for rain later in the week.",
}
if len(got.KeyMessages) != len(wantMessages) {
t.Fatalf("KeyMessages len = %d, want %d", len(got.KeyMessages), len(wantMessages))
}
for i := range wantMessages {
if got.KeyMessages[i] != wantMessages[i] {
t.Fatalf("KeyMessages[%d] = %q, want %q", i, got.KeyMessages[i], wantMessages[i])
}
}
if got.ShortTerm == nil {
t.Fatalf("ShortTerm is nil")
}
if got.ShortTerm.Qualifier != "(Through Late Sunday Night)" {
t.Fatalf("ShortTerm.Qualifier = %q", got.ShortTerm.Qualifier)
}
if got.ShortTerm.IssuedAt == nil || !got.ShortTerm.IssuedAt.Equal(time.Date(2026, 3, 28, 19, 19, 0, 0, time.UTC)) {
t.Fatalf("ShortTerm.IssuedAt = %v", got.ShortTerm.IssuedAt)
}
if !strings.Contains(got.ShortTerm.Text, "After a chilly morning") {
t.Fatalf("ShortTerm.Text missing expected prose: %q", got.ShortTerm.Text)
}
if strings.Contains(got.ShortTerm.Text, "BRC") {
t.Fatalf("ShortTerm.Text should not include signature: %q", got.ShortTerm.Text)
}
if strings.Contains(got.ShortTerm.Text, "\n\n\n") {
t.Fatalf("ShortTerm.Text contains unexpected paragraph breaks: %q", got.ShortTerm.Text)
}
if got.LongTerm == nil {
t.Fatalf("LongTerm is nil")
}
if got.LongTerm.Qualifier != "(Monday through Next Saturday)" {
t.Fatalf("LongTerm.Qualifier = %q", got.LongTerm.Qualifier)
}
if got.LongTerm.IssuedAt == nil || !got.LongTerm.IssuedAt.Equal(time.Date(2026, 3, 28, 19, 19, 0, 0, time.UTC)) {
t.Fatalf("LongTerm.IssuedAt = %v", got.LongTerm.IssuedAt)
}
if !strings.Contains(got.LongTerm.Text, "The peak of the warmth arrives Monday and Tuesday") {
t.Fatalf("LongTerm.Text missing expected prose: %q", got.LongTerm.Text)
}
if strings.Contains(got.LongTerm.Text, "AVIATION") || strings.Contains(got.LongTerm.Text, "WATCHES/WARNINGS/ADVISORIES") {
t.Fatalf("LongTerm.Text includes content from other sections: %q", got.LongTerm.Text)
}
}
func TestParseForecastDiscussionHTMLMissingPreBlock(t *testing.T) {
_, err := ParseForecastDiscussionHTML("<html><body><div>no pre block</div></body></html>")
if err == nil {
t.Fatalf("ParseForecastDiscussionHTML() error = nil, want error")
}
if !strings.Contains(err.Error(), "glossaryProduct") {
t.Fatalf("error = %q, want glossaryProduct context", err)
}
}
func TestParseForecastDiscussionTextMissingIssueTime(t *testing.T) {
_, err := ParseForecastDiscussionText("National Weather Service Saint Louis MO\n\n.KEY MESSAGES...\n- Test")
if err == nil {
t.Fatalf("ParseForecastDiscussionText() error = nil, want error")
}
if !strings.Contains(err.Error(), "issue time") {
t.Fatalf("error = %q, want issue time context", err)
}
}
func loadForecastDiscussionSampleHTML(t *testing.T) string {
t.Helper()
path := filepath.Join("testdata", "forecast_discussion_sample.html")
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("os.ReadFile(%q) error = %v", path, err)
}
return string(b)
}

View File

@@ -0,0 +1,84 @@
<!DOCTYPE html>
<html class="no-js">
<head>
<meta name="DC.date.created" scheme="ISO8601" content="2026-03-28T20:29:47+00:00" />
<title>National Weather Service</title>
</head>
<body>
<pre class="glossaryProduct">
988
FXUS63 KLSX 281924
AFDLSX
Area Forecast Discussion
National Weather Service Saint Louis MO
224 PM CDT Sat Mar 28 2026
.KEY MESSAGES...
- Elevated fire danger conditions are expected across a broad area
tomorrow afternoon due to breezy southwest winds and low
humidity.
- Very warm temperatures are expected once again Monday and
Tuesday, with highs well into the 80s.
- A cold front late Tuesday or early Wednesday brings our next
chance of thunderstorms, followed by a cooldown and possibly
more chances for rain later in the week.
&&
.SHORT TERM... (Through Late Sunday Night)
Issued at 219 PM CDT Sat Mar 28 2026
After a chilly morning that saw widespread freezing temperatures,
another warmup is in store over the next several days as southerly
winds become re-established. We will also see the return of
shower/thunderstorm chances Tuesday onward as we enter a more
unsettled pattern.
In the near-term, the focus continues to be on some lingering fire
weather potential thanks to the presence of an exceptionally dry
airmass.
BRC
&&
.LONG TERM... (Monday through Next Saturday)
Issued at 219 PM CDT Sat Mar 28 2026
The peak of the warmth arrives Monday and Tuesday, as a broad, but
low-amplitude ridge nudges eastward and steady warm/moist advection
continues on both days.
Wednesday onward, the day-to-day details become much less clear, but
latest trends suggest that an active/wet pattern will likely
continue as another more substantial trough follows with additional
chances for showers/thunderstorms late in the week.
BRC
&&
.AVIATION... (For the 18z TAFs through 18z Sunday Afternoon)
Issued at 1133 AM CDT Sat Mar 28 2026
VFR conditions are expected throughout the 18Z TAF period.
BRC
&&
.LSX WATCHES/WARNINGS/ADVISORIES...
MO...None.
IL...None.
&&
$$
WFO LSX
</pre>
</body>
</html>

View File

@@ -8,11 +8,15 @@
// Canonical input schemas: // Canonical input schemas:
// - weather.observation.v1 -> model.WeatherObservation // - weather.observation.v1 -> model.WeatherObservation
// - weather.forecast.v1 -> model.WeatherForecastRun // - weather.forecast.v1 -> model.WeatherForecastRun
// - weather.forecast_discussion.v1 -> model.WeatherForecastDiscussion
// - weather.weather_story.v1 -> model.WeatherStoryRun
// - weather.alert.v1 -> model.WeatherAlertRun // - weather.alert.v1 -> model.WeatherAlertRun
// //
// Parent/child relationships: // Parent/child relationships:
// - observations.event_id -> observation_present_weather.event_id // - observations.event_id -> observation_present_weather.event_id
// - forecasts.event_id -> forecast_periods.run_event_id // - forecasts.event_id -> forecast_periods.run_event_id
// - forecast_discussions.event_id -> forecast_discussion_key_messages.run_event_id
// - weather_story_runs.event_id -> weather_stories.run_event_id
// - alert_runs.event_id -> alerts.run_event_id // - alert_runs.event_id -> alerts.run_event_id
// - alerts.(run_event_id, alert_index) -> alert_references.(run_event_id, alert_index) // - alerts.(run_event_id, alert_index) -> alert_references.(run_event_id, alert_index)
// //
@@ -24,13 +28,18 @@
// - observation_present_weather.observed_at // - observation_present_weather.observed_at
// - forecasts.issued_at // - forecasts.issued_at
// - forecast_periods.issued_at // - forecast_periods.issued_at
// - forecast_discussions.issued_at
// - forecast_discussion_key_messages.issued_at
// - weather_story_runs.as_of
// - weather_stories.as_of
// - alert_runs.as_of // - alert_runs.as_of
// - alerts.as_of // - alerts.as_of
// - alert_references.as_of // - alert_references.as_of
// //
// Envelope field mapping (shared parent columns) // Envelope field mapping (shared parent columns)
// //
// These columns exist on observations, forecasts, and alert_runs: // These columns exist on parent tables such as observations, forecasts,
// forecast_discussions, weather_story_runs, and alert_runs:
// - event_id TEXT -> event.id // - event_id TEXT -> event.id
// - event_kind TEXT -> event.kind // - event_kind TEXT -> event.kind
// - event_source TEXT -> event.source // - event_source TEXT -> event.source
@@ -101,7 +110,7 @@
// - end_time TIMESTAMPTZ -> payload.periods[i].endTime // - end_time TIMESTAMPTZ -> payload.periods[i].endTime
// - name TEXT NULL -> payload.periods[i].name // - name TEXT NULL -> payload.periods[i].name
// - is_day BOOLEAN NULL -> payload.periods[i].isDay // - is_day BOOLEAN NULL -> payload.periods[i].isDay
// - condition_code INTEGER -> payload.periods[i].conditionCode // - condition_code INTEGER NULL -> payload.periods[i].conditionCode
// - text_description TEXT NULL -> payload.periods[i].textDescription // - text_description TEXT NULL -> payload.periods[i].textDescription
// - temperature_c DOUBLE PRECISION NULL -> payload.periods[i].temperatureC // - temperature_c DOUBLE PRECISION NULL -> payload.periods[i].temperatureC
// - temperature_c_min DOUBLE PRECISION NULL -> payload.periods[i].temperatureCMin // - temperature_c_min DOUBLE PRECISION NULL -> payload.periods[i].temperatureCMin
@@ -120,7 +129,35 @@
// - snowfall_depth_mm DOUBLE PRECISION NULL -> payload.periods[i].snowfallDepthMm // - snowfall_depth_mm DOUBLE PRECISION NULL -> payload.periods[i].snowfallDepthMm
// - uv_index DOUBLE PRECISION NULL -> payload.periods[i].uvIndex // - uv_index DOUBLE PRECISION NULL -> payload.periods[i].uvIndex
// //
// 5. alert_runs (PK: event_id) // 5. weather_story_runs (PK: event_id)
//
// - event_id TEXT -> event.id
// - event_kind TEXT -> event.kind
// - event_source TEXT -> event.source
// - event_schema TEXT -> event.schema
// - event_emitted_at TIMESTAMPTZ -> event.emitted_at
// - event_effective_at TIMESTAMPTZ NULL -> event.effective_at
// - office_id TEXT NULL -> payload.officeId
// - as_of TIMESTAMPTZ -> payload.asOf
// - story_count INTEGER -> len(payload.stories)
//
// 6. weather_stories (PK: run_event_id, story_index)
//
// - run_event_id TEXT -> weather_story_runs.event_id / payload.stories[i]
// - story_index INTEGER -> i (array position in payload.stories)
// - as_of TIMESTAMPTZ -> payload.asOf (copied from parent)
// - office_id TEXT NULL -> payload.stories[i].officeId
// - start_time TIMESTAMPTZ -> payload.stories[i].startTime
// - end_time TIMESTAMPTZ -> payload.stories[i].endTime
// - updated_at TIMESTAMPTZ -> payload.stories[i].updatedAt
// - title TEXT NULL -> payload.stories[i].title
// - description TEXT NULL -> payload.stories[i].description
// - alt_text TEXT NULL -> payload.stories[i].altText
// - priority BOOLEAN -> payload.stories[i].priority
// - story_order INTEGER -> payload.stories[i].order
// - download_url TEXT NULL -> payload.stories[i].downloadUrl
//
// 7. alert_runs (PK: event_id)
// //
// - event_id TEXT -> event.id // - event_id TEXT -> event.id
// - event_kind TEXT -> event.kind // - event_kind TEXT -> event.kind
@@ -135,7 +172,7 @@
// - longitude DOUBLE PRECISION NULL -> payload.longitude // - longitude DOUBLE PRECISION NULL -> payload.longitude
// - alert_count INTEGER -> len(payload.alerts) // - alert_count INTEGER -> len(payload.alerts)
// //
// 6. alerts (PK: run_event_id, alert_index) // 8. alerts (PK: run_event_id, alert_index)
// //
// - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i] // - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i]
// - alert_index INTEGER -> i (array position in payload.alerts) // - alert_index INTEGER -> i (array position in payload.alerts)
@@ -160,7 +197,7 @@
// - sender_name TEXT NULL -> payload.alerts[i].senderName // - sender_name TEXT NULL -> payload.alerts[i].senderName
// - reference_count INTEGER -> len(payload.alerts[i].references) // - reference_count INTEGER -> len(payload.alerts[i].references)
// //
// 7. alert_references (PK: run_event_id, alert_index, reference_index) // 9. alert_references (PK: run_event_id, alert_index, reference_index)
// //
// - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i].references[j] // - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i].references[j]
// - alert_index INTEGER -> i (array position in payload.alerts) // - alert_index INTEGER -> i (array position in payload.alerts)
@@ -181,6 +218,10 @@
// read one row from forecasts, then join forecast_periods by run_event_id // read one row from forecasts, then join forecast_periods by run_event_id
// ordered by period_index to rebuild periods. // ordered by period_index to rebuild periods.
// //
// - WeatherStoryRun:
// read one row from weather_story_runs, then join weather_stories by
// run_event_id ordered by story_index to rebuild stories.
//
// - WeatherAlertRun: // - WeatherAlertRun:
// read one row from alert_runs, join alerts by run_event_id ordered by // read one row from alert_runs, join alerts by run_event_id ordered by
// alert_index, then join alert_references by (run_event_id, alert_index) // alert_index, then join alert_references by (run_event_id, alert_index)

View File

@@ -20,6 +20,10 @@ func mapPostgresEvent(_ context.Context, e fkevent.Event) ([]fksinks.PostgresWri
return mapObservationEvent(e) return mapObservationEvent(e)
case standards.SchemaWeatherForecastV1: case standards.SchemaWeatherForecastV1:
return mapForecastEvent(e) return mapForecastEvent(e)
case standards.SchemaWeatherForecastDiscussionV1:
return mapForecastDiscussionEvent(e)
case standards.SchemaWeatherStoryV1:
return mapWeatherStoryEvent(e)
case standards.SchemaWeatherAlertV1: case standards.SchemaWeatherAlertV1:
return mapAlertEvent(e) return mapAlertEvent(e)
default: default:
@@ -135,7 +139,7 @@ func mapForecastEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
"end_time": p.EndTime.UTC(), "end_time": p.EndTime.UTC(),
"name": nullableString(p.Name), "name": nullableString(p.Name),
"is_day": nullableBool(p.IsDay), "is_day": nullableBool(p.IsDay),
"condition_code": int(p.ConditionCode), "condition_code": nullableWMOCode(p.ConditionCode),
"text_description": nullableString(p.TextDescription), "text_description": nullableString(p.TextDescription),
"temperature_c": nullableFloat64(p.TemperatureC), "temperature_c": nullableFloat64(p.TemperatureC),
"temperature_c_min": nullableFloat64(p.TemperatureCMin), "temperature_c_min": nullableFloat64(p.TemperatureCMin),
@@ -160,6 +164,115 @@ func mapForecastEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
return writes, nil return writes, nil
} }
func mapForecastDiscussionEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
run, err := decodePayload[model.WeatherForecastDiscussion](e.Payload)
if err != nil {
return nil, fmt.Errorf("decode forecast discussion payload: %w", err)
}
if run.IssuedAt.IsZero() {
return nil, fmt.Errorf("decode forecast discussion payload: issuedAt is required")
}
if strings.TrimSpace(string(run.Product)) == "" {
return nil, fmt.Errorf("decode forecast discussion payload: product is required")
}
issuedAt := run.IssuedAt.UTC()
shortTermQualifier, shortTermIssuedAt, shortTermText := nullableDiscussionSection(run.ShortTerm)
longTermQualifier, longTermIssuedAt, longTermText := nullableDiscussionSection(run.LongTerm)
writes := make([]fksinks.PostgresWrite, 0, 1+len(run.KeyMessages))
writes = append(writes, fksinks.PostgresWrite{
Table: tableForecastDiscussions,
Values: map[string]any{
"event_id": e.ID,
"event_kind": string(e.Kind),
"event_source": e.Source,
"event_schema": e.Schema,
"event_emitted_at": e.EmittedAt.UTC(),
"event_effective_at": nullableTime(e.EffectiveAt),
"office_id": nullableString(run.OfficeID),
"office_name": nullableString(run.OfficeName),
"issued_at": issuedAt,
"updated_at": nullableTime(run.UpdatedAt),
"product": string(run.Product),
"short_term_qualifier": shortTermQualifier,
"short_term_issued_at": shortTermIssuedAt,
"short_term_text": shortTermText,
"long_term_qualifier": longTermQualifier,
"long_term_issued_at": longTermIssuedAt,
"long_term_text": longTermText,
"key_message_count": len(run.KeyMessages),
},
})
for i, msg := range run.KeyMessages {
writes = append(writes, fksinks.PostgresWrite{
Table: tableForecastDiscussionKeyMessages,
Values: map[string]any{
"run_event_id": e.ID,
"message_index": i,
"issued_at": issuedAt,
"message_text": nullableString(msg),
},
})
}
return writes, nil
}
func mapWeatherStoryEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
run, err := decodePayload[model.WeatherStoryRun](e.Payload)
if err != nil {
return nil, fmt.Errorf("decode weather story payload: %w", err)
}
if run.AsOf.IsZero() {
return nil, fmt.Errorf("decode weather story payload: asOf is required")
}
asOf := run.AsOf.UTC()
writes := make([]fksinks.PostgresWrite, 0, 1+len(run.Stories))
writes = append(writes, fksinks.PostgresWrite{
Table: tableWeatherStoryRuns,
Values: map[string]any{
"event_id": e.ID,
"event_kind": string(e.Kind),
"event_source": e.Source,
"event_schema": e.Schema,
"event_emitted_at": e.EmittedAt.UTC(),
"event_effective_at": nullableTime(e.EffectiveAt),
"office_id": nullableString(run.OfficeID),
"as_of": asOf,
"story_count": len(run.Stories),
},
})
for i, story := range run.Stories {
if story.StartTime.IsZero() || story.EndTime.IsZero() || story.UpdatedAt.IsZero() {
return nil, fmt.Errorf("decode weather story payload: stories[%d] startTime/endTime/updatedAt are required", i)
}
writes = append(writes, fksinks.PostgresWrite{
Table: tableWeatherStories,
Values: map[string]any{
"run_event_id": e.ID,
"story_index": i,
"as_of": asOf,
"office_id": nullableString(story.OfficeID),
"start_time": story.StartTime.UTC(),
"end_time": story.EndTime.UTC(),
"updated_at": story.UpdatedAt.UTC(),
"title": nullableString(story.Title),
"description": nullableString(story.Description),
"alt_text": nullableString(story.AltText),
"priority": story.Priority,
"story_order": story.Order,
"download_url": nullableString(story.DownloadURL),
},
})
}
return writes, nil
}
func mapAlertEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) { func mapAlertEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
run, err := decodePayload[model.WeatherAlertRun](e.Payload) run, err := decodePayload[model.WeatherAlertRun](e.Payload)
if err != nil { if err != nil {
@@ -269,6 +382,13 @@ func decodePayload[T any](payload any) (T, error) {
return out, nil return out, nil
} }
func nullableDiscussionSection(section *model.WeatherForecastDiscussionSection) (any, any, any) {
if section == nil {
return nil, nil, nil
}
return nullableString(section.Qualifier), nullableTime(section.IssuedAt), nullableString(section.Text)
}
func countAlertReferences(alerts []model.WeatherAlert) int { func countAlertReferences(alerts []model.WeatherAlert) int {
total := 0 total := 0
for _, a := range alerts { for _, a := range alerts {
@@ -305,6 +425,13 @@ func nullableTime(v *time.Time) any {
return v.UTC() return v.UTC()
} }
func nullableWMOCode(v *model.WMOCode) any {
if v == nil {
return nil
}
return int(*v)
}
func compactJSONText(v any) (any, error) { func compactJSONText(v any) (any, error) {
if v == nil { if v == nil {
return nil, nil return nil, nil

View File

@@ -63,13 +63,13 @@ func TestMapPostgresEventForecastStructPayload(t *testing.T) {
StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC), StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC),
EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC), EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
IsDay: &isDay, IsDay: &isDay,
ConditionCode: model.WMOCode(2), ConditionCode: wmoCodePtr(model.WMOCode(2)),
TemperatureC: &temp, TemperatureC: &temp,
}, },
{ {
StartTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC), StartTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
EndTime: time.Date(2026, 3, 16, 21, 0, 0, 0, time.UTC), EndTime: time.Date(2026, 3, 16, 21, 0, 0, 0, time.UTC),
ConditionCode: model.WMOCode(3), ConditionCode: nil,
}, },
}, },
} }
@@ -94,6 +94,9 @@ func TestMapPostgresEventForecastStructPayload(t *testing.T) {
if got := writes[1].Values["period_index"]; got != 0 { if got := writes[1].Values["period_index"]; got != 0 {
t.Fatalf("first period index = %#v, want 0", got) t.Fatalf("first period index = %#v, want 0", got)
} }
if got := writes[2].Values["condition_code"]; got != nil {
t.Fatalf("second period condition_code = %#v, want nil", got)
}
assertAllWritesIncludeAllColumns(t, writes) assertAllWritesIncludeAllColumns(t, writes)
} }
@@ -146,6 +149,120 @@ func TestMapPostgresEventAlertStructPayload(t *testing.T) {
assertAllWritesIncludeAllColumns(t, writes) assertAllWritesIncludeAllColumns(t, writes)
} }
func TestMapPostgresEventForecastDiscussionStructPayload(t *testing.T) {
updatedAt := time.Date(2026, 3, 28, 20, 29, 47, 0, time.UTC)
shortIssuedAt := time.Date(2026, 3, 28, 19, 19, 0, 0, time.UTC)
run := model.WeatherForecastDiscussion{
OfficeID: "LSX",
OfficeName: "National Weather Service Saint Louis MO",
Product: model.ForecastDiscussionProductAFD,
IssuedAt: time.Date(2026, 3, 28, 19, 24, 0, 0, time.UTC),
UpdatedAt: &updatedAt,
KeyMessages: []string{"msg one", "msg two"},
ShortTerm: &model.WeatherForecastDiscussionSection{Qualifier: "(Tonight)", IssuedAt: &shortIssuedAt, Text: "Short term text"},
LongTerm: &model.WeatherForecastDiscussionSection{Text: "Long term text"},
}
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastDiscussionV1, "forecast_discussion", run))
if err != nil {
t.Fatalf("mapPostgresEvent() error = %v", err)
}
if len(writes) != 3 {
t.Fatalf("mapPostgresEvent() writes len = %d, want 3", len(writes))
}
if writes[0].Table != tableForecastDiscussions {
t.Fatalf("writes[0].Table = %q, want %q", writes[0].Table, tableForecastDiscussions)
}
if got := writes[0].Values["key_message_count"]; got != 2 {
t.Fatalf("forecast_discussions key_message_count = %#v, want 2", got)
}
if got := writes[0].Values["short_term_qualifier"]; got != "(Tonight)" {
t.Fatalf("forecast_discussions short_term_qualifier = %#v, want (Tonight)", got)
}
if got := writes[0].Values["long_term_issued_at"]; got != nil {
t.Fatalf("forecast_discussions long_term_issued_at = %#v, want nil", got)
}
if writes[1].Table != tableForecastDiscussionKeyMessages || writes[2].Table != tableForecastDiscussionKeyMessages {
t.Fatalf("forecast discussion key message writes not in expected order")
}
if got := writes[2].Values["message_index"]; got != 1 {
t.Fatalf("second key message index = %#v, want 1", got)
}
assertAllWritesIncludeAllColumns(t, writes)
}
func TestMapPostgresEventWeatherStoryStructPayload(t *testing.T) {
run := model.WeatherStoryRun{
OfficeID: "LSX",
AsOf: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
Stories: []model.WeatherStory{
{
OfficeID: "LSX",
StartTime: time.Date(2026, 5, 30, 8, 46, 0, 0, time.UTC),
EndTime: time.Date(2026, 5, 31, 11, 0, 0, 0, time.UTC),
UpdatedAt: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
Title: "Several Chances for Rain Through Monday",
Description: "Scattered showers and thunderstorms.",
AltText: "This slide shows the forecast.",
Priority: true,
Order: 1,
DownloadURL: "https://api.weather.gov/offices/LSX/weatherstories/download/story-1",
},
},
}
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, "weather_story", run))
if err != nil {
t.Fatalf("mapPostgresEvent() error = %v", err)
}
if len(writes) != 2 {
t.Fatalf("mapPostgresEvent() writes len = %d, want 2", len(writes))
}
if writes[0].Table != tableWeatherStoryRuns {
t.Fatalf("writes[0].Table = %q, want %q", writes[0].Table, tableWeatherStoryRuns)
}
if got := writes[0].Values["story_count"]; got != 1 {
t.Fatalf("weather_story_runs story_count = %#v, want 1", got)
}
if writes[1].Table != tableWeatherStories {
t.Fatalf("writes[1].Table = %q, want %q", writes[1].Table, tableWeatherStories)
}
if got := writes[1].Values["download_url"]; got != "https://api.weather.gov/offices/LSX/weatherstories/download/story-1" {
t.Fatalf("weather_stories download_url = %#v", got)
}
if got := writes[1].Values["story_order"]; got != 1 {
t.Fatalf("weather_stories story_order = %#v, want 1", got)
}
assertAllWritesIncludeAllColumns(t, writes)
}
func TestMapPostgresEventWeatherStoryRejectsMissingAsOf(t *testing.T) {
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, "weather_story", model.WeatherStoryRun{}))
if err == nil {
t.Fatalf("mapPostgresEvent() error = nil, want missing asOf error")
}
if !strings.Contains(err.Error(), "asOf is required") {
t.Fatalf("error = %q, want asOf context", err)
}
}
func TestMapPostgresEventWeatherStoryRejectsMissingStoryTimes(t *testing.T) {
run := model.WeatherStoryRun{
AsOf: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
Stories: []model.WeatherStory{{Title: "missing times"}},
}
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, "weather_story", run))
if err == nil {
t.Fatalf("mapPostgresEvent() error = nil, want missing story times error")
}
if !strings.Contains(err.Error(), "stories[0] startTime/endTime/updatedAt are required") {
t.Fatalf("error = %q, want story time context", err)
}
}
func TestMapPostgresEventMapPayload(t *testing.T) { func TestMapPostgresEventMapPayload(t *testing.T) {
run := model.WeatherForecastRun{ run := model.WeatherForecastRun{
IssuedAt: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC), IssuedAt: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
@@ -154,7 +271,7 @@ func TestMapPostgresEventMapPayload(t *testing.T) {
{ {
StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC), StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC),
EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC), EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
ConditionCode: model.WMOCode(2), ConditionCode: wmoCodePtr(model.WMOCode(2)),
}, },
}, },
} }
@@ -201,6 +318,16 @@ func TestMapPostgresEventMalformedPayload(t *testing.T) {
} }
} }
func TestMapPostgresEventForecastDiscussionMalformedPayload(t *testing.T) {
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastDiscussionV1, "forecast_discussion", "bad"))
if err == nil {
t.Fatalf("mapPostgresEvent() expected error for malformed payload")
}
if !strings.Contains(err.Error(), "decode forecast discussion payload") {
t.Fatalf("error = %q, want decode forecast discussion payload context", err)
}
}
func testEvent(schema string, kind fkevent.Kind, payload any) fkevent.Event { func testEvent(schema string, kind fkevent.Kind, payload any) fkevent.Event {
effectiveAt := time.Date(2026, 3, 16, 18, 30, 0, 0, time.UTC) effectiveAt := time.Date(2026, 3, 16, 18, 30, 0, 0, time.UTC)
return fkevent.Event{ return fkevent.Event{
@@ -245,3 +372,8 @@ func tableColumnCounts() map[string]int {
} }
return m return m
} }
func wmoCodePtr(v model.WMOCode) *model.WMOCode {
out := v
return &out
}

View File

@@ -9,6 +9,10 @@ const (
tableObservationPresentWeather = "observation_present_weather" tableObservationPresentWeather = "observation_present_weather"
tableForecasts = "forecasts" tableForecasts = "forecasts"
tableForecastPeriods = "forecast_periods" tableForecastPeriods = "forecast_periods"
tableForecastDiscussions = "forecast_discussions"
tableForecastDiscussionKeyMessages = "forecast_discussion_key_messages"
tableWeatherStoryRuns = "weather_story_runs"
tableWeatherStories = "weather_stories"
tableAlertRuns = "alert_runs" tableAlertRuns = "alert_runs"
tableAlerts = "alerts" tableAlerts = "alerts"
tableAlertReferences = "alert_references" tableAlertReferences = "alert_references"
@@ -102,7 +106,7 @@ func PostgresSchema() fksinks.PostgresSchema {
{Name: "end_time", Type: "TIMESTAMPTZ", Nullable: false}, {Name: "end_time", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "name", Type: "TEXT", Nullable: true}, {Name: "name", Type: "TEXT", Nullable: true},
{Name: "is_day", Type: "BOOLEAN", Nullable: true}, {Name: "is_day", Type: "BOOLEAN", Nullable: true},
{Name: "condition_code", Type: "INTEGER", Nullable: false}, {Name: "condition_code", Type: "INTEGER", Nullable: true},
{Name: "text_description", Type: "TEXT", Nullable: true}, {Name: "text_description", Type: "TEXT", Nullable: true},
{Name: "temperature_c", Type: "DOUBLE PRECISION", Nullable: true}, {Name: "temperature_c", Type: "DOUBLE PRECISION", Nullable: true},
{Name: "temperature_c_min", Type: "DOUBLE PRECISION", Nullable: true}, {Name: "temperature_c_min", Type: "DOUBLE PRECISION", Nullable: true},
@@ -129,6 +133,94 @@ func PostgresSchema() fksinks.PostgresSchema {
{Name: "idx_wf_fc_period_run_start", Columns: []string{"run_event_id", "start_time"}}, {Name: "idx_wf_fc_period_run_start", Columns: []string{"run_event_id", "start_time"}},
}, },
}, },
{
Name: tableForecastDiscussions,
Columns: []fksinks.PostgresColumn{
{Name: "event_id", Type: "TEXT", Nullable: false},
{Name: "event_kind", Type: "TEXT", Nullable: false},
{Name: "event_source", Type: "TEXT", Nullable: false},
{Name: "event_schema", Type: "TEXT", Nullable: false},
{Name: "event_emitted_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "event_effective_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "office_id", Type: "TEXT", Nullable: true},
{Name: "office_name", Type: "TEXT", Nullable: true},
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "updated_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "product", Type: "TEXT", Nullable: false},
{Name: "short_term_qualifier", Type: "TEXT", Nullable: true},
{Name: "short_term_issued_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "short_term_text", Type: "TEXT", Nullable: true},
{Name: "long_term_qualifier", Type: "TEXT", Nullable: true},
{Name: "long_term_issued_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "long_term_text", Type: "TEXT", Nullable: true},
{Name: "key_message_count", Type: "INTEGER", Nullable: false},
},
PrimaryKey: []string{"event_id"},
PruneColumn: "issued_at",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_discussion_office_product_issued_at", Columns: []string{"office_id", "product", "issued_at"}},
{Name: "idx_wf_discussion_issued_at", Columns: []string{"issued_at"}},
},
},
{
Name: tableForecastDiscussionKeyMessages,
Columns: []fksinks.PostgresColumn{
{Name: "run_event_id", Type: "TEXT REFERENCES forecast_discussions(event_id) ON DELETE CASCADE", Nullable: false},
{Name: "message_index", Type: "INTEGER", Nullable: false},
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "message_text", Type: "TEXT", Nullable: true},
},
PrimaryKey: []string{"run_event_id", "message_index"},
PruneColumn: "issued_at",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_discussion_message_issued_at", Columns: []string{"issued_at"}},
},
},
{
Name: tableWeatherStoryRuns,
Columns: []fksinks.PostgresColumn{
{Name: "event_id", Type: "TEXT", Nullable: false},
{Name: "event_kind", Type: "TEXT", Nullable: false},
{Name: "event_source", Type: "TEXT", Nullable: false},
{Name: "event_schema", Type: "TEXT", Nullable: false},
{Name: "event_emitted_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "event_effective_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "office_id", Type: "TEXT", Nullable: true},
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "story_count", Type: "INTEGER", Nullable: false},
},
PrimaryKey: []string{"event_id"},
PruneColumn: "as_of",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_story_run_office_as_of", Columns: []string{"office_id", "as_of"}},
{Name: "idx_wf_story_run_as_of", Columns: []string{"as_of"}},
},
},
{
Name: tableWeatherStories,
Columns: []fksinks.PostgresColumn{
{Name: "run_event_id", Type: "TEXT REFERENCES weather_story_runs(event_id) ON DELETE CASCADE", Nullable: false},
{Name: "story_index", Type: "INTEGER", Nullable: false},
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "office_id", Type: "TEXT", Nullable: true},
{Name: "start_time", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "end_time", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "updated_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "title", Type: "TEXT", Nullable: true},
{Name: "description", Type: "TEXT", Nullable: true},
{Name: "alt_text", Type: "TEXT", Nullable: true},
{Name: "priority", Type: "BOOLEAN", Nullable: false},
{Name: "story_order", Type: "INTEGER", Nullable: false},
{Name: "download_url", Type: "TEXT", Nullable: true},
},
PrimaryKey: []string{"run_event_id", "story_index"},
PruneColumn: "as_of",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_stories_start_time", Columns: []string{"start_time"}},
{Name: "idx_wf_stories_end_time", Columns: []string{"end_time"}},
{Name: "idx_wf_stories_updated_at", Columns: []string{"updated_at"}},
},
},
{ {
Name: tableAlertRuns, Name: tableAlertRuns,
Columns: []fksinks.PostgresColumn{ Columns: []fksinks.PostgresColumn{

View File

@@ -1,58 +1,6 @@
package postgres package postgres
import ( import "testing"
"fmt"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
fksinks "gitea.maximumdirect.net/ejr/feedkit/sinks"
)
func TestRegisterPostgresSchemasNilConfig(t *testing.T) {
err := fksinks.RegisterPostgresSchemaForConfiguredSinks(nil, PostgresSchema())
if err == nil {
t.Fatalf("RegisterPostgresSchemas(nil) expected error")
}
if !strings.Contains(err.Error(), "config is nil") {
t.Fatalf("error = %q, want config is nil", err)
}
}
func TestRegisterPostgresSchemasNonPostgresNoOp(t *testing.T) {
cfg := &config.Config{
Sinks: []config.SinkConfig{
{Name: "stdout_only", Driver: "stdout"},
{Name: "nats_only", Driver: "nats"},
},
}
if err := fksinks.RegisterPostgresSchemaForConfiguredSinks(cfg, PostgresSchema()); err != nil {
t.Fatalf("RegisterPostgresSchemas(non-postgres) error = %v", err)
}
}
func TestRegisterPostgresSchemasDuplicateRegistrationFails(t *testing.T) {
sinkName := uniqueSinkName("pg_test")
cfg := &config.Config{
Sinks: []config.SinkConfig{
{Name: sinkName, Driver: "postgres"},
},
}
if err := fksinks.RegisterPostgresSchemaForConfiguredSinks(cfg, PostgresSchema()); err != nil {
t.Fatalf("first RegisterPostgresSchemas() error = %v", err)
}
err := fksinks.RegisterPostgresSchemaForConfiguredSinks(cfg, PostgresSchema())
if err == nil {
t.Fatalf("second RegisterPostgresSchemas() expected duplicate error")
}
if !strings.Contains(err.Error(), "already registered") {
t.Fatalf("error = %q, want already registered", err)
}
}
func TestWeatherPostgresSchemaShape(t *testing.T) { func TestWeatherPostgresSchemaShape(t *testing.T) {
s := PostgresSchema() s := PostgresSchema()
@@ -65,6 +13,10 @@ func TestWeatherPostgresSchemaShape(t *testing.T) {
tableObservationPresentWeather: true, tableObservationPresentWeather: true,
tableForecasts: true, tableForecasts: true,
tableForecastPeriods: true, tableForecastPeriods: true,
tableForecastDiscussions: true,
tableForecastDiscussionKeyMessages: true,
tableWeatherStoryRuns: true,
tableWeatherStories: true,
tableAlertRuns: true, tableAlertRuns: true,
tableAlerts: true, tableAlerts: true,
tableAlertReferences: true, tableAlertReferences: true,
@@ -91,6 +43,37 @@ func TestWeatherPostgresSchemaShape(t *testing.T) {
} }
} }
func uniqueSinkName(prefix string) string { func TestWeatherPostgresSchemaIncludesWeatherStoryColumns(t *testing.T) {
return fmt.Sprintf("%s_%d", prefix, time.Now().UnixNano()) runColumns := columnsForTable(t, tableWeatherStoryRuns)
if !runColumns["as_of"] {
t.Fatalf("%s missing as_of column", tableWeatherStoryRuns)
}
if !runColumns["story_count"] {
t.Fatalf("%s missing story_count column", tableWeatherStoryRuns)
}
storyColumns := columnsForTable(t, tableWeatherStories)
for _, col := range []string{"start_time", "end_time", "updated_at", "title", "description", "alt_text", "priority", "story_order", "download_url"} {
if !storyColumns[col] {
t.Fatalf("%s missing %s column", tableWeatherStories, col)
}
}
}
func columnsForTable(t *testing.T, table string) map[string]bool {
t.Helper()
schema := PostgresSchema()
for _, tbl := range schema.Tables {
if tbl.Name != table {
continue
}
cols := make(map[string]bool, len(tbl.Columns))
for _, col := range tbl.Columns {
cols[col.Name] = true
}
return cols
}
t.Fatalf("missing table %q", table)
return nil
} }

View File

@@ -9,33 +9,34 @@ import (
fksource "gitea.maximumdirect.net/ejr/feedkit/sources" fksource "gitea.maximumdirect.net/ejr/feedkit/sources"
) )
type pollDriverRegistration struct {
driver string
factory func(config.SourceConfig) (fksource.PollSource, error)
}
var pollDriverRegistrations = []pollDriverRegistration{
{driver: "nws_observation", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewObservationSource(cfg) }},
{driver: "nws_alerts", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewAlertsSource(cfg) }},
{driver: "nws_forecast_hourly", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewHourlyForecastSource(cfg) }},
{driver: "nws_forecast_narrative", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewNarrativeForecastSource(cfg) }},
{driver: "nws_forecast_discussion", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {
return nws.NewForecastDiscussionSource(cfg)
}},
{driver: "nws_weatherstories", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewWeatherStoriesSource(cfg) }},
{driver: "openmeteo_observation", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return openmeteo.NewObservationSource(cfg) }},
{driver: "openmeteo_forecast", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return openmeteo.NewForecastSource(cfg) }},
{driver: "openweather_observation", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {
return openweather.NewObservationSource(cfg)
}},
}
// RegisterBuiltins registers the source drivers that ship with this binary. // RegisterBuiltins registers the source drivers that ship with this binary.
// Keeping this in one place makes main.go very readable. // Keeping this in one place makes main.go very readable.
func RegisterBuiltins(r *fksource.Registry) { func RegisterBuiltins(r *fksource.Registry) {
// NWS drivers for _, reg := range pollDriverRegistrations {
r.RegisterPoll("nws_observation", func(cfg config.SourceConfig) (fksource.PollSource, error) { reg := reg
return nws.NewObservationSource(cfg) r.RegisterPoll(reg.driver, func(cfg config.SourceConfig) (fksource.PollSource, error) {
}) return reg.factory(cfg)
r.RegisterPoll("nws_alerts", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return nws.NewAlertsSource(cfg)
})
r.RegisterPoll("nws_forecast_hourly", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return nws.NewHourlyForecastSource(cfg)
})
r.RegisterPoll("nws_forecast_narrative", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return nws.NewNarrativeForecastSource(cfg)
})
// Open-Meteo drivers
r.RegisterPoll("openmeteo_observation", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return openmeteo.NewObservationSource(cfg)
})
r.RegisterPoll("openmeteo_forecast", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return openmeteo.NewForecastSource(cfg)
})
// OpenWeatherMap drivers
r.RegisterPoll("openweather_observation", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return openweather.NewObservationSource(cfg)
}) })
}
} }

View File

@@ -34,6 +34,32 @@ func TestRegisterBuiltinsRegistersNWSNarrativeForecastDriver(t *testing.T) {
} }
} }
func TestRegisterBuiltinsRegistersNWSForecastDiscussionDriver(t *testing.T) {
reg := fksource.NewRegistry()
RegisterBuiltins(reg)
in, err := reg.BuildInput(sourceConfigForDriver("nws_forecast_discussion"))
if err != nil {
t.Fatalf("BuildInput(nws_forecast_discussion) error = %v", err)
}
if _, ok := in.(fksource.PollSource); !ok {
t.Fatalf("BuildInput(nws_forecast_discussion) type = %T, want PollSource", in)
}
}
func TestRegisterBuiltinsRegistersNWSWeatherStoriesDriver(t *testing.T) {
reg := fksource.NewRegistry()
RegisterBuiltins(reg)
in, err := reg.BuildInput(sourceConfigForDriver("nws_weatherstories"))
if err != nil {
t.Fatalf("BuildInput(nws_weatherstories) error = %v", err)
}
if _, ok := in.(fksource.PollSource); !ok {
t.Fatalf("BuildInput(nws_weatherstories) type = %T, want PollSource", in)
}
}
func TestRegisterBuiltinsDoesNotRegisterLegacyNWSForecastDriver(t *testing.T) { func TestRegisterBuiltinsDoesNotRegisterLegacyNWSForecastDriver(t *testing.T) {
reg := fksource.NewRegistry() reg := fksource.NewRegistry()
RegisterBuiltins(reg) RegisterBuiltins(reg)
@@ -47,13 +73,44 @@ func TestRegisterBuiltinsDoesNotRegisterLegacyNWSForecastDriver(t *testing.T) {
} }
} }
func TestRegisterBuiltinsRegistersAllCurrentDrivers(t *testing.T) {
reg := fksource.NewRegistry()
RegisterBuiltins(reg)
drivers := []string{
"nws_observation",
"nws_alerts",
"nws_forecast_hourly",
"nws_forecast_narrative",
"nws_forecast_discussion",
"nws_weatherstories",
"openmeteo_observation",
"openmeteo_forecast",
"openweather_observation",
}
for _, driver := range drivers {
in, err := reg.BuildInput(sourceConfigForDriver(driver))
if err != nil {
t.Fatalf("BuildInput(%s) error = %v", driver, err)
}
if _, ok := in.(fksource.PollSource); !ok {
t.Fatalf("BuildInput(%s) type = %T, want PollSource", driver, in)
}
}
}
func sourceConfigForDriver(driver string) config.SourceConfig { func sourceConfigForDriver(driver string) config.SourceConfig {
url := "https://example.invalid"
if driver == "openweather_observation" {
url = "https://example.invalid?units=metric"
}
return config.SourceConfig{ return config.SourceConfig{
Name: "test-source", Name: "test-source",
Driver: driver, Driver: driver,
Mode: config.SourceModePoll, Mode: config.SourceModePoll,
Params: map[string]any{ Params: map[string]any{
"url": "https://example.invalid", "url": url,
"user_agent": "test-agent", "user_agent": "test-agent",
}, },
} }

View File

@@ -39,8 +39,8 @@ func NewAlertsSource(cfg config.SourceConfig) (*AlertsSource, error) {
func (s *AlertsSource) Name() string { return s.http.Name } func (s *AlertsSource) Name() string { return s.http.Name }
// Kind is used for routing/policy. // Kinds is used for routing/policy.
func (s *AlertsSource) Kind() event.Kind { return event.Kind("alert") } func (s *AlertsSource) Kinds() []event.Kind { return []event.Kind{event.Kind("alert")} }
func (s *AlertsSource) Poll(ctx context.Context) ([]event.Event, error) { func (s *AlertsSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx) raw, meta, changed, err := s.fetchRaw(ctx)
@@ -71,7 +71,7 @@ func (s *AlertsSource) Poll(ctx context.Context) ([]event.Event, error) {
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt) eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent( return fksources.SingleEvent(
s.Kind(), event.Kind("alert"),
s.http.Name, s.http.Name,
standards.SchemaRawNWSAlertsV1, standards.SchemaRawNWSAlertsV1,
eventID, eventID,

View File

@@ -0,0 +1,114 @@
package nws
import (
"context"
"encoding/json"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
)
const nwsForecastAccept = "application/geo+json, application/json"
type forecastSource struct {
http *fksources.HTTPSource
rawSchema string
}
type forecastMeta struct {
Properties struct {
GeneratedAt string `json:"generatedAt"`
UpdateTime string `json:"updateTime"`
Updated string `json:"updated"`
} `json:"properties"`
ParsedGeneratedAt time.Time `json:"-"`
ParsedUpdateTime time.Time `json:"-"`
}
func newForecastSource(cfg config.SourceConfig, driver, rawSchema string) (*forecastSource, error) {
hs, err := fksources.NewHTTPSource(driver, cfg, nwsForecastAccept)
if err != nil {
return nil, err
}
return &forecastSource{
http: hs,
rawSchema: rawSchema,
}, nil
}
func (s *forecastSource) Name() string { return s.http.Name }
func (s *forecastSource) Kinds() []event.Kind { return []event.Kind{event.Kind("forecast")} }
func (s *forecastSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx)
if err != nil {
return nil, err
}
if !changed {
return nil, nil
}
var effectiveAt *time.Time
switch {
case !meta.ParsedGeneratedAt.IsZero():
t := meta.ParsedGeneratedAt.UTC()
effectiveAt = &t
case !meta.ParsedUpdateTime.IsZero():
t := meta.ParsedUpdateTime.UTC()
effectiveAt = &t
}
emittedAt := time.Now().UTC()
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent(
event.Kind("forecast"),
s.http.Name,
s.rawSchema,
eventID,
emittedAt,
effectiveAt,
raw,
)
}
func (s *forecastSource) fetchRaw(ctx context.Context) (json.RawMessage, forecastMeta, bool, error) {
raw, changed, err := s.http.FetchJSONIfChanged(ctx)
if err != nil {
return nil, forecastMeta{}, false, err
}
if !changed {
return nil, forecastMeta{}, false, nil
}
var meta forecastMeta
if err := json.Unmarshal(raw, &meta); err != nil {
return raw, forecastMeta{}, true, nil
}
genStr := strings.TrimSpace(meta.Properties.GeneratedAt)
if genStr != "" {
if t, err := nwscommon.ParseTime(genStr); err == nil {
meta.ParsedGeneratedAt = t.UTC()
}
}
updStr := strings.TrimSpace(meta.Properties.UpdateTime)
if updStr == "" {
updStr = strings.TrimSpace(meta.Properties.Updated)
}
if updStr != "" {
if t, err := nwscommon.ParseTime(updStr); err == nil {
meta.ParsedUpdateTime = t.UTC()
}
}
return raw, meta, true, nil
}

View File

@@ -0,0 +1,68 @@
package nws
import (
"context"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
// ForecastDiscussionSource polls an NWS forecast discussion HTML page and emits a RAW discussion Event.
//
// Output schema:
// - standards.SchemaRawNWSForecastDiscussionV1
type ForecastDiscussionSource struct {
http *fksources.HTTPSource
}
func NewForecastDiscussionSource(cfg config.SourceConfig) (*ForecastDiscussionSource, error) {
const driver = "nws_forecast_discussion"
hs, err := fksources.NewHTTPSource(driver, cfg, "text/html, application/xhtml+xml")
if err != nil {
return nil, err
}
return &ForecastDiscussionSource{http: hs}, nil
}
func (s *ForecastDiscussionSource) Name() string { return s.http.Name }
func (s *ForecastDiscussionSource) Kinds() []event.Kind {
return []event.Kind{event.Kind("forecast_discussion")}
}
func (s *ForecastDiscussionSource) Poll(ctx context.Context) ([]event.Event, error) {
body, changed, err := s.http.FetchBytesIfChanged(ctx)
if err != nil {
return nil, err
}
if !changed {
return nil, nil
}
rawHTML := string(body)
parsed, err := nwscommon.ParseForecastDiscussionHTML(rawHTML)
if err != nil {
return nil, err
}
issuedAt := parsed.IssuedAt.UTC()
effectiveAt := &issuedAt
emittedAt := time.Now().UTC()
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent(
event.Kind("forecast_discussion"),
s.http.Name,
standards.SchemaRawNWSForecastDiscussionV1,
eventID,
emittedAt,
effectiveAt,
rawHTML,
)
}

View File

@@ -0,0 +1,138 @@
package nws
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
func TestForecastDiscussionSourcePollEmitsExpectedEvent(t *testing.T) {
rawHTML := loadForecastDiscussionSampleHTML(t)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
_, _ = w.Write([]byte(rawHTML))
}))
defer srv.Close()
src, err := NewForecastDiscussionSource(forecastDiscussionSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewForecastDiscussionSource() error = %v", err)
}
if got := src.Kinds(); len(got) != 1 || got[0] != event.Kind("forecast_discussion") {
t.Fatalf("Kinds() = %#v, want [forecast_discussion]", got)
}
events, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(events) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(events))
}
got := events[0]
if got.Kind != event.Kind("forecast_discussion") {
t.Fatalf("Kind = %q, want forecast_discussion", got.Kind)
}
if got.Schema != standards.SchemaRawNWSForecastDiscussionV1 {
t.Fatalf("Schema = %q, want %q", got.Schema, standards.SchemaRawNWSForecastDiscussionV1)
}
wantEffectiveAt := time.Date(2026, 3, 28, 19, 24, 0, 0, time.UTC)
if got.EffectiveAt == nil || !got.EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("EffectiveAt = %v, want %s", got.EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
}
payload, ok := got.Payload.(string)
if !ok {
t.Fatalf("Payload type = %T, want string", got.Payload)
}
if payload != rawHTML {
t.Fatalf("Payload did not preserve exact HTML")
}
}
func TestForecastDiscussionSourcePollReturnsNoEventsWhenUnchanged(t *testing.T) {
rawHTML := loadForecastDiscussionSampleHTML(t)
const etag = `"discussion-v1"`
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("If-None-Match") == etag {
w.WriteHeader(http.StatusNotModified)
return
}
w.Header().Set("ETag", etag)
_, _ = w.Write([]byte(rawHTML))
}))
defer srv.Close()
src, err := NewForecastDiscussionSource(forecastDiscussionSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewForecastDiscussionSource() error = %v", err)
}
first, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("first Poll() error = %v", err)
}
if len(first) != 1 {
t.Fatalf("first Poll() len = %d, want 1", len(first))
}
second, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("second Poll() error = %v", err)
}
if len(second) != 0 {
t.Fatalf("second Poll() len = %d, want 0", len(second))
}
}
func TestForecastDiscussionSourcePollRejectsInvalidHTML(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte("<html><body><div>missing discussion block</div></body></html>"))
}))
defer srv.Close()
src, err := NewForecastDiscussionSource(forecastDiscussionSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewForecastDiscussionSource() error = %v", err)
}
_, err = src.Poll(context.Background())
if err == nil {
t.Fatalf("Poll() error = nil, want error")
}
if !strings.Contains(err.Error(), "glossaryProduct") {
t.Fatalf("error = %q, want glossaryProduct context", err)
}
}
func forecastDiscussionSourceConfig(url string) config.SourceConfig {
return config.SourceConfig{
Name: "test-forecast-discussion-source",
Driver: "nws_forecast_discussion",
Mode: config.SourceModePoll,
Params: map[string]any{
"url": url,
"user_agent": "test-agent",
},
}
}
func loadForecastDiscussionSampleHTML(t *testing.T) string {
t.Helper()
path := filepath.Join("..", "..", "providers", "nws", "testdata", "forecast_discussion_sample.html")
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("os.ReadFile(%q) error = %v", path, err)
}
return string(b)
}

View File

@@ -2,15 +2,7 @@
package nws package nws
import ( import (
"context"
"encoding/json"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config" "gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards" "gitea.maximumdirect.net/ejr/weatherfeeder/standards"
) )
@@ -22,114 +14,15 @@ import (
// Output schema (current implementation): // Output schema (current implementation):
// - standards.SchemaRawNWSHourlyForecastV1 // - standards.SchemaRawNWSHourlyForecastV1
type HourlyForecastSource struct { type HourlyForecastSource struct {
http *fksources.HTTPSource *forecastSource
} }
func NewHourlyForecastSource(cfg config.SourceConfig) (*HourlyForecastSource, error) { func NewHourlyForecastSource(cfg config.SourceConfig) (*HourlyForecastSource, error) {
const driver = "nws_forecast_hourly" const driver = "nws_forecast_hourly"
src, err := newForecastSource(cfg, driver, standards.SchemaRawNWSHourlyForecastV1)
// NWS forecast endpoints are GeoJSON (and sometimes also advertise json-ld/json).
hs, err := fksources.NewHTTPSource(driver, cfg, "application/geo+json, application/json")
if err != nil { if err != nil {
return nil, err return nil, err
} }
return &HourlyForecastSource{http: hs}, nil return &HourlyForecastSource{forecastSource: src}, nil
}
func (s *HourlyForecastSource) Name() string { return s.http.Name }
// Kind is used for routing/policy.
func (s *HourlyForecastSource) Kind() event.Kind { return event.Kind("forecast") }
func (s *HourlyForecastSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx)
if err != nil {
return nil, err
}
if !changed {
return nil, nil
}
// EffectiveAt is optional; for forecasts its most naturally the run “issued” time.
// NWS gridpoint forecasts expose generatedAt (preferred) and updateTime/updated.
var effectiveAt *time.Time
switch {
case !meta.ParsedGeneratedAt.IsZero():
t := meta.ParsedGeneratedAt.UTC()
effectiveAt = &t
case !meta.ParsedUpdateTime.IsZero():
t := meta.ParsedUpdateTime.UTC()
effectiveAt = &t
}
emittedAt := time.Now().UTC()
// NWS gridpoint forecast GeoJSON commonly has a stable "id" equal to the endpoint URL.
// That is *not* unique per issued run, so we intentionally do not use it for Event.ID.
// Instead we rely on Source:EffectiveAt (or Source:EmittedAt fallback).
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent(
s.Kind(),
s.http.Name,
standards.SchemaRawNWSHourlyForecastV1,
eventID,
emittedAt,
effectiveAt,
raw,
)
}
// ---- RAW fetch + minimal metadata decode ----
type hourlyForecastMeta struct {
// Present for GeoJSON Feature responses, but often stable (endpoint URL).
ID string `json:"id"`
Properties struct {
GeneratedAt string `json:"generatedAt"` // preferred “issued/run generated” time
UpdateTime string `json:"updateTime"` // last update time of underlying data
Updated string `json:"updated"` // deprecated alias for updateTime
} `json:"properties"`
ParsedGeneratedAt time.Time `json:"-"`
ParsedUpdateTime time.Time `json:"-"`
}
func (s *HourlyForecastSource) fetchRaw(ctx context.Context) (json.RawMessage, hourlyForecastMeta, bool, error) {
raw, changed, err := s.http.FetchJSONIfChanged(ctx)
if err != nil {
return nil, hourlyForecastMeta{}, false, err
}
if !changed {
return nil, hourlyForecastMeta{}, false, nil
}
var meta hourlyForecastMeta
if err := json.Unmarshal(raw, &meta); err != nil {
// If metadata decode fails, still return raw; Poll will fall back to Source:EmittedAt.
return raw, hourlyForecastMeta{}, true, nil
}
// generatedAt (preferred)
genStr := strings.TrimSpace(meta.Properties.GeneratedAt)
if genStr != "" {
if t, err := nwscommon.ParseTime(genStr); err == nil {
meta.ParsedGeneratedAt = t.UTC()
}
}
// updateTime, with fallback to deprecated "updated"
updStr := strings.TrimSpace(meta.Properties.UpdateTime)
if updStr == "" {
updStr = strings.TrimSpace(meta.Properties.Updated)
}
if updStr != "" {
if t, err := nwscommon.ParseTime(updStr); err == nil {
meta.ParsedUpdateTime = t.UTC()
}
}
return raw, meta, true, nil
} }

View File

@@ -2,15 +2,7 @@
package nws package nws
import ( import (
"context"
"encoding/json"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config" "gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards" "gitea.maximumdirect.net/ejr/weatherfeeder/standards"
) )
@@ -22,114 +14,15 @@ import (
// Output schema: // Output schema:
// - standards.SchemaRawNWSNarrativeForecastV1 // - standards.SchemaRawNWSNarrativeForecastV1
type NarrativeForecastSource struct { type NarrativeForecastSource struct {
http *fksources.HTTPSource *forecastSource
} }
func NewNarrativeForecastSource(cfg config.SourceConfig) (*NarrativeForecastSource, error) { func NewNarrativeForecastSource(cfg config.SourceConfig) (*NarrativeForecastSource, error) {
const driver = "nws_forecast_narrative" const driver = "nws_forecast_narrative"
src, err := newForecastSource(cfg, driver, standards.SchemaRawNWSNarrativeForecastV1)
// NWS forecast endpoints are GeoJSON (and sometimes also advertise json-ld/json).
hs, err := fksources.NewHTTPSource(driver, cfg, "application/geo+json, application/json")
if err != nil { if err != nil {
return nil, err return nil, err
} }
return &NarrativeForecastSource{http: hs}, nil return &NarrativeForecastSource{forecastSource: src}, nil
}
func (s *NarrativeForecastSource) Name() string { return s.http.Name }
// Kind is used for routing/policy.
func (s *NarrativeForecastSource) Kind() event.Kind { return event.Kind("forecast") }
func (s *NarrativeForecastSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx)
if err != nil {
return nil, err
}
if !changed {
return nil, nil
}
// EffectiveAt is optional; for forecasts its most naturally the run “issued” time.
// NWS gridpoint forecasts expose generatedAt (preferred) and updateTime/updated.
var effectiveAt *time.Time
switch {
case !meta.ParsedGeneratedAt.IsZero():
t := meta.ParsedGeneratedAt.UTC()
effectiveAt = &t
case !meta.ParsedUpdateTime.IsZero():
t := meta.ParsedUpdateTime.UTC()
effectiveAt = &t
}
emittedAt := time.Now().UTC()
// NWS gridpoint forecast GeoJSON commonly has a stable "id" equal to the endpoint URL.
// That is *not* unique per issued run, so we intentionally do not use it for Event.ID.
// Instead we rely on Source:EffectiveAt (or Source:EmittedAt fallback).
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent(
s.Kind(),
s.http.Name,
standards.SchemaRawNWSNarrativeForecastV1,
eventID,
emittedAt,
effectiveAt,
raw,
)
}
// ---- RAW fetch + minimal metadata decode ----
type narrativeForecastMeta struct {
// Present for GeoJSON Feature responses, but often stable (endpoint URL).
ID string `json:"id"`
Properties struct {
GeneratedAt string `json:"generatedAt"` // preferred “issued/run generated” time
UpdateTime string `json:"updateTime"` // last update time of underlying data
Updated string `json:"updated"` // deprecated alias for updateTime
} `json:"properties"`
ParsedGeneratedAt time.Time `json:"-"`
ParsedUpdateTime time.Time `json:"-"`
}
func (s *NarrativeForecastSource) fetchRaw(ctx context.Context) (json.RawMessage, narrativeForecastMeta, bool, error) {
raw, changed, err := s.http.FetchJSONIfChanged(ctx)
if err != nil {
return nil, narrativeForecastMeta{}, false, err
}
if !changed {
return nil, narrativeForecastMeta{}, false, nil
}
var meta narrativeForecastMeta
if err := json.Unmarshal(raw, &meta); err != nil {
// If metadata decode fails, still return raw; Poll will fall back to Source:EmittedAt.
return raw, narrativeForecastMeta{}, true, nil
}
// generatedAt (preferred)
genStr := strings.TrimSpace(meta.Properties.GeneratedAt)
if genStr != "" {
if t, err := nwscommon.ParseTime(genStr); err == nil {
meta.ParsedGeneratedAt = t.UTC()
}
}
// updateTime, with fallback to deprecated "updated"
updStr := strings.TrimSpace(meta.Properties.UpdateTime)
if updStr == "" {
updStr = strings.TrimSpace(meta.Properties.Updated)
}
if updStr != "" {
if t, err := nwscommon.ParseTime(updStr); err == nil {
meta.ParsedUpdateTime = t.UTC()
}
}
return raw, meta, true, nil
} }

View File

@@ -0,0 +1,189 @@
package nws
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
type forecastPoller interface {
Poll(ctx context.Context) ([]event.Event, error)
}
func TestForecastSourcesEmitExpectedSchemaAndPreferGeneratedAt(t *testing.T) {
tests := []struct {
name string
driver string
wantSchema string
newSource func(config.SourceConfig) (forecastPoller, error)
}{
{
name: "hourly",
driver: "nws_forecast_hourly",
wantSchema: standards.SchemaRawNWSHourlyForecastV1,
newSource: func(cfg config.SourceConfig) (forecastPoller, error) {
return NewHourlyForecastSource(cfg)
},
},
{
name: "narrative",
driver: "nws_forecast_narrative",
wantSchema: standards.SchemaRawNWSNarrativeForecastV1,
newSource: func(cfg config.SourceConfig) (forecastPoller, error) {
return NewNarrativeForecastSource(cfg)
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"properties":{"generatedAt":"2026-03-28T12:00:00Z","updateTime":"2026-03-28T11:00:00Z"}}`))
}))
defer srv.Close()
src, err := tt.newSource(forecastSourceConfig(tt.driver, srv.URL))
if err != nil {
t.Fatalf("newSource() error = %v", err)
}
if ks, ok := src.(interface{ Kinds() []event.Kind }); !ok {
t.Fatalf("source does not implement Kinds()")
} else if gotKinds := ks.Kinds(); len(gotKinds) != 1 || gotKinds[0] != event.Kind("forecast") {
t.Fatalf("Kinds() = %#v, want [forecast]", gotKinds)
}
got, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(got) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(got))
}
if got[0].Schema != tt.wantSchema {
t.Fatalf("Poll() schema = %q, want %q", got[0].Schema, tt.wantSchema)
}
if got[0].Kind != event.Kind("forecast") {
t.Fatalf("Poll() kind = %q, want forecast", got[0].Kind)
}
wantEffectiveAt := time.Date(2026, 3, 28, 12, 0, 0, 0, time.UTC)
if got[0].EffectiveAt == nil || !got[0].EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("Poll() effectiveAt = %v, want %s", got[0].EffectiveAt, wantEffectiveAt)
}
})
}
}
func TestForecastSourcePollEffectiveAtFallbackOrder(t *testing.T) {
tests := []struct {
name string
body string
wantEffectiveAt *time.Time
}{
{
name: "updateTime fallback",
body: `{"properties":{"updateTime":"2026-03-28T11:00:00Z"}}`,
wantEffectiveAt: func() *time.Time {
t := time.Date(2026, 3, 28, 11, 0, 0, 0, time.UTC)
return &t
}(),
},
{
name: "updated fallback",
body: `{"properties":{"updated":"2026-03-28T10:00:00Z"}}`,
wantEffectiveAt: func() *time.Time {
t := time.Date(2026, 3, 28, 10, 0, 0, 0, time.UTC)
return &t
}(),
},
{
name: "omitted when metadata lacks timestamps",
body: `{"properties":{}}`,
wantEffectiveAt: nil,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(tt.body))
}))
defer srv.Close()
src, err := NewHourlyForecastSource(forecastSourceConfig("nws_forecast_hourly", srv.URL))
if err != nil {
t.Fatalf("NewHourlyForecastSource() error = %v", err)
}
got, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(got) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(got))
}
if tt.wantEffectiveAt == nil {
if got[0].EffectiveAt != nil {
t.Fatalf("Poll() effectiveAt = %v, want nil", got[0].EffectiveAt)
}
return
}
if got[0].EffectiveAt == nil || !got[0].EffectiveAt.Equal(*tt.wantEffectiveAt) {
t.Fatalf("Poll() effectiveAt = %v, want %s", got[0].EffectiveAt, *tt.wantEffectiveAt)
}
})
}
}
func TestForecastSourcePollMetadataDecodeFailureStillEmitsRawEvent(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`not-json`))
}))
defer srv.Close()
src, err := NewNarrativeForecastSource(forecastSourceConfig("nws_forecast_narrative", srv.URL))
if err != nil {
t.Fatalf("NewNarrativeForecastSource() error = %v", err)
}
got, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(got) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(got))
}
if got[0].EffectiveAt != nil {
t.Fatalf("Poll() effectiveAt = %v, want nil", got[0].EffectiveAt)
}
if got[0].Schema != standards.SchemaRawNWSNarrativeForecastV1 {
t.Fatalf("Poll() schema = %q, want %q", got[0].Schema, standards.SchemaRawNWSNarrativeForecastV1)
}
raw, ok := got[0].Payload.(json.RawMessage)
if !ok {
t.Fatalf("Poll() payload type = %T, want json.RawMessage", got[0].Payload)
}
if string(raw) != "not-json" {
t.Fatalf("Poll() payload = %q, want %q", string(raw), "not-json")
}
}
func forecastSourceConfig(driver, url string) config.SourceConfig {
return config.SourceConfig{
Name: "test-forecast-source",
Driver: driver,
Mode: config.SourceModePoll,
Params: map[string]any{
"url": url,
"user_agent": "test-agent",
},
}
}

View File

@@ -32,7 +32,7 @@ func NewObservationSource(cfg config.SourceConfig) (*ObservationSource, error) {
func (s *ObservationSource) Name() string { return s.http.Name } func (s *ObservationSource) Name() string { return s.http.Name }
func (s *ObservationSource) Kind() event.Kind { return event.Kind("observation") } func (s *ObservationSource) Kinds() []event.Kind { return []event.Kind{event.Kind("observation")} }
func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) { func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx) raw, meta, changed, err := s.fetchRaw(ctx)
@@ -54,7 +54,7 @@ func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
eventID := fksources.DefaultEventID(meta.ID, s.http.Name, effectiveAt, emittedAt) eventID := fksources.DefaultEventID(meta.ID, s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent( return fksources.SingleEvent(
s.Kind(), event.Kind("observation"),
s.http.Name, s.http.Name,
standards.SchemaRawNWSObservationV1, standards.SchemaRawNWSObservationV1,
eventID, eventID,

View File

@@ -41,6 +41,9 @@ func TestObservationSourcePollReturnsNoEventsOn304(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("NewObservationSource() error = %v", err) t.Fatalf("NewObservationSource() error = %v", err)
} }
if got := src.Kinds(); len(got) != 1 || got[0] != event.Kind("observation") {
t.Fatalf("Kinds() = %#v, want [observation]", got)
}
first, err := src.Poll(context.Background()) first, err := src.Poll(context.Background())
if err != nil { if err != nil {

View File

@@ -0,0 +1,118 @@
package nws
import (
"context"
"encoding/json"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
// WeatherStoriesSource polls an NWS weatherstories endpoint and emits a RAW weather story Event.
//
// Output schema:
// - standards.SchemaRawNWSWeatherStoriesV1
type WeatherStoriesSource struct {
http *fksources.HTTPSource
}
func NewWeatherStoriesSource(cfg config.SourceConfig) (*WeatherStoriesSource, error) {
const driver = "nws_weatherstories"
hs, err := fksources.NewHTTPSource(driver, cfg, "application/geo+json, application/json")
if err != nil {
return nil, err
}
return &WeatherStoriesSource{http: hs}, nil
}
func (s *WeatherStoriesSource) Name() string { return s.http.Name }
func (s *WeatherStoriesSource) Kinds() []event.Kind {
return []event.Kind{event.Kind("weather_story")}
}
func (s *WeatherStoriesSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx)
if err != nil {
return nil, err
}
if !changed {
return nil, nil
}
var effectiveAt *time.Time
switch {
case !meta.ParsedLatestUpdateTime.IsZero():
t := meta.ParsedLatestUpdateTime.UTC()
effectiveAt = &t
case !meta.ParsedLatestStartTime.IsZero():
t := meta.ParsedLatestStartTime.UTC()
effectiveAt = &t
}
emittedAt := time.Now().UTC()
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent(
event.Kind("weather_story"),
s.http.Name,
standards.SchemaRawNWSWeatherStoriesV1,
eventID,
emittedAt,
effectiveAt,
raw,
)
}
type weatherStoriesMeta struct {
Stories []struct {
StartTime string `json:"startTime"`
UpdateTime string `json:"updateTime"`
} `json:"stories"`
ParsedLatestUpdateTime time.Time `json:"-"`
ParsedLatestStartTime time.Time `json:"-"`
}
func (s *WeatherStoriesSource) fetchRaw(ctx context.Context) (json.RawMessage, weatherStoriesMeta, bool, error) {
raw, changed, err := s.http.FetchJSONIfChanged(ctx)
if err != nil {
return nil, weatherStoriesMeta{}, false, err
}
if !changed {
return nil, weatherStoriesMeta{}, false, nil
}
var meta weatherStoriesMeta
if err := json.Unmarshal(raw, &meta); err != nil {
return raw, weatherStoriesMeta{}, true, nil
}
for _, story := range meta.Stories {
if ts := strings.TrimSpace(story.UpdateTime); ts != "" {
if t, err := nwscommon.ParseTime(ts); err == nil {
t = t.UTC()
if meta.ParsedLatestUpdateTime.IsZero() || t.After(meta.ParsedLatestUpdateTime) {
meta.ParsedLatestUpdateTime = t
}
}
}
if ts := strings.TrimSpace(story.StartTime); ts != "" {
if t, err := nwscommon.ParseTime(ts); err == nil {
t = t.UTC()
if meta.ParsedLatestStartTime.IsZero() || t.After(meta.ParsedLatestStartTime) {
meta.ParsedLatestStartTime = t
}
}
}
}
return raw, meta, true, nil
}

View File

@@ -0,0 +1,158 @@
package nws
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
func TestWeatherStoriesSourcePollEmitsExpectedEventAndPrefersLatestUpdateTime(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{
"stories": [
{"startTime":"2026-05-30T08:46:00+00:00","updateTime":"2026-05-30T09:00:34+00:00"},
{"startTime":"2026-05-30T10:00:00+00:00","updateTime":"2026-05-30T11:00:34+00:00"}
]
}`))
}))
defer srv.Close()
src, err := NewWeatherStoriesSource(weatherStoriesSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewWeatherStoriesSource() error = %v", err)
}
if got := src.Kinds(); len(got) != 1 || got[0] != event.Kind("weather_story") {
t.Fatalf("Kinds() = %#v, want [weather_story]", got)
}
events, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(events) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(events))
}
got := events[0]
if got.Kind != event.Kind("weather_story") {
t.Fatalf("Kind = %q, want weather_story", got.Kind)
}
if got.Schema != standards.SchemaRawNWSWeatherStoriesV1 {
t.Fatalf("Schema = %q, want %q", got.Schema, standards.SchemaRawNWSWeatherStoriesV1)
}
wantEffectiveAt := time.Date(2026, 5, 30, 11, 0, 34, 0, time.UTC)
if got.EffectiveAt == nil || !got.EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("EffectiveAt = %v, want %s", got.EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
}
if _, ok := got.Payload.(json.RawMessage); !ok {
t.Fatalf("Payload type = %T, want json.RawMessage", got.Payload)
}
}
func TestWeatherStoriesSourcePollEffectiveAtFallsBackToLatestStartTime(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{
"stories": [
{"startTime":"2026-05-30T08:46:00+00:00","updateTime":"bad"},
{"startTime":"2026-05-31T11:00:00+00:00","updateTime":""}
]
}`))
}))
defer srv.Close()
src, err := NewWeatherStoriesSource(weatherStoriesSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewWeatherStoriesSource() error = %v", err)
}
events, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(events) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(events))
}
wantEffectiveAt := time.Date(2026, 5, 31, 11, 0, 0, 0, time.UTC)
if events[0].EffectiveAt == nil || !events[0].EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("EffectiveAt = %v, want %s", events[0].EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
}
}
func TestWeatherStoriesSourcePollReturnsNoEventsWhenUnchanged(t *testing.T) {
const etag = `"stories-v1"`
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("If-None-Match") == etag {
w.WriteHeader(http.StatusNotModified)
return
}
w.Header().Set("ETag", etag)
_, _ = w.Write([]byte(`{"stories":[]}`))
}))
defer srv.Close()
src, err := NewWeatherStoriesSource(weatherStoriesSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewWeatherStoriesSource() error = %v", err)
}
first, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("first Poll() error = %v", err)
}
if len(first) != 1 {
t.Fatalf("first Poll() len = %d, want 1", len(first))
}
second, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("second Poll() error = %v", err)
}
if len(second) != 0 {
t.Fatalf("second Poll() len = %d, want 0", len(second))
}
}
func TestWeatherStoriesSourcePollMetadataDecodeFailureStillEmitsRawEvent(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`not-json`))
}))
defer srv.Close()
src, err := NewWeatherStoriesSource(weatherStoriesSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewWeatherStoriesSource() error = %v", err)
}
events, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(events) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(events))
}
if events[0].EffectiveAt != nil {
t.Fatalf("EffectiveAt = %v, want nil", events[0].EffectiveAt)
}
if events[0].Schema != standards.SchemaRawNWSWeatherStoriesV1 {
t.Fatalf("Schema = %q, want %q", events[0].Schema, standards.SchemaRawNWSWeatherStoriesV1)
}
}
func weatherStoriesSourceConfig(url string) config.SourceConfig {
return config.SourceConfig{
Name: "test-weatherstories-source",
Driver: "nws_weatherstories",
Mode: config.SourceModePoll,
Params: map[string]any{
"url": url,
"user_agent": "test-agent",
},
}
}

View File

@@ -31,7 +31,7 @@ func NewForecastSource(cfg config.SourceConfig) (*ForecastSource, error) {
func (s *ForecastSource) Name() string { return s.http.Name } func (s *ForecastSource) Name() string { return s.http.Name }
func (s *ForecastSource) Kind() event.Kind { return event.Kind("forecast") } func (s *ForecastSource) Kinds() []event.Kind { return []event.Kind{event.Kind("forecast")} }
func (s *ForecastSource) Poll(ctx context.Context) ([]event.Event, error) { func (s *ForecastSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx) raw, meta, changed, err := s.fetchRaw(ctx)
@@ -55,7 +55,7 @@ func (s *ForecastSource) Poll(ctx context.Context) ([]event.Event, error) {
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt) eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent( return fksources.SingleEvent(
s.Kind(), event.Kind("forecast"),
s.http.Name, s.http.Name,
standards.SchemaRawOpenMeteoHourlyForecastV1, standards.SchemaRawOpenMeteoHourlyForecastV1,
eventID, eventID,

View File

@@ -31,7 +31,7 @@ func NewObservationSource(cfg config.SourceConfig) (*ObservationSource, error) {
func (s *ObservationSource) Name() string { return s.http.Name } func (s *ObservationSource) Name() string { return s.http.Name }
func (s *ObservationSource) Kind() event.Kind { return event.Kind("observation") } func (s *ObservationSource) Kinds() []event.Kind { return []event.Kind{event.Kind("observation")} }
func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) { func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx) raw, meta, changed, err := s.fetchRaw(ctx)
@@ -52,7 +52,7 @@ func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt) eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent( return fksources.SingleEvent(
s.Kind(), event.Kind("observation"),
s.http.Name, s.http.Name,
standards.SchemaRawOpenMeteoCurrentV1, standards.SchemaRawOpenMeteoCurrentV1,
eventID, eventID,

View File

@@ -0,0 +1,44 @@
package openmeteo
import (
"testing"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
)
func TestObservationSourceAdvertisesKinds(t *testing.T) {
src, err := NewObservationSource(config.SourceConfig{
Name: "openmeteo-observation-test",
Driver: "openmeteo_observation",
Mode: config.SourceModePoll,
Params: map[string]any{
"url": "https://example.invalid",
"user_agent": "test-agent",
},
})
if err != nil {
t.Fatalf("NewObservationSource() error = %v", err)
}
if got := src.Kinds(); len(got) != 1 || got[0] != event.Kind("observation") {
t.Fatalf("Kinds() = %#v, want [observation]", got)
}
}
func TestForecastSourceAdvertisesKinds(t *testing.T) {
src, err := NewForecastSource(config.SourceConfig{
Name: "openmeteo-forecast-test",
Driver: "openmeteo_forecast",
Mode: config.SourceModePoll,
Params: map[string]any{
"url": "https://example.invalid",
"user_agent": "test-agent",
},
})
if err != nil {
t.Fatalf("NewForecastSource() error = %v", err)
}
if got := src.Kinds(); len(got) != 1 || got[0] != event.Kind("forecast") {
t.Fatalf("Kinds() = %#v, want [forecast]", got)
}
}

View File

@@ -35,7 +35,7 @@ func NewObservationSource(cfg config.SourceConfig) (*ObservationSource, error) {
func (s *ObservationSource) Name() string { return s.http.Name } func (s *ObservationSource) Name() string { return s.http.Name }
func (s *ObservationSource) Kind() event.Kind { return event.Kind("observation") } func (s *ObservationSource) Kinds() []event.Kind { return []event.Kind{event.Kind("observation")} }
func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) { func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
if err := owcommon.RequireMetricUnits(s.http.URL); err != nil { if err := owcommon.RequireMetricUnits(s.http.URL); err != nil {
@@ -60,7 +60,7 @@ func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt) eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent( return fksources.SingleEvent(
s.Kind(), event.Kind("observation"),
s.http.Name, s.http.Name,
standards.SchemaRawOpenWeatherCurrentV1, standards.SchemaRawOpenWeatherCurrentV1,
eventID, eventID,

View File

@@ -0,0 +1,26 @@
package openweather
import (
"testing"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
)
func TestObservationSourceAdvertisesKinds(t *testing.T) {
src, err := NewObservationSource(config.SourceConfig{
Name: "openweather-observation-test",
Driver: "openweather_observation",
Mode: config.SourceModePoll,
Params: map[string]any{
"url": "https://example.invalid?units=metric",
"user_agent": "test-agent",
},
})
if err != nil {
t.Fatalf("NewObservationSource() error = %v", err)
}
if got := src.Kinds(); len(got) != 1 || got[0] != event.Kind("observation") {
t.Fatalf("Kinds() = %#v, want [observation]", got)
}
}

View File

@@ -71,9 +71,9 @@ type WeatherForecastPeriod struct {
// Providers vary in whether they explicitly include this. // Providers vary in whether they explicitly include this.
IsDay *bool `json:"isDay,omitempty"` IsDay *bool `json:"isDay,omitempty"`
// Canonical internal representation (provider-independent). // Canonical internal representation (provider-independent), when applicable.
// Like WeatherObservation, this is required; use an “unknown” WMOCode if unmappable. // Some products (notably narrative) may not provide or imply a canonical WMO code.
ConditionCode WMOCode `json:"conditionCode"` ConditionCode *WMOCode `json:"conditionCode,omitempty"`
// Human-facing narrative summary for this period. // Human-facing narrative summary for this period.
TextDescription string `json:"textDescription,omitempty"` TextDescription string `json:"textDescription,omitempty"`

View File

@@ -0,0 +1,40 @@
package model
import "time"
// ForecastDiscussionProduct distinguishes the discussion bulletin family.
//
// Today weatherfeeder only normalizes Area Forecast Discussion (AFD) products,
// but this remains a distinct type so additional discussion-like products can be
// added without changing the payload field type.
type ForecastDiscussionProduct string
const (
ForecastDiscussionProductAFD ForecastDiscussionProduct = "afd"
)
// WeatherForecastDiscussion is a canonical issued discussion bulletin for an NWS office.
//
// Unlike WeatherForecastRun, this is authored narrative text rather than a sequence
// of forecast periods.
type WeatherForecastDiscussion struct {
OfficeID string `json:"officeId,omitempty"`
OfficeName string `json:"officeName,omitempty"`
Product ForecastDiscussionProduct `json:"product"`
IssuedAt time.Time `json:"issuedAt"`
UpdatedAt *time.Time `json:"updatedAt,omitempty"`
KeyMessages []string `json:"keyMessages,omitempty"`
ShortTerm *WeatherForecastDiscussionSection `json:"shortTerm,omitempty"`
LongTerm *WeatherForecastDiscussionSection `json:"longTerm,omitempty"`
}
// WeatherForecastDiscussionSection is a fixed prose section within a discussion bulletin.
type WeatherForecastDiscussionSection struct {
Qualifier string `json:"qualifier,omitempty"`
IssuedAt *time.Time `json:"issuedAt,omitempty"`
Text string `json:"text,omitempty"`
}

28
model/weather_story.go Normal file
View File

@@ -0,0 +1,28 @@
package model
import "time"
// WeatherStoryRun is a snapshot of NWS weather stories for an office as-of a point in time.
type WeatherStoryRun struct {
OfficeID string `json:"officeId,omitempty"`
AsOf time.Time `json:"asOf"`
Stories []WeatherStory `json:"stories"`
}
// WeatherStory is a provider-independent representation of a single story card.
type WeatherStory struct {
OfficeID string `json:"officeId,omitempty"`
StartTime time.Time `json:"startTime"`
EndTime time.Time `json:"endTime"`
UpdatedAt time.Time `json:"updatedAt"`
Title string `json:"title,omitempty"`
Description string `json:"description,omitempty"`
AltText string `json:"altText,omitempty"`
Priority bool `json:"priority"`
Order int `json:"order"`
DownloadURL string `json:"downloadUrl,omitempty"`
}

View File

@@ -17,6 +17,8 @@ const (
SchemaRawNWSHourlyForecastV1 = "raw.nws.hourly.forecast.v1" SchemaRawNWSHourlyForecastV1 = "raw.nws.hourly.forecast.v1"
SchemaRawNWSNarrativeForecastV1 = "raw.nws.narrative.forecast.v1" SchemaRawNWSNarrativeForecastV1 = "raw.nws.narrative.forecast.v1"
SchemaRawNWSForecastDiscussionV1 = "raw.nws.forecast_discussion.v1"
SchemaRawNWSWeatherStoriesV1 = "raw.nws.weatherstories.v1"
SchemaRawOpenMeteoHourlyForecastV1 = "raw.openmeteo.hourly.forecast.v1" SchemaRawOpenMeteoHourlyForecastV1 = "raw.openmeteo.hourly.forecast.v1"
SchemaRawOpenWeatherHourlyForecastV1 = "raw.openweather.hourly.forecast.v1" SchemaRawOpenWeatherHourlyForecastV1 = "raw.openweather.hourly.forecast.v1"
@@ -25,5 +27,7 @@ const (
// Canonical domain schemas (emitted after normalization). // Canonical domain schemas (emitted after normalization).
SchemaWeatherObservationV1 = "weather.observation.v1" SchemaWeatherObservationV1 = "weather.observation.v1"
SchemaWeatherForecastV1 = "weather.forecast.v1" SchemaWeatherForecastV1 = "weather.forecast.v1"
SchemaWeatherForecastDiscussionV1 = "weather.forecast_discussion.v1"
SchemaWeatherStoryV1 = "weather.weather_story.v1"
SchemaWeatherAlertV1 = "weather.alert.v1" SchemaWeatherAlertV1 = "weather.alert.v1"
) )