31 Commits

Author SHA1 Message Date
dec05821bf Add a staged cleanup roadmap to address the code quality audit
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-06-10 21:02:07 -05:00
1a9f462fbf Audit code quality and deduplication opportunities 2026-06-10 20:04:00 -05:00
a990da957b Finalize SPC outlook feature addition and clean up implemented roadmap documentation
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-06-10 19:54:39 -05:00
a4cd63ca4e Update the SPC outlook implementation plan to identify remaining gaps and corrections
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-06-10 19:40:46 -05:00
fba519cab0 Update documentation for SPC outlook support
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-06-11 00:30:49 +00:00
da8ff81692 Verify SPC outlook implementation 2026-06-11 00:28:30 +00:00
f91a185f9d Document SPC outlook support 2026-06-11 00:27:23 +00:00
e966276c40 Add Postgres mapping for SPC outlooks 2026-06-11 00:24:12 +00:00
1e2db468ea Add SPC outlook normalization 2026-06-11 00:20:31 +00:00
cefd4dfc7c Add raw SPC convective outlook source 2026-06-11 00:15:40 +00:00
42c646c328 Add GeoJSON point containment helper 2026-06-11 00:10:01 +00:00
b2c429983c Add SPC provider parsing helpers 2026-06-11 00:06:55 +00:00
0f20d1e4cb Add AGENTS.md 2026-06-10 23:59:45 +00:00
979d754d18 Added an implementation plan for SPC convective outlook support, and completed the documentation cleanup
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-06-10 18:59:16 -05:00
002f9d0ba6 Clean up documentation consistency
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-06-10 20:28:41 +00:00
ec115ba152 Document provider integration contracts 2026-06-10 20:25:13 +00:00
47176520bb Document development workflow and internals 2026-06-10 20:22:38 +00:00
9e27a431a1 Add maintained configuration examples 2026-06-10 20:17:40 +00:00
f0605781a2 Document operations and troubleshooting 2026-06-10 20:15:08 +00:00
abb8f218ec Document event and Postgres integration contracts 2026-06-10 20:12:27 +00:00
4ac4e401ed Document weatherfeeder CLI and configuration 2026-06-10 20:07:56 +00:00
77bd59cb87 Add policy documents and a roadmap to implement a full documentation set
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-06-10 14:59:20 -05:00
bb5abf798b Add an initial roadmap for SPC convective outlook support 2026-06-10 14:49:28 -05:00
fd820fd964 Implemented NWS weather stories support
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-05-30 06:47:18 -05:00
cca873cafb Made forecast-period conditionCode optional
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
2026-05-28 07:47:07 -05:00
f457bab039 Updated dependencies to feedkit v0.9.1
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-03-29 10:54:36 -05:00
6712c16167 Updated to feedkit v0.9.0
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-03-29 08:35:56 -05:00
a0389ebce8 Added support for Area Forecast Discussions issued by the NWS
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-03-28 16:17:03 -05:00
40f17c9d86 Updates to track upstream feedkit v0.8.2
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-03-28 13:53:54 -05:00
c76088c38c Code cleanup and deduplication pass through weatherfeeder
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
ci/woodpecker/manual/build-image Pipeline was successful
2026-03-28 12:01:07 -05:00
2c1278a70a Moved generic and broadly useful helper functions upstream into feedkit
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-03-28 11:30:20 -05:00
118 changed files with 10788 additions and 1181 deletions

4
AGENTS.md Normal file
View File

@@ -0,0 +1,4 @@
Please carefully review the documents in `docs/policy` before making any changes to this repository.
- `architecture.md` provides the canonical high-level architecture policy for this repository.
- `development.md` provides more granular development policy for this repository.
- `documentation.md` provides the canonical documentation policy for this repository.

299
API.md
View File

@@ -1,299 +0,0 @@
# weatherfeeder API (Wire Contract)
This document defines the stable, consumer-facing JSON contract emitted by weatherfeeder sinks.
weatherfeeder emits **events** encoded as JSON. Each event has:
- an **envelope** (metadata + schema identifier), and
- a **payload** whose shape is determined by `schema`.
Downstream consumers should:
1. parse the event envelope,
2. switch on `schema`, then
3. decode `payload` into the matching schema.
---
## Event envelope
All events are JSON objects with these fields:
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `id` | string | yes | Stable event identifier. Treat as opaque. |
| `schema` | string | yes | Schema identifier (e.g. `weather.observation.v1`). |
| `source` | string | yes | Provider/source identifier (stable within configuration). |
| `effectiveAt` | string (timestamp) | yes | RFC3339Nano timestamp indicating when this event is effective. |
| `payload` | object | yes | Schema-specific payload (see below). |
### Timestamp format
All timestamps are encoded as JSON strings using Gos `time.Time` JSON encoding (RFC3339Nano).
Examples:
- `"2026-01-17T14:27:00Z"`
- `"2026-01-17T08:27:00-06:00"`
---
## Canonical schemas
weatherfeeder emits three canonical domain schemas:
- `weather.observation.v1`
- `weather.forecast.v1`
- `weather.alert.v1`
Each payload is described below using the JSON field names as the contract.
---
## Shared Conventions
- Timestamps are JSON strings in RFC3339Nano format.
- Optional fields are omitted when unknown (`omitempty` behavior).
- Numeric measurements are normalized to metric units:
- `*C` = Celsius
- `*Kmh` = kilometers/hour
- `*Pa` = Pascals
- `*Meters` = meters
- `*Mm` = millimeters
- `*Percent` = percent (0-100)
- `conditionCode` is a WMO weather interpretation code (`int`).
- Unknown/unmappable is `-1`.
- Downstream consumers should treat unknown codes as “unknown conditions” rather than failing decoding.
- For readability and stability, weatherfeeder rounds floating-point values in canonical payloads to
**4 digits after the decimal** during normalization.
---
## Schema: `weather.observation.v1`
Payload type: `WeatherObservation`
A `WeatherObservation` represents a point-in-time observation for a station/location.
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `stationId` | string | no | Provider station/location identifier |
| `stationName` | string | no | Human station name |
| `timestamp` | timestamp string | yes | Observation timestamp |
| `conditionCode` | int | yes | WMO code (`-1` unknown) |
| `isDay` | bool | no | Day/night hint |
| `textDescription` | string | no | Human-facing short description |
| `temperatureC` | number | no | Celsius |
| `dewpointC` | number | no | Celsius |
| `windDirectionDegrees` | number | no | Degrees |
| `windSpeedKmh` | number | no | km/h |
| `windGustKmh` | number | no | km/h |
| `barometricPressurePa` | number | no | Pascals |
| `visibilityMeters` | number | no | Meters |
| `relativeHumidityPercent` | number | no | Percent |
| `apparentTemperatureC` | number | no | Celsius |
| `presentWeather` | array | no | Provider-specific structured weather fragments |
### Nested: `presentWeather[]`
Each `presentWeather[]` element:
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `raw` | object | no | Provider-specific JSON object |
---
## Schema: `weather.forecast.v1`
Payload type: `WeatherForecastRun`
A `WeatherForecastRun` is a single issued forecast snapshot for a location and a specific product
(hourly / narrative / daily). The run contains an ordered list of forecast periods.
### `product` values
`product` is one of:
- `"hourly"`
- `"narrative"`
- `"daily"`
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `locationId` | string | no | Provider location identifier |
| `locationName` | string | no | Human name, if available |
| `issuedAt` | string (timestamp) | yes | When this run was generated/issued |
| `updatedAt` | string (timestamp) | no | Optional later update time |
| `product` | string | yes | One of `hourly`, `narrative`, `daily` |
| `latitude` | number | no | Degrees |
| `longitude` | number | no | Degrees |
| `elevationMeters` | number | no | meters |
| `periods` | array | yes | Chronological forecast periods |
### Nested: `periods[]` (`WeatherForecastPeriod`)
A `WeatherForecastPeriod` is valid for `[startTime, endTime)`.
| Field | Type | Required | Units / Notes |
|---|---:|:---:|---|
| `startTime` | string (timestamp) | yes | Period start |
| `endTime` | string (timestamp) | yes | Period end |
| `name` | string | no | Human label (often empty for hourly) |
| `isDay` | bool | no | Day/night hint |
| `conditionCode` | int | yes | WMO code (`-1` for unknown) |
| `textDescription` | string | no | Human-facing short phrase |
| `temperatureC` | number | no | °C |
| `temperatureCMin` | number | no | °C (aggregated products) |
| `temperatureCMax` | number | no | °C (aggregated products) |
| `dewpointC` | number | no | °C |
| `relativeHumidityPercent` | number | no | percent |
| `windDirectionDegrees` | number | no | degrees |
| `windSpeedKmh` | number | no | km/h |
| `windGustKmh` | number | no | km/h |
| `barometricPressurePa` | number | no | Pa |
| `visibilityMeters` | number | no | meters |
| `apparentTemperatureC` | number | no | °C |
| `cloudCoverPercent` | number | no | percent |
| `probabilityOfPrecipitationPercent` | number | no | percent |
| `precipitationAmountMm` | number | no | mm (liquid equivalent) |
| `snowfallDepthMm` | number | no | mm |
| `uvIndex` | number | no | unitless index |
---
## Schema: `weather.alert.v1`
Payload type: `WeatherAlertRun`
A `WeatherAlertRun` is a snapshot of *active* alerts for a location as-of a point in time.
A run may contain zero, one, or many alerts.
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `locationId` | string | no | Provider location identifier |
| `locationName` | string | no | Human name, if available |
| `asOf` | string (timestamp) | yes | When the provider asserted this snapshot is current |
| `latitude` | number | no | Degrees |
| `longitude` | number | no | Degrees |
| `alerts` | array | yes | Active alerts (order provider-dependent) |
### Nested: `alerts[]` (`WeatherAlert`)
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `id` | string | yes | Provider-stable identifier (often a URL/URI) |
| `event` | string | no | Classification/event label |
| `headline` | string | no | Alert headline |
| `severity` | string | no | Example: Extreme/Severe/Moderate/Minor/Unknown |
| `urgency` | string | no | Example: Immediate/Expected/Future/Past/Unknown |
| `certainty` | string | no | Example: Observed/Likely/Possible/Unlikely/Unknown |
| `status` | string | no | Example: Actual/Exercise/Test/System/Unknown |
| `messageType` | string | no | Example: Alert/Update/Cancel |
| `category` | string | no | Example: Met/Geo/Safety/Rescue/Fire/Health/Env/Transport/Infra/CBRNE/Other |
| `response` | string | no | Example: Shelter/Evacuate/Prepare/Execute/Avoid/Monitor/Assess/AllClear/None |
| `response` | string | no | e.g. Shelter/Evacuate/Prepare/... |
| `description` | string | no | Narrative |
| `instruction` | string | no | What to do |
| `sent` | string (timestamp) | no | Provider-dependent |
| `effective` | string (timestamp) | no | Provider-dependent |
| `onset` | string (timestamp) | no | Provider-dependent |
| `expires` | string (timestamp) | no | Provider-dependent |
| `areaDescription` | string | no | Often a provider string |
| `senderName` | string | no | Provenance |
| `references` | array | no | Related alert references |
### Nested: `references[]` (`AlertReference`)
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `id` | string | no | Provider reference ID/URI |
| `identifier` | string | no | Provider identifier string, if distinct |
| `sender` | string | no | Sender |
| `sent` | string (timestamp) | no | Timestamp |
---
## Compatibility rules
- Consumers **must** ignore unknown fields.
- Producers (weatherfeeder) prefer **additive changes** within a schema version.
- Renames/removals/semantic breaks normally require a **schema version bump** (`weather.*.v2`); pre-1.0 projects may choose in-place changes.
---
## Examples
### Observation event (`weather.observation.v1`)
```json
{
"id": "nws:KSTL:2026-01-17T14:00:00Z",
"schema": "weather.observation.v1",
"source": "nws_observation",
"effectiveAt": "2026-01-17T14:00:00Z",
"payload": {
"stationId": "KSTL",
"timestamp": "2026-01-17T14:00:00Z",
"conditionCode": 1,
"textDescription": "Mainly Sunny",
"temperatureC": 3.25,
"windSpeedKmh": 18.5
}
}
```
### Forecast event (`weather.forecast.v1`)
```json
{
"id": "openmeteo:38.63,-90.20:2026-01-17T13:00:00Z",
"schema": "weather.forecast.v1",
"source": "openmeteo_forecast",
"effectiveAt": "2026-01-17T13:00:00Z",
"payload": {
"locationName": "St. Louis, MO",
"issuedAt": "2026-01-17T13:00:00Z",
"product": "hourly",
"latitude": 38.63,
"longitude": -90.2,
"periods": [
{
"startTime": "2026-01-17T14:00:00Z",
"endTime": "2026-01-17T15:00:00Z",
"conditionCode": 2,
"textDescription": "Partly Cloudy",
"temperatureC": 3.5,
"probabilityOfPrecipitationPercent": 10
}
]
}
}
```
### Alert event (`weather.alert.v1`)
```json
{
"id": "nws:alerts:2026-01-17T14:10:00Z",
"schema": "weather.alert.v1",
"source": "nws_alerts",
"effectiveAt": "2026-01-17T14:10:00Z",
"payload": {
"asOf": "2026-01-17T14:05:00Z",
"alerts": [
{
"id": "https://api.weather.gov/alerts/abc123",
"event": "Winter Weather Advisory",
"headline": "Winter Weather Advisory issued January 17 at 8:05AM CST",
"severity": "Moderate",
"description": "Mixed precipitation expected...",
"expires": "2026-01-18T06:00:00Z"
}
]
}
}
```

View File

@@ -1,34 +1,40 @@
# weatherfeeder
weatherfeeder is a small daemon that polls weather observations, forecasts, and alerts from multiple upstream
providers, normalizes them into a provider-independent format, and emits them to a sink.
`weatherfeeder` is a config-driven daemon that polls weather providers, normalizes
provider-specific responses into canonical weather events, and routes those
events to configured sinks.
Today, the only implemented sink is `stdout`, which prints JSON-encoded events.
It currently supports NWS observations, alerts, hourly forecasts, narrative
forecasts, forecast discussions, and weather stories; SPC Day 1-3 convective
outlooks; Open-Meteo observations and hourly forecasts; and OpenWeather
observations. Implemented sinks are stdout, NATS, and Postgres.
## What weatherfeeder emits
## Quickstart
weatherfeeder emits **feed events** encoded as JSON. Each event includes a schema identifier and a payload.
Downstream consumers should key off the `schema` value and decode the `payload` accordingly.
Run the checked-in sample config:
Canonical domain schemas emitted after normalization:
```sh
cd cmd/weatherfeeder
go run .
```
- `weather.observation.v1``WeatherObservation`
- `weather.forecast.v1``WeatherForecastRun`
- `weather.alert.v1``WeatherAlertRun`
The sample config at `cmd/weatherfeeder/config.yml` is load-tested and can be
used as a starting point. The executable always reads `config.yml` from its
current working directory.
For the complete wire contract (event envelope + payload schemas, fields, units, and compatibility rules), see:
## Documentation
- **API.md**
## Upstream providers (current MVP)
- NWS: observations, hourly forecasts, narrative forecasts, alerts
- Open-Meteo: observations, hourly forecasts
- OpenWeather: observations
## Versioning & compatibility
The JSON field names on canonical payload types are treated as part of the wire contract.
Additive changes are preferred. Renames/removals require a schema version bump.
See **API.md** for details.
- [CLI reference](docs/cli.md)
- [Configuration reference](docs/config.md)
- [Operations guide](docs/operations.md)
- [Troubleshooting guide](docs/troubleshooting.md)
- [Example configs](examples/)
- [Event wire contract](docs/integrations/events.md)
- [Postgres table contract](docs/integrations/postgres.md)
- [NWS integration notes](docs/integrations/nws.md)
- [SPC integration notes](docs/integrations/spc.md)
- [Open-Meteo integration notes](docs/integrations/openmeteo.md)
- [OpenWeather integration notes](docs/integrations/openweather.md)
- [Architecture policy](docs/policy/architecture.md)
- [Development policy](docs/policy/development.md)
- [Documentation policy](docs/policy/documentation.md)

View File

@@ -15,7 +15,7 @@ sources:
# driver: openmeteo_observation
# every: 10m
# params:
# url: "https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&current=temperature_2m,relative_humidity_2m,weather_code,wind_speed_10m,wind_direction_10m,precipitation,surface_pressure,rain,showers,snowfall,cloud_cover,apparent_temperature,is_day,wind_gusts_10m,pressure_msl&forecast_days=1"
# url: "https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&current=temperature_2m,relative_humidity_2m,weather_code,wind_speed_10m,wind_direction_10m,precipitation,surface_pressure,rain,showers,snowfall,cloud_cover,apparent_temperature,is_day,wind_gusts_10m,pressure_msl"
# user_agent: "HomeOps (eric@maximumdirect.net)"
# - name: OpenWeatherObservation
@@ -24,7 +24,7 @@ sources:
# driver: openweather_observation
# every: 10m
# params:
# url: "https://api.openweathermap.org/data/2.5/weather?lat=38.6239&lon=-90.3571&appid=c954f2566cb7ccb56b43737b52e88fc6&units=metric"
# url: "https://api.openweathermap.org/data/2.5/weather?lat=38.6239&lon=-90.3571&units=metric"
# user_agent: "HomeOps (eric@maximumdirect.net)"
# - name: NWSObservationKSUS
@@ -63,6 +63,24 @@ sources:
url: "https://api.weather.gov/gridpoints/LSX/90,74/forecast?units=us"
user_agent: "HomeOps (eric@maximumdirect.net)"
- name: NWSForecastDiscussionSTL
mode: poll
kinds: ["forecast_discussion"]
driver: nws_forecast_discussion
every: 30m
params:
url: "https://forecast.weather.gov/product.php?site=LSX&issuedby=LSX&product=AFD&format=TXT&version=1&glossary=0"
user_agent: "HomeOps (eric@maximumdirect.net)"
- name: NWSWeatherStoriesSTL
mode: poll
kinds: ["weather_story"]
driver: nws_weatherstories
every: 30m
params:
url: "https://api.weather.gov/offices/LSX/weatherstories"
user_agent: "HomeOps (eric@maximumdirect.net)"
- name: OpenMeteoHourlyForecastSTL
mode: poll
kinds: ["forecast"]
@@ -81,6 +99,18 @@ sources:
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
user_agent: "HomeOps (eric@maximumdirect.net)"
- name: SPCConvectiveOutlookSTL
mode: poll
kinds: ["outlook"]
driver: spc_convective_outlook
every: 30m
params:
latitude: 38.6239
longitude: -90.3571
location_id: "stl"
location_name: "St. Louis, MO"
user_agent: "HomeOps (eric@maximumdirect.net)"
sinks:
- name: stdout
driver: stdout
@@ -90,14 +120,14 @@ sinks:
driver: nats
params:
url: nats://nats:4222
exchange: weatherfeeder
subject: weatherfeeder
# - name: pg_weatherfeeder
# driver: postgres
# params:
# uri: postgres://weatherdb:5432/weatherdb?sslmode=disable
# username: weatherdb
# password: weatherdb
# password: <database_password>
# prune: 3d
# # Prunes rows older than now-3d on each write transaction.
@@ -108,13 +138,13 @@ sinks:
routes:
- sink: stdout
kinds: ["observation", "forecast", "alert"]
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"]
- sink: nats_weatherfeeder
kinds: ["observation", "forecast", "alert"]
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"]
# - sink: pg_weatherfeeder
# kinds: ["observation", "forecast", "alert"]
# kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"]
# - sink: logfile
# kinds: ["observation", "alert", "forecast"]
# kinds: ["observation", "alert", "forecast", "forecast_discussion", "weather_story", "outlook"]

View File

@@ -3,14 +3,10 @@ package main
import (
"context"
"errors"
"fmt"
"log"
"os"
"os/signal"
"sort"
"strings"
"syscall"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
fkdispatch "gitea.maximumdirect.net/ejr/feedkit/dispatch"
@@ -41,9 +37,6 @@ func main() {
if err != nil {
log.Fatalf("config load failed: %v", err)
}
if err := wfpgsink.RegisterPostgresSchemas(cfg); err != nil {
log.Fatalf("postgres schema registration failed: %v", err)
}
// --- Registries ---
srcReg := fksources.NewRegistry()
@@ -51,15 +44,8 @@ func main() {
// Compile stdout, Postgres, and NATS sinks for weatherfeeder. The former is useful for debugging and the latter are the main intended outputs.
sinkReg := fksinks.NewRegistry()
sinkReg.Register("stdout", func(cfg config.SinkConfig) (fksinks.Sink, error) {
return fksinks.NewStdoutSink(cfg.Name), nil
})
sinkReg.Register("postgres", func(cfg config.SinkConfig) (fksinks.Sink, error) {
return fksinks.NewPostgresSinkFromConfig(cfg)
})
sinkReg.Register("nats", func(cfg config.SinkConfig) (fksinks.Sink, error) {
return fksinks.NewNATSSinkFromConfig(cfg)
})
fksinks.RegisterBuiltins(sinkReg)
sinkReg.Register("postgres", fksinks.PostgresFactory(wfpgsink.PostgresSchema()))
// --- Build sources into scheduler jobs ---
var jobs []fkscheduler.Job
@@ -69,23 +55,15 @@ func main() {
log.Fatalf("build source failed (sources[%d] name=%q driver=%q): %v", i, sc.Name, sc.Driver, err)
}
if err := validateSourceExpectedKinds(sc, in); err != nil {
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
log.Fatalf("source expected kinds validation failed (sources[%d] name=%q driver=%q): %v", i, sc.Name, sc.Driver, err)
}
// If this is a polling source, every is required.
if _, ok := in.(fksources.PollSource); ok && sc.Every.Duration <= 0 {
log.Fatalf(
"polling source missing/invalid interval (sources[%d] name=%q driver=%q): sources[].every must be > 0",
i, sc.Name, sc.Driver,
)
job, err := fkscheduler.JobFromSourceConfig(in, sc)
if err != nil {
log.Fatalf("build scheduler job failed (sources[%d] name=%q driver=%q): %v", i, sc.Name, sc.Driver, err)
}
// For stream sources, Every is ignored; it is fine if omitted/zero.
jobs = append(jobs, fkscheduler.Job{
Source: in,
Every: sc.Every.Duration,
})
jobs = append(jobs, job)
}
// --- Build sinks ---
@@ -99,7 +77,7 @@ func main() {
}
// --- Compile routes ---
routes, err := compileRoutes(cfg, builtSinks)
routes, err := fkdispatch.CompileRoutes(cfg)
if err != nil {
log.Fatalf("compile routes failed: %v", err)
}
@@ -160,124 +138,6 @@ func main() {
log.Printf("shutdown complete")
}
func compileRoutes(cfg *config.Config, builtSinks map[string]fksinks.Sink) ([]fkdispatch.Route, error) {
if len(cfg.Routes) == 0 {
return defaultRoutes(builtSinks), nil
}
var routes []fkdispatch.Route
for i, r := range cfg.Routes {
if strings.TrimSpace(r.Sink) == "" {
return nil, fmt.Errorf("routes[%d].sink is empty", i)
}
if _, ok := builtSinks[r.Sink]; !ok {
return nil, fmt.Errorf("routes[%d].sink references unknown sink %q", i, r.Sink)
}
kinds := map[fkevent.Kind]bool{}
for j, k := range r.Kinds {
kind, err := fkevent.ParseKind(k)
if err != nil {
return nil, fmt.Errorf("routes[%d].kinds[%d]: %w", i, j, err)
}
kinds[kind] = true
}
routes = append(routes, fkdispatch.Route{
SinkName: r.Sink,
Kinds: kinds,
})
}
return routes, nil
}
func defaultRoutes(builtSinks map[string]fksinks.Sink) []fkdispatch.Route {
// nil Kinds means "match all kinds" by convention
var allKinds map[fkevent.Kind]bool = nil
routes := make([]fkdispatch.Route, 0, len(builtSinks))
for name := range builtSinks {
routes = append(routes, fkdispatch.Route{
SinkName: name,
Kinds: allKinds,
})
}
return routes
}
func isContextShutdown(err error) bool {
return errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)
}
func validateSourceExpectedKinds(sc config.SourceConfig, in fksources.Input) error {
expectedKinds, err := parseExpectedKinds(sc.ExpectedKinds())
if err != nil {
return err
}
if len(expectedKinds) == 0 {
return nil
}
advertisedKinds := advertisedSourceKinds(in)
if len(advertisedKinds) == 0 {
return nil
}
for kind := range expectedKinds {
if !advertisedKinds[kind] {
return fmt.Errorf(
"configured expected kind %q not advertised by source (configured=%v advertised=%v)",
kind,
sortedKinds(expectedKinds),
sortedKinds(advertisedKinds),
)
}
}
return nil
}
func parseExpectedKinds(raw []string) (map[fkevent.Kind]bool, error) {
kinds := map[fkevent.Kind]bool{}
for i, k := range raw {
kind, err := fkevent.ParseKind(k)
if err != nil {
return nil, fmt.Errorf("invalid expected kind at index %d (%q): %w", i, k, err)
}
kinds[kind] = true
}
return kinds, nil
}
func advertisedSourceKinds(in fksources.Input) map[fkevent.Kind]bool {
if in == nil {
return nil
}
kinds := map[fkevent.Kind]bool{}
if ks, ok := in.(fksources.KindsSource); ok {
for _, kind := range ks.Kinds() {
kinds[kind] = true
}
return kinds
}
if ks, ok := in.(fksources.KindSource); ok {
kinds[ks.Kind()] = true
return kinds
}
return nil
}
func sortedKinds(kindSet map[fkevent.Kind]bool) []string {
out := make([]string, 0, len(kindSet))
for kind := range kindSet {
out = append(out, string(kind))
}
sort.Strings(out)
return out
}
// keep time imported (mirrors your previous main.go defensive trick)
var _ = time.Second

View File

@@ -2,7 +2,9 @@ package main
import (
"context"
"path/filepath"
"reflect"
"sort"
"strings"
"testing"
"time"
@@ -13,8 +15,11 @@ import (
fkprocessors "gitea.maximumdirect.net/ejr/feedkit/processors"
fkdedupe "gitea.maximumdirect.net/ejr/feedkit/processors/dedupe"
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
fkscheduler "gitea.maximumdirect.net/ejr/feedkit/scheduler"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
wfnormalizers "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers"
wfsources "gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources"
)
type testInput struct {
@@ -23,13 +28,6 @@ type testInput struct {
func (s testInput) Name() string { return s.name }
type testKindSource struct {
testInput
kind fkevent.Kind
}
func (s testKindSource) Kind() fkevent.Kind { return s.kind }
type testKindsSource struct {
testInput
kinds []fkevent.Kind
@@ -37,18 +35,6 @@ type testKindsSource struct {
func (s testKindsSource) Kinds() []fkevent.Kind { return s.kinds }
func TestValidateSourceExpectedKindsLegacyKindFallback(t *testing.T) {
sc := config.SourceConfig{Kind: "observation"}
in := testKindSource{
testInput: testInput{name: "test"},
kind: fkevent.Kind("observation"),
}
if err := validateSourceExpectedKinds(sc, in); err != nil {
t.Fatalf("validateSourceExpectedKinds() unexpected error: %v", err)
}
}
func TestValidateSourceExpectedKindsSubsetAllowed(t *testing.T) {
sc := config.SourceConfig{Kinds: []string{"observation"}}
in := testKindsSource{
@@ -56,8 +42,8 @@ func TestValidateSourceExpectedKindsSubsetAllowed(t *testing.T) {
kinds: []fkevent.Kind{"observation", "forecast"},
}
if err := validateSourceExpectedKinds(sc, in); err != nil {
t.Fatalf("validateSourceExpectedKinds() unexpected error: %v", err)
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
t.Fatalf("ValidateExpectedKinds() unexpected error: %v", err)
}
}
@@ -68,12 +54,12 @@ func TestValidateSourceExpectedKindsMismatchFails(t *testing.T) {
kinds: []fkevent.Kind{"observation", "forecast"},
}
err := validateSourceExpectedKinds(sc, in)
err := fksources.ValidateExpectedKinds(sc, in)
if err == nil {
t.Fatalf("validateSourceExpectedKinds() expected mismatch error, got nil")
t.Fatalf("ValidateExpectedKinds() expected mismatch error, got nil")
}
if !strings.Contains(err.Error(), "configured expected kind") {
t.Fatalf("validateSourceExpectedKinds() error %q does not include expected message", err)
t.Fatalf("ValidateExpectedKinds() error %q does not include expected message", err)
}
}
@@ -81,14 +67,8 @@ func TestValidateSourceExpectedKindsNoMetadataSkipsCheck(t *testing.T) {
sc := config.SourceConfig{Kinds: []string{"alert"}}
in := testInput{name: "test"}
if err := validateSourceExpectedKinds(sc, in); err != nil {
t.Fatalf("validateSourceExpectedKinds() unexpected error: %v", err)
}
}
func TestParseExpectedKindsRejectsEmptyValues(t *testing.T) {
if _, err := parseExpectedKinds([]string{""}); err == nil {
t.Fatalf("parseExpectedKinds() expected error for empty kind")
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
t.Fatalf("ValidateExpectedKinds() unexpected error: %v", err)
}
}
@@ -98,6 +78,62 @@ func TestExampleConfigLoads(t *testing.T) {
}
}
func TestExampleConfigSourcesBuildSchedulerJobs(t *testing.T) {
cfg, err := config.Load("config.yml")
if err != nil {
t.Fatalf("config.Load(config.yml) unexpected error: %v", err)
}
assertConfigSourcesBuildSchedulerJobs(t, cfg)
}
func TestMaintainedConfigExamplesLoad(t *testing.T) {
paths, err := filepath.Glob("../../examples/*.yml")
if err != nil {
t.Fatalf("filepath.Glob examples: %v", err)
}
sort.Strings(paths)
if len(paths) == 0 {
t.Fatalf("expected maintained config examples")
}
for _, path := range paths {
t.Run(filepath.Base(path), func(t *testing.T) {
cfg, err := config.Load(path)
if err != nil {
t.Fatalf("config.Load(%s) unexpected error: %v", path, err)
}
assertConfigSourcesBuildSchedulerJobs(t, cfg)
})
}
}
func assertConfigSourcesBuildSchedulerJobs(t *testing.T, cfg *config.Config) {
t.Helper()
reg := fksources.NewRegistry()
wfsources.RegisterBuiltins(reg)
for i, sc := range cfg.Sources {
in, err := reg.BuildInput(sc)
if err != nil {
t.Fatalf("BuildInput(sources[%d]) error = %v", i, err)
}
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
t.Fatalf("ValidateExpectedKinds(sources[%d]) error = %v", i, err)
}
job, err := fkscheduler.JobFromSourceConfig(in, sc)
if err != nil {
t.Fatalf("JobFromSourceConfig(sources[%d]) error = %v", i, err)
}
if job.Source == nil {
t.Fatalf("JobFromSourceConfig(sources[%d]) returned nil source", i)
}
}
}
func TestProcessorRegistryBuildsNormalizeThenDedupeChain(t *testing.T) {
chain, err := buildProcessorChainForTests()
if err != nil {

72
docs/cli.md Normal file
View File

@@ -0,0 +1,72 @@
# CLI Reference
## Shortest Useful Command
Run `weatherfeeder` from a directory containing `config.yml`:
```sh
cd cmd/weatherfeeder
go run .
```
When using a built binary:
```sh
./weatherfeeder
```
## Command Overview
`weatherfeeder` starts a long-running polling daemon. On startup it:
1. reads `config.yml` from the current working directory;
2. builds configured sources, sinks, and routes;
3. starts polling sources on their configured intervals;
4. normalizes and deduplicates events;
5. dispatches matching events to configured sinks.
The command logs startup, runtime, and shutdown messages to stderr using the Go
standard logger.
## Flags
There are currently no CLI flags, subcommands, or environment-variable based
configuration controls.
The config path is fixed at `config.yml` relative to the process current working
directory. To run with a different config, change the working directory or place
the desired file at that path.
## Common Workflows
Run the checked-in sample config:
```sh
cd cmd/weatherfeeder
go run .
```
Maintained copyable configs are available under [`examples/`](../examples/).
Build and run a local binary:
```sh
go build -o weatherfeeder ./cmd/weatherfeeder
cp cmd/weatherfeeder/config.yml .
./weatherfeeder
```
Run in the project container image with a mounted config:
```sh
docker run --rm -v "$PWD/config.yml:/weatherfeeder/config.yml:ro" weatherfeeder
```
The Docker image sets `/weatherfeeder` as the working directory, so the mounted
file must appear at `/weatherfeeder/config.yml`.
## Shutdown
Stop the daemon with `Ctrl-C` or `SIGTERM`. The process uses context-aware
shutdown for scheduler, dispatcher, processors, sources, and sinks, then logs
`shutdown complete`.

273
docs/config.md Normal file
View File

@@ -0,0 +1,273 @@
# Configuration Reference
## Config File
`weatherfeeder` reads exactly one YAML file named `config.yml` from the current
working directory. There is no config path flag and no search path.
YAML decoding is strict for config struct fields: misspelled fields such as
`sources[].drviver` fail startup. Driver-specific `params` maps are validated by
the source or sink constructor that consumes them.
The top-level file contains:
```yaml
sources:
- name: NWSObservationKSTL
mode: poll
driver: nws_observation
every: 10m
kinds: ["observation"]
params:
url: "https://api.weather.gov/stations/KSTL/observations/latest"
user_agent: "Example weatherfeeder operator (ops@example.com)"
sinks:
- name: stdout
driver: stdout
params: {}
routes:
- sink: stdout
kinds: ["observation"]
```
`sources` and `sinks` must each contain at least one entry. `routes` is optional.
When `routes` is omitted, every configured sink receives every event kind.
Maintained copyable configs are available under [`examples/`](../examples/).
## Production-Oriented Shape
A typical deployment uses multiple polling sources and sends the same canonical
event stream to a broker or database:
```yaml
sources:
- name: NWSAlertsLocal
mode: poll
driver: nws_alerts
every: 1m
kinds: ["alert"]
params:
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
user_agent: "Example weatherfeeder operator (ops@example.com)"
- name: SPCConvectiveOutlookLocal
mode: poll
driver: spc_convective_outlook
every: 30m
kinds: ["outlook"]
params:
latitude: 38.6239
longitude: -90.3571
location_id: "local"
location_name: "Configured point"
user_agent: "Example weatherfeeder operator (ops@example.com)"
sinks:
- name: nats_weather
driver: nats
params:
url: nats://nats:4222
subject: weatherfeeder
- name: pg_weather
driver: postgres
params:
uri: postgres://weatherdb:5432/weatherdb?sslmode=disable
username: weatherdb
password: <database_password>
prune: 3d
routes:
- sink: nats_weather
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"]
- sink: pg_weather
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"]
```
Do not commit real API keys, database passwords, or personal contact addresses in
copyable configs.
## Top-Level Fields
| Field | Required | Description |
|---|:---:|---|
| `sources` | yes | List of configured input sources. |
| `sinks` | yes | List of configured output sinks. |
| `routes` | no | List of sink routing rules. If omitted, all sinks receive all kinds. |
## Source Fields
| Field | Required | Description |
|---|:---:|---|
| `name` | yes | Unique source name. Used as the event source identifier. |
| `driver` | yes | Source driver name. |
| `mode` | no | `poll`, `stream`, or omitted for auto. Current weatherfeeder drivers are polling drivers. |
| `every` | yes | Poll interval for current weatherfeeder source drivers. |
| `kinds` | no | Expected event kinds. If present, startup verifies they match the source driver. |
| `params` | driver-specific | Driver parameters. See the source-specific sections below. |
Current event kinds are `observation`, `forecast`, `forecast_discussion`,
`weather_story`, `alert`, and `outlook`.
## Source Drivers
| Driver | Kind | Upstream product |
|---|---|---|
| `nws_observation` | `observation` | NWS station latest observation. |
| `nws_alerts` | `alert` | NWS alerts collection. |
| `nws_forecast_hourly` | `forecast` | NWS hourly gridpoint forecast. |
| `nws_forecast_narrative` | `forecast` | NWS narrative gridpoint forecast. |
| `nws_forecast_discussion` | `forecast_discussion` | NWS forecast discussion HTML product. |
| `nws_weatherstories` | `weather_story` | NWS office weather stories. |
| `openmeteo_observation` | `observation` | Open-Meteo current conditions. |
| `openmeteo_forecast` | `forecast` | Open-Meteo hourly forecast. |
| `openweather_observation` | `observation` | OpenWeather current weather. |
| `spc_convective_outlook` | `outlook` | SPC Day 1-3 convective outlooks. |
## HTTP Source Params
Most source drivers use the shared HTTP polling helper.
| Param | Required | Description |
|---|:---:|---|
| `url` | yes | Full upstream request URL. `URL` is also accepted by the helper. |
| `user_agent` | yes | User-Agent sent to the upstream provider. `userAgent` is also accepted by the helper. |
| `conditional` | no | Boolean. Defaults to `true`; enables ETag and Last-Modified conditional requests. |
| `http_timeout` | no | Positive duration for the HTTP client timeout. |
| `http_response_body_limit_bytes` | no | Positive integer response body limit in bytes. |
When `conditional` is enabled and the upstream returns `304 Not Modified`, the
source emits no events for that poll.
OpenWeather observation URLs must include `units=metric`. Startup fails if the
URL omits it or sets another unit system.
## SPC Convective Outlook Params
`spc_convective_outlook` fetches the twelve required Day 1-3 GeoJSON outlook
products and the three required Day 1-3 print pages as one atomic bundle.
| Param | Required | Description |
|---|:---:|---|
| `latitude` | yes | Location latitude in decimal degrees. |
| `longitude` | yes | Location longitude in decimal degrees. |
| `user_agent` | yes | User-Agent sent to SPC. `userAgent` is also accepted. |
| `location_id` | no | Operator-defined location identifier copied into canonical outlook runs. |
| `location_name` | no | Human location label copied into canonical outlook runs. |
| `geojson_urls` | no | Map of product key to override URL. Used for tests and upstream URL changes. |
| `discussion_urls` | no | Map of day key to override print-page URL. Used for tests and upstream URL changes. |
| `rss_url` | no | Optional RSS URL. RSS is not fetched unless this is configured. |
| `http_timeout` | no | Positive duration for the HTTP client timeout. |
| `http_response_body_limit_bytes` | no | Positive integer response body limit in bytes. |
GeoJSON product keys are `day1_categorical`, `day1_tornado`, `day1_hail`,
`day1_wind`, `day2_categorical`, `day2_tornado`, `day2_hail`, `day2_wind`,
`day3_categorical`, `day3_tornado`, `day3_hail`, and `day3_wind`.
Discussion keys are `day1`, `day2`, and `day3`.
```yaml
sources:
- name: SPCConvectiveOutlookSTL
mode: poll
kinds: ["outlook"]
driver: spc_convective_outlook
every: 30m
params:
latitude: 38.6239
longitude: -90.3571
location_id: "stl"
location_name: "St. Louis, MO"
user_agent: "Example weatherfeeder operator (ops@example.com)"
```
## Sink Fields
| Field | Required | Description |
|---|:---:|---|
| `name` | yes | Unique sink name. Routes refer to this value. |
| `driver` | yes | Sink driver name. |
| `params` | driver-specific | Sink parameters. |
## Sink Drivers
### `stdout`
Prints each event as JSON to stdout.
```yaml
sinks:
- name: stdout
driver: stdout
params: {}
```
### `nats`
Publishes each event as JSON to a NATS subject.
| Param | Required | Description |
|---|:---:|---|
| `url` | yes | NATS server URL, such as `nats://localhost:4222`. |
| `subject` | yes | Subject to publish events to. |
### `postgres`
Writes supported canonical weather events to Postgres using weatherfeeder's
registered schema mapping. The table contract is documented in
[Postgres integration](integrations/postgres.md).
| Param | Required | Description |
|---|:---:|---|
| `uri` | yes | PostgreSQL connection URI. |
| `username` | yes | Database username. |
| `password` | yes | Database password. |
| `prune` | no | Retention window. If set, rows older than the window are pruned on each write transaction. |
`prune` accepts Go duration strings such as `72h`, plus day and week suffixes
such as `3d` and `2w`.
## Routes
Routes connect event kinds to sinks:
```yaml
routes:
- sink: stdout
kinds: ["observation", "alert"]
```
| Field | Required | Description |
|---|:---:|---|
| `sink` | yes | Name of a configured sink. |
| `kinds` | no | Event kinds to send to that sink. Omit or use an empty list to match all kinds. |
Route `kinds` values are trimmed and lowercased by the dispatcher. Blank entries
are rejected.
## Duration Formats
Top-level source `every` accepts:
- Go duration strings such as `30s`, `10m`, or `1h`;
- integer values, interpreted as minutes;
- numeric strings such as `"15"`, also interpreted as minutes.
HTTP param durations such as `http_timeout` accept Go duration strings. Numeric
values and numeric strings are interpreted as seconds.
Postgres `prune` must be a string duration.
## Secrets
The config file is read directly from disk and has no built-in secret expansion.
Keep real credentials out of repository-tracked configs. Use deployment tooling
to render `config.yml` with the needed secret values before starting the daemon.
## Maintained Examples
- [Minimal stdout config](../examples/config.minimal.yml)
- [NATS publishing config](../examples/config.nats.yml)
- [Postgres persistence config](../examples/config.postgres.yml)

289
docs/integrations/events.md Normal file
View File

@@ -0,0 +1,289 @@
# Event Wire Contract
This document is the canonical JSON contract for events emitted by
`weatherfeeder` JSON sinks, including stdout and NATS. Postgres stores the same
event envelope fields in parent table columns; see
[Postgres integration](postgres.md).
Downstream consumers should read the envelope, switch on `schema`, and decode
`payload` according to that schema.
## Envelope
Every emitted event is a JSON object with these fields:
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `id` | string | yes | Stable event identifier. Treat as opaque. |
| `kind` | string | yes | Routing kind, such as `observation` or `alert`. |
| `source` | string | yes | Configured source name. |
| `emitted_at` | timestamp | yes | When the daemon emitted the event. |
| `effective_at` | timestamp | no | Timestamp the payload is about, when known. |
| `schema` | string | no | Schema identifier. Weatherfeeder sources and normalizers set this. |
| `payload` | object, array, string, or scalar | yes | Schema-specific payload. |
Timestamps are JSON strings using Go `time.Time` JSON encoding, which is
RFC3339Nano-compatible. Weatherfeeder normalizers use UTC timestamps for
canonical payloads.
## Kinds And Schemas
Canonical schemas emitted after normalization:
| Kind | Schema | Payload |
|---|---|---|
| `observation` | `weather.observation.v1` | `WeatherObservation` |
| `forecast` | `weather.forecast.v1` | `WeatherForecastRun` |
| `forecast_discussion` | `weather.forecast_discussion.v1` | `WeatherForecastDiscussion` |
| `weather_story` | `weather.weather_story.v1` | `WeatherStoryRun` |
| `alert` | `weather.alert.v1` | `WeatherAlertRun` |
| `outlook` | `weather.outlook.v1` | `WeatherOutlookRun` |
Raw upstream schemas emitted by current sources:
| Kind | Schema | Payload |
|---|---|---|
| `observation` | `raw.nws.observation.v1` | NWS observation JSON |
| `observation` | `raw.openmeteo.current.v1` | Open-Meteo current JSON |
| `observation` | `raw.openweather.current.v1` | OpenWeather current JSON |
| `forecast` | `raw.nws.hourly.forecast.v1` | NWS hourly forecast JSON |
| `forecast` | `raw.nws.narrative.forecast.v1` | NWS narrative forecast JSON |
| `forecast_discussion` | `raw.nws.forecast_discussion.v1` | NWS forecast discussion HTML string |
| `weather_story` | `raw.nws.weatherstories.v1` | NWS weather stories JSON |
| `forecast` | `raw.openmeteo.hourly.forecast.v1` | Open-Meteo hourly forecast JSON |
| `alert` | `raw.nws.alerts.v1` | NWS alerts JSON |
| `outlook` | `raw.spc.convective_outlook.v1` | SPC convective outlook raw bundle |
`standards.SchemaRawOpenWeatherHourlyForecastV1` exists in code, but no current
registered source emits it.
## Shared Conventions
- Canonical numeric measurements use metric units.
- Floating-point values in canonical payloads are rounded to 4 digits after the
decimal point during normalization.
- Optional fields use JSON `omitempty`; absent fields should be treated as
unknown.
- `conditionCode` is a WMO weather interpretation code. Unknown observation
conditions use `-1`. Forecast period `conditionCode` is optional.
- Additive fields are compatible within a schema version. Removing, renaming, or
changing the meaning of a field requires a new schema identifier.
## `weather.observation.v1`
Payload type: `WeatherObservation`.
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `stationId` | string | no | Provider station/location identifier. |
| `stationName` | string | no | Human station name. |
| `timestamp` | timestamp | yes | Observation timestamp. |
| `conditionCode` | integer | yes | WMO code; `-1` means unknown. |
| `isDay` | boolean | no | Day/night hint. |
| `textDescription` | string | no | Short human description. |
| `temperatureC` | number | no | Celsius. |
| `dewpointC` | number | no | Celsius. |
| `windDirectionDegrees` | number | no | Degrees. |
| `windSpeedKmh` | number | no | Kilometers per hour. |
| `windGustKmh` | number | no | Kilometers per hour. |
| `barometricPressurePa` | number | no | Pascals. |
| `visibilityMeters` | number | no | Meters. |
| `relativeHumidityPercent` | number | no | Percent from 0 to 100. |
| `apparentTemperatureC` | number | no | Celsius. |
| `presentWeather` | array | no | Provider-specific present weather fragments. |
`presentWeather[]` entries contain optional `raw` objects.
## `weather.forecast.v1`
Payload type: `WeatherForecastRun`.
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `locationId` | string | no | Provider location identifier. |
| `locationName` | string | no | Human location name. |
| `issuedAt` | timestamp | yes | When the forecast run was generated or issued. |
| `updatedAt` | timestamp | no | Subsequent provider update time. |
| `product` | string | yes | Current emitted values are `hourly` and `narrative`. |
| `latitude` | number | no | Degrees. |
| `longitude` | number | no | Degrees. |
| `elevationMeters` | number | no | Meters. |
| `periods` | array | yes | Ordered forecast periods. |
`periods[]` entries:
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `startTime` | timestamp | yes | Period start. |
| `endTime` | timestamp | yes | Period end. |
| `name` | string | no | Human label. |
| `isDay` | boolean | no | Day/night hint. |
| `conditionCode` | integer | no | WMO code when applicable. |
| `textDescription` | string | no | Human summary. |
| `temperatureC` | number | no | Celsius. |
| `temperatureCMin` | number | no | Celsius. |
| `temperatureCMax` | number | no | Celsius. |
| `dewpointC` | number | no | Celsius. |
| `relativeHumidityPercent` | number | no | Percent from 0 to 100. |
| `windDirectionDegrees` | number | no | Degrees. |
| `windSpeedKmh` | number | no | Kilometers per hour. |
| `windGustKmh` | number | no | Kilometers per hour. |
| `barometricPressurePa` | number | no | Pascals. |
| `visibilityMeters` | number | no | Meters. |
| `apparentTemperatureC` | number | no | Celsius. |
| `cloudCoverPercent` | number | no | Percent from 0 to 100. |
| `probabilityOfPrecipitationPercent` | number | no | Percent from 0 to 100. |
| `precipitationAmountMm` | number | no | Liquid-equivalent millimeters. |
| `snowfallDepthMm` | number | no | Millimeters. |
| `uvIndex` | number | no | Unitless index. |
## `weather.forecast_discussion.v1`
Payload type: `WeatherForecastDiscussion`.
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `officeId` | string | no | NWS office identifier. |
| `officeName` | string | no | Office name. |
| `product` | string | yes | Current value is `afd`. |
| `issuedAt` | timestamp | yes | Bulletin issue time. |
| `updatedAt` | timestamp | no | Subsequent update time. |
| `keyMessages` | array of strings | no | Extracted key messages. |
| `shortTerm` | object | no | Short-term section. |
| `longTerm` | object | no | Long-term section. |
`shortTerm` and `longTerm` sections contain optional `qualifier`, `issuedAt`,
and `text` fields.
## `weather.weather_story.v1`
Payload type: `WeatherStoryRun`.
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `officeId` | string | no | NWS office identifier. |
| `asOf` | timestamp | yes | Snapshot time. |
| `stories` | array | yes | Ordered story cards. |
`stories[]` entries:
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `officeId` | string | no | Office identifier. |
| `startTime` | timestamp | yes | Story start. |
| `endTime` | timestamp | yes | Story end. |
| `updatedAt` | timestamp | yes | Story update time. |
| `title` | string | no | Story title. |
| `description` | string | no | Story description. |
| `altText` | string | no | Image alternate text. |
| `priority` | boolean | yes | Provider priority flag. |
| `order` | integer | yes | Provider display order. |
| `downloadUrl` | string | no | Story image URL. |
## `weather.alert.v1`
Payload type: `WeatherAlertRun`.
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `locationId` | string | no | Provider location identifier. |
| `locationName` | string | no | Human location name. |
| `asOf` | timestamp | yes | Snapshot time. |
| `latitude` | number | no | Degrees. |
| `longitude` | number | no | Degrees. |
| `alerts` | array | yes | Active alerts. |
`alerts[]` entries:
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `id` | string | yes | Provider-stable alert identifier. |
| `event` | string | no | Alert event label. |
| `headline` | string | no | Alert headline. |
| `severity` | string | no | Provider severity. |
| `urgency` | string | no | Provider urgency. |
| `certainty` | string | no | Provider certainty. |
| `status` | string | no | Alert status. |
| `messageType` | string | no | Alert message type. |
| `category` | string | no | Alert category. |
| `response` | string | no | Recommended response. |
| `description` | string | no | Alert description. |
| `instruction` | string | no | Alert instruction. |
| `sent` | timestamp | no | Provider sent time. |
| `effective` | timestamp | no | Effective time. |
| `onset` | timestamp | no | Onset time. |
| `expires` | timestamp | no | Expiration time. |
| `areaDescription` | string | no | Affected area description. |
| `senderName` | string | no | Provider sender name. |
| `references` | array | no | Related alerts. |
`references[]` entries contain optional `id`, `identifier`, `sender`, and
`sent` fields.
## `weather.outlook.v1`
Payload type: `WeatherOutlookRun`.
The current producer is the SPC convective outlook normalizer. It emits Day 1-3
convective outlook polygons for categorical, tornado, hail, and wind products.
All timestamps are UTC.
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `locationId` | string | no | Operator-configured location identifier. |
| `locationName` | string | no | Operator-configured location label. |
| `latitude` | number | no | Configured point latitude in decimal degrees. |
| `longitude` | number | no | Configured point longitude in decimal degrees. |
| `asOf` | timestamp | yes | Snapshot time. For SPC, this is the latest outlook issue time when available. |
| `issuedAt` | timestamp | no | Latest issue time across outlook features when any feature exists. |
| `outlooks` | array | yes | Ordered outlook polygons. |
`outlooks[]` entries:
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `id` | string | yes | Deterministic weatherfeeder outlook identifier. |
| `provider` | string | yes | Current value is `spc`. |
| `product` | string | yes | Current value is `convective`. |
| `day` | integer | yes | SPC outlook day, currently `1`, `2`, or `3`. |
| `outlookType` | string | yes | `categorical`, `tornado`, `hail`, or `wind`. |
| `label` | string | yes | SPC outlook label such as `SLGT` or `15`. |
| `labelText` | string | no | Human label text from SPC, when present. |
| `severityRank` | integer | no | SPC `DN` value, when present. |
| `validFrom` | timestamp | yes | Valid period start. |
| `validTo` | timestamp | yes | Valid period end. |
| `issuedAt` | timestamp | yes | Feature issue time. |
| `expiresAt` | timestamp | yes | Expiration time; currently equal to `validTo`. |
| `forecaster` | string | no | SPC forecaster text, when present. |
| `headline` | string | no | Matching Day 1-3 print-page product title. |
| `summary` | string | no | Text from the print-page `...SUMMARY...` section. |
| `discussion` | string | no | Cleaned full print-page product text. |
| `sourceUrl` | string | no | GeoJSON product URL for this outlook feature. |
| `imageUrl` | string | no | Reserved for provider image URLs; currently empty. |
| `containsLocation` | boolean | yes | Whether the configured point is inside or on the boundary of the polygon. |
| `geometry` | object | yes | Compact GeoJSON `Polygon` or `MultiPolygon` geometry. |
`geometry` preserves the SPC feature geometry as compact GeoJSON using
`[longitude, latitude]` coordinate order. `containsLocation` is computed with
that geometry and the configured source `latitude`/`longitude`; boundary points
count as contained. All outlook polygons are emitted, including polygons that do
not contain the configured point.
## Compact Example
```json
{
"id": "NWSObservationKSTL:2026-06-10T12:00:00Z",
"kind": "observation",
"source": "NWSObservationKSTL",
"emitted_at": "2026-06-10T12:00:05Z",
"effective_at": "2026-06-10T12:00:00Z",
"schema": "weather.observation.v1",
"payload": {
"stationId": "KSTL",
"timestamp": "2026-06-10T12:00:00Z",
"conditionCode": 0,
"temperatureC": 22.5
}
}
```

119
docs/integrations/nws.md Normal file
View File

@@ -0,0 +1,119 @@
# NWS Integration Notes
## Purpose
This document describes the NWS products that `weatherfeeder` currently polls
and normalizes. It is for developers and operators maintaining NWS source URLs,
normalizers, fixtures, and tests.
General config syntax belongs in [configuration](../config.md). Emitted JSON
events are documented in [event wire contract](events.md).
## Implemented Drivers
| Driver | Kind | Raw schema | Canonical schema |
| --- | --- | --- | --- |
| `nws_observation` | `observation` | `raw.nws.observation.v1` | `weather.observation.v1` |
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` | `weather.alert.v1` |
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
## Config Requirements
All NWS drivers require HTTP source params:
- `url`
- `user_agent`
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
an upstream `304 Not Modified` response emits no event for that poll.
NWS expects a descriptive `User-Agent`. Do not use anonymous or placeholder
contact values in production configs.
## Upstream Shapes Used
`nws_observation` expects the latest station observation GeoJSON shape. The
normalizer uses fields under `properties` such as `stationId`, `stationName`,
`timestamp`, `textDescription`, measured values, `presentWeather`, and
`cloudLayers`, plus point geometry for day/night inference.
`nws_alerts` expects an alerts FeatureCollection. The normalizer uses the
collection `updated` timestamp, `title`, each feature ID, alert classification
fields, narrative fields, timing fields, sender fields, and references.
`nws_forecast_hourly` and `nws_forecast_narrative` expect gridpoint forecast
GeoJSON with `properties.generatedAt`, `properties.updateTime`, elevation,
polygon geometry, and ordered `periods`.
`nws_forecast_discussion` expects an HTML page containing the discussion text in
a `<pre>` block. The provider helper extracts office identity, product, issue
time, update time, key messages, and short/long term sections.
`nws_weatherstories` expects a JSON response with a `stories` array. The
normalizer uses office ID, start/end/update times, title, description, alt text,
priority, order, and download URL.
## Accept Headers
NWS JSON sources request:
```text
application/geo+json, application/json
```
The forecast discussion source requests:
```text
text/html, application/xhtml+xml
```
## Effective Time
Source events set `effective_at` from the best metadata available:
- observations: `properties.timestamp`;
- alerts: collection `updated`, otherwise latest per-alert timestamp;
- hourly and narrative forecasts: `properties.generatedAt`, otherwise update
time;
- forecast discussions: parsed issue time;
- weather stories: latest story update time, otherwise latest story start time.
Normalizers use canonical payload time as the normalized event effective time.
Alerts and weather stories fall back to the incoming event envelope when the
payload does not provide a better snapshot time.
## Mapping Notes
Observations preserve raw `presentWeather` fragments and infer WMO condition
codes from METAR phenomena, provider text, and cloud-layer fallback. Sea-level
pressure is preferred over barometric pressure when present.
Hourly forecasts infer WMO condition codes from `shortForecast` and icon tokens.
Narrative forecasts preserve text but intentionally leave period condition codes
unset. Forecast temperatures are converted to Celsius when NWS supplies
Fahrenheit, and wind speed strings are converted to kilometers per hour.
Alert timing fields are parsed best-effort. Invalid per-alert timestamps are
left unset rather than failing the whole alert run. Missing alert IDs are
synthesized from the run snapshot time and array position.
Forecast discussion parsing requires an issue time. Weather story entries require
start time, end time, and update time.
## Failure Behavior
Constructor validation failures stop daemon startup. Polling failures are
returned to the scheduler. JSON sources still emit raw payloads when only
minimal metadata decoding fails. Forecast discussion polling fails if the HTML
cannot be parsed enough to determine the issue time.
## Tests To Inspect
- `internal/sources/nws/*_test.go`
- `internal/normalizers/nws/*_test.go`
- `internal/providers/nws/*_test.go`
- fixtures under `internal/providers/nws/testdata`

View File

@@ -0,0 +1,101 @@
# Open-Meteo Integration Notes
## Purpose
This document describes the Open-Meteo API usage currently implemented by
`weatherfeeder`. It is for developers and operators maintaining Open-Meteo
source URLs, normalizers, fixtures, and tests.
General config syntax belongs in [configuration](../config.md). Emitted JSON
events are documented in [event wire contract](events.md).
## Implemented Drivers
| Driver | Kind | Raw schema | Canonical schema |
| --- | --- | --- | --- |
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` | `weather.observation.v1` |
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` |
## Config Requirements
Both drivers require HTTP source params:
- `url`
- `user_agent`
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
an upstream `304 Not Modified` response emits no event for that poll.
## Upstream Shapes Used
`openmeteo_observation` expects a JSON response with top-level location/timezone
metadata and a `current` object. The normalizer uses:
- `latitude`, `longitude`, `timezone`, `utc_offset_seconds`;
- `current.time`;
- current temperature, apparent temperature, relative humidity, weather code,
wind speed/direction/gusts, pressure, and `is_day`.
`openmeteo_forecast` expects top-level location/timezone metadata and an
array-oriented `hourly` object. The normalizer uses:
- `hourly.time`;
- hourly temperature, apparent temperature, dew point, relative humidity,
precipitation probability, precipitation amount, snowfall, weather code,
pressure, wind speed/direction/gusts, `is_day`, cloud cover, visibility, and
UV index.
Open-Meteo field presence is allowed to vary. Missing optional arrays produce
nil canonical fields for the affected periods.
## Accept Header
Open-Meteo sources request:
```text
application/json
```
## Time Handling
Open-Meteo timestamps often omit an explicit offset. The provider helper parses
times by using the returned `timezone` or `utc_offset_seconds` when needed.
Observation source events set `effective_at` from `current.time` when it can be
parsed. Hourly forecast source events prefer `current.time`, then the first
non-empty `hourly.time` entry.
The hourly forecast normalizer sets canonical `issuedAt` from the incoming event
`emitted_at` when present, otherwise from the first hourly period start.
Normalized forecast `effective_at` matches `issuedAt`.
## Mapping Notes
Open-Meteo is not a station feed. Weatherfeeder synthesizes canonical
station/location IDs from latitude and longitude when both are available.
Open-Meteo weather codes are WMO codes and are treated as authoritative.
Canonical text is derived from the WMO code and day/night hint.
Wind speed and gust fields are treated as kilometers per hour. Pressure values
are treated as hPa and converted to Pa. Snowfall values are treated as
centimeters and converted to millimeters.
Hourly forecast period end time is the next period start. The last period uses
the previous interval length, or one hour when there is no previous interval.
## Failure Behavior
Constructor validation failures stop daemon startup. Polling failures are
returned to the scheduler. Metadata decoding failures in sources still allow raw
payload emission when the HTTP response itself succeeded.
Normalization fails when required time data is missing or invalid, such as an
empty `hourly.time` array for hourly forecasts.
## Tests To Inspect
- `internal/sources/openmeteo/source_test.go`
- `internal/normalizers/openmeteo/*_test.go`
- `internal/providers/openmeteo/*_test.go`

View File

@@ -0,0 +1,101 @@
# OpenWeather Integration Notes
## Purpose
This document describes the OpenWeather current-weather usage implemented by
`weatherfeeder`. It is for developers and operators maintaining OpenWeather
source URLs, normalizers, fixtures, and tests.
General config syntax belongs in [configuration](../config.md). Emitted JSON
events are documented in [event wire contract](events.md).
## Implemented Driver
| Driver | Kind | Raw schema | Canonical schema |
| --- | --- | --- | --- |
| `openweather_observation` | `observation` | `raw.openweather.current.v1` | `weather.observation.v1` |
Only current-weather observation polling is registered for OpenWeather.
## Config Requirements
The driver requires HTTP source params:
- `url`
- `user_agent`
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
an upstream `304 Not Modified` response emits no event for that poll.
The configured URL must include:
```text
units=metric
```
Startup fails if `units` is omitted or set to another value. Keep OpenWeather
API keys out of committed configs. Use local config management or deployment
secrets for the `appid` query parameter.
## Upstream Shape Used
The source emits the full current-weather JSON payload as a raw event. The
normalizer uses:
- `coord.lat`, `coord.lon`;
- primary `weather[0]` condition ID, description, and icon;
- `main.temp`, `main.feels_like`, `main.pressure`, `main.humidity`, and
optional `main.sea_level`;
- `visibility`;
- `wind.speed`, `wind.deg`, and `wind.gust`;
- `dt`;
- `sys.sunrise` and `sys.sunset`;
- `id` and `name`.
## Accept Header
OpenWeather sources request:
```text
application/json
```
## Time Handling
Source events set `effective_at` from `dt` when it is present and positive.
The normalizer also uses `dt` as the canonical observation timestamp and
normalized effective time.
## Mapping Notes
Metric units are required so canonical unit conversion is deterministic:
- `main.temp` and `main.feels_like` are treated as Celsius;
- `wind.speed` and `wind.gust` are treated as meters per second and converted to
kilometers per hour;
- pressure values are treated as hPa and converted to Pa.
The primary condition is `weather[0]`. OpenWeather condition IDs are mapped into
the canonical WMO code vocabulary. The human text description is preserved from
the provider description.
Day/night is inferred from the OpenWeather icon suffix when available, otherwise
from sunrise and sunset bounds.
The station ID uses the OpenWeather city ID when present. If no city ID is
present, weatherfeeder synthesizes an ID from coordinates. The station name uses
the provider `name`, falling back to `OpenWeatherMap` when blank.
## Failure Behavior
Constructor validation failures stop daemon startup. Polling also re-checks the
metric-unit requirement before fetching. HTTP failures are returned to the
scheduler. Metadata decoding failures in the source still allow raw payload
emission when the HTTP response itself succeeded.
## Tests To Inspect
- `internal/sources/openweather/source_test.go`
- `internal/normalizers/openweather/*_test.go`
- `internal/providers/openweather/*_test.go`

View File

@@ -0,0 +1,476 @@
# Postgres Integration
This document is the canonical table contract for the optional `postgres` sink.
It describes the schema created and written by weatherfeeder through feedkit's
Postgres sink.
Configure the sink as described in [configuration](../config.md#postgres).
## Initialization And Writes
At startup, each configured Postgres sink opens the database and runs
`CREATE TABLE IF NOT EXISTS` for every weatherfeeder table, followed by
`CREATE INDEX IF NOT EXISTS` for every configured index.
This initialization creates missing tables and indexes only. It does not alter
existing tables, migrate column definitions, drop old objects, or backfill data.
Schema changes require operator-managed database migration.
Events are mapped only for canonical weather schemas:
- `weather.observation.v1`
- `weather.forecast.v1`
- `weather.forecast_discussion.v1`
- `weather.weather_story.v1`
- `weather.alert.v1`
- `weather.outlook.v1`
Unsupported schemas produce no writes for this sink. Mapped events are inserted
transactionally. Inserts use ordinary `INSERT`; duplicate primary keys fail the
write.
## Shared Envelope Columns
Parent tables store the feed event envelope:
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
## Table Overview
| Table | Primary key | Prune column |
|---|---|---|
| `observations` | `event_id` | `observed_at` |
| `observation_present_weather` | `event_id`, `weather_index` | `observed_at` |
| `forecasts` | `event_id` | `issued_at` |
| `forecast_periods` | `run_event_id`, `period_index` | `issued_at` |
| `forecast_discussions` | `event_id` | `issued_at` |
| `forecast_discussion_key_messages` | `run_event_id`, `message_index` | `issued_at` |
| `weather_story_runs` | `event_id` | `as_of` |
| `weather_stories` | `run_event_id`, `story_index` | `as_of` |
| `alert_runs` | `event_id` | `as_of` |
| `alerts` | `run_event_id`, `alert_index` | `as_of` |
| `alert_references` | `run_event_id`, `alert_index`, `reference_index` | `as_of` |
| `outlook_runs` | `event_id` | `as_of` |
| `outlooks` | `run_event_id`, `outlook_index` | `as_of` |
## Table Contract
### `observations`
Primary key: `event_id`
Prune column: `observed_at`
Indexes:
- `idx_wf_obs_station_observed_at` on `station_id`, `observed_at`
- `idx_wf_obs_observed_at` on `observed_at`
- `idx_wf_obs_condition_code` on `condition_code`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
| `station_id` | `TEXT` | yes | `payload.stationId` |
| `station_name` | `TEXT` | yes | `payload.stationName` |
| `observed_at` | `TIMESTAMPTZ` | no | `payload.timestamp` |
| `condition_code` | `INTEGER` | no | `payload.conditionCode` |
| `is_day` | `BOOLEAN` | yes | `payload.isDay` |
| `text_description` | `TEXT` | yes | `payload.textDescription` |
| `temperature_c` | `DOUBLE PRECISION` | yes | `payload.temperatureC` |
| `dewpoint_c` | `DOUBLE PRECISION` | yes | `payload.dewpointC` |
| `wind_direction_degrees` | `DOUBLE PRECISION` | yes | `payload.windDirectionDegrees` |
| `wind_speed_kmh` | `DOUBLE PRECISION` | yes | `payload.windSpeedKmh` |
| `wind_gust_kmh` | `DOUBLE PRECISION` | yes | `payload.windGustKmh` |
| `barometric_pressure_pa` | `DOUBLE PRECISION` | yes | `payload.barometricPressurePa` |
| `visibility_meters` | `DOUBLE PRECISION` | yes | `payload.visibilityMeters` |
| `relative_humidity_percent` | `DOUBLE PRECISION` | yes | `payload.relativeHumidityPercent` |
| `apparent_temperature_c` | `DOUBLE PRECISION` | yes | `payload.apparentTemperatureC` |
### `observation_present_weather`
Primary key: `event_id`, `weather_index`
Prune column: `observed_at`
Foreign key: `event_id` references `observations(event_id)` with cascade delete.
Index: `idx_wf_obs_present_observed_at` on `observed_at`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT REFERENCES observations(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `weather_index` | `INTEGER` | no | `payload.presentWeather[]` index. |
| `observed_at` | `TIMESTAMPTZ` | no | `payload.timestamp` |
| `raw_text` | `TEXT` | yes | Compact JSON text from `payload.presentWeather[].raw` |
### `forecasts`
Primary key: `event_id`
Prune column: `issued_at`
Indexes:
- `idx_wf_fc_location_product_issued_at` on `location_id`, `product`, `issued_at`
- `idx_wf_fc_issued_at` on `issued_at`
- `idx_wf_fc_product_issued_at` on `product`, `issued_at`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
| `location_id` | `TEXT` | yes | `payload.locationId` |
| `location_name` | `TEXT` | yes | `payload.locationName` |
| `issued_at` | `TIMESTAMPTZ` | no | `payload.issuedAt` |
| `updated_at` | `TIMESTAMPTZ` | yes | `payload.updatedAt` |
| `product` | `TEXT` | no | `payload.product` |
| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` |
| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` |
| `elevation_meters` | `DOUBLE PRECISION` | yes | `payload.elevationMeters` |
| `period_count` | `INTEGER` | no | `len(payload.periods)` |
### `forecast_periods`
Primary key: `run_event_id`, `period_index`
Prune column: `issued_at`
Foreign key: `run_event_id` references `forecasts(event_id)` with cascade delete.
Indexes:
- `idx_wf_fc_period_start_time` on `start_time`
- `idx_wf_fc_period_end_time` on `end_time`
- `idx_wf_fc_period_run_start` on `run_event_id`, `start_time`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `run_event_id` | `TEXT REFERENCES forecasts(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `period_index` | `INTEGER` | no | `payload.periods[]` index. |
| `issued_at` | `TIMESTAMPTZ` | no | Parent `payload.issuedAt` |
| `start_time` | `TIMESTAMPTZ` | no | `payload.periods[].startTime` |
| `end_time` | `TIMESTAMPTZ` | no | `payload.periods[].endTime` |
| `name` | `TEXT` | yes | `payload.periods[].name` |
| `is_day` | `BOOLEAN` | yes | `payload.periods[].isDay` |
| `condition_code` | `INTEGER` | yes | `payload.periods[].conditionCode` |
| `text_description` | `TEXT` | yes | `payload.periods[].textDescription` |
| `temperature_c` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureC` |
| `temperature_c_min` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureCMin` |
| `temperature_c_max` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureCMax` |
| `dewpoint_c` | `DOUBLE PRECISION` | yes | `payload.periods[].dewpointC` |
| `relative_humidity_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].relativeHumidityPercent` |
| `wind_direction_degrees` | `DOUBLE PRECISION` | yes | `payload.periods[].windDirectionDegrees` |
| `wind_speed_kmh` | `DOUBLE PRECISION` | yes | `payload.periods[].windSpeedKmh` |
| `wind_gust_kmh` | `DOUBLE PRECISION` | yes | `payload.periods[].windGustKmh` |
| `barometric_pressure_pa` | `DOUBLE PRECISION` | yes | `payload.periods[].barometricPressurePa` |
| `visibility_meters` | `DOUBLE PRECISION` | yes | `payload.periods[].visibilityMeters` |
| `apparent_temperature_c` | `DOUBLE PRECISION` | yes | `payload.periods[].apparentTemperatureC` |
| `cloud_cover_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].cloudCoverPercent` |
| `probability_of_precipitation_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].probabilityOfPrecipitationPercent` |
| `precipitation_amount_mm` | `DOUBLE PRECISION` | yes | `payload.periods[].precipitationAmountMm` |
| `snowfall_depth_mm` | `DOUBLE PRECISION` | yes | `payload.periods[].snowfallDepthMm` |
| `uv_index` | `DOUBLE PRECISION` | yes | `payload.periods[].uvIndex` |
### `forecast_discussions`
Primary key: `event_id`
Prune column: `issued_at`
Indexes:
- `idx_wf_discussion_office_product_issued_at` on `office_id`, `product`, `issued_at`
- `idx_wf_discussion_issued_at` on `issued_at`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
| `office_id` | `TEXT` | yes | `payload.officeId` |
| `office_name` | `TEXT` | yes | `payload.officeName` |
| `issued_at` | `TIMESTAMPTZ` | no | `payload.issuedAt` |
| `updated_at` | `TIMESTAMPTZ` | yes | `payload.updatedAt` |
| `product` | `TEXT` | no | `payload.product` |
| `short_term_qualifier` | `TEXT` | yes | `payload.shortTerm.qualifier` |
| `short_term_issued_at` | `TIMESTAMPTZ` | yes | `payload.shortTerm.issuedAt` |
| `short_term_text` | `TEXT` | yes | `payload.shortTerm.text` |
| `long_term_qualifier` | `TEXT` | yes | `payload.longTerm.qualifier` |
| `long_term_issued_at` | `TIMESTAMPTZ` | yes | `payload.longTerm.issuedAt` |
| `long_term_text` | `TEXT` | yes | `payload.longTerm.text` |
| `key_message_count` | `INTEGER` | no | `len(payload.keyMessages)` |
### `forecast_discussion_key_messages`
Primary key: `run_event_id`, `message_index`
Prune column: `issued_at`
Foreign key: `run_event_id` references `forecast_discussions(event_id)` with
cascade delete.
Index: `idx_wf_discussion_message_issued_at` on `issued_at`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `run_event_id` | `TEXT REFERENCES forecast_discussions(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `message_index` | `INTEGER` | no | `payload.keyMessages[]` index. |
| `issued_at` | `TIMESTAMPTZ` | no | Parent `payload.issuedAt` |
| `message_text` | `TEXT` | yes | `payload.keyMessages[]` value |
### `weather_story_runs`
Primary key: `event_id`
Prune column: `as_of`
Indexes:
- `idx_wf_story_run_office_as_of` on `office_id`, `as_of`
- `idx_wf_story_run_as_of` on `as_of`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
| `office_id` | `TEXT` | yes | `payload.officeId` |
| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` |
| `story_count` | `INTEGER` | no | `len(payload.stories)` |
### `weather_stories`
Primary key: `run_event_id`, `story_index`
Prune column: `as_of`
Foreign key: `run_event_id` references `weather_story_runs(event_id)` with
cascade delete.
Indexes:
- `idx_wf_stories_start_time` on `start_time`
- `idx_wf_stories_end_time` on `end_time`
- `idx_wf_stories_updated_at` on `updated_at`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `run_event_id` | `TEXT REFERENCES weather_story_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `story_index` | `INTEGER` | no | `payload.stories[]` index. |
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
| `office_id` | `TEXT` | yes | `payload.stories[].officeId` |
| `start_time` | `TIMESTAMPTZ` | no | `payload.stories[].startTime` |
| `end_time` | `TIMESTAMPTZ` | no | `payload.stories[].endTime` |
| `updated_at` | `TIMESTAMPTZ` | no | `payload.stories[].updatedAt` |
| `title` | `TEXT` | yes | `payload.stories[].title` |
| `description` | `TEXT` | yes | `payload.stories[].description` |
| `alt_text` | `TEXT` | yes | `payload.stories[].altText` |
| `priority` | `BOOLEAN` | no | `payload.stories[].priority` |
| `story_order` | `INTEGER` | no | `payload.stories[].order` |
| `download_url` | `TEXT` | yes | `payload.stories[].downloadUrl` |
### `alert_runs`
Primary key: `event_id`
Prune column: `as_of`
Indexes:
- `idx_wf_alert_run_location_as_of` on `location_id`, `as_of`
- `idx_wf_alert_run_as_of` on `as_of`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
| `location_id` | `TEXT` | yes | `payload.locationId` |
| `location_name` | `TEXT` | yes | `payload.locationName` |
| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` |
| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` |
| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` |
| `alert_count` | `INTEGER` | no | `len(payload.alerts)` |
### `alerts`
Primary key: `run_event_id`, `alert_index`
Prune column: `as_of`
Foreign key: `run_event_id` references `alert_runs(event_id)` with cascade
delete.
Indexes:
- `idx_wf_alerts_alert_id` on `alert_id`
- `idx_wf_alerts_severity_expires` on `severity`, `expires`
- `idx_wf_alerts_as_of` on `as_of`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `run_event_id` | `TEXT REFERENCES alert_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `alert_index` | `INTEGER` | no | `payload.alerts[]` index. |
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
| `alert_id` | `TEXT` | no | `payload.alerts[].id` |
| `event` | `TEXT` | yes | `payload.alerts[].event` |
| `headline` | `TEXT` | yes | `payload.alerts[].headline` |
| `severity` | `TEXT` | yes | `payload.alerts[].severity` |
| `urgency` | `TEXT` | yes | `payload.alerts[].urgency` |
| `certainty` | `TEXT` | yes | `payload.alerts[].certainty` |
| `status` | `TEXT` | yes | `payload.alerts[].status` |
| `message_type` | `TEXT` | yes | `payload.alerts[].messageType` |
| `category` | `TEXT` | yes | `payload.alerts[].category` |
| `response` | `TEXT` | yes | `payload.alerts[].response` |
| `description` | `TEXT` | yes | `payload.alerts[].description` |
| `instruction` | `TEXT` | yes | `payload.alerts[].instruction` |
| `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].sent` |
| `effective` | `TIMESTAMPTZ` | yes | `payload.alerts[].effective` |
| `onset` | `TIMESTAMPTZ` | yes | `payload.alerts[].onset` |
| `expires` | `TIMESTAMPTZ` | yes | `payload.alerts[].expires` |
| `area_description` | `TEXT` | yes | `payload.alerts[].areaDescription` |
| `sender_name` | `TEXT` | yes | `payload.alerts[].senderName` |
| `reference_count` | `INTEGER` | no | `len(payload.alerts[].references)` |
### `alert_references`
Primary key: `run_event_id`, `alert_index`, `reference_index`
Prune column: `as_of`
Foreign key: `run_event_id` references `alert_runs(event_id)` with cascade
delete.
Indexes:
- `idx_wf_alert_refs_as_of` on `as_of`
- `idx_wf_alert_refs_sent` on `sent`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `run_event_id` | `TEXT REFERENCES alert_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `alert_index` | `INTEGER` | no | Parent alert index. |
| `reference_index` | `INTEGER` | no | `payload.alerts[].references[]` index. |
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
| `id` | `TEXT` | yes | `payload.alerts[].references[].id` |
| `identifier` | `TEXT` | yes | `payload.alerts[].references[].identifier` |
| `sender` | `TEXT` | yes | `payload.alerts[].references[].sender` |
| `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].references[].sent` |
### `outlook_runs`
Primary key: `event_id`
Prune column: `as_of`
Indexes:
- `idx_wf_outlook_run_location_as_of` on `location_id`, `as_of`
- `idx_wf_outlook_run_as_of` on `as_of`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
| `location_id` | `TEXT` | yes | `payload.locationId` |
| `location_name` | `TEXT` | yes | `payload.locationName` |
| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` |
| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` |
| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` |
| `issued_at` | `TIMESTAMPTZ` | yes | `payload.issuedAt` |
| `outlook_count` | `INTEGER` | no | `len(payload.outlooks)` |
### `outlooks`
Primary key: `run_event_id`, `outlook_index`
Prune column: `as_of`
Foreign key: `run_event_id` references `outlook_runs(event_id)` with cascade
delete.
Indexes:
- `idx_wf_outlooks_contains_valid` on `contains_location`, `valid_from`, `valid_to`
- `idx_wf_outlooks_day_type_label` on `day`, `outlook_type`, `label`
- `idx_wf_outlooks_valid` on `valid_from`, `valid_to`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `run_event_id` | `TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `outlook_index` | `INTEGER` | no | `payload.outlooks[]` index. |
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
| `outlook_id` | `TEXT` | no | `payload.outlooks[].id` |
| `provider` | `TEXT` | no | `payload.outlooks[].provider` |
| `product` | `TEXT` | no | `payload.outlooks[].product` |
| `day` | `INTEGER` | no | `payload.outlooks[].day` |
| `outlook_type` | `TEXT` | no | `payload.outlooks[].outlookType` |
| `label` | `TEXT` | no | `payload.outlooks[].label` |
| `label_text` | `TEXT` | yes | `payload.outlooks[].labelText` |
| `severity_rank` | `INTEGER` | yes | `payload.outlooks[].severityRank` |
| `valid_from` | `TIMESTAMPTZ` | no | `payload.outlooks[].validFrom` |
| `valid_to` | `TIMESTAMPTZ` | no | `payload.outlooks[].validTo` |
| `issued_at` | `TIMESTAMPTZ` | no | `payload.outlooks[].issuedAt` |
| `expires_at` | `TIMESTAMPTZ` | no | `payload.outlooks[].expiresAt` |
| `forecaster` | `TEXT` | yes | `payload.outlooks[].forecaster` |
| `headline` | `TEXT` | yes | `payload.outlooks[].headline` |
| `summary` | `TEXT` | yes | `payload.outlooks[].summary` |
| `discussion` | `TEXT` | yes | `payload.outlooks[].discussion` |
| `source_url` | `TEXT` | yes | `payload.outlooks[].sourceUrl` |
| `image_url` | `TEXT` | yes | `payload.outlooks[].imageUrl` |
| `contains_location` | `BOOLEAN` | no | `payload.outlooks[].containsLocation` |
| `geometry_json` | `TEXT` | no | Compact JSON from `payload.outlooks[].geometry` |
## Retention
When sink param `prune` is set, every successful write transaction deletes rows
older than `now - prune` from every table using that table's prune column.
The sink also exposes manual prune helpers in code, but the `weatherfeeder`
binary does not provide CLI commands for them.
## Reconstructing Canonical Payloads
- `WeatherObservation`: read `observations`, then join
`observation_present_weather` by `event_id` ordered by `weather_index`.
- `WeatherForecastRun`: read `forecasts`, then join `forecast_periods` by
`run_event_id` ordered by `period_index`.
- `WeatherForecastDiscussion`: read `forecast_discussions`, then join
`forecast_discussion_key_messages` by `run_event_id` ordered by
`message_index`.
- `WeatherStoryRun`: read `weather_story_runs`, then join `weather_stories` by
`run_event_id` ordered by `story_index`.
- `WeatherAlertRun`: read `alert_runs`, join `alerts` by `run_event_id` ordered
by `alert_index`, then join `alert_references` by `run_event_id` and
`alert_index` ordered by `reference_index`.
- `WeatherOutlookRun`: read `outlook_runs`, then join `outlooks` by
`run_event_id` ordered by `outlook_index`.

134
docs/integrations/spc.md Normal file
View File

@@ -0,0 +1,134 @@
# SPC Integration Notes
## Purpose
This document describes the Storm Prediction Center convective outlook usage
implemented by `weatherfeeder`. It is for developers and operators maintaining
SPC source configuration, provider helpers, fixtures, and tests.
General config syntax belongs in [configuration](../config.md). Emitted JSON
events are documented in [event wire contract](events.md).
## Implemented Driver
| Driver | Kind | Raw schema | Canonical schema |
| --- | --- | --- | --- |
| `spc_convective_outlook` | `outlook` | `raw.spc.convective_outlook.v1` | `weather.outlook.v1` |
## Config Requirements
The driver requires:
- `latitude`
- `longitude`
- `user_agent`
Optional params are:
- `location_id`
- `location_name`
- `geojson_urls`
- `discussion_urls`
- `rss_url`
- `http_timeout`
- `http_response_body_limit_bytes`
RSS is not fetched unless `rss_url` is configured. URL override maps are intended
for tests and upstream URL changes; the default driver configuration owns the
current Day 1-3 SPC product URLs.
## Upstream Products Used
The source fetches twelve required GeoJSON products every poll:
- Day 1 categorical, tornado, hail, and wind
- Day 2 categorical, tornado, hail, and wind
- Day 3 categorical, tornado, hail, and wind
It also fetches three required print pages:
- Day 1 convective outlook print page
- Day 2 convective outlook print page
- Day 3 convective outlook print page
GeoJSON products are authoritative for outlook polygons, valid windows, issue
times, labels, and severity rank. Print pages are authoritative for headline,
summary, and discussion text.
## Accept Headers
GeoJSON requests use:
```text
application/geo+json, application/json
```
Print-page requests use:
```text
text/html, application/xhtml+xml
```
RSS requests, when configured, use:
```text
application/rss+xml, application/xml, text/xml
```
## Polling And Raw Events
The source polls all required GeoJSON and print-page products as one bundle. If
any required request fails or returns a non-2xx response, the poll returns an
error and emits no partial event.
The raw payload contains fetched bodies plus configured location metadata and
per-product metadata. The source parses only the timestamp metadata needed for
event effective time selection; canonical mapping belongs to the normalizer.
The source emits no event when a complete fetched bundle is unchanged from the
previous successful poll. It does this with a source-local hash of the fetched
document bodies.
## Time Handling
Raw source `effective_at` prefers:
1. the latest valid GeoJSON `ISSUE_ISO`;
2. the latest print-page `Updated:` timestamp;
3. RSS `lastBuildDate` when RSS was fetched and parseable;
4. fetch time.
The normalizer sets canonical `asOf` and normalized event `effective_at` from
the latest valid outlook feature `issuedAt`, with fallback to print-page update
time and then the incoming event envelope.
## Mapping Notes
Each GeoJSON feature becomes one canonical outlook. Products are ordered by day,
then categorical, tornado, hail, and wind. Feature order is preserved within
each product.
The normalizer computes `containsLocation` with the configured latitude and
longitude against compact GeoJSON `Polygon` or `MultiPolygon` geometry.
Coordinates use GeoJSON order, `[longitude, latitude]`, and boundary points
count as contained.
All outlook polygons are preserved, including polygons that do not contain the
configured point. Matching print-page headline, summary, and discussion text is
attached to every outlook for the same day.
## Failure Behavior
Constructor validation failures stop daemon startup. Polling failures are
returned to the scheduler and emit no event for that poll.
Normalization fails when required GeoJSON timestamps, labels, geometry, or
configured coordinates are missing or invalid. Print-page extraction failures
also fail normalization because print pages are required inputs.
## Tests To Inspect
- `internal/providers/spc/*_test.go`
- `internal/sources/spc/*_test.go`
- `internal/normalizers/spc/*_test.go`
- fixtures under `internal/providers/spc/testdata`

View File

@@ -0,0 +1,101 @@
# Normalizer Internals
## Purpose
Normalizers convert raw provider events into canonical weather events. They are
weather-domain mapping code and should stay independent of runtime wiring,
source polling, and sink persistence.
Detailed package conventions live in `internal/normalizers/doc.go`.
## Inputs And Outputs
Inputs are raw feed events whose schemas identify provider payload shape.
Outputs are canonical feed events using `model` payloads and `weather.*`
schemas.
Current mappings:
| Raw schema | Canonical schema |
| --- | --- |
| `raw.nws.observation.v1` | `weather.observation.v1` |
| `raw.openmeteo.current.v1` | `weather.observation.v1` |
| `raw.openweather.current.v1` | `weather.observation.v1` |
| `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
| `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
| `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` |
| `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
| `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
| `raw.nws.alerts.v1` | `weather.alert.v1` |
| `raw.spc.convective_outlook.v1` | `weather.outlook.v1` |
## Boundaries
- Normalizers match by `Event.Schema`.
- Normalizers decode raw payloads into provider structs.
- Normalizers map provider data into canonical `model` payloads.
- Normalizers do not fetch network data, read config, route events, or write
sinks.
- Shared cross-provider behavior belongs in `internal/normalizers/common`.
- Provider-specific helper logic shared with sources belongs in
`internal/providers/<provider>`.
## Config Fields Used
Normalizers do not read config. They operate only on incoming events.
## External Adapters Used
Runtime composition creates feedkit's normalize processor with
`RequireMatch=false`. Events without a matching normalizer pass through
unchanged.
Weatherfeeder registers normalizers in a stable order:
1. NWS
2. Open-Meteo
3. OpenWeather
4. SPC
The current normalizers avoid ambiguous matches by using schema equality.
The SPC outlook normalizer decodes the raw multi-document bundle, maps each
GeoJSON feature to a canonical outlook, and enriches all outlooks for a day with
the matching print-page headline, summary, and discussion. It preserves compact
GeoJSON feature geometry and computes `containsLocation` with
`internal/geo.ContainsPoint` using the source-configured point. Boundary points
count as contained, and all polygons are preserved whether or not they contain
the point.
## State
Normalizers should be stateless. Shared helpers should be deterministic and free
of I/O.
## Failure Behavior
Malformed required raw payload data should produce contextual errors from the
owning normalizer. Successful normalization validates the output event before it
continues through the pipeline.
`internal/normalizers/common.Finalize` preserves the input event envelope except
for schema, payload, and effective time. It also rounds canonical float values
to four digits after the decimal point.
## Tests To Inspect
- `internal/normalizers/builtins_test.go`
- provider normalizer tests under `internal/normalizers/nws`
- provider normalizer tests under `internal/normalizers/openmeteo`
- provider normalizer tests under `internal/normalizers/openweather`
- provider normalizer tests under `internal/normalizers/spc`
- common helper tests under `internal/normalizers/common`
## Invariants
- Match by schema constants from `standards`.
- Preserve the event envelope except for intentional canonical changes.
- Produce canonical payload structs from `model`.
- Validate normalized events before returning them.
- Keep normalizers independent of sources, sinks, config loading, and runtime
composition.

View File

@@ -0,0 +1,124 @@
# Postgres Sink Internals
## Purpose
`internal/sinks/postgres` defines weatherfeeder's canonical-event-to-Postgres
mapping. It supplies a schema definition and mapper to feedkit's generic
Postgres sink.
The consumer-facing table contract is
[`docs/integrations/postgres.md`](../integrations/postgres.md). This document
describes the internal ownership boundary.
## Inputs And Outputs
Inputs are canonical feed events. The mapper currently handles these schemas:
- `weather.observation.v1`
- `weather.forecast.v1`
- `weather.forecast_discussion.v1`
- `weather.weather_story.v1`
- `weather.alert.v1`
- `weather.outlook.v1`
Outputs are feedkit `PostgresWrite` values for weatherfeeder-owned tables.
Unsupported schemas produce no writes and no error.
## Boundaries
- Weatherfeeder owns table definitions in `schema.go`.
- Weatherfeeder owns canonical payload mapping in `map.go`.
- Feedkit owns database opening, table and index creation, transactions,
inserts, context-aware consumption, and prune execution.
- Postgres mapping consumes canonical events only. It should not understand raw
provider schemas.
## Config Fields Used
Weatherfeeder registers the `postgres` sink by passing `PostgresSchema()` to
feedkit. Feedkit parses sink params:
- `uri`
- `username`
- `password`
- `prune`, optional duration
Weatherfeeder-owned mapper code does not read config directly.
## External Adapters Used
The runtime registers the sink with:
```go
sinkReg.Register("postgres", fksinks.PostgresFactory(wfpgsink.PostgresSchema()))
```
Feedkit validates events at the sink boundary, calls the weatherfeeder mapper,
validates writes against the compiled schema, inserts rows in a transaction, and
optionally prunes rows older than the configured window.
## State
The mapper is stateless. Durable state is stored in Postgres through feedkit's
sink implementation.
## Mapping Rules
Parent rows preserve event envelope fields where the table supports them:
- `event_id`
- `event_kind`
- `event_source`
- `event_schema`
- `event_emitted_at`
- `event_effective_at`
Child rows use positional indexes to preserve canonical array order:
- `weather_index`
- `period_index`
- `message_index`
- `story_index`
- `alert_index`
- `reference_index`
- `outlook_index`
Required canonical fields are validated before writes are returned:
- observations require `timestamp`;
- forecasts require `issuedAt` and `product`, and each period requires
`startTime` and `endTime`;
- forecast discussions require `issuedAt` and `product`;
- weather story runs require `asOf`, and each story requires `startTime`,
`endTime`, and `updatedAt`;
- alert runs require `asOf`, and each alert requires `id`;
- outlook runs require `asOf`, and each outlook requires `id`, `provider`,
`product`, `day`, `outlookType`, `label`, `validFrom`, `validTo`, `issuedAt`,
`expiresAt`, and `geometry`.
Nullable canonical values are converted to SQL nulls by mapper helpers.
Observation present-weather raw values and outlook geometry values are stored as
compact JSON text.
## Failure Behavior
Payload decode failures, missing required fields, invalid compact JSON values,
or schema/write mismatches return errors to feedkit's sink. Feedkit rolls back
the transaction when a write fails.
Unsupported canonical schemas are ignored by this mapper so other routed events
can use different sinks without Postgres-specific failures.
## Tests To Inspect
- `internal/sinks/postgres/schema_test.go`
- `internal/sinks/postgres/map_test.go`
- feedkit Postgres sink tests when changing generic sink behavior assumptions
## Invariants
- Persist only canonical schemas.
- Preserve event envelope fields in parent rows.
- Preserve array order with child positional indexes.
- Validate required fields before writing.
- Keep table-contract docs synchronized with schema and mapper changes.

110
docs/internal/runtime.md Normal file
View File

@@ -0,0 +1,110 @@
# Runtime Internals
## Purpose
`cmd/weatherfeeder` wires the daemon together. It owns process setup and runtime
composition; provider mapping, source fetching details, and sink persistence
rules stay in their owning packages.
## Inputs And Outputs
The executable reads `config.yml` from the current working directory through
feedkit config loading. It builds configured sources, scheduler jobs, sinks, and
routes, then runs source polling and sink dispatch until shutdown.
Inputs are configured source polls. Outputs are feed events delivered to the
configured sinks.
## Runtime Flow
The implemented flow is:
1. load `config.yml`;
2. register weatherfeeder source drivers;
3. register feedkit built-in sinks and the weatherfeeder Postgres sink;
4. build source inputs and scheduler jobs;
5. validate configured expected kinds against source-advertised kinds;
6. build sinks and compile routes;
7. run the processor chain `normalize`, then `dedupe`;
8. run the scheduler and dispatcher concurrently;
9. shut down on signal or fatal scheduler/dispatcher error.
The in-process event channel is buffered to 256 events. The dedupe processor is
bounded by `dedupeMaxEntries`, currently 2048.
## Boundaries
- Runtime composition belongs in `cmd/weatherfeeder/main.go`.
- Source driver behavior belongs under `internal/sources`.
- Normalizer behavior belongs under `internal/normalizers`.
- Canonical payloads and schema strings belong in `model` and `standards`.
- Postgres mapping belongs under `internal/sinks/postgres`.
`cmd/weatherfeeder` should stay thin and should not contain provider parsing,
canonical mapping, or table-mapping rules.
## Config Fields Used
Runtime wiring consumes the feedkit top-level config sections:
- `sources`: source driver selection, source name, mode, cadence, expected kinds,
and driver params;
- `sinks`: sink driver selection, sink name, and sink params;
- `routes`: event-kind routing to named sinks.
The executable does not expose CLI flags or config path discovery.
## External Adapters Used
Runtime composition uses feedkit for:
- config loading;
- source registry and expected-kind validation;
- scheduler job construction;
- processor registry and chain execution;
- normalization and dedupe processors;
- sink registry and built-in sinks;
- route compilation and dispatch.
Weatherfeeder registers its own source drivers and its Postgres schema mapper.
## State
Weatherfeeder-owned runtime state is in process:
- event channel contents;
- the bounded dedupe key set;
- source instances and their in-memory unchanged-content state;
- scheduler and dispatcher goroutines.
There is no weatherfeeder-owned durable scheduler state, checkpoint, replay log,
or resume marker. Durable persistence is owned by configured external sinks.
## Failure Behavior
Startup failures are fatal and include context such as config index, source name,
sink name, driver name, or the operation that failed.
At runtime, scheduler and dispatcher errors are sent to a shared error channel.
Context cancellation and deadline errors are treated as normal shutdown. Any
other scheduler or dispatcher error is logged as fatal and cancels the process
context.
The daemon handles `os.Interrupt` and `SIGTERM` with `signal.NotifyContext`.
After both runtime goroutines return, it logs `shutdown complete`.
## Tests To Inspect
- `cmd/weatherfeeder/main_test.go`
- source registry tests under `internal/sources`
- normalizer registration tests under `internal/normalizers`
- feedkit scheduler, processor, dispatch, and sink tests when changing runtime
infrastructure usage
## Invariants
- Keep normalization before dedupe.
- Keep queue sizes and dedupe bounds explicit.
- Preserve context-aware shutdown.
- Keep runtime wiring separate from domain mapping and persistence rules.
- Keep startup validation failures loud and contextual.

122
docs/internal/sources.md Normal file
View File

@@ -0,0 +1,122 @@
# Source Internals
## Purpose
Source packages poll upstream weather providers and emit raw feed events. They
are adapters, not canonical mappers.
Sources should decode only the metadata needed for event identity, effective
time, and routing policy. Full provider payload interpretation belongs in
normalizers.
## Inputs And Outputs
Inputs are feedkit `config.SourceConfig` values and upstream HTTP responses.
Outputs are feed events whose payloads are raw provider JSON and whose schemas
come from `standards`.
Current drivers:
| Driver | Kind | Raw schema |
| --- | --- | --- |
| `nws_observation` | `observation` | `raw.nws.observation.v1` |
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` |
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` |
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` |
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` |
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` |
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` |
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` |
| `openweather_observation` | `observation` | `raw.openweather.current.v1` |
| `spc_convective_outlook` | `outlook` | `raw.spc.convective_outlook.v1` |
## Boundaries
- Source constructors validate source-specific params.
- Sources use feedkit HTTP helpers for HTTP polling.
- Sources emit raw events and should not build canonical `model` payloads.
- Provider helper packages under `internal/providers/<provider>` hold shared
parsing and validation helpers.
- Registration is centralized in `internal/sources/builtins.go`.
## Config Fields Used
Most source drivers use feedkit `HTTPSource`.
Required params:
- `url`
- `user_agent`
Optional params:
- `conditional`, default `true`;
- `http_timeout`;
- `http_response_body_limit_bytes`.
OpenWeather observation sources additionally require the configured URL to use
metric units. This is enforced by `internal/providers/openweather`.
The SPC convective outlook source is a multi-document poll source rather than a
single-URL `HTTPSource`. It requires `latitude`, `longitude`, and `user_agent`;
accepts optional `location_id`, `location_name`, `geojson_urls`,
`discussion_urls`, and `rss_url`; and supports `http_timeout` and
`http_response_body_limit_bytes`.
Source-level `kinds`, when configured, are validated against the source's
advertised `Kinds()`.
## External Adapters Used
Most sources use feedkit's HTTP helper for:
- request construction;
- `User-Agent` and `Accept` headers;
- optional conditional GET validators;
- response body size limits;
- JSON raw-message fetches.
NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times
with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs.
SPC helpers define Day 1-3 product metadata, parse GeoJSON timestamps, extract
cleaned print-page discussion text, and parse optional RSS metadata.
## State
HTTP conditional validators are held in each single-document HTTP source
instance. They are not persisted across process restarts. The SPC outlook source
keeps only a source-local hash of the most recent complete required product
bundle and emits no event when a later complete bundle is unchanged.
## Failure Behavior
Constructor failures are returned during startup and stop the daemon. Polling
failures are returned to the scheduler.
If a source cannot decode minimal metadata from an otherwise successful upstream
response, it still emits the raw event when possible. The event then falls back
to default ID/effective-time behavior from feedkit source helpers.
Unchanged conditional responses return no events and no error.
The SPC outlook source fetches all configured Day 1-3 GeoJSON products and print
pages atomically. If any required GeoJSON or print-page request fails, the poll
returns an error and emits no partial bundle. RSS is fetched only when `rss_url`
is configured.
## Tests To Inspect
- `internal/sources/builtins_test.go`
- provider source tests under `internal/sources/nws`
- provider source tests under `internal/sources/openmeteo`
- provider source tests under `internal/sources/openweather`
- provider source tests under `internal/sources/spc`
- provider helper tests under `internal/providers`
## Invariants
- Emit raw provider schemas from `standards`.
- Keep provider-to-canonical mapping out of sources.
- Keep HTTP behavior context-aware.
- Keep driver registration explicit and stable.
- Keep source tests independent of live upstream services.

163
docs/operations.md Normal file
View File

@@ -0,0 +1,163 @@
# Operations
This document describes how to run and observe the `weatherfeeder` daemon in its
current form. For configuration syntax, see [configuration](config.md). For the
CLI surface, see [CLI reference](cli.md).
## Normal Workflow
1. Prepare `config.yml` in the process working directory.
2. Start the daemon with `./weatherfeeder` or `go run .` from
`cmd/weatherfeeder`.
3. Watch stderr logs for startup or runtime errors.
4. Consume events from the configured sinks.
5. Stop the process with `Ctrl-C` or `SIGTERM`.
The daemon has no admin subcommands and no runtime reload command. Change the
config file and restart the process to apply configuration changes.
Maintained copyable configs are available under [`examples/`](../examples/).
## Runtime Lifecycle
On startup, `weatherfeeder`:
1. loads `config.yml` from the current working directory;
2. registers built-in source drivers;
3. registers stdout, NATS, and weatherfeeder Postgres sink drivers;
4. builds sources and validates configured `kinds` against source metadata;
5. builds sinks and compiles routes;
6. starts the scheduler and dispatcher;
7. processes events through normalization, then in-memory dedupe;
8. routes processed events to configured sinks.
Startup errors are fatal and terminate the process. Runtime poll, pipeline, and
sink errors are logged and the process continues unless the scheduler or
dispatcher returns a fatal error.
## Logs
The process uses the Go standard logger with date, time, and microseconds. Logs
go to stderr.
Common log prefixes:
| Prefix | Meaning |
|---|---|
| `config load failed` | `config.yml` could not be read, parsed, or validated. |
| `build source failed` | A source driver or its params are invalid. |
| `source expected kinds validation failed` | Configured source `kinds` do not match the source driver. |
| `build sink failed` | A sink driver or its params are invalid, or a sink could not initialize. |
| `compile routes failed` | Routes reference invalid sinks or kinds. |
| `scheduler: poll failed` | A source poll failed; the source will be polled again on its next interval. |
| `dispatcher: pipeline error` | Normalization or dedupe failed for one event. |
| `dispatch: sink ... failed consuming event` | A sink failed to consume one event. |
| `shutdown complete` | Scheduler and dispatcher have exited. |
## Scheduling And Polling
Current weatherfeeder sources are polling sources. Each source uses its
configured `every` interval. The scheduler applies jitter before the first poll
and before each interval tick. If no jitter is configured in code, feedkit uses
`min(every/10, 30s)`, capped at half the interval.
Poll failures are logged and do not stop the daemon. A failed poll emits no
events for that source until a subsequent poll succeeds.
## Unchanged Upstream Content
Most sources use feedkit's single-document HTTP polling helper. By default,
`params.conditional` is `true` for those sources, so the helper keeps ETag and
Last-Modified validators in memory for each source instance.
If the upstream returns `304 Not Modified`, the source emits no events for that
poll. Validator state is in memory only; restarting the process starts with no
cached validators.
The SPC convective outlook source polls multiple required documents as one
bundle. It emits no event when a later complete bundle has the same fetched
document bodies as the previous successful poll.
## Processing And Dedupe
Every event passes through normalization first and dedupe second.
Normalizers match raw source schemas and produce canonical `weather.*.v1`
payloads. If an event has no matching normalizer, the normalize processor passes
it through unchanged.
Dedupe keys by event ID and stores a bounded in-memory set of 2048 recent IDs.
Duplicate IDs are dropped. Dedupe state is not persisted, so a restart starts
with an empty dedupe set.
## Routing And Sink Fanout
Routes choose sinks by event kind. If `routes` is omitted, every sink receives
every event kind. If a route omits `kinds`, that route also matches all kinds.
The dispatcher creates one queue and one worker goroutine per sink. The default
per-sink queue size is 64. `weatherfeeder` does not currently expose config
fields for sink queue size, enqueue timeout, or consume timeout.
Sink errors are logged per event. A sink failure does not stop other sinks from
receiving the same event.
## Sink Behavior
### stdout
The stdout sink validates each event and prints one JSON object per line to
stdout. This is useful for local inspection and log forwarding.
### NATS
The NATS sink connects lazily on the first event, reuses the connection while it
is open, and publishes each event as JSON to the configured subject. Connection,
marshal, and publish failures are logged by the dispatch worker.
### Postgres
The Postgres sink opens the database during startup. It creates missing tables
and indexes with `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`.
It does not modify existing table definitions.
Each mapped canonical event is written in one transaction. If `params.prune` is
set, the sink deletes rows older than `now - prune` from every weatherfeeder
table in that same transaction. See the
[Postgres table contract](integrations/postgres.md).
## State And Recovery
`weatherfeeder` keeps only runtime state in process memory:
- scheduler goroutines and timers;
- HTTP conditional request validators;
- source-local unchanged-content state;
- event channel buffers;
- per-sink fanout queues;
- the dedupe ID set.
Durable state is external sink state: NATS broker state outside this process and
Postgres tables managed by the configured database.
There is no internal checkpoint, replay log, or resume marker. To recover from a
process failure, fix the underlying issue and restart the daemon from a working
directory containing the desired `config.yml`.
## Shutdown
`weatherfeeder` listens for `os.Interrupt` and `SIGTERM`. On shutdown, the
shared context is canceled. Scheduler jobs stop polling, dispatch workers stop,
and the process logs `shutdown complete`.
Queued sink work may be dropped when shutdown context cancellation reaches the
fanout workers. Use external sink durability, such as Postgres or broker
retention, for durable downstream state.
## Caveats
- There is no health-check endpoint.
- There is no runtime config reload.
- There are no built-in metrics.
- Source conditional request state and dedupe state are reset by restart.
- Existing Postgres schemas are not migrated automatically.

186
docs/policy/architecture.md Normal file
View File

@@ -0,0 +1,186 @@
# Architecture Policy
## Purpose
This document defines `weatherfeeder`'s development architecture and invariants for maintainers and LLM coding agents. It describes how the implemented system is built and how subsequent changes should preserve its boundaries.
This is an inward-facing policy document. User-facing wire contracts belong in
[`docs/integrations/events.md`](../integrations/events.md), and roadmap items
belong under [`docs/roadmap/`](../roadmap/).
## Project Shape
`weatherfeeder` is a config-driven Go daemon that polls upstream weather providers, emits feed events, normalizes provider-specific raw payloads into canonical weather payloads, and routes those events to configured sinks.
The implemented runtime flow is:
1. `cmd/weatherfeeder` loads `config.yml` from the working directory.
2. Source drivers are built through the source registry.
3. Feedkit scheduler jobs poll sources and publish raw events onto an in-process event channel.
4. A pipeline runs normalization, then in-memory dedupe.
5. The dispatcher routes processed events to configured sinks by event kind.
6. Sinks consume events independently through feedkit fanout workers.
Canonical payload structs live in `model`. Schema identifiers and cross-provider wire conventions live in `standards`. Source adapters live under `internal/sources`. Normalizers live under `internal/normalizers`. Provider-specific parsing helpers shared by sources and normalizers live under `internal/providers`. Sink-specific persistence mapping lives under `internal/sinks`.
## Core Design Principles
- Hexagonal boundaries: provider APIs, config loading, scheduling, dispatch, and sinks are external mechanisms around the weather domain model and normalization logic.
- Raw-to-canonical flow: sources should fetch and envelope raw provider payloads; normalizers should own provider-to-canonical mapping.
- Schema-based routing: normalizers match on event schema, not source name or event kind.
- Composable registries: source drivers, normalizers, processors, and sinks are assembled explicitly through registries.
- Bounded concurrency: scheduling and sink fanout are concurrent, but the application should keep queues, goroutine ownership, logging, and cancellation behavior visible.
- Standard-library-first: use the Go standard library unless a narrow dependency materially improves maintainability or interoperability.
- Current-behavior docs: outside roadmap files, document only implemented behavior.
## Architectural Boundaries
Core/domain logic:
- `model` defines canonical payload structs and JSON field names.
- `standards` defines schema strings, versioning conventions, WMO constants, and shared wire policy.
- Normalizer mapping code is domain logic and should stay independent of CLI setup, filesystem paths, sink details, and runtime orchestration.
Adapters:
- Source adapters under `internal/sources/<provider>` fetch upstream data and produce raw feed events.
- Sink adapters under `internal/sinks/<sink>` map canonical events to external systems.
- Provider helper packages under `internal/providers/<provider>` contain pure parsing or provider-specific helper logic shared by sources and normalizers.
Runtime composition:
- `cmd/weatherfeeder/main.go` owns process wiring: config load, registry setup, scheduler jobs, processor chain, dispatcher, signal cancellation, and logging.
- It should remain thin. Do not move provider mapping or sink persistence rules into `cmd/weatherfeeder`.
Tests and examples:
- The sample `cmd/weatherfeeder/config.yml` is executable test input and is load-tested.
- Tests should keep exercising package contracts directly rather than relying only on full-daemon execution.
## Modules Or Processing Steps
The implemented processing steps are source polling, normalization, dedupe, and sink dispatch.
Source contract:
- Build from `feedkit/config.SourceConfig`.
- Validate required driver params inside the source constructor.
- Advertise emitted event kinds through `Kinds()` when possible.
- Emit raw schemas from `standards`.
- Decode only minimal metadata needed for event identity and effective time; leave full provider decoding to normalizers.
- Respect `context.Context` during network work.
Normalizer contract:
- One normalizer type per normalizer file.
- Match by `Event.Schema`.
- Decode raw payloads into provider structs.
- Map to canonical `model` payloads.
- Preserve the incoming event envelope except for intentional schema, payload, and effective-time changes.
- Use shared helpers in `internal/normalizers/common` for cross-provider behavior.
- Follow the detailed normalizer conventions in `internal/normalizers/doc.go`.
Sink contract:
- Consume canonical schemas, not provider raw schemas.
- Keep sink mapping isolated from normalizers and sources.
- Preserve event envelope fields in durable storage where the sink schema supports it.
- Validate required canonical fields before writing.
## State, Inputs, and Outputs
Inputs are configured source polls. The daemon currently uses feedkit's YAML config model with sources, sinks, and routes.
Outputs are feed events sent to configured sinks. Implemented sink support comes from feedkit built-ins plus weatherfeeder's Postgres schema mapping. The sample config includes stdout and NATS routes; Postgres is configured as an optional commented example.
The daemon's own state is in-process:
- the event channel buffers events during runtime;
- the dedupe processor stores a bounded in-memory key set;
- source instances may keep HTTP conditional request state through feedkit HTTP source helpers;
- scheduler and dispatcher state is not persisted by `weatherfeeder`.
Durable persistence is an external sink concern. The Postgres table contract is
documented in [`docs/integrations/postgres.md`](../integrations/postgres.md);
the consumer-facing event contract is documented in
[`docs/integrations/events.md`](../integrations/events.md).
## Configuration and CLI Boundaries
The implemented executable reads `config.yml` from the current working directory. It does not currently expose CLI flags or config path discovery.
Configuration shape is owned by feedkit's config package:
- `sources` define named source drivers, mode, poll cadence, expected kinds, and driver params.
- `sinks` define named sink drivers and sink params.
- `routes` connect event kinds to sinks.
Weatherfeeder-specific config policy belongs in source and sink constructors, registry setup, and tests. Do not spread config parsing through domain model or normalizer packages.
[`docs/config.md`](../config.md) and [`docs/cli.md`](../cli.md) are the
canonical user/operator references. This policy should stay architectural and
avoid duplicating those references.
## Errors, Logging, and Diagnostics
`cmd/weatherfeeder` uses the standard library `log` package with timestamps and microseconds.
Startup errors are fatal and include config index, source/sink name, driver name, or operation context where available. Runtime scheduler and dispatcher errors are reported through logs; context cancellation and deadline errors are treated as shutdown conditions.
Normalizers and sink mappers should wrap errors with operation and payload context, for example decode, parse, map, scan, or required-field context. Avoid logging or returning whole upstream payloads by default.
The daemon handles `os.Interrupt` and `SIGTERM` through `signal.NotifyContext`. Sources, sinks, scheduler jobs, dispatcher, and processors should respect `context.Context`.
## Testing Expectations
When changing behavior, inspect or add focused tests in the owning package.
Expected coverage by change type:
- Source drivers: constructor behavior, advertised kinds, poll event schema/kind, effective-time policy, unchanged responses, and malformed metadata handling.
- Normalizers: schema matching, canonical schema output, key field mapping, effective time, malformed required fields, and wire-shape regressions.
- Provider helpers: parsing edge cases and fixtures.
- Runtime wiring: config loading, source registry build, scheduler job creation, processor ordering, pass-through behavior, and dedupe behavior.
- Postgres sink: schema shape, mapper writes, required-field validation, nullable handling, and compact JSON behavior.
- Documentation-sensitive examples: keep sample config loadable.
Use local test servers and fixtures rather than real upstream services. Full-package tests should remain fast and deterministic.
## Dependency Policy
Prefer the Go standard library for parsing, HTTP handling, time handling, logging, and tests where reasonable.
Existing broad runtime composition is delegated to `feedkit`, which provides config, sources, scheduler, processors, dispatch, and sinks. Keep weatherfeeder-specific code from depending directly on low-level external clients when feedkit or a small adapter can contain that dependency.
Third-party dependencies should be narrow, justified, and preferably de facto standard for their purpose. YAML parsing through feedkit is an acceptable example. Do not add dependencies for small conveniences, and do not let dependency-specific types leak across package boundaries unless that dependency is the package's explicit contract.
## Documentation Expectations
Documentation must follow [`docs/policy/documentation.md`](documentation.md).
Rules for architecture-related docs:
- Current-behavior docs must describe implemented behavior only.
- Roadmap or speculative work belongs only under `docs/roadmap/`.
- Prefer links to canonical docs over repeated reference material.
- Update docs in the same change when modifying schemas, config behavior, runtime behavior, adapters, or persistence contracts.
## Architectural Invariants
- Keep `cmd/weatherfeeder` as composition code, not domain logic.
- Keep source fetching separate from normalization.
- Keep normalizers matched by schema constants from `standards`.
- Keep canonical payload structs in `model` and treat JSON tags as wire contract.
- Keep provider-specific helpers under `internal/providers/<provider>` when shared by sources and normalizers.
- Keep cross-provider normalizer helpers pure and deterministic.
- Keep sink persistence mapping isolated under `internal/sinks/<sink>`.
- Preserve explicit registry-based extension points for sources and normalizers.
- Preserve context-aware shutdown and bounded in-process queues.
- Avoid broad dependencies without clear architectural value.
## Non-Goals
- `weatherfeeder` is not an HTTP API. API serving belongs to separate consumers such as `weatherapi`.
- `weatherfeeder` does not own long-term durable state except through configured external sinks.
- `weatherfeeder` does not provide a general plugin runtime; new built-in providers and sinks are registered in code.
- Architecture policy is not a CLI, config, or wire-contract reference.

202
docs/policy/development.md Normal file
View File

@@ -0,0 +1,202 @@
# Development Policy
## Purpose
This document describes how to change `weatherfeeder` safely. It is for
maintainers and coding agents working in the repository.
Use this alongside the [architecture policy](architecture.md). User-facing CLI,
configuration, operations, and wire-contract details belong in their canonical
docs, not here.
## Repository Layout
- `cmd/weatherfeeder/`: executable wiring, sample `config.yml`, and runtime
composition tests.
- `model/`: canonical weather payload structs. JSON tags are part of the wire
contract.
- `standards/`: schema strings, versioning conventions, WMO constants, and
shared wire-format policy.
- `internal/sources/`: source adapters that poll upstream providers and emit raw
feed events.
- `internal/normalizers/`: raw-to-canonical event transforms.
- `internal/providers/`: pure provider helper code shared by sources and
normalizers.
- `internal/sinks/postgres/`: weatherfeeder-owned Postgres schema and canonical
event mapper.
- `docs/`: current behavior, policies, integration contracts, and roadmap files.
- `examples/`: maintained, copyable configuration examples.
## Build And Test
Run the full test suite before committing behavior or documentation changes that
depend on code behavior:
```sh
go test ./...
```
Use narrower commands while iterating:
```sh
go test ./cmd/weatherfeeder
go test ./internal/sources/...
go test ./internal/normalizers/...
go test ./internal/sinks/postgres
```
Format Go code before committing:
```sh
gofmt -w <changed-go-files>
```
Do not require live upstream weather services, NATS, or Postgres for unit tests.
Use fixtures, local test servers, and package-level tests.
## Coding Conventions
- Keep `cmd/weatherfeeder` focused on composition: config load, registry setup,
scheduler jobs, processor chain, dispatch, signal handling, and logging.
- Keep source fetching separate from normalizer mapping.
- Match normalizers by schema constants from `standards`, not source names.
- Keep provider-specific helper code under `internal/providers/<provider>` when
both sources and normalizers use it.
- Keep cross-provider normalizer helpers pure and deterministic under
`internal/normalizers/common`.
- Keep sink persistence mapping isolated under `internal/sinks/<sink>`.
- Wrap errors with operation context, but do not include whole upstream payloads
in errors or logs by default.
- Prefer explicit registries and small package-level constructors over hidden
global behavior.
## Dependency Policy
Prefer the Go standard library unless a dependency materially improves
maintainability or interoperability.
`feedkit` owns generic daemon infrastructure for config, HTTP source helpers,
scheduling, processors, dispatch, and sinks. Weatherfeeder code should contain
weather-domain behavior and narrow adapter logic rather than duplicating feedkit
infrastructure.
Do not add broad dependencies for small conveniences. Do not let
dependency-specific types leak across package boundaries unless that dependency
is the package contract.
## Adding Config Fields
Generic config shape is owned by feedkit. Weatherfeeder-specific config behavior
belongs in source or sink constructors, registry setup, and tests.
When adding config behavior:
- validate required params at the adapter boundary;
- keep secrets in environment variables or placeholders, not committed values;
- update [configuration docs](../config.md);
- update maintained examples when the change affects normal operation;
- add or update config-load tests for example files when practical.
## Adding CLI Flags
The executable currently reads `config.yml` from the current working directory.
If CLI flags are added:
- keep parsing in `cmd/weatherfeeder`;
- avoid moving config policy into domain packages;
- update [CLI docs](../cli.md);
- update tests that exercise command behavior.
## Adding A Source Driver
Source drivers should fetch upstream data and emit raw events with minimal
metadata decoding.
Checklist:
- implement the driver under `internal/sources/<provider>`;
- build from `config.SourceConfig`;
- validate required params in the constructor;
- use feedkit HTTP helpers for HTTP polling when applicable;
- emit raw schema constants from `standards`;
- advertise emitted kinds through `Kinds()`;
- decode only metadata needed for event ID and effective time;
- register the driver in `internal/sources/builtins.go`;
- add constructor, kind, and polling tests;
- update config docs and examples when operators need new configuration;
- add provider integration notes when the provider contract needs maintenance
context.
## Adding A Normalizer
Normalizers own provider-to-canonical mapping.
Checklist:
- add one normalizer type per normalizer file;
- match using `Event.Schema`;
- decode raw payloads into provider structs;
- map to canonical `model` payloads;
- use `internal/normalizers/common.Finalize` so envelope handling and float
rounding stay consistent;
- preserve input envelope fields except schema, payload, and effective time;
- register through the provider package and `internal/normalizers/builtins.go`;
- add tests for schema matching, key payload fields, effective time, malformed
required data, and output validation.
## Adding Canonical Models Or Schemas
Canonical event changes affect multiple contracts.
Checklist:
- update payload structs in `model`;
- add or update schema constants in `standards`;
- update [event wire contract docs](../integrations/events.md);
- update normalizers that produce the schema;
- update Postgres mapping if the schema is persisted;
- add tests for wire shape and mapper behavior.
## Adding Postgres Mapping
Weatherfeeder owns the canonical-event-to-table mapping. Feedkit owns the
generic Postgres sink mechanics.
Checklist:
- update `internal/sinks/postgres/schema.go`;
- update `internal/sinks/postgres/map.go`;
- preserve event envelope columns in parent rows when the table supports them;
- validate required canonical fields before writing;
- use positional indexes for child rows that represent arrays;
- update mapper and schema tests;
- update [Postgres integration docs](../integrations/postgres.md) when the table
contract changes.
## Examples And Documentation
Documentation must follow the [documentation policy](documentation.md).
When behavior changes, update the canonical docs in the same change:
- config shape: `docs/config.md`;
- CLI behavior: `docs/cli.md`;
- operations and recovery: `docs/operations.md`;
- troubleshooting: `docs/troubleshooting.md`;
- external contracts: `docs/integrations/`;
- internal component behavior: `docs/internal/`;
- copyable configs: `examples/`.
Keep roadmap content under `docs/roadmap/`. Current-behavior docs must describe
implemented behavior only.
## Review Checklist
Before committing:
- run focused tests for changed packages;
- run `go test ./...` for broad behavior or documentation changes tied to code;
- verify maintained examples still load when examples or config docs changed;
- check links in changed docs;
- search for stale paths, unsupported features, and secret-like values;
- keep unrelated refactors out of the change.

View File

@@ -0,0 +1,356 @@
# Go Project Documentation Policy
## Purpose
Project documentation must help four audiences:
1. users who need to run the application;
2. administrators/operators who need to configure and operate it;
3. developers who need to understand and change it safely;
4. LLM coding agents that need clear scope, boundaries, and invariants.
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
## Core Rules
### 1. Keep docs concise
Each document should cover a defined scope and only the essentials for that scope.
Avoid:
- long background explanations;
- repeated reference material;
- implementation detail in user-facing docs;
- aspirational language outside roadmap docs;
- verbose examples where one minimal example is clearer.
### 2. Document only implemented behavior outside roadmap files
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
- `docs/roadmap/`
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
### 3. Use canonical homes
Each type of information should have one canonical location.
Canonical homes:
- project purpose and quickstart: `README.md`
- development principles: `docs/policy/architecture.md`
- configuration reference: `docs/config.md`
- CLI reference: `docs/cli.md`
- operations and recovery: `docs/operations.md`
- troubleshooting: `docs/troubleshooting.md`
- implemented internals: `docs/internal/`
- future work: `docs/roadmap/`
- contributor workflow: `docs/policy/development.md`
- copyable examples: `examples/`
Other files should summarize briefly and link to the canonical source.
### 4. Keep examples real
Examples should be valid, maintained, and free of secrets.
Where practical:
- example configs should load successfully;
- example commands should match real CLI syntax;
- important examples should be covered by tests.
## Documentation Profiles
All projects require:
- `README.md`
- `docs/policy/architecture.md`
Additional docs depend on the project.
### Small library
Recommended:
- `docs/policy/development.md`, if contributor conventions are non-obvious
### Simple CLI
Required:
- `docs/cli.md`
Recommended:
- `docs/policy/development.md`
### Config-driven CLI
Required:
- `docs/cli.md`
- `docs/config.md`
Recommended:
- `examples/`
- `docs/policy/development.md`
### Stateful or operator-facing application
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
Recommended:
- `docs/troubleshooting.md`
- `examples/`
- `docs/policy/development.md`
### Modular, staged, service-oriented, or orchestration application
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
- `docs/internal/`
- `docs/policy/development.md`
Recommended:
- `docs/troubleshooting.md`
- validated examples under `examples/`
## Required Documents
### README.md
**Audience:** users, administrators, operators
The README is the outward-facing project orientation page.
It should include, in order:
1. concise description;
2. elevator pitch;
3. shortest useful command or usage example;
4. links to targeted docs.
The README should be short. It is not a manual.
The “shortest useful command” means the simplest command that performs the projects core use case. (It does not mean `app --help`.)
### docs/policy/architecture.md
**Audience:** developers, LLM coding agents
`docs/policy/architecture.md` is required for every project.
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
It should include:
- project shape;
- core design principles;
- package and boundary philosophy;
- state/persistence philosophy, if applicable;
- external integration philosophy, if applicable;
- error-handling and logging principles;
- testing expectations;
- documentation expectations;
- architectural invariants;
- explicit non-goals, if useful.
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
### docs/policy/development.md
**Audience:** developers, LLM coding agents
Required for projects maintained by humans and LLM coding agents.
It should include:
- repository layout;
- build/test commands;
- coding conventions;
- dependency policy;
- how to add config fields;
- how to add CLI flags;
- how to add stages/modules/adapters, if applicable;
- how to update examples;
- documentation update expectations.
### docs/config.md
**Audience:** administrators, operators, advanced users
Required for applications with configuration files.
It should include, in order:
1. config file locations and discovery precedence;
2. minimal working config;
3. production-oriented config;
4. full configuration reference;
5. secrets handling, if applicable;
6. links to maintained examples.
The full configuration reference should be canonical.
### docs/cli.md
**Audience:** users, administrators, operators
Required for CLI applications.
It should include, in order:
1. shortest useful command;
2. command overview;
3. complete flag reference;
4. common workflows;
5. diagnostic or recovery commands, if applicable.
Explain when commands are useful, not just their syntax.
### docs/operations.md
**Audience:** administrators, operators
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
It should cover:
- normal workflow;
- filesystem layout;
- remote storage layout, if applicable;
- logs and manifests;
- resume/retry behavior;
- cleanup behavior;
- archive/backup behavior;
- safe recovery procedures;
- operational caveats.
### docs/troubleshooting.md
**Audience:** administrators, operators
Recommended once recurring failure modes exist.
Each entry should include:
- symptom;
- likely cause;
- diagnostic command or inspection step;
- safe fix;
- relevant links.
### docs/internal/
**Audience:** developers, LLM coding agents
Required for modular, staged, service-oriented, or orchestration projects.
This directory describes implemented internal components. It is not the roadmap.
Use one file per major component where useful.
Each component doc should include:
1. purpose;
2. inputs and outputs;
3. boundaries;
4. config fields used;
5. external adapters used;
6. state or manifest behavior, if applicable;
7. skip/resume behavior, if applicable;
8. failure behavior;
9. tests to inspect before changing;
10. architectural invariants.
### docs/roadmap/
**Audience:** maintainers, developers, LLM coding agents
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
Roadmap docs should clearly distinguish:
- proposed work;
- accepted plans;
- deferred ideas;
- rejected ideas;
- implementation prompts or task breakdowns, if useful.
Roadmap docs should not be confused with current behavior.
### docs/integrations/
**Audience:** developers, LLM coding agents
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
Use one file per integration where useful.
## Examples Directory
Projects with non-trivial configuration or workflows should include `examples/`.
Useful examples include:
- minimal working config;
- production-oriented config;
- full annotated config;
- local development config;
- remote/object-storage config;
- minimal session/input file.
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
## Security and Privacy
Docs and examples must not include:
- real API keys;
- tokens;
- passwords;
- private keys;
- private environment dumps;
- sensitive user data;
- raw private transcripts;
- private infrastructure details unless intentionally public.
Document secret-handling mechanisms, not actual secret values.
## Maintenance Rules
When docs change, verify the affected behavior.
Where practical:
- load example config files in tests;
- test CLI examples or command parser behavior;
- validate documented flags against real flags;
- remove stale references;
- update links after renames;
- keep roadmap content out of non-roadmap docs.
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
Documentation is complete only when it matches the current code.
## Documentation Change Checklist
Before merging documentation changes, verify:
- README is concise and orientation-focused.
- `docs/policy/architecture.md` describes development principles.
- Future work appears only under `docs/roadmap/`.
- User-facing docs avoid unnecessary internals.
- Developer-facing docs preserve boundaries and invariants.
- Config examples match the schema.
- CLI examples match real commands and flags.
- Defaults appear in the canonical config reference.
- No secrets or private data are included.
- Links are accurate.

558
docs/roadmap/audit.md Normal file
View File

@@ -0,0 +1,558 @@
# Code Quality And Deduplication Audit
## Executive Summary
`weatherfeeder` is in good shape for a limited cleanup pass before the next major release. The current architecture is coherent: runtime composition is thin, provider fetching lives in source adapters, provider-specific parsing lives under `internal/providers`, canonical mapping lives in normalizers, and Postgres persistence is isolated under `internal/sinks/postgres`.
The codebase does not show a major architectural risk that would require broad redesign. Most duplication is the predictable result of recent feature growth across sources, canonical payloads, and persistence tables. The highest-value cleanup should be narrow and behavior-preserving.
Top refactoring targets:
1. Source adapter HTTP/config scaffolding: single-document sources and the SPC multi-document source share config, request, effective-time, and event-envelope policy in similar but not identical forms.
2. Postgres mapper boilerplate: parent envelope columns, UTC/null conversion, required-field checks, and child-row construction are repeated across every canonical product mapper.
3. Event kind and driver-name strings: event kinds and source driver names are repeated across registries, source implementations, tests, examples, and docs without code constants comparable to the centralized schema constants.
Recommended posture: perform a limited cleanup pass in small commits. Avoid broad framework changes, generic workflow engines, plugin systems, or ORM-like abstractions.
## Repository Map Reviewed
Inspected directories and packages:
- `cmd/weatherfeeder`: runtime composition, sample config, config-load and pipeline tests.
- `internal/sources`: source registry and provider source adapters for NWS, Open-Meteo, OpenWeather, and SPC.
- `internal/providers`: provider-specific parsing helpers for NWS, Open-Meteo, OpenWeather, and SPC.
- `internal/normalizers`: built-in normalizer registration, common helpers, and provider normalizers.
- `internal/sinks/postgres`: weatherfeeder-owned table schema and canonical-event mapper.
- `internal/geo`: point-in-geometry helper used by SPC outlook normalization.
- `model`: canonical payload structs.
- `standards`: schema and WMO constants.
- `examples`: maintained config examples.
- `docs`: policy, config, CLI, operations, troubleshooting, internal docs, integration docs, and `docs/roadmap/future.md`.
Major execution paths reviewed:
- daemon startup from `cmd/weatherfeeder/main.go`;
- source driver registration and source construction;
- source polling for single-document HTTP products and SPC multi-document bundles;
- normalizer registration and raw-schema dispatch;
- raw-to-canonical mapping for observations, forecasts, discussions, weather stories, alerts, and outlooks;
- Postgres schema and write mapping;
- maintained config loading tests.
Important areas not deeply inspected:
- Feedkit internals were not audited because they are a dependency and outside this repository's ownership boundary.
- Live upstream service behavior was not tested; this audit used local source inspection, fixtures, and existing docs.
- Full test execution was not run because this is a report-only task and no code behavior changed.
## High-Confidence Deduplication Opportunities
### 1. Centralize Repeated Source HTTP And Config Scaffolding
Affected files/packages:
- `internal/sources/nws/observation.go`
- `internal/sources/nws/alerts.go`
- `internal/sources/nws/forecast_common.go`
- `internal/sources/nws/forecast_discussion.go`
- `internal/sources/nws/weatherstories.go`
- `internal/sources/openmeteo/observation.go`
- `internal/sources/openmeteo/forecast.go`
- `internal/sources/openweather/observation.go`
- `internal/sources/spc/convective_outlook.go`
- `docs/internal/sources.md`
- `docs/config.md`
Duplicated or near-duplicated behavior:
- Most sources wrap `fksources.NewHTTPSource`, implement `Name`, advertise one `Kinds` value, fetch raw content if changed, compute `effectiveAt`, call `DefaultEventID`, and emit a single event.
- The SPC source cannot use `HTTPSource` directly because it fetches an atomic multi-document bundle, but it repeats the same user-agent, HTTP timeout, body limit, request accept header, and unchanged-content policy at a lower level through `transport.FetchBodyWithLimit`.
- HTTP config params are documented as shared, but source construction has no weatherfeeder-owned helper for the common param names and error shape when a source cannot use `HTTPSource`.
Why it matters:
- Adding future providers or multi-document products increases the chance of drift in `user_agent`, `http_timeout`, `http_response_body_limit_bytes`, body-limit, and error-message behavior.
- The single-document and SPC paths both implement operator-facing HTTP policy, but the shared policy is visible only in docs and feedkit conventions.
- Bug fixes to source envelope construction or shared HTTP params would likely need multiple package edits.
Recommended refactor:
- Add a small source-internal helper package or file, for example `internal/sources/sourceconfig` or `internal/sources/internal/httpconfig`, that owns weatherfeeder-specific HTTP param extraction for sources that cannot directly use `fksources.NewHTTPSource`.
- Keep `fksources.NewHTTPSource` as the implementation for simple sources. Do not replace it with a custom framework.
- Add a helper for common single-event envelope construction only if it remains explicit about `kind`, `source`, `schema`, `eventID`, `emittedAt`, `effectiveAt`, and payload. Avoid hiding provider-specific effective-time selection.
- For SPC, replace local parsing of `user_agent`, `http_timeout`, and `http_response_body_limit_bytes` with the shared helper while preserving its atomic multi-fetch and hash semantics.
Suggested tests:
- Add helper-level tests for param aliases, positive timeout/body-limit validation, missing `user_agent`, and error messages.
- Keep existing source tests for emitted kind/schema/effective time unchanged.
- Add one SPC constructor test that proves shared timeout/body-limit validation still applies.
Risk level: Low to Medium. The behavior is straightforward, but source constructor errors are user-facing and should be protected by tests.
### 2. Reduce Postgres Mapper Boilerplate Without Creating An ORM
Affected files/packages:
- `internal/sinks/postgres/map.go`
- `internal/sinks/postgres/schema.go`
- `internal/sinks/postgres/map_test.go`
- `internal/sinks/postgres/schema_test.go`
- `docs/internal/postgres-sink.md`
- `docs/integrations/postgres.md`
Duplicated or near-duplicated behavior:
- Every parent table mapper repeats the same event envelope columns: `event_id`, `event_kind`, `event_source`, `event_schema`, `event_emitted_at`, and `event_effective_at`.
- Every run mapper follows the same pattern: decode payload, validate required run time/product fields, normalize to UTC, write one parent row, then write child rows with positional indexes.
- Nullable conversion helpers already exist, but each mapper repeats the same map literal shape and required-field error phrasing.
- Schema definitions repeat the same envelope column declarations across parent tables.
Why it matters:
- The mapper is now the largest concentration of cross-product persistence policy. As more canonical products are added, missing an envelope column, count field, UTC conversion, or required-field check becomes easier.
- Recent SPC work showed that canonical fields can be accidentally omitted from persistence even when the model and docs are correct.
- Refactoring this area would reduce maintenance risk and make mapper tests easier to read.
Recommended refactor:
- Add a small `eventEnvelopeValues(e)` helper returning the parent envelope value map, then merge product-specific columns into it.
- Add a small `eventEnvelopeColumns()` helper for schema definitions if feedkit schema construction remains readable.
- Add required-field helper functions for common checks such as `requireTime`, `requireString`, and `requireJSON`, but keep product-specific validation functions where policy differs.
- Keep explicit per-product mapper functions. Do not introduce reflection-based table mapping, struct tags, or a generic ORM layer.
Suggested tests:
- Add focused helper tests for envelope value UTC/null behavior.
- Keep product mapper tests asserting important columns per product.
- Add a regression test that each parent table with event envelope columns receives all envelope values from mapper output.
- Keep schema tests for nullable/required columns, especially recent outlook and forecast condition semantics.
Risk level: Medium. The target is low-level persistence code; refactor only with existing mapper tests passing and add tests before moving column/value construction.
### 3. Centralize Event Kind And Driver Name Constants
Affected files/packages:
- `internal/sources/builtins.go`
- `internal/sources/builtins_test.go`
- `internal/sources/*/*.go`
- `internal/normalizers/*/*_test.go`
- `cmd/weatherfeeder/main_test.go`
- `cmd/weatherfeeder/config.yml`
- `examples/*.yml`
- `docs/config.md`
- `docs/internal/sources.md`
- `docs/integrations/events.md`
- `standards/schema.go`
Duplicated or near-duplicated behavior:
- Schemas are centralized in `standards/schema.go`, but event kinds are repeatedly typed as string literals such as `event.Kind("forecast")`, `event.Kind("weather_story")`, and `event.Kind("outlook")`.
- Driver names are repeated in source constructors, registry entries, tests, config examples, docs, and troubleshooting text.
- The all-current-drivers test duplicates the registry table manually.
Why it matters:
- Kinds and driver names are public operator-facing strings. A typo or stale test value can produce startup failures or documentation drift.
- The mismatch between centralized schemas and non-centralized kinds/drivers makes future feature additions more error-prone.
- The source registry already has a structured `pollDriverRegistrations` slice that can become the canonical source for driver tests.
Recommended refactor:
- Add event kind constants in `standards`, for example `KindObservation`, `KindForecast`, `KindForecastDiscussion`, `KindWeatherStory`, `KindAlert`, and `KindOutlook`, typed as `event.Kind` if dependency direction is acceptable. If `standards` should not import feedkit, use string constants and convert at adapter boundaries.
- Add source driver constants near source registration, for example in `internal/sources/drivers.go`, and have constructors/tests use those constants.
- Update source registry tests to derive the all-current-drivers list from `pollDriverRegistrations`, while keeping explicit negative tests for removed legacy names.
- Keep docs and YAML examples literal; they are user-facing examples and should not be generated for this cleanup pass.
Suggested tests:
- Update source registry tests to assert every registered driver builds as a `PollSource` using the registry slice.
- Add a small test that configured source `Kinds()` match the central kind constants.
- Keep example config load tests as the docs/example guardrail.
Risk level: Low. This is mostly mechanical, but care is needed to avoid import cycles if kind constants are typed with feedkit's `event.Kind`.
### 4. Centralize Config Example Coverage Around All Maintained YAML Files
Affected files/packages:
- `cmd/weatherfeeder/main_test.go`
- `cmd/weatherfeeder/config.yml`
- `examples/config.minimal.yml`
- `examples/config.nats.yml`
- `examples/config.postgres.yml`
- `docs/config.md`
Duplicated or near-duplicated behavior:
- The sample and copyable configs repeat driver names, event kinds, route kind lists, NWS user-agent conventions, source cadences, and sink shapes.
- `main_test.go` already verifies that `cmd/weatherfeeder/config.yml` and `examples/*.yml` load and that sources build scheduler jobs.
- There is no single test that compares example route kind lists against current source-advertised kinds or documented current kinds.
Why it matters:
- Config examples are part of the operator contract. They tend to drift when a new canonical kind is added or renamed.
- Routes are easy to leave stale because a config can load successfully while omitting newly supported kinds from a production-oriented route example.
Recommended refactor:
- Keep examples explicit and copyable.
- Add tests that collect advertised source kinds from configured examples and verify route examples either intentionally match all kinds or document why they are selective.
- Add a small helper in tests for building the weatherfeeder source registry and validating all maintained configs, so config coverage stays obvious.
Suggested tests:
- Extend `TestMaintainedConfigExamplesLoad` to assert source expected kinds and scheduler jobs, which it already does, and add route-kind sanity checks if feedkit exposes compiled routes clearly enough.
- Add a docs/config example guard only if it can be kept simple; avoid parsing Markdown tables unless this repo already uses doc extraction tests.
Risk level: Low. This is test-only cleanup unless route semantics in examples are intentionally selective.
## Medium-Confidence Opportunities
### 1. Normalize Required-Time Helper Patterns Where Semantics Match
Affected files/packages:
- `internal/providers/nws/time.go`
- `internal/providers/openmeteo/time.go`
- `internal/providers/spc/time.go`
- `internal/normalizers/nws/forecast.go`
- `internal/normalizers/nws/weatherstories.go`
- `internal/normalizers/spc/convective_outlook.go`
- `internal/sources/nws/*`
- `internal/sources/spc/convective_outlook.go`
Duplicated or near-duplicated behavior:
- Multiple normalizers implement required timestamp parsing with field-specific error messages.
- Multiple sources parse provider timestamps best-effort for effective-time selection.
- Providers correctly differ in timestamp formats, but callers often repeat trim/empty/UTC/error-context patterns.
Why it matters:
- Time parsing is a domain policy hotspot. Small drift in required vs optional parsing, UTC normalization, or error wording can create subtle behavior differences.
- Required field names in errors are useful and should be preserved.
Recommended refactor:
- Do not force all providers through one cross-provider parser; NWS, Open-Meteo, and SPC formats differ for good reasons.
- Consider small provider-local helpers such as `ParseRequiredTime(value, field)` and `ParseOptionalTime(value)` where a provider already has a canonical parser.
- Use common helper signatures only when the failure behavior is truly identical.
Suggested tests:
- Provider helper tests for empty, malformed, and UTC-normalized timestamps.
- Normalizer tests that assert required timestamp errors include the field path.
Risk level: Medium. The duplication is real, but over-centralization could obscure provider-specific formats.
### 2. Table-Drive Source Registry Tests More Aggressively
Affected files/packages:
- `internal/sources/builtins_test.go`
Duplicated or near-duplicated behavior:
- Several tests independently instantiate a registry and build one named driver.
- `TestRegisterBuiltinsRegistersAllCurrentDrivers` duplicates the same driver list that exists in `pollDriverRegistrations`.
Why it matters:
- Adding new drivers currently requires touching both the registration table and a manually duplicated test list.
- The test suite already has the structure needed to derive cases from the registration table.
Recommended refactor:
- Replace individual positive registration tests with a table derived from `pollDriverRegistrations`.
- Keep one or two named tests only when they assert special policy, such as legacy driver removal.
- Keep `sourceConfigForDriver` but make it keyed off driver constants.
Suggested tests:
- One table-driven positive registration test for every driver.
- One explicit negative test for `nws_forecast` legacy driver.
Risk level: Low. This is test cleanup with minimal behavior risk.
### 3. Package-Local Fixture Helpers Are Duplicated
Affected files/packages:
- `internal/providers/nws/forecast_discussion_test.go`
- `internal/providers/spc/geojson_test.go`
- `internal/sources/nws/forecast_discussion_test.go`
- `internal/sources/spc/convective_outlook_test.go`
- `internal/normalizers/nws/forecast_discussion_test.go`
- `internal/normalizers/spc/convective_outlook_test.go`
Duplicated or near-duplicated behavior:
- Several tests define local helpers that read from `testdata` using `os.ReadFile` and `filepath.Join`.
- Helper names differ by package, but behavior is mostly identical.
Why it matters:
- This is low-risk duplication, but fixture read failures and paths could be made more consistent.
- Cleaner fixture helpers would reduce noise in parser/source/normalizer tests.
Recommended refactor:
- Prefer package-local test helpers, not a cross-package test utility. Go package tests are easier to understand when fixtures remain near the package under test.
- Within each package with multiple test files, consolidate repeated `readTestFile` helpers into one `_test.go` helper file.
Suggested tests:
- No new behavior tests required; this cleanup is test-only.
- Run affected package tests.
Risk level: Low.
### 4. Schema, Model, And Postgres Documentation Lists Require Manual Synchronization
Affected files/packages:
- `standards/schema.go`
- `model/*.go`
- `docs/integrations/events.md`
- `docs/integrations/postgres.md`
- `docs/internal/normalizers.md`
- `docs/internal/sources.md`
- `docs/config.md`
Duplicated or near-duplicated behavior:
- Current schemas, raw mappings, canonical mappings, event kinds, and Postgres table contracts are documented in multiple current-behavior docs.
- This is partly intentional because docs serve different audiences, but all lists must be manually updated when a feature is added.
Why it matters:
- Recent feature additions touched many docs. Manual sync is workable now but will remain a recurring release risk.
- Documentation policy requires current-behavior docs to avoid speculative or stale content.
Recommended refactor:
- Do not generate docs wholesale.
- Add targeted doc consistency tests only for compact, machine-checkable facts, such as ensuring every schema constant appears in `docs/integrations/events.md` and every source driver appears in `docs/config.md`.
- Keep prose manual.
Suggested tests:
- A small standards/docs test that reads selected docs and checks for schema constants and driver constants.
- Keep examples load-tested.
Risk level: Medium. Doc tests can become brittle if they parse prose too deeply; keep them shallow.
## Boundary And Responsibility Concerns
### Source Adapter HTTP Policy Is Split Between Feedkit And Weatherfeeder
Most single-document sources rely on feedkit `HTTPSource`, while SPC implements a custom multi-document fetch loop. This boundary is acceptable because SPC's atomic bundle semantics differ from single-document polling. The concern is not the custom source itself; the concern is that shared weatherfeeder HTTP config policy is partly reimplemented in the SPC adapter.
Recommended home: keep generic HTTP mechanics in feedkit, but add a small weatherfeeder source helper for weatherfeeder-owned parameter names and validation when a source cannot use `HTTPSource` directly.
### Postgres Mapping Is Correctly Isolated But Becoming Too Dense
The Postgres mapper is in the right package and does not leak into domain or normalizer code. The package responsibility is clear. The concern is density and repeated policy, not boundary drift.
Recommended home: keep mapper helpers under `internal/sinks/postgres`. Do not move persistence concerns into `model` or normalizers.
### Event Kind Strings Lack A Canonical Home
Schemas have a clear home in `standards`; event kinds do not. Because kinds are part of routing and operator config, they deserve a comparable canonical code location.
Recommended home: `standards` is the best conceptual location if dependency direction remains clean. If importing feedkit's `event` package into `standards` is undesirable, use string constants in `standards` and convert in source adapters.
### Runtime Composition Is Appropriately Thin
`cmd/weatherfeeder/main.go` is mostly process wiring. It does not contain provider parsing or sink mapping. No refactor is recommended here beyond possibly extracting tiny helper functions if future CLI flags make startup more complex.
## Path, Key, And Naming Construction Review
Centralized enough:
- Schema strings are centralized in `standards/schema.go`.
- SPC product keys, day numbers, outlook types, and default URLs are centralized in `internal/providers/spc/product.go`.
- Postgres table names are centralized as constants in `internal/sinks/postgres/schema.go`.
- Test fixture paths are local and simple.
Needs cleanup:
- Event kind strings are repeated across source adapters, tests, YAML examples, and docs.
- Source driver names are repeated across constructors, registration, tests, docs, and examples.
- Postgres envelope column names are repeated in schema and mapper literals.
- Config route kind lists in examples are manually synchronized with supported canonical kinds.
Recommended approach:
- Add code constants for kinds and drivers first.
- Add narrow Postgres helpers for envelope column/value names second.
- Leave user-facing YAML and Markdown examples explicit, but test them against the code constants where practical.
## Resolution And Catalog Review
Current resolution model:
- Source drivers resolve through `internal/sources.RegisterBuiltins` and feedkit's source registry.
- Normalizers resolve by schema matching through `internal/normalizers.RegisterBuiltins` and feedkit's normalize processor.
- Sinks resolve through feedkit's sink registry, with weatherfeeder registering a Postgres schema mapper.
- Schemas resolve through `standards` constants.
- SPC product catalogs resolve through `internal/providers/spc` product metadata helpers.
Consistency assessment:
- Normalizer resolution is strong: schema equality is explicit and follows policy.
- Source driver resolution is explicit and readable, but test coverage duplicates driver lists rather than deriving from the registry table.
- SPC product resolution is strong and should remain provider-local.
- There is no artifact, prompt, module, profile, manifest, or object-key catalog in this repository.
Recommended centralization:
- Treat source driver constants and event kind constants as small catalogs.
- Avoid building a generic catalog framework; explicit registry tables are appropriate for this codebase.
## Config And Command-Loading Review
Current behavior:
- The executable reads exactly `config.yml` from the current working directory.
- There are no CLI flags, subcommands, profiles, environment-variable config overlays, or config path precedence rules.
- Feedkit owns top-level config loading and validation.
- Weatherfeeder source/sink constructors own driver-specific param validation.
- Maintained examples are load-tested and source-build-tested.
Consistency assessment:
- There is no duplicated command-loading behavior because there is only one command path.
- Driver-specific config validation is mostly consistent, but SPC has to duplicate some HTTP param parsing because it cannot use feedkit's single-document `HTTPSource`.
- OpenWeather's `units=metric` invariant is correctly located in `internal/providers/openweather` and enforced by the source constructor.
Intentional differences:
- SPC does not require `params.url` because it owns a fixed product catalog plus optional override maps.
- OpenWeather has stricter URL validation because unit semantics affect canonical mapping correctness.
- Single-document sources use conditional HTTP validators; SPC uses a bundle hash because it fetches multiple documents atomically.
Likely accidental drift risk:
- HTTP timeout and body-limit validation wording can differ between feedkit-backed HTTP sources and SPC.
- Future multi-document sources may copy SPC's config parsing rather than sharing a narrow helper.
## State, Manifest, Or Progress Handling Review
Current state handling:
- The daemon has no durable internal run state, manifest, checkpoint, or resume marker.
- Feedkit scheduler, dispatcher, sink fanout queues, and dedupe operate in memory.
- Single-document HTTP conditional validators are source-instance memory only.
- SPC unchanged-response behavior uses a source-local hash of the last complete bundle.
- Postgres persistence is external sink state.
Consistency assessment:
- The state model is documented and consistent with the architecture policy.
- There is no hidden filesystem state substituting for declared state.
- There is no resume/force/dry-run behavior to drift across commands.
Cleanup recommendation:
- No state/manifest refactor is needed now.
- If future durable checkpoints are added, design them explicitly rather than expanding the current in-memory dedupe or source-local hash semantics.
## Refactors To Avoid
Avoid these refactors in the next cleanup pass:
- A generic workflow engine or stage abstraction. The current runtime has source polling, normalization, dedupe, and dispatch; adding a stage framework would be speculative.
- A plugin runtime. The policy explicitly favors built-in registries over a general plugin system.
- Replacing feedkit HTTP, scheduler, dispatch, or sink infrastructure with weatherfeeder-owned equivalents.
- A generic Postgres ORM or reflection-driven mapper. The table contract is explicit and should remain readable.
- Cross-provider timestamp parsing that ignores provider-specific timestamp formats.
- Consolidating WMO mapping too aggressively. Provider-specific condition signals differ; only shared text fallback belongs in common helpers.
- Generating all docs from code. Shallow consistency tests are useful; generated manuals would fight the documentation policy's audience-specific structure.
- Collapsing all source adapters into one generic source type. The effective-time and payload policies are similar but still product-specific.
- Moving persistence tags or database column names into `model`. Canonical payloads should not depend on the Postgres sink.
## Recommended Implementation Sequence
1. Add event kind and source driver constants.
Scope: constants plus mechanical usage in source adapters, registry tests, and normalizer tests where appropriate. Keep docs/YAML literal. Run `go test ./internal/sources ./internal/normalizers/... ./cmd/weatherfeeder`.
2. Table-drive source registry tests.
Scope: derive positive source driver cases from `pollDriverRegistrations`; keep legacy-driver negative tests. Run `go test ./internal/sources ./cmd/weatherfeeder`.
3. Add source HTTP config helper for non-`HTTPSource` adapters.
Scope: centralize `user_agent`, `http_timeout`, and `http_response_body_limit_bytes` parsing for SPC and future multi-document sources. Do not alter simple `HTTPSource` adapters. Run `go test ./internal/sources/spc ./internal/sources ./cmd/weatherfeeder`.
4. Add Postgres envelope helper tests, then helper functions.
Scope: add `eventEnvelopeValues`, optionally envelope column helpers, and product-specific required-field helpers. Keep explicit mapper functions. Run `go test ./internal/sinks/postgres`.
5. Consolidate package-local fixture helpers.
Scope: per package only; no cross-package testing utility. Run affected provider/source/normalizer package tests.
6. Add shallow docs consistency tests.
Scope: verify schema constants and source driver constants appear in canonical docs. Avoid parsing Markdown tables deeply. Run `go test ./standards ./cmd/weatherfeeder` or place tests in a suitable package that can read repo docs.
7. Dead-code and legacy sweep.
Scope: after constants/tests are in place, search for obsolete schema/driver/kind literals such as removed legacy driver names. Keep explicit negative tests where they document supported removals.
## Test Strategy
Tests to add before refactoring:
- Source HTTP config helper tests for aliases, missing params, positive duration/body-limit validation, and error context.
- Postgres mapper tests that assert parent envelope columns are consistently present for every mapped canonical parent row.
- Source driver/kind constant tests if constants are introduced.
Tests to update during refactoring:
- `internal/sources/builtins_test.go` for table-driven registry coverage.
- `internal/sources/spc/convective_outlook_test.go` for shared HTTP config validation.
- `internal/sinks/postgres/map_test.go` and `schema_test.go` for envelope helper preservation.
- Existing provider/source/normalizer fixture tests if fixture helpers move.
Focused verification commands:
```sh
go test ./cmd/weatherfeeder ./internal/sources ./internal/sources/... ./internal/providers/... ./internal/normalizers/... ./internal/sinks/postgres ./model ./standards
```
Full verification command before merging cleanup:
```sh
go test ./...
```
## Appendix: Findings Not Worth Acting On
### Provider-Specific Source Files Share A Similar Shape
NWS, Open-Meteo, and OpenWeather source files all implement `Name`, `Kinds`, `Poll`, metadata decode, and event emission. This is acceptable because each product has distinct effective-time and metadata policy. Extract only the clearly shared HTTP/config pieces.
### Normalizer Match Methods Are Repetitive By Design
Most normalizers implement a one-line `Match` against a schema constant. This repetition is good: it keeps routing explicit and cheap. A generic schema-to-builder registry would add indirection without meaningful risk reduction.
### Provider Time Parsers Should Remain Provider-Specific
NWS, Open-Meteo, and SPC timestamp formats differ. The current provider-local parsers are easier to reason about than a broad cross-provider parser. Only required/optional wrapper patterns should be considered for cleanup.
### Documentation Repeats Some Lists Intentionally
`README.md`, `docs/config.md`, `docs/internal/sources.md`, and `docs/integrations/events.md` repeat selected feature lists for different audiences. Do not eliminate that repetition wholesale. Prefer shallow consistency tests for high-risk identifiers.
### SPC Bundle Hash State Should Stay Local
The SPC source's last-bundle hash is source-local unchanged-content state, not a general manifest/checkpoint system. Generalizing it now would be premature.
### Runtime Wiring Could Be Split Into Helpers, But Need Not Be
`cmd/weatherfeeder/main.go` is readable and policy-aligned. Extracting helper functions now would mostly move code around. Revisit only if CLI flags, config path options, metrics, or health checks are added.

281
docs/roadmap/cleanup.md Normal file
View File

@@ -0,0 +1,281 @@
# Cleanup Roadmap
## Summary
This roadmap turns the findings in `docs/roadmap/audit.md` into staged, behavior-preserving cleanup work for `weatherfeeder`.
The goal is to reduce duplication and drift risk before the next major release without changing public contracts. Implement these stages in order. Each stage should be small enough for one focused implementation prompt or commit unless the implementing agent discovers unexpected coupling.
Preserve the current architecture:
- `feedkit` remains generic daemon infrastructure.
- `weatherfeeder` keeps weather-domain policy in sources, providers, normalizers, `model`, `standards`, and Postgres mapping.
- Current-behavior docs should change only when implementation changes require them.
- Roadmap-only cleanup instructions belong here until implemented.
## Global Guardrails
- Do not change public schema strings, event kind strings, source driver names, config keys, table names, column names, column nullability, or canonical model JSON field names.
- Do not include database migrations; this is cleanup-only work.
- Do not change `feedkit` in this cleanup pass.
- Do not introduce broad framework abstractions, plugin systems, workflow engines, generic source frameworks, ORM-style mapping, reflection mapping, or persistence annotations in `model`.
- Keep `cmd/weatherfeeder` focused on runtime composition.
- Keep source fetching separate from normalizer mapping.
- Keep provider-specific timestamp parsing and WMO mapping provider-specific unless an existing common helper already has exactly matching semantics.
- Keep user-facing YAML examples and Markdown examples literal; do not generate docs.
- Run focused tests after each stage and `go test ./...` after all stages.
## Stage 1: Centralize Event Kind And Driver Constants
Add code constants for repeated weatherfeeder identifiers while preserving the literal string values.
Implementation requirements:
- Add event kind string constants in `standards`, for example:
- `KindObservation = "observation"`
- `KindForecast = "forecast"`
- `KindForecastDiscussion = "forecast_discussion"`
- `KindWeatherStory = "weather_story"`
- `KindAlert = "alert"`
- `KindOutlook = "outlook"`
- Keep kind constants typed as plain strings, not `event.Kind`, so `standards` does not import `feedkit`.
- Add source driver string constants in the provider source packages to avoid import cycles:
- NWS driver constants under `internal/sources/nws`.
- Open-Meteo driver constants under `internal/sources/openmeteo`.
- OpenWeather driver constants under `internal/sources/openweather`.
- SPC driver constant under `internal/sources/spc`.
- Update source constructors to use driver constants instead of local string literals.
- Update `Kinds()` methods and `SingleEvent` calls to use `event.Kind(standards.Kind...)`.
- Update source registration to use provider driver constants.
- Update internal tests to use constants where doing so reduces drift.
- Keep docs and YAML examples literal because they are user-facing examples.
- Do not add weather-specific kind constants to `feedkit`.
Acceptance criteria:
- All driver and event kind string values remain unchanged.
- No import cycle is introduced.
- `standards` does not import `feedkit`.
- Source constructors, emitted events, and advertised kinds behave identically.
Focused tests:
```sh
go test ./internal/sources ./cmd/weatherfeeder
go test ./internal/normalizers/...
```
## Stage 2: Table-Drive Source Registry Tests
Reduce source registry test duplication while preserving registry coverage.
Implementation requirements:
- Replace repeated positive tests in `internal/sources/builtins_test.go` with one table-driven test derived from `pollDriverRegistrations`.
- Keep the explicit negative test proving legacy `nws_forecast` is not registered.
- Keep `sourceConfigForDriver`, but update it to use driver constants or registry-derived driver names.
- Preserve coverage that every current registered driver builds as a `PollSource`.
- Do not weaken `ValidateExpectedKinds` or scheduler job build coverage in `cmd/weatherfeeder` tests.
Acceptance criteria:
- Adding a new driver to `pollDriverRegistrations` automatically includes it in the positive registry test.
- Legacy-driver rejection remains explicitly tested.
- Test behavior remains deterministic and independent of live upstream services.
Focused tests:
```sh
go test ./internal/sources ./cmd/weatherfeeder
```
## Stage 3: Add Shared HTTP Config Helper For Multi-Document Sources
Centralize common HTTP config parsing for sources that cannot use feedkit's single-document `HTTPSource`.
Implementation requirements:
- Add a narrow helper under `internal/sources/internal/httpconfig`.
- The helper should parse only the common source HTTP client params needed by non-`HTTPSource` sources:
- trimmed source name;
- required `params.user_agent` / `params.userAgent`;
- optional `params.http_timeout` using feedkit config duration semantics;
- optional `params.http_response_body_limit_bytes` as a positive integer.
- The helper should return values sufficient for callers to build a `transport.NewHTTPClient(timeout)` and pass a body limit to `transport.FetchBodyWithLimit`.
- Refactor only the SPC source to use this helper.
- Preserve SPC-specific config parsing in the SPC source:
- `latitude`;
- `longitude`;
- `location_id` / `locationID`;
- `location_name` / `locationName`;
- `geojson_urls`;
- `discussion_urls`;
- `rss_url` / `rssURL`.
- Do not replace `fksources.NewHTTPSource` for normal single-URL sources.
- Preserve SPC atomic bundle fetch behavior, bundle hash behavior, effective-time policy, raw payload shape, and error context.
Acceptance criteria:
- SPC constructor accepts and rejects the same configs as before.
- SPC still fetches required documents atomically and emits no partial bundle.
- Existing SPC source tests pass unchanged except for expected helper-related error wording if the wording becomes more consistent.
- No single-document source is refactored away from `fksources.NewHTTPSource`.
Focused tests:
```sh
go test ./internal/sources/spc ./internal/sources
```
## Stage 4: Reduce Postgres Mapper Envelope Duplication
Centralize repeated Postgres parent envelope mapping without changing the table contract.
Implementation requirements:
- Add small helper functions inside `internal/sinks/postgres`.
- Centralize parent event envelope values:
- `event_id`;
- `event_kind`;
- `event_source`;
- `event_schema`;
- `event_emitted_at`;
- `event_effective_at`.
- Use the helper in every parent table mapper that stores event envelope columns.
- Optionally centralize event envelope column declarations if it keeps `schema.go` readable. If it makes the schema definition harder to scan, leave column declarations explicit.
- Keep explicit per-product mapper functions.
- Keep product-specific validation in product-specific functions where policy differs.
- Do not introduce reflection mapping, struct tags, ORM-style abstractions, generated table mapping, or persistence annotations in `model`.
- Preserve every existing table, column, nullability rule, required-field check, compact JSON behavior, UTC normalization, child positional index, and write count.
Acceptance criteria:
- Mapper output for existing valid payloads is equivalent before and after the refactor.
- Unsupported schemas still map to zero writes and no error.
- Required-field failures still include useful product/path context.
- Feedkit Postgres schema validation still receives complete rows for every declared column.
Focused tests:
```sh
go test ./internal/sinks/postgres
```
## Stage 5: Tighten Config Example And Documentation Consistency Tests
Add shallow tests that detect identifier drift without generating or over-parsing documentation.
Implementation requirements:
- Extend maintained config tests so `cmd/weatherfeeder/config.yml` and every `examples/*.yml` remain loadable and source-buildable.
- Add shallow docs consistency tests for stable identifiers only:
- every schema constant in `standards` appears in `docs/integrations/events.md` when it is part of the current event contract;
- every registered source driver name appears in `docs/config.md`;
- every registered source driver name appears in `docs/internal/sources.md`.
- Avoid parsing Markdown tables deeply; simple file-content checks are sufficient.
- Do not generate docs.
- Do not modify current-behavior docs unless the implementation uncovers an actual stale documented identifier.
- Keep documentation policy intact: implemented behavior outside `docs/roadmap/`, future plans under `docs/roadmap/`.
Acceptance criteria:
- Identifier consistency tests fail when a new source driver or current schema is added without updating canonical docs.
- Tests are shallow and low maintenance.
- Tests do not assert prose formatting or table layout.
Focused tests:
```sh
go test ./cmd/weatherfeeder ./standards ./internal/sources
```
## Stage 6: Consolidate Package-Local Test Fixture Helpers
Remove low-value duplicated fixture-reading code only where it is local and obvious.
Implementation requirements:
- Consolidate duplicated fixture readers only within the same Go package.
- Do not create a cross-package test utility package.
- Keep fixtures under each package's `testdata` directory.
- If a package has only one fixture helper, leave it alone.
- Do not change fixture contents unless a test already requires it.
- Do not mix this stage with parser behavior changes.
Acceptance criteria:
- Test helper duplication is reduced where multiple files in one package share the same fixture-reading behavior.
- Tests remain easy to read locally.
- No package imports a helper solely for tests from another package.
Focused tests:
```sh
go test ./internal/providers/nws ./internal/providers/spc
go test ./internal/sources/nws ./internal/sources/spc
go test ./internal/normalizers/nws ./internal/normalizers/spc
```
## Stage 7: Dead-Code And Literal Sweep
Perform a final cleanup sweep after constants and helper stages are complete.
Implementation requirements:
- Search for stale driver, kind, and schema literals after earlier stages.
- Replace internal code/test literals with constants where it reduces typo or drift risk.
- Keep user-facing docs and YAML examples literal.
- Keep intentional legacy-driver negative tests.
- Do not remove compatibility tests unless they are clearly obsolete and no longer document supported behavior.
- Do not broaden the cleanup into unrelated refactors.
Suggested searches:
```sh
rg 'event\.Kind\("|nws_forecast|nws_weatherstories|openmeteo_|openweather_|spc_convective_outlook|weather_story|forecast_discussion|raw\.|weather\.' .
rg 'TODO|legacy|deprecated|unknown source driver' internal cmd docs examples
```
Acceptance criteria:
- Internal literals are reduced where constants now exist.
- Intentional literals in docs, YAML examples, raw schema docs, and negative tests remain readable.
- No behavior changes are introduced.
Focused tests:
```sh
go test ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./cmd/weatherfeeder
```
## Final Verification
After all stages are complete, run:
```sh
go test ./...
git status --short
```
Before committing the implemented cleanup, verify:
- No public contracts changed unintentionally.
- Current-behavior docs still describe implemented behavior only.
- `docs/roadmap/cleanup.md` is either updated to remove completed work or moved to future/remediation tracking according to the repository's roadmap practice.
- No unrelated changes are included.
## Refactors To Avoid
Do not perform these changes as part of this cleanup roadmap:
- Generic workflow or stage engine.
- Runtime plugin system.
- Weather-specific constants in `feedkit`.
- Replacing feedkit scheduler, dispatch, HTTP helpers, or sink mechanics.
- Generic source abstraction covering every source type.
- Reflection-based or generated Postgres mapper.
- ORM-style persistence layer.
- Database column metadata on canonical model structs.
- Cross-provider timestamp parser that hides provider-specific formats.
- Broad WMO mapper consolidation beyond existing common text fallback.
- Generated documentation system.

60
docs/roadmap/future.md Normal file
View File

@@ -0,0 +1,60 @@
# Future Work
## Purpose
This document is the catch-all roadmap for planned, deferred, aspirational, experimental, or unimplemented weatherfeeder work. Current behavior belongs in the canonical docs outside `docs/roadmap/`.
## SPC Convective Outlook Follow-Ups
### Weatherapi Outlook Endpoints
Expose persisted SPC convective outlooks through `weatherapi` after the weatherfeeder storage contract is stable.
Likely endpoints:
- `GET /outlooks/convective`
- `GET /outlooks/convective/active`
- `GET /outlooks/convective/location`
Recommended behavior:
- Return the latest outlook run by default.
- Support active outlook filtering by current time and `containsLocation=true`.
- Consider optional query filters for `day`, `outlookType`, and `containsLocation`.
- Preserve canonical outlook geometry for downstream display and audit use.
### SPC Day 4-8 Outlooks
Add SPC Day 4-8 convective outlook support as a schema-compatible extension only after Day 1-3 operation is proven.
Notes:
- Day 4-8 products have different semantics from Day 1-3 categorical/tornado/hail/wind products.
- Avoid forcing Day 4-8 assumptions into the current Day 1-3 model until the source shapes and consumer needs are reviewed.
- Prefer reusing `weather.outlook.v1` if the fields remain accurate; otherwise write a separate roadmap before changing the canonical contract.
### Degraded SPC Bundle Mode
Evaluate whether the SPC source should support degraded partial bundles when one required upstream product fails.
Current behavior should remain atomic: if a required GeoJSON or print-page fetch fails, emit no event for that poll.
Future degraded mode would need a clear contract for:
- distinguishing "no risk polygon" from "product missing";
- exposing per-product fetch errors without leaking raw provider internals into canonical events;
- deciding whether downstream sinks and APIs should store or serve partial snapshots.
### Richer SPC Page Assets And Tables
Evaluate whether to parse additional SPC print-page metadata beyond the current discussion text.
Possible additions:
- archive GeoJSON/shapefile/KML links;
- image URLs;
- page risk tables;
- city tables;
- richer discussion section metadata.
Keep GeoJSON products authoritative for polygons, validity windows, and point matching unless a future roadmap explicitly changes that contract.

251
docs/troubleshooting.md Normal file
View File

@@ -0,0 +1,251 @@
# Troubleshooting
Use this guide with [configuration](config.md) and [operations](operations.md).
Messages are emitted through the standard logger on stderr.
## `config load failed: ... read "config.yml"`
Symptom: startup exits before building sources or sinks.
Likely cause: the process working directory does not contain `config.yml`, or
the runtime user cannot read it.
Diagnostic: run `pwd` in the same working directory used by the process, then
check `ls -l config.yml`.
Safe fix: place the intended config at `./config.yml`, change the working
directory, or mount the file at `/weatherfeeder/config.yml` when using the
provided container image.
## `config load failed: ... parse YAML`
Symptom: startup exits with a YAML parse error or an unknown field error.
Likely cause: invalid YAML syntax, multiple YAML documents, or a misspelled
config struct field.
Diagnostic: inspect the line and field in the error. Feedkit uses strict YAML
field decoding for config struct fields.
Safe fix: correct the YAML and compare the shape with
[configuration](config.md). Driver-specific `params` keys are validated by their
source or sink constructors.
## `config validation failed`
Symptom: startup exits and prints one or more validation messages.
Likely cause: missing `sources` or `sinks`, blank names, duplicate source or sink
names, invalid `mode`, missing `every` for a polling source with `mode: poll`, or
a route that references an unknown sink.
Diagnostic: read every bullet under `config validation failed`; the loader sorts
these messages so multiple issues can be fixed in one edit.
Safe fix: update the top-level config fields as documented in
[configuration](config.md).
## `unknown source driver`
Symptom: startup exits with `build source failed`.
Likely cause: `sources[].driver` does not match a registered weatherfeeder
source driver.
Diagnostic: compare the configured driver with the source driver table in
[configuration](config.md#source-drivers).
Safe fix: correct the driver name. Current drivers include `nws_observation`,
`nws_alerts`, `nws_forecast_hourly`, `nws_forecast_narrative`,
`nws_forecast_discussion`, `nws_weatherstories`, `openmeteo_observation`,
`openmeteo_forecast`, `openweather_observation`, and
`spc_convective_outlook`.
## `unknown sink driver`
Symptom: startup exits with `build sink failed`.
Likely cause: `sinks[].driver` is not registered.
Diagnostic: compare the configured driver with the sink driver table in
[configuration](config.md#sink-drivers).
Safe fix: use `stdout`, `nats`, or `postgres`.
## `source expected kinds validation failed`
Symptom: startup exits after building a source.
Likely cause: `sources[].kinds` declares a kind the source does not emit.
Diagnostic: compare the configured `kinds` list with the source driver kind in
[configuration](config.md#source-drivers).
Safe fix: remove `kinds` or set it to the kind emitted by that driver.
## `params.url is required` Or `params.user_agent is required`
Symptom: startup exits with `build source failed`.
Likely cause: a source is missing required HTTP params, or the values are blank
or not strings.
Diagnostic: inspect the named source in the error and check its `params`.
Safe fix: add non-empty `url` and `user_agent` values. See
[HTTP source params](config.md#http-source-params).
SPC convective outlook sources do not use `params.url`; they require
`latitude`, `longitude`, and `user_agent`. See
[SPC convective outlook params](config.md#spc-convective-outlook-params).
## `params.latitude is required` Or `params.longitude is required`
Symptom: startup exits for an `spc_convective_outlook` source.
Likely cause: the source is missing one of the configured point coordinates.
Diagnostic: inspect the named SPC source in the error and check its `params`.
Safe fix: add numeric `latitude` and `longitude` values in decimal degrees.
## `url must include units=metric`
Symptom: startup exits for an `openweather_observation` source.
Likely cause: the OpenWeather URL omits `units=metric` or sets another unit
system.
Diagnostic: inspect the query string in `params.url`.
Safe fix: add `units=metric` to the OpenWeather current-weather URL. Keep API
keys out of committed configs.
## `source ... sources[].every must be > 0 for polling sources`
Symptom: startup exits while building scheduler jobs.
Likely cause: a current weatherfeeder polling source has no usable `every`
interval.
Diagnostic: inspect the named `sources[]` entry and check `every`.
Safe fix: set a positive duration such as `1m`, `10m`, or `1h`.
## `build sink failed ... params.url is required`
Symptom: startup exits while building a NATS sink.
Likely cause: the NATS sink is missing `params.url`, or the value is blank or
not a string.
Diagnostic: inspect the named sink in the error and check its `params`.
Safe fix: set a NATS URL such as `nats://localhost:4222`.
## `build sink failed ... params.subject is required`
Symptom: startup exits while building a NATS sink.
Likely cause: the NATS sink is missing `params.subject`, or the value is blank
or not a string.
Diagnostic: inspect the named sink in the error and check its `params`.
Safe fix: set a non-empty subject such as `weatherfeeder`.
## `dispatch: sink ... failed consuming event ... NATS sink: connect`
Symptom: the daemon starts, but NATS events are not published.
Likely cause: the NATS server URL is unreachable, the server is not accepting
connections, or the configured URL is wrong for the runtime network.
Diagnostic: from the same runtime environment, check that the host and port in
`sinks[].params.url` are reachable.
Safe fix: correct the NATS URL or restore broker connectivity. Other configured
sinks continue receiving events.
## `postgres sink ... open db`
Symptom: startup exits while building a Postgres sink.
Likely cause: the database URI, username, password, network path, or database
availability is wrong.
Diagnostic: inspect `sinks[].params.uri`, `username`, and `password`; verify
that the same runtime environment can reach the database.
Safe fix: correct the credentials or URI, restore database connectivity, then
restart the daemon.
## `postgres sink ... ensure table` Or `ensure index`
Symptom: startup exits during Postgres initialization.
Likely cause: the database user cannot create required tables or indexes, an
existing object conflicts with weatherfeeder's expected table contract, or the
database is unavailable during initialization.
Diagnostic: inspect the named table or index in the error and compare existing
database objects with the [Postgres table contract](integrations/postgres.md).
Safe fix: grant the needed database privileges, create a compatible schema, or
perform an operator-managed migration before restarting.
## `postgres sink: insert into ...`
Symptom: the daemon starts, but Postgres writes for some events fail.
Likely cause: a duplicate primary key, incompatible existing table definition,
database constraint error, or connection failure during a write transaction.
Diagnostic: inspect the table name and database error in the log. Compare the
table with [Postgres integration](integrations/postgres.md).
Safe fix: repair the database schema or address the duplicate/connection issue.
Other configured sinks continue receiving events.
## No Events Appear On A Sink
Symptom: the daemon is running but the expected sink receives no events.
Likely cause: the route does not match the event kind, the source has not
emitted changed content, or the sink is failing per event.
Diagnostic: check `routes`, source `kinds`, and logs for `scheduler: poll
failed`, `dispatcher: pipeline error`, or `dispatch: sink ... failed consuming
event`.
Safe fix: correct the route or source configuration. If the source uses
conditional HTTP and the upstream has not changed, no event is emitted for a
`304 Not Modified` response; wait for changed upstream content or temporarily
set `params.conditional: false` for diagnosis.
## `scheduler: poll failed`
Symptom: one source logs poll failures while the daemon keeps running.
Likely cause: upstream HTTP error, bad URL, timeout, response body limit, or
provider response shape that the source cannot parse.
Diagnostic: inspect the source name in the log and review its HTTP params.
Safe fix: correct the URL, user agent, timeout, or body limit. The next
scheduled poll will retry.
## `dispatcher: pipeline error`
Symptom: source polling succeeds, but one event is dropped before sinks.
Likely cause: a normalizer could not decode or map the raw payload, or dedupe
received an invalid event ID.
Diagnostic: inspect the error text and the source/schema that produced the
event. Review the [event wire contract](integrations/events.md) for expected
canonical fields.
Safe fix: correct source configuration if it points to the wrong upstream
product. If the upstream payload changed shape, update the relevant normalizer
and tests.

View File

@@ -0,0 +1,19 @@
---
sources:
- name: NWSObservationKSTL
mode: poll
kinds: ["observation"]
driver: nws_observation
every: 10m
params:
url: "https://api.weather.gov/stations/KSTL/observations/latest"
user_agent: "weatherfeeder example (operator@example.com)"
sinks:
- name: stdout
driver: stdout
params: {}
routes:
- sink: stdout
kinds: ["observation"]

42
examples/config.nats.yml Normal file
View File

@@ -0,0 +1,42 @@
---
sources:
- name: NWSObservationKSTL
mode: poll
kinds: ["observation"]
driver: nws_observation
every: 10m
params:
url: "https://api.weather.gov/stations/KSTL/observations/latest"
user_agent: "weatherfeeder example (operator@example.com)"
- name: NWSAlertsSTL
mode: poll
kinds: ["alert"]
driver: nws_alerts
every: 1m
params:
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
user_agent: "weatherfeeder example (operator@example.com)"
- name: SPCConvectiveOutlookSTL
mode: poll
kinds: ["outlook"]
driver: spc_convective_outlook
every: 30m
params:
latitude: 38.6239
longitude: -90.3571
location_id: "stl"
location_name: "St. Louis, MO"
user_agent: "weatherfeeder example (operator@example.com)"
sinks:
- name: nats_weather
driver: nats
params:
url: nats://localhost:4222
subject: weatherfeeder.events
routes:
- sink: nats_weather
kinds: ["observation", "alert", "outlook"]

View File

@@ -0,0 +1,44 @@
---
sources:
- name: NWSObservationKSTL
mode: poll
kinds: ["observation"]
driver: nws_observation
every: 10m
params:
url: "https://api.weather.gov/stations/KSTL/observations/latest"
user_agent: "weatherfeeder example (operator@example.com)"
- name: OpenMeteoHourlyForecastSTL
mode: poll
kinds: ["forecast"]
driver: openmeteo_forecast
every: 1h
params:
url: "https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&hourly=temperature_2m,relative_humidity_2m,dew_point_2m,apparent_temperature,precipitation_probability,precipitation,snowfall,weather_code,surface_pressure,wind_speed_10m,wind_direction_10m&forecast_days=3"
user_agent: "weatherfeeder example (operator@example.com)"
- name: SPCConvectiveOutlookSTL
mode: poll
kinds: ["outlook"]
driver: spc_convective_outlook
every: 30m
params:
latitude: 38.6239
longitude: -90.3571
location_id: "stl"
location_name: "St. Louis, MO"
user_agent: "weatherfeeder example (operator@example.com)"
sinks:
- name: pg_weather
driver: postgres
params:
uri: "postgres://postgres.example.invalid:5432/weatherfeeder?sslmode=disable"
username: <database_username>
password: <database_password>
prune: 3d
routes:
- sink: pg_weather
kinds: ["observation", "forecast", "outlook"]

2
go.mod
View File

@@ -2,7 +2,7 @@ module gitea.maximumdirect.net/ejr/weatherfeeder
go 1.25
require gitea.maximumdirect.net/ejr/feedkit v0.8.0
require gitea.maximumdirect.net/ejr/feedkit v0.9.1
require (
github.com/klauspost/compress v1.17.2 // indirect

4
go.sum
View File

@@ -1,5 +1,5 @@
gitea.maximumdirect.net/ejr/feedkit v0.8.0 h1:JdEEy6T3AQ97alLNYcQ3crN3tOEZPLMBD0Qr/MH5/dw=
gitea.maximumdirect.net/ejr/feedkit v0.8.0/go.mod h1:U6xC9xZLN3cL4yi7YBVyzGoHYRLJXusFCAKlj2kdYYQ=
gitea.maximumdirect.net/ejr/feedkit v0.9.1 h1:YghBQT1podqc+FJuPGuIZImV4A9dMr56Hikd5xuniig=
gitea.maximumdirect.net/ejr/feedkit v0.9.1/go.mod h1:U6xC9xZLN3cL4yi7YBVyzGoHYRLJXusFCAKlj2kdYYQ=
github.com/klauspost/compress v1.17.2 h1:RlWWUY/Dr4fL8qk9YG7DTZ7PDgME2V4csBXA8L/ixi4=
github.com/klauspost/compress v1.17.2/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=

107
internal/geo/geojson.go Normal file
View File

@@ -0,0 +1,107 @@
package geo
import (
"encoding/json"
"fmt"
)
type geometry struct {
Type string `json:"type"`
Coordinates json.RawMessage `json:"coordinates"`
}
// ContainsPoint reports whether a GeoJSON Polygon or MultiPolygon contains p.
// GeoJSON coordinate order is [longitude, latitude].
func ContainsPoint(raw []byte, p Point) (bool, error) {
if len(raw) == 0 {
return false, fmt.Errorf("geojson geometry is empty")
}
var geom geometry
if err := json.Unmarshal(raw, &geom); err != nil {
return false, fmt.Errorf("decode geojson geometry: %w", err)
}
switch geom.Type {
case "Polygon":
polygon, err := decodePolygon(geom.Coordinates)
if err != nil {
return false, fmt.Errorf("decode polygon: %w", err)
}
return polygonContainsPoint(polygon, p), nil
case "MultiPolygon":
multiPolygon, err := decodeMultiPolygon(geom.Coordinates)
if err != nil {
return false, fmt.Errorf("decode multipolygon: %w", err)
}
for _, polygon := range multiPolygon {
if polygonContainsPoint(polygon, p) {
return true, nil
}
}
return false, nil
case "":
return false, fmt.Errorf("geojson geometry type is required")
default:
return false, fmt.Errorf("unsupported geojson geometry type %q", geom.Type)
}
}
func decodePolygon(raw json.RawMessage) (Polygon, error) {
var coords [][][]float64
if err := json.Unmarshal(raw, &coords); err != nil {
return nil, err
}
return polygonFromCoordinates(coords)
}
func decodeMultiPolygon(raw json.RawMessage) ([]Polygon, error) {
var coords [][][][]float64
if err := json.Unmarshal(raw, &coords); err != nil {
return nil, err
}
if len(coords) == 0 {
return nil, fmt.Errorf("multipolygon has no polygons")
}
out := make([]Polygon, 0, len(coords))
for i, polygonCoords := range coords {
polygon, err := polygonFromCoordinates(polygonCoords)
if err != nil {
return nil, fmt.Errorf("polygons[%d]: %w", i, err)
}
out = append(out, polygon)
}
return out, nil
}
func polygonFromCoordinates(coords [][][]float64) (Polygon, error) {
if len(coords) == 0 {
return nil, fmt.Errorf("polygon has no rings")
}
polygon := make(Polygon, 0, len(coords))
for i, ringCoords := range coords {
ring, err := ringFromCoordinates(ringCoords)
if err != nil {
return nil, fmt.Errorf("rings[%d]: %w", i, err)
}
polygon = append(polygon, ring)
}
return polygon, nil
}
func ringFromCoordinates(coords [][]float64) (Ring, error) {
if len(coords) == 0 {
return nil, fmt.Errorf("ring has no points")
}
ring := make(Ring, 0, len(coords))
for i, pair := range coords {
if len(pair) < 2 {
return nil, fmt.Errorf("points[%d] has %d values, need longitude and latitude", i, len(pair))
}
ring = append(ring, Point{Longitude: pair[0], Latitude: pair[1]})
}
return ring, nil
}

105
internal/geo/point.go Normal file
View File

@@ -0,0 +1,105 @@
package geo
import "math"
const epsilon = 1e-9
// Point is a geographic coordinate in decimal degrees.
type Point struct {
Longitude float64
Latitude float64
}
// Ring is one GeoJSON linear ring.
type Ring []Point
// Polygon is a GeoJSON polygon. The first ring is the exterior ring; subsequent
// rings are holes.
type Polygon []Ring
func polygonContainsPoint(polygon Polygon, p Point) bool {
if len(polygon) == 0 {
return false
}
if pointOnRing(polygon[0], p) {
return true
}
if !ringContainsPoint(polygon[0], p) {
return false
}
for _, hole := range polygon[1:] {
if pointOnRing(hole, p) {
return true
}
if ringContainsPoint(hole, p) {
return false
}
}
return true
}
func ringContainsPoint(ring Ring, p Point) bool {
inside := false
n := len(ring)
if n == 0 {
return false
}
for i, j := 0, n-1; i < n; j, i = i, i+1 {
a := ring[j]
b := ring[i]
if pointOnSegment(p, a, b) {
return true
}
intersects := (a.Latitude > p.Latitude) != (b.Latitude > p.Latitude)
if intersects {
x := (b.Longitude-a.Longitude)*(p.Latitude-a.Latitude)/(b.Latitude-a.Latitude) + a.Longitude
if almostEqual(x, p.Longitude) {
return true
}
if x > p.Longitude {
inside = !inside
}
}
}
return inside
}
func pointOnRing(ring Ring, p Point) bool {
n := len(ring)
if n == 0 {
return false
}
for i, j := 0, n-1; i < n; j, i = i, i+1 {
if pointOnSegment(p, ring[j], ring[i]) {
return true
}
}
return false
}
func pointOnSegment(p, a, b Point) bool {
cross := (p.Latitude-a.Latitude)*(b.Longitude-a.Longitude) - (p.Longitude-a.Longitude)*(b.Latitude-a.Latitude)
if math.Abs(cross) > epsilon {
return false
}
minLon, maxLon := minMax(a.Longitude, b.Longitude)
minLat, maxLat := minMax(a.Latitude, b.Latitude)
return p.Longitude >= minLon-epsilon &&
p.Longitude <= maxLon+epsilon &&
p.Latitude >= minLat-epsilon &&
p.Latitude <= maxLat+epsilon
}
func minMax(a, b float64) (float64, float64) {
if a < b {
return a, b
}
return b, a
}
func almostEqual(a, b float64) bool {
return math.Abs(a-b) <= epsilon
}

181
internal/geo/point_test.go Normal file
View File

@@ -0,0 +1,181 @@
package geo
import (
"encoding/json"
"strings"
"testing"
)
const squarePolygon = `{
"type": "Polygon",
"coordinates": [[
[-91.0, 38.0],
[-90.0, 38.0],
[-90.0, 39.0],
[-91.0, 39.0],
[-91.0, 38.0]
]]
}`
func TestContainsPointInsideSimplePolygon(t *testing.T) {
got, err := ContainsPoint([]byte(squarePolygon), Point{Longitude: -90.5, Latitude: 38.5})
if err != nil {
t.Fatalf("ContainsPoint() error = %v", err)
}
if !got {
t.Fatalf("ContainsPoint() = false, want true")
}
}
func TestContainsPointAcceptsRawMessage(t *testing.T) {
got, err := ContainsPoint(json.RawMessage(squarePolygon), Point{Longitude: -90.5, Latitude: 38.5})
if err != nil {
t.Fatalf("ContainsPoint() error = %v", err)
}
if !got {
t.Fatalf("ContainsPoint() = false, want true")
}
}
func TestContainsPointOutsideSimplePolygon(t *testing.T) {
got, err := ContainsPoint([]byte(squarePolygon), Point{Longitude: -89.5, Latitude: 38.5})
if err != nil {
t.Fatalf("ContainsPoint() error = %v", err)
}
if got {
t.Fatalf("ContainsPoint() = true, want false")
}
}
func TestContainsPointOnBoundary(t *testing.T) {
got, err := ContainsPoint([]byte(squarePolygon), Point{Longitude: -91.0, Latitude: 38.5})
if err != nil {
t.Fatalf("ContainsPoint() error = %v", err)
}
if !got {
t.Fatalf("ContainsPoint() = false, want true")
}
}
func TestContainsPointInHoleReturnsFalse(t *testing.T) {
const polygonWithHole = `{
"type": "Polygon",
"coordinates": [
[[0,0],[10,0],[10,10],[0,10],[0,0]],
[[4,4],[6,4],[6,6],[4,6],[4,4]]
]
}`
got, err := ContainsPoint([]byte(polygonWithHole), Point{Longitude: 5, Latitude: 5})
if err != nil {
t.Fatalf("ContainsPoint() error = %v", err)
}
if got {
t.Fatalf("ContainsPoint() = true, want false")
}
}
func TestContainsPointOnHoleBoundaryReturnsTrue(t *testing.T) {
const polygonWithHole = `{
"type": "Polygon",
"coordinates": [
[[0,0],[10,0],[10,10],[0,10],[0,0]],
[[4,4],[6,4],[6,6],[4,6],[4,4]]
]
}`
got, err := ContainsPoint([]byte(polygonWithHole), Point{Longitude: 4, Latitude: 5})
if err != nil {
t.Fatalf("ContainsPoint() error = %v", err)
}
if !got {
t.Fatalf("ContainsPoint() = false, want true")
}
}
func TestContainsPointInsideOneMultiPolygonMember(t *testing.T) {
const multiPolygon = `{
"type": "MultiPolygon",
"coordinates": [
[[[0,0],[1,0],[1,1],[0,1],[0,0]]],
[[[10,10],[12,10],[12,12],[10,12],[10,10]]]
]
}`
got, err := ContainsPoint([]byte(multiPolygon), Point{Longitude: 11, Latitude: 11})
if err != nil {
t.Fatalf("ContainsPoint() error = %v", err)
}
if !got {
t.Fatalf("ContainsPoint() = false, want true")
}
}
func TestContainsPointUsesLongitudeLatitudeOrder(t *testing.T) {
const narrowPolygon = `{
"type": "Polygon",
"coordinates": [[
[-91.0, 38.0],
[-90.0, 38.0],
[-90.0, 39.0],
[-91.0, 39.0],
[-91.0, 38.0]
]]
}`
got, err := ContainsPoint([]byte(narrowPolygon), Point{Longitude: -90.5, Latitude: 38.5})
if err != nil {
t.Fatalf("ContainsPoint() error = %v", err)
}
if !got {
t.Fatalf("ContainsPoint() = false, want true")
}
got, err = ContainsPoint([]byte(narrowPolygon), Point{Longitude: 38.5, Latitude: -90.5})
if err != nil {
t.Fatalf("ContainsPoint() reversed error = %v", err)
}
if got {
t.Fatalf("ContainsPoint() with reversed coordinate values = true, want false")
}
}
func TestContainsPointUnsupportedGeometryError(t *testing.T) {
_, err := ContainsPoint([]byte(`{"type":"Point","coordinates":[-90,38]}`), Point{Longitude: -90, Latitude: 38})
if err == nil {
t.Fatalf("ContainsPoint() error = nil, want error")
}
if !strings.Contains(err.Error(), `unsupported geojson geometry type "Point"`) {
t.Fatalf("ContainsPoint() error = %q", err)
}
}
func TestContainsPointInvalidJSONError(t *testing.T) {
_, err := ContainsPoint([]byte(`{"type":"Polygon"`), Point{Longitude: -90, Latitude: 38})
if err == nil {
t.Fatalf("ContainsPoint() error = nil, want error")
}
if !strings.Contains(err.Error(), "decode geojson geometry") {
t.Fatalf("ContainsPoint() error = %q", err)
}
}
func TestContainsPointMalformedCoordinatesError(t *testing.T) {
_, err := ContainsPoint([]byte(`{"type":"Polygon","coordinates":[[[1]]]}`), Point{Longitude: 1, Latitude: 1})
if err == nil {
t.Fatalf("ContainsPoint() error = nil, want error")
}
if !strings.Contains(err.Error(), "need longitude and latitude") {
t.Fatalf("ContainsPoint() error = %q", err)
}
}
func TestContainsPointEmptyRingError(t *testing.T) {
_, err := ContainsPoint([]byte(`{"type":"Polygon","coordinates":[[]]}`), Point{Longitude: 1, Latitude: 1})
if err == nil {
t.Fatalf("ContainsPoint() error = nil, want error")
}
if !strings.Contains(err.Error(), "ring has no points") {
t.Fatalf("ContainsPoint() error = %q", err)
}
}

View File

@@ -7,8 +7,16 @@ import (
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openmeteo"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openweather"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/spc"
)
var builtinRegistrations = []func([]fknormalize.Normalizer) []fknormalize.Normalizer{
nws.Register,
openmeteo.Register,
openweather.Register,
spc.Register,
}
// RegisterBuiltins registers all normalizers shipped with this binary.
//
// This mirrors internal/sources.RegisterBuiltins, but note the selection model:
@@ -27,9 +35,9 @@ func RegisterBuiltins(in []fknormalize.Normalizer) []fknormalize.Normalizer {
//
// Order here should be stable across releases to reduce surprises when adding
// new normalizers.
out = nws.Register(out)
out = openmeteo.Register(out)
out = openweather.Register(out)
for _, register := range builtinRegistrations {
out = register(out)
}
return out
}

View File

@@ -8,6 +8,7 @@ import (
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openmeteo"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openweather"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/spc"
)
func TestRegisterBuiltinsOrder(t *testing.T) {
@@ -19,10 +20,13 @@ func TestRegisterBuiltinsOrder(t *testing.T) {
want := []fknormalize.Normalizer{
nws.ObservationNormalizer{},
nws.ForecastNormalizer{},
nws.ForecastDiscussionNormalizer{},
nws.WeatherStoriesNormalizer{},
nws.AlertsNormalizer{},
openmeteo.ObservationNormalizer{},
openmeteo.ForecastNormalizer{},
openweather.ObservationNormalizer{},
spc.ConvectiveOutlookNormalizer{},
}
if len(got) != len(want) {

View File

@@ -5,31 +5,19 @@ import (
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
)
// Finalize builds the output event envelope by copying the input and applying the
// canonical schema/payload, plus (optionally) EffectiveAt.
// canonical schema/payload, plus an optional effective time.
//
// Important behavior:
// - ID/Kind/Source/EmittedAt are preserved by copying the input event.
// - EffectiveAt is only overwritten when effectiveAt is non-zero.
// If effectiveAt is zero, any existing in.EffectiveAt is preserved.
// - EffectiveAt is only overwritten when the supplied effective time is non-zero.
// If the supplied time is zero, any existing in.EffectiveAt is preserved.
// - Payload floats are rounded to a stable wire-friendly precision (see round.go).
func Finalize(in event.Event, outSchema string, outPayload any, effectiveAt time.Time) (*event.Event, error) {
out := in
out.Schema = outSchema
// Enforce stable numeric presentation for sinks: round floats in the canonical payload.
out.Payload = RoundFloats(outPayload, DefaultFloatPrecision)
if !effectiveAt.IsZero() {
t := effectiveAt.UTC()
out.EffectiveAt = &t
}
if err := out.Validate(); err != nil {
return nil, err
}
return &out, nil
// Enforce stable numeric presentation for weather payloads before delegating to feedkit's
// generic envelope finalizer.
return fknormalize.FinalizeEvent(in, outSchema, RoundFloats(outPayload, DefaultFloatPrecision), effectiveAt)
}

View File

@@ -0,0 +1,36 @@
package common
import (
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
)
func TestFinalizeRoundsWeatherPayloadFloats(t *testing.T) {
type payload struct {
Value float64
}
in := event.Event{
ID: "evt-1",
Kind: event.Kind("observation"),
Source: "source-a",
EmittedAt: time.Date(2026, 3, 28, 12, 0, 0, 0, time.UTC),
Schema: "raw.example.v1",
Payload: map[string]any{"old": true},
}
out, err := Finalize(in, "weather.example.v1", payload{Value: 1.234567}, time.Time{})
if err != nil {
t.Fatalf("Finalize() unexpected error: %v", err)
}
got, ok := out.Payload.(payload)
if !ok {
t.Fatalf("Finalize() payload type = %T, want payload", out.Payload)
}
if got.Value != 1.2346 {
t.Fatalf("Finalize() rounded value = %v, want 1.2346", got.Value)
}
}

View File

@@ -2,11 +2,11 @@
package common
import (
"encoding/json"
"fmt"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
)
// DecodeJSONPayload extracts the event payload as bytes and unmarshals it into T.
@@ -15,36 +15,24 @@ import (
// - sources emit raw JSON payloads (typically json.RawMessage)
// - normalizers decode into provider structs
//
// Errors include a small amount of stage context ("extract payload", "decode raw payload").
// Errors include a small amount of operation context ("extract payload", "decode raw payload").
// Callers typically wrap these with a provider/kind label.
func DecodeJSONPayload[T any](in event.Event) (T, error) {
var zero T
b, err := PayloadBytes(in)
if err != nil {
return zero, fmt.Errorf("extract payload: %w", err)
}
var parsed T
if err := json.Unmarshal(b, &parsed); err != nil {
return zero, fmt.Errorf("decode raw payload: %w", err)
}
return parsed, nil
return fknormalize.DecodeJSONPayload[T](in)
}
// NormalizeJSON is a convenience wrapper for the common JSON-normalizer pattern:
//
// 1. Decode raw JSON payload into provider struct T
// 2. Map T into canonical payload P (plus an EffectiveAt timestamp)
// 3. Finalize the event envelope (schema/payload/effectiveAt) + Validate
// 2. Map T into canonical payload P (plus an effective time)
// 3. Finalize the event envelope (schema/payload/effective time) + Validate
//
// label should be short and specific, e.g. "openweather observation".
// outSchema should be the canonical schema constant.
// build should contain ONLY provider/domain mapping logic.
//
// Error policy:
// - NormalizeJSON wraps ALL failures with consistent context: "<label> normalize: <stage>: ..."
// - NormalizeJSON wraps ALL failures with consistent context: "<label> normalize: <operation>: ..."
// - build() should return specific errors without repeating the label prefix.
func NormalizeJSON[T any, P any](
in event.Event,

View File

@@ -1,53 +0,0 @@
package common
import (
"encoding/json"
"fmt"
"gitea.maximumdirect.net/ejr/feedkit/event"
)
// PayloadBytes extracts a JSON payload into bytes suitable for json.Unmarshal.
//
// Supported payload shapes (weatherfeeder convention):
// - json.RawMessage (recommended for raw events)
// - []byte
// - string (assumed to contain JSON)
// - map[string]any (re-marshaled to JSON)
//
// If you add other raw representations later, extend this function.
func PayloadBytes(e event.Event) ([]byte, error) {
if e.Payload == nil {
return nil, fmt.Errorf("payload is nil")
}
switch v := e.Payload.(type) {
case json.RawMessage:
if len(v) == 0 {
return nil, fmt.Errorf("payload is empty json.RawMessage")
}
return []byte(v), nil
case []byte:
if len(v) == 0 {
return nil, fmt.Errorf("payload is empty []byte")
}
return v, nil
case string:
if v == "" {
return nil, fmt.Errorf("payload is empty string")
}
return []byte(v), nil
case map[string]any:
b, err := json.Marshal(v)
if err != nil {
return nil, fmt.Errorf("marshal map payload: %w", err)
}
return b, nil
default:
return nil, fmt.Errorf("unsupported payload type %T", e.Payload)
}
}

View File

@@ -70,7 +70,7 @@
//
// weather.<kind>.vN
//
// weatherfeeder centralizes schema strings in internal/standards/schema.go.
// weatherfeeder centralizes schema strings in standards/schema.go.
// Always use those constants (do not inline schema strings).
//
// Example mappings:
@@ -101,8 +101,8 @@
// Every normalizer type must have a doc comment that states:
//
// - what it converts (e.g., “OpenWeather current -> WeatherObservation”)
// - which raw schema it matches (constant identifier from internal/standards)
// - which canonical schema it produces (constant identifier from internal/standards)
// - which raw schema it matches (constant identifier from standards)
// - which canonical schema it produces (constant identifier from standards)
// - any special caveats (units, day/night inference, missing fields, etc.)
//
// Including literal schema string values is optional,

View File

@@ -29,7 +29,7 @@ import (
// 2. Alert timing fields are best-effort parsed; invalid timestamps do not fail the
// entire normalization (they are left nil).
// 3. Some fields are intentionally passed through as strings (severity/urgency/etc.)
// since canonical vocabularies may evolve later.
// because the canonical model currently preserves provider vocabulary there.
type AlertsNormalizer struct{}
func (AlertsNormalizer) Match(e event.Event) bool {
@@ -37,7 +37,7 @@ func (AlertsNormalizer) Match(e event.Event) bool {
}
func (AlertsNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
// If we can't derive AsOf from the payload, fall back to the existing event envelope.
fallbackAsOf := in.EmittedAt.UTC()

View File

@@ -23,10 +23,11 @@ import (
// builders by raw schema.
//
// Caveats / policy:
// 1. NWS forecast periods do not include METAR presentWeather phenomena, so ConditionCode
// is inferred from period.shortForecast (with a conservative icon-based fallback).
// 2. Temperature is converted to °C when NWS supplies °F.
// 3. WindSpeed is parsed from strings like "9 mph" / "10 to 15 mph" and converted to km/h.
// 1. Hourly NWS forecast periods do not include METAR presentWeather phenomena, so
// ConditionCode is inferred from period.shortForecast (with a conservative icon fallback).
// 2. Narrative NWS periods intentionally leave ConditionCode unset.
// 3. Temperature is converted to °C when NWS supplies °F.
// 4. WindSpeed is parsed from strings like "9 mph" / "10 to 15 mph" and converted to km/h.
type ForecastNormalizer struct{}
func (ForecastNormalizer) Match(e event.Event) bool {
@@ -41,7 +42,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
}
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
return normalizeForecastEventBySchema(in)
}
@@ -75,62 +76,63 @@ func normalizeNarrativeForecastEvent(in event.Event) (*event.Event, error) {
)
}
type forecastPeriodMapper[T any] func(idx int, period T) (model.WeatherForecastPeriod, error)
// buildHourlyForecast contains hourly forecast mapping logic (provider -> canonical model).
func buildHourlyForecast(parsed nwsHourlyForecastResponse) (model.WeatherForecastRun, time.Time, error) {
issuedAt, updatedAt, err := parseForecastRunTimes(parsed.Properties.GeneratedAt, parsed.Properties.UpdateTime)
if err != nil {
return model.WeatherForecastRun{}, time.Time{}, err
}
// Best-effort location centroid from the GeoJSON polygon (optional).
lat, lon := centroidLatLon(parsed.Geometry.Coordinates)
run := newForecastRunBase(
issuedAt,
updatedAt,
model.ForecastProductHourly,
lat,
lon,
return buildForecastRun(
parsed.Properties.GeneratedAt,
parsed.Properties.UpdateTime,
parsed.Geometry.Coordinates,
parsed.Properties.Elevation.Value,
model.ForecastProductHourly,
parsed.Properties.Periods,
mapHourlyForecastPeriod,
)
periods := make([]model.WeatherForecastPeriod, 0, len(parsed.Properties.Periods))
for i, p := range parsed.Properties.Periods {
period, err := mapHourlyForecastPeriod(i, p)
if err != nil {
return model.WeatherForecastRun{}, time.Time{}, err
}
periods = append(periods, period)
}
run.Periods = periods
// EffectiveAt policy for forecasts: treat IssuedAt as the effective time (dedupe-friendly).
return run, issuedAt, nil
}
// buildNarrativeForecast contains narrative forecast mapping logic (provider -> canonical model).
func buildNarrativeForecast(parsed nwsNarrativeForecastResponse) (model.WeatherForecastRun, time.Time, error) {
issuedAt, updatedAt, err := parseForecastRunTimes(parsed.Properties.GeneratedAt, parsed.Properties.UpdateTime)
return buildForecastRun(
parsed.Properties.GeneratedAt,
parsed.Properties.UpdateTime,
parsed.Geometry.Coordinates,
parsed.Properties.Elevation.Value,
model.ForecastProductNarrative,
parsed.Properties.Periods,
mapNarrativeForecastPeriod,
)
}
func buildForecastRun[T any](
generatedAt string,
updateTime string,
coordinates [][][]float64,
elevation *float64,
product model.ForecastProduct,
srcPeriods []T,
mapPeriod forecastPeriodMapper[T],
) (model.WeatherForecastRun, time.Time, error) {
issuedAt, updatedAt, err := parseForecastRunTimes(generatedAt, updateTime)
if err != nil {
return model.WeatherForecastRun{}, time.Time{}, err
}
// Best-effort location centroid from the GeoJSON polygon (optional).
lat, lon := centroidLatLon(parsed.Geometry.Coordinates)
lat, lon := centroidLatLon(coordinates)
run := newForecastRunBase(
issuedAt,
updatedAt,
model.ForecastProductNarrative,
product,
lat,
lon,
parsed.Properties.Elevation.Value,
elevation,
)
periods := make([]model.WeatherForecastPeriod, 0, len(parsed.Properties.Periods))
for i, p := range parsed.Properties.Periods {
period, err := mapNarrativeForecastPeriod(i, p)
periods := make([]model.WeatherForecastPeriod, 0, len(srcPeriods))
for i, p := range srcPeriods {
period, err := mapPeriod(i, p)
if err != nil {
return model.WeatherForecastRun{}, time.Time{}, err
}
@@ -223,6 +225,7 @@ func mapHourlyForecastPeriod(idx int, p nwsHourlyForecastPeriod) (model.WeatherF
// Infer WMO from shortForecast (and fall back to icon token).
providerDesc := strings.TrimSpace(p.ShortForecast)
wmo := wmoFromNWSForecast(providerDesc, p.Icon, tempC)
wmoPtr := wmoCodePtr(wmo)
return model.WeatherForecastPeriod{
StartTime: start,
@@ -231,7 +234,7 @@ func mapHourlyForecastPeriod(idx int, p nwsHourlyForecastPeriod) (model.WeatherF
Name: strings.TrimSpace(p.Name),
IsDay: isDay,
ConditionCode: wmo,
ConditionCode: wmoPtr,
// For forecasts, keep provider short forecast text as the human-facing description.
TextDescription: providerDesc,
@@ -263,9 +266,7 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
tempC := tempCFromNWS(p.Temperature, p.TemperatureUnit)
// Infer WMO from shortForecast (and fall back to icon token).
shortForecast := strings.TrimSpace(p.ShortForecast)
wmo := wmoFromNWSForecast(shortForecast, p.Icon, tempC)
textDescription := strings.TrimSpace(p.DetailedForecast)
if textDescription == "" {
@@ -279,7 +280,7 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
Name: strings.TrimSpace(p.Name),
IsDay: isDay,
ConditionCode: wmo,
ConditionCode: nil,
TextDescription: textDescription,
@@ -291,3 +292,8 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
ProbabilityOfPrecipitationPercent: p.ProbabilityOfPrecipitation.Value,
}, nil
}
func wmoCodePtr(code model.WMOCode) *model.WMOCode {
out := code
return &out
}

View File

@@ -0,0 +1,72 @@
package nws
import (
"context"
"fmt"
"strings"
"gitea.maximumdirect.net/ejr/feedkit/event"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
type ForecastDiscussionNormalizer struct{}
func (ForecastDiscussionNormalizer) Match(e event.Event) bool {
return strings.TrimSpace(e.Schema) == standards.SchemaRawNWSForecastDiscussionV1
}
func (ForecastDiscussionNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx
rawHTML, err := decodeStringPayload(in.Payload)
if err != nil {
return nil, fmt.Errorf("nws forecast discussion normalize: %w", err)
}
parsed, err := nwscommon.ParseForecastDiscussionHTML(rawHTML)
if err != nil {
return nil, fmt.Errorf("nws forecast discussion normalize: build: %w", err)
}
payload := model.WeatherForecastDiscussion{
OfficeID: strings.TrimSpace(parsed.OfficeID),
OfficeName: strings.TrimSpace(parsed.OfficeName),
Product: model.ForecastDiscussionProduct(strings.TrimSpace(parsed.Product)),
IssuedAt: parsed.IssuedAt.UTC(),
UpdatedAt: parsed.UpdatedAt,
KeyMessages: append([]string(nil), parsed.KeyMessages...),
ShortTerm: mapForecastDiscussionSection(parsed.ShortTerm),
LongTerm: mapForecastDiscussionSection(parsed.LongTerm),
}
out, err := normcommon.Finalize(in, standards.SchemaWeatherForecastDiscussionV1, payload, payload.IssuedAt)
if err != nil {
return nil, fmt.Errorf("nws forecast discussion normalize: %w", err)
}
return out, nil
}
func mapForecastDiscussionSection(in *nwscommon.ForecastDiscussionSection) *model.WeatherForecastDiscussionSection {
if in == nil {
return nil
}
return &model.WeatherForecastDiscussionSection{
Qualifier: strings.TrimSpace(in.Qualifier),
IssuedAt: in.IssuedAt,
Text: strings.TrimSpace(in.Text),
}
}
func decodeStringPayload(payload any) (string, error) {
switch v := payload.(type) {
case string:
return v, nil
case []byte:
return string(v), nil
default:
return "", fmt.Errorf("extract payload: expected string payload, got %T", payload)
}
}

View File

@@ -0,0 +1,130 @@
package nws
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
func TestForecastDiscussionNormalizerProducesCanonicalSchema(t *testing.T) {
rawHTML := loadForecastDiscussionSampleHTML(t)
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
ID: "evt-discussion-1",
Kind: event.Kind("forecast_discussion"),
Source: "nws-discussion-test",
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
Schema: standards.SchemaRawNWSForecastDiscussionV1,
Payload: rawHTML,
})
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
if out == nil {
t.Fatalf("Normalize() returned nil output")
}
if out.Schema != standards.SchemaWeatherForecastDiscussionV1 {
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastDiscussionV1)
}
if out.Kind != event.Kind("forecast_discussion") {
t.Fatalf("Kind = %q, want forecast_discussion", out.Kind)
}
payload, ok := out.Payload.(model.WeatherForecastDiscussion)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherForecastDiscussion", out.Payload)
}
if payload.OfficeID != "LSX" {
t.Fatalf("OfficeID = %q, want LSX", payload.OfficeID)
}
if payload.Product != model.ForecastDiscussionProductAFD {
t.Fatalf("Product = %q, want %q", payload.Product, model.ForecastDiscussionProductAFD)
}
if len(payload.KeyMessages) != 3 {
t.Fatalf("KeyMessages len = %d, want 3", len(payload.KeyMessages))
}
if payload.ShortTerm == nil || payload.LongTerm == nil {
t.Fatalf("ShortTerm=%v LongTerm=%v, want both populated", payload.ShortTerm, payload.LongTerm)
}
if payload.ShortTerm.Qualifier != "(Through Late Sunday Night)" {
t.Fatalf("ShortTerm.Qualifier = %q", payload.ShortTerm.Qualifier)
}
if !strings.Contains(payload.ShortTerm.Text, "After a chilly morning") {
t.Fatalf("ShortTerm.Text = %q, want normalized prose", payload.ShortTerm.Text)
}
if strings.Contains(payload.ShortTerm.Text, "BRC") {
t.Fatalf("ShortTerm.Text contains signature: %q", payload.ShortTerm.Text)
}
if strings.Contains(payload.LongTerm.Text, "AVIATION") || strings.Contains(payload.LongTerm.Text, "WATCHES/WARNINGS/ADVISORIES") {
t.Fatalf("LongTerm.Text includes downstream sections: %q", payload.LongTerm.Text)
}
wantEffectiveAt := time.Date(2026, 3, 28, 19, 24, 0, 0, time.UTC)
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
}
}
func TestForecastDiscussionNormalizerRejectsMissingIssueTime(t *testing.T) {
_, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
ID: "evt-discussion-bad",
Kind: event.Kind("forecast_discussion"),
Source: "nws-discussion-test",
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
Schema: standards.SchemaRawNWSForecastDiscussionV1,
Payload: "<html><body><pre class=\"glossaryProduct\">National Weather Service Saint Louis MO</pre></body></html>",
})
if err == nil {
t.Fatalf("Normalize() error = nil, want error")
}
if !strings.Contains(err.Error(), "issue time") {
t.Fatalf("error = %q, want issue time context", err)
}
}
func TestForecastDiscussionNormalizerWireShapeHasNoUnexpectedKeys(t *testing.T) {
rawHTML := loadForecastDiscussionSampleHTML(t)
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
ID: "evt-discussion-2",
Kind: event.Kind("forecast_discussion"),
Source: "nws-discussion-test",
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
Schema: standards.SchemaRawNWSForecastDiscussionV1,
Payload: rawHTML,
})
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
b, err := json.Marshal(out.Payload)
if err != nil {
t.Fatalf("json.Marshal(payload) error = %v", err)
}
var got map[string]any
if err := json.Unmarshal(b, &got); err != nil {
t.Fatalf("json.Unmarshal(payload) error = %v", err)
}
for _, key := range []string{"sections", "aviation"} {
if _, ok := got[key]; ok {
t.Fatalf("unexpected key %q in canonical payload", key)
}
}
}
func loadForecastDiscussionSampleHTML(t *testing.T) string {
t.Helper()
path := filepath.Join("..", "..", "providers", "nws", "testdata", "forecast_discussion_sample.html")
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("os.ReadFile(%q) error = %v", path, err)
}
return string(b)
}

View File

@@ -35,6 +35,9 @@ func TestBuildHourlyForecastUsesShortForecastAsTextDescription(t *testing.T) {
if got, want := run.Periods[0].TextDescription, "Mostly Cloudy"; got != want {
t.Fatalf("TextDescription = %q, want %q", got, want)
}
if run.Periods[0].ConditionCode == nil {
t.Fatalf("ConditionCode is nil, want inferred hourly WMO code")
}
wantIssued := time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC)
if !run.IssuedAt.Equal(wantIssued) {
@@ -47,6 +50,55 @@ func TestBuildHourlyForecastUsesShortForecastAsTextDescription(t *testing.T) {
assertNoLegacyForecastDescriptionKeys(t, run.Periods[0])
}
func TestBuildHourlyForecastPreservesUpdatedAtCentroidAndElevation(t *testing.T) {
parsed := nwsHourlyForecastResponse{}
parsed.Properties.GeneratedAt = "2026-03-16T18:00:00Z"
parsed.Properties.UpdateTime = "2026-03-16T18:30:00Z"
elevation := 123.4
parsed.Properties.Elevation.Value = &elevation
parsed.Geometry.Coordinates = [][][]float64{
{
{-90.0, 38.0},
{-89.0, 38.0},
{-89.0, 39.0},
{-90.0, 39.0},
},
}
parsed.Properties.Periods = []nwsHourlyForecastPeriod{
{
StartTime: "2026-03-16T19:00:00Z",
EndTime: "2026-03-16T20:00:00Z",
ShortForecast: "Cloudy",
},
}
run, effectiveAt, err := buildHourlyForecast(parsed)
if err != nil {
t.Fatalf("buildHourlyForecast() error = %v", err)
}
wantIssued := time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC)
wantUpdated := time.Date(2026, 3, 16, 18, 30, 0, 0, time.UTC)
if !run.IssuedAt.Equal(wantIssued) {
t.Fatalf("IssuedAt = %s, want %s", run.IssuedAt.Format(time.RFC3339), wantIssued.Format(time.RFC3339))
}
if run.UpdatedAt == nil || !run.UpdatedAt.Equal(wantUpdated) {
t.Fatalf("UpdatedAt = %v, want %s", run.UpdatedAt, wantUpdated.Format(time.RFC3339))
}
if run.Latitude == nil || math.Abs(*run.Latitude-38.5) > 0.0001 {
t.Fatalf("Latitude = %v, want 38.5", run.Latitude)
}
if run.Longitude == nil || math.Abs(*run.Longitude+89.5) > 0.0001 {
t.Fatalf("Longitude = %v, want -89.5", run.Longitude)
}
if run.ElevationMeters == nil || math.Abs(*run.ElevationMeters-elevation) > 0.0001 {
t.Fatalf("ElevationMeters = %v, want %.1f", run.ElevationMeters, elevation)
}
if !effectiveAt.Equal(wantIssued) {
t.Fatalf("effectiveAt = %s, want %s", effectiveAt.Format(time.RFC3339), wantIssued.Format(time.RFC3339))
}
}
func TestNormalizeForecastEventBySchemaRejectsUnsupportedSchema(t *testing.T) {
_, err := normalizeForecastEventBySchema(event.Event{
Schema: "raw.nws.daily.forecast.v1",
@@ -85,6 +137,70 @@ func TestNormalizeForecastEventBySchemaRoutesNarrative(t *testing.T) {
}
}
func TestNormalizeForecastEventBySchemaProducesCanonicalWeatherForecastSchema(t *testing.T) {
tests := []struct {
name string
schema string
payload map[string]any
}{
{
name: "hourly",
schema: standards.SchemaRawNWSHourlyForecastV1,
payload: map[string]any{
"properties": map[string]any{
"generatedAt": "2026-03-16T18:00:00Z",
"periods": []map[string]any{
{
"startTime": "2026-03-16T19:00:00Z",
"endTime": "2026-03-16T20:00:00Z",
"shortForecast": "Cloudy",
},
},
},
},
},
{
name: "narrative",
schema: standards.SchemaRawNWSNarrativeForecastV1,
payload: map[string]any{
"properties": map[string]any{
"generatedAt": "2026-03-16T18:00:00Z",
"periods": []map[string]any{
{
"startTime": "2026-03-16T19:00:00Z",
"endTime": "2026-03-16T20:00:00Z",
"shortForecast": "Cloudy",
"detailedForecast": "Cloudy",
},
},
},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
out, err := normalizeForecastEventBySchema(event.Event{
ID: "evt-1",
Kind: event.Kind("forecast"),
Source: "nws-test",
EmittedAt: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
Schema: tt.schema,
Payload: tt.payload,
})
if err != nil {
t.Fatalf("normalizeForecastEventBySchema() error = %v", err)
}
if out == nil {
t.Fatalf("normalizeForecastEventBySchema() returned nil output")
}
if out.Schema != standards.SchemaWeatherForecastV1 {
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastV1)
}
})
}
}
func TestBuildNarrativeForecastMapsExpectedFields(t *testing.T) {
parsed := nwsNarrativeForecastResponse{}
parsed.Properties.GeneratedAt = "2026-03-27T15:17:01Z"
@@ -148,6 +264,9 @@ func TestBuildNarrativeForecastMapsExpectedFields(t *testing.T) {
if p.ProbabilityOfPrecipitationPercent == nil || *p.ProbabilityOfPrecipitationPercent != 20 {
t.Fatalf("ProbabilityOfPrecipitationPercent = %v, want 20", p.ProbabilityOfPrecipitationPercent)
}
if p.ConditionCode != nil {
t.Fatalf("ConditionCode = %v, want nil for narrative period", p.ConditionCode)
}
wantIssued := time.Date(2026, 3, 27, 15, 17, 1, 0, time.UTC)
if !run.IssuedAt.Equal(wantIssued) {

View File

@@ -32,7 +32,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
}
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
return normcommon.NormalizeJSON(
in,

View File

@@ -5,18 +5,15 @@ import (
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
)
var builtins = []fknormalize.Normalizer{
ObservationNormalizer{},
ForecastNormalizer{},
ForecastDiscussionNormalizer{},
WeatherStoriesNormalizer{},
AlertsNormalizer{},
}
// Register appends NWS normalizers in stable order.
func Register(in []fknormalize.Normalizer) []fknormalize.Normalizer {
out := in
// Observations
out = append(out, ObservationNormalizer{})
// Forecasts
out = append(out, ForecastNormalizer{})
// Alerts
out = append(out, AlertsNormalizer{})
return out
return append(in, builtins...)
}

View File

@@ -262,6 +262,25 @@ type nwsAlertProperties struct {
References json.RawMessage `json:"references"`
}
// nwsWeatherStoriesResponse is a minimal representation of the NWS /weatherstories
// payload needed for mapping into model.WeatherStoryRun.
type nwsWeatherStoriesResponse struct {
Stories []nwsWeatherStory `json:"stories"`
}
type nwsWeatherStory struct {
OfficeID string `json:"officeId"`
StartTime string `json:"startTime"`
EndTime string `json:"endTime"`
UpdateTime string `json:"updateTime"`
Title string `json:"title"`
Description string `json:"description"`
AltText string `json:"altText"`
Priority bool `json:"priority"`
Order int `json:"order"`
Download string `json:"download"`
}
type nwsAlertReference struct {
ID string `json:"id"`
Identifier string `json:"identifier"`

View File

@@ -0,0 +1,107 @@
package nws
import (
"context"
"fmt"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
// WeatherStoriesNormalizer converts:
//
// standards.SchemaRawNWSWeatherStoriesV1 -> standards.SchemaWeatherStoryV1
//
// It maps the NWS /weatherstories JSON response into a canonical story snapshot.
type WeatherStoriesNormalizer struct{}
func (WeatherStoriesNormalizer) Match(e event.Event) bool {
return strings.TrimSpace(e.Schema) == standards.SchemaRawNWSWeatherStoriesV1
}
func (WeatherStoriesNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx
fallbackAsOf := in.EmittedAt.UTC()
if in.EffectiveAt != nil && !in.EffectiveAt.IsZero() {
fallbackAsOf = in.EffectiveAt.UTC()
}
return normcommon.NormalizeJSON(
in,
"nws weatherstories",
standards.SchemaWeatherStoryV1,
func(parsed nwsWeatherStoriesResponse) (model.WeatherStoryRun, time.Time, error) {
return buildWeatherStories(parsed, fallbackAsOf)
},
)
}
func buildWeatherStories(parsed nwsWeatherStoriesResponse, fallbackAsOf time.Time) (model.WeatherStoryRun, time.Time, error) {
stories := make([]model.WeatherStory, 0, len(parsed.Stories))
var officeID string
var asOf time.Time
for i, raw := range parsed.Stories {
startTime, err := parseRequiredNWSTime(raw.StartTime, fmt.Sprintf("stories[%d].startTime", i))
if err != nil {
return model.WeatherStoryRun{}, time.Time{}, err
}
endTime, err := parseRequiredNWSTime(raw.EndTime, fmt.Sprintf("stories[%d].endTime", i))
if err != nil {
return model.WeatherStoryRun{}, time.Time{}, err
}
updatedAt, err := parseRequiredNWSTime(raw.UpdateTime, fmt.Sprintf("stories[%d].updateTime", i))
if err != nil {
return model.WeatherStoryRun{}, time.Time{}, err
}
storyOfficeID := strings.TrimSpace(raw.OfficeID)
if officeID == "" && storyOfficeID != "" {
officeID = storyOfficeID
}
if asOf.IsZero() || updatedAt.After(asOf) {
asOf = updatedAt
}
stories = append(stories, model.WeatherStory{
OfficeID: storyOfficeID,
StartTime: startTime,
EndTime: endTime,
UpdatedAt: updatedAt,
Title: strings.TrimSpace(raw.Title),
Description: strings.TrimSpace(raw.Description),
AltText: strings.TrimSpace(raw.AltText),
Priority: raw.Priority,
Order: raw.Order,
DownloadURL: strings.TrimSpace(raw.Download),
})
}
if asOf.IsZero() {
asOf = fallbackAsOf.UTC()
}
run := model.WeatherStoryRun{
OfficeID: officeID,
AsOf: asOf,
Stories: stories,
}
return run, asOf, nil
}
func parseRequiredNWSTime(raw, field string) (time.Time, error) {
if strings.TrimSpace(raw) == "" {
return time.Time{}, fmt.Errorf("%s is required", field)
}
t, err := nwscommon.ParseTime(raw)
if err != nil {
return time.Time{}, fmt.Errorf("%s: %w", field, err)
}
return t.UTC(), nil
}

View File

@@ -0,0 +1,146 @@
package nws
import (
"encoding/json"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
func TestWeatherStoriesNormalizerProducesCanonicalSchemaAndMapsSample(t *testing.T) {
out, err := (WeatherStoriesNormalizer{}).Normalize(nil, weatherStoriesRawEvent(weatherStoriesSamplePayload()))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
if out == nil {
t.Fatalf("Normalize() returned nil output")
}
if out.Schema != standards.SchemaWeatherStoryV1 {
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherStoryV1)
}
if out.Kind != event.Kind("weather_story") {
t.Fatalf("Kind = %q, want weather_story", out.Kind)
}
payload, ok := out.Payload.(model.WeatherStoryRun)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherStoryRun", out.Payload)
}
if payload.OfficeID != "LSX" {
t.Fatalf("OfficeID = %q, want LSX", payload.OfficeID)
}
wantAsOf := time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC)
if !payload.AsOf.Equal(wantAsOf) {
t.Fatalf("AsOf = %s, want %s", payload.AsOf, wantAsOf)
}
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantAsOf) {
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantAsOf)
}
if len(payload.Stories) != 1 {
t.Fatalf("Stories len = %d, want 1", len(payload.Stories))
}
story := payload.Stories[0]
if story.Title != "Several Chances for Rain Through Monday" {
t.Fatalf("Title = %q", story.Title)
}
if story.Description != "A stagnant weather pattern." {
t.Fatalf("Description = %q", story.Description)
}
if story.AltText != "This slide shows the forecast." {
t.Fatalf("AltText = %q", story.AltText)
}
if story.Priority {
t.Fatalf("Priority = true, want false")
}
if story.Order != 1 {
t.Fatalf("Order = %d, want 1", story.Order)
}
if story.DownloadURL != "https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f" {
t.Fatalf("DownloadURL = %q", story.DownloadURL)
}
}
func TestWeatherStoriesNormalizerEmptyStoriesUsesFallbackAsOf(t *testing.T) {
effectiveAt := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC)
in := weatherStoriesRawEvent(`{"stories":[]}`)
in.EffectiveAt = &effectiveAt
out, err := (WeatherStoriesNormalizer{}).Normalize(nil, in)
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
payload, ok := out.Payload.(model.WeatherStoryRun)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherStoryRun", out.Payload)
}
if !payload.AsOf.Equal(effectiveAt) {
t.Fatalf("AsOf = %s, want fallback %s", payload.AsOf, effectiveAt)
}
if payload.Stories == nil {
t.Fatalf("Stories = nil, want empty slice")
}
if len(payload.Stories) != 0 {
t.Fatalf("Stories len = %d, want 0", len(payload.Stories))
}
}
func TestWeatherStoriesNormalizerRejectsInvalidRequiredStoryTime(t *testing.T) {
_, err := (WeatherStoriesNormalizer{}).Normalize(nil, weatherStoriesRawEvent(`{
"stories": [{
"officeId": "LSX",
"startTime": "bad",
"endTime": "2026-05-31T11:00:00+00:00",
"updateTime": "2026-05-30T09:00:34+00:00"
}]
}`))
if err == nil {
t.Fatalf("Normalize() error = nil, want error")
}
if !strings.Contains(err.Error(), "stories[0].startTime") {
t.Fatalf("error = %q, want field context", err)
}
}
func TestWeatherStoriesNormalizerMatch(t *testing.T) {
n := WeatherStoriesNormalizer{}
if !n.Match(event.Event{Schema: standards.SchemaRawNWSWeatherStoriesV1}) {
t.Fatalf("Match(raw weatherstories) = false, want true")
}
if n.Match(event.Event{Schema: standards.SchemaRawNWSAlertsV1}) {
t.Fatalf("Match(raw alerts) = true, want false")
}
}
func weatherStoriesRawEvent(payload string) event.Event {
return event.Event{
ID: "evt-weatherstories-1",
Kind: event.Kind("weather_story"),
Source: "nws-weatherstories-test",
EmittedAt: time.Date(2026, 5, 30, 9, 5, 0, 0, time.UTC),
Schema: standards.SchemaRawNWSWeatherStoriesV1,
Payload: json.RawMessage(payload),
}
}
func weatherStoriesSamplePayload() string {
return `{
"stories": [
{
"officeId": " LSX ",
"startTime": "2026-05-30T08:46:00+00:00",
"endTime": "2026-05-31T11:00:00+00:00",
"updateTime": "2026-05-30T09:00:34+00:00",
"title": " Several Chances for Rain Through Monday ",
"description": " A stagnant weather pattern. ",
"altText": " This slide shows the forecast. ",
"priority": false,
"order": 1,
"download": " https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f "
}
]
}`
}

View File

@@ -33,7 +33,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
}
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
// If present, prefer the existing event EmittedAt as IssuedAt.
var fallbackIssued time.Time
@@ -98,6 +98,7 @@ func buildForecast(parsed omForecastResponse, fallbackIssued time.Time) (model.W
}
wmo := wmoAt(parsed.Hourly.WeatherCode, i)
wmoPtr := wmoCodePtr(wmo)
canonicalText := standards.WMOText(wmo, isDay)
period := model.WeatherForecastPeriod{
@@ -107,7 +108,7 @@ func buildForecast(parsed omForecastResponse, fallbackIssued time.Time) (model.W
Name: "",
IsDay: isDay,
ConditionCode: wmo,
ConditionCode: wmoPtr,
TextDescription: canonicalText,
}
@@ -237,3 +238,8 @@ func wmoAt(vals []*int, idx int) model.WMOCode {
}
return model.WMOUnknown
}
func wmoCodePtr(code model.WMOCode) *model.WMOCode {
out := code
return &out
}

View File

@@ -35,6 +35,9 @@ func TestBuildForecastUsesCanonicalTextDescription(t *testing.T) {
if got := run.Periods[0].TextDescription; got != expectedText {
t.Fatalf("TextDescription = %q, want %q", got, expectedText)
}
if run.Periods[0].ConditionCode == nil {
t.Fatalf("ConditionCode is nil, want mapped WMO code")
}
wantIssued := time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC)
if !run.IssuedAt.Equal(wantIssued) {

View File

@@ -40,7 +40,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
}
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
return normcommon.NormalizeJSON(
in,

View File

@@ -5,14 +5,12 @@ import (
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
)
var builtins = []fknormalize.Normalizer{
ObservationNormalizer{},
ForecastNormalizer{},
}
// Register appends Open-Meteo normalizers in stable order.
func Register(in []fknormalize.Normalizer) []fknormalize.Normalizer {
out := in
// Observations
out = append(out, ObservationNormalizer{})
// Forecasts
out = append(out, ForecastNormalizer{})
return out
return append(in, builtins...)
}

View File

@@ -8,8 +8,7 @@ import (
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
)
// This file holds provider-specific helpers that are shared across multiple
// OpenWeather normalizers (observations today; forecasts/alerts later).
// This file holds provider-specific helpers for OpenWeather normalizers.
// Keeping these out of observation.go helps preserve the "one normalizer per file"
// convention while avoiding duplication.

View File

@@ -37,7 +37,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
}
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
return normcommon.NormalizeJSON(
in,

View File

@@ -5,12 +5,11 @@ import (
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
)
var builtins = []fknormalize.Normalizer{
ObservationNormalizer{},
}
// Register appends OpenWeather normalizers in stable order.
func Register(in []fknormalize.Normalizer) []fknormalize.Normalizer {
out := in
// Observations
out = append(out, ObservationNormalizer{})
return out
return append(in, builtins...)
}

View File

@@ -0,0 +1,313 @@
package spc
import (
"context"
"encoding/json"
"fmt"
"math"
"regexp"
"sort"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/geo"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
spcprovider "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/spc"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
const (
providerSPC = "spc"
productConvective = "convective"
outlookNormalizer = "spc convective outlook"
outlookKind = "outlook"
outlookTypeUnknown = 99
)
var idTokenRE = regexp.MustCompile(`[^a-z0-9]+`)
// ConvectiveOutlookNormalizer converts:
//
// standards.SchemaRawSPCConvectiveOutlookV1 -> standards.SchemaWeatherOutlookV1
//
// It maps SPC GeoJSON outlook features into canonical outlook polygons and
// enriches each day with the matching required print-page discussion.
type ConvectiveOutlookNormalizer struct{}
func (ConvectiveOutlookNormalizer) Match(e event.Event) bool {
return strings.TrimSpace(e.Schema) == standards.SchemaRawSPCConvectiveOutlookV1
}
func (ConvectiveOutlookNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx
fallbackAsOf := in.EmittedAt.UTC()
if in.EffectiveAt != nil && !in.EffectiveAt.IsZero() {
fallbackAsOf = in.EffectiveAt.UTC()
}
return normcommon.NormalizeJSON(
in,
outlookNormalizer,
standards.SchemaWeatherOutlookV1,
func(parsed spcprovider.RawConvectiveOutlookBundle) (model.WeatherOutlookRun, time.Time, error) {
return buildConvectiveOutlook(parsed, fallbackAsOf)
},
)
}
func buildConvectiveOutlook(bundle spcprovider.RawConvectiveOutlookBundle, fallbackAsOf time.Time) (model.WeatherOutlookRun, time.Time, error) {
if err := validateCoordinates(bundle.Latitude, bundle.Longitude); err != nil {
return model.WeatherOutlookRun{}, time.Time{}, err
}
discussions, latestDiscussionUpdated, err := parseDiscussions(bundle.Discussions)
if err != nil {
return model.WeatherOutlookRun{}, time.Time{}, err
}
products := orderedProducts(bundle.Products)
point := geo.Point{Latitude: bundle.Latitude, Longitude: bundle.Longitude}
outlooks := make([]model.WeatherOutlook, 0)
var latestIssue time.Time
for _, product := range products {
if err := validateProductMetadata(product); err != nil {
return model.WeatherOutlookRun{}, time.Time{}, err
}
discussion, ok := discussions[product.Day]
if !ok {
return model.WeatherOutlookRun{}, time.Time{}, fmt.Errorf("product %s: discussion for day %d is required", product.Key, product.Day)
}
collection, err := spcprovider.DecodeGeoJSON(product.Body)
if err != nil {
return model.WeatherOutlookRun{}, time.Time{}, fmt.Errorf("product %s: %w", product.Key, err)
}
for i, feature := range collection.Features {
outlook, err := mapFeature(product, feature, i, point, discussion)
if err != nil {
return model.WeatherOutlookRun{}, time.Time{}, err
}
if latestIssue.IsZero() || outlook.IssuedAt.After(latestIssue) {
latestIssue = outlook.IssuedAt
}
outlooks = append(outlooks, outlook)
}
}
asOf := latestIssue
if asOf.IsZero() {
asOf = latestDiscussionUpdated
}
if asOf.IsZero() {
asOf = fallbackAsOf.UTC()
}
var issuedAt *time.Time
if !latestIssue.IsZero() {
t := latestIssue.UTC()
issuedAt = &t
}
lat := bundle.Latitude
lon := bundle.Longitude
run := model.WeatherOutlookRun{
LocationID: strings.TrimSpace(bundle.LocationID),
LocationName: strings.TrimSpace(bundle.LocationName),
Latitude: &lat,
Longitude: &lon,
AsOf: asOf.UTC(),
IssuedAt: issuedAt,
Outlooks: outlooks,
}
return run, run.AsOf, nil
}
type parsedDiscussion struct {
Headline string
Summary string
Discussion string
UpdatedAt *time.Time
}
func parseDiscussions(pages []spcprovider.RawDiscussionPage) (map[int]parsedDiscussion, time.Time, error) {
out := map[int]parsedDiscussion{}
var latestUpdated time.Time
for _, page := range pages {
parsed, err := spcprovider.ParseDiscussionHTML(page.Body)
if err != nil {
return nil, time.Time{}, fmt.Errorf("discussion %s: %w", page.Key, err)
}
day := page.Day
if day == 0 {
if meta, ok := spcprovider.DiscussionProductByKey(page.Key); ok {
day = meta.Day
}
}
if day < 1 || day > 3 {
return nil, time.Time{}, fmt.Errorf("discussion %s: day must be 1, 2, or 3, got %d", page.Key, page.Day)
}
disc := parsedDiscussion{
Headline: strings.TrimSpace(parsed.Headline),
Summary: strings.TrimSpace(parsed.Summary),
Discussion: strings.TrimSpace(parsed.Discussion),
UpdatedAt: parsed.UpdatedAt,
}
out[day] = disc
if parsed.UpdatedAt != nil && (latestUpdated.IsZero() || parsed.UpdatedAt.After(latestUpdated)) {
latestUpdated = parsed.UpdatedAt.UTC()
}
}
return out, latestUpdated, nil
}
func orderedProducts(products []spcprovider.RawOutlookProduct) []spcprovider.RawOutlookProduct {
out := make([]spcprovider.RawOutlookProduct, len(products))
copy(out, products)
sort.SliceStable(out, func(i, j int) bool {
if out[i].Day != out[j].Day {
return out[i].Day < out[j].Day
}
left := outlookTypeOrder(out[i].OutlookType)
right := outlookTypeOrder(out[j].OutlookType)
if left != right {
return left < right
}
return out[i].Key < out[j].Key
})
return out
}
func outlookTypeOrder(outlookType string) int {
switch strings.TrimSpace(outlookType) {
case spcprovider.OutlookTypeCategorical:
return 0
case spcprovider.OutlookTypeTornado:
return 1
case spcprovider.OutlookTypeHail:
return 2
case spcprovider.OutlookTypeWind:
return 3
default:
return outlookTypeUnknown
}
}
func validateProductMetadata(product spcprovider.RawOutlookProduct) error {
if product.Day < 1 || product.Day > 3 {
return fmt.Errorf("product %s: day must be 1, 2, or 3, got %d", product.Key, product.Day)
}
switch strings.TrimSpace(product.OutlookType) {
case spcprovider.OutlookTypeCategorical, spcprovider.OutlookTypeTornado, spcprovider.OutlookTypeHail, spcprovider.OutlookTypeWind:
return nil
default:
return fmt.Errorf("product %s: unsupported outlook type %q", product.Key, product.OutlookType)
}
}
func mapFeature(product spcprovider.RawOutlookProduct, feature spcprovider.GeoJSONFeature, index int, point geo.Point, discussion parsedDiscussion) (model.WeatherOutlook, error) {
fieldPrefix := fmt.Sprintf("product %s feature %d", product.Key, index)
props := feature.Properties
validFrom, err := parseRequiredSPCTime(props.ValidISO, fieldPrefix+".VALID_ISO")
if err != nil {
return model.WeatherOutlook{}, err
}
validTo, err := parseRequiredSPCTime(props.ExpireISO, fieldPrefix+".EXPIRE_ISO")
if err != nil {
return model.WeatherOutlook{}, err
}
issuedAt, err := parseRequiredSPCTime(props.IssueISO, fieldPrefix+".ISSUE_ISO")
if err != nil {
return model.WeatherOutlook{}, err
}
label := strings.TrimSpace(props.Label)
if label == "" {
return model.WeatherOutlook{}, fmt.Errorf("%s.LABEL is required", fieldPrefix)
}
if len(feature.Geometry) == 0 {
return model.WeatherOutlook{}, fmt.Errorf("%s.geometry is required", fieldPrefix)
}
containsLocation, err := geo.ContainsPoint(feature.Geometry, point)
if err != nil {
return model.WeatherOutlook{}, fmt.Errorf("%s.geometry: %w", fieldPrefix, err)
}
geometry := make(json.RawMessage, len(feature.Geometry))
copy(geometry, feature.Geometry)
return model.WeatherOutlook{
ID: outlookID(product.Day, product.OutlookType, label, issuedAt, validFrom, index),
Provider: providerSPC,
Product: productConvective,
Day: product.Day,
OutlookType: strings.TrimSpace(product.OutlookType),
Label: label,
LabelText: strings.TrimSpace(props.Label2),
SeverityRank: props.DN,
ValidFrom: validFrom,
ValidTo: validTo,
IssuedAt: issuedAt,
ExpiresAt: validTo,
Forecaster: strings.TrimSpace(props.Forecaster),
Headline: discussion.Headline,
Summary: discussion.Summary,
Discussion: discussion.Discussion,
SourceURL: strings.TrimSpace(product.URL),
ImageURL: "",
ContainsLocation: containsLocation,
Geometry: geometry,
}, nil
}
func parseRequiredSPCTime(value, field string) (time.Time, error) {
if strings.TrimSpace(value) == "" {
return time.Time{}, fmt.Errorf("%s is required", field)
}
t, err := spcprovider.ParseISOTimestamp(value)
if err != nil {
return time.Time{}, fmt.Errorf("%s: %w", field, err)
}
return t.UTC(), nil
}
func outlookID(day int, outlookType, label string, issuedAt time.Time, validFrom time.Time, index int) string {
return fmt.Sprintf(
"spc-convective-day%d-%s-%s-%s-%s-%d",
day,
safeIDToken(outlookType),
safeIDToken(label),
issuedAt.UTC().Format(time.RFC3339),
validFrom.UTC().Format(time.RFC3339),
index,
)
}
func safeIDToken(value string) string {
value = strings.ToLower(strings.TrimSpace(value))
value = idTokenRE.ReplaceAllString(value, "-")
value = strings.Trim(value, "-")
if value == "" {
return "unknown"
}
return value
}
func validateCoordinates(latitude, longitude float64) error {
switch {
case math.IsNaN(latitude) || math.IsInf(latitude, 0):
return fmt.Errorf("latitude must be finite")
case math.IsNaN(longitude) || math.IsInf(longitude, 0):
return fmt.Errorf("longitude must be finite")
case latitude < -90 || latitude > 90:
return fmt.Errorf("latitude must be between -90 and 90, got %v", latitude)
case longitude < -180 || longitude > 180:
return fmt.Errorf("longitude must be between -180 and 180, got %v", longitude)
default:
return nil
}
}

View File

@@ -0,0 +1,357 @@
package spc
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
spcprovider "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/spc"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
func TestConvectiveOutlookNormalizerMatch(t *testing.T) {
n := ConvectiveOutlookNormalizer{}
if !n.Match(event.Event{Schema: standards.SchemaRawSPCConvectiveOutlookV1}) {
t.Fatalf("Match(raw SPC outlook) = false, want true")
}
if n.Match(event.Event{Schema: standards.SchemaRawNWSAlertsV1}) {
t.Fatalf("Match(raw NWS alerts) = true, want false")
}
}
func TestConvectiveOutlookNormalizerProducesCanonicalSchemaAndMapsSample(t *testing.T) {
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, spcBundle(t, 38.5, -90.5)))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
if out.Schema != standards.SchemaWeatherOutlookV1 {
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherOutlookV1)
}
if out.Kind != event.Kind("outlook") {
t.Fatalf("Kind = %q, want outlook", out.Kind)
}
run, ok := out.Payload.(model.WeatherOutlookRun)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherOutlookRun", out.Payload)
}
wantAsOf := time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC)
if !run.AsOf.Equal(wantAsOf) {
t.Fatalf("AsOf = %s, want %s", run.AsOf, wantAsOf)
}
if run.IssuedAt == nil || !run.IssuedAt.Equal(wantAsOf) {
t.Fatalf("IssuedAt = %v, want %s", run.IssuedAt, wantAsOf)
}
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantAsOf) {
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantAsOf)
}
if run.LocationID != "stl" || run.LocationName != "St. Louis, MO" {
t.Fatalf("location metadata = %q/%q", run.LocationID, run.LocationName)
}
if run.Latitude == nil || *run.Latitude != 38.5 || run.Longitude == nil || *run.Longitude != -90.5 {
t.Fatalf("coordinates = %v,%v", run.Latitude, run.Longitude)
}
if len(run.Outlooks) != 12 {
t.Fatalf("Outlooks length = %d, want 12", len(run.Outlooks))
}
got := run.Outlooks[0]
if got.Provider != "spc" || got.Product != "convective" {
t.Fatalf("provider/product = %q/%q", got.Provider, got.Product)
}
if got.Day != 1 || got.OutlookType != spcprovider.OutlookTypeCategorical {
t.Fatalf("day/type = %d/%q", got.Day, got.OutlookType)
}
if got.Label != "SLGT" || got.LabelText != "Slight Risk" {
t.Fatalf("label fields = %q/%q", got.Label, got.LabelText)
}
if got.SeverityRank == nil || *got.SeverityRank != 3 {
t.Fatalf("SeverityRank = %v, want 3", got.SeverityRank)
}
assertTime(t, "ValidFrom", got.ValidFrom, 2026, 6, 11, 13, 0, 0)
assertTime(t, "ValidTo", got.ValidTo, 2026, 6, 12, 12, 0, 0)
assertTime(t, "IssuedAt", got.IssuedAt, 2026, 6, 11, 12, 34, 56)
assertTime(t, "ExpiresAt", got.ExpiresAt, 2026, 6, 12, 12, 0, 0)
if got.Forecaster != "SMITH" {
t.Fatalf("Forecaster = %q, want SMITH", got.Forecaster)
}
if got.SourceURL != "https://example.invalid/day1_categorical.geojson" {
t.Fatalf("SourceURL = %q", got.SourceURL)
}
wantGeometry := `{"type":"Polygon","coordinates":[[[-91.0,38.0],[-90.0,38.0],[-90.0,39.0],[-91.0,39.0],[-91.0,38.0]]]}`
if string(got.Geometry) != wantGeometry {
t.Fatalf("Geometry = %s, want %s", got.Geometry, wantGeometry)
}
if !got.ContainsLocation {
t.Fatalf("ContainsLocation = false, want true")
}
if got.Headline != "Day 1 Convective Outlook" {
t.Fatalf("Headline = %q", got.Headline)
}
if !strings.Contains(got.Summary, "central Plains") {
t.Fatalf("Summary = %q", got.Summary)
}
if !strings.Contains(got.Discussion, "...DISCUSSION...") {
t.Fatalf("Discussion missing product text: %q", got.Discussion)
}
if !strings.HasPrefix(got.Discussion, "SPC AC 111234") {
t.Fatalf("Discussion = %q, want SPC product code prefix", got.Discussion)
}
if got.ID != "spc-convective-day1-categorical-slgt-2026-06-11T12:34:56Z-2026-06-11T13:00:00Z-0" {
t.Fatalf("ID = %q", got.ID)
}
}
func TestConvectiveOutlookNormalizerOrdersProductsByDayAndType(t *testing.T) {
bundle := spcBundle(t, 0, 0)
for i, j := 0, len(bundle.Products)-1; i < j; i, j = i+1, j-1 {
bundle.Products[i], bundle.Products[j] = bundle.Products[j], bundle.Products[i]
}
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
run := out.Payload.(model.WeatherOutlookRun)
got := []string{
run.Outlooks[0].OutlookType,
run.Outlooks[1].OutlookType,
run.Outlooks[2].OutlookType,
run.Outlooks[3].OutlookType,
}
want := []string{
spcprovider.OutlookTypeCategorical,
spcprovider.OutlookTypeTornado,
spcprovider.OutlookTypeHail,
spcprovider.OutlookTypeWind,
}
for i := range want {
if got[i] != want[i] || run.Outlooks[i].Day != 1 {
t.Fatalf("outlook[%d] = day %d type %q, want day 1 type %q", i, run.Outlooks[i].Day, got[i], want[i])
}
}
}
func TestConvectiveOutlookNormalizerMapsProbabilisticOutlookTypes(t *testing.T) {
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, spcBundle(t, 0, 0)))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
run := out.Payload.(model.WeatherOutlookRun)
for _, outlookType := range []string{
spcprovider.OutlookTypeTornado,
spcprovider.OutlookTypeHail,
spcprovider.OutlookTypeWind,
} {
if findOutlook(run.Outlooks, 1, outlookType) == nil {
t.Fatalf("missing day 1 outlook type %q", outlookType)
}
}
}
func TestConvectiveOutlookNormalizerContainsLocationFalseOutsidePolygon(t *testing.T) {
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, spcBundle(t, 0, 0)))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
run := out.Payload.(model.WeatherOutlookRun)
if run.Outlooks[0].ContainsLocation {
t.Fatalf("ContainsLocation = true, want false")
}
}
func TestConvectiveOutlookNormalizerPreservesCorrectionMarker(t *testing.T) {
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, spcBundle(t, 0, 0)))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
run := out.Payload.(model.WeatherOutlookRun)
got := findOutlook(run.Outlooks, 2, spcprovider.OutlookTypeTornado)
if got == nil {
t.Fatalf("missing day 2 tornado outlook")
}
if !strings.Contains(got.Headline, "CORR 1") {
t.Fatalf("Headline = %q, want correction marker", got.Headline)
}
if !strings.Contains(got.Discussion, "CORR 1") {
t.Fatalf("Discussion = %q, want correction marker", got.Discussion)
}
}
func TestConvectiveOutlookNormalizerMissingRSSNormalizes(t *testing.T) {
bundle := spcBundle(t, 0, 0)
bundle.RSS = nil
if _, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle)); err != nil {
t.Fatalf("Normalize() error = %v", err)
}
}
func TestConvectiveOutlookNormalizerInvalidRequiredTimestampFailsWithContext(t *testing.T) {
bundle := spcBundle(t, 0, 0)
bundle.Products[0].Body = json.RawMessage(strings.Replace(
string(bundle.Products[0].Body),
`"ISSUE_ISO": "2026-06-11T12:34:56Z"`,
`"ISSUE_ISO": "bad"`,
1,
))
_, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle))
if err == nil {
t.Fatalf("Normalize() error = nil, want error")
}
if !strings.Contains(err.Error(), "product day1_categorical feature 0.ISSUE_ISO") {
t.Fatalf("error = %q, want product and feature context", err)
}
}
func TestConvectiveOutlookNormalizerInvalidGeometryFailsWithContext(t *testing.T) {
bundle := spcBundle(t, 0, 0)
bundle.Products[0].Body = json.RawMessage(strings.Replace(
string(bundle.Products[0].Body),
`"geometry": {`,
`"geometry": {"type":"LineString","coordinates":[[-91,38],[-90,39]]}, "oldGeometry": {`,
1,
))
_, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle))
if err == nil {
t.Fatalf("Normalize() error = nil, want error")
}
if !strings.Contains(err.Error(), "product day1_categorical feature 0.geometry") {
t.Fatalf("error = %q, want product and feature context", err)
}
}
func TestConvectiveOutlookNormalizerRejectsMissingLabel(t *testing.T) {
bundle := spcBundle(t, 0, 0)
bundle.Products[0].Body = json.RawMessage(strings.Replace(
string(bundle.Products[0].Body),
`"LABEL": "SLGT"`,
`"LABEL": ""`,
1,
))
_, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle))
if err == nil {
t.Fatalf("Normalize() error = nil, want error")
}
if !strings.Contains(err.Error(), "product day1_categorical feature 0.LABEL") {
t.Fatalf("error = %q, want label context", err)
}
}
func TestConvectiveOutlookNormalizerOutputJSONShape(t *testing.T) {
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, spcBundle(t, 38.5, -90.5)))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
raw, err := json.Marshal(out.Payload)
if err != nil {
t.Fatalf("Marshal(payload) error = %v", err)
}
got := string(raw)
for _, want := range []string{`"asOf"`, `"outlooks"`, `"containsLocation"`, `"geometry"`} {
if !strings.Contains(got, want) {
t.Fatalf("payload JSON missing %s: %s", want, got)
}
}
for _, unwanted := range []string{`"products"`, `"discussions"`, `"fetchedAt"`, `"body"`} {
if strings.Contains(got, unwanted) {
t.Fatalf("payload JSON exposed raw key %s: %s", unwanted, got)
}
}
}
func spcRawEvent(t *testing.T, bundle spcprovider.RawConvectiveOutlookBundle) event.Event {
t.Helper()
raw, err := json.Marshal(bundle)
if err != nil {
t.Fatalf("Marshal(bundle) error = %v", err)
}
effectiveAt := time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC)
return event.Event{
ID: "evt-spc-outlook-1",
Kind: event.Kind("outlook"),
Source: "spc-test",
EmittedAt: time.Date(2026, 6, 11, 20, 5, 0, 0, time.UTC),
EffectiveAt: &effectiveAt,
Schema: standards.SchemaRawSPCConvectiveOutlookV1,
Payload: json.RawMessage(raw),
}
}
func spcBundle(t *testing.T, latitude, longitude float64) spcprovider.RawConvectiveOutlookBundle {
t.Helper()
fetchedAt := time.Date(2026, 6, 11, 20, 0, 0, 0, time.UTC)
products := make([]spcprovider.RawOutlookProduct, 0, len(spcprovider.GeoJSONProducts()))
for _, product := range spcprovider.GeoJSONProducts() {
products = append(products, spcprovider.RawOutlookProduct{
Key: product.Key,
Day: product.Day,
OutlookType: product.OutlookType,
URL: "https://example.invalid/" + product.Key + ".geojson",
FetchedAt: fetchedAt,
Body: json.RawMessage(geoJSONFixtureForProduct(t, product.Key)),
})
}
return spcprovider.RawConvectiveOutlookBundle{
LocationID: "stl",
LocationName: "St. Louis, MO",
Latitude: latitude,
Longitude: longitude,
FetchedAt: fetchedAt,
Products: products,
Discussions: []spcprovider.RawDiscussionPage{
{Key: "day1", Day: 1, URL: "https://example.invalid/day1.html", FetchedAt: fetchedAt, Body: string(readSPCTestFixture(t, "day1_prt.html"))},
{Key: "day2", Day: 2, URL: "https://example.invalid/day2.html", FetchedAt: fetchedAt, Body: string(readSPCTestFixture(t, "day2_prt_corr.html"))},
{Key: "day3", Day: 3, URL: "https://example.invalid/day3.html", FetchedAt: fetchedAt, Body: string(readSPCTestFixture(t, "day3_prt.html"))},
},
}
}
func geoJSONFixtureForProduct(t *testing.T, key string) []byte {
t.Helper()
switch {
case strings.HasPrefix(key, "day1_"):
return readSPCTestFixture(t, "day1_cat.geojson")
case strings.HasPrefix(key, "day2_"):
return readSPCTestFixture(t, "day2_torn.geojson")
case strings.HasPrefix(key, "day3_"):
return readSPCTestFixture(t, "day3_wind.geojson")
default:
t.Fatalf("unknown product key %q", key)
return nil
}
}
func readSPCTestFixture(t *testing.T, name string) []byte {
t.Helper()
path := filepath.Join("..", "..", "providers", "spc", "testdata", name)
raw, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read fixture %s: %v", path, err)
}
return raw
}
func findOutlook(outlooks []model.WeatherOutlook, day int, outlookType string) *model.WeatherOutlook {
for i := range outlooks {
if outlooks[i].Day == day && outlooks[i].OutlookType == outlookType {
return &outlooks[i]
}
}
return nil
}
func assertTime(t *testing.T, name string, got time.Time, year int, month time.Month, day int, hour int, minute int, second int) {
t.Helper()
want := time.Date(year, month, day, hour, minute, second, 0, time.UTC)
if !got.Equal(want) {
t.Fatalf("%s = %s, want %s", name, got, want)
}
}

View File

@@ -0,0 +1,14 @@
package spc
import (
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
)
var builtins = []fknormalize.Normalizer{
ConvectiveOutlookNormalizer{},
}
// Register appends SPC normalizers in stable order.
func Register(in []fknormalize.Normalizer) []fknormalize.Normalizer {
return append(in, builtins...)
}

View File

@@ -0,0 +1,552 @@
package nws
import (
"fmt"
"html"
"regexp"
"strconv"
"strings"
"time"
)
type ForecastDiscussion struct {
OfficeID string
OfficeName string
Product string
IssuedAt time.Time
UpdatedAt *time.Time
KeyMessages []string
ShortTerm *ForecastDiscussionSection
LongTerm *ForecastDiscussionSection
}
type ForecastDiscussionSection struct {
Qualifier string
IssuedAt *time.Time
Text string
}
var (
forecastDiscussionHeaderRE = regexp.MustCompile(`^\.(KEY MESSAGES|SHORT TERM|LONG TERM|AVIATION)\.\.\.(.*)$`)
forecastDiscussionAFDRE = regexp.MustCompile(`^AFD([A-Z]{3})$`)
forecastDiscussionWMORE = regexp.MustCompile(`\bK([A-Z]{3})\b`)
forecastDiscussionSigRE = regexp.MustCompile(`^[A-Z]{2,6}$`)
)
func ParseForecastDiscussionHTML(raw string) (ForecastDiscussion, error) {
text, err := ExtractForecastDiscussionText(raw)
if err != nil {
return ForecastDiscussion{}, err
}
parsed, err := ParseForecastDiscussionText(text)
if err != nil {
return ForecastDiscussion{}, err
}
parsed.UpdatedAt = parseForecastDiscussionUpdatedAt(raw)
return parsed, nil
}
func ExtractForecastDiscussionText(raw string) (string, error) {
lower := strings.ToLower(raw)
searchFrom := 0
for {
openStart := strings.Index(lower[searchFrom:], "<pre")
if openStart < 0 {
return "", fmt.Errorf("missing <pre class=\"glossaryProduct\"> block")
}
openStart += searchFrom
openEnd := strings.Index(lower[openStart:], ">")
if openEnd < 0 {
return "", fmt.Errorf("unterminated <pre> tag")
}
openEnd += openStart
tag := lower[openStart : openEnd+1]
if isGlossaryProductTag(tag) {
closeStart := strings.Index(lower[openEnd+1:], "</pre>")
if closeStart < 0 {
return "", fmt.Errorf("missing closing </pre> for glossaryProduct block")
}
closeStart += openEnd + 1
text := html.UnescapeString(raw[openEnd+1 : closeStart])
text = strings.ReplaceAll(text, "\r\n", "\n")
text = strings.ReplaceAll(text, "\r", "\n")
return text, nil
}
searchFrom = openEnd + 1
}
}
func ParseForecastDiscussionText(text string) (ForecastDiscussion, error) {
lines := splitLines(text)
officeID := parseForecastDiscussionOfficeID(lines)
officeName, issuedAt, err := parseForecastDiscussionHeader(lines)
if err != nil {
return ForecastDiscussion{}, err
}
out := ForecastDiscussion{
OfficeID: officeID,
OfficeName: officeName,
Product: "afd",
IssuedAt: issuedAt.UTC(),
}
if block, ok := extractForecastDiscussionSection(lines, "KEY MESSAGES"); ok {
out.KeyMessages = parseForecastDiscussionKeyMessages(block)
}
if block, ok := extractForecastDiscussionSection(lines, "SHORT TERM"); ok {
section, err := parseForecastDiscussionTextSection(block)
if err != nil {
return ForecastDiscussion{}, fmt.Errorf("parse SHORT TERM: %w", err)
}
out.ShortTerm = &section
}
if block, ok := extractForecastDiscussionSection(lines, "LONG TERM"); ok {
section, err := parseForecastDiscussionTextSection(block)
if err != nil {
return ForecastDiscussion{}, fmt.Errorf("parse LONG TERM: %w", err)
}
out.LongTerm = &section
}
return out, nil
}
func isGlossaryProductTag(tag string) bool {
tag = strings.ToLower(tag)
return strings.Contains(tag, `class="glossaryproduct"`) ||
strings.Contains(tag, `class='glossaryproduct'`) ||
strings.Contains(tag, `class="glossaryproduct `) ||
strings.Contains(tag, `class='glossaryproduct `)
}
func parseForecastDiscussionUpdatedAt(raw string) *time.Time {
lower := strings.ToLower(raw)
searchFrom := 0
for {
metaStart := strings.Index(lower[searchFrom:], "<meta")
if metaStart < 0 {
return nil
}
metaStart += searchFrom
metaEnd := strings.Index(lower[metaStart:], ">")
if metaEnd < 0 {
return nil
}
metaEnd += metaStart
tag := raw[metaStart : metaEnd+1]
if !strings.EqualFold(strings.TrimSpace(extractHTMLAttr(tag, "name")), "DC.date.created") {
searchFrom = metaEnd + 1
continue
}
content := strings.TrimSpace(extractHTMLAttr(tag, "content"))
if content == "" {
return nil
}
t, err := ParseTime(content)
if err != nil {
return nil
}
tt := t.UTC()
return &tt
}
}
func extractHTMLAttr(tag, attr string) string {
lower := strings.ToLower(tag)
attrLower := strings.ToLower(attr)
for i := 0; i < len(lower); i++ {
idx := strings.Index(lower[i:], attrLower)
if idx < 0 {
return ""
}
idx += i
if idx > 0 {
prev := lower[idx-1]
if isAttrNameChar(prev) {
i = idx + len(attrLower)
continue
}
}
j := idx + len(attrLower)
for j < len(lower) && isHTMLSpace(lower[j]) {
j++
}
if j >= len(lower) || lower[j] != '=' {
i = idx + len(attrLower)
continue
}
j++
for j < len(lower) && isHTMLSpace(lower[j]) {
j++
}
if j >= len(tag) {
return ""
}
quote := tag[j]
if quote != '"' && quote != '\'' {
return ""
}
j++
k := j
for k < len(tag) && tag[k] != quote {
k++
}
if k >= len(tag) {
return ""
}
return html.UnescapeString(tag[j:k])
}
return ""
}
func isHTMLSpace(b byte) bool {
switch b {
case ' ', '\n', '\r', '\t', '\f':
return true
default:
return false
}
}
func isAttrNameChar(b byte) bool {
switch {
case b >= 'a' && b <= 'z':
return true
case b >= 'A' && b <= 'Z':
return true
case b >= '0' && b <= '9':
return true
case b == '-' || b == '_' || b == ':':
return true
default:
return false
}
}
func splitLines(text string) []string {
text = strings.ReplaceAll(text, "\r\n", "\n")
text = strings.ReplaceAll(text, "\r", "\n")
return strings.Split(text, "\n")
}
func parseForecastDiscussionOfficeID(lines []string) string {
for _, raw := range lines {
line := strings.TrimSpace(raw)
if m := forecastDiscussionAFDRE.FindStringSubmatch(line); len(m) == 2 {
return m[1]
}
}
for _, raw := range lines {
line := strings.TrimSpace(raw)
if m := forecastDiscussionWMORE.FindStringSubmatch(line); len(m) == 2 {
return m[1]
}
}
return ""
}
func parseForecastDiscussionHeader(lines []string) (string, time.Time, error) {
for i, raw := range lines {
line := strings.TrimSpace(raw)
if !strings.HasPrefix(line, "National Weather Service ") {
continue
}
officeName := line
for j := i + 1; j < len(lines); j++ {
tsLine := strings.TrimSpace(lines[j])
if tsLine == "" {
continue
}
issuedAt, err := parseForecastDiscussionIssueTime(tsLine)
if err != nil {
return "", time.Time{}, fmt.Errorf("parse bulletin issuedAt %q: %w", tsLine, err)
}
return officeName, issuedAt.UTC(), nil
}
return "", time.Time{}, fmt.Errorf("missing bulletin issue time after office line")
}
return "", time.Time{}, fmt.Errorf("missing office header")
}
func parseForecastDiscussionIssueTime(line string) (time.Time, error) {
line = strings.TrimSpace(line)
line = strings.TrimPrefix(line, "Issued at ")
line = strings.TrimSpace(line)
parts := strings.Fields(line)
if len(parts) != 7 {
return time.Time{}, fmt.Errorf("unexpected issue time format")
}
loc, err := forecastDiscussionLocation(parts[2])
if err != nil {
return time.Time{}, err
}
datePart, err := time.Parse("Mon Jan 2 2006", strings.Join(parts[3:], " "))
if err != nil {
return time.Time{}, err
}
hour, minute, err := parseForecastDiscussionClock(parts[0], parts[1])
if err != nil {
return time.Time{}, err
}
return time.Date(
datePart.Year(),
datePart.Month(),
datePart.Day(),
hour,
minute,
0,
0,
loc,
), nil
}
func parseForecastDiscussionClock(rawClock, rawAMPM string) (int, int, error) {
clock := strings.TrimSpace(rawClock)
ampm := strings.ToUpper(strings.TrimSpace(rawAMPM))
if ampm != "AM" && ampm != "PM" {
return 0, 0, fmt.Errorf("unexpected meridiem %q", rawAMPM)
}
n, err := strconv.Atoi(clock)
if err != nil {
return 0, 0, fmt.Errorf("invalid clock %q", rawClock)
}
hour := n
minute := 0
if len(clock) >= 3 {
hour = n / 100
minute = n % 100
}
if hour < 1 || hour > 12 {
return 0, 0, fmt.Errorf("invalid hour %q", rawClock)
}
if minute < 0 || minute > 59 {
return 0, 0, fmt.Errorf("invalid minute %q", rawClock)
}
if ampm == "AM" {
if hour == 12 {
hour = 0
}
return hour, minute, nil
}
if hour != 12 {
hour += 12
}
return hour, minute, nil
}
func forecastDiscussionLocation(abbrev string) (*time.Location, error) {
offsets := map[string]int{
"AST": -4 * 3600,
"ADT": -3 * 3600,
"EST": -5 * 3600,
"EDT": -4 * 3600,
"CST": -6 * 3600,
"CDT": -5 * 3600,
"MST": -7 * 3600,
"MDT": -6 * 3600,
"PST": -8 * 3600,
"PDT": -7 * 3600,
"AKST": -9 * 3600,
"AKDT": -8 * 3600,
"HST": -10 * 3600,
"UTC": 0,
"GMT": 0,
}
abbr := strings.ToUpper(strings.TrimSpace(abbrev))
offset, ok := offsets[abbr]
if !ok {
return nil, fmt.Errorf("unsupported time zone %q", abbrev)
}
return time.FixedZone(abbr, offset), nil
}
func extractForecastDiscussionSection(lines []string, section string) ([]string, bool) {
target := "." + section + "..."
for i, raw := range lines {
line := strings.TrimSpace(raw)
if !strings.HasPrefix(line, target) {
continue
}
out := []string{line}
for j := i + 1; j < len(lines); j++ {
next := strings.TrimSpace(lines[j])
if next == "&&" || next == "$$" || strings.Contains(next, "WATCHES/WARNINGS/ADVISORIES") {
break
}
if j > i+1 && isForecastDiscussionSectionHeader(next) {
break
}
out = append(out, lines[j])
}
return out, true
}
return nil, false
}
func isForecastDiscussionSectionHeader(line string) bool {
return forecastDiscussionHeaderRE.MatchString(strings.TrimSpace(line))
}
func parseForecastDiscussionKeyMessages(block []string) []string {
if len(block) <= 1 {
return nil
}
body := trimBlankLines(block[1:])
var messages []string
var current strings.Builder
flush := func() {
msg := strings.TrimSpace(current.String())
if msg != "" {
messages = append(messages, msg)
}
current.Reset()
}
for _, raw := range body {
line := strings.TrimSpace(raw)
if line == "" {
continue
}
if strings.HasPrefix(line, "-") {
flush()
line = strings.TrimSpace(strings.TrimPrefix(line, "-"))
current.WriteString(line)
continue
}
if current.Len() > 0 {
current.WriteByte(' ')
}
current.WriteString(line)
}
flush()
return messages
}
func parseForecastDiscussionTextSection(block []string) (ForecastDiscussionSection, error) {
if len(block) == 0 {
return ForecastDiscussionSection{}, fmt.Errorf("empty section")
}
section := ForecastDiscussionSection{
Qualifier: parseForecastDiscussionQualifier(strings.TrimSpace(block[0])),
}
body := trimBlankLines(block[1:])
if len(body) == 0 {
return section, nil
}
first := strings.TrimSpace(body[0])
if strings.HasPrefix(first, "Issued at ") {
issuedAt, err := parseForecastDiscussionIssueTime(first)
if err != nil {
return ForecastDiscussionSection{}, fmt.Errorf("parse section issuedAt %q: %w", first, err)
}
tt := issuedAt.UTC()
section.IssuedAt = &tt
body = trimBlankLines(body[1:])
}
body = trimForecastDiscussionSignatureLines(body)
section.Text = joinForecastDiscussionParagraphs(body)
return section, nil
}
func parseForecastDiscussionQualifier(header string) string {
m := forecastDiscussionHeaderRE.FindStringSubmatch(header)
if len(m) != 3 {
return ""
}
return strings.TrimSpace(m[2])
}
func trimBlankLines(lines []string) []string {
start := 0
for start < len(lines) && strings.TrimSpace(lines[start]) == "" {
start++
}
end := len(lines)
for end > start && strings.TrimSpace(lines[end-1]) == "" {
end--
}
return lines[start:end]
}
func trimForecastDiscussionSignatureLines(lines []string) []string {
lines = trimBlankLines(lines)
for len(lines) > 0 {
last := strings.TrimSpace(lines[len(lines)-1])
if last == "" {
lines = lines[:len(lines)-1]
continue
}
if forecastDiscussionSigRE.MatchString(last) {
lines = trimBlankLines(lines[:len(lines)-1])
continue
}
break
}
return lines
}
func joinForecastDiscussionParagraphs(lines []string) string {
lines = trimBlankLines(lines)
if len(lines) == 0 {
return ""
}
var paragraphs []string
current := make([]string, 0, len(lines))
flush := func() {
if len(current) == 0 {
return
}
paragraphs = append(paragraphs, strings.Join(current, " "))
current = current[:0]
}
for _, raw := range lines {
line := strings.TrimSpace(raw)
if line == "" {
flush()
continue
}
current = append(current, line)
}
flush()
return strings.Join(paragraphs, "\n\n")
}

View File

@@ -0,0 +1,118 @@
package nws
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestParseForecastDiscussionHTMLParsesExpectedFields(t *testing.T) {
raw := loadForecastDiscussionSampleHTML(t)
got, err := ParseForecastDiscussionHTML(raw)
if err != nil {
t.Fatalf("ParseForecastDiscussionHTML() error = %v", err)
}
if got.OfficeID != "LSX" {
t.Fatalf("OfficeID = %q, want LSX", got.OfficeID)
}
if got.OfficeName != "National Weather Service Saint Louis MO" {
t.Fatalf("OfficeName = %q", got.OfficeName)
}
if got.Product != "afd" {
t.Fatalf("Product = %q, want afd", got.Product)
}
wantIssuedAt := time.Date(2026, 3, 28, 19, 24, 0, 0, time.UTC)
if !got.IssuedAt.Equal(wantIssuedAt) {
t.Fatalf("IssuedAt = %s, want %s", got.IssuedAt.Format(time.RFC3339), wantIssuedAt.Format(time.RFC3339))
}
wantUpdatedAt := time.Date(2026, 3, 28, 20, 29, 47, 0, time.UTC)
if got.UpdatedAt == nil || !got.UpdatedAt.Equal(wantUpdatedAt) {
t.Fatalf("UpdatedAt = %v, want %s", got.UpdatedAt, wantUpdatedAt.Format(time.RFC3339))
}
wantMessages := []string{
"Elevated fire danger conditions are expected across a broad area tomorrow afternoon due to breezy southwest winds and low humidity.",
"Very warm temperatures are expected once again Monday and Tuesday, with highs well into the 80s.",
"A cold front late Tuesday or early Wednesday brings our next chance of thunderstorms, followed by a cooldown and possibly more chances for rain later in the week.",
}
if len(got.KeyMessages) != len(wantMessages) {
t.Fatalf("KeyMessages len = %d, want %d", len(got.KeyMessages), len(wantMessages))
}
for i := range wantMessages {
if got.KeyMessages[i] != wantMessages[i] {
t.Fatalf("KeyMessages[%d] = %q, want %q", i, got.KeyMessages[i], wantMessages[i])
}
}
if got.ShortTerm == nil {
t.Fatalf("ShortTerm is nil")
}
if got.ShortTerm.Qualifier != "(Through Late Sunday Night)" {
t.Fatalf("ShortTerm.Qualifier = %q", got.ShortTerm.Qualifier)
}
if got.ShortTerm.IssuedAt == nil || !got.ShortTerm.IssuedAt.Equal(time.Date(2026, 3, 28, 19, 19, 0, 0, time.UTC)) {
t.Fatalf("ShortTerm.IssuedAt = %v", got.ShortTerm.IssuedAt)
}
if !strings.Contains(got.ShortTerm.Text, "After a chilly morning") {
t.Fatalf("ShortTerm.Text missing expected prose: %q", got.ShortTerm.Text)
}
if strings.Contains(got.ShortTerm.Text, "BRC") {
t.Fatalf("ShortTerm.Text should not include signature: %q", got.ShortTerm.Text)
}
if strings.Contains(got.ShortTerm.Text, "\n\n\n") {
t.Fatalf("ShortTerm.Text contains unexpected paragraph breaks: %q", got.ShortTerm.Text)
}
if got.LongTerm == nil {
t.Fatalf("LongTerm is nil")
}
if got.LongTerm.Qualifier != "(Monday through Next Saturday)" {
t.Fatalf("LongTerm.Qualifier = %q", got.LongTerm.Qualifier)
}
if got.LongTerm.IssuedAt == nil || !got.LongTerm.IssuedAt.Equal(time.Date(2026, 3, 28, 19, 19, 0, 0, time.UTC)) {
t.Fatalf("LongTerm.IssuedAt = %v", got.LongTerm.IssuedAt)
}
if !strings.Contains(got.LongTerm.Text, "The peak of the warmth arrives Monday and Tuesday") {
t.Fatalf("LongTerm.Text missing expected prose: %q", got.LongTerm.Text)
}
if strings.Contains(got.LongTerm.Text, "AVIATION") || strings.Contains(got.LongTerm.Text, "WATCHES/WARNINGS/ADVISORIES") {
t.Fatalf("LongTerm.Text includes content from other sections: %q", got.LongTerm.Text)
}
}
func TestParseForecastDiscussionHTMLMissingPreBlock(t *testing.T) {
_, err := ParseForecastDiscussionHTML("<html><body><div>no pre block</div></body></html>")
if err == nil {
t.Fatalf("ParseForecastDiscussionHTML() error = nil, want error")
}
if !strings.Contains(err.Error(), "glossaryProduct") {
t.Fatalf("error = %q, want glossaryProduct context", err)
}
}
func TestParseForecastDiscussionTextMissingIssueTime(t *testing.T) {
_, err := ParseForecastDiscussionText("National Weather Service Saint Louis MO\n\n.KEY MESSAGES...\n- Test")
if err == nil {
t.Fatalf("ParseForecastDiscussionText() error = nil, want error")
}
if !strings.Contains(err.Error(), "issue time") {
t.Fatalf("error = %q, want issue time context", err)
}
}
func loadForecastDiscussionSampleHTML(t *testing.T) string {
t.Helper()
path := filepath.Join("testdata", "forecast_discussion_sample.html")
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("os.ReadFile(%q) error = %v", path, err)
}
return string(b)
}

View File

@@ -0,0 +1,84 @@
<!DOCTYPE html>
<html class="no-js">
<head>
<meta name="DC.date.created" scheme="ISO8601" content="2026-03-28T20:29:47+00:00" />
<title>National Weather Service</title>
</head>
<body>
<pre class="glossaryProduct">
988
FXUS63 KLSX 281924
AFDLSX
Area Forecast Discussion
National Weather Service Saint Louis MO
224 PM CDT Sat Mar 28 2026
.KEY MESSAGES...
- Elevated fire danger conditions are expected across a broad area
tomorrow afternoon due to breezy southwest winds and low
humidity.
- Very warm temperatures are expected once again Monday and
Tuesday, with highs well into the 80s.
- A cold front late Tuesday or early Wednesday brings our next
chance of thunderstorms, followed by a cooldown and possibly
more chances for rain later in the week.
&&
.SHORT TERM... (Through Late Sunday Night)
Issued at 219 PM CDT Sat Mar 28 2026
After a chilly morning that saw widespread freezing temperatures,
another warmup is in store over the next several days as southerly
winds become re-established. We will also see the return of
shower/thunderstorm chances Tuesday onward as we enter a more
unsettled pattern.
In the near-term, the focus continues to be on some lingering fire
weather potential thanks to the presence of an exceptionally dry
airmass.
BRC
&&
.LONG TERM... (Monday through Next Saturday)
Issued at 219 PM CDT Sat Mar 28 2026
The peak of the warmth arrives Monday and Tuesday, as a broad, but
low-amplitude ridge nudges eastward and steady warm/moist advection
continues on both days.
Wednesday onward, the day-to-day details become much less clear, but
latest trends suggest that an active/wet pattern will likely
continue as another more substantial trough follows with additional
chances for showers/thunderstorms late in the week.
BRC
&&
.AVIATION... (For the 18z TAFs through 18z Sunday Afternoon)
Issued at 1133 AM CDT Sat Mar 28 2026
VFR conditions are expected throughout the 18Z TAF period.
BRC
&&
.LSX WATCHES/WARNINGS/ADVISORIES...
MO...None.
IL...None.
&&
$$
WFO LSX
</pre>
</body>
</html>

View File

@@ -1,5 +1,5 @@
// Package openweather contains provider-specific helper code for OpenWeather used by
// both sources and normalizers.
// Package openweather contains provider-specific helper code for OpenWeather
// used by sources and normalizers.
//
// Rules:
// - No network I/O here.

View File

@@ -0,0 +1,188 @@
package spc
import (
"fmt"
"html"
"regexp"
"strings"
"time"
)
var (
scriptBlockRE = regexp.MustCompile(`(?is)<script\b[^>]*>.*?</script>`)
preBlockRE = regexp.MustCompile(`(?is)<pre\b[^>]*>(.*?)</pre>`)
tagRE = regexp.MustCompile(`(?is)<[^>]+>`)
updatedRE = regexp.MustCompile(`(?im)^\s*Updated:\s*(.+?)\s*$`)
pageUpdatedRE = regexp.MustCompile(`(?i)\bUpdated:\s*((?:\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z)|(?:[A-Z][a-z]{2}\s+[A-Z][a-z]{2}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}\s+UTC\s+\d{4})|(?:\d{4}\s+UTC\s+[A-Z][a-z]{2}\s+[A-Z][a-z]{2}\s+\d{1,2}\s+\d{4})|(?:\d{4}Z\s+[A-Z][a-z]{2}\s+[A-Z][a-z]{2}\s+\d{1,2}\s+\d{4}))`)
productCodeRE = regexp.MustCompile(`(?i)^SPC\s+AC\s+\d+\s*$`)
sectionRE = regexp.MustCompile(`^\s*\.\.\.[A-Z0-9 /-]+\.{3}\s*$`)
)
// DiscussionText contains parsed text from an SPC print page.
type DiscussionText struct {
ProductTitle string
Headline string
Summary string
Discussion string
UpdatedAt *time.Time
}
// ExtractProductText extracts and cleans the first useful preformatted SPC
// product text block from a print page.
func ExtractProductText(rawHTML string) (string, error) {
matches := preBlockRE.FindAllStringSubmatch(rawHTML, -1)
for _, match := range matches {
if len(match) < 2 {
continue
}
text := cleanHTMLText(match[1])
if strings.TrimSpace(text) != "" {
return text, nil
}
}
return "", fmt.Errorf("no useful pre block found")
}
// ParseDiscussionHTML extracts SPC product text and page-level metadata from a
// print-page HTML document.
func ParseDiscussionHTML(rawHTML string) (DiscussionText, error) {
text, err := ExtractProductText(rawHTML)
if err != nil {
return DiscussionText{}, err
}
parsed := ParseDiscussionText(text)
if updatedAt := ParsePageUpdatedTimestamp(rawHTML); updatedAt != nil {
parsed.UpdatedAt = updatedAt
}
return parsed, nil
}
// ParseDiscussionText extracts common SPC narrative metadata from cleaned
// product text.
func ParseDiscussionText(text string) DiscussionText {
text = trimBlankLines(normalizeNewlines(text))
title := ParseProductTitle(text)
return DiscussionText{
ProductTitle: title,
Headline: title,
Summary: ExtractSummary(text),
Discussion: text,
UpdatedAt: ParseUpdatedTimestamp(text),
}
}
// ParsePageUpdatedTimestamp parses the page-level Updated row from an SPC print
// page. SPC currently places this outside the product <pre> block.
func ParsePageUpdatedTimestamp(rawHTML string) *time.Time {
text := cleanHTMLText(rawHTML)
text = strings.ReplaceAll(text, "\u00a0", " ")
text = strings.Join(strings.Fields(text), " ")
match := pageUpdatedRE.FindStringSubmatch(text)
if len(match) != 2 {
return nil
}
return parseUpdatedValue(match[1])
}
// ParseUpdatedTimestamp parses an SPC print-page Updated line when present.
func ParseUpdatedTimestamp(text string) *time.Time {
match := updatedRE.FindStringSubmatch(normalizeNewlines(text))
if len(match) != 2 {
return nil
}
return parseUpdatedValue(match[1])
}
// ParseProductTitle returns the first non-empty product line from cleaned text.
func ParseProductTitle(text string) string {
for _, line := range strings.Split(normalizeNewlines(text), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "Updated:") || productCodeRE.MatchString(line) {
continue
}
return line
}
return ""
}
// ParseHeadline returns the human-facing headline from cleaned text.
func ParseHeadline(text string) string {
return ParseProductTitle(text)
}
// ExtractSummary returns text under the ...SUMMARY... section through the next
// SPC section heading.
func ExtractSummary(text string) string {
lines := strings.Split(normalizeNewlines(text), "\n")
start := -1
for i, line := range lines {
if strings.EqualFold(strings.TrimSpace(line), "...SUMMARY...") {
start = i + 1
break
}
}
if start < 0 {
return ""
}
var out []string
for _, line := range lines[start:] {
if sectionRE.MatchString(line) {
break
}
out = append(out, line)
}
return trimBlankLines(strings.Join(out, "\n"))
}
func cleanHTMLText(raw string) string {
raw = scriptBlockRE.ReplaceAllString(raw, "")
raw = tagRE.ReplaceAllString(raw, "")
raw = html.UnescapeString(raw)
raw = normalizeNewlines(raw)
return trimBlankLines(raw)
}
func normalizeNewlines(text string) string {
text = strings.ReplaceAll(text, "\r\n", "\n")
text = strings.ReplaceAll(text, "\r", "\n")
return text
}
func trimBlankLines(text string) string {
lines := strings.Split(normalizeNewlines(text), "\n")
start := 0
for start < len(lines) && strings.TrimSpace(lines[start]) == "" {
start++
}
end := len(lines)
for end > start && strings.TrimSpace(lines[end-1]) == "" {
end--
}
return strings.Join(lines[start:end], "\n")
}
func parseUpdatedValue(value string) *time.Time {
value = strings.TrimSpace(value)
if value == "" {
return nil
}
if t := parseOptionalISOTimestamp(value); t != nil {
return t
}
for _, layout := range []string{
"Mon Jan 2 15:04:05 UTC 2006",
"1504 UTC Mon Jan 2 2006",
"1504Z Mon Jan 2 2006",
"3:04 PM UTC Mon Jan 2 2006",
time.RFC1123,
time.RFC1123Z,
} {
t, err := time.Parse(layout, value)
if err == nil {
tt := t.UTC()
return &tt
}
}
return nil
}

View File

@@ -0,0 +1,109 @@
package spc
import (
"strings"
"testing"
"time"
)
func TestExtractProductTextCleansPreBlock(t *testing.T) {
raw := string(readTestFile(t, "day1_prt.html"))
got, err := ExtractProductText(raw)
if err != nil {
t.Fatalf("ExtractProductText() error = %v", err)
}
if strings.Contains(got, "<script") || strings.Contains(got, "<pre") {
t.Fatalf("ExtractProductText() retained HTML: %q", got)
}
if strings.Contains(got, "ignore me") {
t.Fatalf("ExtractProductText() retained script content: %q", got)
}
if !strings.Contains(got, "Day 1 Convective Outlook") {
t.Fatalf("ExtractProductText() missing headline: %q", got)
}
if !strings.HasPrefix(got, "SPC AC 111234") {
t.Fatalf("ExtractProductText() = %q, want product code prefix", got)
}
if strings.HasPrefix(got, "\n") || strings.HasSuffix(got, "\n") {
t.Fatalf("ExtractProductText() retained surrounding blank lines: %q", got)
}
}
func TestParseDiscussionHTMLExtractsHeadlineSummaryAndUpdated(t *testing.T) {
got, err := ParseDiscussionHTML(string(readTestFile(t, "day1_prt.html")))
if err != nil {
t.Fatalf("ParseDiscussionHTML() error = %v", err)
}
if got.ProductTitle != "Day 1 Convective Outlook" {
t.Fatalf("ProductTitle = %q", got.ProductTitle)
}
if got.Headline != "Day 1 Convective Outlook" {
t.Fatalf("Headline = %q", got.Headline)
}
wantSummary := "Severe thunderstorms are possible across parts of the central Plains\nand mid Mississippi Valley this afternoon and evening."
if got.Summary != wantSummary {
t.Fatalf("Summary = %q, want %q", got.Summary, wantSummary)
}
if !strings.Contains(got.Discussion, "...DISCUSSION...") {
t.Fatalf("Discussion missing full text: %q", got.Discussion)
}
if !strings.HasPrefix(got.Discussion, "SPC AC 111234") {
t.Fatalf("Discussion = %q, want product code prefix", got.Discussion)
}
wantUpdated := time.Date(2026, 6, 11, 12, 45, 0, 0, time.UTC)
if got.UpdatedAt == nil || !got.UpdatedAt.Equal(wantUpdated) {
t.Fatalf("UpdatedAt = %v, want %s", got.UpdatedAt, wantUpdated)
}
}
func TestParseProductTitleSkipsSPCProductCode(t *testing.T) {
got := ParseProductTitle("SPC AC 101959\nDay 1 Convective Outlook\nNWS Storm Prediction Center Norman OK")
if got != "Day 1 Convective Outlook" {
t.Fatalf("ParseProductTitle() = %q, want Day 1 Convective Outlook", got)
}
}
func TestParseDiscussionTextPreservesCorrectionMarker(t *testing.T) {
got, err := ParseDiscussionHTML(string(readTestFile(t, "day2_prt_corr.html")))
if err != nil {
t.Fatalf("ParseDiscussionHTML() error = %v", err)
}
if !strings.Contains(got.Headline, "CORR 1") {
t.Fatalf("Headline = %q, want correction marker", got.Headline)
}
if !strings.Contains(got.Discussion, "CORR 1") {
t.Fatalf("Discussion = %q, want correction marker", got.Discussion)
}
wantUpdated := time.Date(2026, 6, 11, 17, 30, 0, 0, time.UTC)
if got.UpdatedAt == nil || !got.UpdatedAt.Equal(wantUpdated) {
t.Fatalf("UpdatedAt = %v, want %s", got.UpdatedAt, wantUpdated)
}
}
func TestParseUpdatedTimestampReturnsNilWhenAbsent(t *testing.T) {
text, err := ExtractProductText(string(readTestFile(t, "day2_prt_corr.html")))
if err != nil {
t.Fatalf("ExtractProductText() error = %v", err)
}
if got := ParseUpdatedTimestamp(text); got != nil {
t.Fatalf("ParseUpdatedTimestamp() = %v, want nil", got)
}
}
func TestParseUpdatedTimestampAcceptsSPCUTCFormat(t *testing.T) {
got := ParseUpdatedTimestamp("Updated: 1945 UTC Thu Jun 11 2026")
want := time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC)
if got == nil || !got.Equal(want) {
t.Fatalf("ParseUpdatedTimestamp() = %v, want %s", got, want)
}
}
func TestParsePageUpdatedTimestampAcceptsLiveSPCShape(t *testing.T) {
got := ParsePageUpdatedTimestamp(string(readTestFile(t, "day3_prt.html")))
want := time.Date(2026, 6, 11, 20, 0, 0, 0, time.UTC)
if got == nil || !got.Equal(want) {
t.Fatalf("ParsePageUpdatedTimestamp() = %v, want %s", got, want)
}
}

View File

@@ -0,0 +1,8 @@
// Package spc contains deterministic helper code for Storm Prediction Center
// products used by sources and normalizers.
//
// Rules:
// - No network I/O here.
// - Keep helpers deterministic and easy to unit test.
// - Preserve upstream payload fragments needed for canonical mapping.
package spc

View File

@@ -0,0 +1,105 @@
package spc
import (
"bytes"
"encoding/json"
"fmt"
"strconv"
"strings"
)
// GeoJSONFeatureCollection is the minimal SPC outlook FeatureCollection shape
// needed by weatherfeeder.
type GeoJSONFeatureCollection struct {
Type string `json:"type"`
Features []GeoJSONFeature `json:"features"`
}
// GeoJSONFeature preserves typed SPC properties and compact raw geometry.
type GeoJSONFeature struct {
Type string `json:"type"`
Properties GeoJSONProperties `json:"properties"`
Geometry json.RawMessage `json:"geometry"`
}
// GeoJSONProperties contains the SPC fields used by canonical mapping.
type GeoJSONProperties struct {
ValidISO string `json:"VALID_ISO"`
ExpireISO string `json:"EXPIRE_ISO"`
IssueISO string `json:"ISSUE_ISO"`
Forecaster string `json:"FORECASTER"`
Label string `json:"LABEL"`
Label2 string `json:"LABEL2"`
DN *int `json:"DN"`
}
// DecodeGeoJSON decodes an SPC GeoJSON outlook product and compacts feature
// geometry JSON for stable downstream storage.
func DecodeGeoJSON(raw []byte) (GeoJSONFeatureCollection, error) {
var collection GeoJSONFeatureCollection
if err := json.Unmarshal(raw, &collection); err != nil {
return GeoJSONFeatureCollection{}, fmt.Errorf("decode geojson: %w", err)
}
for i := range collection.Features {
geom, err := compactJSON(collection.Features[i].Geometry)
if err != nil {
return GeoJSONFeatureCollection{}, fmt.Errorf("features[%d].geometry: %w", i, err)
}
collection.Features[i].Geometry = geom
}
return collection, nil
}
func (p *GeoJSONProperties) UnmarshalJSON(raw []byte) error {
type alias GeoJSONProperties
var aux struct {
alias
DN any `json:"DN"`
}
if err := json.Unmarshal(raw, &aux); err != nil {
return err
}
*p = GeoJSONProperties(aux.alias)
dn, err := parseSeverityRank(aux.DN)
if err != nil {
return err
}
p.DN = dn
return nil
}
func parseSeverityRank(value any) (*int, error) {
switch v := value.(type) {
case nil:
return nil, nil
case float64:
rank := int(v)
if float64(rank) != v {
return nil, fmt.Errorf("DN must be an integer, got %v", v)
}
return &rank, nil
case string:
v = strings.TrimSpace(v)
if v == "" {
return nil, nil
}
rank, err := strconv.Atoi(v)
if err != nil {
return nil, fmt.Errorf("DN must be an integer, got %q", v)
}
return &rank, nil
default:
return nil, fmt.Errorf("DN must be an integer or string, got %T", value)
}
}
func compactJSON(raw json.RawMessage) (json.RawMessage, error) {
if len(raw) == 0 {
return nil, fmt.Errorf("missing")
}
var buf bytes.Buffer
if err := json.Compact(&buf, raw); err != nil {
return nil, err
}
return json.RawMessage(buf.Bytes()), nil
}

View File

@@ -0,0 +1,90 @@
package spc
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestDecodeGeoJSONExposesSPCPropertiesAndCompactGeometry(t *testing.T) {
raw := readTestFile(t, "day1_cat.geojson")
got, err := DecodeGeoJSON(raw)
if err != nil {
t.Fatalf("DecodeGeoJSON() error = %v", err)
}
if got.Type != "FeatureCollection" {
t.Fatalf("Type = %q, want FeatureCollection", got.Type)
}
if len(got.Features) != 1 {
t.Fatalf("Features length = %d, want 1", len(got.Features))
}
feature := got.Features[0]
props := feature.Properties
if props.ValidISO != "2026-06-11T13:00:00Z" {
t.Fatalf("VALID_ISO = %q", props.ValidISO)
}
if props.ExpireISO != "2026-06-12T12:00:00Z" {
t.Fatalf("EXPIRE_ISO = %q", props.ExpireISO)
}
if props.IssueISO != "2026-06-11T12:34:56Z" {
t.Fatalf("ISSUE_ISO = %q", props.IssueISO)
}
if props.Forecaster != "SMITH" {
t.Fatalf("FORECASTER = %q", props.Forecaster)
}
if props.Label != "SLGT" {
t.Fatalf("LABEL = %q", props.Label)
}
if props.Label2 != "Slight Risk" {
t.Fatalf("LABEL2 = %q", props.Label2)
}
if props.DN == nil || *props.DN != 3 {
t.Fatalf("DN = %v, want 3", props.DN)
}
wantGeometry := `{"type":"Polygon","coordinates":[[[-91.0,38.0],[-90.0,38.0],[-90.0,39.0],[-91.0,39.0],[-91.0,38.0]]]}`
if string(feature.Geometry) != wantGeometry {
t.Fatalf("Geometry = %s, want %s", feature.Geometry, wantGeometry)
}
if strings.Contains(string(feature.Geometry), "\n") || strings.Contains(string(feature.Geometry), " ") {
t.Fatalf("Geometry is not compact: %q", feature.Geometry)
}
}
func TestDecodeGeoJSONParsesSeverityRankString(t *testing.T) {
raw := readTestFile(t, "day2_torn.geojson")
got, err := DecodeGeoJSON(raw)
if err != nil {
t.Fatalf("DecodeGeoJSON() error = %v", err)
}
props := got.Features[0].Properties
if props.DN == nil || *props.DN != 5 {
t.Fatalf("DN = %v, want 5", props.DN)
}
}
func TestParseISOTimestampTrimsAndReturnsUTC(t *testing.T) {
got, err := ParseISOTimestamp(" 2026-06-11T12:34:56Z ")
if err != nil {
t.Fatalf("ParseISOTimestamp() error = %v", err)
}
want := time.Date(2026, 6, 11, 12, 34, 56, 0, time.UTC)
if !got.Equal(want) {
t.Fatalf("ParseISOTimestamp() = %s, want %s", got, want)
}
}
func readTestFile(t *testing.T, name string) []byte {
t.Helper()
path := filepath.Join("testdata", name)
raw, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read %s: %v", path, err)
}
return raw
}

View File

@@ -0,0 +1,89 @@
package spc
import "fmt"
const (
OutlookTypeCategorical = "categorical"
OutlookTypeTornado = "tornado"
OutlookTypeHail = "hail"
OutlookTypeWind = "wind"
)
// GeoJSONProduct describes one required SPC convective outlook GeoJSON product.
type GeoJSONProduct struct {
Key string
Day int
OutlookType string
URL string
}
// DiscussionProduct describes one required SPC convective outlook print page.
type DiscussionProduct struct {
Key string
Day int
URL string
}
var geoJSONProducts = []GeoJSONProduct{
{Key: "day1_categorical", Day: 1, OutlookType: OutlookTypeCategorical, URL: "https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson"},
{Key: "day1_tornado", Day: 1, OutlookType: OutlookTypeTornado, URL: "https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson"},
{Key: "day1_hail", Day: 1, OutlookType: OutlookTypeHail, URL: "https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson"},
{Key: "day1_wind", Day: 1, OutlookType: OutlookTypeWind, URL: "https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson"},
{Key: "day2_categorical", Day: 2, OutlookType: OutlookTypeCategorical, URL: "https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson"},
{Key: "day2_tornado", Day: 2, OutlookType: OutlookTypeTornado, URL: "https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson"},
{Key: "day2_hail", Day: 2, OutlookType: OutlookTypeHail, URL: "https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson"},
{Key: "day2_wind", Day: 2, OutlookType: OutlookTypeWind, URL: "https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson"},
{Key: "day3_categorical", Day: 3, OutlookType: OutlookTypeCategorical, URL: "https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson"},
{Key: "day3_tornado", Day: 3, OutlookType: OutlookTypeTornado, URL: "https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson"},
{Key: "day3_hail", Day: 3, OutlookType: OutlookTypeHail, URL: "https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson"},
{Key: "day3_wind", Day: 3, OutlookType: OutlookTypeWind, URL: "https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson"},
}
var discussionProducts = []DiscussionProduct{
{Key: "day1", Day: 1, URL: "https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html"},
{Key: "day2", Day: 2, URL: "https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html"},
{Key: "day3", Day: 3, URL: "https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html"},
}
// GeoJSONProducts returns the required SPC convective outlook GeoJSON products
// in stable day/type order.
func GeoJSONProducts() []GeoJSONProduct {
out := make([]GeoJSONProduct, len(geoJSONProducts))
copy(out, geoJSONProducts)
return out
}
// DiscussionProducts returns the required SPC convective outlook print pages in
// stable day order.
func DiscussionProducts() []DiscussionProduct {
out := make([]DiscussionProduct, len(discussionProducts))
copy(out, discussionProducts)
return out
}
// GeoJSONProductByKey returns product metadata for a configured product key.
func GeoJSONProductByKey(key string) (GeoJSONProduct, bool) {
for _, product := range geoJSONProducts {
if product.Key == key {
return product, true
}
}
return GeoJSONProduct{}, false
}
// DiscussionProductByKey returns discussion metadata for a configured day key.
func DiscussionProductByKey(key string) (DiscussionProduct, bool) {
for _, product := range discussionProducts {
if product.Key == key {
return product, true
}
}
return DiscussionProduct{}, false
}
func validateProductDay(day int) error {
if day < 1 || day > 3 {
return fmt.Errorf("day must be 1, 2, or 3, got %d", day)
}
return nil
}

View File

@@ -0,0 +1,56 @@
package spc
import "testing"
func TestGeoJSONProductsStableOrder(t *testing.T) {
got := GeoJSONProducts()
if len(got) != 12 {
t.Fatalf("GeoJSONProducts() length = %d, want 12", len(got))
}
wantKeys := []string{
"day1_categorical",
"day1_tornado",
"day1_hail",
"day1_wind",
"day2_categorical",
"day2_tornado",
"day2_hail",
"day2_wind",
"day3_categorical",
"day3_tornado",
"day3_hail",
"day3_wind",
}
for i, want := range wantKeys {
if got[i].Key != want {
t.Fatalf("GeoJSONProducts()[%d].Key = %q, want %q", i, got[i].Key, want)
}
if err := validateProductDay(got[i].Day); err != nil {
t.Fatalf("GeoJSONProducts()[%d].Day invalid: %v", i, err)
}
if got[i].URL == "" {
t.Fatalf("GeoJSONProducts()[%d].URL is empty", i)
}
}
}
func TestDiscussionProductsStableOrder(t *testing.T) {
got := DiscussionProducts()
if len(got) != 3 {
t.Fatalf("DiscussionProducts() length = %d, want 3", len(got))
}
wantKeys := []string{"day1", "day2", "day3"}
for i, want := range wantKeys {
if got[i].Key != want {
t.Fatalf("DiscussionProducts()[%d].Key = %q, want %q", i, got[i].Key, want)
}
if got[i].Day != i+1 {
t.Fatalf("DiscussionProducts()[%d].Day = %d, want %d", i, got[i].Day, i+1)
}
if got[i].URL == "" {
t.Fatalf("DiscussionProducts()[%d].URL is empty", i)
}
}
}

View File

@@ -0,0 +1,45 @@
package spc
import (
"encoding/json"
"time"
)
// RawConvectiveOutlookBundle is the provider payload shape for SPC convective
// outlook fetch bundles.
type RawConvectiveOutlookBundle struct {
LocationID string `json:"locationId,omitempty"`
LocationName string `json:"locationName,omitempty"`
Latitude float64 `json:"latitude"`
Longitude float64 `json:"longitude"`
FetchedAt time.Time `json:"fetchedAt"`
Products []RawOutlookProduct `json:"products"`
Discussions []RawDiscussionPage `json:"discussions"`
RSS *RawRSSFeed `json:"rss,omitempty"`
}
// RawOutlookProduct contains one fetched SPC GeoJSON product.
type RawOutlookProduct struct {
Key string `json:"key"`
Day int `json:"day"`
OutlookType string `json:"outlookType"`
URL string `json:"url"`
FetchedAt time.Time `json:"fetchedAt"`
Body json.RawMessage `json:"body"`
}
// RawDiscussionPage contains one fetched SPC print page.
type RawDiscussionPage struct {
Key string `json:"key"`
Day int `json:"day"`
URL string `json:"url"`
FetchedAt time.Time `json:"fetchedAt"`
Body string `json:"body"`
}
// RawRSSFeed contains optional fetched SPC RSS metadata.
type RawRSSFeed struct {
URL string `json:"url"`
FetchedAt time.Time `json:"fetchedAt"`
Body string `json:"body"`
}

View File

@@ -0,0 +1,51 @@
package spc
import (
"encoding/json"
"testing"
"time"
)
func TestRawConvectiveOutlookBundleJSONShape(t *testing.T) {
fetchedAt := time.Date(2026, 6, 11, 20, 0, 0, 0, time.UTC)
bundle := RawConvectiveOutlookBundle{
LocationID: "stl",
LocationName: "St. Louis, MO",
Latitude: 38.6239,
Longitude: -90.3571,
FetchedAt: fetchedAt,
Products: []RawOutlookProduct{{
Key: "day1_categorical",
Day: 1,
OutlookType: OutlookTypeCategorical,
URL: "https://example.invalid/day1.geojson",
FetchedAt: fetchedAt,
Body: json.RawMessage(`{"type":"FeatureCollection","features":[]}`),
}},
Discussions: []RawDiscussionPage{{
Key: "day1",
Day: 1,
URL: "https://example.invalid/day1.html",
FetchedAt: fetchedAt,
Body: "Day 1 Convective Outlook",
}},
}
raw, err := json.Marshal(bundle)
if err != nil {
t.Fatalf("Marshal() error = %v", err)
}
var got map[string]any
if err := json.Unmarshal(raw, &got); err != nil {
t.Fatalf("Unmarshal() error = %v", err)
}
for _, key := range []string{"locationId", "locationName", "latitude", "longitude", "fetchedAt", "products", "discussions"} {
if _, ok := got[key]; !ok {
t.Fatalf("marshaled bundle missing key %q in %s", key, raw)
}
}
if _, ok := got["rss"]; ok {
t.Fatalf("marshaled bundle included empty rss: %s", raw)
}
}

View File

@@ -0,0 +1,81 @@
package spc
import (
"encoding/xml"
"fmt"
"strings"
"time"
)
// RSSFeed is a minimal view of the optional SPC RSS feed.
type RSSFeed struct {
Title string
Link string
Description string
LastBuildDate *time.Time
Items []RSSItem
}
// RSSItem is a minimal view of one optional SPC RSS item.
type RSSItem struct {
Title string
Link string
Description string
PubDate string
GUID string
}
// ParseRSSFeed decodes supplemental SPC RSS metadata.
func ParseRSSFeed(raw string) (RSSFeed, error) {
var doc struct {
Channel struct {
Title string `xml:"title"`
Link string `xml:"link"`
Description string `xml:"description"`
LastBuildDate string `xml:"lastBuildDate"`
Items []struct {
Title string `xml:"title"`
Link string `xml:"link"`
Description string `xml:"description"`
PubDate string `xml:"pubDate"`
GUID string `xml:"guid"`
} `xml:"item"`
} `xml:"channel"`
}
if err := xml.Unmarshal([]byte(raw), &doc); err != nil {
return RSSFeed{}, fmt.Errorf("decode rss: %w", err)
}
feed := RSSFeed{
Title: strings.TrimSpace(doc.Channel.Title),
Link: strings.TrimSpace(doc.Channel.Link),
Description: strings.TrimSpace(doc.Channel.Description),
LastBuildDate: parseRSSDate(doc.Channel.LastBuildDate),
Items: make([]RSSItem, 0, len(doc.Channel.Items)),
}
for _, item := range doc.Channel.Items {
feed.Items = append(feed.Items, RSSItem{
Title: strings.TrimSpace(item.Title),
Link: strings.TrimSpace(item.Link),
Description: strings.TrimSpace(item.Description),
PubDate: strings.TrimSpace(item.PubDate),
GUID: strings.TrimSpace(item.GUID),
})
}
return feed, nil
}
func parseRSSDate(value string) *time.Time {
value = strings.TrimSpace(value)
if value == "" {
return nil
}
for _, layout := range []string{time.RFC1123Z, time.RFC1123} {
t, err := time.Parse(layout, value)
if err == nil {
tt := t.UTC()
return &tt
}
}
return nil
}

View File

@@ -0,0 +1,43 @@
package spc
import (
"testing"
"time"
)
func TestParseRSSFeed(t *testing.T) {
const raw = `<?xml version="1.0"?>
<rss version="2.0">
<channel>
<title>SPC AC RSS</title>
<link>https://www.spc.noaa.gov/products/</link>
<description>SPC products</description>
<lastBuildDate>Thu, 11 Jun 2026 19:00:00 +0000</lastBuildDate>
<item>
<title>Day 1 Convective Outlook</title>
<link>https://www.spc.noaa.gov/products/outlook/day1otlk.html</link>
<description>Outlook text</description>
<pubDate>Thu, 11 Jun 2026 18:55:00 +0000</pubDate>
<guid>day1</guid>
</item>
</channel>
</rss>`
got, err := ParseRSSFeed(raw)
if err != nil {
t.Fatalf("ParseRSSFeed() error = %v", err)
}
if got.Title != "SPC AC RSS" {
t.Fatalf("Title = %q", got.Title)
}
wantBuild := time.Date(2026, 6, 11, 19, 0, 0, 0, time.UTC)
if got.LastBuildDate == nil || !got.LastBuildDate.Equal(wantBuild) {
t.Fatalf("LastBuildDate = %v, want %s", got.LastBuildDate, wantBuild)
}
if len(got.Items) != 1 {
t.Fatalf("Items length = %d, want 1", len(got.Items))
}
if got.Items[0].GUID != "day1" {
t.Fatalf("Item GUID = %q", got.Items[0].GUID)
}
}

View File

@@ -0,0 +1,29 @@
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"VALID_ISO": "2026-06-11T13:00:00Z",
"EXPIRE_ISO": "2026-06-12T12:00:00Z",
"ISSUE_ISO": "2026-06-11T12:34:56Z",
"FORECASTER": "SMITH",
"LABEL": "SLGT",
"LABEL2": "Slight Risk",
"DN": 3
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[-91.0, 38.0],
[-90.0, 38.0],
[-90.0, 39.0],
[-91.0, 39.0],
[-91.0, 38.0]
]
]
}
}
]
}

View File

@@ -0,0 +1,22 @@
<!doctype html>
<html>
<head><title>Day 1 Convective Outlook</title></head>
<body>
<table>
<tr><td align="center" class="rpttext" nowrap>Updated:&nbsp;Thu Jun 11 12:45:00 UTC 2026&nbsp;(<a href="archive/day1-geojson.zip">geojson</a>)</td></tr>
</table>
<pre>
<script>window.bad = "<b>ignore me</b>";</script>
SPC AC 111234
Day 1 Convective Outlook
NWS Storm Prediction Center Norman OK
...SUMMARY...
Severe thunderstorms are possible across parts of the central Plains
and mid Mississippi Valley this afternoon and evening.
...DISCUSSION...
The primary threats will be damaging wind and large hail.
</pre>
</body>
</html>

View File

@@ -0,0 +1,19 @@
<!doctype html>
<html>
<body>
<table>
<tr><td class="rpttext">Updated:&nbsp;Thu Jun 11 17:30:00 UTC 2026&nbsp;</td></tr>
</table>
<pre>
SPC AC 111730
Day 2 Convective Outlook CORR 1
NWS Storm Prediction Center Norman OK
...SUMMARY...
Scattered severe thunderstorms remain possible across the southern Plains.
...DISCUSSION...
Corrected outlook text remains otherwise unchanged.
</pre>
</body>
</html>

View File

@@ -0,0 +1,29 @@
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"VALID_ISO": "2026-06-12T12:00:00Z",
"EXPIRE_ISO": "2026-06-13T12:00:00Z",
"ISSUE_ISO": "2026-06-11T17:30:00Z",
"FORECASTER": "DOE",
"LABEL": "5",
"LABEL2": "5% Tornado",
"DN": "5"
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[-100.0, 35.0],
[-98.0, 35.0],
[-98.0, 37.0],
[-100.0, 37.0],
[-100.0, 35.0]
]
]
}
}
]
}

View File

@@ -0,0 +1,19 @@
<!doctype html>
<html>
<body>
<table>
<tr><td class="rpttext">Updated:&nbsp;Thu Jun 11 20:00:00 UTC 2026&nbsp;</td></tr>
</table>
<pre>
SPC AC 112000
Day 3 Convective Outlook
NWS Storm Prediction Center Norman OK
...SUMMARY...
A corridor of strong to severe storms may develop near a frontal zone.
...DISCUSSION...
Confidence remains moderate for organized storms.
</pre>
</body>
</html>

View File

@@ -0,0 +1,31 @@
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"VALID_ISO": "2026-06-13T12:00:00Z",
"EXPIRE_ISO": "2026-06-14T12:00:00Z",
"ISSUE_ISO": "2026-06-11T19:45:00Z",
"FORECASTER": "LEE",
"LABEL": "15",
"LABEL2": "15% Wind",
"DN": 15
},
"geometry": {
"type": "MultiPolygon",
"coordinates": [
[
[
[-103.0, 34.0],
[-101.0, 34.0],
[-101.0, 36.0],
[-103.0, 36.0],
[-103.0, 34.0]
]
]
]
}
}
]
}

View File

@@ -0,0 +1,24 @@
package spc
import (
"strings"
"time"
)
// ParseISOTimestamp parses SPC ISO timestamps from GeoJSON properties.
func ParseISOTimestamp(value string) (time.Time, error) {
return time.Parse(time.RFC3339, strings.TrimSpace(value))
}
func parseOptionalISOTimestamp(value string) *time.Time {
value = strings.TrimSpace(value)
if value == "" {
return nil
}
t, err := ParseISOTimestamp(value)
if err != nil {
return nil
}
tt := t.UTC()
return &tt
}

View File

@@ -8,29 +8,42 @@
// Canonical input schemas:
// - weather.observation.v1 -> model.WeatherObservation
// - weather.forecast.v1 -> model.WeatherForecastRun
// - weather.forecast_discussion.v1 -> model.WeatherForecastDiscussion
// - weather.weather_story.v1 -> model.WeatherStoryRun
// - weather.alert.v1 -> model.WeatherAlertRun
// - weather.outlook.v1 -> model.WeatherOutlookRun
//
// Parent/child relationships:
// - observations.event_id -> observation_present_weather.event_id
// - forecasts.event_id -> forecast_periods.run_event_id
// - forecast_discussions.event_id -> forecast_discussion_key_messages.run_event_id
// - weather_story_runs.event_id -> weather_stories.run_event_id
// - alert_runs.event_id -> alerts.run_event_id
// - alerts.(run_event_id, alert_index) -> alert_references.(run_event_id, alert_index)
// - outlook_runs.event_id -> outlooks.run_event_id
//
// Dedupe and retention behavior:
// - Parent primary keys (event_id): observations, forecasts, alert_runs.
// - Parent primary keys (event_id): observations, forecasts, alert_runs, outlook_runs.
// - Child primary keys use positional indexes to preserve payload order.
// - Prune columns:
// - observations.observed_at
// - observation_present_weather.observed_at
// - forecasts.issued_at
// - forecast_periods.issued_at
// - forecast_discussions.issued_at
// - forecast_discussion_key_messages.issued_at
// - weather_story_runs.as_of
// - weather_stories.as_of
// - alert_runs.as_of
// - alerts.as_of
// - alert_references.as_of
// - outlook_runs.as_of
// - outlooks.as_of
//
// Envelope field mapping (shared parent columns)
//
// These columns exist on observations, forecasts, and alert_runs:
// These columns exist on parent tables such as observations, forecasts,
// forecast_discussions, weather_story_runs, alert_runs, and outlook_runs:
// - event_id TEXT -> event.id
// - event_kind TEXT -> event.kind
// - event_source TEXT -> event.source
@@ -101,7 +114,7 @@
// - end_time TIMESTAMPTZ -> payload.periods[i].endTime
// - name TEXT NULL -> payload.periods[i].name
// - is_day BOOLEAN NULL -> payload.periods[i].isDay
// - condition_code INTEGER -> payload.periods[i].conditionCode
// - condition_code INTEGER NULL -> payload.periods[i].conditionCode
// - text_description TEXT NULL -> payload.periods[i].textDescription
// - temperature_c DOUBLE PRECISION NULL -> payload.periods[i].temperatureC
// - temperature_c_min DOUBLE PRECISION NULL -> payload.periods[i].temperatureCMin
@@ -120,7 +133,35 @@
// - snowfall_depth_mm DOUBLE PRECISION NULL -> payload.periods[i].snowfallDepthMm
// - uv_index DOUBLE PRECISION NULL -> payload.periods[i].uvIndex
//
// 5. alert_runs (PK: event_id)
// 5. weather_story_runs (PK: event_id)
//
// - event_id TEXT -> event.id
// - event_kind TEXT -> event.kind
// - event_source TEXT -> event.source
// - event_schema TEXT -> event.schema
// - event_emitted_at TIMESTAMPTZ -> event.emitted_at
// - event_effective_at TIMESTAMPTZ NULL -> event.effective_at
// - office_id TEXT NULL -> payload.officeId
// - as_of TIMESTAMPTZ -> payload.asOf
// - story_count INTEGER -> len(payload.stories)
//
// 6. weather_stories (PK: run_event_id, story_index)
//
// - run_event_id TEXT -> weather_story_runs.event_id / payload.stories[i]
// - story_index INTEGER -> i (array position in payload.stories)
// - as_of TIMESTAMPTZ -> payload.asOf (copied from parent)
// - office_id TEXT NULL -> payload.stories[i].officeId
// - start_time TIMESTAMPTZ -> payload.stories[i].startTime
// - end_time TIMESTAMPTZ -> payload.stories[i].endTime
// - updated_at TIMESTAMPTZ -> payload.stories[i].updatedAt
// - title TEXT NULL -> payload.stories[i].title
// - description TEXT NULL -> payload.stories[i].description
// - alt_text TEXT NULL -> payload.stories[i].altText
// - priority BOOLEAN -> payload.stories[i].priority
// - story_order INTEGER -> payload.stories[i].order
// - download_url TEXT NULL -> payload.stories[i].downloadUrl
//
// 7. alert_runs (PK: event_id)
//
// - event_id TEXT -> event.id
// - event_kind TEXT -> event.kind
@@ -135,7 +176,7 @@
// - longitude DOUBLE PRECISION NULL -> payload.longitude
// - alert_count INTEGER -> len(payload.alerts)
//
// 6. alerts (PK: run_event_id, alert_index)
// 8. alerts (PK: run_event_id, alert_index)
//
// - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i]
// - alert_index INTEGER -> i (array position in payload.alerts)
@@ -160,7 +201,7 @@
// - sender_name TEXT NULL -> payload.alerts[i].senderName
// - reference_count INTEGER -> len(payload.alerts[i].references)
//
// 7. alert_references (PK: run_event_id, alert_index, reference_index)
// 9. alert_references (PK: run_event_id, alert_index, reference_index)
//
// - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i].references[j]
// - alert_index INTEGER -> i (array position in payload.alerts)
@@ -171,6 +212,48 @@
// - sender TEXT NULL -> payload.alerts[i].references[j].sender
// - sent TIMESTAMPTZ NULL -> payload.alerts[i].references[j].sent
//
// 10. outlook_runs (PK: event_id)
//
// - event_id TEXT -> event.id
// - event_kind TEXT -> event.kind
// - event_source TEXT -> event.source
// - event_schema TEXT -> event.schema
// - event_emitted_at TIMESTAMPTZ -> event.emitted_at
// - event_effective_at TIMESTAMPTZ NULL -> event.effective_at
// - location_id TEXT NULL -> payload.locationId
// - location_name TEXT NULL -> payload.locationName
// - latitude DOUBLE PRECISION NULL -> payload.latitude
// - longitude DOUBLE PRECISION NULL -> payload.longitude
// - as_of TIMESTAMPTZ -> payload.asOf
// - issued_at TIMESTAMPTZ NULL -> payload.issuedAt
// - outlook_count INTEGER -> len(payload.outlooks)
//
// 11. outlooks (PK: run_event_id, outlook_index)
//
// - run_event_id TEXT -> outlook_runs.event_id / payload.outlooks[i]
// - outlook_index INTEGER -> i (array position in payload.outlooks)
// - as_of TIMESTAMPTZ -> payload.asOf (copied from parent)
// - outlook_id TEXT -> payload.outlooks[i].id
// - provider TEXT -> payload.outlooks[i].provider
// - product TEXT -> payload.outlooks[i].product
// - day INTEGER -> payload.outlooks[i].day
// - outlook_type TEXT -> payload.outlooks[i].outlookType
// - label TEXT -> payload.outlooks[i].label
// - label_text TEXT NULL -> payload.outlooks[i].labelText
// - severity_rank INTEGER NULL -> payload.outlooks[i].severityRank
// - valid_from TIMESTAMPTZ -> payload.outlooks[i].validFrom
// - valid_to TIMESTAMPTZ -> payload.outlooks[i].validTo
// - issued_at TIMESTAMPTZ -> payload.outlooks[i].issuedAt
// - expires_at TIMESTAMPTZ -> payload.outlooks[i].expiresAt
// - forecaster TEXT NULL -> payload.outlooks[i].forecaster
// - headline TEXT NULL -> payload.outlooks[i].headline
// - summary TEXT NULL -> payload.outlooks[i].summary
// - discussion TEXT NULL -> payload.outlooks[i].discussion
// - source_url TEXT NULL -> payload.outlooks[i].sourceUrl
// - image_url TEXT NULL -> payload.outlooks[i].imageUrl
// - contains_location BOOLEAN -> payload.outlooks[i].containsLocation
// - geometry_json TEXT -> compact JSON payload.outlooks[i].geometry
//
// Reconstructing canonical JSON payloads
//
// - WeatherObservation:
@@ -181,8 +264,16 @@
// read one row from forecasts, then join forecast_periods by run_event_id
// ordered by period_index to rebuild periods.
//
// - WeatherStoryRun:
// read one row from weather_story_runs, then join weather_stories by
// run_event_id ordered by story_index to rebuild stories.
//
// - WeatherAlertRun:
// read one row from alert_runs, join alerts by run_event_id ordered by
// alert_index, then join alert_references by (run_event_id, alert_index)
// ordered by reference_index to rebuild references per alert.
//
// - WeatherOutlookRun:
// read one row from outlook_runs, then join outlooks by run_event_id ordered
// by outlook_index to rebuild outlooks.
package postgres

View File

@@ -1,6 +1,7 @@
package postgres
import (
"bytes"
"context"
"encoding/json"
"fmt"
@@ -20,8 +21,14 @@ func mapPostgresEvent(_ context.Context, e fkevent.Event) ([]fksinks.PostgresWri
return mapObservationEvent(e)
case standards.SchemaWeatherForecastV1:
return mapForecastEvent(e)
case standards.SchemaWeatherForecastDiscussionV1:
return mapForecastDiscussionEvent(e)
case standards.SchemaWeatherStoryV1:
return mapWeatherStoryEvent(e)
case standards.SchemaWeatherAlertV1:
return mapAlertEvent(e)
case standards.SchemaWeatherOutlookV1:
return mapOutlookEvent(e)
default:
return nil, nil
}
@@ -135,7 +142,7 @@ func mapForecastEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
"end_time": p.EndTime.UTC(),
"name": nullableString(p.Name),
"is_day": nullableBool(p.IsDay),
"condition_code": int(p.ConditionCode),
"condition_code": nullableWMOCode(p.ConditionCode),
"text_description": nullableString(p.TextDescription),
"temperature_c": nullableFloat64(p.TemperatureC),
"temperature_c_min": nullableFloat64(p.TemperatureCMin),
@@ -160,6 +167,115 @@ func mapForecastEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
return writes, nil
}
func mapForecastDiscussionEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
run, err := decodePayload[model.WeatherForecastDiscussion](e.Payload)
if err != nil {
return nil, fmt.Errorf("decode forecast discussion payload: %w", err)
}
if run.IssuedAt.IsZero() {
return nil, fmt.Errorf("decode forecast discussion payload: issuedAt is required")
}
if strings.TrimSpace(string(run.Product)) == "" {
return nil, fmt.Errorf("decode forecast discussion payload: product is required")
}
issuedAt := run.IssuedAt.UTC()
shortTermQualifier, shortTermIssuedAt, shortTermText := nullableDiscussionSection(run.ShortTerm)
longTermQualifier, longTermIssuedAt, longTermText := nullableDiscussionSection(run.LongTerm)
writes := make([]fksinks.PostgresWrite, 0, 1+len(run.KeyMessages))
writes = append(writes, fksinks.PostgresWrite{
Table: tableForecastDiscussions,
Values: map[string]any{
"event_id": e.ID,
"event_kind": string(e.Kind),
"event_source": e.Source,
"event_schema": e.Schema,
"event_emitted_at": e.EmittedAt.UTC(),
"event_effective_at": nullableTime(e.EffectiveAt),
"office_id": nullableString(run.OfficeID),
"office_name": nullableString(run.OfficeName),
"issued_at": issuedAt,
"updated_at": nullableTime(run.UpdatedAt),
"product": string(run.Product),
"short_term_qualifier": shortTermQualifier,
"short_term_issued_at": shortTermIssuedAt,
"short_term_text": shortTermText,
"long_term_qualifier": longTermQualifier,
"long_term_issued_at": longTermIssuedAt,
"long_term_text": longTermText,
"key_message_count": len(run.KeyMessages),
},
})
for i, msg := range run.KeyMessages {
writes = append(writes, fksinks.PostgresWrite{
Table: tableForecastDiscussionKeyMessages,
Values: map[string]any{
"run_event_id": e.ID,
"message_index": i,
"issued_at": issuedAt,
"message_text": nullableString(msg),
},
})
}
return writes, nil
}
func mapWeatherStoryEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
run, err := decodePayload[model.WeatherStoryRun](e.Payload)
if err != nil {
return nil, fmt.Errorf("decode weather story payload: %w", err)
}
if run.AsOf.IsZero() {
return nil, fmt.Errorf("decode weather story payload: asOf is required")
}
asOf := run.AsOf.UTC()
writes := make([]fksinks.PostgresWrite, 0, 1+len(run.Stories))
writes = append(writes, fksinks.PostgresWrite{
Table: tableWeatherStoryRuns,
Values: map[string]any{
"event_id": e.ID,
"event_kind": string(e.Kind),
"event_source": e.Source,
"event_schema": e.Schema,
"event_emitted_at": e.EmittedAt.UTC(),
"event_effective_at": nullableTime(e.EffectiveAt),
"office_id": nullableString(run.OfficeID),
"as_of": asOf,
"story_count": len(run.Stories),
},
})
for i, story := range run.Stories {
if story.StartTime.IsZero() || story.EndTime.IsZero() || story.UpdatedAt.IsZero() {
return nil, fmt.Errorf("decode weather story payload: stories[%d] startTime/endTime/updatedAt are required", i)
}
writes = append(writes, fksinks.PostgresWrite{
Table: tableWeatherStories,
Values: map[string]any{
"run_event_id": e.ID,
"story_index": i,
"as_of": asOf,
"office_id": nullableString(story.OfficeID),
"start_time": story.StartTime.UTC(),
"end_time": story.EndTime.UTC(),
"updated_at": story.UpdatedAt.UTC(),
"title": nullableString(story.Title),
"description": nullableString(story.Description),
"alt_text": nullableString(story.AltText),
"priority": story.Priority,
"story_order": story.Order,
"download_url": nullableString(story.DownloadURL),
},
})
}
return writes, nil
}
func mapAlertEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
run, err := decodePayload[model.WeatherAlertRun](e.Payload)
if err != nil {
@@ -243,6 +359,109 @@ func mapAlertEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
return writes, nil
}
func mapOutlookEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
run, err := decodePayload[model.WeatherOutlookRun](e.Payload)
if err != nil {
return nil, fmt.Errorf("decode outlook payload: %w", err)
}
if run.AsOf.IsZero() {
return nil, fmt.Errorf("decode outlook payload: asOf is required")
}
asOf := run.AsOf.UTC()
writes := make([]fksinks.PostgresWrite, 0, 1+len(run.Outlooks))
writes = append(writes, fksinks.PostgresWrite{
Table: tableOutlookRuns,
Values: map[string]any{
"event_id": e.ID,
"event_kind": string(e.Kind),
"event_source": e.Source,
"event_schema": e.Schema,
"event_emitted_at": e.EmittedAt.UTC(),
"event_effective_at": nullableTime(e.EffectiveAt),
"location_id": nullableString(run.LocationID),
"location_name": nullableString(run.LocationName),
"latitude": nullableFloat64(run.Latitude),
"longitude": nullableFloat64(run.Longitude),
"as_of": asOf,
"issued_at": nullableTime(run.IssuedAt),
"outlook_count": len(run.Outlooks),
},
})
for i, outlook := range run.Outlooks {
if err := validateOutlook(outlook, i); err != nil {
return nil, err
}
geometryJSON, err := requiredCompactJSONText(outlook.Geometry)
if err != nil {
return nil, fmt.Errorf("decode outlook payload: outlooks[%d].geometry: %w", i, err)
}
writes = append(writes, fksinks.PostgresWrite{
Table: tableOutlooks,
Values: map[string]any{
"run_event_id": e.ID,
"outlook_index": i,
"as_of": asOf,
"outlook_id": outlook.ID,
"provider": outlook.Provider,
"product": outlook.Product,
"day": outlook.Day,
"outlook_type": outlook.OutlookType,
"label": outlook.Label,
"label_text": nullableString(outlook.LabelText),
"severity_rank": nullableInt(outlook.SeverityRank),
"valid_from": outlook.ValidFrom.UTC(),
"valid_to": outlook.ValidTo.UTC(),
"issued_at": outlook.IssuedAt.UTC(),
"expires_at": outlook.ExpiresAt.UTC(),
"forecaster": nullableString(outlook.Forecaster),
"headline": nullableString(outlook.Headline),
"summary": nullableString(outlook.Summary),
"discussion": nullableString(outlook.Discussion),
"source_url": nullableString(outlook.SourceURL),
"image_url": nullableString(outlook.ImageURL),
"contains_location": outlook.ContainsLocation,
"geometry_json": geometryJSON,
},
})
}
return writes, nil
}
func validateOutlook(outlook model.WeatherOutlook, index int) error {
if strings.TrimSpace(outlook.ID) == "" {
return fmt.Errorf("decode outlook payload: outlooks[%d].id is required", index)
}
if strings.TrimSpace(outlook.Provider) == "" {
return fmt.Errorf("decode outlook payload: outlooks[%d].provider is required", index)
}
if strings.TrimSpace(outlook.Product) == "" {
return fmt.Errorf("decode outlook payload: outlooks[%d].product is required", index)
}
if outlook.Day == 0 {
return fmt.Errorf("decode outlook payload: outlooks[%d].day is required", index)
}
if strings.TrimSpace(outlook.OutlookType) == "" {
return fmt.Errorf("decode outlook payload: outlooks[%d].outlookType is required", index)
}
if strings.TrimSpace(outlook.Label) == "" {
return fmt.Errorf("decode outlook payload: outlooks[%d].label is required", index)
}
if outlook.ValidFrom.IsZero() || outlook.ValidTo.IsZero() {
return fmt.Errorf("decode outlook payload: outlooks[%d] validFrom/validTo are required", index)
}
if outlook.IssuedAt.IsZero() || outlook.ExpiresAt.IsZero() {
return fmt.Errorf("decode outlook payload: outlooks[%d] issuedAt/expiresAt are required", index)
}
if len(outlook.Geometry) == 0 {
return fmt.Errorf("decode outlook payload: outlooks[%d].geometry is required", index)
}
return nil
}
func decodePayload[T any](payload any) (T, error) {
var out T
if payload == nil {
@@ -269,6 +488,13 @@ func decodePayload[T any](payload any) (T, error) {
return out, nil
}
func nullableDiscussionSection(section *model.WeatherForecastDiscussionSection) (any, any, any) {
if section == nil {
return nil, nil, nil
}
return nullableString(section.Qualifier), nullableTime(section.IssuedAt), nullableString(section.Text)
}
func countAlertReferences(alerts []model.WeatherAlert) int {
total := 0
for _, a := range alerts {
@@ -298,6 +524,13 @@ func nullableBool(v *bool) any {
return *v
}
func nullableInt(v *int) any {
if v == nil {
return nil
}
return *v
}
func nullableTime(v *time.Time) any {
if v == nil || v.IsZero() {
return nil
@@ -305,6 +538,13 @@ func nullableTime(v *time.Time) any {
return v.UTC()
}
func nullableWMOCode(v *model.WMOCode) any {
if v == nil {
return nil
}
return int(*v)
}
func compactJSONText(v any) (any, error) {
if v == nil {
return nil, nil
@@ -318,3 +558,19 @@ func compactJSONText(v any) (any, error) {
}
return string(b), nil
}
func requiredCompactJSONText(v any) (string, error) {
compact, err := compactJSONText(v)
if err != nil {
return "", err
}
s, ok := compact.(string)
if !ok || strings.TrimSpace(s) == "" || strings.TrimSpace(s) == "null" {
return "", fmt.Errorf("is required")
}
var buf bytes.Buffer
if err := json.Compact(&buf, []byte(s)); err != nil {
return "", err
}
return buf.String(), nil
}

View File

@@ -63,13 +63,13 @@ func TestMapPostgresEventForecastStructPayload(t *testing.T) {
StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC),
EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
IsDay: &isDay,
ConditionCode: model.WMOCode(2),
ConditionCode: wmoCodePtr(model.WMOCode(2)),
TemperatureC: &temp,
},
{
StartTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
EndTime: time.Date(2026, 3, 16, 21, 0, 0, 0, time.UTC),
ConditionCode: model.WMOCode(3),
ConditionCode: nil,
},
},
}
@@ -94,6 +94,9 @@ func TestMapPostgresEventForecastStructPayload(t *testing.T) {
if got := writes[1].Values["period_index"]; got != 0 {
t.Fatalf("first period index = %#v, want 0", got)
}
if got := writes[2].Values["condition_code"]; got != nil {
t.Fatalf("second period condition_code = %#v, want nil", got)
}
assertAllWritesIncludeAllColumns(t, writes)
}
@@ -146,6 +149,320 @@ func TestMapPostgresEventAlertStructPayload(t *testing.T) {
assertAllWritesIncludeAllColumns(t, writes)
}
func TestMapPostgresEventForecastDiscussionStructPayload(t *testing.T) {
updatedAt := time.Date(2026, 3, 28, 20, 29, 47, 0, time.UTC)
shortIssuedAt := time.Date(2026, 3, 28, 19, 19, 0, 0, time.UTC)
run := model.WeatherForecastDiscussion{
OfficeID: "LSX",
OfficeName: "National Weather Service Saint Louis MO",
Product: model.ForecastDiscussionProductAFD,
IssuedAt: time.Date(2026, 3, 28, 19, 24, 0, 0, time.UTC),
UpdatedAt: &updatedAt,
KeyMessages: []string{"msg one", "msg two"},
ShortTerm: &model.WeatherForecastDiscussionSection{Qualifier: "(Tonight)", IssuedAt: &shortIssuedAt, Text: "Short term text"},
LongTerm: &model.WeatherForecastDiscussionSection{Text: "Long term text"},
}
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastDiscussionV1, "forecast_discussion", run))
if err != nil {
t.Fatalf("mapPostgresEvent() error = %v", err)
}
if len(writes) != 3 {
t.Fatalf("mapPostgresEvent() writes len = %d, want 3", len(writes))
}
if writes[0].Table != tableForecastDiscussions {
t.Fatalf("writes[0].Table = %q, want %q", writes[0].Table, tableForecastDiscussions)
}
if got := writes[0].Values["key_message_count"]; got != 2 {
t.Fatalf("forecast_discussions key_message_count = %#v, want 2", got)
}
if got := writes[0].Values["short_term_qualifier"]; got != "(Tonight)" {
t.Fatalf("forecast_discussions short_term_qualifier = %#v, want (Tonight)", got)
}
if got := writes[0].Values["long_term_issued_at"]; got != nil {
t.Fatalf("forecast_discussions long_term_issued_at = %#v, want nil", got)
}
if writes[1].Table != tableForecastDiscussionKeyMessages || writes[2].Table != tableForecastDiscussionKeyMessages {
t.Fatalf("forecast discussion key message writes not in expected order")
}
if got := writes[2].Values["message_index"]; got != 1 {
t.Fatalf("second key message index = %#v, want 1", got)
}
assertAllWritesIncludeAllColumns(t, writes)
}
func TestMapPostgresEventWeatherStoryStructPayload(t *testing.T) {
run := model.WeatherStoryRun{
OfficeID: "LSX",
AsOf: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
Stories: []model.WeatherStory{
{
OfficeID: "LSX",
StartTime: time.Date(2026, 5, 30, 8, 46, 0, 0, time.UTC),
EndTime: time.Date(2026, 5, 31, 11, 0, 0, 0, time.UTC),
UpdatedAt: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
Title: "Several Chances for Rain Through Monday",
Description: "Scattered showers and thunderstorms.",
AltText: "This slide shows the forecast.",
Priority: true,
Order: 1,
DownloadURL: "https://api.weather.gov/offices/LSX/weatherstories/download/story-1",
},
},
}
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, "weather_story", run))
if err != nil {
t.Fatalf("mapPostgresEvent() error = %v", err)
}
if len(writes) != 2 {
t.Fatalf("mapPostgresEvent() writes len = %d, want 2", len(writes))
}
if writes[0].Table != tableWeatherStoryRuns {
t.Fatalf("writes[0].Table = %q, want %q", writes[0].Table, tableWeatherStoryRuns)
}
if got := writes[0].Values["story_count"]; got != 1 {
t.Fatalf("weather_story_runs story_count = %#v, want 1", got)
}
if writes[1].Table != tableWeatherStories {
t.Fatalf("writes[1].Table = %q, want %q", writes[1].Table, tableWeatherStories)
}
if got := writes[1].Values["download_url"]; got != "https://api.weather.gov/offices/LSX/weatherstories/download/story-1" {
t.Fatalf("weather_stories download_url = %#v", got)
}
if got := writes[1].Values["story_order"]; got != 1 {
t.Fatalf("weather_stories story_order = %#v, want 1", got)
}
assertAllWritesIncludeAllColumns(t, writes)
}
func TestMapPostgresEventOutlookStructPayload(t *testing.T) {
lat := 38.6239
lon := -90.3571
issuedAt := time.Date(2026, 6, 11, 19, 45, 0, 0, time.FixedZone("UTC-5", -5*60*60))
severity := 3
run := model.WeatherOutlookRun{
LocationID: "stl",
LocationName: "St. Louis, MO",
Latitude: &lat,
Longitude: &lon,
AsOf: time.Date(2026, 6, 12, 0, 45, 0, 0, time.UTC),
IssuedAt: &issuedAt,
Outlooks: []model.WeatherOutlook{
{
ID: "outlook-1",
Provider: "spc",
Product: "convective",
Day: 1,
OutlookType: "categorical",
Label: "SLGT",
LabelText: "Slight Risk",
SeverityRank: &severity,
ValidFrom: time.Date(2026, 6, 11, 13, 0, 0, 0, time.FixedZone("UTC-5", -5*60*60)),
ValidTo: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
IssuedAt: issuedAt,
ExpiresAt: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
Forecaster: "SMITH",
Headline: "Day 1 Convective Outlook",
Summary: "Severe thunderstorms are possible.",
Discussion: "Full discussion text.",
SourceURL: "https://example.invalid/day1.geojson",
ContainsLocation: true,
Geometry: json.RawMessage(`{ "type" : "Polygon", "coordinates" : [ [ [ -91.0, 38.0 ], [ -90.0, 38.0 ], [ -90.0, 39.0 ], [ -91.0, 39.0 ], [ -91.0, 38.0 ] ] ] }`),
},
{
ID: "outlook-2",
Provider: "spc",
Product: "convective",
Day: 1,
OutlookType: "wind",
Label: "15",
ValidFrom: time.Date(2026, 6, 11, 13, 0, 0, 0, time.UTC),
ValidTo: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
IssuedAt: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
ExpiresAt: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
ContainsLocation: false,
Geometry: json.RawMessage(`{"type":"Polygon","coordinates":[[[-100,35],[-98,35],[-98,37],[-100,37],[-100,35]]]}`),
},
},
}
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV1, "outlook", run))
if err != nil {
t.Fatalf("mapPostgresEvent() error = %v", err)
}
if len(writes) != 3 {
t.Fatalf("mapPostgresEvent() writes len = %d, want 3", len(writes))
}
if writes[0].Table != tableOutlookRuns {
t.Fatalf("writes[0].Table = %q, want %q", writes[0].Table, tableOutlookRuns)
}
if got := writes[0].Values["outlook_count"]; got != 2 {
t.Fatalf("outlook_runs outlook_count = %#v, want 2", got)
}
if got := writes[0].Values["issued_at"]; got != issuedAt.UTC() {
t.Fatalf("outlook_runs issued_at = %#v, want UTC %s", got, issuedAt.UTC())
}
if writes[1].Table != tableOutlooks || writes[2].Table != tableOutlooks {
t.Fatalf("outlook writes not in expected order")
}
if got := writes[1].Values["outlook_index"]; got != 0 {
t.Fatalf("first outlook index = %#v, want 0", got)
}
if got := writes[1].Values["outlook_id"]; got != "outlook-1" {
t.Fatalf("first outlook_id = %#v, want outlook-1", got)
}
if got := writes[1].Values["provider"]; got != "spc" {
t.Fatalf("first provider = %#v, want spc", got)
}
if got := writes[1].Values["valid_from"]; got != run.Outlooks[0].ValidFrom.UTC() {
t.Fatalf("first valid_from = %#v, want UTC %s", got, run.Outlooks[0].ValidFrom.UTC())
}
if got := writes[1].Values["geometry_json"]; got != `{"type":"Polygon","coordinates":[[[-91.0,38.0],[-90.0,38.0],[-90.0,39.0],[-91.0,39.0],[-91.0,38.0]]]}` {
t.Fatalf("first geometry_json = %#v", got)
}
if got := writes[2].Values["contains_location"]; got != false {
t.Fatalf("second contains_location = %#v, want false", got)
}
assertAllWritesIncludeAllColumns(t, writes)
}
func TestMapPostgresEventOutlookRejectsMissingAsOf(t *testing.T) {
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV1, "outlook", model.WeatherOutlookRun{}))
if err == nil {
t.Fatalf("mapPostgresEvent() error = nil, want missing asOf error")
}
if !strings.Contains(err.Error(), "asOf is required") {
t.Fatalf("error = %q, want asOf context", err)
}
}
func TestMapPostgresEventOutlookRejectsMissingIDAndProvider(t *testing.T) {
base := model.WeatherOutlook{
ID: "outlook-1",
Provider: "spc",
Product: "convective",
Day: 1,
OutlookType: "categorical",
Label: "SLGT",
ValidFrom: time.Date(2026, 6, 11, 13, 0, 0, 0, time.UTC),
ValidTo: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
IssuedAt: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
ExpiresAt: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
Geometry: json.RawMessage(`{"type":"Polygon","coordinates":[[[-91,38],[-90,38],[-90,39],[-91,39],[-91,38]]]}`),
}
tests := []struct {
name string
mutate func(*model.WeatherOutlook)
wantErr string
}{
{
name: "missing id",
mutate: func(outlook *model.WeatherOutlook) { outlook.ID = "" },
wantErr: "outlooks[0].id is required",
},
{
name: "missing provider",
mutate: func(outlook *model.WeatherOutlook) { outlook.Provider = "" },
wantErr: "outlooks[0].provider is required",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
outlook := base
tt.mutate(&outlook)
run := model.WeatherOutlookRun{
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
Outlooks: []model.WeatherOutlook{outlook},
}
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV1, "outlook", run))
if err == nil {
t.Fatalf("mapPostgresEvent() error = nil, want %q", tt.wantErr)
}
if !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("error = %q, want %q", err, tt.wantErr)
}
})
}
}
func TestMapPostgresEventOutlookRejectsMissingRequiredTimes(t *testing.T) {
run := model.WeatherOutlookRun{
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
Outlooks: []model.WeatherOutlook{{
ID: "outlook-1",
Provider: "spc",
Product: "convective",
Day: 1,
OutlookType: "categorical",
Label: "SLGT",
Geometry: json.RawMessage(`{"type":"Polygon","coordinates":[[[-91,38],[-90,38],[-90,39],[-91,39],[-91,38]]]}`),
}},
}
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV1, "outlook", run))
if err == nil {
t.Fatalf("mapPostgresEvent() error = nil, want missing time error")
}
if !strings.Contains(err.Error(), "outlooks[0] validFrom/validTo are required") {
t.Fatalf("error = %q, want outlook time context", err)
}
}
func TestMapPostgresEventOutlookRejectsEmptyGeometry(t *testing.T) {
run := model.WeatherOutlookRun{
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
Outlooks: []model.WeatherOutlook{{
ID: "outlook-1",
Provider: "spc",
Product: "convective",
Day: 1,
OutlookType: "categorical",
Label: "SLGT",
ValidFrom: time.Date(2026, 6, 11, 13, 0, 0, 0, time.UTC),
ValidTo: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
IssuedAt: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
ExpiresAt: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
}},
}
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV1, "outlook", run))
if err == nil {
t.Fatalf("mapPostgresEvent() error = nil, want geometry error")
}
if !strings.Contains(err.Error(), "outlooks[0].geometry is required") {
t.Fatalf("error = %q, want geometry context", err)
}
}
func TestMapPostgresEventWeatherStoryRejectsMissingAsOf(t *testing.T) {
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, "weather_story", model.WeatherStoryRun{}))
if err == nil {
t.Fatalf("mapPostgresEvent() error = nil, want missing asOf error")
}
if !strings.Contains(err.Error(), "asOf is required") {
t.Fatalf("error = %q, want asOf context", err)
}
}
func TestMapPostgresEventWeatherStoryRejectsMissingStoryTimes(t *testing.T) {
run := model.WeatherStoryRun{
AsOf: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
Stories: []model.WeatherStory{{Title: "missing times"}},
}
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, "weather_story", run))
if err == nil {
t.Fatalf("mapPostgresEvent() error = nil, want missing story times error")
}
if !strings.Contains(err.Error(), "stories[0] startTime/endTime/updatedAt are required") {
t.Fatalf("error = %q, want story time context", err)
}
}
func TestMapPostgresEventMapPayload(t *testing.T) {
run := model.WeatherForecastRun{
IssuedAt: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
@@ -154,7 +471,7 @@ func TestMapPostgresEventMapPayload(t *testing.T) {
{
StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC),
EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
ConditionCode: model.WMOCode(2),
ConditionCode: wmoCodePtr(model.WMOCode(2)),
},
},
}
@@ -201,6 +518,16 @@ func TestMapPostgresEventMalformedPayload(t *testing.T) {
}
}
func TestMapPostgresEventForecastDiscussionMalformedPayload(t *testing.T) {
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastDiscussionV1, "forecast_discussion", "bad"))
if err == nil {
t.Fatalf("mapPostgresEvent() expected error for malformed payload")
}
if !strings.Contains(err.Error(), "decode forecast discussion payload") {
t.Fatalf("error = %q, want decode forecast discussion payload context", err)
}
}
func testEvent(schema string, kind fkevent.Kind, payload any) fkevent.Event {
effectiveAt := time.Date(2026, 3, 16, 18, 30, 0, 0, time.UTC)
return fkevent.Event{
@@ -238,10 +565,15 @@ func assertAllWritesIncludeAllColumns(t *testing.T, writes []fksinks.PostgresWri
}
func tableColumnCounts() map[string]int {
s := weatherPostgresSchema()
s := PostgresSchema()
m := make(map[string]int, len(s.Tables))
for _, tbl := range s.Tables {
m[tbl.Name] = len(tbl.Columns)
}
return m
}
func wmoCodePtr(v model.WMOCode) *model.WMOCode {
out := v
return &out
}

View File

@@ -1,10 +1,6 @@
package postgres
import (
"fmt"
"strings"
"gitea.maximumdirect.net/ejr/feedkit/config"
fksinks "gitea.maximumdirect.net/ejr/feedkit/sinks"
)
@@ -13,36 +9,19 @@ const (
tableObservationPresentWeather = "observation_present_weather"
tableForecasts = "forecasts"
tableForecastPeriods = "forecast_periods"
tableForecastDiscussions = "forecast_discussions"
tableForecastDiscussionKeyMessages = "forecast_discussion_key_messages"
tableWeatherStoryRuns = "weather_story_runs"
tableWeatherStories = "weather_stories"
tableAlertRuns = "alert_runs"
tableAlerts = "alerts"
tableAlertReferences = "alert_references"
tableOutlookRuns = "outlook_runs"
tableOutlooks = "outlooks"
)
// RegisterPostgresSchemas registers weatherfeeder's Postgres schema for each
// configured sink using driver=postgres.
func RegisterPostgresSchemas(cfg *config.Config) error {
if cfg == nil {
return fmt.Errorf("register postgres schemas: config is nil")
}
schema := weatherPostgresSchema()
for i, sk := range cfg.Sinks {
if !isPostgresDriver(sk.Driver) {
continue
}
if err := fksinks.RegisterPostgresSchema(sk.Name, schema); err != nil {
return fmt.Errorf("register postgres schema for sinks[%d] name=%q: %w", i, sk.Name, err)
}
}
return nil
}
func isPostgresDriver(driver string) bool {
return strings.EqualFold(strings.TrimSpace(driver), "postgres")
}
func weatherPostgresSchema() fksinks.PostgresSchema {
// PostgresSchema returns weatherfeeder's Postgres schema definition.
func PostgresSchema() fksinks.PostgresSchema {
return fksinks.PostgresSchema{
Tables: []fksinks.PostgresTable{
{
@@ -129,7 +108,7 @@ func weatherPostgresSchema() fksinks.PostgresSchema {
{Name: "end_time", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "name", Type: "TEXT", Nullable: true},
{Name: "is_day", Type: "BOOLEAN", Nullable: true},
{Name: "condition_code", Type: "INTEGER", Nullable: false},
{Name: "condition_code", Type: "INTEGER", Nullable: true},
{Name: "text_description", Type: "TEXT", Nullable: true},
{Name: "temperature_c", Type: "DOUBLE PRECISION", Nullable: true},
{Name: "temperature_c_min", Type: "DOUBLE PRECISION", Nullable: true},
@@ -156,6 +135,94 @@ func weatherPostgresSchema() fksinks.PostgresSchema {
{Name: "idx_wf_fc_period_run_start", Columns: []string{"run_event_id", "start_time"}},
},
},
{
Name: tableForecastDiscussions,
Columns: []fksinks.PostgresColumn{
{Name: "event_id", Type: "TEXT", Nullable: false},
{Name: "event_kind", Type: "TEXT", Nullable: false},
{Name: "event_source", Type: "TEXT", Nullable: false},
{Name: "event_schema", Type: "TEXT", Nullable: false},
{Name: "event_emitted_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "event_effective_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "office_id", Type: "TEXT", Nullable: true},
{Name: "office_name", Type: "TEXT", Nullable: true},
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "updated_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "product", Type: "TEXT", Nullable: false},
{Name: "short_term_qualifier", Type: "TEXT", Nullable: true},
{Name: "short_term_issued_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "short_term_text", Type: "TEXT", Nullable: true},
{Name: "long_term_qualifier", Type: "TEXT", Nullable: true},
{Name: "long_term_issued_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "long_term_text", Type: "TEXT", Nullable: true},
{Name: "key_message_count", Type: "INTEGER", Nullable: false},
},
PrimaryKey: []string{"event_id"},
PruneColumn: "issued_at",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_discussion_office_product_issued_at", Columns: []string{"office_id", "product", "issued_at"}},
{Name: "idx_wf_discussion_issued_at", Columns: []string{"issued_at"}},
},
},
{
Name: tableForecastDiscussionKeyMessages,
Columns: []fksinks.PostgresColumn{
{Name: "run_event_id", Type: "TEXT REFERENCES forecast_discussions(event_id) ON DELETE CASCADE", Nullable: false},
{Name: "message_index", Type: "INTEGER", Nullable: false},
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "message_text", Type: "TEXT", Nullable: true},
},
PrimaryKey: []string{"run_event_id", "message_index"},
PruneColumn: "issued_at",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_discussion_message_issued_at", Columns: []string{"issued_at"}},
},
},
{
Name: tableWeatherStoryRuns,
Columns: []fksinks.PostgresColumn{
{Name: "event_id", Type: "TEXT", Nullable: false},
{Name: "event_kind", Type: "TEXT", Nullable: false},
{Name: "event_source", Type: "TEXT", Nullable: false},
{Name: "event_schema", Type: "TEXT", Nullable: false},
{Name: "event_emitted_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "event_effective_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "office_id", Type: "TEXT", Nullable: true},
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "story_count", Type: "INTEGER", Nullable: false},
},
PrimaryKey: []string{"event_id"},
PruneColumn: "as_of",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_story_run_office_as_of", Columns: []string{"office_id", "as_of"}},
{Name: "idx_wf_story_run_as_of", Columns: []string{"as_of"}},
},
},
{
Name: tableWeatherStories,
Columns: []fksinks.PostgresColumn{
{Name: "run_event_id", Type: "TEXT REFERENCES weather_story_runs(event_id) ON DELETE CASCADE", Nullable: false},
{Name: "story_index", Type: "INTEGER", Nullable: false},
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "office_id", Type: "TEXT", Nullable: true},
{Name: "start_time", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "end_time", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "updated_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "title", Type: "TEXT", Nullable: true},
{Name: "description", Type: "TEXT", Nullable: true},
{Name: "alt_text", Type: "TEXT", Nullable: true},
{Name: "priority", Type: "BOOLEAN", Nullable: false},
{Name: "story_order", Type: "INTEGER", Nullable: false},
{Name: "download_url", Type: "TEXT", Nullable: true},
},
PrimaryKey: []string{"run_event_id", "story_index"},
PruneColumn: "as_of",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_stories_start_time", Columns: []string{"start_time"}},
{Name: "idx_wf_stories_end_time", Columns: []string{"end_time"}},
{Name: "idx_wf_stories_updated_at", Columns: []string{"updated_at"}},
},
},
{
Name: tableAlertRuns,
Columns: []fksinks.PostgresColumn{
@@ -232,6 +299,65 @@ func weatherPostgresSchema() fksinks.PostgresSchema {
{Name: "idx_wf_alert_refs_sent", Columns: []string{"sent"}},
},
},
{
Name: tableOutlookRuns,
Columns: []fksinks.PostgresColumn{
{Name: "event_id", Type: "TEXT", Nullable: false},
{Name: "event_kind", Type: "TEXT", Nullable: false},
{Name: "event_source", Type: "TEXT", Nullable: false},
{Name: "event_schema", Type: "TEXT", Nullable: false},
{Name: "event_emitted_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "event_effective_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "location_id", Type: "TEXT", Nullable: true},
{Name: "location_name", Type: "TEXT", Nullable: true},
{Name: "latitude", Type: "DOUBLE PRECISION", Nullable: true},
{Name: "longitude", Type: "DOUBLE PRECISION", Nullable: true},
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "outlook_count", Type: "INTEGER", Nullable: false},
},
PrimaryKey: []string{"event_id"},
PruneColumn: "as_of",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_outlook_run_location_as_of", Columns: []string{"location_id", "as_of"}},
{Name: "idx_wf_outlook_run_as_of", Columns: []string{"as_of"}},
},
},
{
Name: tableOutlooks,
Columns: []fksinks.PostgresColumn{
{Name: "run_event_id", Type: "TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE", Nullable: false},
{Name: "outlook_index", Type: "INTEGER", Nullable: false},
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "outlook_id", Type: "TEXT", Nullable: false},
{Name: "provider", Type: "TEXT", Nullable: false},
{Name: "product", Type: "TEXT", Nullable: false},
{Name: "day", Type: "INTEGER", Nullable: false},
{Name: "outlook_type", Type: "TEXT", Nullable: false},
{Name: "label", Type: "TEXT", Nullable: false},
{Name: "label_text", Type: "TEXT", Nullable: true},
{Name: "severity_rank", Type: "INTEGER", Nullable: true},
{Name: "valid_from", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "valid_to", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "expires_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "forecaster", Type: "TEXT", Nullable: true},
{Name: "headline", Type: "TEXT", Nullable: true},
{Name: "summary", Type: "TEXT", Nullable: true},
{Name: "discussion", Type: "TEXT", Nullable: true},
{Name: "source_url", Type: "TEXT", Nullable: true},
{Name: "image_url", Type: "TEXT", Nullable: true},
{Name: "contains_location", Type: "BOOLEAN", Nullable: false},
{Name: "geometry_json", Type: "TEXT", Nullable: false},
},
PrimaryKey: []string{"run_event_id", "outlook_index"},
PruneColumn: "as_of",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_outlooks_contains_valid", Columns: []string{"contains_location", "valid_from", "valid_to"}},
{Name: "idx_wf_outlooks_day_type_label", Columns: []string{"day", "outlook_type", "label"}},
{Name: "idx_wf_outlooks_valid", Columns: []string{"valid_from", "valid_to"}},
},
},
},
MapEvent: mapPostgresEvent,
}

View File

@@ -1,62 +1,14 @@
package postgres
import (
"fmt"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
)
func TestRegisterPostgresSchemasNilConfig(t *testing.T) {
err := RegisterPostgresSchemas(nil)
if err == nil {
t.Fatalf("RegisterPostgresSchemas(nil) expected error")
}
if !strings.Contains(err.Error(), "config is nil") {
t.Fatalf("error = %q, want config is nil", err)
}
}
func TestRegisterPostgresSchemasNonPostgresNoOp(t *testing.T) {
cfg := &config.Config{
Sinks: []config.SinkConfig{
{Name: "stdout_only", Driver: "stdout"},
{Name: "nats_only", Driver: "nats"},
},
}
if err := RegisterPostgresSchemas(cfg); err != nil {
t.Fatalf("RegisterPostgresSchemas(non-postgres) error = %v", err)
}
}
func TestRegisterPostgresSchemasDuplicateRegistrationFails(t *testing.T) {
sinkName := uniqueSinkName("pg_test")
cfg := &config.Config{
Sinks: []config.SinkConfig{
{Name: sinkName, Driver: "postgres"},
},
}
if err := RegisterPostgresSchemas(cfg); err != nil {
t.Fatalf("first RegisterPostgresSchemas() error = %v", err)
}
err := RegisterPostgresSchemas(cfg)
if err == nil {
t.Fatalf("second RegisterPostgresSchemas() expected duplicate error")
}
if !strings.Contains(err.Error(), "already registered") {
t.Fatalf("error = %q, want already registered", err)
}
}
func TestWeatherPostgresSchemaShape(t *testing.T) {
s := weatherPostgresSchema()
s := PostgresSchema()
if s.MapEvent == nil {
t.Fatalf("weatherPostgresSchema().MapEvent is nil")
t.Fatalf("PostgresSchema().MapEvent is nil")
}
wantTables := map[string]bool{
@@ -64,13 +16,19 @@ func TestWeatherPostgresSchemaShape(t *testing.T) {
tableObservationPresentWeather: true,
tableForecasts: true,
tableForecastPeriods: true,
tableForecastDiscussions: true,
tableForecastDiscussionKeyMessages: true,
tableWeatherStoryRuns: true,
tableWeatherStories: true,
tableAlertRuns: true,
tableAlerts: true,
tableAlertReferences: true,
tableOutlookRuns: true,
tableOutlooks: true,
}
if len(s.Tables) != len(wantTables) {
t.Fatalf("weatherPostgresSchema().Tables len = %d, want %d", len(s.Tables), len(wantTables))
t.Fatalf("PostgresSchema().Tables len = %d, want %d", len(s.Tables), len(wantTables))
}
seenIndexes := map[string]bool{}
@@ -90,6 +48,93 @@ func TestWeatherPostgresSchemaShape(t *testing.T) {
}
}
func uniqueSinkName(prefix string) string {
return fmt.Sprintf("%s_%d", prefix, time.Now().UnixNano())
func TestWeatherPostgresSchemaIncludesOutlookTables(t *testing.T) {
runColumns := columnsForTable(t, tableOutlookRuns)
for _, col := range []string{"event_id", "event_kind", "event_source", "event_schema", "event_emitted_at", "event_effective_at", "location_id", "location_name", "latitude", "longitude", "as_of", "issued_at", "outlook_count"} {
if !runColumns[col] {
t.Fatalf("%s missing %s column", tableOutlookRuns, col)
}
}
assertTablePrimaryKey(t, tableOutlookRuns, []string{"event_id"})
assertTableIndex(t, tableOutlookRuns, "idx_wf_outlook_run_location_as_of", []string{"location_id", "as_of"})
assertTableIndex(t, tableOutlookRuns, "idx_wf_outlook_run_as_of", []string{"as_of"})
outlookColumns := columnsForTable(t, tableOutlooks)
for _, col := range []string{"run_event_id", "outlook_index", "as_of", "outlook_id", "provider", "product", "day", "outlook_type", "label", "label_text", "severity_rank", "valid_from", "valid_to", "issued_at", "expires_at", "forecaster", "headline", "summary", "discussion", "source_url", "image_url", "contains_location", "geometry_json"} {
if !outlookColumns[col] {
t.Fatalf("%s missing %s column", tableOutlooks, col)
}
}
assertTablePrimaryKey(t, tableOutlooks, []string{"run_event_id", "outlook_index"})
assertTableIndex(t, tableOutlooks, "idx_wf_outlooks_contains_valid", []string{"contains_location", "valid_from", "valid_to"})
assertTableIndex(t, tableOutlooks, "idx_wf_outlooks_day_type_label", []string{"day", "outlook_type", "label"})
assertTableIndex(t, tableOutlooks, "idx_wf_outlooks_valid", []string{"valid_from", "valid_to"})
}
func TestWeatherPostgresSchemaIncludesWeatherStoryColumns(t *testing.T) {
runColumns := columnsForTable(t, tableWeatherStoryRuns)
if !runColumns["as_of"] {
t.Fatalf("%s missing as_of column", tableWeatherStoryRuns)
}
if !runColumns["story_count"] {
t.Fatalf("%s missing story_count column", tableWeatherStoryRuns)
}
storyColumns := columnsForTable(t, tableWeatherStories)
for _, col := range []string{"start_time", "end_time", "updated_at", "title", "description", "alt_text", "priority", "story_order", "download_url"} {
if !storyColumns[col] {
t.Fatalf("%s missing %s column", tableWeatherStories, col)
}
}
}
func assertTablePrimaryKey(t *testing.T, table string, want []string) {
t.Helper()
for _, tbl := range PostgresSchema().Tables {
if tbl.Name != table {
continue
}
if strings.Join(tbl.PrimaryKey, ",") != strings.Join(want, ",") {
t.Fatalf("%s primary key = %#v, want %#v", table, tbl.PrimaryKey, want)
}
return
}
t.Fatalf("missing table %q", table)
}
func assertTableIndex(t *testing.T, table string, name string, want []string) {
t.Helper()
for _, tbl := range PostgresSchema().Tables {
if tbl.Name != table {
continue
}
for _, idx := range tbl.Indexes {
if idx.Name == name {
if strings.Join(idx.Columns, ",") != strings.Join(want, ",") {
t.Fatalf("%s index %s columns = %#v, want %#v", table, name, idx.Columns, want)
}
return
}
}
t.Fatalf("%s missing index %s", table, name)
}
t.Fatalf("missing table %q", table)
}
func columnsForTable(t *testing.T, table string) map[string]bool {
t.Helper()
schema := PostgresSchema()
for _, tbl := range schema.Tables {
if tbl.Name != table {
continue
}
cols := make(map[string]bool, len(tbl.Columns))
for _, col := range tbl.Columns {
cols[col.Name] = true
}
return cols
}
t.Fatalf("missing table %q", table)
return nil
}

View File

@@ -4,38 +4,43 @@ import (
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/openmeteo"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/openweather"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/spc"
"gitea.maximumdirect.net/ejr/feedkit/config"
fksource "gitea.maximumdirect.net/ejr/feedkit/sources"
)
type pollDriverRegistration struct {
driver string
factory func(config.SourceConfig) (fksource.PollSource, error)
}
var pollDriverRegistrations = []pollDriverRegistration{
{driver: "nws_observation", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewObservationSource(cfg) }},
{driver: "nws_alerts", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewAlertsSource(cfg) }},
{driver: "nws_forecast_hourly", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewHourlyForecastSource(cfg) }},
{driver: "nws_forecast_narrative", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewNarrativeForecastSource(cfg) }},
{driver: "nws_forecast_discussion", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {
return nws.NewForecastDiscussionSource(cfg)
}},
{driver: "nws_weatherstories", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewWeatherStoriesSource(cfg) }},
{driver: "openmeteo_observation", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return openmeteo.NewObservationSource(cfg) }},
{driver: "openmeteo_forecast", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return openmeteo.NewForecastSource(cfg) }},
{driver: "openweather_observation", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {
return openweather.NewObservationSource(cfg)
}},
{driver: "spc_convective_outlook", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {
return spc.NewConvectiveOutlookSource(cfg)
}},
}
// RegisterBuiltins registers the source drivers that ship with this binary.
// Keeping this in one place makes main.go very readable.
func RegisterBuiltins(r *fksource.Registry) {
// NWS drivers
r.RegisterPoll("nws_observation", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return nws.NewObservationSource(cfg)
})
r.RegisterPoll("nws_alerts", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return nws.NewAlertsSource(cfg)
})
r.RegisterPoll("nws_forecast_hourly", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return nws.NewHourlyForecastSource(cfg)
})
r.RegisterPoll("nws_forecast_narrative", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return nws.NewNarrativeForecastSource(cfg)
})
// Open-Meteo drivers
r.RegisterPoll("openmeteo_observation", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return openmeteo.NewObservationSource(cfg)
})
r.RegisterPoll("openmeteo_forecast", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return openmeteo.NewForecastSource(cfg)
})
// OpenWeatherMap drivers
r.RegisterPoll("openweather_observation", func(cfg config.SourceConfig) (fksource.PollSource, error) {
return openweather.NewObservationSource(cfg)
for _, reg := range pollDriverRegistrations {
reg := reg
r.RegisterPoll(reg.driver, func(cfg config.SourceConfig) (fksource.PollSource, error) {
return reg.factory(cfg)
})
}
}

View File

@@ -34,6 +34,32 @@ func TestRegisterBuiltinsRegistersNWSNarrativeForecastDriver(t *testing.T) {
}
}
func TestRegisterBuiltinsRegistersNWSForecastDiscussionDriver(t *testing.T) {
reg := fksource.NewRegistry()
RegisterBuiltins(reg)
in, err := reg.BuildInput(sourceConfigForDriver("nws_forecast_discussion"))
if err != nil {
t.Fatalf("BuildInput(nws_forecast_discussion) error = %v", err)
}
if _, ok := in.(fksource.PollSource); !ok {
t.Fatalf("BuildInput(nws_forecast_discussion) type = %T, want PollSource", in)
}
}
func TestRegisterBuiltinsRegistersNWSWeatherStoriesDriver(t *testing.T) {
reg := fksource.NewRegistry()
RegisterBuiltins(reg)
in, err := reg.BuildInput(sourceConfigForDriver("nws_weatherstories"))
if err != nil {
t.Fatalf("BuildInput(nws_weatherstories) error = %v", err)
}
if _, ok := in.(fksource.PollSource); !ok {
t.Fatalf("BuildInput(nws_weatherstories) type = %T, want PollSource", in)
}
}
func TestRegisterBuiltinsDoesNotRegisterLegacyNWSForecastDriver(t *testing.T) {
reg := fksource.NewRegistry()
RegisterBuiltins(reg)
@@ -47,14 +73,51 @@ func TestRegisterBuiltinsDoesNotRegisterLegacyNWSForecastDriver(t *testing.T) {
}
}
func TestRegisterBuiltinsRegistersAllCurrentDrivers(t *testing.T) {
reg := fksource.NewRegistry()
RegisterBuiltins(reg)
drivers := []string{
"nws_observation",
"nws_alerts",
"nws_forecast_hourly",
"nws_forecast_narrative",
"nws_forecast_discussion",
"nws_weatherstories",
"openmeteo_observation",
"openmeteo_forecast",
"openweather_observation",
"spc_convective_outlook",
}
for _, driver := range drivers {
in, err := reg.BuildInput(sourceConfigForDriver(driver))
if err != nil {
t.Fatalf("BuildInput(%s) error = %v", driver, err)
}
if _, ok := in.(fksource.PollSource); !ok {
t.Fatalf("BuildInput(%s) type = %T, want PollSource", driver, in)
}
}
}
func sourceConfigForDriver(driver string) config.SourceConfig {
url := "https://example.invalid"
if driver == "openweather_observation" {
url = "https://example.invalid?units=metric"
}
params := map[string]any{
"url": url,
"user_agent": "test-agent",
}
if driver == "spc_convective_outlook" {
params["latitude"] = 38.6239
params["longitude"] = -90.3571
}
return config.SourceConfig{
Name: "test-source",
Driver: driver,
Mode: config.SourceModePoll,
Params: map[string]any{
"url": "https://example.invalid",
"user_agent": "test-agent",
},
Params: params,
}
}

View File

@@ -1,54 +0,0 @@
// FILE: ./internal/sources/common/event.go
package common
import (
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
)
// SingleRawEvent constructs, validates, and returns a slice containing exactly one event.
//
// This removes repetitive "event envelope ceremony" from individual sources.
// Sources remain responsible for:
// - fetching bytes (raw payload)
// - choosing Schema (raw schema identifier)
// - computing Event.ID and (optional) EffectiveAt
//
// emittedAt is explicit so callers can compute IDs using the same timestamp (or
// so tests can provide a stable value).
func SingleRawEvent(
kind event.Kind,
sourceName string,
schema string,
id string,
emittedAt time.Time,
effectiveAt *time.Time,
payload any,
) ([]event.Event, error) {
if emittedAt.IsZero() {
emittedAt = time.Now().UTC()
} else {
emittedAt = emittedAt.UTC()
}
e := event.Event{
ID: id,
Kind: kind,
Source: sourceName,
EmittedAt: emittedAt,
EffectiveAt: effectiveAt,
// RAW schema (normalizer matches on this).
Schema: schema,
// Raw payload (usually json.RawMessage). Normalizer will decode and map to canonical model.
Payload: payload,
}
if err := e.Validate(); err != nil {
return nil, err
}
return []event.Event{e}, nil
}

View File

@@ -1,39 +0,0 @@
// FILE: ./internal/sources/common/id.go
package common
import (
"fmt"
"strings"
"time"
)
// ChooseEventID applies weatherfeeder's opinionated Event.ID policy:
//
// - If upstream provides an ID, use it (trimmed).
// - Otherwise, ID is "<Source>:<EffectiveAt>" when available.
// - If EffectiveAt is unavailable, fall back to "<Source>:<EmittedAt>".
//
// Timestamps are encoded as RFC3339Nano in UTC.
func ChooseEventID(upstreamID, sourceName string, effectiveAt *time.Time, emittedAt time.Time) string {
if id := strings.TrimSpace(upstreamID); id != "" {
return id
}
src := strings.TrimSpace(sourceName)
if src == "" {
src = "UNKNOWN_SOURCE"
}
// Prefer EffectiveAt for dedupe friendliness.
if effectiveAt != nil && !effectiveAt.IsZero() {
return fmt.Sprintf("%s:%s", src, effectiveAt.UTC().Format(time.RFC3339Nano))
}
// Fall back to EmittedAt (still stable within a poll invocation).
t := emittedAt.UTC()
if t.IsZero() {
t = time.Now().UTC()
}
return fmt.Sprintf("%s:%s", src, t.Format(time.RFC3339Nano))
}

View File

@@ -11,7 +11,6 @@ import (
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
@@ -40,8 +39,8 @@ func NewAlertsSource(cfg config.SourceConfig) (*AlertsSource, error) {
func (s *AlertsSource) Name() string { return s.http.Name }
// Kind is used for routing/policy.
func (s *AlertsSource) Kind() event.Kind { return event.Kind("alert") }
// Kinds is used for routing/policy.
func (s *AlertsSource) Kinds() []event.Kind { return []event.Kind{event.Kind("alert")} }
func (s *AlertsSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx)
@@ -69,10 +68,10 @@ func (s *AlertsSource) Poll(ctx context.Context) ([]event.Event, error) {
// NWS alerts collections do not provide a stable per-snapshot ID.
// Use Source:EffectiveAt (or Source:EmittedAt fallback) for dedupe friendliness.
eventID := common.ChooseEventID("", s.http.Name, effectiveAt, emittedAt)
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return common.SingleRawEvent(
s.Kind(),
return fksources.SingleEvent(
event.Kind("alert"),
s.http.Name,
standards.SchemaRawNWSAlertsV1,
eventID,

View File

@@ -0,0 +1,114 @@
package nws
import (
"context"
"encoding/json"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
)
const nwsForecastAccept = "application/geo+json, application/json"
type forecastSource struct {
http *fksources.HTTPSource
rawSchema string
}
type forecastMeta struct {
Properties struct {
GeneratedAt string `json:"generatedAt"`
UpdateTime string `json:"updateTime"`
Updated string `json:"updated"`
} `json:"properties"`
ParsedGeneratedAt time.Time `json:"-"`
ParsedUpdateTime time.Time `json:"-"`
}
func newForecastSource(cfg config.SourceConfig, driver, rawSchema string) (*forecastSource, error) {
hs, err := fksources.NewHTTPSource(driver, cfg, nwsForecastAccept)
if err != nil {
return nil, err
}
return &forecastSource{
http: hs,
rawSchema: rawSchema,
}, nil
}
func (s *forecastSource) Name() string { return s.http.Name }
func (s *forecastSource) Kinds() []event.Kind { return []event.Kind{event.Kind("forecast")} }
func (s *forecastSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx)
if err != nil {
return nil, err
}
if !changed {
return nil, nil
}
var effectiveAt *time.Time
switch {
case !meta.ParsedGeneratedAt.IsZero():
t := meta.ParsedGeneratedAt.UTC()
effectiveAt = &t
case !meta.ParsedUpdateTime.IsZero():
t := meta.ParsedUpdateTime.UTC()
effectiveAt = &t
}
emittedAt := time.Now().UTC()
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent(
event.Kind("forecast"),
s.http.Name,
s.rawSchema,
eventID,
emittedAt,
effectiveAt,
raw,
)
}
func (s *forecastSource) fetchRaw(ctx context.Context) (json.RawMessage, forecastMeta, bool, error) {
raw, changed, err := s.http.FetchJSONIfChanged(ctx)
if err != nil {
return nil, forecastMeta{}, false, err
}
if !changed {
return nil, forecastMeta{}, false, nil
}
var meta forecastMeta
if err := json.Unmarshal(raw, &meta); err != nil {
return raw, forecastMeta{}, true, nil
}
genStr := strings.TrimSpace(meta.Properties.GeneratedAt)
if genStr != "" {
if t, err := nwscommon.ParseTime(genStr); err == nil {
meta.ParsedGeneratedAt = t.UTC()
}
}
updStr := strings.TrimSpace(meta.Properties.UpdateTime)
if updStr == "" {
updStr = strings.TrimSpace(meta.Properties.Updated)
}
if updStr != "" {
if t, err := nwscommon.ParseTime(updStr); err == nil {
meta.ParsedUpdateTime = t.UTC()
}
}
return raw, meta, true, nil
}

View File

@@ -0,0 +1,68 @@
package nws
import (
"context"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
// ForecastDiscussionSource polls an NWS forecast discussion HTML page and emits a RAW discussion Event.
//
// Output schema:
// - standards.SchemaRawNWSForecastDiscussionV1
type ForecastDiscussionSource struct {
http *fksources.HTTPSource
}
func NewForecastDiscussionSource(cfg config.SourceConfig) (*ForecastDiscussionSource, error) {
const driver = "nws_forecast_discussion"
hs, err := fksources.NewHTTPSource(driver, cfg, "text/html, application/xhtml+xml")
if err != nil {
return nil, err
}
return &ForecastDiscussionSource{http: hs}, nil
}
func (s *ForecastDiscussionSource) Name() string { return s.http.Name }
func (s *ForecastDiscussionSource) Kinds() []event.Kind {
return []event.Kind{event.Kind("forecast_discussion")}
}
func (s *ForecastDiscussionSource) Poll(ctx context.Context) ([]event.Event, error) {
body, changed, err := s.http.FetchBytesIfChanged(ctx)
if err != nil {
return nil, err
}
if !changed {
return nil, nil
}
rawHTML := string(body)
parsed, err := nwscommon.ParseForecastDiscussionHTML(rawHTML)
if err != nil {
return nil, err
}
issuedAt := parsed.IssuedAt.UTC()
effectiveAt := &issuedAt
emittedAt := time.Now().UTC()
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent(
event.Kind("forecast_discussion"),
s.http.Name,
standards.SchemaRawNWSForecastDiscussionV1,
eventID,
emittedAt,
effectiveAt,
rawHTML,
)
}

View File

@@ -0,0 +1,138 @@
package nws
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
func TestForecastDiscussionSourcePollEmitsExpectedEvent(t *testing.T) {
rawHTML := loadForecastDiscussionSampleHTML(t)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
_, _ = w.Write([]byte(rawHTML))
}))
defer srv.Close()
src, err := NewForecastDiscussionSource(forecastDiscussionSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewForecastDiscussionSource() error = %v", err)
}
if got := src.Kinds(); len(got) != 1 || got[0] != event.Kind("forecast_discussion") {
t.Fatalf("Kinds() = %#v, want [forecast_discussion]", got)
}
events, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(events) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(events))
}
got := events[0]
if got.Kind != event.Kind("forecast_discussion") {
t.Fatalf("Kind = %q, want forecast_discussion", got.Kind)
}
if got.Schema != standards.SchemaRawNWSForecastDiscussionV1 {
t.Fatalf("Schema = %q, want %q", got.Schema, standards.SchemaRawNWSForecastDiscussionV1)
}
wantEffectiveAt := time.Date(2026, 3, 28, 19, 24, 0, 0, time.UTC)
if got.EffectiveAt == nil || !got.EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("EffectiveAt = %v, want %s", got.EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
}
payload, ok := got.Payload.(string)
if !ok {
t.Fatalf("Payload type = %T, want string", got.Payload)
}
if payload != rawHTML {
t.Fatalf("Payload did not preserve exact HTML")
}
}
func TestForecastDiscussionSourcePollReturnsNoEventsWhenUnchanged(t *testing.T) {
rawHTML := loadForecastDiscussionSampleHTML(t)
const etag = `"discussion-v1"`
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("If-None-Match") == etag {
w.WriteHeader(http.StatusNotModified)
return
}
w.Header().Set("ETag", etag)
_, _ = w.Write([]byte(rawHTML))
}))
defer srv.Close()
src, err := NewForecastDiscussionSource(forecastDiscussionSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewForecastDiscussionSource() error = %v", err)
}
first, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("first Poll() error = %v", err)
}
if len(first) != 1 {
t.Fatalf("first Poll() len = %d, want 1", len(first))
}
second, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("second Poll() error = %v", err)
}
if len(second) != 0 {
t.Fatalf("second Poll() len = %d, want 0", len(second))
}
}
func TestForecastDiscussionSourcePollRejectsInvalidHTML(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte("<html><body><div>missing discussion block</div></body></html>"))
}))
defer srv.Close()
src, err := NewForecastDiscussionSource(forecastDiscussionSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewForecastDiscussionSource() error = %v", err)
}
_, err = src.Poll(context.Background())
if err == nil {
t.Fatalf("Poll() error = nil, want error")
}
if !strings.Contains(err.Error(), "glossaryProduct") {
t.Fatalf("error = %q, want glossaryProduct context", err)
}
}
func forecastDiscussionSourceConfig(url string) config.SourceConfig {
return config.SourceConfig{
Name: "test-forecast-discussion-source",
Driver: "nws_forecast_discussion",
Mode: config.SourceModePoll,
Params: map[string]any{
"url": url,
"user_agent": "test-agent",
},
}
}
func loadForecastDiscussionSampleHTML(t *testing.T) string {
t.Helper()
path := filepath.Join("..", "..", "providers", "nws", "testdata", "forecast_discussion_sample.html")
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("os.ReadFile(%q) error = %v", path, err)
}
return string(b)
}

View File

@@ -2,16 +2,7 @@
package nws
import (
"context"
"encoding/json"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
@@ -23,114 +14,15 @@ import (
// Output schema (current implementation):
// - standards.SchemaRawNWSHourlyForecastV1
type HourlyForecastSource struct {
http *fksources.HTTPSource
*forecastSource
}
func NewHourlyForecastSource(cfg config.SourceConfig) (*HourlyForecastSource, error) {
const driver = "nws_forecast_hourly"
// NWS forecast endpoints are GeoJSON (and sometimes also advertise json-ld/json).
hs, err := fksources.NewHTTPSource(driver, cfg, "application/geo+json, application/json")
src, err := newForecastSource(cfg, driver, standards.SchemaRawNWSHourlyForecastV1)
if err != nil {
return nil, err
}
return &HourlyForecastSource{http: hs}, nil
}
func (s *HourlyForecastSource) Name() string { return s.http.Name }
// Kind is used for routing/policy.
func (s *HourlyForecastSource) Kind() event.Kind { return event.Kind("forecast") }
func (s *HourlyForecastSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx)
if err != nil {
return nil, err
}
if !changed {
return nil, nil
}
// EffectiveAt is optional; for forecasts its most naturally the run “issued” time.
// NWS gridpoint forecasts expose generatedAt (preferred) and updateTime/updated.
var effectiveAt *time.Time
switch {
case !meta.ParsedGeneratedAt.IsZero():
t := meta.ParsedGeneratedAt.UTC()
effectiveAt = &t
case !meta.ParsedUpdateTime.IsZero():
t := meta.ParsedUpdateTime.UTC()
effectiveAt = &t
}
emittedAt := time.Now().UTC()
// NWS gridpoint forecast GeoJSON commonly has a stable "id" equal to the endpoint URL.
// That is *not* unique per issued run, so we intentionally do not use it for Event.ID.
// Instead we rely on Source:EffectiveAt (or Source:EmittedAt fallback).
eventID := common.ChooseEventID("", s.http.Name, effectiveAt, emittedAt)
return common.SingleRawEvent(
s.Kind(),
s.http.Name,
standards.SchemaRawNWSHourlyForecastV1,
eventID,
emittedAt,
effectiveAt,
raw,
)
}
// ---- RAW fetch + minimal metadata decode ----
type hourlyForecastMeta struct {
// Present for GeoJSON Feature responses, but often stable (endpoint URL).
ID string `json:"id"`
Properties struct {
GeneratedAt string `json:"generatedAt"` // preferred “issued/run generated” time
UpdateTime string `json:"updateTime"` // last update time of underlying data
Updated string `json:"updated"` // deprecated alias for updateTime
} `json:"properties"`
ParsedGeneratedAt time.Time `json:"-"`
ParsedUpdateTime time.Time `json:"-"`
}
func (s *HourlyForecastSource) fetchRaw(ctx context.Context) (json.RawMessage, hourlyForecastMeta, bool, error) {
raw, changed, err := s.http.FetchJSONIfChanged(ctx)
if err != nil {
return nil, hourlyForecastMeta{}, false, err
}
if !changed {
return nil, hourlyForecastMeta{}, false, nil
}
var meta hourlyForecastMeta
if err := json.Unmarshal(raw, &meta); err != nil {
// If metadata decode fails, still return raw; Poll will fall back to Source:EmittedAt.
return raw, hourlyForecastMeta{}, true, nil
}
// generatedAt (preferred)
genStr := strings.TrimSpace(meta.Properties.GeneratedAt)
if genStr != "" {
if t, err := nwscommon.ParseTime(genStr); err == nil {
meta.ParsedGeneratedAt = t.UTC()
}
}
// updateTime, with fallback to deprecated "updated"
updStr := strings.TrimSpace(meta.Properties.UpdateTime)
if updStr == "" {
updStr = strings.TrimSpace(meta.Properties.Updated)
}
if updStr != "" {
if t, err := nwscommon.ParseTime(updStr); err == nil {
meta.ParsedUpdateTime = t.UTC()
}
}
return raw, meta, true, nil
return &HourlyForecastSource{forecastSource: src}, nil
}

View File

@@ -2,16 +2,7 @@
package nws
import (
"context"
"encoding/json"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
@@ -23,114 +14,15 @@ import (
// Output schema:
// - standards.SchemaRawNWSNarrativeForecastV1
type NarrativeForecastSource struct {
http *fksources.HTTPSource
*forecastSource
}
func NewNarrativeForecastSource(cfg config.SourceConfig) (*NarrativeForecastSource, error) {
const driver = "nws_forecast_narrative"
// NWS forecast endpoints are GeoJSON (and sometimes also advertise json-ld/json).
hs, err := fksources.NewHTTPSource(driver, cfg, "application/geo+json, application/json")
src, err := newForecastSource(cfg, driver, standards.SchemaRawNWSNarrativeForecastV1)
if err != nil {
return nil, err
}
return &NarrativeForecastSource{http: hs}, nil
}
func (s *NarrativeForecastSource) Name() string { return s.http.Name }
// Kind is used for routing/policy.
func (s *NarrativeForecastSource) Kind() event.Kind { return event.Kind("forecast") }
func (s *NarrativeForecastSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx)
if err != nil {
return nil, err
}
if !changed {
return nil, nil
}
// EffectiveAt is optional; for forecasts its most naturally the run “issued” time.
// NWS gridpoint forecasts expose generatedAt (preferred) and updateTime/updated.
var effectiveAt *time.Time
switch {
case !meta.ParsedGeneratedAt.IsZero():
t := meta.ParsedGeneratedAt.UTC()
effectiveAt = &t
case !meta.ParsedUpdateTime.IsZero():
t := meta.ParsedUpdateTime.UTC()
effectiveAt = &t
}
emittedAt := time.Now().UTC()
// NWS gridpoint forecast GeoJSON commonly has a stable "id" equal to the endpoint URL.
// That is *not* unique per issued run, so we intentionally do not use it for Event.ID.
// Instead we rely on Source:EffectiveAt (or Source:EmittedAt fallback).
eventID := common.ChooseEventID("", s.http.Name, effectiveAt, emittedAt)
return common.SingleRawEvent(
s.Kind(),
s.http.Name,
standards.SchemaRawNWSNarrativeForecastV1,
eventID,
emittedAt,
effectiveAt,
raw,
)
}
// ---- RAW fetch + minimal metadata decode ----
type narrativeForecastMeta struct {
// Present for GeoJSON Feature responses, but often stable (endpoint URL).
ID string `json:"id"`
Properties struct {
GeneratedAt string `json:"generatedAt"` // preferred “issued/run generated” time
UpdateTime string `json:"updateTime"` // last update time of underlying data
Updated string `json:"updated"` // deprecated alias for updateTime
} `json:"properties"`
ParsedGeneratedAt time.Time `json:"-"`
ParsedUpdateTime time.Time `json:"-"`
}
func (s *NarrativeForecastSource) fetchRaw(ctx context.Context) (json.RawMessage, narrativeForecastMeta, bool, error) {
raw, changed, err := s.http.FetchJSONIfChanged(ctx)
if err != nil {
return nil, narrativeForecastMeta{}, false, err
}
if !changed {
return nil, narrativeForecastMeta{}, false, nil
}
var meta narrativeForecastMeta
if err := json.Unmarshal(raw, &meta); err != nil {
// If metadata decode fails, still return raw; Poll will fall back to Source:EmittedAt.
return raw, narrativeForecastMeta{}, true, nil
}
// generatedAt (preferred)
genStr := strings.TrimSpace(meta.Properties.GeneratedAt)
if genStr != "" {
if t, err := nwscommon.ParseTime(genStr); err == nil {
meta.ParsedGeneratedAt = t.UTC()
}
}
// updateTime, with fallback to deprecated "updated"
updStr := strings.TrimSpace(meta.Properties.UpdateTime)
if updStr == "" {
updStr = strings.TrimSpace(meta.Properties.Updated)
}
if updStr != "" {
if t, err := nwscommon.ParseTime(updStr); err == nil {
meta.ParsedUpdateTime = t.UTC()
}
}
return raw, meta, true, nil
return &NarrativeForecastSource{forecastSource: src}, nil
}

View File

@@ -0,0 +1,189 @@
package nws
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
type forecastPoller interface {
Poll(ctx context.Context) ([]event.Event, error)
}
func TestForecastSourcesEmitExpectedSchemaAndPreferGeneratedAt(t *testing.T) {
tests := []struct {
name string
driver string
wantSchema string
newSource func(config.SourceConfig) (forecastPoller, error)
}{
{
name: "hourly",
driver: "nws_forecast_hourly",
wantSchema: standards.SchemaRawNWSHourlyForecastV1,
newSource: func(cfg config.SourceConfig) (forecastPoller, error) {
return NewHourlyForecastSource(cfg)
},
},
{
name: "narrative",
driver: "nws_forecast_narrative",
wantSchema: standards.SchemaRawNWSNarrativeForecastV1,
newSource: func(cfg config.SourceConfig) (forecastPoller, error) {
return NewNarrativeForecastSource(cfg)
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"properties":{"generatedAt":"2026-03-28T12:00:00Z","updateTime":"2026-03-28T11:00:00Z"}}`))
}))
defer srv.Close()
src, err := tt.newSource(forecastSourceConfig(tt.driver, srv.URL))
if err != nil {
t.Fatalf("newSource() error = %v", err)
}
if ks, ok := src.(interface{ Kinds() []event.Kind }); !ok {
t.Fatalf("source does not implement Kinds()")
} else if gotKinds := ks.Kinds(); len(gotKinds) != 1 || gotKinds[0] != event.Kind("forecast") {
t.Fatalf("Kinds() = %#v, want [forecast]", gotKinds)
}
got, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(got) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(got))
}
if got[0].Schema != tt.wantSchema {
t.Fatalf("Poll() schema = %q, want %q", got[0].Schema, tt.wantSchema)
}
if got[0].Kind != event.Kind("forecast") {
t.Fatalf("Poll() kind = %q, want forecast", got[0].Kind)
}
wantEffectiveAt := time.Date(2026, 3, 28, 12, 0, 0, 0, time.UTC)
if got[0].EffectiveAt == nil || !got[0].EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("Poll() effectiveAt = %v, want %s", got[0].EffectiveAt, wantEffectiveAt)
}
})
}
}
func TestForecastSourcePollEffectiveAtFallbackOrder(t *testing.T) {
tests := []struct {
name string
body string
wantEffectiveAt *time.Time
}{
{
name: "updateTime fallback",
body: `{"properties":{"updateTime":"2026-03-28T11:00:00Z"}}`,
wantEffectiveAt: func() *time.Time {
t := time.Date(2026, 3, 28, 11, 0, 0, 0, time.UTC)
return &t
}(),
},
{
name: "updated fallback",
body: `{"properties":{"updated":"2026-03-28T10:00:00Z"}}`,
wantEffectiveAt: func() *time.Time {
t := time.Date(2026, 3, 28, 10, 0, 0, 0, time.UTC)
return &t
}(),
},
{
name: "omitted when metadata lacks timestamps",
body: `{"properties":{}}`,
wantEffectiveAt: nil,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(tt.body))
}))
defer srv.Close()
src, err := NewHourlyForecastSource(forecastSourceConfig("nws_forecast_hourly", srv.URL))
if err != nil {
t.Fatalf("NewHourlyForecastSource() error = %v", err)
}
got, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(got) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(got))
}
if tt.wantEffectiveAt == nil {
if got[0].EffectiveAt != nil {
t.Fatalf("Poll() effectiveAt = %v, want nil", got[0].EffectiveAt)
}
return
}
if got[0].EffectiveAt == nil || !got[0].EffectiveAt.Equal(*tt.wantEffectiveAt) {
t.Fatalf("Poll() effectiveAt = %v, want %s", got[0].EffectiveAt, *tt.wantEffectiveAt)
}
})
}
}
func TestForecastSourcePollMetadataDecodeFailureStillEmitsRawEvent(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`not-json`))
}))
defer srv.Close()
src, err := NewNarrativeForecastSource(forecastSourceConfig("nws_forecast_narrative", srv.URL))
if err != nil {
t.Fatalf("NewNarrativeForecastSource() error = %v", err)
}
got, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(got) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(got))
}
if got[0].EffectiveAt != nil {
t.Fatalf("Poll() effectiveAt = %v, want nil", got[0].EffectiveAt)
}
if got[0].Schema != standards.SchemaRawNWSNarrativeForecastV1 {
t.Fatalf("Poll() schema = %q, want %q", got[0].Schema, standards.SchemaRawNWSNarrativeForecastV1)
}
raw, ok := got[0].Payload.(json.RawMessage)
if !ok {
t.Fatalf("Poll() payload type = %T, want json.RawMessage", got[0].Payload)
}
if string(raw) != "not-json" {
t.Fatalf("Poll() payload = %q, want %q", string(raw), "not-json")
}
}
func forecastSourceConfig(driver, url string) config.SourceConfig {
return config.SourceConfig{
Name: "test-forecast-source",
Driver: driver,
Mode: config.SourceModePoll,
Params: map[string]any{
"url": url,
"user_agent": "test-agent",
},
}
}

View File

@@ -11,7 +11,6 @@ import (
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
@@ -33,7 +32,7 @@ func NewObservationSource(cfg config.SourceConfig) (*ObservationSource, error) {
func (s *ObservationSource) Name() string { return s.http.Name }
func (s *ObservationSource) Kind() event.Kind { return event.Kind("observation") }
func (s *ObservationSource) Kinds() []event.Kind { return []event.Kind{event.Kind("observation")} }
func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx)
@@ -52,10 +51,10 @@ func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
}
emittedAt := time.Now().UTC()
eventID := common.ChooseEventID(meta.ID, s.http.Name, effectiveAt, emittedAt)
eventID := fksources.DefaultEventID(meta.ID, s.http.Name, effectiveAt, emittedAt)
return common.SingleRawEvent(
s.Kind(),
return fksources.SingleEvent(
event.Kind("observation"),
s.http.Name,
standards.SchemaRawNWSObservationV1,
eventID,

Some files were not shown because too many files have changed in this diff Show More