Eric Rakestraw
50215d2105
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
124 lines
5.0 KiB
Markdown
124 lines
5.0 KiB
Markdown
# NWS Integration Notes
|
|
|
|
## Purpose
|
|
|
|
This document describes the NWS products that `weatherfeeder` currently polls
|
|
and normalizes. It is for developers and operators maintaining NWS source URLs,
|
|
normalizers, fixtures, and tests.
|
|
|
|
General config syntax belongs in [configuration](../config.md). Emitted JSON
|
|
events are documented in [event wire contract](events.md).
|
|
|
|
## Implemented Drivers
|
|
|
|
| Driver | Kind | Raw schema | Canonical schema |
|
|
| --- | --- | --- | --- |
|
|
| `nws_observation` | `observation` | `raw.nws.observation.v1` | `weather.observation.v1` |
|
|
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` | `weather.alert.v1` |
|
|
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
|
|
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
|
|
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
|
|
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
|
|
|
|
## Config Requirements
|
|
|
|
All NWS drivers require HTTP source params:
|
|
|
|
- `url`
|
|
- `user_agent`
|
|
|
|
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
|
|
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
|
|
an upstream `304 Not Modified` response emits no event for that poll.
|
|
|
|
NWS expects a descriptive `User-Agent`. Do not use anonymous or placeholder
|
|
contact values in production configs.
|
|
|
|
## Upstream Shapes Used
|
|
|
|
`nws_observation` expects the latest station observation GeoJSON shape. The
|
|
normalizer uses fields under `properties` such as `stationId`, `stationName`,
|
|
`timestamp`, `textDescription`, measured values, `presentWeather`, and
|
|
`cloudLayers`, plus point geometry for day/night inference.
|
|
|
|
`nws_alerts` expects an alerts FeatureCollection. The normalizer uses the
|
|
collection `updated` timestamp, `title`, each feature ID, alert classification
|
|
fields, narrative fields, timing fields, sender fields, and references.
|
|
`properties.onset` and `properties.ends` map to the canonical alert period
|
|
start and end. `properties.expires` maps only to canonical `expires` provider
|
|
metadata and is not treated as the alert period end.
|
|
|
|
`nws_forecast_hourly` and `nws_forecast_narrative` expect gridpoint forecast
|
|
GeoJSON with `properties.generatedAt`, `properties.updateTime`, elevation,
|
|
polygon geometry, and ordered `periods`.
|
|
|
|
`nws_forecast_discussion` expects an HTML page containing the discussion text in
|
|
a `<pre>` block. The provider helper extracts office identity, product, issue
|
|
time, update time, key messages, and short/long term sections.
|
|
|
|
`nws_weatherstories` expects a JSON response with a `stories` array. The
|
|
normalizer uses office ID, start/end/update times, title, description, alt text,
|
|
priority, order, and download URL.
|
|
|
|
## Accept Headers
|
|
|
|
NWS JSON sources request:
|
|
|
|
```text
|
|
application/geo+json, application/json
|
|
```
|
|
|
|
The forecast discussion source requests:
|
|
|
|
```text
|
|
text/html, application/xhtml+xml
|
|
```
|
|
|
|
## Effective Time
|
|
|
|
Source events set `effective_at` from the best metadata available:
|
|
|
|
- observations: `properties.timestamp`;
|
|
- alerts: collection `updated`, otherwise latest per-alert timestamp;
|
|
- hourly and narrative forecasts: `properties.generatedAt`, otherwise update
|
|
time;
|
|
- forecast discussions: parsed issue time;
|
|
- weather stories: latest story update time, otherwise latest story start time.
|
|
|
|
Normalizers use canonical payload time as the normalized event effective time.
|
|
Alerts and weather stories fall back to the incoming event envelope when the
|
|
payload does not provide a better snapshot time.
|
|
|
|
## Mapping Notes
|
|
|
|
Observations preserve raw `presentWeather` fragments and infer WMO condition
|
|
codes from METAR phenomena, provider text, and cloud-layer fallback. Sea-level
|
|
pressure is preferred over barometric pressure when present.
|
|
|
|
Hourly forecasts infer WMO condition codes from `shortForecast` and icon tokens.
|
|
Narrative forecasts preserve text but intentionally leave period condition codes
|
|
unset. Forecast temperatures are converted to Celsius when NWS supplies
|
|
Fahrenheit, and wind speed strings are converted to kilometers per hour.
|
|
|
|
Alert timing fields are parsed best-effort. Invalid per-alert timestamps are
|
|
left unset rather than failing the whole alert run. NWS `ends` is preserved
|
|
separately from `expires`; `expires` does not fall back to `ends`. Missing alert
|
|
IDs are synthesized from the run snapshot time and array position.
|
|
|
|
Forecast discussion parsing requires an issue time. Weather story entries require
|
|
start time, end time, and update time.
|
|
|
|
## Failure Behavior
|
|
|
|
Constructor validation failures stop daemon startup. Polling failures are
|
|
returned to the scheduler. JSON sources still emit raw payloads when only
|
|
minimal metadata decoding fails. Forecast discussion polling fails if the HTML
|
|
cannot be parsed enough to determine the issue time.
|
|
|
|
## Tests To Inspect
|
|
|
|
- `internal/sources/nws/*_test.go`
|
|
- `internal/normalizers/nws/*_test.go`
|
|
- `internal/providers/nws/*_test.go`
|
|
- fixtures under `internal/providers/nws/testdata`
|