diff --git a/README.md b/README.md index beea461..4db5010 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,9 @@ current working directory. - [Example configs](examples/) - [Event wire contract](docs/integrations/events.md) - [Postgres table contract](docs/integrations/postgres.md) +- [NWS integration notes](docs/integrations/nws.md) +- [Open-Meteo integration notes](docs/integrations/openmeteo.md) +- [OpenWeather integration notes](docs/integrations/openweather.md) - [Architecture policy](docs/policy/architecture.md) - [Development policy](docs/policy/development.md) - [Documentation policy](docs/policy/documentation.md) diff --git a/docs/integrations/nws.md b/docs/integrations/nws.md new file mode 100644 index 0000000..ba669eb --- /dev/null +++ b/docs/integrations/nws.md @@ -0,0 +1,119 @@ +# NWS Integration Notes + +## Purpose + +This document describes the NWS products that `weatherfeeder` currently polls +and normalizes. It is for developers and operators maintaining NWS source URLs, +normalizers, fixtures, and tests. + +General config syntax belongs in [configuration](../config.md). Emitted JSON +events are documented in [event wire contract](events.md). + +## Implemented Drivers + +| Driver | Kind | Raw schema | Canonical schema | +| --- | --- | --- | --- | +| `nws_observation` | `observation` | `raw.nws.observation.v1` | `weather.observation.v1` | +| `nws_alerts` | `alert` | `raw.nws.alerts.v1` | `weather.alert.v1` | +| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` | +| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` | +| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` | +| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` | `weather.weather_story.v1` | + +## Config Requirements + +All NWS drivers require HTTP source params: + +- `url` +- `user_agent` + +The shared HTTP helper also accepts `conditional`, `http_timeout`, and +`http_response_body_limit_bytes`. Conditional requests are enabled by default; +an upstream `304 Not Modified` response emits no event for that poll. + +NWS expects a descriptive `User-Agent`. Do not use anonymous or placeholder +contact values in production configs. + +## Upstream Shapes Used + +`nws_observation` expects the latest station observation GeoJSON shape. The +normalizer uses fields under `properties` such as `stationId`, `stationName`, +`timestamp`, `textDescription`, measured values, `presentWeather`, and +`cloudLayers`, plus point geometry for day/night inference. + +`nws_alerts` expects an alerts FeatureCollection. The normalizer uses the +collection `updated` timestamp, `title`, each feature ID, alert classification +fields, narrative fields, timing fields, sender fields, and references. + +`nws_forecast_hourly` and `nws_forecast_narrative` expect gridpoint forecast +GeoJSON with `properties.generatedAt`, `properties.updateTime`, elevation, +polygon geometry, and ordered `periods`. + +`nws_forecast_discussion` expects an HTML page containing the discussion text in +a `
` block. The provider helper extracts office identity, product, issue +time, update time, key messages, and short/long term sections. + +`nws_weatherstories` expects a JSON response with a `stories` array. The +normalizer uses office ID, start/end/update times, title, description, alt text, +priority, order, and download URL. + +## Accept Headers + +NWS JSON sources request: + +```text +application/geo+json, application/json +``` + +The forecast discussion source requests: + +```text +text/html, application/xhtml+xml +``` + +## Effective Time + +Source events set `effective_at` from the best metadata available: + +- observations: `properties.timestamp`; +- alerts: collection `updated`, otherwise latest per-alert timestamp; +- hourly and narrative forecasts: `properties.generatedAt`, otherwise update + time; +- forecast discussions: parsed issue time; +- weather stories: latest story update time, otherwise latest story start time. + +Normalizers use canonical payload time as the normalized event effective time. +Alerts and weather stories fall back to the incoming event envelope when the +payload does not provide a better snapshot time. + +## Mapping Notes + +Observations preserve raw `presentWeather` fragments and infer WMO condition +codes from METAR phenomena, provider text, and cloud-layer fallback. Sea-level +pressure is preferred over barometric pressure when present. + +Hourly forecasts infer WMO condition codes from `shortForecast` and icon tokens. +Narrative forecasts preserve text but intentionally leave period condition codes +unset. Forecast temperatures are converted to Celsius when NWS supplies +Fahrenheit, and wind speed strings are converted to kilometers per hour. + +Alert timing fields are parsed best-effort. Invalid per-alert timestamps are +left unset rather than failing the whole alert run. Missing alert IDs are +synthesized from the run snapshot time and array position. + +Forecast discussion parsing requires an issue time. Weather story entries require +start time, end time, and update time. + +## Failure Behavior + +Constructor validation failures stop daemon startup. Polling failures are +returned to the scheduler. JSON sources still emit raw payloads when only +minimal metadata decoding fails. Forecast discussion polling fails if the HTML +cannot be parsed enough to determine the issue time. + +## Tests To Inspect + +- `internal/sources/nws/*_test.go` +- `internal/normalizers/nws/*_test.go` +- `internal/providers/nws/*_test.go` +- fixtures under `internal/providers/nws/testdata` diff --git a/docs/integrations/openmeteo.md b/docs/integrations/openmeteo.md new file mode 100644 index 0000000..80e9289 --- /dev/null +++ b/docs/integrations/openmeteo.md @@ -0,0 +1,101 @@ +# Open-Meteo Integration Notes + +## Purpose + +This document describes the Open-Meteo API usage currently implemented by +`weatherfeeder`. It is for developers and operators maintaining Open-Meteo +source URLs, normalizers, fixtures, and tests. + +General config syntax belongs in [configuration](../config.md). Emitted JSON +events are documented in [event wire contract](events.md). + +## Implemented Drivers + +| Driver | Kind | Raw schema | Canonical schema | +| --- | --- | --- | --- | +| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` | `weather.observation.v1` | +| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` | + +## Config Requirements + +Both drivers require HTTP source params: + +- `url` +- `user_agent` + +The shared HTTP helper also accepts `conditional`, `http_timeout`, and +`http_response_body_limit_bytes`. Conditional requests are enabled by default; +an upstream `304 Not Modified` response emits no event for that poll. + +## Upstream Shapes Used + +`openmeteo_observation` expects a JSON response with top-level location/timezone +metadata and a `current` object. The normalizer uses: + +- `latitude`, `longitude`, `timezone`, `utc_offset_seconds`; +- `current.time`; +- current temperature, apparent temperature, relative humidity, weather code, + wind speed/direction/gusts, pressure, and `is_day`. + +`openmeteo_forecast` expects top-level location/timezone metadata and an +array-oriented `hourly` object. The normalizer uses: + +- `hourly.time`; +- hourly temperature, apparent temperature, dew point, relative humidity, + precipitation probability, precipitation amount, snowfall, weather code, + pressure, wind speed/direction/gusts, `is_day`, cloud cover, visibility, and + UV index. + +Open-Meteo field presence is allowed to vary. Missing optional arrays produce +nil canonical fields for the affected periods. + +## Accept Header + +Open-Meteo sources request: + +```text +application/json +``` + +## Time Handling + +Open-Meteo timestamps often omit an explicit offset. The provider helper parses +times by using the returned `timezone` or `utc_offset_seconds` when needed. + +Observation source events set `effective_at` from `current.time` when it can be +parsed. Hourly forecast source events prefer `current.time`, then the first +non-empty `hourly.time` entry. + +The hourly forecast normalizer sets canonical `issuedAt` from the incoming event +`emitted_at` when present, otherwise from the first hourly period start. +Normalized forecast `effective_at` matches `issuedAt`. + +## Mapping Notes + +Open-Meteo is not a station feed. Weatherfeeder synthesizes canonical +station/location IDs from latitude and longitude when both are available. + +Open-Meteo weather codes are WMO codes and are treated as authoritative. +Canonical text is derived from the WMO code and day/night hint. + +Wind speed and gust fields are treated as kilometers per hour. Pressure values +are treated as hPa and converted to Pa. Snowfall values are treated as +centimeters and converted to millimeters. + +Hourly forecast period end time is the next period start. The last period uses +the previous interval length, or one hour when there is no previous interval. + +## Failure Behavior + +Constructor validation failures stop daemon startup. Polling failures are +returned to the scheduler. Metadata decoding failures in sources still allow raw +payload emission when the HTTP response itself succeeded. + +Normalization fails when required time data is missing or invalid, such as an +empty `hourly.time` array for hourly forecasts. + +## Tests To Inspect + +- `internal/sources/openmeteo/source_test.go` +- `internal/normalizers/openmeteo/*_test.go` +- `internal/providers/openmeteo/*_test.go` diff --git a/docs/integrations/openweather.md b/docs/integrations/openweather.md new file mode 100644 index 0000000..e3db4e1 --- /dev/null +++ b/docs/integrations/openweather.md @@ -0,0 +1,101 @@ +# OpenWeather Integration Notes + +## Purpose + +This document describes the OpenWeather current-weather usage implemented by +`weatherfeeder`. It is for developers and operators maintaining OpenWeather +source URLs, normalizers, fixtures, and tests. + +General config syntax belongs in [configuration](../config.md). Emitted JSON +events are documented in [event wire contract](events.md). + +## Implemented Driver + +| Driver | Kind | Raw schema | Canonical schema | +| --- | --- | --- | --- | +| `openweather_observation` | `observation` | `raw.openweather.current.v1` | `weather.observation.v1` | + +Only current-weather observation polling is registered for OpenWeather. + +## Config Requirements + +The driver requires HTTP source params: + +- `url` +- `user_agent` + +The shared HTTP helper also accepts `conditional`, `http_timeout`, and +`http_response_body_limit_bytes`. Conditional requests are enabled by default; +an upstream `304 Not Modified` response emits no event for that poll. + +The configured URL must include: + +```text +units=metric +``` + +Startup fails if `units` is omitted or set to another value. Keep OpenWeather +API keys out of committed configs. Use local config management or deployment +secrets for the `appid` query parameter. + +## Upstream Shape Used + +The source emits the full current-weather JSON payload as a raw event. The +normalizer uses: + +- `coord.lat`, `coord.lon`; +- primary `weather[0]` condition ID, description, and icon; +- `main.temp`, `main.feels_like`, `main.pressure`, `main.humidity`, and + optional `main.sea_level`; +- `visibility`; +- `wind.speed`, `wind.deg`, and `wind.gust`; +- `dt`; +- `sys.sunrise` and `sys.sunset`; +- `id` and `name`. + +## Accept Header + +OpenWeather sources request: + +```text +application/json +``` + +## Time Handling + +Source events set `effective_at` from `dt` when it is present and positive. +The normalizer also uses `dt` as the canonical observation timestamp and +normalized effective time. + +## Mapping Notes + +Metric units are required so canonical unit conversion is deterministic: + +- `main.temp` and `main.feels_like` are treated as Celsius; +- `wind.speed` and `wind.gust` are treated as meters per second and converted to + kilometers per hour; +- pressure values are treated as hPa and converted to Pa. + +The primary condition is `weather[0]`. OpenWeather condition IDs are mapped into +the canonical WMO code vocabulary. The human text description is preserved from +the provider description. + +Day/night is inferred from the OpenWeather icon suffix when available, otherwise +from sunrise and sunset bounds. + +The station ID uses the OpenWeather city ID when present. If no city ID is +present, weatherfeeder synthesizes an ID from coordinates. The station name uses +the provider `name`, falling back to `OpenWeatherMap` when blank. + +## Failure Behavior + +Constructor validation failures stop daemon startup. Polling also re-checks the +metric-unit requirement before fetching. HTTP failures are returned to the +scheduler. Metadata decoding failures in the source still allow raw payload +emission when the HTTP response itself succeeded. + +## Tests To Inspect + +- `internal/sources/openweather/source_test.go` +- `internal/normalizers/openweather/*_test.go` +- `internal/providers/openweather/*_test.go`