Source Internals

Purpose

Source packages poll upstream weather providers and emit raw feed events. They are adapters, not canonical mappers.

Sources should decode only the metadata needed for event identity, effective time, and routing policy. Full provider payload interpretation belongs in normalizers.

Inputs And Outputs

Inputs are feedkit config.SourceConfig values and upstream HTTP responses. Outputs are feed events whose payloads are raw provider JSON and whose schemas come from standards.

Current drivers:

Driver	Kind	Raw schema
`nws_observation`	`observation`	`raw.nws.observation.v1`
`nws_alerts`	`alert`	`raw.nws.alerts.v1`
`nws_forecast_hourly`	`forecast`	`raw.nws.hourly.forecast.v1`
`nws_forecast_narrative`	`forecast`	`raw.nws.narrative.forecast.v1`
`nws_forecast_discussion`	`forecast_discussion`	`raw.nws.forecast_discussion.v1`
`nws_weatherstories`	`weather_story`	`raw.nws.weatherstories.v1`
`openmeteo_observation`	`observation`	`raw.openmeteo.current.v1`
`openmeteo_forecast`	`forecast`	`raw.openmeteo.hourly.forecast.v1`
`openweather_observation`	`observation`	`raw.openweather.current.v1`
`spc_convective_outlook`	`outlook`	`raw.spc.convective_outlook.v1`

Boundaries

Source constructors validate source-specific params.
Sources use feedkit HTTP helpers for HTTP polling.
Sources emit raw events and should not build canonical model payloads.
Provider helper packages under internal/providers/<provider> hold shared parsing and validation helpers.
Registration is centralized in internal/sources/builtins.go.

Config Fields Used

Most source drivers use feedkit HTTPSource.

Required params:

url
user_agent

Optional params:

conditional, default true;
http_timeout;
http_response_body_limit_bytes.

OpenWeather observation sources additionally require the configured URL to use metric units. This is enforced by internal/providers/openweather.

The SPC convective outlook source is a multi-document poll source rather than a single-URL HTTPSource. It requires latitude, longitude, and user_agent; accepts optional location_id, location_name, geojson_urls, discussion_urls, and rss_url; and supports http_timeout and http_response_body_limit_bytes.

Source-level kinds, when configured, are validated against the source's advertised Kinds().

External Adapters Used

Most sources use feedkit's HTTP helper for:

request construction;
User-Agent and Accept headers;
optional conditional GET validators;
response body size limits;
JSON raw-message fetches.

NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs. SPC helpers define Day 1-3 product metadata, parse GeoJSON timestamps, extract cleaned print-page discussion text, and parse optional RSS metadata.

State

HTTP conditional validators are held in each single-document HTTP source instance. They are not persisted across process restarts. The SPC outlook source keeps only a source-local hash of the most recent complete required product bundle and emits no event when a later complete bundle is unchanged.

Failure Behavior

Constructor failures are returned during startup and stop the daemon. Polling failures are returned to the scheduler.

If a source cannot decode minimal metadata from an otherwise successful upstream response, it still emits the raw event when possible. The event then falls back to default ID/effective-time behavior from feedkit source helpers.

Unchanged conditional responses return no events and no error.

The SPC outlook source fetches all configured Day 1-3 GeoJSON products and print pages atomically. If any required GeoJSON or print-page request fails, the poll returns an error and emits no partial bundle. RSS is fetched only when rss_url is configured.

Tests To Inspect

internal/sources/builtins_test.go
provider source tests under internal/sources/nws
provider source tests under internal/sources/openmeteo
provider source tests under internal/sources/openweather
provider source tests under internal/sources/spc
provider helper tests under internal/providers

Invariants

Emit raw provider schemas from standards.
Keep provider-to-canonical mapping out of sources.
Keep HTTP behavior context-aware.
Keep driver registration explicit and stable.
Keep source tests independent of live upstream services.

4.5 KiB Raw Blame History