123 lines
4.5 KiB
Markdown
123 lines
4.5 KiB
Markdown
# Source Internals
|
|
|
|
## Purpose
|
|
|
|
Source packages poll upstream weather providers and emit raw feed events. They
|
|
are adapters, not canonical mappers.
|
|
|
|
Sources should decode only the metadata needed for event identity, effective
|
|
time, and routing policy. Full provider payload interpretation belongs in
|
|
normalizers.
|
|
|
|
## Inputs And Outputs
|
|
|
|
Inputs are feedkit `config.SourceConfig` values and upstream HTTP responses.
|
|
Outputs are feed events whose payloads are raw provider JSON and whose schemas
|
|
come from `standards`.
|
|
|
|
Current drivers:
|
|
|
|
| Driver | Kind | Raw schema |
|
|
| --- | --- | --- |
|
|
| `nws_observation` | `observation` | `raw.nws.observation.v1` |
|
|
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` |
|
|
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` |
|
|
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` |
|
|
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` |
|
|
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` |
|
|
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` |
|
|
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` |
|
|
| `openweather_observation` | `observation` | `raw.openweather.current.v1` |
|
|
| `spc_convective_outlook` | `outlook` | `raw.spc.convective_outlook.v1` |
|
|
|
|
## Boundaries
|
|
|
|
- Source constructors validate source-specific params.
|
|
- Sources use feedkit HTTP helpers for HTTP polling.
|
|
- Sources emit raw events and should not build canonical `model` payloads.
|
|
- Provider helper packages under `internal/providers/<provider>` hold shared
|
|
parsing and validation helpers.
|
|
- Registration is centralized in `internal/sources/builtins.go`.
|
|
|
|
## Config Fields Used
|
|
|
|
Most source drivers use feedkit `HTTPSource`.
|
|
|
|
Required params:
|
|
|
|
- `url`
|
|
- `user_agent`
|
|
|
|
Optional params:
|
|
|
|
- `conditional`, default `true`;
|
|
- `http_timeout`;
|
|
- `http_response_body_limit_bytes`.
|
|
|
|
OpenWeather observation sources additionally require the configured URL to use
|
|
metric units. This is enforced by `internal/providers/openweather`.
|
|
|
|
The SPC convective outlook source is a multi-document poll source rather than a
|
|
single-URL `HTTPSource`. It requires `latitude`, `longitude`, and `user_agent`;
|
|
accepts optional `location_id`, `location_name`, `geojson_urls`,
|
|
`discussion_urls`, and `rss_url`; and supports `http_timeout` and
|
|
`http_response_body_limit_bytes`.
|
|
|
|
Source-level `kinds`, when configured, are validated against the source's
|
|
advertised `Kinds()`.
|
|
|
|
## External Adapters Used
|
|
|
|
Most sources use feedkit's HTTP helper for:
|
|
|
|
- request construction;
|
|
- `User-Agent` and `Accept` headers;
|
|
- optional conditional GET validators;
|
|
- response body size limits;
|
|
- JSON raw-message fetches.
|
|
|
|
NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times
|
|
with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs.
|
|
SPC helpers define Day 1-3 product metadata, parse GeoJSON timestamps, extract
|
|
cleaned print-page discussion text, and parse optional RSS metadata.
|
|
|
|
## State
|
|
|
|
HTTP conditional validators are held in each single-document HTTP source
|
|
instance. They are not persisted across process restarts. The SPC outlook source
|
|
keeps only a source-local hash of the most recent complete required product
|
|
bundle and emits no event when a later complete bundle is unchanged.
|
|
|
|
## Failure Behavior
|
|
|
|
Constructor failures are returned during startup and stop the daemon. Polling
|
|
failures are returned to the scheduler.
|
|
|
|
If a source cannot decode minimal metadata from an otherwise successful upstream
|
|
response, it still emits the raw event when possible. The event then falls back
|
|
to default ID/effective-time behavior from feedkit source helpers.
|
|
|
|
Unchanged conditional responses return no events and no error.
|
|
|
|
The SPC outlook source fetches all configured Day 1-3 GeoJSON products and print
|
|
pages atomically. If any required GeoJSON or print-page request fails, the poll
|
|
returns an error and emits no partial bundle. RSS is fetched only when `rss_url`
|
|
is configured.
|
|
|
|
## Tests To Inspect
|
|
|
|
- `internal/sources/builtins_test.go`
|
|
- provider source tests under `internal/sources/nws`
|
|
- provider source tests under `internal/sources/openmeteo`
|
|
- provider source tests under `internal/sources/openweather`
|
|
- provider source tests under `internal/sources/spc`
|
|
- provider helper tests under `internal/providers`
|
|
|
|
## Invariants
|
|
|
|
- Emit raw provider schemas from `standards`.
|
|
- Keep provider-to-canonical mapping out of sources.
|
|
- Keep HTTP behavior context-aware.
|
|
- Keep driver registration explicit and stable.
|
|
- Keep source tests independent of live upstream services.
|