106 lines
3.5 KiB
Markdown
106 lines
3.5 KiB
Markdown
# Source Internals
|
|
|
|
## Purpose
|
|
|
|
Source packages poll upstream weather providers and emit raw feed events. They
|
|
are adapters, not canonical mappers.
|
|
|
|
Sources should decode only the metadata needed for event identity, effective
|
|
time, and routing policy. Full provider payload interpretation belongs in
|
|
normalizers.
|
|
|
|
## Inputs And Outputs
|
|
|
|
Inputs are feedkit `config.SourceConfig` values and upstream HTTP responses.
|
|
Outputs are feed events whose payloads are raw provider JSON and whose schemas
|
|
come from `standards`.
|
|
|
|
Current drivers:
|
|
|
|
| Driver | Kind | Raw schema |
|
|
| --- | --- | --- |
|
|
| `nws_observation` | `observation` | `raw.nws.observation.v1` |
|
|
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` |
|
|
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` |
|
|
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` |
|
|
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` |
|
|
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` |
|
|
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` |
|
|
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` |
|
|
| `openweather_observation` | `observation` | `raw.openweather.current.v1` |
|
|
|
|
## Boundaries
|
|
|
|
- Source constructors validate source-specific params.
|
|
- Sources use feedkit HTTP helpers for HTTP polling.
|
|
- Sources emit raw events and should not build canonical `model` payloads.
|
|
- Provider helper packages under `internal/providers/<provider>` hold shared
|
|
parsing and validation helpers.
|
|
- Registration is centralized in `internal/sources/builtins.go`.
|
|
|
|
## Config Fields Used
|
|
|
|
All current source drivers use feedkit `HTTPSource`.
|
|
|
|
Required params:
|
|
|
|
- `url`
|
|
- `user_agent`
|
|
|
|
Optional params:
|
|
|
|
- `conditional`, default `true`;
|
|
- `http_timeout`;
|
|
- `http_response_body_limit_bytes`.
|
|
|
|
OpenWeather observation sources additionally require the configured URL to use
|
|
metric units. This is enforced by `internal/providers/openweather`.
|
|
|
|
Source-level `kinds`, when configured, are validated against the source's
|
|
advertised `Kinds()`.
|
|
|
|
## External Adapters Used
|
|
|
|
Sources use feedkit's HTTP helper for:
|
|
|
|
- request construction;
|
|
- `User-Agent` and `Accept` headers;
|
|
- optional conditional GET validators;
|
|
- response body size limits;
|
|
- JSON raw-message fetches.
|
|
|
|
NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times
|
|
with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs.
|
|
|
|
## State
|
|
|
|
HTTP conditional validators are held in each source instance. They are not
|
|
persisted across process restarts.
|
|
|
|
## Failure Behavior
|
|
|
|
Constructor failures are returned during startup and stop the daemon. Polling
|
|
failures are returned to the scheduler.
|
|
|
|
If a source cannot decode minimal metadata from an otherwise successful upstream
|
|
response, it still emits the raw event when possible. The event then falls back
|
|
to default ID/effective-time behavior from feedkit source helpers.
|
|
|
|
Unchanged conditional responses return no events and no error.
|
|
|
|
## Tests To Inspect
|
|
|
|
- `internal/sources/builtins_test.go`
|
|
- provider source tests under `internal/sources/nws`
|
|
- provider source tests under `internal/sources/openmeteo`
|
|
- provider source tests under `internal/sources/openweather`
|
|
- provider helper tests under `internal/providers`
|
|
|
|
## Invariants
|
|
|
|
- Emit raw provider schemas from `standards`.
|
|
- Keep provider-to-canonical mapping out of sources.
|
|
- Keep HTTP behavior context-aware.
|
|
- Keep driver registration explicit and stable.
|
|
- Keep source tests independent of live upstream services.
|