# Source Internals

## Purpose

Source packages poll upstream weather providers and emit raw feed events. They
are adapters, not canonical mappers.

Sources should decode only the metadata needed for event identity, effective
time, and routing policy. Full provider payload interpretation belongs in
normalizers.

## Inputs And Outputs

Inputs are feedkit `config.SourceConfig` values and upstream HTTP responses.
Outputs are feed events whose payloads are raw provider JSON and whose schemas
come from `standards`.

Current drivers:

| Driver | Kind | Raw schema |
| --- | --- | --- |
| `nws_observation` | `observation` | `raw.nws.observation.v1` |
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` |
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` |
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` |
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` |
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` |
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` |
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` |
| `openweather_observation` | `observation` | `raw.openweather.current.v1` |
| `spc_convective_outlook` | `outlook` | `raw.spc.convective_outlook.v1` |

## Boundaries

- Source constructors validate source-specific params.
- Sources use feedkit HTTP helpers for HTTP polling.
- Sources emit raw events and should not build canonical `model` payloads.
- Provider helper packages under `internal/providers/<provider>` hold shared
  parsing and validation helpers.
- Registration is centralized in `internal/sources/builtins.go`.

## Config Fields Used

Most source drivers use feedkit `HTTPSource`.

Required params:

- `url`
- `user_agent`

Optional params:

- `conditional`, default `true`;
- `http_timeout`;
- `http_response_body_limit_bytes`.

OpenWeather observation sources additionally require the configured URL to use
metric units. This is enforced by `internal/providers/openweather`.

The SPC convective outlook source is a multi-document poll source rather than a
single-URL `HTTPSource`. It requires `latitude`, `longitude`, and `user_agent`;
accepts optional `location_id`, `location_name`, `geojson_urls`,
`discussion_urls`, and `rss_url`; and supports `http_timeout` and
`http_response_body_limit_bytes`.

Source-level `kinds`, when configured, are validated against the source's
advertised `Kinds()`.

## External Adapters Used

Most sources use feedkit's HTTP helper for:

- request construction;
- `User-Agent` and `Accept` headers;
- optional conditional GET validators;
- response body size limits;
- JSON raw-message fetches.

NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times
with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs.
SPC helpers define Day 1-3 product metadata, parse GeoJSON timestamps, extract
cleaned print-page discussion text, and parse optional RSS metadata.

## State

HTTP conditional validators are held in each single-document HTTP source
instance. They are not persisted across process restarts. The SPC outlook source
keeps only a source-local hash of the most recent complete required product
bundle and emits no event when a later complete bundle is unchanged.

## Failure Behavior

Constructor failures are returned during startup and stop the daemon. Polling
failures are returned to the scheduler.

If a source cannot decode minimal metadata from an otherwise successful upstream
response, it still emits the raw event when possible. The event then falls back
to default ID/effective-time behavior from feedkit source helpers.

Unchanged conditional responses return no events and no error.

The SPC outlook source fetches all configured Day 1-3 GeoJSON products and print
pages atomically. If any required GeoJSON or print-page request fails, the poll
returns an error and emits no partial bundle. RSS is fetched only when `rss_url`
is configured.

## Tests To Inspect

- `internal/sources/builtins_test.go`
- provider source tests under `internal/sources/nws`
- provider source tests under `internal/sources/openmeteo`
- provider source tests under `internal/sources/openweather`
- provider source tests under `internal/sources/spc`
- provider helper tests under `internal/providers`

## Invariants

- Emit raw provider schemas from `standards`.
- Keep provider-to-canonical mapping out of sources.
- Keep HTTP behavior context-aware.
- Keep driver registration explicit and stable.
- Keep source tests independent of live upstream services.