Document development workflow and internals

docs/internal/sources.md

@@ -0,0 +1,105 @@
+# Source Internals
+
+## Purpose
+
+Source packages poll upstream weather providers and emit raw feed events. They
+are adapters, not canonical mappers.
+
+Sources should decode only the metadata needed for event identity, effective
+time, and routing policy. Full provider payload interpretation belongs in
+normalizers.
+
+## Inputs And Outputs
+
+Inputs are feedkit `config.SourceConfig` values and upstream HTTP responses.
+Outputs are feed events whose payloads are raw provider JSON and whose schemas
+come from `standards`.
+
+Current drivers:
+
+| Driver | Kind | Raw schema |
+| --- | --- | --- |
+| `nws_observation` | `observation` | `raw.nws.observation.v1` |
+| `nws_alerts` | `alert` | `raw.nws.alerts.v1` |
+| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` |
+| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` |
+| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` |
+| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` |
+| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` |
+| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` |
+| `openweather_observation` | `observation` | `raw.openweather.current.v1` |
+
+## Boundaries
+
+- Source constructors validate source-specific params.
+- Sources use feedkit HTTP helpers for HTTP polling.
+- Sources emit raw events and should not build canonical `model` payloads.
+- Provider helper packages under `internal/providers/<provider>` hold shared
+  parsing and validation helpers.
+- Registration is centralized in `internal/sources/builtins.go`.
+
+## Config Fields Used
+
+All current source drivers use feedkit `HTTPSource`.
+
+Required params:
+
+- `url`
+- `user_agent`
+
+Optional params:
+
+- `conditional`, default `true`;
+- `http_timeout`;
+- `http_response_body_limit_bytes`.
+
+OpenWeather observation sources additionally require the configured URL to use
+metric units. This is enforced by `internal/providers/openweather`.
+
+Source-level `kinds`, when configured, are validated against the source's
+advertised `Kinds()`.
+
+## External Adapters Used
+
+Sources use feedkit's HTTP helper for:
+
+- request construction;
+- `User-Agent` and `Accept` headers;
+- optional conditional GET validators;
+- response body size limits;
+- JSON raw-message fetches.
+
+NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times
+with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs.
+
+## State
+
+HTTP conditional validators are held in each source instance. They are not
+persisted across process restarts.
+
+## Failure Behavior
+
+Constructor failures are returned during startup and stop the daemon. Polling
+failures are returned to the scheduler.
+
+If a source cannot decode minimal metadata from an otherwise successful upstream
+response, it still emits the raw event when possible. The event then falls back
+to default ID/effective-time behavior from feedkit source helpers.
+
+Unchanged conditional responses return no events and no error.
+
+## Tests To Inspect
+
+- `internal/sources/builtins_test.go`
+- provider source tests under `internal/sources/nws`
+- provider source tests under `internal/sources/openmeteo`
+- provider source tests under `internal/sources/openweather`
+- provider helper tests under `internal/providers`
+
+## Invariants
+
+- Emit raw provider schemas from `standards`.
+- Keep provider-to-canonical mapping out of sources.
+- Keep HTTP behavior context-aware.
+- Keep driver registration explicit and stable.
+- Keep source tests independent of live upstream services.