11 Commits

Author SHA1 Message Date
002f9d0ba6 Clean up documentation consistency
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-06-10 20:28:41 +00:00
ec115ba152 Document provider integration contracts 2026-06-10 20:25:13 +00:00
47176520bb Document development workflow and internals 2026-06-10 20:22:38 +00:00
9e27a431a1 Add maintained configuration examples 2026-06-10 20:17:40 +00:00
f0605781a2 Document operations and troubleshooting 2026-06-10 20:15:08 +00:00
abb8f218ec Document event and Postgres integration contracts 2026-06-10 20:12:27 +00:00
4ac4e401ed Document weatherfeeder CLI and configuration 2026-06-10 20:07:56 +00:00
77bd59cb87 Add policy documents and a roadmap to implement a full documentation set
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-06-10 14:59:20 -05:00
bb5abf798b Add an initial roadmap for SPC convective outlook support 2026-06-10 14:49:28 -05:00
fd820fd964 Implemented NWS weather stories support
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-05-30 06:47:18 -05:00
cca873cafb Made forecast-period conditionCode optional
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
2026-05-28 07:47:07 -05:00
58 changed files with 4719 additions and 417 deletions

341
API.md
View File

@@ -1,339 +1,4 @@
# weatherfeeder API (Wire Contract)
# Event Wire Contract
This document defines the stable, consumer-facing JSON contract emitted by weatherfeeder sinks.
weatherfeeder emits **events** encoded as JSON. Each event has:
- an **envelope** (metadata + schema identifier), and
- a **payload** whose shape is determined by `schema`.
Downstream consumers should:
1. parse the event envelope,
2. switch on `schema`, then
3. decode `payload` into the matching schema.
---
## Event envelope
All events are JSON objects with these fields:
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `id` | string | yes | Stable event identifier. Treat as opaque. |
| `schema` | string | yes | Schema identifier (e.g. `weather.observation.v1`). |
| `source` | string | yes | Provider/source identifier (stable within configuration). |
| `effectiveAt` | string (timestamp) | yes | RFC3339Nano timestamp indicating when this event is effective. |
| `payload` | object | yes | Schema-specific payload (see below). |
### Timestamp format
All timestamps are encoded as JSON strings using Gos `time.Time` JSON encoding (RFC3339Nano).
Examples:
- `"2026-01-17T14:27:00Z"`
- `"2026-01-17T08:27:00-06:00"`
---
## Canonical schemas
weatherfeeder emits four canonical domain schemas:
- `weather.observation.v1`
- `weather.forecast.v1`
- `weather.forecast_discussion.v1`
- `weather.alert.v1`
Each payload is described below using the JSON field names as the contract.
### Raw upstream schemas
weatherfeeder sources also emit provider-specific raw schemas before normalization.
For this feature, the raw source schema is:
- `raw.nws.forecast_discussion.v1`
- payload type: string
- payload contents: exact fetched HTML response body
---
## Shared Conventions
- Timestamps are JSON strings in RFC3339Nano format.
- Optional fields are omitted when unknown (`omitempty` behavior).
- Numeric measurements are normalized to metric units:
- `*C` = Celsius
- `*Kmh` = kilometers/hour
- `*Pa` = Pascals
- `*Meters` = meters
- `*Mm` = millimeters
- `*Percent` = percent (0-100)
- `conditionCode` is a WMO weather interpretation code (`int`).
- Unknown/unmappable is `-1`.
- Downstream consumers should treat unknown codes as “unknown conditions” rather than failing decoding.
- For readability and stability, weatherfeeder rounds floating-point values in canonical payloads to
**4 digits after the decimal** during normalization.
---
## Schema: `weather.observation.v1`
Payload type: `WeatherObservation`
A `WeatherObservation` represents a point-in-time observation for a station/location.
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `stationId` | string | no | Provider station/location identifier |
| `stationName` | string | no | Human station name |
| `timestamp` | timestamp string | yes | Observation timestamp |
| `conditionCode` | int | yes | WMO code (`-1` unknown) |
| `isDay` | bool | no | Day/night hint |
| `textDescription` | string | no | Human-facing short description |
| `temperatureC` | number | no | Celsius |
| `dewpointC` | number | no | Celsius |
| `windDirectionDegrees` | number | no | Degrees |
| `windSpeedKmh` | number | no | km/h |
| `windGustKmh` | number | no | km/h |
| `barometricPressurePa` | number | no | Pascals |
| `visibilityMeters` | number | no | Meters |
| `relativeHumidityPercent` | number | no | Percent |
| `apparentTemperatureC` | number | no | Celsius |
| `presentWeather` | array | no | Provider-specific structured weather fragments |
### Nested: `presentWeather[]`
Each `presentWeather[]` element:
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `raw` | object | no | Provider-specific JSON object |
---
## Schema: `weather.forecast.v1`
Payload type: `WeatherForecastRun`
A `WeatherForecastRun` is a single issued forecast snapshot for a location and a specific product
(hourly / narrative / daily). The run contains an ordered list of forecast periods.
### `product` values
`product` is one of:
- `"hourly"`
- `"narrative"`
- `"daily"`
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `locationId` | string | no | Provider location identifier |
| `locationName` | string | no | Human name, if available |
| `issuedAt` | string (timestamp) | yes | When this run was generated/issued |
| `updatedAt` | string (timestamp) | no | Optional later update time |
| `product` | string | yes | One of `hourly`, `narrative`, `daily` |
| `latitude` | number | no | Degrees |
| `longitude` | number | no | Degrees |
| `elevationMeters` | number | no | meters |
| `periods` | array | yes | Chronological forecast periods |
### Nested: `periods[]` (`WeatherForecastPeriod`)
A `WeatherForecastPeriod` is valid for `[startTime, endTime)`.
| Field | Type | Required | Units / Notes |
|---|---:|:---:|---|
| `startTime` | string (timestamp) | yes | Period start |
| `endTime` | string (timestamp) | yes | Period end |
| `name` | string | no | Human label (often empty for hourly) |
| `isDay` | bool | no | Day/night hint |
| `conditionCode` | int | yes | WMO code (`-1` for unknown) |
| `textDescription` | string | no | Human-facing short phrase |
| `temperatureC` | number | no | °C |
| `temperatureCMin` | number | no | °C (aggregated products) |
| `temperatureCMax` | number | no | °C (aggregated products) |
| `dewpointC` | number | no | °C |
| `relativeHumidityPercent` | number | no | percent |
| `windDirectionDegrees` | number | no | degrees |
| `windSpeedKmh` | number | no | km/h |
| `windGustKmh` | number | no | km/h |
| `barometricPressurePa` | number | no | Pa |
| `visibilityMeters` | number | no | meters |
| `apparentTemperatureC` | number | no | °C |
| `cloudCoverPercent` | number | no | percent |
| `probabilityOfPrecipitationPercent` | number | no | percent |
| `precipitationAmountMm` | number | no | mm (liquid equivalent) |
| `snowfallDepthMm` | number | no | mm |
| `uvIndex` | number | no | unitless index |
---
## Schema: `weather.alert.v1`
Payload type: `WeatherAlertRun`
A `WeatherAlertRun` is a snapshot of *active* alerts for a location as-of a point in time.
A run may contain zero, one, or many alerts.
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `locationId` | string | no | Provider location identifier |
| `locationName` | string | no | Human name, if available |
| `asOf` | string (timestamp) | yes | When the provider asserted this snapshot is current |
| `latitude` | number | no | Degrees |
| `longitude` | number | no | Degrees |
| `alerts` | array | yes | Active alerts (order provider-dependent) |
### Nested: `alerts[]` (`WeatherAlert`)
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `id` | string | yes | Provider-stable identifier (often a URL/URI) |
| `event` | string | no | Classification/event label |
| `headline` | string | no | Alert headline |
| `severity` | string | no | Example: Extreme/Severe/Moderate/Minor/Unknown |
| `urgency` | string | no | Example: Immediate/Expected/Future/Past/Unknown |
| `certainty` | string | no | Example: Observed/Likely/Possible/Unlikely/Unknown |
| `status` | string | no | Example: Actual/Exercise/Test/System/Unknown |
| `messageType` | string | no | Example: Alert/Update/Cancel |
| `category` | string | no | Example: Met/Geo/Safety/Rescue/Fire/Health/Env/Transport/Infra/CBRNE/Other |
| `response` | string | no | Example: Shelter/Evacuate/Prepare/Execute/Avoid/Monitor/Assess/AllClear/None |
| `response` | string | no | e.g. Shelter/Evacuate/Prepare/... |
| `description` | string | no | Narrative |
| `instruction` | string | no | What to do |
| `sent` | string (timestamp) | no | Provider-dependent |
| `effective` | string (timestamp) | no | Provider-dependent |
| `onset` | string (timestamp) | no | Provider-dependent |
| `expires` | string (timestamp) | no | Provider-dependent |
| `areaDescription` | string | no | Often a provider string |
| `senderName` | string | no | Provenance |
| `references` | array | no | Related alert references |
### Nested: `references[]` (`AlertReference`)
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `id` | string | no | Provider reference ID/URI |
| `identifier` | string | no | Provider identifier string, if distinct |
| `sender` | string | no | Sender |
| `sent` | string (timestamp) | no | Timestamp |
---
## Schema: `weather.forecast_discussion.v1`
Payload type: `WeatherForecastDiscussion`
A `WeatherForecastDiscussion` is an issued narrative bulletin for an NWS office.
It is distinct from `weather.forecast.v1`, which is period-based.
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `officeId` | string | no | NWS office identifier, e.g. `LSX` |
| `officeName` | string | no | Human office name |
| `product` | string | yes | Currently `afd` |
| `issuedAt` | string (timestamp) | yes | Bulletin issue time |
| `updatedAt` | string (timestamp) | no | Optional page/update timestamp |
| `keyMessages` | array | no | Ordered key-message bullet list |
| `shortTerm` | object | no | Short-term discussion section |
| `longTerm` | object | no | Long-term discussion section |
### Nested: `shortTerm` / `longTerm`
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `qualifier` | string | no | Header qualifier such as `(Through Late Sunday Night)` |
| `issuedAt` | string (timestamp) | no | Optional section-local issue time |
| `text` | string | no | Paragraph-preserved prose text |
---
## Compatibility rules
- Consumers **must** ignore unknown fields.
- Producers (weatherfeeder) prefer **additive changes** within a schema version.
- Renames/removals/semantic breaks normally require a **schema version bump** (`weather.*.v2`); pre-1.0 projects may choose in-place changes.
---
## Examples
### Observation event (`weather.observation.v1`)
```json
{
"id": "nws:KSTL:2026-01-17T14:00:00Z",
"schema": "weather.observation.v1",
"source": "nws_observation",
"effectiveAt": "2026-01-17T14:00:00Z",
"payload": {
"stationId": "KSTL",
"timestamp": "2026-01-17T14:00:00Z",
"conditionCode": 1,
"textDescription": "Mainly Sunny",
"temperatureC": 3.25,
"windSpeedKmh": 18.5
}
}
```
### Forecast event (`weather.forecast.v1`)
```json
{
"id": "openmeteo:38.63,-90.20:2026-01-17T13:00:00Z",
"schema": "weather.forecast.v1",
"source": "openmeteo_forecast",
"effectiveAt": "2026-01-17T13:00:00Z",
"payload": {
"locationName": "St. Louis, MO",
"issuedAt": "2026-01-17T13:00:00Z",
"product": "hourly",
"latitude": 38.63,
"longitude": -90.2,
"periods": [
{
"startTime": "2026-01-17T14:00:00Z",
"endTime": "2026-01-17T15:00:00Z",
"conditionCode": 2,
"textDescription": "Partly Cloudy",
"temperatureC": 3.5,
"probabilityOfPrecipitationPercent": 10
}
]
}
}
```
### Alert event (`weather.alert.v1`)
```json
{
"id": "nws:alerts:2026-01-17T14:10:00Z",
"schema": "weather.alert.v1",
"source": "nws_alerts",
"effectiveAt": "2026-01-17T14:10:00Z",
"payload": {
"asOf": "2026-01-17T14:05:00Z",
"alerts": [
{
"id": "https://api.weather.gov/alerts/abc123",
"event": "Winter Weather Advisory",
"headline": "Winter Weather Advisory issued January 17 at 8:05AM CST",
"severity": "Moderate",
"description": "Mixed precipitation expected...",
"expires": "2026-01-18T06:00:00Z"
}
]
}
}
```
The canonical event wire contract has moved to
[docs/integrations/events.md](docs/integrations/events.md).

View File

@@ -1,35 +1,39 @@
# weatherfeeder
weatherfeeder is a small daemon that polls weather observations, forecasts, and alerts from multiple upstream
providers, normalizes them into a provider-independent format, and emits them to a sink.
`weatherfeeder` is a config-driven daemon that polls weather providers, normalizes
provider-specific responses into canonical weather events, and routes those
events to configured sinks.
Today, the only implemented sink is `stdout`, which prints JSON-encoded events.
It currently supports NWS observations, alerts, hourly forecasts, narrative
forecasts, forecast discussions, and weather stories; Open-Meteo observations
and hourly forecasts; and OpenWeather observations. Implemented sinks are
stdout, NATS, and Postgres.
## What weatherfeeder emits
## Quickstart
weatherfeeder emits **feed events** encoded as JSON. Each event includes a schema identifier and a payload.
Downstream consumers should key off the `schema` value and decode the `payload` accordingly.
Run the checked-in sample config:
Canonical domain schemas emitted after normalization:
```sh
cd cmd/weatherfeeder
go run .
```
- `weather.observation.v1``WeatherObservation`
- `weather.forecast.v1``WeatherForecastRun`
- `weather.forecast_discussion.v1``WeatherForecastDiscussion`
- `weather.alert.v1``WeatherAlertRun`
The sample config at `cmd/weatherfeeder/config.yml` is load-tested and can be
used as a starting point. The executable always reads `config.yml` from its
current working directory.
For the complete wire contract (event envelope + payload schemas, fields, units, and compatibility rules), see:
## Documentation
- **API.md**
## Upstream providers (current MVP)
- NWS: observations, hourly forecasts, narrative forecasts, forecast discussions, alerts
- Open-Meteo: observations, hourly forecasts
- OpenWeather: observations
## Versioning & compatibility
The JSON field names on canonical payload types are treated as part of the wire contract.
Additive changes are preferred. Renames/removals require a schema version bump.
See **API.md** for details.
- [CLI reference](docs/cli.md)
- [Configuration reference](docs/config.md)
- [Operations guide](docs/operations.md)
- [Troubleshooting guide](docs/troubleshooting.md)
- [Example configs](examples/)
- [Event wire contract](docs/integrations/events.md)
- [Postgres table contract](docs/integrations/postgres.md)
- [NWS integration notes](docs/integrations/nws.md)
- [Open-Meteo integration notes](docs/integrations/openmeteo.md)
- [OpenWeather integration notes](docs/integrations/openweather.md)
- [Architecture policy](docs/policy/architecture.md)
- [Development policy](docs/policy/development.md)
- [Documentation policy](docs/policy/documentation.md)

View File

@@ -24,7 +24,7 @@ sources:
# driver: openweather_observation
# every: 10m
# params:
# url: "https://api.openweathermap.org/data/2.5/weather?lat=38.6239&lon=-90.3571&appid=c954f2566cb7ccb56b43737b52e88fc6&units=metric"
# url: "https://api.openweathermap.org/data/2.5/weather?lat=38.6239&lon=-90.3571&units=metric"
# user_agent: "HomeOps (eric@maximumdirect.net)"
# - name: NWSObservationKSUS
@@ -72,6 +72,15 @@ sources:
url: "https://forecast.weather.gov/product.php?site=LSX&issuedby=LSX&product=AFD&format=TXT&version=1&glossary=0"
user_agent: "HomeOps (eric@maximumdirect.net)"
- name: NWSWeatherStoriesSTL
mode: poll
kinds: ["weather_story"]
driver: nws_weatherstories
every: 30m
params:
url: "https://api.weather.gov/offices/LSX/weatherstories"
user_agent: "HomeOps (eric@maximumdirect.net)"
- name: OpenMeteoHourlyForecastSTL
mode: poll
kinds: ["forecast"]
@@ -106,7 +115,7 @@ sinks:
# params:
# uri: postgres://weatherdb:5432/weatherdb?sslmode=disable
# username: weatherdb
# password: weatherdb
# password: <database_password>
# prune: 3d
# # Prunes rows older than now-3d on each write transaction.
@@ -117,13 +126,13 @@ sinks:
routes:
- sink: stdout
kinds: ["observation", "forecast", "forecast_discussion", "alert"]
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"]
- sink: nats_weatherfeeder
kinds: ["observation", "forecast", "forecast_discussion", "alert"]
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"]
# - sink: pg_weatherfeeder
# kinds: ["observation", "forecast", "forecast_discussion", "alert"]
# kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"]
# - sink: logfile
# kinds: ["observation", "alert", "forecast", "forecast_discussion"]
# kinds: ["observation", "alert", "forecast", "forecast_discussion", "weather_story"]

View File

@@ -2,7 +2,9 @@ package main
import (
"context"
"path/filepath"
"reflect"
"sort"
"strings"
"testing"
"time"
@@ -82,6 +84,33 @@ func TestExampleConfigSourcesBuildSchedulerJobs(t *testing.T) {
t.Fatalf("config.Load(config.yml) unexpected error: %v", err)
}
assertConfigSourcesBuildSchedulerJobs(t, cfg)
}
func TestMaintainedConfigExamplesLoad(t *testing.T) {
paths, err := filepath.Glob("../../examples/*.yml")
if err != nil {
t.Fatalf("filepath.Glob examples: %v", err)
}
sort.Strings(paths)
if len(paths) == 0 {
t.Fatalf("expected maintained config examples")
}
for _, path := range paths {
t.Run(filepath.Base(path), func(t *testing.T) {
cfg, err := config.Load(path)
if err != nil {
t.Fatalf("config.Load(%s) unexpected error: %v", path, err)
}
assertConfigSourcesBuildSchedulerJobs(t, cfg)
})
}
}
func assertConfigSourcesBuildSchedulerJobs(t *testing.T, cfg *config.Config) {
t.Helper()
reg := fksources.NewRegistry()
wfsources.RegisterBuiltins(reg)
@@ -91,6 +120,10 @@ func TestExampleConfigSourcesBuildSchedulerJobs(t *testing.T) {
t.Fatalf("BuildInput(sources[%d]) error = %v", i, err)
}
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
t.Fatalf("ValidateExpectedKinds(sources[%d]) error = %v", i, err)
}
job, err := fkscheduler.JobFromSourceConfig(in, sc)
if err != nil {
t.Fatalf("JobFromSourceConfig(sources[%d]) error = %v", i, err)

72
docs/cli.md Normal file
View File

@@ -0,0 +1,72 @@
# CLI Reference
## Shortest Useful Command
Run `weatherfeeder` from a directory containing `config.yml`:
```sh
cd cmd/weatherfeeder
go run .
```
When using a built binary:
```sh
./weatherfeeder
```
## Command Overview
`weatherfeeder` starts a long-running polling daemon. On startup it:
1. reads `config.yml` from the current working directory;
2. builds configured sources, sinks, and routes;
3. starts polling sources on their configured intervals;
4. normalizes and deduplicates events;
5. dispatches matching events to configured sinks.
The command logs startup, runtime, and shutdown messages to stderr using the Go
standard logger.
## Flags
There are currently no CLI flags, subcommands, or environment-variable based
configuration controls.
The config path is fixed at `config.yml` relative to the process current working
directory. To run with a different config, change the working directory or place
the desired file at that path.
## Common Workflows
Run the checked-in sample config:
```sh
cd cmd/weatherfeeder
go run .
```
Maintained copyable configs are available under [`examples/`](../examples/).
Build and run a local binary:
```sh
go build -o weatherfeeder ./cmd/weatherfeeder
cp cmd/weatherfeeder/config.yml .
./weatherfeeder
```
Run in the project container image with a mounted config:
```sh
docker run --rm -v "$PWD/config.yml:/weatherfeeder/config.yml:ro" weatherfeeder
```
The Docker image sets `/weatherfeeder` as the working directory, so the mounted
file must appear at `/weatherfeeder/config.yml`.
## Shutdown
Stop the daemon with `Ctrl-C` or `SIGTERM`. The process uses context-aware
shutdown for scheduler, dispatcher, processors, sources, and sinks, then logs
`shutdown complete`.

222
docs/config.md Normal file
View File

@@ -0,0 +1,222 @@
# Configuration Reference
## Config File
`weatherfeeder` reads exactly one YAML file named `config.yml` from the current
working directory. There is no config path flag and no search path.
YAML decoding is strict for config struct fields: misspelled fields such as
`sources[].drviver` fail startup. Driver-specific `params` maps are validated by
the source or sink constructor that consumes them.
The top-level file contains:
```yaml
sources:
- name: NWSObservationKSTL
mode: poll
driver: nws_observation
every: 10m
kinds: ["observation"]
params:
url: "https://api.weather.gov/stations/KSTL/observations/latest"
user_agent: "Example weatherfeeder operator (ops@example.com)"
sinks:
- name: stdout
driver: stdout
params: {}
routes:
- sink: stdout
kinds: ["observation"]
```
`sources` and `sinks` must each contain at least one entry. `routes` is optional.
When `routes` is omitted, every configured sink receives every event kind.
Maintained copyable configs are available under [`examples/`](../examples/).
## Production-Oriented Shape
A typical deployment uses multiple polling sources and sends the same canonical
event stream to a broker or database:
```yaml
sources:
- name: NWSAlertsLocal
mode: poll
driver: nws_alerts
every: 1m
kinds: ["alert"]
params:
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
user_agent: "Example weatherfeeder operator (ops@example.com)"
sinks:
- name: nats_weather
driver: nats
params:
url: nats://nats:4222
subject: weatherfeeder
- name: pg_weather
driver: postgres
params:
uri: postgres://weatherdb:5432/weatherdb?sslmode=disable
username: weatherdb
password: <database_password>
prune: 3d
routes:
- sink: nats_weather
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"]
- sink: pg_weather
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"]
```
Do not commit real API keys, database passwords, or personal contact addresses in
copyable configs.
## Top-Level Fields
| Field | Required | Description |
|---|:---:|---|
| `sources` | yes | List of configured input sources. |
| `sinks` | yes | List of configured output sinks. |
| `routes` | no | List of sink routing rules. If omitted, all sinks receive all kinds. |
## Source Fields
| Field | Required | Description |
|---|:---:|---|
| `name` | yes | Unique source name. Used as the event source identifier. |
| `driver` | yes | Source driver name. |
| `mode` | no | `poll`, `stream`, or omitted for auto. Current weatherfeeder drivers are polling drivers. |
| `every` | yes | Poll interval for current weatherfeeder source drivers. |
| `kinds` | no | Expected event kinds. If present, startup verifies they match the source driver. |
| `params` | driver-specific | Driver parameters. Current source drivers require HTTP params. |
Current event kinds are `observation`, `forecast`, `forecast_discussion`,
`weather_story`, and `alert`.
## Source Drivers
| Driver | Kind | Upstream product |
|---|---|---|
| `nws_observation` | `observation` | NWS station latest observation. |
| `nws_alerts` | `alert` | NWS alerts collection. |
| `nws_forecast_hourly` | `forecast` | NWS hourly gridpoint forecast. |
| `nws_forecast_narrative` | `forecast` | NWS narrative gridpoint forecast. |
| `nws_forecast_discussion` | `forecast_discussion` | NWS forecast discussion HTML product. |
| `nws_weatherstories` | `weather_story` | NWS office weather stories. |
| `openmeteo_observation` | `observation` | Open-Meteo current conditions. |
| `openmeteo_forecast` | `forecast` | Open-Meteo hourly forecast. |
| `openweather_observation` | `observation` | OpenWeather current weather. |
## HTTP Source Params
All current source drivers use the shared HTTP polling helper.
| Param | Required | Description |
|---|:---:|---|
| `url` | yes | Full upstream request URL. `URL` is also accepted by the helper. |
| `user_agent` | yes | User-Agent sent to the upstream provider. `userAgent` is also accepted by the helper. |
| `conditional` | no | Boolean. Defaults to `true`; enables ETag and Last-Modified conditional requests. |
| `http_timeout` | no | Positive duration for the HTTP client timeout. |
| `http_response_body_limit_bytes` | no | Positive integer response body limit in bytes. |
When `conditional` is enabled and the upstream returns `304 Not Modified`, the
source emits no events for that poll.
OpenWeather observation URLs must include `units=metric`. Startup fails if the
URL omits it or sets another unit system.
## Sink Fields
| Field | Required | Description |
|---|:---:|---|
| `name` | yes | Unique sink name. Routes refer to this value. |
| `driver` | yes | Sink driver name. |
| `params` | driver-specific | Sink parameters. |
## Sink Drivers
### `stdout`
Prints each event as JSON to stdout.
```yaml
sinks:
- name: stdout
driver: stdout
params: {}
```
### `nats`
Publishes each event as JSON to a NATS subject.
| Param | Required | Description |
|---|:---:|---|
| `url` | yes | NATS server URL, such as `nats://localhost:4222`. |
| `subject` | yes | Subject to publish events to. |
### `postgres`
Writes supported canonical weather events to Postgres using weatherfeeder's
registered schema mapping. The table contract is documented in
[Postgres integration](integrations/postgres.md).
| Param | Required | Description |
|---|:---:|---|
| `uri` | yes | PostgreSQL connection URI. |
| `username` | yes | Database username. |
| `password` | yes | Database password. |
| `prune` | no | Retention window. If set, rows older than the window are pruned on each write transaction. |
`prune` accepts Go duration strings such as `72h`, plus day and week suffixes
such as `3d` and `2w`.
## Routes
Routes connect event kinds to sinks:
```yaml
routes:
- sink: stdout
kinds: ["observation", "alert"]
```
| Field | Required | Description |
|---|:---:|---|
| `sink` | yes | Name of a configured sink. |
| `kinds` | no | Event kinds to send to that sink. Omit or use an empty list to match all kinds. |
Route `kinds` values are trimmed and lowercased by the dispatcher. Blank entries
are rejected.
## Duration Formats
Top-level source `every` accepts:
- Go duration strings such as `30s`, `10m`, or `1h`;
- integer values, interpreted as minutes;
- numeric strings such as `"15"`, also interpreted as minutes.
HTTP param durations such as `http_timeout` accept Go duration strings. Numeric
values and numeric strings are interpreted as seconds.
Postgres `prune` must be a string duration.
## Secrets
The config file is read directly from disk and has no built-in secret expansion.
Keep real credentials out of repository-tracked configs. Use deployment tooling
to render `config.yml` with the needed secret values before starting the daemon.
## Maintained Examples
- [Minimal stdout config](../examples/config.minimal.yml)
- [NATS publishing config](../examples/config.nats.yml)
- [Postgres persistence config](../examples/config.postgres.yml)

238
docs/integrations/events.md Normal file
View File

@@ -0,0 +1,238 @@
# Event Wire Contract
This document is the canonical JSON contract for events emitted by
`weatherfeeder` JSON sinks, including stdout and NATS. Postgres stores the same
event envelope fields in parent table columns; see
[Postgres integration](postgres.md).
Downstream consumers should read the envelope, switch on `schema`, and decode
`payload` according to that schema.
## Envelope
Every emitted event is a JSON object with these fields:
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `id` | string | yes | Stable event identifier. Treat as opaque. |
| `kind` | string | yes | Routing kind, such as `observation` or `alert`. |
| `source` | string | yes | Configured source name. |
| `emitted_at` | timestamp | yes | When the daemon emitted the event. |
| `effective_at` | timestamp | no | Timestamp the payload is about, when known. |
| `schema` | string | no | Schema identifier. Weatherfeeder sources and normalizers set this. |
| `payload` | object, array, string, or scalar | yes | Schema-specific payload. |
Timestamps are JSON strings using Go `time.Time` JSON encoding, which is
RFC3339Nano-compatible. Weatherfeeder normalizers use UTC timestamps for
canonical payloads.
## Kinds And Schemas
Canonical schemas emitted after normalization:
| Kind | Schema | Payload |
|---|---|---|
| `observation` | `weather.observation.v1` | `WeatherObservation` |
| `forecast` | `weather.forecast.v1` | `WeatherForecastRun` |
| `forecast_discussion` | `weather.forecast_discussion.v1` | `WeatherForecastDiscussion` |
| `weather_story` | `weather.weather_story.v1` | `WeatherStoryRun` |
| `alert` | `weather.alert.v1` | `WeatherAlertRun` |
Raw upstream schemas emitted by current sources:
| Kind | Schema | Payload |
|---|---|---|
| `observation` | `raw.nws.observation.v1` | NWS observation JSON |
| `observation` | `raw.openmeteo.current.v1` | Open-Meteo current JSON |
| `observation` | `raw.openweather.current.v1` | OpenWeather current JSON |
| `forecast` | `raw.nws.hourly.forecast.v1` | NWS hourly forecast JSON |
| `forecast` | `raw.nws.narrative.forecast.v1` | NWS narrative forecast JSON |
| `forecast_discussion` | `raw.nws.forecast_discussion.v1` | NWS forecast discussion HTML string |
| `weather_story` | `raw.nws.weatherstories.v1` | NWS weather stories JSON |
| `forecast` | `raw.openmeteo.hourly.forecast.v1` | Open-Meteo hourly forecast JSON |
| `alert` | `raw.nws.alerts.v1` | NWS alerts JSON |
`standards.SchemaRawOpenWeatherHourlyForecastV1` exists in code, but no current
registered source emits it.
## Shared Conventions
- Canonical numeric measurements use metric units.
- Floating-point values in canonical payloads are rounded to 4 digits after the
decimal point during normalization.
- Optional fields use JSON `omitempty`; absent fields should be treated as
unknown.
- `conditionCode` is a WMO weather interpretation code. Unknown observation
conditions use `-1`. Forecast period `conditionCode` is optional.
- Additive fields are compatible within a schema version. Removing, renaming, or
changing the meaning of a field requires a new schema identifier.
## `weather.observation.v1`
Payload type: `WeatherObservation`.
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `stationId` | string | no | Provider station/location identifier. |
| `stationName` | string | no | Human station name. |
| `timestamp` | timestamp | yes | Observation timestamp. |
| `conditionCode` | integer | yes | WMO code; `-1` means unknown. |
| `isDay` | boolean | no | Day/night hint. |
| `textDescription` | string | no | Short human description. |
| `temperatureC` | number | no | Celsius. |
| `dewpointC` | number | no | Celsius. |
| `windDirectionDegrees` | number | no | Degrees. |
| `windSpeedKmh` | number | no | Kilometers per hour. |
| `windGustKmh` | number | no | Kilometers per hour. |
| `barometricPressurePa` | number | no | Pascals. |
| `visibilityMeters` | number | no | Meters. |
| `relativeHumidityPercent` | number | no | Percent from 0 to 100. |
| `apparentTemperatureC` | number | no | Celsius. |
| `presentWeather` | array | no | Provider-specific present weather fragments. |
`presentWeather[]` entries contain optional `raw` objects.
## `weather.forecast.v1`
Payload type: `WeatherForecastRun`.
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `locationId` | string | no | Provider location identifier. |
| `locationName` | string | no | Human location name. |
| `issuedAt` | timestamp | yes | When the forecast run was generated or issued. |
| `updatedAt` | timestamp | no | Subsequent provider update time. |
| `product` | string | yes | Current emitted values are `hourly` and `narrative`. |
| `latitude` | number | no | Degrees. |
| `longitude` | number | no | Degrees. |
| `elevationMeters` | number | no | Meters. |
| `periods` | array | yes | Ordered forecast periods. |
`periods[]` entries:
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `startTime` | timestamp | yes | Period start. |
| `endTime` | timestamp | yes | Period end. |
| `name` | string | no | Human label. |
| `isDay` | boolean | no | Day/night hint. |
| `conditionCode` | integer | no | WMO code when applicable. |
| `textDescription` | string | no | Human summary. |
| `temperatureC` | number | no | Celsius. |
| `temperatureCMin` | number | no | Celsius. |
| `temperatureCMax` | number | no | Celsius. |
| `dewpointC` | number | no | Celsius. |
| `relativeHumidityPercent` | number | no | Percent from 0 to 100. |
| `windDirectionDegrees` | number | no | Degrees. |
| `windSpeedKmh` | number | no | Kilometers per hour. |
| `windGustKmh` | number | no | Kilometers per hour. |
| `barometricPressurePa` | number | no | Pascals. |
| `visibilityMeters` | number | no | Meters. |
| `apparentTemperatureC` | number | no | Celsius. |
| `cloudCoverPercent` | number | no | Percent from 0 to 100. |
| `probabilityOfPrecipitationPercent` | number | no | Percent from 0 to 100. |
| `precipitationAmountMm` | number | no | Liquid-equivalent millimeters. |
| `snowfallDepthMm` | number | no | Millimeters. |
| `uvIndex` | number | no | Unitless index. |
## `weather.forecast_discussion.v1`
Payload type: `WeatherForecastDiscussion`.
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `officeId` | string | no | NWS office identifier. |
| `officeName` | string | no | Office name. |
| `product` | string | yes | Current value is `afd`. |
| `issuedAt` | timestamp | yes | Bulletin issue time. |
| `updatedAt` | timestamp | no | Subsequent update time. |
| `keyMessages` | array of strings | no | Extracted key messages. |
| `shortTerm` | object | no | Short-term section. |
| `longTerm` | object | no | Long-term section. |
`shortTerm` and `longTerm` sections contain optional `qualifier`, `issuedAt`,
and `text` fields.
## `weather.weather_story.v1`
Payload type: `WeatherStoryRun`.
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `officeId` | string | no | NWS office identifier. |
| `asOf` | timestamp | yes | Snapshot time. |
| `stories` | array | yes | Ordered story cards. |
`stories[]` entries:
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `officeId` | string | no | Office identifier. |
| `startTime` | timestamp | yes | Story start. |
| `endTime` | timestamp | yes | Story end. |
| `updatedAt` | timestamp | yes | Story update time. |
| `title` | string | no | Story title. |
| `description` | string | no | Story description. |
| `altText` | string | no | Image alternate text. |
| `priority` | boolean | yes | Provider priority flag. |
| `order` | integer | yes | Provider display order. |
| `downloadUrl` | string | no | Story image URL. |
## `weather.alert.v1`
Payload type: `WeatherAlertRun`.
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `locationId` | string | no | Provider location identifier. |
| `locationName` | string | no | Human location name. |
| `asOf` | timestamp | yes | Snapshot time. |
| `latitude` | number | no | Degrees. |
| `longitude` | number | no | Degrees. |
| `alerts` | array | yes | Active alerts. |
`alerts[]` entries:
| Field | Type | Required | Notes |
|---|---|:---:|---|
| `id` | string | yes | Provider-stable alert identifier. |
| `event` | string | no | Alert event label. |
| `headline` | string | no | Alert headline. |
| `severity` | string | no | Provider severity. |
| `urgency` | string | no | Provider urgency. |
| `certainty` | string | no | Provider certainty. |
| `status` | string | no | Alert status. |
| `messageType` | string | no | Alert message type. |
| `category` | string | no | Alert category. |
| `response` | string | no | Recommended response. |
| `description` | string | no | Alert description. |
| `instruction` | string | no | Alert instruction. |
| `sent` | timestamp | no | Provider sent time. |
| `effective` | timestamp | no | Effective time. |
| `onset` | timestamp | no | Onset time. |
| `expires` | timestamp | no | Expiration time. |
| `areaDescription` | string | no | Affected area description. |
| `senderName` | string | no | Provider sender name. |
| `references` | array | no | Related alerts. |
`references[]` entries contain optional `id`, `identifier`, `sender`, and
`sent` fields.
## Compact Example
```json
{
"id": "NWSObservationKSTL:2026-06-10T12:00:00Z",
"kind": "observation",
"source": "NWSObservationKSTL",
"emitted_at": "2026-06-10T12:00:05Z",
"effective_at": "2026-06-10T12:00:00Z",
"schema": "weather.observation.v1",
"payload": {
"stationId": "KSTL",
"timestamp": "2026-06-10T12:00:00Z",
"conditionCode": 0,
"temperatureC": 22.5
}
}
```

119
docs/integrations/nws.md Normal file
View File

@@ -0,0 +1,119 @@
# NWS Integration Notes
## Purpose
This document describes the NWS products that `weatherfeeder` currently polls
and normalizes. It is for developers and operators maintaining NWS source URLs,
normalizers, fixtures, and tests.
General config syntax belongs in [configuration](../config.md). Emitted JSON
events are documented in [event wire contract](events.md).
## Implemented Drivers
| Driver | Kind | Raw schema | Canonical schema |
| --- | --- | --- | --- |
| `nws_observation` | `observation` | `raw.nws.observation.v1` | `weather.observation.v1` |
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` | `weather.alert.v1` |
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
## Config Requirements
All NWS drivers require HTTP source params:
- `url`
- `user_agent`
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
an upstream `304 Not Modified` response emits no event for that poll.
NWS expects a descriptive `User-Agent`. Do not use anonymous or placeholder
contact values in production configs.
## Upstream Shapes Used
`nws_observation` expects the latest station observation GeoJSON shape. The
normalizer uses fields under `properties` such as `stationId`, `stationName`,
`timestamp`, `textDescription`, measured values, `presentWeather`, and
`cloudLayers`, plus point geometry for day/night inference.
`nws_alerts` expects an alerts FeatureCollection. The normalizer uses the
collection `updated` timestamp, `title`, each feature ID, alert classification
fields, narrative fields, timing fields, sender fields, and references.
`nws_forecast_hourly` and `nws_forecast_narrative` expect gridpoint forecast
GeoJSON with `properties.generatedAt`, `properties.updateTime`, elevation,
polygon geometry, and ordered `periods`.
`nws_forecast_discussion` expects an HTML page containing the discussion text in
a `<pre>` block. The provider helper extracts office identity, product, issue
time, update time, key messages, and short/long term sections.
`nws_weatherstories` expects a JSON response with a `stories` array. The
normalizer uses office ID, start/end/update times, title, description, alt text,
priority, order, and download URL.
## Accept Headers
NWS JSON sources request:
```text
application/geo+json, application/json
```
The forecast discussion source requests:
```text
text/html, application/xhtml+xml
```
## Effective Time
Source events set `effective_at` from the best metadata available:
- observations: `properties.timestamp`;
- alerts: collection `updated`, otherwise latest per-alert timestamp;
- hourly and narrative forecasts: `properties.generatedAt`, otherwise update
time;
- forecast discussions: parsed issue time;
- weather stories: latest story update time, otherwise latest story start time.
Normalizers use canonical payload time as the normalized event effective time.
Alerts and weather stories fall back to the incoming event envelope when the
payload does not provide a better snapshot time.
## Mapping Notes
Observations preserve raw `presentWeather` fragments and infer WMO condition
codes from METAR phenomena, provider text, and cloud-layer fallback. Sea-level
pressure is preferred over barometric pressure when present.
Hourly forecasts infer WMO condition codes from `shortForecast` and icon tokens.
Narrative forecasts preserve text but intentionally leave period condition codes
unset. Forecast temperatures are converted to Celsius when NWS supplies
Fahrenheit, and wind speed strings are converted to kilometers per hour.
Alert timing fields are parsed best-effort. Invalid per-alert timestamps are
left unset rather than failing the whole alert run. Missing alert IDs are
synthesized from the run snapshot time and array position.
Forecast discussion parsing requires an issue time. Weather story entries require
start time, end time, and update time.
## Failure Behavior
Constructor validation failures stop daemon startup. Polling failures are
returned to the scheduler. JSON sources still emit raw payloads when only
minimal metadata decoding fails. Forecast discussion polling fails if the HTML
cannot be parsed enough to determine the issue time.
## Tests To Inspect
- `internal/sources/nws/*_test.go`
- `internal/normalizers/nws/*_test.go`
- `internal/providers/nws/*_test.go`
- fixtures under `internal/providers/nws/testdata`

View File

@@ -0,0 +1,101 @@
# Open-Meteo Integration Notes
## Purpose
This document describes the Open-Meteo API usage currently implemented by
`weatherfeeder`. It is for developers and operators maintaining Open-Meteo
source URLs, normalizers, fixtures, and tests.
General config syntax belongs in [configuration](../config.md). Emitted JSON
events are documented in [event wire contract](events.md).
## Implemented Drivers
| Driver | Kind | Raw schema | Canonical schema |
| --- | --- | --- | --- |
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` | `weather.observation.v1` |
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` |
## Config Requirements
Both drivers require HTTP source params:
- `url`
- `user_agent`
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
an upstream `304 Not Modified` response emits no event for that poll.
## Upstream Shapes Used
`openmeteo_observation` expects a JSON response with top-level location/timezone
metadata and a `current` object. The normalizer uses:
- `latitude`, `longitude`, `timezone`, `utc_offset_seconds`;
- `current.time`;
- current temperature, apparent temperature, relative humidity, weather code,
wind speed/direction/gusts, pressure, and `is_day`.
`openmeteo_forecast` expects top-level location/timezone metadata and an
array-oriented `hourly` object. The normalizer uses:
- `hourly.time`;
- hourly temperature, apparent temperature, dew point, relative humidity,
precipitation probability, precipitation amount, snowfall, weather code,
pressure, wind speed/direction/gusts, `is_day`, cloud cover, visibility, and
UV index.
Open-Meteo field presence is allowed to vary. Missing optional arrays produce
nil canonical fields for the affected periods.
## Accept Header
Open-Meteo sources request:
```text
application/json
```
## Time Handling
Open-Meteo timestamps often omit an explicit offset. The provider helper parses
times by using the returned `timezone` or `utc_offset_seconds` when needed.
Observation source events set `effective_at` from `current.time` when it can be
parsed. Hourly forecast source events prefer `current.time`, then the first
non-empty `hourly.time` entry.
The hourly forecast normalizer sets canonical `issuedAt` from the incoming event
`emitted_at` when present, otherwise from the first hourly period start.
Normalized forecast `effective_at` matches `issuedAt`.
## Mapping Notes
Open-Meteo is not a station feed. Weatherfeeder synthesizes canonical
station/location IDs from latitude and longitude when both are available.
Open-Meteo weather codes are WMO codes and are treated as authoritative.
Canonical text is derived from the WMO code and day/night hint.
Wind speed and gust fields are treated as kilometers per hour. Pressure values
are treated as hPa and converted to Pa. Snowfall values are treated as
centimeters and converted to millimeters.
Hourly forecast period end time is the next period start. The last period uses
the previous interval length, or one hour when there is no previous interval.
## Failure Behavior
Constructor validation failures stop daemon startup. Polling failures are
returned to the scheduler. Metadata decoding failures in sources still allow raw
payload emission when the HTTP response itself succeeded.
Normalization fails when required time data is missing or invalid, such as an
empty `hourly.time` array for hourly forecasts.
## Tests To Inspect
- `internal/sources/openmeteo/source_test.go`
- `internal/normalizers/openmeteo/*_test.go`
- `internal/providers/openmeteo/*_test.go`

View File

@@ -0,0 +1,101 @@
# OpenWeather Integration Notes
## Purpose
This document describes the OpenWeather current-weather usage implemented by
`weatherfeeder`. It is for developers and operators maintaining OpenWeather
source URLs, normalizers, fixtures, and tests.
General config syntax belongs in [configuration](../config.md). Emitted JSON
events are documented in [event wire contract](events.md).
## Implemented Driver
| Driver | Kind | Raw schema | Canonical schema |
| --- | --- | --- | --- |
| `openweather_observation` | `observation` | `raw.openweather.current.v1` | `weather.observation.v1` |
Only current-weather observation polling is registered for OpenWeather.
## Config Requirements
The driver requires HTTP source params:
- `url`
- `user_agent`
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
an upstream `304 Not Modified` response emits no event for that poll.
The configured URL must include:
```text
units=metric
```
Startup fails if `units` is omitted or set to another value. Keep OpenWeather
API keys out of committed configs. Use local config management or deployment
secrets for the `appid` query parameter.
## Upstream Shape Used
The source emits the full current-weather JSON payload as a raw event. The
normalizer uses:
- `coord.lat`, `coord.lon`;
- primary `weather[0]` condition ID, description, and icon;
- `main.temp`, `main.feels_like`, `main.pressure`, `main.humidity`, and
optional `main.sea_level`;
- `visibility`;
- `wind.speed`, `wind.deg`, and `wind.gust`;
- `dt`;
- `sys.sunrise` and `sys.sunset`;
- `id` and `name`.
## Accept Header
OpenWeather sources request:
```text
application/json
```
## Time Handling
Source events set `effective_at` from `dt` when it is present and positive.
The normalizer also uses `dt` as the canonical observation timestamp and
normalized effective time.
## Mapping Notes
Metric units are required so canonical unit conversion is deterministic:
- `main.temp` and `main.feels_like` are treated as Celsius;
- `wind.speed` and `wind.gust` are treated as meters per second and converted to
kilometers per hour;
- pressure values are treated as hPa and converted to Pa.
The primary condition is `weather[0]`. OpenWeather condition IDs are mapped into
the canonical WMO code vocabulary. The human text description is preserved from
the provider description.
Day/night is inferred from the OpenWeather icon suffix when available, otherwise
from sunrise and sunset bounds.
The station ID uses the OpenWeather city ID when present. If no city ID is
present, weatherfeeder synthesizes an ID from coordinates. The station name uses
the provider `name`, falling back to `OpenWeatherMap` when blank.
## Failure Behavior
Constructor validation failures stop daemon startup. Polling also re-checks the
metric-unit requirement before fetching. HTTP failures are returned to the
scheduler. Metadata decoding failures in the source still allow raw payload
emission when the HTTP response itself succeeded.
## Tests To Inspect
- `internal/sources/openweather/source_test.go`
- `internal/normalizers/openweather/*_test.go`
- `internal/providers/openweather/*_test.go`

View File

@@ -0,0 +1,403 @@
# Postgres Integration
This document is the canonical table contract for the optional `postgres` sink.
It describes the schema created and written by weatherfeeder through feedkit's
Postgres sink.
Configure the sink as described in [configuration](../config.md#postgres).
## Initialization And Writes
At startup, each configured Postgres sink opens the database and runs
`CREATE TABLE IF NOT EXISTS` for every weatherfeeder table, followed by
`CREATE INDEX IF NOT EXISTS` for every configured index.
This initialization creates missing tables and indexes only. It does not alter
existing tables, migrate column definitions, drop old objects, or backfill data.
Schema changes require operator-managed database migration.
Events are mapped only for canonical weather schemas:
- `weather.observation.v1`
- `weather.forecast.v1`
- `weather.forecast_discussion.v1`
- `weather.weather_story.v1`
- `weather.alert.v1`
Unsupported schemas produce no writes for this sink. Mapped events are inserted
transactionally. Inserts use ordinary `INSERT`; duplicate primary keys fail the
write.
## Shared Envelope Columns
Parent tables store the feed event envelope:
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
## Table Overview
| Table | Primary key | Prune column |
|---|---|---|
| `observations` | `event_id` | `observed_at` |
| `observation_present_weather` | `event_id`, `weather_index` | `observed_at` |
| `forecasts` | `event_id` | `issued_at` |
| `forecast_periods` | `run_event_id`, `period_index` | `issued_at` |
| `forecast_discussions` | `event_id` | `issued_at` |
| `forecast_discussion_key_messages` | `run_event_id`, `message_index` | `issued_at` |
| `weather_story_runs` | `event_id` | `as_of` |
| `weather_stories` | `run_event_id`, `story_index` | `as_of` |
| `alert_runs` | `event_id` | `as_of` |
| `alerts` | `run_event_id`, `alert_index` | `as_of` |
| `alert_references` | `run_event_id`, `alert_index`, `reference_index` | `as_of` |
## Table Contract
### `observations`
Primary key: `event_id`
Prune column: `observed_at`
Indexes:
- `idx_wf_obs_station_observed_at` on `station_id`, `observed_at`
- `idx_wf_obs_observed_at` on `observed_at`
- `idx_wf_obs_condition_code` on `condition_code`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
| `station_id` | `TEXT` | yes | `payload.stationId` |
| `station_name` | `TEXT` | yes | `payload.stationName` |
| `observed_at` | `TIMESTAMPTZ` | no | `payload.timestamp` |
| `condition_code` | `INTEGER` | no | `payload.conditionCode` |
| `is_day` | `BOOLEAN` | yes | `payload.isDay` |
| `text_description` | `TEXT` | yes | `payload.textDescription` |
| `temperature_c` | `DOUBLE PRECISION` | yes | `payload.temperatureC` |
| `dewpoint_c` | `DOUBLE PRECISION` | yes | `payload.dewpointC` |
| `wind_direction_degrees` | `DOUBLE PRECISION` | yes | `payload.windDirectionDegrees` |
| `wind_speed_kmh` | `DOUBLE PRECISION` | yes | `payload.windSpeedKmh` |
| `wind_gust_kmh` | `DOUBLE PRECISION` | yes | `payload.windGustKmh` |
| `barometric_pressure_pa` | `DOUBLE PRECISION` | yes | `payload.barometricPressurePa` |
| `visibility_meters` | `DOUBLE PRECISION` | yes | `payload.visibilityMeters` |
| `relative_humidity_percent` | `DOUBLE PRECISION` | yes | `payload.relativeHumidityPercent` |
| `apparent_temperature_c` | `DOUBLE PRECISION` | yes | `payload.apparentTemperatureC` |
### `observation_present_weather`
Primary key: `event_id`, `weather_index`
Prune column: `observed_at`
Foreign key: `event_id` references `observations(event_id)` with cascade delete.
Index: `idx_wf_obs_present_observed_at` on `observed_at`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT REFERENCES observations(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `weather_index` | `INTEGER` | no | `payload.presentWeather[]` index. |
| `observed_at` | `TIMESTAMPTZ` | no | `payload.timestamp` |
| `raw_text` | `TEXT` | yes | Compact JSON text from `payload.presentWeather[].raw` |
### `forecasts`
Primary key: `event_id`
Prune column: `issued_at`
Indexes:
- `idx_wf_fc_location_product_issued_at` on `location_id`, `product`, `issued_at`
- `idx_wf_fc_issued_at` on `issued_at`
- `idx_wf_fc_product_issued_at` on `product`, `issued_at`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
| `location_id` | `TEXT` | yes | `payload.locationId` |
| `location_name` | `TEXT` | yes | `payload.locationName` |
| `issued_at` | `TIMESTAMPTZ` | no | `payload.issuedAt` |
| `updated_at` | `TIMESTAMPTZ` | yes | `payload.updatedAt` |
| `product` | `TEXT` | no | `payload.product` |
| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` |
| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` |
| `elevation_meters` | `DOUBLE PRECISION` | yes | `payload.elevationMeters` |
| `period_count` | `INTEGER` | no | `len(payload.periods)` |
### `forecast_periods`
Primary key: `run_event_id`, `period_index`
Prune column: `issued_at`
Foreign key: `run_event_id` references `forecasts(event_id)` with cascade delete.
Indexes:
- `idx_wf_fc_period_start_time` on `start_time`
- `idx_wf_fc_period_end_time` on `end_time`
- `idx_wf_fc_period_run_start` on `run_event_id`, `start_time`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `run_event_id` | `TEXT REFERENCES forecasts(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `period_index` | `INTEGER` | no | `payload.periods[]` index. |
| `issued_at` | `TIMESTAMPTZ` | no | Parent `payload.issuedAt` |
| `start_time` | `TIMESTAMPTZ` | no | `payload.periods[].startTime` |
| `end_time` | `TIMESTAMPTZ` | no | `payload.periods[].endTime` |
| `name` | `TEXT` | yes | `payload.periods[].name` |
| `is_day` | `BOOLEAN` | yes | `payload.periods[].isDay` |
| `condition_code` | `INTEGER` | yes | `payload.periods[].conditionCode` |
| `text_description` | `TEXT` | yes | `payload.periods[].textDescription` |
| `temperature_c` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureC` |
| `temperature_c_min` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureCMin` |
| `temperature_c_max` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureCMax` |
| `dewpoint_c` | `DOUBLE PRECISION` | yes | `payload.periods[].dewpointC` |
| `relative_humidity_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].relativeHumidityPercent` |
| `wind_direction_degrees` | `DOUBLE PRECISION` | yes | `payload.periods[].windDirectionDegrees` |
| `wind_speed_kmh` | `DOUBLE PRECISION` | yes | `payload.periods[].windSpeedKmh` |
| `wind_gust_kmh` | `DOUBLE PRECISION` | yes | `payload.periods[].windGustKmh` |
| `barometric_pressure_pa` | `DOUBLE PRECISION` | yes | `payload.periods[].barometricPressurePa` |
| `visibility_meters` | `DOUBLE PRECISION` | yes | `payload.periods[].visibilityMeters` |
| `apparent_temperature_c` | `DOUBLE PRECISION` | yes | `payload.periods[].apparentTemperatureC` |
| `cloud_cover_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].cloudCoverPercent` |
| `probability_of_precipitation_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].probabilityOfPrecipitationPercent` |
| `precipitation_amount_mm` | `DOUBLE PRECISION` | yes | `payload.periods[].precipitationAmountMm` |
| `snowfall_depth_mm` | `DOUBLE PRECISION` | yes | `payload.periods[].snowfallDepthMm` |
| `uv_index` | `DOUBLE PRECISION` | yes | `payload.periods[].uvIndex` |
### `forecast_discussions`
Primary key: `event_id`
Prune column: `issued_at`
Indexes:
- `idx_wf_discussion_office_product_issued_at` on `office_id`, `product`, `issued_at`
- `idx_wf_discussion_issued_at` on `issued_at`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
| `office_id` | `TEXT` | yes | `payload.officeId` |
| `office_name` | `TEXT` | yes | `payload.officeName` |
| `issued_at` | `TIMESTAMPTZ` | no | `payload.issuedAt` |
| `updated_at` | `TIMESTAMPTZ` | yes | `payload.updatedAt` |
| `product` | `TEXT` | no | `payload.product` |
| `short_term_qualifier` | `TEXT` | yes | `payload.shortTerm.qualifier` |
| `short_term_issued_at` | `TIMESTAMPTZ` | yes | `payload.shortTerm.issuedAt` |
| `short_term_text` | `TEXT` | yes | `payload.shortTerm.text` |
| `long_term_qualifier` | `TEXT` | yes | `payload.longTerm.qualifier` |
| `long_term_issued_at` | `TIMESTAMPTZ` | yes | `payload.longTerm.issuedAt` |
| `long_term_text` | `TEXT` | yes | `payload.longTerm.text` |
| `key_message_count` | `INTEGER` | no | `len(payload.keyMessages)` |
### `forecast_discussion_key_messages`
Primary key: `run_event_id`, `message_index`
Prune column: `issued_at`
Foreign key: `run_event_id` references `forecast_discussions(event_id)` with
cascade delete.
Index: `idx_wf_discussion_message_issued_at` on `issued_at`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `run_event_id` | `TEXT REFERENCES forecast_discussions(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `message_index` | `INTEGER` | no | `payload.keyMessages[]` index. |
| `issued_at` | `TIMESTAMPTZ` | no | Parent `payload.issuedAt` |
| `message_text` | `TEXT` | yes | `payload.keyMessages[]` value |
### `weather_story_runs`
Primary key: `event_id`
Prune column: `as_of`
Indexes:
- `idx_wf_story_run_office_as_of` on `office_id`, `as_of`
- `idx_wf_story_run_as_of` on `as_of`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
| `office_id` | `TEXT` | yes | `payload.officeId` |
| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` |
| `story_count` | `INTEGER` | no | `len(payload.stories)` |
### `weather_stories`
Primary key: `run_event_id`, `story_index`
Prune column: `as_of`
Foreign key: `run_event_id` references `weather_story_runs(event_id)` with
cascade delete.
Indexes:
- `idx_wf_stories_start_time` on `start_time`
- `idx_wf_stories_end_time` on `end_time`
- `idx_wf_stories_updated_at` on `updated_at`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `run_event_id` | `TEXT REFERENCES weather_story_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `story_index` | `INTEGER` | no | `payload.stories[]` index. |
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
| `office_id` | `TEXT` | yes | `payload.stories[].officeId` |
| `start_time` | `TIMESTAMPTZ` | no | `payload.stories[].startTime` |
| `end_time` | `TIMESTAMPTZ` | no | `payload.stories[].endTime` |
| `updated_at` | `TIMESTAMPTZ` | no | `payload.stories[].updatedAt` |
| `title` | `TEXT` | yes | `payload.stories[].title` |
| `description` | `TEXT` | yes | `payload.stories[].description` |
| `alt_text` | `TEXT` | yes | `payload.stories[].altText` |
| `priority` | `BOOLEAN` | no | `payload.stories[].priority` |
| `story_order` | `INTEGER` | no | `payload.stories[].order` |
| `download_url` | `TEXT` | yes | `payload.stories[].downloadUrl` |
### `alert_runs`
Primary key: `event_id`
Prune column: `as_of`
Indexes:
- `idx_wf_alert_run_location_as_of` on `location_id`, `as_of`
- `idx_wf_alert_run_as_of` on `as_of`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `event_id` | `TEXT` | no | `event.id` |
| `event_kind` | `TEXT` | no | `event.kind` |
| `event_source` | `TEXT` | no | `event.source` |
| `event_schema` | `TEXT` | no | `event.schema` |
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
| `location_id` | `TEXT` | yes | `payload.locationId` |
| `location_name` | `TEXT` | yes | `payload.locationName` |
| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` |
| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` |
| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` |
| `alert_count` | `INTEGER` | no | `len(payload.alerts)` |
### `alerts`
Primary key: `run_event_id`, `alert_index`
Prune column: `as_of`
Foreign key: `run_event_id` references `alert_runs(event_id)` with cascade
delete.
Indexes:
- `idx_wf_alerts_alert_id` on `alert_id`
- `idx_wf_alerts_severity_expires` on `severity`, `expires`
- `idx_wf_alerts_as_of` on `as_of`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `run_event_id` | `TEXT REFERENCES alert_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `alert_index` | `INTEGER` | no | `payload.alerts[]` index. |
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
| `alert_id` | `TEXT` | no | `payload.alerts[].id` |
| `event` | `TEXT` | yes | `payload.alerts[].event` |
| `headline` | `TEXT` | yes | `payload.alerts[].headline` |
| `severity` | `TEXT` | yes | `payload.alerts[].severity` |
| `urgency` | `TEXT` | yes | `payload.alerts[].urgency` |
| `certainty` | `TEXT` | yes | `payload.alerts[].certainty` |
| `status` | `TEXT` | yes | `payload.alerts[].status` |
| `message_type` | `TEXT` | yes | `payload.alerts[].messageType` |
| `category` | `TEXT` | yes | `payload.alerts[].category` |
| `response` | `TEXT` | yes | `payload.alerts[].response` |
| `description` | `TEXT` | yes | `payload.alerts[].description` |
| `instruction` | `TEXT` | yes | `payload.alerts[].instruction` |
| `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].sent` |
| `effective` | `TIMESTAMPTZ` | yes | `payload.alerts[].effective` |
| `onset` | `TIMESTAMPTZ` | yes | `payload.alerts[].onset` |
| `expires` | `TIMESTAMPTZ` | yes | `payload.alerts[].expires` |
| `area_description` | `TEXT` | yes | `payload.alerts[].areaDescription` |
| `sender_name` | `TEXT` | yes | `payload.alerts[].senderName` |
| `reference_count` | `INTEGER` | no | `len(payload.alerts[].references)` |
### `alert_references`
Primary key: `run_event_id`, `alert_index`, `reference_index`
Prune column: `as_of`
Foreign key: `run_event_id` references `alert_runs(event_id)` with cascade
delete.
Indexes:
- `idx_wf_alert_refs_as_of` on `as_of`
- `idx_wf_alert_refs_sent` on `sent`
| Column | Type | Null | Source |
|---|---|:---:|---|
| `run_event_id` | `TEXT REFERENCES alert_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
| `alert_index` | `INTEGER` | no | Parent alert index. |
| `reference_index` | `INTEGER` | no | `payload.alerts[].references[]` index. |
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
| `id` | `TEXT` | yes | `payload.alerts[].references[].id` |
| `identifier` | `TEXT` | yes | `payload.alerts[].references[].identifier` |
| `sender` | `TEXT` | yes | `payload.alerts[].references[].sender` |
| `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].references[].sent` |
## Retention
When sink param `prune` is set, every successful write transaction deletes rows
older than `now - prune` from every table using that table's prune column.
The sink also exposes manual prune helpers in code, but the `weatherfeeder`
binary does not provide CLI commands for them.
## Reconstructing Canonical Payloads
- `WeatherObservation`: read `observations`, then join
`observation_present_weather` by `event_id` ordered by `weather_index`.
- `WeatherForecastRun`: read `forecasts`, then join `forecast_periods` by
`run_event_id` ordered by `period_index`.
- `WeatherForecastDiscussion`: read `forecast_discussions`, then join
`forecast_discussion_key_messages` by `run_event_id` ordered by
`message_index`.
- `WeatherStoryRun`: read `weather_story_runs`, then join `weather_stories` by
`run_event_id` ordered by `story_index`.
- `WeatherAlertRun`: read `alert_runs`, join `alerts` by `run_event_id` ordered
by `alert_index`, then join `alert_references` by `run_event_id` and
`alert_index` ordered by `reference_index`.

View File

@@ -0,0 +1,90 @@
# Normalizer Internals
## Purpose
Normalizers convert raw provider events into canonical weather events. They are
weather-domain mapping code and should stay independent of runtime wiring,
source polling, and sink persistence.
Detailed package conventions live in `internal/normalizers/doc.go`.
## Inputs And Outputs
Inputs are raw feed events whose schemas identify provider payload shape.
Outputs are canonical feed events using `model` payloads and `weather.*`
schemas.
Current mappings:
| Raw schema | Canonical schema |
| --- | --- |
| `raw.nws.observation.v1` | `weather.observation.v1` |
| `raw.openmeteo.current.v1` | `weather.observation.v1` |
| `raw.openweather.current.v1` | `weather.observation.v1` |
| `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
| `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
| `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` |
| `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
| `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
| `raw.nws.alerts.v1` | `weather.alert.v1` |
## Boundaries
- Normalizers match by `Event.Schema`.
- Normalizers decode raw payloads into provider structs.
- Normalizers map provider data into canonical `model` payloads.
- Normalizers do not fetch network data, read config, route events, or write
sinks.
- Shared cross-provider behavior belongs in `internal/normalizers/common`.
- Provider-specific helper logic shared with sources belongs in
`internal/providers/<provider>`.
## Config Fields Used
Normalizers do not read config. They operate only on incoming events.
## External Adapters Used
Runtime composition creates feedkit's normalize processor with
`RequireMatch=false`. Events without a matching normalizer pass through
unchanged.
Weatherfeeder registers normalizers in a stable order:
1. NWS
2. Open-Meteo
3. OpenWeather
The current normalizers avoid ambiguous matches by using schema equality.
## State
Normalizers should be stateless. Shared helpers should be deterministic and free
of I/O.
## Failure Behavior
Malformed required raw payload data should produce contextual errors from the
owning normalizer. Successful normalization validates the output event before it
continues through the pipeline.
`internal/normalizers/common.Finalize` preserves the input event envelope except
for schema, payload, and effective time. It also rounds canonical float values
to four digits after the decimal point.
## Tests To Inspect
- `internal/normalizers/builtins_test.go`
- provider normalizer tests under `internal/normalizers/nws`
- provider normalizer tests under `internal/normalizers/openmeteo`
- provider normalizer tests under `internal/normalizers/openweather`
- common helper tests under `internal/normalizers/common`
## Invariants
- Match by schema constants from `standards`.
- Preserve the event envelope except for intentional canonical changes.
- Produce canonical payload structs from `model`.
- Validate normalized events before returning them.
- Keep normalizers independent of sources, sinks, config loading, and runtime
composition.

View File

@@ -0,0 +1,118 @@
# Postgres Sink Internals
## Purpose
`internal/sinks/postgres` defines weatherfeeder's canonical-event-to-Postgres
mapping. It supplies a schema definition and mapper to feedkit's generic
Postgres sink.
The consumer-facing table contract is
[`docs/integrations/postgres.md`](../integrations/postgres.md). This document
describes the internal ownership boundary.
## Inputs And Outputs
Inputs are canonical feed events. The mapper currently handles these schemas:
- `weather.observation.v1`
- `weather.forecast.v1`
- `weather.forecast_discussion.v1`
- `weather.weather_story.v1`
- `weather.alert.v1`
Outputs are feedkit `PostgresWrite` values for weatherfeeder-owned tables.
Unsupported schemas produce no writes and no error.
## Boundaries
- Weatherfeeder owns table definitions in `schema.go`.
- Weatherfeeder owns canonical payload mapping in `map.go`.
- Feedkit owns database opening, table and index creation, transactions,
inserts, context-aware consumption, and prune execution.
- Postgres mapping consumes canonical events only. It should not understand raw
provider schemas.
## Config Fields Used
Weatherfeeder registers the `postgres` sink by passing `PostgresSchema()` to
feedkit. Feedkit parses sink params:
- `uri`
- `username`
- `password`
- `prune`, optional duration
Weatherfeeder-owned mapper code does not read config directly.
## External Adapters Used
The runtime registers the sink with:
```go
sinkReg.Register("postgres", fksinks.PostgresFactory(wfpgsink.PostgresSchema()))
```
Feedkit validates events at the sink boundary, calls the weatherfeeder mapper,
validates writes against the compiled schema, inserts rows in a transaction, and
optionally prunes rows older than the configured window.
## State
The mapper is stateless. Durable state is stored in Postgres through feedkit's
sink implementation.
## Mapping Rules
Parent rows preserve event envelope fields where the table supports them:
- `event_id`
- `event_kind`
- `event_source`
- `event_schema`
- `event_emitted_at`
- `event_effective_at`
Child rows use positional indexes to preserve canonical array order:
- `weather_index`
- `period_index`
- `message_index`
- `story_index`
- `alert_index`
- `reference_index`
Required canonical fields are validated before writes are returned:
- observations require `timestamp`;
- forecasts require `issuedAt` and `product`, and each period requires
`startTime` and `endTime`;
- forecast discussions require `issuedAt` and `product`;
- weather story runs require `asOf`, and each story requires `startTime`,
`endTime`, and `updatedAt`;
- alert runs require `asOf`, and each alert requires `id`.
Nullable canonical values are converted to SQL nulls by mapper helpers.
Observation present-weather raw values are stored as compact JSON text.
## Failure Behavior
Payload decode failures, missing required fields, invalid compact JSON values,
or schema/write mismatches return errors to feedkit's sink. Feedkit rolls back
the transaction when a write fails.
Unsupported canonical schemas are ignored by this mapper so other routed events
can use different sinks without Postgres-specific failures.
## Tests To Inspect
- `internal/sinks/postgres/schema_test.go`
- `internal/sinks/postgres/map_test.go`
- feedkit Postgres sink tests when changing generic sink behavior assumptions
## Invariants
- Persist only canonical schemas.
- Preserve event envelope fields in parent rows.
- Preserve array order with child positional indexes.
- Validate required fields before writing.
- Keep table-contract docs synchronized with schema and mapper changes.

110
docs/internal/runtime.md Normal file
View File

@@ -0,0 +1,110 @@
# Runtime Internals
## Purpose
`cmd/weatherfeeder` wires the daemon together. It owns process setup and runtime
composition; provider mapping, source fetching details, and sink persistence
rules stay in their owning packages.
## Inputs And Outputs
The executable reads `config.yml` from the current working directory through
feedkit config loading. It builds configured sources, scheduler jobs, sinks, and
routes, then runs source polling and sink dispatch until shutdown.
Inputs are configured source polls. Outputs are feed events delivered to the
configured sinks.
## Runtime Flow
The implemented flow is:
1. load `config.yml`;
2. register weatherfeeder source drivers;
3. register feedkit built-in sinks and the weatherfeeder Postgres sink;
4. build source inputs and scheduler jobs;
5. validate configured expected kinds against source-advertised kinds;
6. build sinks and compile routes;
7. run the processor chain `normalize`, then `dedupe`;
8. run the scheduler and dispatcher concurrently;
9. shut down on signal or fatal scheduler/dispatcher error.
The in-process event channel is buffered to 256 events. The dedupe processor is
bounded by `dedupeMaxEntries`, currently 2048.
## Boundaries
- Runtime composition belongs in `cmd/weatherfeeder/main.go`.
- Source driver behavior belongs under `internal/sources`.
- Normalizer behavior belongs under `internal/normalizers`.
- Canonical payloads and schema strings belong in `model` and `standards`.
- Postgres mapping belongs under `internal/sinks/postgres`.
`cmd/weatherfeeder` should stay thin and should not contain provider parsing,
canonical mapping, or table-mapping rules.
## Config Fields Used
Runtime wiring consumes the feedkit top-level config sections:
- `sources`: source driver selection, source name, mode, cadence, expected kinds,
and driver params;
- `sinks`: sink driver selection, sink name, and sink params;
- `routes`: event-kind routing to named sinks.
The executable does not expose CLI flags or config path discovery.
## External Adapters Used
Runtime composition uses feedkit for:
- config loading;
- source registry and expected-kind validation;
- scheduler job construction;
- processor registry and chain execution;
- normalization and dedupe processors;
- sink registry and built-in sinks;
- route compilation and dispatch.
Weatherfeeder registers its own source drivers and its Postgres schema mapper.
## State
Weatherfeeder-owned runtime state is in process:
- event channel contents;
- the bounded dedupe key set;
- source instances and their HTTP conditional validators;
- scheduler and dispatcher goroutines.
There is no weatherfeeder-owned durable scheduler state, checkpoint, replay log,
or resume marker. Durable persistence is owned by configured external sinks.
## Failure Behavior
Startup failures are fatal and include context such as config index, source name,
sink name, driver name, or the operation that failed.
At runtime, scheduler and dispatcher errors are sent to a shared error channel.
Context cancellation and deadline errors are treated as normal shutdown. Any
other scheduler or dispatcher error is logged as fatal and cancels the process
context.
The daemon handles `os.Interrupt` and `SIGTERM` with `signal.NotifyContext`.
After both runtime goroutines return, it logs `shutdown complete`.
## Tests To Inspect
- `cmd/weatherfeeder/main_test.go`
- source registry tests under `internal/sources`
- normalizer registration tests under `internal/normalizers`
- feedkit scheduler, processor, dispatch, and sink tests when changing runtime
infrastructure usage
## Invariants
- Keep normalization before dedupe.
- Keep queue sizes and dedupe bounds explicit.
- Preserve context-aware shutdown.
- Keep runtime wiring separate from domain mapping and persistence rules.
- Keep startup validation failures loud and contextual.

105
docs/internal/sources.md Normal file
View File

@@ -0,0 +1,105 @@
# Source Internals
## Purpose
Source packages poll upstream weather providers and emit raw feed events. They
are adapters, not canonical mappers.
Sources should decode only the metadata needed for event identity, effective
time, and routing policy. Full provider payload interpretation belongs in
normalizers.
## Inputs And Outputs
Inputs are feedkit `config.SourceConfig` values and upstream HTTP responses.
Outputs are feed events whose payloads are raw provider JSON and whose schemas
come from `standards`.
Current drivers:
| Driver | Kind | Raw schema |
| --- | --- | --- |
| `nws_observation` | `observation` | `raw.nws.observation.v1` |
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` |
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` |
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` |
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` |
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` |
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` |
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` |
| `openweather_observation` | `observation` | `raw.openweather.current.v1` |
## Boundaries
- Source constructors validate source-specific params.
- Sources use feedkit HTTP helpers for HTTP polling.
- Sources emit raw events and should not build canonical `model` payloads.
- Provider helper packages under `internal/providers/<provider>` hold shared
parsing and validation helpers.
- Registration is centralized in `internal/sources/builtins.go`.
## Config Fields Used
All current source drivers use feedkit `HTTPSource`.
Required params:
- `url`
- `user_agent`
Optional params:
- `conditional`, default `true`;
- `http_timeout`;
- `http_response_body_limit_bytes`.
OpenWeather observation sources additionally require the configured URL to use
metric units. This is enforced by `internal/providers/openweather`.
Source-level `kinds`, when configured, are validated against the source's
advertised `Kinds()`.
## External Adapters Used
Sources use feedkit's HTTP helper for:
- request construction;
- `User-Agent` and `Accept` headers;
- optional conditional GET validators;
- response body size limits;
- JSON raw-message fetches.
NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times
with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs.
## State
HTTP conditional validators are held in each source instance. They are not
persisted across process restarts.
## Failure Behavior
Constructor failures are returned during startup and stop the daemon. Polling
failures are returned to the scheduler.
If a source cannot decode minimal metadata from an otherwise successful upstream
response, it still emits the raw event when possible. The event then falls back
to default ID/effective-time behavior from feedkit source helpers.
Unchanged conditional responses return no events and no error.
## Tests To Inspect
- `internal/sources/builtins_test.go`
- provider source tests under `internal/sources/nws`
- provider source tests under `internal/sources/openmeteo`
- provider source tests under `internal/sources/openweather`
- provider helper tests under `internal/providers`
## Invariants
- Emit raw provider schemas from `standards`.
- Keep provider-to-canonical mapping out of sources.
- Keep HTTP behavior context-aware.
- Keep driver registration explicit and stable.
- Keep source tests independent of live upstream services.

158
docs/operations.md Normal file
View File

@@ -0,0 +1,158 @@
# Operations
This document describes how to run and observe the `weatherfeeder` daemon in its
current form. For configuration syntax, see [configuration](config.md). For the
CLI surface, see [CLI reference](cli.md).
## Normal Workflow
1. Prepare `config.yml` in the process working directory.
2. Start the daemon with `./weatherfeeder` or `go run .` from
`cmd/weatherfeeder`.
3. Watch stderr logs for startup or runtime errors.
4. Consume events from the configured sinks.
5. Stop the process with `Ctrl-C` or `SIGTERM`.
The daemon has no admin subcommands and no runtime reload command. Change the
config file and restart the process to apply configuration changes.
Maintained copyable configs are available under [`examples/`](../examples/).
## Runtime Lifecycle
On startup, `weatherfeeder`:
1. loads `config.yml` from the current working directory;
2. registers built-in source drivers;
3. registers stdout, NATS, and weatherfeeder Postgres sink drivers;
4. builds sources and validates configured `kinds` against source metadata;
5. builds sinks and compiles routes;
6. starts the scheduler and dispatcher;
7. processes events through normalization, then in-memory dedupe;
8. routes processed events to configured sinks.
Startup errors are fatal and terminate the process. Runtime poll, pipeline, and
sink errors are logged and the process continues unless the scheduler or
dispatcher returns a fatal error.
## Logs
The process uses the Go standard logger with date, time, and microseconds. Logs
go to stderr.
Common log prefixes:
| Prefix | Meaning |
|---|---|
| `config load failed` | `config.yml` could not be read, parsed, or validated. |
| `build source failed` | A source driver or its params are invalid. |
| `source expected kinds validation failed` | Configured source `kinds` do not match the source driver. |
| `build sink failed` | A sink driver or its params are invalid, or a sink could not initialize. |
| `compile routes failed` | Routes reference invalid sinks or kinds. |
| `scheduler: poll failed` | A source poll failed; the source will be polled again on its next interval. |
| `dispatcher: pipeline error` | Normalization or dedupe failed for one event. |
| `dispatch: sink ... failed consuming event` | A sink failed to consume one event. |
| `shutdown complete` | Scheduler and dispatcher have exited. |
## Scheduling And Polling
Current weatherfeeder sources are polling sources. Each source uses its
configured `every` interval. The scheduler applies jitter before the first poll
and before each interval tick. If no jitter is configured in code, feedkit uses
`min(every/10, 30s)`, capped at half the interval.
Poll failures are logged and do not stop the daemon. A failed poll emits no
events for that source until a subsequent poll succeeds.
## Conditional HTTP Fetches
All current sources use feedkit's HTTP polling helper. By default,
`params.conditional` is `true`, so the helper keeps ETag and Last-Modified
validators in memory for each source instance.
If the upstream returns `304 Not Modified`, the source emits no events for that
poll. Validator state is in memory only; restarting the process starts with no
cached validators.
## Processing And Dedupe
Every event passes through normalization first and dedupe second.
Normalizers match raw source schemas and produce canonical `weather.*.v1`
payloads. If an event has no matching normalizer, the normalize processor passes
it through unchanged.
Dedupe keys by event ID and stores a bounded in-memory set of 2048 recent IDs.
Duplicate IDs are dropped. Dedupe state is not persisted, so a restart starts
with an empty dedupe set.
## Routing And Sink Fanout
Routes choose sinks by event kind. If `routes` is omitted, every sink receives
every event kind. If a route omits `kinds`, that route also matches all kinds.
The dispatcher creates one queue and one worker goroutine per sink. The default
per-sink queue size is 64. `weatherfeeder` does not currently expose config
fields for sink queue size, enqueue timeout, or consume timeout.
Sink errors are logged per event. A sink failure does not stop other sinks from
receiving the same event.
## Sink Behavior
### stdout
The stdout sink validates each event and prints one JSON object per line to
stdout. This is useful for local inspection and log forwarding.
### NATS
The NATS sink connects lazily on the first event, reuses the connection while it
is open, and publishes each event as JSON to the configured subject. Connection,
marshal, and publish failures are logged by the dispatch worker.
### Postgres
The Postgres sink opens the database during startup. It creates missing tables
and indexes with `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`.
It does not modify existing table definitions.
Each mapped canonical event is written in one transaction. If `params.prune` is
set, the sink deletes rows older than `now - prune` from every weatherfeeder
table in that same transaction. See the
[Postgres table contract](integrations/postgres.md).
## State And Recovery
`weatherfeeder` keeps only runtime state in process memory:
- scheduler goroutines and timers;
- HTTP conditional request validators;
- event channel buffers;
- per-sink fanout queues;
- the dedupe ID set.
Durable state is external sink state: NATS broker state outside this process and
Postgres tables managed by the configured database.
There is no internal checkpoint, replay log, or resume marker. To recover from a
process failure, fix the underlying issue and restart the daemon from a working
directory containing the desired `config.yml`.
## Shutdown
`weatherfeeder` listens for `os.Interrupt` and `SIGTERM`. On shutdown, the
shared context is canceled. Scheduler jobs stop polling, dispatch workers stop,
and the process logs `shutdown complete`.
Queued sink work may be dropped when shutdown context cancellation reaches the
fanout workers. Use external sink durability, such as Postgres or broker
retention, for durable downstream state.
## Caveats
- There is no health-check endpoint.
- There is no runtime config reload.
- There are no built-in metrics.
- Source conditional request state and dedupe state are reset by restart.
- Existing Postgres schemas are not migrated automatically.

186
docs/policy/architecture.md Normal file
View File

@@ -0,0 +1,186 @@
# Architecture Policy
## Purpose
This document defines `weatherfeeder`'s development architecture and invariants for maintainers and LLM coding agents. It describes how the implemented system is built and how subsequent changes should preserve its boundaries.
This is an inward-facing policy document. User-facing wire contracts belong in
[`docs/integrations/events.md`](../integrations/events.md), and roadmap items
belong under [`docs/roadmap/`](../roadmap/).
## Project Shape
`weatherfeeder` is a config-driven Go daemon that polls upstream weather providers, emits feed events, normalizes provider-specific raw payloads into canonical weather payloads, and routes those events to configured sinks.
The implemented runtime flow is:
1. `cmd/weatherfeeder` loads `config.yml` from the working directory.
2. Source drivers are built through the source registry.
3. Feedkit scheduler jobs poll sources and publish raw events onto an in-process event channel.
4. A pipeline runs normalization, then in-memory dedupe.
5. The dispatcher routes processed events to configured sinks by event kind.
6. Sinks consume events independently through feedkit fanout workers.
Canonical payload structs live in `model`. Schema identifiers and cross-provider wire conventions live in `standards`. Source adapters live under `internal/sources`. Normalizers live under `internal/normalizers`. Provider-specific parsing helpers shared by sources and normalizers live under `internal/providers`. Sink-specific persistence mapping lives under `internal/sinks`.
## Core Design Principles
- Hexagonal boundaries: provider APIs, config loading, scheduling, dispatch, and sinks are external mechanisms around the weather domain model and normalization logic.
- Raw-to-canonical flow: sources should fetch and envelope raw provider payloads; normalizers should own provider-to-canonical mapping.
- Schema-based routing: normalizers match on event schema, not source name or event kind.
- Composable registries: source drivers, normalizers, processors, and sinks are assembled explicitly through registries.
- Bounded concurrency: scheduling and sink fanout are concurrent, but the application should keep queues, goroutine ownership, logging, and cancellation behavior visible.
- Standard-library-first: use the Go standard library unless a narrow dependency materially improves maintainability or interoperability.
- Current-behavior docs: outside roadmap files, document only implemented behavior.
## Architectural Boundaries
Core/domain logic:
- `model` defines canonical payload structs and JSON field names.
- `standards` defines schema strings, versioning conventions, WMO constants, and shared wire policy.
- Normalizer mapping code is domain logic and should stay independent of CLI setup, filesystem paths, sink details, and runtime orchestration.
Adapters:
- Source adapters under `internal/sources/<provider>` fetch upstream data and produce raw feed events.
- Sink adapters under `internal/sinks/<sink>` map canonical events to external systems.
- Provider helper packages under `internal/providers/<provider>` contain pure parsing or provider-specific helper logic shared by sources and normalizers.
Runtime composition:
- `cmd/weatherfeeder/main.go` owns process wiring: config load, registry setup, scheduler jobs, processor chain, dispatcher, signal cancellation, and logging.
- It should remain thin. Do not move provider mapping or sink persistence rules into `cmd/weatherfeeder`.
Tests and examples:
- The sample `cmd/weatherfeeder/config.yml` is executable test input and is load-tested.
- Tests should keep exercising package contracts directly rather than relying only on full-daemon execution.
## Modules Or Processing Steps
The implemented processing steps are source polling, normalization, dedupe, and sink dispatch.
Source contract:
- Build from `feedkit/config.SourceConfig`.
- Validate required driver params inside the source constructor.
- Advertise emitted event kinds through `Kinds()` when possible.
- Emit raw schemas from `standards`.
- Decode only minimal metadata needed for event identity and effective time; leave full provider decoding to normalizers.
- Respect `context.Context` during network work.
Normalizer contract:
- One normalizer type per normalizer file.
- Match by `Event.Schema`.
- Decode raw payloads into provider structs.
- Map to canonical `model` payloads.
- Preserve the incoming event envelope except for intentional schema, payload, and effective-time changes.
- Use shared helpers in `internal/normalizers/common` for cross-provider behavior.
- Follow the detailed normalizer conventions in `internal/normalizers/doc.go`.
Sink contract:
- Consume canonical schemas, not provider raw schemas.
- Keep sink mapping isolated from normalizers and sources.
- Preserve event envelope fields in durable storage where the sink schema supports it.
- Validate required canonical fields before writing.
## State, Inputs, and Outputs
Inputs are configured source polls. The daemon currently uses feedkit's YAML config model with sources, sinks, and routes.
Outputs are feed events sent to configured sinks. Implemented sink support comes from feedkit built-ins plus weatherfeeder's Postgres schema mapping. The sample config includes stdout and NATS routes; Postgres is configured as an optional commented example.
The daemon's own state is in-process:
- the event channel buffers events during runtime;
- the dedupe processor stores a bounded in-memory key set;
- source instances may keep HTTP conditional request state through feedkit HTTP source helpers;
- scheduler and dispatcher state is not persisted by `weatherfeeder`.
Durable persistence is an external sink concern. The Postgres table contract is
documented in [`docs/integrations/postgres.md`](../integrations/postgres.md);
the consumer-facing event contract is documented in
[`docs/integrations/events.md`](../integrations/events.md).
## Configuration and CLI Boundaries
The implemented executable reads `config.yml` from the current working directory. It does not currently expose CLI flags or config path discovery.
Configuration shape is owned by feedkit's config package:
- `sources` define named source drivers, mode, poll cadence, expected kinds, and driver params.
- `sinks` define named sink drivers and sink params.
- `routes` connect event kinds to sinks.
Weatherfeeder-specific config policy belongs in source and sink constructors, registry setup, and tests. Do not spread config parsing through domain model or normalizer packages.
[`docs/config.md`](../config.md) and [`docs/cli.md`](../cli.md) are the
canonical user/operator references. This policy should stay architectural and
avoid duplicating those references.
## Errors, Logging, and Diagnostics
`cmd/weatherfeeder` uses the standard library `log` package with timestamps and microseconds.
Startup errors are fatal and include config index, source/sink name, driver name, or operation context where available. Runtime scheduler and dispatcher errors are reported through logs; context cancellation and deadline errors are treated as shutdown conditions.
Normalizers and sink mappers should wrap errors with operation and payload context, for example decode, parse, map, scan, or required-field context. Avoid logging or returning whole upstream payloads by default.
The daemon handles `os.Interrupt` and `SIGTERM` through `signal.NotifyContext`. Sources, sinks, scheduler jobs, dispatcher, and processors should respect `context.Context`.
## Testing Expectations
When changing behavior, inspect or add focused tests in the owning package.
Expected coverage by change type:
- Source drivers: constructor behavior, advertised kinds, poll event schema/kind, effective-time policy, unchanged responses, and malformed metadata handling.
- Normalizers: schema matching, canonical schema output, key field mapping, effective time, malformed required fields, and wire-shape regressions.
- Provider helpers: parsing edge cases and fixtures.
- Runtime wiring: config loading, source registry build, scheduler job creation, processor ordering, pass-through behavior, and dedupe behavior.
- Postgres sink: schema shape, mapper writes, required-field validation, nullable handling, and compact JSON behavior.
- Documentation-sensitive examples: keep sample config loadable.
Use local test servers and fixtures rather than real upstream services. Full-package tests should remain fast and deterministic.
## Dependency Policy
Prefer the Go standard library for parsing, HTTP handling, time handling, logging, and tests where reasonable.
Existing broad runtime composition is delegated to `feedkit`, which provides config, sources, scheduler, processors, dispatch, and sinks. Keep weatherfeeder-specific code from depending directly on low-level external clients when feedkit or a small adapter can contain that dependency.
Third-party dependencies should be narrow, justified, and preferably de facto standard for their purpose. YAML parsing through feedkit is an acceptable example. Do not add dependencies for small conveniences, and do not let dependency-specific types leak across package boundaries unless that dependency is the package's explicit contract.
## Documentation Expectations
Documentation must follow [`docs/policy/documentation.md`](documentation.md).
Rules for architecture-related docs:
- Current-behavior docs must describe implemented behavior only.
- Roadmap or speculative work belongs only under `docs/roadmap/`.
- Prefer links to canonical docs over repeated reference material.
- Update docs in the same change when modifying schemas, config behavior, runtime behavior, adapters, or persistence contracts.
## Architectural Invariants
- Keep `cmd/weatherfeeder` as composition code, not domain logic.
- Keep source fetching separate from normalization.
- Keep normalizers matched by schema constants from `standards`.
- Keep canonical payload structs in `model` and treat JSON tags as wire contract.
- Keep provider-specific helpers under `internal/providers/<provider>` when shared by sources and normalizers.
- Keep cross-provider normalizer helpers pure and deterministic.
- Keep sink persistence mapping isolated under `internal/sinks/<sink>`.
- Preserve explicit registry-based extension points for sources and normalizers.
- Preserve context-aware shutdown and bounded in-process queues.
- Avoid broad dependencies without clear architectural value.
## Non-Goals
- `weatherfeeder` is not an HTTP API. API serving belongs to separate consumers such as `weatherapi`.
- `weatherfeeder` does not own long-term durable state except through configured external sinks.
- `weatherfeeder` does not provide a general plugin runtime; new built-in providers and sinks are registered in code.
- Architecture policy is not a CLI, config, or wire-contract reference.

202
docs/policy/development.md Normal file
View File

@@ -0,0 +1,202 @@
# Development Policy
## Purpose
This document describes how to change `weatherfeeder` safely. It is for
maintainers and coding agents working in the repository.
Use this alongside the [architecture policy](architecture.md). User-facing CLI,
configuration, operations, and wire-contract details belong in their canonical
docs, not here.
## Repository Layout
- `cmd/weatherfeeder/`: executable wiring, sample `config.yml`, and runtime
composition tests.
- `model/`: canonical weather payload structs. JSON tags are part of the wire
contract.
- `standards/`: schema strings, versioning conventions, WMO constants, and
shared wire-format policy.
- `internal/sources/`: source adapters that poll upstream providers and emit raw
feed events.
- `internal/normalizers/`: raw-to-canonical event transforms.
- `internal/providers/`: pure provider helper code shared by sources and
normalizers.
- `internal/sinks/postgres/`: weatherfeeder-owned Postgres schema and canonical
event mapper.
- `docs/`: current behavior, policies, integration contracts, and roadmap files.
- `examples/`: maintained, copyable configuration examples.
## Build And Test
Run the full test suite before committing behavior or documentation changes that
depend on code behavior:
```sh
go test ./...
```
Use narrower commands while iterating:
```sh
go test ./cmd/weatherfeeder
go test ./internal/sources/...
go test ./internal/normalizers/...
go test ./internal/sinks/postgres
```
Format Go code before committing:
```sh
gofmt -w <changed-go-files>
```
Do not require live upstream weather services, NATS, or Postgres for unit tests.
Use fixtures, local test servers, and package-level tests.
## Coding Conventions
- Keep `cmd/weatherfeeder` focused on composition: config load, registry setup,
scheduler jobs, processor chain, dispatch, signal handling, and logging.
- Keep source fetching separate from normalizer mapping.
- Match normalizers by schema constants from `standards`, not source names.
- Keep provider-specific helper code under `internal/providers/<provider>` when
both sources and normalizers use it.
- Keep cross-provider normalizer helpers pure and deterministic under
`internal/normalizers/common`.
- Keep sink persistence mapping isolated under `internal/sinks/<sink>`.
- Wrap errors with operation context, but do not include whole upstream payloads
in errors or logs by default.
- Prefer explicit registries and small package-level constructors over hidden
global behavior.
## Dependency Policy
Prefer the Go standard library unless a dependency materially improves
maintainability or interoperability.
`feedkit` owns generic daemon infrastructure for config, HTTP source helpers,
scheduling, processors, dispatch, and sinks. Weatherfeeder code should contain
weather-domain behavior and narrow adapter logic rather than duplicating feedkit
infrastructure.
Do not add broad dependencies for small conveniences. Do not let
dependency-specific types leak across package boundaries unless that dependency
is the package contract.
## Adding Config Fields
Generic config shape is owned by feedkit. Weatherfeeder-specific config behavior
belongs in source or sink constructors, registry setup, and tests.
When adding config behavior:
- validate required params at the adapter boundary;
- keep secrets in environment variables or placeholders, not committed values;
- update [configuration docs](../config.md);
- update maintained examples when the change affects normal operation;
- add or update config-load tests for example files when practical.
## Adding CLI Flags
The executable currently reads `config.yml` from the current working directory.
If CLI flags are added:
- keep parsing in `cmd/weatherfeeder`;
- avoid moving config policy into domain packages;
- update [CLI docs](../cli.md);
- update tests that exercise command behavior.
## Adding A Source Driver
Source drivers should fetch upstream data and emit raw events with minimal
metadata decoding.
Checklist:
- implement the driver under `internal/sources/<provider>`;
- build from `config.SourceConfig`;
- validate required params in the constructor;
- use feedkit HTTP helpers for HTTP polling when applicable;
- emit raw schema constants from `standards`;
- advertise emitted kinds through `Kinds()`;
- decode only metadata needed for event ID and effective time;
- register the driver in `internal/sources/builtins.go`;
- add constructor, kind, and polling tests;
- update config docs and examples when operators need new configuration;
- add provider integration notes when the provider contract needs maintenance
context.
## Adding A Normalizer
Normalizers own provider-to-canonical mapping.
Checklist:
- add one normalizer type per normalizer file;
- match using `Event.Schema`;
- decode raw payloads into provider structs;
- map to canonical `model` payloads;
- use `internal/normalizers/common.Finalize` so envelope handling and float
rounding stay consistent;
- preserve input envelope fields except schema, payload, and effective time;
- register through the provider package and `internal/normalizers/builtins.go`;
- add tests for schema matching, key payload fields, effective time, malformed
required data, and output validation.
## Adding Canonical Models Or Schemas
Canonical event changes affect multiple contracts.
Checklist:
- update payload structs in `model`;
- add or update schema constants in `standards`;
- update [event wire contract docs](../integrations/events.md);
- update normalizers that produce the schema;
- update Postgres mapping if the schema is persisted;
- add tests for wire shape and mapper behavior.
## Adding Postgres Mapping
Weatherfeeder owns the canonical-event-to-table mapping. Feedkit owns the
generic Postgres sink mechanics.
Checklist:
- update `internal/sinks/postgres/schema.go`;
- update `internal/sinks/postgres/map.go`;
- preserve event envelope columns in parent rows when the table supports them;
- validate required canonical fields before writing;
- use positional indexes for child rows that represent arrays;
- update mapper and schema tests;
- update [Postgres integration docs](../integrations/postgres.md) when the table
contract changes.
## Examples And Documentation
Documentation must follow the [documentation policy](documentation.md).
When behavior changes, update the canonical docs in the same change:
- config shape: `docs/config.md`;
- CLI behavior: `docs/cli.md`;
- operations and recovery: `docs/operations.md`;
- troubleshooting: `docs/troubleshooting.md`;
- external contracts: `docs/integrations/`;
- internal component behavior: `docs/internal/`;
- copyable configs: `examples/`.
Keep roadmap content under `docs/roadmap/`. Current-behavior docs must describe
implemented behavior only.
## Review Checklist
Before committing:
- run focused tests for changed packages;
- run `go test ./...` for broad behavior or documentation changes tied to code;
- verify maintained examples still load when examples or config docs changed;
- check links in changed docs;
- search for stale paths, unsupported features, and secret-like values;
- keep unrelated refactors out of the change.

View File

@@ -0,0 +1,356 @@
# Go Project Documentation Policy
## Purpose
Project documentation must help four audiences:
1. users who need to run the application;
2. administrators/operators who need to configure and operate it;
3. developers who need to understand and change it safely;
4. LLM coding agents that need clear scope, boundaries, and invariants.
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
## Core Rules
### 1. Keep docs concise
Each document should cover a defined scope and only the essentials for that scope.
Avoid:
- long background explanations;
- repeated reference material;
- implementation detail in user-facing docs;
- aspirational language outside roadmap docs;
- verbose examples where one minimal example is clearer.
### 2. Document only implemented behavior outside roadmap files
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
- `docs/roadmap/`
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
### 3. Use canonical homes
Each type of information should have one canonical location.
Canonical homes:
- project purpose and quickstart: `README.md`
- development principles: `docs/policy/architecture.md`
- configuration reference: `docs/config.md`
- CLI reference: `docs/cli.md`
- operations and recovery: `docs/operations.md`
- troubleshooting: `docs/troubleshooting.md`
- implemented internals: `docs/internal/`
- future work: `docs/roadmap/`
- contributor workflow: `docs/policy/development.md`
- copyable examples: `examples/`
Other files should summarize briefly and link to the canonical source.
### 4. Keep examples real
Examples should be valid, maintained, and free of secrets.
Where practical:
- example configs should load successfully;
- example commands should match real CLI syntax;
- important examples should be covered by tests.
## Documentation Profiles
All projects require:
- `README.md`
- `docs/policy/architecture.md`
Additional docs depend on the project.
### Small library
Recommended:
- `docs/policy/development.md`, if contributor conventions are non-obvious
### Simple CLI
Required:
- `docs/cli.md`
Recommended:
- `docs/policy/development.md`
### Config-driven CLI
Required:
- `docs/cli.md`
- `docs/config.md`
Recommended:
- `examples/`
- `docs/policy/development.md`
### Stateful or operator-facing application
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
Recommended:
- `docs/troubleshooting.md`
- `examples/`
- `docs/policy/development.md`
### Modular, staged, service-oriented, or orchestration application
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
- `docs/internal/`
- `docs/policy/development.md`
Recommended:
- `docs/troubleshooting.md`
- validated examples under `examples/`
## Required Documents
### README.md
**Audience:** users, administrators, operators
The README is the outward-facing project orientation page.
It should include, in order:
1. concise description;
2. elevator pitch;
3. shortest useful command or usage example;
4. links to targeted docs.
The README should be short. It is not a manual.
The “shortest useful command” means the simplest command that performs the projects core use case. (It does not mean `app --help`.)
### docs/policy/architecture.md
**Audience:** developers, LLM coding agents
`docs/policy/architecture.md` is required for every project.
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
It should include:
- project shape;
- core design principles;
- package and boundary philosophy;
- state/persistence philosophy, if applicable;
- external integration philosophy, if applicable;
- error-handling and logging principles;
- testing expectations;
- documentation expectations;
- architectural invariants;
- explicit non-goals, if useful.
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
### docs/policy/development.md
**Audience:** developers, LLM coding agents
Required for projects maintained by humans and LLM coding agents.
It should include:
- repository layout;
- build/test commands;
- coding conventions;
- dependency policy;
- how to add config fields;
- how to add CLI flags;
- how to add stages/modules/adapters, if applicable;
- how to update examples;
- documentation update expectations.
### docs/config.md
**Audience:** administrators, operators, advanced users
Required for applications with configuration files.
It should include, in order:
1. config file locations and discovery precedence;
2. minimal working config;
3. production-oriented config;
4. full configuration reference;
5. secrets handling, if applicable;
6. links to maintained examples.
The full configuration reference should be canonical.
### docs/cli.md
**Audience:** users, administrators, operators
Required for CLI applications.
It should include, in order:
1. shortest useful command;
2. command overview;
3. complete flag reference;
4. common workflows;
5. diagnostic or recovery commands, if applicable.
Explain when commands are useful, not just their syntax.
### docs/operations.md
**Audience:** administrators, operators
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
It should cover:
- normal workflow;
- filesystem layout;
- remote storage layout, if applicable;
- logs and manifests;
- resume/retry behavior;
- cleanup behavior;
- archive/backup behavior;
- safe recovery procedures;
- operational caveats.
### docs/troubleshooting.md
**Audience:** administrators, operators
Recommended once recurring failure modes exist.
Each entry should include:
- symptom;
- likely cause;
- diagnostic command or inspection step;
- safe fix;
- relevant links.
### docs/internal/
**Audience:** developers, LLM coding agents
Required for modular, staged, service-oriented, or orchestration projects.
This directory describes implemented internal components. It is not the roadmap.
Use one file per major component where useful.
Each component doc should include:
1. purpose;
2. inputs and outputs;
3. boundaries;
4. config fields used;
5. external adapters used;
6. state or manifest behavior, if applicable;
7. skip/resume behavior, if applicable;
8. failure behavior;
9. tests to inspect before changing;
10. architectural invariants.
### docs/roadmap/
**Audience:** maintainers, developers, LLM coding agents
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
Roadmap docs should clearly distinguish:
- proposed work;
- accepted plans;
- deferred ideas;
- rejected ideas;
- implementation prompts or task breakdowns, if useful.
Roadmap docs should not be confused with current behavior.
### docs/integrations/
**Audience:** developers, LLM coding agents
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
Use one file per integration where useful.
## Examples Directory
Projects with non-trivial configuration or workflows should include `examples/`.
Useful examples include:
- minimal working config;
- production-oriented config;
- full annotated config;
- local development config;
- remote/object-storage config;
- minimal session/input file.
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
## Security and Privacy
Docs and examples must not include:
- real API keys;
- tokens;
- passwords;
- private keys;
- private environment dumps;
- sensitive user data;
- raw private transcripts;
- private infrastructure details unless intentionally public.
Document secret-handling mechanisms, not actual secret values.
## Maintenance Rules
When docs change, verify the affected behavior.
Where practical:
- load example config files in tests;
- test CLI examples or command parser behavior;
- validate documented flags against real flags;
- remove stale references;
- update links after renames;
- keep roadmap content out of non-roadmap docs.
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
Documentation is complete only when it matches the current code.
## Documentation Change Checklist
Before merging documentation changes, verify:
- README is concise and orientation-focused.
- `docs/policy/architecture.md` describes development principles.
- Future work appears only under `docs/roadmap/`.
- User-facing docs avoid unnecessary internals.
- Developer-facing docs preserve boundaries and invariants.
- Config examples match the schema.
- CLI examples match real commands and flags.
- Defaults appear in the canonical config reference.
- No secrets or private data are included.
- Links are accurate.

View File

@@ -0,0 +1,491 @@
# Documentation Roadmap
## Purpose
This roadmap defines the work required to bring `weatherfeeder` documentation into compliance with [`docs/policy/documentation.md`](../policy/documentation.md) and the current implementation described by [`docs/policy/architecture.md`](../policy/architecture.md).
This is a planning document only. Implementation stages must update current-behavior documentation so it reflects implemented code, and keep proposed or unimplemented work under `docs/roadmap/`.
## Repository Documentation Inventory
- `README.md`: keep and rewrite. It is the correct project orientation file, but it is stale: it says stdout is the only implemented sink, uses “current MVP” framing, and lacks a shortest useful run command and links to targeted docs.
- `API.md`: move and rewrite. It currently acts as the event wire contract, but external integration contracts belong under `docs/integrations/`. It is also stale: it omits the emitted `kind` and `emitted_at` envelope fields, documents `effectiveAt` instead of the actual `effective_at` JSON field, lists only some raw schemas, and has duplicate alert `response` rows.
- `docs/policy/documentation.md`: keep and lightly update only if needed. It is the controlling documentation policy and should not be duplicated elsewhere.
- `docs/policy/architecture.md`: keep and lightly update after the documentation tree is migrated. Its architecture content is current, but its link to `API.md` should be updated when the wire contract moves to `docs/integrations/events.md`.
- `docs/roadmap/spc.md`: keep. It is correctly located under roadmap because it describes unimplemented SPC outlook support.
- `docs/roadmap/documentation.md`: create. This file is the current deliverable.
- `cmd/weatherfeeder/config.yml`: keep and treat as an executable sample, but do not use it as the only operator-facing config reference. It includes live endpoints and a commented OpenWeather URL with an apparent real API key; future cleanup should remove or replace that secret-like value.
- `internal/normalizers/doc.go`: keep and lightly update. It contains implemented developer guidance but incorrectly references `internal/standards/schema.go`; the actual package is `standards` at repository root.
- `internal/normalizers/common/doc.go`: keep and lightly update if needed. It is concise and current.
- `internal/providers/nws/doc.go`: keep and lightly update if needed. It documents implemented provider-helper boundaries.
- `internal/providers/openweather/doc.go`: keep and lightly update. It references future forecasts/alerts in a comment; outside roadmap docs this should be rewritten to current behavior only.
- `standards/doc.go`: keep and lightly update. It incorrectly says `internal/standards/doc.go`; actual path is `standards/doc.go`.
- `model/doc.go`: keep and lightly update. It says JSON tags are wire contract for “stdout today; others later,” which is stale because NATS and Postgres are implemented.
- `internal/sinks/postgres/doc.go`: keep and lightly update. It is the authoritative implemented Postgres table contract, but should be linked from `docs/integrations/postgres.md` instead of being the only discoverable operator/developer reference.
- `examples/`: create new. No examples directory exists. Current example material is embedded in `cmd/weatherfeeder/config.yml` only.
- `docs/config.md`: create new. Required because `weatherfeeder` is config-driven.
- `docs/cli.md`: create new. Required because `weatherfeeder` is a command-line executable, even though it currently has no flags.
- `docs/operations.md`: create new. Required because `weatherfeeder` is an operator-facing daemon with polling, concurrency, external sinks, retention/pruning, and shutdown behavior.
- `docs/troubleshooting.md`: create new. Recommended and useful because recurring operator failures are visible in constructors and config validation.
- `docs/policy/development.md`: create new. Required for a modular project maintained by humans and coding agents.
- `docs/internal/`: create new. Required because the project is modular and staged.
- `docs/integrations/`: create new. Required because the project depends on external weather APIs, NATS, Postgres, and an emitted event contract.
## Policy Compliance Assessment
Required missing documents:
- `docs/cli.md`
- `docs/config.md`
- `docs/operations.md`
- `docs/internal/`
- `docs/policy/development.md`
- `docs/integrations/` entries for important external contracts
Recommended missing documents:
- `docs/troubleshooting.md`
- `examples/` with maintained config examples
Existing stale or misplaced content:
- `README.md` is not concise enough as a current quickstart and contains stale sink status.
- `API.md` is in the wrong canonical home for an external integration contract and does not match `feedkit/event.Event` JSON tags.
- Some package comments include stale path references or future-looking language outside roadmap files.
- The sample config contains live endpoint examples and an OpenWeather URL with an apparent credential-like `appid` value in a commented block; examples should be secret-free.
Unimplemented, historical, or roadmap-only content outside roadmap:
- README “current MVP” phrasing is development-history framing and should be removed.
- Comments such as “others later,” “forecasts/alerts later,” and “future expensive steps” should be rewritten or removed unless they describe an implemented boundary.
- Do not move SPC outlook content out of `docs/roadmap/spc.md` until it is implemented.
Examples needing work:
- There is no `examples/` directory.
- `cmd/weatherfeeder/config.yml` is load-tested by `cmd/weatherfeeder/main_test.go`, but it is not an ideal copyable public example because it includes local/operator-specific values and a commented API key-like OpenWeather URL.
- Future examples should be checked with `feedkit/config.Load` or a weatherfeeder-specific test.
Links needing verification:
- Links from `README.md` to `API.md` should be changed to the new canonical event contract path if `API.md` is moved.
- `docs/policy/architecture.md` links to `../../API.md`; update this after moving the contract.
- Internal package comments referring to `internal/standards` need correction to `standards`.
## Target Documentation Set
### `README.md`
- Audience: users, administrators, operators.
- Purpose: orient readers and get them to the shortest useful run path.
- Canonical scope: project purpose, elevator pitch, shortest useful command, and links to deeper docs.
- Recommended outline: description; what it does; shortest useful command; emitted products and providers summary; links to config, CLI, operations, event contract, development policy.
- Source of truth: `cmd/weatherfeeder/main.go`, `cmd/weatherfeeder/config.yml`, `internal/sources/builtins.go`, `internal/normalizers/builtins.go`, `standards/schema.go`, `README.md` stale claims.
- Acceptance criteria: concise, no stale stdout-only claim, no full config reference, no development-history framing, links target canonical docs.
### `docs/cli.md`
- Audience: users, administrators, operators.
- Purpose: document the executable interface.
- Canonical scope: how to run `weatherfeeder` and what CLI flags exist.
- Recommended outline: shortest useful command; command overview; complete flag reference stating there are currently no flags; working-directory requirement for `config.yml`; common local/container workflows; shutdown signal behavior.
- Source of truth: `cmd/weatherfeeder/main.go`, `Dockerfile`, `cmd/weatherfeeder/main_test.go`.
- Acceptance criteria: accurately states that `config.yml` is loaded from current working directory and no config path flag exists; does not invent flags.
### `docs/config.md`
- Audience: administrators, operators, advanced users.
- Purpose: canonical configuration reference.
- Canonical scope: YAML shape, field rules, source/sink/route definitions, driver params, and defaults.
- Recommended outline: config file location; minimal config; production-oriented config; top-level reference; source drivers; source params; sink drivers; sink params; routes; duration formats; secrets handling; links to examples.
- Source of truth: `cmd/weatherfeeder/main.go`, `cmd/weatherfeeder/config.yml`, `../feedkit/config/config.go`, `../feedkit/config/load.go`, `../feedkit/sources/http.go`, `../feedkit/sinks/*.go`, `internal/sources/builtins.go`, source constructors.
- Acceptance criteria: documents strict YAML known-field behavior, required `sources` and `sinks`, optional `routes`, `mode`, `every`, `kinds`, HTTP params including `url`, `user_agent`, `conditional`, `http_timeout`, and `http_response_body_limit_bytes`; documents OpenWeather `units=metric` requirement; avoids embedding secrets.
### `docs/operations.md`
- Audience: administrators and operators.
- Purpose: explain how to run, observe, shut down, and recover the daemon.
- Canonical scope: runtime behavior, logs, scheduling, sink behavior, Postgres initialization/pruning, and operational caveats.
- Recommended outline: normal workflow; runtime lifecycle; logs; scheduler and polling behavior; conditional HTTP fetches; routing and sink fanout; Postgres table creation and retention; shutdown; recovery; caveats.
- Source of truth: `cmd/weatherfeeder/main.go`, `../feedkit/scheduler`, `../feedkit/dispatch`, `../feedkit/sources/http.go`, `../feedkit/sinks/postgres.go`, `internal/sinks/postgres/schema.go`, `Dockerfile`.
- Acceptance criteria: describes implemented behavior only; no unsupported admin commands; clearly states durable state is external sink state.
### `docs/troubleshooting.md`
- Audience: administrators and operators.
- Purpose: provide safe diagnosis for common implemented failure modes.
- Canonical scope: symptoms, likely causes, checks, fixes, and links.
- Recommended outline: missing `config.yml`; YAML parse or unknown-field errors; unknown source/sink driver; source kind mismatch; missing `params.url` or `params.user_agent`; OpenWeather missing `units=metric`; NATS connection failure; Postgres connection/table/credential failure; no events due to 304 unchanged responses; route sends no events.
- Source of truth: constructors and validation in feedkit/weatherfeeder, `cmd/weatherfeeder/main.go`, source tests, sink tests.
- Acceptance criteria: every entry has symptom, likely cause, diagnostic step, safe fix, and link to config/operations where useful.
### `docs/policy/development.md`
- Audience: developers and LLM coding agents.
- Purpose: contributor workflow and safe-change guidance.
- Canonical scope: repository layout, build/test commands, coding conventions, dependency policy, adding drivers/normalizers/schemas/config fields/docs.
- Recommended outline: layout; Go/test commands; style; dependency policy; adding a source; adding a normalizer; adding a canonical model/schema; adding Postgres mapping; updating sample config/examples/docs; review checklist.
- Source of truth: `docs/policy/architecture.md`, `go.mod`, `cmd/weatherfeeder/main_test.go`, package docs, builtins registries, tests.
- Acceptance criteria: no user/operator reference duplication; links to config/CLI docs; gives enough steps for coding agents to preserve architecture.
### `docs/internal/runtime.md`
- Audience: developers and LLM coding agents.
- Purpose: describe implemented runtime composition.
- Canonical scope: config load, registries, scheduler, processor chain, dispatcher, sink fanout, shutdown.
- Recommended outline: purpose; inputs/outputs; boundaries; config fields used; adapters used; state; failure behavior; tests; invariants.
- Source of truth: `cmd/weatherfeeder/main.go`, `cmd/weatherfeeder/main_test.go`, `../feedkit/scheduler`, `../feedkit/dispatch`, `../feedkit/processors`.
- Acceptance criteria: accurately describes normalize then dedupe, `dedupeMaxEntries = 2048`, event channel buffer 256, and context shutdown behavior.
### `docs/internal/sources.md`
- Audience: developers and LLM coding agents.
- Purpose: document implemented source drivers and source contracts.
- Canonical scope: source boundaries, registered drivers, kinds, raw schemas, params, effective-time policies at a summary level.
- Recommended outline: source contract; common HTTP behavior; driver table; provider notes; failure behavior; tests; invariants.
- Source of truth: `internal/sources/builtins.go`, `internal/sources/*`, `../feedkit/sources/http.go`, source tests, `standards/schema.go`.
- Acceptance criteria: includes all current drivers and no legacy `nws_forecast`; documents raw payload preservation and conditional GET behavior without duplicating full config reference.
### `docs/internal/normalizers.md`
- Audience: developers and LLM coding agents.
- Purpose: document raw-to-canonical mapping boundaries and registration.
- Canonical scope: normalizer contract, schema matching, provider packages, output schemas, tests.
- Recommended outline: contract; registration/order; mapping table from raw schemas to canonical schemas; common helpers; failure behavior; tests; invariants.
- Source of truth: `internal/normalizers/doc.go`, `internal/normalizers/builtins.go`, provider normalizer packages, `standards/schema.go`, normalizer tests.
- Acceptance criteria: fixes stale `internal/standards` references; avoids duplicating every payload field from event contract.
### `docs/internal/postgres-sink.md`
- Audience: developers and LLM coding agents.
- Purpose: explain weatherfeeder-owned Postgres mapping internals.
- Canonical scope: schema registration, mapper behavior, required-field validation, table contract link.
- Recommended outline: purpose; input schemas; writes; parent/child tables; nullable rules; pruning columns; tests; invariants.
- Source of truth: `internal/sinks/postgres/doc.go`, `schema.go`, `map.go`, sink tests, feedkit Postgres sink.
- Acceptance criteria: links to `docs/integrations/postgres.md` for operator-facing table contract; does not duplicate full table definitions unless that integration doc remains intentionally concise.
### `docs/integrations/events.md`
- Audience: downstream consumers and developers.
- Purpose: canonical emitted event wire contract.
- Canonical scope: feedkit event envelope JSON and canonical weather payload schemas.
- Recommended outline: envelope; timestamp rules; kinds and schemas; raw schema note; shared conventions; canonical payloads; compatibility rules; compact examples.
- Source of truth: `../feedkit/event/event.go`, `standards/schema.go`, `model/*.go`, `internal/normalizers/common/round.go`, normalizer tests.
- Acceptance criteria: documents actual JSON field names (`id`, `kind`, `source`, `emitted_at`, `effective_at`, `schema`, `payload`); includes all current raw and canonical schemas; states forecast period `conditionCode` is optional; examples validate against current structs.
### `docs/integrations/postgres.md`
- Audience: downstream SQL consumers, administrators, developers.
- Purpose: canonical Postgres table contract for weatherfeeder writes.
- Canonical scope: tables, columns, keys, indexes, pruning, reconstruction notes, migration caveats.
- Recommended outline: scope; sink config link; initialization behavior; table overview; full table contract; pruning; migrations; reconstruction of canonical payloads.
- Source of truth: `internal/sinks/postgres/doc.go`, `schema.go`, `map.go`, `../feedkit/sinks/postgres.go`, schema/map tests.
- Acceptance criteria: matches generated schema exactly; notes `CREATE TABLE IF NOT EXISTS` does not alter existing schemas; documents `forecast_periods.condition_code` nullable.
### `docs/integrations/nws.md`
- Audience: developers and operators maintaining NWS integrations.
- Purpose: concise notes for implemented NWS endpoint usage.
- Canonical scope: only endpoints and fields weatherfeeder currently uses.
- Recommended outline: supported drivers; endpoint shapes; required user agent; accept headers; effective-time policies; parser caveats; tests.
- Source of truth: `internal/sources/nws`, `internal/normalizers/nws`, `internal/providers/nws`, NWS tests and fixtures.
- Acceptance criteria: no undocumented SPC outlook content; weather stories and forecast discussions included as implemented.
### `docs/integrations/openmeteo.md`
- Audience: developers and operators maintaining Open-Meteo integrations.
- Purpose: concise notes for implemented Open-Meteo observation and hourly forecast usage.
- Canonical scope: current endpoint usage, time parsing, fields mapped, effective time.
- Source of truth: `internal/sources/openmeteo`, `internal/normalizers/openmeteo`, `internal/providers/openmeteo`, tests.
- Acceptance criteria: no daily forecast or unsupported products documented as current.
### `docs/integrations/openweather.md`
- Audience: developers and operators maintaining OpenWeather integration.
- Purpose: concise notes for implemented OpenWeather observation usage.
- Canonical scope: current observation driver, `units=metric` requirement, timestamp behavior, mapped fields.
- Source of truth: `internal/sources/openweather`, `internal/normalizers/openweather`, `internal/providers/openweather`, tests.
- Acceptance criteria: no forecast/alert OpenWeather support documented as current.
### `examples/config.minimal.yml`
- Audience: users and operators.
- Purpose: copyable minimal config.
- Canonical scope: one source, one stdout sink, one route.
- Source of truth: `cmd/weatherfeeder/config.yml`, config tests, source constructors.
- Acceptance criteria: loads with `config.Load`; no secrets; uses placeholder-safe or public endpoints as appropriate.
### `examples/config.nats.yml`
- Audience: operators.
- Purpose: copyable NATS publishing config.
- Canonical scope: selected sources, NATS sink, route examples.
- Source of truth: sample config, feedkit NATS sink.
- Acceptance criteria: loads with `config.Load`; no secrets; clearly uses replaceable NATS URL/subject.
### `examples/config.postgres.yml`
- Audience: operators.
- Purpose: copyable Postgres persistence config.
- Canonical scope: selected sources, Postgres sink params, pruning example, routes.
- Source of truth: sample config, feedkit Postgres sink, weatherfeeder Postgres schema.
- Acceptance criteria: loads with `config.Load`; uses placeholders for username/password/URI; links from config and operations docs.
## File-by-File Rewrite Guidance
### `README.md`
Cover what the daemon does, the shortest useful command, and where to go next. Use implemented provider and product summaries from source/normalizer registries. Avoid full config tables, full wire schemas, history, “MVP” language, and stale sink limitations. Link to `docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/integrations/events.md`, and `docs/policy/development.md`.
Do not carry forward the claim that stdout is the only implemented sink.
### `API.md`
Move its canonical content to `docs/integrations/events.md` and either delete `API.md` or replace it with a short pointer only if backward-compatible repository links are required. The rewritten contract must inspect `../feedkit/event/event.go` before documenting envelope fields. Do not preserve camelCase envelope examples unless the code changes first.
### `docs/config.md`
Use feedkit config structs for generic fields and weatherfeeder source/sink constructors for driver-specific params. Link to examples instead of embedding all large configs. Avoid documenting feedkit stream-only params as actively used by weatherfeeder unless clearly marked as generic feedkit config accepted by the current config model; all current weatherfeeder sources are poll sources.
### `docs/cli.md`
State the current no-flags behavior plainly. Include that `weatherfeeder` expects `config.yml` in the current working directory. Do not document environment variables, subcommands, or alternate config paths unless implemented.
### `docs/operations.md`
Focus on running the daemon and operating sinks. Include signal shutdown, log behavior, conditional HTTP fetches, Postgres table creation, retention pruning, and no internal durable scheduler state. Avoid unsupported recovery commands.
### `docs/troubleshooting.md`
Base entries on actual validation and constructor errors. Link to `docs/config.md` for fixes. Avoid speculative upstream outages beyond generic HTTP/source errors unless tests or code reveal specific behavior.
### `docs/policy/development.md`
Use `docs/policy/architecture.md` as the authority. Give concrete safe-change steps for adding source drivers, normalizers, schemas, Postgres tables, config examples, and docs. Avoid repeating the architecture policy wholesale.
### `docs/internal/*.md`
Create concise component docs. They should guide changes, not become manuals. Each should include tests to inspect before changing. Link to package docs when those are more precise.
### `docs/integrations/*.md`
Document only external contracts actually used by the implementation. Keep provider docs narrow and maintenance-oriented. Do not describe unimplemented SPC support outside `docs/roadmap/spc.md`.
### Package comments
Fix stale path and future-language comments during the documentation migration stage that touches developer docs. Specific likely fixes:
- `model/doc.go`: remove “stdout today; others later.”
- `standards/doc.go`: correct `internal/standards/doc.go` path.
- `internal/normalizers/doc.go`: correct `internal/standards/schema.go` path.
- `internal/providers/openweather/doc.go`: remove “forecasts/alerts later.”
## Examples Plan
No `examples/` directory currently exists. Create one because the project has non-trivial configuration and operator workflows.
Recommended examples:
- `examples/config.minimal.yml`: one public HTTP source, stdout sink, route. Validity check: `config.Load` in a new or existing config/example test. Link from README, config, CLI.
- `examples/config.nats.yml`: NATS sink publishing example. Validity check: `config.Load`; do not require a live NATS server. Link from config and operations.
- `examples/config.postgres.yml`: Postgres sink with placeholder credentials and optional `prune`. Validity check: `config.Load`; do not require a live Postgres server. Link from config, operations, Postgres integration docs.
- `examples/config.full.yml`: optional, only if the team wants one annotated config covering all implemented drivers. Validity check: `config.Load`; must use placeholders for secrets and avoid real OpenWeather API keys. Link from config docs only.
Do not add SPC examples until SPC support is implemented.
## Internal Documentation Plan
### Runtime Pipeline
- Path: `docs/internal/runtime.md`
- Purpose: explain daemon composition.
- Inputs and outputs: YAML config in; feed events through scheduler/pipeline/dispatcher to sinks out.
- Boundaries: composition belongs in `cmd/weatherfeeder`; domain mapping belongs in normalizers/model.
- Config fields used: sources, sinks, routes, source `every`, mode/kinds, sink params.
- Adapters used: feedkit scheduler, processors, dispatch, sinks.
- Failure behavior: fatal startup errors; runtime errors logged and cancel context; interrupt/SIGTERM shutdown.
- Tests to inspect: `cmd/weatherfeeder/main_test.go`, feedkit scheduler/dispatch tests.
- Invariants: normalize before dedupe; bounded event channel; context-aware shutdown.
### Sources
- Path: `docs/internal/sources.md`
- Purpose: document source driver architecture.
- Inputs and outputs: `config.SourceConfig` and upstream HTTP responses in; raw feed events out.
- Boundaries: sources fetch and envelope raw payloads; normalizers decode full payloads.
- Config fields used: name, driver, mode, every, kinds, params URL/user agent/HTTP options.
- Adapters used: feedkit source registry and HTTP helper.
- Failure behavior: constructor validation, HTTP errors, unchanged 304 returns no events, some malformed metadata still emits raw events.
- Tests to inspect: `internal/sources/builtins_test.go`, provider source tests.
- Invariants: drivers registered in one place; raw schemas from `standards`; no canonical mapping in sources.
### Normalizers
- Path: `docs/internal/normalizers.md`
- Purpose: document raw-to-canonical mapping architecture.
- Inputs and outputs: raw events matched by schema in; canonical model payload events out.
- Boundaries: no HTTP, no sink persistence, no CLI/config behavior.
- Config fields used: none directly.
- Adapters used: feedkit normalize processor.
- Failure behavior: decode/parse/missing required fields return contextual errors; no-match passthrough configured in main.
- Tests to inspect: `internal/normalizers/builtins_test.go`, provider normalizer tests, `internal/normalizers/common` tests.
- Invariants: schema matching only, stable registration order, canonical schema constants.
### Postgres Sink Mapping
- Path: `docs/internal/postgres-sink.md`
- Purpose: document weatherfeeder-specific Postgres mapping internals.
- Inputs and outputs: canonical events in; feedkit `PostgresWrite` rows out.
- Boundaries: mapper validates and maps; feedkit owns DB connection, DDL, transactions, pruning.
- Config fields used: sink `uri`, `username`, `password`, optional `prune` via feedkit.
- Adapters used: feedkit Postgres sink.
- Failure behavior: unsupported schemas map to no writes or errors per mapper behavior; required missing fields fail before write.
- Tests to inspect: `internal/sinks/postgres/map_test.go`, `schema_test.go`, feedkit Postgres tests.
- Invariants: consume canonical schemas only; preserve envelope columns; keep parent/child order indexes.
## Integration Documentation Plan
### Event Wire Contract
- Path: `docs/integrations/events.md`
- External system or contract: JSON events emitted to stdout/NATS and represented in Postgres parent envelope columns.
- Current usage: sinks marshal feedkit events or map canonical events to tables.
- Version notes: schema identifiers use `raw.*.v1` and `weather.*.v1`; JSON tags are compatibility contract.
- Document: actual envelope fields, schema list, payload fields, units, optionality, examples.
- Do not document: weatherapi HTTP endpoints, unimplemented outlooks, or future schemas.
### Postgres
- Path: `docs/integrations/postgres.md`
- External system or contract: PostgreSQL schema created/written by feedkit Postgres sink using weatherfeeder schema definition.
- Current usage: optional configured sink; create-if-missing tables and indexes; transactional writes; optional pruning.
- Version notes: no migration framework is implemented; existing DBs may need manual schema changes when table definitions change.
- Document: params, table contract, pruning, reconstruction, migration caveats.
- Do not document: weatherapi query behavior except as a downstream consumer link if needed.
### NWS
- Path: `docs/integrations/nws.md`
- External system or contract: NWS API and NWS forecast discussion/weather story endpoints used by current drivers.
- Current usage: observations, alerts, hourly forecast, narrative forecast, forecast discussion, weather stories.
- Version notes: no explicit upstream API version in code; weatherfeeder pins its own raw schema names.
- Document: endpoint shapes used, required User-Agent, accept headers, effective time, raw schemas.
- Do not document: SPC outlooks or unsupported NWS products.
### Open-Meteo
- Path: `docs/integrations/openmeteo.md`
- External system or contract: Open-Meteo forecast API for current observations and hourly forecasts.
- Current usage: JSON HTTP source, time parsing from timezone/UTC offset, hourly fields mapped to canonical forecast.
- Version notes: no explicit upstream API version in code.
- Document: required URL/user agent params, expected current/hourly response pieces, effective time behavior.
- Do not document: daily forecast unless implemented.
### OpenWeather
- Path: `docs/integrations/openweather.md`
- External system or contract: OpenWeather current weather endpoint.
- Current usage: observation source/normalizer only.
- Version notes: source requires `units=metric` in URL.
- Document: metric units requirement, timestamp behavior, mapped fields, secret handling for API key.
- Do not document: OpenWeather forecasts or alerts.
### NATS
- Path: include in `docs/operations.md` and `docs/config.md`; create `docs/integrations/nats.md` only if NATS-specific maintenance grows.
- External system or contract: NATS publish subject configured by feedkit sink.
- Current usage: optional sink publishes each event as JSON.
- Document: `url`, `subject`, JSON event payload, connection failure behavior.
- Do not document: subscriptions or server administration beyond this sink contract.
## Recommended Implementation Sequence
### Stage 1: Current-State README, CLI, and Config Docs
- Goal: establish accurate user/operator entry points.
- Files to create/update/delete/move: rewrite `README.md`; create `docs/cli.md`; create `docs/config.md`.
- Repository areas to inspect: `cmd/weatherfeeder/main.go`, `cmd/weatherfeeder/config.yml`, `../feedkit/config`, `../feedkit/sources/http.go`, `../feedkit/sinks`, `internal/sources/builtins.go`, source constructors.
- Acceptance criteria: README concise and current; CLI doc states no flags and cwd `config.yml`; config doc includes complete implemented config reference and no secrets.
- Suggested validation commands: `go test ./cmd/weatherfeeder`; `rg -n "only implemented sink|current MVP|--config|API.md" README.md docs/cli.md docs/config.md`.
- Size: small enough for one implementation prompt.
### Stage 2: Event and Postgres Integration Contracts
- Goal: move and correct external contracts.
- Files to create/update/delete/move: create `docs/integrations/events.md`; create `docs/integrations/postgres.md`; delete or replace `API.md` with a pointer; update links in `README.md` and `docs/policy/architecture.md`.
- Repository areas to inspect: `../feedkit/event/event.go`, `standards/schema.go`, `model/*.go`, `internal/sinks/postgres/doc.go`, `schema.go`, `map.go`, normalizer tests, sink tests.
- Acceptance criteria: event envelope field names match code; all current schemas listed; Postgres contract matches schema definition; no duplicate or stale API rows.
- Suggested validation commands: `go test ./internal/sinks/postgres ./internal/normalizers/...`; `rg -n "effectiveAt|emittedAt|API.md|response \| string.*response" README.md docs API.md`.
- Size: one implementation prompt if kept focused; split if examples are expanded heavily.
### Stage 3: Operations and Troubleshooting Docs
- Goal: document running and recovering the daemon.
- Files to create/update/delete/move: create `docs/operations.md`; create `docs/troubleshooting.md`.
- Repository areas to inspect: `cmd/weatherfeeder/main.go`, `Dockerfile`, feedkit scheduler/dispatch/source/sink implementations, config validation, source/sink constructors.
- Acceptance criteria: operational docs describe logs, shutdown, scheduling, conditional HTTP, fanout, Postgres create/prune behavior, and common failures without inventing commands.
- Suggested validation commands: `go test ./cmd/weatherfeeder`; `rg -n "TODO|future|planned|unimplemented" docs/operations.md docs/troubleshooting.md`.
- Size: small enough for one implementation prompt.
### Stage 4: Examples Directory and Example Validation
- Goal: add maintained copyable configs.
- Files to create/update/delete/move: create `examples/config.minimal.yml`, `examples/config.nats.yml`, `examples/config.postgres.yml`; optionally create `examples/config.full.yml`; update tests to load examples; update links in README/config/operations.
- Repository areas to inspect: `cmd/weatherfeeder/config.yml`, `cmd/weatherfeeder/main_test.go`, feedkit config validation, source/sink params.
- Acceptance criteria: examples contain no secrets, load successfully, and are linked from canonical docs.
- Suggested validation commands: `go test ./cmd/weatherfeeder`; `rg -n "appid=|password: [^<]|token|secret" examples cmd/weatherfeeder/config.yml docs`.
- Size: one implementation prompt.
### Stage 5: Developer and Internal Docs
- Goal: document safe change workflow and internal component boundaries.
- Files to create/update/delete/move: create `docs/policy/development.md`; create `docs/internal/runtime.md`; create `docs/internal/sources.md`; create `docs/internal/normalizers.md`; create `docs/internal/postgres-sink.md`; lightly update package comments with stale paths/future language.
- Repository areas to inspect: architecture policy, package docs, registries, source/normalizer/sink tests, `go.mod`.
- Acceptance criteria: docs are inward-facing, concise, and action-oriented; package comments no longer contain stale path or future-support language outside roadmap.
- Suggested validation commands: `go test ./...`; `rg -n "internal/standards|others later|forecasts/alerts later|future expensive steps" . -g '*.go' -g '*.md'`.
- Size: likely one implementation prompt, but split package-comment cleanup if code comment edits are considered too broad.
### Stage 6: Provider Integration Notes
- Goal: document external provider contracts actually used by implemented source/normalizer pairs.
- Files to create/update/delete/move: create `docs/integrations/nws.md`, `docs/integrations/openmeteo.md`, `docs/integrations/openweather.md`; optionally create `docs/integrations/nats.md` only if NATS content should stand alone.
- Repository areas to inspect: provider source packages, provider normalizer packages, `internal/providers`, provider tests/fixtures.
- Acceptance criteria: each integration doc is narrow, current, and excludes unimplemented products; NWS doc does not include SPC outlooks except linking to roadmap if needed.
- Suggested validation commands: `go test ./internal/sources/... ./internal/normalizers/... ./internal/providers/...`; `rg -n "SPC|outlook|daily forecast|OpenWeather forecast|OpenWeather alert" docs/integrations`.
- Size: one implementation prompt.
### Stage 7: Final Documentation Consistency Pass
- Goal: remove stale references and verify policy compliance.
- Files to create/update/delete/move: any docs touched in prior stages; remove obsolete root `API.md` if not kept as pointer.
- Repository areas to inspect: whole documentation tree, package comments, tests.
- Acceptance criteria: every non-roadmap doc describes implemented behavior only; every doc has clear audience/scope; roadmap content remains only in `docs/roadmap`; links resolve by inspection.
- Suggested validation commands: `go test ./...`; `find docs -type f -name '*.md' -print | sort`; `rg -n "MVP|only implemented sink|planned|future|later|deprecated|experimental|TODO|API.md|effectiveAt|emittedAt|internal/standards|appid=" README.md docs internal model standards cmd examples`.
- Size: small enough for one implementation prompt.
## Validation Plan
Automated checks available now:
- `go test ./...` verifies code behavior and existing load-tested sample config.
- `go test ./cmd/weatherfeeder` verifies `cmd/weatherfeeder/config.yml` loads and sources build scheduler jobs.
- Focused package tests verify source drivers, normalizers, and Postgres schema/mapping.
Recommended new checks during implementation:
- Add tests that load all `examples/*.yml` with `feedkit/config.Load`.
- Add or extend tests to ensure documented source drivers in config examples build through `internal/sources.RegisterBuiltins`.
- Use `rg` checks for stale or prohibited terms: `only implemented sink`, `current MVP`, `future`, `later`, `planned`, `deprecated`, `experimental`, `API.md`, `effectiveAt`, `emittedAt`, `internal/standards`, and secret-like strings such as `appid=`.
- Manually verify Markdown links because no dedicated Markdown/link checker is currently present.
- Manually compare `docs/integrations/events.md` against `../feedkit/event/event.go`, `standards/schema.go`, and `model/*.go` before accepting.
- Manually compare `docs/integrations/postgres.md` against `internal/sinks/postgres/schema.go` before accepting.
No repository-local Markdown formatter, Markdown linter, Makefile, justfile, Taskfile, or `package.json` docs tooling was found during this planning pass.
## Open Questions
No question blocks implementation of this documentation migration. Recommended decisions:
- Move the canonical event contract from `API.md` to `docs/integrations/events.md` to comply with the documentation policy.
- Replace root `API.md` with a short pointer only if backward-compatible links are considered important; otherwise delete it during the migration.
- Keep `cmd/weatherfeeder/config.yml` as the executable in-repo sample used by tests, but create public copyable examples under `examples/` and remove secret-like values from examples.

323
docs/roadmap/spc.md Normal file
View File

@@ -0,0 +1,323 @@
# SPC Convective Outlook Support Roadmap
## Summary
Add `weatherfeeder` support for Storm Prediction Center convective outlooks as a new canonical outlook domain. The feature should poll SPC GeoJSON outlook products, optionally enrich them with RSS discussion metadata, compute whether the configured forecast point is inside each active outlook polygon, normalize the result into a provider-independent schema, and persist it through existing sinks.
This is a new domain, not an extension of `weather.alert.v1`. SPC outlooks describe probabilistic/categorical risk areas over a validity window; alerts describe active actionable hazard messages.
## Public Contract
Add schema constants:
- Raw schema: `raw.spc.convective_outlook.v1`
- Canonical schema: `weather.outlook.v1`
Add source driver:
- `spc_convective_outlook`
Add event kind:
- `outlook`
Add canonical model types:
- `model.WeatherOutlookRun`
- `model.WeatherOutlook`
Recommended canonical run fields:
- `locationId`, `locationName`
- `latitude`, `longitude`
- `asOf`
- `issuedAt`
- `outlooks`
Recommended canonical outlook fields:
- `id`
- `provider`
- `product`
- `day`
- `outlookType`
- `label`
- `labelText`
- `severityRank`
- `validFrom`
- `validTo`
- `issuedAt`
- `expiresAt`
- `forecaster`
- `headline`
- `summary`
- `discussion`
- `sourceUrl`
- `imageUrl`
- `containsLocation`
- `geometry`
Contract defaults:
- `product` should be `convective`.
- `outlookType` should be one of `categorical`, `tornado`, `hail`, `wind`.
- `day` should be `1`, `2`, or `3` for this first implementation.
- `containsLocation` is computed against configured forecast coordinates.
- `geometry` should preserve compact GeoJSON geometry for auditability and future API use.
## Source Scope
The source should fetch a bundle of SPC products in one poll cycle and emit one raw event containing the fetched RSS metadata, GeoJSON products, configured point, and per-product fetch metadata.
Poll these GeoJSON URLs:
- `https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson`
Note: the initial candidate list duplicated Day 3 wind and omitted Day 2 wind. Use the corrected 12-product matrix above.
Also poll:
- `https://www.spc.noaa.gov/products/spcacrss.xml`
RSS usage:
- Use RSS as discussion/discovery metadata, not as the geometry source.
- Preserve item title, link, description text, pubDate, and guid where useful.
- Extract the narrative outlook text from the RSS item description when practical.
- Do not attempt to parse polygons from RSS HTML.
Polling cadence:
- Use a default cadence of `30m`, not daily/twice-daily. SPC current outlook files can update several times per day, and a 30-minute poll with HTTP caching is low cost and less likely to miss updates.
- Rely on ETag/Last-Modified handling from the HTTP source layer where available.
- Keep the cadence configurable via `every`.
Recommended config shape:
```yaml
- name: SPCConvectiveOutlookSTL
mode: poll
kinds: ["outlook"]
driver: spc_convective_outlook
every: 30m
params:
latitude: 38.6239
longitude: -90.3571
location_id: "stl"
location_name: "St. Louis, MO"
user_agent: "HomeOps (eric@maximumdirect.net)"
```
The source should own default SPC URLs, with optional params to override the RSS URL and product URLs for tests or future provider changes.
## Implementation Stages
### Stage 1: Raw Source and Schema
- Add schema constants and model placeholders.
- Add `internal/sources/spc` package.
- Implement `spc_convective_outlook` as a poll source.
- Fetch all configured GeoJSON products and RSS during a poll cycle.
- Emit one raw event with kind `outlook` and schema `raw.spc.convective_outlook.v1`.
- Use the latest valid `ISSUE_ISO`, RSS `lastBuildDate`, or fetch time for `effectiveAt`, in that order.
- Preserve partial fetch errors as source errors for the first implementation; do not emit incomplete outlook bundles unless a later explicit degraded-mode policy is added.
Tests:
- source driver builds as a `PollSource`
- source advertises kind `outlook`
- source emits one raw bundled event
- source chooses expected effective time
- source uses configured location metadata
- source fails clearly on missing latitude/longitude
### Stage 2: GeoJSON and Spatial Matching
- Add an internal geometry helper package, for example `internal/geo`.
- Support GeoJSON `Polygon` and `MultiPolygon`.
- Implement point-in-polygon with ring support:
- first ring is the exterior
- subsequent rings are holes
- boundary counts as inside
- GeoJSON coordinate order is `[longitude, latitude]`
- Use planar ray casting for this operational point-in-polygon check.
Tests:
- point inside polygon
- point outside polygon
- point on polygon boundary
- point inside a hole is outside
- point inside one multipolygon member is inside
- longitude/latitude order regression test
### Stage 3: Normalizer and Canonical Mapping
- Add `SPCConvectiveOutlookNormalizer`.
- Match only `raw.spc.convective_outlook.v1`.
- Decode the raw source bundle.
- Map each GeoJSON feature to one `WeatherOutlook`.
- Preserve SPC feature order within each product, then order products by day and type.
- Map properties:
- `VALID_ISO` -> `validFrom`
- `EXPIRE_ISO` -> `validTo` and `expiresAt`
- `ISSUE_ISO` -> `issuedAt`
- `FORECASTER` -> `forecaster`
- `LABEL` -> `label`
- `LABEL2` -> `labelText`
- `DN` -> `severityRank`
- Derive:
- `day` from product key or URL
- `outlookType` from product key or URL
- `id` from day, type, label, issuedAt, validFrom, and feature index
- `containsLocation` from configured point and GeoJSON geometry
- Enrich narrative fields from RSS where a matching day/product item can be determined.
- Set run `asOf` to the latest valid `issuedAt` across features, falling back to RSS `lastBuildDate`, then input event time.
- Set run `issuedAt` to the latest valid `issuedAt` across features.
- Set output event `effectiveAt` to run `asOf`.
Tests:
- normalizer routes only raw SPC schema
- categorical and probabilistic products map expected fields
- `containsLocation` is true for a known point inside a fixture polygon
- `containsLocation` is false outside
- missing optional RSS still permits GeoJSON normalization
- malformed required GeoJSON timestamps fail with useful context
- canonical JSON shape does not expose raw-provider-only bundle internals
### Stage 4: Postgres Sink
Add tables:
- `outlook_runs`
- `outlooks`
Suggested `outlook_runs` columns:
- event envelope columns
- `location_id`
- `location_name`
- `latitude`
- `longitude`
- `as_of`
- `issued_at`
- `outlook_count`
Suggested `outlooks` columns:
- `run_event_id`
- `outlook_index`
- `as_of`
- `product`
- `day`
- `outlook_type`
- `label`
- `label_text`
- `severity_rank`
- `valid_from`
- `valid_to`
- `issued_at`
- `expires_at`
- `forecaster`
- `headline`
- `summary`
- `discussion`
- `source_url`
- `image_url`
- `contains_location`
- `geometry_json`
Suggested indexes:
- `outlook_runs(location_id, as_of)`
- `outlooks(contains_location, valid_from, valid_to)`
- `outlooks(day, outlook_type, label)`
- `outlooks(valid_from, valid_to)`
Mapping rules:
- Store `geometry` as compact GeoJSON text in `geometry_json`.
- Require run `asOf`.
- Require outlook `validFrom`, `validTo`, `issuedAt`, `day`, `outlookType`, and `label`.
- Preserve all outlook polygons, not only polygons containing the configured point.
Tests:
- schema includes outlook tables and indexes
- mapper writes one run row plus one row per outlook
- mapper stores compact geometry JSON
- mapper rejects missing required run/outlook fields
### Stage 5: Config and Documentation
- Update sample config with `SPCConvectiveOutlookSTL`.
- Add `outlook` to route examples.
- Update README provider capabilities.
- Update `API.md` with `weather.outlook.v1`.
- Update Postgres sink docs with outlook table contract.
Docs should clearly state:
- RSS is used for narrative/discussion metadata.
- GeoJSON is used for polygons and point matching.
- `containsLocation` is computed by weatherfeeder at ingestion time.
- Geometry is stored for downstream audit/display.
### Stage 6: Weatherapi Follow-Up
Do not include weatherapi changes in the first weatherfeeder implementation unless explicitly requested.
Likely future weatherapi endpoints:
- `GET /outlooks/convective`
- `GET /outlooks/convective/active`
- `GET /outlooks/convective/location`
Recommended API behavior:
- latest run by default
- active outlooks filtered by current time and `containsLocation=true`
- optional filter query params for `day`, `outlookType`, and `containsLocation`
## Open Questions for Implementation
- Whether to include Day 4-8 probabilistic outlooks in a later version. Exclude them from v1.
- Whether to make partial source bundles acceptable if one product URL fails. Default v1 behavior should fail the poll and retry.
- Whether to parse detailed discussion text from RSS only, HTML pages, or both. Default v1 should use RSS only.
- Whether to keep `geometry` in canonical JSON permanently. Default v1 should include it because it preserves source context and enables downstream display.
## Verification Commands
Run focused tests:
```sh
go test ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./model
```
Run full weatherfeeder tests:
```sh
go test ./...
```
## Acceptance Criteria
- A configured SPC source emits raw outlook bundles.
- Normalization produces `weather.outlook.v1` events.
- Each outlook indicates whether the configured forecast point is inside its polygon.
- All current Day 1-3 categorical/tornado/hail/wind products are represented.
- RSS discussion metadata is preserved where available.
- Postgres sink persists outlook runs and outlook rows.
- Sample config and public docs describe the new kind, driver, schema, and storage contract.

236
docs/troubleshooting.md Normal file
View File

@@ -0,0 +1,236 @@
# Troubleshooting
Use this guide with [configuration](config.md) and [operations](operations.md).
Messages are emitted through the standard logger on stderr.
## `config load failed: ... read "config.yml"`
Symptom: startup exits before building sources or sinks.
Likely cause: the process working directory does not contain `config.yml`, or
the runtime user cannot read it.
Diagnostic: run `pwd` in the same working directory used by the process, then
check `ls -l config.yml`.
Safe fix: place the intended config at `./config.yml`, change the working
directory, or mount the file at `/weatherfeeder/config.yml` when using the
provided container image.
## `config load failed: ... parse YAML`
Symptom: startup exits with a YAML parse error or an unknown field error.
Likely cause: invalid YAML syntax, multiple YAML documents, or a misspelled
config struct field.
Diagnostic: inspect the line and field in the error. Feedkit uses strict YAML
field decoding for config struct fields.
Safe fix: correct the YAML and compare the shape with
[configuration](config.md). Driver-specific `params` keys are validated by their
source or sink constructors.
## `config validation failed`
Symptom: startup exits and prints one or more validation messages.
Likely cause: missing `sources` or `sinks`, blank names, duplicate source or sink
names, invalid `mode`, missing `every` for a polling source with `mode: poll`, or
a route that references an unknown sink.
Diagnostic: read every bullet under `config validation failed`; the loader sorts
these messages so multiple issues can be fixed in one edit.
Safe fix: update the top-level config fields as documented in
[configuration](config.md).
## `unknown source driver`
Symptom: startup exits with `build source failed`.
Likely cause: `sources[].driver` does not match a registered weatherfeeder
source driver.
Diagnostic: compare the configured driver with the source driver table in
[configuration](config.md#source-drivers).
Safe fix: correct the driver name. Current drivers include `nws_observation`,
`nws_alerts`, `nws_forecast_hourly`, `nws_forecast_narrative`,
`nws_forecast_discussion`, `nws_weatherstories`, `openmeteo_observation`,
`openmeteo_forecast`, and `openweather_observation`.
## `unknown sink driver`
Symptom: startup exits with `build sink failed`.
Likely cause: `sinks[].driver` is not registered.
Diagnostic: compare the configured driver with the sink driver table in
[configuration](config.md#sink-drivers).
Safe fix: use `stdout`, `nats`, or `postgres`.
## `source expected kinds validation failed`
Symptom: startup exits after building a source.
Likely cause: `sources[].kinds` declares a kind the source does not emit.
Diagnostic: compare the configured `kinds` list with the source driver kind in
[configuration](config.md#source-drivers).
Safe fix: remove `kinds` or set it to the kind emitted by that driver.
## `params.url is required` Or `params.user_agent is required`
Symptom: startup exits with `build source failed`.
Likely cause: a source is missing required HTTP params, or the values are blank
or not strings.
Diagnostic: inspect the named source in the error and check its `params`.
Safe fix: add non-empty `url` and `user_agent` values. See
[HTTP source params](config.md#http-source-params).
## `url must include units=metric`
Symptom: startup exits for an `openweather_observation` source.
Likely cause: the OpenWeather URL omits `units=metric` or sets another unit
system.
Diagnostic: inspect the query string in `params.url`.
Safe fix: add `units=metric` to the OpenWeather current-weather URL. Keep API
keys out of committed configs.
## `source ... sources[].every must be > 0 for polling sources`
Symptom: startup exits while building scheduler jobs.
Likely cause: a current weatherfeeder polling source has no usable `every`
interval.
Diagnostic: inspect the named `sources[]` entry and check `every`.
Safe fix: set a positive duration such as `1m`, `10m`, or `1h`.
## `build sink failed ... params.url is required`
Symptom: startup exits while building a NATS sink.
Likely cause: the NATS sink is missing `params.url`, or the value is blank or
not a string.
Diagnostic: inspect the named sink in the error and check its `params`.
Safe fix: set a NATS URL such as `nats://localhost:4222`.
## `build sink failed ... params.subject is required`
Symptom: startup exits while building a NATS sink.
Likely cause: the NATS sink is missing `params.subject`, or the value is blank
or not a string.
Diagnostic: inspect the named sink in the error and check its `params`.
Safe fix: set a non-empty subject such as `weatherfeeder`.
## `dispatch: sink ... failed consuming event ... NATS sink: connect`
Symptom: the daemon starts, but NATS events are not published.
Likely cause: the NATS server URL is unreachable, the server is not accepting
connections, or the configured URL is wrong for the runtime network.
Diagnostic: from the same runtime environment, check that the host and port in
`sinks[].params.url` are reachable.
Safe fix: correct the NATS URL or restore broker connectivity. Other configured
sinks continue receiving events.
## `postgres sink ... open db`
Symptom: startup exits while building a Postgres sink.
Likely cause: the database URI, username, password, network path, or database
availability is wrong.
Diagnostic: inspect `sinks[].params.uri`, `username`, and `password`; verify
that the same runtime environment can reach the database.
Safe fix: correct the credentials or URI, restore database connectivity, then
restart the daemon.
## `postgres sink ... ensure table` Or `ensure index`
Symptom: startup exits during Postgres initialization.
Likely cause: the database user cannot create required tables or indexes, an
existing object conflicts with weatherfeeder's expected table contract, or the
database is unavailable during initialization.
Diagnostic: inspect the named table or index in the error and compare existing
database objects with the [Postgres table contract](integrations/postgres.md).
Safe fix: grant the needed database privileges, create a compatible schema, or
perform an operator-managed migration before restarting.
## `postgres sink: insert into ...`
Symptom: the daemon starts, but Postgres writes for some events fail.
Likely cause: a duplicate primary key, incompatible existing table definition,
database constraint error, or connection failure during a write transaction.
Diagnostic: inspect the table name and database error in the log. Compare the
table with [Postgres integration](integrations/postgres.md).
Safe fix: repair the database schema or address the duplicate/connection issue.
Other configured sinks continue receiving events.
## No Events Appear On A Sink
Symptom: the daemon is running but the expected sink receives no events.
Likely cause: the route does not match the event kind, the source has not
emitted changed content, or the sink is failing per event.
Diagnostic: check `routes`, source `kinds`, and logs for `scheduler: poll
failed`, `dispatcher: pipeline error`, or `dispatch: sink ... failed consuming
event`.
Safe fix: correct the route or source configuration. If the source uses
conditional HTTP and the upstream has not changed, no event is emitted for a
`304 Not Modified` response; wait for changed upstream content or temporarily
set `params.conditional: false` for diagnosis.
## `scheduler: poll failed`
Symptom: one source logs poll failures while the daemon keeps running.
Likely cause: upstream HTTP error, bad URL, timeout, response body limit, or
provider response shape that the source cannot parse.
Diagnostic: inspect the source name in the log and review its HTTP params.
Safe fix: correct the URL, user agent, timeout, or body limit. The next
scheduled poll will retry.
## `dispatcher: pipeline error`
Symptom: source polling succeeds, but one event is dropped before sinks.
Likely cause: a normalizer could not decode or map the raw payload, or dedupe
received an invalid event ID.
Diagnostic: inspect the error text and the source/schema that produced the
event. Review the [event wire contract](integrations/events.md) for expected
canonical fields.
Safe fix: correct source configuration if it points to the wrong upstream
product. If the upstream payload changed shape, update the relevant normalizer
and tests.

View File

@@ -0,0 +1,19 @@
---
sources:
- name: NWSObservationKSTL
mode: poll
kinds: ["observation"]
driver: nws_observation
every: 10m
params:
url: "https://api.weather.gov/stations/KSTL/observations/latest"
user_agent: "weatherfeeder example (operator@example.com)"
sinks:
- name: stdout
driver: stdout
params: {}
routes:
- sink: stdout
kinds: ["observation"]

30
examples/config.nats.yml Normal file
View File

@@ -0,0 +1,30 @@
---
sources:
- name: NWSObservationKSTL
mode: poll
kinds: ["observation"]
driver: nws_observation
every: 10m
params:
url: "https://api.weather.gov/stations/KSTL/observations/latest"
user_agent: "weatherfeeder example (operator@example.com)"
- name: NWSAlertsSTL
mode: poll
kinds: ["alert"]
driver: nws_alerts
every: 1m
params:
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
user_agent: "weatherfeeder example (operator@example.com)"
sinks:
- name: nats_weather
driver: nats
params:
url: nats://localhost:4222
subject: weatherfeeder.events
routes:
- sink: nats_weather
kinds: ["observation", "alert"]

View File

@@ -0,0 +1,32 @@
---
sources:
- name: NWSObservationKSTL
mode: poll
kinds: ["observation"]
driver: nws_observation
every: 10m
params:
url: "https://api.weather.gov/stations/KSTL/observations/latest"
user_agent: "weatherfeeder example (operator@example.com)"
- name: OpenMeteoHourlyForecastSTL
mode: poll
kinds: ["forecast"]
driver: openmeteo_forecast
every: 1h
params:
url: "https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&hourly=temperature_2m,relative_humidity_2m,dew_point_2m,apparent_temperature,precipitation_probability,precipitation,snowfall,weather_code,surface_pressure,wind_speed_10m,wind_direction_10m&forecast_days=3"
user_agent: "weatherfeeder example (operator@example.com)"
sinks:
- name: pg_weather
driver: postgres
params:
uri: "postgres://postgres.example.invalid:5432/weatherfeeder?sslmode=disable"
username: <database_username>
password: <database_password>
prune: 3d
routes:
- sink: pg_weather
kinds: ["observation", "forecast"]

View File

@@ -20,6 +20,7 @@ func TestRegisterBuiltinsOrder(t *testing.T) {
nws.ObservationNormalizer{},
nws.ForecastNormalizer{},
nws.ForecastDiscussionNormalizer{},
nws.WeatherStoriesNormalizer{},
nws.AlertsNormalizer{},
openmeteo.ObservationNormalizer{},
openmeteo.ForecastNormalizer{},

View File

@@ -9,12 +9,12 @@ import (
)
// Finalize builds the output event envelope by copying the input and applying the
// canonical schema/payload, plus (optionally) EffectiveAt.
// canonical schema/payload, plus an optional effective time.
//
// Important behavior:
// - ID/Kind/Source/EmittedAt are preserved by copying the input event.
// - EffectiveAt is only overwritten when effectiveAt is non-zero.
// If effectiveAt is zero, any existing in.EffectiveAt is preserved.
// - EffectiveAt is only overwritten when the supplied effective time is non-zero.
// If the supplied time is zero, any existing in.EffectiveAt is preserved.
// - Payload floats are rounded to a stable wire-friendly precision (see round.go).
func Finalize(in event.Event, outSchema string, outPayload any, effectiveAt time.Time) (*event.Event, error) {
// Enforce stable numeric presentation for weather payloads before delegating to feedkit's

View File

@@ -15,7 +15,7 @@ import (
// - sources emit raw JSON payloads (typically json.RawMessage)
// - normalizers decode into provider structs
//
// Errors include a small amount of stage context ("extract payload", "decode raw payload").
// Errors include a small amount of operation context ("extract payload", "decode raw payload").
// Callers typically wrap these with a provider/kind label.
func DecodeJSONPayload[T any](in event.Event) (T, error) {
return fknormalize.DecodeJSONPayload[T](in)
@@ -24,15 +24,15 @@ func DecodeJSONPayload[T any](in event.Event) (T, error) {
// NormalizeJSON is a convenience wrapper for the common JSON-normalizer pattern:
//
// 1. Decode raw JSON payload into provider struct T
// 2. Map T into canonical payload P (plus an EffectiveAt timestamp)
// 3. Finalize the event envelope (schema/payload/effectiveAt) + Validate
// 2. Map T into canonical payload P (plus an effective time)
// 3. Finalize the event envelope (schema/payload/effective time) + Validate
//
// label should be short and specific, e.g. "openweather observation".
// outSchema should be the canonical schema constant.
// build should contain ONLY provider/domain mapping logic.
//
// Error policy:
// - NormalizeJSON wraps ALL failures with consistent context: "<label> normalize: <stage>: ..."
// - NormalizeJSON wraps ALL failures with consistent context: "<label> normalize: <operation>: ..."
// - build() should return specific errors without repeating the label prefix.
func NormalizeJSON[T any, P any](
in event.Event,

View File

@@ -70,7 +70,7 @@
//
// weather.<kind>.vN
//
// weatherfeeder centralizes schema strings in internal/standards/schema.go.
// weatherfeeder centralizes schema strings in standards/schema.go.
// Always use those constants (do not inline schema strings).
//
// Example mappings:
@@ -101,8 +101,8 @@
// Every normalizer type must have a doc comment that states:
//
// - what it converts (e.g., “OpenWeather current -> WeatherObservation”)
// - which raw schema it matches (constant identifier from internal/standards)
// - which canonical schema it produces (constant identifier from internal/standards)
// - which raw schema it matches (constant identifier from standards)
// - which canonical schema it produces (constant identifier from standards)
// - any special caveats (units, day/night inference, missing fields, etc.)
//
// Including literal schema string values is optional,

View File

@@ -29,7 +29,7 @@ import (
// 2. Alert timing fields are best-effort parsed; invalid timestamps do not fail the
// entire normalization (they are left nil).
// 3. Some fields are intentionally passed through as strings (severity/urgency/etc.)
// since canonical vocabularies may evolve later.
// because the canonical model currently preserves provider vocabulary there.
type AlertsNormalizer struct{}
func (AlertsNormalizer) Match(e event.Event) bool {
@@ -37,7 +37,7 @@ func (AlertsNormalizer) Match(e event.Event) bool {
}
func (AlertsNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
// If we can't derive AsOf from the payload, fall back to the existing event envelope.
fallbackAsOf := in.EmittedAt.UTC()

View File

@@ -23,10 +23,11 @@ import (
// builders by raw schema.
//
// Caveats / policy:
// 1. NWS forecast periods do not include METAR presentWeather phenomena, so ConditionCode
// is inferred from period.shortForecast (with a conservative icon-based fallback).
// 2. Temperature is converted to °C when NWS supplies °F.
// 3. WindSpeed is parsed from strings like "9 mph" / "10 to 15 mph" and converted to km/h.
// 1. Hourly NWS forecast periods do not include METAR presentWeather phenomena, so
// ConditionCode is inferred from period.shortForecast (with a conservative icon fallback).
// 2. Narrative NWS periods intentionally leave ConditionCode unset.
// 3. Temperature is converted to °C when NWS supplies °F.
// 4. WindSpeed is parsed from strings like "9 mph" / "10 to 15 mph" and converted to km/h.
type ForecastNormalizer struct{}
func (ForecastNormalizer) Match(e event.Event) bool {
@@ -41,7 +42,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
}
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
return normalizeForecastEventBySchema(in)
}
@@ -224,6 +225,7 @@ func mapHourlyForecastPeriod(idx int, p nwsHourlyForecastPeriod) (model.WeatherF
// Infer WMO from shortForecast (and fall back to icon token).
providerDesc := strings.TrimSpace(p.ShortForecast)
wmo := wmoFromNWSForecast(providerDesc, p.Icon, tempC)
wmoPtr := wmoCodePtr(wmo)
return model.WeatherForecastPeriod{
StartTime: start,
@@ -232,7 +234,7 @@ func mapHourlyForecastPeriod(idx int, p nwsHourlyForecastPeriod) (model.WeatherF
Name: strings.TrimSpace(p.Name),
IsDay: isDay,
ConditionCode: wmo,
ConditionCode: wmoPtr,
// For forecasts, keep provider short forecast text as the human-facing description.
TextDescription: providerDesc,
@@ -264,9 +266,7 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
tempC := tempCFromNWS(p.Temperature, p.TemperatureUnit)
// Infer WMO from shortForecast (and fall back to icon token).
shortForecast := strings.TrimSpace(p.ShortForecast)
wmo := wmoFromNWSForecast(shortForecast, p.Icon, tempC)
textDescription := strings.TrimSpace(p.DetailedForecast)
if textDescription == "" {
@@ -280,7 +280,7 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
Name: strings.TrimSpace(p.Name),
IsDay: isDay,
ConditionCode: wmo,
ConditionCode: nil,
TextDescription: textDescription,
@@ -292,3 +292,8 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
ProbabilityOfPrecipitationPercent: p.ProbabilityOfPrecipitation.Value,
}, nil
}
func wmoCodePtr(code model.WMOCode) *model.WMOCode {
out := code
return &out
}

View File

@@ -35,6 +35,9 @@ func TestBuildHourlyForecastUsesShortForecastAsTextDescription(t *testing.T) {
if got, want := run.Periods[0].TextDescription, "Mostly Cloudy"; got != want {
t.Fatalf("TextDescription = %q, want %q", got, want)
}
if run.Periods[0].ConditionCode == nil {
t.Fatalf("ConditionCode is nil, want inferred hourly WMO code")
}
wantIssued := time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC)
if !run.IssuedAt.Equal(wantIssued) {
@@ -261,6 +264,9 @@ func TestBuildNarrativeForecastMapsExpectedFields(t *testing.T) {
if p.ProbabilityOfPrecipitationPercent == nil || *p.ProbabilityOfPrecipitationPercent != 20 {
t.Fatalf("ProbabilityOfPrecipitationPercent = %v, want 20", p.ProbabilityOfPrecipitationPercent)
}
if p.ConditionCode != nil {
t.Fatalf("ConditionCode = %v, want nil for narrative period", p.ConditionCode)
}
wantIssued := time.Date(2026, 3, 27, 15, 17, 1, 0, time.UTC)
if !run.IssuedAt.Equal(wantIssued) {

View File

@@ -32,7 +32,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
}
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
return normcommon.NormalizeJSON(
in,

View File

@@ -9,6 +9,7 @@ var builtins = []fknormalize.Normalizer{
ObservationNormalizer{},
ForecastNormalizer{},
ForecastDiscussionNormalizer{},
WeatherStoriesNormalizer{},
AlertsNormalizer{},
}

View File

@@ -262,6 +262,25 @@ type nwsAlertProperties struct {
References json.RawMessage `json:"references"`
}
// nwsWeatherStoriesResponse is a minimal representation of the NWS /weatherstories
// payload needed for mapping into model.WeatherStoryRun.
type nwsWeatherStoriesResponse struct {
Stories []nwsWeatherStory `json:"stories"`
}
type nwsWeatherStory struct {
OfficeID string `json:"officeId"`
StartTime string `json:"startTime"`
EndTime string `json:"endTime"`
UpdateTime string `json:"updateTime"`
Title string `json:"title"`
Description string `json:"description"`
AltText string `json:"altText"`
Priority bool `json:"priority"`
Order int `json:"order"`
Download string `json:"download"`
}
type nwsAlertReference struct {
ID string `json:"id"`
Identifier string `json:"identifier"`

View File

@@ -0,0 +1,107 @@
package nws
import (
"context"
"fmt"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
// WeatherStoriesNormalizer converts:
//
// standards.SchemaRawNWSWeatherStoriesV1 -> standards.SchemaWeatherStoryV1
//
// It maps the NWS /weatherstories JSON response into a canonical story snapshot.
type WeatherStoriesNormalizer struct{}
func (WeatherStoriesNormalizer) Match(e event.Event) bool {
return strings.TrimSpace(e.Schema) == standards.SchemaRawNWSWeatherStoriesV1
}
func (WeatherStoriesNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx
fallbackAsOf := in.EmittedAt.UTC()
if in.EffectiveAt != nil && !in.EffectiveAt.IsZero() {
fallbackAsOf = in.EffectiveAt.UTC()
}
return normcommon.NormalizeJSON(
in,
"nws weatherstories",
standards.SchemaWeatherStoryV1,
func(parsed nwsWeatherStoriesResponse) (model.WeatherStoryRun, time.Time, error) {
return buildWeatherStories(parsed, fallbackAsOf)
},
)
}
func buildWeatherStories(parsed nwsWeatherStoriesResponse, fallbackAsOf time.Time) (model.WeatherStoryRun, time.Time, error) {
stories := make([]model.WeatherStory, 0, len(parsed.Stories))
var officeID string
var asOf time.Time
for i, raw := range parsed.Stories {
startTime, err := parseRequiredNWSTime(raw.StartTime, fmt.Sprintf("stories[%d].startTime", i))
if err != nil {
return model.WeatherStoryRun{}, time.Time{}, err
}
endTime, err := parseRequiredNWSTime(raw.EndTime, fmt.Sprintf("stories[%d].endTime", i))
if err != nil {
return model.WeatherStoryRun{}, time.Time{}, err
}
updatedAt, err := parseRequiredNWSTime(raw.UpdateTime, fmt.Sprintf("stories[%d].updateTime", i))
if err != nil {
return model.WeatherStoryRun{}, time.Time{}, err
}
storyOfficeID := strings.TrimSpace(raw.OfficeID)
if officeID == "" && storyOfficeID != "" {
officeID = storyOfficeID
}
if asOf.IsZero() || updatedAt.After(asOf) {
asOf = updatedAt
}
stories = append(stories, model.WeatherStory{
OfficeID: storyOfficeID,
StartTime: startTime,
EndTime: endTime,
UpdatedAt: updatedAt,
Title: strings.TrimSpace(raw.Title),
Description: strings.TrimSpace(raw.Description),
AltText: strings.TrimSpace(raw.AltText),
Priority: raw.Priority,
Order: raw.Order,
DownloadURL: strings.TrimSpace(raw.Download),
})
}
if asOf.IsZero() {
asOf = fallbackAsOf.UTC()
}
run := model.WeatherStoryRun{
OfficeID: officeID,
AsOf: asOf,
Stories: stories,
}
return run, asOf, nil
}
func parseRequiredNWSTime(raw, field string) (time.Time, error) {
if strings.TrimSpace(raw) == "" {
return time.Time{}, fmt.Errorf("%s is required", field)
}
t, err := nwscommon.ParseTime(raw)
if err != nil {
return time.Time{}, fmt.Errorf("%s: %w", field, err)
}
return t.UTC(), nil
}

View File

@@ -0,0 +1,146 @@
package nws
import (
"encoding/json"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
func TestWeatherStoriesNormalizerProducesCanonicalSchemaAndMapsSample(t *testing.T) {
out, err := (WeatherStoriesNormalizer{}).Normalize(nil, weatherStoriesRawEvent(weatherStoriesSamplePayload()))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
if out == nil {
t.Fatalf("Normalize() returned nil output")
}
if out.Schema != standards.SchemaWeatherStoryV1 {
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherStoryV1)
}
if out.Kind != event.Kind("weather_story") {
t.Fatalf("Kind = %q, want weather_story", out.Kind)
}
payload, ok := out.Payload.(model.WeatherStoryRun)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherStoryRun", out.Payload)
}
if payload.OfficeID != "LSX" {
t.Fatalf("OfficeID = %q, want LSX", payload.OfficeID)
}
wantAsOf := time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC)
if !payload.AsOf.Equal(wantAsOf) {
t.Fatalf("AsOf = %s, want %s", payload.AsOf, wantAsOf)
}
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantAsOf) {
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantAsOf)
}
if len(payload.Stories) != 1 {
t.Fatalf("Stories len = %d, want 1", len(payload.Stories))
}
story := payload.Stories[0]
if story.Title != "Several Chances for Rain Through Monday" {
t.Fatalf("Title = %q", story.Title)
}
if story.Description != "A stagnant weather pattern." {
t.Fatalf("Description = %q", story.Description)
}
if story.AltText != "This slide shows the forecast." {
t.Fatalf("AltText = %q", story.AltText)
}
if story.Priority {
t.Fatalf("Priority = true, want false")
}
if story.Order != 1 {
t.Fatalf("Order = %d, want 1", story.Order)
}
if story.DownloadURL != "https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f" {
t.Fatalf("DownloadURL = %q", story.DownloadURL)
}
}
func TestWeatherStoriesNormalizerEmptyStoriesUsesFallbackAsOf(t *testing.T) {
effectiveAt := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC)
in := weatherStoriesRawEvent(`{"stories":[]}`)
in.EffectiveAt = &effectiveAt
out, err := (WeatherStoriesNormalizer{}).Normalize(nil, in)
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
payload, ok := out.Payload.(model.WeatherStoryRun)
if !ok {
t.Fatalf("Payload type = %T, want model.WeatherStoryRun", out.Payload)
}
if !payload.AsOf.Equal(effectiveAt) {
t.Fatalf("AsOf = %s, want fallback %s", payload.AsOf, effectiveAt)
}
if payload.Stories == nil {
t.Fatalf("Stories = nil, want empty slice")
}
if len(payload.Stories) != 0 {
t.Fatalf("Stories len = %d, want 0", len(payload.Stories))
}
}
func TestWeatherStoriesNormalizerRejectsInvalidRequiredStoryTime(t *testing.T) {
_, err := (WeatherStoriesNormalizer{}).Normalize(nil, weatherStoriesRawEvent(`{
"stories": [{
"officeId": "LSX",
"startTime": "bad",
"endTime": "2026-05-31T11:00:00+00:00",
"updateTime": "2026-05-30T09:00:34+00:00"
}]
}`))
if err == nil {
t.Fatalf("Normalize() error = nil, want error")
}
if !strings.Contains(err.Error(), "stories[0].startTime") {
t.Fatalf("error = %q, want field context", err)
}
}
func TestWeatherStoriesNormalizerMatch(t *testing.T) {
n := WeatherStoriesNormalizer{}
if !n.Match(event.Event{Schema: standards.SchemaRawNWSWeatherStoriesV1}) {
t.Fatalf("Match(raw weatherstories) = false, want true")
}
if n.Match(event.Event{Schema: standards.SchemaRawNWSAlertsV1}) {
t.Fatalf("Match(raw alerts) = true, want false")
}
}
func weatherStoriesRawEvent(payload string) event.Event {
return event.Event{
ID: "evt-weatherstories-1",
Kind: event.Kind("weather_story"),
Source: "nws-weatherstories-test",
EmittedAt: time.Date(2026, 5, 30, 9, 5, 0, 0, time.UTC),
Schema: standards.SchemaRawNWSWeatherStoriesV1,
Payload: json.RawMessage(payload),
}
}
func weatherStoriesSamplePayload() string {
return `{
"stories": [
{
"officeId": " LSX ",
"startTime": "2026-05-30T08:46:00+00:00",
"endTime": "2026-05-31T11:00:00+00:00",
"updateTime": "2026-05-30T09:00:34+00:00",
"title": " Several Chances for Rain Through Monday ",
"description": " A stagnant weather pattern. ",
"altText": " This slide shows the forecast. ",
"priority": false,
"order": 1,
"download": " https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f "
}
]
}`
}

View File

@@ -33,7 +33,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
}
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
// If present, prefer the existing event EmittedAt as IssuedAt.
var fallbackIssued time.Time
@@ -98,6 +98,7 @@ func buildForecast(parsed omForecastResponse, fallbackIssued time.Time) (model.W
}
wmo := wmoAt(parsed.Hourly.WeatherCode, i)
wmoPtr := wmoCodePtr(wmo)
canonicalText := standards.WMOText(wmo, isDay)
period := model.WeatherForecastPeriod{
@@ -107,7 +108,7 @@ func buildForecast(parsed omForecastResponse, fallbackIssued time.Time) (model.W
Name: "",
IsDay: isDay,
ConditionCode: wmo,
ConditionCode: wmoPtr,
TextDescription: canonicalText,
}
@@ -237,3 +238,8 @@ func wmoAt(vals []*int, idx int) model.WMOCode {
}
return model.WMOUnknown
}
func wmoCodePtr(code model.WMOCode) *model.WMOCode {
out := code
return &out
}

View File

@@ -35,6 +35,9 @@ func TestBuildForecastUsesCanonicalTextDescription(t *testing.T) {
if got := run.Periods[0].TextDescription; got != expectedText {
t.Fatalf("TextDescription = %q, want %q", got, expectedText)
}
if run.Periods[0].ConditionCode == nil {
t.Fatalf("ConditionCode is nil, want mapped WMO code")
}
wantIssued := time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC)
if !run.IssuedAt.Equal(wantIssued) {

View File

@@ -40,7 +40,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
}
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
return normcommon.NormalizeJSON(
in,

View File

@@ -8,8 +8,7 @@ import (
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
)
// This file holds provider-specific helpers that are shared across multiple
// OpenWeather normalizers (observations today; forecasts/alerts later).
// This file holds provider-specific helpers for OpenWeather normalizers.
// Keeping these out of observation.go helps preserve the "one normalizer per file"
// convention while avoiding duplication.

View File

@@ -37,7 +37,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
}
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
return normcommon.NormalizeJSON(
in,

View File

@@ -1,5 +1,5 @@
// Package openweather contains provider-specific helper code for OpenWeather used by
// both sources and normalizers.
// Package openweather contains provider-specific helper code for OpenWeather
// used by sources and normalizers.
//
// Rules:
// - No network I/O here.

View File

@@ -8,11 +8,15 @@
// Canonical input schemas:
// - weather.observation.v1 -> model.WeatherObservation
// - weather.forecast.v1 -> model.WeatherForecastRun
// - weather.forecast_discussion.v1 -> model.WeatherForecastDiscussion
// - weather.weather_story.v1 -> model.WeatherStoryRun
// - weather.alert.v1 -> model.WeatherAlertRun
//
// Parent/child relationships:
// - observations.event_id -> observation_present_weather.event_id
// - forecasts.event_id -> forecast_periods.run_event_id
// - forecast_discussions.event_id -> forecast_discussion_key_messages.run_event_id
// - weather_story_runs.event_id -> weather_stories.run_event_id
// - alert_runs.event_id -> alerts.run_event_id
// - alerts.(run_event_id, alert_index) -> alert_references.(run_event_id, alert_index)
//
@@ -24,13 +28,18 @@
// - observation_present_weather.observed_at
// - forecasts.issued_at
// - forecast_periods.issued_at
// - forecast_discussions.issued_at
// - forecast_discussion_key_messages.issued_at
// - weather_story_runs.as_of
// - weather_stories.as_of
// - alert_runs.as_of
// - alerts.as_of
// - alert_references.as_of
//
// Envelope field mapping (shared parent columns)
//
// These columns exist on observations, forecasts, and alert_runs:
// These columns exist on parent tables such as observations, forecasts,
// forecast_discussions, weather_story_runs, and alert_runs:
// - event_id TEXT -> event.id
// - event_kind TEXT -> event.kind
// - event_source TEXT -> event.source
@@ -101,7 +110,7 @@
// - end_time TIMESTAMPTZ -> payload.periods[i].endTime
// - name TEXT NULL -> payload.periods[i].name
// - is_day BOOLEAN NULL -> payload.periods[i].isDay
// - condition_code INTEGER -> payload.periods[i].conditionCode
// - condition_code INTEGER NULL -> payload.periods[i].conditionCode
// - text_description TEXT NULL -> payload.periods[i].textDescription
// - temperature_c DOUBLE PRECISION NULL -> payload.periods[i].temperatureC
// - temperature_c_min DOUBLE PRECISION NULL -> payload.periods[i].temperatureCMin
@@ -120,7 +129,35 @@
// - snowfall_depth_mm DOUBLE PRECISION NULL -> payload.periods[i].snowfallDepthMm
// - uv_index DOUBLE PRECISION NULL -> payload.periods[i].uvIndex
//
// 5. alert_runs (PK: event_id)
// 5. weather_story_runs (PK: event_id)
//
// - event_id TEXT -> event.id
// - event_kind TEXT -> event.kind
// - event_source TEXT -> event.source
// - event_schema TEXT -> event.schema
// - event_emitted_at TIMESTAMPTZ -> event.emitted_at
// - event_effective_at TIMESTAMPTZ NULL -> event.effective_at
// - office_id TEXT NULL -> payload.officeId
// - as_of TIMESTAMPTZ -> payload.asOf
// - story_count INTEGER -> len(payload.stories)
//
// 6. weather_stories (PK: run_event_id, story_index)
//
// - run_event_id TEXT -> weather_story_runs.event_id / payload.stories[i]
// - story_index INTEGER -> i (array position in payload.stories)
// - as_of TIMESTAMPTZ -> payload.asOf (copied from parent)
// - office_id TEXT NULL -> payload.stories[i].officeId
// - start_time TIMESTAMPTZ -> payload.stories[i].startTime
// - end_time TIMESTAMPTZ -> payload.stories[i].endTime
// - updated_at TIMESTAMPTZ -> payload.stories[i].updatedAt
// - title TEXT NULL -> payload.stories[i].title
// - description TEXT NULL -> payload.stories[i].description
// - alt_text TEXT NULL -> payload.stories[i].altText
// - priority BOOLEAN -> payload.stories[i].priority
// - story_order INTEGER -> payload.stories[i].order
// - download_url TEXT NULL -> payload.stories[i].downloadUrl
//
// 7. alert_runs (PK: event_id)
//
// - event_id TEXT -> event.id
// - event_kind TEXT -> event.kind
@@ -135,7 +172,7 @@
// - longitude DOUBLE PRECISION NULL -> payload.longitude
// - alert_count INTEGER -> len(payload.alerts)
//
// 6. alerts (PK: run_event_id, alert_index)
// 8. alerts (PK: run_event_id, alert_index)
//
// - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i]
// - alert_index INTEGER -> i (array position in payload.alerts)
@@ -160,7 +197,7 @@
// - sender_name TEXT NULL -> payload.alerts[i].senderName
// - reference_count INTEGER -> len(payload.alerts[i].references)
//
// 7. alert_references (PK: run_event_id, alert_index, reference_index)
// 9. alert_references (PK: run_event_id, alert_index, reference_index)
//
// - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i].references[j]
// - alert_index INTEGER -> i (array position in payload.alerts)
@@ -181,6 +218,10 @@
// read one row from forecasts, then join forecast_periods by run_event_id
// ordered by period_index to rebuild periods.
//
// - WeatherStoryRun:
// read one row from weather_story_runs, then join weather_stories by
// run_event_id ordered by story_index to rebuild stories.
//
// - WeatherAlertRun:
// read one row from alert_runs, join alerts by run_event_id ordered by
// alert_index, then join alert_references by (run_event_id, alert_index)

View File

@@ -22,6 +22,8 @@ func mapPostgresEvent(_ context.Context, e fkevent.Event) ([]fksinks.PostgresWri
return mapForecastEvent(e)
case standards.SchemaWeatherForecastDiscussionV1:
return mapForecastDiscussionEvent(e)
case standards.SchemaWeatherStoryV1:
return mapWeatherStoryEvent(e)
case standards.SchemaWeatherAlertV1:
return mapAlertEvent(e)
default:
@@ -137,7 +139,7 @@ func mapForecastEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
"end_time": p.EndTime.UTC(),
"name": nullableString(p.Name),
"is_day": nullableBool(p.IsDay),
"condition_code": int(p.ConditionCode),
"condition_code": nullableWMOCode(p.ConditionCode),
"text_description": nullableString(p.TextDescription),
"temperature_c": nullableFloat64(p.TemperatureC),
"temperature_c_min": nullableFloat64(p.TemperatureCMin),
@@ -218,6 +220,59 @@ func mapForecastDiscussionEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error
return writes, nil
}
func mapWeatherStoryEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
run, err := decodePayload[model.WeatherStoryRun](e.Payload)
if err != nil {
return nil, fmt.Errorf("decode weather story payload: %w", err)
}
if run.AsOf.IsZero() {
return nil, fmt.Errorf("decode weather story payload: asOf is required")
}
asOf := run.AsOf.UTC()
writes := make([]fksinks.PostgresWrite, 0, 1+len(run.Stories))
writes = append(writes, fksinks.PostgresWrite{
Table: tableWeatherStoryRuns,
Values: map[string]any{
"event_id": e.ID,
"event_kind": string(e.Kind),
"event_source": e.Source,
"event_schema": e.Schema,
"event_emitted_at": e.EmittedAt.UTC(),
"event_effective_at": nullableTime(e.EffectiveAt),
"office_id": nullableString(run.OfficeID),
"as_of": asOf,
"story_count": len(run.Stories),
},
})
for i, story := range run.Stories {
if story.StartTime.IsZero() || story.EndTime.IsZero() || story.UpdatedAt.IsZero() {
return nil, fmt.Errorf("decode weather story payload: stories[%d] startTime/endTime/updatedAt are required", i)
}
writes = append(writes, fksinks.PostgresWrite{
Table: tableWeatherStories,
Values: map[string]any{
"run_event_id": e.ID,
"story_index": i,
"as_of": asOf,
"office_id": nullableString(story.OfficeID),
"start_time": story.StartTime.UTC(),
"end_time": story.EndTime.UTC(),
"updated_at": story.UpdatedAt.UTC(),
"title": nullableString(story.Title),
"description": nullableString(story.Description),
"alt_text": nullableString(story.AltText),
"priority": story.Priority,
"story_order": story.Order,
"download_url": nullableString(story.DownloadURL),
},
})
}
return writes, nil
}
func mapAlertEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
run, err := decodePayload[model.WeatherAlertRun](e.Payload)
if err != nil {
@@ -370,6 +425,13 @@ func nullableTime(v *time.Time) any {
return v.UTC()
}
func nullableWMOCode(v *model.WMOCode) any {
if v == nil {
return nil
}
return int(*v)
}
func compactJSONText(v any) (any, error) {
if v == nil {
return nil, nil

View File

@@ -63,13 +63,13 @@ func TestMapPostgresEventForecastStructPayload(t *testing.T) {
StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC),
EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
IsDay: &isDay,
ConditionCode: model.WMOCode(2),
ConditionCode: wmoCodePtr(model.WMOCode(2)),
TemperatureC: &temp,
},
{
StartTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
EndTime: time.Date(2026, 3, 16, 21, 0, 0, 0, time.UTC),
ConditionCode: model.WMOCode(3),
ConditionCode: nil,
},
},
}
@@ -94,6 +94,9 @@ func TestMapPostgresEventForecastStructPayload(t *testing.T) {
if got := writes[1].Values["period_index"]; got != 0 {
t.Fatalf("first period index = %#v, want 0", got)
}
if got := writes[2].Values["condition_code"]; got != nil {
t.Fatalf("second period condition_code = %#v, want nil", got)
}
assertAllWritesIncludeAllColumns(t, writes)
}
@@ -190,6 +193,76 @@ func TestMapPostgresEventForecastDiscussionStructPayload(t *testing.T) {
assertAllWritesIncludeAllColumns(t, writes)
}
func TestMapPostgresEventWeatherStoryStructPayload(t *testing.T) {
run := model.WeatherStoryRun{
OfficeID: "LSX",
AsOf: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
Stories: []model.WeatherStory{
{
OfficeID: "LSX",
StartTime: time.Date(2026, 5, 30, 8, 46, 0, 0, time.UTC),
EndTime: time.Date(2026, 5, 31, 11, 0, 0, 0, time.UTC),
UpdatedAt: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
Title: "Several Chances for Rain Through Monday",
Description: "Scattered showers and thunderstorms.",
AltText: "This slide shows the forecast.",
Priority: true,
Order: 1,
DownloadURL: "https://api.weather.gov/offices/LSX/weatherstories/download/story-1",
},
},
}
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, "weather_story", run))
if err != nil {
t.Fatalf("mapPostgresEvent() error = %v", err)
}
if len(writes) != 2 {
t.Fatalf("mapPostgresEvent() writes len = %d, want 2", len(writes))
}
if writes[0].Table != tableWeatherStoryRuns {
t.Fatalf("writes[0].Table = %q, want %q", writes[0].Table, tableWeatherStoryRuns)
}
if got := writes[0].Values["story_count"]; got != 1 {
t.Fatalf("weather_story_runs story_count = %#v, want 1", got)
}
if writes[1].Table != tableWeatherStories {
t.Fatalf("writes[1].Table = %q, want %q", writes[1].Table, tableWeatherStories)
}
if got := writes[1].Values["download_url"]; got != "https://api.weather.gov/offices/LSX/weatherstories/download/story-1" {
t.Fatalf("weather_stories download_url = %#v", got)
}
if got := writes[1].Values["story_order"]; got != 1 {
t.Fatalf("weather_stories story_order = %#v, want 1", got)
}
assertAllWritesIncludeAllColumns(t, writes)
}
func TestMapPostgresEventWeatherStoryRejectsMissingAsOf(t *testing.T) {
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, "weather_story", model.WeatherStoryRun{}))
if err == nil {
t.Fatalf("mapPostgresEvent() error = nil, want missing asOf error")
}
if !strings.Contains(err.Error(), "asOf is required") {
t.Fatalf("error = %q, want asOf context", err)
}
}
func TestMapPostgresEventWeatherStoryRejectsMissingStoryTimes(t *testing.T) {
run := model.WeatherStoryRun{
AsOf: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
Stories: []model.WeatherStory{{Title: "missing times"}},
}
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, "weather_story", run))
if err == nil {
t.Fatalf("mapPostgresEvent() error = nil, want missing story times error")
}
if !strings.Contains(err.Error(), "stories[0] startTime/endTime/updatedAt are required") {
t.Fatalf("error = %q, want story time context", err)
}
}
func TestMapPostgresEventMapPayload(t *testing.T) {
run := model.WeatherForecastRun{
IssuedAt: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
@@ -198,7 +271,7 @@ func TestMapPostgresEventMapPayload(t *testing.T) {
{
StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC),
EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
ConditionCode: model.WMOCode(2),
ConditionCode: wmoCodePtr(model.WMOCode(2)),
},
},
}
@@ -299,3 +372,8 @@ func tableColumnCounts() map[string]int {
}
return m
}
func wmoCodePtr(v model.WMOCode) *model.WMOCode {
out := v
return &out
}

View File

@@ -11,6 +11,8 @@ const (
tableForecastPeriods = "forecast_periods"
tableForecastDiscussions = "forecast_discussions"
tableForecastDiscussionKeyMessages = "forecast_discussion_key_messages"
tableWeatherStoryRuns = "weather_story_runs"
tableWeatherStories = "weather_stories"
tableAlertRuns = "alert_runs"
tableAlerts = "alerts"
tableAlertReferences = "alert_references"
@@ -104,7 +106,7 @@ func PostgresSchema() fksinks.PostgresSchema {
{Name: "end_time", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "name", Type: "TEXT", Nullable: true},
{Name: "is_day", Type: "BOOLEAN", Nullable: true},
{Name: "condition_code", Type: "INTEGER", Nullable: false},
{Name: "condition_code", Type: "INTEGER", Nullable: true},
{Name: "text_description", Type: "TEXT", Nullable: true},
{Name: "temperature_c", Type: "DOUBLE PRECISION", Nullable: true},
{Name: "temperature_c_min", Type: "DOUBLE PRECISION", Nullable: true},
@@ -174,6 +176,51 @@ func PostgresSchema() fksinks.PostgresSchema {
{Name: "idx_wf_discussion_message_issued_at", Columns: []string{"issued_at"}},
},
},
{
Name: tableWeatherStoryRuns,
Columns: []fksinks.PostgresColumn{
{Name: "event_id", Type: "TEXT", Nullable: false},
{Name: "event_kind", Type: "TEXT", Nullable: false},
{Name: "event_source", Type: "TEXT", Nullable: false},
{Name: "event_schema", Type: "TEXT", Nullable: false},
{Name: "event_emitted_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "event_effective_at", Type: "TIMESTAMPTZ", Nullable: true},
{Name: "office_id", Type: "TEXT", Nullable: true},
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "story_count", Type: "INTEGER", Nullable: false},
},
PrimaryKey: []string{"event_id"},
PruneColumn: "as_of",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_story_run_office_as_of", Columns: []string{"office_id", "as_of"}},
{Name: "idx_wf_story_run_as_of", Columns: []string{"as_of"}},
},
},
{
Name: tableWeatherStories,
Columns: []fksinks.PostgresColumn{
{Name: "run_event_id", Type: "TEXT REFERENCES weather_story_runs(event_id) ON DELETE CASCADE", Nullable: false},
{Name: "story_index", Type: "INTEGER", Nullable: false},
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "office_id", Type: "TEXT", Nullable: true},
{Name: "start_time", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "end_time", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "updated_at", Type: "TIMESTAMPTZ", Nullable: false},
{Name: "title", Type: "TEXT", Nullable: true},
{Name: "description", Type: "TEXT", Nullable: true},
{Name: "alt_text", Type: "TEXT", Nullable: true},
{Name: "priority", Type: "BOOLEAN", Nullable: false},
{Name: "story_order", Type: "INTEGER", Nullable: false},
{Name: "download_url", Type: "TEXT", Nullable: true},
},
PrimaryKey: []string{"run_event_id", "story_index"},
PruneColumn: "as_of",
Indexes: []fksinks.PostgresIndex{
{Name: "idx_wf_stories_start_time", Columns: []string{"start_time"}},
{Name: "idx_wf_stories_end_time", Columns: []string{"end_time"}},
{Name: "idx_wf_stories_updated_at", Columns: []string{"updated_at"}},
},
},
{
Name: tableAlertRuns,
Columns: []fksinks.PostgresColumn{

View File

@@ -15,6 +15,8 @@ func TestWeatherPostgresSchemaShape(t *testing.T) {
tableForecastPeriods: true,
tableForecastDiscussions: true,
tableForecastDiscussionKeyMessages: true,
tableWeatherStoryRuns: true,
tableWeatherStories: true,
tableAlertRuns: true,
tableAlerts: true,
tableAlertReferences: true,
@@ -40,3 +42,38 @@ func TestWeatherPostgresSchemaShape(t *testing.T) {
}
}
}
func TestWeatherPostgresSchemaIncludesWeatherStoryColumns(t *testing.T) {
runColumns := columnsForTable(t, tableWeatherStoryRuns)
if !runColumns["as_of"] {
t.Fatalf("%s missing as_of column", tableWeatherStoryRuns)
}
if !runColumns["story_count"] {
t.Fatalf("%s missing story_count column", tableWeatherStoryRuns)
}
storyColumns := columnsForTable(t, tableWeatherStories)
for _, col := range []string{"start_time", "end_time", "updated_at", "title", "description", "alt_text", "priority", "story_order", "download_url"} {
if !storyColumns[col] {
t.Fatalf("%s missing %s column", tableWeatherStories, col)
}
}
}
func columnsForTable(t *testing.T, table string) map[string]bool {
t.Helper()
schema := PostgresSchema()
for _, tbl := range schema.Tables {
if tbl.Name != table {
continue
}
cols := make(map[string]bool, len(tbl.Columns))
for _, col := range tbl.Columns {
cols[col.Name] = true
}
return cols
}
t.Fatalf("missing table %q", table)
return nil
}

View File

@@ -22,6 +22,7 @@ var pollDriverRegistrations = []pollDriverRegistration{
{driver: "nws_forecast_discussion", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {
return nws.NewForecastDiscussionSource(cfg)
}},
{driver: "nws_weatherstories", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewWeatherStoriesSource(cfg) }},
{driver: "openmeteo_observation", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return openmeteo.NewObservationSource(cfg) }},
{driver: "openmeteo_forecast", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return openmeteo.NewForecastSource(cfg) }},
{driver: "openweather_observation", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {

View File

@@ -47,6 +47,19 @@ func TestRegisterBuiltinsRegistersNWSForecastDiscussionDriver(t *testing.T) {
}
}
func TestRegisterBuiltinsRegistersNWSWeatherStoriesDriver(t *testing.T) {
reg := fksource.NewRegistry()
RegisterBuiltins(reg)
in, err := reg.BuildInput(sourceConfigForDriver("nws_weatherstories"))
if err != nil {
t.Fatalf("BuildInput(nws_weatherstories) error = %v", err)
}
if _, ok := in.(fksource.PollSource); !ok {
t.Fatalf("BuildInput(nws_weatherstories) type = %T, want PollSource", in)
}
}
func TestRegisterBuiltinsDoesNotRegisterLegacyNWSForecastDriver(t *testing.T) {
reg := fksource.NewRegistry()
RegisterBuiltins(reg)
@@ -70,6 +83,7 @@ func TestRegisterBuiltinsRegistersAllCurrentDrivers(t *testing.T) {
"nws_forecast_hourly",
"nws_forecast_narrative",
"nws_forecast_discussion",
"nws_weatherstories",
"openmeteo_observation",
"openmeteo_forecast",
"openweather_observation",

View File

@@ -0,0 +1,118 @@
package nws
import (
"context"
"encoding/json"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
fksources "gitea.maximumdirect.net/ejr/feedkit/sources"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
// WeatherStoriesSource polls an NWS weatherstories endpoint and emits a RAW weather story Event.
//
// Output schema:
// - standards.SchemaRawNWSWeatherStoriesV1
type WeatherStoriesSource struct {
http *fksources.HTTPSource
}
func NewWeatherStoriesSource(cfg config.SourceConfig) (*WeatherStoriesSource, error) {
const driver = "nws_weatherstories"
hs, err := fksources.NewHTTPSource(driver, cfg, "application/geo+json, application/json")
if err != nil {
return nil, err
}
return &WeatherStoriesSource{http: hs}, nil
}
func (s *WeatherStoriesSource) Name() string { return s.http.Name }
func (s *WeatherStoriesSource) Kinds() []event.Kind {
return []event.Kind{event.Kind("weather_story")}
}
func (s *WeatherStoriesSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, changed, err := s.fetchRaw(ctx)
if err != nil {
return nil, err
}
if !changed {
return nil, nil
}
var effectiveAt *time.Time
switch {
case !meta.ParsedLatestUpdateTime.IsZero():
t := meta.ParsedLatestUpdateTime.UTC()
effectiveAt = &t
case !meta.ParsedLatestStartTime.IsZero():
t := meta.ParsedLatestStartTime.UTC()
effectiveAt = &t
}
emittedAt := time.Now().UTC()
eventID := fksources.DefaultEventID("", s.http.Name, effectiveAt, emittedAt)
return fksources.SingleEvent(
event.Kind("weather_story"),
s.http.Name,
standards.SchemaRawNWSWeatherStoriesV1,
eventID,
emittedAt,
effectiveAt,
raw,
)
}
type weatherStoriesMeta struct {
Stories []struct {
StartTime string `json:"startTime"`
UpdateTime string `json:"updateTime"`
} `json:"stories"`
ParsedLatestUpdateTime time.Time `json:"-"`
ParsedLatestStartTime time.Time `json:"-"`
}
func (s *WeatherStoriesSource) fetchRaw(ctx context.Context) (json.RawMessage, weatherStoriesMeta, bool, error) {
raw, changed, err := s.http.FetchJSONIfChanged(ctx)
if err != nil {
return nil, weatherStoriesMeta{}, false, err
}
if !changed {
return nil, weatherStoriesMeta{}, false, nil
}
var meta weatherStoriesMeta
if err := json.Unmarshal(raw, &meta); err != nil {
return raw, weatherStoriesMeta{}, true, nil
}
for _, story := range meta.Stories {
if ts := strings.TrimSpace(story.UpdateTime); ts != "" {
if t, err := nwscommon.ParseTime(ts); err == nil {
t = t.UTC()
if meta.ParsedLatestUpdateTime.IsZero() || t.After(meta.ParsedLatestUpdateTime) {
meta.ParsedLatestUpdateTime = t
}
}
}
if ts := strings.TrimSpace(story.StartTime); ts != "" {
if t, err := nwscommon.ParseTime(ts); err == nil {
t = t.UTC()
if meta.ParsedLatestStartTime.IsZero() || t.After(meta.ParsedLatestStartTime) {
meta.ParsedLatestStartTime = t
}
}
}
}
return raw, meta, true, nil
}

View File

@@ -0,0 +1,158 @@
package nws
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
func TestWeatherStoriesSourcePollEmitsExpectedEventAndPrefersLatestUpdateTime(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{
"stories": [
{"startTime":"2026-05-30T08:46:00+00:00","updateTime":"2026-05-30T09:00:34+00:00"},
{"startTime":"2026-05-30T10:00:00+00:00","updateTime":"2026-05-30T11:00:34+00:00"}
]
}`))
}))
defer srv.Close()
src, err := NewWeatherStoriesSource(weatherStoriesSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewWeatherStoriesSource() error = %v", err)
}
if got := src.Kinds(); len(got) != 1 || got[0] != event.Kind("weather_story") {
t.Fatalf("Kinds() = %#v, want [weather_story]", got)
}
events, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(events) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(events))
}
got := events[0]
if got.Kind != event.Kind("weather_story") {
t.Fatalf("Kind = %q, want weather_story", got.Kind)
}
if got.Schema != standards.SchemaRawNWSWeatherStoriesV1 {
t.Fatalf("Schema = %q, want %q", got.Schema, standards.SchemaRawNWSWeatherStoriesV1)
}
wantEffectiveAt := time.Date(2026, 5, 30, 11, 0, 34, 0, time.UTC)
if got.EffectiveAt == nil || !got.EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("EffectiveAt = %v, want %s", got.EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
}
if _, ok := got.Payload.(json.RawMessage); !ok {
t.Fatalf("Payload type = %T, want json.RawMessage", got.Payload)
}
}
func TestWeatherStoriesSourcePollEffectiveAtFallsBackToLatestStartTime(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{
"stories": [
{"startTime":"2026-05-30T08:46:00+00:00","updateTime":"bad"},
{"startTime":"2026-05-31T11:00:00+00:00","updateTime":""}
]
}`))
}))
defer srv.Close()
src, err := NewWeatherStoriesSource(weatherStoriesSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewWeatherStoriesSource() error = %v", err)
}
events, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(events) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(events))
}
wantEffectiveAt := time.Date(2026, 5, 31, 11, 0, 0, 0, time.UTC)
if events[0].EffectiveAt == nil || !events[0].EffectiveAt.Equal(wantEffectiveAt) {
t.Fatalf("EffectiveAt = %v, want %s", events[0].EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
}
}
func TestWeatherStoriesSourcePollReturnsNoEventsWhenUnchanged(t *testing.T) {
const etag = `"stories-v1"`
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("If-None-Match") == etag {
w.WriteHeader(http.StatusNotModified)
return
}
w.Header().Set("ETag", etag)
_, _ = w.Write([]byte(`{"stories":[]}`))
}))
defer srv.Close()
src, err := NewWeatherStoriesSource(weatherStoriesSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewWeatherStoriesSource() error = %v", err)
}
first, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("first Poll() error = %v", err)
}
if len(first) != 1 {
t.Fatalf("first Poll() len = %d, want 1", len(first))
}
second, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("second Poll() error = %v", err)
}
if len(second) != 0 {
t.Fatalf("second Poll() len = %d, want 0", len(second))
}
}
func TestWeatherStoriesSourcePollMetadataDecodeFailureStillEmitsRawEvent(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`not-json`))
}))
defer srv.Close()
src, err := NewWeatherStoriesSource(weatherStoriesSourceConfig(srv.URL))
if err != nil {
t.Fatalf("NewWeatherStoriesSource() error = %v", err)
}
events, err := src.Poll(context.Background())
if err != nil {
t.Fatalf("Poll() error = %v", err)
}
if len(events) != 1 {
t.Fatalf("Poll() len = %d, want 1", len(events))
}
if events[0].EffectiveAt != nil {
t.Fatalf("EffectiveAt = %v, want nil", events[0].EffectiveAt)
}
if events[0].Schema != standards.SchemaRawNWSWeatherStoriesV1 {
t.Fatalf("Schema = %q, want %q", events[0].Schema, standards.SchemaRawNWSWeatherStoriesV1)
}
}
func weatherStoriesSourceConfig(url string) config.SourceConfig {
return config.SourceConfig{
Name: "test-weatherstories-source",
Driver: "nws_weatherstories",
Mode: config.SourceModePoll,
Params: map[string]any{
"url": url,
"user_agent": "test-agent",
},
}
}

View File

@@ -42,7 +42,7 @@ type WeatherAlert struct {
Headline string `json:"headline,omitempty"`
Severity string `json:"severity,omitempty"` // e.g. Extreme/Severe/Moderate/Minor/Unknown
Urgency string `json:"urgency,omitempty"` // e.g. Immediate/Expected/Future/Past/Unknown
Urgency string `json:"urgency,omitempty"` // provider-defined urgency value
Certainty string `json:"certainty,omitempty"` // e.g. Observed/Likely/Possible/Unlikely/Unknown
Status string `json:"status,omitempty"` // e.g. Actual/Exercise/Test/System/Unknown

View File

@@ -2,7 +2,7 @@
// Package model defines weatherfeeder's canonical domain payload types.
//
// These structs are emitted as the Payload of canonical events (schemas "weather.*.vN").
// JSON tags are treated as part of the wire contract for sinks (stdout today; others later).
// JSON tags are treated as part of the wire contract for configured sinks.
//
// Compatibility guidance:
// - Prefer additive changes.

View File

@@ -34,7 +34,7 @@ type WeatherForecastRun struct {
LocationName string `json:"locationName,omitempty"`
IssuedAt time.Time `json:"issuedAt"` // required: when this run was generated/issued
// Some providers include both a generated time and a later update time.
// Some providers include both a generated time and a subsequent update time.
// Keep UpdatedAt optional; many providers wont supply it.
UpdatedAt *time.Time `json:"updatedAt,omitempty"`
@@ -71,9 +71,9 @@ type WeatherForecastPeriod struct {
// Providers vary in whether they explicitly include this.
IsDay *bool `json:"isDay,omitempty"`
// Canonical internal representation (provider-independent).
// Like WeatherObservation, this is required; use an “unknown” WMOCode if unmappable.
ConditionCode WMOCode `json:"conditionCode"`
// Canonical internal representation (provider-independent), when applicable.
// Some products (notably narrative) may not provide or imply a canonical WMO code.
ConditionCode *WMOCode `json:"conditionCode,omitempty"`
// Human-facing narrative summary for this period.
TextDescription string `json:"textDescription,omitempty"`

28
model/weather_story.go Normal file
View File

@@ -0,0 +1,28 @@
package model
import "time"
// WeatherStoryRun is a snapshot of NWS weather stories for an office as-of a point in time.
type WeatherStoryRun struct {
OfficeID string `json:"officeId,omitempty"`
AsOf time.Time `json:"asOf"`
Stories []WeatherStory `json:"stories"`
}
// WeatherStory is a provider-independent representation of a single story card.
type WeatherStory struct {
OfficeID string `json:"officeId,omitempty"`
StartTime time.Time `json:"startTime"`
EndTime time.Time `json:"endTime"`
UpdatedAt time.Time `json:"updatedAt"`
Title string `json:"title,omitempty"`
Description string `json:"description,omitempty"`
AltText string `json:"altText,omitempty"`
Priority bool `json:"priority"`
Order int `json:"order"`
DownloadURL string `json:"downloadUrl,omitempty"`
}

View File

@@ -1,4 +1,4 @@
// File: internal/standards/doc.go
// File: standards/doc.go
//
// Package standards defines weatherfeeders provider-agnostic “project law”:
//

View File

@@ -18,6 +18,7 @@ const (
SchemaRawNWSHourlyForecastV1 = "raw.nws.hourly.forecast.v1"
SchemaRawNWSNarrativeForecastV1 = "raw.nws.narrative.forecast.v1"
SchemaRawNWSForecastDiscussionV1 = "raw.nws.forecast_discussion.v1"
SchemaRawNWSWeatherStoriesV1 = "raw.nws.weatherstories.v1"
SchemaRawOpenMeteoHourlyForecastV1 = "raw.openmeteo.hourly.forecast.v1"
SchemaRawOpenWeatherHourlyForecastV1 = "raw.openweather.hourly.forecast.v1"
@@ -27,5 +28,6 @@ const (
SchemaWeatherObservationV1 = "weather.observation.v1"
SchemaWeatherForecastV1 = "weather.forecast.v1"
SchemaWeatherForecastDiscussionV1 = "weather.forecast_discussion.v1"
SchemaWeatherStoryV1 = "weather.weather_story.v1"
SchemaWeatherAlertV1 = "weather.alert.v1"
)