Compare commits
7 Commits
77bd59cb87
...
002f9d0ba6
| Author | SHA1 | Date | |
|---|---|---|---|
| 002f9d0ba6 | |||
| ec115ba152 | |||
| 47176520bb | |||
| 9e27a431a1 | |||
| f0605781a2 | |||
| abb8f218ec | |||
| 4ac4e401ed |
406
API.md
406
API.md
@@ -1,404 +1,4 @@
|
||||
# weatherfeeder API (Wire Contract)
|
||||
# Event Wire Contract
|
||||
|
||||
This document defines the stable, consumer-facing JSON contract emitted by weatherfeeder sinks.
|
||||
|
||||
weatherfeeder emits **events** encoded as JSON. Each event has:
|
||||
- an **envelope** (metadata + schema identifier), and
|
||||
- a **payload** whose shape is determined by `schema`.
|
||||
|
||||
Downstream consumers should:
|
||||
1. parse the event envelope,
|
||||
2. switch on `schema`, then
|
||||
3. decode `payload` into the matching schema.
|
||||
|
||||
---
|
||||
|
||||
## Event envelope
|
||||
|
||||
All events are JSON objects with these fields:
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `id` | string | yes | Stable event identifier. Treat as opaque. |
|
||||
| `schema` | string | yes | Schema identifier (e.g. `weather.observation.v1`). |
|
||||
| `source` | string | yes | Provider/source identifier (stable within configuration). |
|
||||
| `effectiveAt` | string (timestamp) | yes | RFC3339Nano timestamp indicating when this event is effective. |
|
||||
| `payload` | object | yes | Schema-specific payload (see below). |
|
||||
|
||||
### Timestamp format
|
||||
|
||||
All timestamps are encoded as JSON strings using Go’s `time.Time` JSON encoding (RFC3339Nano).
|
||||
|
||||
Examples:
|
||||
- `"2026-01-17T14:27:00Z"`
|
||||
- `"2026-01-17T08:27:00-06:00"`
|
||||
|
||||
---
|
||||
|
||||
## Canonical schemas
|
||||
|
||||
weatherfeeder emits five canonical domain schemas:
|
||||
|
||||
- `weather.observation.v1`
|
||||
- `weather.forecast.v1`
|
||||
- `weather.forecast_discussion.v1`
|
||||
- `weather.weather_story.v1`
|
||||
- `weather.alert.v1`
|
||||
|
||||
Each payload is described below using the JSON field names as the contract.
|
||||
|
||||
### Raw upstream schemas
|
||||
|
||||
weatherfeeder sources also emit provider-specific raw schemas before normalization.
|
||||
Relevant raw source schemas include:
|
||||
|
||||
- `raw.nws.forecast_discussion.v1`
|
||||
- payload type: string
|
||||
- payload contents: exact fetched HTML response body
|
||||
- `raw.nws.weatherstories.v1`
|
||||
- payload type: object
|
||||
- payload contents: exact fetched JSON response body
|
||||
|
||||
---
|
||||
|
||||
## Shared Conventions
|
||||
|
||||
- Timestamps are JSON strings in RFC3339Nano format.
|
||||
- Optional fields are omitted when unknown (`omitempty` behavior).
|
||||
- Numeric measurements are normalized to metric units:
|
||||
- `*C` = Celsius
|
||||
- `*Kmh` = kilometers/hour
|
||||
- `*Pa` = Pascals
|
||||
- `*Meters` = meters
|
||||
- `*Mm` = millimeters
|
||||
- `*Percent` = percent (0-100)
|
||||
- `conditionCode` is a WMO weather interpretation code (`int`).
|
||||
- Unknown/unmappable is `-1`.
|
||||
- Downstream consumers should treat unknown codes as “unknown conditions” rather than failing decoding.
|
||||
- For readability and stability, weatherfeeder rounds floating-point values in canonical payloads to
|
||||
**4 digits after the decimal** during normalization.
|
||||
|
||||
---
|
||||
|
||||
## Schema: `weather.observation.v1`
|
||||
|
||||
Payload type: `WeatherObservation`
|
||||
|
||||
A `WeatherObservation` represents a point-in-time observation for a station/location.
|
||||
|
||||
### Fields
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `stationId` | string | no | Provider station/location identifier |
|
||||
| `stationName` | string | no | Human station name |
|
||||
| `timestamp` | timestamp string | yes | Observation timestamp |
|
||||
| `conditionCode` | int | yes | WMO code (`-1` unknown) |
|
||||
| `isDay` | bool | no | Day/night hint |
|
||||
| `textDescription` | string | no | Human-facing short description |
|
||||
| `temperatureC` | number | no | Celsius |
|
||||
| `dewpointC` | number | no | Celsius |
|
||||
| `windDirectionDegrees` | number | no | Degrees |
|
||||
| `windSpeedKmh` | number | no | km/h |
|
||||
| `windGustKmh` | number | no | km/h |
|
||||
| `barometricPressurePa` | number | no | Pascals |
|
||||
| `visibilityMeters` | number | no | Meters |
|
||||
| `relativeHumidityPercent` | number | no | Percent |
|
||||
| `apparentTemperatureC` | number | no | Celsius |
|
||||
| `presentWeather` | array | no | Provider-specific structured weather fragments |
|
||||
|
||||
### Nested: `presentWeather[]`
|
||||
|
||||
Each `presentWeather[]` element:
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `raw` | object | no | Provider-specific JSON object |
|
||||
|
||||
---
|
||||
|
||||
## Schema: `weather.forecast.v1`
|
||||
|
||||
Payload type: `WeatherForecastRun`
|
||||
|
||||
A `WeatherForecastRun` is a single issued forecast snapshot for a location and a specific product
|
||||
(hourly / narrative / daily). The run contains an ordered list of forecast periods.
|
||||
|
||||
### `product` values
|
||||
|
||||
`product` is one of:
|
||||
|
||||
- `"hourly"`
|
||||
- `"narrative"`
|
||||
- `"daily"`
|
||||
|
||||
### Fields
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `locationId` | string | no | Provider location identifier |
|
||||
| `locationName` | string | no | Human name, if available |
|
||||
| `issuedAt` | string (timestamp) | yes | When this run was generated/issued |
|
||||
| `updatedAt` | string (timestamp) | no | Optional later update time |
|
||||
| `product` | string | yes | One of `hourly`, `narrative`, `daily` |
|
||||
| `latitude` | number | no | Degrees |
|
||||
| `longitude` | number | no | Degrees |
|
||||
| `elevationMeters` | number | no | meters |
|
||||
| `periods` | array | yes | Chronological forecast periods |
|
||||
|
||||
### Nested: `periods[]` (`WeatherForecastPeriod`)
|
||||
|
||||
A `WeatherForecastPeriod` is valid for `[startTime, endTime)`.
|
||||
|
||||
| Field | Type | Required | Units / Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `startTime` | string (timestamp) | yes | Period start |
|
||||
| `endTime` | string (timestamp) | yes | Period end |
|
||||
| `name` | string | no | Human label (often empty for hourly) |
|
||||
| `isDay` | bool | no | Day/night hint |
|
||||
| `conditionCode` | int | no | WMO code when applicable (`-1` for unknown) |
|
||||
| `textDescription` | string | no | Human-facing short phrase |
|
||||
| `temperatureC` | number | no | °C |
|
||||
| `temperatureCMin` | number | no | °C (aggregated products) |
|
||||
| `temperatureCMax` | number | no | °C (aggregated products) |
|
||||
| `dewpointC` | number | no | °C |
|
||||
| `relativeHumidityPercent` | number | no | percent |
|
||||
| `windDirectionDegrees` | number | no | degrees |
|
||||
| `windSpeedKmh` | number | no | km/h |
|
||||
| `windGustKmh` | number | no | km/h |
|
||||
| `barometricPressurePa` | number | no | Pa |
|
||||
| `visibilityMeters` | number | no | meters |
|
||||
| `apparentTemperatureC` | number | no | °C |
|
||||
| `cloudCoverPercent` | number | no | percent |
|
||||
| `probabilityOfPrecipitationPercent` | number | no | percent |
|
||||
| `precipitationAmountMm` | number | no | mm (liquid equivalent) |
|
||||
| `snowfallDepthMm` | number | no | mm |
|
||||
| `uvIndex` | number | no | unitless index |
|
||||
|
||||
---
|
||||
|
||||
## Schema: `weather.alert.v1`
|
||||
|
||||
Payload type: `WeatherAlertRun`
|
||||
|
||||
A `WeatherAlertRun` is a snapshot of *active* alerts for a location as-of a point in time.
|
||||
A run may contain zero, one, or many alerts.
|
||||
|
||||
### Fields
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `locationId` | string | no | Provider location identifier |
|
||||
| `locationName` | string | no | Human name, if available |
|
||||
| `asOf` | string (timestamp) | yes | When the provider asserted this snapshot is current |
|
||||
| `latitude` | number | no | Degrees |
|
||||
| `longitude` | number | no | Degrees |
|
||||
| `alerts` | array | yes | Active alerts (order provider-dependent) |
|
||||
|
||||
### Nested: `alerts[]` (`WeatherAlert`)
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `id` | string | yes | Provider-stable identifier (often a URL/URI) |
|
||||
| `event` | string | no | Classification/event label |
|
||||
| `headline` | string | no | Alert headline |
|
||||
| `severity` | string | no | Example: Extreme/Severe/Moderate/Minor/Unknown |
|
||||
| `urgency` | string | no | Example: Immediate/Expected/Future/Past/Unknown |
|
||||
| `certainty` | string | no | Example: Observed/Likely/Possible/Unlikely/Unknown |
|
||||
| `status` | string | no | Example: Actual/Exercise/Test/System/Unknown |
|
||||
| `messageType` | string | no | Example: Alert/Update/Cancel |
|
||||
| `category` | string | no | Example: Met/Geo/Safety/Rescue/Fire/Health/Env/Transport/Infra/CBRNE/Other |
|
||||
| `response` | string | no | Example: Shelter/Evacuate/Prepare/Execute/Avoid/Monitor/Assess/AllClear/None |
|
||||
| `response` | string | no | e.g. Shelter/Evacuate/Prepare/... |
|
||||
| `description` | string | no | Narrative |
|
||||
| `instruction` | string | no | What to do |
|
||||
| `sent` | string (timestamp) | no | Provider-dependent |
|
||||
| `effective` | string (timestamp) | no | Provider-dependent |
|
||||
| `onset` | string (timestamp) | no | Provider-dependent |
|
||||
| `expires` | string (timestamp) | no | Provider-dependent |
|
||||
| `areaDescription` | string | no | Often a provider string |
|
||||
| `senderName` | string | no | Provenance |
|
||||
| `references` | array | no | Related alert references |
|
||||
|
||||
### Nested: `references[]` (`AlertReference`)
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `id` | string | no | Provider reference ID/URI |
|
||||
| `identifier` | string | no | Provider identifier string, if distinct |
|
||||
| `sender` | string | no | Sender |
|
||||
| `sent` | string (timestamp) | no | Timestamp |
|
||||
|
||||
---
|
||||
|
||||
## Schema: `weather.weather_story.v1`
|
||||
|
||||
Payload type: `WeatherStoryRun`
|
||||
|
||||
A `WeatherStoryRun` is a snapshot of NWS weather stories for an office as-of a point in time.
|
||||
The run may contain zero, one, or many stories.
|
||||
|
||||
### Fields
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `officeId` | string | no | NWS office identifier, e.g. `LSX` |
|
||||
| `asOf` | string (timestamp) | yes | Latest story update time or source fallback |
|
||||
| `stories` | array | yes | Weather stories (order provider-dependent) |
|
||||
|
||||
### Nested: `stories[]` (`WeatherStory`)
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `officeId` | string | no | NWS office identifier |
|
||||
| `startTime` | string (timestamp) | yes | Story validity start |
|
||||
| `endTime` | string (timestamp) | yes | Story validity end |
|
||||
| `updatedAt` | string (timestamp) | yes | Story update time |
|
||||
| `title` | string | no | Human story title |
|
||||
| `description` | string | no | Story narrative text |
|
||||
| `altText` | string | no | Accessibility text for the provider graphic |
|
||||
| `priority` | bool | yes | Provider priority flag |
|
||||
| `order` | int | yes | Provider display order |
|
||||
| `downloadUrl` | string | no | Provider download URL; weatherfeeder does not fetch the asset |
|
||||
|
||||
---
|
||||
|
||||
## Schema: `weather.forecast_discussion.v1`
|
||||
|
||||
Payload type: `WeatherForecastDiscussion`
|
||||
|
||||
A `WeatherForecastDiscussion` is an issued narrative bulletin for an NWS office.
|
||||
It is distinct from `weather.forecast.v1`, which is period-based.
|
||||
|
||||
### Fields
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `officeId` | string | no | NWS office identifier, e.g. `LSX` |
|
||||
| `officeName` | string | no | Human office name |
|
||||
| `product` | string | yes | Currently `afd` |
|
||||
| `issuedAt` | string (timestamp) | yes | Bulletin issue time |
|
||||
| `updatedAt` | string (timestamp) | no | Optional page/update timestamp |
|
||||
| `keyMessages` | array | no | Ordered key-message bullet list |
|
||||
| `shortTerm` | object | no | Short-term discussion section |
|
||||
| `longTerm` | object | no | Long-term discussion section |
|
||||
|
||||
### Nested: `shortTerm` / `longTerm`
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---:|:---:|---|
|
||||
| `qualifier` | string | no | Header qualifier such as `(Through Late Sunday Night)` |
|
||||
| `issuedAt` | string (timestamp) | no | Optional section-local issue time |
|
||||
| `text` | string | no | Paragraph-preserved prose text |
|
||||
|
||||
---
|
||||
|
||||
## Compatibility rules
|
||||
|
||||
- Consumers **must** ignore unknown fields.
|
||||
- Producers (weatherfeeder) prefer **additive changes** within a schema version.
|
||||
- Renames/removals/semantic breaks normally require a **schema version bump** (`weather.*.v2`); pre-1.0 projects may choose in-place changes.
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
### Observation event (`weather.observation.v1`)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "nws:KSTL:2026-01-17T14:00:00Z",
|
||||
"schema": "weather.observation.v1",
|
||||
"source": "nws_observation",
|
||||
"effectiveAt": "2026-01-17T14:00:00Z",
|
||||
"payload": {
|
||||
"stationId": "KSTL",
|
||||
"timestamp": "2026-01-17T14:00:00Z",
|
||||
"conditionCode": 1,
|
||||
"textDescription": "Mainly Sunny",
|
||||
"temperatureC": 3.25,
|
||||
"windSpeedKmh": 18.5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Forecast event (`weather.forecast.v1`)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "openmeteo:38.63,-90.20:2026-01-17T13:00:00Z",
|
||||
"schema": "weather.forecast.v1",
|
||||
"source": "openmeteo_forecast",
|
||||
"effectiveAt": "2026-01-17T13:00:00Z",
|
||||
"payload": {
|
||||
"locationName": "St. Louis, MO",
|
||||
"issuedAt": "2026-01-17T13:00:00Z",
|
||||
"product": "hourly",
|
||||
"latitude": 38.63,
|
||||
"longitude": -90.2,
|
||||
"periods": [
|
||||
{
|
||||
"startTime": "2026-01-17T14:00:00Z",
|
||||
"endTime": "2026-01-17T15:00:00Z",
|
||||
"conditionCode": 2,
|
||||
"textDescription": "Partly Cloudy",
|
||||
"temperatureC": 3.5,
|
||||
"probabilityOfPrecipitationPercent": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Alert event (`weather.alert.v1`)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "nws:alerts:2026-01-17T14:10:00Z",
|
||||
"schema": "weather.alert.v1",
|
||||
"source": "nws_alerts",
|
||||
"effectiveAt": "2026-01-17T14:10:00Z",
|
||||
"payload": {
|
||||
"asOf": "2026-01-17T14:05:00Z",
|
||||
"alerts": [
|
||||
{
|
||||
"id": "https://api.weather.gov/alerts/abc123",
|
||||
"event": "Winter Weather Advisory",
|
||||
"headline": "Winter Weather Advisory issued January 17 at 8:05AM CST",
|
||||
"severity": "Moderate",
|
||||
"description": "Mixed precipitation expected...",
|
||||
"expires": "2026-01-18T06:00:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Weather story event (`weather.weather_story.v1`)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "nws:weatherstories:2026-05-30T09:00:34Z",
|
||||
"schema": "weather.weather_story.v1",
|
||||
"source": "nws_weatherstories",
|
||||
"effectiveAt": "2026-05-30T09:00:34Z",
|
||||
"payload": {
|
||||
"officeId": "LSX",
|
||||
"asOf": "2026-05-30T09:00:34Z",
|
||||
"stories": [
|
||||
{
|
||||
"officeId": "LSX",
|
||||
"startTime": "2026-05-30T08:46:00Z",
|
||||
"endTime": "2026-05-31T11:00:00Z",
|
||||
"updatedAt": "2026-05-30T09:00:34Z",
|
||||
"title": "Several Chances for Rain Through Monday",
|
||||
"description": "Scattered showers and thunderstorms remain possible.",
|
||||
"altText": "This slide shows the forecast for today through Tuesday.",
|
||||
"priority": false,
|
||||
"order": 1,
|
||||
"downloadUrl": "https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
The canonical event wire contract has moved to
|
||||
[docs/integrations/events.md](docs/integrations/events.md).
|
||||
|
||||
57
README.md
57
README.md
@@ -1,36 +1,39 @@
|
||||
# weatherfeeder
|
||||
|
||||
weatherfeeder is a small daemon that polls weather observations, forecasts, and alerts from multiple upstream
|
||||
providers, normalizes them into a provider-independent format, and emits them to a sink.
|
||||
`weatherfeeder` is a config-driven daemon that polls weather providers, normalizes
|
||||
provider-specific responses into canonical weather events, and routes those
|
||||
events to configured sinks.
|
||||
|
||||
Today, the only implemented sink is `stdout`, which prints JSON-encoded events.
|
||||
It currently supports NWS observations, alerts, hourly forecasts, narrative
|
||||
forecasts, forecast discussions, and weather stories; Open-Meteo observations
|
||||
and hourly forecasts; and OpenWeather observations. Implemented sinks are
|
||||
stdout, NATS, and Postgres.
|
||||
|
||||
## What weatherfeeder emits
|
||||
## Quickstart
|
||||
|
||||
weatherfeeder emits **feed events** encoded as JSON. Each event includes a schema identifier and a payload.
|
||||
Downstream consumers should key off the `schema` value and decode the `payload` accordingly.
|
||||
Run the checked-in sample config:
|
||||
|
||||
Canonical domain schemas emitted after normalization:
|
||||
```sh
|
||||
cd cmd/weatherfeeder
|
||||
go run .
|
||||
```
|
||||
|
||||
- `weather.observation.v1` → `WeatherObservation`
|
||||
- `weather.forecast.v1` → `WeatherForecastRun`
|
||||
- `weather.forecast_discussion.v1` → `WeatherForecastDiscussion`
|
||||
- `weather.weather_story.v1` → `WeatherStoryRun`
|
||||
- `weather.alert.v1` → `WeatherAlertRun`
|
||||
The sample config at `cmd/weatherfeeder/config.yml` is load-tested and can be
|
||||
used as a starting point. The executable always reads `config.yml` from its
|
||||
current working directory.
|
||||
|
||||
For the complete wire contract (event envelope + payload schemas, fields, units, and compatibility rules), see:
|
||||
## Documentation
|
||||
|
||||
- **API.md**
|
||||
|
||||
## Upstream providers (current MVP)
|
||||
|
||||
- NWS: observations, hourly forecasts, narrative forecasts, forecast discussions, weather stories, alerts
|
||||
- Open-Meteo: observations, hourly forecasts
|
||||
- OpenWeather: observations
|
||||
|
||||
## Versioning & compatibility
|
||||
|
||||
The JSON field names on canonical payload types are treated as part of the wire contract.
|
||||
Additive changes are preferred. Renames/removals require a schema version bump.
|
||||
|
||||
See **API.md** for details.
|
||||
- [CLI reference](docs/cli.md)
|
||||
- [Configuration reference](docs/config.md)
|
||||
- [Operations guide](docs/operations.md)
|
||||
- [Troubleshooting guide](docs/troubleshooting.md)
|
||||
- [Example configs](examples/)
|
||||
- [Event wire contract](docs/integrations/events.md)
|
||||
- [Postgres table contract](docs/integrations/postgres.md)
|
||||
- [NWS integration notes](docs/integrations/nws.md)
|
||||
- [Open-Meteo integration notes](docs/integrations/openmeteo.md)
|
||||
- [OpenWeather integration notes](docs/integrations/openweather.md)
|
||||
- [Architecture policy](docs/policy/architecture.md)
|
||||
- [Development policy](docs/policy/development.md)
|
||||
- [Documentation policy](docs/policy/documentation.md)
|
||||
|
||||
@@ -24,7 +24,7 @@ sources:
|
||||
# driver: openweather_observation
|
||||
# every: 10m
|
||||
# params:
|
||||
# url: "https://api.openweathermap.org/data/2.5/weather?lat=38.6239&lon=-90.3571&appid=c954f2566cb7ccb56b43737b52e88fc6&units=metric"
|
||||
# url: "https://api.openweathermap.org/data/2.5/weather?lat=38.6239&lon=-90.3571&units=metric"
|
||||
# user_agent: "HomeOps (eric@maximumdirect.net)"
|
||||
|
||||
# - name: NWSObservationKSUS
|
||||
@@ -115,7 +115,7 @@ sinks:
|
||||
# params:
|
||||
# uri: postgres://weatherdb:5432/weatherdb?sslmode=disable
|
||||
# username: weatherdb
|
||||
# password: weatherdb
|
||||
# password: <database_password>
|
||||
# prune: 3d
|
||||
# # Prunes rows older than now-3d on each write transaction.
|
||||
|
||||
|
||||
@@ -2,7 +2,9 @@ package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -82,6 +84,33 @@ func TestExampleConfigSourcesBuildSchedulerJobs(t *testing.T) {
|
||||
t.Fatalf("config.Load(config.yml) unexpected error: %v", err)
|
||||
}
|
||||
|
||||
assertConfigSourcesBuildSchedulerJobs(t, cfg)
|
||||
}
|
||||
|
||||
func TestMaintainedConfigExamplesLoad(t *testing.T) {
|
||||
paths, err := filepath.Glob("../../examples/*.yml")
|
||||
if err != nil {
|
||||
t.Fatalf("filepath.Glob examples: %v", err)
|
||||
}
|
||||
sort.Strings(paths)
|
||||
if len(paths) == 0 {
|
||||
t.Fatalf("expected maintained config examples")
|
||||
}
|
||||
|
||||
for _, path := range paths {
|
||||
t.Run(filepath.Base(path), func(t *testing.T) {
|
||||
cfg, err := config.Load(path)
|
||||
if err != nil {
|
||||
t.Fatalf("config.Load(%s) unexpected error: %v", path, err)
|
||||
}
|
||||
assertConfigSourcesBuildSchedulerJobs(t, cfg)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func assertConfigSourcesBuildSchedulerJobs(t *testing.T, cfg *config.Config) {
|
||||
t.Helper()
|
||||
|
||||
reg := fksources.NewRegistry()
|
||||
wfsources.RegisterBuiltins(reg)
|
||||
|
||||
@@ -91,6 +120,10 @@ func TestExampleConfigSourcesBuildSchedulerJobs(t *testing.T) {
|
||||
t.Fatalf("BuildInput(sources[%d]) error = %v", i, err)
|
||||
}
|
||||
|
||||
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
|
||||
t.Fatalf("ValidateExpectedKinds(sources[%d]) error = %v", i, err)
|
||||
}
|
||||
|
||||
job, err := fkscheduler.JobFromSourceConfig(in, sc)
|
||||
if err != nil {
|
||||
t.Fatalf("JobFromSourceConfig(sources[%d]) error = %v", i, err)
|
||||
|
||||
72
docs/cli.md
Normal file
72
docs/cli.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# CLI Reference
|
||||
|
||||
## Shortest Useful Command
|
||||
|
||||
Run `weatherfeeder` from a directory containing `config.yml`:
|
||||
|
||||
```sh
|
||||
cd cmd/weatherfeeder
|
||||
go run .
|
||||
```
|
||||
|
||||
When using a built binary:
|
||||
|
||||
```sh
|
||||
./weatherfeeder
|
||||
```
|
||||
|
||||
## Command Overview
|
||||
|
||||
`weatherfeeder` starts a long-running polling daemon. On startup it:
|
||||
|
||||
1. reads `config.yml` from the current working directory;
|
||||
2. builds configured sources, sinks, and routes;
|
||||
3. starts polling sources on their configured intervals;
|
||||
4. normalizes and deduplicates events;
|
||||
5. dispatches matching events to configured sinks.
|
||||
|
||||
The command logs startup, runtime, and shutdown messages to stderr using the Go
|
||||
standard logger.
|
||||
|
||||
## Flags
|
||||
|
||||
There are currently no CLI flags, subcommands, or environment-variable based
|
||||
configuration controls.
|
||||
|
||||
The config path is fixed at `config.yml` relative to the process current working
|
||||
directory. To run with a different config, change the working directory or place
|
||||
the desired file at that path.
|
||||
|
||||
## Common Workflows
|
||||
|
||||
Run the checked-in sample config:
|
||||
|
||||
```sh
|
||||
cd cmd/weatherfeeder
|
||||
go run .
|
||||
```
|
||||
|
||||
Maintained copyable configs are available under [`examples/`](../examples/).
|
||||
|
||||
Build and run a local binary:
|
||||
|
||||
```sh
|
||||
go build -o weatherfeeder ./cmd/weatherfeeder
|
||||
cp cmd/weatherfeeder/config.yml .
|
||||
./weatherfeeder
|
||||
```
|
||||
|
||||
Run in the project container image with a mounted config:
|
||||
|
||||
```sh
|
||||
docker run --rm -v "$PWD/config.yml:/weatherfeeder/config.yml:ro" weatherfeeder
|
||||
```
|
||||
|
||||
The Docker image sets `/weatherfeeder` as the working directory, so the mounted
|
||||
file must appear at `/weatherfeeder/config.yml`.
|
||||
|
||||
## Shutdown
|
||||
|
||||
Stop the daemon with `Ctrl-C` or `SIGTERM`. The process uses context-aware
|
||||
shutdown for scheduler, dispatcher, processors, sources, and sinks, then logs
|
||||
`shutdown complete`.
|
||||
222
docs/config.md
Normal file
222
docs/config.md
Normal file
@@ -0,0 +1,222 @@
|
||||
# Configuration Reference
|
||||
|
||||
## Config File
|
||||
|
||||
`weatherfeeder` reads exactly one YAML file named `config.yml` from the current
|
||||
working directory. There is no config path flag and no search path.
|
||||
|
||||
YAML decoding is strict for config struct fields: misspelled fields such as
|
||||
`sources[].drviver` fail startup. Driver-specific `params` maps are validated by
|
||||
the source or sink constructor that consumes them.
|
||||
|
||||
The top-level file contains:
|
||||
|
||||
```yaml
|
||||
sources:
|
||||
- name: NWSObservationKSTL
|
||||
mode: poll
|
||||
driver: nws_observation
|
||||
every: 10m
|
||||
kinds: ["observation"]
|
||||
params:
|
||||
url: "https://api.weather.gov/stations/KSTL/observations/latest"
|
||||
user_agent: "Example weatherfeeder operator (ops@example.com)"
|
||||
|
||||
sinks:
|
||||
- name: stdout
|
||||
driver: stdout
|
||||
params: {}
|
||||
|
||||
routes:
|
||||
- sink: stdout
|
||||
kinds: ["observation"]
|
||||
```
|
||||
|
||||
`sources` and `sinks` must each contain at least one entry. `routes` is optional.
|
||||
When `routes` is omitted, every configured sink receives every event kind.
|
||||
Maintained copyable configs are available under [`examples/`](../examples/).
|
||||
|
||||
## Production-Oriented Shape
|
||||
|
||||
A typical deployment uses multiple polling sources and sends the same canonical
|
||||
event stream to a broker or database:
|
||||
|
||||
```yaml
|
||||
sources:
|
||||
- name: NWSAlertsLocal
|
||||
mode: poll
|
||||
driver: nws_alerts
|
||||
every: 1m
|
||||
kinds: ["alert"]
|
||||
params:
|
||||
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
|
||||
user_agent: "Example weatherfeeder operator (ops@example.com)"
|
||||
|
||||
sinks:
|
||||
- name: nats_weather
|
||||
driver: nats
|
||||
params:
|
||||
url: nats://nats:4222
|
||||
subject: weatherfeeder
|
||||
|
||||
- name: pg_weather
|
||||
driver: postgres
|
||||
params:
|
||||
uri: postgres://weatherdb:5432/weatherdb?sslmode=disable
|
||||
username: weatherdb
|
||||
password: <database_password>
|
||||
prune: 3d
|
||||
|
||||
routes:
|
||||
- sink: nats_weather
|
||||
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"]
|
||||
|
||||
- sink: pg_weather
|
||||
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"]
|
||||
```
|
||||
|
||||
Do not commit real API keys, database passwords, or personal contact addresses in
|
||||
copyable configs.
|
||||
|
||||
## Top-Level Fields
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|:---:|---|
|
||||
| `sources` | yes | List of configured input sources. |
|
||||
| `sinks` | yes | List of configured output sinks. |
|
||||
| `routes` | no | List of sink routing rules. If omitted, all sinks receive all kinds. |
|
||||
|
||||
## Source Fields
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|:---:|---|
|
||||
| `name` | yes | Unique source name. Used as the event source identifier. |
|
||||
| `driver` | yes | Source driver name. |
|
||||
| `mode` | no | `poll`, `stream`, or omitted for auto. Current weatherfeeder drivers are polling drivers. |
|
||||
| `every` | yes | Poll interval for current weatherfeeder source drivers. |
|
||||
| `kinds` | no | Expected event kinds. If present, startup verifies they match the source driver. |
|
||||
| `params` | driver-specific | Driver parameters. Current source drivers require HTTP params. |
|
||||
|
||||
Current event kinds are `observation`, `forecast`, `forecast_discussion`,
|
||||
`weather_story`, and `alert`.
|
||||
|
||||
## Source Drivers
|
||||
|
||||
| Driver | Kind | Upstream product |
|
||||
|---|---|---|
|
||||
| `nws_observation` | `observation` | NWS station latest observation. |
|
||||
| `nws_alerts` | `alert` | NWS alerts collection. |
|
||||
| `nws_forecast_hourly` | `forecast` | NWS hourly gridpoint forecast. |
|
||||
| `nws_forecast_narrative` | `forecast` | NWS narrative gridpoint forecast. |
|
||||
| `nws_forecast_discussion` | `forecast_discussion` | NWS forecast discussion HTML product. |
|
||||
| `nws_weatherstories` | `weather_story` | NWS office weather stories. |
|
||||
| `openmeteo_observation` | `observation` | Open-Meteo current conditions. |
|
||||
| `openmeteo_forecast` | `forecast` | Open-Meteo hourly forecast. |
|
||||
| `openweather_observation` | `observation` | OpenWeather current weather. |
|
||||
|
||||
## HTTP Source Params
|
||||
|
||||
All current source drivers use the shared HTTP polling helper.
|
||||
|
||||
| Param | Required | Description |
|
||||
|---|:---:|---|
|
||||
| `url` | yes | Full upstream request URL. `URL` is also accepted by the helper. |
|
||||
| `user_agent` | yes | User-Agent sent to the upstream provider. `userAgent` is also accepted by the helper. |
|
||||
| `conditional` | no | Boolean. Defaults to `true`; enables ETag and Last-Modified conditional requests. |
|
||||
| `http_timeout` | no | Positive duration for the HTTP client timeout. |
|
||||
| `http_response_body_limit_bytes` | no | Positive integer response body limit in bytes. |
|
||||
|
||||
When `conditional` is enabled and the upstream returns `304 Not Modified`, the
|
||||
source emits no events for that poll.
|
||||
|
||||
OpenWeather observation URLs must include `units=metric`. Startup fails if the
|
||||
URL omits it or sets another unit system.
|
||||
|
||||
## Sink Fields
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|:---:|---|
|
||||
| `name` | yes | Unique sink name. Routes refer to this value. |
|
||||
| `driver` | yes | Sink driver name. |
|
||||
| `params` | driver-specific | Sink parameters. |
|
||||
|
||||
## Sink Drivers
|
||||
|
||||
### `stdout`
|
||||
|
||||
Prints each event as JSON to stdout.
|
||||
|
||||
```yaml
|
||||
sinks:
|
||||
- name: stdout
|
||||
driver: stdout
|
||||
params: {}
|
||||
```
|
||||
|
||||
### `nats`
|
||||
|
||||
Publishes each event as JSON to a NATS subject.
|
||||
|
||||
| Param | Required | Description |
|
||||
|---|:---:|---|
|
||||
| `url` | yes | NATS server URL, such as `nats://localhost:4222`. |
|
||||
| `subject` | yes | Subject to publish events to. |
|
||||
|
||||
### `postgres`
|
||||
|
||||
Writes supported canonical weather events to Postgres using weatherfeeder's
|
||||
registered schema mapping. The table contract is documented in
|
||||
[Postgres integration](integrations/postgres.md).
|
||||
|
||||
| Param | Required | Description |
|
||||
|---|:---:|---|
|
||||
| `uri` | yes | PostgreSQL connection URI. |
|
||||
| `username` | yes | Database username. |
|
||||
| `password` | yes | Database password. |
|
||||
| `prune` | no | Retention window. If set, rows older than the window are pruned on each write transaction. |
|
||||
|
||||
`prune` accepts Go duration strings such as `72h`, plus day and week suffixes
|
||||
such as `3d` and `2w`.
|
||||
|
||||
## Routes
|
||||
|
||||
Routes connect event kinds to sinks:
|
||||
|
||||
```yaml
|
||||
routes:
|
||||
- sink: stdout
|
||||
kinds: ["observation", "alert"]
|
||||
```
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|:---:|---|
|
||||
| `sink` | yes | Name of a configured sink. |
|
||||
| `kinds` | no | Event kinds to send to that sink. Omit or use an empty list to match all kinds. |
|
||||
|
||||
Route `kinds` values are trimmed and lowercased by the dispatcher. Blank entries
|
||||
are rejected.
|
||||
|
||||
## Duration Formats
|
||||
|
||||
Top-level source `every` accepts:
|
||||
|
||||
- Go duration strings such as `30s`, `10m`, or `1h`;
|
||||
- integer values, interpreted as minutes;
|
||||
- numeric strings such as `"15"`, also interpreted as minutes.
|
||||
|
||||
HTTP param durations such as `http_timeout` accept Go duration strings. Numeric
|
||||
values and numeric strings are interpreted as seconds.
|
||||
|
||||
Postgres `prune` must be a string duration.
|
||||
|
||||
## Secrets
|
||||
|
||||
The config file is read directly from disk and has no built-in secret expansion.
|
||||
Keep real credentials out of repository-tracked configs. Use deployment tooling
|
||||
to render `config.yml` with the needed secret values before starting the daemon.
|
||||
|
||||
## Maintained Examples
|
||||
|
||||
- [Minimal stdout config](../examples/config.minimal.yml)
|
||||
- [NATS publishing config](../examples/config.nats.yml)
|
||||
- [Postgres persistence config](../examples/config.postgres.yml)
|
||||
238
docs/integrations/events.md
Normal file
238
docs/integrations/events.md
Normal file
@@ -0,0 +1,238 @@
|
||||
# Event Wire Contract
|
||||
|
||||
This document is the canonical JSON contract for events emitted by
|
||||
`weatherfeeder` JSON sinks, including stdout and NATS. Postgres stores the same
|
||||
event envelope fields in parent table columns; see
|
||||
[Postgres integration](postgres.md).
|
||||
|
||||
Downstream consumers should read the envelope, switch on `schema`, and decode
|
||||
`payload` according to that schema.
|
||||
|
||||
## Envelope
|
||||
|
||||
Every emitted event is a JSON object with these fields:
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---|:---:|---|
|
||||
| `id` | string | yes | Stable event identifier. Treat as opaque. |
|
||||
| `kind` | string | yes | Routing kind, such as `observation` or `alert`. |
|
||||
| `source` | string | yes | Configured source name. |
|
||||
| `emitted_at` | timestamp | yes | When the daemon emitted the event. |
|
||||
| `effective_at` | timestamp | no | Timestamp the payload is about, when known. |
|
||||
| `schema` | string | no | Schema identifier. Weatherfeeder sources and normalizers set this. |
|
||||
| `payload` | object, array, string, or scalar | yes | Schema-specific payload. |
|
||||
|
||||
Timestamps are JSON strings using Go `time.Time` JSON encoding, which is
|
||||
RFC3339Nano-compatible. Weatherfeeder normalizers use UTC timestamps for
|
||||
canonical payloads.
|
||||
|
||||
## Kinds And Schemas
|
||||
|
||||
Canonical schemas emitted after normalization:
|
||||
|
||||
| Kind | Schema | Payload |
|
||||
|---|---|---|
|
||||
| `observation` | `weather.observation.v1` | `WeatherObservation` |
|
||||
| `forecast` | `weather.forecast.v1` | `WeatherForecastRun` |
|
||||
| `forecast_discussion` | `weather.forecast_discussion.v1` | `WeatherForecastDiscussion` |
|
||||
| `weather_story` | `weather.weather_story.v1` | `WeatherStoryRun` |
|
||||
| `alert` | `weather.alert.v1` | `WeatherAlertRun` |
|
||||
|
||||
Raw upstream schemas emitted by current sources:
|
||||
|
||||
| Kind | Schema | Payload |
|
||||
|---|---|---|
|
||||
| `observation` | `raw.nws.observation.v1` | NWS observation JSON |
|
||||
| `observation` | `raw.openmeteo.current.v1` | Open-Meteo current JSON |
|
||||
| `observation` | `raw.openweather.current.v1` | OpenWeather current JSON |
|
||||
| `forecast` | `raw.nws.hourly.forecast.v1` | NWS hourly forecast JSON |
|
||||
| `forecast` | `raw.nws.narrative.forecast.v1` | NWS narrative forecast JSON |
|
||||
| `forecast_discussion` | `raw.nws.forecast_discussion.v1` | NWS forecast discussion HTML string |
|
||||
| `weather_story` | `raw.nws.weatherstories.v1` | NWS weather stories JSON |
|
||||
| `forecast` | `raw.openmeteo.hourly.forecast.v1` | Open-Meteo hourly forecast JSON |
|
||||
| `alert` | `raw.nws.alerts.v1` | NWS alerts JSON |
|
||||
|
||||
`standards.SchemaRawOpenWeatherHourlyForecastV1` exists in code, but no current
|
||||
registered source emits it.
|
||||
|
||||
## Shared Conventions
|
||||
|
||||
- Canonical numeric measurements use metric units.
|
||||
- Floating-point values in canonical payloads are rounded to 4 digits after the
|
||||
decimal point during normalization.
|
||||
- Optional fields use JSON `omitempty`; absent fields should be treated as
|
||||
unknown.
|
||||
- `conditionCode` is a WMO weather interpretation code. Unknown observation
|
||||
conditions use `-1`. Forecast period `conditionCode` is optional.
|
||||
- Additive fields are compatible within a schema version. Removing, renaming, or
|
||||
changing the meaning of a field requires a new schema identifier.
|
||||
|
||||
## `weather.observation.v1`
|
||||
|
||||
Payload type: `WeatherObservation`.
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---|:---:|---|
|
||||
| `stationId` | string | no | Provider station/location identifier. |
|
||||
| `stationName` | string | no | Human station name. |
|
||||
| `timestamp` | timestamp | yes | Observation timestamp. |
|
||||
| `conditionCode` | integer | yes | WMO code; `-1` means unknown. |
|
||||
| `isDay` | boolean | no | Day/night hint. |
|
||||
| `textDescription` | string | no | Short human description. |
|
||||
| `temperatureC` | number | no | Celsius. |
|
||||
| `dewpointC` | number | no | Celsius. |
|
||||
| `windDirectionDegrees` | number | no | Degrees. |
|
||||
| `windSpeedKmh` | number | no | Kilometers per hour. |
|
||||
| `windGustKmh` | number | no | Kilometers per hour. |
|
||||
| `barometricPressurePa` | number | no | Pascals. |
|
||||
| `visibilityMeters` | number | no | Meters. |
|
||||
| `relativeHumidityPercent` | number | no | Percent from 0 to 100. |
|
||||
| `apparentTemperatureC` | number | no | Celsius. |
|
||||
| `presentWeather` | array | no | Provider-specific present weather fragments. |
|
||||
|
||||
`presentWeather[]` entries contain optional `raw` objects.
|
||||
|
||||
## `weather.forecast.v1`
|
||||
|
||||
Payload type: `WeatherForecastRun`.
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---|:---:|---|
|
||||
| `locationId` | string | no | Provider location identifier. |
|
||||
| `locationName` | string | no | Human location name. |
|
||||
| `issuedAt` | timestamp | yes | When the forecast run was generated or issued. |
|
||||
| `updatedAt` | timestamp | no | Subsequent provider update time. |
|
||||
| `product` | string | yes | Current emitted values are `hourly` and `narrative`. |
|
||||
| `latitude` | number | no | Degrees. |
|
||||
| `longitude` | number | no | Degrees. |
|
||||
| `elevationMeters` | number | no | Meters. |
|
||||
| `periods` | array | yes | Ordered forecast periods. |
|
||||
|
||||
`periods[]` entries:
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---|:---:|---|
|
||||
| `startTime` | timestamp | yes | Period start. |
|
||||
| `endTime` | timestamp | yes | Period end. |
|
||||
| `name` | string | no | Human label. |
|
||||
| `isDay` | boolean | no | Day/night hint. |
|
||||
| `conditionCode` | integer | no | WMO code when applicable. |
|
||||
| `textDescription` | string | no | Human summary. |
|
||||
| `temperatureC` | number | no | Celsius. |
|
||||
| `temperatureCMin` | number | no | Celsius. |
|
||||
| `temperatureCMax` | number | no | Celsius. |
|
||||
| `dewpointC` | number | no | Celsius. |
|
||||
| `relativeHumidityPercent` | number | no | Percent from 0 to 100. |
|
||||
| `windDirectionDegrees` | number | no | Degrees. |
|
||||
| `windSpeedKmh` | number | no | Kilometers per hour. |
|
||||
| `windGustKmh` | number | no | Kilometers per hour. |
|
||||
| `barometricPressurePa` | number | no | Pascals. |
|
||||
| `visibilityMeters` | number | no | Meters. |
|
||||
| `apparentTemperatureC` | number | no | Celsius. |
|
||||
| `cloudCoverPercent` | number | no | Percent from 0 to 100. |
|
||||
| `probabilityOfPrecipitationPercent` | number | no | Percent from 0 to 100. |
|
||||
| `precipitationAmountMm` | number | no | Liquid-equivalent millimeters. |
|
||||
| `snowfallDepthMm` | number | no | Millimeters. |
|
||||
| `uvIndex` | number | no | Unitless index. |
|
||||
|
||||
## `weather.forecast_discussion.v1`
|
||||
|
||||
Payload type: `WeatherForecastDiscussion`.
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---|:---:|---|
|
||||
| `officeId` | string | no | NWS office identifier. |
|
||||
| `officeName` | string | no | Office name. |
|
||||
| `product` | string | yes | Current value is `afd`. |
|
||||
| `issuedAt` | timestamp | yes | Bulletin issue time. |
|
||||
| `updatedAt` | timestamp | no | Subsequent update time. |
|
||||
| `keyMessages` | array of strings | no | Extracted key messages. |
|
||||
| `shortTerm` | object | no | Short-term section. |
|
||||
| `longTerm` | object | no | Long-term section. |
|
||||
|
||||
`shortTerm` and `longTerm` sections contain optional `qualifier`, `issuedAt`,
|
||||
and `text` fields.
|
||||
|
||||
## `weather.weather_story.v1`
|
||||
|
||||
Payload type: `WeatherStoryRun`.
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---|:---:|---|
|
||||
| `officeId` | string | no | NWS office identifier. |
|
||||
| `asOf` | timestamp | yes | Snapshot time. |
|
||||
| `stories` | array | yes | Ordered story cards. |
|
||||
|
||||
`stories[]` entries:
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---|:---:|---|
|
||||
| `officeId` | string | no | Office identifier. |
|
||||
| `startTime` | timestamp | yes | Story start. |
|
||||
| `endTime` | timestamp | yes | Story end. |
|
||||
| `updatedAt` | timestamp | yes | Story update time. |
|
||||
| `title` | string | no | Story title. |
|
||||
| `description` | string | no | Story description. |
|
||||
| `altText` | string | no | Image alternate text. |
|
||||
| `priority` | boolean | yes | Provider priority flag. |
|
||||
| `order` | integer | yes | Provider display order. |
|
||||
| `downloadUrl` | string | no | Story image URL. |
|
||||
|
||||
## `weather.alert.v1`
|
||||
|
||||
Payload type: `WeatherAlertRun`.
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---|:---:|---|
|
||||
| `locationId` | string | no | Provider location identifier. |
|
||||
| `locationName` | string | no | Human location name. |
|
||||
| `asOf` | timestamp | yes | Snapshot time. |
|
||||
| `latitude` | number | no | Degrees. |
|
||||
| `longitude` | number | no | Degrees. |
|
||||
| `alerts` | array | yes | Active alerts. |
|
||||
|
||||
`alerts[]` entries:
|
||||
|
||||
| Field | Type | Required | Notes |
|
||||
|---|---|:---:|---|
|
||||
| `id` | string | yes | Provider-stable alert identifier. |
|
||||
| `event` | string | no | Alert event label. |
|
||||
| `headline` | string | no | Alert headline. |
|
||||
| `severity` | string | no | Provider severity. |
|
||||
| `urgency` | string | no | Provider urgency. |
|
||||
| `certainty` | string | no | Provider certainty. |
|
||||
| `status` | string | no | Alert status. |
|
||||
| `messageType` | string | no | Alert message type. |
|
||||
| `category` | string | no | Alert category. |
|
||||
| `response` | string | no | Recommended response. |
|
||||
| `description` | string | no | Alert description. |
|
||||
| `instruction` | string | no | Alert instruction. |
|
||||
| `sent` | timestamp | no | Provider sent time. |
|
||||
| `effective` | timestamp | no | Effective time. |
|
||||
| `onset` | timestamp | no | Onset time. |
|
||||
| `expires` | timestamp | no | Expiration time. |
|
||||
| `areaDescription` | string | no | Affected area description. |
|
||||
| `senderName` | string | no | Provider sender name. |
|
||||
| `references` | array | no | Related alerts. |
|
||||
|
||||
`references[]` entries contain optional `id`, `identifier`, `sender`, and
|
||||
`sent` fields.
|
||||
|
||||
## Compact Example
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "NWSObservationKSTL:2026-06-10T12:00:00Z",
|
||||
"kind": "observation",
|
||||
"source": "NWSObservationKSTL",
|
||||
"emitted_at": "2026-06-10T12:00:05Z",
|
||||
"effective_at": "2026-06-10T12:00:00Z",
|
||||
"schema": "weather.observation.v1",
|
||||
"payload": {
|
||||
"stationId": "KSTL",
|
||||
"timestamp": "2026-06-10T12:00:00Z",
|
||||
"conditionCode": 0,
|
||||
"temperatureC": 22.5
|
||||
}
|
||||
}
|
||||
```
|
||||
119
docs/integrations/nws.md
Normal file
119
docs/integrations/nws.md
Normal file
@@ -0,0 +1,119 @@
|
||||
# NWS Integration Notes
|
||||
|
||||
## Purpose
|
||||
|
||||
This document describes the NWS products that `weatherfeeder` currently polls
|
||||
and normalizes. It is for developers and operators maintaining NWS source URLs,
|
||||
normalizers, fixtures, and tests.
|
||||
|
||||
General config syntax belongs in [configuration](../config.md). Emitted JSON
|
||||
events are documented in [event wire contract](events.md).
|
||||
|
||||
## Implemented Drivers
|
||||
|
||||
| Driver | Kind | Raw schema | Canonical schema |
|
||||
| --- | --- | --- | --- |
|
||||
| `nws_observation` | `observation` | `raw.nws.observation.v1` | `weather.observation.v1` |
|
||||
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` | `weather.alert.v1` |
|
||||
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
|
||||
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
|
||||
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
|
||||
|
||||
## Config Requirements
|
||||
|
||||
All NWS drivers require HTTP source params:
|
||||
|
||||
- `url`
|
||||
- `user_agent`
|
||||
|
||||
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
|
||||
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
|
||||
an upstream `304 Not Modified` response emits no event for that poll.
|
||||
|
||||
NWS expects a descriptive `User-Agent`. Do not use anonymous or placeholder
|
||||
contact values in production configs.
|
||||
|
||||
## Upstream Shapes Used
|
||||
|
||||
`nws_observation` expects the latest station observation GeoJSON shape. The
|
||||
normalizer uses fields under `properties` such as `stationId`, `stationName`,
|
||||
`timestamp`, `textDescription`, measured values, `presentWeather`, and
|
||||
`cloudLayers`, plus point geometry for day/night inference.
|
||||
|
||||
`nws_alerts` expects an alerts FeatureCollection. The normalizer uses the
|
||||
collection `updated` timestamp, `title`, each feature ID, alert classification
|
||||
fields, narrative fields, timing fields, sender fields, and references.
|
||||
|
||||
`nws_forecast_hourly` and `nws_forecast_narrative` expect gridpoint forecast
|
||||
GeoJSON with `properties.generatedAt`, `properties.updateTime`, elevation,
|
||||
polygon geometry, and ordered `periods`.
|
||||
|
||||
`nws_forecast_discussion` expects an HTML page containing the discussion text in
|
||||
a `<pre>` block. The provider helper extracts office identity, product, issue
|
||||
time, update time, key messages, and short/long term sections.
|
||||
|
||||
`nws_weatherstories` expects a JSON response with a `stories` array. The
|
||||
normalizer uses office ID, start/end/update times, title, description, alt text,
|
||||
priority, order, and download URL.
|
||||
|
||||
## Accept Headers
|
||||
|
||||
NWS JSON sources request:
|
||||
|
||||
```text
|
||||
application/geo+json, application/json
|
||||
```
|
||||
|
||||
The forecast discussion source requests:
|
||||
|
||||
```text
|
||||
text/html, application/xhtml+xml
|
||||
```
|
||||
|
||||
## Effective Time
|
||||
|
||||
Source events set `effective_at` from the best metadata available:
|
||||
|
||||
- observations: `properties.timestamp`;
|
||||
- alerts: collection `updated`, otherwise latest per-alert timestamp;
|
||||
- hourly and narrative forecasts: `properties.generatedAt`, otherwise update
|
||||
time;
|
||||
- forecast discussions: parsed issue time;
|
||||
- weather stories: latest story update time, otherwise latest story start time.
|
||||
|
||||
Normalizers use canonical payload time as the normalized event effective time.
|
||||
Alerts and weather stories fall back to the incoming event envelope when the
|
||||
payload does not provide a better snapshot time.
|
||||
|
||||
## Mapping Notes
|
||||
|
||||
Observations preserve raw `presentWeather` fragments and infer WMO condition
|
||||
codes from METAR phenomena, provider text, and cloud-layer fallback. Sea-level
|
||||
pressure is preferred over barometric pressure when present.
|
||||
|
||||
Hourly forecasts infer WMO condition codes from `shortForecast` and icon tokens.
|
||||
Narrative forecasts preserve text but intentionally leave period condition codes
|
||||
unset. Forecast temperatures are converted to Celsius when NWS supplies
|
||||
Fahrenheit, and wind speed strings are converted to kilometers per hour.
|
||||
|
||||
Alert timing fields are parsed best-effort. Invalid per-alert timestamps are
|
||||
left unset rather than failing the whole alert run. Missing alert IDs are
|
||||
synthesized from the run snapshot time and array position.
|
||||
|
||||
Forecast discussion parsing requires an issue time. Weather story entries require
|
||||
start time, end time, and update time.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Constructor validation failures stop daemon startup. Polling failures are
|
||||
returned to the scheduler. JSON sources still emit raw payloads when only
|
||||
minimal metadata decoding fails. Forecast discussion polling fails if the HTML
|
||||
cannot be parsed enough to determine the issue time.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/sources/nws/*_test.go`
|
||||
- `internal/normalizers/nws/*_test.go`
|
||||
- `internal/providers/nws/*_test.go`
|
||||
- fixtures under `internal/providers/nws/testdata`
|
||||
101
docs/integrations/openmeteo.md
Normal file
101
docs/integrations/openmeteo.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# Open-Meteo Integration Notes
|
||||
|
||||
## Purpose
|
||||
|
||||
This document describes the Open-Meteo API usage currently implemented by
|
||||
`weatherfeeder`. It is for developers and operators maintaining Open-Meteo
|
||||
source URLs, normalizers, fixtures, and tests.
|
||||
|
||||
General config syntax belongs in [configuration](../config.md). Emitted JSON
|
||||
events are documented in [event wire contract](events.md).
|
||||
|
||||
## Implemented Drivers
|
||||
|
||||
| Driver | Kind | Raw schema | Canonical schema |
|
||||
| --- | --- | --- | --- |
|
||||
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` | `weather.observation.v1` |
|
||||
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||
|
||||
## Config Requirements
|
||||
|
||||
Both drivers require HTTP source params:
|
||||
|
||||
- `url`
|
||||
- `user_agent`
|
||||
|
||||
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
|
||||
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
|
||||
an upstream `304 Not Modified` response emits no event for that poll.
|
||||
|
||||
## Upstream Shapes Used
|
||||
|
||||
`openmeteo_observation` expects a JSON response with top-level location/timezone
|
||||
metadata and a `current` object. The normalizer uses:
|
||||
|
||||
- `latitude`, `longitude`, `timezone`, `utc_offset_seconds`;
|
||||
- `current.time`;
|
||||
- current temperature, apparent temperature, relative humidity, weather code,
|
||||
wind speed/direction/gusts, pressure, and `is_day`.
|
||||
|
||||
`openmeteo_forecast` expects top-level location/timezone metadata and an
|
||||
array-oriented `hourly` object. The normalizer uses:
|
||||
|
||||
- `hourly.time`;
|
||||
- hourly temperature, apparent temperature, dew point, relative humidity,
|
||||
precipitation probability, precipitation amount, snowfall, weather code,
|
||||
pressure, wind speed/direction/gusts, `is_day`, cloud cover, visibility, and
|
||||
UV index.
|
||||
|
||||
Open-Meteo field presence is allowed to vary. Missing optional arrays produce
|
||||
nil canonical fields for the affected periods.
|
||||
|
||||
## Accept Header
|
||||
|
||||
Open-Meteo sources request:
|
||||
|
||||
```text
|
||||
application/json
|
||||
```
|
||||
|
||||
## Time Handling
|
||||
|
||||
Open-Meteo timestamps often omit an explicit offset. The provider helper parses
|
||||
times by using the returned `timezone` or `utc_offset_seconds` when needed.
|
||||
|
||||
Observation source events set `effective_at` from `current.time` when it can be
|
||||
parsed. Hourly forecast source events prefer `current.time`, then the first
|
||||
non-empty `hourly.time` entry.
|
||||
|
||||
The hourly forecast normalizer sets canonical `issuedAt` from the incoming event
|
||||
`emitted_at` when present, otherwise from the first hourly period start.
|
||||
Normalized forecast `effective_at` matches `issuedAt`.
|
||||
|
||||
## Mapping Notes
|
||||
|
||||
Open-Meteo is not a station feed. Weatherfeeder synthesizes canonical
|
||||
station/location IDs from latitude and longitude when both are available.
|
||||
|
||||
Open-Meteo weather codes are WMO codes and are treated as authoritative.
|
||||
Canonical text is derived from the WMO code and day/night hint.
|
||||
|
||||
Wind speed and gust fields are treated as kilometers per hour. Pressure values
|
||||
are treated as hPa and converted to Pa. Snowfall values are treated as
|
||||
centimeters and converted to millimeters.
|
||||
|
||||
Hourly forecast period end time is the next period start. The last period uses
|
||||
the previous interval length, or one hour when there is no previous interval.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Constructor validation failures stop daemon startup. Polling failures are
|
||||
returned to the scheduler. Metadata decoding failures in sources still allow raw
|
||||
payload emission when the HTTP response itself succeeded.
|
||||
|
||||
Normalization fails when required time data is missing or invalid, such as an
|
||||
empty `hourly.time` array for hourly forecasts.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/sources/openmeteo/source_test.go`
|
||||
- `internal/normalizers/openmeteo/*_test.go`
|
||||
- `internal/providers/openmeteo/*_test.go`
|
||||
101
docs/integrations/openweather.md
Normal file
101
docs/integrations/openweather.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# OpenWeather Integration Notes
|
||||
|
||||
## Purpose
|
||||
|
||||
This document describes the OpenWeather current-weather usage implemented by
|
||||
`weatherfeeder`. It is for developers and operators maintaining OpenWeather
|
||||
source URLs, normalizers, fixtures, and tests.
|
||||
|
||||
General config syntax belongs in [configuration](../config.md). Emitted JSON
|
||||
events are documented in [event wire contract](events.md).
|
||||
|
||||
## Implemented Driver
|
||||
|
||||
| Driver | Kind | Raw schema | Canonical schema |
|
||||
| --- | --- | --- | --- |
|
||||
| `openweather_observation` | `observation` | `raw.openweather.current.v1` | `weather.observation.v1` |
|
||||
|
||||
Only current-weather observation polling is registered for OpenWeather.
|
||||
|
||||
## Config Requirements
|
||||
|
||||
The driver requires HTTP source params:
|
||||
|
||||
- `url`
|
||||
- `user_agent`
|
||||
|
||||
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
|
||||
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
|
||||
an upstream `304 Not Modified` response emits no event for that poll.
|
||||
|
||||
The configured URL must include:
|
||||
|
||||
```text
|
||||
units=metric
|
||||
```
|
||||
|
||||
Startup fails if `units` is omitted or set to another value. Keep OpenWeather
|
||||
API keys out of committed configs. Use local config management or deployment
|
||||
secrets for the `appid` query parameter.
|
||||
|
||||
## Upstream Shape Used
|
||||
|
||||
The source emits the full current-weather JSON payload as a raw event. The
|
||||
normalizer uses:
|
||||
|
||||
- `coord.lat`, `coord.lon`;
|
||||
- primary `weather[0]` condition ID, description, and icon;
|
||||
- `main.temp`, `main.feels_like`, `main.pressure`, `main.humidity`, and
|
||||
optional `main.sea_level`;
|
||||
- `visibility`;
|
||||
- `wind.speed`, `wind.deg`, and `wind.gust`;
|
||||
- `dt`;
|
||||
- `sys.sunrise` and `sys.sunset`;
|
||||
- `id` and `name`.
|
||||
|
||||
## Accept Header
|
||||
|
||||
OpenWeather sources request:
|
||||
|
||||
```text
|
||||
application/json
|
||||
```
|
||||
|
||||
## Time Handling
|
||||
|
||||
Source events set `effective_at` from `dt` when it is present and positive.
|
||||
The normalizer also uses `dt` as the canonical observation timestamp and
|
||||
normalized effective time.
|
||||
|
||||
## Mapping Notes
|
||||
|
||||
Metric units are required so canonical unit conversion is deterministic:
|
||||
|
||||
- `main.temp` and `main.feels_like` are treated as Celsius;
|
||||
- `wind.speed` and `wind.gust` are treated as meters per second and converted to
|
||||
kilometers per hour;
|
||||
- pressure values are treated as hPa and converted to Pa.
|
||||
|
||||
The primary condition is `weather[0]`. OpenWeather condition IDs are mapped into
|
||||
the canonical WMO code vocabulary. The human text description is preserved from
|
||||
the provider description.
|
||||
|
||||
Day/night is inferred from the OpenWeather icon suffix when available, otherwise
|
||||
from sunrise and sunset bounds.
|
||||
|
||||
The station ID uses the OpenWeather city ID when present. If no city ID is
|
||||
present, weatherfeeder synthesizes an ID from coordinates. The station name uses
|
||||
the provider `name`, falling back to `OpenWeatherMap` when blank.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Constructor validation failures stop daemon startup. Polling also re-checks the
|
||||
metric-unit requirement before fetching. HTTP failures are returned to the
|
||||
scheduler. Metadata decoding failures in the source still allow raw payload
|
||||
emission when the HTTP response itself succeeded.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/sources/openweather/source_test.go`
|
||||
- `internal/normalizers/openweather/*_test.go`
|
||||
- `internal/providers/openweather/*_test.go`
|
||||
403
docs/integrations/postgres.md
Normal file
403
docs/integrations/postgres.md
Normal file
@@ -0,0 +1,403 @@
|
||||
# Postgres Integration
|
||||
|
||||
This document is the canonical table contract for the optional `postgres` sink.
|
||||
It describes the schema created and written by weatherfeeder through feedkit's
|
||||
Postgres sink.
|
||||
|
||||
Configure the sink as described in [configuration](../config.md#postgres).
|
||||
|
||||
## Initialization And Writes
|
||||
|
||||
At startup, each configured Postgres sink opens the database and runs
|
||||
`CREATE TABLE IF NOT EXISTS` for every weatherfeeder table, followed by
|
||||
`CREATE INDEX IF NOT EXISTS` for every configured index.
|
||||
|
||||
This initialization creates missing tables and indexes only. It does not alter
|
||||
existing tables, migrate column definitions, drop old objects, or backfill data.
|
||||
Schema changes require operator-managed database migration.
|
||||
|
||||
Events are mapped only for canonical weather schemas:
|
||||
|
||||
- `weather.observation.v1`
|
||||
- `weather.forecast.v1`
|
||||
- `weather.forecast_discussion.v1`
|
||||
- `weather.weather_story.v1`
|
||||
- `weather.alert.v1`
|
||||
|
||||
Unsupported schemas produce no writes for this sink. Mapped events are inserted
|
||||
transactionally. Inserts use ordinary `INSERT`; duplicate primary keys fail the
|
||||
write.
|
||||
|
||||
## Shared Envelope Columns
|
||||
|
||||
Parent tables store the feed event envelope:
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `event_id` | `TEXT` | no | `event.id` |
|
||||
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||
| `event_source` | `TEXT` | no | `event.source` |
|
||||
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||
|
||||
## Table Overview
|
||||
|
||||
| Table | Primary key | Prune column |
|
||||
|---|---|---|
|
||||
| `observations` | `event_id` | `observed_at` |
|
||||
| `observation_present_weather` | `event_id`, `weather_index` | `observed_at` |
|
||||
| `forecasts` | `event_id` | `issued_at` |
|
||||
| `forecast_periods` | `run_event_id`, `period_index` | `issued_at` |
|
||||
| `forecast_discussions` | `event_id` | `issued_at` |
|
||||
| `forecast_discussion_key_messages` | `run_event_id`, `message_index` | `issued_at` |
|
||||
| `weather_story_runs` | `event_id` | `as_of` |
|
||||
| `weather_stories` | `run_event_id`, `story_index` | `as_of` |
|
||||
| `alert_runs` | `event_id` | `as_of` |
|
||||
| `alerts` | `run_event_id`, `alert_index` | `as_of` |
|
||||
| `alert_references` | `run_event_id`, `alert_index`, `reference_index` | `as_of` |
|
||||
|
||||
## Table Contract
|
||||
|
||||
### `observations`
|
||||
|
||||
Primary key: `event_id`
|
||||
|
||||
Prune column: `observed_at`
|
||||
|
||||
Indexes:
|
||||
|
||||
- `idx_wf_obs_station_observed_at` on `station_id`, `observed_at`
|
||||
- `idx_wf_obs_observed_at` on `observed_at`
|
||||
- `idx_wf_obs_condition_code` on `condition_code`
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `event_id` | `TEXT` | no | `event.id` |
|
||||
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||
| `event_source` | `TEXT` | no | `event.source` |
|
||||
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||
| `station_id` | `TEXT` | yes | `payload.stationId` |
|
||||
| `station_name` | `TEXT` | yes | `payload.stationName` |
|
||||
| `observed_at` | `TIMESTAMPTZ` | no | `payload.timestamp` |
|
||||
| `condition_code` | `INTEGER` | no | `payload.conditionCode` |
|
||||
| `is_day` | `BOOLEAN` | yes | `payload.isDay` |
|
||||
| `text_description` | `TEXT` | yes | `payload.textDescription` |
|
||||
| `temperature_c` | `DOUBLE PRECISION` | yes | `payload.temperatureC` |
|
||||
| `dewpoint_c` | `DOUBLE PRECISION` | yes | `payload.dewpointC` |
|
||||
| `wind_direction_degrees` | `DOUBLE PRECISION` | yes | `payload.windDirectionDegrees` |
|
||||
| `wind_speed_kmh` | `DOUBLE PRECISION` | yes | `payload.windSpeedKmh` |
|
||||
| `wind_gust_kmh` | `DOUBLE PRECISION` | yes | `payload.windGustKmh` |
|
||||
| `barometric_pressure_pa` | `DOUBLE PRECISION` | yes | `payload.barometricPressurePa` |
|
||||
| `visibility_meters` | `DOUBLE PRECISION` | yes | `payload.visibilityMeters` |
|
||||
| `relative_humidity_percent` | `DOUBLE PRECISION` | yes | `payload.relativeHumidityPercent` |
|
||||
| `apparent_temperature_c` | `DOUBLE PRECISION` | yes | `payload.apparentTemperatureC` |
|
||||
|
||||
### `observation_present_weather`
|
||||
|
||||
Primary key: `event_id`, `weather_index`
|
||||
|
||||
Prune column: `observed_at`
|
||||
|
||||
Foreign key: `event_id` references `observations(event_id)` with cascade delete.
|
||||
|
||||
Index: `idx_wf_obs_present_observed_at` on `observed_at`
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `event_id` | `TEXT REFERENCES observations(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||
| `weather_index` | `INTEGER` | no | `payload.presentWeather[]` index. |
|
||||
| `observed_at` | `TIMESTAMPTZ` | no | `payload.timestamp` |
|
||||
| `raw_text` | `TEXT` | yes | Compact JSON text from `payload.presentWeather[].raw` |
|
||||
|
||||
### `forecasts`
|
||||
|
||||
Primary key: `event_id`
|
||||
|
||||
Prune column: `issued_at`
|
||||
|
||||
Indexes:
|
||||
|
||||
- `idx_wf_fc_location_product_issued_at` on `location_id`, `product`, `issued_at`
|
||||
- `idx_wf_fc_issued_at` on `issued_at`
|
||||
- `idx_wf_fc_product_issued_at` on `product`, `issued_at`
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `event_id` | `TEXT` | no | `event.id` |
|
||||
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||
| `event_source` | `TEXT` | no | `event.source` |
|
||||
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||
| `location_id` | `TEXT` | yes | `payload.locationId` |
|
||||
| `location_name` | `TEXT` | yes | `payload.locationName` |
|
||||
| `issued_at` | `TIMESTAMPTZ` | no | `payload.issuedAt` |
|
||||
| `updated_at` | `TIMESTAMPTZ` | yes | `payload.updatedAt` |
|
||||
| `product` | `TEXT` | no | `payload.product` |
|
||||
| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` |
|
||||
| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` |
|
||||
| `elevation_meters` | `DOUBLE PRECISION` | yes | `payload.elevationMeters` |
|
||||
| `period_count` | `INTEGER` | no | `len(payload.periods)` |
|
||||
|
||||
### `forecast_periods`
|
||||
|
||||
Primary key: `run_event_id`, `period_index`
|
||||
|
||||
Prune column: `issued_at`
|
||||
|
||||
Foreign key: `run_event_id` references `forecasts(event_id)` with cascade delete.
|
||||
|
||||
Indexes:
|
||||
|
||||
- `idx_wf_fc_period_start_time` on `start_time`
|
||||
- `idx_wf_fc_period_end_time` on `end_time`
|
||||
- `idx_wf_fc_period_run_start` on `run_event_id`, `start_time`
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `run_event_id` | `TEXT REFERENCES forecasts(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||
| `period_index` | `INTEGER` | no | `payload.periods[]` index. |
|
||||
| `issued_at` | `TIMESTAMPTZ` | no | Parent `payload.issuedAt` |
|
||||
| `start_time` | `TIMESTAMPTZ` | no | `payload.periods[].startTime` |
|
||||
| `end_time` | `TIMESTAMPTZ` | no | `payload.periods[].endTime` |
|
||||
| `name` | `TEXT` | yes | `payload.periods[].name` |
|
||||
| `is_day` | `BOOLEAN` | yes | `payload.periods[].isDay` |
|
||||
| `condition_code` | `INTEGER` | yes | `payload.periods[].conditionCode` |
|
||||
| `text_description` | `TEXT` | yes | `payload.periods[].textDescription` |
|
||||
| `temperature_c` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureC` |
|
||||
| `temperature_c_min` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureCMin` |
|
||||
| `temperature_c_max` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureCMax` |
|
||||
| `dewpoint_c` | `DOUBLE PRECISION` | yes | `payload.periods[].dewpointC` |
|
||||
| `relative_humidity_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].relativeHumidityPercent` |
|
||||
| `wind_direction_degrees` | `DOUBLE PRECISION` | yes | `payload.periods[].windDirectionDegrees` |
|
||||
| `wind_speed_kmh` | `DOUBLE PRECISION` | yes | `payload.periods[].windSpeedKmh` |
|
||||
| `wind_gust_kmh` | `DOUBLE PRECISION` | yes | `payload.periods[].windGustKmh` |
|
||||
| `barometric_pressure_pa` | `DOUBLE PRECISION` | yes | `payload.periods[].barometricPressurePa` |
|
||||
| `visibility_meters` | `DOUBLE PRECISION` | yes | `payload.periods[].visibilityMeters` |
|
||||
| `apparent_temperature_c` | `DOUBLE PRECISION` | yes | `payload.periods[].apparentTemperatureC` |
|
||||
| `cloud_cover_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].cloudCoverPercent` |
|
||||
| `probability_of_precipitation_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].probabilityOfPrecipitationPercent` |
|
||||
| `precipitation_amount_mm` | `DOUBLE PRECISION` | yes | `payload.periods[].precipitationAmountMm` |
|
||||
| `snowfall_depth_mm` | `DOUBLE PRECISION` | yes | `payload.periods[].snowfallDepthMm` |
|
||||
| `uv_index` | `DOUBLE PRECISION` | yes | `payload.periods[].uvIndex` |
|
||||
|
||||
### `forecast_discussions`
|
||||
|
||||
Primary key: `event_id`
|
||||
|
||||
Prune column: `issued_at`
|
||||
|
||||
Indexes:
|
||||
|
||||
- `idx_wf_discussion_office_product_issued_at` on `office_id`, `product`, `issued_at`
|
||||
- `idx_wf_discussion_issued_at` on `issued_at`
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `event_id` | `TEXT` | no | `event.id` |
|
||||
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||
| `event_source` | `TEXT` | no | `event.source` |
|
||||
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||
| `office_id` | `TEXT` | yes | `payload.officeId` |
|
||||
| `office_name` | `TEXT` | yes | `payload.officeName` |
|
||||
| `issued_at` | `TIMESTAMPTZ` | no | `payload.issuedAt` |
|
||||
| `updated_at` | `TIMESTAMPTZ` | yes | `payload.updatedAt` |
|
||||
| `product` | `TEXT` | no | `payload.product` |
|
||||
| `short_term_qualifier` | `TEXT` | yes | `payload.shortTerm.qualifier` |
|
||||
| `short_term_issued_at` | `TIMESTAMPTZ` | yes | `payload.shortTerm.issuedAt` |
|
||||
| `short_term_text` | `TEXT` | yes | `payload.shortTerm.text` |
|
||||
| `long_term_qualifier` | `TEXT` | yes | `payload.longTerm.qualifier` |
|
||||
| `long_term_issued_at` | `TIMESTAMPTZ` | yes | `payload.longTerm.issuedAt` |
|
||||
| `long_term_text` | `TEXT` | yes | `payload.longTerm.text` |
|
||||
| `key_message_count` | `INTEGER` | no | `len(payload.keyMessages)` |
|
||||
|
||||
### `forecast_discussion_key_messages`
|
||||
|
||||
Primary key: `run_event_id`, `message_index`
|
||||
|
||||
Prune column: `issued_at`
|
||||
|
||||
Foreign key: `run_event_id` references `forecast_discussions(event_id)` with
|
||||
cascade delete.
|
||||
|
||||
Index: `idx_wf_discussion_message_issued_at` on `issued_at`
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `run_event_id` | `TEXT REFERENCES forecast_discussions(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||
| `message_index` | `INTEGER` | no | `payload.keyMessages[]` index. |
|
||||
| `issued_at` | `TIMESTAMPTZ` | no | Parent `payload.issuedAt` |
|
||||
| `message_text` | `TEXT` | yes | `payload.keyMessages[]` value |
|
||||
|
||||
### `weather_story_runs`
|
||||
|
||||
Primary key: `event_id`
|
||||
|
||||
Prune column: `as_of`
|
||||
|
||||
Indexes:
|
||||
|
||||
- `idx_wf_story_run_office_as_of` on `office_id`, `as_of`
|
||||
- `idx_wf_story_run_as_of` on `as_of`
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `event_id` | `TEXT` | no | `event.id` |
|
||||
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||
| `event_source` | `TEXT` | no | `event.source` |
|
||||
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||
| `office_id` | `TEXT` | yes | `payload.officeId` |
|
||||
| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` |
|
||||
| `story_count` | `INTEGER` | no | `len(payload.stories)` |
|
||||
|
||||
### `weather_stories`
|
||||
|
||||
Primary key: `run_event_id`, `story_index`
|
||||
|
||||
Prune column: `as_of`
|
||||
|
||||
Foreign key: `run_event_id` references `weather_story_runs(event_id)` with
|
||||
cascade delete.
|
||||
|
||||
Indexes:
|
||||
|
||||
- `idx_wf_stories_start_time` on `start_time`
|
||||
- `idx_wf_stories_end_time` on `end_time`
|
||||
- `idx_wf_stories_updated_at` on `updated_at`
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `run_event_id` | `TEXT REFERENCES weather_story_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||
| `story_index` | `INTEGER` | no | `payload.stories[]` index. |
|
||||
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
|
||||
| `office_id` | `TEXT` | yes | `payload.stories[].officeId` |
|
||||
| `start_time` | `TIMESTAMPTZ` | no | `payload.stories[].startTime` |
|
||||
| `end_time` | `TIMESTAMPTZ` | no | `payload.stories[].endTime` |
|
||||
| `updated_at` | `TIMESTAMPTZ` | no | `payload.stories[].updatedAt` |
|
||||
| `title` | `TEXT` | yes | `payload.stories[].title` |
|
||||
| `description` | `TEXT` | yes | `payload.stories[].description` |
|
||||
| `alt_text` | `TEXT` | yes | `payload.stories[].altText` |
|
||||
| `priority` | `BOOLEAN` | no | `payload.stories[].priority` |
|
||||
| `story_order` | `INTEGER` | no | `payload.stories[].order` |
|
||||
| `download_url` | `TEXT` | yes | `payload.stories[].downloadUrl` |
|
||||
|
||||
### `alert_runs`
|
||||
|
||||
Primary key: `event_id`
|
||||
|
||||
Prune column: `as_of`
|
||||
|
||||
Indexes:
|
||||
|
||||
- `idx_wf_alert_run_location_as_of` on `location_id`, `as_of`
|
||||
- `idx_wf_alert_run_as_of` on `as_of`
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `event_id` | `TEXT` | no | `event.id` |
|
||||
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||
| `event_source` | `TEXT` | no | `event.source` |
|
||||
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||
| `location_id` | `TEXT` | yes | `payload.locationId` |
|
||||
| `location_name` | `TEXT` | yes | `payload.locationName` |
|
||||
| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` |
|
||||
| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` |
|
||||
| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` |
|
||||
| `alert_count` | `INTEGER` | no | `len(payload.alerts)` |
|
||||
|
||||
### `alerts`
|
||||
|
||||
Primary key: `run_event_id`, `alert_index`
|
||||
|
||||
Prune column: `as_of`
|
||||
|
||||
Foreign key: `run_event_id` references `alert_runs(event_id)` with cascade
|
||||
delete.
|
||||
|
||||
Indexes:
|
||||
|
||||
- `idx_wf_alerts_alert_id` on `alert_id`
|
||||
- `idx_wf_alerts_severity_expires` on `severity`, `expires`
|
||||
- `idx_wf_alerts_as_of` on `as_of`
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `run_event_id` | `TEXT REFERENCES alert_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||
| `alert_index` | `INTEGER` | no | `payload.alerts[]` index. |
|
||||
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
|
||||
| `alert_id` | `TEXT` | no | `payload.alerts[].id` |
|
||||
| `event` | `TEXT` | yes | `payload.alerts[].event` |
|
||||
| `headline` | `TEXT` | yes | `payload.alerts[].headline` |
|
||||
| `severity` | `TEXT` | yes | `payload.alerts[].severity` |
|
||||
| `urgency` | `TEXT` | yes | `payload.alerts[].urgency` |
|
||||
| `certainty` | `TEXT` | yes | `payload.alerts[].certainty` |
|
||||
| `status` | `TEXT` | yes | `payload.alerts[].status` |
|
||||
| `message_type` | `TEXT` | yes | `payload.alerts[].messageType` |
|
||||
| `category` | `TEXT` | yes | `payload.alerts[].category` |
|
||||
| `response` | `TEXT` | yes | `payload.alerts[].response` |
|
||||
| `description` | `TEXT` | yes | `payload.alerts[].description` |
|
||||
| `instruction` | `TEXT` | yes | `payload.alerts[].instruction` |
|
||||
| `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].sent` |
|
||||
| `effective` | `TIMESTAMPTZ` | yes | `payload.alerts[].effective` |
|
||||
| `onset` | `TIMESTAMPTZ` | yes | `payload.alerts[].onset` |
|
||||
| `expires` | `TIMESTAMPTZ` | yes | `payload.alerts[].expires` |
|
||||
| `area_description` | `TEXT` | yes | `payload.alerts[].areaDescription` |
|
||||
| `sender_name` | `TEXT` | yes | `payload.alerts[].senderName` |
|
||||
| `reference_count` | `INTEGER` | no | `len(payload.alerts[].references)` |
|
||||
|
||||
### `alert_references`
|
||||
|
||||
Primary key: `run_event_id`, `alert_index`, `reference_index`
|
||||
|
||||
Prune column: `as_of`
|
||||
|
||||
Foreign key: `run_event_id` references `alert_runs(event_id)` with cascade
|
||||
delete.
|
||||
|
||||
Indexes:
|
||||
|
||||
- `idx_wf_alert_refs_as_of` on `as_of`
|
||||
- `idx_wf_alert_refs_sent` on `sent`
|
||||
|
||||
| Column | Type | Null | Source |
|
||||
|---|---|:---:|---|
|
||||
| `run_event_id` | `TEXT REFERENCES alert_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||
| `alert_index` | `INTEGER` | no | Parent alert index. |
|
||||
| `reference_index` | `INTEGER` | no | `payload.alerts[].references[]` index. |
|
||||
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
|
||||
| `id` | `TEXT` | yes | `payload.alerts[].references[].id` |
|
||||
| `identifier` | `TEXT` | yes | `payload.alerts[].references[].identifier` |
|
||||
| `sender` | `TEXT` | yes | `payload.alerts[].references[].sender` |
|
||||
| `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].references[].sent` |
|
||||
|
||||
## Retention
|
||||
|
||||
When sink param `prune` is set, every successful write transaction deletes rows
|
||||
older than `now - prune` from every table using that table's prune column.
|
||||
|
||||
The sink also exposes manual prune helpers in code, but the `weatherfeeder`
|
||||
binary does not provide CLI commands for them.
|
||||
|
||||
## Reconstructing Canonical Payloads
|
||||
|
||||
- `WeatherObservation`: read `observations`, then join
|
||||
`observation_present_weather` by `event_id` ordered by `weather_index`.
|
||||
- `WeatherForecastRun`: read `forecasts`, then join `forecast_periods` by
|
||||
`run_event_id` ordered by `period_index`.
|
||||
- `WeatherForecastDiscussion`: read `forecast_discussions`, then join
|
||||
`forecast_discussion_key_messages` by `run_event_id` ordered by
|
||||
`message_index`.
|
||||
- `WeatherStoryRun`: read `weather_story_runs`, then join `weather_stories` by
|
||||
`run_event_id` ordered by `story_index`.
|
||||
- `WeatherAlertRun`: read `alert_runs`, join `alerts` by `run_event_id` ordered
|
||||
by `alert_index`, then join `alert_references` by `run_event_id` and
|
||||
`alert_index` ordered by `reference_index`.
|
||||
90
docs/internal/normalizers.md
Normal file
90
docs/internal/normalizers.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# Normalizer Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
Normalizers convert raw provider events into canonical weather events. They are
|
||||
weather-domain mapping code and should stay independent of runtime wiring,
|
||||
source polling, and sink persistence.
|
||||
|
||||
Detailed package conventions live in `internal/normalizers/doc.go`.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs are raw feed events whose schemas identify provider payload shape.
|
||||
Outputs are canonical feed events using `model` payloads and `weather.*`
|
||||
schemas.
|
||||
|
||||
Current mappings:
|
||||
|
||||
| Raw schema | Canonical schema |
|
||||
| --- | --- |
|
||||
| `raw.nws.observation.v1` | `weather.observation.v1` |
|
||||
| `raw.openmeteo.current.v1` | `weather.observation.v1` |
|
||||
| `raw.openweather.current.v1` | `weather.observation.v1` |
|
||||
| `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||
| `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
|
||||
| `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||
| `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
|
||||
| `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
|
||||
| `raw.nws.alerts.v1` | `weather.alert.v1` |
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Normalizers match by `Event.Schema`.
|
||||
- Normalizers decode raw payloads into provider structs.
|
||||
- Normalizers map provider data into canonical `model` payloads.
|
||||
- Normalizers do not fetch network data, read config, route events, or write
|
||||
sinks.
|
||||
- Shared cross-provider behavior belongs in `internal/normalizers/common`.
|
||||
- Provider-specific helper logic shared with sources belongs in
|
||||
`internal/providers/<provider>`.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
Normalizers do not read config. They operate only on incoming events.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
Runtime composition creates feedkit's normalize processor with
|
||||
`RequireMatch=false`. Events without a matching normalizer pass through
|
||||
unchanged.
|
||||
|
||||
Weatherfeeder registers normalizers in a stable order:
|
||||
|
||||
1. NWS
|
||||
2. Open-Meteo
|
||||
3. OpenWeather
|
||||
|
||||
The current normalizers avoid ambiguous matches by using schema equality.
|
||||
|
||||
## State
|
||||
|
||||
Normalizers should be stateless. Shared helpers should be deterministic and free
|
||||
of I/O.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Malformed required raw payload data should produce contextual errors from the
|
||||
owning normalizer. Successful normalization validates the output event before it
|
||||
continues through the pipeline.
|
||||
|
||||
`internal/normalizers/common.Finalize` preserves the input event envelope except
|
||||
for schema, payload, and effective time. It also rounds canonical float values
|
||||
to four digits after the decimal point.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/normalizers/builtins_test.go`
|
||||
- provider normalizer tests under `internal/normalizers/nws`
|
||||
- provider normalizer tests under `internal/normalizers/openmeteo`
|
||||
- provider normalizer tests under `internal/normalizers/openweather`
|
||||
- common helper tests under `internal/normalizers/common`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Match by schema constants from `standards`.
|
||||
- Preserve the event envelope except for intentional canonical changes.
|
||||
- Produce canonical payload structs from `model`.
|
||||
- Validate normalized events before returning them.
|
||||
- Keep normalizers independent of sources, sinks, config loading, and runtime
|
||||
composition.
|
||||
118
docs/internal/postgres-sink.md
Normal file
118
docs/internal/postgres-sink.md
Normal file
@@ -0,0 +1,118 @@
|
||||
# Postgres Sink Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/sinks/postgres` defines weatherfeeder's canonical-event-to-Postgres
|
||||
mapping. It supplies a schema definition and mapper to feedkit's generic
|
||||
Postgres sink.
|
||||
|
||||
The consumer-facing table contract is
|
||||
[`docs/integrations/postgres.md`](../integrations/postgres.md). This document
|
||||
describes the internal ownership boundary.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs are canonical feed events. The mapper currently handles these schemas:
|
||||
|
||||
- `weather.observation.v1`
|
||||
- `weather.forecast.v1`
|
||||
- `weather.forecast_discussion.v1`
|
||||
- `weather.weather_story.v1`
|
||||
- `weather.alert.v1`
|
||||
|
||||
Outputs are feedkit `PostgresWrite` values for weatherfeeder-owned tables.
|
||||
Unsupported schemas produce no writes and no error.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Weatherfeeder owns table definitions in `schema.go`.
|
||||
- Weatherfeeder owns canonical payload mapping in `map.go`.
|
||||
- Feedkit owns database opening, table and index creation, transactions,
|
||||
inserts, context-aware consumption, and prune execution.
|
||||
- Postgres mapping consumes canonical events only. It should not understand raw
|
||||
provider schemas.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
Weatherfeeder registers the `postgres` sink by passing `PostgresSchema()` to
|
||||
feedkit. Feedkit parses sink params:
|
||||
|
||||
- `uri`
|
||||
- `username`
|
||||
- `password`
|
||||
- `prune`, optional duration
|
||||
|
||||
Weatherfeeder-owned mapper code does not read config directly.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
The runtime registers the sink with:
|
||||
|
||||
```go
|
||||
sinkReg.Register("postgres", fksinks.PostgresFactory(wfpgsink.PostgresSchema()))
|
||||
```
|
||||
|
||||
Feedkit validates events at the sink boundary, calls the weatherfeeder mapper,
|
||||
validates writes against the compiled schema, inserts rows in a transaction, and
|
||||
optionally prunes rows older than the configured window.
|
||||
|
||||
## State
|
||||
|
||||
The mapper is stateless. Durable state is stored in Postgres through feedkit's
|
||||
sink implementation.
|
||||
|
||||
## Mapping Rules
|
||||
|
||||
Parent rows preserve event envelope fields where the table supports them:
|
||||
|
||||
- `event_id`
|
||||
- `event_kind`
|
||||
- `event_source`
|
||||
- `event_schema`
|
||||
- `event_emitted_at`
|
||||
- `event_effective_at`
|
||||
|
||||
Child rows use positional indexes to preserve canonical array order:
|
||||
|
||||
- `weather_index`
|
||||
- `period_index`
|
||||
- `message_index`
|
||||
- `story_index`
|
||||
- `alert_index`
|
||||
- `reference_index`
|
||||
|
||||
Required canonical fields are validated before writes are returned:
|
||||
|
||||
- observations require `timestamp`;
|
||||
- forecasts require `issuedAt` and `product`, and each period requires
|
||||
`startTime` and `endTime`;
|
||||
- forecast discussions require `issuedAt` and `product`;
|
||||
- weather story runs require `asOf`, and each story requires `startTime`,
|
||||
`endTime`, and `updatedAt`;
|
||||
- alert runs require `asOf`, and each alert requires `id`.
|
||||
|
||||
Nullable canonical values are converted to SQL nulls by mapper helpers.
|
||||
Observation present-weather raw values are stored as compact JSON text.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Payload decode failures, missing required fields, invalid compact JSON values,
|
||||
or schema/write mismatches return errors to feedkit's sink. Feedkit rolls back
|
||||
the transaction when a write fails.
|
||||
|
||||
Unsupported canonical schemas are ignored by this mapper so other routed events
|
||||
can use different sinks without Postgres-specific failures.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/sinks/postgres/schema_test.go`
|
||||
- `internal/sinks/postgres/map_test.go`
|
||||
- feedkit Postgres sink tests when changing generic sink behavior assumptions
|
||||
|
||||
## Invariants
|
||||
|
||||
- Persist only canonical schemas.
|
||||
- Preserve event envelope fields in parent rows.
|
||||
- Preserve array order with child positional indexes.
|
||||
- Validate required fields before writing.
|
||||
- Keep table-contract docs synchronized with schema and mapper changes.
|
||||
110
docs/internal/runtime.md
Normal file
110
docs/internal/runtime.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# Runtime Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
`cmd/weatherfeeder` wires the daemon together. It owns process setup and runtime
|
||||
composition; provider mapping, source fetching details, and sink persistence
|
||||
rules stay in their owning packages.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
The executable reads `config.yml` from the current working directory through
|
||||
feedkit config loading. It builds configured sources, scheduler jobs, sinks, and
|
||||
routes, then runs source polling and sink dispatch until shutdown.
|
||||
|
||||
Inputs are configured source polls. Outputs are feed events delivered to the
|
||||
configured sinks.
|
||||
|
||||
## Runtime Flow
|
||||
|
||||
The implemented flow is:
|
||||
|
||||
1. load `config.yml`;
|
||||
2. register weatherfeeder source drivers;
|
||||
3. register feedkit built-in sinks and the weatherfeeder Postgres sink;
|
||||
4. build source inputs and scheduler jobs;
|
||||
5. validate configured expected kinds against source-advertised kinds;
|
||||
6. build sinks and compile routes;
|
||||
7. run the processor chain `normalize`, then `dedupe`;
|
||||
8. run the scheduler and dispatcher concurrently;
|
||||
9. shut down on signal or fatal scheduler/dispatcher error.
|
||||
|
||||
The in-process event channel is buffered to 256 events. The dedupe processor is
|
||||
bounded by `dedupeMaxEntries`, currently 2048.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Runtime composition belongs in `cmd/weatherfeeder/main.go`.
|
||||
- Source driver behavior belongs under `internal/sources`.
|
||||
- Normalizer behavior belongs under `internal/normalizers`.
|
||||
- Canonical payloads and schema strings belong in `model` and `standards`.
|
||||
- Postgres mapping belongs under `internal/sinks/postgres`.
|
||||
|
||||
`cmd/weatherfeeder` should stay thin and should not contain provider parsing,
|
||||
canonical mapping, or table-mapping rules.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
Runtime wiring consumes the feedkit top-level config sections:
|
||||
|
||||
- `sources`: source driver selection, source name, mode, cadence, expected kinds,
|
||||
and driver params;
|
||||
- `sinks`: sink driver selection, sink name, and sink params;
|
||||
- `routes`: event-kind routing to named sinks.
|
||||
|
||||
The executable does not expose CLI flags or config path discovery.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
Runtime composition uses feedkit for:
|
||||
|
||||
- config loading;
|
||||
- source registry and expected-kind validation;
|
||||
- scheduler job construction;
|
||||
- processor registry and chain execution;
|
||||
- normalization and dedupe processors;
|
||||
- sink registry and built-in sinks;
|
||||
- route compilation and dispatch.
|
||||
|
||||
Weatherfeeder registers its own source drivers and its Postgres schema mapper.
|
||||
|
||||
## State
|
||||
|
||||
Weatherfeeder-owned runtime state is in process:
|
||||
|
||||
- event channel contents;
|
||||
- the bounded dedupe key set;
|
||||
- source instances and their HTTP conditional validators;
|
||||
- scheduler and dispatcher goroutines.
|
||||
|
||||
There is no weatherfeeder-owned durable scheduler state, checkpoint, replay log,
|
||||
or resume marker. Durable persistence is owned by configured external sinks.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Startup failures are fatal and include context such as config index, source name,
|
||||
sink name, driver name, or the operation that failed.
|
||||
|
||||
At runtime, scheduler and dispatcher errors are sent to a shared error channel.
|
||||
Context cancellation and deadline errors are treated as normal shutdown. Any
|
||||
other scheduler or dispatcher error is logged as fatal and cancels the process
|
||||
context.
|
||||
|
||||
The daemon handles `os.Interrupt` and `SIGTERM` with `signal.NotifyContext`.
|
||||
After both runtime goroutines return, it logs `shutdown complete`.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `cmd/weatherfeeder/main_test.go`
|
||||
- source registry tests under `internal/sources`
|
||||
- normalizer registration tests under `internal/normalizers`
|
||||
- feedkit scheduler, processor, dispatch, and sink tests when changing runtime
|
||||
infrastructure usage
|
||||
|
||||
## Invariants
|
||||
|
||||
- Keep normalization before dedupe.
|
||||
- Keep queue sizes and dedupe bounds explicit.
|
||||
- Preserve context-aware shutdown.
|
||||
- Keep runtime wiring separate from domain mapping and persistence rules.
|
||||
- Keep startup validation failures loud and contextual.
|
||||
105
docs/internal/sources.md
Normal file
105
docs/internal/sources.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# Source Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
Source packages poll upstream weather providers and emit raw feed events. They
|
||||
are adapters, not canonical mappers.
|
||||
|
||||
Sources should decode only the metadata needed for event identity, effective
|
||||
time, and routing policy. Full provider payload interpretation belongs in
|
||||
normalizers.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs are feedkit `config.SourceConfig` values and upstream HTTP responses.
|
||||
Outputs are feed events whose payloads are raw provider JSON and whose schemas
|
||||
come from `standards`.
|
||||
|
||||
Current drivers:
|
||||
|
||||
| Driver | Kind | Raw schema |
|
||||
| --- | --- | --- |
|
||||
| `nws_observation` | `observation` | `raw.nws.observation.v1` |
|
||||
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` |
|
||||
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` |
|
||||
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` |
|
||||
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` |
|
||||
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` |
|
||||
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` |
|
||||
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` |
|
||||
| `openweather_observation` | `observation` | `raw.openweather.current.v1` |
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Source constructors validate source-specific params.
|
||||
- Sources use feedkit HTTP helpers for HTTP polling.
|
||||
- Sources emit raw events and should not build canonical `model` payloads.
|
||||
- Provider helper packages under `internal/providers/<provider>` hold shared
|
||||
parsing and validation helpers.
|
||||
- Registration is centralized in `internal/sources/builtins.go`.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
All current source drivers use feedkit `HTTPSource`.
|
||||
|
||||
Required params:
|
||||
|
||||
- `url`
|
||||
- `user_agent`
|
||||
|
||||
Optional params:
|
||||
|
||||
- `conditional`, default `true`;
|
||||
- `http_timeout`;
|
||||
- `http_response_body_limit_bytes`.
|
||||
|
||||
OpenWeather observation sources additionally require the configured URL to use
|
||||
metric units. This is enforced by `internal/providers/openweather`.
|
||||
|
||||
Source-level `kinds`, when configured, are validated against the source's
|
||||
advertised `Kinds()`.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
Sources use feedkit's HTTP helper for:
|
||||
|
||||
- request construction;
|
||||
- `User-Agent` and `Accept` headers;
|
||||
- optional conditional GET validators;
|
||||
- response body size limits;
|
||||
- JSON raw-message fetches.
|
||||
|
||||
NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times
|
||||
with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs.
|
||||
|
||||
## State
|
||||
|
||||
HTTP conditional validators are held in each source instance. They are not
|
||||
persisted across process restarts.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Constructor failures are returned during startup and stop the daemon. Polling
|
||||
failures are returned to the scheduler.
|
||||
|
||||
If a source cannot decode minimal metadata from an otherwise successful upstream
|
||||
response, it still emits the raw event when possible. The event then falls back
|
||||
to default ID/effective-time behavior from feedkit source helpers.
|
||||
|
||||
Unchanged conditional responses return no events and no error.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/sources/builtins_test.go`
|
||||
- provider source tests under `internal/sources/nws`
|
||||
- provider source tests under `internal/sources/openmeteo`
|
||||
- provider source tests under `internal/sources/openweather`
|
||||
- provider helper tests under `internal/providers`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Emit raw provider schemas from `standards`.
|
||||
- Keep provider-to-canonical mapping out of sources.
|
||||
- Keep HTTP behavior context-aware.
|
||||
- Keep driver registration explicit and stable.
|
||||
- Keep source tests independent of live upstream services.
|
||||
158
docs/operations.md
Normal file
158
docs/operations.md
Normal file
@@ -0,0 +1,158 @@
|
||||
# Operations
|
||||
|
||||
This document describes how to run and observe the `weatherfeeder` daemon in its
|
||||
current form. For configuration syntax, see [configuration](config.md). For the
|
||||
CLI surface, see [CLI reference](cli.md).
|
||||
|
||||
## Normal Workflow
|
||||
|
||||
1. Prepare `config.yml` in the process working directory.
|
||||
2. Start the daemon with `./weatherfeeder` or `go run .` from
|
||||
`cmd/weatherfeeder`.
|
||||
3. Watch stderr logs for startup or runtime errors.
|
||||
4. Consume events from the configured sinks.
|
||||
5. Stop the process with `Ctrl-C` or `SIGTERM`.
|
||||
|
||||
The daemon has no admin subcommands and no runtime reload command. Change the
|
||||
config file and restart the process to apply configuration changes.
|
||||
|
||||
Maintained copyable configs are available under [`examples/`](../examples/).
|
||||
|
||||
## Runtime Lifecycle
|
||||
|
||||
On startup, `weatherfeeder`:
|
||||
|
||||
1. loads `config.yml` from the current working directory;
|
||||
2. registers built-in source drivers;
|
||||
3. registers stdout, NATS, and weatherfeeder Postgres sink drivers;
|
||||
4. builds sources and validates configured `kinds` against source metadata;
|
||||
5. builds sinks and compiles routes;
|
||||
6. starts the scheduler and dispatcher;
|
||||
7. processes events through normalization, then in-memory dedupe;
|
||||
8. routes processed events to configured sinks.
|
||||
|
||||
Startup errors are fatal and terminate the process. Runtime poll, pipeline, and
|
||||
sink errors are logged and the process continues unless the scheduler or
|
||||
dispatcher returns a fatal error.
|
||||
|
||||
## Logs
|
||||
|
||||
The process uses the Go standard logger with date, time, and microseconds. Logs
|
||||
go to stderr.
|
||||
|
||||
Common log prefixes:
|
||||
|
||||
| Prefix | Meaning |
|
||||
|---|---|
|
||||
| `config load failed` | `config.yml` could not be read, parsed, or validated. |
|
||||
| `build source failed` | A source driver or its params are invalid. |
|
||||
| `source expected kinds validation failed` | Configured source `kinds` do not match the source driver. |
|
||||
| `build sink failed` | A sink driver or its params are invalid, or a sink could not initialize. |
|
||||
| `compile routes failed` | Routes reference invalid sinks or kinds. |
|
||||
| `scheduler: poll failed` | A source poll failed; the source will be polled again on its next interval. |
|
||||
| `dispatcher: pipeline error` | Normalization or dedupe failed for one event. |
|
||||
| `dispatch: sink ... failed consuming event` | A sink failed to consume one event. |
|
||||
| `shutdown complete` | Scheduler and dispatcher have exited. |
|
||||
|
||||
## Scheduling And Polling
|
||||
|
||||
Current weatherfeeder sources are polling sources. Each source uses its
|
||||
configured `every` interval. The scheduler applies jitter before the first poll
|
||||
and before each interval tick. If no jitter is configured in code, feedkit uses
|
||||
`min(every/10, 30s)`, capped at half the interval.
|
||||
|
||||
Poll failures are logged and do not stop the daemon. A failed poll emits no
|
||||
events for that source until a subsequent poll succeeds.
|
||||
|
||||
## Conditional HTTP Fetches
|
||||
|
||||
All current sources use feedkit's HTTP polling helper. By default,
|
||||
`params.conditional` is `true`, so the helper keeps ETag and Last-Modified
|
||||
validators in memory for each source instance.
|
||||
|
||||
If the upstream returns `304 Not Modified`, the source emits no events for that
|
||||
poll. Validator state is in memory only; restarting the process starts with no
|
||||
cached validators.
|
||||
|
||||
## Processing And Dedupe
|
||||
|
||||
Every event passes through normalization first and dedupe second.
|
||||
|
||||
Normalizers match raw source schemas and produce canonical `weather.*.v1`
|
||||
payloads. If an event has no matching normalizer, the normalize processor passes
|
||||
it through unchanged.
|
||||
|
||||
Dedupe keys by event ID and stores a bounded in-memory set of 2048 recent IDs.
|
||||
Duplicate IDs are dropped. Dedupe state is not persisted, so a restart starts
|
||||
with an empty dedupe set.
|
||||
|
||||
## Routing And Sink Fanout
|
||||
|
||||
Routes choose sinks by event kind. If `routes` is omitted, every sink receives
|
||||
every event kind. If a route omits `kinds`, that route also matches all kinds.
|
||||
|
||||
The dispatcher creates one queue and one worker goroutine per sink. The default
|
||||
per-sink queue size is 64. `weatherfeeder` does not currently expose config
|
||||
fields for sink queue size, enqueue timeout, or consume timeout.
|
||||
|
||||
Sink errors are logged per event. A sink failure does not stop other sinks from
|
||||
receiving the same event.
|
||||
|
||||
## Sink Behavior
|
||||
|
||||
### stdout
|
||||
|
||||
The stdout sink validates each event and prints one JSON object per line to
|
||||
stdout. This is useful for local inspection and log forwarding.
|
||||
|
||||
### NATS
|
||||
|
||||
The NATS sink connects lazily on the first event, reuses the connection while it
|
||||
is open, and publishes each event as JSON to the configured subject. Connection,
|
||||
marshal, and publish failures are logged by the dispatch worker.
|
||||
|
||||
### Postgres
|
||||
|
||||
The Postgres sink opens the database during startup. It creates missing tables
|
||||
and indexes with `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`.
|
||||
It does not modify existing table definitions.
|
||||
|
||||
Each mapped canonical event is written in one transaction. If `params.prune` is
|
||||
set, the sink deletes rows older than `now - prune` from every weatherfeeder
|
||||
table in that same transaction. See the
|
||||
[Postgres table contract](integrations/postgres.md).
|
||||
|
||||
## State And Recovery
|
||||
|
||||
`weatherfeeder` keeps only runtime state in process memory:
|
||||
|
||||
- scheduler goroutines and timers;
|
||||
- HTTP conditional request validators;
|
||||
- event channel buffers;
|
||||
- per-sink fanout queues;
|
||||
- the dedupe ID set.
|
||||
|
||||
Durable state is external sink state: NATS broker state outside this process and
|
||||
Postgres tables managed by the configured database.
|
||||
|
||||
There is no internal checkpoint, replay log, or resume marker. To recover from a
|
||||
process failure, fix the underlying issue and restart the daemon from a working
|
||||
directory containing the desired `config.yml`.
|
||||
|
||||
## Shutdown
|
||||
|
||||
`weatherfeeder` listens for `os.Interrupt` and `SIGTERM`. On shutdown, the
|
||||
shared context is canceled. Scheduler jobs stop polling, dispatch workers stop,
|
||||
and the process logs `shutdown complete`.
|
||||
|
||||
Queued sink work may be dropped when shutdown context cancellation reaches the
|
||||
fanout workers. Use external sink durability, such as Postgres or broker
|
||||
retention, for durable downstream state.
|
||||
|
||||
## Caveats
|
||||
|
||||
- There is no health-check endpoint.
|
||||
- There is no runtime config reload.
|
||||
- There are no built-in metrics.
|
||||
- Source conditional request state and dedupe state are reset by restart.
|
||||
- Existing Postgres schemas are not migrated automatically.
|
||||
@@ -2,9 +2,11 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines `weatherfeeder`'s development architecture and invariants for maintainers and LLM coding agents. It describes how the implemented system is built and how future changes should preserve its boundaries.
|
||||
This document defines `weatherfeeder`'s development architecture and invariants for maintainers and LLM coding agents. It describes how the implemented system is built and how subsequent changes should preserve its boundaries.
|
||||
|
||||
This is an inward-facing policy document. User-facing wire contracts belong in [`API.md`](../../API.md), and future work belongs under [`docs/roadmap/`](../roadmap/).
|
||||
This is an inward-facing policy document. User-facing wire contracts belong in
|
||||
[`docs/integrations/events.md`](../integrations/events.md), and roadmap items
|
||||
belong under [`docs/roadmap/`](../roadmap/).
|
||||
|
||||
## Project Shape
|
||||
|
||||
@@ -55,9 +57,9 @@ Tests and examples:
|
||||
- The sample `cmd/weatherfeeder/config.yml` is executable test input and is load-tested.
|
||||
- Tests should keep exercising package contracts directly rather than relying only on full-daemon execution.
|
||||
|
||||
## Modules or Stages
|
||||
## Modules Or Processing Steps
|
||||
|
||||
The implemented processing stages are source polling, normalization, dedupe, and sink dispatch.
|
||||
The implemented processing steps are source polling, normalization, dedupe, and sink dispatch.
|
||||
|
||||
Source contract:
|
||||
|
||||
@@ -98,7 +100,10 @@ The daemon's own state is in-process:
|
||||
- source instances may keep HTTP conditional request state through feedkit HTTP source helpers;
|
||||
- scheduler and dispatcher state is not persisted by `weatherfeeder`.
|
||||
|
||||
Durable persistence is an external sink concern. The Postgres table contract is documented in `internal/sinks/postgres/doc.go`; the consumer-facing event contract is documented in [`API.md`](../../API.md).
|
||||
Durable persistence is an external sink concern. The Postgres table contract is
|
||||
documented in [`docs/integrations/postgres.md`](../integrations/postgres.md);
|
||||
the consumer-facing event contract is documented in
|
||||
[`docs/integrations/events.md`](../integrations/events.md).
|
||||
|
||||
## Configuration and CLI Boundaries
|
||||
|
||||
@@ -112,7 +117,9 @@ Configuration shape is owned by feedkit's config package:
|
||||
|
||||
Weatherfeeder-specific config policy belongs in source and sink constructors, registry setup, and tests. Do not spread config parsing through domain model or normalizer packages.
|
||||
|
||||
If dedicated `docs/config.md` or `docs/cli.md` files are added later, they should become the canonical user/operator references. This policy should stay architectural and avoid duplicating those references.
|
||||
[`docs/config.md`](../config.md) and [`docs/cli.md`](../cli.md) are the
|
||||
canonical user/operator references. This policy should stay architectural and
|
||||
avoid duplicating those references.
|
||||
|
||||
## Errors, Logging, and Diagnostics
|
||||
|
||||
|
||||
202
docs/policy/development.md
Normal file
202
docs/policy/development.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# Development Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
This document describes how to change `weatherfeeder` safely. It is for
|
||||
maintainers and coding agents working in the repository.
|
||||
|
||||
Use this alongside the [architecture policy](architecture.md). User-facing CLI,
|
||||
configuration, operations, and wire-contract details belong in their canonical
|
||||
docs, not here.
|
||||
|
||||
## Repository Layout
|
||||
|
||||
- `cmd/weatherfeeder/`: executable wiring, sample `config.yml`, and runtime
|
||||
composition tests.
|
||||
- `model/`: canonical weather payload structs. JSON tags are part of the wire
|
||||
contract.
|
||||
- `standards/`: schema strings, versioning conventions, WMO constants, and
|
||||
shared wire-format policy.
|
||||
- `internal/sources/`: source adapters that poll upstream providers and emit raw
|
||||
feed events.
|
||||
- `internal/normalizers/`: raw-to-canonical event transforms.
|
||||
- `internal/providers/`: pure provider helper code shared by sources and
|
||||
normalizers.
|
||||
- `internal/sinks/postgres/`: weatherfeeder-owned Postgres schema and canonical
|
||||
event mapper.
|
||||
- `docs/`: current behavior, policies, integration contracts, and roadmap files.
|
||||
- `examples/`: maintained, copyable configuration examples.
|
||||
|
||||
## Build And Test
|
||||
|
||||
Run the full test suite before committing behavior or documentation changes that
|
||||
depend on code behavior:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Use narrower commands while iterating:
|
||||
|
||||
```sh
|
||||
go test ./cmd/weatherfeeder
|
||||
go test ./internal/sources/...
|
||||
go test ./internal/normalizers/...
|
||||
go test ./internal/sinks/postgres
|
||||
```
|
||||
|
||||
Format Go code before committing:
|
||||
|
||||
```sh
|
||||
gofmt -w <changed-go-files>
|
||||
```
|
||||
|
||||
Do not require live upstream weather services, NATS, or Postgres for unit tests.
|
||||
Use fixtures, local test servers, and package-level tests.
|
||||
|
||||
## Coding Conventions
|
||||
|
||||
- Keep `cmd/weatherfeeder` focused on composition: config load, registry setup,
|
||||
scheduler jobs, processor chain, dispatch, signal handling, and logging.
|
||||
- Keep source fetching separate from normalizer mapping.
|
||||
- Match normalizers by schema constants from `standards`, not source names.
|
||||
- Keep provider-specific helper code under `internal/providers/<provider>` when
|
||||
both sources and normalizers use it.
|
||||
- Keep cross-provider normalizer helpers pure and deterministic under
|
||||
`internal/normalizers/common`.
|
||||
- Keep sink persistence mapping isolated under `internal/sinks/<sink>`.
|
||||
- Wrap errors with operation context, but do not include whole upstream payloads
|
||||
in errors or logs by default.
|
||||
- Prefer explicit registries and small package-level constructors over hidden
|
||||
global behavior.
|
||||
|
||||
## Dependency Policy
|
||||
|
||||
Prefer the Go standard library unless a dependency materially improves
|
||||
maintainability or interoperability.
|
||||
|
||||
`feedkit` owns generic daemon infrastructure for config, HTTP source helpers,
|
||||
scheduling, processors, dispatch, and sinks. Weatherfeeder code should contain
|
||||
weather-domain behavior and narrow adapter logic rather than duplicating feedkit
|
||||
infrastructure.
|
||||
|
||||
Do not add broad dependencies for small conveniences. Do not let
|
||||
dependency-specific types leak across package boundaries unless that dependency
|
||||
is the package contract.
|
||||
|
||||
## Adding Config Fields
|
||||
|
||||
Generic config shape is owned by feedkit. Weatherfeeder-specific config behavior
|
||||
belongs in source or sink constructors, registry setup, and tests.
|
||||
|
||||
When adding config behavior:
|
||||
|
||||
- validate required params at the adapter boundary;
|
||||
- keep secrets in environment variables or placeholders, not committed values;
|
||||
- update [configuration docs](../config.md);
|
||||
- update maintained examples when the change affects normal operation;
|
||||
- add or update config-load tests for example files when practical.
|
||||
|
||||
## Adding CLI Flags
|
||||
|
||||
The executable currently reads `config.yml` from the current working directory.
|
||||
If CLI flags are added:
|
||||
|
||||
- keep parsing in `cmd/weatherfeeder`;
|
||||
- avoid moving config policy into domain packages;
|
||||
- update [CLI docs](../cli.md);
|
||||
- update tests that exercise command behavior.
|
||||
|
||||
## Adding A Source Driver
|
||||
|
||||
Source drivers should fetch upstream data and emit raw events with minimal
|
||||
metadata decoding.
|
||||
|
||||
Checklist:
|
||||
|
||||
- implement the driver under `internal/sources/<provider>`;
|
||||
- build from `config.SourceConfig`;
|
||||
- validate required params in the constructor;
|
||||
- use feedkit HTTP helpers for HTTP polling when applicable;
|
||||
- emit raw schema constants from `standards`;
|
||||
- advertise emitted kinds through `Kinds()`;
|
||||
- decode only metadata needed for event ID and effective time;
|
||||
- register the driver in `internal/sources/builtins.go`;
|
||||
- add constructor, kind, and polling tests;
|
||||
- update config docs and examples when operators need new configuration;
|
||||
- add provider integration notes when the provider contract needs maintenance
|
||||
context.
|
||||
|
||||
## Adding A Normalizer
|
||||
|
||||
Normalizers own provider-to-canonical mapping.
|
||||
|
||||
Checklist:
|
||||
|
||||
- add one normalizer type per normalizer file;
|
||||
- match using `Event.Schema`;
|
||||
- decode raw payloads into provider structs;
|
||||
- map to canonical `model` payloads;
|
||||
- use `internal/normalizers/common.Finalize` so envelope handling and float
|
||||
rounding stay consistent;
|
||||
- preserve input envelope fields except schema, payload, and effective time;
|
||||
- register through the provider package and `internal/normalizers/builtins.go`;
|
||||
- add tests for schema matching, key payload fields, effective time, malformed
|
||||
required data, and output validation.
|
||||
|
||||
## Adding Canonical Models Or Schemas
|
||||
|
||||
Canonical event changes affect multiple contracts.
|
||||
|
||||
Checklist:
|
||||
|
||||
- update payload structs in `model`;
|
||||
- add or update schema constants in `standards`;
|
||||
- update [event wire contract docs](../integrations/events.md);
|
||||
- update normalizers that produce the schema;
|
||||
- update Postgres mapping if the schema is persisted;
|
||||
- add tests for wire shape and mapper behavior.
|
||||
|
||||
## Adding Postgres Mapping
|
||||
|
||||
Weatherfeeder owns the canonical-event-to-table mapping. Feedkit owns the
|
||||
generic Postgres sink mechanics.
|
||||
|
||||
Checklist:
|
||||
|
||||
- update `internal/sinks/postgres/schema.go`;
|
||||
- update `internal/sinks/postgres/map.go`;
|
||||
- preserve event envelope columns in parent rows when the table supports them;
|
||||
- validate required canonical fields before writing;
|
||||
- use positional indexes for child rows that represent arrays;
|
||||
- update mapper and schema tests;
|
||||
- update [Postgres integration docs](../integrations/postgres.md) when the table
|
||||
contract changes.
|
||||
|
||||
## Examples And Documentation
|
||||
|
||||
Documentation must follow the [documentation policy](documentation.md).
|
||||
|
||||
When behavior changes, update the canonical docs in the same change:
|
||||
|
||||
- config shape: `docs/config.md`;
|
||||
- CLI behavior: `docs/cli.md`;
|
||||
- operations and recovery: `docs/operations.md`;
|
||||
- troubleshooting: `docs/troubleshooting.md`;
|
||||
- external contracts: `docs/integrations/`;
|
||||
- internal component behavior: `docs/internal/`;
|
||||
- copyable configs: `examples/`.
|
||||
|
||||
Keep roadmap content under `docs/roadmap/`. Current-behavior docs must describe
|
||||
implemented behavior only.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
Before committing:
|
||||
|
||||
- run focused tests for changed packages;
|
||||
- run `go test ./...` for broad behavior or documentation changes tied to code;
|
||||
- verify maintained examples still load when examples or config docs changed;
|
||||
- check links in changed docs;
|
||||
- search for stale paths, unsupported features, and secret-like values;
|
||||
- keep unrelated refactors out of the change.
|
||||
236
docs/troubleshooting.md
Normal file
236
docs/troubleshooting.md
Normal file
@@ -0,0 +1,236 @@
|
||||
# Troubleshooting
|
||||
|
||||
Use this guide with [configuration](config.md) and [operations](operations.md).
|
||||
Messages are emitted through the standard logger on stderr.
|
||||
|
||||
## `config load failed: ... read "config.yml"`
|
||||
|
||||
Symptom: startup exits before building sources or sinks.
|
||||
|
||||
Likely cause: the process working directory does not contain `config.yml`, or
|
||||
the runtime user cannot read it.
|
||||
|
||||
Diagnostic: run `pwd` in the same working directory used by the process, then
|
||||
check `ls -l config.yml`.
|
||||
|
||||
Safe fix: place the intended config at `./config.yml`, change the working
|
||||
directory, or mount the file at `/weatherfeeder/config.yml` when using the
|
||||
provided container image.
|
||||
|
||||
## `config load failed: ... parse YAML`
|
||||
|
||||
Symptom: startup exits with a YAML parse error or an unknown field error.
|
||||
|
||||
Likely cause: invalid YAML syntax, multiple YAML documents, or a misspelled
|
||||
config struct field.
|
||||
|
||||
Diagnostic: inspect the line and field in the error. Feedkit uses strict YAML
|
||||
field decoding for config struct fields.
|
||||
|
||||
Safe fix: correct the YAML and compare the shape with
|
||||
[configuration](config.md). Driver-specific `params` keys are validated by their
|
||||
source or sink constructors.
|
||||
|
||||
## `config validation failed`
|
||||
|
||||
Symptom: startup exits and prints one or more validation messages.
|
||||
|
||||
Likely cause: missing `sources` or `sinks`, blank names, duplicate source or sink
|
||||
names, invalid `mode`, missing `every` for a polling source with `mode: poll`, or
|
||||
a route that references an unknown sink.
|
||||
|
||||
Diagnostic: read every bullet under `config validation failed`; the loader sorts
|
||||
these messages so multiple issues can be fixed in one edit.
|
||||
|
||||
Safe fix: update the top-level config fields as documented in
|
||||
[configuration](config.md).
|
||||
|
||||
## `unknown source driver`
|
||||
|
||||
Symptom: startup exits with `build source failed`.
|
||||
|
||||
Likely cause: `sources[].driver` does not match a registered weatherfeeder
|
||||
source driver.
|
||||
|
||||
Diagnostic: compare the configured driver with the source driver table in
|
||||
[configuration](config.md#source-drivers).
|
||||
|
||||
Safe fix: correct the driver name. Current drivers include `nws_observation`,
|
||||
`nws_alerts`, `nws_forecast_hourly`, `nws_forecast_narrative`,
|
||||
`nws_forecast_discussion`, `nws_weatherstories`, `openmeteo_observation`,
|
||||
`openmeteo_forecast`, and `openweather_observation`.
|
||||
|
||||
## `unknown sink driver`
|
||||
|
||||
Symptom: startup exits with `build sink failed`.
|
||||
|
||||
Likely cause: `sinks[].driver` is not registered.
|
||||
|
||||
Diagnostic: compare the configured driver with the sink driver table in
|
||||
[configuration](config.md#sink-drivers).
|
||||
|
||||
Safe fix: use `stdout`, `nats`, or `postgres`.
|
||||
|
||||
## `source expected kinds validation failed`
|
||||
|
||||
Symptom: startup exits after building a source.
|
||||
|
||||
Likely cause: `sources[].kinds` declares a kind the source does not emit.
|
||||
|
||||
Diagnostic: compare the configured `kinds` list with the source driver kind in
|
||||
[configuration](config.md#source-drivers).
|
||||
|
||||
Safe fix: remove `kinds` or set it to the kind emitted by that driver.
|
||||
|
||||
## `params.url is required` Or `params.user_agent is required`
|
||||
|
||||
Symptom: startup exits with `build source failed`.
|
||||
|
||||
Likely cause: a source is missing required HTTP params, or the values are blank
|
||||
or not strings.
|
||||
|
||||
Diagnostic: inspect the named source in the error and check its `params`.
|
||||
|
||||
Safe fix: add non-empty `url` and `user_agent` values. See
|
||||
[HTTP source params](config.md#http-source-params).
|
||||
|
||||
## `url must include units=metric`
|
||||
|
||||
Symptom: startup exits for an `openweather_observation` source.
|
||||
|
||||
Likely cause: the OpenWeather URL omits `units=metric` or sets another unit
|
||||
system.
|
||||
|
||||
Diagnostic: inspect the query string in `params.url`.
|
||||
|
||||
Safe fix: add `units=metric` to the OpenWeather current-weather URL. Keep API
|
||||
keys out of committed configs.
|
||||
|
||||
## `source ... sources[].every must be > 0 for polling sources`
|
||||
|
||||
Symptom: startup exits while building scheduler jobs.
|
||||
|
||||
Likely cause: a current weatherfeeder polling source has no usable `every`
|
||||
interval.
|
||||
|
||||
Diagnostic: inspect the named `sources[]` entry and check `every`.
|
||||
|
||||
Safe fix: set a positive duration such as `1m`, `10m`, or `1h`.
|
||||
|
||||
## `build sink failed ... params.url is required`
|
||||
|
||||
Symptom: startup exits while building a NATS sink.
|
||||
|
||||
Likely cause: the NATS sink is missing `params.url`, or the value is blank or
|
||||
not a string.
|
||||
|
||||
Diagnostic: inspect the named sink in the error and check its `params`.
|
||||
|
||||
Safe fix: set a NATS URL such as `nats://localhost:4222`.
|
||||
|
||||
## `build sink failed ... params.subject is required`
|
||||
|
||||
Symptom: startup exits while building a NATS sink.
|
||||
|
||||
Likely cause: the NATS sink is missing `params.subject`, or the value is blank
|
||||
or not a string.
|
||||
|
||||
Diagnostic: inspect the named sink in the error and check its `params`.
|
||||
|
||||
Safe fix: set a non-empty subject such as `weatherfeeder`.
|
||||
|
||||
## `dispatch: sink ... failed consuming event ... NATS sink: connect`
|
||||
|
||||
Symptom: the daemon starts, but NATS events are not published.
|
||||
|
||||
Likely cause: the NATS server URL is unreachable, the server is not accepting
|
||||
connections, or the configured URL is wrong for the runtime network.
|
||||
|
||||
Diagnostic: from the same runtime environment, check that the host and port in
|
||||
`sinks[].params.url` are reachable.
|
||||
|
||||
Safe fix: correct the NATS URL or restore broker connectivity. Other configured
|
||||
sinks continue receiving events.
|
||||
|
||||
## `postgres sink ... open db`
|
||||
|
||||
Symptom: startup exits while building a Postgres sink.
|
||||
|
||||
Likely cause: the database URI, username, password, network path, or database
|
||||
availability is wrong.
|
||||
|
||||
Diagnostic: inspect `sinks[].params.uri`, `username`, and `password`; verify
|
||||
that the same runtime environment can reach the database.
|
||||
|
||||
Safe fix: correct the credentials or URI, restore database connectivity, then
|
||||
restart the daemon.
|
||||
|
||||
## `postgres sink ... ensure table` Or `ensure index`
|
||||
|
||||
Symptom: startup exits during Postgres initialization.
|
||||
|
||||
Likely cause: the database user cannot create required tables or indexes, an
|
||||
existing object conflicts with weatherfeeder's expected table contract, or the
|
||||
database is unavailable during initialization.
|
||||
|
||||
Diagnostic: inspect the named table or index in the error and compare existing
|
||||
database objects with the [Postgres table contract](integrations/postgres.md).
|
||||
|
||||
Safe fix: grant the needed database privileges, create a compatible schema, or
|
||||
perform an operator-managed migration before restarting.
|
||||
|
||||
## `postgres sink: insert into ...`
|
||||
|
||||
Symptom: the daemon starts, but Postgres writes for some events fail.
|
||||
|
||||
Likely cause: a duplicate primary key, incompatible existing table definition,
|
||||
database constraint error, or connection failure during a write transaction.
|
||||
|
||||
Diagnostic: inspect the table name and database error in the log. Compare the
|
||||
table with [Postgres integration](integrations/postgres.md).
|
||||
|
||||
Safe fix: repair the database schema or address the duplicate/connection issue.
|
||||
Other configured sinks continue receiving events.
|
||||
|
||||
## No Events Appear On A Sink
|
||||
|
||||
Symptom: the daemon is running but the expected sink receives no events.
|
||||
|
||||
Likely cause: the route does not match the event kind, the source has not
|
||||
emitted changed content, or the sink is failing per event.
|
||||
|
||||
Diagnostic: check `routes`, source `kinds`, and logs for `scheduler: poll
|
||||
failed`, `dispatcher: pipeline error`, or `dispatch: sink ... failed consuming
|
||||
event`.
|
||||
|
||||
Safe fix: correct the route or source configuration. If the source uses
|
||||
conditional HTTP and the upstream has not changed, no event is emitted for a
|
||||
`304 Not Modified` response; wait for changed upstream content or temporarily
|
||||
set `params.conditional: false` for diagnosis.
|
||||
|
||||
## `scheduler: poll failed`
|
||||
|
||||
Symptom: one source logs poll failures while the daemon keeps running.
|
||||
|
||||
Likely cause: upstream HTTP error, bad URL, timeout, response body limit, or
|
||||
provider response shape that the source cannot parse.
|
||||
|
||||
Diagnostic: inspect the source name in the log and review its HTTP params.
|
||||
|
||||
Safe fix: correct the URL, user agent, timeout, or body limit. The next
|
||||
scheduled poll will retry.
|
||||
|
||||
## `dispatcher: pipeline error`
|
||||
|
||||
Symptom: source polling succeeds, but one event is dropped before sinks.
|
||||
|
||||
Likely cause: a normalizer could not decode or map the raw payload, or dedupe
|
||||
received an invalid event ID.
|
||||
|
||||
Diagnostic: inspect the error text and the source/schema that produced the
|
||||
event. Review the [event wire contract](integrations/events.md) for expected
|
||||
canonical fields.
|
||||
|
||||
Safe fix: correct source configuration if it points to the wrong upstream
|
||||
product. If the upstream payload changed shape, update the relevant normalizer
|
||||
and tests.
|
||||
19
examples/config.minimal.yml
Normal file
19
examples/config.minimal.yml
Normal file
@@ -0,0 +1,19 @@
|
||||
---
|
||||
sources:
|
||||
- name: NWSObservationKSTL
|
||||
mode: poll
|
||||
kinds: ["observation"]
|
||||
driver: nws_observation
|
||||
every: 10m
|
||||
params:
|
||||
url: "https://api.weather.gov/stations/KSTL/observations/latest"
|
||||
user_agent: "weatherfeeder example (operator@example.com)"
|
||||
|
||||
sinks:
|
||||
- name: stdout
|
||||
driver: stdout
|
||||
params: {}
|
||||
|
||||
routes:
|
||||
- sink: stdout
|
||||
kinds: ["observation"]
|
||||
30
examples/config.nats.yml
Normal file
30
examples/config.nats.yml
Normal file
@@ -0,0 +1,30 @@
|
||||
---
|
||||
sources:
|
||||
- name: NWSObservationKSTL
|
||||
mode: poll
|
||||
kinds: ["observation"]
|
||||
driver: nws_observation
|
||||
every: 10m
|
||||
params:
|
||||
url: "https://api.weather.gov/stations/KSTL/observations/latest"
|
||||
user_agent: "weatherfeeder example (operator@example.com)"
|
||||
|
||||
- name: NWSAlertsSTL
|
||||
mode: poll
|
||||
kinds: ["alert"]
|
||||
driver: nws_alerts
|
||||
every: 1m
|
||||
params:
|
||||
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
|
||||
user_agent: "weatherfeeder example (operator@example.com)"
|
||||
|
||||
sinks:
|
||||
- name: nats_weather
|
||||
driver: nats
|
||||
params:
|
||||
url: nats://localhost:4222
|
||||
subject: weatherfeeder.events
|
||||
|
||||
routes:
|
||||
- sink: nats_weather
|
||||
kinds: ["observation", "alert"]
|
||||
32
examples/config.postgres.yml
Normal file
32
examples/config.postgres.yml
Normal file
@@ -0,0 +1,32 @@
|
||||
---
|
||||
sources:
|
||||
- name: NWSObservationKSTL
|
||||
mode: poll
|
||||
kinds: ["observation"]
|
||||
driver: nws_observation
|
||||
every: 10m
|
||||
params:
|
||||
url: "https://api.weather.gov/stations/KSTL/observations/latest"
|
||||
user_agent: "weatherfeeder example (operator@example.com)"
|
||||
|
||||
- name: OpenMeteoHourlyForecastSTL
|
||||
mode: poll
|
||||
kinds: ["forecast"]
|
||||
driver: openmeteo_forecast
|
||||
every: 1h
|
||||
params:
|
||||
url: "https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&hourly=temperature_2m,relative_humidity_2m,dew_point_2m,apparent_temperature,precipitation_probability,precipitation,snowfall,weather_code,surface_pressure,wind_speed_10m,wind_direction_10m&forecast_days=3"
|
||||
user_agent: "weatherfeeder example (operator@example.com)"
|
||||
|
||||
sinks:
|
||||
- name: pg_weather
|
||||
driver: postgres
|
||||
params:
|
||||
uri: "postgres://postgres.example.invalid:5432/weatherfeeder?sslmode=disable"
|
||||
username: <database_username>
|
||||
password: <database_password>
|
||||
prune: 3d
|
||||
|
||||
routes:
|
||||
- sink: pg_weather
|
||||
kinds: ["observation", "forecast"]
|
||||
@@ -9,12 +9,12 @@ import (
|
||||
)
|
||||
|
||||
// Finalize builds the output event envelope by copying the input and applying the
|
||||
// canonical schema/payload, plus (optionally) EffectiveAt.
|
||||
// canonical schema/payload, plus an optional effective time.
|
||||
//
|
||||
// Important behavior:
|
||||
// - ID/Kind/Source/EmittedAt are preserved by copying the input event.
|
||||
// - EffectiveAt is only overwritten when effectiveAt is non-zero.
|
||||
// If effectiveAt is zero, any existing in.EffectiveAt is preserved.
|
||||
// - EffectiveAt is only overwritten when the supplied effective time is non-zero.
|
||||
// If the supplied time is zero, any existing in.EffectiveAt is preserved.
|
||||
// - Payload floats are rounded to a stable wire-friendly precision (see round.go).
|
||||
func Finalize(in event.Event, outSchema string, outPayload any, effectiveAt time.Time) (*event.Event, error) {
|
||||
// Enforce stable numeric presentation for weather payloads before delegating to feedkit's
|
||||
|
||||
@@ -15,7 +15,7 @@ import (
|
||||
// - sources emit raw JSON payloads (typically json.RawMessage)
|
||||
// - normalizers decode into provider structs
|
||||
//
|
||||
// Errors include a small amount of stage context ("extract payload", "decode raw payload").
|
||||
// Errors include a small amount of operation context ("extract payload", "decode raw payload").
|
||||
// Callers typically wrap these with a provider/kind label.
|
||||
func DecodeJSONPayload[T any](in event.Event) (T, error) {
|
||||
return fknormalize.DecodeJSONPayload[T](in)
|
||||
@@ -24,15 +24,15 @@ func DecodeJSONPayload[T any](in event.Event) (T, error) {
|
||||
// NormalizeJSON is a convenience wrapper for the common JSON-normalizer pattern:
|
||||
//
|
||||
// 1. Decode raw JSON payload into provider struct T
|
||||
// 2. Map T into canonical payload P (plus an EffectiveAt timestamp)
|
||||
// 3. Finalize the event envelope (schema/payload/effectiveAt) + Validate
|
||||
// 2. Map T into canonical payload P (plus an effective time)
|
||||
// 3. Finalize the event envelope (schema/payload/effective time) + Validate
|
||||
//
|
||||
// label should be short and specific, e.g. "openweather observation".
|
||||
// outSchema should be the canonical schema constant.
|
||||
// build should contain ONLY provider/domain mapping logic.
|
||||
//
|
||||
// Error policy:
|
||||
// - NormalizeJSON wraps ALL failures with consistent context: "<label> normalize: <stage>: ..."
|
||||
// - NormalizeJSON wraps ALL failures with consistent context: "<label> normalize: <operation>: ..."
|
||||
// - build() should return specific errors without repeating the label prefix.
|
||||
func NormalizeJSON[T any, P any](
|
||||
in event.Event,
|
||||
|
||||
@@ -70,7 +70,7 @@
|
||||
//
|
||||
// weather.<kind>.vN
|
||||
//
|
||||
// weatherfeeder centralizes schema strings in internal/standards/schema.go.
|
||||
// weatherfeeder centralizes schema strings in standards/schema.go.
|
||||
// Always use those constants (do not inline schema strings).
|
||||
//
|
||||
// Example mappings:
|
||||
@@ -101,8 +101,8 @@
|
||||
// Every normalizer type must have a doc comment that states:
|
||||
//
|
||||
// - what it converts (e.g., “OpenWeather current -> WeatherObservation”)
|
||||
// - which raw schema it matches (constant identifier from internal/standards)
|
||||
// - which canonical schema it produces (constant identifier from internal/standards)
|
||||
// - which raw schema it matches (constant identifier from standards)
|
||||
// - which canonical schema it produces (constant identifier from standards)
|
||||
// - any special caveats (units, day/night inference, missing fields, etc.)
|
||||
//
|
||||
// Including literal schema string values is optional,
|
||||
|
||||
@@ -29,7 +29,7 @@ import (
|
||||
// 2. Alert timing fields are best-effort parsed; invalid timestamps do not fail the
|
||||
// entire normalization (they are left nil).
|
||||
// 3. Some fields are intentionally passed through as strings (severity/urgency/etc.)
|
||||
// since canonical vocabularies may evolve later.
|
||||
// because the canonical model currently preserves provider vocabulary there.
|
||||
type AlertsNormalizer struct{}
|
||||
|
||||
func (AlertsNormalizer) Match(e event.Event) bool {
|
||||
@@ -37,7 +37,7 @@ func (AlertsNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (AlertsNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
// If we can't derive AsOf from the payload, fall back to the existing event envelope.
|
||||
fallbackAsOf := in.EmittedAt.UTC()
|
||||
|
||||
@@ -42,7 +42,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
return normalizeForecastEventBySchema(in)
|
||||
}
|
||||
|
||||
@@ -32,7 +32,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
return normcommon.NormalizeJSON(
|
||||
in,
|
||||
|
||||
@@ -33,7 +33,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
// If present, prefer the existing event EmittedAt as IssuedAt.
|
||||
var fallbackIssued time.Time
|
||||
|
||||
@@ -40,7 +40,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
return normcommon.NormalizeJSON(
|
||||
in,
|
||||
|
||||
@@ -8,8 +8,7 @@ import (
|
||||
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
|
||||
)
|
||||
|
||||
// This file holds provider-specific helpers that are shared across multiple
|
||||
// OpenWeather normalizers (observations today; forecasts/alerts later).
|
||||
// This file holds provider-specific helpers for OpenWeather normalizers.
|
||||
// Keeping these out of observation.go helps preserve the "one normalizer per file"
|
||||
// convention while avoiding duplication.
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
return normcommon.NormalizeJSON(
|
||||
in,
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// Package openweather contains provider-specific helper code for OpenWeather used by
|
||||
// both sources and normalizers.
|
||||
// Package openweather contains provider-specific helper code for OpenWeather
|
||||
// used by sources and normalizers.
|
||||
//
|
||||
// Rules:
|
||||
// - No network I/O here.
|
||||
|
||||
@@ -42,7 +42,7 @@ type WeatherAlert struct {
|
||||
Headline string `json:"headline,omitempty"`
|
||||
|
||||
Severity string `json:"severity,omitempty"` // e.g. Extreme/Severe/Moderate/Minor/Unknown
|
||||
Urgency string `json:"urgency,omitempty"` // e.g. Immediate/Expected/Future/Past/Unknown
|
||||
Urgency string `json:"urgency,omitempty"` // provider-defined urgency value
|
||||
Certainty string `json:"certainty,omitempty"` // e.g. Observed/Likely/Possible/Unlikely/Unknown
|
||||
|
||||
Status string `json:"status,omitempty"` // e.g. Actual/Exercise/Test/System/Unknown
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
// Package model defines weatherfeeder's canonical domain payload types.
|
||||
//
|
||||
// These structs are emitted as the Payload of canonical events (schemas "weather.*.vN").
|
||||
// JSON tags are treated as part of the wire contract for sinks (stdout today; others later).
|
||||
// JSON tags are treated as part of the wire contract for configured sinks.
|
||||
//
|
||||
// Compatibility guidance:
|
||||
// - Prefer additive changes.
|
||||
|
||||
@@ -34,7 +34,7 @@ type WeatherForecastRun struct {
|
||||
LocationName string `json:"locationName,omitempty"`
|
||||
IssuedAt time.Time `json:"issuedAt"` // required: when this run was generated/issued
|
||||
|
||||
// Some providers include both a generated time and a later update time.
|
||||
// Some providers include both a generated time and a subsequent update time.
|
||||
// Keep UpdatedAt optional; many providers won’t supply it.
|
||||
UpdatedAt *time.Time `json:"updatedAt,omitempty"`
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// File: internal/standards/doc.go
|
||||
// File: standards/doc.go
|
||||
//
|
||||
// Package standards defines weatherfeeder’s provider-agnostic “project law”:
|
||||
//
|
||||
|
||||
Reference in New Issue
Block a user