From abb8f218ec1986785fa7c8ea1e2093eb99b3e782 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 10 Jun 2026 20:12:27 +0000 Subject: [PATCH] Document event and Postgres integration contracts --- API.md | 406 +--------------------------------- README.md | 2 + docs/config.md | 3 +- docs/integrations/events.md | 238 ++++++++++++++++++++ docs/integrations/postgres.md | 403 +++++++++++++++++++++++++++++++++ docs/policy/architecture.md | 13 +- 6 files changed, 658 insertions(+), 407 deletions(-) create mode 100644 docs/integrations/events.md create mode 100644 docs/integrations/postgres.md diff --git a/API.md b/API.md index 5e93758..cdddd7e 100644 --- a/API.md +++ b/API.md @@ -1,404 +1,4 @@ -# weatherfeeder API (Wire Contract) +# Event Wire Contract -This document defines the stable, consumer-facing JSON contract emitted by weatherfeeder sinks. - -weatherfeeder emits **events** encoded as JSON. Each event has: -- an **envelope** (metadata + schema identifier), and -- a **payload** whose shape is determined by `schema`. - -Downstream consumers should: -1. parse the event envelope, -2. switch on `schema`, then -3. decode `payload` into the matching schema. - ---- - -## Event envelope - -All events are JSON objects with these fields: - -| Field | Type | Required | Notes | -|---|---:|:---:|---| -| `id` | string | yes | Stable event identifier. Treat as opaque. | -| `schema` | string | yes | Schema identifier (e.g. `weather.observation.v1`). | -| `source` | string | yes | Provider/source identifier (stable within configuration). | -| `effectiveAt` | string (timestamp) | yes | RFC3339Nano timestamp indicating when this event is effective. | -| `payload` | object | yes | Schema-specific payload (see below). | - -### Timestamp format - -All timestamps are encoded as JSON strings using Go’s `time.Time` JSON encoding (RFC3339Nano). - -Examples: -- `"2026-01-17T14:27:00Z"` -- `"2026-01-17T08:27:00-06:00"` - ---- - -## Canonical schemas - -weatherfeeder emits five canonical domain schemas: - -- `weather.observation.v1` -- `weather.forecast.v1` -- `weather.forecast_discussion.v1` -- `weather.weather_story.v1` -- `weather.alert.v1` - -Each payload is described below using the JSON field names as the contract. - -### Raw upstream schemas - -weatherfeeder sources also emit provider-specific raw schemas before normalization. -Relevant raw source schemas include: - -- `raw.nws.forecast_discussion.v1` - - payload type: string - - payload contents: exact fetched HTML response body -- `raw.nws.weatherstories.v1` - - payload type: object - - payload contents: exact fetched JSON response body - ---- - -## Shared Conventions - -- Timestamps are JSON strings in RFC3339Nano format. -- Optional fields are omitted when unknown (`omitempty` behavior). -- Numeric measurements are normalized to metric units: - - `*C` = Celsius - - `*Kmh` = kilometers/hour - - `*Pa` = Pascals - - `*Meters` = meters - - `*Mm` = millimeters - - `*Percent` = percent (0-100) -- `conditionCode` is a WMO weather interpretation code (`int`). - - Unknown/unmappable is `-1`. - - Downstream consumers should treat unknown codes as “unknown conditions” rather than failing decoding. -- For readability and stability, weatherfeeder rounds floating-point values in canonical payloads to -**4 digits after the decimal** during normalization. - ---- - -## Schema: `weather.observation.v1` - -Payload type: `WeatherObservation` - -A `WeatherObservation` represents a point-in-time observation for a station/location. - -### Fields - -| Field | Type | Required | Notes | -|---|---:|:---:|---| -| `stationId` | string | no | Provider station/location identifier | -| `stationName` | string | no | Human station name | -| `timestamp` | timestamp string | yes | Observation timestamp | -| `conditionCode` | int | yes | WMO code (`-1` unknown) | -| `isDay` | bool | no | Day/night hint | -| `textDescription` | string | no | Human-facing short description | -| `temperatureC` | number | no | Celsius | -| `dewpointC` | number | no | Celsius | -| `windDirectionDegrees` | number | no | Degrees | -| `windSpeedKmh` | number | no | km/h | -| `windGustKmh` | number | no | km/h | -| `barometricPressurePa` | number | no | Pascals | -| `visibilityMeters` | number | no | Meters | -| `relativeHumidityPercent` | number | no | Percent | -| `apparentTemperatureC` | number | no | Celsius | -| `presentWeather` | array | no | Provider-specific structured weather fragments | - -### Nested: `presentWeather[]` - -Each `presentWeather[]` element: - -| Field | Type | Required | Notes | -|---|---:|:---:|---| -| `raw` | object | no | Provider-specific JSON object | - ---- - -## Schema: `weather.forecast.v1` - -Payload type: `WeatherForecastRun` - -A `WeatherForecastRun` is a single issued forecast snapshot for a location and a specific product -(hourly / narrative / daily). The run contains an ordered list of forecast periods. - -### `product` values - -`product` is one of: - -- `"hourly"` -- `"narrative"` -- `"daily"` - -### Fields - -| Field | Type | Required | Notes | -|---|---:|:---:|---| -| `locationId` | string | no | Provider location identifier | -| `locationName` | string | no | Human name, if available | -| `issuedAt` | string (timestamp) | yes | When this run was generated/issued | -| `updatedAt` | string (timestamp) | no | Optional later update time | -| `product` | string | yes | One of `hourly`, `narrative`, `daily` | -| `latitude` | number | no | Degrees | -| `longitude` | number | no | Degrees | -| `elevationMeters` | number | no | meters | -| `periods` | array | yes | Chronological forecast periods | - -### Nested: `periods[]` (`WeatherForecastPeriod`) - -A `WeatherForecastPeriod` is valid for `[startTime, endTime)`. - -| Field | Type | Required | Units / Notes | -|---|---:|:---:|---| -| `startTime` | string (timestamp) | yes | Period start | -| `endTime` | string (timestamp) | yes | Period end | -| `name` | string | no | Human label (often empty for hourly) | -| `isDay` | bool | no | Day/night hint | -| `conditionCode` | int | no | WMO code when applicable (`-1` for unknown) | -| `textDescription` | string | no | Human-facing short phrase | -| `temperatureC` | number | no | °C | -| `temperatureCMin` | number | no | °C (aggregated products) | -| `temperatureCMax` | number | no | °C (aggregated products) | -| `dewpointC` | number | no | °C | -| `relativeHumidityPercent` | number | no | percent | -| `windDirectionDegrees` | number | no | degrees | -| `windSpeedKmh` | number | no | km/h | -| `windGustKmh` | number | no | km/h | -| `barometricPressurePa` | number | no | Pa | -| `visibilityMeters` | number | no | meters | -| `apparentTemperatureC` | number | no | °C | -| `cloudCoverPercent` | number | no | percent | -| `probabilityOfPrecipitationPercent` | number | no | percent | -| `precipitationAmountMm` | number | no | mm (liquid equivalent) | -| `snowfallDepthMm` | number | no | mm | -| `uvIndex` | number | no | unitless index | - ---- - -## Schema: `weather.alert.v1` - -Payload type: `WeatherAlertRun` - -A `WeatherAlertRun` is a snapshot of *active* alerts for a location as-of a point in time. -A run may contain zero, one, or many alerts. - -### Fields - -| Field | Type | Required | Notes | -|---|---:|:---:|---| -| `locationId` | string | no | Provider location identifier | -| `locationName` | string | no | Human name, if available | -| `asOf` | string (timestamp) | yes | When the provider asserted this snapshot is current | -| `latitude` | number | no | Degrees | -| `longitude` | number | no | Degrees | -| `alerts` | array | yes | Active alerts (order provider-dependent) | - -### Nested: `alerts[]` (`WeatherAlert`) - -| Field | Type | Required | Notes | -|---|---:|:---:|---| -| `id` | string | yes | Provider-stable identifier (often a URL/URI) | -| `event` | string | no | Classification/event label | -| `headline` | string | no | Alert headline | -| `severity` | string | no | Example: Extreme/Severe/Moderate/Minor/Unknown | -| `urgency` | string | no | Example: Immediate/Expected/Future/Past/Unknown | -| `certainty` | string | no | Example: Observed/Likely/Possible/Unlikely/Unknown | -| `status` | string | no | Example: Actual/Exercise/Test/System/Unknown | -| `messageType` | string | no | Example: Alert/Update/Cancel | -| `category` | string | no | Example: Met/Geo/Safety/Rescue/Fire/Health/Env/Transport/Infra/CBRNE/Other | -| `response` | string | no | Example: Shelter/Evacuate/Prepare/Execute/Avoid/Monitor/Assess/AllClear/None | -| `response` | string | no | e.g. Shelter/Evacuate/Prepare/... | -| `description` | string | no | Narrative | -| `instruction` | string | no | What to do | -| `sent` | string (timestamp) | no | Provider-dependent | -| `effective` | string (timestamp) | no | Provider-dependent | -| `onset` | string (timestamp) | no | Provider-dependent | -| `expires` | string (timestamp) | no | Provider-dependent | -| `areaDescription` | string | no | Often a provider string | -| `senderName` | string | no | Provenance | -| `references` | array | no | Related alert references | - -### Nested: `references[]` (`AlertReference`) - -| Field | Type | Required | Notes | -|---|---:|:---:|---| -| `id` | string | no | Provider reference ID/URI | -| `identifier` | string | no | Provider identifier string, if distinct | -| `sender` | string | no | Sender | -| `sent` | string (timestamp) | no | Timestamp | - ---- - -## Schema: `weather.weather_story.v1` - -Payload type: `WeatherStoryRun` - -A `WeatherStoryRun` is a snapshot of NWS weather stories for an office as-of a point in time. -The run may contain zero, one, or many stories. - -### Fields - -| Field | Type | Required | Notes | -|---|---:|:---:|---| -| `officeId` | string | no | NWS office identifier, e.g. `LSX` | -| `asOf` | string (timestamp) | yes | Latest story update time or source fallback | -| `stories` | array | yes | Weather stories (order provider-dependent) | - -### Nested: `stories[]` (`WeatherStory`) - -| Field | Type | Required | Notes | -|---|---:|:---:|---| -| `officeId` | string | no | NWS office identifier | -| `startTime` | string (timestamp) | yes | Story validity start | -| `endTime` | string (timestamp) | yes | Story validity end | -| `updatedAt` | string (timestamp) | yes | Story update time | -| `title` | string | no | Human story title | -| `description` | string | no | Story narrative text | -| `altText` | string | no | Accessibility text for the provider graphic | -| `priority` | bool | yes | Provider priority flag | -| `order` | int | yes | Provider display order | -| `downloadUrl` | string | no | Provider download URL; weatherfeeder does not fetch the asset | - ---- - -## Schema: `weather.forecast_discussion.v1` - -Payload type: `WeatherForecastDiscussion` - -A `WeatherForecastDiscussion` is an issued narrative bulletin for an NWS office. -It is distinct from `weather.forecast.v1`, which is period-based. - -### Fields - -| Field | Type | Required | Notes | -|---|---:|:---:|---| -| `officeId` | string | no | NWS office identifier, e.g. `LSX` | -| `officeName` | string | no | Human office name | -| `product` | string | yes | Currently `afd` | -| `issuedAt` | string (timestamp) | yes | Bulletin issue time | -| `updatedAt` | string (timestamp) | no | Optional page/update timestamp | -| `keyMessages` | array | no | Ordered key-message bullet list | -| `shortTerm` | object | no | Short-term discussion section | -| `longTerm` | object | no | Long-term discussion section | - -### Nested: `shortTerm` / `longTerm` - -| Field | Type | Required | Notes | -|---|---:|:---:|---| -| `qualifier` | string | no | Header qualifier such as `(Through Late Sunday Night)` | -| `issuedAt` | string (timestamp) | no | Optional section-local issue time | -| `text` | string | no | Paragraph-preserved prose text | - ---- - -## Compatibility rules - -- Consumers **must** ignore unknown fields. -- Producers (weatherfeeder) prefer **additive changes** within a schema version. -- Renames/removals/semantic breaks normally require a **schema version bump** (`weather.*.v2`); pre-1.0 projects may choose in-place changes. - ---- - -## Examples - -### Observation event (`weather.observation.v1`) - -```json -{ - "id": "nws:KSTL:2026-01-17T14:00:00Z", - "schema": "weather.observation.v1", - "source": "nws_observation", - "effectiveAt": "2026-01-17T14:00:00Z", - "payload": { - "stationId": "KSTL", - "timestamp": "2026-01-17T14:00:00Z", - "conditionCode": 1, - "textDescription": "Mainly Sunny", - "temperatureC": 3.25, - "windSpeedKmh": 18.5 - } -} -``` - -### Forecast event (`weather.forecast.v1`) - -```json -{ - "id": "openmeteo:38.63,-90.20:2026-01-17T13:00:00Z", - "schema": "weather.forecast.v1", - "source": "openmeteo_forecast", - "effectiveAt": "2026-01-17T13:00:00Z", - "payload": { - "locationName": "St. Louis, MO", - "issuedAt": "2026-01-17T13:00:00Z", - "product": "hourly", - "latitude": 38.63, - "longitude": -90.2, - "periods": [ - { - "startTime": "2026-01-17T14:00:00Z", - "endTime": "2026-01-17T15:00:00Z", - "conditionCode": 2, - "textDescription": "Partly Cloudy", - "temperatureC": 3.5, - "probabilityOfPrecipitationPercent": 10 - } - ] - } -} -``` - -### Alert event (`weather.alert.v1`) - -```json -{ - "id": "nws:alerts:2026-01-17T14:10:00Z", - "schema": "weather.alert.v1", - "source": "nws_alerts", - "effectiveAt": "2026-01-17T14:10:00Z", - "payload": { - "asOf": "2026-01-17T14:05:00Z", - "alerts": [ - { - "id": "https://api.weather.gov/alerts/abc123", - "event": "Winter Weather Advisory", - "headline": "Winter Weather Advisory issued January 17 at 8:05AM CST", - "severity": "Moderate", - "description": "Mixed precipitation expected...", - "expires": "2026-01-18T06:00:00Z" - } - ] - } -} -``` - -### Weather story event (`weather.weather_story.v1`) - -```json -{ - "id": "nws:weatherstories:2026-05-30T09:00:34Z", - "schema": "weather.weather_story.v1", - "source": "nws_weatherstories", - "effectiveAt": "2026-05-30T09:00:34Z", - "payload": { - "officeId": "LSX", - "asOf": "2026-05-30T09:00:34Z", - "stories": [ - { - "officeId": "LSX", - "startTime": "2026-05-30T08:46:00Z", - "endTime": "2026-05-31T11:00:00Z", - "updatedAt": "2026-05-30T09:00:34Z", - "title": "Several Chances for Rain Through Monday", - "description": "Scattered showers and thunderstorms remain possible.", - "altText": "This slide shows the forecast for today through Tuesday.", - "priority": false, - "order": 1, - "downloadUrl": "https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f" - } - ] - } -} -``` +The canonical event wire contract has moved to +[docs/integrations/events.md](docs/integrations/events.md). diff --git a/README.md b/README.md index 9e87768..a7009f6 100644 --- a/README.md +++ b/README.md @@ -26,5 +26,7 @@ current working directory. - [CLI reference](docs/cli.md) - [Configuration reference](docs/config.md) +- [Event wire contract](docs/integrations/events.md) +- [Postgres table contract](docs/integrations/postgres.md) - [Architecture policy](docs/policy/architecture.md) - [Documentation policy](docs/policy/documentation.md) diff --git a/docs/config.md b/docs/config.md index 4bfb897..85ac9ef 100644 --- a/docs/config.md +++ b/docs/config.md @@ -164,7 +164,8 @@ Publishes each event as JSON to a NATS subject. ### `postgres` Writes supported canonical weather events to Postgres using weatherfeeder's -registered schema mapping. +registered schema mapping. The table contract is documented in +[Postgres integration](integrations/postgres.md). | Param | Required | Description | |---|:---:|---| diff --git a/docs/integrations/events.md b/docs/integrations/events.md new file mode 100644 index 0000000..3209296 --- /dev/null +++ b/docs/integrations/events.md @@ -0,0 +1,238 @@ +# Event Wire Contract + +This document is the canonical JSON contract for events emitted by +`weatherfeeder` JSON sinks, including stdout and NATS. Postgres stores the same +event envelope fields in parent table columns; see +[Postgres integration](postgres.md). + +Downstream consumers should read the envelope, switch on `schema`, and decode +`payload` according to that schema. + +## Envelope + +Every emitted event is a JSON object with these fields: + +| Field | Type | Required | Notes | +|---|---|:---:|---| +| `id` | string | yes | Stable event identifier. Treat as opaque. | +| `kind` | string | yes | Routing kind, such as `observation` or `alert`. | +| `source` | string | yes | Configured source name. | +| `emitted_at` | timestamp | yes | When the daemon emitted the event. | +| `effective_at` | timestamp | no | Timestamp the payload is about, when known. | +| `schema` | string | no | Schema identifier. Weatherfeeder sources and normalizers set this. | +| `payload` | object, array, string, or scalar | yes | Schema-specific payload. | + +Timestamps are JSON strings using Go `time.Time` JSON encoding, which is +RFC3339Nano-compatible. Weatherfeeder normalizers use UTC timestamps for +canonical payloads. + +## Kinds And Schemas + +Canonical schemas emitted after normalization: + +| Kind | Schema | Payload | +|---|---|---| +| `observation` | `weather.observation.v1` | `WeatherObservation` | +| `forecast` | `weather.forecast.v1` | `WeatherForecastRun` | +| `forecast_discussion` | `weather.forecast_discussion.v1` | `WeatherForecastDiscussion` | +| `weather_story` | `weather.weather_story.v1` | `WeatherStoryRun` | +| `alert` | `weather.alert.v1` | `WeatherAlertRun` | + +Raw upstream schemas emitted by current sources: + +| Kind | Schema | Payload | +|---|---|---| +| `observation` | `raw.nws.observation.v1` | NWS observation JSON | +| `observation` | `raw.openmeteo.current.v1` | Open-Meteo current JSON | +| `observation` | `raw.openweather.current.v1` | OpenWeather current JSON | +| `forecast` | `raw.nws.hourly.forecast.v1` | NWS hourly forecast JSON | +| `forecast` | `raw.nws.narrative.forecast.v1` | NWS narrative forecast JSON | +| `forecast_discussion` | `raw.nws.forecast_discussion.v1` | NWS forecast discussion HTML string | +| `weather_story` | `raw.nws.weatherstories.v1` | NWS weather stories JSON | +| `forecast` | `raw.openmeteo.hourly.forecast.v1` | Open-Meteo hourly forecast JSON | +| `alert` | `raw.nws.alerts.v1` | NWS alerts JSON | + +`standards.SchemaRawOpenWeatherHourlyForecastV1` exists in code, but no current +registered source emits it. + +## Shared Conventions + +- Canonical numeric measurements use metric units. +- Floating-point values in canonical payloads are rounded to 4 digits after the + decimal point during normalization. +- Optional fields use JSON `omitempty`; absent fields should be treated as + unknown. +- `conditionCode` is a WMO weather interpretation code. Unknown observation + conditions use `-1`. Forecast period `conditionCode` is optional. +- Additive fields are compatible within a schema version. Removing, renaming, or + changing the meaning of a field requires a new schema identifier. + +## `weather.observation.v1` + +Payload type: `WeatherObservation`. + +| Field | Type | Required | Notes | +|---|---|:---:|---| +| `stationId` | string | no | Provider station/location identifier. | +| `stationName` | string | no | Human station name. | +| `timestamp` | timestamp | yes | Observation timestamp. | +| `conditionCode` | integer | yes | WMO code; `-1` means unknown. | +| `isDay` | boolean | no | Day/night hint. | +| `textDescription` | string | no | Short human description. | +| `temperatureC` | number | no | Celsius. | +| `dewpointC` | number | no | Celsius. | +| `windDirectionDegrees` | number | no | Degrees. | +| `windSpeedKmh` | number | no | Kilometers per hour. | +| `windGustKmh` | number | no | Kilometers per hour. | +| `barometricPressurePa` | number | no | Pascals. | +| `visibilityMeters` | number | no | Meters. | +| `relativeHumidityPercent` | number | no | Percent from 0 to 100. | +| `apparentTemperatureC` | number | no | Celsius. | +| `presentWeather` | array | no | Provider-specific present weather fragments. | + +`presentWeather[]` entries contain optional `raw` objects. + +## `weather.forecast.v1` + +Payload type: `WeatherForecastRun`. + +| Field | Type | Required | Notes | +|---|---|:---:|---| +| `locationId` | string | no | Provider location identifier. | +| `locationName` | string | no | Human location name. | +| `issuedAt` | timestamp | yes | When the forecast run was generated or issued. | +| `updatedAt` | timestamp | no | Later provider update time. | +| `product` | string | yes | `hourly`, `narrative`, or `daily`. | +| `latitude` | number | no | Degrees. | +| `longitude` | number | no | Degrees. | +| `elevationMeters` | number | no | Meters. | +| `periods` | array | yes | Ordered forecast periods. | + +`periods[]` entries: + +| Field | Type | Required | Notes | +|---|---|:---:|---| +| `startTime` | timestamp | yes | Period start. | +| `endTime` | timestamp | yes | Period end. | +| `name` | string | no | Human label. | +| `isDay` | boolean | no | Day/night hint. | +| `conditionCode` | integer | no | WMO code when applicable. | +| `textDescription` | string | no | Human summary. | +| `temperatureC` | number | no | Celsius. | +| `temperatureCMin` | number | no | Celsius. | +| `temperatureCMax` | number | no | Celsius. | +| `dewpointC` | number | no | Celsius. | +| `relativeHumidityPercent` | number | no | Percent from 0 to 100. | +| `windDirectionDegrees` | number | no | Degrees. | +| `windSpeedKmh` | number | no | Kilometers per hour. | +| `windGustKmh` | number | no | Kilometers per hour. | +| `barometricPressurePa` | number | no | Pascals. | +| `visibilityMeters` | number | no | Meters. | +| `apparentTemperatureC` | number | no | Celsius. | +| `cloudCoverPercent` | number | no | Percent from 0 to 100. | +| `probabilityOfPrecipitationPercent` | number | no | Percent from 0 to 100. | +| `precipitationAmountMm` | number | no | Liquid-equivalent millimeters. | +| `snowfallDepthMm` | number | no | Millimeters. | +| `uvIndex` | number | no | Unitless index. | + +## `weather.forecast_discussion.v1` + +Payload type: `WeatherForecastDiscussion`. + +| Field | Type | Required | Notes | +|---|---|:---:|---| +| `officeId` | string | no | NWS office identifier. | +| `officeName` | string | no | Office name. | +| `product` | string | yes | Current value is `afd`. | +| `issuedAt` | timestamp | yes | Bulletin issue time. | +| `updatedAt` | timestamp | no | Later update time. | +| `keyMessages` | array of strings | no | Extracted key messages. | +| `shortTerm` | object | no | Short-term section. | +| `longTerm` | object | no | Long-term section. | + +`shortTerm` and `longTerm` sections contain optional `qualifier`, `issuedAt`, +and `text` fields. + +## `weather.weather_story.v1` + +Payload type: `WeatherStoryRun`. + +| Field | Type | Required | Notes | +|---|---|:---:|---| +| `officeId` | string | no | NWS office identifier. | +| `asOf` | timestamp | yes | Snapshot time. | +| `stories` | array | yes | Ordered story cards. | + +`stories[]` entries: + +| Field | Type | Required | Notes | +|---|---|:---:|---| +| `officeId` | string | no | Office identifier. | +| `startTime` | timestamp | yes | Story start. | +| `endTime` | timestamp | yes | Story end. | +| `updatedAt` | timestamp | yes | Story update time. | +| `title` | string | no | Story title. | +| `description` | string | no | Story description. | +| `altText` | string | no | Image alternate text. | +| `priority` | boolean | yes | Provider priority flag. | +| `order` | integer | yes | Provider display order. | +| `downloadUrl` | string | no | Story image URL. | + +## `weather.alert.v1` + +Payload type: `WeatherAlertRun`. + +| Field | Type | Required | Notes | +|---|---|:---:|---| +| `locationId` | string | no | Provider location identifier. | +| `locationName` | string | no | Human location name. | +| `asOf` | timestamp | yes | Snapshot time. | +| `latitude` | number | no | Degrees. | +| `longitude` | number | no | Degrees. | +| `alerts` | array | yes | Active alerts. | + +`alerts[]` entries: + +| Field | Type | Required | Notes | +|---|---|:---:|---| +| `id` | string | yes | Provider-stable alert identifier. | +| `event` | string | no | Alert event label. | +| `headline` | string | no | Alert headline. | +| `severity` | string | no | Provider severity. | +| `urgency` | string | no | Provider urgency. | +| `certainty` | string | no | Provider certainty. | +| `status` | string | no | Alert status. | +| `messageType` | string | no | Alert message type. | +| `category` | string | no | Alert category. | +| `response` | string | no | Recommended response. | +| `description` | string | no | Alert description. | +| `instruction` | string | no | Alert instruction. | +| `sent` | timestamp | no | Provider sent time. | +| `effective` | timestamp | no | Effective time. | +| `onset` | timestamp | no | Onset time. | +| `expires` | timestamp | no | Expiration time. | +| `areaDescription` | string | no | Affected area description. | +| `senderName` | string | no | Provider sender name. | +| `references` | array | no | Related alerts. | + +`references[]` entries contain optional `id`, `identifier`, `sender`, and +`sent` fields. + +## Compact Example + +```json +{ + "id": "NWSObservationKSTL:2026-06-10T12:00:00Z", + "kind": "observation", + "source": "NWSObservationKSTL", + "emitted_at": "2026-06-10T12:00:05Z", + "effective_at": "2026-06-10T12:00:00Z", + "schema": "weather.observation.v1", + "payload": { + "stationId": "KSTL", + "timestamp": "2026-06-10T12:00:00Z", + "conditionCode": 0, + "temperatureC": 22.5 + } +} +``` diff --git a/docs/integrations/postgres.md b/docs/integrations/postgres.md new file mode 100644 index 0000000..2ff6731 --- /dev/null +++ b/docs/integrations/postgres.md @@ -0,0 +1,403 @@ +# Postgres Integration + +This document is the canonical table contract for the optional `postgres` sink. +It describes the schema created and written by weatherfeeder through feedkit's +Postgres sink. + +Configure the sink as described in [configuration](../config.md#postgres). + +## Initialization And Writes + +At startup, each configured Postgres sink opens the database and runs +`CREATE TABLE IF NOT EXISTS` for every weatherfeeder table, followed by +`CREATE INDEX IF NOT EXISTS` for every configured index. + +This initialization creates missing tables and indexes only. It does not alter +existing tables, migrate column definitions, drop old objects, or backfill data. +Schema changes require operator-managed database migration. + +Events are mapped only for canonical weather schemas: + +- `weather.observation.v1` +- `weather.forecast.v1` +- `weather.forecast_discussion.v1` +- `weather.weather_story.v1` +- `weather.alert.v1` + +Unsupported schemas produce no writes for this sink. Mapped events are inserted +transactionally. Inserts use ordinary `INSERT`; duplicate primary keys fail the +write. + +## Shared Envelope Columns + +Parent tables store the feed event envelope: + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `event_id` | `TEXT` | no | `event.id` | +| `event_kind` | `TEXT` | no | `event.kind` | +| `event_source` | `TEXT` | no | `event.source` | +| `event_schema` | `TEXT` | no | `event.schema` | +| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` | +| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` | + +## Table Overview + +| Table | Primary key | Prune column | +|---|---|---| +| `observations` | `event_id` | `observed_at` | +| `observation_present_weather` | `event_id`, `weather_index` | `observed_at` | +| `forecasts` | `event_id` | `issued_at` | +| `forecast_periods` | `run_event_id`, `period_index` | `issued_at` | +| `forecast_discussions` | `event_id` | `issued_at` | +| `forecast_discussion_key_messages` | `run_event_id`, `message_index` | `issued_at` | +| `weather_story_runs` | `event_id` | `as_of` | +| `weather_stories` | `run_event_id`, `story_index` | `as_of` | +| `alert_runs` | `event_id` | `as_of` | +| `alerts` | `run_event_id`, `alert_index` | `as_of` | +| `alert_references` | `run_event_id`, `alert_index`, `reference_index` | `as_of` | + +## Table Contract + +### `observations` + +Primary key: `event_id` + +Prune column: `observed_at` + +Indexes: + +- `idx_wf_obs_station_observed_at` on `station_id`, `observed_at` +- `idx_wf_obs_observed_at` on `observed_at` +- `idx_wf_obs_condition_code` on `condition_code` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `event_id` | `TEXT` | no | `event.id` | +| `event_kind` | `TEXT` | no | `event.kind` | +| `event_source` | `TEXT` | no | `event.source` | +| `event_schema` | `TEXT` | no | `event.schema` | +| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` | +| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` | +| `station_id` | `TEXT` | yes | `payload.stationId` | +| `station_name` | `TEXT` | yes | `payload.stationName` | +| `observed_at` | `TIMESTAMPTZ` | no | `payload.timestamp` | +| `condition_code` | `INTEGER` | no | `payload.conditionCode` | +| `is_day` | `BOOLEAN` | yes | `payload.isDay` | +| `text_description` | `TEXT` | yes | `payload.textDescription` | +| `temperature_c` | `DOUBLE PRECISION` | yes | `payload.temperatureC` | +| `dewpoint_c` | `DOUBLE PRECISION` | yes | `payload.dewpointC` | +| `wind_direction_degrees` | `DOUBLE PRECISION` | yes | `payload.windDirectionDegrees` | +| `wind_speed_kmh` | `DOUBLE PRECISION` | yes | `payload.windSpeedKmh` | +| `wind_gust_kmh` | `DOUBLE PRECISION` | yes | `payload.windGustKmh` | +| `barometric_pressure_pa` | `DOUBLE PRECISION` | yes | `payload.barometricPressurePa` | +| `visibility_meters` | `DOUBLE PRECISION` | yes | `payload.visibilityMeters` | +| `relative_humidity_percent` | `DOUBLE PRECISION` | yes | `payload.relativeHumidityPercent` | +| `apparent_temperature_c` | `DOUBLE PRECISION` | yes | `payload.apparentTemperatureC` | + +### `observation_present_weather` + +Primary key: `event_id`, `weather_index` + +Prune column: `observed_at` + +Foreign key: `event_id` references `observations(event_id)` with cascade delete. + +Index: `idx_wf_obs_present_observed_at` on `observed_at` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `event_id` | `TEXT REFERENCES observations(event_id) ON DELETE CASCADE` | no | Parent event ID. | +| `weather_index` | `INTEGER` | no | `payload.presentWeather[]` index. | +| `observed_at` | `TIMESTAMPTZ` | no | `payload.timestamp` | +| `raw_text` | `TEXT` | yes | Compact JSON text from `payload.presentWeather[].raw` | + +### `forecasts` + +Primary key: `event_id` + +Prune column: `issued_at` + +Indexes: + +- `idx_wf_fc_location_product_issued_at` on `location_id`, `product`, `issued_at` +- `idx_wf_fc_issued_at` on `issued_at` +- `idx_wf_fc_product_issued_at` on `product`, `issued_at` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `event_id` | `TEXT` | no | `event.id` | +| `event_kind` | `TEXT` | no | `event.kind` | +| `event_source` | `TEXT` | no | `event.source` | +| `event_schema` | `TEXT` | no | `event.schema` | +| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` | +| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` | +| `location_id` | `TEXT` | yes | `payload.locationId` | +| `location_name` | `TEXT` | yes | `payload.locationName` | +| `issued_at` | `TIMESTAMPTZ` | no | `payload.issuedAt` | +| `updated_at` | `TIMESTAMPTZ` | yes | `payload.updatedAt` | +| `product` | `TEXT` | no | `payload.product` | +| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` | +| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` | +| `elevation_meters` | `DOUBLE PRECISION` | yes | `payload.elevationMeters` | +| `period_count` | `INTEGER` | no | `len(payload.periods)` | + +### `forecast_periods` + +Primary key: `run_event_id`, `period_index` + +Prune column: `issued_at` + +Foreign key: `run_event_id` references `forecasts(event_id)` with cascade delete. + +Indexes: + +- `idx_wf_fc_period_start_time` on `start_time` +- `idx_wf_fc_period_end_time` on `end_time` +- `idx_wf_fc_period_run_start` on `run_event_id`, `start_time` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `run_event_id` | `TEXT REFERENCES forecasts(event_id) ON DELETE CASCADE` | no | Parent event ID. | +| `period_index` | `INTEGER` | no | `payload.periods[]` index. | +| `issued_at` | `TIMESTAMPTZ` | no | Parent `payload.issuedAt` | +| `start_time` | `TIMESTAMPTZ` | no | `payload.periods[].startTime` | +| `end_time` | `TIMESTAMPTZ` | no | `payload.periods[].endTime` | +| `name` | `TEXT` | yes | `payload.periods[].name` | +| `is_day` | `BOOLEAN` | yes | `payload.periods[].isDay` | +| `condition_code` | `INTEGER` | yes | `payload.periods[].conditionCode` | +| `text_description` | `TEXT` | yes | `payload.periods[].textDescription` | +| `temperature_c` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureC` | +| `temperature_c_min` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureCMin` | +| `temperature_c_max` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureCMax` | +| `dewpoint_c` | `DOUBLE PRECISION` | yes | `payload.periods[].dewpointC` | +| `relative_humidity_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].relativeHumidityPercent` | +| `wind_direction_degrees` | `DOUBLE PRECISION` | yes | `payload.periods[].windDirectionDegrees` | +| `wind_speed_kmh` | `DOUBLE PRECISION` | yes | `payload.periods[].windSpeedKmh` | +| `wind_gust_kmh` | `DOUBLE PRECISION` | yes | `payload.periods[].windGustKmh` | +| `barometric_pressure_pa` | `DOUBLE PRECISION` | yes | `payload.periods[].barometricPressurePa` | +| `visibility_meters` | `DOUBLE PRECISION` | yes | `payload.periods[].visibilityMeters` | +| `apparent_temperature_c` | `DOUBLE PRECISION` | yes | `payload.periods[].apparentTemperatureC` | +| `cloud_cover_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].cloudCoverPercent` | +| `probability_of_precipitation_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].probabilityOfPrecipitationPercent` | +| `precipitation_amount_mm` | `DOUBLE PRECISION` | yes | `payload.periods[].precipitationAmountMm` | +| `snowfall_depth_mm` | `DOUBLE PRECISION` | yes | `payload.periods[].snowfallDepthMm` | +| `uv_index` | `DOUBLE PRECISION` | yes | `payload.periods[].uvIndex` | + +### `forecast_discussions` + +Primary key: `event_id` + +Prune column: `issued_at` + +Indexes: + +- `idx_wf_discussion_office_product_issued_at` on `office_id`, `product`, `issued_at` +- `idx_wf_discussion_issued_at` on `issued_at` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `event_id` | `TEXT` | no | `event.id` | +| `event_kind` | `TEXT` | no | `event.kind` | +| `event_source` | `TEXT` | no | `event.source` | +| `event_schema` | `TEXT` | no | `event.schema` | +| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` | +| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` | +| `office_id` | `TEXT` | yes | `payload.officeId` | +| `office_name` | `TEXT` | yes | `payload.officeName` | +| `issued_at` | `TIMESTAMPTZ` | no | `payload.issuedAt` | +| `updated_at` | `TIMESTAMPTZ` | yes | `payload.updatedAt` | +| `product` | `TEXT` | no | `payload.product` | +| `short_term_qualifier` | `TEXT` | yes | `payload.shortTerm.qualifier` | +| `short_term_issued_at` | `TIMESTAMPTZ` | yes | `payload.shortTerm.issuedAt` | +| `short_term_text` | `TEXT` | yes | `payload.shortTerm.text` | +| `long_term_qualifier` | `TEXT` | yes | `payload.longTerm.qualifier` | +| `long_term_issued_at` | `TIMESTAMPTZ` | yes | `payload.longTerm.issuedAt` | +| `long_term_text` | `TEXT` | yes | `payload.longTerm.text` | +| `key_message_count` | `INTEGER` | no | `len(payload.keyMessages)` | + +### `forecast_discussion_key_messages` + +Primary key: `run_event_id`, `message_index` + +Prune column: `issued_at` + +Foreign key: `run_event_id` references `forecast_discussions(event_id)` with +cascade delete. + +Index: `idx_wf_discussion_message_issued_at` on `issued_at` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `run_event_id` | `TEXT REFERENCES forecast_discussions(event_id) ON DELETE CASCADE` | no | Parent event ID. | +| `message_index` | `INTEGER` | no | `payload.keyMessages[]` index. | +| `issued_at` | `TIMESTAMPTZ` | no | Parent `payload.issuedAt` | +| `message_text` | `TEXT` | yes | `payload.keyMessages[]` value | + +### `weather_story_runs` + +Primary key: `event_id` + +Prune column: `as_of` + +Indexes: + +- `idx_wf_story_run_office_as_of` on `office_id`, `as_of` +- `idx_wf_story_run_as_of` on `as_of` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `event_id` | `TEXT` | no | `event.id` | +| `event_kind` | `TEXT` | no | `event.kind` | +| `event_source` | `TEXT` | no | `event.source` | +| `event_schema` | `TEXT` | no | `event.schema` | +| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` | +| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` | +| `office_id` | `TEXT` | yes | `payload.officeId` | +| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` | +| `story_count` | `INTEGER` | no | `len(payload.stories)` | + +### `weather_stories` + +Primary key: `run_event_id`, `story_index` + +Prune column: `as_of` + +Foreign key: `run_event_id` references `weather_story_runs(event_id)` with +cascade delete. + +Indexes: + +- `idx_wf_stories_start_time` on `start_time` +- `idx_wf_stories_end_time` on `end_time` +- `idx_wf_stories_updated_at` on `updated_at` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `run_event_id` | `TEXT REFERENCES weather_story_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. | +| `story_index` | `INTEGER` | no | `payload.stories[]` index. | +| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` | +| `office_id` | `TEXT` | yes | `payload.stories[].officeId` | +| `start_time` | `TIMESTAMPTZ` | no | `payload.stories[].startTime` | +| `end_time` | `TIMESTAMPTZ` | no | `payload.stories[].endTime` | +| `updated_at` | `TIMESTAMPTZ` | no | `payload.stories[].updatedAt` | +| `title` | `TEXT` | yes | `payload.stories[].title` | +| `description` | `TEXT` | yes | `payload.stories[].description` | +| `alt_text` | `TEXT` | yes | `payload.stories[].altText` | +| `priority` | `BOOLEAN` | no | `payload.stories[].priority` | +| `story_order` | `INTEGER` | no | `payload.stories[].order` | +| `download_url` | `TEXT` | yes | `payload.stories[].downloadUrl` | + +### `alert_runs` + +Primary key: `event_id` + +Prune column: `as_of` + +Indexes: + +- `idx_wf_alert_run_location_as_of` on `location_id`, `as_of` +- `idx_wf_alert_run_as_of` on `as_of` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `event_id` | `TEXT` | no | `event.id` | +| `event_kind` | `TEXT` | no | `event.kind` | +| `event_source` | `TEXT` | no | `event.source` | +| `event_schema` | `TEXT` | no | `event.schema` | +| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` | +| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` | +| `location_id` | `TEXT` | yes | `payload.locationId` | +| `location_name` | `TEXT` | yes | `payload.locationName` | +| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` | +| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` | +| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` | +| `alert_count` | `INTEGER` | no | `len(payload.alerts)` | + +### `alerts` + +Primary key: `run_event_id`, `alert_index` + +Prune column: `as_of` + +Foreign key: `run_event_id` references `alert_runs(event_id)` with cascade +delete. + +Indexes: + +- `idx_wf_alerts_alert_id` on `alert_id` +- `idx_wf_alerts_severity_expires` on `severity`, `expires` +- `idx_wf_alerts_as_of` on `as_of` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `run_event_id` | `TEXT REFERENCES alert_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. | +| `alert_index` | `INTEGER` | no | `payload.alerts[]` index. | +| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` | +| `alert_id` | `TEXT` | no | `payload.alerts[].id` | +| `event` | `TEXT` | yes | `payload.alerts[].event` | +| `headline` | `TEXT` | yes | `payload.alerts[].headline` | +| `severity` | `TEXT` | yes | `payload.alerts[].severity` | +| `urgency` | `TEXT` | yes | `payload.alerts[].urgency` | +| `certainty` | `TEXT` | yes | `payload.alerts[].certainty` | +| `status` | `TEXT` | yes | `payload.alerts[].status` | +| `message_type` | `TEXT` | yes | `payload.alerts[].messageType` | +| `category` | `TEXT` | yes | `payload.alerts[].category` | +| `response` | `TEXT` | yes | `payload.alerts[].response` | +| `description` | `TEXT` | yes | `payload.alerts[].description` | +| `instruction` | `TEXT` | yes | `payload.alerts[].instruction` | +| `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].sent` | +| `effective` | `TIMESTAMPTZ` | yes | `payload.alerts[].effective` | +| `onset` | `TIMESTAMPTZ` | yes | `payload.alerts[].onset` | +| `expires` | `TIMESTAMPTZ` | yes | `payload.alerts[].expires` | +| `area_description` | `TEXT` | yes | `payload.alerts[].areaDescription` | +| `sender_name` | `TEXT` | yes | `payload.alerts[].senderName` | +| `reference_count` | `INTEGER` | no | `len(payload.alerts[].references)` | + +### `alert_references` + +Primary key: `run_event_id`, `alert_index`, `reference_index` + +Prune column: `as_of` + +Foreign key: `run_event_id` references `alert_runs(event_id)` with cascade +delete. + +Indexes: + +- `idx_wf_alert_refs_as_of` on `as_of` +- `idx_wf_alert_refs_sent` on `sent` + +| Column | Type | Null | Source | +|---|---|:---:|---| +| `run_event_id` | `TEXT REFERENCES alert_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. | +| `alert_index` | `INTEGER` | no | Parent alert index. | +| `reference_index` | `INTEGER` | no | `payload.alerts[].references[]` index. | +| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` | +| `id` | `TEXT` | yes | `payload.alerts[].references[].id` | +| `identifier` | `TEXT` | yes | `payload.alerts[].references[].identifier` | +| `sender` | `TEXT` | yes | `payload.alerts[].references[].sender` | +| `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].references[].sent` | + +## Retention + +When sink param `prune` is set, every successful write transaction deletes rows +older than `now - prune` from every table using that table's prune column. + +The sink also exposes manual prune helpers in code, but the `weatherfeeder` +binary does not provide CLI commands for them. + +## Reconstructing Canonical Payloads + +- `WeatherObservation`: read `observations`, then join + `observation_present_weather` by `event_id` ordered by `weather_index`. +- `WeatherForecastRun`: read `forecasts`, then join `forecast_periods` by + `run_event_id` ordered by `period_index`. +- `WeatherForecastDiscussion`: read `forecast_discussions`, then join + `forecast_discussion_key_messages` by `run_event_id` ordered by + `message_index`. +- `WeatherStoryRun`: read `weather_story_runs`, then join `weather_stories` by + `run_event_id` ordered by `story_index`. +- `WeatherAlertRun`: read `alert_runs`, join `alerts` by `run_event_id` ordered + by `alert_index`, then join `alert_references` by `run_event_id` and + `alert_index` ordered by `reference_index`. diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 6ff9584..7209846 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -4,7 +4,9 @@ This document defines `weatherfeeder`'s development architecture and invariants for maintainers and LLM coding agents. It describes how the implemented system is built and how future changes should preserve its boundaries. -This is an inward-facing policy document. User-facing wire contracts belong in [`API.md`](../../API.md), and future work belongs under [`docs/roadmap/`](../roadmap/). +This is an inward-facing policy document. User-facing wire contracts belong in +[`docs/integrations/events.md`](../integrations/events.md), and future work +belongs under [`docs/roadmap/`](../roadmap/). ## Project Shape @@ -98,7 +100,10 @@ The daemon's own state is in-process: - source instances may keep HTTP conditional request state through feedkit HTTP source helpers; - scheduler and dispatcher state is not persisted by `weatherfeeder`. -Durable persistence is an external sink concern. The Postgres table contract is documented in `internal/sinks/postgres/doc.go`; the consumer-facing event contract is documented in [`API.md`](../../API.md). +Durable persistence is an external sink concern. The Postgres table contract is +documented in [`docs/integrations/postgres.md`](../integrations/postgres.md); +the consumer-facing event contract is documented in +[`docs/integrations/events.md`](../integrations/events.md). ## Configuration and CLI Boundaries @@ -112,7 +117,9 @@ Configuration shape is owned by feedkit's config package: Weatherfeeder-specific config policy belongs in source and sink constructors, registry setup, and tests. Do not spread config parsing through domain model or normalizer packages. -If dedicated `docs/config.md` or `docs/cli.md` files are added later, they should become the canonical user/operator references. This policy should stay architectural and avoid duplicating those references. +[`docs/config.md`](../config.md) and [`docs/cli.md`](../cli.md) are the +canonical user/operator references. This policy should stay architectural and +avoid duplicating those references. ## Errors, Logging, and Diagnostics